Should pandoc be bundled with manuskript's packages?

[olivierkes]

I'd like to continue in a separated thread the discussion we started here: #186 (comment)

@olivierkes: From a user standpoint, I guess they want out of the box the ability to export. So why not avoid them the supplementary step of downloading and installing a second software?

@gedakc: That's a good question. There are a few reasons I can think of why we should not bundle pandoc with the manuskript PyInstaller package:

1. Manuskript should not crash regardless of whether pandoc is installed or not. We probably have a bug in our code.
2. We do not wish to take on added packaging responsibility. If a security bug is found in pandoc and we included pandoc in our package, then we should update our package every time a pandoc bug is fixed.
3. Pandoc may be used with other programs outside of manuskript. Multiple copies of pandoc should not be required for each and every program that uses pandoc.
4. The user should have the freedom to install (or not install) a specific version of pandoc if they so wish.
5. When we eventually have OS specific install packages these should not include a copy of pandoc. For example in Debian and Ubuntu, packages can require or suggest other packages, but never actually include a separate copy of the package.

1. That's right, we need to fix that.

2. We already do. In pyInstaller packages, we ship: python, Qt, PyQt, lxml, pyenchant, pdf.js, and few other libraries.

3. Same as 2.

4. Same as 2. :)

5. I know that in GNU/Linux distributions with proper package system that can handle dependency, this is the case. So when manuksript will ship as .deb package or whatever, it won't include any of the libs mentions in 2.

   But how does it work in Windows or Mac OS? If we can do that, great ! But if it add works for the user, not optimal.

And I add:

6. I just had a twitter exchange, and a guy told me manuskript should export to ePUB. I thought he was running 0.3.0 (since ePUB export was added with pandoc on 0.4.0). Turns out, he was running 0.5.0.

   If we dont ship pandoc by default (but I still fail to see the reason), we should at least say something explicitly about installing pandoc, directly in the list of export formats, under manuskript's internally supported formats.

7. Other advantage I see of statically linking libraries (is that the right terms?) is that we can control the version of the software we ship, so we can be sure of compatibility. We've just seen a bug with the -v / --version. If the trade-of is a few Mb more, that's a relatively small price to pay.

So my opinion is: as long as we package with pyInstaller, we should ship pandoc as well. As soon as we find a better option, with better system integration, we might rethink this. Let me hear your thoughts :)

[gedakc]

Hi @olivierkes. You provide some good counter points. My responses follow:

@gedakc 2. We do not wish to take on added packaging responsibility.

@olivierkes We already do. In pyInstaller packages, we ship: python, Qt, PyQt, lxml, pyenchant, pdf.js, and few other libraries.

My point here would be these are libraries important to a programmer and not an end user. Generally these must be called by a software application to provide value to the end-user.

Pandoc; however; is directly useful to an end-user from the command line. In fact a user might wish to install pandoc only, and not use manuskript.

@olivierkes 6. If we dont ship pandoc by default (but I still fail to see the reason), we should at least say something explicitly about installing pandoc, directly in the list of export formats, under manuskript's internally supported formats.

I think a lot of the confusion around manuskript, how it works, and extra programs to install is because we do not have good documentation. I've been working to improve this situation, but it is a huge undertaking. I agree that we should at least say something explicitly about installing pandoc to get access to more export formats.

Perhaps we might start a manuskript manual on the wiki so that all users can contribute?

More Thoughts

I have been impressed with your efforts to create and share manuskript. I think that you have approached creating manuskript in the best way possible. You created a working program and then shared this program with the world under a Free Software license. These two key things make it possible for all of us to work together to improve the software.

Conclusion

I am open to bundling pandoc with manuskript. My caution on trying to bundle pandoc is that the more things we include the greater the chance that a package will fail to run on different OS versions.

For example I discovered that the manuskript-0.4.0-linux.zip package you created ran on all distros I tested. Your package did not include PyEnchant. This makes it a much easier package for 64 bit Linux users to install and run.

The manusrkipt-0.4.0-1-linux64.zip package I built included PyEnchant and consequently did not run unmodified on all distros I tested. Instead the user has to either use a different package or remove files on some distros in order to permit manuskript to run properly. This is an extra level of complexity I would prefer to avoid if it was reasonably possible. For now the extra complexity does provide access to the spell checker.

In conclusion you've convinced me to at least try bundling pandoc with the manuskript PyInstaller package. This leads to the logical next question:

How does one include pandoc in the PyInstaller package?

[gedakc]

To help inform our users about extra export formats available with pandoc I added the following section to our download pages:

--- Begin section ---

Extra Export Formats

Install pandoc for more export document formats such as ePub, OpenDocument (LibreOffice), and DocX (MS Word).

--- End section ---

[olivierkes]

Great, thanks!!!

[gedakc]

Some more discussion:

@gedakc 2. We do not wish to take on added packaging responsibility.

@olivierkes We already do. In pyInstaller packages, we ship: python, Qt, PyQt, lxml, pyenchant, pdf.js, and few other libraries.

@gedakc My point here would be these are libraries important to a programmer and not an end user. Generally these must be called by a software application to provide value to the end-user.

There is a further difference regarding these python libraries (e.g., PyQt, pyenchant), and separate programs (e.g., pandoc). The python libraries must be present when the PyInstaller package is created. If not then the libraries are not seen if added to Linux later.

For example with the manuskript-0.4.0-linuz.zip package you created it seems to have not been built with pyenchant. This can be seen in the Tools menu which contains the entry Install PyEnchant to use spellcheck. This is a great tip for people running manuskript from source code.

However for users of the PyInstaller package, if they install the Python pyenchant library then it is still not seen by manuskript in the PyInstaller package. I have tested this myself to confirm this behaviour. I believe this is also the reason behind the following issue 122 - Installed PyEnchant but Manuskript still asks me to "Install PyEnchant to use Spellcheck".

@olivierkes 6. If we dont ship pandoc by default (but I still fail to see the reason), we should at least say something explicitly about installing pandoc, directly in the list of export formats, under manuskript's internally supported formats.

I agree that it would be helpful to add this tip to the manuskript code to suggest installing pandoc for more export formats. This would be useful regardless of whether we bundle pandoc with the PyInstaller package. It would be useful to users running manuskript from source and from PyInstaller package.

[olivierkes]

Thanks, this makes sense.

So in summary up to that point, we should:

• Add a tip if pandoc is not installed, similar to PyEnchant
• Work on more documentation
• Check out how to bundle pandoc with PyInstaller to see if that creates more complications or not

Do you know why users had to remove files in some distro on packages including pyenchant? And how did you detect that libstdc++.so.6 was the cause of the trouble? And is that file necessary? Can't we just ship the package for everyone without libstdc++.so.6?

However for users of the PyInstaller package, if they install the Python pyenchant library then it is still not seen by manuskript in the PyInstaller package.

I wonder if there is a way to tell a PyInstaller package to search the system's python modules anyway. Because otherwise, even if pyenchant is there, I guess the user cannot add dictionaries?

[gedakc]

Thanks for summarizing. That helps us keep focused on the next steps. :-)

Following is some background research, responses to questions, our dilemma, and a question.

Background Research

With the PyInstaller packages I'm trying to ensure that manuskript runs on as many versions of currently supported GNU/Linux distros as possible. Unfortunately we might be caught at a time when the oldest distros are not recent enough to support PyQt5 AND be bug free.

Following are the release dates of the oldest currently supported distros

Ascending    |              |         |
Release Date | Distribution | Version | Problem
-------------+--------------+---------+---------------------------------
2013-05-04   | Debian       | 7       | No PyQt5 package, distro too old
2014-04-17   | Ubuntu       | 14.04   | Plugin "xcb" missing issue [1]
2015-04-25   | Debian       | 8       | Best compromise [2]
2016-11-16   | openSUSE     | 42.2    |
2016-11-22   | Fedora       | 25      |
...          | ...          | ...     | Later distros all newer.  Need
             |              |         | oldest for forward compatibility.

[1]: Appears to be a problem with Qt 5.2.1.

@olivierkes you encountered this when you tried to package Manuskript 0.2.0 using Kubuntu 14.04 with Qt 5.2.1. See Qt platform plugin "xcb" missing.

This issue crops up when the PyInstaller package is built on Ubuntu 14.04. When running the package on another distro, like Debian 8 or Fedora 25, the following message is displayed:

This application failed to start because it could not find or load the Qt platform plugin "xcb".

Reinstalling the application may fix this problem.
Fatal Python error: Aborted

Current thread 0x00007faa37030700 (most recent call first):
  File "manuskript/main.py", line 19 in run
  File "manuskript", line 13 in <module>
Aborted

As best I can determine this was a bug in Qt 5.2.1 that was fixed in a later version.

[2]: Best compromise

Using Debian 8 to build the manuskript PyInstaller package seems to be the best compromise. It is one of the oldest currently supported distros, and can be made to work on other distros by including or removing manuskript/libstdc++.so.6.

Responses to Questions

Following are responses to questions:

Do you know why users had to remove files in some distro on packages including pyenchant?

The short answer is I don't know exactly why.

The problem seems to arise when PyEnchant is included in the PyInstaller package.

Pyenchant is listed as being supported with PyInstaller on Windows only. See PyInstaller - Supported Packages and search for pyenchant. The results on Linux may be unpredictable.

For the distros I tested I discovered that the work around was to remove the libstdc++.so.6 file.

As to the exact reason why pyenchant doesn't work on all distros, I don't know. There is a slim chance this might be related to the issues that @katafrakt encountered. See issue 188 Manuskript does not start with PyEnchant on MacOS.

And how did you detect that libstdc++.so.6 was the cause of the trouble?

I came across the work around of removing libstdc++.so.6 by searching the Internet to see how other projects resolved the issue.

And is that file necessary?

When the PyInstaller package is built on Debian 8, it requires libstdc++.so.6 to work on Ubuntu 14.04.

Can't we just ship the package for everyone without libstdc++.so.6?

From my testing libstdc++.so.6 can be removed on all the other distros and the PyInstaller package will work correctly. The only one I know for sure that won't work is Ubuntu 14.04.

I wonder if there is a way to tell a PyInstaller package to search the system's python modules anyway.

It might be possible, though I don't know how to do accomplish this.

Because otherwise, even if pyenchant is there, I guess the user cannot add dictionaries?

Actually users can install additional dictionaries and these will be seen by manuskript in the PyInstaller package (assuming PyEnhant was included at build time). I suspect that this is because the dictionaries are global in nature and used by more than one application or library. At least that has been my experience on GNU/Linux.

Dilemma

I believe we have a dilemma regarding whether or not to include libstdc++.so.6 in the manuskript PyInstaller package.

• If we include libstdc++.so.6 then it will run on all distros after Debian 8 (some having to remove the file), and also Ubuntu 14.04.

• If we don't include libstdc++.so.6 then it will run on all distros after Debian 8 (no modifications needed), but NOT on Ubuntu 14.04.

Neither choice is a perfect solution.

Question

Do you think it would be better to have two packages, one that works only on ubuntu 14.04, and another that works on all of the other currently supported distros?

[olivierkes]

Thanks for that detailed and documented answer.
I think the easiest solution would be indeed to have two packages. It will be less work than trying to understand and fix all those issues, especially since we hope to let go of PyInstaller once packages exists for those distros.

[olivierkes]

• Adds a tip if pandoc is not installed, similar to PyEnchant

[gedakc]

Good work adding the pandoc tip. 👍

Are you planning a 0.5.1 release soon?

If so then I will wait for the 0.5.1 release before creating two PyInstaller packages: one for ubuntu 14.04, and another for all the other distros.

[olivierkes]

I don't know. If we follow the RERO principle then yes we should.
Especially if we can fix something with Travis to have automatic builds :)

What do you think, when should we release 0.5.1?

[gedakc]

Regarding the RERO principle I think a balance is needed. Too many releases can overly burden users and developers alike. Too few releases and users and contributors might lose interest because there appears to be no progress.

If we can get manuskript fixed so that the Mac OS X build can be created with Travis then that would be of huge benefit to Mac users. I think we should give @katafrakt some time to develop a fix for Mac OS X.

After that I think it would be helpful to our users to create a 0.5.1 release that includes the updated language translations, the Mac OS X packaging fix, and two separate Linux PyInstaller packages.

[olivierkes]

Sounds good to me !

[gedakc]

@olivierkes Especially if we can fix something with Travis to have automatic builds :)

From the Travis CI - Build Environment Overview it appears that only Ubuntu 12.04 Precise and Ubuntu 14.04 Trusty are available as GNU/Linux offerings.

Because Ubuntu 12.04 is too told to have PyQt5, and because we encountered issues creating packages on Ubuntu 14.04, it doesn't look good for us to use Travis CI to automatically build Linux PyInstaller packages. :(