Binaries on PyPI only support Python 3.9

[BenjaminRodenberg]

I installed pythonfmu from PyPI via pip3 install --user python3fmu. However, when I tried to use the generated FMU OMEdit crashed. After some investigations I found out that the binaries provided on PyPI are only compatible with Python 3.9 (as far as I understand). According to the documentation pythonfmu should work with any python3 distribution, if I don't understand it wrong.

Why only python 3.9?

I downloaded pythonfmu-0.6.2-py3-none-any.whl from here, unpacked it and ran ldd on libpythonfmu-export.so:

/pythonfmu-0.6.2/pythonfmu/resources/binaries/linux64# ldd libpythonfmu-export.so 
	linux-vdso.so.1 (0x00007ffdb51e8000)
	libpython3.9.so.1.0 => not found
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0b76d8e000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0b76d73000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0b76b81000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0b76a32000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f0b771aa000)

Further details

I assume that this is due to the way how the binaries of pythonfmu are generated in this action.

I'm not sure whether this is something that one can change easily. Distributing binaries for all python platforms is probably out of scope.

I finally fixed this issue by cloning this repository, building the binaries myself via cmake & make in pythonfmu/pythonfmu-export and installing pythonfmu via setup.py.

[markaren]

It should work in Python 3.6 <= as the ABI is kept binary compatible. I for one, use Python 3.7.
However, I'm on Windows, so there might be a difference there.

[BenjaminRodenberg]

Thanks for the reply! On Ubuntu 20.04 I ran into trouble with the system python (3.8) when trying to use the FMU in OMEdit. I can try to reproduce the issue on my system and post the error here.

[BenjaminRodenberg]

I generate the example PythonFMU/examples/demoslave.py. Once with pythonfmu from PyPI, once with the version I have built using this repository.

In both cases I ran

1. ~/PythonFMU/examples$ pythonfmu build -f demoslave.py
2. Extracted Resistor.fmu
3. cd Resistor_{github,pypi}/binaries/linux64

I will describe the behavior below.

PyPI

• Importing in OMEdit gives many errors and I cannot "simulate".

[1] 13:33:47 Scripting Error
[/home/benjamin/.openmodelica/libraries/index.json: 0:0-0:0]: The package index /home/benjamin/.openmodelica/libraries/index.json could not be parsed.

[2] 13:33:47 Scripting Error
Failed to load package Modelica (default) using MODELICAPATH /usr/bin/../lib/omlibrary:/home/benjamin/.openmodelica/libraries/

[3] 13:33:47 Scripting Error
[/home/benjamin/.openmodelica/libraries/index.json: 0:0-0:0]: The package index /home/benjamin/.openmodelica/libraries/index.json could not be parsed.

[4] 13:33:47 Scripting Error
Failed to load package ModelicaReference (default) using MODELICAPATH /usr/bin/../lib/omlibrary:/home/benjamin/.openmodelica/libraries/.

[5] 13:34:47 Scripting Error
[fmiLogger] module FMICAPI: Could not load the FMU binary: libpython3.9.so.1.0: cannot open shared object file: No such file or directory

[6] 13:34:47 Scripting Error
[instantiate] Could not load "resources/0001_Resistor_pypi.fmu" which is associated with "test.Root.Resistor_pypi"; it may be corrupted or may not support your platform

[7] 13:34:47 Scripting Error
[initialize] Model "test" is in wrong model state

[8] 13:34:47 Scripting Error
[terminate] Model "test" is in wrong model state

• libpython3.9.so.1.0 is not found by ldd

~/PythonFMU/examples/Resistor_pypi/binaries/linux64$ ldd Resistor.so 
	linux-vdso.so.1 (0x00007fff7e796000)
	libpython3.9.so.1.0 => not found
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc210351000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc210336000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc210144000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc20fff3000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc2107b7000)

This Repo (github)

• Importing in OMEdit gives fewer errors, "simulate" works:

[1] 13:36:08 Scripting Error
[/home/benjamin/.openmodelica/libraries/index.json: 0:0-0:0]: The package index /home/benjamin/.openmodelica/libraries/index.json could not be parsed.

[2] 13:36:08 Scripting Error
Failed to load package Modelica (default) using MODELICAPATH /usr/bin/../lib/omlibrary:/home/benjamin/.openmodelica/libraries/.

[3] 13:36:08 Scripting Error
[/home/benjamin/.openmodelica/libraries/index.json: 0:0-0:0]: The package index /home/benjamin/.openmodelica/libraries/index.json could not be parsed.

[4] 13:36:08 Scripting Error
Failed to load package ModelicaReference (default) using MODELICAPATH /usr/bin/../lib/omlibrary:/home/benjamin/.openmodelica/libraries/.

[5] 13:36:35 Scripting Notification
Result file: test_res.mat (bufferSize=10)

• libpython3.8.so.1.0 is found by ldd

~/PythonFMU/examples/Resistor_github/binaries/linux64$ ldd Resistor.so 
	linux-vdso.so.1 (0x00007ffcd016e000)
	libpython3.8.so.1.0 => /lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f06e0cdc000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f06e0afb000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f06e0ae0000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f06e08ee000)
	libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f06e08be000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f06e08a2000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f06e087f000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f06e0879000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f06e0874000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f06e0725000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f06e1367000)

[markaren]

Yeah, I can replicate through WSL (Ubuntu 18.04).
I guess the same issue will appear when using e.g. 3.6 to build the binary.

[markaren]

This may be the reason why it work on different Python versions on Windows:

if (WIN32)
  set(Python3_LIBRARIES ${Python3_LIBRARY_DIRS}/python3.lib)
endif ()

I dunno if something similar is possible to do for Linux.

[BenjaminRodenberg]

I think libpython3.so has a similar purpose on UNIX systems (see here: https://stackoverflow.com/a/42388242). I unsuccessfully tried the following modification:

# Force to use stable Python ABI https://docs.python.org/3/c-api/stable.html
add_compile_definitions(Py_LIMITED_API)
find_package(Python3 REQUIRED COMPONENTS Interpreter Development)
if (WIN32)
  set(Python3_LIBRARIES ${Python3_LIBRARY_DIRS}/python3.lib)
else (UNIX)
  set(Python3_LIBRARIES ${Python3_LIBRARY_DIRS}/libpython3.so)
endif ()

I got stuck at the point that libpython3.so could not be found and I also don't know how to get it.

[markaren]

I ran the locate command on my Ubuntu installation and there is no libpython3.so..
So I'm kind of blank as to facilitate this for linux users. It is clearly an issue as the FMUs generated for that platform will have major portability issues.

[markaren]

Ok, so now I'm a little bit confused... You see, the CI compiles the binaries once. Presumably using 3.9..
Tests are then carried out on 3.7, 3.8 and 3.9 using the same binary. They all run and they all pass (Ubuntu and Windows)..

[BenjaminRodenberg]

Ok, so now I'm a little bit confused... You see, the CI compiles the binaries once. Presumably using 3.9..
Tests are then carried out on 3.7, 3.8 and 3.9 using the same binary. They all run and they all pass (Ubuntu and Windows)..

Do you know whether on the CI instance executing the tests libpython3.9.so.1.0 is present or not? I can imagine that if a test is carried out with (for example) python 3.7 while both 3.7 and 3.9 are installed on the system it might still work. However, if libpython3.9.so.1.0 is not present (which is what I would assume for the general case when somebody runs python 3.7) I assume the tests should fail.

I once also was in the situation that both python 3.8 and 3.9 were installed on my system and everything worked nicely with python 3.8, if I remember correctly.

[markaren]

Yeah, the virtual environment has:

2.7.18
3.5.10
3.6.13
3.7.10
3.8.10
3.9.5

[HenningScheufler]

I could also reproduce this issue on ubuntu 20.04 using a miniconda with a python version 3.8:

Following output is produced:

pyfmi.fmi.InvalidBinaryException: The FMU could not be loaded. Error loading the binary. Could not load the FMU binary: libpython3.9.so.1.0: cannot open shared object file: No such file or directory

After creating a new conda environment 3.9:

conda create -n py39 python=3.9
conda activate py39
pip install pythonfmu
conda create -n py39 python=3.9

The problem persist despite using the correct python version. However, if we add python 3.9 to our the library path this issue can be resolved

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:ANACONDA_PATH/envs/py39/lib/

Why is python 3.9 linked by default on ubuntu?

[markaren]

As far as I can tell, there is no way for PythonFMU to support more than one python distribution on Ubuntu as it links to a specific version of Python. Windows does not.. I guess Python 3.9 is the version used by default on GitHub Actions (Python 3.x) so that is the one that is linked. The only way to make it work with a different Python on Linux is to manually build the library and replace the .so.

[HenningScheufler]

The python interpreter gets found by:
find_package(Python3 REQUIRED COMPONENTS Interpreter Development)
However, this line (tested with cmake version 3.21.3) always find the newest interpreter which explains why it always links to 3.9.

After adding following line

set(Python3_FIND_STRATEGY LOCATION) # added requires cmake 3.15
find_package(Python3 REQUIRED COMPONENTS Interpreter Development)

the activated interpreter is used

ubuntu 1804 only comes with cmake 3.10 upgrading to ubuntu 2004 should change the cmake version to 3.16