About python : Can-a-statically-linked-python-interpretter-load-python-modules-that-contain-native-dynamic-libraries

I’ve statically linked a python interpretter (libpython.a) into a C++ program.

When I try to use it to import a python module that contains a native dynamic library component (a .so), I get an error that the core python library symbols are unavailable.

For example, if I try to:

import subprocess

I get:

ImportError: _posixsubprocess.so: undefined symbol: PyTuple_Type

Likewise, if I try:

import ctypes

I get:

ImportError: _ctypes.so: undefined symbol: PyFloat_Type

These two functions PyTuple_Type and PyFloat_Type are part of the core python library (libpython.a), so should be statically linked into the program.

Yet, when these dynamic libraries (_posixsubprocess.so or _ctypes.so) are loaded, they can’t seem to find them.

Any ideas?

In general, when you load a dynamic library can its needed undefined external symbols be resolved by symbols that are statically defined/linked in the main program? I’m not sure I understand how that works (or if it’s supposed to work).

The relevant parts of the test program are:

#include <Python.h>
int main() {
    Py_Initialize();
    PyRun_SimpleString("import subprocess")
}

I’ve figured out part of it:

By default the symbols in a program are not exported and made available to dynamic libraries that are loaded.

In order to export the symbols you can pass the --export-dynamic option to the linker. Also you may need --whole-archive so that the linker doesn’t link out unused functions:

$ gcc -Wl,--export-dynamic myprogram.cc -Wl,--whole-archive libpython.a -Wl,--no-whole-archive

See: https://sourceware.org/binutils/docs-2.34/ld/Options.html#Options

Doing so fixes the problem, and the python modules now work, as the dynamic libraries can find the python API symbols from the main program.

I’m not sure if it’s possible to just export the symbols from one particular library.

Also I’m not sure yet if Windows and Mac will have similar problems, or the equivalent solution on those platforms.