SWIG and Python: Create a package compatible with any 3.x version

Question

SWIG generates two wrapping layers for Python module: the C wrapper code and Python wrapper code. And, as I understand, both are agnostic to the specific subversion of Python 3.x (3.1-3.6 currently).

However, when compiling the code, I have to include the headers of specific 3.x version of Python, which includes the specific library version in PyConfig.h, e.g. 3.4: pragma comment(lib,"python34.lib") - which results in the Python package that has to find and load python34.dll in the runtime, and the loading will fail with any other version.

But all 3.x versions are compatible and simple editing of the produced .pyd binary - replacing the python34.dll with say python36.dll makes the final module to work with Python 3.6 just fine.

Is there any way to create a Python 3.x package with SWIG that would be able to search and find any available 3.x version installed on the system?

Bodice

Given that the pragma comes from PyConfig.h I think this is more of a Windows/DLL/Python thing than a SWIG thing, but it's a good question. – Gav

@Flexo yes, I think you're right. However, I'm thinking that there might be some "tricky" workaround. This pragma can be removed. If the Python dll and functions could be loaded dynamically with say LoadLibrary/GetProcAddress, then it'd be easy to search for any available python3x.dll binary. But I don't immediately see a way to do this given the SWIG structure, which links to Python explicitly – Bodice

Hi! Have you learned anything about this? – Noel

You can define Py_LIMITED_API and then it will link to python3.lib rather than the specific version. Check the Python.h header – Digestible

Answer 1

You can 'force' link your project to python3.lib instead of linking to python3X.lib (where X is the minor version). The wrapper will then have python3.dll as part of its imports, and python3.dll will do forwarding to python3X.dll.

However, note that the APIs that you are using need to come from the common set of Python 3 APIs (I believe starting on 3.2, see reference below). If you ever need to use an API which does not exist on 3.2, but exists on 3.3 onward, your SWIG wrapper will still be linked to the minor library.

For example, if you are using PyUnicode_AsUTF8 or PyUnicode_AsUTF8AndSize which exist starting Python 3.3, and then you try to compile your wrapper for 3.6 and 3.7, although those APIs are common for both versions, they will be included as part of python36.dll and python37.dll imports.

Reference: PEP 384

Caskey

Answer 2

I was trying to build a Python 3 generic SWIG library as well, since users of my library may have different versions of Python 3 installed and at the moment I need to build a different library for each version of Python (3.7, 3.8, 3.9, ...), and this is getting hard to maintain and is confusing for customers.

I tried defining Py_LIMITED_API, however unfortunately it looks like SWIG makes use of functions that are in the full API, e.g. PyInstanceMethod_New() found in classobject.h.

I was able to hack the SWIG generated *_wrap.c file to remove references to PyInstanceMethod_New(), and the same compiled module now does seem to load in Python 3.7 and 3.11, but is only a very simple example that calls a C function and has no classes. This was very hacky though, and I'm worried that a more complex C++ library with classes will not work in the same way. If anyone has any more up to date ideas on how to create a generic Python 3 SWIG wrapped library it would be really appreciated.

Anterior

This does not really answer the question. If you have a different question, you can ask it by clicking Ask Question. To get notified when this question gets new answers, you can follow this question. Once you have enough reputation, you can also add a bounty to draw more attention to this question. - From Review – Broadway