how to write setup.py to install python extention (xxx.so file) built by SWIG?

Question

Using SWIG to build python extensions (xxx.so) is easier than distutils. You don't need to write a wrap.c program to wrap your original c program. So I'd like using SWIG than Distutils. But There is no methods to install extensions(xxx.so) automatically by writing a setup.py file. This troubled me for many days.

To explain the question more intelligible, I use the Step strategy to make it clear.

STEP 1： There is a original C/C++ code. And then use SWIG to create a Python wrapper.

STEP 2: Use Distuils (write a setup.py) or just gcc command line to create Python extension file. The name of the Python extension is xxx.so. Obviously, in this step, you can combine the compiling and installation together by using Distuils.

STEP 3: install the xxx.so Python extension by writing a setup.py.

I just don't know the STEP 3 that how to write a setup.py to install xxx.so Python extensions.

Is there any way to install python extensions (xxx.so) by writing setup.py and not using sys.path.append() or sys.path.insert() methods?

Thank you very much for any answer provided by you !

In the end, I tried the STEP 2 method that combine compilation and installation together. It can work correctly, but it's not my purpose that only install _xxx.so Python extension using setup.py.

directory structure:

setup.py
src/
    libdemo/
        __init__.py
        LibHello.py
lib/
    palindrome.c   # original C code
    palindrome_wrap.c  # Python wrapper built by SWIG

setup.py code:

#-*- coding:utf-8 -*-
import sys
import os
from setuptools import setup, find_packages, Extension

libPath = "lib"

moduleLibPalindrome = Extension(
      name = 'libdemo._palindrome', 
      sources = [os.path.join(libPath, 'palindrome_wrap.c'), 
            os.path.join(libPath, 'palindrome.c')]
)
setup (
      name = 'test_example',
      version = '0.1',
      author = "Hobart",
      description = "Install extension from original C/C++",
      packages= find_packages(where = 'src'),
      package_dir = {'':'src'},
      ext_modules=[moduleLibPalindrome]
)

result check :

>>> from libdemo import _palindrome
>>> dir(_palindrome)
['SWIG_PyInstanceMethod_New', '__builtins__', '__doc__', '__file__',      '__name__', '__package__', 'is_palindrome']
>>> _palindrome.is_palindrome('asddsa')
1
>>> 

The C code source file and _palindrome.so file are shown as following. you can download and do experiment on your computer.

Sackey

You should import palindrome, not _palindrome. – Origin

Answer 1

Distutils is not about writing extensions, but about compiling and installing them. SWIG can help you creating the Python wrappers for your C/C++ code, but does not handle compiling and installing them.

So, distutils can (and should) be used to install the SWIG generated extensions. See the SWIG docs on using distutils.

EDIT: I am a little confused about what you are trying to do. Normally, you would use distutils for compiling and installing your extension. This can still be done in two separate steps by calling

python setup.py build_ext
python setup.py install

However, if I understand you correctly, for whatever reason, you have a precompiled extension (i.e., xxx.py and _xxx.so), and wish to merely install these using e.g. distutils.

You can use the package_data keyword for distutils, i.e., the following setup.py:

from distutils.core import setup
setup (name = 'xxx',
       version = '0.1',
       author = "Nobody",
       description = """Install precompiled extension""",
       py_modules = ["xxx"],
       packages=[''],
       package_data={'': ['_xxx.so']},
       )

Origin

Thanks! You clearly explained the relationship and the difference between SWIG and very easy to understand. But I just want install Python extensions (xxx.so) which already have been built by myself. As you said, SWIG help me easily creating the Python wrappers. But the question is not how to using swig to creating a Python wrapper for a specified C/C++ code. – Sackey

STEP 1： There is a original C/C++ code. And then use SWIG to create a Python wrapper. STEP 2: Use Distuils or just gcc command line to create Python extension file. The name is xxx.so. STEP 3: install the xxx.so Python extension by writing a setup.py. I just don't know the STEP 3 that how to write a setup.py to install xxx.so Python extensions. – Sackey

It doesn't work ! I don't know how to make it. In the end, I tried the STEP 2 method that combine compilation and installation together. you can see the question description in detail. Again, Thank you very much for your helpful answer! – Sackey

Actually , package_data is same as MANIFEST.in that is used to create a source distribution. Maybe I don't know how to use package_data keyword to install Python extension(_xxx.so). If you can, It's better to provide a practical setup.py example. Thanks a lot! – Sackey

The current example in your question looks perfectly sensible. If you still want to only install a precompiled extension, you should specify where your xxx.py and _xxx.so are in your file structure. Yes, package_data and MANIFEST.in are somewhat equivalent, so you can also use MANIFEST.in. – Origin

@m7thonYes , I want to only install a precompiled extension. I don't know how to install _xxx.so using package_data keywords or MANIFEST.in. I only know how to create a source distribution including some specific files. But it's unable to install it. – Sackey

if _xxx.so file and setup.py in the same directory, there is no need to do any thing and setup.py automatically install the xxx.so。But that's temporary. that's to say, when you close the current process, and you will not be able to use this module again. This way is same as sys.path.append() method. – Sackey

You need to say in what directory _xxx.so and xxx.py are. To install, execute python setup.py install. – Origin

you can use “init.py” to tell the Python installation mechanism where directory contain the xxx.py module that need to install.But how to identify where the _xxx.so are? Do you have any simple example to explain it? Thanks! – Sackey

I think we misunderstand each other. I thought you already have files _xxx.so and xxx.py. So, where are they? – Origin

You know what's my question and show me code to solve it. But it didn't work. I don't know why it can't work. So I ask you again. _xxx.so file is given as URL in my question description. You can download to check it and explore it how to only install _xxx.so Python extension. Sorry for bother you for many times. If you know how to install Python extension(_xxx.so), which is already built but isn't installed, code may be a appropriate way to show your answer. Thanks！ – Sackey

The sample code above works if _xxx.so and xxx.py are in the same directory as the setup.py file (I actually tested it). If the files are in some other directory, you need to adapt the package_data entry accordingly. If you don't say where the files are in your case, but only that "it doesn't work", then I cannot help you. Sorry. – Origin