Include *.pyd files in Python Packages

Question

I have a python module module.pyd that works pretty fine once it is put manually onto the site-packages of python installation folder.

The problem starts when I upload my solution to a cloud enviroment, the buildpack requests that I pass every module as a package to be installed with pip install module. I ve created a folder with a simple __init__.py file that just imports everything of the module.pyd so that my module is treated like a folder.

Then I read here http://peterdowns.com/posts/first-time-with-pypi.html how to upload my own module and I succeeded, but when I install my module, the module.pyd file is not copied. I also tried to install it direct by the repository pip install git+repository but the same thing happened.

I have read here https://docs.python.org/2/distutils/sourcedist.html#specifying-the-files-to-distribute that I might have to explicitly say I want to copy *.pyd files in a MANIFEST.in file, I have done it, but it seems not working yet.

I currently using python 2.7.10

I am new on python so I d appreciate you guys help

Woodcut

Not much of this makes sense, unfortunately. Why do you not want to distribute the actual .py files? Why do you need to put it in site-packages, rather than in your project itself, if it's your own code? – Dustheap

@daniel-roseman Actually I dont have the py files of this module (it is a C++ compilation I think). I just want to place it as a distribution or at least in a repository so the buildpack of the cloud enviroment can access it. But once it install the module, the 'init.py' file is dowloaded, but not the pyd file. I already tried to upload it together with the solution, but I dont know why, it doesn't work – Woodcut

Answer 1

Just use the MANIFEST.in:

recursive-include module *.pyd

This will include all pyd files in the module directory.

Your package layout should be the following:

module/
--- __init__.py
--- _module.pyd
--- module.py
MANIFEST.in
README.rst
setup.py

And don't forget to add include_package_data=True in setup() in your setup.py in order to force using MANIFEST.in when building wheels and win32 installers (else MANIFEST.in will only be used for source tarball/zip).

Minimal example of setup():

README_rst = ''
with open('README.rst', mode='r', encoding='utf-8') as fd:
    README_rst = fd.read()

setup(
    name='module',
    version='0.0.1',
    description='Cool short description',
    author='Author',
    author_email='[email protected]',
    url='repo.com',
    packages=['module'],
    long_description=README_rst,
    include_package_data=True,
    classifiers=[
        # Trove classifiers
        # The full list is here: https://pypi.python.org/pypi?%3Aaction=list_classifiers
        'Development Status :: 3 - Alpha',
    ]
)

Townie

Can you elaborate on why you're using module.py and _module.pyd (why add the .py file and rename the .pyd file with an underscore)? – Mostly

@Mostly ah sorry it might indeed be confusing, _module.pyd is here a SWIG-generated Python module interface to a C++ program, so there is no relation between _module.pyd and module.py, it would probably be less confusing to rename the pyd file to _core.pyd. – Townie

Then why keep the underscore? You want to make the *.pyd file public, right? – Mostly

We wanted to provide a more pythonic interface with docstrings and default named keywords, so the py file basically just redefined functions and classes of the pyd :-) Also the py file managed cross platform compatibility by loading the appropriate pyd, so in my concrete case i had multiple pyd files generated for various platforms. – Townie

@Townie ... and you shipped every .pyd file to every platform? Why not just wheel the right .pyd to the right platform? – Windswept

@TylerGubala For simplicity in the package definition, but yes indeed once it works like this, it's possible to make specific definitions for each platform. I'm not yet much used to Python 3, so it might be easier to do so now. – Townie

This was the only solution I could find that actually worked – Muoimuon