package only cythonized binary python files and resource data but ignoring python .py source files

Question

I need to obfuscate my python code, to achieve it I am using cythonize extension, I am able to achieve it and get the binary compiled a.so files from a.py files but after doing bdist_wheel the .whl package only packages the a.so files and ignores the resource_folder.

My project file structure is

|main_project
|__,setup.py
|__,main_folder
|____,a.py
|____,__init__.py
|____resource_folder
     |__,a.model
     |__,a.json

I used following links to make obfuscated python wheel package,

https://bucharjan.cz/blog/using-cython-to-protect-a-python-codebase.html

https://medium.com/@xpl/protecting-python-sources-using-cython-dcd940bb188e

Package only binary compiled .so files of a python library compiled with Cython

Following is the snippet from my setup.py

packages = find_packages(exclude=('tests',))

def get_package_files_in_directory(directory):
    paths = []
    for (path, directories, filenames) in os.walk(directory):
        for filename in filenames:
            paths.append(os.path.join('..', path, filename))
    return paths


setup(
    packages=[],

    ext_modules=cythonize(
        [
           Extension("main_folder.*", ["main_folder/*.py"])

        ],
        build_dir="build",
        compiler_directives=dict(
        always_allow_keywords=True
        )),
package_data={p: package_files + get_package_files_in_directory(os.path.join(here, p, 'resources')) for p in packages},
,....
,...
)

To package I am using following command

python setup.py build_ext

python setup.py bdist_wheel

---

expected result is .whl file containing a.so file and resource folder

actual result is .whl file contain only a.so file.

---

to also package the resource_folder I used get_package_files_in_directory() function as suggested in this link "( How do you add additional files to a wheel?)" but this also didnot work for me

Redneck

You are defining package data for package main_folder, but not including main_folder package in the dist. – Marcellusmarcelo

Yes I did that to exclude a.py from getting packaged in wheel as suggested in this answer #39499953 – Redneck

I see. However, the suggested (and accepted) answer you linked is wrong as excluding packages when you are actually packaging them will break a lot of stuff, including package data (no packages, no package data). The correct approach should be including the packages, but excluding all the modules underneath. I will post an answer once found the correct approach for that. – Marcellusmarcelo

I have added an answer here. If you encounter any problems, update your question and I will add a specific answer for it. – Marcellusmarcelo

Answer 1

Based on this Answer by @hoefling I was able to package my resource_folder and obfuscated binary a.so file.

The recipe for setup.py

from Cython.Distutils import build_ext
from Cython.Build import cythonize
from setuptools.extension import Extension
from setuptools.command.build_py import build_py as build_py_orig
from pathlib import Path
from setuptools import find_packages, setup, Command
import os
import shutil

here = os.path.abspath(os.path.dirname(__file__))    
packages = find_packages(exclude=('tests',))

def get_package_files_in_directory(directory):
    paths = []
    for (path, directories, filenames) in os.walk(directory):
        for filename in filenames:
            paths.append(os.path.join('..', path, filename))
    return paths
#to copy the __init__.py as specified in above references links

class MyBuildExt(build_ext):
    def run(self):
        build_ext.run(self)

        build_dir = Path(self.build_lib)
        root_dir = Path(__file__).parent

        target_dir = build_dir if not self.inplace else root_dir

        self.copy_file(Path('main_folder') / '__init__.py', root_dir, target_dir)


    def copy_file(self, path, source_dir, destination_dir):
        if not (source_dir / path).exists():
            return

        shutil.copyfile(str(source_dir / path), str(destination_dir / path))

#as specified by @hoefling to ignore .py and not resource_folder
class build_py(build_py_orig):
    def build_packages(self):
        pass

setup(
    packages=find_packages(),  # needed for obfuscation
    ext_modules=cythonize(
        [
           Extension("main_folder.*", ["main_folder/*.py"])

        ],
        build_dir="build",
        compiler_directives=dict(
        always_allow_keywords=True
        )),
    package_data={p: get_package_files_in_directory(os.path.join(here, p, 'resource_folder')) for p in packages}, #package_data as found in another reference
    cmdclass={
        'build_py': build_py
    },
    entry_points={
    },
)

---

To create obfuscated *.whl package set of commands

python setup.py build_ext  #creates the a.so
python setup.py build_py   #copies the resource_folder excluding .py
python setup.py bdist_wheel # then whl generation

Redneck