setup.py: renaming src package to project name

Question

Let's say you have a project called proj and in this project you have the following structure:

proj/
  dists/
  doc/
  src/
    __init__.py
    xyz.py
    abc.py
  test/
  setup.py

As you can see all the content of your project is in the src subfolder. How to go about making a distutils distribution package out of the src folder?

My naive idea, following the tutorial, would've been to write the setup.py like this:

#omitting basics
setup(
   name='proj',
   packages=['src'],
   package_dir={'proj':'src'}
)

But after installing the resulting package to my system, I still have to import src.xyz and not proj.xyz, which would've been the goal and the expected result.

Toxin

Answer 1

You could fix it by putting Python package files into proj/ directory:

proj/
  src/
    proj/
      __init__.py
      xyz.py
      abc.py
  setup.py

And changing setup.py to:

# ...
setup(
   name='proj',
   packages=['proj'],
   package_dir={'':'src'}
)

It is not required by distutils but other tools might expect the parent directory name of __init__.py file to be the same as Python package name i.e., proj in this case.

Chinatown

I did that and it works. Not sure if I understand your last comment, though. You mean there are tools that expect me to have a folder structure like proj/src/proj to work as expected? Seems so inefficient. Or am I doing something inefficient here? – Toxin

I meant that the directory name: <name>/__init__.py should be the same as Python package name i.e., if I see __init__.py then just by looking at its parent directory name I know what Python package it is. – Chinatown

Answer 2

This is due to a bug in setuptools reported here: https://github.com/pypa/setuptools/issues/250

Basically, it does work but not in dev mode. Now on you have 3 solutions:

• symlink the src package as proj (and ignore it when comitting), it will works out of the box but is dirty
• change from src to proj
• create a subdirectory proj in src and use the following options:

packages=['proj'],
package_dir={'proj': 'src/proj'},

Ivy

Link to setuptools issue: github.com/pypa/setuptools/issues/250 – Arching

Issue's now closed for 3.5+ – Whitleywhitlock

Answer 3

the correct settings is:

#omitting basics
setup(
   name='proj',
   packages=['proj'],
   package_dir={'proj':'src'}
)

the src folder should contains __init__.py (if file is empty, everthing is exported by default)

in another project: requirements.txt:

../relativePathToProject or name of package:version

Metsky

*edit: well maybe it does actually work... I might have judged too quickly. I need to check it first. – Toxin

For me, with python 3.8 and setuptools 58.0.4, this still installs the package as src rather than proj. – Ovida

The example was tested with python 3.7. I used venv: pip install -r requirements.txt with the relative path to the dependency project – Metsky

The solution works also if src contains subpackages, but in my case it just cloned the src folder. – Pointer

I am running Python 3.10 with setuptools 67.8.0 but am running into the same issues as @Ovida when installing in editable mode (pip install -e .). Any recommendations on how to navigate that? – Spore

this doesn't seem to work in python3.10, the pip package is called proj but in python I still need to do import src – Week

Answer 4

You can first use find_packages to find all the package names in src, then rename them manually to the desired name. Finally, use the package_dir option to specify the package convention.

import re

from setuptools import find_packages, setup

PACKAGE_NAME = 'proj'
SOURCE_DIRECTORY = 'src'
SOURCE_PACKAGE_REGEX = re.compile(rf'^{SOURCE_DIRECTORY}')

source_packages = find_packages(include=[SOURCE_DIRECTORY, f'{SOURCE_DIRECTORY}.*'])
proj_packages = [SOURCE_PACKAGE_REGEX.sub(PACKAGE_NAME, name) for name in source_packages]

setup(
    name=PACKAGE_NAME,
    packages=proj_packages,
    package_dir={PACKAGE_NAME: SOURCE_DIRECTORY},
    ...
)

Enrika

I still need to import src with this method in python3.10 – Week

Answer 5

Building on Jfs' Answer, if like me you had a directory structure already established that you didn't want to/couldn't change for other reasons, one solution is to temporary copy all of the code to be packaged and then build that.

Here's a makefile with targets which copies all the files across in src to a temp location, and then calls setup.py to build the package, before finally cleaning up after itself.

SRC_DIR='src'
TEMP_PACKAGE_DIR='your_shiny_package'

package: prepare_package_dir build_package deploy_package remove_package_dir

# I'm using gemfury, your deployment will probably look different
deploy_package:
    twine upload --repository fury dist/* --verbose

clean_package:
    rm -r dist || echo 'dist removed'
    rm -r ${TEMP_PACKAGE_DIR}.egg-info || echo 'egg-info removed'

build_package: clean_package
    python setup.py sdist

prepare_package_dir:
    mkdir ${TEMP_PACKAGE_DIR}
    cp -R ${SRC_DIR}/* ${TEMP_PACKAGE_DIR}/
    mv ${TEMP_PACKAGE_DIR} ${SRC_DIR}/${TEMP_PACKAGE_DIR}

remove_package_dir:
    rm -rf ${SRC_DIR}/${TEMP_PACKAGE_DIR}

and then a setup.py which looks a bit like this:

setup(
    name='your_shiny_package',
    version=version,
    description='some great package...',
    long_description=readme,
    url='https://whereever',
    author='you',
    packages=['src/your_shiny_package'],
    install_requires=parse_all_requirements(),
    zip_safe=False,
    include_package_data=True)

Might not be the most efficient, but saves you having to restructure your whole repo permanently.

Usage: just call make package either as part of your build pipeline or manually.

Array

Answer 6

The src dir is commonly used to keep all the py files in your project. Based on this some developers prefer to rename src to the repository name. That is:

proj/
  dists/
  doc/
  proj/
    __init__.py
    xyz.py
    abc.py
  test/
  setup.py

It means that when you import any module or subpackage, you can always import from proj as from proj import ....

If the repository name is not that good to be a package name, e.g. neetcode-challenges-level-hard, you can just use another name to call your src folder to make it seem to be part of the job:

neetcode-challenges-level-hard/
  dists/
  doc/

  neetcode/
    __init__.py
    optimization.py
    tree.py

  test/
  setup.py

To import any module you'd use from neetcode import ....

Osmund

Please note that "scripts" and ".py files in a module" are not the same thing at all in Python. The confusion between these two makes this answer less valuable. It needs some clarification. – Ovida

What is the difference? – Osmund

It's complicated: #16982421 – Ovida

Answer 7

You can try adding the src folder to the PYTHONPATH before you call the setup function:

import sys, os
src_path = os.path.join(os.path.realpath(os.path.dirname(__file__)), 'src')
sys.path.append(src_path)

And also, just to be on the safe side, you then change the working directory:

os.chdir(src_path)

After that, it should all be OK.

Some other tools for packaging your app support that from within. I thought it was setuptools, turns out it's PyInstaller. But basically, that's what should be done, just enough for your packages to be imported directly.

Turns out distutils has the package_dir directive. That is what you should use, but it might work by only adding your package to the PYTHONPATH.

Asyut

I'm not sure you got the problem yet. I'm trying to write a python package that other people can install by themself easily. It's not just about importing stuff you wrote in another folder. – Toxin

Yes I understand, but the setup.py has to import stuff in another folder in order to work. Once these packages are in your PYTHONPATH, it doesn't matter where they are. If it did not work for you, you'll have to see another way, sorry. – Asyut

Sorry, but that's not the problem. – Toxin