How can I make setuptools (or distribute) install a package from the local file system

Question

Is it possible to specify (editable) source dependencies in setup.py that are known to reside on the local file system?

Consider the following directory structure, all of which lives in a single VCS repository:

projects
  utils
    setup.py
    ...
  app1
     setup.py
     ... # app1 files depend on ../utils
  app2
     setup.py
     ... # app2 files depend on ../utils

Given the following commands:

cd projects
mkvirtualenv app1
pip install -e app1

I'd like to have all the dependencies for app1 installed, including "utils", which is an "editable" dependency. Likewise, if I did the same for app2.

I've tried playing with all different combinations of file://... URLs in install_requires and dependency_links to no avail. I'd like to use a dependency link URL like src+file://../utils, which would tell setuptools that the source for the package is on the file system at this relative path. Is there a way to do this?

Maddening

This might be of some help: packages.python.org/distribute/… – Viens

Thanks, Rob. I reviewed that page extensively before asking the question here. The page lists two categories of URLs: (1) direct download URLs, and (2) URLs of web pages that contain direct download links. I was hoping my issue would fall into the first category, but I can't figure out how to construct the URL. – Maddening

I submitted a patch to pip to support relative file: URLs in dependency_links, which solves my issue. Hopefully it gets merged soon. – Maddening

Just ran into this question, and the link Rob Wouters posted is broken. Here's one that works as of this comment: pythonhosted.org/setuptools/… – Dimitris

Why don't you create an account at #github and host your package/fork there. This would allow you refer to a specific commit of that repository as a dependency. Also, do you just want to get it to work on your localhost? Or do you want a robust solution that works for everyone interested in the package? – Hobart

Package metadata is not the right place for this kind of information. For this there's requirements.txt where you can use local paths as shown in the Danver Braganza's answer. – Lukash

That link from Rob appears to be broken again @Wiscocrew. Here is the latest (I think): setuptools.pypa.io/en/latest/deprecated/dependency_links.html – Bottoms

I'm afraid I don't recall what exactly I linked 8 years ago, but that looks right @Bottoms – Dimitris

Answer 1

i managed to provide relative local dependency in setup.py with:

setup(
    install_requires=[
        'utils @ file://localhost/%s/../utils/' % os.getcwd().replace('\\', '/'),
        ],
    )

but maybe someone know better solution

Hardee

Took me forever to find this. So far this is the only working solution I've seen. Very useful for working with editable installed local packages. – Peneus

Same solution but simplified with f-strings for python 3.x. This also assumes the package you're referencing is above one level in your file path hence the "..": python f"my_package @ file://localhost/{os.getcwd()}/../my_package/" – Peneus

Useful, but only applies if installing via develop option, AKA -e flag. If you're installing the package via pip install ., you immediately lose your starting CWD. – Floccose

@Floccose os.getcwd works for me when I use pip install . (no -e flag) – Advantage

Answer 2

I had an identical problem where I needed to depend on modules in a sibling folder. I was able to find a solution after stumbling upon https://caremad.io/2013/07/setup-vs-requirement/

I ended up requirements.txt to refer specifically to the file I wanted, and then installing everything with

pip install -r requirements.txt

---

requirements.txt

-e ../utils                                                                                                                                                                    
-e .

And setup.py has all my other dependencies, including utils. When pip tries to install app1 itself, it realizes that the utils dependency has already been filled, and so passes over it, while installing the other requirements.

Glaze

the relative path in the requirements.txt work in case you entered the directory and executed the command...executing same command in the parent directory would not work. There are issues with relative path specification for pip requirements.txt file github.com/pypa/pip/issues/3772 – Plusch

Answer 3

When I want to work with a set of projects interrelated, I install all of them using /setup.py develop.

If I mistakenly or later I want to make a pip-installed module editable, I clone the source, and do a python setup.py develop on it too, substituting the existing one.

Just to get sure, I erase the reference in the virtualenv's site-packages and the package itself.

Phatic

Answer 4

The current answers don't explain how to use setuptools to install editable, local dependencies.

The below will install local dependencies as editable for use with setuptools.

setup.py

from pip import main as pip_main
from pip._internal.network.session import PipSession
from pip._internal.models.link import Link
from pip._internal.req import parse_requirements
from pip._internal.req.req_file import ParsedRequirement
from pip._internal.req.constructors import parse_req_from_line
import os
from setuptools import find_packages, setup


def pip_to_setup(requirement: ParsedRequirement, editable=True) -> str:
  '''
  Convert a pip requirement to a setup.py install_requires
    If `editable` then pre-install via `pip install -e ...` and return ''
    Else return local path requirement
  '''

  requirement_parts = parse_req_from_line(
    requirement.requirement,
    requirement.line_source,
  )

  if (
    isinstance(requirement_parts.link, Link)
    and requirement_parts.link.is_existing_dir()
  ):
    package_path = requirement_parts.link.file_path
    if editable:
      pip_main(['install', '-e', package_path])
      return ''  # https://medium.com/@ashley.e.shultz/python-mono-repo-with-only-built-in-tooling-7c2d52c2fc66
    else:
      package_name = os.path.basename(package_path)
      return f'{package_name} @ file://{package_path}'
  else:
    return requirement.requirement


def main():
  dir_path = os.path.dirname(__file__)
  package_name = os.path.basename(dir_path)

  requirements_path = os.path.join(dir_path, 'requirements.txt')
  test_requirements_path = os.path.join(dir_path, 'requirements-test.txt')

  requirements = parse_requirements(requirements_path, session=PipSession())
  install_requires = [pip_to_setup(r) for r in requirements]

  test_requires = []
  if os.path.isfile(test_requirements_path):
    test_requirements = parse_requirements(test_requirements_path, session=PipSession())
    test_requires.extend(pip_to_setup(r) for r in test_requirements)

  setup(
    name=package_name,
    setup_requires=['setuptools_scm'],
    install_requires=install_requires,
    tests_require=test_requires,
    extras_require={'test': test_requires},
    packages=find_packages(where='src'),
    package_dir={'': 'src'},
    package_data={'': ['**/*']},
  )


if __name__ == '__main__':
  main()

(You may have to modify packages and package_dir depending on your project structure.

Example requirements.txt

boto3
../your_local_package

Haakon