I have a requirements.txt file that I'm using with Travis-CI. It seems silly to duplicate the requirements in both requirements.txt and setup.py, so I was hoping to pass a file handle to the install_requires kwarg in setuptools.setup.
Is this possible? If so, how should I go about doing it?
Here is my requirements.txt file:
guessit>=0.5.2
tvdb_api>=1.8.2
hachoir-metadata>=1.3.3
hachoir-core>=1.3.3
hachoir-parser>=1.3.4
You can flip it around and list the dependencies in setup.py and have a single character — a dot . — in requirements.txt instead.
Alternatively, even if not advised, it is still possible to parse the requirements.txt file (if it doesn't refer any external requirements by URL) with the following hack (tested with pip 9.0.1):
install_reqs = parse_requirements('requirements.txt', session='hack')
This doesn't filter environment markers though.
In old versions of pip, more specifically older than 6.0, there is a public API that can be used to achieve this. A requirement file can contain comments (#) and can include some other files (--requirement or -r). Thus, if you really want to parse a requirements.txt you can use the pip parser:
from pip.req import parse_requirements
# parse_requirements() returns generator of pip.req.InstallRequirement objects
install_reqs = parse_requirements(<requirements_path>)
# reqs is a list of requirement
# e.g. ['django==1.5.1', 'mezzanine==1.4.6']
reqs = [str(ir.req) for ir in install_reqs]
setup(
...
install_requires=reqs
)
setup(..., dependency_links=[str(req_line.url) for req_line in parse_requirements(<requirements_path>)], ...) –
Hoarding parse_requirements that is being removed, or just the process-dependecy-links option? I did this and it required me to run pip install -e . --process-dependency-links and it kindly let me know I was doing things wrong and that process-dependency-links is deprecated –
Longlongan parse_requirements function is being moved to a different location in pip 1.6. Additionally --process-dependency-links will be completely gone in pip 1.6. Installing things from github should be done using a requirements file. See caremad.io/blog/setup-vs-requirement –
Photomap pip defaults to parsing dependencies from setup.py in the absence of requirements.txt, the simple answer astutely noted by Tobu below is to list all dependencies in setup.py and remove requirements.txt. For applications requiring both, simply reduce the dependency list in requirements.txt to merely the . character. Done. –
Ebner session not expected to be a boolean. –
Dael session=False because session must be an instance of pip.download.PipSession. Maybe a boolean was possible some times ago. I edited my answer anyway because the . requirements.txt is neat, even if it's the opposite ;-) –
Corrigible pip 10. –
Mertiemerton . character does in fact work... but it is slow. When all requirements are already satisfied, pip install -r requirements.txt goes from taking 8 seconds with an explicit list to taking 54 seconds with .... 7x slower. –
Poisson On the face of it, it does seem that requirements.txt and setup.py are silly duplicates, but it's important to understand that while the form is similar, the intended function is very different.
The goal of a package author, when specifying dependencies, is to say "wherever you install this package, these are the other packages you need, in order for this package to work."
In contrast, the deployment author (which may be the same person at a different time) has a different job, in that they say "here's the list of packages that we've gathered together and tested and that I now need to install".
The package author writes for a wide variety of scenarios, because they're putting their work out there to be used in ways they may not know about, and have no way of knowing what packages will be installed alongside their package. In order to be a good neighbor and avoid dependency version conflicts with other packages, they need to specify as wide a range of dependency versions as can possibly work. This is what install_requires in setup.py does.
The deployment author writes for a very different, very specific goal: a single instance of an installed application or service, installed on a particular computer. In order to precisely control a deployment, and be sure that the right packages are tested and deployed, the deployment author must specify the exact version and source-location of every package to be installed, including dependencies and dependency's dependencies. With this spec, a deployment can be repeatably applied to several machines, or tested on a test machine, and the deployment author can be confident that the same packages are deployed every time. This is what a requirements.txt does.
So you can see that, while they both look like a big list of packages and versions, these two things have very different jobs. And it's definitely easy to mix this up and get it wrong! But the right way to think about this is that requirements.txt is an "answer" to the "question" posed by the requirements in all the various setup.py package files. Rather than write it by hand, it's often generated by telling pip to look at all the setup.py files in a set of desired packages, find a set of packages that it thinks fits all the requirements, and then, after they're installed, "freeze" that list of packages into a text file (this is where the pip freeze name comes from).
So the takeaway:
setup.py should declare the loosest possible dependency versions that are still workable. Its job is to say what a particular package can work with.requirements.txt is a deployment manifest that defines an entire installation job, and shouldn't be thought of as tied to any one package. Its job is to declare an exhaustive list of all the necessary packages to make a deployment work.requirements.txt along with the source of the package that contains the concrete/frozen requirements for installation or test. Surely setup.py can be used for this purpose within the project itself? I can only imagine using such a file for tools used to support managing the project (e.g. refactoring, making releases etc.). –
Widgeon requirements.txt is more documentation for the state of the world that produced a given build, even though it is not usually used in the build process itself? That makes sense. However, it looks like several systems rely on duplication: Travis installs some default (old) packages in your virtualenv and says to use requirements.txt. If I ask how to ensure dependencies are at latest using setup.py, people insist that I should use requirements.txt. –
Widgeon setup.py should use >= and requirements.txt should perhaps use ==. –
Poisson You can flip it around and list the dependencies in setup.py and have a single character — a dot . — in requirements.txt instead.
Alternatively, even if not advised, it is still possible to parse the requirements.txt file (if it doesn't refer any external requirements by URL) with the following hack (tested with pip 9.0.1):
install_reqs = parse_requirements('requirements.txt', session='hack')
This doesn't filter environment markers though.
In old versions of pip, more specifically older than 6.0, there is a public API that can be used to achieve this. A requirement file can contain comments (#) and can include some other files (--requirement or -r). Thus, if you really want to parse a requirements.txt you can use the pip parser:
from pip.req import parse_requirements
# parse_requirements() returns generator of pip.req.InstallRequirement objects
install_reqs = parse_requirements(<requirements_path>)
# reqs is a list of requirement
# e.g. ['django==1.5.1', 'mezzanine==1.4.6']
reqs = [str(ir.req) for ir in install_reqs]
setup(
...
install_requires=reqs
)
setup(..., dependency_links=[str(req_line.url) for req_line in parse_requirements(<requirements_path>)], ...) –
Hoarding parse_requirements that is being removed, or just the process-dependecy-links option? I did this and it required me to run pip install -e . --process-dependency-links and it kindly let me know I was doing things wrong and that process-dependency-links is deprecated –
Longlongan parse_requirements function is being moved to a different location in pip 1.6. Additionally --process-dependency-links will be completely gone in pip 1.6. Installing things from github should be done using a requirements file. See caremad.io/blog/setup-vs-requirement –
Photomap pip defaults to parsing dependencies from setup.py in the absence of requirements.txt, the simple answer astutely noted by Tobu below is to list all dependencies in setup.py and remove requirements.txt. For applications requiring both, simply reduce the dependency list in requirements.txt to merely the . character. Done. –
Ebner session not expected to be a boolean. –
Dael session=False because session must be an instance of pip.download.PipSession. Maybe a boolean was possible some times ago. I edited my answer anyway because the . requirements.txt is neat, even if it's the opposite ;-) –
Corrigible pip 10. –
Mertiemerton . character does in fact work... but it is slow. When all requirements are already satisfied, pip install -r requirements.txt goes from taking 8 seconds with an explicit list to taking 54 seconds with .... 7x slower. –
Poisson It can't take a file handle. The install_requires argument can only be a string or a list of strings.
You can, of course, read your file in the setup script and pass it as a list of strings to install_requires.
import os
from setuptools import setup
with open('requirements.txt') as f:
required = f.read().splitlines()
setup(...
install_requires=required,
...)
install_requires. However, it does not work if you don't use declarative syntax. –
Axolotl setup.py is a program that should be run, not a data file that should be parsed. That doesn't make this answer any worse. –
Anschauung include requirements.txt into your MANIFEST.in or you won't be able to install your library from a source distribution. –
Dall requirements.txt files by default. –
Mertiemerton pip defaults to installing all dependencies listed by setup.py in the absence of requirements.txt, the sane solution is to list all dependencies in setup.py and remove requirements.txt entirely. See also famousgarkin and Tobu's explanatory answers. –
Ebner Requirements files use an expanded pip format, which is only useful if you need to complement your setup.py with stronger constraints, for example specifying the exact urls some of the dependencies must come from, or the output of pip freeze to freeze the entire package set to known-working versions. If you don't need the extra constraints, use only a setup.py. If you feel like you really need to ship a requirements.txt anyway, you can make it a single line:
.
It will be valid and refer exactly to the contents of the setup.py that is in the same directory.
pip install -r requirements.txt ) without installing the package itself? –
Retro -e . should be enough. Check this page: caremad.io/posts/2013/07/setup-vs-requirement –
Blameworthy While not an exact answer to the question, I recommend Donald Stufft's blog post at https://caremad.io/2013/07/setup-vs-requirement/ for a good take on this problem. I've been using it to great success.
In short, requirements.txt is not a setup.py alternative, but a deployment complement. Keep an appropriate abstraction of package dependencies in setup.py. Set requirements.txt or more of 'em to fetch specific versions of package dependencies for development, testing, or production.
E.g. with packages included in the repo under deps/:
# fetch specific dependencies
--no-index
--find-links deps/
# install package
# NOTE: -e . for editable mode
.
pip executes package's setup.py and installs the specific versions of dependencies declared in install_requires. There's no duplicity and the purpose of both artifacts is preserved.
pip install my-package. If dependencies for my-package are not listed in my-package/setup.py, they are not installed by pip install my-package. I've been unable to determine how to provide a package for others that includes dependencies without explicitly stating them in setup.py. Would love to know if someone has figured out how to keep it DRY while allowing others to install my-package + dependencies without downloading the requirements file and manually calling pip install -r my-package/requirements.txt. –
Hunchback requirements.txt. That's the whole point. Updated the question to make things more clear. Also updated obsolete blog post link. –
Crushing First, I believe parsing requirements.txt to fill the list of dependencies in package metadata is not a good idea. The requirements.txt file and the list of "install dependencies" are two different concepts, they are not interchangeable. It should be the other way around, the list of dependencies in package metadata should be considered as some kind of source of truth, and files such as requirements.txt should be generated from there. For example with a tool such as pip-compile. See the notes at the bottom of this answer.
But everyone has different needs, that lead to different workflows. So with that said... There are 3 possibilities to handle this, depending on where you want your project's package metadata to be written: pyproject.toml, setup.cfg, or setup.py.
Words of caution!
If you insist on having the list of dependencies in package metadata be read from a requirements.txt file then make sure that this requirements.txt file is included in the "source distribution" (sdist) otherwise installation will fail, for obvious reasons.
These techniques will work only for simple requirements.txt files. See Requirements parsing in the documentation page for pkg_resources to get details about what is handled. In short, each line should be a valid PEP 508 requirement. Notations that are really specific to pip are not supported and it will cause a failure.
pyproject.toml
[project]
# ...
dynamic = ["dependencies"]
[tool.setuptools.dynamic]
# ...
dependencies = requirements.txt
setup.cfg
Since setuptools version 62.6 it is possible to write something like this in setup.cfg:
[options]
install_requires = file: requirements.txt
setup.py
It is possible to parse a relatively simple requirements.txt file from a setuptools setup.py script without pip. The setuptools project already contains necessary tools in its top level package pkg_resources.
It could more or less look like this:
#!/usr/bin/env python
import pathlib
import pkg_resources
import setuptools
with pathlib.Path('requirements.txt').open() as requirements_txt:
install_requires = [
str(requirement)
for requirement
in pkg_resources.parse_requirements(requirements_txt)
]
setuptools.setup(
install_requires=install_requires,
)
Notes:
pip's parsing and not pkg_resources's since before 2015 are bugs such as github.com/pypa/setuptools/issues/470 . This exact one is fixed nowadays, but I'm still a bit scared to use it, since both implementations appear to be developed separately. –
Valiancy parse_requirements will fail with a syntax error if the requirements.txt file contains options such as -r. –
Dedifferentiation Using parse_requirements is problematic because the pip API isn't publicly documented and supported. In pip 1.6, that function is actually moving, so existing uses of it are likely to break.
A more reliable way to eliminate duplication between setup.py and requirements.txt is to specific your dependencies in setup.py and then put -e . into your requirements.txt file. Some information from one of the pip developers about why that's a better way to go is available here: https://caremad.io/blog/setup-vs-requirement/
Most of the other answers above don't work with the current version of pip's API. Here is the correct* way to do it with the current version of pip (6.0.8 at the time of writing, also worked in 7.1.2. You can check your version with pip -V).
from pip.req import parse_requirements
from pip.download import PipSession
install_reqs = parse_requirements(<requirements_path>, session=PipSession())
reqs = [str(ir.req) for ir in install_reqs]
setup(
...
install_requires=reqs
....
)
* Correct, in that it is the way to use parse_requirements with the current pip. It still probably isn't the best way to do it, since, as posters above said, pip doesn't really maintain an API.
Install the current package in Travis. This avoids the use of a requirements.txt file.
For example:
language: python
python:
- "2.7"
- "2.6"
install:
- pip install -q -e .
script:
- python runtests.py
pip freeze and export that file somewhere as an artifact (like S3 or something), then you'd have a great way to repeatably install exactly what you tested. –
Medan from pip.req import parse_requirements did not work for me and I think it's for the blank lines in my requirements.txt, but this function does work
def parse_requirements(requirements):
with open(requirements) as f:
return [l.strip('\n') for l in f if l.strip('\n') and not l.startswith('#')]
reqs = parse_requirements(<requirements_path>)
setup(
...
install_requires=reqs,
...
)
The following interface became deprecated in pip 10:
from pip.req import parse_requirements
from pip.download import PipSession
So I switched it just to simple text parsing:
with open('requirements.txt', 'r') as f:
install_reqs = [
s for s in [
line.split('#', 1)[0].strip(' \t\n') for line in f
] if s != ''
]
BEWARE OF parse_requirements BEHAVIOUR!
Please note that pip.req.parse_requirements will change underscores to dashes. This was enraging me for a few days before I discovered it. Example demonstrating:
from pip.req import parse_requirements # tested with v.1.4.1
reqs = '''
example_with_underscores
example-with-dashes
'''
with open('requirements.txt', 'w') as f:
f.write(reqs)
req_deps = parse_requirements('requirements.txt')
result = [str(ir.req) for ir in req_deps if ir.req is not None]
print result
produces
['example-with-underscores', 'example-with-dashes']
[ir.req.unsafe_name for ir in req_deps if ir.req is not None] –
Guthrun If you don't want to force your users to install pip, you can emulate its behavior with this:
import sys
from os import path as p
try:
from setuptools import setup, find_packages
except ImportError:
from distutils.core import setup, find_packages
def read(filename, parent=None):
parent = (parent or __file__)
try:
with open(p.join(p.dirname(parent), filename)) as f:
return f.read()
except IOError:
return ''
def parse_requirements(filename, parent=None):
parent = (parent or __file__)
filepath = p.join(p.dirname(parent), filename)
content = read(filename, parent)
for line_number, line in enumerate(content.splitlines(), 1):
candidate = line.strip()
if candidate.startswith('-r'):
for item in parse_requirements(candidate[2:].strip(), filepath):
yield item
else:
yield candidate
setup(
...
install_requires=list(parse_requirements('requirements.txt'))
)
distutils has been deprecated and is scheduled for removal in Python 3.12. –
Sacci I created a reusable function for this. It actually parses an entire directory of requirements files and sets them to extras_require.
Latest always available here: https://gist.github.com/akatrevorjay/293c26fefa24a7b812f5
import glob
import itertools
import os
# This is getting ridiculous
try:
from pip._internal.req import parse_requirements
from pip._internal.network.session import PipSession
except ImportError:
try:
from pip._internal.req import parse_requirements
from pip._internal.download import PipSession
except ImportError:
from pip.req import parse_requirements
from pip.download import PipSession
def setup_requirements(
patterns=[
'requirements.txt', 'requirements/*.txt', 'requirements/*.pip'
],
combine=True):
"""
Parse a glob of requirements and return a dictionary of setup() options.
Create a dictionary that holds your options to setup() and update it using this.
Pass that as kwargs into setup(), viola
Any files that are not a standard option name (ie install, tests, setup) are added to extras_require with their
basename minus ext. An extra key is added to extras_require: 'all', that contains all distinct reqs combined.
Keep in mind all literally contains `all` packages in your extras.
This means if you have conflicting packages across your extras, then you're going to have a bad time.
(don't use all in these cases.)
If you're running this for a Docker build, set `combine=True`.
This will set `install_requires` to all distinct reqs combined.
Example:
>>> import setuptools
>>> _conf = dict(
... name='mainline',
... version='0.0.1',
... description='Mainline',
... author='Trevor Joynson <[email protected],io>',
... url='https://trevor.joynson.io',
... namespace_packages=['mainline'],
... packages=setuptools.find_packages(),
... zip_safe=False,
... include_package_data=True,
... )
>>> _conf.update(setup_requirements())
>>> # setuptools.setup(**_conf)
:param str pattern: Glob pattern to find requirements files
:param bool combine: Set True to set install_requires to extras_require['all']
:return dict: Dictionary of parsed setup() options
"""
session = PipSession()
# Handle setuptools insanity
key_map = {
'requirements': 'install_requires',
'install': 'install_requires',
'tests': 'tests_require',
'setup': 'setup_requires',
}
ret = {v: set() for v in key_map.values()}
extras = ret['extras_require'] = {}
all_reqs = set()
files = [glob.glob(pat) for pat in patterns]
files = itertools.chain(*files)
for full_fn in files:
# Parse
reqs = {
str(r.req)
for r in parse_requirements(full_fn, session=session)
# Must match env marker, eg:
# yarl ; python_version >= '3.0'
if r.match_markers()
}
all_reqs.update(reqs)
# Add in the right section
fn = os.path.basename(full_fn)
barefn, _ = os.path.splitext(fn)
key = key_map.get(barefn)
if key:
ret[key].update(reqs)
extras[key] = reqs
extras[barefn] = reqs
if 'all' not in extras:
extras['all'] = list(all_reqs)
if combine:
extras['install'] = ret['install_requires']
ret['install_requires'] = list(all_reqs)
def _listify(dikt):
ret = {}
for k, v in dikt.items():
if isinstance(v, set):
v = list(v)
elif isinstance(v, dict):
v = _listify(v)
ret[k] = v
return ret
ret = _listify(ret)
return ret
__all__ = ['setup_requirements']
if __name__ == '__main__':
reqs = setup_requirements()
print(reqs)
pip._internal.. If you don't provide a usable external API, then you shouldn't break all those that are using all that you provide. –
Valiancy Another possible solution...
def gather_requirements(top_path=None):
"""Captures requirements from repo.
Expected file format is: requirements[-_]<optional-extras>.txt
For example:
pip install -e .[foo]
Would require:
requirements-foo.txt
or
requirements_foo.txt
"""
from pip.download import PipSession
from pip.req import parse_requirements
import re
session = PipSession()
top_path = top_path or os.path.realpath(os.getcwd())
extras = {}
for filepath in tree(top_path):
filename = os.path.basename(filepath)
basename, ext = os.path.splitext(filename)
if ext == '.txt' and basename.startswith('requirements'):
if filename == 'requirements.txt':
extra_name = 'requirements'
else:
_, extra_name = re.split(r'[-_]', basename, 1)
if extra_name:
reqs = [str(ir.req) for ir in parse_requirements(filepath, session=session)]
extras.setdefault(extra_name, []).extend(reqs)
all_reqs = set()
for key, values in extras.items():
all_reqs.update(values)
extras['all'] = list(all_reqs)
return extras
and then to use...
reqs = gather_requirements()
install_reqs = reqs.pop('requirements', [])
test_reqs = reqs.pop('test', [])
...
setup(
...
'install_requires': install_reqs,
'test_requires': test_reqs,
'extras_require': reqs,
...
)
tree come from? –
Berardo A workaround that works for me
WARNING: Only use when you are not using setup.cfg already !!! This will overwrite setup.cfg
Keep requirements.txt and requirements-dev.txt at same level as setup.py
Remove these lines from setup.py
install_requires=...
extras_require=...
Add this to setup.py to create setup.cfg and load requirements from files - requirements.txt and requirements-dev.txt
# Read the dependencies from requirements.txt
try:
with open("requirements.txt", "r", encoding="utf8") as f:
requirements = f.read().splitlines()
requirements = [
f"\t{req.strip()}\n" for req in requirements if req.strip()[0] != "#"
]
except FileNotFoundError:
requirements = []
finally:
if requirements:
requirements = ["[options]\n", "install_requires =\n"] + requirements
with open("setup.cfg", "w", encoding="utf8") as cfg:
cfg.writelines(requirements)
# Read the extra dependencies from requirements_extra.txt
try:
with open("requirements-dev.txt", "r", encoding="utf8") as f:
dev_requirements = f.read().splitlines()
dev_requirements = [
f"\t{req.strip()}\n" for req in dev_requirements if req.strip()[0] != "#"
]
except FileNotFoundError:
dev_requirements = []
finally:
if dev_requirements:
dev_requirements = [
"\n[options.extras_require]\n",
"dev =\n",
] + dev_requirements
with open("setup.cfg", "a", encoding="utf8") as cfg:
cfg.writelines(dev_requirements)
setup.cfg should look like this after you run python setup.py sdist
[options]
install_requires =
package_1
...
[options.extras_require]
dev =
package_2
...
Cross posting my answer from this SO question for another simple, pip version proof solution.
try: # for pip >= 10
from pip._internal.req import parse_requirements
from pip._internal.download import PipSession
except ImportError: # for pip <= 9.0.3
from pip.req import parse_requirements
from pip.download import PipSession
requirements = parse_requirements(os.path.join(os.path.dirname(__file__), 'requirements.txt'), session=PipSession())
if __name__ == '__main__':
setup(
...
install_requires=[str(requirement.req) for requirement in requirements],
...
)
Then just throw in all your requirements under requirements.txt under project root directory.
Yet another parse_requirements hack that also parses environment markers into extras_require:
from collections import defaultdict
from pip.req import parse_requirements
requirements = []
extras = defaultdict(list)
for r in parse_requirements('requirements.txt', session='hack'):
if r.markers:
extras[':' + str(r.markers)].append(str(r.req))
else:
requirements.append(str(r.req))
setup(
...,
install_requires=requirements,
extras_require=extras
)
It should support both sdist and binary dists.
As stated by others, parse_requirements has several shortcomings, so this is not what you should do on public projects, but it may suffice for internal/personal projects.
parse_requirements() anymore, so this now fails. –
Ut I did this:
import re
def requirements(filename):
with open(filename) as f:
ll = f.read().splitlines()
d = {}
for l in ll:
k, v = re.split(r'==|>=', l)
d[k] = v
return d
def packageInfo():
try:
from pip._internal.operations import freeze
except ImportError:
from pip.operations import freeze
d = {}
for kv in freeze.freeze():
k, v = re.split(r'==|>=', kv)
d[k] = v
return d
req = getpackver('requirements.txt')
pkginfo = packageInfo()
for k, v in req.items():
print(f'{k:<16}: {v:<6} -> {pkginfo[k]}')
Here is a complete hack (tested with pip 9.0.1) based on Romain's answer that parses requirements.txt and filters it according to current environment markers:
from pip.req import parse_requirements
requirements = []
for r in parse_requirements('requirements.txt', session='hack'):
# check markers, such as
#
# rope_py3k ; python_version >= '3.0'
#
if r.match_markers():
requirements.append(str(r.req))
print(requirements)
r.match_markers() you're actually evaluating the markers, which is correct thing to do for a sdist. However, if you're building a binary dist (e.g. wheel), the package would only list those libraries that matched your build-time environment. –
Ut wheel environment (if it is the thing person tries to do) to evaluate markers against it? –
Brey bdist_wheel. It doesn't evaluate markers, it just adds them to extras_require. –
Ut © 2022 - 2024 — McMap. All rights reserved.
install_requiresis used to declare dependencies on packages that are required for the package to work and are used by developer of the package, whilerequirements.txtis used to automate installing of environments, which allows installing extra software and do the version pinning and are used by sysadmins deploying the package. Their role and target audience differ significantly, so trying to combine them like OP wishes is a genuine design mistake imho. – Unciform[line.strip() for line in open("requirements.txt").readlines()]? – Saltwaterpkg_resources.parse_requirements()– Benedicto