Pip pyproject.toml: Can optional dependency groups require other optional dependency groups?

Question

I am using the latest version of pip, 23.01. I have a pyproject.toml file with dependencies and optional dependency groups (aka "extras"). To avoid redundancies and make managing optional dependency groups easier, I would like to know how to have optional dependency groups require other optional dependency groups.

I have a pyproject.toml where the optional dependency groups have redundant overlaps in dependencies. I guess they could described as "hierarchical". It looks like this:

[project]
name = 'my-package'
dependencies = [
    'pandas',
    'numpy>=1.22.0',
    # ...
]

[project.optional-dependencies]
# development dependency groups
test = [
    'my-package[chem]',
    'pytest>=4.6',
    'pytest-cov',
    # ...
    # Redundant overlap with chem and torch dependencies
    'rdkit',
    # ...
    'torch>=1.9',
    # ...
]

# feature dependency groups
chem = [
    'rdkit',
    # ...
    # Redundant overlap with torch dependencies
    'torch>=1.9',
    # ...

]
torch = [
    'torch>=1.9',
    # ...
]

In the above example, pip install .[test] will include all of chem and torch groups' packages, and pip install .[chem] will include torch group's packages.

Removing overlaps and references from one group to another, a user can still get packages required for chem by doing pip install .[chem,torch], but I work with data scientists who may not realize immediately that the torch group is a requirement for the chem group, etc.

Therefore, I want a file that's something like this:

[project]
name = 'my-package'
dependencies = [
    'pandas',
    'numpy>=1.22.0',
    # ...
]

[project.optional-dependencies]
# development dependency groups
test = [
    'my-package[chem]',
    'pytest>=4.6',
    'pytest-cov',
    # ...
]

# feature dependency groups
chem = [
    'my-package[torch]',
    'rdkit',
    # ...
]
torch = [
    'torch>=1.9',
    # ...
]

This approach can't work because my-package is hosted in our private pip repository, so having'my-package[chem]' like the above example fetches the previously built version's chem group packages.

It appears that using Poetry and its pyproject.toml format/features can make this possible, but I would prefer not to switch our build system around too much. Is this possible with pip?

Studbook

I would use a Makefile instead of directly calling pip, this is usually better anyway, because of consistency across projects, if all your projects having make install or install-xx for xx group, then you have a make install as a standard way across the organization – Homonym

I do not understand what the blocker is. What does the fact that my-package is private have to do with your issue? Can you clarify this part? -- From what I can tell this should work, you already found the solution. -- As far as I know Poetry would not help, it has something called "dependency groups" that are not the same thing as "extras", but they are for development dependencies only, so not what you want. – Pitchstone

@Omar I will look into Makefiles. Is there a web page or some other resource that describes more of what you're suggesting here? – Studbook

@Pitchstone Apologies for confusion! The private nature of the pip package isn't the problem, it's simply that it is hosted in a pip repository to begin with. What happens is that if I write my-package[torch] as a chem group dependency, it will install from the repository's latest hosted version instead of installing the newly specified torch group in the pyproject.toml. I tried using the dot syntax, e.g. .[torch], hoping that would work but apparently pip doesn't recognize it within pyproject.toml files. And good to know about the Poetry differences! I'm not too familiar with Poetry yet – Studbook

Can you clarify this issue in the question itself? What should I picture? If I do python -m pip install 'my-package[chem] @ git+https://github.com/user/my-package.git' then the dependency my-package[torch] ends up triggering pip to install from the index instead of from git? Is that right? If yes, that might be a bug in pip, if not, then I still have not understood what you mean. – Pitchstone

Maybe related: github.com/pypa/pip/issues/10393 and discuss.python.org/t/8428 – Pitchstone

@Pitchstone I just uploaded an example Makefile that deals with docker compose routine tasks, it supports help and passing arguments to make targets gist.github.com/omars-code/cc4c8f2e24c1f8a65365d56a99b910f6 – Homonym

I think I was mistaken in what the problem is. In the question, the approach I desired (e.g. 'my-package[chem]') works so long as my-package is not hosted in an index somewhere. If I change the name to something that doesn't exist on pypi or on my PIP_EXTRA_INDEX_URL, then it works fine. Otherwise, like I mentioned earlier, pip seems to prioritize grabbing the already-hosted package instead of from the file's groups. @Pitchstone you're correct in your understanding except it's installing from pypi/pip indices not git. Sorry for the confusion, should I start a new question or edit this? – Studbook

Also the issues you related are pretty spot on. Seems like this was never addressed? Is it worth raising an issue on pip? – Studbook

@Omar thank you for the example Makefile. I've never used make before (except to install Python). Gives me something to think about for this project as well as my other projects. – Studbook

Edit your question. And maybe remove the parts that are now known to not be relevant. -- According to the discussions and GitHub issue, this should work. So I am a bit confused. If you manage to create a minimal reproducible example, that would be the best. – Pitchstone

Answer 1

Maybe it's worth binding the 2 dependency groups into one, and see if that solves the problem, like shown in 'all'

[project]
name = "foo"
version = "0.1.0"

[project.optional-dependencies]
socks = ["pysocks"]
jwt = ["pyjwt"]
all = ["foo[socks,jwt]"]

Homonym

This is not specific to PDM, and is indeed already covered in the question itself. So I am not sure what this answer adds to the discussion. – Pitchstone

It's not covered in the question itself, the idea is to bind foo[socks,jwt] into all, and then the precedence should be resolved, or am I misunderstanding something here ? – Homonym

Ah, I see. But from my understanding the issue is at another level. Maybe a bug in pip? Although that seems unlikely. Hard to tell. -- Anyway the notation you are presenting is not specific to PDM,it is part of this specification which is implemented by many tools, not just PDM. – Pitchstone

Thanks @sinoroc, will update, so not to mislead anyone – Homonym

Awesome, thanks! Didn't know that this is possible. – Waterer

Perhaps even simpler? all = [".[socks,jwt]"] – Eyeless

No. That produces a ValueError. Must be pep508. – Eyeless