How to add optional dependencies of a library as 'extra' in poetry and pyproject.toml?

Question

I am building a Python package using pyproject and poetry. My pyproject.toml looks like this:

[tool.poetry]
authors = ["test"]
description = ""
name = "test"
version = "0.1.0"

[tool.poetry.dependencies]
spacy = {extras = ["cuda113"], version = "^3.2.3"}
faiss-gpu = {version = "1.7.2", optional = true}

[tool.poetry.extras]
gpu = ["faiss-gpu"]

This successfully installs faiss-gpu as an extra using poetry install -E gpu.

However, I would like to install spacy[cuda113] (GPU version) only when poetry install -E gpu is provided. A normal poetry install should only install spacy (CPU version).

I've tried using the following configuration, but this makes all of spacy optional and doesn't install it. Only spacy[cuda113] (GPU version) must be optional.

[tool.poetry]
authors = ["test"]
description = ""
name = "test"
version = "0.1.0"

[tool.poetry.dependencies]
spacy = {extras = ["cuda113"], version = "^3.2.3", optional = true}
faiss-gpu = {version = "1.7.2", optional = true}

[tool.poetry.extras]
gpu = ["faiss-gpu", "spacy"]

Is there a way to make spacy[cuda113] optional but spacy as a required dependency?

Dyke

Answer 1

Try like this:

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
authors = ["test"]
description = ""
name = "test"
version = "0.1.0"

[tool.poetry.dependencies]
python = "^3.10"
faiss-gpu = {version = "1.7.2", optional = true}
spacy = {extras = ["cuda113"], version = "^3.2.3", optional = true}
Spacy = "^3.2.3"

[tool.poetry.extras]
gpu = ["faiss-gpu", "spacy"]

It generates the dist-info METADATA lines like this, which looks correct in terms of how pip will install test vs test[gpu]:

Metadata-Version: 2.1
Name: test
Version: 0.1.0
Summary: 
Author: test
Requires-Python: >=3.10,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Provides-Extra: gpu
Requires-Dist: Spacy (>=3.2.3,<4.0.0)
Requires-Dist: faiss-gpu (==1.7.2); extra == "gpu"
Requires-Dist: spacy[cuda113] (>=3.2.3,<4.0.0); extra == "gpu"

Disclaimer: I think I am relying on bugs and/or implementation detail in poetry for this to work. Note the case sensitivity difference between "spacy" and "Spacy" in the dependency specification. The ordering of the lines is also crucial.

Amboceptor

Hi @wim. Thanks for answering but isn't quite the solution. The problem with your solution is that both "Spacy" and "spacy[cuda113]" are installed as extra gpu dependencies. As stated in my question A normal poetry install should only install spacy (CPU version). So I want the base spacy library to not be an extra and always be installed. – Dyke

could you provide some documentation on what happens when you specify a library using capital letters e.g. "Spacy" ? – Dyke

@Dyke I think you have some misunderstanding about how extras work - there is no difference between installing both "Spacy" and "spacy[cuda113]" vs installing only "spacy[cuda113]". Extras can only add dependencies. In this case it adds the dependency for cupy-cuda113. The capital letter is only there to trick the TOML loader into considering Spacy and spacy as unique keys - otherwise the file in the format that poetry expects is not valid TOML due to duplicate keys in a mapping. – Amboceptor

@vim I very much appreciate the effort in the answer, but with your settings, if I run poetry install -E gpu, only the normal spacy version gets installed, so it doesn't meet the requirements – Dyke

@Dyke It does work. I've tested and verified this as working on two different machines with different poetry versions now (Python 3.10 + poetry 1.1.13, and Python 3.8 + poetry 1.1.11). See replit.com/@wimglenn/ExampleSpacyGPU#pyproject.toml - created a fresh environment, removed the lockfile and then ran poetry install -E gpu which pulls in cupy-cuda113 (check the pip freeze in the shell tab). The only way cupy-cuda113 can be installed is via the gpu extra of spacy. – Amboceptor

@vim thanks for providing the replit. It does seem to work, so thank you! As the answer relies on implementation details that are not made explicit by poetry, I'll wait until they have a proper feature for this. – Dyke