pip installing environment.yml as if it's a requirements.txt

Question

I have an environment.yml file, but don't want to use Conda:

name: foo
channels:
  - defaults
dependencies:
  - matplotlib=2.2.2

Is it possible to have pip install the dependencies inside an environment.yml file as if it's a requirements.txt file?

I tried pip install -r environment.yml and it doesn't work with pip==22.1.2.

Fumed

No, but you can always parse the yaml yourself using Python. I'm not familiar with Conda, but it looks like it might be as simple as extracting the dependencies list and then either invoking pip directly using subprocess` or outputting each entry as one line in a requirements.txt file – Epitasis

Answer 1

No, pip does not support this format. The format it expects for a requirements file is documented here. You'll have to convert the environment.yml file to a requirements.txt format either manually or via a script that automates this process. However, keep in mind that not all packages on Conda will be available on PyPI.

Stereoscopy

Answer 2

Based on Beni implementation, I just wanted to adjust the code since it has lots of errors;

import os
import yaml

with open("environment.yaml") as file_handle:
    environment_data = yaml.safe_load(file_handle)

for dependency in environment_data["dependencies"]:
    if isinstance(dependency, dict):
      for lib in dependency['pip']:
        os.system(f"pip install {lib}")

Massotherapy

This answer solves the problem and should be marked as the solution! – Fenny

This answer only solves the very limited case where there are pip: dependencies declared the YAML. Any dependencies listed as Conda packages are completely ignored. – Cubeb

First, maybe because the question is asking about how to install packages within the .yaml file using pip. I don't think this is limiting installation. In many cases, you can't use Conda or installing it is time expensive – we have Colab, for example, or a remote server where you want to do a specific task. – Massotherapy

It is limited and take the OP YAML as an example: it lacks a pip: section, so this code would do nothing given it. On Colab, condacolab works fine directly - no need to convert. For other situations, Micromamba is standalone (no install) and fast. – Cubeb

I have not known that before. I will check this out! – Massotherapy

Here is a pastable oneliner version python -c "import os, yaml; [os.system(f'pip install {lib}') for dependency in yaml.safe_load(open('environment.yaml'))['dependencies'] if isinstance(dependency, dict) for lib in dependency.get('pip', [])]" – Brink

I just wish it would ignore versions – Brink

This batch one liner, lets you test all the lines in environement.yaml to make sure they're installed @echo off & for /f "tokens=2* delims=-" %i in (environment.yaml) do ( for /f "tokens=1 delims=<>=@" %j in ("%i") do ( python -c "import sys; exec('try: import %j; sys.exit(0)\nexcept ImportError: sys.exit(1)')" && echo %j: OK || echo %j: missing ) ) – Brink

Answer 3

I've implemented what Brian suggests in his comment.

This converts the environment.yaml to requirements.txt:

import yaml

with open("environment.yaml") as file_handle:
    environment_data = yaml.load(file_handle)

with open("requirements.txt", "w") as file_handle:
    for dependency in environment_data["dependencies"]:
        package_name, package_version = dependency.split("=")
        file_handle.write("{} == {}".format(package_name, package_version))

And this installs the dependencies directly with pip:

import os
import yaml

with open("environment.yaml") as file_handle:
    environment_data = yaml.load(file_handle)

for dependency in environment_data["dependencies"]:
    package_name, package_version = dependency.split("=")
    os.system("pip install {}=={}".format(package_name, package_version))

NOTE: I've omitted error handling and any other variations of package definitions (e.g., specification of a package version greater than or equal to a certain version) to keep it simple.

Trotman

Answer 4

The first answer makes important points: there is not direct conversion because Conda is a general package manager and so includes additional packages. Furthermore, Conda packages can often go by different names. None of the proposed parsing solutions cover this situation.

Personally, I think the most efficacious complete approach is to recreate the environment with Mamba, then use pip in the environment to dump out a legitimate requirements.txt.

# use mamba, not conda
mamba env create -n foo -f environment.yaml
mamba install -yn foo pip
mamba run -n foo pip list --format freeze > requirements.txt
mamba env remove -n foo

That is, don't overthink it and use the reliable tools at hand.

Cubeb

Answer 5

If you don't want to use a script I've used the following one-liner to generate the requirements.txt file from an env.yml file:

❯ grep -n "pip:" env.yml | cut -d":" -f1 | xargs -I{} expr {} + 1 | xargs -I{} tail -n +{} env.yml | sed 's/^[[:space:]]*- //' | cut -d"=" -f 1 > requirements.txt

This just takes all of the dependencies under the pip dependency in the yml and removes the pinned version for pip to handle on its own.

Eddings