How do I keep track of pip-installed packages in an Anaconda (Conda) environment?

Question

I've installed and have been using the Anaconda Python distribution, and I have started using the Anaconda (Conda) environment. I can use the standard conda install... command to put packages from the distribution into my environments, but to use anything outside (i.e. Flask-WTF, flask-sqlalchemy, and alembic) I need to use pip install in the active environment. However, when I look at the contents of the environment, either in the directory, or using conda list these pip installed packages don't show up.

Using pip freeze and pip list just lists every package I've ever installed.

Is there a way to keep track of what is in each of my Anaconda envs (both pip and conda installed)?

Fulfil

Use pip installed with conda, e.g. ~/anaconda/bin/pip. Use it to install packages into a conda environment, as well as to see the union of packages installed with this pip and with conda install. – Monachism

conda list has for a while now included python packages that were installed by other means (e.g. pip, easy-install, directly with python setup.py install, etc etc) – Cockcrow

Do not upgrade your pip or this export will get hosed – Existentialism

@andrew-cassidy could you please explain a bit more about this? Sometimes when I do a conda update --all it will update pip too. We cannot easily prevent pip updates from happening. Does pip (behave badly) by deleting its own index whenever it is updated? – Khoury

Answer 1

conda-env now does this automatically (if pip was installed with conda).

You can see how this works by using the export tool used for migrating an environment:

conda env export -n <env-name> > environment.yml

The file will list both conda packages and pip packages:

name: stats
channels:
  - javascript
dependencies:
  - python=3.4
  - bokeh=0.9.2
  - numpy=1.9.*
  - nodejs=0.10.*
  - flask
  - pip:
    - Flask-Testing

If you're looking to follow through with exporting the environment, move environment.yml to the new host machine and run:

conda env create -f path/to/environment.yml

Fact

Just to make this clear for future readers: the -pip: section flags all of the packages that were installed via pip. Also, to my knowledge, conda env export will never generate semvar matching values, like the -nodejs=0.10.* shown in this example. It will give an explicit value like -nodejs=0.10.1 and you will need to put the * if that is what you want. If it is able to handle semvar matching, that would be awesome and I'd be glad to hear how to make that magic happen. :) – Bitternut

I have found cases in which this failed, with 'ResolvePackageNotFound' errors. It was solved by adding --from-history in the command that creates the yml file. – Orpington

If you just want to look at the list in the console, you can omit the > environment.yml. – Penurious

Answer 2

conda will only keep track of the packages it installed. And pip will give you the packages that were either installed using the pip installer itself or they used setuptools in their setup.py so conda build generated the egg information. So you have basically three options.

1. You can take the union of the conda list and pip freeze and manage packages that were installed using conda (that show in the conda list) with the conda package manager and the ones that are installed with pip (that show in pip freeze but not in conda list) with pip.

2. Install in your environment only the python, pip and distribute packages and manage everything with pip. (This is not that trivial if you're on Windows...)

3. Build your own conda packages, and manage everything with conda.

I would personally recommend the third option since it's very easy to build conda packages. There is a git repository of example recipes on the continuum's github account. But it usually boils down to:

 conda skeleton pypi PACKAGE
 conda build PACKAGE

or just:

conda pipbuild PACKAGE

Also when you have built them once, you can upload them to https://binstar.org/ and just install from there.

Then you'll have everything managed using conda.

Rigadoon

I just tried to ~/anaconda/bin/pip install memory-profiler and conda install numba, and they are both shown in ~/anaconda/bin/pip freeze. – Monachism

"git repository of example recipes on the continuum's github account" Do you mean github.com/ContinuumIO/anaconda-recipes ? Please link if you can – Crisscross

This doesn't actually install the package so that you can import it, though. You have to do conda install C:\...PACKAGE-0.0.0-py27_0.tar.bz2 as described in https://mcmap.net/q/149279/-how-do-i-keep-track-of-pip-installed-packages-in-an-anaconda-conda-environment – Crisscross

I'd say first always have conda install python=3.x.y in your conda environment such that subsequent pip installs go through conda installed pip. – Amadus

This answer (edited Jun 17 '14 at 11:36 answered Sep 5 '13 at 15:40 by Viktor Kerkez) is now out of date. The full answer above by @Fact (answered Nov 13 '15 at 14:21) provides a more accurate and up-to-date answer. – Khoury

Answer 3

There is a branch of conda (new-pypi-install) that adds better integration with pip and PyPI. In particular conda list will also show pip installed packages and conda install will first try to find a conda package and failing that will use pip to install the package.

This branch is scheduled to be merged later this week so that version 2.1 of conda will have better pip-integration with conda.

Holmquist

I'd like to point out that this is already working in conda 2.2.3 (and probably before that). The way it works is that if you try to conda install package and it can't find a conda recipe, it tries to install it via pip (and then keeps track of it). It even helpfully suggests installing pip if not in your environment. Thanks Travis for all your work! – Affenpinscher

Clarification, since I've misread this several times. @Travis means I think that if one is running inside a conda, then running pip will first search for conda packages? The reverse is definitely not true - conda install does not look for pypi packages, not even items on pypi.binstar.org. This is as of conda 3.5.2. – Irrecoverable

Has this been removed? If I run, say, conda install autopep8 it tells me "No package found" and that I should search anaconda.org manually. – Crisscross

@Irrecoverable running pip... also does not work, it will not look for conda package at all. – Touchstone

Answer 4

I followed @Viktor Kerkez's answer and have had mixed success. I found that sometimes this recipe of

conda skeleton pypi PACKAGE

conda build PACKAGE

would look like everything worked but I could not successfully import PACKAGE. Recently I asked about this on the Anaconda user group and heard from @Travis Oliphant himself on the best way to use conda to build and manage packages that do not ship with Anaconda. You can read this thread here, but I'll describe the approach below to hopefully make the answers to the OP's question more complete...

Example: I am going to install the excellent prettyplotlib package on Windows using conda 2.2.5.

1a) conda build --build-recipe prettyplotlib

You'll see the build messages all look good until the final TEST section of the build. I saw this error

File "C:\Anaconda\conda-bld\test-tmp_dir\run_test.py", line 23 import None SyntaxError: cannot assign to None TESTS FAILED: prettyplotlib-0.1.3-py27_0

1b) Go into /conda-recipes/prettyplotlib and edit the meta.yaml file. Presently, the packages being set up like in step 1a result in yaml files that have an error in the test section. For example, here is how mine looked for prettyplotlib

test:   # Python imports   imports:
    - 
    - prettyplotlib
    - prettyplotlib

Edit this section to remove the blank line preceded by the - and also remove the redundant prettyplotlib line. At the time of this writing I have found that I need to edit most meta.yaml files like this for external packages I am installing with conda, meaning that there is a blank import line causing the error along with a redundant import of the given package.

1c) Rerun the command from 1a, which should complete with out error this time. At the end of the build you'll be asked if you want to upload the build to binstar. I entered No and then saw this message:

If you want to upload this package to binstar.org later, type:

$ binstar upload C:\Anaconda\conda-bld\win-64\prettyplotlib-0.1.3-py27_0.tar.bz2

That tar.bz2 file is the build that you now need to actually install.

2) conda install C:\Anaconda\conda-bld\win-64\prettyplotlib-0.1.3-py27_0.tar.bz2

Following these steps I have successfully used conda to install a number of packages that do not come with Anaconda. Previously, I had installed some of these using pip, so I did pip uninstall PACKAGE prior to installing PACKAGE with conda. Using conda, I can now manage (almost) all of my packages with a single approach rather than having a mix of stuff installed with conda, pip, easy_install, and python setup.py install.

For context, I think this recent blog post by @Travis Oliphant will be helpful for people like me who do not appreciate everything that goes into robust Python packaging but certainly appreciate when stuff "just works". conda seems like a great way forward...

Oahu

Using conda version 3.5.2, the command $ conda build --build-recipe XXX ; Errors out with ' unrecognized arguments: --build-recipe'. Full response is; usage: conda-build [-h] [-c] [--no-binstar-upload] [--output] [-s] [-t] [--no-test] [-V] [-q] RECIPE_PATH [RECIPE_PATH ...] conda-build: error: unrecognized arguments: --build-recipe – Irrecoverable

This answer ( answered Dec 23 '13 at 20:04 Josh Hemann) is now out of date. The full answer above by @Fact (answered Nov 13 '15 at 14:21) provides a more accurate and up-to-date answer. – Khoury

Answer 5

I'd recommend conda-lock for this nowadays.

Appendicle

This package is apparently no longer maintained, according to github.com/Simplistix/picky-conda and github.com/Simplistix/picky-pip. – Dunlap

@Dunlap - yep, updated answer for currently maintained tooling, thanks for highlighting! – Appendicle

Answer 6

I usually prefix the 'bin/pip' folder for the specific environment you want to install the package before the 'pip' command. For instance, if you would like to install pymc3 in the environment py34, you should use this command:

~/anaconda/envs/py34/bin/pip install git+https://github.com/pymc-devs/pymc3 

You basically just need to find the right path to your environment 'bin/pip' folder and put it before the install command.

Misadvise

Answer 7

I think what's missing here is that when you do:

>pip install .

to install a local package with a setup.py, it installs a package that is visible to all the conda envs that use the same version of python. Note I am using the conda version of pip!

e.g., if I'm using python2.7 it puts the local package here:

/usr/local/anaconda/lib/python2.7/site-packages

If I then later create a new conda env with python=2.7 (= the default):

>conda create --name new

>source activate new

And then do:

(new)>conda list    // empty - conda is not aware of any packages yet

However, if I do:

(new)>pip list      // the local package installed above is present

So in this case, conda does not know about the pip package, but the package is available to python.

However, If I instead install the local package (again using pip) after I've created (and activated) the new conda env, now conda sees it:

(new)>conda list   // sees that the package is there and was installed by pip

So I think the interaction between conda and pip has some issues - ie, using pip to install a local package from within one conda env makes that package available (but not seen via conda list) to all other conda envs of the same python version.

Cipolin

Answer 8

conda env export lists all conda and pip packages in an environment. conda-env must be installed in the conda root (conda install -c conda conda-env).

To write an environment.yml file describing the current environment:

conda env export > environment.yml

References:

• https://github.com/conda/conda-env/
• https://github.com/conda/conda-env/issues/12

Maugham

Answer 9

You can start by installing the below given command in the conda environment:

conda install pip

Followed by installing all pip packages you need in the environment.

After installing all the conda and pip packages to export the environment use:

conda env export -n <env-name> > environment.yml

This will create the required file in the folder

Chante

Answer 10

My which pip shows the following path:

$ which pip
/home/kmario23/anaconda3/bin/pip

So, whatever package I install using pip install <package-name> will have to be reflected in the list of packages when the list is exported using:

$ conda list --export > conda_list.txt

But, I don't. So, instead I used the following command as suggested by several others:

# get environment name by
$ conda-env list

# get list of all installed packages by (conda, pip, etc.,)
$ conda-env export -n <my-environment-name> > all_packages.yml
# if you haven't created any specific env, then just use 'root'

Now, I can see all the packages in my all-packages.yml file.

Hoenir

Answer 11

Use your environment's pip to install packages like so

~/anaconda3/envs/<ENV_NAME_HERE>/bin/pip install <PACKAGE_NAME>

This should help conda track all your pip installed packages as well when you use conda list

Tonicity