Get all modules/packages used by a python project

Question

I have a python GUI application. And now I need to know what all libraries the application links to. So that I can check the license compatibility of all the libraries.

I have tried using strace, but strace seems to report all the packages even if they are not used by the application.

And, I tried python ModuleFinder but it just returns the modules that are inside python2.7 and not system level packages that are linked.

So is there any way I can get all the libraries that are linked from my application?

Idolize

pip freeze for all installed packages – Evolutionary

pip install yolk, which gives you a list of all the eggs in your environment using: yolk -l – Commissar

@wolendranh, I am concerned with the use of pip freeze as I guess that pip freeze will simply list down all the packages present in that virtual env , but that might not be used by my application and I might have messed up my env by adding some unwanted packages.. how to solve that? I am more interested in cleaning up the dependencies list, sort of!! – Wallaroo

Answer 1

You can give a try to the library https://github.com/bndr/pipreqs found following the guide https://www.fullstackpython.com/application-dependencies.html

---

The library pipreqs is pip installable and automatically generates the file requirements.txt.

It contains all the imports libraries with versions you are using in the virtualenv or in the python correctly installed. Just type:

pip install pipreqs
pipreqs /home/project/location

It will print:

INFO: Successfully saved requirements file in /home/project/location/requirements.txt

In addition it is compatible with the pip install -r command: if you need to create a venv of your project, or update your current python version with compatible libraries, you just need to type:

pip install -r requirements.txt

I had the same problem and this library solved it for me. Not sure if it works for multiple layers of dependencies i.e. in case you have nested level of dependent libraries.

-- Edit 1:

If looking for a more sophisticated version manager, please consider as well pyvenv https://github.com/pyenv/pyenv. It wraps virtualenv producing some improvements over the version specification that is created by pipreqs.

-- Edit 2:

If, after creating the file with the dependency libraries of your module with pipreqs, you want to pin the whole dependency tree, take a look at pip-compile. It figures out a way to get the dependencies of your top level libraries, and it pins them in a new requirement files, indicating the dependency tree.

-- Edit 3:

If you want to split your dependency tree into different files (e.g. base, test, dev, docs) and have a way of managing the dependency tree, please take a look at pip-compile-multi.

-- Edit 4:

At this point in time, I would recommend taking a look at poetry/PDM/Hatch and start projects directly with this dependency manager. Dependency tree comes out of the box.

Leigh

seems that this install latest version of the packages and not the current version you are using. – Brought

One of the best answers one could possible get – Dun

@ennepi: does not look the case for me. Are you using it in a virtual-environment? – Leigh

@ennepi, ok, got it. You may need to add the flag --use-local to get what you are looking for – Leigh

Important! both pipreqs and other scanning packages fail on BOM character (UTF-8) on start of any .py file in your path. It fails ugly... So, if you get a bad character syntax error, remove bom character (using nodepad++ for example...) and you're good to go. – Miniature

I am interested in knowing if a module inside my env is actually needed by my application code or not... how to do that? thanks! – Wallaroo

You can use pre-commit, if your application is versioned controlled, or simply a linter, like pylint. Pylint will indicate as warnings all the modules that are imported but not used. – Leigh

pipreqs works, but I dont see how to use your other suggesion: pyenv. I have installed it, but what should I do to generate a requirements file of my python package? – Bout

Yep, I was indicating it as a possible package manager, after having found the packages with pipreqs. (I'll amend the answer to make it clear). – Leigh

Any possibility of a package that scans a directory for all .py files, and scans those for all imports, and creates a list of packages that way? Also referring to host machine (or a named venv) for package versions would be nice. – Accent

@Accent as far as I know pipreqs does exactly this scan, but the version are the latest ones, not the one on the virtualenv. You can combine the requirements file with the result of pip freeze to get what you are looking for. – Leigh

@Leigh Thanks for mentioning that. I'm looking into it, and maybe it can do both! pypi.org/project/pipreqs: --use-local: Use ONLY local package info instead of querying PyPI – Accent

Answer 2

Install yolk for python2 with:

pip install yolk

Or install yolk for python3 with:

pip install yolk3k

Call the following to get the list of eggs in your environment:

yolk -l

Alternatively, you can use snakefood for graphing your dependencies, as answered in this question.

You could try going into the site-packages folder where the unpacked eggs are stored, and running this:

ls -l */LICENSE*

That will give you a list of the licence files for each project (if they're stored in the root of the egg, which they usually are).

Commissar

Answer 3

for those the answer from @SeF is not working and getting command not found error , you can also try

pip install pipreqs
pipreqs /home/project/location

'pipreqs' is not recognized as an internal or external command,
operable program or batch file.

Or

python3 -m  pipreqs.pipreqs .

"." is the location of your project, in my case i was in the project directory

Redmer

Answer 4

To get all the installed packages or modules. A very easy way is by going to your virtual environment directory on the terminal (The one with (venv) behind your computer's username) and run on the terminal one of these commands

pip freeze > requirements.txt

If you are using python3

pip3 freeze > requirements.txt

This would get all your installed packages from your virtual environment's library of packages used during the project, and store them in the 'requirements.txt' file that would automatically be created upon running the command.

Picaroon

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. – Inheritor