What does pipenv do after installing that takes up so much time and downloads huge amounts of data?

Question

After encountering a few nightmares with Python versions, I tried pyenv and pipenv. But when installing pygame and seaborn with pipenv, I noticed the installation happens in a few seconds and the Installation Succeeded message would appear immediately. Then there are some locking messages shown and there's a long waiting time of a few minutes where it shows a loading icon saying Locking.
During this time, there's a huge amount of data being downloaded. Image shown below. What is this data being downloaded? Why is it necessary? Can it be disabled? I'm wary of using pipenv now.

Grazynagreabe

Answer 1

This sounds related to https://github.com/pypa/pipenv/issues/3827:

pipenv lock downloads every available artifact of installed packages and their dependencies. It does this to calculate their hashes, even when the artifact url includes the hash in a fragment. For some large packages, such as scipy, which have large dependencies and many artifacts per version, this behavior can result in unreasonably long delays for some users (893MB vs. 50MB download).

A workaround in the form of a patch for the pipenv source code is given in this bug report itself. It takes the hash from the artifact URL if possible instead of always recomputing it, which seems to drastically improve locking time.

Link to workaround: https://github.com/pypa/pipenv/blob/4c003521052d3b607be5abedf989744a5c172bda/pipenv/patched/piptools/repositories/pypi.py#L60-L71

Margertmargery

Answer 2

if pipenv locking gets stuck somewhere do

$ pipenv install --skip-lock
$ pipenv lock

first skip the lock part by --skip-lock then manually do locking later when you have time
it works.

Potentiality

new pipenv is much faster lincolnloop.github.io/python-package-manager-shootout – Genisia

Answer 3

Because the developers of pipenv are strange. Yes, they are strange.

In short, pipenv is trying to download every dependency to calculate the hash. So it can generate a lock file with hash. Well, easy to understand that this is important to ensure a consistent environment.

But the problem is, in the past, this is the only way, as Pypi didn't provide hash for the packages. While for now, it is just ridiculous as Pypi does provide the hash for every package. There is no need for downloading the whole package to just get the hashtag. At least if you can get the hashtag directly from the metadata.

For an unknown reason, the developers of pipenv just don't want to make any change on this.

Boastful

Non-native speaker question: Is a hash really called a "hashtag"? I thought this was only for #annotations. – Margertmargery

If the community use a trustability chain to store the package hashes (blockchain), should this problem be solved, right? – Hayashi

@FelixDombek I think Sraw was maybe being inconsistent in terminology. If you check the pypi documentation pip.pypa.io/en/latest/reference/pip_install/#hashes-from-pypi you'll see that a hashtag (or number sign, or pound sign, or octothorp; they are all the same thing) is included as part of the download url, marking the url fragment; in that fragment the hash is specified using the key md5. For more information on the parts of a url, you can take a look at doepud.co.uk/blog/anatomy-of-a-url. – Integer