How are Pipfile and Pipfile.lock used?

Question

It seems that Pipfile / Pipfile.lock are intended to be replacements for requirements.txt, in the context of Python packaging. There isn't much documentation out there on how these actually work, however. I found an evolving description of pipfile on the PyPi section of the Python website here but it's pretty messy and doesn't explain the semantics of the different sections of the file.

Any pointers on how to understand these files?

Scalpel

This is a direct analogue of Gemfile and Gemfile.lock from the Ruby world: The .lock file has specific versions for each dependency; the one without that extension has only the known-to-controlling-humans versions. That said, asking for an explanation of something that's still evolving and a long way off from being well-defined, much less standardized, is perhaps a bit premature. – Qualm

(And similarly, the difference between Pipfile and requirements.txt is largely that the former tries to adopt features from the Ruby world, ie. being able to specify dependency sets for multiple environments and with conditions/options/etc. within a single file). – Qualm

It seems like it's already been deployed in the Heroku "getting started with Python" repo (github.com/heroku/python-getting-started.git) so like it or not, seems like it's productionized. – Scalpel

Gotcha. That said -- the docs look pretty solid to me. I don't know what I could write in an answer that wouldn't just be restating them. – Qualm

What docs are you referring to? – Scalpel

If you're referring to the link that I made in the OP then there are a number of things omitted, for example what does it actually mean for something to be in a section called source. – Scalpel

I'd suggest filing an issue with the project, then, describing in detail the specific items which the documentation doesn't cover. That way you'll get directly to the folks most familiar with the codebase. – Qualm

Answer 1

The concept behind these files is simple and analogue to other already existing tools, if you have some familiarity with Ruby's Bundler or Node's Npm. Pipenv is both a package and virtual environment management tool that uses the Pipfile and Pipfile.lock files to achieve these goals.

Pipenv handles the virtual environment for you (no more activate and deactivate required). Below, some basics to get you started, see more at pipenv website.

Getting Started

Start using pipenv is easy, in your project folder type...

$ pipenv install

... and if it already has a requirements.txt file, it will generate a Pipfile file with the requirements and a virtual environment folder, otherwise, it will generate an empty Pipfile file. If you disliked or changed your mind about something that you have installed, just type...

$ pipenv uninstall <package>

... and you're good to go. To activate the virtual environment that pipenv already generated, go with...

$ pipenv shell

... and your virtual environment will be activated. To leave the environment...

$ exit

... and you will be back to your original terminal session.

Pipfile

The Pipfile file is intended to specify packages requirements for your Python application or library, both to development and execution. You can install a package by simply using...

$ pipenv install flask

... and it will be added as a dependency for deployment and execution or by using ...

$ pipenv install --dev pytest

... and it will be used as a dependency for development time. In both cases, if you need to be more specific about the package version, as stated in the documentation pipenv makes use of the same version specifiers used by pip. The file syntax is pretty straight forward, as follows.

[[source]] # Here goes your package sources (where you are downloading your packages from).
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages] # Here goes your package requirements for running the application and its versions (which packages you will use when running the application).
requests = "*"
flask = "*"
pandas = "*"

[dev-packages] # Here goes your package requirements for developing the application and its versions (which packages you will use when developing the application)
pylint = "*"
wheel = "*"

[requires] # Here goes your required Python version.
python_version = "3.6"

Pipfile.lock

The Pipfile.lock is intended to specify, based on the packages present in Pipfile, which specific version of those should be used, avoiding the risks of automatically upgrading packages that depend upon each other and breaking your project dependency tree.

You can lock your currently installed packages using...

$ pipenv lock

... and the tool will lookup your virtual environment folder to generate the lock file for you automatically, based on the currently installed versions. The file syntax is not as obvious as is for Pipfile , so for the sake of conciseness, it will not be displayed here.

Vogel

This is an interesting answer, but I think when I was asking I couldn't quite understand what is actually meant by [[source]], like is that the source for the packages which are downloaded by pipenv or something else? I guess the rest of the two files is obvious enough. – Scalpel

@Stephen, when using package managers such as dnf, npm and pip, the package manager have to know where from it will pull the packages that you require, that is the source. On python, by default, we can use the PyPI with pip, but we can also have a package server on premises (for instance a JFrog Artifactory install), so we can have multiple sources of packages from where we pull. By the way, PyPI just has updated it's interface, it is awesome! :) – Vogel

Wait, why would pytest and coverage be in [packages] instead of [dev-packages]? Aren't these used by developers, rather than users? – Turnip

@AlSweigart, it was just a random selection of packages from an already existing file in my computer, if you think it may be confusing for people reaching this answer, let me now so I can change them for clarity sake. In the original file, this setup is used so that my continuous integration pipeline can install packages required for running unit tests and code coverage. – Vogel

I'll admit, I found it confusing because I'm working on a tutorial where I have to decide if the [package]/[dev-package] difference is important, or if I should just put everything in [package]. (Googling for this is what led me to this page.) But having both while putting dev stuff in [package] kind of threw me. I could see why requests should be in [package] though. – Turnip

Changed, as requested, to avoid confusion on development packages, thanks for you comment, hope it is less confusing now :) – Vogel

@XChikuX, as described here, the main commands are install, uninstall, and lock, which generates a Pipfile.lock, but you can skip it by using the --skip-lock option if you don't want it for some reason ;) – Vogel

Are we meant to also commit Pipfile.lock into source control? Especially when working with medium-sized and distributed teams? – Zwinglian

I think it is a good idea, if you want to keep track of ones package versions along the source code evolution, since most of the packages will be version starred on Pipfile. If something breaks on a package upgrade, one should then be able to fix it by reviewing the package version history, and indeed, one of the creators of the requests library do recommend doing it here. – Vogel

Do you need a flag when using pipenv install to force pipenv to use Pipfile.lock to determine what packages to install, or does that just happen automatically? – Scalpel

@Stephen, it is actually the opposite, as stated in pipenv install documentation you can specify the --skip-lock flag to force it to ignore the Pipfile.lock file rather than forcing it to use it ;) – Vogel

what I want ask is that after 'pipenv lock' is done and I tried to add new things to the project after a long time and that time the packages that I used in that particular project got updated. When I hit something like 'pipenv install' the version of packages in both Pipfile and Pipfile.lock will be updates or it will stay as the same version that we locked even tho there will be new versions. – Grummet

@ChinkySight, the lock file is intended to keep the exact same versions, so that your project keeps working, even after a while. So, when you pipenv install, the previously added package versions are preserved. – Vogel

@ChinkySight, if you want to update, either a specific package or all packages though, you can run a pipenv update <pkg>, where pkg is optional, as stated here – Vogel

@danieldeveloper001 Thank you for your response I really appreciate that. But okay lets I have locked the lock file which contain openpyxl == 3.0.1 only. When I install the environment using 'pipenv install -ignore-Pipfile', it install the packages and set the V-environment it as 'Pipfile.lock' but real things is that when I perform 'pipenv install' on that it will update the openpyxl version to the latest which is 3.0.2 why is that ? – Grummet

@ChinkySight, could you provide the steps that you've performed? I cannot reproduce the behaviour you are experiencing here. Below, the steps I've performed trying to reproduce the issue: (1) pipenv install --python 3 ; (2) pipenv shell ; (3) pipenv install openpyxl==3.0.1 ; (4) pipenv install --ignore-pipfile – Vogel

@danieldeveloper001 my question was that, let assume that I locked the v-environment two months before where I had install the 'openpyxl' only whose version was 3.0.1. If I install the V-environment now via pipenv install -ignore-Pipfile then what will be the openpyxl version, 3.0.1 or 3.0.2 ? – Grummet

@danieldeveloper001 3.0.2 is the latest version :D – Grummet

Let us continue this discussion in chat. – Grummet

I am the OP. The answer as provided is OK, but one of the main things I was getting at when I posted a few years ago was my confusion about how Pipfile.lock actually locks anything, and I don't think it's convincingly addressed yet. My confusion was that I thought Pipfile.lock could lock my OWN environment at the versions that it contained, so I was confused when it got updated later. – Scalpel

Now I think that that Pipfile.lock is only useful for the workflow where a new developer/system gets Pipfile.lock and runs pipenv install. In that case they should get the exact same packages. So in that sense it's a lock, but it never actually prevents any local package updates by developers who have it in their folder. Is that correct? – Scalpel

@Vogel If specified python version is 3.6 , then does it mean the latest 3.6 version e.g 3.6.15 or the version 3.6.0 ? – Palpitant

It has been a while since the last time I wrote Python, but from what I can infer from this documentation, there are both python_version and python_full_version, so I would assume that if not using the "full_version" option, it would use the closest version that matches the "version" option. Hence, using your example, 3.6 could mean any 3.6.x and using 3 would mean any 3.x.y. Can you test it and share the results? – Vogel

"I thought Pipfile.lock could lock my OWN environment at the versions that it contained". Hello OP, from what I can recall, if a Pipfile.lock was created from an existing environment and a library foo was at version x.y.z, then the same number would appear in the lockfile. If changes were made to the Pipfile itself, the Pipfile.lock would be generated again, unless when the --skip-lock option was used (last documented option from this link). – Vogel

Answer 2

It is worth noting that as of now (2022), pipfile and pipfile.lock are only used by pipenv, a tool written by the same authors. There is currently no standard (i.e. PEP) that mentions those files.

Regarding pipenv there's other popular tools (like poetry) that try to solve a similar use case without the use of pipfiles.

On a related note, there's also the dawn of pyproject.toml supported by various PEPs like 660 and 621, which is slowly gaining support by setuptools, flit, hatch.

Update: Since Summer 2022 the python packaging tutorial replaced the setup.py and setup.cfg based tutorial completely with pyproject.toml.

Longcloth

Answer 3

As explained above by @Charles Duffy, it is a direct analog of Gemfile and Gemfile.lock from the Ruby world. See reference below for more details.

Reference: https://medium.com/never-hop-on-the-bandwagon/gemfile-and-gemfile-lock-in-ruby-65adc918b856

Katmandu