How to use poetry with docker?

Question

How do I install poetry in my image? (should I use pip?)

Which version of poetry should I use?

Do I need a virtual environment?

There are many examples and opinions in the wild which offer different solutions.

Viperish

For inspiration, you can find a production example Poetry + Dockerimage here. – Underestimate

Answer 1

TL;DR

Install poetry with pip, configure virtualenv, install dependencies, run your app.

FROM python:3.10

# Configure Poetry
ENV POETRY_VERSION=1.2.0
ENV POETRY_HOME=/opt/poetry
ENV POETRY_VENV=/opt/poetry-venv
ENV POETRY_CACHE_DIR=/opt/.cache

# Install poetry separated from system interpreter
RUN python3 -m venv $POETRY_VENV \
    && $POETRY_VENV/bin/pip install -U pip setuptools \
    && $POETRY_VENV/bin/pip install poetry==${POETRY_VERSION}

# Add `poetry` to PATH
ENV PATH="${PATH}:${POETRY_VENV}/bin"

WORKDIR /app

# Install dependencies
COPY poetry.lock pyproject.toml ./
RUN poetry install

# Run your app
COPY . /app
CMD [ "poetry", "run", "python", "-c", "print('Hello, World!')" ]

In Detail

Installing Poetry

How do I install poetry in my image? (should I use pip?)

Install it with pip

You should install poetry with pip. but you need to isolate it from the system interpreter and the project's virtual environment.

For maximum control in your CI environment, installation with pip is fully supported ... offers the best debugging experience, and leaves you subject to the fewest external tools.

ENV POETRY_VERSION=1.2.0
ENV POETRY_VENV=/opt/poetry-venv

# Install poetry separated from system interpreter
RUN python3 -m venv $POETRY_VENV \
    && $POETRY_VENV/bin/pip install -U pip setuptools \
    && $POETRY_VENV/bin/pip install poetry==${POETRY_VERSION}

# Add `poetry` to PATH
ENV PATH="${PATH}:${POETRY_VENV}/bin"

Poetry Version

Which version of poetry should I use?

Specify the latest stable version explicitly in your installation.

Forgetting to specify POETRY_VERSION will result in undeterministic builds, as the installer will always install the latest version - which may introduce breaking changes

Virtual Environment (virtualenv)

Do I need a virtual environment?

Yes, and you need to configure it a bit.

ENV POETRY_CACHE_DIR=/opt/.cache

The reasons for this are somewhat off topic:

By default, poetry creates a virtual environment in $HOME/.cache/pypoetry/virtualenvs to isolate the system interpreter from your application. This is the desired behavior for most development scenarios. When using a container, the $HOME variable may be changed by certain runtimes, so creating the virtual environment in an independent directory solves any reproducibility issues that may arise.

Bringing It All Together

To use poetry in a docker image you need to:

1. Install your desired version of poetry
2. Configure virtual environment location
3. Install your dependencies
4. Use poetry run python ... to run your application

A Working Example:

This is a minimal flask project managed with poetry.

You can copy these contents to your machine to test it out (expect for poerty.lock)

Project structure

python-poetry-docker/
|- Dockerfile
|- app.py
|- pyproject.toml
|- poetry.lock

Dockerfile

FROM python:3.10 as python-base

# https://python-poetry.org/docs#ci-recommendations
ENV POETRY_VERSION=1.2.0
ENV POETRY_HOME=/opt/poetry
ENV POETRY_VENV=/opt/poetry-venv

# Tell Poetry where to place its cache and virtual environment
ENV POETRY_CACHE_DIR=/opt/.cache

# Create stage for Poetry installation
FROM python-base as poetry-base

# Creating a virtual environment just for poetry and install it with pip
RUN python3 -m venv $POETRY_VENV \
    && $POETRY_VENV/bin/pip install -U pip setuptools \
    && $POETRY_VENV/bin/pip install poetry==${POETRY_VERSION}

# Create a new stage from the base python image
FROM python-base as example-app

# Copy Poetry to app image
COPY --from=poetry-base ${POETRY_VENV} ${POETRY_VENV}

# Add Poetry to PATH
ENV PATH="${PATH}:${POETRY_VENV}/bin"

WORKDIR /app

# Copy Dependencies
COPY poetry.lock pyproject.toml ./

# [OPTIONAL] Validate the project is properly configured
RUN poetry check

# Install Dependencies
RUN poetry install --no-interaction --no-cache --without dev

# Copy Application
COPY . /app

# Run Application
EXPOSE 5000
CMD [ "poetry", "run", "python", "-m", "flask", "run", "--host=0.0.0.0" ]

app.py

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, Docker!'

pyproject.toml

[tool.poetry]
name = "python-poetry-docker-example"
version = "0.1.0"
description = ""
authors = ["Someone <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.10"
Flask = "^2.1.2"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

poetry.lock

[[package]]
name = "click"
version = "8.1.3"
description = "Composable command line interface toolkit"
category = "main"
optional = false
python-versions = ">=3.7"

[package.dependencies]
... more lines ommitted

Full contents in gist.

Viperish

This answer turned out to be quite long, edits welcome to reduce mental overhead – Viperish

Your answer is very good and will written. I just disagree with the point to disable virtual environments in a docker image. venv's isolate the dependencies of your application from the one installed in the system. And those exists in a docker images the same way as in a "normal" system. – Velez

[off topic] @finswimmer, disabling virtualenv within docker actually solved a reproducibility bug I experienced with Google Cloud Run. In my first version, I left poetry with its default config, and it created a virtual environment during the build. Running locally with docker run worked as expected. When starting up the same image (same sha256) with Cloud Run poetry creates another virtualenv, empty of all dependencies and the app fails to start (due to a missing pacakage). This was kind of shocked about docker reproducibility in general, but wanted to post a valid solution. – Viperish

[still off topic] Apparently Cloud Run modify the HOME variable (🤯), that is the root cause of my reproducibility issue. – Viperish

For the most part, I like this approach; however, I'm curious about installing Poetry via pip rather than the official installer. What are the tradeoffs? – Curriery

(My answer is mostly taken from poetries CI recommendations.) @KyleAdams , The official installer does a lot of under-the-hood magic to install poetry, that is not desirable in a CI environment and might be harder to debug compared to installing with pip. You trade-off ease of use (official installer) for fine grained control on what happens inside your container. A trivial example: the contents of the installer change over time compared installing a pinned version with pip. – Viperish

The POETRY_HOME environment variable isn't used. The example code uses POETRY_VENV instead (the poetry CI instructions use POETRY_HOME). – Niobe

I am getting 127.0.0.1 refused to connect. Any suggestions? This site can’t be reached127.0.0.1 refused to connect. Try: Checking the connection Checking the proxy and the firewall ERR_CONNECTION_REFUSED – Bureaucratize

Answer 2

I prefer to use multistage builds so I can get rid of poetry in my actual release images and keep those images slim.

FROM python:3.10-slim AS builder

ENV POETRY_HOME="/opt/poetry" \
    POETRY_VIRTUALENVS_IN_PROJECT=1 \
    POETRY_NO_INTERACTION=1

# to run poetry directly as soon as it's installed
ENV PATH="$POETRY_HOME/bin:$PATH"

# install poetry
RUN apt-get update \
    && apt-get install -y --no-install-recommends curl \
    && curl -sSL https://install.python-poetry.org | python3 -

WORKDIR /app

# copy only pyproject.toml and poetry.lock file nothing else here
COPY poetry.lock pyproject.toml ./

# this will create the folder /app/.venv (might need adjustment depending on which poetry version you are using)
RUN poetry install --no-root --no-ansi --without dev

# ---------------------------------------------------------------------

FROM python:3.10-slim

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PATH="/app/.venv/bin:$PATH"

WORKDIR /app

# copy the venv folder from builder image 
COPY --from=builder /app/.venv ./.venv

Nari

This does not seem to work, : Couldn't import Django. Are you sure it's installed and available on your PYTHONPATH environment variable? Also, why did you change the .venv to venv? – Trierarch

@Trierarch you are right, I did not meant to rename .venv (typo) and I edited my answer – Nari

if add a command like poetry main.py, the packages installed in the virtualenv are not found – Showmanship

If we just COPY poetry.lock and pyproject.toml, where does the project specific Python code get copied? Should there be a "COPY . /app" line somewhere in the last image? – Niobe

@MikeDalrymple yes, do a COPY . . or something similar and adjust the PYTHONPATH if needed. – Nari

If you have a package in a src folder: /src/my_package. Don't forget to copy /src/my_package/__init__.py so poetry will install your package as well. – Zoochemistry

Answer 3

I provide a Python Docker Image with Poetry installed which is ready to use as builder base for your own images. Images are available for the last three Poetry and Python versions: https://github.com/max-pfeiffer/python-poetry

I also provide some other images which incorporate the already mentioned practices (virtual environment, multistage builds). There you can also find some additional examples on how to utilise Poetry and to build images for web applications:

• https://github.com/max-pfeiffer/uvicorn-poetry
• https://github.com/max-pfeiffer/uvicorn-gunicorn-poetry

Tyrr

Answer 4

Just an add-on to any of the ideas above.

you could split the poetry install into two steps.

To avoid source file changes busting RUN layer cache for the 3rd party dependency.

COPY pyproject.toml poetry.lock /app
RUN poetry install --only main --no-root --no-directory
COPY . /app
RUN poetry install --only main

Scruggs