Using Poetry with PyEnv and having Python version issues

Question

I'm working on WSL2 Ubuntu. I've been taking a course in backend API development using Fastapi. My Ubuntu default python is 3.8 I believe. I'm trying to use python 3.10.0 for my development. I've done the following:

1. pyenv install 3.10.0
2. cd to my project folder and pyenv local 3.10.0 (creates .python-version file containing 3.10.0)
3. poetry shell (this creates a .venv folder and all dependencies are added there during development). Bash Prompt has (.venv) in it.
4. python --version shows 3.10.0 as current version

My problem is that when I look into the .venv/bin file, all of the symbolic links show python 3.8.

drwxr-xr-x 2 mryan mryan 4096 Feb  1 23:12 .
drwxr-xr-x 4 mryan mryan 4096 Feb  1 23:12 ..
-rw-r--r-- 1 mryan mryan 2152 Feb  1 23:12 activate
-rw-r--r-- 1 mryan mryan 1444 Feb  1 23:12 activate.csh
-rw-r--r-- 1 mryan mryan 3033 Feb  1 23:12 activate.fish
-rw-r--r-- 1 mryan mryan 1303 Feb  1 23:12 activate.nu
-rw-r--r-- 1 mryan mryan 1754 Feb  1 23:12 activate.ps1
-rw-r--r-- 1 mryan mryan 1199 Feb  1 23:12 activate_this.py
-rw-r--r-- 1 mryan mryan  333 Feb  1 23:12 deactivate.nu
-rwxr-xr-x 1 mryan mryan  249 Feb  1 23:12 pip
-rwxr-xr-x 1 mryan mryan  249 Feb  1 23:12 pip-3.8
-rwxr-xr-x 1 mryan mryan  249 Feb  1 23:12 pip3
-rwxr-xr-x 1 mryan mryan  249 Feb  1 23:12 pip3.8
lrwxrwxrwx 1 mryan mryan   18 Feb  1 23:12 python -> /usr/bin/python3.8
lrwxrwxrwx 1 mryan mryan    6 Feb  1 23:12 python3 -> python
lrwxrwxrwx 1 mryan mryan    6 Feb  1 23:12 python3.8 -> python
-rwxr-xr-x 1 mryan mryan  236 Feb  1 23:12 wheel
-rwxr-xr-x 1 mryan mryan  236 Feb  1 23:12 wheel-3.8
-rwxr-xr-x 1 mryan mryan  236 Feb  1 23:12 wheel3
-rwxr-xr-x 1 mryan mryan  236 Feb  1 23:12 wheel3.8

When I try using the command poetry env use 3.10.0 I get the following:

(.venv) mryan ~/python/fastapi1/.venv/bin $poetry env use 3.10.0
/bin/sh: 1: python3.10: not found

  EnvCommandError

  Command python3.10 -c "import sys; print('.'.join([str(s) for s in sys.version_info[:3]]))" errored with the following return code 127, and output:


  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:378 in activate
       374│                     shell=True,
       375│                 )
       376│             )
       377│         except CalledProcessError as e:
    →  378│             raise EnvCommandError(e)
       379│
       380│         python_version = Version.parse(python_version.strip())
       381│         minor = "{}.{}".format(python_version.major, python_version.minor)
       382│         patch = python_version.text
(.venv) mryan ~/python/fastapi1/.venv/bin $python3.10
python3.10: command not found

What am I doing wrong here? I feel I'm missing something significant here. I've been through a zillion web pages on poetry and virtual environments.

Mulligrubs

Which version of poetry are you using? – Colton

I'm using version 1.1.12 installed via the curl method as shown on the poetry web site. – Mulligrubs

Btw Mark, do you create a new poetry project using 'poetry new' or 'poetry init'? That might make a difference. – Dumbstruck

Answer 1

I had the same problem you are having, I used to be able to use:

pyenv local 3.10.0

poetry new example-project

And poetry would create example-project with a venv in it based on python 3.10.0. This no longer worked for me so I found that these commands did work for me:

poetry new example-project
cd example-project/
poetry env use 3.10.0
poetry update

In case your project already had a venv installed it might be a good idea to remove that first before using the poetry env command. I'm pretty sure you haven't added pyenv to your PATH properly. Check out the installation instructions for pyenv, specifically the 'Configure your shell's environment for Pyenv' section.

Dumbstruck

I have to apologize, I want to put in code and bash comments to answer your questions but three ticks doesn't seem to format and my space is limited. I'm researching how to provide information. – Mulligrubs

Unfortunately I get the EnvCommandError as shown in my post anytime I use the Poetry env use 3.10.0. I can do a pyenv local 3.10.0 without any issue and the project uses it. But using the poetry env use command errors out. I do have pyenv and poetry setup in my bash login script. For poetry, I make sure $HOME/.local/bin is added to the path. For pyenv I use the example from the pyenv site. I've been doing this all along. – Mulligrubs

I've also confirmed that pyenv versions lists my current version as 3.10.0 (downloaded and set with pyenv local 3.10.0) and I can run python --version and get 3.10.0. If I back out of the project folder, I get 3.8.1 (wsl Ubuntu default). – Mulligrubs

Also to mention it, I have the following in my bash login script.. alias python='pyenv exec python3' alias py='python'. I'm going to try making a very bare minimum bash login script and see if maybe I'm doing something to create the error. – Mulligrubs

Good idea, try resetting some 'dot-files' (.bashrc / .bash_profile / .profile) and try adding pyenv to PATH again to see if it changes anything, keep me posted. Ubuntu probably stores defaults of these dot-files somewhere, I know Linux Mint does. – Dumbstruck

I've just replicated your exact error and I'll post another answer to your initial question. – Dumbstruck

I had some similar problems that were mostly pyenv not poetry and worked through it #71454522 – Disentangle

Thanks! For me I hadn't set up my PATH properly for pyenv, and followed your linked installation instructions to fix it. – Rintoul

Answer 2

I am surprised that poetry shell creates .venv. Are you certain about that? Or did you maybe create it with virtualenv?

I just kind of replicated what you probably did and did not get the error. I would recommend you to create a minimal, reproducible example. Then you will probably catch what is wrong. For this I case I recommend creating a bash script. A start could be:

mkdir ~/python/test_error
cd ~/python/test_error
cp ~/python/fastapi1/pyproject.toml .
pyenv local 3.10.0
poetry env use 3.10.0

This script works for me (when I change the folders).

Colton

It creates it in the project folder because I setup poetry to do so with the following command 'poetry config virtualenvs.in-project true' – Mulligrubs

If I follow your example above, I get the same error when I issue the 'poetry env use 3.10.0' command. < Command python3.10 -c "import sys; print('.'.join([str(s) for s in sys.version_info[:3]]))" errored with the following return code 127, and output:>. If I try executing python --version, I get 3.10 (I set the pyenv global 3.10). If I try python3.10 I get command not found. If I enter pyenv exec python3.10 --version then I get the expected version number. – Mulligrubs

This solved the issue for me, the old flow in the poetry docs no longer works github.com/python-poetry/poetry/issues/4317 – Atilt

Answer 3

Thanks SWHJ, I also come up with that solution. Just a small point here is we need to set this config:

poetry config virtualenvs.in-project true 

so poetry will create a .venv inside the project's folder.

Interlocutrix

There is a another poetry config command that tells poetry to use the local environment Python in your.venv folder instead of the installed Python for your operating system. poetry config virtualenvs.in-project true poetry config virtualenvs.prefer-active-python true Additionally: Type poetry env use 3.9.15 Or whatever version you have installed. – Mulligrubs

Answer 4

I'm able to replicate your exact error using these commands:

poetry new test-project

cd test-project/

poetry env use 3.10

Result in the exact same error you are getting. However, this works for me:

poetry new test-project

cd test-project/

pyenv local 3.10

poetry env use 3.10

Following these commands my poetry will create a .venv inside the project's folder successfully. By the way, you'll have to edit the pyproject.toml file manually, because that doesn't change using this method.

If this doesn't work I suspect something went wrong while adding pyenv to your PATH. In that case you could try resetting your 'dot-files' (.bashrc, .bash_profile and/or .profile) and add pyenv again using these commands from the pyenv github:

# the sed invocation inserts the lines at the start of the file
# after any initial comment lines
sed -Ei -e '/^([^#]|$)/ {a \
export PYENV_ROOT="$HOME/.pyenv"
a \
export PATH="$PYENV_ROOT/bin:$PATH"
a \
' -e ':a' -e '$!{n;ba};}' ~/.profile
echo 'eval "$(pyenv init --path)"' >>~/.profile

echo 'eval "$(pyenv init -)"' >> ~/.bashrc

Dumbstruck