Linking pyenv python to homebrew in order to avoid homebrew [email protected] installation

Question

Some packages from brew require [email protected] as a dependency. For example:

$ brew deps vim
gdbm
gettext
libyaml
lua
[email protected]
perl
[email protected]
readline
ruby
sqlite
xz

However, I want to manage all my python installations with pyenv and I would not like to download [email protected] with brew. This would imply having to exactly same versions of python installed in 2 different locations, which I would like to avoid. Reading up on brew, pyenv and python I´ve come to understand that having python installed in different parts of the system may cause some trouble in the future, which I'd also like to avoid. Unfortunately I cannot seem to resolve the python dependency in brew packages through pyenv. Below follow the steps I've tried to overcome this.

I have installed pyenv with brew, and necessary python versions from there.

$ pyenv versions
  system
* 3.8.2 (set by PYENV_VERSION environment variable)

I have tried solving this according to this Github discussion by setting a brew alias such as:

alias brew='env PATH=${PATH//$(pyenv root)\/shims:/} brew'

As that did not resolve the dependency issue, I tried create a link for [email protected] in /usr/local/Cellar which would point to the pyenv python, somehow similar to this issue with:

ln -s  ~/.pyenv/versions/3.8.2 $(brew --cellar python)@3.8

This did not work, so I have also tried adding the link to ´/usr/local/bin´:

ln -s  ~/.pyenv/versions/3.8.2 /usr/local/bin/[email protected]

However, running brew info vim still shows that the [email protected] dependency is not satisfied.

$ brew info vim
vim: stable 8.2.0900 (bottled), HEAD
Vi 'workalike' with many additional features
https://www.vim.org/
Conflicts with:
  ex-vi (because vim and ex-vi both install bin/ex and bin/view)
  macvim (because vim and macvim both install vi* binaries)
Not installed
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/vim.rb
==> Dependencies
Required: gettext ✘, lua ✘, perl ✘, [email protected] ✘, ruby ✘

Any ideas how can I link/connect my pyenv python installation to homebrew so that the additional [email protected] is not installed? Or maybe solve the issue in another manner using pyenv global/local/shell? I am currently using macOs Catalina.

Any help is greatly appreciated!

Sewole

Answer 1

2023-09-16 EDIT

Since brew version 4.1.0 (released 2023-07-20) there is a new command called pyenv-sync

$  brew pyenv-sync --help
Usage: brew pyenv-sync

Create symlinks for Homebrew's installed Python versions in ~/.pyenv/versions.

Note that older patch version symlinks will be created and linked to the minor
version so e.g. Python 3.11.0 will also be symlinked to 3.11.3.

  -d, --debug                      Display any debugging information.
  -q, --quiet                      Make some output more quiet.
  -v, --verbose                    Make some output more verbose.
  -h, --help                       Show this message.

While I haven't tried it, this sounds like the best option for linking Homebrew's Python with pyenv.

END OF EDIT

---

Unfortunately "unlinking" python from all formulae it is somewhat unnecessary as with each update you must "relink" them.

Take black, as an example:

$ ls -al $(brew --prefix black)/libexec/bin
total 104
drwxr-xr-x  16 thecesrom  staff   512 Jun 11 08:32 .
drwxr-xr-x   6 thecesrom  staff   192 Jun 11 08:32 ..
-rw-r--r--   1 thecesrom  staff  8834 Jun 10 15:27 Activate.ps1
-rw-r--r--   1 thecesrom  staff  1916 Jun 11 08:32 activate
-rw-r--r--   1 thecesrom  staff   865 Jun 11 08:32 activate.csh
-rw-r--r--   1 thecesrom  staff  2005 Jun 11 08:32 activate.fish
-rwxr-xr-x   1 thecesrom  staff   256 Jun 11 08:32 black
-rwxr-xr-x   1 thecesrom  staff   251 Jun 11 08:32 black-primer
-rwxr-xr-x   1 thecesrom  staff   257 Jun 11 08:32 blackd
-rwxr-xr-x   1 thecesrom  staff  1000 Jun 11 08:32 chardetect
-rwxr-xr-x   1 thecesrom  staff   257 Jun 11 08:32 pip
-rwxr-xr-x   1 thecesrom  staff   257 Jun 11 08:32 pip3
-rwxr-xr-x   1 thecesrom  staff   257 Jun 11 08:32 pip3.9
lrwxr-xr-x   1 thecesrom  staff    84 Jun 10 15:27 python -> ../../../../../opt/[email protected]/Frameworks/Python.framework/Versions/3.9/bin/python3.9
lrwxr-xr-x   1 thecesrom  staff    84 Jun 10 15:27 python3 -> ../../../../../opt/[email protected]/Frameworks/Python.framework/Versions/3.9/bin/python3.9
lrwxr-xr-x   1 thecesrom  staff    84 Jun 10 15:27 python3.9 -> ../../../../../opt/[email protected]/Frameworks/Python.framework/Versions/3.9/bin/python3.9

Notice that python, python3 and python3.9 all point to Homebrew's python dependency. So every time a formula depending on python is updated you must unlink all symlinks and recreate them pointing to your pyenv-installed version. But this still would not remove python downloaded via brew, and you would potentially end up with two installations of the same Python version in your machine.

A different way to see this problem is to add brew's python to pyenv.

As you can see, in my case [email protected] was installed as a dependency to black and other formulae, so this is what I did.

First, create a symlink at ~/.pyenv/versions.

$ cd ~/.pyenv/versions
$ ln -sfv "$(brew --prefix [email protected])" 3.9
$ ls -al
total 0
drwxr-xr-x  4 thecesrom  staff  128 Aug 31 07:51 .
drwxr-xr-x  6 thecesrom  staff  192 Jun  8 10:58 ..
drwxr-xr-x  6 thecesrom  staff  192 Aug 31 07:48 2.7.18
lrwxr-xr-x  1 thecesrom  staff   25 Aug 31 07:51 3.9 -> /usr/local/opt/[email protected]

2021-09-06 edit: I suggest using 3.9 to avoid having to change the symlink with every Python upgrade.

Since I wanted to include the same folders (bin, include, lib, share) as my other Python installation (2.7.18), which I installed via pyenv, then I created a symlink for include.

$ cd "$(brew --prefix [email protected])"
$ ln -sfv Frameworks/Python.framework/Versions/3.9/include/python3.9 include
include -> Frameworks/Python.framework/Versions/3.9/include/python3.9

Once I did that, I created symlinks for idle, pip, python and wheel at the following location:

$ cd "$(brew --prefix [email protected])/bin"
$ ln -sfv idle3 idle
idle -> idle3
$ ln -sfv pip3 pip 
pip -> pip3
$ ln -sfv python3 python
python -> python3
$ ln -sfv wheel3 wheel
wheel -> wheel3

I did that, because I got the following output when I ran brew link [email protected]:

$ brew link [email protected]
Warning: Already linked: /usr/local/Cellar/[email protected]/3.9.7
To relink, run:
  brew unlink [email protected] && brew link [email protected]

Once that's done, run pyenv rehash as recommended by pyenv whenever you install new versions.

$ pyenv rehash

Then, verify all versions managed by pyenv.

$ pyenv versions
  system
  2.7.18
* 3.9 (set by /Users/thecesrom/.pyenv/version)

And finally configure pyenv as needed. In my case I set 3.9 as global by running pyenv global 3.9, and now when I run python --version I get the following output:

$ python --version
Python 3.9.7

Ophthalmoscopy

Thank you for the detailed instructions of adding a Brew installation of Python to pyenv. This was perfect, the only caveat is that with the latest installation of Brew and Python 3.9.10, I only had to symlink the top level folder and nothing else. – Goddord

Answer 2

First of all, I appreciate the desire to keep redundancy to a minimum! I was sad to see no answers, but I think that I've figured it out. You can do this, though Homebrew warns against it and you're wading into the territory of relying on your own understanding to support this use case (as Homebrew states in their post). The SO issue you linked to has the right idea with the links, but from what I can tell they don't get the links quite right. I'm all for experimenting and learning (it's the only way?), so by all means go for it, but again you won't get much support if you're not able to dig into your own $PATH and follow the links that you've set up yourself.

If you already have any traces of Homebrew's python3, first get rid of them (brew unlink python3 and brew uninstall python3). If you already had [email protected] from Homebrew, when uninstalling it, note the packages that depended on it (for example, ffmpeg), and reinstall them after.

As of this post, Homebrew is expected Python 3.8.6 for it's [email protected], so first install that version with pyenv following their documentation:

pyenv install 3.8.6

That will put (by default) the actual Python install in

~/.pyenv/versions/3.8.6

Now we just need to add one link, and then let brew do the rest. I'll use full paths here, so you can run this from anywhere (and remember to read ln -s ... ... in your head as Link -Symbolic Target LinkName):

ln -s ~/.pyenv/versions/3.8.6 $(brew --cellar python)/3.8.6

With the -f flag, you could have omitted the trailing /3.8.6, as ln will use the name of the target. To be as explicit as possible on the link, you should have

ln -s ~/.pyenv/versions/3.8.6 /usr/local/Cellar/[email protected]/3.8.6

Here's what the result should look like:

➜  ~ ls -alF $(brew --cellar [email protected]) 
total 0
lrwxr-xr-x  1 [my username]  admin    36B Oct 14 16:52 3.8.6 -> 
/Users/[my username]/.pyenv/versions/3.8.6

Finally, let Homebrew manage the rest of necessary linking:

brew link python3

Pict

looks like this doesn't work anymore on Big Sur Homebrew still installs it's own python version 3.9.1_4 while Global Pyenv Python 3.9.0 was already linked with brew link python3. gist.github.com/mkschulze/b906afbad7be13e596a3d747a998ad0b – Chronograph

eek, though it looks like you're not matching the version all the way (3.9.1 is what homebrew wants). I tried to make it clear in the post, but in case I wasn't: this "solution" is very much a self-supported one...so good luck? – Pict

Well, so it’s homebrew that’s asks for a specific version or the depencie? Having Python3 linked would ideally be enough for any Python 3 requirements. However there is not such a version available with pyenv so, nice try? – Chronograph

Wish I could help more. I'd be surprised if pyenv doesn't have specifically 3.9.1 – Pict

I'm not sure I understand. Without an installation of Python by brew, there is no "/usr/local/Cellar/python@xxx" to link to. Or did I miss a step? – Patellate

No worries, just pointing out problems others might face as well. At least currently there is no 3.9.1 on pyenv, also that would mean I would have to install a different pyenv Python Version whenever a dependency asks for it, even while I can easily install it to any global Python3 version set with pyenv. So question remains, is it brew or the dependency that requires a specific Python Version here.@Patellate needs to be created I think, because it gets removed once you unlink and uninstall the brew version, by following above steps given that you initially have a brew Pyversion linked. – Chronograph

I just installed python3.9.1 on Macos with pyenv. 3.10-dev is even in the list. [pyenv install -l] – Patellate

It must have been added within the last two days then. – Chronograph

@Patellate yes you missed that linking step. – Pict

@Patellate how did your resolve the fact that was no python directory to put the link into after the instructions said to remove the homebrew version? Did you just create a directory using mkdir [email protected]? – Frons

@AndyReagan please clarify (or edit) that in your commands ll is your personal alias for ls -alF $(brew --cellar [email protected]) – Frons

@Frons good catch, yes ll is my alias. I edited the post. – Pict