Activate a virtualenv via fabric as deploy user

Question

I want to run my fabric script locally, which will in turn, log into my server, switch user to deploy, activate the projects .virtualenv, which will change dir to the project and issue a git pull.

def git_pull():
    sudo('su deploy')
    # here i need to switch to the virtualenv
    run('git pull')

I typically use the workon command from virtualenvwrapper which sources the activate file and the postactivate file will put me in the project folder. In this case, it seems that because fabric runs from within shell, control is give over to fabric, so I can't use bash's source built-in to '$source ~/.virtualenv/myvenv/bin/activate'

Anybody have an example and explanation of how they have done this?

Slight

Out of curiosity, why aren't you using workon as a prefix? – Clincher

Answer 1

Right now, you can do what I do, which is kludgy but works perfectly well* (this usage assumes you're using virtualenvwrapper -- which you should be -- but you can easily substitute in the rather longer 'source' call you mentioned, if not):

def task():
    workon = 'workon myvenv && '
    run(workon + 'git pull')
    run(workon + 'do other stuff, etc')

Since version 1.0, Fabric has a prefix context manager which uses this technique so you can for example:

def task():
    with prefix('workon myvenv'):
        run('git pull')
        run('do other stuff, etc')

---

* There are bound to be cases where using the command1 && command2 approach may blow up on you, such as when command1 fails (command2 will never run) or if command1 isn't properly escaped and contains special shell characters, and so forth.

Croaky

But workon is unknown by sh. How can we tell fabric to use bash instead ? – Symphonia

IMHO you should just use source venv/bin/activate. It's easier and works out of the box. workon is an additional dependency and even if it's installed you have to add it in .bashrc - too complicated for fabric deploys. – Tanguy

@PierredeLESPINAY see #11272872 for a solution for your problem. – Loincloth

Answer 2

As an update to bitprophet's forecast: With Fabric 1.0 you can make use of prefix() and your own context managers.

from __future__ import with_statement
from fabric.api import *
from contextlib import contextmanager as _contextmanager

env.hosts = ['servername']
env.user = 'deploy'
env.keyfile = ['$HOME/.ssh/deploy_rsa']
env.directory = '/path/to/virtualenvs/project'
env.activate = 'source /path/to/virtualenvs/project/bin/activate'

@_contextmanager
def virtualenv():
    with cd(env.directory):
        with prefix(env.activate):
            yield

def deploy():
    with virtualenv():
        run('pip freeze')

Valorie

@simon, by writing your own prefix method that calls .bashrc and wraps both the prefix and the command within the -c argument for bash. See below – Volumed

But source is unknown by sh. How can we tell fabric to use bash instead ? – Symphonia

@PierredeLESPINAY you can use . instead of source – Fasten

Why do you use cd() when you're fully specifying the path to activate in prefix()? – Falcongentle

@NickT Because prefix() doesn't seem to cd there - see these docs which do the same. We want to cd there so that when we yield to execute other commands (pip freeze in my example), those commands can be relative to that directory. – Valorie

Answer 3

I'm just using a simple wrapper function virtualenv() that can be called instead of run(). It doesn't use the cd context manager, so relative paths can be used.

def virtualenv(command):
    """
    Run a command in the virtualenv. This prefixes the command with the source
    command.
    Usage:
        virtualenv('pip install django')
    """
    source = 'source %(project_directory)s/bin/activate && ' % env
    run(source + command)

Driskill

Answer 4

virtualenvwrapper can make this a little simpler

1. Using @nh2's approach (this approach also works when using local, but only for virtualenvwrapper installations where workon is in $PATH, in other words -- Windows)

   from contextlib import contextmanager
   from fabric.api import prefix
   
   @contextmanager
   def virtualenv():
       with prefix("workon env1"):
           yield
   
   def deploy():
       with virtualenv():
           run("pip freeze > requirements.txt")
   

2. Or deploy your fab file and run this locally. This setup lets you activate the virtualenv for local or remote commands. This approach is powerful because it works around local's inability to run .bashrc using bash -l:

   @contextmanager
   def local_prefix(shell, prefix):
       def local_call(command):
           return local("%(sh)s \"%(pre)s && %(cmd)s\"" % 
               {"sh": shell, "pre": prefix, "cmd": command})
       yield local_prefix
   
   def write_requirements(shell="/bin/bash -lic", env="env1"):
       with local_prefix(shell, "workon %s" % env) as local:
           local("pip freeze > requirements.txt")
   
   write_requirements()  # locally
   run("fab write_requirements")
   

Volumed

Thanks for summarizing nh2's answer, virtualenv contextmanager declaration could be done in 5 lines on Python 2.6+, however it is never guaranteed that 'workon' alias is always imported correctly, and it's much more reliable to use `source .../activate' command – Artel

Answer 5

This is my approach on using virtualenv with local deployments.

Using fabric's path() context manager you can run pip or python with binaries from virtualenv.

from fabric.api import lcd, local, path

project_dir = '/www/my_project/sms/'
env_bin_dir = project_dir + '../env/bin/'

def deploy():
    with lcd(project_dir):
        local('git pull origin')
        local('git checkout -f')
        with path(env_bin_dir, behavior='prepend'):
            local('pip freeze')
            local('pip install -r requirements/staging.txt')
            local('./manage.py migrate') # Django related

            # Note: previous line is the same as:
            local('python manage.py migrate')

            # Using next line, you can make sure that python 
            # from virtualenv directory is used:
            local('which python')

Westerman

I like this very much -- I don't see any obvious disadvantages to this approach, and it's very clean. Thanks :) – Lagerkvist

still the best and most clean answer here – Arredondo

Answer 6

Thanks to all answers posted and I would like to add one more alternative for this. There is an module, fabric-virtualenv, which can provide the function as the same code:

>>> from fabvenv import virtualenv
>>> with virtualenv('/home/me/venv/'):
...     run('python foo')

fabric-virtualenv makes use of fabric.context_managers.prefix, which might be a good way :)

Popularize

Interesting but I don't like the fact that there is not link to SCM/issue tracker. A package that is only published on PYPI without a link to source code and and issue tracker does not inspire much trust.... but is easy to fix. – Manno

Answer 7

If you want to install the packages to environment or want to run commands according to the packages you have in environment, I have found this hack to solve my problem, instead of writing complex methods of fabric or installing new OS packages:

/path/to/virtualenv/bin/python manage.py migrate/runserver/makemigrations  # for running commands under virtualenv

local("/home/user/env/bin/python manage.py migrate")    # fabric command


/path/to/virtualenv/bin/pip install -r requirements.txt   # installing/upgrading virtualenv

local("/home/user/env/bin/pip install -r requirements.txt")  #  fabric command

This way you might not need to activate the environment, but you can execute commands under the environment.

Additive

Answer 8

Here is code for a decorator that will result in the use of Virtual Environment for any run/sudo calls:

# This is the bash code to update the $PATH as activate does
UPDATE_PYTHON_PATH = r'PATH="{}:$PATH"'.format(VIRTUAL_ENV_BIN_DIR)

def with_venv(func, *args, **kwargs):
  "Use Virtual Environment for the command"

  def wrapped(*args, **kwargs):
    with prefix(UPDATE_PYTHON_PATH):
      return func(*args, **kwargs)

  wrapped.__name__ = func.__name__
  wrapped.__doc__ = func.__doc__
  return wrapped

and then to use the decorator, note the order of the decorators is important:

@task
@with_venv
def which_python():
  "Gets which python is being used"
  run("which python")

Glass

Answer 9

This approach worked for me, you can apply this too.

from fabric.api import run 
# ... other code...
def install_pip_requirements():
    run("/bin/bash -l -c 'source venv/bin/activate' "
        "&& pip install -r requirements.txt "
        "&& /bin/bash -l -c 'deactivate'")

Assuming venv is your virtual env directory and add this method wherever appropriate.

Whimsy

Answer 10

I use pyenv with plugin pyenv-virtualenvwrapper. I had no success with workon, instead I use this (fabric 2.5):

with c.prefix('source /home/mirek/.virtualenvs/%s/bin/activate' % PROJECT):
    with c.prefix('cd /home/mirek/%s/%s' % (PROJECT, PROJECT)):
        c.run('python manage.py ....')

For git pull agent forwarding is good, ie. ssh -A .. or better in ~/.ssh/config something like this:

Host forpsi
    HostName xx.xx.xx.xx
    IdentityFile /home/mirek/.ssh/id_ed25519_xxx
    ForwardAgent yes

And now on the development machine if you have a private key in the agent (after ssh-add or if you have AddKeysToAgent yes in ~/.ssh/config after you made git push) then git pull should not ask for the passphrase of the key.

Megalopolis

Works perfect with Pyenv. Thanks – Westbound