Trouble activating virtualenv on server via Fabric

Question

I am trying to run some Django management commands via Fabric on my staging server.

The problem is it seems Fabric is not able to activate the virtualenv and thus using system python/libs when executing the commands.

On the server the Django app is run using a virtualenv (no, I don' use virtualenvwrapper yet...)

Using Fabric (1.0.1) a command might look like this when run from my box:

The fabfile method:

def collectstatic():
    require('settings', provided_by=[production, staging])

    with settings(warn_only=True):
        run('source %(env_path)s/bin/activate && python %(repo_path)s/%(project_name)s/configs/%(settings)s/manage.py collectstatic --noinput -v0' % env)

The output:

$ fab staging master collectstatic
[myserver.no] Executing task 'master'
[myserver.no] Executing task 'collectstatic'
[myserver.no] run: source /home/newsapps/sites/mysite/env/bin/activate && python /home/newsapps/sites/mysite/repository/mysite/configs/staging/manage.py collectstatic --noinput -v0
[myserver.no] Login password: 
[myserver.no] out: Unknown command: 'collectstatic'
[myserver.no] out: Type 'manage.py help' for usage.

I know of course that the Django command collectstatic does not exist in versions prior to 1.3 which leads med to think that system python (which has Django 1.2) is beeing used.

My fabfile/project layout is based on the great fabfile of the Tribapps guys

So I created a fabric method to test pythonversion:

def pythonver():
    require('settings', provided_by=[production, staging])

    with settings(warn_only=True):

    run('source %(env_path)s/bin/activate && echo "import sys; print sys.path" | python ' % env)

When run it gives the following output:

$ fab staging master pythonver
[myserver.no] Executing task 'master'
[myserver.no] Executing task 'pythonver'
[myserver.no] run: source /home/newsapps/sites/mysite/env/bin/activate && echo "import sys; print sys.path" | python 
[myserver.no] Login password: 
[myserver.no] out: ['', '/usr/lib/python2.6', '/usr/lib/python2.6/plat-linux2', '/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', '/usr/lib/python2.6/lib-dynload', '/usr/lib/python2.6/dist-packages', '/usr/lib/pymodules/python2.6', '/usr/lib/pymodules/python2.6/gtk-2.0', 

As you can see it uses system python and not my virtualenv located in home/newsapps/sites/mysite/env

But if I run this command directly on the server

source /home/newsapps/sites/mysite/env/bin/activate && echo "import sys; print sys.path" | python 

.. then it outputs the right paths from the virtualenv

What am I doing wrong since the commands are not run with the python from my virtualenv using Fabric?

Fulmar

Answer 1

You should call the python version from your virtualenv bin directory, then you will be sure it uses the virtualenv's version of python.

/home/newsapps/sites/mysite/env/bin/python /home/newsapps/sites/mysite/repository/mysite/configs/staging/manage.py collectstatic --noinput -v0

Rosinski

@lajarre -v VERBOSITY, --verbosity=VERBOSITY Verbosity level; 0=minimal output, 1=normal output, 2=verbose output, 3=very verbose output – Rosinski

Answer 2

I wouldn't bother with activating the virtualenv, just give the full path to the virtualenv's python interpreter. That will then use the correct PYTHONPATH, etc.

Kentish

I am adding my additional PYTHONPATH entries (such as your project's root directory) in my virtualenv's postactivate file. I would assume this method doesn't add that. Should I be placing these someplace else? – Counterattraction

You can add <name>.pth files to your <virtualenv>/lib/python2.x/site-packages/ directory. These can contain a path that will be added to your PYTHONPATH. I do this as part of my deployment process. – Kentish

Or, you can make your project into an installable package (give it a setup.py file), and install it using pip -e, which kind of does the same thing. – Kentish

Answer 3

I had the same problem. Couldn't solve it the easy way. So I just used the full path to the python bin file inside the virtualenv. I'm not a pro in Python, but I guess it's the same thing in the end. It goes something like this in my fab file:

PYTHON = '/home/dudus/.virtualenvs/pai/bin/python'
PIP = '/home/dudus/.virtualenvs/pai/bin/pip'

def update_db():
    with cd(REMOTE_DIR + 'application/'):
        run('%s ./manage.py syncdb --settings="%s"' % 
            (PYTHON, SETTINGS)) # syncdb
        run('%s ./manage.py migrate --settings="%s"' % 
            (PYTHON, SETTINGS)) # south migrate

Saudra

Answer 4

This will work perfectly :)

from __future__ import with_statement
from fabric.api import *
from contextlib import contextmanager as _contextmanager

env.hosts = ['servername']
env.user = 'username' 
env.directory = '/path/to/virtualenvs/project' 
env.activate = 'source /path/to/virtualenvs/project/bin/activate'

@_contextmanager
def virtualenv():
    with cd(env.directory):
        with prefix(env.activate):
            yield

def deploy():
    with virtualenv():
        run('pip freeze')

Readability

Answer 5

This approach worked for me, you can apply this too.

from fabric.api import run 
# ... other code...
def install_pip_requirements():
    run("/bin/bash -l -c 'source venv/bin/activate' "
        "&& pip install -r requirements.txt "
        "&& /bin/bash -l -c 'deactivate'")

Assuming venv is your virtual env directory and add this method wherever appropriate.

Hellcat