How to set target hosts in Fabric file

Question

I want to use Fabric to deploy my web app code to development, staging and production servers. My fabfile:

def deploy_2_dev():
  deploy('dev')

def deploy_2_staging():
  deploy('staging')

def deploy_2_prod():
  deploy('prod')

def deploy(server):
  print 'env.hosts:', env.hosts
  env.hosts = [server]
  print 'env.hosts:', env.hosts

Sample output:

host:folder user$ fab deploy_2_dev
env.hosts: []
env.hosts: ['dev']
No hosts found. Please specify (single) host string for connection:

When I create a set_hosts() task as shown in the Fabric docs, env.hosts is set properly. However, this is not a viable option, neither is a decorator. Passing hosts on the command line would ultimately result in some kind of shell script that calls the fabfile, I would prefer having one single tool do the job properly.

It says in the Fabric docs that 'env.hosts is simply a Python list object'. From my observations, this is simply not true.

Can anyone explain what is going on here ? How can I set the host to deploy to ?

Dag

I have the same problem, have you found any solution to this? – Rubinstein

to run the same task against multiple servers, use "fab -H staging-server,production-server deploy"... more in my answer below: https://mcmap.net/q/195199/-how-to-set-target-hosts-in-fabric-file – Mendelism

Try this: docs.fabfile.org/en/1.13/usage/env.html#passwords – Colliery

This answer does not apply to fabric 2+. If someone more familiar with Stackoverflow conventions could edit the question or question title to refer to fabric 1 it might be helpful. – Pycno

Answer 1

I do this by declaring an actual function for each environment. For example:

def test():
    env.user = 'testuser'
    env.hosts = ['test.server.com']

def prod():
    env.user = 'produser'
    env.hosts = ['prod.server.com']

def deploy():
    ...

Using the above functions, I would type the following to deploy to my test environment:

fab test deploy

...and the following to deploy to production:

fab prod deploy

The nice thing about doing it this way is that the test and prod functions can be used before any fab function, not just deploy. It is incredibly useful.

Orlop

Due to a bug in fabric (code.fabfile.org/issues/show/138#change-1497) it is better to include user in host string (like [email protected]) instead of setting env.user. – Sudatorium

I had the same problem, and this seems like the best solution. I define the hosts, user and a lot of other settings in a YAML file that is loaded by the dev() and prod() functions. (So that I can reuse the same Fabric script for similar projects.) – Dude

@MikhailKorobov: When I followed your link, I saw "Welcome to nginx!". All the requests to code.fabfile.org domain have responses like that. – Therefrom

Yeah, it seems all bugs were migrated to github. – Sudatorium

Unfortunately, it looks like this no longer works - fabric won't run tasks without env.hosts already defined, and won't run functions in the fab A B C style without them being defined as tasks. – Exclusive

docs.fabfile.org/en/1.10/usage/… claims it still works – Dasha

I'm using 1.11.1 and can confirm this approach does work for me. – Dichromic

Answer 2

Use roledefs

from fabric.api import env, run

env.roledefs = {
    'test': ['localhost'],
    'dev': ['[email protected]'],
    'staging': ['[email protected]'],
    'production': ['[email protected]']
} 

def deploy():
    run('echo test')

Choose role with -R:

$ fab -R test deploy
[localhost] Executing task 'deploy'
...

Orelu

Or if the task is always run on the same role, you can use the @roles() decorator on the task. – Inordinate

Sounds like roledefs is a better solution than defining them in separate tasks. – Tully

Does anybody know how I can include a password for the provided username in a roledef? A further dictionary entry 'password': 'some_password' seems to be ignored and leads to a prompt at runtime. – Szechwan

@Szechwan you can use env.passwords which is a dictionary containing user+host+port as key and password as value. E.g. env.passwords={'user@host:22' : 'password'} – Whitman

Answer 3

Here's a simpler version of serverhorror's answer:

from fabric.api import settings

def mystuff():
    with settings(host_string='192.0.2.78'):
        run("hostname -f")

Nimrod

Per the docs, the settings context manager is for overriding env variables, not for setting them initially. I think using roledefs, as thomie suggested, is more appropriate for defining hosts like stage, dev and test. – Eccrinology

Answer 4

Was stuck on this myself, but finally figured it out. You simply can't set the env.hosts configuration from within a task. Each task is executed N times, once for each Host specified, so the setting is fundamentally outside of task scope.

Looking at your code above, you could simply do this:

@hosts('dev')
def deploy_dev():
    deploy()

@hosts('staging')
def deploy_staging():
    deploy()

def deploy():
    # do stuff...

Which seems like it would do what you're intending.

Or you can write some custom code in the global scope that parses the arguments manually, and sets env.hosts before your task function is defined. For a few reasons, that's actually how I've set mine up.

Skulk

Found a way: from fabric.api import env; env.host_string = "dev" – Austerity

Answer 5

Since fab 1.5 this is a documented way to dynamically set hosts.

http://docs.fabfile.org/en/1.7/usage/execution.html#dynamic-hosts

Quote from the doc below.

Using execute with dynamically-set host lists

A common intermediate-to-advanced use case for Fabric is to parameterize lookup of one’s target host list at runtime (when use of Roles does not suffice). execute can make this extremely simple, like so:

from fabric.api import run, execute, task

# For example, code talking to an HTTP API, or a database, or ...
from mylib import external_datastore

# This is the actual algorithm involved. It does not care about host
# lists at all.
def do_work():
    run("something interesting on a host")

# This is the user-facing task invoked on the command line.
@task
def deploy(lookup_param):
    # This is the magic you don't get with @hosts or @roles.
    # Even lazy-loading roles require you to declare available roles
    # beforehand. Here, the sky is the limit.
    host_list = external_datastore.query(lookup_param)
    # Put this dynamically generated host list together with the work to be
    # done.
    execute(do_work, hosts=host_list)

Bolger

+1. A lot of really good answers toward the bottom of the page here. – Poree

Answer 6

Contrary to some other answers, it is possible to modify the env environment variables within a task. However, this env will only be used for subsequent tasks executed using the fabric.tasks.execute function.

from fabric.api import task, roles, run, env
from fabric.tasks import execute

# Not a task, plain old Python to dynamically retrieve list of hosts
def get_stressors():
    hosts = []
    # logic ...
    return hosts

@task
def stress_test():
    # 1) Dynamically generate hosts/roles
    stressors = get_stressors()
    env.roledefs['stressors'] = map(lambda x: x.public_ip, stressors)

    # 2) Wrap sub-tasks you want to execute on new env in execute(...)
    execute(stress)

    # 3) Note that sub-tasks not nested in execute(...) will use original env
    clean_up()

@roles('stressors')
def stress():
    # this function will see any changes to env, as it was wrapped in execute(..)
    run('echo "Running stress test..."')
    # ...

@task
def clean_up():
    # this task will NOT see any dynamic changes to env

Without wrapping sub-tasks in execute(...), your module-level env settings or whatever is passed from the fab CLI will be used.

Choreographer

This is the best answer if you want to dynamically set env.hosts. – Carious

Answer 7

You need to set host_string an example would be:

from fabric.context_managers import settings as _settings

def _get_hardware_node(virtualized):
    return "localhost"

def mystuff(virtualized):
    real_host = _get_hardware_node(virtualized)
    with _settings(
        host_string=real_host):
        run("echo I run on the host %s :: `hostname -f`" % (real_host, ))

Rubinstein

Sweet. I've posted a simpler version of the code in another answer here. – Nimrod

Answer 8

To explain why it's even an issue. The command fab is leveraging fabric the library to run the tasks on the host lists. If you try and change the host list inside a task, you're esentially attempting to change a list while iterating over it. Or in the case where you have no hosts defined, loop over an empty list where the code where you set the list to loop over is never executed.

The use of env.host_string is a work around for this behavior only in that it's specifying directly to the functions what hosts to connect with. This causes some issues in that you'll be remaking the execution loop if you want to have a number of hosts to execute on.

The simplest way the people make the ability to set hosts at run time, is to keep the env populatiing as a distinct task, that sets up all the host strings, users, etc. Then they run the deploy task. It looks like this:

fab production deploy

or

fab staging deploy

Where staging and production are like the tasks you have given, but they do not call the next task themselves. The reason it has to work like this, is that the task has to finish, and break out of the loop (of hosts, in the env case None, but it's a loop of one at that point), and then have the loop over the hosts (now defined by the preceding task) anew.

Delatorre

Answer 9

You need to modify env.hosts at the module level, not within a task function. I made the same mistake.

from fabric.api import *

def _get_hosts():
    hosts = []
    ... populate 'hosts' list ...
    return hosts

env.hosts = _get_hosts()

def your_task():
    ... your task ...

Farmhouse

Answer 10

It's very simple. Just initialize the env.host_string variable and all of the following commands will be executed on this host.

from fabric.api import env, run

env.host_string = '[email protected]'

def foo:
    run("hostname -f")

Workwoman

Answer 11

I'm totally new to fabric, but to get fabric to run the same commands on multiple hosts (e.g. to deploy to multiple servers, in one command) you can run:

fab -H staging-server,production-server deploy 

where staging-server and production-server are 2 servers you want to run the deploy action against. Here's a simple fabfile.py that will display the OS name. Note that the fabfile.py should be in the same directory as where you run the fab command.

from fabric.api import *

def deploy():
    run('uname -s')

This works with fabric 1.8.1 at least.

Mendelism

Answer 12

So, in order to set the hosts, and have the commands run across all the hosts, you have to start with:

def PROD():
    env.hosts = ['10.0.0.1', '10.0.0.2']

def deploy(version='0.0'):
    sudo('deploy %s' % version)

Once those are defined, then run the command on the command line:

fab PROD deploy:1.5

What will run the deploy task across all of the servers listed in the PROD function, as it sets the env.hosts before running the task.

Tsan

Suppose the deployment on the first host worked but the one on the second failed, how do I do it again only on the second one? – Oligarchy

Answer 13

You can assign to env.hoststring before executing a subtask. Assign to this global variable in a loop if you want to iterate over multiple hosts.

Unfortunately for you and me, fabric is not designed for this use case. Check out the main function at http://github.com/bitprophet/fabric/blob/master/fabric/main.py to see how it works.

Edeline

Answer 14

Here's another "summersault" pattern that enables the fab my_env_1 my_command usage:

With this pattern, we only have to define environments one time using a dictionary. env_factory creates functions based on the keynames of ENVS. I put ENVS in its own directory and file secrets.config.py to separate config from the fabric code.

The drawback is that, as written, adding the @task decorator will break it.

Notes: We use def func(k=k): instead of def func(): in the factory because of late binding. We get the running module with this solution and patch it to define the function.

secrets.config.py

ENVS = {
    'my_env_1': {
        'HOSTS': [
            'host_1',
            'host_2',
        ],
        'MY_OTHER_SETTING': 'value_1',
    },
    'my_env_2': {
        'HOSTS': ['host_3'],
        'MY_OTHER_SETTING': 'value_2'
    }
}

fabfile.py

import sys
from fabric.api import env
from secrets import config


def _set_env(env_name):
    # can easily customize for various use cases
    selected_config = config.ENVS[env_name]
    for k, v in selected_config.items():
        setattr(env, k, v)


def _env_factory(env_dict):
    for k in env_dict:
        def func(k=k):
            _set_env(k)
        setattr(sys.modules[__name__], k, func)


_env_factory(config.ENVS)

def my_command():
    # do work

Reproof

Answer 15

Using roles is currently considered to be the "proper" and "correct" way of doing this and is what you "should" do it.

That said, if you are like most of what you "would like" or "desire" is the ability to perform a "twisted syster" or switching target systems on the fly.

So for entertainment purposes only (!) the following example illustrates what many might consider to a risky, and yet somehow thoroughly satisfying, manoeuvre that goes something like this:

env.remote_hosts       = env.hosts = ['10.0.1.6']
env.remote_user        = env.user = 'bob'
env.remote_password    = env.password = 'password1'
env.remote_host_string = env.host_string

env.local_hosts        = ['127.0.0.1']
env.local_user         = 'mark'
env.local_password     = 'password2'

def perform_sumersault():
    env_local_host_string = env.host_string = env.local_user + '@' + env.local_hosts[0]
    env.password = env.local_password
    run("hostname -f")
    env.host_string = env.remote_host_string
    env.remote_password = env.password
    run("hostname -f")

Then running:

fab perform_sumersault

Eyestrain