How can you "source" an environment inside a Buildbot step?

Question

Within Buildbot I need to be able to "source" an environment before doing a compilation step.

If I was building the application from command line using bash I would have to do:

. envrionment-set-up-script
build_command

Within the build bot master.cfg file I have tried the following:

factory.addStep(ShellCommand(command=["source","environment-set-up-script"])
factory.addStep(ShellCommand(command=[".","environment-set-up-script"]))
factory.addStep(Configure(command=["source","environment-set-up-script"]))
factory.addStep(Configure(command=[".","environment-set-up-script"]))

All of which fail, this is because the command cannot be found, which makes sense as it is a bash builtin.

Also I do not think that this is the correct approach as the environment would not necessarily be used when the next step of the factory is called.

Ninebark

Answer 1

After some experimenting I have found a way in which to achieve this. You need to:

• run a bash sub-shell setting the environment that should be used for that shell, i.e. call bash with the environment variable BASH_ENV set to the file that should be sourced into the environment.
• run the env command in bash to capture the environment
• parse the result of the env command into a property (using a SetProperty step)
• use the property within further steps as the env parameter

Note: that the environment should be parsed as a dictionary that can be used as an env parameter

    from buildbot.process.factory import BuildFactory
    from buildbot.steps.shell import ShellCommand, SetProperty
    from buildbot.process.properties import Property  

    def glob2list(rc, stdout, stderr):
        ''' Function used as the extrat_fn function for SetProperty class
            This takes the output from env command and creates a dictionary of 
            the environment, the result of which is stored in a property names
            env'''
        if not rc:
            env_list = [ l.strip() for l in stdout.split('\n') ]
            env_dict={ l.split('=',1)[0]:l.split('=',1)[1] for l in 
                          env_list if len(l.split('=',1))==2}
            return {'env':env_dict}

    #This is the equivalent of running source MyWorkdir/my-shell-script then  
    #capturing the environment afterwords.
    factory.addStep(SetProperty(command="bash -c env",
                extract_fn=glob2list,       
                workdir='MyWorkdir',
                env={BASH_ENV':'my-shell-script' }))

    #Do another step with the environment that was sourced from 
    #MyWorkdir/my-shell-script
    factory.addStep(ShellCommand(command=["env"],
                workdir="MyWorkdir",
                env=Property('env')))

Ninebark

The only case that might fail is if the sources script starts some process that a) doesn't survive its shell dying, and b) is referenced by the environment. – Igorot

That's pretty intense, compared to some of the simpler options described here, but certainly gives you a lot of flexibility. Consider generalizing this and submitting it as a patch to Buildbot? – Timmy

Answer 2

While working with OpenEmbedded/Yocto, we solved the problem in a way similar to this:

class EnvStep(ShellCommand):
    def __init__(self, command='', **kwargs):
        kwargs['command'] = [
            'bash', '-c', 'source environment-set-up-script; %s' % command
        ]
        ShellCommand.__init__(self, **kwargs)

Then, adding an EnvStep that sets its command parameter to foo lets us run foo within the environment sourced with environment-set-up-script. In other words, you would use the step by calling

factory.addStep(EnvStep(command='foo'))

and the sourcing would happen automagically.

We also have a slew of other custom buildsteps that require the build-env to be sourced, so we just let them subclass EnvStep instead of ShellCommand for the environment to be dealt with automatically.

Bonnice

Answer 3

You can use the env parameter to the ShellCommand build step to setup the environment for your command. (http://docs.buildbot.net/0.8.1/Using-ShellCommands.html)

You can also set the environment before starting the build slave, if you want the environment to apply to all build command equally.

Basically either:

1. Work out the environment variables required and pass in in env.
2. Source the configuration before starting the build slave.
3. Wrap the source command and your builds command(s) is a shell script, and run as a single build step.

Igorot

The env parameter adds a known set of environment values to the environment when executing a specific command. I could very well go through the shell script that needs to be sourced and workout the environment. The shell scrip in question is not mine but part of open embedded tools. Therefore is likely to change in source control outside my control. Hence, the question asked how to "source" the environment and apply this to further build steps. – Ninebark

I'd probably just go with running the whole lot in a single build step, by wrapping everything in a shell script. – Igorot

Answer 4

An example for Visual Studio development is also useful.

By using common tools scripts, the right env is set for each machine, multiple versions of VS can be used by the same builders, and multiple architectures native and cross are supported.

# A raw text string to get around windows quoting problems in buildbot.
vscomntools=r""""%VS120COMNTOOLS%\VsDevCmd.bat" %PROCESSOR_ARCHITECTURE% & set"""

# Determine MSVC Environment using VS Common Tools for build steps.
f.addStep(SetProperty(
    command=vscomntools,
    extract_fn=glob2list))

The alternative is to start each command by trying to quote the batch file, &, and command.

Bootery

Answer 5

    steps.ShellCommand(
            name = "Example",
            command = ["/bin/bash", "-c", "source <BASH SCRIPT>"],
            haltOnFailure = True,
            description = "Example"
    ),

Schoenfelder

The issue with this is that the following buildstep will spawn a new shell and won't inherit those sourced functions. – Merovingian

Answer 6

I solved this problem a little differently. I install a batch file on the Windows worker which, upon launching, calls the environment setting batch file, then executes the intended command. This is, of course, made harder by the fact that batch files are horrible at forwarding arguments, and by the fact that Visual Studio 2017's VsDevCmd.bat clobbers your cwd. However, if you install the following file on the worker, you can build with VS2017:

@ECHO OFF
@REM Calls a single command from the environment set from visual studio vcvars.
@REM Usage:
@REM withvcvars-2017.bat <arch> [command]
@REM 
@REM Run VsDevCmd.bat /help to see available arches
@REM See below instantiation for more fine grained option setting.

set ARCH=%1
shift

setlocal enabledelayedexpansion

@rem Replace __ with = in batch files.
@rem This works around idiotic lack of equals signs in args
@rem args contains full args string with substitutions in place
:argloop
if "%~1" NEQ "" (

    set str=%~1
    set out=!str:__==!
    set %~1=!out!
    set args=!args!!out! 

    SHIFT
    goto :argloop
)

@rem Aside from batch files being too moronic to allow for equals signs in args,
@rem VsDevCmd.bat idiotically clobbers the current working directory when it's called.
set CWD=%cd%

echo Calling VsDevCmd.bat for arch %ARCH%
call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\Tools\VsDevCmd.bat" -arch=%ARCH% -winsdk=8.1 -app_platform=Desktop -no_logo

@rem Who lets these things out the door?
cd %CWD%

@ECHO ON
%args%

Once this is done, you can create a function in your bulidbot master logic which appends this batch file:

def _withvcvars(year, arch, cmd):     
    base_cmd = ["%swithvcvars-%s.bat" % ('\\path\\to\\batchfile\\', year), arch]      
    return base+cmd

This lets you runs commands wherein you call msbuild.exe which expects equals signs in its arguments. Just specify them as double underscores:

withvcvars-2017.bat amd64 msbuild.exe your.sln /p:Configuration__Release /p:Platform__x64

Pinery