How to scale buildbot in a company
Asked Answered
C

2

12

I've been looking into buildbot lately, and the lack of good documentation and sample configurations makes it hard to understand how buildbot is commonly used.

According to the buildbot manual, each buildmaster is responsible for 1 code base. That means that a company that wants to use buildbot on, say, 10 projects, needs to maintain 10 different sets of buildbot installations (master-slaves configurations, open ports, websites with output etc.). Is this really the way things are done? Am I missing an option that creates a mash-up that is easy to maintain and monitor?

Thanks!

Cryology answered 2/8, 2009 at 20:22 Comment(1)
FWIW, there is currently work underway to improve buildbot's scaling, sponsored by the Mozilla project. (Who runs a very large buildmaster). See the buildbot mail list for details.Riddick
N
4

At my place of work we use Buildbot to test a single program over several architectures and versions of Python. I use one build master to oversee about 16 slaves. Each set of slaves pulls from a different repo and tests it against Python 2.X.

From my experience, it would be easy to configure a single build master to run a mash-up of projects. This might not be a good idea because the waterfall page (where the build slaves report results) can get very congested with more than a few slaves. If you are comfortable scrolling through a long waterfall page, then this will not be an issue.

EDIT:

The update command in master.cfg:

test_python26_linux.addStep(ShellCommand, name = "update pygr",
    command = ["/u/opierce/PygrBuildBot/update.sh","000-buildbot","ctb"], workdir=".")

000-buildbot and ctb are additional parameters to specify which branch and repo to pull from to get the information. The script update.sh is something I wrote to avoid an unrelated git problem. If you wanted to run different projects, you could write something like:

builder1.addStep(ShellCommand, name = "update project 1",
    command = ["git","pull","git://github.com/your_id/project1.git"], workdir=".")

 (the rest of builder1 steps)

builder2.addStep(ShellCommand, name = "update project 2",
    command = ["git","pull","git://github.com/your_id/project2.git"], workdir=".")

(the rest of builder2 steps)

The two projects don't have to be related. Buildbot creates a directory for each builder and runs all the steps in that directory.

Napier answered 2/8, 2009 at 20:45 Comment(4)
The master configuration just tells the slave to run a series of terminal commands. To run a different version or different project, you just tell it to pull from a different repo. Because you can individualize the commands for each slave you can have each slave running a different version/project/whatever.Napier
No, that's the opposite of what I am saying. Pulling from a repository and running real time builds is the idea behind continuous integration. We use git for version control.Napier
I finally understood what you mean by this, and this is actually a good idea. Only note worth giving - we decided to use categories, so that we can now divide the waterfall view according to those categories, and so each developer sees only the projects that interest him.Cryology
Why are you using a ShellCommand with 'git pull' instead of an actual Git step? djmitche.github.com/buildbot/docs/0.7.11/#GitAbshire
R
2

FYI, BuildBot 0.8.x supports several repositories on one master, simplifying things a bit.

Riddick answered 28/10, 2010 at 19:53 Comment(1)
You can see an example in a related questionSchweitzer

© 2022 - 2024 — McMap. All rights reserved.