Setting up a deployment / build / CI cycle for PHP projects

Question

I am a lone developer most of my time, working on a number of big, mainly PHP-based projects. I want to professionalize and automate how changes to the code base are handled, and create a Continuous Integration process that makes the transition to work in a team possible without having to make fundamental changes.

What I am doing right now is, I have a local test environment for every project; I use SVN for each project; changes are tested locally, and then transferred to the on-line version, usually via FTP. API documentation is generated manually from the source code; Unit tests are something I am getting into slowly, and it's not yet part of my daily routine.

The "build cycle" I am envisioning would do the following:

• A changeset gets checked into SVN after having been tested locally.

• I start the build process. The SVN HEAD revision gets checked out, modified if necessary, and made ready for upload.

• API Documentation gets generated automatically - if I haven't set it up in detail yet, using a default template, scanning the whole code base.

• The new revision is deployed to the remote location via FTP (Including some directory renaming, chmodding, importing databases, and the likes.) This is something I already like phing for very much, but I'm open for alternatives of course.

• Unit tests residing in a predefined location are run. I am informed about their failure or success using E-Mail, RSS or (preferably) HTML output that I can grab and put into a web page.

• (optionally) a end-user "changelog" text file in a pre-defined location gets updated with a pre-defined part of the commit message ("It is now possible to filter for both "foo" and "bar" at the same time). This message is not necessarily identical with the SVN commit message, which probably contains much more internal information.

• Stuff like code metrics, code style checking and so on are not my primary focus right now, but on the long run, they certainly will. Solutions that bring this out-of-the-box are very kindly looked upon.

I am looking for

• Feedback and experiences from people who are or were in a similar situation, and have successfully implemented a solution for this

• Especially, good step-by-step tutorials and walkthroughs on how to set this up

• Solutions that provide as much automation as possible, for example by creating a skeleton API, test cases and so on for each new project.

and also

• Product recommendations. What I know so far is phing/ant for building, and phpUnderControl or Hudson for the reporting part. I like them all as far as I can see, but I have of course no detailed experience with them.

I am swamped with work, so I have a strong inclination towards simple solutions. On the other hand, if a feature is missing, I'll cry about it being too limited. :) Point-and-click solutions are welcome, too. I am also to commercial product recommendations that can work with PHP projects.

My setup

I am working on Windows locally (7, to be exact) and most client projects are run on a LAMP stack, often on shared hosting (= no remote SSH). I am looking for solutions that I can run in my own environment. I am ready to set up a Linux VM for this, no problem. Hosted solutions are interesting for me only if they provide all of the aspects described, or are flexible enough to interact with the other parts of the process.

Bounty I am accepting the answer that I feel will give me the most mileage. There is a lot of excellent input here, I wish I could accept more than one answer. Thanks everyone!

Dimitry

Answer 1

I've been through buildbot, CruiseControl.net, CruiseControl and Hudson. All though I really liked CruiseControl*, it was just too much of a hassle with really complex dependency cases. buildbot is not easy to set up, but it's got a nice aura (I just like python, that's all). But hudson won over the former three because:

1. It's just easy to set up
2. It's easy to customize
3. It looks good and got nice overview functionality
4. It got point-and-click updates, for itself and all installed plugins. This is a really nice feature, that I appreciate more and more

Caveat: I only ever used linux as base for the above mentioned build servers (CC.net ran on mono), but they should all - according to the docs - run cross-platform.

Setting up a hudson server

Prerequisites:

• Java (1.5 will serve you just fine)
• Read access to the subversion server (I have a separate account for the hudson user)

From here, it's just:

java -jar hudson.war

This will run a small server instance right off your console, and you should be able to browse the installation at your http://localhost:8080, if you don't have anything else running on that port in advance (you can specify another port by passing the --httpPort=ANOTHER_HTTP_PORT option to the above command) and everything went well in the 'installation' process.

If you go to the available plugins directory (http://localhost:8080/pluginManager/available), you'll find plugins for supporting your above mentioned tasks (subversion support is installed per default).

If that has whet you appetite, you should install a java application server, such as tomcat or jetty. Installation instructions are available for all major application servers

Update: Kohsuke Kawaguchi has constructed a windows service installer for hudson

Setting up a project in hudson

The links in the following walk-through assumes a running instance of hudson located at http://localhost:8080

1. Select new Job (http://localhost:8080/view/All/newJob) from the menu on the left
2. Give the job a name and tick Build a free-style software project on the list
3. Pressing 'ok' will take you to the configuration page of the job. All the options have a little question mark besides them. Pressing this will bring up a help text regarding the option.
4. Under the option group 'Source Code Management' you would be using Subversion. Hudson accepts both url access as well as local module access
5. Under the option group 'Build Triggers', you would use 'Poll SCM'. The syntax used here is that of cron, so polling the subversion repository every 5 minutes would be */5 * * * *
6. The process of building the project is specified under the option group 'Build'. If you already have an ant build file with all the targets you need, you're in luck. Just choose 'Invoke ant' and write the name of the target. The option group supports maven and shell commands as well out of the box, but there is also a plugin available for phing.
7. Tick off additional build actions in 'Post Build Actions', such as e-mail notifications or archiving of build artefacts.

For setting up processes for which hudson have no plugins, you can either call them directly through a shell script from within the build setup, or you could write you own plugin

Pitfalls:

• If you have it produce build artefacts, remember to have hudson clean up after itself in regular intervals.
• If you have more than 20 projects set up, consider not displaying their build status as the default main page on hudson

Good luck!

Minuet

I think there's an error with your cron syntax. That would run on the fifth minute of every hour. If you wanted to poll the repo every 5 minutes you would need to use */5 – Batty

@BrianWigginton: you are completely right. And I even tried to correct the post, but I cannot save it because I have "a reference to an invalid hostname", namely 'localhost'. I hope people look down here for the correction. – Minuet

Answer 2

The term you are looking for is "continous integration."

Here is an example of someone who uses GIT + phpundercontrol: http://maff.ailoo.net/2009/09/continuous-integration-phpundercontrol-git/

CruiseControl (which is a CI server), can use Hosted SVN/GIT as a source. So you can even use it with GitHub or Beanstalk or something else.

Then you can integrate that with the following kind of software:

• PHPUnit
• php-codesniffer
• phpdocumentor
• PHP Gcov
• PHPXref
• Yasca
• etc.

You could also try this hosted CI: http://www.php-ci.net/hosting/create-project

Keep in mind though, that those tools need custom support if you integrate them yourself.

Have you also thought about project management and patch management?

You can use Redmine for project management. It has integrated continuous integration support, but only as client side (not as CI server).

Try using a hosted SVN/GIT/etc. solution, because they will cover your backups and keep their servers running, so you can focus on development.

For a tutorial on how to setup Hudson, see: http://toptopic.wordpress.com/2009/02/26/php-and-hudson/

Camorra

Thanks for the link. The products I am familiar with, what I'm hoping for is more tutorial-style walkthroughs like the one you provided. – Electronic

Hi, as of Oct 2012, is php-ci.net down or pulled already from service? I can't seem to connect to that address, nor can I find new information from the past year. – Shakeup

@Ryan, I don't know what happend. I do know, that you can try others. Maybe you could try: CircleCI or you could check this topic: Hosted Continuous Integration for PHP? – Camorra

@Ryan, no it's not down, I have been using that for a while now. They have moved to phptesting.org – Pyrope

Answer 3

I use Atlassian's Bamboo continous integration server for my main PHP project (along with their other products such as fisheye (repository browsing), jira (issue tracker) and clover (code coverage)).

It supports SVN and now supports Git and it has a great user interface. It is available for linux, windows and mac and can run standalone on its own tomcat server which is great for people (like me) who does not like to take days to setup their tools). Although it may look expensive, being a lone developer myself I purchased the starter kit license for 10$ (10$ by software). This is great for small teams and it is worth the look.

Gothic

How can you configure bamboo to support phing? I am currently use ant, but seems phing is more apporipriate to use for php depooyment. Thanks – Cesura

Answer 4

PHPTesting PHPCI This is nice, continuous integration server built in php.

Plus, its free and open source. :)

it has number of plugins..

PHPCI includes integration plugins for:

• Atoum
• Behat
• Campfire
• Codeception
• Composer
• Email
• Grunt
• IRC
• PHP
• Lint
• MySQL
• PDepend
• PostgreSQL
• PHP Code Sniffer
• PHP Copy/Paste Detector
• PHP Spec
• PHP Unit
• Shell Commands
• Tar / Zip

Overtrade

Although I like PHPCI I don't recommend it. It's incredibly buggy and non-reliable. It's sad that common basic functionality isn't being properly unit tested so basic thing such as adding a project URL is difficult without it messing up. – Chouest

@Chouest what is your CI strategy for PHP? – Pyrope

@Pyrope I'm stuck with PHPCI unfortunately. All the other software is so much more bulky and takes a lot to set up. Everything has it's pros and cons. You just have to go with what works best for you. – Chouest

Answer 5

I would suggest using Jenkins http://jenkins-ci.org/ it's free and it's open source.

It's pretty straight forward to setup, works on multiple platforms and integrates well with other continuous integration tools like SonarQube (+ SQUALE) to measure technical debt and Thucydides for testing automation.

I would highly suggest using GIT or GIT Hub for version control instead of SVN. From my point of view it's just a better version control system that will help you scale your development efforts later.

Since you're working mostly with PHP project there are some other tools you can use.

PHPUnit - For unit testing

PHP CodeSniffer - Check for coding standards

PHP Depend - Shows your PHP code dependencies

XDEBUG - For performance testing

All of these tools and be triggered with a Jenkins job and helps with the quality and performance of your code.

Good luck and Enjoy!

Silvasilvain

Answer 6

I am mostly a sys admin but sometimes I code PHP as well. As a side project I created some scripts that will make it simple and painless to set up a full blown PHP CI environment using Jenkins. It also runs a sample project for you so you can see how each build step is configured.

If you want to try it out all you need is a Debian/Ubuntu box and shell access.

http://yauh.de/articles/379/setting-up-a-ci-environment-for-php-projects-using-jenkins-ci

Update To add some content to my answer:

You can simply set up a Jenkins CI for PHP using Ansible. Since v1.4 it supports roles which you can download from their galaxy.ansibleworks.com community site and it will do the heavy lifting for you. It is called jenkins-php.

Orji

Answer 7

I do not use many of the products, or even types of products that you use, but I will give you my experience.

I run a TEST environment in parrallel with my PROD environment. I have no local testing per se. If it is too hard to get soemthing up into a real TEST environment, then I fix my build process. I don't see the point in testing locally, as the environments are different. UPDATE: The only thing I do locally is run "php -l" before I upload anything. Stops the stupid mistakes.

The build process works with whatever is in the current workspace, which includes uncommitted code. This is not everyone's cup of tea, but I am going to TEST very often. Everything gets committed before going to PROD.

Part of my build process (similar to yours) creates two META files. One contains the last (typically) 100 changes and also gives me the current changelist number. The shows me what changes are installed. The other contains the CLIENTSPEC (in Perforce terms) which shows me exactly what branches were used in this build. Together these give me reproducible builds.

I do not build straight to the target environment, but to a staging area on the server. I use SSH so this makes sense. This gives me a few advantages. Most importantly it avoids dying half way through a large upload. It also gives me a place to store META files, and all the build files are automatically archived (so I can go straight back to any build). The script also logs the update (so there is an entry in the log stream and I can see pre- and post-) and kicks all daemons (I use daemontools so "svc -t"). All of these are better off on the target machine.

One other issue is DB changes. I keep a master script of the DB schema, which I update every time the schema changes. Each of the changes also go into a changes.sql script, which is uploaded with the build to the staging area. The script is run as part of the install script.

Checkerbloom

Thanks for your input Phil, this is a very, very interesting setup and I think I can use parts of this. In my case however, I will very often not have SSH access to the remote end, so I need a lot of "intelligence" locally, which is why I think I'm going to need a CI product like Hudson. – Electronic

Answer 8

Consider fazend.com, a free hosted CI platform, which automates configuration and installation procedures. You don't need to setup version control, bug tracking, CI server, test environment, etc. Everything is done on-demand.

Amabelle

Mmm, this looks very interesting! But how do they survive? Seeing as everything's free? How do they pay for the server, and the E3 fees? – Electronic

@Electronic This project is sponsored by a software development company, that's why the service is free – Amabelle

Answer 9

I've recently begun the same kind of process, and am using Beanstalk for svn hosting.

There are two nifty features in the paid accounts (start at $15pm i think):

• deployment allows the user to create ftp targets for staging and production servers, which can be deployed at the click of a button (inc specifying a revision and branch)
• webhooks allow the user to set up a url that is called on each commit/deploy, passing across things like revision number, description and user. This could be used to update docs, run unit tests and update changelogs.

I'm sure there are other hosted or self-hosting svn servers with these two features, but beanstalk is the one i have experience of and it's working very, very well

There's also an API, which I imagine could be used to integrate deployment further in to your process.

Eady

Cheers @adam. I am looking not so much for hosted services for this, as I fear they will always be too limited for what I need. But what you say sounds interesting, and the API might be a connecting point for the rest. I will take a look at this. – Electronic

No worries, thought you might say that. It suits me for now, as I've not quite got my head round svn (and I'm a one-man team), plus they offer a full repo export if you want to move. – Eady