Tips for speeding up build time on Linux using ANT, Javacc, JUnit and compiling Java classes

Question

We have a large codebase that takes approx 12 minutes on the developer machines to auto-generate some Java 5 classes using JavaCC and then compiles all the classes as well as running the units test.

The project consists of multiple projects which can be built in groups, but we are aiming for a full a build in under 10 minutes

What tips are there for reducing this build time?

Thanks

Method

If I understand correctly, you're aiming for a clean build (i.e., from scratch) to take less than 10 minutes. However, why do you need to perform a full rebuild so often? Just curious here. – Swelling

Its part of of our process. We try to maintain a constant working build of committed code. We have a build sever hat constantly builds when new code is committed. Also most developers in our place run a full build of their code before checking in. Those that don't run the risk of breaking the build. – Method

In my experience, you're in trouble with the policy of running a full build/test before every commit. This will prevent developers from making finer-grained commits. Not to mention the annoyance where you've waited for 12 minutes, and then discovered that more commits have occurred in the meantime. – Swelling

"This will prevent developers from making finer-grained commits." Good point never thought of it from that point of view "Not to mention the annoyance where you've waited for 12 minutes, and then discovered that more commits have occurred in the meantime" True this does happen but not too often – Method

Answer 1

One quick fix that might shave some time off is to ensure that you are running Ant using the server JVM (by default it uses the client VM). Set ANT_OPTS to include "-server".

Likker

Will this speed up times on a desktop machine as well as the server? – Method

Yes, this simply changes the way the JIT optimizes bytecode when running Ant. – Swelling

On Windows I knocked about a minute off a 6 minute build with this switch. – Likker

It might not make a significant difference, but it's worth trying since it's trivial to do. – Likker

Answer 2

• Profile the build process and see where the bottlenecks are. This can give you some ideas as to how to improve the process.
• Try building independent projects in parallel on multi-core/CPU machines. As an extension of this idea, you may want to look around for a Java equivalent of distcc (don't know whether it exists) to distribute your build over a number of machines.
• Get better machines.

Swelling

"Get better machines." we have recently all converted to Linux as the build times went from approx 40 minutes to approx 12 minutes. – Method

"Try building independent projects in parallel on multi-core/CPU machines. " Most of are on Dual core machines and we have a parallel build process that is slightly faster than the normal build – Method

Answer 3

some tips for reducing build time:

1. do less work. e.g. remove unnecessary logging/echoing to files and console

2. make your build 'incremental'. Compile only changes classes.

3. eliminate duplicated effort. Easier said than done, but if you run the build in debug mode ("ant -debug") you can sometimes see redundant tasks or targets.

4. avoid expensive operations. copying of files, and packaging of jars into wars are necessary for release.Signing jars is expensive and should only be done, if possible' for milestone releases rather than every build

Collywobbles

Answer 4

Try be inspired by pragmatic programmer. Compile only what is necessary, have two or more test suites. One for quick tests, other for full tests. Consider if there is real need to use each build-step every time. It necessary try to use jikes compiler instead of javac. After project spans several hundreds of classes I switch to jikes to improve speed. But be aware of potential incompatibility issues. Don't forget to include one all in one target to perform every step with full rebuild and full test of project.

Hirsutism

Answer 5

Now that you've explained the process in more detail, here are two more options:

1. A dedicated machine/cluster where the build is performed much quicker than on a normal workstation. The developers would then, before a commit, run a script that builds their code on the dedicated machine/cluster.

2. Change the partitioning into sub-projects so that it's harder to break one project by modifying another. This should then make it less important to do a full build before every commit. Only commits that are touching sensitive sub-projects, or those spanning multiple projects would then need to be "checked" by means of a full build.

Swelling

Answer 6

This probably wouldn't help in the very near term, but figured I should throw it out there anyway.

If your project is breakable into smaller projects (a database subsystem, logging, as examples), you may be interested in using something like maven to handle the build. You can run each smaller bite as a separate project or module, and maven will be able to maintain what needs to be built if changes exist. In this the build can focus onthe main portion of your project and it won't take nearly as long.

Radii

Answer 7

What is the breakdown in time spent:

1. generating the classes
2. compiling the classes
3. running the tests

Depending on your project, you may see significant increases in build time by allocating a larger heap size to javac(memoryMaximumSize) and junit(maxmemory).

Stricken

Answer 8

Is it very important that the entire build lasts less than 10 minutes? If you make the sub-projects independent from one another, you could work on one sub-project while having already compiled the other ones (think Maven or Ivy to manage the dependencies).

Another solution (and if your modules are reasonably stable) is to treat your sub-projects as standalone projects. Each project would then follow their own release cycle and be available from a local Maven/Ivy repository. This of course works well if at least parts of the project are reasonably stable.

Patisserie