GCC/Make Build Time Optimizations

Question

We have project which uses gcc and make files. Project also contains of one big subproject (SDK) and a lot of relatively small subprojects which use that SDK and some shared framework.

We use precompiled headers, but that helps only for re-compilation to be faster.

Is there any known techniques and tools to help with build-time optimizations? Or maybe you know some articles/resources about this or related topics?

Purr

Answer 1

You can tackle the problem from two sides: refactor the code to reduce the complexity the compiler is seeing, or speed up the compiler execution.

Without touching the code, you can add more compilation power into it. Use ccache to avoid recompiling files you have already compiled and distcc to distribute the build time among more machines. Use make -j where N is the number of cores+1 if you compile locally, or a bigger number for distributed builds. That flag will run more than one compiler in parallel.

Refactoring the code. Prefer forward declaration to includes (simple). Decouple as much as you can to avoid dependencies (use the PIMPL idiom).

Template instantiation is expensive, they are recompiled in every compilation unit that uses them. If you can refactor your templates as to forward declare them and then instantiate them in only one compilation unit.

Nippers

Answer 2

The best I can think of with make is the -j option. This tells make to run as many jobs as possible in parallel:

make -j

If you want to limit the number of concurrent jobs to n you can use:

make -j n

---

Make sure the dependencies are correct so make doesn't run jobs it doesn't have to.

---

Another thing to take into account is optimizations that gcc does with the -O switch. You can specify various levels of optimization. The higher the optimization, the longer the compile and link times. A project I work with runs takes 2 minutes to link with -O3, and half a minute with -O1. You should make sure you're not optimizing more than you need to. You could build without optimization for development builds and with optimization for deployment builds.

---

Compiling with debug info (gcc -g) will probably increase the size of your executable and may impact your build time. If you don't need it, try removing it to see if it affects you.

---

The type of linking (static vs. dynamic) should make a difference. As far as I understand static linking takes longer (though I may be wrong here). You should see if this affects your build.

Disject

Also make sure that any recursion is done so that make -j actually works. For example, no calling "make -C subdir", use $(MAKE), etc. – Nightstick

I'm not sure I understand the suggestion. – Disject

If your makefiles recurse by calling just make, you won't get much scalability with -j since only the top level make will use the -j arg, you should be using $(MAKE) because it will invoke submakes with the same -j and you will have a better opportunity to get all jobs running. – Theotokos

Answer 3

From the description of the project I guess that you have one Makefile per directory and are using recursive make a lot. In that case techniques from "Recursive Make Considered Harmful" should help very much.

Decern

Answer 4

If you have multiple computers available gcc is well distributed by distcc.

You can also use ccache in addition.

All this works with very little changes of the makefiles.

Josey

Answer 5

Also, you'll probably want to keep your source code files as small and self-contained as possible/feasible, i.e. prefer many smaller object files over one huge single object file.

This will also help avoid unnecessary recompilations, in addition you can have one static library with object files for each source code directory or module, basically allowing the compiler to reuse as much previously compiled code as possible.

Something else, which wasn't yet mentioned in any of the previous responses, is making symbol linkage as 'private' as possible, i.e. prefer static linkage (functions, variables) for your code if it doesn't have to be visible externally.

In addition, you may also want to look into using the GNU gold linker, which is much more efficient for compiling C++ code for ELF targets.

Basically, I'd advise you to carefully profile your build process and check where the most time is spend, that'll give you some hints as to how to optimize your build process or your projects source code structure.

Pachston

Answer 6

You could consider switching to a different build system (which obviously won't work for everyone), such as SCons. SCons is much smarter than make. It automatically scans header dependencies, so you always have the smallest set of rebuild dependencies. By adding the line Decider('MD5-timestamp') to your SConstruct file, SCons will first look at the time stamp of a file, and if it's newer than the previously built time stamp, it will use the MD5 of the file to make sure you actually changed something. This works not just on source files but object files as well. This means that if you change a comment, for instance, you don't have to re-link.

The automatic scanning of header files has also ensured that I never have to type scons --clean. It always does the right thing.

Longspur

Answer 7

If you have a LAN with developer machines, perhaps you should try implementing a distributed compiler solution, such as distcc.

This might not help if all of the time during the build is spent analyzing dependencies, or doing some single serial task. For the raw crunch of compiling many source files into object files, parallel building obviously helps, as suggested (on a single machine) by Nathan. Parallelizing across multiple machines can take it even further.

Kirtley

Answer 8

http://ccache.samba.org/ speeds up big time.

I work on a middle sized project, and that's the only thing we do to speed up the compile time.

Rabelais

As long as you don't do stupid trickery with DATE__/__TIME or related macros, ccache is nice for accelerating repeated clean builds. On the other hand, if you're doing clean builds often it means your rebuild system is broken and you should fix that first... – Assemble

Answer 9

You can use distcc distributed compiler to reduce the build time if you have access to several machines. Here's an article from from IBM developerWorks related to distcc and how you can use it: http://www.ibm.com/developerworks/linux/library/l-distcc.html

Another method to reduce build time is to use precompiled headers. Here's a starting point for gcc.

Also don't forget to use -j when building with make if your machine has more than one cpu/core(2x the number of cores/cpus is just fine).

Crass

Answer 10

Using small files may not always be a good recommendation. A disk have a 32 or 64K min sector size, with a file taking at least a sector. So 1024 files of 3K size (small code inside) will actually take 32 or 64 Meg on disk, instead of the expected 3 meg. 32/64 meg that needs to be read by the drive. If files are dispersed around on the disk you increase read time even more with seek time. This is helped with Disk Cache obviously, to a limit. pre-compiled header can also be of good help alleviating this.

So with due respect to coding guidelines, there is no point in going out of them just to place each strcuct, typedef or utility class into separate files.

Idealize