How to improve Visual C++ compilation times?

Question

I am compiling 2 C++ projects in a buildbot, on each commit. Both are around 1000 files, one is 100 kloc, the other 170 kloc. Compilation times are very different from gcc (4.4) to Visual C++ (2008).

Visual C++ compilations for one project take in the 20 minutes. They cannot take advantage of the multiple cores because a project depend on the other. In the end, a full compilation of both projects in Debug and Release, in 32 and 64 bits takes more than 2 1/2 hours.

gcc compilations for one project take in the 4 minutes. It can be parallelized on the 4 cores and takes around 1 min 10 secs. All 8 builds for 4 versions (Debug/Release, 32/64 bits) of the 2 projects are compiled in less than 10 minutes.

What is happening with Visual C++ compilation times? They are basically 5 times slower.

What is the average time that can be expected to compile a C++ kloc? Mine are 7 s/kloc with vc++ and 1.4 s/kloc with gcc.

Can anything be done to speed-up compilation times on Visual C++?

Amelina

The disk is only 1/3 full. I doubt fragmentation and DLL copies can cut compilation times by 5! Barely 10 % maybe, but no more. – Amelina

Thanks for the points... But just curious... was it the instantiations in the header files that was the problem, or one of the other things I mentioned? – Jugal

Something that helped us a huge amount was moving all the temporary, intermediate and pch files to a ram disk, but still never as fast as GCC despite the Windows box having ssds and the linux box having hds. We were compiling on a Dual Xeon E5440, and I found "make -j33" gave radically better compilation times than the "-j9" that people recommended (I ran a simple script to benchmark every -j setting from 0-99) – Adjoining

Answer 1

One thing that slows down the VC++ compiler is if you have a header file that initializes concrete instances of non-trival const value types. You may see this happen with constants of type std::string or GUIDs. It affects both compilation and link time.

For a single dll, this caused a 10x slowdown. It helps if you put them in a precompiled header file, or, just declare them in a header and initialize them in a cpp file.

Do take a look into the virus scanner, and be sure to experiment with precompiled headers, without it you won't see VC++ at its best.

Oh yeah, and make sure the %TMP% folder is on the same partition as where your build is written to, as VC++ makes temp files and moves them later.

Jugal

I'm curious about that first point; How can you even define a variable in a header file without getting multiple definitions? Unless, of course, it's not the same variable because it's changed by a define or an unnamed namespace. But in those cases, I imagine that's the same as defining them in the cpp file, no? – Minded

In my case, They were declared as consts. I will edit that in. I did not write the code. I fixed it by forward declaring the names, and initialize them in a cpp file. – Jugal

#31029441 in the accepted answer removing the static keyword caused HUGE build times, might be related – Toulouse

Answer 2

The projects depending on each other doesn't imply that no parallelization is possible. The build systems are smart enough to figure out and avoid critical depenedancies, Otherwise gcc wouldn't be able to use 4 cores.

So (in addition to other steps), why not just try enabling multiprocessing in Visual Studio using /MP (See http://msdn.microsoft.com/en-us/library/bb385193.aspx).

Adria

Because multiprocessing in Visual Studio cannot split the work by compilation units (.cpp files), but by project only! – Amelina

Stange: at least om my machine (and according to the documentation quoted above) both VS2008, VS2010 Beta2 use multiple cores even when building just one project. – Adria

@dtrosset: VS2008 can split by compilation units. That's what the /MP flag does. In VS2005, the only form of parallelization available was per-project. – Christopher

As /MP is not compatible with /Gm (incremental builds), which one to choose? Either slow on developer workstation, or on automated builds. – Amelina

Why do they both have to use the exact same flags? Make automated builds use /MP and if /Gm is faster on developer machines, use that there instead. – Christopher

Answer 3

How are you building the Visual Studio projects? Are you just running the ide (devenv) with the project and /build or do you have a makefile similar to what I assume you are using for gcc. I'm assuming that both builds use a similar makefile but I thought it worth checking.

Are you using precompiled headers for either compiler? If you're NOT using precompiled headers for VS then you might like to switch to using them. Personally I'd recommend using the #pragma hdrstop approach rather than a single all inclusive header file but if you're currently NOT using precompiled headers and want to try it out a single all inclusive header file that is force included (using the /FI compiler command line switch) can be tested quickly without any code changes.

I wrote about both /FI and #pragma hdrstop here: http://www.lenholgate.com/blog/2004/07/fi-stlport-precompiled-headers-warning-level-4-and-pragma-hdrstop.html

Shatzer

I am compiling the Visual Studio projects on a command line using vcbuild.exe. (MSBuild.exe, or devenv.exe /build are the same duration) – Amelina

+1 for /FI flag. Not a great fan of #pragma hdrstop though – Lactation

@Manuel, why don't you like #pragma hdrstop ? It's a very handy way to allow clean minimal dependency builds with no precompiled as well as fast builds with precompiled headers which is, IMHO, a great way to make sure that code only includes what it needs which makes it easier to reuse... – Shatzer

@Len - Maybe I'm missing the point of hdrstop but I think I achieve the same with /FI and I don't have to modify my source files in any way – Lactation

@Manuel, #pragma hdrstop allows more fine tuning of what you need to put into the precompiled header and what can be included in the C++ file, and is a little less 'hidden' than the /FI route. – Shatzer

Answer 4

It's not the direct answer for the question but at my company we are using IncrediBuild for distributed compilation. It really speeds up the compilation process. http://incredibuild.com/visual_studio.htm

Deforest

Incredibuild can use CPU cores on multiple machines, but the current project setup fails to utilize all cores on the local machine. I.e. you first need to tackle that level of parallelization. – Refractory

Answer 5

The book "Large-Scale C++ Software Design" by John Lakos has many tips on structuring your code and design for large-scale projects. Including many tips on speeding up compilation. Not directly related to Visual C++, but well worth reading anyway.

Kotick

Answer 6

I've written two articles on techniques that reduce the compilation time. Among these techniques a post on precompiled header and unity builds that may help you improve compilation times. They ship with CMake scripts that handle the techniques transparently.

Filaria

Answer 7

First of all, in most cases you can build debug and release configurations of the same project in parallel.

Also what you describe sounds horribly slow - looks like you don't use precompiled headers in VC++ or using them incorrectly - they are specifically intended to improve compilation time.

Insistence

I am not using precompiled headers at all. Neither for vc++ nor g++; – Amelina

Answer 8

Perhaps there is an issue with the dependency checking, unless you are forcing a complete rebuild.

You could make some static libraries. Put code that seldom changes into libraries.

The slowest parts of building a program:

1. Opening and closing files.
2. Parsing and translating source files.

In general, the linking and executable creation phases are the fastest.

Have you determined:

1. which phases are the slowest?
2. Which files are slowest to compile?

Remember, when determining efficiency, always profile (in one manner or another).

Manualmanubrium

Answer 9

Are you building on the same machine? Are you using the same OS? I've seen speed differences in the region of 3-10x when comparing GCC in Cygwin and GCC in a VirtualBox machine running inside the Windows hosting Cygwin.

Leaguer

It's not the same machine, but they are equivalent. Visual Studio runs on Windows XP 32 bits system. gcc runs on virtualized (!) Debian Linux 32 bits system. – Amelina

Well, in my experience, that could explain the entire difference in time. Not sure what to do about it though - perhaps running the MS compilers through wine in a debian environment? Do you require the MS compilers, or can you switch to mingw? In that case you could cross compile from your debian setting. – Leaguer

Answer 10

It seems very strange that there would be such a difference... but there is no reason that you cannot take advantage of the multicores on Visual either!

Basically you have 4 compilations modes: (Debug/Release)x(32bits/64bits), each one being totally independent of the other, you could perfectly run the 4 in parallel, taking full advantage of the 4 cores available. Or simply try out the MultiProcessor approach on Visual Studio too.

However that's not going to cut it. 150 minutes versus 10 minutes is a huge gap. From my personal experience there are 2 major factors in reducing compilation time:

• have all files used on a local disk (using replication from remote ones if necessary) and all files created locally too (.o .so)
• use all the cores at your disposal, and if you can, even go Multi Machines (distcc etc...)

Umbra

Maybe I can try this, and run vcbuild.exe to build the same project but different configurations at the same time. – Amelina

There is a reason I cannot take advantage of multicores on Visual: Visual C++ does not support compiling multiple files from the same project at the same time. – Amelina

Partially true, as the /MP option is not compatible with the /Gm option (incremental builds) that is set for all our Debug builds. Curiously enough, we use the same projects for developer workstation builds and automated buildbot builds. – Amelina

Answer 11

Compile and link one cpp file at a time, even when header-file-changes affect multiple cpp files. This can be accomplised with visual studio macro:

Dim WithEvents myTimer As Timers.Timer

Sub CompileAndLinkCurrentCppFile()
    DTE.ExecuteCommand("Build.Compile")
    myTimer = New Timers.Timer
    myTimer.Interval = 0.05
    myTimer.Start()
End Sub

Sub myTimer_Elapsed(ByVal ee As Object, ByVal dd As Timers.ElapsedEventArgs) Handles myTimer.Elapsed
    If DTE.Solution.SolutionBuild.BuildState <> vsBuildState.vsBuildStateInProgress And DTE.Solution.SolutionBuild.LastBuildInfo <> 1 Then
        myTimer.Stop()
        DTE.ExecuteCommand("Build.Link")
    End If
End Sub

Dissuade

How is this different to Ctrl+F7? – Homogenetic

Answer 12

Don't know if this is still an issue and how much improvement you got in the menatime, but if it still stands that msbuild doesn't know how to orchestrate concurrently within one project (each cpp should be separately buildable unless you have some codegens - codegens are best moved to a separate project) you hay have to download driver development kit or .NET SSCLI since they both have nmake, build which are known to paralelize things well. SSCLI already had the build build setup, don't remeber if DDK has some build samples of do you have to start from scratch.

Also a bit oldish article on MSBuild parallelization doesn't go into details but mentiones some diff between actual msbuild and msbuild + sln. If the /MP vs. /Gm was the only issue then you may have to make a little script or C# exe to edit .proj files for lab build. Or use explicit cmd line override in projects and take that option from an env var.

Demon