Header inclusion optimization

Question

Is there an automatic way to optimize inclusion of header files in C++, so that compilation time is improved ? With the word "automatic" I mean a tool or program. Is it possible to find which headers files are obsolete (e.g exposed functionality is not used) ?

Edit: Having each include header "included only once is one important thing, but is there a way to even change the contents of files so that frequently used "functionality" is on specific includes and less frequently used functionality is on other includes? Am i asking too much ? Unfortunately, we are talking about an existing code base with thousands of files. Could it be a refactoring tool what I am actually asking for ?

Dahl

Are you starting from scratch, or are you trying to optimize headers for an existing code base? – Dorpat

There is an existing code base, around 3000 source files. – Dahl

Have you tried pre-compiled headers ? They are widely used by IDE (Visual Studio or XCode) in order to pre-compile all the UI and system stuff, so they are built every-time. – Osullivan

This question has some suggestions for tools that check for unused includes as well as classes that can forward declared: #1302350 – Witching

Does Visual Assist X do this in the same manner as Resharper? – Gelinas

Not a list of tools, but this question has a good number of tips for speeding up c++ compiles: #373642 – Often

The tool you seem to need already exists and is built into most modern compilers. See your compilers documentation about "Pre-Compiled headders" – Diaconate

Answer 1

Update

I think what you really want is "include what you use" rather than a minimal set of headers. IWYU means forward declare as much as possible, and include headers that directly declare the symbols you use. You cannot mindlessly convert a file to be IWYU clean as it may no longer compile. When that occurs, you need to find the missing header and add it. However, if every file is IWYU clean your compiles will be faster overall even if you have to add headers occasionally. Not to mention you headers will be more meaningful/self-documenting.

As my previous answer points out it is technically possible to include even fewer headers than necessary for IWYU, but it's generally a waste of time.

Now if only there was a tool to most of the IWYU refactoring grunt work for you :)

• Google's IWYU
• Include What You Use

---

I had considered a creating/using a tool like this once. The idea is to use binary search and repeated compilation to find the minimal set of includes. Upon further investigation it didn't seem that useful.

Some issues:

• Changing the included header files can change the behavior, and still allow the file to compile. One example in particular, if you defined your own std::swap in a separate header file. You could remove that header and your code would still compile using the default std::swap implementation. However, the std::swap may be: inefficient, cause a runtime error, or worse produce subtly wrong logic.

• Sometimes a header file inclusion works as documentation. For instance, to used std::foreach, often including <vector> is sufficient to get it to compile. The code is more meaningful with the extra #include <algorithm>.

• The minimal compilation set may not be portable, between compilers or compiler versions. Using the std::foreach example again, there is no guarantee that std::foreach will provided in by <vector>.

• The minimal set of includes may not affect compile time significantly anyway. Visual studio and gcc support #pragma once which make repeated included essentially non-existent performance wise. And at least gcc's preprocessor has been optimized to process include guards very fast (as fast as #pragma once).

Haemophilic

I'm a big fan of #pragma once – Cann

I think there is also potentially big gain in forward declaring, not including. – Aegospotami

@T.E.D.: And people were using #include guards a long time before #pragma once (or any pragma) were available. I worked on a project a few years ago that didn't have either, and it was a real pain. – Razor

Both MSVC and GCC are able to skip repeated inclusions with regular #include guards as well, so there's little need for nonstandard pragmas. – Horribly

I've recently come to like #pragma once as well. I kinda worry about portability, but it sure makes headers look cleaner with one short line at the top instead of a multi-line complex include guard. – Haemophilic

There is one thing. Sometimes a #include is no longer semantically appropriate to have there. Nothing in the file doing the #include actually uses any symbols of any kind from the #includeed file. If this is the case, it would be best to remove the #include, mostly to avoid giving people the wrong idea. Also if an extraneous #include introduces a dependency removing it can greatly increase compilation speed, partly by not compiling the #include but also by avoiding erroneous recompiles. – Empale

I think Omnifarious's final point is the one I'd like to make, too. The performance gains I expect to get out of include optimization is on recompile. It's really annoying when I change a header file and a zillion unrelated files need to be recompiled. – Reiss

Thanks for this answer. I am working on a project to clean up some code and I've noticed a lot of header drift. I generally find it faster just to read through a file and guess which headers are no longer used. I start with simple stuff and work up to the larger files. – Taurus

Answer 2

Most compilers have some kind of support for precompiled header files.

A tool that removes totally unneeded includes might be nice. You seem to be implying that you'd like to see one that removes needed includes that are rendered redundant by other includes. That I wouldn't be a big fan of. Somebody could remove the other redundant include one day, and then some other poor slob will have to track down what include file somewhere on the hard disk has all those missing symbols that suddenly appeared on them for no apparent reason.

Cann

But the problem you mention would mean a poor organization of include files, wouldn't it? – Dahl

Include files are pretty much by definition poorly organized. For instance: If I'm suddenly missing symbol foo from an API three #includes deep, how am I to know that foo happens to be defined in jobob.h somewhere in one of the include subdirectories under the Windows distribution? Other than searching the whole harddrive (or Googling "foo" and hoping I get lucky), I'm not. – Cann

Answer 3

PC-Lint will report unused include files.

Ec

Answer 4

The GamesFromWithin blog has a good article on C++ headers, and even a tool that lists which headers are included the most (and thus prime candidates for pimpl/pch/forwarding). Good tool, albeit in perl, and I've actually used it a few times to dig up some good data when build times make me (too much) nuts.

Specialist

Answer 5

There has been some work in GCC on the topic of precompiled headers, but there are some restrictions (only one precompiled header can be included per compilation, if I remember).

If you are not using GCC this solution will not help you, because it doesn't produce a simplified header that could be used by another compiler.

Vincenzovincible

I found that with precompiled headers gcc sucks again. I'm using Eiffel which compiles to C and generates a very huge header file. It is C not C++ and using compiling with a precompiled is slower then compiling without it. Somebody should verify if this is also true for C++. With Sun Studio compiler (i recommend it) or Microsofts VC get indeed a speed burst. – Dendrology

Answer 6

Google's cppclean will help you remove unnecessary header files. It produces some false positives, but it provides a great starting point. See my answer to a similar question and comments here.

Oversupply

Answer 7

I never heard of any tool tracking useless includes. And this would be only the trivial case.

However, you should pay some attention to it anyways. How to organize the headers and the implementation depends a lot on how you organize you code. Thinking your project as indepents parts is important, and headers will partly enforce you to think about it.

Bohemianism

Answer 8

I recent found a tool to do exactly what you asked.

On Ben Ziegler's blog Double Buffered, he posted about some build system optimizations.

So, I came up with the sophisticated solution of Brute Force Perl Script. Basically, I would scan through all of the .c files in our code and attempt to comment out each #include directive. It would then recompile the code base and see if there were errors. If there were it would revert the comment out and then go on to the next one.

The results weren't stellar. No noticeable build time improvements.

I still stand by my first answer; it's a waste of time, and ill advised.

Either way here is his perl script. You'll need to hack it a bit to get it to work on your code but core should work the same.

Haemophilic

This does not necessarily do the job. Just because you can comment out a header file and have the system still work doesn't mean you shouldn't include it. You should #include <string> if you use ::std::string even if the other headers you include also #include <string>. At best a tool like this can point you towards possible extraneous headers. – Empale

I absolutely agree. See my first answer. A tool like this can alter the behavior of c++ code (c code may be safe, but still a bad idea). – Haemophilic

Answer 9

I wrote a trivially simple tool to do this once, and it worked really well, albeit in the most hideously brute force manner.

It recursed over all h/cpp files in the project.
For each, it searched for the #includes, and added // in front of each of them in turn, then triggered a build. If the build returned an error it undid the //. If the build succeeded, it left the include line commented out.

It was pretty quick to process all the cpp files (an hour or so) but I'd have to leave it running over the codebase all weekend to deal with the headers, but it was very simple to implement and extremely effective. And it only needed to be run about once a year to clean away the accumulated fluff.

It made a small but significant difference (about 5 minutes on a 70min build IIRC).

Thorium

Note that this isn't necessarily going to work - overloads and template resolutions can compile successfully, but go terribly wrong at runtime. – Often

I'm just saying that an automated system like this can have subtle, hard to diagnose effects - make sure that there's a human element involved. – Often

Answer 10

Precompiled headers are awful.

This is really a linker issue, and there's nothing to do about it except either get a better linker (good luck) or make your code in a way that's easier on the linker.

Bituminize

Sorry, but linkers don't deal with header files. This is a compiler issue. The problem is when an include file is changed. – Astrometry

Blanket statement that doesn't make sense, combined with false information? You should research the compile/link/run process more, on your platform of choice. That, and precompiled headers can be non-intrusively added to your application on some platforms (gcc/icc), making the choice whether or not to use them strictly based on compile-time performance. – Foochow

Sorry, but you obviously have no idea what you are talking about. – Bituminize

Tom, sure it works for rinky dink application but it's only going to cause headache in anything else. Include a million headers in a million source files? It's just not going to work out for anything but the simplest cases, and shielding people is what leads to dumb questions like this followed by ignorant responses when someone says the right thing. – Bituminize

Antiguru - a SO comment isn't really suitable for an explanation, so I've posted this out-of-band. See tombarta.wordpress.com/2009/12/12/precompiled-headers-with-gcc for a demonstration of non-intrusive precompiled headers. – Foochow