Ignore missing headers with clang AST parser

Question

I'm on Windows, using MSVC to compile my project, but I need clang for its neat AST parser, which allow me to write a little code generator.
Problem is, clang cannot parse MSVC headers (a very-well known and understandable problem).

I tried two options :

1. I include MSVC header folder, parsing the built-in headers included in my code will end-up leading to a fatal error at some point, preventing me from parsing the parts I want correctly.
2. What I did before is simply not provide any built-in headers and forward declare the types I needed. It worked fine and somehow it doesn't anymore with latest Clang. I don't really know if the parser policy on missing header changed, but it is causing complete failure every time something like <string> is included and not much get parsed.

I am using the python bindings (libclang), but I would consider switching to C/C++ API if there would be a solution there.

Is there anyway I can alter this behavior and make clang continue parsing even when some headers are not found ?

Ximenes

Do you want to know about solutions that can parse MS headers? – Foveola

Any solution that makes can make this work yes ! – Ximenes

Any workaround solution regarding this? @Ximenes – Peregrination

Answer 1

Use SetSuppressIncludeNotFoundError. Took me an hour to find! You can imagine how glad I was to find it!

https://clang.llvm.org/doxygen/classclang_1_1Preprocessor.html#ac7bafe67fc32e41460855b39d20ff6af

Mellen

That would be great, now I gotta check if this option is exposed in the python bindings – Ximenes

hey @Martin, I'm running into the same issue with the headers, I'm following this tutorial to create a refactoring tool clang.llvm.org/docs/LibASTMatchersTutorial.html Do you know where or how I can set the SetSuppressIncludeNotFoundError option using ClangTool? – Kozloski

Any answers to @Kozloski question ? – Sweated

Answer 2

One way to ignore the errors due to missing headers is to set SetSuppressIncludeNotFoundError to true in your definition of ASTFrontendAction. An example for the same is given below.

{
public:
    virtual std::unique_ptr<clang::ASTConsumer> CreateASTConsumer(
        clang::CompilerInstance &Compiler, llvm::StringRef InFile)
    {
        Compiler.getPreprocessor().SetSuppressIncludeNotFoundError(true);
        return std::unique_ptr<clang::ASTConsumer>(
            new CustomASTConsumer(&Compiler.getASTContext()));
    }
};

For a complete example using ASTFrontendAction, please visit at https://clang.llvm.org/docs/RAVFrontendAction.html

Heliotaxis

Answer 3

So you want to process C++ code that uses MS headers, and you want access to ASTs so that you can generate code. And Clang won't handle MS headers. So Clang can't be the answer unless it gets a radical upgrade.

You asked for "any solution that can make this work".

Our DMS Software Reengineering Tookit with its C++14 Front End can do this.

DMS provides general parsing, AST construction/inspection/transformation/generation, and inverse parsing (conversion of ASTs back into compilable code), parameterized by language definitions.

The C++ front end provides a full C++14 parser, preprocessor handling, AST construction, and full name and type resolution. It has been tested with GCC and MS VS 2013 header files; we're testing with 2015 header files now. (It also handles MS VS 2013 syntax, too).

It handles the tough parsing cases completely, including the C++ famous "most vexing parse". You can see parse trees at get human readable AST from c++ code.

DMS does not provide Python bindings, nor a direct C++ interface. Rather, it is a standalone tool designed to support the construction of custom tools (e.g., your "little code generator"). It has its own very extensive set of internal APIs, coded in metaprogramming language PARLANSE, which is LISP-like. Other aspects of DMS are managed by using DSLs for lexers, grammars, and transformations. See below.

A word of caution: any tool that can process C++ is gauranteed to be complex. DMS is correspondingly complex, and it takes a while to learn to use it, so you're not going to get instant answers. The good news here is that some things are easier to do. Your code generation problem is likely "read a skeleton file, and then replace key entries in it with problem specific code". If that's the case, a DMS tool with the following code (simplified for presentation here) will likely do the trick:

    ...
    (= myAST (Registry:ParseFile (. filename)  (. `CppVisualStudio2013') ...)
    (Registry:ApplyTransforms myAST (. `MyTransforms.rsl'))
    (Registry:PrettyPrint myAST (concat filename `.modified'))
    ...

with a transforms file MyTransforms.rsl containing source-to-source surface-syntax (e.g, C++ syntax) transformation rules of the conceptual form

        rule rulename if_you_see THIS then replace_by ("-->") THAT

An actual C++ rule might look like (making this up because I don't know your actual code generation goals)

     rule replace_abstraction(s: STRING_LITERAL):
      " abstraction_place_holder(\s) "
     ->  " my_DSL_library(\s,17); "

The ApplyTransforms call above will apply all the rules in this file until none apply any further.

Writing surface syntax transforms, where you can do it, is way easier than making calls on a procedure library (which, like Clang, DMS offers) that hack at the tree.

You can write more complex metaprograms using PARLANSE to apply some rules in one place, other rules someplace else, and you can mix source-to-source transforms with procedural transforms that hack directly at the tree if you want.

If you want more details on what transforms look like, ask and I'll provide a link.

Foveola