Multiple parsers in flex/bison : include fails

Question

I am trying to build a program using multiple parsers. I have put the flex / bison files of the different parsers in separate folders to avoid conflicts.

I have used different prefixes to be able to use them. However, my program won't build from scratch, the scanner classes produce errors:

#pragma once

#if ! defined(yyFlexLexerOnce)
#define yyFlexLexer spFlexLexer
#include <FlexLexer.h>
#undef yyFlexLexer
#endif

#include "split_pattern_parser.h"

namespace SP {

class SP_Scanner : public spFlexLexer {
...
}

I get this error :

error: expected class-name before ‘{’ token
 class SP_Scanner : public spFlexLexer {

I manage to compile by removing the #if ! defined(yyFlexLexerOnce) condition several times, and putting it again when I get this error :

src/actions/actions_scanner.h:4:21: error: redefinition of ‘class actFlexLexer’
#define yyFlexLexer actFlexLexer
                 ^
actions_lexer.cpp:32:25: error: previous definition of ‘class actFlexLexer’

(actions is just another lexer)

Even if it is not so big a problem for me (I can continue to work on my project), it gets really problematic when I have to hand out my project: it is not neat to have to explain this procedure to build the code.

Thanks in advance for your help

Isolated

C is not C++ is not C! – Kipp

Yes I know, but on every example I have seen on using flex/bison, they use this weird mix of the two – Isolated

Typically they use C only. Buit your code defintively is not C. – Kipp

Yes, because my whole project is built in C++. Flex / Bison is supposed to support C++, and there are many examples of this. However, they do not feature multiple parsers. – Isolated

Answer 1

You shouldn't be fooling around with yyFlexLexerOnce. That's internal to the implementation of Flex.

You need to declare every variety of FlexLexer which you use, ideally precisely once. You also need to declare FlexLexer itself precisely once. The yyFlexLexerOnce macro is the way whoever put together the C++ interface for flex scanners chose to make this possible. (Multiple header files would have been my choice, but I'm sure they had their reasons.)

So just forget ever having seen yyFlexLexerOnce, and do it the way the manual tells you to:

#define yyFlexLexer spFlexLexer
#include "FlexLexer.h"
#undef yyFlexLexer

// This will not normally be in the same header file, but it could be.

#define yyFlexLexer actFlexLexer
#include "FlexLexer.h"
#undef yyFlexLexer

There are two important issues with FlexLexer.h.

The first one, as illustrated above, is that it is designed to be #included more than once, so it does not have a header guard. That means that you have to ensure that it is never included twice with the same preprocessor define of yyFlexLexer. A good idiom is probably to create a little wrapper:

/*** File: sp_scanner.h ***/

#pragma once
#define yyFlexLexer spFlexLexer
#include "FlexLexer.h"
#undef yyFlexLexer

But that runs into the second issue: yyFlexLexer is automatically #included in the generated scanner implementation file, with an appropriate #define of yyFlexLexer. So you must not #include it (even indirectly) in any inserted code in your .l scanner definition file.

This creates an annoyance in the not-uncommon case where you have other declarations which are required for the implementation of the scanner. The temptation would be to put these declarations in the sp_lexer.h header file (as above), but that won't work because you can't #include "sp_lexer.h" in your scanner definition file. Instead, you need to create yet another header file, and #include it from both the scanner.l file and in the sp_lexer.h file.

Concretely, suppose that you are using the yyclass option to insert the scanner implementation method in a derived class. Obviously, you need to declare this derived class before the generated scanner, and you will also want to declare this derived class in any consumer of the generated scanner. So you will end up with something like this:

/*** File: sp_scanner_internal.h ***/
#pragma once
namespace sp {
  class Scanner : public spFlexLexer {
    /* ... */
  };
}

/*** File: sp_scanner.h ***/
#pragma once
#define yyFlexLexer spFlexLexer
#include "FlexLexer.h"
#undef yyFlexLexer
#include "sp_scanner_internal.h"

/*** File: sp_scanner.l ***/
%option prefix="sp"
%option outfile="sp_scanner.cpp"
%option yyclass="sp::Scanner"

%{
  #include "sp_scanner_internal.h"
  #include "sp_parser.h"
%}

I removed the %option header-file from your example files, because that header file should not be used in C++ projects. See the Flex manual:

The --header-file option is not compatible with the --c++ option, since the C++ scanner provides its own header in yyFlexLexer.h.

I also removed #include <iostream>, since it is included from FlexLexer.h, but of course it is not a problem if you prefer to keep it.

---

Clance

Yes, I also tried that, which seems to be the most natural way to do it. However, the include is oddly conflicting with the lexer cpp file generated by flex, and I get the error in the last grey frame of my question. I'm obliged to alternate between the two versions (with and without yyFlexLexerOnce) to be able to compile. – Isolated

I can give you the github link to help you see the problem, if you want :) – Isolated

@pvallet: that sounds more like your header files lack include guards ( or you are doing something like #include a source file). A minimal compilable example would be useful. – Clance

Here is the repository, I think it is simpler than creating a minimal example: github.com/pvallet/CGA-_interpreter – Isolated

@pvallet: it might be easier for you, but what about the rest of us? But then again, it might not be, because creating minimal examples is a good learning/debugging technique, so not doing so might complicate your life, too. Anyway, if I have time to trawl through all that code later on, I'll let you know. – Clance

Yes you're right, creating minimal examples is usually a good technique to debug, and even to add features to a project. I will provide a simple example so that it is easier for you to read, I'm sorry I didn't do that earlier :/ – Isolated

There you go, I created a minimal example : github.com/pvallet/minimal_example Thank you so much for your time – Isolated

Thanks, that's helpful. I'm not at a keyboard right now, but I think the problem is that you #include scanner.h from within the scanner.l file. That will cause yyFlexLexer to bevincludef twice, because the generated scanner also includes it. I'll update the answer when I get to a machine. – Clance

Ah ok that explains the conflict with lexer.cpp. However, I need to know about the class scanner in the lexer file, otherwise I get a "invalid use of incomplete type" error :/ – Isolated

@pvallet: Yes. You need to separate what you define from what flex defines for you. (I'm not saying this is at all a good design. Personally, I refuse to use the C++ flex interface; I don't believe it produces good C++ code, and I prefer to just use a C scanner.) I'll edit the answer in a few minutes; I just need to produce a minimal compilable answer. – Clance

@pvallet: I updated the answer, finally. Hope it helps. I never use the C++ code generators; for C++, I prefer to generate the C API (even if I compile it as C++ in order to be able to use the C++ standard library), and then wrap my own classes around the generated global symbols. It's a bit more pollution of the global namespace, but it has the advantage of actually letting me write a decent C++ class without having to jump through hoops because of the idiotic and incompatible bison and flex C++ generated interfaces. But that's all opinion so it doesn't belong in an answer. – Clance

Thank you so much for your help ! It's working now. I felt really stuck on this topic, I needed someone's help. I don't really have an opinion on using C or C++, this is the first time I'm using flex/bison, I naturally tried to put the C++ implementation on a C++ project. Well, thanks again, have a good day ! – Isolated