bison end of file

Question

If I forget to put an empty line at the end of any of my files my program gets a syntax error. The problem is my grammar expects a newline to end the current line. Since a newline doesn't exist bison generates a syntax error because it does not finish the rule.

How do I solve this? I tried making <<EOF>> return MY_EOF BUT when I do that lex crashes a horrible death. I guess there's code in its default EOF that I am not calling. I have no idea what functions they may be. Using EOF create the error symbol EOF is used, but is not defined as a token and has no rules

Hysteresis

Answer 1

You could use a flex EOF rule to append a newline to the input:

<<EOF>> { static int once = 0; return once++ ? 0 : '\n' }

Terrorist

Note that if you want to do this in a re-entrant lexer, you must instead use { static int once = 0; return (once = !once) ? NEWLINE : 0; } since otherwise the first file will prevent newlines being recognized on any future invocations. – Coessential

@Hach-Que Good point - in that case, use: { if( !yyextra->eof ) { yyextra->eof = true; return NEWLINE; } else { yyterminate(); } } and add an 'eof' field to your context. – Sledge

When will we having multiple EOF ? – Sculpturesque

@AnkurGautam: If, after getting EOF, you reset the input (with yyrestart or yy_scan_string or similar), you can start reading again, and will later get another EOF – Terrorist

Answer 2

In your lex file

#define yyterminate() return token::END

In your yacc file

%token END 0 "end of file"

Teens

I havent seen token:: before, are you using the C++ version? I had to change it to return MY_END But %token MY_END 0 "end of file" get me the warning ./test.y:90.11-75: warning: rule never reduced because of conflicts: $accept: program "end of file". It seems to work but i am not 100% sure. – Hysteresis

Answer 3

In fact to catch end of file in lex|flex you can use the yywrap() function which is called by lexical analyzer if end of the input file is reached.

this solution is available by both lex and flex.the callback of yywrap() indicates the EOFso you can reimplement this function and inject the work you need to do at the end of your input stream.

Cholecystectomy

@blueshift No i didn't misspel yywarp, it is the macro, automatically called when we reach the end of the current buffer it returns 1 when there is no buffer to read or 0 if there is – Cholecystectomy

I use flex and have yywrap() but no yywarp(). Are you sure? – Judoka

@Judoka OMG am sorry i didn't get the point from the beginning it is yywrap() – Cholecystectomy

Answer 4

Previous works fine for me.

If you use C Bizon (not C++), just use END for token::END and in yacc file %token END

Had another issue after that, if the macros return not YY_NULL, it never terminates (infinite loop)

It can be solved like this:

bool term = false;
#define yyterminate() return (term = !term)?END : YY_NULL

Constantin

Answer 5

An alternative approach would be to restructure your grammar to not need a newline at the end. As long as your language allows blank lines (usually the case), you can write your grammar using newline as a line separator rather than a line terminator

input: line | input '\n' line ;
line: /* empty */
    | ... various other rules ...

Now if you do have a newline at the end of the input, this gets treated as a blank line after that newline, which is probably fine.

Terrorist