difficulty getting c-style comments in flex/lex

Question

I want to make a rule in flex to consume a c-style comment like /* */

i have the following

c_comment "/*"[\n.]*"*/"

But it doesn't ever get matched. Any idea why? if you need more of my code please let me know and I'll submit the whole thing. Thanks to anyone who replies.

Hardback

I'm not sure why you get no match there, but your expression will eat everything in the file between the first "/*" and the last "*/". Your expression to match the contents of the comment must exclude "*/" from being consumed. One way to do this: flex.sourceforge.net/manual/… – Lapidary

thanks, that site was helpful – Hardback

Answer 1

I suggest you use start conditions instead.

%x C_COMMENT

"/*"            { BEGIN(C_COMMENT); }
<C_COMMENT>"*/" { BEGIN(INITIAL); }
<C_COMMENT>\n   { }
<C_COMMENT>.    { }

Do note that there must not be any whitespace between the <condition> and the rule.

%x C_COMMENT defines the C_COMMENT state, and the rule /* has it start. Once it's started, */ will have it go back to the initial state (INITIAL is predefined), and every other characters will just be consumed without any particular action. When two rules match, Flex disambiguates by taking the one that has the longest match, so the dot rule does not prevent */ from matching. The \n rule is necessary because a dot matches everything except a newline.

The %x definition makes C_COMMENT an exclusive state, which means the lexer will only match rules that are "tagged" <C_COMMENT> once it enters the state.

Here is a tiny example lexer that implements this answer by printing everything except what's inside /* comments */.

Hercule

I understand that I am too late to the party, but this regex would incorrectly identify /* rubbish */ */ as a complete block comment(from /* to 2nd */), as opposed to the C style block comments in which opening /* is terminated by the nearest closing */ and the other */ is identified as stray character in the program. The following regex (for flex/lex) handles this case as well "/*"((("*"[^/])?)|[^*])*"*/" Source - [link] (#16160690) – Jodhpur

The problem here was with <C_COMMENT>. { }, If @Hercule would have used follopwing, It would have been resolved <C_COMMENT>[^*\n]*<C_COMMENT>"*"+[^*/\n]*. it would eat up everything except the * followed by /. So in this case, it would end up in first * followed by /. so /* rubbish */ foolosh */, it would comment /* rubbish */ and follow next token for foolish */ – Prelatism

@NitinTripathi, are you sure that this is necessary? I don't have access to flex here, but the documentation states that when multiple rules match, the longest match is chosen. It seems to me that the . rule should never match the * of a closing comment because the closing comment is longer than one of any characters. – Hercule

@NitinTripathi, this very simple flex lexer does not suffer from the /* rubbish */ foolosh */ issue that you (and @Shobhit) describe. – Hercule

I compiled the example "tiny example lexer" into a.out, then ran: echo "/* this is a multiline comment */abc" | ./a.out, where the comment block had four newlines, and the result was four newlines followed by 'abc'. I don't think this is correct-- the entire comment block should be ignored, so the newlines in the comment block should not impact the output. – Moresque

@mwag, looking back I see that dot doesn't match a newline, so it might be as simpe as adding a <C_COMMENT>\n { } rule. I'll test that when I get back to a computer that has flex. – Hercule

that seems to work. I'll try it out in the app I was using the other approach with and see if anything else turns up – Moresque

Answer 2

Here's an example just in case anyone is confused about how to work zneak's answer:

(Basically, you put "%x C_COMMENT" in the first section and the rest in the second section, as explained by his helpful link)

foo.l

%{
// c code..
%}
%x C_COMMENT

%%
"/*"            { BEGIN(C_COMMENT); }
<C_COMMENT>"*/" { BEGIN(INITIAL); }
<C_COMMENT>.    { }

%%
// c code..

Hope that helps someone! Tiff

Storz

Answer 3

Not sure why it's not being picked up but I do know that a pattern of that sort can produce large lexical elements. It's more efficient to detect just the start comment marker and toss everything in the bitbucket until you find the end marker.

This site has code which will do that:

"/*" {
    for (;;) {
        while ((c = input()) != '*' && c != EOF)
            ; /* eat up text of comment */
        if (c == '*') {
            while ((c = input()) == '*')
                ;
            if (c == '/')
                break; /* found the end */
        }
        if (c == EOF) {
            error ("EOF in comment");
            break;
        }
    }
}

Lashondra

I'm not sure it's really good to consume input that way. =/ Isn't that a mix of concerns? – Hercule

I usually tend towards pragmatism than dogmatism :-) – Lashondra

I see only one concern here, and that is eating up the comment so you can proceed with lexing real tokens. However, you could argue that this example is not taking advantage of the abstraction mechanisms that flex offers to make what you're doing clearer. – Lapidary

@Nate, I don't doubt there are better ways to do it, I only offer up one solution. My experiences are with lex/yacc, I've never used flex/bison at all since they weren't available on the platforms I needed to develop on. This is quite a while ago and, in those days, the compiler never even saw comments - they were stripped out by the pre-processor, then a separate program in our development environment: AT&T 3B2 vintage which should give an indication as to my age :-) – Lashondra

IMO this is as good a way as any to solve this particular problem. C-style comments can't be expressed very cleanly in the lex/flex framework so you might as well just write some code to handle it, as you've done. This has the advantage of not requiring lex states, which I feel make a grammar harder to follow. My comment was more in response to zneak's: as long as the code here is strictly doing lexical analysis (which it is), I feel it is in the right place and does not present a problem regarding separation of concerns. – Lapidary

Answer 4

I believe this solution is simpler:

"/*"((\*+[^/*])|([^*]))*\**"*/"

Burgrave

Even if it is correct (difficult for me see), its inefficient since a rather long lexeme might need to be buffered in yytext. – Geis

Answer 5

I've tried several of the suggested solutions and here are the results.

• I could not get the C_COMMENT solution, which has the most up-votes and looks great, to work at all in practice (one of the comments to it explains at least one reason why). It should be downvoted and certainly should not be the highest-voted solution
• The solution from Mugen seemed to work in all of the code I ran it on
• Could not get the solution from Andrey to even compile at all in lex. I looked at the referenced website and using patterns from there did not help

• the answer from paxdiablo worked and had the advantage of being easy to read. I further modified as follows:

  "/*" { int c1 = 0, c2 = input();
         for(;;) {
           if(c2 == EOF) break;
           if(c1 == '*' && c2 == '/')
             break;
           c1 = c2;
           c2 = input();
         }
       }
  

Moresque

It's not entirely clear to me why the solution in my answer is not working for you. In case two flex rules match, the longest rule has precedence. This means that the . rule should never consume the * of a */ token. This lexer doesn't suffer from the issue that you describe: the input /* hello */world */ produces the output world */ as expected. – Hercule

I've added a comment to your answer that explains the issue I had, which is related to embedded newlines in the comment block – Moresque

Answer 6

There's a worked example in the Flex manual, which gets the gnarly edge cases right:

<INITIAL>"/*"         BEGIN(IN_COMMENT);
<IN_COMMENT>"*/"      BEGIN(INITIAL);
<IN_COMMENT>[^*\n]+   // eat comment in chunks
<IN_COMMENT>"*"       // eat the lone star
<IN_COMMENT>\n        yylineno++;

Sheers

Answer 7

Another example:

"/*"([^*]*|(\*+[^/]))*"*/"

Scammon

Answer 8

The worked example is:

\/\*([^*]|[\r\n]|(\*+([^*/]|[\r\n])))*\*+\/

which found in ostermiller.org

Lullaby

In Flex, [^*] includes both \r and \n (and every other 8-bit code except for *) so the |[\r\n] is unnecessary. (Just like most of the other regex environments in the linked article, with the exception of nedit.) – Quinze

Answer 9

ignore the space and newline

"/*"
  (
    "/"*
      (
        "*"*
        [^*/]
        "/"*
      )*
    "*"*
  )*
"*/"

Kenneth C. Louden - Compiler Construction_ Principles and Practice (1997) section 2.2.3

Dorothadorothea