Flex not counting lines properly on multiline comments

Question

I`m using the above regex to identify multiline comments in Flex:

[/][*][^*]*[*]+([^*/][^*]*[*]+)*[/]       { /* DO NOTHING */ }

But seems to me that flex/bison is not returning properly the line counter. For example:

Input:

1  ___bqmu7ftc
2 // _qXnFEgQL9Zsyn8Ohtx7zhToLK68xbu3XRrOvRi 
3 /* "{ output 6 = <=W if u7 do nN)T!=$||JN,a9vR)7" 
4  -758939 
5 -31943.6165480 
6 // "RND" 
7  '_' 
8 */
9 [br _int]

Output:

1 TK_IDENT [___bqmu7ftc]
4 [
4 TK_IDENT [br]
4 TK_IDENT [_int]
4 ]

The line should be 9 instead of 4.

Any ideas?

Dimenhydrinate

You'll need to include a bit more of your code :) All I can say is that when I include %option yylineno, the patterns for identifiers, operators and whitespace, and print yylineno and the returned token, it works fine. – Helen

Answer 1

This is the solution I found on Flex manual

Remember to declare int comment_caller; on your definition scope.

%x comment 
%x foo
%%

"/*" {comment_caller = INITIAL;
    BEGIN(comment);
      }

     <foo>"/*"  {
      comment_caller = foo;
      BEGIN(comment);
      }

     <comment>[^*\n]*    {}
     <comment>"*"+[^*/\n]*   {}
     <comment>\n       {++line_num;}
     <comment>"*"+"/"    BEGIN(comment_caller);

Dimenhydrinate

Answer 2

I don't know how you generated the test output in your question, but here's an (almost) minimal example of how to use yylineno. It works fine for me:

%{
  #define ID 257
%}

%option yylineno
%option noinput nounput noyywrap

%%

[[:space:]]+                            { /* DO NOTHING */ }
"//".*                                  { /* DO NOTHING */ }
[/][*][^*]*[*]+([^*/][^*]*[*]+)*[/]     { /* DO NOTHING */ }
[[:alpha:]_][[:alnum:]_]*               { return ID; }
.                                       { return *yytext; }

%%

int main(int argc, char** argv) {
  for (;;) {
    int token = yylex();
    switch (token) {
      case 0:   printf("%4d: %s\n",       yylineno, "EOF"); return 0;
      case ID:  printf("%4d: %-4s[%s]\n", yylineno, "ID", yytext); break;
      default:  printf("%4d: %c\n",       yylineno, token); break;
    }
  }
}

Helen

Hey @rici. Thanks for your time. I just found another fix for this. Im posting the solution. – Dimenhydrinate

@adolfosrs: Como você gosta, mas o meu é mais fácil. Porque você faz o trabalho que faz o flex sem carga? ... In the long run, letting flex handle yylineno is a lot more maintainable. – Helen

Answer 3

I had the same problem with multi-line comments with flex. I used the regex that was suggested in this stackoverflow question(this is same as the regex you mentioned in this question)

This regex also gets the new lines in the multi-line comment. So if you are counting the number of the current line with counting the \n you will get into trouble. Because there could be multi-line comments and the regular expression selects the whole multi-line comment at once. So it doesn't let you to count the new lines.

So I found another way to keep the number of lines even with the regular expression. Explain below:

You know that flex keeps the matched expression inyytext variable. So we can count number of new lines in the multi-line comment and that worked perfectly with any code I tested. Here is my code: note: the numberOfCurrentLine variable is the global variable I used to save the number of the current line.

[/][*][^*]*[*]+([^*/][^*]*[*]+)*[/] {
        // The code below, the counts number of occurance of \n and then adds
        // the number to the numberOfCurrentLine variable
        // to keep the number of current line

        char* str = yytext;
        int i = 0;
        char *pch=strchr(str,'\n');
        while (pch!=NULL) {
            i++;
            pch=strchr(pch+1,'\n');
        }

        numberOfCurrentLine+=i;
    }

This code counts the number of \n in the selected comment and adds it to the global variable that is counting the number of the current line.

The code for counting number of occurrences of a char that I used above is from this post. So with the above code, I always have the right number of the current line and the code works perfectly.

Merta