How to determine line/column numbers from boost::spirit::lex tokens?

Question

In its answer to this question, hkaiser said he would write an example of a token type which carries code position information. I really cannot find anything about this.

Can anyone point me to such an example ?

Gasaway

Answer 1

He was probably referring to position_token, which did in fact get released, although there's little documentation. This SO question has good coverage of its usage.

Graff

OK thanks. Do you know if there is an example or a tutorial for how to use it somewhere ? – Gasaway

Hartmut gives further details in his answer here: #5964728 – Graff

Within the Boost sources you can find it at libs/spirit/example/qi/compiler_tutorial/conjure3/lexer.hpp – Graff

This source code is quite complex! I cannot figure out what information is stored into the tokens, and how you read it... – Gasaway

position_token has methods begin() and end() which return iterators to the start and end of the underlying input that they matched. You could use this, for example, to determine the byte offset, within a file, of the matched input. – Graff

I don't see any sign that it tracks line and column information, though. – Graff

@JeffTrull for that I think you'll need to use the line_pos_iterator. – Shoestring

@cv_and_he that sounds great. Care to make it an answer? I think it's probably what OP is looking for. – Graff

@cv_and_he hm actually I think he needed a lex-compatible iterator. line_pos_iterator seems to be for character-based input – Graff

@JeffTrull It is. I think what the OP needs is a combination of both position_token and line_pos_iterator. – Shoestring