Expecting a LexBuffer<char> but given a LexBuffer<byte> The type 'char' does not match the type 'byte'

Question

Type mismatch. Expecting a LexBuffer<char> but given a LexBuffer<byte> The type 'char' does not match the type 'byte'

This is the error message that I am getting while using fslex. I have tried manually checking every single occurrence of lexbuf and its type. It's LexBuffer<char> everywhere. But still the compiler is giving me the above error. Can you please tell me why this error occurs and how to go about resolving it.

{
    open System
    open Microsoft.FSharp.Text.Lexing
    open Microsoft.FSharp.Text.Parsing

    let lexeme (lexbuf : LexBuffer<char>) = new System.String(lexbuf.Lexeme)
    let newline (lexbuf:LexBuffer<char>) = lexbuf.EndPos <- lexbuf.EndPos.NextLine
    let unexpected_char (lexbuf:LexBuffer<char>) = failwith ("Unexpected character '"+(lexeme lexbuf)+"'")
}

let char = ['a'-'z' 'A'-'Z']
let digit = ['0'-'9']
let float = '-'?digit+ '.' digit+
let ident = char+ (char | digit)*
let whitespace = [' ' '\t']
let newline = ('\n' | '\r' '\n')

rule tokenize = parse
    | "maximize" { MAXIMIZE }
    | "minimize" { MINIMIZE }
    | "where" { WHERE }
    | '+' { PLUS }
    | '-' { MINUS }
    | '*' { MULTIPLY }
    | '=' { EQUALS }
    | '>' { STRICTGREATERTHAN }
    | '<' { STRICTLESSTHAN }
    | ">=" { GREATERTHANEQUALS }
    | "<=" { LESSTHANEQUALS }
    | '[' { LSQUARE }
    | ']' { RSQUARE }
    | whitespace { tokenize lexbuf }
    | newline { newline lexbuf; tokenize lexbuf }     
    | ident { ID (lexeme lexbuf) }
    | float { FLOAT (Double.Parse(lexeme lexbuf)) } 
    | ';' { SEMICOLON }
    | eof { EOF }
    | _ { unexpected_char lexbuf } 

Stlaurent

Welcome to SO. Note that angle brackets in your question text need to be escaped in backquotes to get displayed correctly - I have made this edit for you. – Khalsa

Can't debug code we can't see. Please post your code. – Balbo

This the lexer definition. The context is to write a DSl for a Linear Program using F#. – Stlaurent

Answer 1

There was a mistake with my lexer file definition I believe, it compiled when I made the following my lexer definition. Experts can throw more insight into the reasons, while the understanding that I have is the type of the lexbuf that is used in the lexer should somehow be related to the definition that the parser generates

{

open System
open LanguageParser
open Microsoft.FSharp.Text.Lexing
open Microsoft.FSharp.Text.Parsing
open System.Text

let newline (lexbuf:LexBuffer<_>) = lexbuf.EndPos <- lexbuf.EndPos.NextLine

}

let char = ['a'-'z' 'A'-'Z']
let digit = ['0'-'9']
let float = '-'?digit+ '.' digit+
let ident = char+ (char | digit)*
let whitespace = [' ' '\t']
let newline = ('\n' | '\r' '\n')

rule tokenize = parse
    | "maximize" { MAXIMIZE }
    | "minimize" { MINIMIZE }
    | "where" { WHERE }
    | '+' { PLUS }
    | '-' { MINUS }
    | '*' { MULTIPLY }
    | '=' { EQUALS }
    | '>' { STRICTGREATERTHAN }
    | '<' { STRICTLESSTHAN }
    | ">=" { GREATERTHANEQUALS }
    | "<=" { LESSTHANEQUALS }
    | '[' { LSQUARE }
    | ']' { RSQUARE }
    | whitespace { tokenize lexbuf }
    | newline { newline lexbuf; tokenize lexbuf } 
    | ident { ID <| Encoding.UTF8.GetString(lexbuf.Lexeme) }
    | float { FLOAT <| Double.Parse(Encoding.UTF8.GetString(lexbuf.Lexeme)) } 
    | ';' { SEMICOLON }
    | eof { EOF }
    | _ { failwith ("Unexpected Character") } 

Stlaurent

It "works" because you have removed the cases handling floats and idents. The function "lexeme" is no longer used, but avoiding using this function is not a workable solution in general. – Lemuel

Answer 2

Maybe you need to generate a unicode lexer. A unicode lexer works with a LexBuffer<char> rather than LexBuffer<byte>.

• The "unicode" argument to FsLex is optional, but if enabled generates a unicode lexer.

http://blogs.msdn.com/dsyme/archive/2009/10/21/some-smaller-features-in-the-latest-release-of-f.aspx

Cristal

fslex --unicode Your.fsl - in case other people stumble upon this. – Taproom

Answer 3

Have you tried inserting an explicit cast?

Tralee

I'm guessing you're right; a lot of people seem to be thrown by the fact that there are no implicit casts in F#. – Trader