Scala Parser Token Delimiter Problem

Question

I'm trying to define a grammar for the commands below.

object ParserWorkshop {
    def main(args: Array[String]) = {
        ChoiceParser("todo link todo to database")
        ChoiceParser("todo link todo to database deadline: next tuesday context: app.model")
    }
}

The second command should be tokenized as:

action = todo
message = link todo to database
properties = [deadline: next tuesday, context: app.model]

When I run this input on the grammar defined below, I receive the following error message:

[1.27] parsed: Command(todo,link todo to database,List())
[1.36] failure: string matching regex `\z' expected but `:' found

todo link todo to database deadline: next tuesday context: app.model
                                   ^

As far as I can see it fails because the pattern for matching the words of the message is nearly identical to the pattern for the key of the property key:value pair, so the parser cannot tell where the message ends and the property starts. I can solve this by insisting that start token be used for each property like so:

todo link todo to database :deadline: next tuesday :context: app.model

But i would prefer to keep the command as close natural language as possible. I have two questions:

What does the error message actually mean? And how would I modify the existing grammar to work for the given input strings?

import scala.util.parsing.combinator._

case class Command(action: String, message: String, properties: List[Property])
case class Property(name: String, value: String)

object ChoiceParser extends JavaTokenParsers {
    def apply(input: String) = println(parseAll(command, input))

    def command = action~message~properties ^^ {case a~m~p => new Command(a, m, p)}

    def action = ident

    def message = """[\w\d\s\.]+""".r

    def properties = rep(property)

    def property = propertyName~":"~propertyValue ^^ {
        case n~":"~v => new Property(n, v)
    }

    def propertyName: Parser[String] = ident

    def propertyValue: Parser[String] = """[\w\d\s\.]+""".r
}

Detention

I think you should change your syntax to something like this: todo "link todo to database" :deadline: "next tuesday" :context: "app.model" – Bolen

This is a solution I want to avoid, as I want to keep the Todo grammar as close to a natural language as possible. – Detention

Answer 1

It is really simple. When you use ~, you have to understand that there's no backtracking on individual parsers which have completed succesfully.

So, for instance, message got everything up to before the colon, as all of that is an acceptable pattern. Next, properties is a rep of property, which requires propertyName, but it only finds the colon (the first char not gobbled by message). So propertyName fails, and property fails. Now, properties, as mentioned, is a rep, so it finishes succesfully with 0 repetitions, which then makes command finish succesfully.

So, back to parseAll. The command parser returned succesfully, having consumed everything before the colon. It then asks the question: are we at the end of the input (\z)? No, because there is a colon right next. So, it expected end-of-input, but got a colon.

You'll have to change the regex so it won't consume the last identifier before a colon. For example:

def message = """[\w\d\s\.]+(?![:\w])""".r

By the way, when you use def you force the expression to be reevaluated. In other words, each of these defs create a parser every time each one is called. The regular expressions are instantiated every time the parsers they belong to are processed. If you change everything to val, you'll get much better performance.

Remember, these things define the parser, they do not run it. It is parseAll which runs a parser.

Isolative