how to encapsulate the lex and yacc of PLY in two seperate class

Question

I am writing my own parser with PLY. I want to encapsulate the lex and yacc respectively

Here is the code for class Lex:

class Lex:
    tokens = (
        'NAME', 'NUMBER',
    )

    literals = ['=', '+', '-', '*', '/', '(', ')']

    # Tokens

    t_NAME = r'[a-zA-Z_][a-zA-Z0-9_]*'

    ...

the Parser code (which use yacc):

class Parser:

    # Parsing rules
    tokens = Lex.tokens

    def p_statement_assign(self, p):
        'statement : NAME "=" expression'
        self.names[p[1]] = p[3]

   ...

    def run(self):
        print self.tokens
        lex.lex(module=Lex)   # ----what should I do here?-----
        parser = yacc.yacc(module=self)
        parser.parse("1+2")

And I got the following error： unbound method t_NUMBER() must be called with Lex instance as first argument (got LexToken instance instead)

I tried using module=Lex for the lex, just like yacc.yacc(module=self),but it didn't work, anyone could tell the solution.

Marela

Answer 1

This is the first time I've tried building a interpreter. I really don't like PLY for modern Python. It feels weird to use docstrings to house the regular expressions. The PLY samples also seem pretty outdated. But, hey, the first version of PLY was released in 2001 and it is still alive 16 years later, so kudos to the author/community for maintaining it.

That being said, here is how I got some encapsulation to work:

class MyLexer(object):
    tokens = (
        'NUMBER',
        'PLUS', 'MINUS', 'TIMES', 'DIVIDE',
        'LPAREN', 'RPAREN',
    )

    t_PLUS = r'\+'
    t_MINUS = r'-'
    t_TIMES = r'\*'
    t_DIVIDE = r'/'
    t_LPAREN = r'\('
    t_RPAREN = r'\)'

    ...

    def __init__(self):
        # Build the lexer
        self.lexer = lex.lex(module=self)


class MyParser(object):

    tokens = MyLexer.tokens

    ...

    def p_expression_binop(self, t):
        '''
        expression : expression PLUS expression
                   | expression MINUS expression
                   | expression TIMES expression
                   | expression DIVIDE expression
        '''
        left_hand_side = t[1]
        right_hand_side = t[3]
        operator = t[2]
        if operator == '+':
            value = left_hand_side + right_hand_side
        elif operator == '-':
            value = left_hand_side - right_hand_side
        elif operator == '*':
            value = left_hand_side * right_hand_side
        elif operator == '/':
            value = left_hand_side / right_hand_side
        else:
            raise AssertionError('Unknown operator: {}'.format(operator))

        t[0] = value

    ...

    def __init__(self):
        self.lexer = MyLexer()
        self.parser = yacc.yacc(module=self)

Daven