Problems with PLY LEX and YACC

Question

I am trying to run the first part of a simple example of the PLY but I encounter a strange error. When I run the following code, it gives me an error regarding lex.lex() Anyone knows what the problem is?

import ply.lex as lex

tokens = [ 'NAME','NUMBER','PLUS','MINUS','TIMES', 'DIVIDE', 'EQUALS' ]
t_ignore =  '\t'
t_PLUS = r'\+'
t_MINUS = r'-'
t_TIMES = r'\*'
t_DIVIDE = r'/'
t_EQUALS = r'='
t_NAME = r'[a-zA-Z_][a-zA-Z0-9_]*'
def t_NUMBER(t):
    r'\d+'
    t.value = int(t.value)
    return t

lex.lex() # Build the lexer

This is the error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-e527bd224769> in <module>()
     14     return t
     15 
---> 16 ply.lex.lex() # Build the lexer

c:\python27\lib\site-packages\ply\lex.pyc in lex(module, object, debug, optimize, lextab, reflags, nowarn, outputdir, debuglog, errorlog)
    904     linfo.get_all()
    905     if not optimize:
--> 906         if linfo.validate_all():
    907             raise SyntaxError("Can't build lexer")
    908 

c:\python27\lib\site-packages\ply\lex.pyc in validate_all(self)
    578         self.validate_tokens()
    579         self.validate_literals()
--> 580         self.validate_rules()
    581         return self.error
    582 

c:\python27\lib\site-packages\ply\lex.pyc in validate_rules(self)
    820 
    821         for module in self.modules:
--> 822             self.validate_module(module)
    823 
    824     # -----------------------------------------------------------------------------

c:\python27\lib\site-packages\ply\lex.pyc in validate_module(self, module)
    831 
    832     def validate_module(self, module):
--> 833         lines, linen = inspect.getsourcelines(module)
    834 
    835         fre = re.compile(r'\s*def\s+(t_[a-zA-Z_0-9]*)\(')

c:\python27\lib\inspect.pyc in getsourcelines(object)
    688     original source file the first line of code was found.  An IOError is
    689     raised if the source code cannot be retrieved."""
--> 690     lines, lnum = findsource(object)
    691 
    692     if ismodule(object): return lines, 0

c:\python27\lib\inspect.pyc in findsource(object)
    524     is raised if the source code cannot be retrieved."""
    525 
--> 526     file = getfile(object)
    527     sourcefile = getsourcefile(object)
    528     if not sourcefile and file[:1] + file[-1:] != '<>':

c:\python27\lib\inspect.pyc in getfile(object)
    401         if hasattr(object, '__file__'):
    402             return object.__file__
--> 403         raise TypeError('{!r} is a built-in module'.format(object))
    404     if isclass(object):
    405         object = sys.modules.get(object.__module__)

TypeError: <module '__main__' (built-in)> is a built-in module

Withoutdoors

Copy pasted the code and works fine for me in Python 2 and 3. Only thing I get is a warning: WARNING: No t_error rule is defined – Ehtelehud

Answer 1

You are trying to run ply from some kind of REPL (ipython, at a guess).

For whatever reason, that won't work. Ply insists that the grammar be a module, which means it must be in a file. The error precisely indicates that there was no file associated with the grammar source.

Aceto

I think this should rather be the accepted answer, as this states the same as the accepted answer - which is answered by the questioner himself. – Reptile

Is there any way to fix this in a jupyter notebook? – Tarryn

@EmmanuelMess: Not that I know of. I think Sly will work in a notebook, though. – Aceto

Answer 2

It turned out that the issue is that I was running the code via iPython Notebook and it didn't like it for some reason. Saved the code as a regular .py file and ran it through a command prompt and no errors occurred!

P.S. I appreciate it if anyone can elaborate on why the code doesn't run in an iPython Notebook environment!

Withoutdoors

Well, yeah. That's what I said in my answer. "Ply insists that the grammar be a module, which means it must be in a file." An iPython Notebook is not a file. Files have names and live on the filesystem. I don't know why Ply has this restriction -- I guess it has to do with caching the parsing table -- but it definitely does have it. – Aceto

Seems a bit odd ... any quick edit to make it work in a REPL? – Blavatsky

@JohnDrinane: Not that I know of. – Aceto