File "<string>" in python traceback

Question

I am in the middle of refactoring a huge py module to packages - to not break existing code I moved its contents to package/__init__.py module (Adding code to __init__.py) and went on splitting it from there. I noticed at some point that in my tracebacks I get:

Traceback (most recent call last):
      File "<string>", line 656, in DoItemMenu
      File "bash\balt.py", line 2109, in PopupMenu
        link.AppendToMenu(menu,parent,*args)
      File "bash\balt.py", line 2225, in AppendToMenu
        for link in self.links: link.AppendToMenu(subMenu,window,data)
    ...

where the lines in File "<string>" correspond to the particular package/__init__.py module. Moreover PyCharm's debugger displays a "frame not available" line and does not step into the lines in the __init__.py. Why? Is it related to the import pattern?

The code is imported by a launcher class:

class UnicodeImporter(object):
    def find_module(self,fullname,path=None):
        if isinstance(fullname,unicode):
            fullname = fullname.replace(u'.',u'\\')
            exts = (u'.pyc',u'.pyo',u'.py')
        else:
            fullname = fullname.replace('.','\\')
            exts = ('.pyc','.pyo','.py')
        if os.path.exists(fullname) and os.path.isdir(fullname):
            return self
        for ext in exts:
            if os.path.exists(fullname+ext):
                return self

    def load_module(self,fullname):
        if fullname in sys.modules:
            return sys.modules[fullname]
        else:
            sys.modules[fullname] = imp.new_module(fullname)
        if isinstance(fullname,unicode):
            filename = fullname.replace(u'.',u'\\')
            ext = u'.py'
            initfile = u'__init__'
        else:
            filename = fullname.replace('.','\\')
            ext = '.py'
            initfile = '__init__'
        if os.path.exists(filename+ext):
            try:
                with open(filename+ext,'U') as fp:
                    mod = imp.load_source(fullname,filename+ext,fp)
                    sys.modules[fullname] = mod
                    mod.__loader__ = self
                    return mod
            except:
                print 'fail', filename+ext
                raise
        mod = sys.modules[fullname]
        mod.__loader__ = self
        mod.__file__ = os.path.join(os.getcwd(),filename)
        mod.__path__ = [filename]
        #init file
        initfile = os.path.join(filename,initfile+ext)
        if os.path.exists(initfile):
            with open(initfile,'U') as fp:
                code = fp.read()
            exec code in mod.__dict__
        return mod

Brokenhearted

It means that Python was told to compile that module from a string; that probably is PyCharm's fault, not Python. – Highball

How exactly are you running this code at the moment? – Highball

@MartijnPieters: here is the launcher - I am almost certain that before I copy pasted the file in basher/__init__.py it did not load a string though... – Brokenhearted

No, I meant that PyCharm is instructing the Python interpreter to run that file from a string rather than asking Python to import it as a module. I didn't say anything about the module itself doing anything. – Highball

@MartijnPieters: It happens also when I directly run the launcher - I mentioned Pycharm cause it seems to me that it's the same reason it can't step through the code in __init__.py – Brokenhearted

I only now had a chance to look at the launcher. It loads your code as strings with imp.load_source() and only afterwards sets a __file__ attribute on the module object. That is why your __init__ is listed as loaded from <string> and PyCharm cannot debug it without a filename. – Highball

@MartijnPieters: thanks - will run it from before I split it and let you know if this was it – Brokenhearted

Looking over the function again and not being on a mobile phone, I also see a exec code in mod.__dict__ line; and I also notice the imp.load_source() call including a filename. The latter discounts the load_source() call as the source, but the exec call will definitely lead to the behaviour you see. – Highball

Answer 1

The code is not imported in a traditional manner; instead the launcher code uses an exec statement for loading __init__.py files.

Paring down the flow in the launcher load_module() function for a package (so not a path to a module) you get this:

# the fullname module isn't yet loaded
sys.modules[fullname] = imp.new_module(fullname)
initfile = '__init__'  # or u'__init__' if a unicode path was used

# if no .py file was found, so not a module
mod = sys.modules[fullname]
mod.__loader__ = self
mod.__file__ = os.path.join(os.getcwd(),filename)
mod.__path__ = [filename]
#init file
initfile = os.path.join(filename,initfile+ext)
if os.path.exists(initfile):
    with open(initfile,'U') as fp:
        code = fp.read()
    exec code in mod.__dict__
return mod

This creates an empty module object, loads the source manually and executes it as a string, passing in the module namespace as the globals for the executed code. The resulting code object is always going to list <string> in tracebacks:

>>> import imp
>>> mod = imp.new_module('foo.bar')
>>> mod.__file__ = r'C:\some\location\foo\bar'
>>> mod.__path__ = [r'foo\bar']
>>> exec 'raise ValueError("oops")' in mod.__dict__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <module>
ValueError: oops

Because there is no filename associated with the code, PyCharm cannot find the original source either.

The work-around is to use the compile() function to create a code object first, and attaching a filename to that:

>>> exec compile('raise ValueError("oops")', r'C:\some\location\foo\bar\__init__.py', 'exec') in mod.__dict__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\some\location\foo\bar\__init__.py", line 1, in <module>
ValueError: oops

Note that I included __init__.py in the filename; translating that back to the launcher you'd use:

if os.path.exists(initfile):
    with open(initfile,'U') as fp:
        code = fp.read()
    exec compile(code, initfile, 'exec') in mod.__dict__

Highball

Yep the launcher code was beyond me - will go through and digest all this and get back to you - thanks :) – Brokenhearted

Just to be sure I got it - using the compile() function will only do good in my scenario (remember I am in the middle of refactoring so these files are huge) ? Btw, I had noticed that there is no pyc file for the __init__.py, just did not mention this in the question - now I know why. – Brokenhearted

exec has to compile anyway, but by using an explicit compile() call you get to give the resulting code object a filename. It'll be helpful for all code launched by the launcher. – Highball