Python Multiprocessing error: AttributeError: module '__main__' has no attribute '__spec__'

Question

I'm using Python 3.6 and am trying to follow along with the very first example at the website below (full code also below) and am getting the below error: https://docs.python.org/3.6/library/multiprocessing.html

Error message: AttributeError: module '__main__' has no attribute '__spec__'

Full example code:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

I tried Googling it and searching Stack Overflow but I've only found one other case of this error and it did not have an answer.

Pereira

The posted code works fine on my machine – Reglet

I'm using Anaconda / Spyder with Python 3.6... maybe that has something to do with it? – Pereira

I am using Spyder 3.1.2 with Python 3.6.0 (Anaconda 4.3.1) in Windows 7 – Reglet

I'm on Spyder 3.1.4. I found the code works when I run the script from Command Prompt but gives errors in Spyder. Any idea if there is any configuration steps to make Spyder play nicely with multiprocessing? – Pereira

Are you certain that you are using the correct python executable in Spyder? – Reglet

The Python interpreter is set to "Default (ie the same as Spyder's)". I changed Anaconda's settings to run in an external system terminal and it runs fine. Something about the IPython console is throwing errors. – Pereira

I got this error message today (module '__main__' has no attribute '__spec__') when attempting to do this pytorch tutorial using Spyder. Specifically the line dataiter = iter(trainloader) in the tutorial triggered the error. The __spec__ = None solution below worked for me, but it seems odd that one must resort to this. Have any of the Spyder devs commented on this issue? @CarlosCordoba – Injure

One thing, since it is a problem caused by the Ipython concole, wouldn't it be correct to add the tag "ipython"? I say that based on the answer from user8474060. It might help some people having the same problem. – Thicken

Answer 1

The problem is not with the code / Python 3.6, it is with Spyder.

After some investigation I found that the code runs fine when executed in an external system terminal but not when run in Spyder's IPython console.

I was able to dump the contents of spec and assign them to a variable that was included inside main to allow this code to function within the IPython console.

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    __spec__ = "ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>)"
    with Pool(5) as p:
       print (p.map(f, [1, 2, 3]))

Pereira

I'm actually surprised that worked, since __spec__ is not normally a string. Since it did work, you could probably just use None. – Canzonet

I tried __spec__ = __spec__ and my child processes began to start and stop infinitely. I also checked __spec__ variable when the script is started in system terminal. It is None. So I think __spec__ = None is the "right" way to fix this. – Rambow

Bless you Winand. I'm really glad I tried to execute the pythondocs example before my own code. Got this signature, successfully ran with "spec = None" This issue would have caused me infinite sulk. – Noble

Answer 2

pdb users

The question didn't specifically mention Spyder nor Conda (though it is tagged as such). Hence, I will note that I found this can also happen when using pdb.

E.g.

python -m pdb myprogram.py

Passing __spec__ = None would be a useful workaround if you wanted to persist with pdb.

Alumna

I feel like i"m no dummy, but I actually don't know what you mean here. How do you pass __spec__ = None? – Traci

@cydonian, fwiw, I don't think you're a dummy either. My mind doesn't go back 3 years, but I think what I was trying to say was put this line in the __main__ declaration. I think Winand says it better in the comment above – Alumna

Answer 3

the same probelm in Spyder (Anaconda3, python 3.6) when I try the external terminal.

Error message: AttributeError: module '__main__' has no attribute '__spec__'

I changed the Run console to 'Excute in current console', and applied it. then if that doesnot work, try other conselor and then change back to 'Excute in current console'. Finally, it works. no '__spec__ = None' is needed.

Stuartstub

If anyone still has this pop up randomly (as I did), first try just closing your console, and running again in a new one. Sometimes the settings somehow get messed up. – Botanomancy

Answer 4

I think the best fix or workaround has already been posted, but the real origin was not clarified. And I stumbled upon this issue not in Spyder. So first of all, I would agree with @user8474060's answer except for @Kevin modification:

if __name__ == '__main__':
    __spec__ = None
    with Pool(5) as p:
       print (p.map(f, [1, 2, 3]))

But the origin of the problem is not bound to spyder, nor to multiprocessing but rather an interplay of the two, or more specific of IPython (not spyder). There are a few things to consider:

• multiprocessing does not work in an interactive interpreter, as discussed in the comments of this stack post and explained in the docs (here, only linked to ProcessPoolExecutor, which is jsut a wrapper)
• So you have to use it within a python script, e.g. example.py. As already mentioned in @user8474060's answer, it works nicely when running from a system terminal via e.g. python example.py
• When running a script in Spyder it will run it by default within the current IPython console via runfile, see this stack answer. This does not execute another instance of python as a pure python example.py in an external terminal would do.
• Note that this Run command can be configured to run the file in a dedicated console or an external terminal, as mentioned already by Simin Zuo. However, I don't understand why I didn't work for Simin Zuo with the external terminal.
• Regarding switching between External terminal and Execute in current console, it should always work when running the script for the first time in IPython via runfile, so if it is the first input to the kernel, showing something like In [1]: runfile(/path/to/example.py). It will break upon the second run.
• Now, I would like to mention that it does not affect Spyder, but all IPython systems. I stumbled upon this issue using Jupyter notebooks. It does not matter whether it's used in jupyer lab, jupyter notebook, or any other IDE supporting jupyter notebook.
• In order to run multiprocessing scripts from within jupyter I ran %run example.py. Similar like spyder's runfile the magic command runs the file within the current ipython kernel.
• For jupyter itself it might just be a simple solution to run it as !python example.py, running a fresh python instance from the system shell

Rattigan

Answer 5

Same problem with Spyder (Anaconda3, python 3.7).

I used

from genetic_selection import GeneticSelectionCV

def main(): .... and as I was running the code, an error like this occured:

main_mod_name = getattr(main_module.__spec__, "name", None)

AttributeError: module '__main__' has no attribute '__spec__'

what I did is deleted "__spec__" in main_mod_name = getattr(main_module.__spec__, "name", None)

so I only have this: main_mod_name = getattr(main_module, "name", None)

the code then worked perfectly fine.

Pierette

Answer 6

I stumbled on this question in researching this error. I found that running

with Pool() as mp_pool:

caused the error. Changing this to:

if __name__ == '__main__':
    __spec__ = None
    with Pool() as mp_pool:

resolved it. Python 3.11.4. Fix had nothing to do with Spyder or Anaconda.

Complice

I think this is misleading as it has not only something to do with multiprocessing in python (or python 3.11.4 as you said) but rather with running it from an interactive interpreter. You are right that it is not only a problem in Spyder's interactive interpreter, basically IPython. But also in, for exampe, jupyter. In principle, you cannot run multiprocessing from an interactive interpreter as explained in the docs for ProcessPoolExecutor. But you can run files from interactive sessions as spyder and jupyter. – Rattigan