What could cause a python module to be imported twice?

Question

As far as I understand, a python module is never imported twice, i.e. the code in the module only gets executed the first time it is imported. Subsequent import statements just add the module to the scope of the import.

I have a module called "TiledConvC3D.py" that seems to be imported multiple times though. I use pdb to print the stack at the top of the code for this module.

Here is the end of the stack trace from the first time the module is executed:

File "<anonymized>/python_modules/Theano/theano/gof/cmodule.py", line 328, in refresh
  key = cPickle.load(open(key_pkl, 'rb'))
File "<anonymized>/ops/TiledConvG3D.py", line 565, in <module>
  import TiledConvC3D
File "<anonymized>/ops/TiledConvC3D.py", line 18, in <module>
  pdb.traceback.print_stack()

It goes on to be executed several more times. However, the complete stack trace for the second time it is called does not show any calls to reload, so these executions should not be occurring:

File "sup_train_conj_grad.py", line 103, in <module>
  dataset = Config.get_dataset(dataset_node)
File "<anonymized>/Config.py", line 279, in get_dataset
  from datasets import NewWiskott
File "<anonymized>/datasets/NewWiskott.py", line 16, in <module>
  normalizer_train = video.ContrastNormalizer3D(sigma, global_per_frame = False, input_is_5d = True)
File "<anonymized>/util/video.py", line 204, in __init__
  self.f = theano.function([input],output)
File "<anonymized>/python_modules/Theano/theano/compile/function.py", line 105, in function
  allow_input_downcast=allow_input_downcast)
File "<anonymized>/python_modules/Theano/theano/compile/pfunc.py", line 270, in pfunc
  accept_inplace=accept_inplace, name=name)
File "<anonymized>/python_modules/Theano/theano/compile/function_module.py", line 1105, in orig_function
  fn = Maker(inputs, outputs, mode, accept_inplace = accept_inplace).create(defaults)
File "/u/goodfeli/python_modules/Theano/theano/compile/function_module.py", line 982, in create
  _fn, _i, _o = self.linker.make_thunk(input_storage = input_storage_lists)
File "<anonymized>/python_modules/Theano/theano/gof/link.py", line 321, in make_thunk
  output_storage = output_storage)[:3]
File "<anonymized>/python_modules/Theano/theano/gof/cc.py", line 1178, in make_all
  output_storage = node_output_storage)
File "<anonymized>/python_modules/Theano/theano/gof/cc.py", line 774, in make_thunk
  cthunk, in_storage, out_storage, error_storage = self.__compile__(input_storage, output_storage)
File "<anonymized>/python_modules/Theano/theano/gof/cc.py", line 723, in __compile__
  output_storage)
File "<anonymized>/python_modules/Theano/theano/gof/cc.py", line 1037, in cthunk_factory
  module = get_module_cache().module_from_key(key=key, fn=self.compile_cmodule)
File "<anonymized>/python_modules/Theano/theano/gof/cc.py", line 59, in get_module_cache
  return cmodule.get_module_cache(config.compiledir)
File "<anonymized>/python_modules/Theano/theano/gof/cmodule.py", line 576, in get_module_cache
  _module_cache = ModuleCache(dirname, force_fresh=force_fresh)
File "<anonymized>/python_modules/Theano/theano/gof/cmodule.py", line 268, in __init__
  self.refresh()
File "<anonymized>/python_modules/Theano/theano/gof/cmodule.py", line 326, in refresh
  key = cPickle.load(open(key_pkl, 'rb'))
File "<anonymized>/ops/TiledConvV3D.py", line 504, in <module>
  import TiledConvG3D
File "<anonymized>/ops/TiledConvG3D.py", line 565, in <module>
  import TiledConvC3D
File "<anonymized>/ops/TiledConvC3D.py", line 22, in <module>
  pdb.traceback.print_stack()

Moreover, I also check the id of __builtin__.__import__ . At the very start of my main script, I import __builtin__ and print id(__builtin__.__import__) before doing any other imports. I also print id(__builtin__.import__) from inside my module that is being imported multiple times, and the value of the id does not change.

Are there other mechanisms besides calling reload and overriding __builtin__.__import__ that could explain my module getting loaded multiple times?

Edva

Answer 1

A Python module can be imported twice if the module is found twice in the path. For example, say your project is laid out like so:

• src/
  • package1/
    • spam.py
    • eggs.py

Suppose your PYTHONPATH (sys.path) includes src and src/package1:

PYTHONPATH=/path/to/src:/path/to/src/package1

If that's the case, you can import the same module twice like this:

from package1 import spam
import spam

And Python will think they are different modules. Is that what's going on?

Also, per the discussion below (for users searching this question), another way a module can be imported twice is if there is an exception midway through the first import. For example, if spam imports eggs, but importing eggs results in an exception inside the module, it can be imported again.

Chlorothiazide

That's a good point, but I don't think that's what's happening in this case. As you can see in the stack traces, the module is imported with the same statement in both cases: "import TiledConvC3D" – Edva

Is it possible that the import is failing mid-way through the first time? For example maybe if there's an exception half-way through importing the module? – Chlorothiazide

Yes, this was the problem. I hadn't realized that import was allowed inside try statements. – Edva

I had been hit by the dual-path issue and came here - thanks for having this up here! – Lardaceous

What would be the keys in the sys.modules for each of the ways of importing spam in your example ? Could you link to some docs (like python-notes.curiousefficiency.org/en/latest/python_concepts/…) ? – Tympanitis

Well, that's awkward. Now if anyone decides to import spam, my module-level variable will be initialized all over again, making it terribly unsafe to use. I guess it's most important for decorators since they run at several points during the import process. – Concessionaire

Answer 2

In case this might help anyone, if you're running Flask in debug mode it might load modules not just twice, but several times. It happened to me and I just couldn't wrap my head around it, until I found this question. Here's more info:

Why does running the Flask dev server run itself twice?

Vulgarism