Python Modules: When one imports them, do they go into memory?

Question

I just finished this exercise for beginners on creating and importing modules in python.

I was wondering does everything in the module get imported into the computer's memory?

Will there be implications later on with memory as the code gets longer and the modules imported become more numerous?

Will I need to know memory management to write resource-efficient code because of this?

Qatar

Answer 1

Your modules are automatically compiled (.pyc files) which are then imported into memory, but you do not have to be affraid of getting out of memory: modules are very small; it is common to have thousands of modules loaded at a time!

You do not need to know memory management as Python does all the work for you.

edit: You can also write a lot of documentation of your code and modules in each module itself (and you should, read about docstrings here) without increasing the size or speed of the modules when loading, because the compiling-step takes out all unnecessary text, comments, etc.

Vermouth

Nice. I'm learning to code again after 20 years and <old man voice>back in my day, memory management was important!</old man voice> – Qatar

I see now why you are using a second-life avatar as your image ;) Good luck; you will find Python a lot of fun! – Vermouth

Hahaha. I actually had a blog for work that needed to be associated with my second life character and I signed up for gravatar with that image. Now, I can't get that off. lol Here's me: twitter.com/#!/Dorje – Qatar

The compiling step does not strip docstrings, only comments, unless you start your Python with -OO, which you probably shouldn’t. – Saideman

Answer 2

Yes and no.

Yes, the modules do get imported into the computers memory, but no you shouldn't be writing resource-efficient code because of this. Python modules are very small (a few KiB, in rare cases a few MiB) and have no significant effect on memory usage.

Karlik

Answer 3

I can only imagine one way that imports could be abused to leak memory; You could dynamically create and import modules of arbitrary name (say, for the purpose of creating a plugin system); use them once and stop using them. If you did this through the normal import machinery, ie with __import__(variable_module_name), those modules would be added to sys.modules and even though they will not be used any further.

The solution is well, don't do that. If you are really creating a plugin system, then dynamic imports of that sort are probably fine, since the plugins would get re-used. If you really need to use dynamically generated, single use code; use eval.

If you really, really need to use importing on dynamically generated code (say, for automated testing), then you probably do need to poke around in sys.modules to erase the modules that you imported. Here's a nice article explaining how to do something like that.

Latchstring

For completeness sake, will del sys.modules['mymodule'] suffice to 'un-import' such dynamically generated module? – Vermouth

@Remi: that's necessary, but it's not actually sufficient in the general case, because other references to the module might be hanging around. A more fleshed out description of what that kind of pattern looks like is in the linked article. – Latchstring

Apart from variables that you defined yourself, is there a risk on any specific 'under the hood' references? Thinking of it, even if you would re-generate a module n-times, using the same name, and import it each time using reload(), you also have to take care of variabales referencing something in a previous module-version. – Vermouth

Thanks for the link about the pyunit Approach. But as I understand it, the rollback-importer also does not solve this reference issue, or does it? – Vermouth

There's no one solution that will certainly free any arbitrary module. A module might call logging.getLogger(__name__) and so there's another possible leak (though small), and there are probably other examples. I'm trying to be a bit vague simply because I don't think this is a good pattern in the first place; a Long running process should not do this sort of thing. The rollback importer exists mainly for testing and development and is much less useful for real, production systems. – Latchstring