Way to have compiled python files in a separate folder?

Question

Is it possible to have Python save the .pyc files to a separate folder location that is in sys.path?

/code
    foo.py
    foo.pyc
    bar.py
    bar.pyc

To:

/code
   foo.py
   bar.py
/code_compiled
   foo.pyc
   bar.pyc

I would like this because I feel it'd be more organized. Thanks for any help you can give me.

Baldwin

How you tried Python 3.2? It implements PEP 3147: PYC Repository Directories (python.org/dev/peps/pep-3147). – Cohabit

If you don't care for the .pyc files, you could use os.system('del *.pyc') (windows, e.g.) at the end of your script. – Quartile

Answer 1

Update:

In Python 3.8 -X pycache_prefix=PATH command-line option enables writing .pyc files to a parallel tree rooted at the given directory instead of to the code tree. See $PYTHONPYCACHEPREFIX envvar^{credits: @RobertT' answer}

The location of the cache is reported in sys.pycache_prefix (None indicates the default location in __pycache__ [since Python 3.2] subdirectories).

To turn off caching the compiled Python bytecode, -B may be set, then Python won’t try to write .pyc files on the import of source modules. See $PYTHONDONTWRITEBYTECODE envvar^{credits: @Maleev's answer}

---

^{Old [Python 2] answer:}

There is PEP 304: Controlling Generation of Bytecode Files. Its status is Withdrawn and corresponding patch rejected. Therefore there might be no direct way to do it.

If you don't need source code then you may just delete *.py files. *.pyc files can be used as is or packed in an egg.

Typecast

You could edit your answer with something insightful about the update. – Ottava

Err. See further down. The short answer is no way except the really hacky in Python 2.x; trivial in Python 3.2 and later. – Boardinghouse

Answer 2

In the dark and ancient days of 2003, PEP 304 came forth to challenge this problem. Its patch was found wanting. Environment variable platform dependencies and version skews ripped it to shreds and left its bits scattered across the wastelands.

After years of suffering, a new challenger rose in the last days of 2009. Barry Warsaw summoned PEP 3147 and sent it to do battle, wielding a simple weapon with skill. The PEP crushed the cluttering PYC files, silenced the waring Unladen Swallow and CPython interpreter each trying to argue its PYC file should be triumphant, and allowed Python to rest easy with its dead ghosts occasionally running in the dead of night. PEP 3147 was found worthy by the dictator and was knighted into the official roles in the days of 3.2.

As of 3.2, Python stores a module's PYC files in __pycache__ under the module's directory. Each PYC file contains the name and version of the interpreter, e.g., __pycache__/foo.cpython-33.pyc. You might also have a __pycache__/foo.cpython-32.pyc compiled by an earlier version of Python. The right magic happens: the correct one is used and recompiled if out of sync with the source code. At runtime, look at the module's mymodule.__cached__ for the pyc filename and parse it with imp.get_tag(). See the What's New section for more information.

TL;DR - Just works in Python 3.2 and above. Poor hacks substitute for versions before that.

Boardinghouse

PEP 3147 is only partial solution. __pycache__ directories still clutter source code. Main annoyance for me with pycs is directories left undeleted after svn switch or such. – Lollar

Seems like an aesthetic problem: the cached .pyc will only be used for exact timestamp matches; probably from filecmp.filecmp(shallow=True). While not beautiful, extra caches should not be used. – Boardinghouse

Pray tell that, like most things in 3.2, this was also included in 2.7? – Obovate

Damnit - just ran the suggested test in PEP 3147 and got back the answer False. The test is: import imp; hasattr(imp, 'get_tag'). – Obovate

Specifically, it seems that the PEP was finalized after the first beta of 2.7 had already been released. Still... maybe there is hope? They backported the pip module in 2.7.9... maybe they can backport this in 2.7.11? – Obovate

@Obovate Or may be we can just abandon python2 already! :-D – Scraggly

@Scraggly - No Fabric in Python 3. Also, almost every OS still comes with Python 2. These are the two main reasons I'm still stuck on 2.7. – Obovate

As the new kingdom arose, those of the old kingdom coveted the treasures of the new land. Many raids for class decorators, integer division, and print functions returned victorious. It was only when farmers started bringing back carts of topsoil that the old ones decided to migrate. – Boardinghouse

@CharlesMerriam Out of curiousity, what does the topsoil allegory refer to? – Learn

Topsoil is a moderately valuable good but bulky and not worth transporting. Not all features were back-ported; but is long enough ago I do not remember. – Boardinghouse

Answer 3

And only almost ten years later, Python 3.8 finally provides support for keeping bytecode in separate parallel filesystem tree by setting environment variable PYTHONPYCACHEPREFIX or using -X pycache_prefix=PATH argument (official doc here).

Libbie

Answer 4

If you're willing to sacrifice bytecode generation altogether for it, there's a command line flag:

python -B file_that_imports_others.py

Can be put into IDE's build/run preferences

Kozak

Answer 5

I agree, distributing your code as an egg is a great way to keep it organized. What could be more organized than a single-file containing all of the code and meta-data you would ever need. Changing the way the bytecode compiler works is only going to cause confusion.

If you really do not like the location of those pyc files, an alternative is to run from a read-only folder. Since python will not be able to write, no pyc files ever get made. The hit you take is that every python file will have to be re-compiled as soon as it is loaded, regardless of whether you have changed it or not. That means your start-up time will be a lot worse.

Racy

Answer 6

I disagree. The reasons are wrong or at least not well formulated; but the direction is valid. There are good reasons for being able to segregate source code from compiled objects. Here are a few of them (all of them I have run into at one point or another):

• embedded device reading off a ROM, but able to use an in memory filesystem on RAM.
• multi-os dev environment means sharing (with samba/nfs/whatever) my working directory and building on multiple platforms.
• commercial company wishes to only distribute pyc to protect the IP
• easily run test suite for multiple versions of python using the same working directory
• more easily clean up transitional files (rm -rf $OBJECT_DIR as opposed to find . -name '*.pyc' -exec rm -f {} \;)

There are workarounds for all these problems, BUT they are mostly workarounds NOT solutions. The proper solution in most of these cases would be for the software to accept an alternative location for storing and lookup of these transitional files.

Bowfin

Answer 7

Since Python 3.2 has been implemented PEP 3147: this means that all .pyc files are generated inside a __pycache__ directory (there will be a __pycache__ directory for each directory where you have Python files, and it will hold .pyc files for each version of Python used on the sources)

Forkey

Answer 8

There is ongoing pep that will enable building bytecode to magic directory.

Basically all python files will be compiled to directory __pythoncache__.

Sikang

Answer 9

For Python 3.8 or higher:

The PYTHONPYCACHEPREFIX setting (also available as -X pycache_prefix) configures the implicit bytecode cache to use a separate filesystem tree, rather than the default __pycache__ subdirectories within each source directory.

The location of the cache is reported in sys.pycache_prefix (None indicates the default location in __pycache__ subdirectories).

Ezequieleziechiele

Answer 10

"I feel it'd be more organized" Why? How? What are you trying to accomplish?

The point of saving the compiler output is to save a tiny bit of load time when the module gets imported. Why make this more complex? If you don't like the .pyc's, then run a "delete all the .pyc's" script periodically.

They aren't essential; they're helpful. Why turn off that help?

This isn't C, C++ or Java where the resulting objects are essential. This is just a cache that Python happens to use. We mark them as "ignored" in Subversion so they don't accidentally wind up getting checked in.

Teal

> Why? Apparently because they make ls output or Windows Explorer file listing more viusally cluttered. Subjective but fairly legitimate concern, isn't it? – Kozak

@Maleev: "visually cluttered"? Explorer can be sorted by file type to put the .pyc files elsewhere. Linux ls can be used with a wild-card (i.e., ls *.py). Trying to rearrange the files is a large waste of time when you have to trivial ways to reduce visual clutter. – Teal

Moving the .pyc file location is not the same as turning the feature off. Nor does doing so make the Python interpreter more complex. The argument made is predicated on a misunderstanding of what a cache actually is (a temporary storage facility designed to enhance locality of reference). Caches are, by definition, both lossy and localized. Strewing .pyc files throughout a Python project's hierarchy defeats the purpose of a cache. At best you can say it's an optimization at the cost of file hierarchy hygiene. – Wolfenbarger

Guido disagrees. See the discussion thread on PEP 3147 where this exact problem was fixed - he says that the biggest selling point on the feature is to fix the clutter. A good way down on this mailing thread you'll find Guido's comment "this is a major selling point of this PEP!" with regard to how files will be less cluttered/better organized. mail.python.org/pipermail/python-dev/2010-April/099414.html – Obovate