Supporting different versions of Python

Question

This subject has been disturbing me for some time.

For my Python project I wanted to be able to support Python versions 2.4 to 3.1. I thought a bit about how to do this, and eventually decided to have four separate forks of the source code for four different versions of Python: 2.4, 2.5, 2.6 and 3.1.

I have come to view that as a bad decision, mainly because of Python's distribution annoyances, which I now have to do four times instead of one.

The question is, what to do?

My project is in the scientific computing field. I got the impression that there are still many people who depend on Python 2.4.

Someone suggested I just write my entire project for 2.4, but that is unacceptable for me. That will mean I could not use context managers, and that is something I will not give up on.

How do ordinary Python projects support 2.4? Do they avoid using context managers?

Also, is there any choice but having a separate fork for Python 3.1? I know there are all kinds of hacks for making the same code run on 2.x and 3.x, but one of the reasons I like Python is because the code is beautiful, and I will not tolerate making it ugly with compatibility hacks.

Please, give me your opinion.

Illnatured

Answer 1

There seem be different answers to your problem.

First, if you want to offer all functions for all python versions then yes, you're probably stuck with using the smallest possible functionality subset - hence writing your code for Python 2.4. Or you could backport features from newer interpreters if they're pure python (that's not the case of context managers or coroutines neither).

Or you could split version support into features - if you think there's one (optional) feature which would have great benefit from, let's say, context managers, you can make it available in a separate module and just say that 2.4 users don't have that feature.

In order to support Python 3 take a look at the 2to3 helper, if you write your code properly there's a fair chance you won't need to maintain two separate codebases.

Wee

Answer 2

Yes, you need to write for Python 2.4 syntax to support all of 2.4 - 2.7 in the same codebase.

Some changes in Python 2.6 and 2.7 aim to make it a bit easier to write compatible code with 3.x, but you have to drop support for 2.5 and below to do that.

Outbreed

I see. I'm willing to explore options of NOT doing it in the same codebase. In fact that's what I'm doing now, in the extreme, having separate forks for different versions. Maybe I could do it smarter though. – Illnatured

Answer 3

We have related problem, a large system that supports both jython and cpython back to 2.4. Basically you need to isolate code that needs to be written differently into a hopefully small set of modules, and have things get imported conditionally.

# module svn.py
import sys
if sys.platform.startswith('java'):
    from jythonsvn import *
else:
    from nativesvn import *

In your example you would use tests against sys.version_info, presumably. You could define some simple things in a utility module, that you would use like: from util import *

# module util.py
import sys
if sys.exc_info[0] == 2:
    if sys.exc_info[1] == 4:
        from util_py4 import *
    ...

Then things in util_py4.py like:

def any(seq):                # define workaround functions where possible
    for a in seq:
        if a: return True
    return False
...

Although this is a different problem than porting (since you want to continue to support), this link gives some useful guidance http://python3porting.com/preparing.html (as do a variety of other articles about porting python 2.x).

Your comment that you just cannot live without context managers is a little confusing though. While context managers are powerful and make the code more readable and minimize the risk of errors, you just won't be able to have them in the code of your 2.4 version.

### 2.5 (with appropriate future import) and later
with open('foo','rb')as myfile:
   # do something with myfile

### 2.4 and earlier   
myfile = None
try:
    myfile = open('foo','rb')
    # do something with myfile
finally:
    if myfile: myfile.close()

Since you want to support 2.4 you'll have a body of code that just has to have the second syntax. Will it really be more elegant to write it BOTH ways?

Signify

Answer 4

You could try virtualenv and distribute your application using a single Python version. This may or may not be practical in your case though.

Religieux

Answer 5

First of call you need to keep in mind that Python 2.x shares mostly the same syntax which is backward compatible, new features & additions aside. There are other things to consider that aren't necessarily errors, such as DeprecationWarning messages that while not detrimental, are ugly and can cause confusion.

Python 3.x is backward-INcompatible by design and intends to leave all of the old cruft behind. Python 2.6 introduced many changes that are also in Python 3.x to help ease the transition. To see all of them I would recommend reading up on the What's New in Python 2.6 document. For this reason, it is very possible to write code for Python 2.6 that will also run in Python 3.1, but that is not without its caveats.

Even still there are many minor syntax changes even between 2.x versions that will require you you wrap a lot of your code in try/except blocks, so if this is what you're willing to do then having a 2.x and 3.x branch is totally possible. I think you'll find that you'll be doing a lot of attribute and type tests on your objects to do what you want to do.

I would recommend you check out the code of major projects out there that support various Python versions. Twisted Matrix is the first one that comes to mind. Their code is a wonderful example of how Python code should be written.

In the end, what you're setting out to do will not be easy, so prepare yourself for a lot of work!

Pincer

Where did you see the different branches for 2.4-2.6? I saw different binary downloads for Windows, but the source seems like one folder. Did I miss anything? – Illnatured

set and frozenset were added in Python 2.4. python.org/doc/2.4.4/whatsnew/whatsnew24.html – Ecclesiastic

@cool-RR - No, it was I that missed something. I removed the bad information. @kaizer.se - Thanks, I was confusing the sets module deprecation with the addition of the built-in types. – Pincer

Answer 6

If the differences between versions are not extreme, you can try isolating them into a separate package or module in which you write version-specific code to act as an adaptation layer.

In a trivial fashion, this can be done without the separate module in simple cases, such as when a new version of Python makes standard a package that used to be external, such as (for example) simplejson. We have something similar to this in some code:

try:
    import simplejson as json
except ImportError:
    import json

For non-trivial stuff, such as what you probably have, you wouldn't want such things scattered randomly throughout your code base, so you should collect it all together in one place, when possible, and make that the sole section of your code that is version-specific.

This can't work so well for things where the syntax is different, such as your comment about wanting to use context managers. Sure, you could put the context manager code in a separate module, but that will likely complicate the places where you'd be using it. In such cases, you might backport certain critical features (I think context managers could be simulated somewhat easily) to this adapter module.

Definitely having separate codebases is about the worst thing you could do, so I'd certainly recommend working away from that. At the least, don't arbitrarily use features from newer versions of Python, since although it may look nice to have them in the code (simplifying a particular block of logic perhaps), the fact that you have to duplicate that logic by forking the codebase, even on a single module, is going to more than negate the benefits.

We stick with older versions for legacy code, tweaking as new releases come out to support them but maintaining support for the older ones, sometimes with small adapter layers. At some point, a major release of our code shows up on the schedule, and we consider whether it's time to drop support for an older Python. When that happens, we try to leapfrog several versions, going (for example) from 2.4 to 2.6 directly, and only then start really taking advantage of the new syntax and non-adaptable features.

Mechelle