Python Code Organization Question : Eggs + Packages + Buildout + Unit Tests + SVN

Question

I have several python projects that share common modules. Until now, I've been ... ahem ... keeping multiple copies of the common code and synchronizing by hand. But I'd clearly prefer to do something else.

It looks to me now, as if zc.Buildout maybe what I need. I guess that what I should be doing is putting each reusable component of my system into a separate egg, and then using buildout to assemble them into projects.

I'm also thinking that for any particular module, I should put the unit-tests into a separate package or egg, so that I'm not also installing copies of the component's unit-tests in every project. I only want to unit-test in a place where my library is developed, not where it's just being used.

So maybe I want something like this

projects
  lib1
    tests
    code
  lib2
    tests
    code
  app1
    tests 
    appcode
  app2
    tests
    appcode

etc.

Where both app1 and app2 are independent applications with their own code and tests, but are also including and using both lib1 and lib2. And lib1/test, lib1/code, lib2/test, lib2code, app1, app2 are separate eggs. Does this sound right?

However, I now get confused. I assume that when I develop app1, I want buildout to pull copies of lib1, lib2 and app1 into a separate working directory rather than put copies of these libraries under app1 directly. But how does this work with my SVN source-control? If the working directory is dynamically constructed with buildout, it can't be a live SVN directory from which I can check the changes back into the repository?

Have I misunderstood how buildout is meant to be used? Would I be better going for a completely different approach? How do you mix source-control with module-reuse between projects?

Update : thanks to the two people who've currently answered this question. I'm experimenting more with this.

Shively

Why is $PYTHONPATH not sufficient? – Kiushu

I mean: you keep single instance of each project and modify $PYTHONPATH to find modules you need from other projects. – Kiushu

Answer 1

Do not separate the tests from your code, you need to keep the two closely together. It's not as if tests take up that much disk space or any memory! And tests can be extremely instructive to your library users.

For library packages, include a buildout.cfg and bootstrap.py file with your package to make running the tests easy. See, for example, the plone.reload package; note how it uses zc.recipe.testrunner parts to create a test script that'll autodiscover your tests and run them. This way you can ensure that your library packages are always tested!

Then, your app packages only need to test the integration and application-specific code. Again, include the tests with the package itself, you want to not forget about your tests when working on the code. Use zc.recipe.testrunner parts in your buildout to discover and run these.

Last but not least, use mr.developer to manage your packages. With mr.developer, you can check out packages as your work on them, or rely on the released versions if you do not need to work on the code. A larger project will have many dependencies, many of which do not need you tweaking the code. With mr.developer you can pull in source code at will and turn these into development eggs, until such time you release that code and you can dismiss the checkout again.

To see an actual example of such a project buildout, look no further than the Plone core development buildout.

The sources.cfg file contains a long list of SCM locations for various packages, but normally released versions of eggs are used, until you explicitly activate the packages you plan to work on. checkouts.cfg lists all the packages checked out by default; these packages have changes that will be part of the next version of Plone and have not yet been released. If you work on Plone, you want these around because you cannot ignore these changes. And testing.cfg lists all the packages you need to test if you want to test Plone, a big list.

Note that Plone's sources come from a wide variety of locations. Once you start using buildout and mr.developer to manage your packages, you are free to pull your source code from anywhere.

Disused

Answer 2

This is why you have the site module. It sets the internal sys.path to include all packages and modules from

• lib/site-packages -- including directories, eggs and .pth files.
• PYTHONPATH

This way there is exactly one working copy of your libraries.

There are an unlimited ways to make use of this. Here are two.

1. In each lib, write a setup.py that deploys your lib properly. When you make changes, you do an svn up to collect the changes and a python setup.py install to deploy the one working copy that every application shares.

2. In each app, either depend on things being in the PYTHONPATH environment variable. Be sure that projects/lib1 and projects/lib2 are won the PYTHONPATH. Each app then shares the one working copy of the various libraries.

Centerboard

Answer 3

I've use the following structure quite effectively. in SVN.

Lib1/
   branches/
   tags/
   trunk/
     lib1/
     tests/
     setup.py
Lib2
   branches/
   tags/
   trunk/
     lib2/
     tests/
     setup.py
App1
   branches/
   tags/
   trunk/
     app1/
     tests/
     setup.py
App2
   branches/
   tags/
   trunk/
     app2/
     tests/
     setup.py

I would then create my dev workspace( I use eclipse/pydev) as follows, checking out from either trunk or a branch.

Lib1/
   lib1/
   tests/
   setup.py
Lib2/
   lib2/
   tests/
   setup.py
App1/
   app1/
   tests/
   setup.py
App2/
   app2/
   tests/
   setup.py

I would then use either eclipse project dependencies setup python path, which works well with eclipse code completion. setup.py also works but does not support having multiple workspaces well.

For deployment, I use create a single zip with the following structure.

App1/
   lib1-1.1.0-py2.5.egg/
   lib2-1.1.0-py2.5.egg/
   app1/
   sitecustomize.py

App2/
   lib1-1.2.0-py2.5.egg/
   lib2-1.2.0-py2.5.egg/
   app2/
   sitecustomize.py

I don't use setup install because I want to support multiple versions of the app, also I have some control of the runtime environment, so I don't package python with my deployment but should be easy to add Python into the deployment package if it's needed.

Coronagraph

Answer 4

I'd consider each application and library an egg and use one of the examples already given in terms of laying it out in SVN. Really, the VCS end of things should not be an issue.

Then, for testing each application/library or combination, I'd set up a virtualenv and install each package either via setup.py develop or via actually installing it. Virtualenvwrapper is also a helpful tool to manage these environments as you can simply do things like:

mkvirtualenv lib1-dev

And then later:

workon lib1-dev

Virtualenv uses the PYTHONPATH to gain finer control of the packages installed. Likewise, you can use create environments with:

virtualenv --no-site-packages some-env

And it will leave out any references to your actual system site-packages. This is helpful because not only can you test your library/application, but you can also verify you have the correct dependencies on installation.

Cowen