Using custom packages on my python project

Question

I'm doing a few projects in python right now, and I'm trying to figure out how to work with my own versions of existing open source packages.

For instance, I'm using tipfy with zc.buildout, and I've added in the 'paypal' package. Unfortunately it doesn't have a feature I need, so I've forked it on github and added the feature. I will send the original package maintainers a pull request, but whether they accept my additions or not, I'd like to use my version of the package and keep the convenience of having zc.buildout manage my dependencies. How do I do this?

Do I upload my own take on the library to PyPI and prefix it with my name? Wouldn't that unnecessarily pollute the index?

Or should I make and maintain my own index and package repo? Where do I find the format for this? And is it against to terms of the OSS licenses to host my own repo with modified packages with the same names? (I'd rather not modify every file in the project with new namespaces)

I'm sure this problem comes up quite a lot, and not just with python. I can see this happening with Maven and SBT, too... what do people usually do when they want to use their own versions of popular packages?

Wallop

Answer 1

There are two ways you could go about this. I use both, depending on the context the buildout is being used for:

1. Use mr.developer to include packages from a version control system (mr.developer support a wide range of systems, including git). I use this when developing.

2. Build a private package repository. A simple directory listing in Apache is enough. Add the URL to your private repository as a find-links entry:

   [buildout]
   ...
   find-links =
       ...
       http://username:[email protected]/projectname
   

   In this example I also included a username and password (buildout then will authenticate) and a path on the server that is project specific. You can also just build one big private repository for all your projects, of course.

   In that repository you either put complete eggs, or just a tarball of the package. Repositories named in find-links are searched before PyPI.

   I use this method for deployment buildouts. This way the software will use released packages, which makes release management much clearer and simpler.

Hosting your own forks of OSS packages is perfectly fine! This is one of the freedoms you get when using OSS. Note that when you fork GPL code like this, and are distributing it to a third party, you need to make your changes available to them. A package repository is one way of doing complying with this.

Gerlac

Thanks... will go with this. I see it being a really useful way to maintain deps on multiple projects. – Wallop

It is not good to have username and password in the buildout config. It is better to put those in .pypirc: see #37323892 – Gondola

Answer 2

To keep your headache in check, I would really recommend just bundling all such custom code with your package. Say you made a fork of some package. As long as its license allows it, I would just bundle the modified version of package with my code, as if it's just another directory. You can place it locally under package so it will be easily found. Once the developers of package fix what you need, just remove this directory and make it a dependency on an online package once again.

An added bonus of this approach is making distribution to users/customers easier.

Detrude

I'm going to be using package across several projects... really don't want to maintain so many copies of it, especially when I can have it resolved as a dependency. – Wallop

Answer 3

It's been a while since I've used buildout, but if I recall correctly, there is a pip recipe that allows you to use pip requirements files. You can create a requirements file that contains something like this:

-e git+http://<github url>

That will check the package out locally when installing.

Kill

This is interesting... will check it out. – Wallop

Answer 4

Any paths you put in dependency_links in setup.py of your project will be searched first. So just drop your package in a path and set dependency_links to that path.

local_packages = ['file:///path/to/packages']

setup(
 ...
 dependency_links=local_packages,
...)

Setuptool docs cover a little bit more on how it all works, but that's really all there is to it.

Edit: zc.buildout uses the setuptools dependency_links var by default. If you want to turn it off.

Eliaeliades

Will try this.. and what needs to be in the folder is the tar.gz from sdist, right? – Wallop

buildout offers far better options to manage this. No need to polute your python packages with distribution configuration! – Gerlac

I would think that buildout is just masking this machinery rather than duplicating it. Your answer to use find-links sounds very similar. In addition, I see nothing wrong with distribution configs. Every pypi package has one. – Eliaeliades

I'm nullifying the downvote - Martijn's answer is what I'm going to be using, but I did learn something from this. And I don't believe it's wrong or has a negative effect. – Wallop

Well, you as a package maintainer are deciding where people installing dependencies are going to find them. This is a deployment decision, not a packaging decision. – Gerlac

Answer 5

Do I upload my own take on the library to PyPI and prefix it with my name?

No.

Wouldn't that unnecessarily pollute the index?

Obviously. (Also, you're assuming that your extension is actually useful to others. It might not be all that useful to anyone but you. Indeed, your extension might indicate a failure to understand the package.)

Or should I make and maintain my own index and package repo?

Never. That's completely silly. You have the modified package. You don't need a repository unless you're going to complete against PyPI. You could try, but why bother?

---

Here's what you do.

extend

You can easily extend a given package without modifying or forking.

my_paypal.py

from paypal import *

class MyFancyExtension( SomeExistingClass ):
    def override( self, ... )

def my_other_extension( ... ):
    this()
    that()

It's quite easy. It's often better than forking because you preserve the original package untouched by your extensions.

Archiepiscopate

I could do that, but I want to improve the library itself and contribute changes upstream. I'm also using it as a dependency for several of my projects. Based on that it might be best to host a private repo and fall back to the main when it catches up. – Wallop

Note that there are perfectly good reasons to have a custom PyPI server ("index and package repo"). – Gondola

Answer 6

It makes perfect sense to have your own custom PyPI server for your proprietary, private, forked or development packages.

For example pypiserver is lightweight and easy to set up.

Gondola