How can I create a local own PyPI repository index without a mirror?

Question

We have several own Python packages and want to create local PyPI repository for them using simple interface like https://pypi.python.org/simple/

This repository I want to create for local only without any mirrors due to security reason, and it will be put under Apache's control.

The command pypimirror looks has to be initialized once, which it needs to mirror.

How can I generate a PyPI Simple Index based on local Python packages?

Is there another simple script for this?

Legislature

possible duplicate of Setting up a local PyPi server with custom set of packages – Midrib

possible duplicate of How to roll my own pypi? – Gertudegerty

Answer 1

Take a look at pip2pi. It seems to be exactly what you are looking for.

Brigidbrigida

this is the most simple way to handle for my case – Legislature

It looks like pip2pi is intended for public packages that are already on PyPI, is that correct? I thought the original question was about creating a repository for packages developed in-house. – Whaley

This seems to be unmaintained. A major bug was reported in April, and has not been addressed by the maintainers. A PR to fix it is also going unnoticed. – Naman

I agree with @BrianMcCutchon. I just wasted two days trying to get pip2pi to work before discovering a bug (or bugs) that was causing it not to work for me. If you're developing a web application, I'd suggest pypiserver as that's what I ended up using. If you're developing Python applications, dev.pi would probably be a better choice. Just don't use pip2pi. – Immedicable

@Immedicable Well, the PR was finally merged, and the maintainer added another maintainer, so it might be better maintained now. – Naman

wrote a script that generates a simple repository with N recent versions of 4000 most used packages on pypi. Advantage is it can hold multiple versions as in pypi. gist.github.com/harisankar-krishna-swamy/… – Levitus

Answer 2

We had a similar need at my company. Basically how can we upload "closed source" packages to an index while being able to install them as if they were on PyPI?

We have sponsored a project called devpi which acts as a PyPI cache (packages you access from PyPI will be cached on your server) as well as a powerful and fast index server. The documentation is available at devpi: PyPI server and packaging/testing/release tool.

Next on the roadmap is mirroring for multi geos deployment. To kick the tires on your machine takes about 5 minutes (look at the quick start guides). Finally, devpi is compatible with both pip and easy_install (i.e., you do not need the devpi client installed on your machine).

Parsifal

Little overkill for my simple case, but i vote it since it looks quite stable and easy to use. – Legislature

devpi is exactly what I was looking for. It acts as a pypi server, and you can use the default pip command for managing packages. Easy to setup and easy in use. For use in a virtual environment, the daemon can be started/stopped in post(de)activate hooks. – Danialdaniala

Answer 3

The simplest way is to organize the package distfiles into package-named directories and run a simple HTTP server. No extra packages needed; Python's standard library is enough. Directory structure example:

└── repodir
    ├── setuptools
    │   ├── setuptools-38.1.0-py2.py3-none-any.whl
    │   ├── setuptools-38.1.0.zip
    │   ├── setuptools-39.2.0-py2.py3-none-any.whl
    │   └── setuptools-39.2.0.zip
    ├── wheel
    │   └── wheel-0.31.1-py2.py3-none-any.whl
    ...

Start the server:

cd repodir/
python3 -m http.server 9000
# Or for Python 2:
python2 -m SimpleHTTPServer 9000

The local repository is up and running. Now you can pass the repository to pip:

pip install wheel --extra-index-url=http://127.0.0.1:9000

Or even persist the repository URL in the pip.conf to not to enter it each time:

# pip.conf
[global]
extra-index-url=http://127.0.0.1:9000

Reference: Python Packaging user Guide, Hosting your own simple repository

Encrata

Answer 4

There is nothing special about the mirror, and you can use mod_rewrite to set it up yourself.

1. Dump your packages in a directory that is mapped to a URL. Here I am using /url/to/my/pypi/ an an example. The folder hierarchy should be /foo/bar/simple/[name of package]/[name of tarball]

2. Add the following to .htaccess or the global configuration for that directory where you packages are. The last block of lines is a fall back to the global pypi index:

   Options +Indexes
   
   RewriteEngine On
   
   RewriteRule ^/robots.txt - [L]
   RewriteRule ^/icons/.* - [L]
   RewriteRule ^/index\..* - [L]
   
   RewriteCond /foo/bar/simple/ !-f
   RewriteCond /foo/bar/simple/ !-d
   RewriteRule ^/(.*)/?$ http://pypi.python.org/ [R,L]
   

3. Update your ~/.pip/pip.conf to point to the new repository:

   [global]
   index-url = http://localhost/url/to/my/pypi/
   

   Or use the -i http://localhost/url/to/my/pypi/ option at the command line.

Rader

You would only do the "Options +Indexes" part of step #2 for "local only" as the question requested. But transparently redirecting to pypi for things that aren't local is also useful. :) – Approximation

Answer 5

If you are talking about running simplepypi then you will have your server for adding packages and serve them out. To quote the documentation:

- Running this on the setup.py of your favorite package:

    python setup.py sdist upload -r local

If you were to use either os.walk or glob.glob on your local site-packages directory you could quickly filter for setup.py in each of the packages/directories and invoke the above on them.

If you just need to create a directory of tar.gz files complete with a .html list of them then you can use glob.glob on the top level of your site-packages directory - tar.gz each directory in turn and add the resulting filename to a list - you can then generate your index.html from that list.

You can use any of a large number of template engines for this or generate it yourself:

import glob
filelist = glob.glob("*.tar.gz")
tags = ['<A href="file:Where/%s">%s</A>' % (s,s) for s in tags]
head = """<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<HTML>
<HEAD>
<TITLE></TITLE>
<META NAME="Generator" CONTENT="Python Script">
<META NAME="Keywords" CONTENT="Cheeseshop">
<META NAME="Description" CONTENT="List of local python packages">
</HEAD>

<BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#FF0000" VLINK="#800000" ALINK="#FF00FF" BACKGROUND="?">
"""
tail = """</BODY></HTML>"""
tags.insert(0,head)
tags.append(tail)
page = "\n".join(tags)

Then save or serve you page.

Navigable