if you ever need to customize python packages but your updates are not accepted by upstream developers, then you can host those packages in your own local repo;

host with web server

hosting your own pypi repo is actually very simple; all you need is to build a directory structure like this (check pep-503 for naming normalization):

.
├── bar
│   └── bar-0.1.tar.gz
└── foo
    ├── Foo-1.0.tar.gz
    └── Foo-2.0.tar.gz

then serve this directory using a web server with autoindex: apache, nginx, or even twisted; then set this url as pip index url;

this serves well if all you need is this sole local repo; in real life, you may want something more; for example, you may want to fallback to pypi.org if the requested package does not exist in your own local repo; for this you need more than a simple web server;

host with pypi server

you need a pypi server;

the one behind pypi.org is warehouse, which is powerful and specialized; for personal use, however, i do prefer something simpler; the one getting into my sight is pypiserver, which implements the same interfaces as pypi.org and works with tools like pip and twine; i dont want to talk too much about it because it has a readme; i just show how to use it:

pypi-server -i 127.0.0.1 -p 8080 --fallback-url "https://pypi.org/simple/" {path}

this command starts a pypi server at http://127.0.0.1:8080, hosting packages at local filesystem path {path}; these packages can be source tarballs (sdist), binary wheels (bdist_wheel), etc.; you can install these packages using pip with a custom index url:

  • using cli option --index-url:

    pip install --index-url "http://127.0.0.1:8080/simple/" ...

  • using envar PIP_INDEX_URL:

    PIP_INDEX_URL="http://127.0.0.1:8080/simple/" pip install ...

  • using config file ~/.config/pip/pip.conf:

    [global]
index-url = http://127.0.0.1:8080/simple/

if you request a package which does not exist in {path}, then pypiserver will fallback to https://pypi.org/simple/, as configured; in this way, your local packages have priority over pypi.org packages;

(do not) use --extra-index-url

if you run pip install --help, you may see it has a --extra-index-url option; please do not use it, unless you know what it really means; improper use of this option can lead to severe security issues (including arbitrary code execution);

simply speaking, the --extra-index-url defines a peer package index; there is no priority between the (main) index and these extra indexes; when pip installs a package, it looks for the “best” target it can find, and this often means a higher version number; a bad consequence that can result from this is:

  • you placed a custom package foo==1.0.0 in your own local pypi repo;

  • an evil placed a malicious package foo==9.9.9 on pypi.org;

  • you have pypi.org as the main index url and your own local pypi repo as an extra index url;

  • when you pip install foo, you get the malicious package, not your local package, because the former has a higher version number;

i guess now you see what i mean; it is a very bad consequence, unless you are a security expert who can get multi-$30k from this; alright, so forget about the money and stop using --extra-index-url; it is hardly useful except for mirroring and sharding;

wrap pip to use local pypi repo

you may feel tiresome typing that long index url every time you use pip; putting it in pip.conf is one solution, as shown above; there are multiple pip configuration files you can use, and this scheme indeed offers some flexibility; but what if you sometimes want the local repo, sometimes not? would you go edit that configuration file again and again?

consider using envars in this case; we can wrap the pip command inside an envar wrapper:

_wrap() {
    PIP_INDEX_URL="http://127.0.0.1:8080/simple/" "$@"
}

this allows us to use local repo with any pip:

_wrap pip install ...
_wrap pip3.9 install ...
_wrap pip3.10 install ...

this is better, right? what comes next is even better (note the trailing space):

alias wrap='_wrap '

this further allows us to use local repo with a pip alias:

alias _p='pip'
_wrap _p install ...    # fail
wrap _p install ...     # ok

this works too if the above line is still too long for you:

alias p='wrap _p'
p install ...

all of these together provide a lot of flexibility;

references