Automatically Generating requirements.txt for Python Projects

Posted in Python

permalink

Summary

A quick post to demonstrate a way to automatically generate a requirements.txt file for your Python project.

By the end of this post, you'll be able to do this to refresh and update the versions of all the software your project depends on:

make requirements.txt

All of this code comes from the Human Cell Atlas data-store project!

Creating a requirements.txt.in file

Start by creating a requirements.txt.in, which should look like a normal requirements.txt file, listing software packages for pip to install (and optionally version information - but version information does not need to be specified).

This file is a looser set of specifications of software versions.

Example requirements.txt.in:

numpy
pandas > 0.22
sphinx

Converting requirements.txt.in to requirements.txt

Next, we use the requirements.txt.in file to install the latest versions of each software package (and all dependent software packages) into a virtual environment.

From that virtual environment, we can use pip freeze to output the names of each software package installed in the virtual environment, along with its exact version. This can be used to make a requirements.txt file.

The manual steps are

virtualenv -p $(which python3) venv
venv/bin/pip install -r requirements.txt
venv/bin/pip install -r requirements.txt.in
venv/bin/pip freeze >> requirements.txt
rm -fr venv

Using pip freeze means the resulting results.txt contains detailed version numbers:

alabaster==0.7.12
Babel==2.7.0
certifi==2019.11.28
chardet==3.0.4
docutils==0.15.2
idna==2.8
imagesize==1.1.0
Jinja2==2.10.3
MarkupSafe==1.1.1
numpy==1.17.4
packaging==19.2
pandas==0.25.3
Pygments==2.5.2
pyparsing==2.4.5
python-dateutil==2.8.1
pytz==2019.3
requests==2.22.0
six==1.13.0
snowballstemmer==2.0.0
Sphinx==2.2.2
sphinxcontrib-applehelp==1.0.1
sphinxcontrib-devhelp==1.0.1
sphinxcontrib-htmlhelp==1.0.2
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.2
sphinxcontrib-serializinghtml==1.1.3
urllib3==1.25.7

This is automated with a make rule next.

Automating the step with a make rule

We have a nice Makefile rule that can be dropped into any Makefile that allows users to run

make requirements.txt

and it will use requirements.txt.in, perform the above steps, and output an updated requirements.txt with the latest compatible versions of software.

Here is the Makefile rule:

requirements.txt: %.txt : %.txt.in
    [ ! -e .requirements-env ] || exit 1
    virtualenv -p $(shell which python3) .$<-env
    .$<-env/bin/pip install -r $@
    .$<-env/bin/pip install -r $<
    echo "# You should not edit this file directly.  Instead, you should edit $<." >| $@
    .$<-env/bin/pip freeze >> $@
    rm -rf .$<-env

Summary of the make rule:

  • The first two lines create a virtual environment at .requirements-env/

  • The next two lines run pip install, first on requirements.txt (the existing version), then requirements.txt.in (which installs/updates any software packages in requirements.txt.in)

  • A comment is added to the top of the requirements.txt file to help give users a hint about where to update software requirements.

  • The pip freeze command is used to create a requirements.txt file from the current virtual environment

Refreshing requirements

To update the requirements, update the requirements.txt with these manual steps:

refresh_all_requirements:
    @cat /dev/null > requirements.txt
    @if [ $$(uname -s) == "Darwin" ]; then sleep 1; fi  # this is require because Darwin HFS+ only has second-resolution for timestamps.
    @touch requirements.txt.in
    @$(MAKE) requirements.txt

Now requirements.txt can be updated with

make refresh_all_requirements

This can be done periodically, and the new requirements.txt updated in the version control system.

Tags:    python    pip    version control    make    makefile