In this post, we cover a pattern for automatically generating a
requirements.txt file that has the latest
compatible versions of required software, and that specifies the full and exact version of each package to
make the Python environment reproducible.
This will turn a requirements input file (called
requirements.txt.in for example) that looks like
into a requirements file that specifies the exact version of
numpy and all dependencies, like
By the end of this post, you'll be able to do this to refresh and update the versions of all the software your project depends on:
All of this code comes from the Human Cell Atlas data-store project!
What is requirements.txt?
When developing a Python project, the
requirements.txt file is a plain text file that contains a list of
Python software packages that need to be installed for the current Python software package to work. The software
can be installed using the command
pip install -r requirements.txt
For example, if a package
import numpy at the top of a Python file in the project, the
must be installed before importing
foobar. In this case, the
requirements.txt could just contain
or it could specify a particular version of numpy, or a minimum version of numpy:
numpy >= 1.10
Start by creating a
requirements.txt.in, which should look like a normal
listing software packages for pip to install (and optionally version information - but version information
does not need to be specified).
This file is a looser set of specifications of software versions.
numpy pandas > 0.22 sphinx
Converting requirements.txt.in to requirements.txt
Next, we use the
requirements.txt.in file to install the latest versions of each software package (and all
dependent software packages) into a virtual environment.
From that virtual environment, we can use
pip freeze to output the names of each software package installed in
the virtual environment, along with its exact version. This can be used to make a
The manual steps are
virtualenv -p $(which python3) venv venv/bin/pip install -r requirements.txt venv/bin/pip install -r requirements.txt.in venv/bin/pip freeze >> requirements.txt rm -fr venv
Using pip freeze means the resulting
results.txt contains detailed version numbers:
alabaster==0.7.12 Babel==2.7.0 certifi==2019.11.28 chardet==3.0.4 docutils==0.15.2 idna==2.8 imagesize==1.1.0 Jinja2==2.10.3 MarkupSafe==1.1.1 numpy==1.17.4 packaging==19.2 pandas==0.25.3 Pygments==2.5.2 pyparsing==2.4.5 python-dateutil==2.8.1 pytz==2019.3 requests==2.22.0 six==1.13.0 snowballstemmer==2.0.0 Sphinx==2.2.2 sphinxcontrib-applehelp==1.0.1 sphinxcontrib-devhelp==1.0.1 sphinxcontrib-htmlhelp==1.0.2 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.2 sphinxcontrib-serializinghtml==1.1.3 urllib3==1.25.7
This is automated with a make rule next.
Automating the step with a make rule
We have a nice Makefile rule that can be dropped into any Makefile that allows users to run
and it will use
requirements.txt.in, perform the above steps, and output an updated
requirements.txt with the
latest compatible versions of software.
Here is the Makefile rule:
requirements.txt: %.txt : %.txt.in [ ! -e .requirements-env ] || exit 1 virtualenv -p $(shell which python3) .$<-env .$<-env/bin/pip install -r $@ .$<-env/bin/pip install -r $< echo "# You should not edit this file directly. Instead, you should edit $<." >| $@ .$<-env/bin/pip freeze >> $@ rm -rf .$<-env
Summary of the make rule:
The first two lines create a virtual environment at
The next two lines run
pip install, first on
requirements.txt(the existing version), then
requirements.txt.in(which installs/updates any software packages in
A comment is added to the top of the
requirements.txtfile to help give users a hint about where to update software requirements.
pip freezecommand is used to create a
requirements.txtfile from the current virtual environment
To update the requirements, update the
requirements.txt with these manual steps:
refresh_all_requirements: @cat /dev/null > requirements.txt @if [ $$(uname -s) == "Darwin" ]; then sleep 1; fi # this is require because Darwin HFS+ only has second-resolution for timestamps. @touch requirements.txt.in @$(MAKE) requirements.txt
requirements.txt can be updated with
This can be done periodically, and the new
requirements.txt updated in the version control system.