Many developers use tox as a solution to standardize and automate testing in Python. However, using the tool only for test automation severely limits its power and the full scope of what you could achieve. For example, tox is also a great solution for the “it works on my machine” problem. There are several reasons for this, such as:
- tests can be run against different Python dependency versions
- environment variables can be isolated
- setup commands can be captured and run
In addition, and most importantly, the actions listed above can be performed across Windows, macOS, and Linux OS. In this tutorial, I will dive into how tox works and how it can be used to save precious resources. I will also provide concrete code examples to demonstrate how you can leverage tox.
What is tox?
If you read the tox documentation and take what it says at face value, you will likely think tox is merely a tool used to create virtual environments to install the necessary dependencies required to test a Python package.
The documentation states that “tox aims to automate and standardize testing in Python. It is part of a larger vision of easing the packaging, testing, and release process of Python software.” It later states, “tox is a generic virtualenv management and test command-line tool.” A few examples in the same documentation demonstrate tox being used to build documentation and run development environments.
However, it is better to think of tox as a tool for both automating certain workflows and managing virtual environments. The example configuration file provided in the tox documentation is provided below.
# content of: tox.ini , put in same dir as setup.py
[tox]
envlist = py27,py36
[testenv]
# install pytest in the virtualenv where commands will be executed
deps = pytest
commands =
# NOTE: you can run any command line tool here - not just tests
pytestNotice that the file only installs and runs Pytest; however, the documentation states, “you can run any command-line tool here, not just tests.”
How does tox work?
The System Overview section of the tox documentation presents a workflow diagram (Figure 1). This diagram shows how tox works by breaking down the workflow into a series of steps, outlined below.
- Generate a virtual environment using the version of Python defined in the
tox.inifile. - Install the dependencies listed under the
depssetting of thetox.inifile into the virtual environment. It will also install thesdistof the project if one is created (optional). - Run the commands in the isolated virtual environment; the commands are listed under the commands setting.
- Return the results from each environment to the user.
These steps further show that tox is an effective developer tool that can be used to automate workflows by creating virtual environments and running commands in them. Since the commands you can pass to tox are not limited to those that execute tests, tox is more than just a tool for standardizing and automating tests.
Benefits of tox
This section highlights some of the benefits of using tox to manage virtual environments and automate workflows, including testing.
Ease collaboration
Every time I am given a take-home assignment for a job interview, I use tox. To access the assignment, developers from the hiring company just need to install and run tox. The same is true when working with a team. Team members do not need to reproduce environments or install dependencies. All of these tasks are taken care of in the tox.ini file.
Facilitate continuous integration
Without tox, your continuous integration (CI) script must handle both the creation of virtual environments and the installation of the package dependencies. This means CI scripts built without tox are more complex. This will be explored in more detail below.
Reduce risk of dependency conflicts
For each task, tox creates a new virtual environment. This reduces the chance of dependency conflicts. For example, the dependencies required to run the application and execute it can each be installed into two separate virtual environments.
One major weakness of tox arises during local development; that is, it fails to track changes in the dependencies. Consequently, each time a change is made, it is important to recreate the tox environments. This is done by passing the -r flag when tox is executed (py -m tox -r).
Simple tox use case example
To get to grips with tox, I have created a simple example you can clone from my kurtispykes/tox_example GitHub repository. Follow along to learn how to use tox.
├── __init__.py
├── .gitignore
├── LICENSE
├── README.md
├── string_reversal.py
├── test_string_reversal.py
├── tox.iniAt the center of the tox ecosystem is the configuration file, which may come in one of the following three flavors:
tox.ini
setup.cfg
pyproject.tomlThis example uses tox.ini to configure tox. The contents are provided below:
[tox]
envlist = my_env
skipsdist = true
[testenv]
deps = pytest
commands = pytestThe INI File Structure states that a configuration file using the .ini extension “consists of sections, each led by a [section] header, followed by key/value entries separated by a specific string (= or : by default).”
In tox, a section header translates to a new tox environment. However, notice the [tox] header in this example. The header configures the global settings for tox runs. You can tell tox to use various versions of Python to execute tests in this header.
The [tox] header contains two items:
envlist– Informs tox what environment to execute whenpy -m toxis run from the command line. In this example, theenvlistis namedmy_env. After the initial tox run, other runs will execute much faster because tox tracks the details of the virtual environments and will not recreate or reinstall dependencies.skipsdist– When there is nosetup.pyorpyproject.toml, set theskipsdistflag to true. An error will result if it is not set.
Note that you could also test your package against different versions of Python by adding the version you would like to test against (py27, py37). You must have the versions of Python you would like to test your package against installed in your environment, or an error will be raised.
The [testenv] and [testenv:NAME] headers are used to define the test environments for tox where NAME is the name of a specific environment. The settings defined in the [testenv] (known as the top level) are automatically inherited by the individual environments unless you override them.
This example does not define individual environments, but sets the following two items:
deps– The dependencies required to execute the code.commands– The commands to be triggered as part of the current test environment.
Now with the configuration defined, you can create and test a module to demonstrate tox in action. The module created for this example is string_reversal.py, which contains a function used to reverse a string.
# The contents of string_reversal.py
def reverse_string(text):
reverse_text = text[::-1]
return reverse_textTo test that the module works correctly, use the following test_string_reversal.py script:
# The contents of test_string_reversal.py
from string_reversal import reverse_string
def test_calculate_age():
# Given
text = "Hello World!"
# When
reversed = reverse_string(text)
# Then
assert reversed == "!dlroW olleH"The next step is running tox from the same directory where the tox.ini file is stored. From the command line, use the following command to run tox:
py -m toThe output should look similar to what is displayed in Figure 2.
While this example uses Pytest, you can use any other library to test your module. In fact, you can execute any arbitrary command.
Creating a machine learning package
You can also extend tox to various scenarios. For example, a machine learning (ML) model trained on transaction data to predict when a fraudulent transaction occurs. See the full code in the kurtispykes/fraud-detection-project GitHub repo. The top-level structure of the ML model package is provided below:
├── fraud_detection_model # Contains the code required to build the model
├── requirements
│ ├── requirements.txt
│ ├── test_requirements.txt
├── tests # Contains the unit tests for the model
├── LICENSE
├── MANIFEST.in
├── mypy.ini
├── publish_model.sh
├── pyproject.toml
├── setup.py
├── tox.iniSeveral dependencies were required to build this ML model package. To better manage the package dependencies, create a requirements.txt file.
The tox configuration file for this project includes several environments that tox can run. These individual environments range from fetching the data required for model training to publishing the trained model to a Gemfury repository. This further shows that tox can be a useful tool for managing virtual environments. Some of the configurations in the tox.ini file are shown below.
# Part of the tox.ini file; click on the GitHub link to view the entire file
[tox]
envlist = test_package, typechecks, stylechecks, lint
skipsdist = True
[testenv]
install_command = pip install {opts} {packages}
passenv =
KAGGLE_USERNAME
KAGGLE_KEY
GEMFURY_PUSH_URL
[testenv:test_package]
deps =
-rrequirements/test_requirements.txt
setenv =
PYTHONPATH=.
PYTHONHASHSEED=0
commands=
python fraud_detection_model/train_pipeline.py
pytest \
-s \
-vv \
{posargs:tests/}
[testenv:train]
envdir = {toxworkdir}/test_package
deps =
{[testenv:test_package]deps}
setenv =
{[testenv:test_package]setenv}
commands=
python fraud_detection_model/train_pipeline.py
[testenv:fetch_data]
envdir = {toxworkdir}/test_package
setenv = {[testenv:test_package]setenv}
commands =
# fetch
kaggle competitions download -c ieee-fraud-detection -p ./fraud_detection_model/data/interim
# unzip
unzip ./fraud_detection_model/data/interim/ieee-fraud-detection.zip -d ./fraud_detection_model/data/interim
[testenv:publish_model]
envdir = {toxworkdir}/test_package
deps =
{[testenv:test_package]deps}
setenv =
{[testenv:test_package]setenv}
commands=
python fraud_detection_model/train_pipeline.py
./publish_model.sh .Note that because a setup.py file has been added, the skipsdist argument can be removed from the global tox header. It was not removed in this case because, prior to packaging the model, tox was still being used to manage virtual environments.
You may notice that the global envlist (in the [tox] header) makes a call to test_package, typechecks, stylechecks, lint. When py -m tox is called from your command line, each of these environments will be created and the commands from each environment will be executed.
To select a specific environment to run independently, use the following command: full capabilities of the framework. Instead, it is better to think of tox as a tool for automating certain workflows and managing virtual environments.
py -m tox -e NAMENAME is the name of the testenv you would like tox to create.
A note on using tox in production
Various CI platforms integrate extremely well with tox. The ML model example uses CircleCI for continuous integration. The Circle CI configuration file calls to tox instead of creating several virtual environments directly. For more details, see the CircleCI configuration file on GitHub.
Summary
This post has explained how to use tox for more than standardizing and automating tests in Python. Employing tox only to automate tests is severely underutilizing the full capabilities of the framework. Instead, it is better to think of tox as a tool for automating certain workflows and managing virtual environments.
Register for NVIDIA GTC 2023 for free and join us March 20–23 to learn more about data science and how accelerated computing can transform your work.
