Will It Blend? Testing Across Environments with Tox

As library developers, we have a responsibility that application developers often don’t: our code needs to work reliably in environments we don’t control. Users might install our library on different operating systems, with various Python versions (3.8, 3.9, 3.10, 3.11, 3.12…), and alongside different versions of other libraries.

Just running pytest in your local development environment only verifies it works there. How can you be confident it won’t break for someone using an older Python version you promised to support? Or what if a user has an older version of a dependency installed? This is the problem that tox elegantly solves.

Why Multi-Environment Testing is Essential for Libraries

• Python Version Compatibility: Explicitly testing against all supported Python versions catches syntax errors, deprecated features, or standard library differences that only manifest in specific versions.
• Dependency Variations: Your library likely has dependencies (requests, numpy, etc.). Testing with different versions of these dependencies (e.g., the minimum required vs. the latest) can uncover integration issues.
• Ensuring Packagability: Tox often includes steps to build your library’s distribution package (sdist, wheel) and test the installed package, ensuring your pyproject.toml is correct.
• Automating Tedious Tasks: Manually creating virtual environments for each Python version, installing dependencies, and running tests is slow and error-prone. Tox automates this entirely.
• CI/CD Integration: Tox provides a single command (tox) to run all your environment checks, making it perfect for integration into GitHub Actions or other CI systems.

Introducing Tox: Your Environment Wrangler

Tox is a command-line tool that automates and standardizes testing in Python. Its core function is to create isolated virtual environments for different configurations (e.g., Python versions), install your project and its dependencies into each, and then run your configured test commands within each environment.

How it works:

1. Configuration: You define your test environments and commands in your pyproject.toml file.
2. Environment Creation: When you run tox, it reads the configuration and, for each defined environment, creates a fresh virtual environment.
3. Dependency Installation: It installs the necessary dependencies (including your library itself) into each virtual environment.
4. Command Execution: It runs the specified test commands (like pytest) within the context of each isolated environment.
5. Reporting: It reports the success or failure for each environment.

A Basic pyproject.toml Example

Let’s configure tox to test our library against Python 3.9, 3.10, and 3.11, with both minimum and latest dependency versions:

[project]
name = "my-awesome-library"
version = "0.1.0"
dependencies = [
    "requests>=2.28.0",
    "numpy>=1.21.0"
]

[tool.tox]
legacy_tox_ini = """
[tox]
envlist = py{39,310,311}-deps{min,latest}
isolated_build = True

[testenv]
deps =
    depsmin: requests==2.28.0
    depsmin: numpy==1.21.0
    depslatest: requests
    depslatest: numpy
    pytest
    pytest-cov
commands =
    pytest --cov={envsitepackagesdir}/my_awesome_library tests/
"""

Key configuration elements:

• envlist: Defines a matrix of environments to test. Here we’re testing three Python versions (3.9-3.11) with two dependency sets each (minimum and latest versions).
• isolated_build: Ensures a clean build environment for your package.
• deps: Dependencies are conditionally installed based on the environment. The depsmin factor installs specific versions, while depslatest gets the newest available.
• commands: The actual test commands to run in each environment.

Running Your Tests

First, install tox:

pip install tox

Then, from your project’s root directory, run:

tox

This will create six environments (3 Python versions × 2 dependency sets), install the appropriate dependencies in each, and run your tests. The output might look something like:

py39-depsmin: OK
py39-depslatest: OK
py310-depsmin: OK
py310-depslatest: OK
py311-depsmin: OK
py311-depslatest: OK
congratulations :) all tests passed

Want to run just one environment? No problem:

tox -e py311-depslatest

Testing Different Dependency Sets

One of the most powerful features of tox is its ability to test against different dependency versions. Here’s a more complex example that tests against specific dependency combinations:

[tool.tox]
legacy_tox_ini = """
[tox]
envlist = py311-{stable,edge}

[testenv]
deps =
    stable: requests==2.28.0
    stable: numpy==1.21.0
    edge: requests @ git+https://github.com/psf/requests.git
    edge: numpy @ git+https://github.com/numpy/numpy.git
    pytest
commands = 
    pytest tests/
"""

This configuration lets you test against both stable releases and bleeding-edge versions from Git repositories. Pretty neat, right?

Tox is an indispensable tool for Python library maintainers. It ensures your library works as expected across the diverse landscape of Python environments your users might have. By automating multi-environment testing, it saves you time, catches compatibility bugs early, and gives you (and your users) greater confidence in your code.

Next in our testing series, we’ll dive into mocking – how to test code that interacts with external systems or complex dependencies without actually needing those systems present.