Modern Python Package Publishing: PyGeoHash's New CI/CD Pipeline

Gone are the days of manually building wheels and uploading them to PyPI – we now have a fully automated pipeline using GitHub Actions and PyPI’s Trusted Publisher system. Let me walk you through how we set this up, because it’s pretty cool.

The Old Way vs The New Way

Back in the day (like, last month), releasing PyGeoHash went something like this:

1. Build the source distribution locally
2. Build wheels for each platform (hopefully)
3. Find that PyPI token I stored somewhere “safe”
4. Cross fingers and upload
5. Realize I forgot to build wheels for ARM
6. Do it all again

Now? I just tag a release, and GitHub Actions does everything else. Let’s dive into how this works.

Our Publishing Pipeline

The new workflow kicks in automatically when I push a tag starting with ‘v’ (like v3.0.0). Here’s what happens:

1. Building Wheels

First, we build wheels for every platform we support:

jobs:
  build_wheels:
    name: Build wheels on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
    steps:
      - uses: actions/checkout@v3
      - name: Build wheels
        run: python -m cibuildwheel --output-dir wheelhouse
        env:
          CIBW_SKIP: "pp* *-musllinux*"  # Skip PyPy and musllinux
          CIBW_ARCHS_MACOS: "x86_64 arm64"  # Build for Intel and Apple Silicon

This is where the magic happens. We’re using cibuildwheel, which is like a Swiss Army knife for wheel building. It:

• Builds wheels for Python 3.8 through 3.12
• Handles both Intel and ARM architectures on macOS
• Takes care of all the platform-specific quirks
• Makes sure our C extension compiles everywhere

2. Building the Source Distribution

In parallel, we build the source distribution (sdist):

  build_sdist:
    name: Build source distribution
    runs-on: ubuntu-latest
    steps:
      - name: Build source distribution
        run: |
          # Clean build directories
          rm -fr build/ dist/ .eggs/
          find . -name '*.egg-info' -exec rm -fr {} +
          
          # Build the sdist
          uv run python -m build --sdist

We’re using uv here instead of pip because it’s blazing fast and has excellent dependency resolution. The cleanup step is probably overkill, but hey, better safe than sorry.

3. Publishing to PyPI

Here’s where PyPI’s Trusted Publisher system comes in:

  publish:
    needs: [build_wheels, build_sdist]
    runs-on: ubuntu-latest
    permissions:
      id-token: write  # Required for trusted publishing
    steps:
      - name: Publish to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          packages-dir: dist_flat/

No more API tokens to manage! The Trusted Publisher system uses OpenID Connect to verify that the upload is coming from our specific GitHub Actions workflow. It’s more secure and way less hassle.

You set this up on the PyPI side. I’ve had to do a first publishing manually or with a token to get it started, but then remove the token and do this forward.

Why This Matters

This new setup gives us several benefits:

1. Consistency: Every release is built the same way, every time
2. Coverage: We automatically build wheels for all supported platforms and Python versions
3. Security: No more API tokens floating around
4. Speed: Parallel builds mean faster releases
5. Reliability: If something goes wrong, we can see exactly what happened in the GitHub Actions logs

Setting It Up Yourself

If you maintain a Python package, especially one with C extensions like PyGeoHash, here’s what you need:

1. Set up trusted publishing in your PyPI account settings
2. Create a .github/workflows/publish-pypi.yml file with your build configuration
3. Configure cibuildwheel for your specific needs
4. Start tagging releases!

The hardest part was probably figuring out the right cibuildwheel configuration for our C extension. We had to make sure we were building for both Intel and ARM on macOS.

A Note on PyPy and musllinux

You might have noticed this line in our configuration:

CIBW_SKIP: "pp* *-musllinux*"  # Skip PyPy and musllinux

Let’s talk about why we skip these builds and what you might want to consider for your own package.

PyPy Support

We currently skip PyPy builds because our C extension isn’t tested against PyPy’s JIT compiler. If you want to support PyPy, you’ll need to:

1. Test C Extension Compatibility: PyPy’s C API implementation differs from CPython
2. Provide Pure Python Fallbacks: Consider having a pure Python implementation that PyPy can use
3. Use CFFI Instead: If you need native code, CFFI works better with PyPy than the C API

Here’s what the config might look like if you want to support PyPy:

env:
  CIBW_BUILD: "cp* pp*"  # Build for both CPython and PyPy
  CIBW_BEFORE_BUILD_LINUX: |
    if [[ $PYPY == 1 ]]; then
      pip install cffi  # For PyPy builds
    fi

musllinux Support

musllinux wheels are built against musl libc instead of glibc. We skip these for now for simplicity, but may add them later to better support alpine linux.

But if you want to support musllinux (which can be great for Alpine Linux users), here’s what you need:

env:
  CIBW_BUILD: "*"  # Build all platforms
  CIBW_BEFORE_BUILD_LINUX: |
    # Install musl-dev for musllinux builds
    if [[ "$CIBW_BUILD" == *"musllinux"* ]]; then
      apk add musl-dev
    fi

You’ll also need to ensure your C code is compatible with musl libc, which means:

1. Avoiding glibc-specific features
2. Testing thread-local storage carefully
3. Being cautious with errno handling
4. Checking all your dependencies work with musl

For most packages, I’d recommend starting with standard Linux wheels and only adding musllinux support if users specifically request it. The maintenance overhead can be significant.

Looking Forward

This automation is part of our ongoing efforts to modernize PyGeoHash’s infrastructure. Combined with our recent type hints update, we’re making the library more maintainable and reliable.

Next up? We’re looking at adding some automated performance benchmarking to make sure our C extension stays fast across all platforms. But that’s a story for another post!

Want to see the full workflow? Check out our publish-pypi.yml on GitHub. And as always, if you have suggestions for improvements, our issue tracker is open!