Skip to content

Automate releases to PyPI #105

Closed
Closed

Description

@rok

Set up automated publishing of benchx to PyPI, using a tag-triggered workflow on this repository. The workflow is in #107, and #104 has already fixed the packaging it depends on. Conda packages are tracked separately in #108. Related: #76, on organization accounts for the distribution sites.

Updated after the discussion below: the first version of this plan also uploaded to anaconda.org. That's dropped, because it would need a long-lived token and PyPI's trusted publishing doesn't. Conda packages will come from a conda-forge feedstock instead, which follows the releases on PyPI. The original text is in this issue's edit history.

What #107 does

Pushing a v* tag runs these jobs, and only a tag publishes. PRs that touch the workflow, pyproject.toml or src/benchx/__init__.py run only the first one.

Job What it does Access
build Builds the sdist and wheel, checks the version matches the tag, runs twine check, and runs the tests from the installed wheel Read-only
publish-pypi Uploads to PyPI with pypa/gh-action-pypi-publish pypi environment, using PyPI's trusted publishing
github-release Creates the GitHub release and attaches the sdist and wheel contents: write

It also adds a release checklist in docs/releasing.md, reads the version from src/benchx/__init__.py so it's kept in one place, and renames the runner from benchx-prototype to benchx.

Setup before the first release

  • PyPI: add a pending publisher on the benchx org: project benchx, owner benchx-dev, repository benchx, workflow release.yml, environment pypi. This doesn't reserve the name.
  • GitHub: create the pypi environment with required reviewers, administrator bypass off and self-review prevention on.
  • GitHub: change the environment's v* rule to a tag rule. It's currently a branch rule, which should make GitHub reject the publish job on a tag. Do this before pushing any tag.
  • GitHub: settle the environment's required reviewers (see the open questions below).

Release sequence

  • Merge MAINT: switch to the src/ layout #104.
  • Merge CI: publish releases to PyPI #107.
  • Set the version to 0.1.0rc1, push tag v0.1.0rc1, and have a second reviewer approve the publish job.
  • Test the install on a clean machine with pip install --pre benchx.
  • Set the version to 0.1.0, push tag v0.1.0, and have a second reviewer approve the publish job.
  • Set the version to 0.2.0.dev0, so that results from main don't claim to come from the release.

Once 0.1.0 is on PyPI, #108 can start. docs/releasing.md in #107 has the full checklist for later releases.

A real release candidate is proposed instead of TestPyPI because it tests the actual publishing setup and also claims the name. It stays on PyPI permanently, but pip skips pre-releases by default.

The approval, PyPI trusted publishing and creating the GitHub release can't run from a pull request, so they first run with the release candidate. A problem there would need a follow-up fix and a new release candidate.

Open questions from the first version

  • Approvals: one approval from a listed reviewer is enough, once per release. Self-review prevention is on, so that reviewer can't be whoever pushed the tag, and a release takes two people.
  • Who can publish: still to settle. The pypi environment lists six required reviewers, while the discussion below suggested the three people in the PyPI org. Restricting who can create v* tags is optional on top of that, because GitHub rulesets can't exempt individual users, so it would need a team.
  • conda: decided: conda-forge, not the BenchX channel on anaconda.org. See Distribute benchx through conda-forge #108.
  • Separate release repository: not planned for now. NumPy and SciPy use one to limit who has write access, which seems like more than this project needs today.

Activity

  1. ev-br commented on Nov 6, 2026

    @ev-br
    Collaborator

    LGTM overall!

    A couple of small comments:

    • I don't know about anaconda channels, my impression was that conda-forge is the go-to solution, but I'm not going to press it either way;
    • Might want to start with PyPI first? Having a long-lived token is...uhm well.
    • Having a release checklist would be great. As an example, here's one: https://data-apis.org/array-api-compat/dev/releasing.html --- am not advocating for a specific workflow! What is helpful is the sequence of steps which are otherwise easy to mix up of forget (in that workflow: edit the version, push the tag, check pypi, then merge the PR)

    For open questions:

    • Approvals: I don't know if we need to require multiple approvals. In array-api-compat (which I'm using as a prototypical small pure python project with automated publishing), it's basically whoever has the chops to push a tag, pushes it and that triggers the release.
    • Who can publish: any owner? OTOH, were we to do it manually, it'd be who is in the pypi org. Thus maybe limit it to this collection of people (and if other owners/admins want the chops, then why not).
    • NumPy and SciPy do it differently, with a separate numpy-release repository: the main repo does not have anything release related, and numpy-release has way less people with write access. Don't know if it's an overkill here.
    • Thinking about it for a minute, maybe indeed we want to start it right from the get-go: we have what, three people in the pypi org---limit the release chops to these people.
  2. changed the title [-]Automate releases to PyPI and anaconda.org[/-] [+]Automate releases to PyPI[/+] on Nov 6, 2026
  3. changed the title [-]Automate releases to PyPI[/-] [+]Automate releases to PyPI and conda-forge[/+] on Nov 6, 2026
  4. changed the title [-]Automate releases to PyPI and conda-forge[/-] [+]Automate releases to PyPI[/+] on Nov 6, 2026
  5. rok commented on Nov 6, 2026

    @rok
    MemberAuthor

    I think this is done, conda-forge is a separate issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions