Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/pr_code_changes.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,7 @@ jobs:
uses: ./.github/actions/test-environment
- name: Test documentation builds
run: make documentation
# The YAML tests, the Python tests on six runners, and the check that
# The YAML tests, the Python tests on seven runners, and the check that
# every test ran once and passed, in the locked environment. As one job
# they took 59 minutes on 10 October 2026 (run 38075384588), and runs were
# cancelled at a 60-minute limit (issues #2099 and #2257). The release
Expand Down
12 changes: 9 additions & 3 deletions .github/workflows/test-suite.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@
# publish. All three run the same jobs in the same locked environment.
#
# Jobs: the YAML policy tests (which also list every Python test), the Python
# tests split over six runners, and Result, which fails unless every listed
# tests split over seven runners, and Result, which fails unless every listed
# Python test ran in exactly one shard and passed, apart from the outcomes
# accepted in .github/test_outcome_allowlist.toml. A caller that names its job
# Test gets the checks "Test / YAML", "Test / Python K/6" and "Test / Result";
# Test gets the checks "Test / YAML", "Test / Python K/7" and "Test / Result";
# "Test / Result" is the one to require.
#
# Runners. These jobs, and the check jobs that pull requests, merge queue
Expand Down Expand Up @@ -174,6 +174,12 @@ jobs:
# 15.5 GB (run 40395541241). With six, the listed slow tests come to
# about 134 minutes of worker time per shard instead of 161, and each
# worker runs about a sixth fewer tests.
#
# Seven, not six, for time. Nine successful test runs started on 31
# October (EDT) took 57 to 73 minutes a shard, and the 75-minute limit
# cancelled shard 1/6 with its tests at 99% (run 41191216150, first
# attempt). With seven, the listed slow tests come to about 115 minutes
# of worker time per shard instead of 134.
name: Python ${{ matrix.shard }}/${{ strategy.job-total }}
if: ${{ !inputs.skip }}
runs-on: ${{ vars.CI_RUNNER_ARM != '' && (vars.CI_RUNNER_EVENTS == '' || contains(format(',{0},', vars.CI_RUNNER_EVENTS), format(',{0},', github.event_name))) && vars.CI_RUNNER_ARM || 'ubuntu-24.04-arm' }}
Expand All @@ -189,7 +195,7 @@ jobs:
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4, 5, 6]
shard: [1, 2, 3, 4, 5, 6, 7]
steps:
- name: Checkout repo
uses: actions/checkout@v6
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ test-yaml:

# Every Python test, serially. To spread them over four cores add
# PYTEST_ARGS="-n 4 --dist worksteal" (pytest-xdist); to run what one CI
# shard runs, add "--shard K/6" (policyengine_uk/tests/conftest.py). CI runs
# shard runs, add "--shard K/7" (policyengine_uk/tests/conftest.py). CI runs
# these tests through .github/workflows/test-suite.yaml, in the locked
# environment (uv sync --frozen --extra dev), and also checks each test's
# outcome against .github/test_outcome_allowlist.toml.
Expand Down
1 change: 1 addition & 0 deletions changelog.d/2770.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Spread the Python tests across seven shards to reduce the work per job.
2 changes: 1 addition & 1 deletion docs/book/validation/ubuntu-vs-macos.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ That change verified, on main at `3c48247e`, that:
## Checking for a regression

CI now runs the test suite (`.github/workflows/test-suite.yaml`: the
YAML tests in one job and the Python tests split over six shards) on
YAML tests in one job and the Python tests split over seven shards) on
Linux arm64 (`ubuntu-24.04-arm`), and the install and smoke-import checks
on x86 `ubuntu-latest`. There is no macOS job. If an OS difference is
suspected again:
Expand Down
4 changes: 2 additions & 2 deletions docs/engineering/skills/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,10 +86,10 @@ the same suite,
tree; a release tests the wheel it is about to publish, installed on its own.

The suite runs the YAML policy tests in one job and splits the Python tests
over six runners. Each runner runs one shard:
over seven runners. Each runner runs one shard:

```bash
pytest policyengine_uk/tests -n 4 --dist worksteal --shard K/6 -o xfail_strict=true
pytest policyengine_uk/tests -n 4 --dist worksteal --shard K/7 -o xfail_strict=true
```

`--shard K/N` is defined in `policyengine_uk/tests/conftest.py`. It keeps the
Expand Down
Loading