Repository navigation
CI Test job takes 4-6 hours and has hit the 6-hour limit #2099
Description
The Test job in pr_code_changes.yaml (macos-latest, make test) now runs close to GitHub's 6-hour job limit, and one PR's job has already hit it.
What the logs show
| Run | Branch | pytest policyengine_uk/tests/ |
Job |
|---|---|---|---|
| 36978117926 | remove-is-child-flags (#1896), started 07:21Z | 2,106 passed in 1:27:23 | 1h43m, passed |
| 37017928329 | hb-ctr-non-dep-exemptions | 2,276 passed in 4:11:45 | 4h55m, passed |
| 37020241005 | uc-capital-derived-income | 2,258 passed in 5:03:18 | 5h41m, passed |
| 37019477953 | uc-earned-income-consumers-1986 | — | cancelled: "The job has exceeded the maximum execution time of 6h0m0s" |
All dates are 2026-10-02. The annotations also say "Due to capacity constraints, jobs targeting macOS arm64 runners may experience longer queue times". About 15 Code changes runs were queued for the macOS runner at 19:45Z.
These are per-file pytest times, taken from the gaps between consecutive -v result lines:
| File | #1896 run | uc-capital-derived-income run |
|---|---|---|
test_uc_unearned_benefits_properties.py |
22.4 min | 95.0 min |
test_uc_state_pension_properties.py |
not on that branch | 60.1 min |
test_uc_income_from_capital_properties.py |
not on that branch | 32.2 min |
test_reform_representations.py |
not on that branch | 30.2 min |
test_pension_credit_reported_capital_properties.py |
not on that branch | 11.3 min |
Main has 12 *_properties.py files at 3c48247, up from 5 at 143cc3c (#1896's head). test_uc_unearned_benefits_properties.py grew by 342 lines on main over that range.
Why it matters
Every PR into main now waits several hours in the macOS queue and then about 4-5 hours in Test. The job spends about 24 minutes outside pytest (about 5 before, 19 after in run 37020241005), so a run whose pytest step goes past about 5h35m will be cancelled, whatever the code.
Possible fixes (not yet tried)
- A Hypothesis CI profile with fewer examples for the slow simulation-building properties, or a shared simulation per test module.
pytest-xdiston the runner.- Splitting the property suites into their own job, or moving
Testtoubuntu-latestif nothing needs macOS. - An explicit
timeout-minutesto make the limit visible.
Found while waiting on #2001's CI.
🤖 Generated with Claude Code
Activity
MaxGhenis commentedon Oct 3, 2026
Another instance: #2107's
Testjob (run 37091880974, attempt 1) ranRun tests with coveragefrom 05:01Z to 11:02Z on 2026-10-03 and was cancelled at the 6-hour limit. That PR only changes two property-test fixtures. Attempt 2 is running.🤖 Generated with Claude Code
Apply review r2: net trades in the cap earnings test; relevant UK ear…
MaxGhenis commentedon Oct 3, 2026
Another instance, with timing: #2051's
Testjob (run 37021760212) started at 22:26Z on 2026-10-02. It was cancelled at the 6-hour limit at 04:26Z on 2026-10-03, after pytest had finished.policyengine-core test policyengine_uk/tests/policy: 1,633 passed in 2:18.pytest policyengine_uk/tests/: every test passed or xfailed. The last result (test_uc_unearned_benefits_properties.py::test_carer_support_payment_above_the_cap_does_not_change_uc XFAIL [100%]) is at 04:08:13Z, and there are no FAILED or ERROR lines. The remaining ~18 minutes went on whatever runs after the last test (coverage XML and teardown) until the cancel.Per-file times from the gaps between
-vresult lines:max_examples=25)So a job whose tests all pass can still be cancelled. An explicit
timeout-minuteswould not help; the fixes listed above (a Hypothesis CI profile, xdist, a separate property job) would.🤖 Generated with Claude Code
Pin the key cases in every trading loss property test
MaxGhenis commentedon Oct 3, 2026
Another data point: run 37092342044 for #2089 (branch
uc-hb-split-shared-cap-lha-rules).pytest policyengine_uk/tests/collected 2,341 items. 2,307 had passed, with 0 failures, when the job was cancelled at the 6-hour cap (06:19Z → 12:07Z). It had reached 98%, intest_uc_rebalancing.py.The Hypothesis property files dominate. A
ciprofile with a lowermax_examples, or a separate property-test job, would bring the Test job back under the cap.MaxGhenis commentedon Oct 10, 2026
Status on 10 October 2026, from the UK hub:
Testlimit from 60 to 120 minutes as a stopgap (merged asd98af495). The job now runs onubuntu-24.04-arm.Testruns (Actions API, about 20:30Z): seven successes took 58.5 to 59.6 minutes (median 58.7), and 18 runs were cancelled at 60.1 to 60.3 minutes. So the suite still takes about an hour, three times the 18 minutes the old comment recorded.Two small corrections for whoever next edits the timeout comment or the changelog guidance:
-render as a doubled bullet under towncrier's default template.Split the Test job into a YAML job and three shards of the Python tests
MaxGhenis commentedon Oct 12, 2026
Measured result: #2269 (shards) landed in rollup #2364 (merge f061b21) on 12 October.
Testjob ran the whole Python suite. Run 38075384588 took 59.6 min, and other runs were cancelled at the 60-minute limit (above).YAML took 4.2 min separately.
Still open: #2306 moves to four shards and fails CI on any unexpected skip, xfail or XPASS. #2331 cuts each shard's peak memory.