Repository navigation
Release test fails: test_award_never_rises_with_a_claimants_statutory_pay passes from source, fails from the installed wheel #2397
Description
@vahid-ahmadi flagging this for you: it blocks every release.
What happens
Every versioning.yaml release run on main since 2026-10-12 20:46 UTC fails Release test on one test, so Publish is skipped and nothing has been released since:
FAILED policyengine_uk/tests/test_uc_statutory_pay_properties.py::test_award_never_rises_with_a_claimants_statutory_pay
AssertionError: {'adults': [{'age': 22, 'wages': 12570.0, 'profit': 0.0, 'startup': False, ...}], 'child_ages': [1, 1], ...
Runs: 2e5cf2b (first failure, the #2119 merge), 807e307, and the 07:35, 07:59 and 10:38 runs on 2026-10-13. The pending run for #2031 is expected to fail the same way.
Why PR checks didn't catch it
- The test arrived in rollup Land six reviewed pull requests together (rollup B) #2380 (with Count statutory sick, maternity and paternity pay as Universal Credit earned income #2365), whose PR checks passed.
- Move families onto Universal Credit when a legacy benefit they receive closes #2119 went through the merge queue; the queue's Test / Python 3/4 ran this test on the exact commit
2e5cf2band it passed (run). - The release run on the same
2e5cf2bfailed it. PR and merge-queue checks run tests from the source checkout; Release test runs them from the built wheel installed in a fresh environment.
Reproduction (local, commit 2e5cf2b, Python 3.13)
- From source (
uv sync --frozen --extra dev, thenpytest policyengine_uk/tests/test_uc_statutory_pay_properties.py::test_award_never_rises_with_a_claimants_statutory_pay): passes in 7.6 s. - From the wheel (build the wheel, install it with the locked dependencies into a new venv, run
pytest --pyargs policyengine_uk.tests.test_uc_statutory_pay_properties -k test_award_never_rises_with_a_claimants_statutory_payfrom outside the repo): fails (about 6 minutes, mostly Hypothesis shrinking).
Ruled out
- Missing files: all 2,631 tracked files under
policyengine_uk/are in the wheel. - Dependencies and Python: the passing queue run and the failing release run installed the same 152 packages at the same versions, on Python 3.13.16. The only difference was the package's own version label (2.127.0 vs 2.128.0), and the local wheel was 2.127.0.
- Random inputs: the test uses
derandomize=True, and Hypothesis derives that seed from the test's source and signature, not its path, so both runs try the same households.
So for the same inputs the model gives a different result depending on whether policyengine_uk is imported from a source checkout or an installed wheel. For the failing household (age 22, wages £12,570, two children aged 1), the UC award rises as statutory pay rises in the installed package. Since the installed package is what PyPI users get, this may be a real bug rather than only a test problem. Investigation continues below.
🤖 Generated with Claude Code
Activity
anth-volk commentedon Oct 13, 2026
Root cause
Two separate things.
1. A real model bug (in source and in the wheel)
The failing assertion is
exempt[HIGH] >= exempt[LOW](assert 0.0 >= 1.0). The failing family: one adult, 22, wages £12,570, private pension £6,000, two children aged 1, North East, council tenant; statutory sick pay rises from £0 to £6,000.is_benefit_cap_exempt_earningstestsemployment_income + self_employment_income - income_tax - national_insurance >= 10,152. It subtracts all of the person's income tax and NI, including the tax and NI caused by statutory pay and pension income, while counting only wages and profit. So a rise in statutory pay can end the earnings exemption. Computed directly at 2e5cf2b (identical from source and from the wheel):2. Why PR and merge-queue checks don't see it
Hypothesis 6.169 biases generation with literal constants harvested from local modules, and treats any path containing
/site-packages/as non-local (hypothesis/internal/constants_ast.py: is_local_module_file, used byconjecture/providers.py: _get_local_constants).policyengine_ukis local: Hypothesis collects 37 integers, 51 floats and 1,301 strings from it.The
derandomize=Trueseed is identical in both (same function digest), and the first two examples match, but from the third example on the two environments generate different populations. Each environment is stable across runs and independent ofPYTHONHASHSEEDand the working directory. The release environment's examples include the failing family; the source environment's don't. Soderandomize=Truemakes runs repeatable within one environment, but PR checks and Release test can still explore different cases.Possible fixes (for the owners to decide)
@example(...)for this family so every environment checks it, and/or make PR checks and Release test explore the same cases (for example by running both from the installed wheel).🤖 Generated with Claude Code