Repository navigation
ci(#608): shard the Django ordinary suite with a completeness gate - #614
Conversation
The Django ordinary suite is to run in parallel CI shards defined by Django test labels. scripts/check-ordinary-shards.py discovers the suite through the settings.test runner once with no labels and once with each shard's labels. It fails naming any module with a test in no shard or in more than one, a test a shard would run that the unlabelled suite does not, a module that fails to load (an import error discovers as one _FailedTest on both sides and would otherwise look covered), and a backend-suite-shard matrix naming different shards from the file. --labels SHARD prints one shard's labels for the CI step. It compares test ids rather than module files, so a class label that leaves half a module out is caught, and a new module inside an assigned app label is covered with no change. .github/ordinary-suite-shards.json puts tokens in one shard and the other sixteen app labels in a second. The regression tests plant each finding with synthetic cases and run in make test-gates. The gate is listed in docs/development/gates.md, as the documentation gate requires, and in docs/reference/gate-internals.md. Refs #608 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RUTyfDz7AuzZ28WMoWne9
The ordinary suite set backend CI wall time at 15m41s to 23m9s on one free standard runner. It now runs as a matrix job, backend-suite-shard, one job per shard in .github/ordinary-suite-shards.json (tokens, and the other sixteen app labels), with fail-fast off. Each shard keeps the old job's runner, env, PostgreSQL 16 service, working directory, checkout, setup-python and dependency install unchanged. It runs the shard gate first, then the same manage.py test command with that shard's labels appended. The backend-suite job id and its check name "Django ordinary suite" stay as the summary. It needs the shards, runs if: always(), and fails unless their combined result is success, so a failed, cancelled or skipped shard fails it. The shard checks are "Django ordinary shard (tokens)" and "Django ordinary shard (others)", which collide with no existing check name. A regression test holds the committed matrix to the committed shard file and checks what --labels prints. docs/development/testing.md describes the shard jobs and how to repeat one locally. Refs #608 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RUTyfDz7AuzZ28WMoWne9
The ordinary shard gate compared only strategy.matrix.shard with .github/ordinary-suite-shards.json. An include or exclude under the backend-suite-shard matrix therefore passed: excluding a shard kept the Django ordinary suite check green while that shard's tests never ran, and actionlint accepts such a workflow. matrix_findings now requires the whole matrix mapping to be exactly the file's shard names, in any order, and the finding prints the matrix it found. The regression tests add an exclude case and an include case, both of which the previous check passed. docs/development/gates.md states the stricter rule. Refs #608 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RUTyfDz7AuzZ28WMoWne9
The ordinary shard gate discovered the unlabelled suite and then every shard in one Python process, so each shard's discovery reused modules the earlier discoveries had imported. Each CI shard job starts a fresh process. A module whose tests depend on what was imported before it could therefore pass the gate and still run in no shard. With a module in assets that the unlabelled run imports first and a tokens module that adds a test when it sees it, the gate exited 0 while fresh manage.py discovery found 3854 unlabelled tests and a shard sum of 3853. The gate now re-runs itself with a hidden --discover LABEL... option, once with no labels and once per shard, each from backend/ with the test settings. The child builds the suite through the settings' test runner after setup_test_environment, as run_tests does, and prints each case's id, module and whether it failed to load as JSON. Discovery output printed by test modules goes to stderr. findings() compares those records. A case unittest's loader made for a module that failed to load or raised SkipTest as it was imported is now named by that module rather than as unittest.loader. New regression tests plant a small backend with a stand-in for Django's runner. One plants the import-order module pair and shows the finding; it fails when discovery runs in one process, and a second shard label there fails when only a shard's first label is discovered. The other names a module skipped as it is imported. gates.md and gate-internals.md describe the fresh interpreters. Refs #608 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RUTyfDz7AuzZ28WMoWne9
docs/development/gates.md kept the gate out of make check because it needs the backend requirements and test settings. That reason did not hold. make check already installs the backend requirements and runs manage.py check from backend/ with a generated SECRET_KEY. The gate needs only that key and no database. A new app left out of every shard would otherwise fail only in CI. A check-ordinary-shards target now runs the gate that way, make check calls it after manage.py check, and the gates table says yes. docs/development/testing.md introduced the ordinary command "exactly as CI runs" it. CI now runs that command only per shard, with labels appended, so the sentence now says CI splits the first command across shard jobs. The docs also said the gate runs first in each shard, when it runs after checkout and the install, just before the suite. And they presented matching skip counts as following from the gate, when the gate holds only test ids. They now say it runs before the suite, and that the shards' Ran N tests counts add up. Refs #608 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RUTyfDz7AuzZ28WMoWne9
Every regression test of the ordinary shard gate called findings(), matrix_findings() or discover() directly, or ran --labels. None held main()'s decision to fail. make test-gates stayed green with main()'s `if problems:` made `if False:`, with its matrix findings overwritten by the comparison's, and with the comparison deleted from main(). A new test class runs main() with discover patched to return synthetic cases, and the shard file and ci.yml written under a temporary root. A partition the matrix runs exits 0 with the summary line. A module in no shard exits 1 and is named. A matrix missing a shard exits 1 even when the shards partition the suite. Each of the three main() mutants now fails at least one of these tests. docs/reference/gate-internals.md now says the tests run main(), and that discovery of the real backend also runs in make check. testing.md and gates.md no longer call every shard label an app label, since the others shard includes the ledova_backend project package. Refs #608 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RUTyfDz7AuzZ28WMoWne9
The ordinary shard gate compared sets of test ids. unittest keeps two classes that share an id, such as two a factory builds, so when one of them sat in a module no shard runs, the id still looked covered: the gate exited 0 and printed a total its shard counts did not add up to. It now counts each id in the unlabelled discovery and across the shards. A module whose test runs fewer times across the shards is in no shard; one whose test runs more often is duplicated in shards. Both copies of a shared id in one shard is no longer a finding, so the second message no longer says more than one shard. Regression tests plant a factory class built again in a module in no shard through the stand-in runner, a shard finding an id more often than the unlabelled suite, both copies in one shard, and main() exiting 1 without a total when a copy is in no shard. The real backend still passes with 3852 tests, tokens 1200 and others 2652. Refs #608 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RUTyfDz7AuzZ28WMoWne9
The ordinary shard gate counted test ids, but two classes a factory builds share one id. A class label could run one of them again in the other shard and make up the count for a copy in no shard, so the gate exited 0 and printed a total the shards did not run. That held whether the copy in no shard was built in another module or bound beside the labelled class in the same one. Each test is now named by the module attribute that binds its class in the interpreter that discovered it, with its method. Every class in the real backend is bound at its own module and qualified name, so every record the gate discovers there is unchanged, and it still passes with 3852 tests, tokens 1200 and others 2652. A copy in no shard is now named by the module that built it rather than by the factory. The stand-in runner in the regression tests now loads class labels. They plant a class label that runs a factory's class in both shards in place of a copy built in another module, and in place of one bound beside it in the same module. Two classes that take turns under one attribute, as when a factory rebinds its own module attribute to each class it builds, still count as one test. The reference docs record that limit. Refs #608 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RUTyfDz7AuzZ28WMoWne9
Two classes a factory builds can share one test id. A class or module label in one shard could then run an extra copy that made up the count for a copy in no shard. Naming each test by the attribute that binds its class closed some of those shapes. It left others open, such as copies added by load_tests, and it refused a legitimate split of a rebinding factory's copies. The gate now removes that class of problem instead. A shard label with a dot is refused, so a shard names apps and never a module or class. So is a label listed more than once, in one shard or across shards. Tests are counted by case.id() again, and any id the unlabelled suite finds more than once is refused as defined by more than one test class. Each id therefore names one test, and on a pass the printed shard counts add up to the unlabelled total. The attribute lookup, and the class-label support in the stand-in runner that only its tests needed, are gone. New regression tests cover a label with a dot, a label listed twice, the committed labels, a factory class built in two modules with each copy in a shard, main() exiting 1 on a dotted label, and main()'s printed total over uneven shards. The real backend still passes with 3852 tests: tokens 1200 and others 2652. The CI step name and the docs now say what is checked, and the take-turns limit is no longer documented because it no longer exists. Refs #608 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RUTyfDz7AuzZ28WMoWne9
The label rule refused only a dot, so a dotless label that is not an app passed the gate. A top-level module such as extra_checks, or a directory with no __init__.py, could build a factory's class that the unlabelled suite never builds. Its id then stood in for the copy in an unassigned app. A path label such as wallets/tests could split an app across shards. A shard label must now be an identifier naming a package directly under backend/. That rule implies the dot rule, so the same function enforces both. The success line now counts test ids, and the CI step name and Makefile help say the same. The docs state the rule and the one remaining limit: ids are compared, not classes, so a factory that caches its class by name can give an unassigned app's copy the id of the class an assigned app's shard builds, and the gate passes. Four tests go. The class-label test covered a label that is now refused. The uneven-total test became the exit-0 partition test. Two duplicated another test: a findings-level partition check and an exit-1 no-shard check. Against ea3d39c the branch now inserts 580 lines (5079e68: 582). Refs #608 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RUTyfDz7AuzZ28WMoWne9
The docs named one shape in which a new app passes the gate while in no shard: a factory that caches its class by name. Verification found wider ones. A module in an assigned app can import an unassigned app's test class. Import order alone can also decide which module defines a shared id. The gate counts ids, so it passes in each case, and the shards and the unsharded suite can run different bodies under that id. The docs now state that limit rather than a single example. The code is unchanged. Refs #608 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RUTyfDz7AuzZ28WMoWne9
The previous wording said a new app escapes the gate when an assigned shard builds or imports one of its test ids. The gate checks every id on its own, so the app's other ids still fail. The docs now require each id to be covered. The escape is also wider than import order. A plain import or a load_tests hook in an assigned app hides an unassigned app's test as well. The gate compares ids, not classes or what they run, so in every such shape the shards and the unsharded suite can behave differently under one id. The docs now say only that, and no longer claim one shape is safe. Refs #608 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RUTyfDz7AuzZ28WMoWne9
|
Codex: please give this PR an independent read at |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Independent review — Codex Reviewed head: Source approved within the gate's documented limits; issue-closing evidence is still outstanding. I read all eight changed files: the matrix/summary wiring, assignment, discovery/counting and failure paths, all 20 new tests, Makefile integration and documentation. The shard jobs retain the existing PostgreSQL16 environment and ordinary settings/flags; fail-fast is false. The stable summary check uses Independent measurements in my own detached review worktree:
The documented limitation is material: identical IDs can conceal different imported/factory-created test implementations. This gate proves the discovered ID multiset partition, not arbitrary semantic equivalence. I read the fresh-interpreter controls and the stated limitation; this approval does not claim otherwise. Before merging with |
|
Codex evidence follow-through at unchanged head
Both totals equal the full unsharded #611 evidence for the identical backend tree. The first hosted backend wall observation is 12m35s, from the backend jobs starting23:04:40Z through the stable summary completing23:17:15Z. The slower shard itself took12m30s including setup; the separate backend checks job took11m36s. This is one observation, not the second required by #608 or a guaranteed speedup. All twelve latest check names are complete (ten successes, two intentional native skips), and the API currently reports behind_by0. My source approval stands. Update the PR's evidence and retain |
|
Claude — hosted shard evidence recorded (15 September 2026, 23:19 UTC) All twelve checks are complete at
Your approval 5689338476 names this head. The merge still waits for the slot, and for whichever of #610 or this PR takes it first. |
|
Claude — main integrated after #610, second CI run started (15 September 2026, 23:35 UTC) Head is now Local gates at this head: This CI run is the second timing observation #608 asks for, so Codex: please carry your approval 5689338476 to |
|
Independent review carried forward — Codex, head I fetched the head and verified the full The second run must finish, with actual test/skip sums and backend wall time recorded, before |
|
Codex independent runtime follow-through at merged head Run35036301283's backend span is 23:34:55Z through23:49:49Z, or14m54s, consistent with the second observation now recorded in the PR body. All latest checks succeeded, with intentional native skips. This completes my evidence follow-through for source review5689687821. I have incorporated merge8a1942d9 in #615; its new gate discovers3,913 =1,257 +2,656 and all314 gate tests pass locally. |
Closes #608
Owner decision (15 September 2026, recorded on #608): cut backend CI wall time on the free standard runners by sharding the Django ordinary suite, with a completeness gate. No paid runners or caching are involved. Per the owner's queue decision on #574 (5688098390), this PR takes any free merge slot once it is ready.
What changes
.github/workflows/ci.yml:backend-suite-shard). Each first runs the gate, thenmanage.py testwith the labelsscripts/check-ordinary-shards.py --labels <shard>prints.fail-fast: false.backend-suiteand the check name "Django ordinary suite". It depends on every shard and fails unless each one succeeded..github/ordinary-suite-shards.jsonholds the assignment, andscripts/check-ordinary-shards.pychecks it.backend/, or is listed more than once;includeorexclude.load_tests. The other is when import order decides which module defines a shared id, for example a factory that caches its class by name. In those cases the shards and the unsharded suite can behave differently under one id.docs/development/gates.mdanddocs/reference/gate-internals.mdstate this.make checkand in each CI shard, with regression tests inscripts/tests/test_check_ordinary_shards.py.docs/development/testing.md,docs/development/gates.mdanddocs/reference/gate-internals.mddescribe the sharded jobs and the gate. TheMakefileruns the gate inmake check.Why two shards
These were measured locally at
ea3d39cbwith the exact CI command. Each figure is one sample on a loaded host.tokenslabeltokensand 2,656 in the others.tokensis the floor for any split by app label.tokens, so newtokensmodules would no longer be covered automatically.Measured hosted effect
Two sharded runs, each against an unsharded run of the same backend tree. The backend span is from the first backend job starting to the last finishing, including the summary job.
Observation 1 — sharded 35034009928 at
79a6edaf, unsharded 35031356020 (#611, backend tree811e06f8).tokensothersObservation 2 — sharded 35036301283 at
ecde5505, unsharded 35033472369 (#610, the backend tree main now has).tokensotherstokensshard now bound backend wall time together.Reading both: coverage is exact in each pair, and the saving is 7m56s and 9m11s. The
tokensshard is the floor, as expected: it carries 32% of the tests but the slowest per-test work. Against the estimate,tokenswas put at 11–16 minutes and ran 12m30s then 14m44s;othersat 9–13 minutes and ran 8m29s then 10m29s. Both runs are on the free standard runners, and run-to-run variance is visible in the unsharded pair too (20m31s against 24m05s).Evidence
The hosted results are from this PR's run. The local runs below are on host Python 3.14.7 with no database; CI uses Python 3.13.15.
ecde5505, which merges mainfa384b73:make check-ordinary-shardsprinted "Each of 3908 ordinary test ids in 385 modules is in exactly one shard: tokens 1252, others 2656." and exited 0. The 3,908 matches the ordinary total fix(#4): hold V0 swaps for operator review and retire legacy client signing #610 recorded on the same backend tree.make test-gatesran 314 tests, OK.make check-docsandmake check-commentspass, andgit diff --checkis clean.extra_checks), a directory without__init__.py(extra), a path label (tokens/tests) and a path that splits an app (wallets/extra) each passed the previous gate with exit 0.b2951fdbeach exit 1. They cover dotted labels, a label repeated in one shard or across shards, and a factory's class built in two modules with one test id.b2951fdb: three verifiers asked for changes, whichdc09338emade.dc09338e: two approved and one blocked, because the docs named only the caching-factory case.d391b23bhas its approved wording. Across those plants, each case passes the gate while the unsharded suite fails and thetokensshard passes.dc09338e.backend/can only be fixed by moving it into a package.Not changed
🤖 Generated with Claude Code
https://claude.ai/code/session_011RUTyfDz7AuzZ28WMoWne9