ci: every negative control runs on pull requests, enumerated from the Makefile (#563)

[rowan-claude]

Twice on 2026-09-05 main's certification went red on a merge whose pull-request run was 20 of 20 green, and one of the two was a stale sabotage pattern: #562 rewrote the line the maps fit control's sed aimed at, the sabotage patched nothing, and the control refused where only make test could see it. Every negative control in this tree already refuses that way. What was missing is running them where a pull request is judged.

The rule this leg is held to

Glenn, 2026-09-07, verbatim:

Remember that if we have any CI that runs per-commit it needs to finish 1 minute, 2 minutes max. Anything longer than that needs to be nightly or manually triggered as needed.

It is a rule about a job, and a matrix row is a job, so every row of this leg is cut to fit two minutes rather than the leg as a whole being cut to fit. A control that does not fit the rule on its own is not made to fit by grouping, so it runs nightly instead.

The leg

Two jobs in ci.yml. negative-controls-matrix runs the enumeration gate and prints the plan as a matrix. negative-controls fans out over it, builds bin/schema and the generated tree once, and runs its group's controls in one make -k invocation, so a control that refuses does not hide the ones behind it.

Two more in certify.yml, the same pair over the nightly tier, on the schedule that file already carries.

Red comes from two places: make's exit status, and a scan of the log for the controls' own refusal lines (NEGATIVE CONTROL FAILED: and NEGATIVE CONTROL:, which is where patched nothing and did not apply live). The scan puts the sentence a reader needs into the run's annotations rather than leaving it a thousand compiler lines deep.

The enumeration rule

tools/negativecontrols reads the Makefile and every file the Makefile includes, collects each explicit target whose name carries negative-control, and holds that set against make/negative-controls.json, which is the plan the matrix comes from. go test ./tools/negativecontrols/ fails on a difference in either direction, on an exclusion with an empty reason, and now on a group that names neither tier, so a control cannot leave the pull request without landing on the nightly. A separate test reads both ci.yml and certify.yml and requires each to reach its matrix and its targets through the tool, so a job that typed its own list would not pass.

The reader follows the tree's own include lines rather than a glob, and reads the rule heads a makefile actually writes: several targets on one head, a head continued over a backslash, .PHONY lines, and define blocks and := assignments that only look like rules. 196 controls, up from 159 when this branch opened, all of them enumerated rather than typed.

$ go run ./tools/negativecontrols check
every one of the 196 negative controls is in a group or in an explained exclusion, and every group runs on the pull request or nightly

Measured, per job, on this branch's own run

Run 34116264000, pull_request, head 037becee. Every one of the 38 negative-control jobs is under 120 s, wall time from startedAt to completedAt:

group	wall
wire-fuzz	112 s
wire-fuzz-wide-text	102 s
wire-fuzz-arm-width	99 s
wire-fuzz-message-oracle	95 s
wire-fuzz-index	94 s
base-lists	92 s
wire-fuzz-arm-terminator	91 s
base-cook	90 s
wire-fuzz-message-text	89 s
base-maps-order	88 s
wire-fuzz-length	87 s
cs-block-pitch	86 s
base-conformance	85 s
wire-fuzz-retain-class	84 s
block-fuzz-maximum	83 s
wire-fuzz-message-blob	82 s
block-fuzz	81 s
wire-fuzz-blob-node	80 s
base-maps	76 s
base-keyed	76 s
elixir	72 s
base-maps-keys	69 s
base-shared-node	68 s
cs-block-layout	66 s
base-conformance-absence	66 s
base-maps-depth	65 s
cs	64 s
base-arms	62 s
java	58 s
dart	57 s
cs-cook	56 s
base-packet	55 s
base-keyed-shift	55 s
base	53 s
js	52 s
rust	49 s
message-form	42 s
rust-alloc	39 s
negative-controls-matrix	15 s

Every other job in the file on the same run: msvc 87 s, windows 57 s, go-test 56 s, conformance 27 s to 52 s per leg, lint 44 s, vuln 37 s, generated 24 s, shape-gate 20 s, conformance-matrix 20 s, cpp-lock 11 s.

One job in the file is over the rule and it is not this leg's: big-endian at 263 s. It is 1:49 in ci.yml's own header, measured when #368 wrote it, and it is 4:15 to 4:26 on main today (runs 34111624059, 34111228369, 34110320928) with nobody having touched it. The header now says so. It wants the same treatment this leg got and has not had it, and that is a change to #368's gate rather than to this one.

How the groups were cut

Three passes, each one measured.

1. By toolchain, which is where this branch started: base, the six per-port groups, block-fuzz, conformance, and wire-fuzz and message-form split out of base on cost. On run 34102397965 that was base 8:57, wire-fuzz 7:13, cs 2:46, message-form 2:36, conformance 2:31, block-fuzz 1:59. Five of eleven rows over the rule.
2. By family and by cost inside a toolchain, to 34 groups. On run 34115675206 every group was inside the rule except wire-fuzz-retain, and eight sat between 100 s and 111 s.
3. Again, on the eight, to 38. big-endian going from 1:49 to 4:19 on a job nobody changed is the argument against leaving a job inside the rule by ten seconds.

What the measurement decided, rather than the family names:

• The map gate is 17 controls at about 4 s each on the author's machine and about 2x that on the runner, so it is four groups.
• The tolerant-wire family is one control per job: each rebuilds the compiler under a source overlay and then fuzzes the sabotaged wire, 55 s to 80 s each, so no two of them fit together. wire-fuzz is the tightest job in the leg at 112 s and its floor is the fuzz driver's own build, which is 64 s of it.
• The C# layout check is the instrument for three block controls at 39 s, 31 s and 29 s, so cs is four groups.
• conformance is gone as a group. Its one control had the bare conformance target as its prerequisite, which builds and runs all nine legs and needed all seven SDKs on one runner. The prerequisite is now conformance-rust, the rust leg and the derived manifest the control actually reads, and the control rides one SDK in rust-alloc at 39 s. The soak and the audit keep conformance deliberately: a number measured over a corpus whose matrix is red is a number about a defect, and this control is not a number.

The nightly tier

One control does not fit the rule at any group size, so it runs nightly rather than being grouped into fitting:

• tables-message-form-negative-control, 49 sabotage rows one submake each, 124 s. Its four blades stay on the pull request in message-form.

tables-wire-fuzz-retain-negative-control was here too until the cold read; it is an umbrella with no recipe, and it is in the exclusion list below. The group stays a group at one target: it is where the next control that outgrows the rule lands.

A group names its tier in a when field, matrix renders one tier, and the enumerator refuses a group that names neither. certify.yml runs the nightly tier on the schedule it already carries.

Exclusions

Six, each with a reason the test requires to be non-empty:

• packet-arm-defaults-negative-controls, an alias that re-invokes make once per language for nine targets this leg already runs in their own groups.
• tables-message-form-one-negative-control, a parameterized worker that reads SABOTAGE from the command line. Invoked bare it has nothing to break.
• tables-maps-negative-controls, an umbrella over the seventeen map-gate leaves, which four groups now run between them. An umbrella in any one of them would re-run the other three groups' controls.
• tables-wire-fuzz-negative-control, an umbrella over the twelve tolerant-wire leaves. In any one job it would re-run all twelve and take that job to six minutes.
• tables-wire-fuzz-retain-negative-control, an umbrella over the retain oracle and retain class leaves, which wire-fuzz-message-oracle and wire-fuzz-retain-class run between them.
• tables-big-endian-negative-control, the 263 s body of the big-endian job, which runs it on every pull request already. ci: the big-endian job runs 263 s on main, over the per-commit rule #684 owns moving that job; a second s390x cross-compile on this leg proves nothing the first does not.

Found on the way

• conformance-negative-control-absent did not name build/conformance-go as a prerequisite, though its second half execs it. Inside make test the Go leg is already built, which is why the omission stayed invisible. Fixed here.
• The first stale control this leg found, and the lint on the reader, are in the branch already.
• The 37 controls main gained while this branch was measuring are in the plan, toolchain-negative-control from make test refuses a missing pinned toolchain by name, and names a skip that is asked for (#599) #660 among them. Two of them landed mid-run and the leg went red naming both rather than running 194 of 196, which is the gate doing its job.

From the cold read

Eight findings, each acted on.

1. The nightly group's why described an umbrella as a pass. Makefile:4428 gives tables-wire-fuzz-retain-negative-control no recipe: it names tables-wire-fuzz-retain-oracle-negative-control and tables-wire-fuzz-retain-class-negative-control, both already in groups. The "140 s of fuzzing the whole corpus" was the two leaves' cost, counted a second time. It moves to excluded with the sentence the other umbrellas carry, and the claim is corrected where it is repeated, in ci.yml's leg comment and in docs/CONTRIBUTING.md. The nightly group holds one target and stays a group.

2. Three negative controls the naming convention hid. The marker tools/negativecontrols reads is negative-control, and three controls the Makefile itself introduces as "Its NEGATIVE CONTROL" spelled themselves -negative: tables-big-endian-negative, tables-hostile-negative, tables-pack-negative. The enumerator could not see them, so "every negative control runs on every pull request" passed over three, and two of them ran only inside make test. All three are renamed into the convention, with every reference moved (Makefile, docs/PORTING.md, ci.yml, compiler/porting_test.go), and placed:

target	where it went	why
tables-big-endian-negative-control	excluded	the body of the big-endian job, which runs it on every pull request today; #684 owns moving that job, and the enumerator now sees it without the gate cross-compiling s390x twice
tables-hostile-negative-control	new base-hostile group	14 s on the author's machine
tables-pack-negative-control	new base-pack group	7 s on the author's machine

They started as one base-pack group and are two rows now. On run 34120798793 the pair was 81 s of control time in a 113 s job, the tightest job in the leg, which is the band this branch already cut once. They share less than the name suggests: tables-pack's floor is a go build -overlay and a byte comparison, and tables-hostile's floor is the SANITIZED build of the hostile-value driver, which nothing else in the leg builds and which was most of the 81 s. So base-pack is the pack control at 7 s and base-hostile is the hostile-value control at 14 s, off one warm build. The count is 203 controls now, in 40 pull-request groups and one nightly group.

3. The leg-runs-the-manifest test greped three substrings. A leg that replaced its matrix expression with a hand-typed include list and left the old expression in a comment satisfied all three. gopkg.in/yaml.v3 is not a dependency and this module has none, so the package carries a reader for the block-YAML subset these workflows are written in. The new test asserts, for both tiers, that strategy.matrix IS ${{ fromJSON(needs.<matrix-job>.outputs.matrix) }}, that the leg needs: that job, and that the matrix job's output comes from the step running this tool.

Proven red first. With ci.yml's matrix expression replaced by a hand-typed three-row include list, TestTheLegRunsTheManifestAndNotATypedList still passed:

=== RUN   TestTheLegRunsTheManifestAndNotATypedList
--- PASS: TestTheLegRunsTheManifestAndNotATypedList (0.00s)

and the new test said:

=== RUN   TestEachLegExpandsTheToolsMatrix
    negativecontrols_test.go:291: .github/workflows/ci.yml: the negative-controls job expands
    map[string]interface {}{"include":[]interface {}{map[string]interface {}{"name":"base"},
    map[string]interface {}{"name":"base-packet"}, map[string]interface {}{"name":"base-conformance"}}}
    as its matrix, want "${{ fromJSON(needs.negative-controls-matrix.outputs.matrix) }}": a matrix
    written any other way is a target list this package did not enumerate
--- FAIL: TestEachLegExpandsTheToolsMatrix (0.00s)

The change was reverted; ci.yml is byte-identical to what it was before the experiment.

4. A marked head carrying $ or % was dropped in silence. makefile.go skipped it and the test pinned the skip. A dropped head is a control in no group, in no exclusion and in no job, with every test in the package green, which is the one outcome the package exists to prevent. It is refused by name now, and the test is flipped: four spellings (a variable head, a pattern rule, a .PHONY substitution reference, and a variable head reached over a backslash continuation) each have to refuse and name the head.

5. Five toolchain versions were written twice with nothing joining them. node 20, dart 3.13.2, java 21, otp 29.0.5 and elixir 1.20.4 sit in the plan and in test/conformance/<lang>/ci.json. A new test reads every ci.json and the plan and fails on a mismatch in either direction, and also refuses a registry that pins one field at two versions. dotnet is exempt by construction: the SDK version lives in .github/dotnet-version, both files carry a marker instead, and the test refuses a group that writes a version here.

6. timeout-minutes: 25 on a leg whose rows are held to 120 s. Set to 5, with the comment that says why: 120 s of control time plus about 25 s of checkout, clones and first build, so a row that reaches five minutes is a row that left the rule, and a ceiling wide enough to hide that is not a timeout.

7. ci.yml said certify.yml runs on every push to main. It has no push trigger (#669). The sentence says nightly and on demand, and the claim that followed it, that a regression fails on main minutes after the merge, is corrected to the next nightly or a dispatch.

8. Past-tense narration on this branch's own lines. "It used to sit in a group", "were 82 s", "which were 76 s" and "that job now measures" are present tense. docs/CONTRIBUTING.md no longer hardcodes a control count: it points at go run ./tools/negativecontrols check, which prints it.

And main again. The first run after the cold-read fixes went red on the enumeration gate, naming four controls main had landed while this branch was being read: tables-float-nan-negative-control, tables-maps-cook-check-negative-control, tables-lists-tool-cook-negative-control and tables-message-form-retain-negative-control. main is merged and each is measured and placed in the group whose corpus it already reads, at 1.6 s, 1.6 s, 0.3 s and 2.4 s off one warm build. No group changes tier and none goes near the rule.

Verified: go test ./tools/negativecontrols/... ./... green, make check green, and git diff origin/main...HEAD | grep '^+' | grep -c '—' is 0.

Closes #563

🤖 Generated with Claude Code