Repository navigation
test: 87 arms have survived every mutation aimed at their neighbours: vacuous, or named for something they do not measure #1236
Description
The technique, which is the point; the number is only what it costs
test/check_ledger.tsv records, per check, when it was last observed red and under which mutation. We built that column to keep history. It is also a vacuity detector, and neither session was reading it that way until @OffgridwithJD said so while fixing #1235:
a
neverthat no mutation can move is a question about the arm
The inference has three parts, and the third is the one that does the work:
- A check that has never been observed red may simply never have been attacked. Says nothing.
- A premise legitimately stays
neverforever. Says nothing. - A non-premise check that is
neverin a part where some sibling IS dated is an arm that survived an attack aimed at its neighbours. That is a question.
Measured on 61de3b96:
total ledger rows 1589
never 1527
never AND non-premise 1139
never, non-premise, in a part with a DATED sibling 87
awk -F'\t' '$5!="never"{d[$1"\t"$2]=1} {r[NR]=$0}
END{for(i=1;i<=NR;i++){split(r[i],f,"\t");
if(f[5]=="never" && f[3] !~ /^premise:/ && ((f[1]"\t"f[2]) in d)) print r[i]}}' \
test/check_ledger.tsv1139 is a haystack. 87 is a reading list. The narrowing is what the inference costs, not what it is.
Where it came from: an arm that could not fail, and the ledger knew
#1235's first version excluded the sweep from its own population by basename, and checked it like this:
producer: [ "$b" = "$PGC_CAP_SELF" ] && continue # skips that string
the arm: grep -c "^$PGC_CAP_SELF$" want 0 # greps the SAME variable0 by construction, and blind to the case it existed for: reached under a symlink or a renamed copy the skip stops matching the real file, the sweep re-enters its own population, the recursion returns, and the arm still passes. Driven, same function, only the variable changed:
PGC_CAP_SELF=capability_sweep.sh population 2 arm 0 PASS literal arm 0 PASS
PGC_CAP_SELF=cap.sh population 3 arm 0 PASS literal arm 1 FAIL
(self is back in)
Its ledger row read never, and none of four mutations could move it. That signal was available for free, before anyone read the function.
The second finding: arms named for something they do not measure
Three of the 87 are in harness_selftest/570, written today by @jdatcmd:
a gitfile is a regular file, which a directory test cannot accept
and it is still a .git that a reader must not skip on
These assert stat -c %F on a file the part just created. No change to the product can ever redden them, because they do not measure the product. They are premises wearing non-premise names.
That is milder than a vacuous arm and it is still a defect, for a specific reason: a reader counting what a part proves counts them, and is wrong. The repair is cheap — rename to premise: — but it is a repair, not a relabelling, because the name is the claim.
So this list is not 87 vacuous arms. It is 87 places where either the arm is vacuous, or its name is lying about what it measures. The disjunction is the finding. Anyone working it has to decide which, per arm, by attacking it.
What must NOT happen next
No fixes in the change that files this. The 87 are not a list of defects; they are a list of arms nobody has understood yet. Fixing an item before understanding it converts an open question into a closed answer without passing through the measurement, and what the ledger is worth is precisely that it records what was measured.
Three of them being @jdatcmd's own makes that harder to hold to, not different.
Suggested shape for working it
- Take one PART at a time, not one arm.
- For each candidate, name the change that should redden it and make it. If no such change exists, the arm does not measure the product: rename it
premise:or replace it. - Merge the resulting last-red dates from the RUN, never by hand.
- A part whose candidates all turn out to be fixture facts is a finding about the part, worth saying in its comment.
The largest blocks today are native_join_runtime_filter, index_am_support (the four build-and-answer pairs) and harness_selftest/190. None has been looked at; the list is where to look.
Filed from the #1235 review. The technique is @OffgridwithJD's, the tree-wide measurement and the fixture-facts caveat are @jdatcmd's.
🤖 Generated with Claude Code
Activity
test: attack fifteen unreddened runtime-filter arms (#1236)
Merge pull request #1238 from OffgridwithJD/test/1236-attack-the-runt…
test: produce the runtime-filter rows with the tool, not a script (#1236
Merge pull request #1239 from OffgridwithJD/test/1236-full-attribution
OffgridwithJD commentedon Sep 23, 2026
Second batch: nine
index_am_supportarms, and one of them is your second category#1240 is up. Nine arms attacked, reading list 72 to 64, and the shape repeats exactly:
index_am_support's only dated rows were two premises aboutdocs/features.md, reddened by a docs mutation, which cannot reach one arm about what an index does. Eight behaviour arms satneverbeside them. Unattacked, not vacuous -- the category this list keeps finding and its title does not offer.And M-2 turned up an arm named for something it does not measure
With no index in existence at all, the four answers its operator arms stayed green.
The arm compares a count taken with
enable_seqscan = offagainst one taken withpgcolumnar.enable_custom_scan = off.enable_seqscanis a preference rather than a prohibition: with no index, both sides fall back to the same scan and agree.So an arm named "btree answers its operator with the same rows the scan returns" can pass without an index scan ever happening. It is not vacuous -- M-1 shows it reddens on a wrong answer -- but its name claims the index path was exercised and it does not check that.
That is this issue's second category, found by a mutation rather than by reading, and it is a milder defect than a vacuous arm in the same way the
stat -c %Ffixture facts are: the arm measures something real, just not the thing its name promises.The repair is one premise -- assert the index path's plan actually contains an index scan -- and I have deliberately not made it in #1240. That change records evidence and alters no test; a repair should be reviewed as a repair.
The two batches so far
Both had the same cause: a part whose only mutation was aimed somewhere its behaviour arms could not feel -- a GUC's boot value in one, a documentation file in the other.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP
OffgridwithJD commentedon Sep 23, 2026
Asked of all 64 at once, as @jdatcmd suggested: the hypothesis does not generalise, and the list has structure
The two-for-two pattern — the only mutation ever aimed at this part came from outside its subject — was worth testing before a third batch rather than after. Asked of every part still on the list, by reading what each part's dated rows were reddened by:
It does not hold.
validity_elisionandnative_join_vector_aggcarry genuine behaviour mutations — a reader forced to expect a bitmap that was not written, a writer forced to write one that should have been elided, a fold dropped — and they still have 8 and 7 non-premise arms sittingneverbeside them.So the list is not one population. It splits:
190's twelve, beside a single text mutation on a static guard;validity_elision's eight,native_join_vector_agg's seven, and nownative_join_runtime_filter's remaining twenty-nine, which survived three behaviour mutations in test: attack fifteen unreddened runtime-filter arms (#1236) #1238.The second group is the sharper sublist, and my two batches have moved arms into it rather than only off the list. That is what the reading list is for, but it means the count going down is not the only thing happening to it.
A third category: nine dated rows carry no cause
Each says this was observed red on this date and records nothing that would reproduce it. They are not on the reading list, because they are dated — and they are the same defect from the other side: the column exists to be the catalogue of causes, and for these nine there is a date and no cause.
I am not proposing to fix them. Reconstructing a mutation for a row dated months ago is guesswork, and guesswork is what this file is supposed to exclude. But "dated with no cause" is worth counting alongside "never dated", and it is nine.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP
OffgridwithJD commentedon Sep 23, 2026
Third batch,
validity_elision: one arm reddened, and two guards have no arm that can feel themThis is the batch that did not go as predicted, which makes it the useful one.
validity_elisionis in the sharper sublist: it carries genuine behaviour mutations already (a reader forced to expect a bitmap that was not written, a writer forced to write one that should have been elided) and still had eight non-premise armsnever. Three mutations, aimed at the three kinds of arm sitting there.Both guards removed together: still 25/25
Each guard raises the same SQLSTATE and the arms assert
XX001, so the obvious reading is that each backstops the other. Tested:So the two guards in
columnar_reader.cthat those arms are named for have no arm that fails when they are deleted. AnXX001is still raised under M-D -- the arms pass, and passing requires that code -- so a third refusal covers the same corruption. Which one, I have not identified.What I could not establish, and the mistake that stopped me
I tried to name the third refusal by reproducing the corruption outside the suite: same
UPDATE ... set_byte(encoding_descriptor, 1, 1), same column. With both guards disabled the read succeeded and returned the correct sum, raising nothing at all.That is not evidence about the suite. My fixture was one I invented rather than the suite's own -- different row count, different null pattern -- and the whole finding here is that the suite's fixture provokes a refusal mine does not. The same error I have made twice today in other forms: measuring a question next to the one being asked.
So the claim I am making is the narrow one, measured three ways: removing either guard, or both, changes nothing the suite can see. The claim I am not making is which code refuses instead.
What that means for this issue
These two arms are your second category in its sharpest form so far. They are not vacuous -- they demand
XX001and would fail if nothing refused -- but they are named for specific guards whose removal they cannot detect, and the comments above them describe exactly the mechanism they do not test.It also means the two guards are uncovered: a change that removed either, or both, would pass this suite.
I have not recorded ledger rows for M-1, M-2 or M-D, because nothing reddened and there is nothing to record. M-3's five rows are evidence and will go up with the next batch.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP
OffgridwithJD commentedon Sep 23, 2026
Batch 5,
native_join_vector_agg: three mutations, three different outcomesSeven never arms, from the sharper sublist. The outcomes did not divide the way the arms do, which is the point of running them.
M-2 is ordinary evidence
Two arms covered, exactly the predicted pair: remove the uniqueness gate and a non-unique dim side folds, which both changes the answer and defeats the refusal the arm is named for.
M-3 is redundant code, not an uncovered guard, and the distinction matters
columnar_vector.c:1415returns early whenpgcolumnar_enable_ungrouped_vector_aggis off. Disabling it changes nothing because the same GUC is checked again at:1550on the same path:So "unique join uses core Agg when GUC off" is not an uncovered arm. The behaviour it asserts is enforced twice and the arm cannot distinguish which check enforced it. The first is an early exit rather than a guarantee, and a reader deleting it would be right.
M-1 is the uncovered case, and I cannot name what refuses instead
The explicit outer-join refusal -- the loop over
root->join_info_listthat returns false when a non-INNERSpecialJoinInfocovers the joinrel -- can be deleted and no arm notices, including "LEFT join refuses the join fold". The arm still passes, so the fold still refuses LEFT joins; something downstream does it, structurally rather than by intent.That is the same shape as
validity_elision's two guards, and I am reporting it with the same limit: measured that removing it changes nothing this suite can see; not established which code refuses in its place.The comment above that loop states the intent -- "A join that would multiply fact rows is refused and the ordinary Agg runs" -- and the arm is named for it. Whatever refuses instead is not that loop, and if the downstream refusal ever moved, nothing here would say so.
Running totals
The list is no longer mostly a list of suspicious arms. Five batches in, it has produced three uncovered guards, one redundant check, two fixture facts and 38 arms with evidence -- and the guards are the part that is about the product rather than the tests.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP
test: attack nine index_am_support arms (commandprompt#1236)
OffgridwithJD commentedon Sep 24, 2026
The reading list is not a burn-down chart, and #1244 just took it from 72 to
154. Worth recording before anyone reads the number as a regression.
The list is defined here as:
never, non-premise, in a part where a sibling ISdated. That last clause is what makes it useful -- a dated sibling proves the
part is attackable, so its unattacked neighbours are suspects rather than
unknowns. It is also what makes the count non-monotonic:
#1244 attacked
400-a-check-result-must-be-machinefor the first time. It added7 arms and dated 8 rows -- and in doing so moved the part from "never attacked"
to "attacked", which promoted 82 of its existing arms from invisible to
suspect. None of those 82 changed. The tree did not get worse. The instrument
started pointing at a part it had no opinion about an hour ago.
So the first attack on a large part always makes the number jump, and the
size of the jump is a property of the part, not of the work. The three parts
still contributing most are the shape to expect it from again:
Ten parts in total.
The consequence for this issue is that "reading list size" is the wrong
progress metric, and I have been quoting it as one (87 -> 72 -> 64 in earlier
comments). Two numbers behave better together: parts with a dated arm, which
only ever rises and is 14 -> 15 here, and arms dated per part attacked, which
says whether an attack was thorough. The list size is a work queue, not a score.
Nothing here needs fixing. Recording it so the next person to see 154 does not
go looking for what broke.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP
Merge pull request #1240 from OffgridwithJD/test/1236-index-am-arms
test: attack eighteen arms across three parts (commandprompt#1236)
Merge pull request #1246 from OffgridwithJD/test/1236-batches-345
jdatcmd commentedon Sep 26, 2026
Re-derived on
be31874: 87 has become 393, and the number is not a regressionMy awk reproduces 87 exactly on
61de3b96, so the two numbers are the samemeasurement on two trees and the difference is real.
What moved it, measured rather than guessed
I had two hypotheses and both were wrong. It is not today's dating: only 2 of
the 393 sit in a part dated 2026-09-25 or later. It is not seeding either: the two
parts that dominate were already in the ledger at
61de3b96, at 226 and 96 rows.It is this:
Dating one check in a part promotes every undated non-premise sibling in that
part into the population. Part 410's own never-count did not move at all; one
dating event took it from contributing nothing to contributing 190.
Which means the count cannot be read as progress, and the issue should say so
The inference is still right — a non-premise check that is
neverin a part wherea sibling IS dated survived an attack aimed at its neighbours. But the metric
grows as the work is done, because the third condition is about the part
rather than the check. Attacking one arm in a 228-check part is the single most
expensive move available by this number, and it is also exactly the right thing to
have done.
So
87 -> 393says ten more parts now have someone's attention, not 306 armsgot worse. Anyone reading the count as debt will conclude the opposite of the
truth.
Two things follow for how to work it:
are 272 of the 393 — 69% in two files. A part is a reading list someone can
finish; the total is not.
and moving on is what produced this reading. Date one, then attack its siblings
until the part holds no undated non-premise checks, and the part leaves the
population permanently.
What I have not done
No arms attacked yet. This comment is the population and the correction to how it
reads; the 393 is a reading list, not a verdict on any arm in it. And
471premises are excluded by the rule rather than by inspection, so if any of those is
a non-premise check misnamed
premise:it is invisible to this measuremententirely.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP
jdatcmd commentedon Sep 27, 2026
First part worked end to end, and the population turns out to hold three kinds, not one
I took
harness_selftest/570-a-checkout-test-must-ask-gitbecause it is small: 3non-premise survivors against 9 dated neighbours. Reading all three, they are not
three instances of one thing.
One is a real arm
It asserts that without
_ck_foldjoining line continuations, a test split acrosstwo lines is not matched. Widen
_ck_patto match the split form and thisreddens while its dated sibling stays green. That is an arm and it is attackable.
Two cannot fail for any change to this repository
Two lines above, the fixture is
printf 'gitdir: %s\n' ... > "$_ck_dir/.git", andtwo lines below it is
rm -rf "$_ck_dir". So they assert that a file just createdby
printf >is a regular file and exists. That is POSIX, not pgcolumnar. Theygate nothing — the fixture is destroyed before anything else uses it — and no
change to this tree can move either.
#1236's inference found them correctly. They are exactly what it is for.
But the obvious remedy is not available, and the file says why
The honest arm would be "a directory test rejects a gitfile". It cannot be
written literally, because the sweep in this part reads this file and
[ -d "$_ck_dir/.git" ]spelled out would be counted as a real site. Line 96records that the first version of this part did exactly that and stayed red after
both real sites were fixed.
It could be assembled from a fragment, as the part already does for
_ck_op— butthe resulting arm asserts
[ -d file ]is false for a regular file, which is POSIXagain. And the product property those two are illustrating already has an arm,
already dated:
So the two survivors are scaffolding around a dated arm: they make the
gitfile/directory distinction concrete next to the sweep that depends on it.
Deleting them loses a reader's handhold; keeping them leaves two unfalsifiable
rows in the ledger looking like coverage.
The decision this needs, before I touch 393 rows
Three kinds, three different remedies:
The precedent in this tree is opt: resolve the written stripe limit through the metapage (#1210) #1277's guard, kept with "currently unnecessary,
and untestable from the suite" written in as many words. Applied here that
means a comment saying so, and renaming to
premise:so the metric stopscounting it as an arm.
Renaming is not free: a check name is a ledger key, so each rename costs a
ledger row and a
TESTS.mdline, and the same question applies to an unknownfraction of the 393. I am not making that call unilaterally across hundreds of
rows on the strength of one part.
So: does kind 2 get renamed to
premise:with a comment, or kept as an arm with acomment, or deleted? I will take either and work the parts accordingly. My own
preference is renaming, because an unfalsifiable assertion about the fixture is
what a premise IS, and leaving it named as an arm is what made this part read as
three failures when it holds one.
No arms attacked yet and no rows changed.
570is the worked example, not a fix.🤖 Generated with Claude Code
https://claude.ai/code/session_01MpajdQbkVJ9ey1XyYHcikP
60 remaining items
Merge pull request #1356 from commandprompt/test/1339-undated-premises
test: five premises a product change reddens lose the prefix (#1339)
test: five premises a product change reddens lose the prefix (#1339)
test: five premises a product change reddens lose the prefix (#1339)
test: five premises a product change reddens lose the prefix (#1339)
test: alter_am_cleanup gets its pytest twin, and each arm reddens for…
test: native_groupagg's upper-bound and NULL-key arms can fail, and a…
test: rewrite_storage_oid's five product premises are arms (#1339)
test: native_groupagg's upper-bound and NULL-key arms can fail, and a…
test: rewrite_storage_oid's five product premises are arms (#1339)
test: tier_partition's injected-fault checks are arms, not premises (#…
test: tier_partition's injected-fault checks are arms, not premises (#…
test: analyze_differential's core-shape checks are not premises, and …
test: analyze_differential's core-shape checks are not premises, and …
test: analyze_differential's core-shape checks are not premises, and …