tables: retain-unknown in the Go oracle, and the wire fuzzer's retention leg (#525) by rowan-claude · Pull Request #658 · mas-bandwidth/schema

rowan-claude · 2026-09-07T04:04:13Z

The oracle's half of retain-unknown (docs/SPEC-TABLES.md §6.6) and the wire
fuzzer's retention leg that could not be built until it landed (§4.2). Two of
the four things #525 still owed after #615 put the feature in the C++
reference.

Red first

internal/tablewire carried no retention, so the gate was written against
nothing and quoted red before a line of the oracle existed:

internal/tablewire/retain_test.go:74:22: undefined: tablewire.Retain
internal/tablewire/retain_test.go:76:23: undefined: tablewire.DecodeRetain
internal/tablewire/retain_test.go:89:12: report.Retained undefined
internal/tablewire/retain_test.go:89:36: report.RetainLost undefined
internal/tablewire/retain_test.go:94:24: undefined: tablewire.EncodeRetain

The rows read the same pinned RT1, RT2 and RT3 vectors the C++ reference's own
retain gate reads, and require the same counters, the same retained ids and the
same saved bytes. It is a byte-for-byte mirror and not a second reading.

The oracle's shape

internal/tablewire now carries what §6.6 specifies, each piece held by a row
of the gate:

The caller's two stores. tablewire.Retain is a byte buffer, a capacity,
an id list and an id capacity, and what has been used of each. It grows
nothing and the wire raises neither ceiling.
The resolving walk, one pass each way, linear in the record's bytes. The
capture writes each length AFTER the content it frames; the save takes one
post-order pass that computes each content's wire size and leaves it where the
emit reads it. Kind 13, 15, 30, 14 at those element kinds and 16 at
every element kind resolve; everything else is copied verbatim.
The six excluded classes at one retain_lost each, pinned as six vectors:
a kind 17 field, an array of 17, a 17 met three bodies down inside an
unknown table, an unknown enum variant reference, an unknown union arm id and
an unknown keyed-array slot. The node record whose type id is unnameable is a
row of its own on RT3.
The drop rule. A reference above the entry count, a reference at an entry
of zero, a reference at one of the three reserved ids, a non-canonical length,
an inner body whose terminator falls short, a nested L past its parent and a
kind 17 at any depth each drop the record, count one retain_lost and never
raise malformed on the plain read.
The record dies with the occurrence that carried it. A whole-replacing
field discards AT THE FIELD ID, before a byte of the new occurrence's framing
is read, which is where the reference commits. The collections do not: a map,
a list and a bounded array take the discard where their own read commits to
replace, past a header that can turn the occurrence inert or foreign, and an
enum-keyed array never takes it at all. That is the page's disjoint-parts row.
The retained tail and the merged trailer. A record goes back at the end of
its own body in the order retained, before the node-table field in the root,
and the two stores are numbered into ONE trailer in merged first-use order.
Idempotence. The second save equals the first byte for byte; the first
does not equal the original, because moving a field permutes first-use order.

The leg

make tables-wire-fuzz-retain drives the same mutants through both engines'
retaining paths and compares the two retention counters beside the six and the
saved bytes beside them, plain and under ASan and UBSan:

wire-fuzz retain: seed 24845619678, 23 seeds over 7 roots, 8 seeds absent
(roots the leg has no codec for), 115 seeds outside the arm, 54521
enumerated + 2000 random = 56521 mutants, 0 divergences, 6.1 s, 9249 mutants/s

It is on make test at N=20000 and on make tables-cpp-release at
N=500000 over two seeds, beside the retention-off pair.

Two negative controls, one per engine

A leg that has never gone red proves nothing about either engine, so there is
one control per engine. Each removes ONE check through go build -overlay, so
no tracked file moves, and each blade matches exactly one line of one file.

tables-wire-fuzz-retain-oracle-negative-control removes the oracle's DROP
RULE, the verdict the resolving walk states:

harness: FAILED after 425 mutants: the retention report differs: the leg says
retained=0 retain_lost=1, the oracle says retained=1 retain_lost=0

tables-wire-fuzz-retain-class-negative-control removes the reference's
retain_lost on an EXCLUDED CLASS at the unknown arm, regenerates the corpus
from the sabotaged compiler and builds the same leg against it:

harness: FAILED after 43908 mutants: the retention report differs: the leg says
retained=0 retain_lost=0, the oracle says retained=0 retain_lost=1

Both go red at N=0 too, which is how make test runs them: the enumerated
passes alone carry each mutant. Both are green without the sabotage, which is
the run quoted above.

What the leg made the oracle fix

Each is a divergence the arm printed, not a shape anyone reasoned to:

a framed content is consumed EXACTLY, at every kind; slack a plain read skips
by the length is a shape the walk cannot frame
an array's element kind and count are honoured before its bytes are copied,
and a body too short for its own header is dropped
kinds 14 and 16 resolve as a framed CONTENT, not only as a payload, and so
do 15 and 30
a by-value nested body is the SAME STORAGE before and after a reset, so damage
refills it in place and an address names one body for the life of a load
a record the save walk never reaches counts one retain_lost; a body
SUPERSEDED by a later legal occurrence is discarded at the load and moves
neither counter
a union cell holds the arm body its occurrence built even when that arm's
framing was damaged, so the NEXT occurrence can discard it
a whole-replacing field discards before it reads any framing, which is where
the reference discards

Page lines retired

Present tense, no history:

§4.2 requirement 3 said a retention leg was owed and its cost landed on
the oracle first. It now states the leg, its arm, and its two capacities
declared large on both sides.
§4.2's negative controls gain the leg's own pair, as a table beside the
fuzzer's.
§6.6's closing pair said internal/tablewire carries no retention today
and named the leg among what is owed. The backend status now reads "the C++
REFERENCE and the ORACLE", and what is still owed is the eight ports and the
MESSAGE form's LoadRetain.
VERSIONING.md's Retain-unknown: an opt-in that keeps what this build cannot name (ruling 12 on #523) #525 line: the oracle's retention and the leg move to what
is built; the eight ports, the message form's LoadRetain and the conformance
rows remain.
COMPETITION.md's s23 cell: three things stand between it and green rather
than four.
test/conformance/README.md's wire-fuzz contract gains the two fields the
leg added, because a port reads that page to build its own leg: the roster
entry's RETAIN byte, and retained and retain_lost, which ride EVERY reply
and are zero on every entry the roster did not mark.

Where the page is silent, and what was built

The page settles the feature, not the leg, so these are choices and each took
the fewest cases:

Whether the arm covers MESSAGE-form roots. §6.6 says the form 2 read is
not built and the form 2 write refuses by name, so the arm is the
VARIABLE-CLASS FILE ROOTS and nothing else. The line says how many seeds sat
outside it rather than leaving them uncounted, on the same absent-is-not-
failure rule the roster already uses.
The leg's capacities. The page states the capacity RULE and holds it by
each engine's own gate, where the buffer is pinned one byte short of the last
record; it says nothing about the fuzzer's. Both are declared LARGE on both
sides, because a record's byte cost is the port's own and two engines at one
tight capacity would drop different records and measure the two layouts
rather than the feature.
A variant reference of ZERO. §3 says the zero reference is the enum's
None and §6.6 says kind 30 is a variant id the walk resolves. It is treated
as a VALUE, not a reference: it names no id, so it passes through and takes
no entry in either store. Resolving it as a reference spent an id on a
variant that has none.
Whether the leg belongs on make test and the release gate. The page
puts the retention-off pair on both and is silent about this one. It is on
both, at the same lengths.
How many controls the leg carries. The page says the fuzzer has two and
says nothing here. Two, one per ENGINE, because the leg compares two engines
and a control that can only sabotage one of them leaves the other untested.

A reading, not a disagreement

§6.6's conformance list says "FIVE ROWS, one per excluded class a wire can carry
to the unknown arm" while the excluded-class table has six no rows. They agree:
the sixth, the reserved node-table field, is excluded by construction and a test
cannot reach it. The oracle's gate pins SIX vectors because the NODE-INDEX class
is three shapes on the wire (a kind 17 field, an array of 17, and a 17 met
at depth inside an unknown table) and the page names all three in that one row.
Nothing else in the reference and the page disagreed.

From the cold read

A cold read of the branch found three, all of them in the space the leg cannot
see: none is reachable by mutating a small seed, so the arm stayed at zero
divergences through all three.

The walk's nesting cap counted from a different floor in each engine. The
oracle entered the resolving walk at depth 1 while framed also takes a
level, so its first framed content sat at depth 2 and it dropped at 64 nested
bodies where the page and the reference admit 64 and drop 65. The fix is the
entry depth; TestRetainWalkDepthCap pins both depths in the oracle, red at
the 64 row before it, and the reference's gate already pinned the same two.
A dropped record left its ids in the CALLER'S list. tail undid only the
file's own id table on an overflow, so entries a record took before it went
bad stayed behind and the next record was refused an entry the save does not
owe: one dropped record could cost more than one retain_lost. The reference
undoes both stores at one mark. TestRetainDroppedRecordSpendsNoId is red at
retain_lost=2 before the fix.
The id-list-one-short row measured nothing. It sized the short list off a
DECODE, and a load writes into neither store, so it read zero, ran at a
capacity of -1 and lost every record while asserting only that something was
lost. It now sizes off a save with room for every id and runs at two short
capacities, the page's own and the reference's, pinning one retain_lost and
the reference's saved BYTES at each. The reference's gate writes that new
golden, and its inline comment claiming three distinct retained ids is
corrected to the twelve both engines measure.

Goldens

make update-goldens moved nineteen C++ goldens and no wire byte. Two things
moved them and neither is a hand edit: the resolving walk resolving a kind 15
arm and a kind 30 variant as a framed CONTENT, which landed in the emitter
without its goldens, and origin/main's own new text-map unit, which is
variable-class and so carries the retain family like every other.

Part B stacks on this branch: the message form's LoadRetain and the
conformance manifest's retain surface.

…ead (#525) The retain gate's own wires become ordinary pinned vectors under testdata/wire/tables: the RT2 round-trip wire, RT1's first save with retention, the six excluded-class wires and RT3's unknown node record. internal/tablewire reads the same bytes the C++ reference reads, which is what makes the two engines comparable at all (docs/SPEC-TABLES.md §4.2). The gate writes them under SCHEMA_UPDATE_WIRE_GOLDENS and compares against them on every other run, so the inline kFirstSave pin and the file cannot drift apart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

internal/tablewire carries no retention, so this is the red the feature is built against (docs/SPEC-TABLES.md §6.6, §4.2): internal/tablewire/retain_test.go:74:22: undefined: tablewire.Retain internal/tablewire/retain_test.go:76:23: undefined: tablewire.DecodeRetain internal/tablewire/retain_test.go:89:12: report.Retained undefined internal/tablewire/retain_test.go:89:36: report.RetainLost undefined internal/tablewire/retain_test.go:94:24: undefined: tablewire.EncodeRetain The rows read the same pinned vectors the C++ reference's retain gate reads and require the same counters, the same retained ids and the same saved bytes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

internal/tablewire carries the retain-unknown opt-in (docs/SPEC-TABLES.md §6.6): the caller's two stores, the resolving walk one pass each way, the six excluded classes counted at one each, the drop rule, the idempotent second save and the merged first-use trailer. The oracle's record is READER-PRIVATE, as the page says it may be: a resolved byte string whose references are eight bytes and whose inner lengths are this engine's own fixed width, written AFTER the content they frame so a capture costs one pass. The emit is the post-order pass a canonical LEB128 length riding first forces. A record belongs to the BODY OCCURRENCE that carried it and dies with it, which this engine gets from the tree itself: a later legal occurrence of a known body is a new instance, so the earlier occurrence's records go with the values it held and neither counter moves. The rows now read green against the C++ reference's own pinned save, byte for byte, permuted trailer and all: --- PASS: TestRetainRoundTrip --- PASS: TestRetainTrailerIsMergedInFirstUseOrder --- PASS: TestRetainExcludedClasses (six subtests) --- PASS: TestRetainUnknownNodeRecord --- PASS: TestRetainIsAdditive --- PASS: TestRetainCapacities Decode and Encode are unchanged and retain nothing: the plain pair takes a nil retain state and every branch that tests it is a branch a plain read does not take. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…nt as a framed CONTENT too (#525) The wire fuzzer's retention leg found this on its first long pass. TableRetainInContent carried cases 13, 14, 16 and 17 and copied every other content as bytes. A kind 15 arm's own payload, and a kind 16 slot whose element kind is 15 or 30, both arrive there and both carry a REFERENCE, so a verbatim copy re-emits that reference into a permuted trailer where it names another id. It is also a hole in the walk's own safety property. docs/SPEC-TABLES.md §6.6: "the walk catches every other one, a 17 field or an array of 17 inside a kind 13 body, UNDER A KIND 15 ARM, in a kind 14 or 16 element of either" and "A retained record therefore never carries a node index". A kind 17 under a kind 15 arm went through. The page's own line is the fix: "Kind 15, an arm id and then the arm's payload under this same rule. Kind 30, a variant id." Both sides of the walk gain the two cases; the RT corpus's pinned save does not move, because no RT vector reaches the shape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

) The leg (docs/SPEC-TABLES.md §4.2, §6.6): `wire-fuzz --retain` drives the same mutants through both engines' RETAINING paths and compares the two retention counters beside the six and the saved bytes beside them. The roster entry gains a retain byte and the reply gains the two counters; the arm's roster is the VARIABLE-CLASS FILE ROOTS and nothing else, because a fixed-class root's LoadRetain is refused by name and a form-2 SaveRetain refuses by name, and the line says how many seeds it left out. Both capacities are declared large on both sides: a record's BYTE cost is the port's own, so two engines at one tight capacity would drop different records and the arm would measure the two layouts rather than the feature. What the arm made the oracle fix, each a divergence it printed: - a framed content is consumed EXACTLY, at every kind. Slack a plain read skips by the length is a shape the walk cannot frame. - an array's element kind and count are honoured before its bytes are copied, and a body too short for its own header is dropped. - kinds 14 and 16 resolve as a framed CONTENT, not only as a payload. - a BY-VALUE NESTED BODY IS THE SAME STORAGE before and after a reset: it is a member of its parent in every generated target, so damage refills it in place rather than replacing it, and an address names one body for the life of a load. tabletext.Model.Refill is that reset. - a record the save walk never reaches is one whose path no longer names a body, and it counts one retain_lost; a body SUPERSEDED by a later legal occurrence is discarded at the load instead and moves neither counter. - a union cell holds the arm body its occurrence built even when that arm's framing was damaged, so the NEXT occurrence can discard it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ing (#525) THE REFERENCE DISCARDS BEFORE IT READS ANY FRAMING. A second occurrence of a nested body or a union wins whole (§3, §4), and the reference commits to that at the FIELD ID: the records the earlier occurrence held go before a byte of the new occurrence's framing is read, so an occurrence whose own framing turns out to be damaged still takes them. The oracle discarded at the body instead, one read later, and kept records the reference had already dropped. `discardsWhole` names the shape that is right for: a by-value nested table and a by-value union, both of which replace everything the earlier occurrence held. THE COLLECTIONS DO NOT. A map, a list and a bounded array replace only where their own read COMMITS to replace, which is past a header that can turn the occurrence inert or foreign, and each takes its discard there instead. An enum-keyed array never replaces whole at all: a second occurrence overwrites the slots it carries and leaves the rest standing, so a discard at the field would kill records under slots the writer never touched. The enum's None rides back as a VALUE (§3). A variant reference of zero names no id, so the resolving walk passes it through and the emitter writes it without taking an entry in either store; resolving it as a reference spent an id on a variant that has none. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…525) The leg on the make line (docs/SPEC-TABLES.md §4.2, §6.6), plain and under ASan and UBSan, exactly as the retention-off pair runs: wire-fuzz retain: seed 24845619678, 23 seeds over 7 roots, 8 seeds absent (roots the leg has no codec for), 115 seeds outside the arm, 54521 enumerated + 2000 random = 56521 mutants, 0 divergences TWO NEGATIVE CONTROLS STAND BEHIND IT, one per engine, because a leg that has never gone red proves nothing about either. Each removes ONE check, through `go build -overlay` so no tracked file moves, and each blade matches exactly one line of one file. The ORACLE's DROP RULE. The walk is an interpretation and its verdict is stated (§6.6): a reference above the entry count, a reference at an entry of zero, a reference at a reserved id, a non-canonical length, an inner body whose terminator falls short, a nested `L` past its parent and a kind 17 at any depth all DROP THE RECORD. Without the verdict: harness: FAILED after 425 mutants: the retention report differs: the leg says retained=0 retain_lost=1, the oracle says retained=1 retain_lost=0 The REFERENCE's `retain_lost` ON AN EXCLUDED CLASS. Every exclusion counts, so a caller that needs to know retention held reads ONE NUMBER. `retainLostInline` is the count the emitter puts beside `unknown` at every class a wire can carry to the unknown arm. Without it: harness: FAILED after 43908 mutants: the retention report differs: the leg says retained=0 retain_lost=0, the oracle says retained=0 retain_lost=1 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

) `make update-goldens` over the merge. Two things moved them and neither is a hand edit: the resolving walk resolving a kind 15 arm and a kind 30 variant as a framed CONTENT and not only as a payload, which landed in the emitter without its goldens, and origin/main's own new text-map unit, which is variable-class and so carries the retain family like every other. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Three status lines say the oracle carries no retention and the leg that needs it is not buildable. Both are built, so the lines go, present tense and no history: - §4.2's requirement 3 said a retention leg was owed and its cost landed on the oracle first. It now states the leg, its arm, its two capacities declared large on both sides, and the reason: a record's byte cost is the port's own, so two engines at one tight capacity would measure the two layouts rather than the feature. - §4.2's negative controls gain the leg's own PAIR, one per ENGINE because the leg compares two of them, each blade matching exactly one line of one file. - §6.6's closing pair said `internal/tablewire` carries no retention today and named the fuzzer leg among what is owed. The oracle carries what the subsection specifies and the backend status reads "the C++ REFERENCE and the ORACLE"; what is still owed is the eight ports and the MESSAGE form's `LoadRetain`. VERSIONING's #525 line and COMPETITION's s23 cell move with them: the oracle's retention and the leg are built, and three things stand between s23 and green rather than four. The wire-fuzz driver contract gains the two fields the leg added, because a port reads that page to build its own leg: the roster entry's RETAIN byte and the reply's `retained` and `retain_lost`, which ride EVERY reply and are zero on every entry the roster did not mark. The leg and its controls join the make lines they belong on: `make test` at N=20000 with both controls at N=0, where the enumerated passes alone turn each one red, and `make tables-cpp-release` at N=500000 on two seeds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`modernize` names the loop, and the linter is a CI gate. The behaviour is the one §6.6 states either way: a retained id used by two records takes ONE entry, exactly as any repeat does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`golangci-lint` is a CI gate and it named four things. Three are the gate's own De Morgan spellings, rewritten with the same meaning. The fourth is real: `(*encoder).retains` was dead. A BODY THAT CARRIES A RETAINED FIELD DOES NOT ELIDE (§6.6) holds BY CONSTRUCTION rather than by a second test, because the retained tail is written inside `encodeBody`, so a body holding one is longer than its terminator and the elision length cannot fire. The helper asked a question the code had already answered. The rule now reads where it holds, at the nested-table elision itself, rather than beside a function nothing called. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The page states the cap as 64 NESTED BODIES (docs/SPEC-TABLES.md §6.6), with the last depth it admits still riding and one past it refused. `capture` entered the resolving walk at depth 1 while `framed` also takes a level, so the first framed content sat at depth 2 and the oracle dropped at 64 where the reference admits 64 and drops 65. The reference enters at 0 (internal/codegen/cpptable/retain.go), and a cap read from a different floor is a second wire law: the two engines drop different files. `TestRetainWalkDepthCap` builds the reference's own `deep_wire` shape against §3's grammar, one unknown kind 13 field whose payload is N nested table bodies, and pins both depths in the oracle. It is red before the fix at the 64 row, retained=0 retain_lost=1 where the page and the reference say 1 / 0. The reference's gate already pins the same two depths (test/tables/retain_main.cpp, `walk_is_linear`), so both engines now answer alike. No saved bytes move: the round trip's byte for byte pin against the reference stands unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`tail` took one mark before each record's emit and undid only the file's own id table on overflow (`e.ids.rollback`). `emitter.intern` appends to the caller's list as each id is reached, so a record that went bad PARTWAY left its entries standing there: the next record was then refused an entry the save does not owe, and one dropped record cost more than one `retain_lost`. The page allows exactly one per dropped record and says nothing else about the save changes (§6.6), and the reference undoes both stores at one mark (internal/codegen/cpptable/retain.go, `truncate`). `TestRetainDroppedRecordSpendsNoId` builds a record naming four ids under a list that holds two, so it overflows partway, with a second record naming one new id behind it. Red before the fix at retain_lost=2 where the reference's rule gives 1: the leaked entries killed a record that had room. The row also reads the file's own trailer, where no id the dropped record reached took an entry. The wire builder the depth rows introduced is renamed `wireBuilder` now that a second row builds a wire that is not deep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`TestRetainCapacities` took `IdCapacity: len(whole.Ids())` after a DECODE only. A load writes into neither store (§6.6), so that read zero, the sub-test then set the capacity to -1, and every one of the nine records was refused its id. The old assertion was `RetainLost == 0` is a failure, which nine satisfies, so the page's "EXACTLY ONE `retain_lost` for each such record" was never pinned: red at retain_lost=9 the moment the row asks for one. The list is now sized off a SAVE with room for every id, which is the only place the count exists, and the row runs at two short capacities: one entry short of the twelve the wire carries, which is the page's own words, and the two the reference's gate runs. Both drop exactly one record, `parcel`, the only record naming ids beyond its own, and both answer one: the count is per record and not per entry. The save's BYTES are pinned now as well as its counter, because a drop for want of an id entry takes a record out of a body and renumbers every reference behind it, which a count cannot see. `test/tables/retain_main.cpp` writes the new golden from the same row, and the oracle byte-compares it at both capacities. The gate's inline comment there said the wire carries three distinct retained ids. Measured, both engines report twelve: `parcel` names nine of its own. The capacity of two stands, and the comment now says why it is short. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…#657, the goldens regenerated over the Count member (#525) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

… main left stale (#632) FOUR LINES ARE THIS BRANCH'S, one per emitted message reader whose root's numbering reaches a *string blob: blobs/AssetsTable.h twice, for Asset and for Catalog; maps/DocsTable.h once, for a map whose VALUE is a text buffer; and arms/GateTable.h once, for a blob reached only through a union arm. Each is the same line, the record's bytes handed to TableUtf8Valid. NO WIRE GOLDEN MOVED: the write side is untouched, and a read that refuses damage changes no byte a writer produces. THE REST WAS ALREADY STALE ON MAIN and `make tables-block-zero-cost` was red before this branch: e77093c (#658) added the retain walk's `case 15: case 30:` and e123f1b (#662) moved the map entry readers' text path, and neither re-pinned tables/maps. maps/ChunksTable.h, RunsTable.h, SlotsTable.h and SpansTable.h carry nothing of this branch's at all, and CellsTable.h carries none of its four lines either. The gate now compares 117 Table sources byte-identical to their pins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`LoadRetain` reads a form-2 body as it reads a file's, the resolving walk replacing every reference with the id it names AGAINST THE CONNECTION'S VOCABULARY instead of a trailer (§3.3). `SaveRetain` writing form 2 still refuses by name, so the round trip a caller has is form 2 in and form 1 out, and the file carries its own table and takes §6.6 unchanged. THE RECORD IS THE FILE FORM'S OWN, to the byte, so the capture is a TRANSCODE as well as a resolve: a bitpacked value is read at the width its announced shape states and written at the width the file form spells, and from there it is the same record, laid down in the same slots and read back by the same emit walk. Neither engine carries a second copy of the emit side. THE SKIP RUNS FIRST AND THE CAPTURE SECOND, over the same bits. The plain read's verdict on an unknown entry is the SKIP's, whether or not this build retains, and a capture that lands anywhere but the skip's own end drops the record. It is two flat passes over one record's bits and never a walk that doubles at every level. A BATCH TAKES ONE REGION AND ONE RETENTION BUFFER A BODY: each body carries its own node directory inside that one region (§3.3), so a record's first step stays an index into that body's own directory and SaveRetain's accounting is the file form's unchanged. The reference and the oracle read the same pinned batch and write the same two files, 442 and 93 bytes, byte for byte: retained=10 retain_lost=6 unknown=17 `LoadRetainMessages` is claimed in tableGeneratedVerbs and §11 on that list's own rule, because it is emitted: 52 suffixes become 53 and 68 become 69. Goldens: every C++ table header of a unit with a variable-length table gained the retaining message family. Six under testdata/golden/tables/maps were already stale on main — their committed text predates #658's retain runtime and #620's message text rule — and this repin brings them level; the other eighteen moved on this change alone. No WIRE golden moved: retention is additive and the plain codec is untouched to the byte. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`make conformance-pin update-goldens` on a clean checkout of main rewrites six files and nothing else: testdata/golden/tables/maps/{Cells,Chunks,Docs, Runs,Slots,Spans}Table.h. The pins carry neither the retain walk's arm and enum-reference case (kind 15 and kind 30 resolving as framed content) nor the map entry loader's present clamp, both of which the C++ table emitter writes today. This is the shape #461 describes. The pins were written on the branch that became #662, which was cut before #658's retain-unknown walk landed and merged after it; the two diffs do not touch the same lines, so the merge is clean and nothing in CI reads these files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… (#671) * tables: a *string blob on a message body carries no content rule, red first (#632) docs/SPEC-TABLES.md §3.1 states kinds 12 and 33's content rule MET AT A NODE: "A TEXT blob's CONTENT is refused on the same terms", so a *string blob whose bytes are not well-formed UTF-8, or which carries a zero byte, is damage. §3.3 says a form-2 body's content rules are §3's, unchanged in what they reject, and that what differs is only the recovery, which a bit stream does not have. Neither engine carries it at the message site, where both carry it at the file site (decodenodes.go and pointers.go). The two gates are the page's rows over blobdemo's Catalog, whose numbering reaches a *string blob through note and a *bytes blob through thumb. test/tables/message_blob_main.cpp is the C++ reference's and TestAStringBlobRecordOnAMessageBodyCarriesTheContentRule is the oracle's. Both are RED at this commit: the truncated sequence, the zero byte, the overlong encoding and the lead byte 0xFF all load with a silent report. The last row of each is a *bytes blob carrying the same bytes, which must stay silent: a *bytes blob is bytes and never text, so the rule is the string id's alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: a *string blob record on a message body carries the content rule (#632) Both engines read a form-2 blob record's bytes through the SAME function the FILE form reads one with, so there is one rule and no second copy. The C++ emitter: messagevariable.go's PASS TWO, where the record's bytes are already in hand. The align a blob record spends before its bytes leaves the span on a byte boundary, so the same pointer goes to the runtime's TableUtf8Valid and to the memcpy below it, and the line is emitted under rootReachesStringBlob, the same guard pointers.go emits the file form's under. The check does NOT ride in MessageRecordScan, which is the framing walk LoadMeasure shares: ill-formed content is sizeable, so a measure still answers the region the framing commands and only the decode refuses. The Go oracle: messagedecode.go's placement loop calls textValid on the record's bytes under the TString arm alone, which is decodenodes.go's line for the file form. What differs from the file form is only the recovery, which a bit stream does not have: where a file counts the record malformed and reads on with every slot naming it null, a batch ends there. One malformed counts, the bodies before it stand, and nothing after is read (§3.3). A *bytes blob keeps no rule, because it is bytes and never text (§3.1), and both gates carry the row that says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: re-pin the zero-cost goldens the blob rule moves, and the six main left stale (#632) FOUR LINES ARE THIS BRANCH'S, one per emitted message reader whose root's numbering reaches a *string blob: blobs/AssetsTable.h twice, for Asset and for Catalog; maps/DocsTable.h once, for a map whose VALUE is a text buffer; and arms/GateTable.h once, for a blob reached only through a union arm. Each is the same line, the record's bytes handed to TableUtf8Valid. NO WIRE GOLDEN MOVED: the write side is untouched, and a read that refuses damage changes no byte a writer produces. THE REST WAS ALREADY STALE ON MAIN and `make tables-block-zero-cost` was red before this branch: e77093c (#658) added the retain walk's `case 15: case 30:` and e123f1b (#662) moved the map entry readers' text path, and neither re-pinned tables/maps. maps/ChunksTable.h, RunsTable.h, SlotsTable.h and SpansTable.h carry nothing of this branch's at all, and CellsTable.h carries none of its four lines either. The gate now compares 117 Table sources byte-identical to their pins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: two blades and two pinned vectors under the blob content rule (#632) ONE BLADE AN ENGINE, each removing exactly one line and each naming what it turns red. message-blob-accepts-ill-formed drops textValid from the oracle's message placement loop; message-emitter-blob-accepts-ill-formed drops the emitted TableUtf8Valid call from messagevariable.go's PASS TWO. Each blade is drawn twice. Against the PAGE: the oracle's rides in MESSAGE_FORM_CONTROLS against TestAStringBlobRecordOnAMessageBodyCarriesTheContentRule, and the emitter's in the new tables-message-form-blob-negative-control, which builds test/tables/message_blob_main.cpp against a regenerated tables/blobs. The true run of that program is the target's own first step, so the gate is green before the blade is drawn, and the sabotage reddens four rows and leaves the two silent-report rows standing. Against EACH OTHER: message_blob_ill_formed_text is blob_str8 re-encoded as a batch of one with the fourth byte of note's eight byte payload replaced by 0xFF, and message_blob_zero_byte is the same wire with 0x00 there instead, so the two separate the two halves of one check. Nothing else moves, so what the two readers answer differently is the content rule and nothing else. The *bytes record beside it carries bytes no UTF-8 rule would accept and is never checked, which is the vector's own control. tables-wire-fuzz-message-blob-oracle-negative-control and its leg twin replay both vectors alone and require red on each, with the reports mirrored: the leg says 0,0,0,0,0,true,read, the oracle says 0,0,0,0,0,false,read the leg says 0,0,0,0,0,false,read, the oracle says 0,0,0,0,0,true,read AND THE LEG GREW THE ROOT THE VECTORS NEED. blobdemo's Catalog is the only root on the fuzzer's roster whose numbering can PLACE a text blob record, so it is the only one whose wire can carry §3.1's rule at a node. Its AssetsTable.cpp was already in CONFORMANCE_SOURCES, so the leg costs one include and one MESSAGE_VARIABLE row, and the corpus pass now runs 45 roots rather than 44. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: §3.3's test ledger names the content rule it now holds at a node (#632) No sentence of §3 or §3.1 moves and none is added: §3.1 already says a text blob's CONTENT is refused on the same terms as a kind 12 payload, and §3.3 already says a form-2 body's content rules are §3's. What was stale is HELD BY TEST, which carries one bullet per rule with its red clause and carried none for this one. The bullet names the six rows the two engines' gates run, the *bytes row among them, and the three ways a leg goes red. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

…ace (#525) (#679) * tables: red first for the message form's LoadRetain (#525) The pinned batch and the two rows that read it, before the function exists. RT2 writes a form-2 batch of two bodies over the same values the file rows read: the fields at three depths that are retained, the whole `parcel` table the resolving walk reads, and every class this form can put on the unknown arm — a field of kind 17, an array whose element kind is 17, a table whose payload meets a 17 three bodies down, an unknown enum variant reference, an unknown union arm id and an unknown keyed-array slot. The reference's row and the oracle's read that one vector and require the same counters and the same saved file. Both are RED: test/tables/retain_main.cpp:1023:20: error: no member named 'NodeLoadRetainMessages' in namespace 'tblrt1'; did you mean 'NodeSaveRetainMessages'? internal/tablewire/retain_test.go:716:29: undefined: tablewire.DecodeRetainMessages Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: the message form's LoadRetain, in both engines (#525) `LoadRetain` reads a form-2 body as it reads a file's, the resolving walk replacing every reference with the id it names AGAINST THE CONNECTION'S VOCABULARY instead of a trailer (§3.3). `SaveRetain` writing form 2 still refuses by name, so the round trip a caller has is form 2 in and form 1 out, and the file carries its own table and takes §6.6 unchanged. THE RECORD IS THE FILE FORM'S OWN, to the byte, so the capture is a TRANSCODE as well as a resolve: a bitpacked value is read at the width its announced shape states and written at the width the file form spells, and from there it is the same record, laid down in the same slots and read back by the same emit walk. Neither engine carries a second copy of the emit side. THE SKIP RUNS FIRST AND THE CAPTURE SECOND, over the same bits. The plain read's verdict on an unknown entry is the SKIP's, whether or not this build retains, and a capture that lands anywhere but the skip's own end drops the record. It is two flat passes over one record's bits and never a walk that doubles at every level. A BATCH TAKES ONE REGION AND ONE RETENTION BUFFER A BODY: each body carries its own node directory inside that one region (§3.3), so a record's first step stays an index into that body's own directory and SaveRetain's accounting is the file form's unchanged. The reference and the oracle read the same pinned batch and write the same two files, 442 and 93 bytes, byte for byte: retained=10 retain_lost=6 unknown=17 `LoadRetainMessages` is claimed in tableGeneratedVerbs and §11 on that list's own rule, because it is emitted: 52 suffixes become 53 and 68 become 69. Goldens: every C++ table header of a unit with a variable-length table gained the retaining message family. Six under testdata/golden/tables/maps were already stale on main — their committed text predates #658's retain runtime and #620's message text rule — and this repin brings them level; the other eighteen moved on this change alone. No WIRE golden moved: retention is additive and the plain codec is untouched to the byte. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: the manifest's retain surface, and the message path's control (#525) MANIFEST.txt gains `retain` and `retain-message` lines and the nine rows §6.6 owes: the round trip at depth, the truncated record, the id list one short, the five excluded classes a wire can carry to the unknown arm, and the message form's tail. FORMAT.md's page and the harness's two surfaces follow: `retain` carries the COUNTERS and `retain-save` the BYTES, so each is one shape. THE TWO CAPACITIES TRAVEL AS RULES AND NOT AS NUMBERS. A record's byte cost is the port's own, so `short` says ONE BYTE SHORT OF THE LAST RECORD and a leg resolves it against its own full load; the id list's capacity is a COUNT and rides as one. The reference answers both surfaces and the oracle claims the same nine rows in the harness's own test, against the manifest's counters and the pinned saves. cpp retain pass 9/9 retain-save pass 3/3 c retain absent retain-save absent go retain absent retain-save absent rust retain absent retain-save absent THE NEGATIVE CONTROL is `tables-message-form-retain-negative-control`: it takes the message path's call into the resolving walk out of the emitter through an overlay, regenerates the RT set and requires the retain gate to go red on the pinned batch's own row and on nothing else. FAIL line 1037: report.retained == 10 TestTheTailIsUnconditional gained a map-entry skip. The vocabulary tail never names a map's generated entry (§2.8, §20.2) and the test's own table set did not skip one, because no connection unit carried a map until RT2 did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: retire §6.6's owed lines, and the pages that count them (#525) §6.6's backend status names the reference as carrying `LoadRetainMessages` beside the three verbs, and what is still owed is the eight ports. Its HELD BY TEST list is present tense and says which nine rows are the conformance manifest's own data and why the rest are the two engines' own gates: a record's byte cost is the port's own, an allocation audit is a language's own instrument, and a refusal by name is a compile error rather than an answer a driver writes. §3.3's retention paragraph says both halves are carried, states that the record is the FILE form's own, and states the batch's one buffer a body. §3.3's claimed set names `LoadRetainMessages` beside the three plural verbs. VERSIONING's #525 line: the eight ports only. ROADMAP's retain-unknown cell moves to ✅ for cpp, and 339 cells become 340. Every gate §6.6 names for a backend exists in the reference: the three verbs, the fixed-class refusal by name, the message form's read, the second family of body functions, all sixteen HELD BY TEST rows in test/tables/retain_main.cpp, and the nine conformance rows the harness answers 9/9 and 3/3. No em dash rides in a line this branch adds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: drop the sabotage binary a build left at the root (#525) `go build ./tools/sabotage` writes ./sabotage beside the tree, and one commit picked it up. The tool is run with `go run` everywhere it is used. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: re-anchor the two message sabotages this branch moved (#525) `message-clamp-drops-surplus` and `message-surplus-lands-on-zero` both anchor on the array element loop of the message decoder, which gained the `mine` flag that says whether an element is this build's: a surplus element is decoded into a scratch where a file's reader steps over it by its length, so retention has to be off for it (§6.6). The anchors take the new line and the blades cut what they always cut. The tool caught it, by design: `edit 0 matched 0 times, want exactly 1. The emitter moved away from the anchor, so this negative control no longer breaks what it names.` Both go red on their own rows again, and `make tables-message-form-negative-control` is green end to end. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: size the driver's retain arrays from the batch, and repin after the merge (#525) The message retain codec declared one root, one retention buffer and one id list for the FORM's 256 bound rather than for the batch's own body count, which is a megabyte a body a driver never uses. The batch is opened for its count first, and the arrays are that long. The merge brought three new maps units, Crews, Pairs and Trails, and every one gained the retaining message family, as every unit with a variable-length table does. GateTable, AssetsTable and DocsTable gained a forward declaration each. Every one of the six only GAINED lines. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: gofmt the retain surface test (#525) A blank line the em-dash pass left behind. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: the surplus element's branch is a switch (#525) golangci's gocritic reads the three-way branch as one, and it is: an element of an unbounded array, an element inside this build's bound, and one past it. `message-surplus-lands-on-zero` anchors on that branch and takes the switch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

…e, six baselines committed (#574, #461, #484) (#677) * CLAUDE.md: the horizon section states the present tree (#574) The table layer is in the language: `table` declares a data type on the evolution-tolerant table wire and docs/SPEC-TABLES.md is its normative page beside docs/SPEC.md, so the source-of-truth bullet names both. The parser carries `table` as a declaration (internal/parser/parser.go, scanner.KwTable); the file said it was refused by name. The protocol layer stays out, as the tree has it: `message` and `object` are reserved words the parser refuses by name and `contexts` is refused at file scope. The frozen projection tokens are stated for what they do today, which is holding every existing unit's id stable. Glenn's scope sentence of 2026-08-25 is kept verbatim, in both places it appears. Closes #574 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs, Makefile: the pages state what the tree carries docs/VERSIONING.md's owed list, in the form its #522 and #525 rows already use. #435 and #434: the C++ reference and internal/tablewire carry the id-table form, the enum kind and the escape kind, and the eight ports are what is owed (#511 to #518). #523's message form: the C++ reference carries it and internal/tablenames claims §11's names, so what is owed is the eight ports and the form's LoadRetain. #523's unbounded array: the C++ reference carries the construct and testdata/wire/tables/list_migrates.bin is pinned. The #523 doc-comment row leaves the list: the language carries the `///` comment and a tag at every line kind, and the doc and tags descriptor columns are done in all nine ports on ROADMAP.md. docs/SPEC-TABLES.md §6.1's block-identity backend status: schema#301 is closed and no open issue carries the row, so the page says so and states why the identity is taken over the block's own text and never over a compiler version. docs/SPEC-TABLES.md §6.6: TableRetain and the three verbs are IN §11's claimed set. internal/tablenames/cpp.go:13 registers TableRetain, internal/check/check.go:3646 carries LoadRetain, MeasureRetain and SaveRetain in tableGeneratedVerbs, and :3692 carries Dart's three member spellings. What still lands with the Dart port is TableRetain in that backend's library-scope registry. docs/SPEC-TABLES.md §6.1's backend-status paragraph gains the form the eight ports actually write, which is the one that preceded §3. docs/SPEC-TABLES.md §15's map and list rows state the tool's honest cook status: compiler/cook.go refuses a map or a []T at Cook and Uncook by name, and cook-check refuses a map slot by name while carrying §7.4's element-array clause. Makefile: the zero-cost gate's parenthetical no longer cites ROUND-LOG.md, which no longer exists in the tree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * testdata: six stale map goldens re-pinned from the reference (#461) `make conformance-pin update-goldens` on a clean checkout of main rewrites six files and nothing else: testdata/golden/tables/maps/{Cells,Chunks,Docs, Runs,Slots,Spans}Table.h. The pins carry neither the retain walk's arm and enum-reference case (kind 15 and kind 30 resolving as framed content) nor the map entry loader's present clamp, both of which the C++ table emitter writes today. This is the shape #461 describes. The pins were written on the branch that became #662, which was cut before #658's retain-unknown walk landed and merged after it; the two diffs do not touch the same lines, so the merge is clean and nothing in CI reads these files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ci: a freshness gate for the pins under testdata/ (#461) ci.yml checks freshness for generated/ and for nothing under testdata/, so a conformance pin or a table golden regenerated against an older main auto-merges clean and every job stays green against a pin nobody wrote. The new `pins` job is the `generated` job's shape one door over: check out, clone ../serialize at the shared SERIALIZE_TAG, run the tree's own re-pinning targets (`conformance-pin` and `update-goldens`) in place, and fail on a dirty testdata/, tracked or untracked. Naming the targets rather than the files means a rule that starts writing a new pin is covered the moment it joins one of them. Its first run is the negative control the issue asks for, taken from live material rather than a planted byte: on main the job goes red and names the six map goldens the previous commit re-pins. With those committed it is green. Closes #461 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: a baseline for the six corpora that hold none (#484) `for d in tables/*/; do ./bin/schema check $d; done` names six units with no tables.baseline: blobdemo, blockdemo, blockhome, scalardemo, streamdemo and vocab9demo. Each is one unit in one directory, which is what the notice's advice needs, so each gets the file the tool writes: ./bin/schema tables-baseline --update --reason "first baseline" tables/<d> The files are generated and never hand-written. Every one carries the history entry the reason opens, so the corpus itself carries the coverage clock (docs/SPEC-TABLES.md §18.4). The loop now prints the notice zero times. What still prints it is the multi-unit case #484 carves out: test/tables holds seven units of different packages in one directory, where "commit one in this directory" is advice a caller cannot take, and bench/corpus holds one more. Neither is a baseline this commit can write. Closes #484 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ci: record the pins job's measured wall (#461) 1m44s on run 34113070433, against 4m27s for big-endian in the same run: the job is inside this file's two-minute rule and is not the binding constraint. The comment names where its budget goes, so the next reader who needs the time back knows the first place to look. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

rowan-claude and others added 17 commits September 6, 2026 20:11

Merge origin/main into retain-oracle-fuzz

b8e4892

Merge branch 'main' into retain-oracle-fuzz

d5dd005

Merge origin/main into retain-oracle-fuzz: the #540 entry retired with …

3c77424

…#657, the goldens regenerated over the Count member (#525) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

rowan-claude merged commit e77093c into main Sep 7, 2026
20 checks passed

rowan-claude deleted the retain-oracle-fuzz branch September 7, 2026 09:34

rowan-claude mentioned this pull request Sep 7, 2026

tables: a blob string on a message body carries the content rule (#632) #671

Merged

This was referenced Sep 7, 2026

contract: the page agrees with the tree, the pins get a freshness gate, six baselines committed (#574, #461, #484) #677

Merged

tables: the message form's LoadRetain, and the manifest's retain surface (#525) #679

Merged

Navigation Menu

Uh oh!

tables: retain-unknown in the Go oracle, and the wire fuzzer's retention leg (#525) - #658

tables: retain-unknown in the Go oracle, and the wire fuzzer's retention leg (#525)#658
rowan-claude merged 17 commits into
mainfrom
retain-oracle-fuzz

rowan-claude commented Sep 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

rowan-claude commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Red first

The oracle's shape

The leg

Two negative controls, one per engine

What the leg made the oracle fix

Page lines retired

Where the page is silent, and what was built

A reading, not a disagreement

From the cold read

Goldens

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

rowan-claude commented Sep 7, 2026 •

edited

Loading