tables: the message form's LoadRetain, and the manifest's retain surface (#525)

[rowan-claude]

Two of the three things §6.6 still called owed on the C++ column: the MESSAGE
form's LoadRetain (§3.3) and the conformance manifest's retain surface. The
eight ports are what is left, and VERSIONING's #525 line now says only that.

Red first

The pinned batch and the two rows that read it landed before the function did.
RT2 writes a form-2 batch of two bodies over the same values the file rows read:
the fields at three depths that are retained, the whole parcel table the
resolving walk reads, and every class this form can put on the unknown arm, a
field of kind 17, an array whose element kind is 17, a table whose payload meets
a 17 three bodies down, an unknown enum variant reference, an unknown union arm
id and an unknown keyed-array slot. Both engines were RED (b911247):

test/tables/retain_main.cpp:1023:20: error: no member named
'NodeLoadRetainMessages' in namespace 'tblrt1'; did you mean
'NodeSaveRetainMessages'?

internal/tablewire/retain_test.go:716:29: undefined:
tablewire.DecodeRetainMessages

The compiler's own suggestion is the shape of the change: the write half was
already there, refusing by name, and the read half was not built.

The function, in both engines

LoadRetain reads a form-2 body as it reads a file's, the resolving walk
replacing every reference with the id it names against the CONNECTION'S
VOCABULARY instead of a trailer. SaveRetain writing form 2 still refuses by
name, so the round trip a caller has is form 2 in and form 1 out.

The record is the FILE form's own, to the byte, because a retained record
carries the field's bytes with every reference resolved so that re-emitting it
into any id table is correct, and the table it is re-emitted into is a file's.
So the capture is a TRANSCODE as well as a resolve: a bitpacked value is read at
the width its announced shape states and written at the width the file form
spells. From there it is the same record, laid down in the same slots and read
back by the same emit walk. Neither engine carries a second copy of the emit
side; the reference's message capture writes through TableRetainInSlot,
TableRetainInLeb and TableRetainInId, and the oracle's writes through the
same resolver the file capture fills.

The skip runs first and the capture second, over the same bits. The plain
read's verdict on an unknown entry is the SKIP's, whether or not this build
retains, and a capture that lands anywhere but the skip's own end drops the
record. Two flat passes over one record's bits, never a walk that doubles at
every level.

Reference: <T>LoadRetainMessages, <T>LoadMessageBodyIntoRetain,
<T>LoadMessageBodyRetain and <T>NodeMessageBodyRetain, emitted by the retain
pass with the path threaded at every child-body descent, the four excluded
classes counted at their own arms and TableMessageRetainCapture at the unknown
one. Oracle: DecodeRetainMessages and internal/tablewire/retainmessage.go.

Both read the pinned batch and write the same two files, 442 and 93 bytes, byte
for byte, at retained=10 retain_lost=6 unknown=17.

The manifest's retain surface

MANIFEST.txt gains retain and retain-message lines and the nine rows §6.6
owes: the round trip at depth, the truncated record, the id list one short, the
five excluded classes a wire can carry to the unknown arm, and the message
form's tail. The harness gains two surfaces, retain for the COUNTERS and
retain-save for the BYTES, split for the reason the block surfaces are.

The two capacities travel as RULES and not as numbers. A record's byte cost
is the port's own, so short says ONE BYTE SHORT OF THE LAST RECORD and a leg
resolves it against its own full load; the id list's capacity is a COUNT and
rides as one.

cpp   retain pass 9/9   retain-save pass 3/3
c     retain absent     retain-save absent
go    retain absent     retain-save absent
rust  retain absent     retain-save absent

The eight ports refuse the surface under the harness's own absent-is-not-failure
rule: a port carries no retention at all, so the three verbs are a missing
FEATURE and not a failing test, and §6.6's backend status says so.

The oracle claims the same nine rows in the harness's own test, against the
manifest's counters and the pinned saves.

The controls

make tables-message-form-retain-negative-control takes the message path's call
into the resolving walk out of the emitter through an overlay, regenerates the
RT set and requires the retain gate to go red on the pinned batch's own row:

FAIL line 1042: report.retained == 10

It cuts exactly one line: every file-form row of the gate stays green, so the
blade is the message path and nothing else. Green half: the same gate built from
the unsabotaged compiler is retain: ok, plain and under ASan/UBSan.

The silences, listed

1. The page spells no name for the form-2 retaining read. §6.6 and §3.3 both
   say "the message form's LoadRetain", which is the FILE form's verb, and
   §11's claimed set carries SaveRetainMessages, the write refusal, with no
   read spelling beside it. Reduce-cases: <T>LoadRetainMessages beside
   <T>LoadMessages, plural because the primitive is a batch. It is EMITTED, so
   §11's own rule claims it: tableGeneratedVerbs and the page's fenced block
   both gain it, 52 suffixes become 53 and 68 become 69.
2. The page does not say what a BATCH's retention buffer is. §6.6 says a
   buffer belongs to one loaded region and §3.3 says a batch is ONE region with
   M bodies, each carrying its own node directory. Reduce-cases: one retention
   buffer PER BODY, an array parallel to the caller's array of roots. It keeps a
   record's first step an index into the directory of the body it came from,
   which is what §6.6 says a first step is, and it leaves SaveRetain's
   accounting the file form's unchanged. A single pooled buffer would make every
   save count the other bodies' records as lost.
3. The fuzzer's retention leg keeps its file-form roster, and I did not move
   it. §4.2 states the arm as "the VARIABLE-CLASS FILE ROOTS and nothing else,
   because a fixed-class root's LoadRetain is refused by name and a form-2
   SaveRetain refuses by name". The second reason still stands after this
   change: the leg's third requirement compares the bytes a leg SAVES, and there
   is no retaining form-2 save to compare. Adding message roots would mean
   rewriting that sentence to a cross-form comparison, which is a third thing
   the page does not ask for here. The line prints the seeds it leaves out
   already. Flagged rather than done.
4. unknown is 17 on the message row where the two retention counters sum to
   16. The seventeenth is the plain read's own and predates this change: a bit
   stream has to be walked past, so a DISCARDED SLOT's element body is decoded
   into a sink where a file's reader steps over it by its length, and the field
   inside it this build cannot name counts there. The test asserts the plain
   read gives the same 17, which is what makes it the read's and not retention's.
   Retention keeps nothing under a discarded slot: the slot is the exclusion and
   everything below it went with that one retain_lost.

Two things found in passing

• Six goldens under testdata/golden/tables/maps were already stale on
  main. Their committed text predates tables: retain-unknown in the Go oracle, and the wire fuzzer's retention leg (#525) #658's retain runtime (case 15: case 30: in the capture and the emit) and The message form's content rule and clamp for string(N) are stated in §3.3 and carried by neither engine #620's message text rule. Verified by
  generating from a bin/schema built at this branch's base: CellsTable.h,
  ChunksTable.h, DocsTable.h, RunsTable.h, SlotsTable.h, SpansTable.h.
  This repin brings them level; the other eighteen moved on this change alone,
  and every one of the twenty-seven only GAINED lines.
• TestTheTailIsUnconditional did not skip a map's generated entry. The
  vocabulary tail never names one (§2.8, §20.2) and the test's own table set
  counted it, which no connection unit had exercised because none carried a map
  until RT2 did.

The cell

ROADMAP's retain-unknown cell moves to ✅ for cpp, 339 cells become 340. Every
gate §6.6 names for a backend exists in the reference after this change: the
three verbs on every variable-class root, the refusal by name on every
fixed-class one, the message form's read, the second family of body functions,
every HELD BY TEST row in test/tables/retain_main.cpp, and the nine
conformance rows the harness answers 9/9 and 3/3. The eight port columns stay ❌
and VERSIONING's #525 line says the ports are what remains.

Goldens

make update-goldens moved twenty-seven C++ table headers, every one of a unit
that declares a variable-length table: they gained the retaining message family
(LoadMessageBodyRetain per member, NodeMessageBodyRetain,
LoadMessageBodyIntoRetain and LoadRetainMessages per variable root) and the
message-retain runtime beside the file one. Six of them were stale on main as
above. No WIRE golden moved: retention is additive and the plain codec is
untouched to the byte. Three new pins: retain_message.bin, its two saves, and
retain_conn.bin, RT2's announcement, which the manifest's connection line
names.

make test

Ran to a real red, mine, and fixed: message-clamp-drops-surplus and
message-surplus-lands-on-zero both anchor on the array element loop of the
message decoder, which gained the mine flag that says whether an element is
this build's. The tool caught it by design:

sabotage message-clamp-drops-surplus: edit 0 matched 0 times, want exactly 1.
The emitter moved away from the anchor, so this negative control no longer
breaks what it names.

Both re-anchored, both red on their own rows again, and
make tables-message-form-negative-control is green end to end (rc=0). Every
message sabotage was re-scanned for its anchor after that; all thirty-three
match exactly once.

Green on the way there, in the same run: the whole packet and table suite,
tables-retain plain and under ASan/UBSan, the wire fuzzer at
0 divergences on both the plain and the sanitized leg, and the retention leg:

wire-fuzz retain: seed 24845619678, 23 seeds over 7 roots, 8 seeds absent
(roots the leg has no codec for), 116 seeds outside the arm, 54521 enumerated
+ 20000 random = 74521 mutants, 0 divergences, 7.0 s, 10629 mutants/s

make test cannot reach its end on this machine: node, dart, elixir and dotnet
are absent, and the run stops at the C# conformance leg. SCHEMA_SKIP_LEGS is
in #660 and not merged. Everything before that stop is green.

Toolchain

Absent here and untested by me: node, dart, elixir, dotnet. Present and run:
Go 1.27, clang, cargo, java.