tables: retain-unknown in the C++ reference (#525) - #615
Conversation
…ers (#525) The retain-unknown runtime (docs/SPEC-TABLES.md §6.6): the caller's two stores, the reader-private record, the resolving walk in both directions, the merged trailer over the generated table and the caller's id list, and the retained tail. The three descent helpers now carry a PATH STEP when the retain family is being emitted, and every element loop names its own index. Load, Measure and Save are emitted with the flag false and come out byte for byte what they were: the only golden move so far is TableReport's two new counters.
…efusal (#525) <T>LoadBodyRetain, <T>MeasureBodyRetain and <T>SaveBodyRetainFields for every closure member, <T>LoadRetain, <T>MeasureRetain and <T>SaveRetain on a variable-class root, and the three names refused by name on a fixed-class one. The node table's retaining pair reaches each record through a dispatch the call supplies, so the plain save's numbering gains no store per node. A round trip through the retain family with nothing to retain writes the same bytes the plain save writes.
RT1 is the build that cannot name what RT2 and RT3 wrote. The round trip keeps an unknown field in eight bodies — the root, a nested table, two array elements, a keyed slot, a union arm, a map entry's value and a list element — and RT2 reads every one of them back in the body it came from, which is what the step PAIR exists to prove. The save is idempotent from the first one on, and retention allocates nothing.
…he permuted trailer (#525) A retained record dies with the body occurrence that carried it: the field form of the discard covers a repeated table field, a union whose arm is written again and a map or keyed array written again, and the body form covers a duplicate key inside one occurrence. Neither counter moves. The rows beside it: a buffer one byte short of the last record, an id list one entry short with the save never refused, damage inside sound outer framing that never raises malformed, and the trailer the move permutes.
SaveRetainMessages is refused by name on a form 2 write: a form 2 writer names entries through slots of a vocabulary the compiler settled, and a retained id has neither a slot nor an announced shape. Both refusals are compile errors that name themselves, and each has a control that greps for its sentence.
Eleven suffixes in tableGeneratedVerbs and in section 11's own block, which the page holds itself to: three are the surface the page owes and eight are what carries them. TableRetain in the C++ and Dart unit-scope registries, and Dart's three member spellings, which take that backend's claimed verbs from nine to twelve. The status lines say the present state: the C++ reference carries it, the eight ports and internal/tablewire do not, and the message form's LoadRetain is not built.
…aits for Dart (#525) The zero-cost gate (§2.2) is why: a unit whose tables are all fixed-class carries none of the machinery, so the fixed-class refusal is emitted where retention exists and the checker claims the suffix everywhere either way. The Dart runtime-name registry says what a backend DEFINES and a gate holds it to that, so TableRetain lands there with the Dart port; the three MEMBER spellings are claimed now, which is the half of the page's Dart claim that does not need a definition.
… table record (#525) 73 committed C++ table goldens: 42 byte-identical, 31 grown by pure insertion, and not one line moved or changed. Every pinned WIRE byte string is identical, which is the half that says Load, Measure and Save write what they always wrote.
|
The retain gate inside it, plain and under ASan/UBSan, and both refusals red by name: Every CI check on the pull request passes, big-endian and msvc included. |
Every framed length in the resolved form was a canonical LEB128, so the capture had to know a content's resolved size before writing it, and the only way to know it was to walk that content twice. Nested, that doubled the work at every level: the golden's own text took 0.167 ms at eight levels and 252 ms at twenty, and a two hundred byte hostile field would never have returned. Section 6.6 says the walk cannot take a path a read would not otherwise take, and a cost the file chooses is such a path. A framed length is now a fixed pair of u32. The record is the reader's own storage, so the capture reserves the slot, writes the content and fills the slot in behind it. The save cannot do that, because a wire length is canonical LEB128 and rides first, so it takes one post-order pass instead: measuring leaves each content's wire size in that content's own scratch slot and the emit reads it there, checking the content back against it. The depth cap now counts nested bodies rather than recursion steps, which is what its own comment always claimed, and nothing about the walk's time rests on it any more. A body's terminator in the resolved form was written as a one byte zero and read back as an eight byte id, so any record carrying a nested table body was dropped on the way out. It rides at the width every other reference does. The control is a record forty bodies deep loading under an absolute bound, with a record one past the cap refused by name and the last depth the cap admits still riding. Linear now reads 0.002, 0.005 and 0.008 ms at depths 8, 20 and 40.
TableRetainDiscardField ran at the top of every body-holding field's read, so any second occurrence of the field took every record under it. Two of the four occurrences the page names do not replace whole. An enum-keyed array overwrites the slots the writer carried and leaves the rest standing, so records under untouched slots died. A list whose repeat is inert keeps the value it has, and its records died for a body that changed nothing. Neither moved a counter, so nothing said so. The discard at the field is now taken only where a second occurrence replaces the first whole: a nested table body, which is reset before it is read, and a union, whose tag is written whatever the arm. A map, a list and a bounded array take it at the line their own read commits to replace, past the inert header and past the element kind. An enum-keyed array takes none: its slots are bodies, and a re-put slot discards its own. The control writes banks twice with disjoint slots and reads both sets of records back through the writer, and writes a list twice with the second occurrence inert. Before this both went uncounted: retained stood at two and retain_lost at zero while the records were gone. The reserved-id branch in the unknown arm is deleted. It was dead text in every unit emitted: a body refuses the build version's id and the message vocabulary's before the switch, refuses the node table's in any body but the one whose transport it is, and consumes it in that one, and a fixed-class root gets no retention at all.
…ntences, in progress (#525) The fixer that wrote this died with its stream when the bench changed networks. The two commits before it (one pass each way, the discard scoped) are complete; this one is the rest of the cold read's list mid- work. Measure before trusting. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ut of the page (#525) The merge takes #616, #621 and the arm-defaults rule. Every C++ table golden is re-emitted from the merged compiler rather than hand-merged: the text runtime's clamp now takes the wire's own length, and the fixed-class refusal's two markers ride in every unit. The three em dashes the change had left in the page prose are gone. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…etition cell (#525) The page stated the retain family's factor from the walk that measured each content twice. Re-measured on the one-pass walk at nesting depths 1 and 8, -O3 -DNDEBUG, best of seven: LoadRetain costs 1.5 times the plain Load at depth 1 and 4.1 times at depth 8, and SaveRetain 1.2 and 1.9 times its own. The added cost per record byte is near one nanosecond on each at every reading, and the multiple grows only because the record does: the plain Load skips the field by its outer framing, so the denominator is flat and the multiple is the record's size read against it. The gate's own comments said six excluded classes where the reserved node-table field is excluded by construction and no test can reach it. They say five, and the node record's own comment says fifth. The competition matrix read the retain cell as not built. It is built in the C++ reference, so the honest state is partial: the footnote says what is owed before green, which is tablewire's retention and the fuzzer leg that needs it, the manifest's retain surface, the message form's LoadRetain, and the eight ports. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
) The node-body comment this emitter writes is one a previous build already wrote, and the em-dash pass had rewritten it. That is the one line in the change that a golden could not read as an insertion, and it cost the control its strongest claim: 73 C++ table goldens compared against main, 42 byte-identical, 31 pure insertion, and now zero moved or changed again. The rule is written above the function so the next pass does not take the line again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
) The linear-walk control reads a clock, and the shape gate refuses a clock in a file with no ledger entry. It is the cook open-cost gate's kind of instrument: it measures a cost the FILE can drive rather than a cost the schema has, names no shape, reads no field and reports no rate. Four hits, entered with the reason beside the cook's own. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Not merged at the 2026-09-06 wrap-up: every item of the cold read is landed on the branch (the linear walk with its bound on the shape gate's ledger, the resolution row, the reserved branch deleted, the discard scoped, the page sentences, the measured factor, the regenerate over #616) at twenty green, and the branch went CONFLICTING against main the moment #617 (wstring on the table wire) merged, since both re-pinned the C++ table goldens. The next session merges main into impl-retain, regenerates goldens from the merged compiler rather than hand-merging, confirms twenty green, and lands it. Its honest state on landing is partial, so the cell on #366 stays ❌ with the note until the fuzzer leg and the oracle's retention exist. |
…goldens (#525) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…619) `origin/main` moved under this branch: #615 landed retain-unknown in the C++ reference. `make update-goldens` over the merge, hand-merging nothing. `TextTable.h` is rewritten because it was generated before the merge and now carries #615's emitter whole. The other three maps headers move twenty lines each, and they are the SAME CLASS as the vocabulary rows this branch already moves: `kTableRetainKnownIds` is a unit-wide sorted list of known field ids, 44 -> 50, growing by the six the `Text` unit adds. The value handle's three-way choice is a switch. golangci-lint's gocritic reads the if-else chain as one, and it is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…++ reference (#619) (#626) * tables: a corpus unit whose map values are text, RED on main (#619) `tables/maps/Text.schema` declares the three text value kinds §2.8 names as legal map values: `map[string(8)]string(16)`, `map[uint16]wstring(6)` and `map[int32]bytes(10)`, with an `int32` past them so a stop is visible. Against the emitter as it stands the generated header does not compile. `internal/codegen/cpptable/codecs.go`'s `cppFieldType` has no case for TString, TWString or TBytes, so `mapValueStorageType` answers `/* ? */` and the entry surface reads: inline const /* ? */ * TableEntryFound( const TextNamesEntry * entry ) inline /* ? */ * TableEntryValue( TextNamesEntry * entry ) clang, at the repo's own table flags, over a translation unit that includes `TextTable.h` and nothing else, gives thirteen errors: TextTable.h:4643:24: error: a type specifier is required for all declarations TextTable.h:4643:81: error: cannot initialize return object of type 'const int *' with an rvalue of type 'const char (*)[17]' TextTable.h:4719:80: error: cannot initialize return object of type 'const int *' with an rvalue of type 'const char16_t (*)[7]' TextTable.h:4797:81: error: cannot initialize return object of type 'const int *' with an rvalue of type 'const uint8_t (*)[10]' The entry's own storage is already right — `char value[16 + 1]` beside `int32_t value_length` — so what is missing is the handle, not the layout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: a text map value's handle is the ENTRY, so the header compiles (#619) `string(N)`, `wstring(N)` and `bytes(N)` are legal map values (§2.8, "A VALUE is anything a table field can hold"), and the generated entry already stored each correctly: `char value[N + 1]` beside `int32_t value_length`, the same storage row the field takes anywhere (§7.2). What was missing was the HANDLE. `mapValueStorageType` asked `cppFieldType` for a single type spelling, got `/* ? */` because that function answers types and not pairs, and the entry surface, Insert, Find and IndexFind all spelled it. A pair is two members, so no one member is the value. The handle is therefore the ENTRY, which reaches both: `TableEntryValue` and `TableEntryFound` answer `<Entry> *` and `const <Entry> *`, a caller fills `value` and `value_length`, and `key` stays the map's, which owns the order it carries. Nothing about the entry's layout, its descriptor, its codec or its text form moves: each of the three is an ordinary field of the entry and always was. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: the map gate and the oracle both read the text-value unit (#619) `test_text_values` builds the three text values through the entry handle, out of key order, pins `testdata/wire/tables/map_text.bin`, holds measure == save, loads the region and re-saves it byte for byte, and reads the text form back to the same wire. The wide value is five UTF-16 code units ending in a SURROGATE PAIR, so the transcode runs both ways and a clamp that split the pair would show; the byte values carry a zero byte and a high byte, which text cannot hold. `maps_test.go` gains `{"map_text", "Text"}`, so the compiler's own engine reads the reference's bytes, writes them back byte for byte, renders the text and reads that text to the same bytes. The two engines' texts are the same bytes as well: { "names": { "alpha": "first", "beta": "second" }, "wide": { "5": "Aé中😀" }, "blobs": { "-3": "3q2+7w==", "2": "AAGA/w==" }, "after": 9 } Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: the goldens the text-value unit adds and the vocabulary it moves (#619) NEW, one unit's worth: testdata/golden/tables/maps/TextTable.h, TextTable.cpp testdata/wire/tables/map_text.bin (pinned in the previous commit) MOVED, and every one of them is the UNIT'S SHARED VOCABULARY (§3.3) growing because `mapdemo` gained four tables — `Text` and the three entries its maps generate. The vocabulary is emitted identically into every header of a unit, which is why three unrelated headers move and why no per-type codec does: kCapacity 44 -> 50 the unit's distinct field ids kTableMessageEntriesHere 44 -> 52 the announced vocabulary's size kTableNodeTableFieldSlot 33 -> 40 the reserved node-table id's slot kTableAnnounceBytes 521 -> 615 the announcement, byte for byte BuildVersion 0xe763b68d79174b68 -> 0x28abc7f5927a7539 every `w.put( <slot>, kTableMessageRefBitsHere )` and `node.type_slot` The vocabulary is sorted, so new types shift the slots after them; the two message wires follow from the same fact. `map_conn` IS the announcement, 521 -> 615 bytes. `map_full_message` stays 172 bytes and repacks, because a body's type reference is an index into that vocabulary. `tables/maps/tables.baseline` is ADDITIVE: the `Text` line and the three anonymous entry lines §2.8 keys by the holder's and the field's wire id. The value lines record what the page says a value line records — kind 12 size 16, kind 33 size 6, and kind 14 elem 6 size 10 for the byte buffer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: a negative control on the map value half's reset (#619) `tables-maps-value-reset-negative-control` drops `TableResetMapValue` at its call site in `TableMapPlace`, so a DUPLICATE key no longer resets the value half §2.8 says is reset WHOLE. A TEXT value is where that bites hardest, which is why the control arrives with `Text`: the storage is a PAIR, so the repeat rides the first insert's bytes AND its length. `test_text_values` gains the duplicate row the control needs, and the gate answers by name. RED under the sabotage, three of the four in the new rows: FAIL test/tables/maps_main.cpp:257: again->health = 100, want 0 FAIL test/tables/maps_main.cpp:1453: repeat != NULL && repeat->value_length == 0 FAIL test/tables/maps_main.cpp:1454: repeat != NULL && repeat->value[0] == 0 FAIL test/tables/maps_main.cpp:1468: landed != NULL && landed->value_length == 0 4 map check(s) failed negative control: valuereset turns the MAP GATE red — 4 failures GREEN without it: maps: all checks passed (docs/SPEC-TABLES.md §2.8) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: regenerate over the merge, and the value handle is a switch (#619) `origin/main` moved under this branch: #615 landed retain-unknown in the C++ reference. `make update-goldens` over the merge, hand-merging nothing. `TextTable.h` is rewritten because it was generated before the merge and now carries #615's emitter whole. The other three maps headers move twenty lines each, and they are the SAME CLASS as the vocabulary rows this branch already moves: `kTableRetainKnownIds` is a unit-wide sorted list of known field ids, 44 -> 50, growing by the six the `Text` unit adds. The value handle's three-way choice is a switch. golangci-lint's gocritic reads the if-else chain as one, and it is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * tables: the value-reset control's comment says what its blade does (#619) The control drops the TableResetMapValue call at the placement site; the comment beside its rows said the reset moved to the key half, which no sabotage performs. The Makefile paragraph states the rule in the present tense. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Retain-unknown in the C++ reference, from docs/SPEC-TABLES.md §6.6 as it was
settled by #604 and landed in #608. Closes the last of the original three C++
table-wire rows.
What lands
A second, parallel family of body functions.
Load,MeasureandSaveare not touched: the reference gains
<T>LoadBodyRetain,<T>MeasureBodyRetainand<T>SaveBodyFieldsRetainbeside them, with the PATHthreaded at every child-body descent, and
<T>LoadRetain,<T>MeasureRetainand
<T>SaveRetainon every variable-class root. A caller that does not askinstantiates none of it.
The caller's two stores.
TableRetainis a byte buffer, a capacity, an idlist, an id capacity and what has been used of each. Retention allocates
nothing, in any path, and the test holds that with an
operator newaudit.The retained tail, and the merged trailer. A retained field goes back at
the END of its own body, in the order retained, and in the root body BEFORE the
node-table field. The generated id table is untouched: its capacity, its
overflow rule and its
-1stand exactly as they are, and a retained id takesits entry from the caller's list, the two stores numbered into ONE trailer in
merged first-use order.
The resolving walk, both ways. Every reference in a retained record is the
sixty-four-bit id it names and every length that frames one is recomputed. The
walk reads kind bytes, lengths and references and nothing else, and anything it
cannot frame drops the record, counts one
retain_lostand never raisesmalformedon the plain read.The six excluded classes, one row each, and the two refusals: a fixed-class
root and a form 2 write, each a compile error that names itself.
The proof
Load, Measure and Save are byte-identical
The control compares every committed C++ table golden against what this
compiler emits, and asks whether the old text is a strict SUBSEQUENCE of the
new, so every existing line is present, in order, with nothing moved and
nothing changed:
The 42 that did not move at all are the units with no variable table, which
carry not one byte of retention: the zero-cost gate (§2.2) is why, and the
fixed-class refusal is emitted where retention is rather than everywhere.
And every pinned WIRE byte string is identical.
make update-goldensrewrites
testdata/wire/from the reference's own writers, and after thischange
git diff origin/main -- testdata/wireis empty. That is the half ofthe claim that matters:
Load,MeasureandSavewrite what they alwayswrote, byte for byte, on every instance the corpus pins.
Every gate the page names, with its negative control
retain_lostat one each7is resurrected besideknown = 2, or the discard is countedLoadRetaincompiles, or the refusal is a missing symbolSaveRetainMessagescompiles, or the refusal is a missing symbolThe two refusals are
make tables-retain-fixed-class-negative-controlandmake tables-retain-message-form-negative-control. Each compiles a call andrequires the compile to FAIL with its own sentence in the diagnostic. A
refusal that were a missing symbol would be a linker error with no reason in
it, which is exactly what §11's rule for a surface a class does not carry
refuses.
The cost, measured
The tables read and write benchmark, before and after, one sitting on an M-series
Air,
bench/tables/cpp/legat-O3 -DNDEBUG. Three binaries, rotated so noone of them sits at the same place in the sitting's own drift.
beforeismain at 1472776,
afteris this branch, andctlis this branch with the tworeport counters deleted from the header by hand, which is the only line of this
change the bench unit sees at all.
Round trip, M msg/s, five rounds:
Every difference between the three is inside each binary's OWN spread in that
sitting:
beforealone ranges 0.248 to 0.262, 5.6%. The control is the readingthat settles it, because it differs from
afterby exactly the two countersand nothing else, and it lands 0.8% away in the same band. No measured cost.
The write row is untouched by construction: nothing on the write path changed
in a unit with no variable table, and the bench unit is one.
before write 0.892 / 0.887 / 0.888againstafter write 0.887 / 0.888 / 0.867in thepaired sitting, which drifts downward through the sitting for both.
And the retain family's own cost, stated as a factor, because a caller
choosing it should know. One RT2 wire of 574 bytes carrying 24 retained
records, through RT1,
-O3 -DNDEBUG:The load pays the resolving walk twice per record, once to size the resolved
form and once to write it. The save pays more: it walks the record list at
every body it visits to find the records that body owns, and each record it
places is walked twice again for the same reason the plain save walks a nested
body twice, because a canonical LEB128 length cannot be patched in place. Both
factors scale with the RECORD COUNT the caller admits, which is what the two
capacities bound.
Silences the page left, and what this PR decided
Each of these is a place §6.6 does not reach the case. None was decided
quietly.
excluded class whose conformance row must sit on a POINTERED unit and pin
retain_lostat one, and its own "why not" says a build with no kind17counts it one
unknown, which is a FIXED-class build, and §6.6 deniesthose retention. On a pointered unit the id is the transport the reader
consumes and is never unknown. The rule is implemented, since the retain
family's unknown arm counts one
retain_lostfor any of the three reservedids and captures none of them, and no row is in the corpus, because no wire this
port can read produces one.
LoadRetain's signature. §6.6's sample spellsSceneLoadRetain( scene, region, region_size, wire, wire_size, &retain, &report )with a first argument the sample never declares, and
Loadtakes no valueand returns the root. The reference follows
Load's own shape with the twoextra arguments, and
MeasureRetainandSaveRetainare the sample's own.nesting is the WRITER's and is the one depth on this path a file can drive,
and §6.6 bounds the PATH and says nothing about the walk. A record that nests
past the cap is dropped with one
retain_lost, on the same rule as anythingelse the walk cannot frame.
where zero is the wire's own no-id, the enum's
Noneand the union's emptyarm, and the resolved form spells that as the id zero. And a reference at one of
the three RESERVED ids, which re-emitted into a nested body would be
malformed. Each drops the record and counts one.
array's element index and another for a union's arm ordinal, and this
construct needs both. The element is one step and the arm is the next, and
the two carry the same field ordinal because no body with a field list of
its own sits between them.
LoadRetain,MeasureRetainand
SaveRetain. The reference emits eight more, the body family, the wirepair, the node dispatch and
SaveRetainMessages, and §11's own rule isthat its lists are the checker's claim entire, so all eleven are stated and
claimed.
TableRetainin the DART registry waits for the Dart port. §6.6 asksfor it now. That registry says what a backend DEFINES and a gate holds it to
that, so registering a name Dart does not define would take the gate away.
The three MEMBER spellings, which are field-name refusals with no such gate,
are claimed now and take Dart's claimed verbs from nine to twelve.
tables are all fixed-class carries none of the machinery, which is §2.2's
zero-cost gate, so there the name is free rather than refused. The CHECKER
claims the suffix in every unit either way, which is what protects a
declaration from taking it.
retain_lostat save is counted ONCE, after the save.MeasureRetaintakes no report and counts nothing. The save marks each record it places and
counts what it did not. A body measured twice would otherwise count twice.
directory INDEX the page defines, and the save resolves that index through the
region's directory in one add and compares addresses, so neither side ever
searches a numbering.
What is still owed
Named in §6.6's own status line, which this PR rewrites to the present state
rather than deleting:
internal/tablewire's retention, and the fuzzer's retention leg (§4.2).The leg is not buildable until the oracle retains, and the oracle is a third
reading of §3 written from the page: that is what the leg costs rather than a
detail of it. The fuzzer still runs with retention OFF and its requirement is
the one it has always had.
LoadRetain(§3.3). The form 2 WRITE refuses by nameand is held by a control. The form 2 READ is not built.
retainsurface. The rows live in thereference's own gate for now, because the manifest names no language and a
surface only the reference answers is one the eight ports would each have to
say absent to. The surface lands with the second port.
Refs #525. The page is docs/SPEC-TABLES.md §6.6, settled by #604 and landed as
2fcc2ee in #608.