tables: a clamp answers within its bound, and the text runtime takes the wire's own length - #621
Merged
Merged
Conversation
…the wire's own length Certification's test (ubuntu-latest) job has been red since 28133b4 (#592) on the length negative control, which removes r.room( len ) from a copy of the emitter and requires the wire fuzzer to go red because the leg DIED on the mutant. On Ubuntu it went red on a report difference instead, and the leg's heap was corrupted: munmap_chunk(): invalid pointer. Mutant 629 of seed root_full spells the string field's L as 0xFFFFFFFFFFFFFFFF. #592 added two text helpers that took that length as a signed count, and (int64_t) 0xFFFFFFFFFFFFFFFF is -1. TableUtf8Clamp opens with "if ( length <= bound ) return length", -1 is under every bound, the caller widens the answer back to SIZE_MAX, and the field's memcpy runs at SIZE_MAX. ASan on the sabotaged leg, replaying that mutant, calls it negative-size-param (size=-1) at the field's memcpy. On this Air that copy faults and the leg dies, which is the red the control names; on glibc it writes outside the destination, corrupts the neighbouring chunk header, returns, and the leg reports before free aborts. A LENGTH IS A 64-BIT NUMBER (docs/SPEC-TABLES.md §3), so TableUtf8Valid and TableUtf8Clamp take it as one, and the clamp's answer is never above its bound. The string field read and the string map key drop the casts that narrowed it. The sites passing a reader's own span are non-negative by construction and are unchanged, and the Go oracle takes a slice and cannot express the defect. Nothing changes in a clean build: room( len ) already bounded every length these helpers see, which is what the negative control exists to prove. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…oduces The control removes `r.room( len )` and required the wire fuzzer to go red because the leg DIED on the mutant. With the clamp answering within its bound no leg dies there: the first mutant that exposes the loss is the length pass's 0xFFFFFFFFFFFFFFFF on root_full, mutant 629, where the sabotaged leg takes the payload over bytes the mutant never carried, steps its cursor by a length the body never had, and reports a kind mismatch the oracle does not. That red is an in-bounds deterministic computation, the same on every allocator, where the death it replaces was a wrapped memcpy size that faulted on one libc and corrupted the heap on another. The named reason becomes the report difference, and the comment says what the sabotage now does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Sep 6, 2026
gafferongames
added a commit
that referenced
this pull request
Sep 6, 2026
…ut of the page (#525) The merge takes #616, #621 and the arm-defaults rule. Every C++ table golden is re-emitted from the merged compiler rather than hand-merged: the text runtime's clamp now takes the wire's own length, and the fixed-class refusal's two markers ride in every unit. The three em dashes the change had left in the page prose are gone. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
gafferongames
added a commit
that referenced
this pull request
Sep 6, 2026
The three that landed while this branch sat all reach package wide: #616's arm-defaults rule moves its protocol id once (0xcd3b46b8e6e1a6d7 to 0x4b9b89c718af5d5e) now that Caption.schema is in the unit, #621 takes the text runtime's lengths as the wire's own 64-bit numbers, and #612's unit registry emits WideView.h and WideView.cpp beside the rest. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
gafferongames
added a commit
that referenced
this pull request
Sep 6, 2026
* tables: kind 33 in the C++ reference's storage, codec, cook and text form The id-table wire's wide text (docs/SPEC-TABLES.md §3, schema#522): kind 33 maps onto wstring(N), the C++ table emitter carries the record's char16_t[N + 1] and its int32 used length in code units, the four codec sites, the arm payload, the cook's two-byte scalars, the §16.2 text row and the skip rule. The §11 closure refusal is deleted with its tests inverted to acceptance. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * tables: kind 33 in the Go oracle, the text form and the cook internal/tablewire encodes and decodes wide text on both forms, internal/ tabletext transcodes it at the JSON boundary through one shared escape grammar, and internal/tablecook writes, checks and reads the char16_t[N + 1] piece in both byte orders. The message form's half is terminal, which is the one thing that differs from the id-table wire's recovery. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * wide: the table unit, its gate and five negative controls examples-wide/Caption.schema declares every kind 33 site the wire has — a table's own field, a type a table reaches, a union arm and an element — and test/wide/table_main.cpp states one row per sentence of docs/SPEC-TABLES.md §3 and §4 over them, named from the serialize corpus with -table on them. Each control removes one rule from a copy of the table emitter and names the row that goes red. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * conformance and the wire fuzzer carry kind 33 The wide unit joins the tables corpus as its own directory, five instances pin the kind at every site it has, and the fuzzer gains the ill-formed wide-text and odd-L passes over them. Its control removes the pairing rule from the emitted runtime and the leg then stores what the oracle refuses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: wide text rides both wires, and the closure refusal is deleted SPEC-TABLES §11's wstring clause goes with the kind that made it necessary, SPEC.md §4.12's backend status says which wires C++ carries and drops the clause that contradicted the sentence after it, and USAGE, VERSIONING, the FAQ and the roadmap's cell follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * tables: the JSON walker's blank lines, after the wide text move Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * generated: the bench table unit, after the walker's blank lines Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * goldens: the emitted runtime and the walker, re-pinned The wide-text helpers and the shared escape grammar move every C++ Table source, and the wide unit's build-version golden gains kind 33's record layout. NO WIRE GOLDEN MOVED: adding a kind changes no existing byte, which is the closed set doing its job. Wide text's per-target refusal moves ahead of the form refusals, because a target that laid out no member cannot emit the field under any form. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * wide: the gate runs its rules before its pins, and the lint findings A control that removes a rule must quote the rule's own row rather than a golden mismatch downstream of it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * check: the scaffolding the deleted closure refusal left behind closureEdge, fieldPos and blockFieldPos existed to name the edge and the line of a wstring refusal that no longer exists, and the closure walk's reachedBy map with them. Removal rides the landing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * wide: name the helper the comment describes Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * RED WIP: the cook unit-byte assertion and the read's small fixes, in progress (#522) The fixer died with its stream on a network change. Measure before trusting. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * wide: the cook's unit bytes asserted in both orders, terminator included (#522) The difference check alone is satisfied by the swapped `label_length` and `seq`, so a cook that wrote the units as a run of bytes passed it. All three units of the `char16_t[N + 1]` are now read byte for byte in both orders. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * goldens: the wide unit regenerated from the merged compiler (#522) The three that landed while this branch sat all reach package wide: #616's arm-defaults rule moves its protocol id once (0xcd3b46b8e6e1a6d7 to 0x4b9b89c718af5d5e) now that Caption.schema is in the unit, #621 takes the text runtime's lengths as the wire's own 64-bit numbers, and #612's unit registry emits WideView.h and WideView.cpp beside the rest. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * spec: the cook's per-unit width covers the terminator, and one reflow (#522) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
rowan-claude
added a commit
that referenced
this pull request
Sep 6, 2026
* retain: the C++ runtime and the path threaded through the codec emitters (#525) The retain-unknown runtime (docs/SPEC-TABLES.md §6.6): the caller's two stores, the reader-private record, the resolving walk in both directions, the merged trailer over the generated table and the caller's id list, and the retained tail. The three descent helpers now carry a PATH STEP when the retain family is being emitted, and every element loop names its own index. Load, Measure and Save are emitted with the flag false and come out byte for byte what they were: the only golden move so far is TableReport's two new counters. * retain: emit the retain family, the three verbs and the fixed-class refusal (#525) <T>LoadBodyRetain, <T>MeasureBodyRetain and <T>SaveBodyRetainFields for every closure member, <T>LoadRetain, <T>MeasureRetain and <T>SaveRetain on a variable-class root, and the three names refused by name on a fixed-class one. The node table's retaining pair reaches each record through a dispatch the call supplies, so the plain save's numbering gains no store per node. A round trip through the retain family with nothing to retain writes the same bytes the plain save writes. * retain: the RT1/RT2/RT3 corpus and the round trip at every depth (#525) RT1 is the build that cannot name what RT2 and RT3 wrote. The round trip keeps an unknown field in eight bodies — the root, a nested table, two array elements, a keyed slot, a union arm, a map entry's value and a list element — and RT2 reads every one of them back in the body it came from, which is what the step PAIR exists to prove. The save is idempotent from the first one on, and retention allocates nothing. * retain: record lifetime, the two capacities, the walk's verdict and the permuted trailer (#525) A retained record dies with the body occurrence that carried it: the field form of the discard covers a repeated table field, a union whose arm is written again and a map or keyed array written again, and the body form covers a duplicate key inside one occurrence. Neither counter moves. The rows beside it: a buffer one byte short of the last record, an id list one entry short with the save never refused, damage inside sound outer framing that never raises malformed, and the trailer the move permutes. * retain: the message-form refusal, and the gate in make (#525) SaveRetainMessages is refused by name on a form 2 write: a form 2 writer names entries through slots of a vocabulary the compiler settled, and a retained id has neither a slot nor an announced shape. Both refusals are compile errors that name themselves, and each has a control that greps for its sentence. * retain: claim the names, and the pages say what is built (#525) Eleven suffixes in tableGeneratedVerbs and in section 11's own block, which the page holds itself to: three are the surface the page owes and eight are what carries them. TableRetain in the C++ and Dart unit-scope registries, and Dart's three member spellings, which take that backend's claimed verbs from nine to twelve. The status lines say the present state: the C++ reference carries it, the eight ports and internal/tablewire do not, and the message form's LoadRetain is not built. * retain: the refusal lands where retention does, and Dart's registry waits for Dart (#525) The zero-cost gate (§2.2) is why: a unit whose tables are all fixed-class carries none of the machinery, so the fixed-class refusal is emitted where retention exists and the checker claims the suffix everywhere either way. The Dart runtime-name registry says what a backend DEFINES and a gate holds it to that, so TableRetain lands there with the Dart port; the three MEMBER spellings are claimed now, which is the half of the page's Dart claim that does not need a definition. * retain: plain sentences in the prose this change adds (#525) * retain: re-pin the C++ table goldens, and the void for a root with no table record (#525) 73 committed C++ table goldens: 42 byte-identical, 31 grown by pure insertion, and not one line moved or changed. Every pinned WIRE byte string is identical, which is the half that says Load, Measure and Save write what they always wrote. * retain: slices.Sort for the unit's id set (#525) * retain: the resolving walk is one pass each way (#525) Every framed length in the resolved form was a canonical LEB128, so the capture had to know a content's resolved size before writing it, and the only way to know it was to walk that content twice. Nested, that doubled the work at every level: the golden's own text took 0.167 ms at eight levels and 252 ms at twenty, and a two hundred byte hostile field would never have returned. Section 6.6 says the walk cannot take a path a read would not otherwise take, and a cost the file chooses is such a path. A framed length is now a fixed pair of u32. The record is the reader's own storage, so the capture reserves the slot, writes the content and fills the slot in behind it. The save cannot do that, because a wire length is canonical LEB128 and rides first, so it takes one post-order pass instead: measuring leaves each content's wire size in that content's own scratch slot and the emit reads it there, checking the content back against it. The depth cap now counts nested bodies rather than recursion steps, which is what its own comment always claimed, and nothing about the walk's time rests on it any more. A body's terminator in the resolved form was written as a one byte zero and read back as an eight byte id, so any record carrying a nested table body was dropped on the way out. It rides at the width every other reference does. The control is a record forty bodies deep loading under an absolute bound, with a record one past the cap refused by name and the last depth the cap admits still riding. Linear now reads 0.002, 0.005 and 0.008 ms at depths 8, 20 and 40. * retain: the discard lands where the read replaces (#525) TableRetainDiscardField ran at the top of every body-holding field's read, so any second occurrence of the field took every record under it. Two of the four occurrences the page names do not replace whole. An enum-keyed array overwrites the slots the writer carried and leaves the rest standing, so records under untouched slots died. A list whose repeat is inert keeps the value it has, and its records died for a body that changed nothing. Neither moved a counter, so nothing said so. The discard at the field is now taken only where a second occurrence replaces the first whole: a nested table body, which is reset before it is read, and a union, whose tag is written whatever the arm. A map, a list and a bounded array take it at the line their own read commits to replace, past the inert header and past the element kind. An enum-keyed array takes none: its slots are bodies, and a re-put slot discards its own. The control writes banks twice with disjoint slots and reads both sets of records back through the writer, and writes a list twice with the second occurrence inert. Before this both went uncounted: retained stood at two and retain_lost at zero while the records were gone. The reserved-id branch in the unknown arm is deleted. It was dead text in every unit emitted: a body refuses the build version's id and the message vocabulary's before the switch, refuses the node table's in any body but the one whose transport it is, and consumes it in that one, and a fixed-class root gets no retention at all. * RED WIP: the resolution row, the reserved branch deleted, the page sentences, in progress (#525) The fixer that wrote this died with its stream when the bench changed networks. The two commits before it (one pass each way, the discard scoped) are complete; this one is the rest of the cold read's list mid- work. Measure before trusting. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * retain: regenerate from the merged compiler, and the last em dashes out of the page (#525) The merge takes #616, #621 and the arm-defaults rule. Every C++ table golden is re-emitted from the merged compiler rather than hand-merged: the text runtime's clamp now takes the wire's own length, and the fixed-class refusal's two markers ride in every unit. The three em dashes the change had left in the page prose are gone. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * retain: the measured factor, the five reachable classes, and the competition cell (#525) The page stated the retain family's factor from the walk that measured each content twice. Re-measured on the one-pass walk at nesting depths 1 and 8, -O3 -DNDEBUG, best of seven: LoadRetain costs 1.5 times the plain Load at depth 1 and 4.1 times at depth 8, and SaveRetain 1.2 and 1.9 times its own. The added cost per record byte is near one nanosecond on each at every reading, and the multiple grows only because the record does: the plain Load skips the field by its outer framing, so the denominator is flat and the multiple is the record's size read against it. The gate's own comments said six excluded classes where the reserved node-table field is excluded by construction and no test can reach it. They say five, and the node record's own comment says fifth. The competition matrix read the retain cell as not built. It is built in the C++ reference, so the honest state is partial: the footnote says what is owed before green, which is tablewire's retention and the fuzzer leg that needs it, the manifest's retain surface, the message form's LoadRetain, and the eight ports. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * retain: the plain family's own comment stays spelling for spelling (#525) The node-body comment this emitter writes is one a previous build already wrote, and the em-dash pass had rewritten it. That is the one line in the change that a golden could not read as an insertion, and it cost the control its strongest claim: 73 C++ table goldens compared against main, 42 byte-identical, 31 pure insertion, and now zero moved or changed again. The rule is written above the function so the next pass does not take the line again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * retain: the walk's linearity bound goes on the shape gate's ledger (#525) The linear-walk control reads a clock, and the shape gate refuses a clock in a file with no ledger entry. It is the cook open-cost gate's kind of instrument: it measures a cost the FILE can drive rather than a cost the schema has, names no shape, reads no field and reports no rate. Four hits, entered with the reason beside the cook's own. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Rowan <rowan@mas-bandwidth.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Certification's
test (ubuntu-latest)job has been red on every main commit since 28133b4 (#592, merged 2026-09-06 06:51Z). af66617, the commit before it, was green. Every merge since has landed on that red. This PR is the fix.The red is
make tables-wire-fuzz-length-negative-control, which removesr.room( len )from a copy of the C++ emitter and requires the wire fuzzer to go red because the leg died on the mutant. On Ubuntu it went red for another reason, and the leg's heap was corrupted:The cause
Mutant 629 spells the string field's
Las0xFFFFFFFFFFFFFFFF. #592 added two text helpers that take that length as a signed count:(int64_t) 0xFFFFFFFFFFFFFFFFis-1, andTableUtf8Clampopens withif ( length <= bound ) { return length; }.-1is under every bound, so the clamp hands back-1, the caller widens it toSIZE_MAX, and the field'smemcpyruns atSIZE_MAX. Replaying that exact mutant against a sanitized build of the sabotaged leg:TablesTable.h:6084is the string field'smemcpy, and the region is the mutant's own wire buffer.Why macOS stayed green. On this Air the plain sabotaged leg faults inside that copy, dies on mutant 629, and the control gets the red it names. On glibc/x86-64 the same copy does not fault: it writes outside the destination, corrupts the neighbouring chunk's header, and returns. The leg then writes a full report for mutant 629 (
kind_mismatch=1, clamped=1, what a reader that walked off its buffer counts), the harness sees a report difference instead of a death, and the process aborts later infreewithmunmap_chunk(): invalid pointer. One wrapped size, a loud death on one allocator and a silent out-of-bounds write on the other. The macOS green was the bug crashing conveniently.The ordinary wire fuzz stays green either way, because
r.room( len )boundslenbefore the helpers ever see it. That is exactly what the negative control exists to prove, and it stopped being provable.The fix
A length is a 64-bit number (SPEC-TABLES.md §3), so the text runtime takes it as one.
TableUtf8ValidandTableUtf8Clampnow takeuint64_t length, and the clamp's answer is never above its bound:The two call sites that narrowed a wire length, the string field read in
codecs.goand the string map key inmaps.go, drop their(int64_t)casts. The sites that pass a reader's own span (arms.go,pointers.go,json.go) are non-negative by construction and are unchanged. Goldens are regenerated. The Go oracle'stextBoundarytakes a slice and cannot express the defect, so it is unchanged.Nothing changes in a clean build:
room( len )already bounded every length these helpers see.The red-first test
test_text_clamp_is_boundedintest/tables/main.cppruns the field read's own three lines over the length the fuzzer's length pass plants and asserts that the size reachingmemcpyis within the bound. On the clean tree, with the fix reverted and the test kept:It runs in both
schema_test_tablesandschema_test_tables_asan.The control's named reason
With the clamp answering within its bound, no leg dies on mutant 629 on any platform, so the control's old name no longer describes what the sabotage does. It never did on Ubuntu, and on macOS it only held because the wrapped
memcpyfaulted. The second commit renames it to what the missing fit check now produces:Without
room( len )the leg believes a forgedL: it takes the payload over bytes the mutant never carried and steps its cursor by a length the body never had, so the fields after it decode out of bytes that are not their own and it reports a kind mismatch the oracle does not. That read is in bounds and deterministic, so the control now goes red the same way on every allocator, where the death it replaces depended on which libc ran it.Verification
make tables-wire-fuzz-length-negative-control N=0passes on macOS, andmake testis green. Note thatcertify.ymlruns only on pushes to main, sotest (ubuntu-latest)is not among this PR's checks.