Skip to content

tables: a blob string on a message body carries the content rule (#632) - #671

Merged
rowan-claude merged 7 commits into
mainfrom
fix-632-message-blob-utf8
Sep 7, 2026
Merged

tables: a blob string on a message body carries the content rule (#632)#671
rowan-claude merged 7 commits into
mainfrom
fix-632-message-blob-utf8

Conversation

@rowan-claude

Copy link
Copy Markdown
Collaborator

Closes #632

docs/SPEC-TABLES.md §3.1 states kinds 12 and 33's content rule MET AT A NODE, "A TEXT blob's CONTENT is refused on the same terms", and §3.3 says a form-2 body's content rules are §3's, unchanged in what they reject. A *string blob record carried by a message body was read by BOTH engines without that check, where both apply it at the file site. The two engines were symmetric, so no vector separated them and the corpus could not see it.

Red first

Two gates, one an engine, over blobdemo's Catalog, whose numbering reaches a *string blob through note and a *bytes blob through thumb. test/tables/message_blob_main.cpp is the C++ reference's and TestAStringBlobRecordOnAMessageBodyCarriesTheContentRule is the oracle's. At the first commit both are RED on four rows each: the truncated sequence, the interior zero byte, the overlong encoding and the lead byte 0xFF all load with a silent report.

message blob FAILED: a truncated UTF-8 sequence in a *string blob record is damage, and the batch ends there
message blob FAILED: a zero byte among a *string blob's bytes is damage, and the batch ends there
message blob FAILED: an overlong encoding in a *string blob record is damage, and the batch ends there
message blob FAILED: a lead byte UTF-8 never spells is damage, and the batch ends there

The two silent-report rows stand at that commit and never move: a well-formed blob loads clean, and so do the same ill-formed bytes under the reserved bytes id, because a *bytes blob is bytes and never text.

The fix, one function at the message site

  • The C++ emitter, messagevariable.go's PASS TWO, where the record's bytes are already in hand. A blob record aligns before its bytes, so the same pointer goes to the runtime's TableUtf8Valid and to the memcpy under it, and the line is emitted under rootReachesStringBlob, the guard pointers.go emits the file form's under. The check does NOT ride in MessageRecordScan, which is the framing walk LoadMeasure shares: ill-formed content is sizeable, so a measure still answers the region the framing commands and only the decode refuses.
  • The Go oracle, messagedecode.go's placement loop, calling textValid under the TString arm alone, which is decodenodes.go's line for the file form.

What differs from the file form is only the recovery: where a file counts the record malformed and reads on with every slot naming it null, a batch ends there. One malformed, the bodies before it stand, nothing after is read (§3.3).

The controls

One blade an engine, each matching exactly one line, each drawn twice.

Against the PAGE: message-blob-accepts-ill-formed in MESSAGE_FORM_CONTROLS, and message-emitter-blob-accepts-ill-formed in the new tables-message-form-blob-negative-control, whose first step is the gate's own true run.

Against EACH OTHER, on two pinned vectors: message_blob_ill_formed_text is blob_str8 re-encoded as a batch of one with the fourth byte of note's payload replaced by 0xFF, and message_blob_zero_byte is that wire with 0x00 there instead. tables-wire-fuzz-message-blob-oracle-negative-control and its leg twin replay both alone and require red on each, with the reports mirrored:

the leg says 0,0,0,0,0,true,read, the oracle says 0,0,0,0,0,false,read
the leg says 0,0,0,0,0,false,read, the oracle says 0,0,0,0,0,true,read

The fuzzer's leg grew one root for them: blobdemo's Catalog is the only root on the roster that can PLACE a text blob record. AssetsTable.cpp was already in CONFORMANCE_SOURCES, so it costs one include and one MESSAGE_VARIABLE row, and the corpus pass runs 45 roots rather than 44.

The page

No new rule and no sentence moved: §3.1 and §3.3 already state it. One HELD BY TEST bullet in §3.3 names the six rows and the three ways a leg goes red.

Results

  • go test ./... green
  • make tables-wire-fuzz N=20000: 137 seeds over 45 roots, 123311 enumerated + 20000 random = 143311 mutants, 0 divergences, both the plain and the ASan leg
  • make tables-message-form-negative-control green, 49 blades including the new one
  • make tables-vocab-schema green
  • make tables-block-zero-cost: 117 Table sources byte-identical to their pins
  • the four new controls green
  • make test stops at tables-json-list-walk, which is RED ON MAIN before this branch and unrelated to it: all ten tables/maps units now emit the JSON list half and the gate still calls them list-free. Proven by regenerating tables/maps with main's own emitter through a go build -overlay.

Goldens

Four lines are this branch's, one per emitted message reader whose root reaches a *string blob: blobs/AssetsTable.h twice, maps/DocsTable.h once, arms/GateTable.h once. NO WIRE GOLDEN MOVED. The other re-pinned files were already stale on main from #658 and #662, which moved the emitter without re-pinning tables/maps; the re-pin commit names them.

🤖 Generated with Claude Code

rowan-claude and others added 7 commits September 7, 2026 05:49
… first (#632)

docs/SPEC-TABLES.md §3.1 states kinds 12 and 33's content rule MET AT A NODE:
"A TEXT blob's CONTENT is refused on the same terms", so a *string blob whose
bytes are not well-formed UTF-8, or which carries a zero byte, is damage. §3.3
says a form-2 body's content rules are §3's, unchanged in what they reject,
and that what differs is only the recovery, which a bit stream does not have.
Neither engine carries it at the message site, where both carry it at the file
site (decodenodes.go and pointers.go).

The two gates are the page's rows over blobdemo's Catalog, whose numbering
reaches a *string blob through note and a *bytes blob through thumb.
test/tables/message_blob_main.cpp is the C++ reference's and
TestAStringBlobRecordOnAMessageBodyCarriesTheContentRule is the oracle's.
Both are RED at this commit: the truncated sequence, the zero byte, the
overlong encoding and the lead byte 0xFF all load with a silent report.

The last row of each is a *bytes blob carrying the same bytes, which must stay
silent: a *bytes blob is bytes and never text, so the rule is the string id's
alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ule (#632)

Both engines read a form-2 blob record's bytes through the SAME function the
FILE form reads one with, so there is one rule and no second copy.

The C++ emitter: messagevariable.go's PASS TWO, where the record's bytes are
already in hand. The align a blob record spends before its bytes leaves the
span on a byte boundary, so the same pointer goes to the runtime's
TableUtf8Valid and to the memcpy below it, and the line is emitted under
rootReachesStringBlob, the same guard pointers.go emits the file form's under.
The check does NOT ride in MessageRecordScan, which is the framing walk
LoadMeasure shares: ill-formed content is sizeable, so a measure still answers
the region the framing commands and only the decode refuses.

The Go oracle: messagedecode.go's placement loop calls textValid on the
record's bytes under the TString arm alone, which is decodenodes.go's line for
the file form.

What differs from the file form is only the recovery, which a bit stream does
not have: where a file counts the record malformed and reads on with every slot
naming it null, a batch ends there. One malformed counts, the bodies before it
stand, and nothing after is read (§3.3).

A *bytes blob keeps no rule, because it is bytes and never text (§3.1), and
both gates carry the row that says so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… main left stale (#632)

FOUR LINES ARE THIS BRANCH'S, one per emitted message reader whose root's
numbering reaches a *string blob: blobs/AssetsTable.h twice, for Asset and for
Catalog; maps/DocsTable.h once, for a map whose VALUE is a text buffer; and
arms/GateTable.h once, for a blob reached only through a union arm. Each is the
same line, the record's bytes handed to TableUtf8Valid. NO WIRE GOLDEN MOVED:
the write side is untouched, and a read that refuses damage changes no byte a
writer produces.

THE REST WAS ALREADY STALE ON MAIN and `make tables-block-zero-cost` was red
before this branch: e77093c (#658) added the retain walk's `case 15: case 30:`
and e123f1b (#662) moved the map entry readers' text path, and neither re-pinned
tables/maps. maps/ChunksTable.h, RunsTable.h, SlotsTable.h and SpansTable.h
carry nothing of this branch's at all, and CellsTable.h carries none of its four
lines either. The gate now compares 117 Table sources byte-identical to their
pins.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…#632)

ONE BLADE AN ENGINE, each removing exactly one line and each naming what it
turns red. message-blob-accepts-ill-formed drops textValid from the oracle's
message placement loop; message-emitter-blob-accepts-ill-formed drops the
emitted TableUtf8Valid call from messagevariable.go's PASS TWO.

Each blade is drawn twice. Against the PAGE: the oracle's rides in
MESSAGE_FORM_CONTROLS against
TestAStringBlobRecordOnAMessageBodyCarriesTheContentRule, and the emitter's in
the new tables-message-form-blob-negative-control, which builds
test/tables/message_blob_main.cpp against a regenerated tables/blobs. The true
run of that program is the target's own first step, so the gate is green before
the blade is drawn, and the sabotage reddens four rows and leaves the two
silent-report rows standing.

Against EACH OTHER: message_blob_ill_formed_text is blob_str8 re-encoded as a
batch of one with the fourth byte of note's eight byte payload replaced by
0xFF, and message_blob_zero_byte is the same wire with 0x00 there instead, so
the two separate the two halves of one check. Nothing else moves, so what the
two readers answer differently is the content rule and nothing else. The
*bytes record beside it carries bytes no UTF-8 rule would accept and is never
checked, which is the vector's own control.

tables-wire-fuzz-message-blob-oracle-negative-control and its leg twin replay
both vectors alone and require red on each, with the reports mirrored:

  the leg says 0,0,0,0,0,true,read, the oracle says 0,0,0,0,0,false,read
  the leg says 0,0,0,0,0,false,read, the oracle says 0,0,0,0,0,true,read

AND THE LEG GREW THE ROOT THE VECTORS NEED. blobdemo's Catalog is the only root
on the fuzzer's roster whose numbering can PLACE a text blob record, so it is
the only one whose wire can carry §3.1's rule at a node. Its AssetsTable.cpp
was already in CONFORMANCE_SOURCES, so the leg costs one include and one
MESSAGE_VARIABLE row, and the corpus pass now runs 45 roots rather than 44.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#632)

No sentence of §3 or §3.1 moves and none is added: §3.1 already says a text
blob's CONTENT is refused on the same terms as a kind 12 payload, and §3.3
already says a form-2 body's content rules are §3's. What was stale is HELD BY
TEST, which carries one bullet per rule with its red clause and carried none
for this one. The bullet names the six rows the two engines' gates run, the
*bytes row among them, and the three ways a leg goes red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@rowan-claude
rowan-claude merged commit b1771b6 into main Sep 7, 2026
20 checks passed
@rowan-claude
rowan-claude deleted the fix-632-message-blob-utf8 branch September 7, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A *string blob record on a message body carries no UTF-8 check in either engine, where the file form checks it (§3, §3.1)

1 participant