spec: the wstring table kind, widening, float16, Open reasons, text comments (#523)

[gafferongames]

Rulings 1, 5, 6, 7, 8 and 9 of #523, on the owner's word of 2026-09-04. Specification only, no code. Rebased onto main after #507, #527, #529, #530 and #531.

---

The blocking ruling from the cold read: ill-formed text is framing-class damage

On the table wire, ill-formed text content is damage and not data. A kind 33 payload carrying an unpaired surrogate or an interior zero unit, and a kind 12 payload carrying malformed UTF-8 or an interior zero byte, each leave the field at its declared default, count one malformed, and let the parent read on past L. Kind 12's silent acceptance was an inherited gap, not a precedent, and it is fixed here rather than deferred: the two text kinds now hold one rule.

That inverts what the first draft of this PR said, and it makes the page simpler rather than more complicated:

• SPEC.md §4.12, §4.7 and §5 stay true exactly as written. Every generated reader refuses ill-formed text, in all nine targets, in every build mode. Only the SHAPE of the refusal differs by wire: the packet wire stops, because a bit stream that met one has nowhere to continue; the table wire defaults and counts, because L says where the next field begins. Neither accepts it, which is what lets nine targets owe each other one verdict on one payload.
• The write side needs no rescoping. A tolerant load never produces an instance the reference cannot re-save, because ill-formed content never reaches storage.
• §16.3's U+FFFD rule becomes the text form's rule for its own storage only — an instance a PROGRAM built, or one a lone surrogate escape introduced — and it now states the two spellings separately because the two failures differ: a code point is escaped and preserved (\^@ for a zero byte or zero unit), and what is not a code point at all is replaced (U+FFFD for an ill-formed UTF-8 sequence or an unpaired surrogate).
• SECURITY.md's "not a gap" sentence is gone, replaced by the counted rule: a table reader that STORES any of it is an acceptance bug on that list, exactly as a packet reader that accepts one is.
• Content is checked BEFORE the reader's bound is applied, so no clamp can produce ill-formed storage, and §17.3's UTF-8 carve-out narrows to instances built in code.

Two further rulings landed with it:

• Reserved kind 34 is a NAME reservation for a later major and nothing else. It is not in this major's set, no writer emits it, no reader has a rule for it, and a conforming later-major writer carries a new kind under the escape kind 31 (Reserve one escape kind on the table wire: length-prefixed opaque, skip me #434) so this major's readers skip it and count unknown. A reader of this major therefore meets 34 only as damage, exactly as it meets 35 — which is the correct answer, because a bare 34 is a writer that ignored the escape.
• The Open check order is §7's own enumeration order, one order for both accelerators: magic, byte order, build version, reserved words zero, the alignment word validated, the two part lengths against the caller's length, the root's storage inside the data part, the base alignment. So bad_alignment precedes both truncated clauses, and the page says why — the alignment word is the one field the check computes WITH, so the part-length arithmetic below it would otherwise run over a forgery. §19.2's list is reordered to match, and a block's prologue has NO reserved words and no alignment word (§19.1: exactly magic, build_version, byte_order), so reserved_not_zero and bad_alignment never fire for a block and the page says that rather than leaving it ambiguous.

---

1. wstring on the table wire (folds #522)

wstring(N) gets kind 33 in §3's closed set: L, then L bytes holding L / 2 UTF-16 code units, two bytes each little-endian. An odd L is framing damage. Its unbounded twin is *wstring, a blob node under the third reserved id fnv1a64( "wstring" ), beside bytes and string. The layout row is char16_t[N + 1] then an int32 used length in code units (§7.2), the blob node is a u64 byte length, the units and a zero unit, the text row is a JSON string transcoded at the boundary (§16.2), and the LoadMeasure term is the framed length plus the eight-byte header plus the terminator the reserved type id names. wstring(N) is refused as a map key by name in §2.8, the diagnostic naming string(N). The §11 closure refusal from #510 is deleted, and the arm row, the skip rule, the retain-unknown copy list, the fuzzer strategies and cook-check's companion list all gain the kind.

The table-form goldens take the packet names with -table on them, and the paragraph states the whole verdict table, including which of the corpus's seventeen refusals do not carry over and why.

Cold-reader questions.

• The blob's length is BYTES and always even, not a code unit count. That keeps one arithmetic for all three blob kinds in the wire, the region and cook-check. Is the parallel worth the half step a reader has to do?
• Is *wstring earning its place, or should the ruling's reserved id have been reserved and left unbuilt until a field asks for it?
• The map-key refusal argues from memcmp over little-endian units. Does that argument hold, or is the honest reason simply that no wide order is worth standardizing?
• TableWStringAt's view carries a length in UNITS where the two narrow views carry bytes, and AllocWString( n ) takes a unit count where AllocBytes/AllocString take bytes. Is the family better with one inconsistency and no call-site doubling, or with one unit and a * 2 at every wide call?

2. Widening on read

A new widened counter, sixth in the read report. The signed ladder is kinds 2, 3, 4, 5, 18 and the unsigned one 6, 7, 8, 9, 19, each kind accepting every kind below it on its own ladder, plus 10 into 11. Everything else stays kind_mismatch, the narrowing direction included. The rule is decided by the kind pair alone, so it holds at a field, an arm, an array element and a map key. §4's report line, §4.1's evolution table, §4.2's fuzzer and VERSIONING.md's read report, "Widen a field" pattern and "no widening path" sharp edge all move. The silent class stays at four.

Cold-reader questions.

• The counter is not a loss counter, so the never-clobber condition does not name it. Is that right, given a rewrite writes the wider kind and the file changes?
• Applying the rule at a MAP KEY loads a map that used to read empty. Order is preserved under sign and zero extension, so it is safe, but is it in scope?
• Including i128/u128 as the top rung of each ladder is a reading of "an integer kind", not a ruling. Should it be there?
• A flags field rides as kind 9, so it accepts 6 through 9 like the uint64 it is indistinguishable from. Should flags be excepted, or is the special case worse than the pair?
• The baseline still REFUSES a widening, unchanged, because the tolerance runs one way and a mixed fleet has readers on both sides. Should it warn instead?

3. Flags storage stays uint64

SPEC.md §4.2 states the reason: a mask rides as a raw uint64 on the table wire and a load copies it verbatim, so full-width storage is what lets an older build load bits it cannot name, hold them, and write them back. Derived storage would drop them with nothing able to report it.

Cold-reader question. The property is the TABLE wire's. The packet wire reads W bits and never sees the appended ones. Is the reason stated at the right altitude sitting in SPEC.md?

4. float16

Declined for this major, kind 34 reserved by name in §3's kind table with bits(16) plus application conversion named as the spelling. One bullet in SPEC.md §4.10, one paragraph in §3, one clause in VERSIONING.md's kind space. The fuzzer plants 34 and requires framing damage rather than an escape.

Cold-reader question. Is reserving a number in a closed set the right shape, or does the escape kind 31 already cover a later addition well enough that 34 is a promise the page did not need to make?

5. Open and BlockOpen name their refusal

A TableRefuseReason out-parameter beside the null, filled on every call, with one value per clause of §7's existing check list: ok, not_a_cook, foreign_order, wrong_build_version, truncated, reserved_not_zero, bad_alignment, unaligned_base, plus bad_layout for a block. The check runs in that order and the first failing clause names the reason, so nine languages answer alike on one file. The parameter is optional in every target, so no call site moves. TableRefuseReason joins §11's claimed names. VERSIONING.md's "Open refuses in silence" edge is rewritten.

Cold-reader questions.

• Two alignment values, bad_alignment for the header word and unaligned_base for the caller's pointer. The ruling said "unaligned", one name. Is the split worth it? The argument is that only one of the two is the caller's own fix.
• The byte-order clause is the one clause with no value of its own — it answers not_a_cook, because a header whose byte_order word contradicts its own magic describes no cook in either order. Is folding it right, or does it want a ninth value?
• Should the enum swallow the caller errors JavaScript and Dart throw for? The page says no.

6. Comments in the text form

// and /* */, non-nesting, legal wherever whitespace is, ordinary text inside a string, an unclosed /* is malformed. Neither writer ever emits one. §16.2's "Comments are not JSON and are refused" is replaced, the round-trip guarantee is restated to say what it covers — the form's own output, not a hand-edited text, which loses its comments and trailing commas the same way it loses its line breaks and key order — and §17.3 gains the pack then unpack lap. Held by test in §16.5, including the writer gate that ToJson of every corpus instance contains no / outside a string.

Cold-reader questions.

• The read then write direction is now explicitly lossy for a hand-authored tree. Is stating that enough, or does unpack need a flag that refuses to overwrite a file it did not write?
• Is /* */ worth carrying at all, or is // alone the whole want?

---

Decisions the page made that #523 or the cold read did not

1. A *wstring blob's length is a byte length, always even, rather than a code unit count, so the three blob kinds share one arithmetic in the wire, the region and cook-check.
2. Ill-formed CONTENT in a text blob is malformed too, on §3.1's own terms — every slot naming that blob reads null. The ruling named the two field kinds; a blob is where the same bytes ride at a node, and leaving it out would have been the gap again one level down.
3. The clamp cuts are stated and ordered: content check first, then the bound. A kind 12 clamp cuts at a code point boundary and a kind 33 clamp at N units, dropping a high surrogate whose low half did not fit — the same cuts §16.2 already states for the text form, so both forms land the same bytes.
4. A zero unit and a zero byte are \^@ in the text form, not U+FFFD. They are U+0000, a code point JSON has an escape for. Only what is not a code point is replaced.
5. Widening applies at every site a kind is compared — field, arm, array element, map key — rather than at the field alone, and counts once per field or map, matching the kind_mismatch it replaces. The fuzzer strategy now plants it at all four.
6. The 128-bit kinds are the top rung of each ladder.
7. flags is not excepted from widening, because the rule is the kind pair and nothing else.
8. The baseline still refuses a widening, unchanged, so §18 does not move.
9. widened is not a loss counter — out of the never-clobber condition (stated explicitly in VERSIONING.md now), out of the silent class, and zero in the text form.
10. A table load does not clamp a flags mask, and the page says so and says why the storage is uint64. And it names the cost: an older build that loaded appended bits out of a table and then puts that mask on a PACKET drops them by width, silently, with no counter on either wire able to say so.
11. bad_layout is the block's own value, and the two cook values a block cannot produce are named as never firing rather than left ambiguous.
12. TableRefuseReason is disambiguated from spec: the message form, an id table scoped to the connection (#523) #530's message-form refusal values in one paragraph: different enums, different paths, never returned by one call.
13. TableWStringAt answers a view whose length is a COUNT OF UNITS and AllocWString( n ) takes a unit count, because every other wstring number on the page is a unit count and a byte argument here would be the one call a caller doubles at the call site. Dart spells them tableWStringAt and allocWString. All three trios are claimed in §11.
14. An unclosed /* is malformed; a // ends at end of line or end of input, so a hand-edited last line without a trailing newline is accepted.
15. The round-trip guarantee is restated as the form's own output only. pack then unpack over a hand-authored tree is explicitly not byte-stable: comments and trailing commas go the way line breaks and key order already do.
16. SECURITY.md's text bullet now covers both wires, with the table wire's storing of ill-formed text named as an acceptance bug.
17. Counter sweep: eleven SPEC-TABLES sites, VERSIONING's two, PORTING's, USAGE's two and COMPETITION's. VERSIONING's never-clobber sentence stopped carrying a count that drifts and now says which counters, with widened explicitly outside the condition.

One defect found on main and fixed in passing

#530's change (g) did not land. Its commit message says "3's form-byte pin now uses forms 0, 3 and 0xFF, since 2 is a known form with rows of its own", but §4.2's fuzzer strategy on main still plants form byte 2 and requires "a named refusal and never damage" — which is now wrong, because 2 IS a known form with rules of its own (§3.3). Since this PR edits that same list, the pin is corrected to 3 here, with the reason stated inline. Flagging it rather than burying it, in case #530's author intended a different site.

Reconciliation with #530 and #531

#530, the message form.

• §3.3's vocabulary tail cited "(§3.1, and schema#532 for the third)" for the third blob id. That forward reference is now satisfied and reads "(§3.1, all three)".
• The reserved-id COUNTS spec: the message form, an id table scoped to the connection (#523) #530 moved from two to three — the node-table id and the announcement's build-version id — count ids the LANGUAGE HOLDS BACK, not the blob type ids, which are ordinary hashes of keywords. They are untouched by this PR, correctly. The blob-spelling counts this PR moves from two to three are a different axis.
• TableRefuseReason is disambiguated from §3.3's message-form refusal values in one paragraph: a TableRefuseReason means a FILE was refused, one of §3.3's means a MESSAGE on a connection was, and the recoveries have different owners.

#531, unbounded arrays.

• §6.5's new LoadMeasure reason table says its -1 "carries the SAME ENUM the accelerators' refusals carry" and referred to it only by description, because this PR merges second. It now NAMES TableRefuseReason and cites §7 and §11.
• §7's own paragraph is widened to match: the enum spans the accelerators' eight clause values AND the measure's five, in one vocabulary, on spec: unbounded []T and []*T in the variable class (#523) #531's own ground that a caller asking "why can I not have this file" asks one question whichever call refused it. §11's claim reads "the one refusal vocabulary both accelerators and LoadMeasure's -1 name".
• §2's declaration list and §2.8's VALUE list now carry both spec: unbounded []T and []*T in the variable class (#523) #531's []T entry and this PR's wide spellings.
• spec: unbounded []T and []*T in the variable class (#523) #531 introduces no text-kind, counter or blob rule, so nothing else collides.

The name is TableRefuseReason, ruled. The enum was TableOpenReason in the first draft, and #531's ruling that LoadMeasure's -1 carries the same enum made a name built on Open wrong at half its call sites — nothing is opened by a measure. It is renamed across §6.5, §7, §19.2 and §11's claim, with the Dart spelling stated (TableRefuseReason keeps its PascalCase there, its values take that backend's lowerCamelCase, the shape TableCookRef.outside already takes), and §7 states the reason for the name rather than leaving it to be inferred. #531's §6.5 reference, which could only describe the enum because this PR merges second, now resolves to it by name.

Merge check

git merge-tree against main @ dd39038 (after #530 and #531) is clean, 0 conflicts. Nothing else is open that touches these files.

Two edits are shaped to stay out of #531's way and were kept that way through the rebase: §11's byte-buffers bullet gains *wstring at the END of the bullet, and §2.8's VALUE list carries the wide spellings one line above the *bytes/*string line. No prose was weakened for it.

Known stale, left for the docs pass

docs/COMPARISON-TABLES.md rows 162 and 272 still list five report fields. Those are the D-cells #523's rulings send to the docs pass. testdata/conformance/tables/FORMAT.md and test/conformance/README.md carry the harness's counter tuple and move with the implementation PR, as does the §4 arm-evolution row's manifest pin.

Cold read

Verified at 79176ef. Three named edits landed after it, plus the TableRefuseReason rename the reader's fetch missed:

• §2.6's arm companion list carries wstring(N) beside string(N) and bytes(N).
• §3's gate row says three reserved-id refusals, listing the node-table id, the build-version id and another table's id, so it agrees with §3.3. §3.3's row, which read "the two become three", now names the build-version id as the third of §3's three rather than restating a transition from a count §3 no longer carries.
• §7's refusal table had a stale reason / what the check found header above the clause / reason one, left from the reorder. One table, one header, and the ok row's clause cell now reads "no clause fails".

Ready for review. Do not merge.

[gafferongames]

Rulings from the cold read, under the owner's word of 2026-09-04:

1. On the table wire, ill-formed text content is framing-class damage. A kind 33 payload with an unpaired surrogate or an interior zero unit, and a kind 12 payload with malformed UTF-8 or an interior zero byte, each leave the field at its default, count one malformed, and the parent reads on past L. Kind 12's silent acceptance was an inherited gap, fixed in the same PR. SPEC.md §4.12, §4.7 and §5 stand as written; §16.3's U+FFFD rule is the text form's for its own storage; SECURITY.md's "not a gap" sentence is replaced by the counted rule.
2. Reserved kind 34 is a name reservation for a later major; a conforming later-major writer carries a new kind under the escape kind 31, so a reader of this major meets 34 only as damage.
3. The Open check order is §7's enumeration order, one order for the cook and the block, so the alignment word is validated before the root-fits arithmetic.

The named page changes from the read land on this branch before it leaves draft.