tables: widening on read, the refusal reasons, ill-formed text and comments (#532)

[gafferongames]

Closes #532. The kinds-and-rules page lands in the C++ reference, the oracle
and the tool: the widened counter, TableRefuseReason on Open,
BlockOpen and a variable root's load measure, ill-formed kind 12 text as
damage, comments read and never written in the text form, and reserved kind
34 refused by name. Five BACKEND STATUS: OWED lines are deleted, one per
claim this makes true, and so are the widening caveats in USAGE.md,
VERSIONING.md, COMPETITION.md, COMPARISON-TABLES.md, FAQ.md and
TUTORIAL.md.

origin/main is merged in, so this branch carries #557's bitpacked message
form, #593 and #591.

What lands

1. widened, the sixth report member. An integer kind read into a wider
one of the same signedness, and f32 into f64, decode EXACTLY at the wire
kind's width and count widened once per field and once per map. The branch
is emitted at every site this wire compares kinds: a field's own kind, an
arm's, a positional array's element kind, a keyed body's, a list's, and a
map's key kind. The arm evolution row a1_arm_moved_width_as_a2 moves from
one kind_mismatch and a None union to one widened and the value, and
the manifest's pin moves with it.

1b. And the same row on the MESSAGE FORM, which §3.3 holds to the file
form's word ("NO EVOLUTION ROW OF §4 MOVES"). After the merge the message
reader compared kinds at three sites with no widening branch and the oracle
did the same, so the page's sentence was unmet. Both engines now take it, at
the field, the arm, the positional array's element and the map's key. The
payload already rode at the width the ANNOUNCEMENT states, so the branch sits
inside the mismatch branch the reader already took and the matching path pays
nothing. Two shapes the announced width does not settle on its own are handled
beside it: f32 into f64 carries a NaN's payload on the bits rather than
through the hardware conversion, and a narrow integer into a 128-bit
declaration is sign or zero extended before this reader's own bound meets it.

2. TableRefuseReason. One NATIVE enum per target, a C++ enum here, in
every unit, carried on each language's standard error path: an out-parameter
beside the null in C and C++, never an exception. Open, BlockOpen and
LoadMeasure name the FIRST failing clause in §7's order, bad_alignment
ahead of both truncated clauses and the base's alignment LAST because it is
the one clause that reads nothing out of the file. Every block prologue word
is read bytewise so that ordering holds. It is written on the refusal path
only: a match writes nothing, which is what makes the successful open cost
nothing.

3. Ill-formed kind 12 text is damage. A payload that is not well-formed
UTF-8, or that carries a zero byte, reads the declared default and counts one
malformed, checked AS IT ARRIVES and before the reader's own bound, at a
field, an arm, a map key, an array element and a *string record. A clamp
cuts at a code point boundary, so a clamp can never invent ill-formed storage.

4. Comments in the text form, read and never written: // runs to the end
of the line or of the input, /* */ to its closing delimiter, which does not
nest, and an unclosed /* is malformed on the terms an unclosed string is.
Seven json-hostile rows, five accepted and two refused.

5. Reserved kind 34 is refused as damage by name, in the oracle's skip
and in TableReader::skip, exactly as 35 or 200 is.

The cost rule, and the sitting

The sitting was re-run on the MERGED unit, because #557 moved MeasureBody,
SaveBody and LoadBody and the artifact that merges has to be the one
measured.

The sitting. Apple M2 MacBook Air, 8 cores, macOS 26.6.2, Apple clang
21.0.0, -O3 -DNDEBUG -fno-rtti -ffp-contract=off, 2026-09-05 23:15 local,
one-minute load average 2.35 read with uptime in its own command before
the run, swap 0.00M in use. Seven alternations of four arms round-robin in
one process each, so any drift in the box lands on all four. It is a pairing
check on a shared interactive machine, not a publishable number.

Four arms, one bench/tables/cpp/table_main.cpp, one set of flags.

• before is origin/main's committed generated bench unit, 7,912 lines.
• after is this branch's, 9,157.
• layout is origin/main's with this branch's ADDED RUNTIME pasted in and
  nothing else, 8,063: the refusal enum and the widening and text helpers, none
  of them reached from the bench's codec. It carries the translation unit's
  GROWTH without the behavior, so whatever it does to the write arm is what the
  layout did. The 1,094 lines between it and after are emitted BRANCHES, not
  runtime, and every one of them sits inside a kind-mismatch arm.
• planted is the NEGATIVE CONTROL: the widening predicate moved ONTO the
  matching path and made a real call, if ( kind != N ) becoming
  if ( TableKindWidens( kind, N ) || kind != N ) with TableKindWidens no
  longer inline. It answers the same thing at every site and pays for the
  predicate whether or not the kind moved. 59 sites.

The tables bench, M msg/s, higher is better

arm	write best	write median	round_trip best	round_trip median
before	0.914	0.887	0.274	0.273
after	0.901	0.897	0.273	0.271
layout	0.890	0.887	0.274	0.273
planted (control)	0.904	0.898	0.245	0.245

The read arm is unchanged: 0.274 to 0.273 best and 0.273 to 0.271 median,
a 0.7% median move against per-run spreads of 0.4% to 1.1% on that arm.

THE CONTROL IS RED. Planting the predicate on the matching path costs
10.1% of best and 9.8% of median on the read arm, 0.273 to 0.245 both ways,
so the instrument sees a cost of the size this feature would have had if it had
been put where the payload under the declared kind runs. It is not there: the
widening branch lives inside the kind != N branch a reader already takes,
which today skips the payload, and the generated text shows it. The control
leaves the WRITE arm alone, 0.901 to 0.904 best, which is what says it is
aiming at the read path and nothing else.

The write arm's arms sit within their own spread (0.887, 0.897, 0.887,
0.898 median against a 6.5% worst per-run spread on that arm), and layout
lands on before to three figures, which is the reading bench/LOCK already
gives: a byte-identical source measures differently and the unit's size is what
moves it.

The packet bench

generated/bench/cpp and generated/bench/c are byte-identical to
origin/main on the merged branch, verified with git diff origin/main HEAD -- over both trees, which comes back empty. This PR touches no packet
emitter, so the type wire's read and write paths are unchanged by
construction rather than merely within noise, and that is a stronger statement
than a measurement of them would be.

Gates, each with its control quoted red

• make tables-wire-fuzz: 129 seeds over 42 roots, 121,984 enumerated +
  20,000 random = 141,984 mutants, 0 divergences against the Go oracle,
  including the widening pass (a widening and its reverse at every integer
  and f32 kind byte, over fields, arms, elements and keys) and the text
  pass (a truncated sequence, an overlong encoding, a lone continuation, a
  surrogate, and a zero byte at the front, the middle and the end of every
  kind 12 payload).
• The message form's widening, red first in both engines. The oracle's row
  with the widening predicate planted off:

  === RUN   TestAWidenedKindOnAMessage
      messagerules_test.go:1407: a kind the declaration WIDENS is not a mismatch (§3.3, §4): kind_mismatch=5
      messagerules_test.go:1412: the five widened sites count one each (§3.3, §4): widened=0
      messagerules_test.go:1415: the widened field's value is the writer's own, exactly: score=0, not 1000
      messagerules_test.go:1418: f32 into f64 is exact: ratio=0, not 0.5
      messagerules_test.go:1421: the widened element's value is the writer's own: marks[0]=0, not 7
  --- FAIL: TestAWidenedKindOnAMessage (0.00s)
  

  and the C++ reference's row with the arm's branch planted off:

  FAIL test/tables/message_form.h:2019: report.kind_mismatch == 0
  FAIL test/tables/message_form.h:2020: report.widened == 1
  FAIL test/tables/message_form.h:2021: out.value.type == tbla2::ValueType::A
  FAIL test/tables/message_form.h:2022: out.value.a == 1000
  tables test: 4 failure(s)
  

• make tables-wire-fuzz-length-negative-control: RED, and it changes
  what it quotes: the content rule reads a kind 12 payload as it arrives, over
  the whole of L, so a leg that lost room( len ) walks off the buffer
  instead of clamping a neighbor's bytes.
  harness: FAILED after 629 mutants: the leg died on the mutant (EOF)
• make tables-zero-cost: green, and its control RED:
  negative control: a planted node symbol turns the zero-cost scan RED, and the reserved id alone does not
  The scan now sanctions TableRefuseReason by its whole spelling, because
  TableRef is a prefix of it and the extraction takes the longest
  alternative.
• make conformance: every registered surface passes in all nine
  languages, cook-reason 21/21 and block-reason 12/12, and forgery is now
  12/12 in every port rather than 11/12 in five of them.
• test/tables/block_main.cpp pins BlockOpen's own readings: ok
  untouched on a match, unaligned_base for a null base and for a WHOLE image
  copied to an unaligned address, truncated for a length under the prologue,
  bad_layout for a length whose arrays no longer fit, and truncated for a
  length that stops INSIDE the used extent's own padding.
• test/tables/main.cpp pins unknown_form and blob_over_size_cap.

Page rulings a builder made, for the owner to keep or overturn

Each is a sentence the page did not carry, written onto it because the code
could not be written without answering it.

1. §7's table row for a cook's two caller-side clauses: a null buffer
   answers unaligned_base and a buffer shorter than the header answers
   truncated. Neither is a fact about the file, and the page named no value
   for either.
2. The same two readings for BlockOpen, which owes them for the same
   reason and had the same silence.
3. "A -1 for a trailer that cannot be read whole writes nothing." Damage
   is not a refusal, and the reason enum is the refusal's vocabulary, so a
   measure that fails on damage leaves the caller's own value standing.
4. "Every unit" carries TableRefuseReason, replacing "a unit with
   neither construct carries neither": every unit's cook Open names its
   refusal in it, so every unit needs the enum whether or not it has a pointer
   or a block.
5. §19.2's clause order is the order the arithmetic forces: the per-array
   bad_layout clauses run BEFORE the used-extent truncated clause, because
   the used extent is derived from the arrays and has nothing to say until
   they agree. The page listed them the other way round in both places it
   lists them.
6. unknown_form is any byte that is not THIS CALL's form, form 2
   included at a file measure. A build that carries the message form carries
   it through the message surface, and a batch handed to a file root is a form
   its file measure does not read. Load beside it still distinguishes the
   two, answering message_form_as_file where the byte is 2, because a
   report has room to say which and a -1 has one value.
7. data_cycle gets a carrier rather than losing its name. It is refused
   by Measure, CookMeasure, Save, Cook and Lock, each of which
   answers a bare -1 or false, while LoadMeasure is the only measure
   that takes the out-parameter. An enum value no call can write is a case
   that exists only in the vocabulary, so the ruling is that Measure and
   CookMeasure take the same trailing TableRefuseReason *. Threading it
   through the pack walk is a named follow-on, not part of this row.
8. unaligned_base is the one block value the cross-language surface does
   not carry, because it is a clause over the ADDRESS a caller passes and a
   manifest row hands a driver a path. The page said one row per value. It now
   says which one is held in the C++ gate instead and why.

What the page left silent, and what this had to decide

• A raw ill-formed byte in a JSON string body. The page contemplates a
  text introducing ill-formed storage only through a LONE SURROGATE ESCAPE,
  which both readers already mapped to U+FFFD. A raw byte reached storage
  intact, and the wire this PR then wrote was one this PR's own reader calls
  malformed, so schema pack emitted a wire no conforming reader accepts.
  §16.3's own words ("the rule underneath both is one rule") are applied at
  the point the defect ENTERS: the read replaces it with one U+FFFD, in the
  engine and in the C++ walk both, counted as nothing and clamped at a code
  point boundary like anything else. The page now says so.
  str-illformed-at-bound moves to one clamped, and the text form is
  byte-stable from the FIRST lap rather than after one.
  This decision STANDS pending your yes, and nothing about it moved in
  this round.
• An interior zero byte from the four-hex-digit zero escape is the same hole
  and is NOT closed here. §16.3 states that a narrow field's interior zero
  byte is a code point JSON has an escape for and "the text round-trips it
  exactly", and §3 says a kind 12 payload carrying a zero byte is damage.
  Those two sentences disagree. Filed as Page contradiction: a JSON unicode-zero escape in a string(N) body, round-tripped by §16.3 and damage by §3 #594 for your ruling. No corpus
  row reaches it today, so nothing is red, and nothing here guesses at it.
• The C++ enum's scoping. The page names TableRefuseReason and its
  values and does not say scoped or unscoped. Landed as an UNSCOPED enum at
  namespace scope, matching TableMessageReason beside it, with every value
  name claimed in §11 through internal/tablenames so no declaration can
  collide with one.
• The order among a block's bad_layout clauses. §19.2 lists "a pitch, a
  count, an offset or an extent" under one value, so the order among them is
  unobservable. Landed as pitch, count, offset floor and alignment, offset
  past the caller's bytes, rows past the extent.
• A map whose keys widen and then disagree. §2.8 resets the map to empty
  with one kind_mismatch, and the page does not say whether a widened counted
  at an earlier key is retracted. Landed as KEPT: the counter names an event,
  not a surviving value.
• unknown_form from a MESSAGE measure with no announced vocabulary. The
  page names the form byte and not the announcement. Landed as writing
  nothing: an unannounced connection is §3.3's refusal, not §6.5's.

A divergence found, filed, and closed in the same round

schema#598. BlockOpen's truncated clause over the CALLER's bytes, a
block whose used extent's padding runs past the buffer it was handed, was
refused by the reference and opened by five ports: C#, Dart, Java,
JavaScript and Rust. docs/SECURITY.md names a cross-language acceptance
divergence a security bug on its own terms, so it was filed rather than
absorbed.

The new manifest row block_extent_padding is what found it, on the precedent
cook_claim_one_short set: no patched word, the extent column alone, over
block_render, whose greatest array end is 2784 in a 2816-byte image, so a
claim of 2815 covers every array and not the padding.

forgery       pass 12/12  pass 12/12  FAIL 11/12  FAIL 11/12  pass 12/12  pass 12/12  FAIL 11/12  FAIL 11/12  FAIL 11/12
FAILURES
  cs / forgery: block_extent_padding: 5 bytes out, 7 expected
  dart / forgery: block_extent_padding: 5 bytes out, 7 expected
  java / forgery: block_extent_padding: 5 bytes out, 7 expected
  js / forgery: block_extent_padding: 5 bytes out, 7 expected
  rust / forgery: block_extent_padding: 5 bytes out, 7 expected

The READERS were never the problem: all nine generators emit the clause. The
five drivers were, each raising a claim SHORTER than the image to the image's
own length, so the input had never been expressible in them. Two of them were
worse than that: C# copied bytes.Length into a buffer sized by the claim and
Dart did the same with setRange, so both had a real overrun waiting behind
the clamp. Each now allocates the claim and copies only what fits, as the C++
reference, C, Go and Elixir already did, and all nine refuse:

forgery       pass 12/12  pass 12/12  pass 12/12  pass 12/12  pass 12/12  pass 12/12  pass 12/12  pass 12/12  pass 12/12

Closes #598. #387 stays open for the other half of the same driver work, the
pointer column, which unaligned_base needs.

Named follow-ons

• The wire fuzzer's widen pass has no purchase on form 2, and that is a
  fact about the form rather than a gap in the pass: a form-2 body carries no
  kind byte at all, so the kind a reader compares is the ANNOUNCEMENT's, which
  both the engine and the leg read from their own unit. A widening mutant
  would have to hand both sides a forged announcement, and the driver's
  protocol carries a body. §4's widening row on this form is held by a peer
  whose declaration has MOVED instead, which is the shape every other §4 row
  on this form is held by. Said on the page and in the harness.
• data_cycle's carrier (ruling 7 above): Measure and CookMeasure
  take the out-parameter, threaded through the pack walk.

Two controls this PR had to re-aim

Each still goes red for the reason it names, and each moved because the code
it brackets moved, not because it stopped holding.

• tables-wire-fuzz-length-negative-control quotes the leg died on the mutant rather than the report differs: the content rule reads a kind 12
  payload over the whole of L, so a leg without room( len ) walks off the
  buffer instead of clamping a neighbor's bytes.
• BLOCK_FUZZ_SED_CPP_maximum and the cook-open-walk-cpp sabotage anchor on
  the lines where the declared-maximum check and the alignment refusal now
  sit.

make test is green whole on this branch.