fixed tables: drop elision, ride the bounded payloads at their bound

[gafferongames]

DRAFT. The rule landed for every construct that can carry slack. Three of the ruling's own cases are refused by the wire itself, and the ruling's stated consequence — one constant size per type — does not follow. That is the finding this PR is really carrying; Glenn's ruling on it decides whether the rest should land.

The decisions, as given

"I think for fixed tables we should drop elision."
"Keep it simple. Fixed tables are meant to be the fast equivalent of types, in table form."
on [..Max]int: "it should be [fixed], and we should just always write the array as max, along with the count."
"same for strings, wstrings"
"what about unions? we could just write bytes always matching the largest union in fixed, and now unions are fixed too."
"effectively, fixed tables should only be used for small things."
on the guarded branch, which he calls the lookback conditional: "the intent of the lookback conditional is to make it variable size" / "so that is a disqualifying thing for a fixed table. not supported. only variable."

— the project owner, on the Schema Fixed Table wire (form 1), final form.

The rule, as implemented

A table the compiler derives as FIXED — no pointer, map or unbounded array anywhere in its by-value closure (ir.VariableTables) — writes every declared field, whatever it holds, in declaration order. A VARIABLE table keeps today's wire entirely, and so does a map's generated ENTRY, which is only ever a body inside a variable table's wire. The class is the table's, not the file's: a fixed table nested by value inside a variable one still takes this rule.

field kind	in a fixed body
any scalar, enum, flags, fixed/ufixed, 128-bit	always rides, never elided
a nested FIXED table	always rides, even when its body is its lone terminator
[N]T	always rides, whole
[..Max]T, T of one constant width	live count + Max elements, the slack zero-filled
[..Max]T, T a table/enum/union/pointer	live count + that many elements — Max of a self-framed element is not a number the writer knows
bytes(N)	live length + N bytes, the slack zero-filled
[Enum]T	every slot; no slot elides and the field has no elided form
a union holding None	header + one zero byte (it used to elide)
string(N), wstring(N)	rides — but at its length, not at N (below)
an absent ?T	still elides (below)

Padding is zero and no reader reads it: a count, a length or the enclosing L always stands in front of it. This is the one writer-side exception to §3's "nothing is aligned and nothing is padded", and it is an exception in the SLACK only.

STOPPED CASES — refused by the wire, not by a choice

1. string(N) and wstring(N) cannot ride at their bound. SPEC-TABLES §3, kind 12: "L, then L bytes, WELL-FORMED UTF-8 with no zero byte among them"; kind 33 the same over code units. The generated reader enforces it — TableUtf8Valid opens with if ( lead == 0 ) { return false; }. The L a string carries is its character count; there is no second number a reader could trust. N − length bytes of slack are therefore either malformed (zero bytes) or read back as content. Padding needs a second length inside the payload, which is a READ-side change.

2. A union cannot ride at the widest arm. Two separate refusals:

• A set arm's L is checked against the arm the reference names — a scalar arm against that kind's exact width (if ( sub.size != W ) { … malformed; }), a body arm against its own terminator (if ( sub.offset != sub.size ) { … malformed; }), a string arm by reading the whole L as text. An arm padded to a sibling's width is that arm's own framing damage.
• None's payload is the zero arm reference alone. §3: "arm id reference, and when it is not 0, kind (u8), L, …". The reader does if ( arm_ref == 0 ) { type = None; break; } without ever reading an L, so the field ends at that byte and the next bytes are the next field's header. There is nowhere to put padding.

3. An absent ?T cannot ride. Presence on this wire IS the field riding (§2.3) — the reader sets _present = true because the field arrived. There is no present flag for an absent optional to clear. It stays elided, the one elision a fixed body keeps.

AND THE BODY IS NOT ONE CONSTANT SIZE

Beyond those three: every field header, every enum value and every union arm names its id by a reference — a canonical unsigned LEB128 index into the file's own first-use id table. A reference is one byte for the file's first 127 distinct ids and two for the next 16,256, and which one a given field takes is a property of the FILE the body sits in, not of the type. Canonical LEB128 forbids padding a reference to a fixed width (a non-minimal encoding is malformed), so the widths cannot be normalized without teaching the reader.

MeasureBody therefore stays a function; no constexpr constant is emitted. Emitting one would be a lie the corpus disproves. What did change is that a fixed table's MeasureBody no longer asks what the fields hold — it reads only the counts and lengths of the payloads that carry one, and a body of plain scalars reads nothing at all ((void) value; is emitted for exactly that reason).

Measured, not argued. The paired corpus, 64 identical logical records, one table body each:

wire	corpus	table min..max	mean	spread
elision (today)	guard	2004..2098	2069.84	94
elision	guard dropped	2018..2109	2082.97	91
no elision	guard dropped	2107..2163	2149.88	56
no elision + bound padding	guard dropped	2115..2171	2157.88	56

The spread narrows and does not reach zero.

BYTE ACCOUNT

Paired corpus (64 identical logical records; packet 438 bytes throughout):

	packet	table mean	ratio table/packet
before	438	2069.84	4.726x
after	438	2157.88	4.926x

Growth by cause:

cause	bytes/record	share
drop the guard (idle_ticks now rides)	+13.13	+0.63%
no elision	+66.91	+3.21%
bound padding — entirely payload bytes(16), 8 used → 16	+8.00	+0.39%
total	+88.03	+4.25%

Bound padding reaches exactly one field in this shape. stats [..80]MixedStat and entities [1..8]MixedEntity are TABLE-element arrays (stopped case), player_name string(15) is a string (stopped case), loadout [4]uint8 is already full width.

C++ table conformance corpus, per pinned instance (old → new). The cost is concentrated in default-heavy records, which is where elision was earning its keep:

instance	before	after
scalars_default	10	488	48.8x
keyed_default	10	439	43.9x
profile_elide	64	453	7.1x
chain_value_empty	10	58	5.8x
root_default	10	47	4.7x
keyed_config	303	497	+64%
v2_seams	245	539	+120%
v1_cfg	271	487	+80%
loadout_full	239	321	+34%
wide_blob	210153	280055	+33%
root_full	571	684	+20%
archive	598	711	+19%
scalars_full	484	526	+9%
graph_deep	3319	3356	+1%

The compiler's synthetic wide-table probes move 95 → 4710 (49.6x) and 181 → 8328 (46.0x).

CORPUS IDS

BenchMixed loses its lookback conditional; has_extra, extra and idle_ticks stay as plain fields, so both wires still carry the same logical data. The packet is still 438 bytes — the four new bits land in the tail padding after align — but its content moved, so the id moved.

corpus	before	after
PACKET	6b213fbfa1a03a99	fd0748942dc6ae09
TABLE (paired)	07c57b4fa34b2700	b5867287a028004d

BenchMixed's worst case moves 3626 → 3630 bits and its pinned wire 3504 → 3508 bits, both inside 438 bytes.

The four standalone packet benches (bench/{cpp,c,go,cs}, BENCH-STANDARD.md) all name BenchMixed through generated code and nothing else — no runner carries a field list, and every one reads the variant file blind at filesize/64. No runner source moves. What moves is testdata/wire/bench_mixed.bin, bench/corpus/variants/bench_mixed.variants.bin, the pins in test/bench/main.cpp, the vary mapping in bench/tools/variantgen/main.go, and the corpus id on every published row — rows on 6b213fbfa1a03a99 stop being divisible against rows on the new axis (BENCH-STANDARD §1.6, §5.3).

GATES

Green: go build, go vet, gofmt, make check, make shape-gate (the variantgen ratchet moved 94 → 96 for the two IdleTicks references), internal/goldens (re-pinned), compiler/TestCppTableHeaderAndPut128Bytes (re-pinned via SCHEMA_UPDATE_HEADER_GOLDEN=1), C++ table generation + compile clean under -Wall -Wextra -Werror -Wshadow, every testdata/wire/tables/*.bin re-pinned and save → load → save byte-stable for every pinned instance (the round trip is proven; no codec defect), SCHEMA_UPDATE_WIRE_GOLDENS=1 ./build/schema_test_bench, make bench-variants, ./build/schema_test_bench_paired pin and verify.

Red, C++ — 30 assertions in test/tables/main.cpp that encode the elision contract, and three compiler tests with inline byte pins. These are not re-pinned in this PR on purpose: several of them ask questions the new rule re-answers and that answer is Glenn's, not mine. The clusters:

• wrote == empty_wire_bytes — an all-default table no longer saves the 10-byte empty wire (:374, :395, :2155, :2405, :3388, :3449, :5315, :5399)
• cross-version report counts — a V1 writer now rides every field, so a V2 reader counts more unknown/kind_mismatch (:426, :427, :452, :453, :484, :502, :523, :533, :545, :570, :590, :602, :5345, :5355)
• w_bare_optional > w_bare_value, w_ptr > w_value, !from_bare.link_present (:2309, :2316, :2324, :5406, :5415) — a semantic question for Glenn: with a default-valued T now riding, a bare value read by a build that declares ?T reads as PRESENT, where it used to read absent. §2.3's "?T, *T and a plain nesting are wire-identical" now cuts the other way.
• uint8_t buffer[64] too small for the new wire (:380, :382)
• compiler: TestTableClosureCountDefaults, TestCppTableRefOrdinalBytes, TestCppTableSaveMeasureCacheBytes

Red, the other eight ports. testdata/wire/ is C++-pinned and byte-compared by every leg, so all of them go red until they mirror the emitter: the C, Go, Rust, C#, JS, Dart, Java and Elixir wire tests; the paired gate for C, Go and C# against bench/paired/corpus/*; and make generated-current for generated/bench/{c,cs,go,paired/*}.

Not run (they need the ports mirrored first, or are CI-only with no Makefile target): build/schema_test_tables_asan, tables-wire-fuzz, conformance, the negative-control plan, generated-current, golangci-lint v2.12.2, modernize v0.23.0.

EMITTER HOOKS FOR THE PORTS TO MIRROR

The C++ change is two files: the class predicate and flag in internal/codegen/cpptable/cpptable.go (fixedNoElide, tableGen.noElide, and TableWriter::zeros), and every ride decision in internal/codegen/cpptable/codecs.go (emitTableMeasureField, emitTableWriteField, emitKeyedSlotRides, emitUnionPayloadMeasure, emitUnionPayloadSave, emitArrayField, plus padsAtBound / countedPad).

port	the ride decisions live here
C	internal/codegen/ctable/wire.go:557 emitWireFieldHelper, :627 wireKeyedRides, :642 emitWireScalarLeafMeasure, :674 emitWireWrite; element write at internal/codegen/ctable/codecs.go:422
Go	internal/codegen/gotable/write.go:256 emitFieldRides (the elision predicate), :188 emitWireField, :50 emitTableMeasure, :144 emitTableWrite; the arithmetic-measure fast path at :34 arithMeasureFieldEligible and :95 emitArithMeasureField assumes elision and needs the same gate
C#	internal/codegen/cstable/wire.go:75 leafRideCondition (the elision predicate), :179 emitWriteBodyTyped, :252 emitSaveTyped, :82 emitCollectTyped (the id-interning pass, which must stop skipping defaulted ids); measure in internal/codegen/cstable/measure.go

Rust, JS, Dart, Java and Elixir take the same shape in internal/codegen/{rusttable,jstable,darttable,javatable,elixirtable}.

WHAT I'D ASK GLENN

1. The three stopped cases each need a read-side change to pad. Is that worth a new form byte for the fixed class — where a bounded string could carry its own length and a union its widest-arm L — or do fixed tables live with bounded-and-stable?
2. Without constant size, padding [..Max]T and bytes(N) costs bytes and buys only the writer's branch removal. Keep it, or keep no-elision alone?
3. T-read-as-present-?T (above) is a tolerance change that follows from no-elision. Intended?

🤖 Generated with Claude Code