tables: the fixed form — SPEC §3.4, THE FIRST BYTE, the variable-form rename, and selection by the keyword

[gafferongames]

Draft. Two passes over the fixed form, on one branch.

Pass one (a69f0ea6..07bcdb6a) is SPEC §3.4: the rename Glenn's read
asked for, the version byte his ruling removed, the closed kind set that ruling
implies, the layout validation an untrusted peer makes necessary, and the size
bounds his own rule about fixed tables deserves.

Pass two (073c762a) is Glenn's decisions of 2026-09-09, all locked, put
into the page and the code: form byte 1 is the variable form, THE FIRST
BYTE is one rule at the head of §3, selection is by the fixed table keyword,
a form byte's refusal says which DIRECTION, and the unknown-form negative
control moved off 3. It is the pass that closes the one deviation and the two
follow-ons pass one reported.

---

Pass two: THE FIRST BYTE

A. Form byte 1 is THE VARIABLE FORM

"so let's rename byte 1" … "don't call it fixed form" … "call it variable
form" — Glenn

"The file form" named the CARRIER and not the class, which left the one word
every other page uses for form 1's class — variable — unspoken, and made
"a fixed table on the file form" a sentence with two forms in it. The registry,
every heading and every line of prose now say the variable form: 36 places
in docs/SPEC-TABLES.md, 102 elsewhere.

Identifier renames NOT done, and why. message_form_as_file keeps file:
it names a file EXPECTATION — a batch handed to a reader that wanted a file
— and not form 1, so renaming it would make it wrong. The C bitfield
retention_message_write_requires_file_form keeps its spelling: it is a
diagnostic identifier every C golden in the tree pins, and the sentence beside
it now says VARIABLE form. Nothing else in the emitters spells form 1 with
File — form 1's constant is kTableWireForm, and fileWire, fileEnums,
SourceFile and the rest all mean a source file.

B. THE FIRST BYTE — one rule, at the head of §3

Stated once, cited everywhere. A schema file is a UNION OF FIVE ARMS and the
form byte is its tag — "so in effect, form byte is like a union id" — under
exactly the append-only rule the baseline lock already enforces for a union in a
fixed table: numbered forever, appended at the end, never reinserted, never
renamed. Byte 0 and every value the registry does not carry are refused by
name.

A form's header:

offset  0        the FORM BYTE
offsets 1 .. 7   RESERVED, written zero
offsets 8 .. 15  the form's EIGHT-BYTE HASH, little-endian
offset  16       the BODY

The hash is over the LAYOUT for the fixed and message forms, over the BUILD for
the cook and the block form. Sixteen bytes is the alignment a memory-mapped body
needs, and the fixed form does not need it and pads anyway — Rowan's ruling
— so the bytes do not move a second time when the cook and the block form join
the registry.

The variable form is the one arm whose header is the form byte alone. It
carries no hash (its id table is the wire's own trailer) and maps nothing, so
§3's "nothing is aligned and nothing is padded" is its whole rule and its body
begins at byte 1. Padding it would move every byte of every file on every disk
to buy an alignment its reader does not use.

The one exception beside the rule: inside a packet no form byte is written.
The declaration names the field a fixed table, and "if we KNOW they're all in
message form, we can just drop the byte for type when we put them in packets" —
so a fixed table in a packet is the message form by rule. Stated as a
standing rule and marked as one: a table has no packet wire today, so nothing
writes such bytes yet.

THE REGISTRY, six rows

form	what it is	where
0	never assigned; refused by name	§3
1	the VARIABLE form — form byte, root body, id table	§3
2	the MESSAGE form — planned to become fixed-only, rebuilt on 3	§3.3
3	the FIXED form	§3.4
4	the COOK — planned	§7
5	the BLOCK form — planned	§19

It lives in code too, in ir/formbyte.go, so the compiler, the CLI and the wire
fuzzer read one table rather than three copies of it.

The fixed form's file header moves — and it is the only wire that does

offset  0        form byte 3
offsets 1 .. 7   reserved, zero
offsets 8 .. 15  the LAYOUT HASH (u64 LE)
offset  16       layout length (u32 LE), then the layout, then the records

It was form byte, u32 layout length, layout, records — a five-byte header.
The records are unchanged, hash and body, and every record still carries its
own eight-byte hash: the header's says which layout is in this file, a
record's says which layout stamped this record. A header whose hash is not the
hash of the layout behind it is refused layout_malformed, and that check runs
last of the three so a broken layout is never reported as a lying header.
The seven reserved bytes are written zero and read by nobody — a later form that
wants to say more takes a new form byte, which is the same rule that keeps a
version byte out of the layout.

C. Selection is by the keyword — the deviation closed

"if we add any feature that stops it from being fixed, it is a compile
error… we don't want to surprise the user" — Glenn

"now that we have the fixed form (new) of 3, we select this, and do not want
fixed tables to ever encode as the old way." — Glenn

A fixed table (#823's keyword) encodes as form 3, always; a table
encodes as form 1; there is no path between them. So:

	what happens
4096	a WARNING, always on, naming the table and the size
65536, DECLARED fixed	a COMPILE REFUSAL by name — a declared fixed table never silently falls back to form 1
65536, merely DERIVED into the fixed mode	the form is dropped and the compiler warns; the table asked for nothing and keeps form 1
--fixed-record-limit N	a project's policy, off by default, and it only ever LOWERS

Pass one deviated here and this closes it. The reason it deviated was real:
§12.1's RenderFrame at 7.5 MB and §2.8's WideBlob at 280 KB are legitimate
fixed-MODE tables, and a hard refusal over derived-fixed tables would refuse
this repo's own corpus. Glenn's decision resolves it by separating the two
words: the MODE is derived (§2.2) and gives those tables their by-value
storage, cook and block form; the FORM is selected by the keyword, and the
bounds are the FORM's. So the refusal is armed on the marker the keyword sets
and the derived branch is what #823 deletes.

#823 is not merged, so it is implemented against what marks a table fixed in
this tree today — the derived mode — with ONE pointer to the keyword, on
ir.Struct.FixedDeclared. Two Go tests set that marker and watch the refusal
fire and the small case stay silent.

--fixed-record-limit cannot RAISE the 65536, and that is the one place I
did not follow the item's wording literally. 65536 is the number a peer's
reader holds this build's records to (layout_record_too_large), and a peer's
build has never heard of this project's flag — the read side always checks, so
the write side must not exceed it. Said in the flag's help, in the code and in
§3.4.

D. A form byte's refusal says which DIRECTION

The registry is ORDERED, so one word for both directions was one word too few.
A new previous_form sits beside newer_form:

a fixed reader given	answers
1, the variable form	previous_form — it is OLDER, and newer_form sent a caller looking for a build that does not exist
2, the message form	message_form_as_file
any byte no form defines	newer_form
— and a variable reader given 3	newer_form, which it already did

The unknown-form negative control moved from 3 to 6. 3 is the fixed
form and 4 and 5 are reserved, so 6 is the first byte that is neither
defined nor reserved. test/tables/main.cpp's test_form_byte_refusals, and
form_three became form_six in reports.txt, MANIFEST.txt, FORMAT.md and
testdata/wire/tables/. The wire fuzzer derives its byte from the registry
(ir.WireFormNone), so the next form added moves it without anyone remembering
to.

E. schema check says what byte 0 says, first

Handed a saved file, schema check prints one line before anything else it says
about that file:

$ schema check saved.bin
saved.bin: FORM 3 the fixed form (docs/SPEC-TABLES.md §3.4)
$ schema check zeros.bin
zeros.bin: REFUSED: form 0 is never assigned (docs/SPEC-TABLES.md §3)

Held by test over every row of the registry, over a byte no form defines, over
an empty file, and over the ordering when a saved file and a unit are both named.

Pass two's gates

gate	result
go build / go vet	clean
golangci-lint run ./...	0 issues
go test ./...	green but for the three pre-existing dialect-scan failures below
make tables-fixedform	green on the new header
make conformance	green over cpp, c, cs, go, rust — report 59/59 including the renamed row
goldens + generated/	re-pinned, and byte-stable across two consecutive regenerations
corpus bytes	only byte 0 of the unknown-form control moved (3 → 6); no other testdata/wire file changed

The three pre-existing failures, reported by pass one and reproduced here at
07bcdb6a with the working tree stashed, untouched:
TestTableHeadersAreTheCDialect, TestTableDialectCheckBites,
TestZeroCostForValueOnlyTables — the fixed form's emitted C++ carries
inline constexpr and template, which §13.9's C-like-dialect scan forbids.

Four legs skipped in make conformance: dart, elixir, java, js.
Each has a pinned toolchain under dist/ that this fresh clone has not
fetched, so they cannot build here; nothing about them is red. cs, go,
rust, c and cpp all ran.

What Glenn's decisions left ambiguous, and what I decided

1. Does form 1's header pad too? The item says "one header rule for all
   five", and also that the hash is "over the layout for the fixed and message
   forms; over the build for cook and block" — form 1 is not in that list, it
   has no hash, and it is the one form the item says must not move. Decided:
   the variable form's header is the form byte alone, and that is stated as the
   one arm the rule does not reshape. Padding it would move every byte of
   every file on disk for an alignment nothing reads.
2. Does form 2's header move now? No. The registry says form 2 is planned
   to be rebuilt on form 3; its framing today is its own form byte and count.
   §3 writes the header down in full anyway, because a rule three forms adopt
   one at a time has to be decided once.
3. Are the seven reserved bytes CHECKED on read? Decided: written zero,
   read by nobody. A later form that wants them takes a new form byte, which
   is the rule that already keeps a version byte out of the layout, so checking
   them buys nothing a form byte does not already buy.
4. --fixed-record-limit "raises" the 65536. Decided: it cannot — see C
   above. This is the one place the implementation departs from the wording, and
   the reason is the write-side/read-side asymmetry.
5. The header pads; do the records end up aligned? No — the body begins at
   16 and the body begins with a variable-length layout, so the records land
   wherever that length puts them. Nothing maps them today. Named as a follow-on
   in §15 so that "the header pads" is not read as "the records are aligned".

---

Pass one: SPEC §3.4, the layout

Why

Glenn read §3.4 and asked:

"What is the block format? Isn't this for the render data in space game? Or
is this the cooked format?"

Three different structures in this document already answer to "block" — §7's
cooked block, §19's block form, §3.3's announced vocabulary — and §3.4 was
calling a fourth one that. It is THE LAYOUT now. Said once at first use,
"the layout (form 1 called this the vocabulary block)", and form 1's
vocabulary block, the cook and the blob are untouched.

And on the version byte the two previous children had started building:

"I go with new form byte."

So there is no version byte inside the layout, and there never will be. The
form byte is read before anything else and versions everything behind it, the
layout's own format included. A layout format change is a NEW FORM BYTE — never
a wider entry, never a byte in front of the count.

The consequence nobody had drawn from that ruling

§3.4 used to argue that an older reader steps over a kind it cannot name by the
size the entry states. It cannot, honestly. If the form byte versions the
layout's format, then it versions the KIND SET too — so a kind outside §3's set
does not mean a newer layout of form 3, it means a NEWER FORM BYTE, which is a
different form. Stepping over it would be trusting a size whose meaning this
reader cannot check. layout_kind_unknown says what happened instead. (Johnny
raised skip-vs-refuse; Rowan ruled refuse; this is the argument for it.)

The layout arrives from an untrusted peer

§3.4 now states in its own body what a reader holds a layout to, each rule a
refusal by ITS OWN NAME, every one before a single record byte is touched: the
entry count fits the length exactly, every kind is in the closed set, every
constant size is one its kind admits and its children account for, every kind is
used as its definition allows, the pre-order walk consumes exactly the entries,
no size overflows or passes 65536, and nothing nests past the walk's own bound.

There is no cycle rule and there cannot be a cycle — an entry's children ARE
the entries that follow it, so a child's index always exceeds its parent's and
there is no back reference to make one of. The property is bought by the
representation, not by a check.

The C++ reference applies all of them, and test/tables/fixedform_main.cpp has
one corrupted-layout case per named rule, each breaking exactly one thing in
a layout the reader accepts.

The size bounds

"Effectively, fixed tables should only be used for small things." — Glenn.
Two numbers and a flag, and they are three different kinds of thing:

	what it is	what happens
4096	ADVISORY, always on	the compiler names the table and the size; the table still carries the form
65536	the WIRE's ceiling	a reader refuses a layout stating more (layout_record_too_large), so the fixed form is not emitted for such a table and it keeps form 1 — and the compiler says so rather than dropping it in silence
--fixed-record-limit N	a PROJECT's policy, off by default	a fixed table past N does not compile; a team that wants Glenn's rule enforced rather than advised sets it to 4096

A note on the middle row. It was a deviation from the item as written, and
pass two closes it — see section C above. The item asked for a compile
refusal past 65536; that would have stopped two of this repo's own corpora
compiling, because §12.1's render frame is a legitimate 7.5 MB fixed table
and §2.8's WideBlob a 280 KB one and neither has ever been a form-3 table.
Glenn's ruling separates the MODE from the FORM: the bound is the FORM's, a
DECLARED fixed table past it does not compile, and a table merely derived into
the fixed mode keeps form 1 and warns.

Also stated, each because a reader had to work it out

• The framing in all three carriers, in one table. The form byte and the
  layout appear once per carrier; a record carries only the hash. A file carries
  both up front, a stream's layout rides the announcement, and the message form's
  form byte is once per batch.
• The slack rule, on every row that has slack — ZERO on write, UNSPECIFIED
  on read, never a refusal; the reader validates the USED UNITS ONLY, and
  the content rules apply to those and to nothing else. The unit of every length
  is now on its own row: bytes for string(N) and bytes(N), UTF-16 code units
  for wstring(N).
• A union's tag 0 is None, as an enum ordinal 0 is; the slack behind a
  narrower arm is zero on write and ignored on read. An absent optional's
  payload is likewise ignored on read and zero on write.
• A DEPRECATED field keeps its slot forever (schema lang: fixed table declares the class, and the compiler refuses what breaks it #823, schema lock: fixed tables evolve append-only, and the compiler holds the line #825): the writer emits
  the declared default and the reader yields it. Keeping the slot is what makes
  the deprecation cost nothing.

Where the arithmetic lives

C(f) and C(T) move to ir/tablefixed.go, beside the kind vocabulary, because
a record's constant size is wire law and not a backend's. The reference emits
its layout and its template from the same functions the warning, the ceiling and
the reader's own bound are measured with, so the bytes and the verdict cannot
drift apart.

The bytes do not move

Every layout byte array and every FixedHash this branch emits was diffed
against the same units generated by the compiler at 3bf940e0:

FX1: 2 layouts, BYTE-FOR-BYTE IDENTICAL (195 bytes)
FX2: 3 layouts, BYTE-FOR-BYTE IDENTICAL (301 bytes)
P1:  2 layouts, BYTE-FOR-BYTE IDENTICAL (144 bytes)
V1:  2 layouts, BYTE-FOR-BYTE IDENTICAL (1198 bytes)
bench/corpus paired: hashes identical

Only the comments beside them changed. Removing the version byte is what put
them back where 508b84e2 had them.

What this pass found that the two reads did not

Two things §3.4 asserts that the tree does not do, named in §15 rather than
left as text nobody had checked. Pass two closed the first and half of the
second:

1. §4.2's unknown-form negative control still plants 3. §3.4 already says
   it must move to 4, because the control's whole premise is a byte no form
   defines — and 3 is now the fixed form. It still passes (a form-1 reader
   does refuse byte 3), so nothing is red; what it no longer does is test
   what it says it tests. Moving it renames the shared conformance row
   form_three that every leg reads, so it is a change of its own and §3.4 now
   says it has NOT moved yet. CLOSED in pass two: it plants 6.
2. A generated reader does not dispatch on the form byte. §3.4 says a reader
   for a fixed-table type "accepts BOTH, by the form byte". The reference emits
   two entry points instead, and the fixed one answers newer_form for byte
   1 — which is OLDER. The page's rule is right; the reference is one
   dispatch short of it. The NAMING half is closed in pass two
   (previous_form); the dispatch itself is still a named follow-on.

Three gates that were red on the base commit, before this branch

Verified by regenerating and re-running at 3bf940e0 with the working tree
stashed. None is caused by this pass; two are re-pinned here because a
green branch needs them, the third is left alone.

gate	state at 3bf940e0	here
TestGolden* + the generated/ tree	the same 44 generated files and 45 golden headers regenerate differently — the fixed form landed in 508b84e2 and neither was re-pinned	RE-PINNED, and regeneration verified byte-stable across two runs
TestTableHeadersAreTheCDialect, TestTableDialectCheckBites, TestZeroCostForValueOnlyTables	RED — the fixed form's emitted C++ carries inline constexpr and template, which §13.9's C-like-dialect scan forbids	still red, untouched: it is the fixed form's emission dialect and belongs with the branch that introduced it
make bench-paired-check	RED — paired corpus mismatch: bench/paired/corpus/bench_table.bin	still red, untouched, same reason

One further red was purely environmental and is not a repo problem: this clone's
sibling ../serialize was at v1.8.0 and ../serialize.rs was absent.

---

🤖 Generated with Claude Code

[gafferongames]

Emma Audit & Sign-off on §3.4 (fixed-form-spec-pass)

I have audited PR #829 against Emma's eight review findings from emma-63922dca1393 and the C# match-bytes implementation in PR #827:

1. Little-endian, byte widths, no alignment padding: Cleanly stated. (§3.4: "EVERY FIELD RIDES AS ITS DECLARED STORAGE IMAGE, LITTLE-ENDIAN, AT ITS DECLARED STORAGE WIDTH, AND NOTHING IS PADDED BETWEEN FIELDS.")
2. Layout format version byte removed: Exactly reflects Glenn's ruling. Stated in §3.4: "THERE IS NO VERSION BYTE INSIDE THE LAYOUT, AND THERE NEVER WILL BE. THE FORM BYTE VERSIONS THE LAYOUT'S FORMAT: a change to the layout's format is a NEW FORM BYTE, never a wider entry and never a byte in front of the count."
3. Carrier framing unified: Comprehensive framing table provided for file, stream, and message batch. (File carries form byte + layout up front; records carry 8-byte hash + body; packet carrier omits form byte).
4. Slack rule on strings/bytes: Zero on write, unspecified on read, never a refusal; reader validates used units only.
5. Union None = tag 0, Optional payload, Kind 35:
   • Union tag 0 is strictly None.
   • Absent optional payload ignored on read, zero on write.
   • Kind 35 (TableWireKindOptionalWrapper) documented in the layout child walk table.
6. Deprecated fields: Explicitly documented that deprecated fields retain their constant slot forever, writer emits declared default, and reader yields it.
7. Size bounds: Cleanly tiered:
   • 4096: Advisory compiler warning.
   • 65536: Wire ceiling refusing the fixed form for that table (layout_record_too_large), preserving form 1.
   • --fixed-record-limit N: Project policy flag for strict project-level enforcement.
8. Untrusted peer layout validation: Complete, enumerated validation rules executing before record bytes are touched, with named refusals.

Wire Hashes & Byte Invariance

Verified that layout byte sequences and hashes are 100% byte-for-byte identical with the C++ reference and C# implementation in PR #827 (FxNestedFixedHash, FxRootFixedHash, FX2 FxRootFixedHash, V1 CfgFixedHash, V2 CfgFixedHash, P1 ChainFixedHash, P3 ChainFixedHash).

Verdict: APPROVED & SIGNED OFF.

[gafferongames]

Emma Audit Sign-Off: PR #829 at 073c762

Audited commit 073c762a against SPEC §3.4 and Glenn's architecture directives:

1. SELECTION IS BY THE KEYWORD:

   • Explicitly establishes that declared fixed table encodes as Form 3, always.
   • Clarifies that undeclared table remains unbounded Form 1 variable table, cleanly separating structural Mode from wire Form.
   • Confirms compile refusal for lookback guards, pointers, maps, and unbounded arrays on declared fixed tables per Glenn's ruling.
   • Pinned 65,536 limit to declared fixed tables, with --fixed-record-limit strictly lowering it.

2. The Unified 20-Byte File Header:

   • Bytes 0..7: Form byte 3 at offset 0, bytes 1..7 zeroed padding.
   • Bytes 8..15: Little-endian u64 layout hash at byte 8.
   • Bytes 16..19: Little-endian u32 layout length at byte 16.
   • Bytes 20+: The Layout bytes, followed immediately by records (8-byte hash + body).

3. Form Refusals & Registry:

   • Order-aware refusals: previous_form for Form 1 inputs on fixed readers, newer_form for unallocated/future form bytes, message_form_as_file for Form 2.
   • Form byte 6 used for unknown-form negative control.
   • ir.WireFormOf and ir.WireFormLine verified cleanly in ir/formbyte_test.go.

4. Plan Operations:

   • Ops table cleanly specifies copy, count, text, union, widen, clamp, ordinal.

All tests passing (go test ./ir/..., go test ./compiler -run TestTable).

VERDICT: APPROVED / SIGN-OFF