schema lock: fixed tables evolve append-only, and the compiler holds the line

[gafferongames]

Glenn's decisions

Fixed tables evolve append-only, enforced by a lock file the compiler owns.
His words:

"append only, and deprecation possibly. might keep it light weight?"

On the mechanism:

"OK that sounds cool, go ahead."

And then, of the mechanism itself:

"Is there anything schema check cannot catch in a fixed table? Can we fix
that so it does?"

That question is what the lock's second half answers: the order and the widths
are what a reader stands on ONLY if nothing else about a field can move
underneath it, and six things could.

Network Next did this by hand for years — a uint8 version at the front, bump
on change, append at the bottom, manual upgrades for a limited time:

"manual stuff, and it is a footgun, but it worked."

It worked because the discipline is right. It was a footgun because nothing
enforced it. This PR is that discipline with the footgun taken away.

Why fixed tables need it

A variable-length table rides the id-table wire, where evolution is wire
tolerance: fields may be added, removed and reordered, and the reader finds
each one by its id. A fixed-size table has none of that. It is a plain C
record — no ids, no terminators, no names, walked by offset — so the field
ORDER and the field WIDTHS are the contract. A reordered field, a removed
one, a widened one: every one of those is read as garbage by a reader holding
an older record, silently, with no counter to fire.

The mechanism

1. schema.lock, committed beside the schemas at the unit root, written by
the compiler only. For every fixed table, its field sequence in declared order
— each entry carrying the field's id, kind, width, default
declared or implicit, declared range and resolution, ? and
deprecated marker — and then a block for every type those tables
reach: a nested type as a record of its own, and an enum, a flags mask
or a union as its value list in declared order. Each block carries the
hash of its lines.

fixed table ShipConfig
{
    name     string(32)
    speed    float32 | min = 0, max = 100, resolution = 0.01
    armor    uint8 | deprecated
    shields  uint8 = 3
    hull     Hull = Gunship
    perks    Perks
    lanes    [4]Hull
    by_hull  [Hull]uint16
    gunner   ?GunnerSettings
}

schema-lock 5
package fleet

fixed table ShipConfig layout=0x24c95e6d7dc599a2
    field name id=0xc4bcadba8e631b86 kind=12 width=40 default=bytes:
    field speed id=0x2281498aa0200e40 kind=10 width=4 default=0.0 min=0.0 max=100.0 res=0.01
    field armor id=0xd19988b67e699194 kind=6 width=1 default=0 deprecated
    field shields id=0x798c587767067199 kind=6 width=1 default=3
    field hull id=0x80da8ccc11daadf6 kind=7 width=1 default=variant:Gunship held=Hull@0xc3cc3881c580dd9d
    field perks id=0x4b06f5847096e54a kind=9 width=8 default=mask:0 held=Perks@0x8e31a11d8a742b6a
    field lanes id=0xd14829ec544a2fb4 kind=14 width=4 default=born:0 held=Hull@0xc3cc3881c580dd9d elem=7/1
    field by_hull id=0x17e7d894ce9c89da kind=16 width=6 default=born:0 elem=7/2 key=Hull@0xc3cc3881c580dd9d
    field gunner id=0x40dbb648c0cd44aa kind=13 width=9 default=zero held=GunnerSettings@0xdd5ef1d77231ba48 optional

type GunnerSettings layout=0xdd5ef1d77231ba48
    field reaction id=0xb75aa3662201646a kind=10 width=4 default=0.2
    field tracking id=0xa6bf719a4602b0bc kind=1 width=1 default=false

enum Hull values=0xc3cc3881c580dd9d
    variant Interceptor
    variant Gunship
    variant Freighter

flags Perks values=0x8e31a11d8a742b6a
    variant Shielded
    variant Cloaked

What the lock holds, in full:

fact	why a reader misreads without it
field id, kind, width, in declared order	the record has no ids in it; position and width are the walk
the DEFAULT, declared or implicit	a deprecated slot holds it, an absent field on the id-table wire reads back as it, and a reader handed an older writer's record fills the missing field from it
the RANGE — min, max, a compressed float's res	the bounds are the scale a stored value is read back at
a fixed-point field's frac	fixed(16,16) and fixed(24,8) are one width under one kind and read one stored number as two different values
the ?	an optional carries a presence bool INSIDE the record; a reader expecting one where none is written takes the next field's first byte for the answer
the deprecated marker	one-way; readers were told to ignore the slot
every enum and flags type reached, value list in order, append-only	in a fixed record an enum rides as its dense ordinal and a flags variant is its bit position: the list is the mapping from a stored number to a meaning
every union reached, arm order, append-only	the tag is the arm's position
every type in the closure, not the root alone	a nested type's fields ARE the holder's bytes, so a field retyped one level in moves what a reader of the root gets back without moving one line of the root's entries
which named type a slot holds — held=Hull@0x…, on kinds 13, 15, 7, 9 and an array of any of them	a slot repointed from one type to another of the SAME WIDTH moves no byte and no other fact: Hull → Cargo reads a stored ordinal 1 back as Ice where Gunship was written
an array's element kind and width — elem=7/1, on kinds 14 and 16	the bound and the total width can both hold while each element becomes something else
which enum a keyed array is keyed by — key=Hull@0x…, on kind 16	the key's variants ARE the slots, in declared order, so the list is what says which slot a value was written into

The first row is the whole of the original revision. Everything under it
arrived since, and the rendering version has moved with each widening —
1 → 2 → 3 → 4 → 5 — because every one of these facts sits on a
hashed line, and a lock that does not carry them cannot be repaired by hand.
Each bump makes every committed lock stale at once, deliberately and visibly.

The name is compared, not the hash. Two enums with identical value lists
hash identically — enum Hull and enum Rank over the same three variants
are both values=0xc3cc3881c580dd9d — so held= and key= carry the wire
NAME beside the hash, and it is the name the diff compares. A slot repointed
between two structurally identical types is still a slot that means something
else.

The one class it cannot catch, and the spec says so. A field that keeps its
name, its id, its kind, its width, its default, its range and its ? and
changes only what it MEANS — timeout counted in seconds becoming timeout
counted in milliseconds, an index counted from zero becoming one counted from
one — moves no fact the lock records and no byte a reader could compare. The
rule for it is the rule for every other break: deprecate and add.

An append inside a nested record is not an append. A field at the bottom of
a type a fixed table holds by value grows the holder's slot, which slides
every field after it, so the compile reports both halves: the holder's width
first, and the nested record's own new entry beside it. A nested record has no
bottom of its own; the bottom belongs to the table.

The width is the field's WHOLE storage in the record — an array's elements
together, a string(N)'s buffer and its length companion, an optional's value
and its presence bool — because that is the fact a reader standing at an offset
is standing on. The layout hash is derived and written down anyway: it and the
entries above it are one statement made twice, so a lock whose halves disagree
has been edited by a hand rather than written by the compiler.

2. The check, on every compile (schema check, schema generate): THE
LOCK IN THE TREE IS THE LIVE SEQUENCE, entry for entry, value for value, block
for block. The
APPEND-ONLY rule says what a fixed table may BECOME; the check says the
committed file must BE what the unit declares today, so a lock the declaration
has moved past — an appended field, a new fixed table, a deprecated marker
not written down — is STALE and refused like any other drift, naming the table,
the first entry the lock lacks, and the one command that fixes it. Every change
to a fixed table therefore lands in the same commit as the record of it. No
file, no check.

3. schema lock rewrites the file and only ever appends entries and
values, adds blocks or flips deprecated on. It runs the check's own
comparison first and refuses every break the check refuses — a reorder, a
removal, a widening, a kind change, a moved default, a moved range, a flipped
?, a reordered value list, an un-deprecation, a table withdrawn, a type gone
from the closure, a hand-edited file — so the
one command that moves the file cannot be the one that breaks the rule. The ONE
difference between the two readings is the one this command exists for: where
the check refuses a declaration the lock has not caught up to, this command
writes it down. --print writes nothing.
A lock written under an OLDER rendering version is the one file this command
will not repair — the version is the compiler's own, and unlike a baseline
there is no history to salvage — so its refusal says the remedy that works:
… delete it and write it again with schema lock``. Every bump in this branch
keeps that path live, and it is walked twice by test: once from a two-line
stub, and once from a WHOLE well-formed lock one version back, which is the
file a person actually meets. Both verbs name the one remedy, schema lock
writes nothing over the file it cannot read, and the remedy reproduces the
current rendering byte for byte.

There is no --reason and no history, unlike tables-baseline --update:
moving a baseline declares a break with data already written, while every write
a lock accepts is an append, and an append breaks nothing.

4. | deprecated, a TAG (the valueless half of the qualification section —
there is nothing for it to take a value of). The slot stays exactly where it
is, the writer emits the field's default, the reader ignores it, and the one
NEW USE the compiler refuses is a guard: an if condition may not name a
deprecated field. The other half of "guard or key" the language already gives
for free — a map key takes no qualification at all (§2.8), so a key can never
carry the marker. It is one-way.

5. was = "old_name" keeps the field's id, and the lock records WIRE
names, so a rename moves not one line of the file.

6. Compaction is a new fixed table under a new name. No mechanism, one
sentence in the spec.

The refusals, verbatim

Reorder / insert before the end / removal from the middle:

schema.lock: fixed table ShipConfig: entry 2, field speed (id=0x2281498aa0200e40), is field armor (id=0xd19988b67e699194) in the declaration — a fixed table evolves APPEND-ONLY: a field is added at the BOTTOM, never inserted, moved or removed, because the record has no ids in it and a reader at this offset would read one field as the other (docs/SPEC-TABLES.md §2.10)

Removal from the end:

schema.lock: fixed table ShipConfig: entry 3, field armor (id=0xd19988b67e699194), is in the lock and gone from the declaration — a fixed table evolves APPEND-ONLY: a field is deprecated in place, never removed, because the record has no ids in it and every field after a removed one slides (docs/SPEC-TABLES.md §2.10); restore it and mark it `| deprecated`

Width change (checked before the kind, because it is the fact that slides every field after it):

schema.lock: fixed table ShipConfig: entry 3, field armor (id=0xd19988b67e699194), is 1 bytes wide in the lock and 4 in the declaration — a field already in the lock keeps its width: a fixed record is walked by offset, so widening one field moves every field after it (docs/SPEC-TABLES.md §2.10); deprecate this field and append a new one

Kind change at the same width (float32 read as int32 moves no byte and still lies):

schema.lock: fixed table ShipConfig: entry 2, field speed (id=0x2281498aa0200e40), is kind 10 in the lock and kind 4 in the declaration — a field already in the lock keeps its type: every record already written holds the old one, and nothing on the wire says which (docs/SPEC-TABLES.md §2.10); deprecate this field and append a new one

Un-deprecation:

schema.lock: fixed table ShipConfig: entry 3, field armor (id=0xd19988b67e699194), is deprecated in the lock and live in the declaration — deprecation is ONE-WAY: readers were told to ignore this slot and the writers that have run since left it at its default, so what comes back is not data (docs/SPEC-TABLES.md §2.10); append a new field instead

A moved default, declared or implicit:

schema.lock: fixed table ShipConfig: entry 4, field shields (id=0x798c587767067199), defaults to 3 in the lock and 7 in the declaration — a field already in the lock keeps its default: an older writer's missing field is filled from this default, and a deprecated slot holds it, so a record written before the change reads differently after it (docs/SPEC-TABLES.md §2.10); deprecate this field and append a new one

A moved range, resolution or fixed-point scale — none of which moves a byte:

schema.lock: fixed table ShipConfig: entry 2, field speed (id=0x2281498aa0200e40), is [0.0, 100.0] at resolution 0.01 in the lock and [0.0, 100.0] at resolution 0.001 in the declaration — a field already in the lock keeps its range: the bounds and the resolution are the scale a stored value is read back at, so moving them reads every record already written as a different number (docs/SPEC-TABLES.md §2.10); deprecate this field and append a new one

schema.lock: fixed table ShipConfig: entry 7, field tilt (id=0x1e5088ef2e9bafc6), is [-8, 8] at 16 fractional bits in the lock and [-8, 8] at 8 fractional bits in the declaration — a field already in the lock keeps its range: … (docs/SPEC-TABLES.md §2.10); deprecate this field and append a new one

A flipped ?, checked before the width because it is the more exact account of
the same move:

schema.lock: fixed table ShipConfig: entry 6, field gunner (id=0x40dbb648c0cd44aa), is optional in the lock and plain in the declaration — a field already in the lock keeps its `?`: an optional carries a presence bool beside its value INSIDE the record, so a reader that expects one where none is written takes the next field's first byte for the answer (docs/SPEC-TABLES.md §2.10); deprecate this field and append a new one

A reordered, renamed or inserted enum variant, flags variant or union arm:

schema.lock: enum Hull: variant 1, Interceptor, is Gunship in the declaration — an enum, a flags mask or a union a fixed table reaches evolves APPEND-ONLY: a new variant goes at the END, and one already in the lock is never inserted, moved or renamed, because a fixed record stores the PLACE and a reader would read one value as the other (docs/SPEC-TABLES.md §2.10); restore it, and add the new one at the END

A change one level in, inside a nested type — the root's own entries do not
move one line:

schema.lock: type GunnerSettings: entry 1, field reaction (id=0xb75aa3662201646a), is kind 10 in the lock and kind 4 in the declaration — a field already in the lock keeps its type: every record already written holds the old one, and nothing on the wire says which (docs/SPEC-TABLES.md §2.10); deprecate this field and append a new one

A slot repointed at a different type, at the same width and with every other
fact of the entry unmoved. One sentence covers a nested record, a union, an
enum, a flags mask and an array of any of them, because it is one break:

schema.lock: fixed table ShipConfig: entry 2, field hull (id=0x80da8ccc11daadf6), held Hull in the lock and Cargo in the declaration — a field already in the lock keeps the type it holds: every record already written holds the old one, and nothing on the wire says which (docs/SPEC-TABLES.md §2.10); deprecate this field and append a new one

An array's element respelled at the same total width — a [4]Hull that became
[4]Cargo, or a [Hull]int32 that became [Hull]float32:

schema.lock: fixed table Slots: entry 1, field by_hull (id=0x17e7d894ce9c89da), holds int32 elements in the lock and float32 elements in the declaration — a field already in the lock keeps its element type: every record already written holds the old one, and nothing on the wire says which (docs/SPEC-TABLES.md §2.10); deprecate this field and append a new one

A keyed array pointed at a different key enum. Hull and Cargo have three
variants each, so the slot is the same width and the slot COUNT is the same
too; what moved is which variant each slot belongs to:

schema.lock: fixed table Slots: entry 1, field by_hull (id=0x17e7d894ce9c89da), is keyed by Hull in the lock and Cargo in the declaration — a field already in the lock keeps the enum it is keyed by: the key's variants ARE the slots, in declared order, so every record already written put its values in the slots the old list numbered (docs/SPEC-TABLES.md §2.10); deprecate this field and append a new one

A type gone from the closure:

schema.lock: enum Hull is in the lock and this unit's fixed tables no longer reach it — the lock holds every type a fixed record is made of, so a type leaving the closure is a field that changed what it holds; restore it, or deprecate that field and append a new one (docs/SPEC-TABLES.md §2.10)

An APPEND to an enum, allowed by the rule and still refused until it is
written down — schema lock then records it, with every line before it
untouched:

schema.lock: enum Hull: variant 4, Tanker, is in the declaration and not in the lock — the lock is the committed record of what a fixed table's readers stand on, and this declaration has moved past it: an append, a new block and a deprecation are the changes the rule allows, and a change the rule allows is still a change the record must carry (docs/SPEC-TABLES.md §2.10); write it with `schema lock`

A fixed table that is gone:

schema.lock: fixed table Slot is in the lock and this unit no longer declares it as a fixed table — a fixed table's layout is a promise to every record already written, and a promise is not withdrawn; compaction is a NEW table under a NEW name (docs/SPEC-TABLES.md §2.10)

A lock hand-edited to lie:

schema.lock: fixed table ShipConfig: the lock records layout=0x913d299299757f2a over entries that hash to 0x9132e392996cb3b3 — the lock is written by the compiler and a hand-edit does not hold; regenerate it with `schema lock` and make the schema change you meant instead (docs/SPEC-TABLES.md §2.10)

A stale lock — an appended field, an appended variant or arm, a new block, a
deprecated marker the file has not caught up to. One sentence, one remedy; only the clause naming the difference
varies:

schema.lock: fixed table ShipConfig: entry 4, field shields (id=0x798c587767067199), is in the declaration and not in the lock — the lock is the committed record of a fixed table's layout, and this declaration has moved past it: an append, a new table and a deprecation are the changes the rule allows, and a change the rule allows is still a change the record must carry (docs/SPEC-TABLES.md §2.10); write it with `schema lock`

A deprecation reads … is deprecated in the declaration and live in the lock,
and a new block with no line yet has none to name:
fixed table Marker is in the declaration and not in the lock — ….

A guard on a deprecated field:

if condition active names a field marked `| deprecated` — a deprecated field is retired in place: its slot stays, nothing new may name it, and a guard is a NEW USE (docs/SPEC-TABLES.md §2.10); guard on a live field, or drop the marker

Which tables are FIXED

origin/fixed-table-keyword does not exist yet, so this branches from main
and derives the set with !ir.VariableTables(u)[name], in one place —
lockfile.FixedTables — carrying a TODO(fixed-table-keyword) naming the
declared flag. When the keyword lands, the switch is one line in that
function's body.

Locks added (15)

examples-wide, tables/arms, tables/backend, tables/block,
tables/blockhome, tables/examples, tables/lists, tables/maps,
tables/messages, tables/pointers, tables/scalars, tables/stream,
tables/vocab, tables/vocab9, test/table-base64.

Every other unit in the tree declares no fixed table and carries no lock; a
TestEveryUnitWithFixedTablesIsLocked keeps that register honest and fails the
day a unit grows its first fixed table without one. test/tables/ holds many
single-file units in one directory (SPEC §3.2), so it has no place for a unit
lock and is not covered.

Held by test

internal/lockfile runs the whole mechanism over a fixture package, both ways:
append / new table / new fieldless table / deprecated → the check REFUSES
until schema lock writes it, then passes, and the write is an append (the old
entries are a prefix of the new ones, and ir.RecordLayout proves a
deprecation moved no byte); reorder / removal (middle and end) / widen / kind
change / insert in the middle / un-deprecation / a table withdrawn / a
hand-edited lock → each refused with the entry named; a reorder, a removal and
a kind change are refused BY BOTH VERBS with the file left alone; was =
rename → not a change, so nothing to catch up to and the file does not move;
the driver's own load path refuses.

compiler/lockdeprecated_test.go generates the same schema with and without
the marker across cpp / c / go / cs and requires the outputs to be identical
but for the reflection descriptors' tags column — so the slot is still
declared, still initialized to its default, its offset assertion unchanged, and
still written, and the claim assumes no particular wire.

internal/lockfile/lockclosure_test.go is the widening, measured on a second
fixture built in TWO HALVES that mirror each other field for field: a FIXED
Config and everything it reaches, beside a VARIABLE-LENGTH Bag (it holds a
*Bag) and a closure only it reaches. Every fact the file gained is measured
three ways — TestClosureBreakRefused, TestClosureBreakOnAVariableTableIsNotTheLocksBusiness,
TestClosureBreakRefusedByLockToo — over twenty-one edits: a moved default, a
default given to a slot that had none, a widened integer range, a moved
resolution, a moved fixed-point scale at the same width, a ? turned on, a ?
turned off, a kind change inside a nested type, a nested slot repointed at
another type, an optional nested slot the same way, a union arm's payloads
swapped with the names kept, an array's element type, an enum slot repointed,
a flags slot repointed, an array of enums repointed, a keyed array's key enum
repointed, a keyed array's element respelled at the same width, and a
reordered, renamed or removed enum variant, flags variant or union arm. Each is refused on the
fixed half with the line named; each of them on the variable-length half
passes in silence and moves no byte of the file, because a table with a
pointer in it rides the id-table wire where those edits are what the wire is
FOR; and schema lock refuses every one with the same sentence and leaves the
file exactly as it sat.

TestClosureAppendRefusedUntilLocked runs the appends the rule allows — a
variant at the end of an enum, of a flags mask, an arm at the end of a union
— refused as STALE and then written by schema lock, with every line before
them untouched and the file strictly longer.
TestNestedRecordAppendSlidesTheHolder is the case that is NOT an append: two
refusals, the holder's width first and the nested record's entry beside it.
TestTypeLeavingTheClosureRefused covers all four spellings (type, enum,
flags, union); TestHandEditedValueListRefused catches a value list edited
away from its hash; TestLockHoldsTheWholeClosure and
TestValueListRoundTrips pin the file's shape and its parser; and
TestALockFromAnOlderRenderingSaysWhatToDo walks the version bump's own path —
refused by both verbs with one remedy, and locking cleanly once it is taken.

internal/lockfile/corpus_test.go regenerates every committed lock byte for
byte and compiles each of those units with the check on. All fifteen committed
locks were regenerated under the widened rendering: the corpus's fixed tables
between them reach 99 fixed tables, 16 enums, 3 flags masks, 6 unions and 14
nested type records, every one of which is now held — and eight enum-keyed
array entries across tables/examples, tables/block and tables/scalars
now name their key and their element.

Gates

• go build ./..., go vet ./..., go test ./... — clean
• make check — clean over the whole corpus
• generated/ regenerated with this compiler — byte-identical, zero drift
• golangci-lint — 0 issues; modernize — clean
• the fifteen committed locks regenerate byte for byte, twice: in place, and
  deleted and rewritten from scratch
• make shape-gate — clean
• PORTING register gate (compiler/porting_test.go) — green
• make check over the tree's committed locks — clean; they are current, and
  under the strict reading that is now a claim the gate makes

Each new comparison was sabotaged to prove its row is load-bearing. Drop
the *ir.Enum arm from held() and the enum-slot and enum-array rows fail
and nothing else does; drop *ir.Flags and the flags row fails; stop the
renderer recording key=, or stop the diff comparing it, and the keyed-array
row fails; take kind 16 out of the elem= arm and the keyed-element row fails.
Every one of them turned exactly the intended rows red and left the rest
green, and the sources restored byte-identical after each.

🤖 Generated with Claude Code

[gafferongames]

Coordinator note (Rowan): take the STRICT reading. A schema whose fixed tables have drifted from the lock in any way, including a plain append, fails schema check with a sentence that says to run schema lock; the lock is the record and it must always be current in the tree. schema lock remains the only writer and still refuses anything but an append or a deprecation flip. Change in lockfile.Diff plus one refusal sentence and one test; land it on this branch before the three (keyword #823, this, the new form) are combined. Also: rebase onto fixed-table-keyword when it exists and drop the TODO.