Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,66 @@ jobs:
- name: the tables goldens match a fresh generation (the block zero-cost gate, byte-identity half)
run: make tables-block-zero-cost

# THE PINS ARE CURRENT (issue #461). The job above asks this question of
# generated/ and asks it of nothing under testdata/, so a pin regenerated
# against an older main auto-merges clean and every job stays green against a
# pin nobody wrote. That is not hypothetical: this job's first run found six,
# testdata/golden/tables/maps/*Table.h, pinned on a branch cut before the
# retain-unknown walk landed and merged after it.
#
# THE SHAPE IS THE GENERATED JOB'S, one door over. Every re-pinning rule takes
# bin/schema, the C++ reference and the conformance harness, so this job
# rewrites the whole of testdata/ IN PLACE and staleness is precisely a dirty
# tree afterwards: a tracked pin that changed, or a newly written pin nobody
# committed, which a diff-only check would miss.
#
# THE TARGETS ARE THE TREE'S OWN NAMES, `conformance-pin` and
# `update-goldens`, so a rule that starts writing a new pin is covered the
# moment it joins one of them, with no edit to this file. Locally the same
# check is those two targets and the same `git diff`.
#
# ONE SIBLING, ../serialize, because the legs that write the wire pins are the
# C++ reference's and every other language's half of `update-goldens` is
# `bin/schema generate` and a copy. Its pin is the shared SERIALIZE_TAG above.
#
# MEASURED at 1m44s on the run that introduced it (34113070433), against
# 4m27s for big-endian in the same run: this job is inside the two-minute
# rule and it is not the binding constraint. Its own budget is the C++
# reference binaries `update-goldens` builds plus that target's closing
# `go test ./...`, which is the duplicate of go-test's and the first place
# to look if this job ever wants the time back.
pins:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Check out the C++ serialize runtime (pinned release)
run: |
cd ..
git clone --quiet --depth 1 --branch "$SERIALIZE_TAG" https://github.com/mas-bandwidth/serialize.git serialize

- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version: '1.26'
cache: false

- name: rewrite every pin under testdata/ from the reference
run: make -j"$(nproc)" conformance-pin update-goldens

- name: the pins are current
run: |
git diff --exit-code -- testdata/ || {
echo "::error::a pin under testdata/ is stale: the current compiler and the C++ reference write different bytes. Run 'make conformance-pin update-goldens' and commit what changes."
exit 1
}
untracked=$(git status --porcelain testdata/)
if [ -n "$untracked" ]; then
echo "$untracked"
echo "::error::the re-pin wrote files that are not committed under testdata/. Add them."
exit 1
fi

# THE CONFORMANCE MATRIX (test/conformance/README.md), one job per registered
# driver. The same corpus as data, one driver per language, and the matrix
# that says which surfaces a backend has: cross-language bit identity is the
Expand Down
39 changes: 25 additions & 14 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# schema — working conventions for sessions in this repo

- **docs/SPEC.md is the one source of truth, written as a clean reference.** It states the
- **docs/SPEC.md is the source of truth for the type wire, and docs/SPEC-TABLES.md is
the source of truth for the table wire.** Both are written as clean references, and the
two wires share one language, one unit and one compiler (SPEC.md §1 non-goals). Each states the
most recent specification only — present tense, reference register, no history, no
decision narration (Glenn's directive, 2026-08-18: SPEC must read for a human
implementer, not like a CLAUDE.md). Decision provenance — who ruled what, when, in
Expand Down Expand Up @@ -65,19 +67,28 @@ The long arc (Glenn, 2026-08-04): schema as the single data-definition language,
the opinionated layers built on top of it living elsewhere. The boundary, in Glenn's
words (2026-08-25): *"schema is types and bitpacking and enums and constants."*

- **The table layer left the language (2026-08-25).** Tables, collections and
the JSON data compiler are not part of schema: the language is the realtime
wire — hardcoded structs, one protocol id, same-or-refuse — and content
pipelines are out of scope. `table` stays a reserved word and the parser
refuses it by name.
- **The protocol layer left the language (2026-08-26).** Messages, objects,
the view markers, quantize, round and contexts are not part of schema: the
free offering is types, enums, flags, unions, constants and bitpacking — a
pure data contract, zero protocol conventions. `message` and `object` stay
reserved words and the parser refuses them by name; the projection carries
frozen `message=false` and `round=nearest` tokens beside `table=false` so
the refusals moved no protocol-free unit's id. The positioning is
empowerment: build your own message types with enums and unions.
- **The table layer is part of the language.** `table` declares a data type on
the evolution-tolerant table wire, and docs/SPEC-TABLES.md is its normative page
beside docs/SPEC.md. `type` is the realtime wire (hardcoded structs, one protocol
id, same-or-refuse) and `table` is the tolerant one (identity by name hash, unknown
fields skipped, absent fields defaulted, any reader reads any data). Collections
live in a `table` body alone: the unbounded `map[K]V` and `[]T` are declarable there
and refused by name in a `type` (SPEC.md §1, SPEC-TABLES.md §2.8, §2.9). The data
path is schema's too, as `schema pack`, `unpack`, `cook`, `cook-check` and `uncook`
over a JSON tree. ROADMAP.md's table-wire rows carry each backend's state, C++ the
reference and the other eight ports behind it.
- **The protocol layer stays out of the language.** Messages, objects, the view
markers, quantize, round and contexts are not part of schema: the offering is
types, enums, flags, unions, constants and bitpacking on the type wire, and tables
on the table wire, with zero protocol conventions on either. `message` and `object`
are reserved words the parser refuses by name, and `contexts` is refused at file
scope. The packet projection carries frozen `table=false` and `message=false`
tokens on every `type` line and a frozen `round=nearest` on every compressed-float
line, which hold every existing unit's id stable; moving either is a
ProjectionVersion bump, taken deliberately or not at all. The positioning is
empowerment: build your own message types with enums and unions. The table wire's
own MESSAGE FORM (SPEC-TABLES.md §3.3) is a form byte on that wire and declares
nothing, so it takes no keyword and refuses none.
- **The delta pass** stays out of scope here (SPEC's non-goals): schema
declares types; delta encoding is an application's own layer.
- **Constants migrate through a temporary duplicate set** (schema + flatbuffers) while
Expand Down
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -248,8 +248,8 @@ build/tables-generated/.stamp: bin/schema $(SCHEMAS_WIDE) $(SCHEMAS_TABLES) $(SC
# closure must pay NOTHING for the pointer machinery — no builder, no arena, no
# handles, no lifecycle surface, no extra descriptor columns. The pointer-free
# corpus's generated headers must not contain one symbol of it. (The stronger
# one-time proofbyte-identical emission against the pre-pointer baseline
# is recorded in the round log; this is the standing gate.)
# one-time proof, byte-identical emission against the pre-pointer baseline,
# lives in git history and in no page of this tree; this is the standing gate.)
#
# THE MAP MACHINERY TAKES THE SAME GATE (docs/SPEC-TABLES.md §2.2, §2.8): "not
# one symbol of the map machinery in a map-free unit's generated headers, held
Expand Down
46 changes: 30 additions & 16 deletions docs/SPEC-TABLES.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,14 @@ language schema generates for carries the table wire, and what a fixed-class
port refuses is a pointer in the closure, by name, with this document cited
(§11) — never the `table` declaration itself.

**THE FORM THE EIGHT PORTS CARRY IS THE ONE THAT PRECEDED §3.** §3's id-table
form is the C++ reference and the compiler's own engine (`internal/tablewire`),
held together by the conformance lock; C, C#, Dart, Go, Rust, Java, JavaScript
and Elixir write the earlier form, and the id table, the enum kind and the
escape kind reach none of them yet. Each port's move to §3 is a row of its own
(schema#511 to schema#518), and ROADMAP.md's first table-wire line says which
form a cell means.

**ELIXIR IS THE READING TIER, and the tier is a property of the LANGUAGE rather
than of the port.** A BEAM term has no layout a producer could write, so this
backend never produces a block or a cook: it OPENS one another build wrote and
Expand Down Expand Up @@ -6574,12 +6582,16 @@ assembly, so the runtime is emitted into ONE file per unit (§19.2) and a
second copy is a duplicate-definition error already. There is no include order
to resolve.

**Backend status: OWED, not emitted.** No guarded block carries an identity
today and no generated file asserts one, so the silent case above is live in
every one of the C++ table emitter's guarded blocks. Tracked as schema#301,
with the negative control it must carry: perturb one header's copy of a block
and show the translation unit red, naming that header — the same control that
measured the silence, run in the other direction.
**Backend status: OWED, not emitted, and no open issue carries it.** No guarded
block holds an identity today and no generated file asserts one, so the silent
case above is live in every one of the C++ table emitter's guarded blocks. The
identity is taken over the block's own text and never over a compiler version:
generated output records no compiler version, deliberately (docs/VERSIONING.md),
because a version stamp puts a diff in every generated file of every downstream
tree on every release and says nothing about whether a wire moved. The negative
control the backend half carries is a perturbed copy of one header's block,
which must turn the translation unit red and name that header, and it is the
control that measured the silence run in the other direction.

### 6.2 Variable-length: a lifecycle

Expand Down Expand Up @@ -7064,15 +7076,16 @@ and it is the same buffer across all three by construction.
end is the sum of what the load could not keep and what the save could not
place. That is the number the safety check wants, and it is why the check is
read after the save.
- **`TableRetain` and the three verbs are OWED to §11's claimed set**, and the
claim is deliberately not made in this page's own change: a claim the page
states and the checker does not make is a name a user may still take (§11).
What lands with the feature is `TableRetain` in the unit-scope registry
beside `TableReport`, the three suffixes `LoadRetain`, `MeasureRetain` and
`SaveRetain` in `tableGeneratedVerbs`, and **Dart's three member spellings**
`loadRetain`, `measureRetain` and `saveRetain`, which take that backend's
nine claimed field-name verbs to twelve (§11), plus `TableRetain` in the
Dart library-scope registry, which lands with THAT PORT rather than with the
- **`TableRetain` and the three verbs are IN §11's claimed set**, because a
claim the page states and the checker does not make is a name a user may
still take (§11), so the claims land with the feature rather than with a
port. `TableRetain` sits in the unit-scope registry beside `TableReport`
(`internal/tablenames`), the three suffixes `LoadRetain`, `MeasureRetain`
and `SaveRetain` sit in `tableGeneratedVerbs` beside the body functions that
carry them, and **Dart's three member spellings** `loadRetain`,
`measureRetain` and `saveRetain` take that backend's claimed field-name
verbs from nine to twelve (§11). `TableRetain` in the Dart LIBRARY-SCOPE
registry is the one piece that lands with THAT PORT rather than with the
feature: the registry says what a backend DEFINES, and a gate holds it to
that, so a name Dart does not define yet cannot be registered there without
taking the gate away.
Expand Down Expand Up @@ -11642,7 +11655,8 @@ inspects everything in the schema built:
the construct — the parser's `map[K]V`, the checker's refusals, the generated
entry table with its record and its two constant ids — and every backend
refuses a unit that declares one, by name (§11), until its codec lands. The
C++ reference and the tool are first: the builder surface (insert, erase,
C++ reference is first and carries the whole construct: the builder surface
(insert, erase,
find, iterate), the sort in the four walks, the region load's ascending check
with its `duplicate` and `malformed` events, the const `Find`, the text
form's object and `schema cook-check`'s map-slot clause with its order check
Expand Down
20 changes: 13 additions & 7 deletions docs/VERSIONING.md
Original file line number Diff line number Diff line change
Expand Up @@ -1057,15 +1057,22 @@ Each of these is a claim this page makes in the present tense with the
repository not yet behind it. The 3.0.0 release holds the list at zero.

- #435: the uniform 64-bit wire, the form byte, the id table, the enum kind,
flags bit positions in the build version, the reserved-id refusals.
flags bit positions in the build version, the reserved-id refusals. The C++
reference and `internal/tablewire` carry all of it, held by the conformance
lock. What is owed is the eight ports (#511 to #518), each of which writes
the previous form today.
- #523: the message form, form byte `2`, the batch primitive and the bitpacked
body, the announcement and its two reserved ids, the announced unit vocabulary
of id, kind and shape triples in its compiler-settled order, the two
announcement bounds, the batch bound, the mask's declared width and the
wstring's sixteen bits a unit, and the names §11 owes it.
wstring's sixteen bits a unit, and the names §11 owes it. The C++ reference
carries the form and `internal/tablenames` claims §11's names. What is owed
is the eight ports and the form's `LoadRetain` (SPEC-TABLES.md §3.3), whose
form 2 write refuses by name and whose form 2 read is not built.
- #554: the `column` projection token, reserved by name on the packet wire and
emitted nowhere.
- #434: the reserved escape kind.
- #434: the reserved escape kind. The C++ reference and `internal/tablewire`
carry it; what is owed is the eight ports (#511 to #518).
- #463: the previous-release differential gate — the corpus generated by the
previous release and the new one, byte-compared under an equal id.
- #432: the cook triple, and the byte-order sentences in five places.
Expand All @@ -1081,10 +1088,9 @@ repository not yet behind it. The 3.0.0 release holds the list at zero.
tool with the storage, the cook, the text row and the table-form goldens.
- #523: the unbounded array, `[]T` and `[]*T`, its refusals and the
`list_migrates` golden that pins "the same bytes" as `[..N]T`
(SPEC-TABLES.md §2.9).
- #523: the `///` doc comment, the field-, variant- and arm-level tag, and the
`doc` and `tags` descriptor columns (SPEC.md §4.1, §4.2, SPEC-TABLES.md
§8.1). A tag is carried on a `type` declaration alone today.
(SPEC-TABLES.md §2.9). The C++ reference carries the construct and the
golden is pinned. What is owed is the eight ports and the tool's cook and
uncook halves (SPEC-TABLES.md §7.4).
- #439 and #460: the standard's own contradictions on `T`→`*T`, the flags
row, writer misuse, the declaration-rename row, the count of silent edits,
and the pages that still say schema is not an evolution system.
Expand Down
20 changes: 20 additions & 0 deletions tables/blobs/tables.baseline
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
schema-tables-baseline 7
package blobdemo

table Asset
field name id=0xc4bcadba8e631b86 kind=12 size=32
field kind id=0xef9c96d721673243 kind=8
field data id=0x855b556730a34a05 kind=17 type=bytes
field caption id=0x5daa28eb864c02a5 kind=17 type=string
field next id=0xe5316cbaa025f028 kind=17 type=Asset

table Catalog
field name id=0xc4bcadba8e631b86 kind=12 size=32
field head id=0x0a8f12cc5f9a0c03 kind=17 type=Asset
field thumb id=0x613b19720ff4b203 kind=17 type=bytes
field note id=0x3bf8fbbad1587cdd kind=17 type=string
field alias id=0x509220bb65a646b7 kind=17 type=bytes

## history
### 2026-09-07 (UTC) — first baseline
- baseline created over 2 tables — data written BEFORE this point is not covered by it
Loading
Loading