From b9a379b6f0ae0b8eb5c8245c7dd42dba1913e52c Mon Sep 17 00:00:00 2001 From: Glenn Fiedler Date: Fri, 4 Sep 2026 12:10:27 +1000 Subject: [PATCH 1/2] docs: the prose corrections the whole-standard read and the tutorial found (#460) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prose only — no wire, no emitter, no golden moves — so it lands beside the open wire PRs. Every correction is made where the code is right and the page is wrong, each verified by reading the generated tree or running the binary. **The two-wire front door.** SPEC.md §1's non-goal denied the half of the product SPEC-TABLES.md specifies ("data that must outlive builds is out of its scope entirely"); it now scopes itself to the type wire and names the table wire as the other answer (C12). USAGE.md's "The wire" section and three FAQ answers — "isn't this just Protobuf", "what happens when I change a schema", and the NOT-have list — carried the same denial and are rewritten the same way, keeping the honest half: what schema does not offer is one wire that does both. **Writer misuse, per target, from the generated code.** SPEC.md §5 named a list "exhaustive at all nine" that is wrong on five of them. Read out of generated/: three debug-assert (C++ `serialize_assert`, Dart and Java `assert`), Elixir raises `ArgumentError` in every build, and C, C#, Go, JavaScript and Rust return failure — `0`, `false`, `serialize.ErrValueOutOfRange`, `false`, `Err(ValueOutOfRange)`. Nothing panics and nothing throws. The paragraph's "costlier contracts assert in DEBUG ONLY everywhere" is narrowed to §4.7's own exhaustive list (C1, K7). **Output layout and the CLI, from the tree and from main.go.** §6.1's bullets are rewritten against `schema generate` run over a table-bearing unit for all nine: the C++ Block pair, the C# per-file Block/Cook and three package runtime homes, Go's three table files plus `TableJson.go`, Rust's `Cook` where `Row` was named twice, JavaScript's `Flat.js`, and a Java bullet where there was none. §7's six commands become the binary's own thirteen plus `help`, with the `pack`/`unpack` formatting exception stated. §8's "six table emitters" becomes the nine that exist (K5, K6, K13). **NODE_ENV.** §6.1 said generated JS never reads it; every `Flat.js` forks on `process.env.NODE_ENV` at module load. The claim is now the runtime tier's, and the flat tier's fork is stated where the module is (C15). **The byte order, under #432's ruling.** The build version is target-neutral in effect — `byteorder` rides its projection and no target varies it — so §7's "the byte order is a fact of the build version", §7's pair, §19's "a matching build version already means a matching byte order", USAGE's cook and build-version sections and §20.4's inclusions list all say the same thing now: the cook's header refuses a foreign order, and the content address is the triple `(asset hash, build version, byte order)`, moved everywhere the pair was spelled (V4, F-22). **Counts and rows.** §4.8 against §7.4 rule 9 on union tag order (C11); §16.2's optional row, where `null` is the one key value that reads as absence (C14); `?` added to §4.1's punctuation list (N6); the backend counts that still said two, five, six or seven where nine carry the table wire, across the SPEC-TABLES preamble, §7, §7.5, §11, §15, §16, §20 and USAGE (C13, K13, F-13). §4.1's declaration-rename row gains the qualification it was missing: silent when the table is held by value, but a table's name is its node's type id on the wire, so renaming a POINTER TARGET leaves every node unnameable and every pointer to it null — the row VERSIONING.md already carries since #465 (#464). **USAGE gains two rules a learner met only as a refusal**: `string(N)`'s UTF-8 writer contract and which three targets assert it (F-10), and that `I + F` must equal a storage width (F-11). Its silent-edit count becomes four per §4.1, with the retired-name reuse named as the fifth that the baseline cannot see yet (schema#441). **The examples parse now.** `table Node { value int32 next *Node }` and the comma form appear in five SPEC-TABLES blocks and one USAGE block and none of them parsed; each is expanded to the canonical multi-line form, and the two inline `table Pair { ... }` fragments are respelled as prose. A truncated ```rust fence in USAGE — an unclosed block that swallowed the paragraph after it — is closed. compiler/docs_test.go is the gate: it names each block by an anchor line, pulls the fenced block out of the page and parses it, so an example that stops parsing goes red and an anchor that stops existing goes red with the page and the line to look at. Both failure modes were planted and observed (K8). **docs/COMPARISON-TABLES.md**, from the competitor gather: the silent-edit count takes §4.1's four; the required-fields claim narrows to Protobuf, whose own guidance says "Never add a required field" where FlatBuffers describes `required` neutrally; FlatBuffers' verifier is C++, C and Swift and it is the reflection row that reads basic in C; the Protobuf enum cell gains protobuf.dev's own nonconformance list; and the `PB-dos` source URL follows its redirect. Co-Authored-By: Claude Fable 5.1 --- compiler/docs_test.go | 102 ++++++++++++++ docs/COMPARISON-TABLES.md | 20 +-- docs/FAQ.md | 87 ++++++++---- docs/SPEC-TABLES.md | 270 ++++++++++++++++++++++++++------------ docs/SPEC.md | 181 ++++++++++++++++++------- docs/USAGE.md | 120 ++++++++++++----- 6 files changed, 577 insertions(+), 203 deletions(-) create mode 100644 compiler/docs_test.go diff --git a/compiler/docs_test.go b/compiler/docs_test.go new file mode 100644 index 000000000..f8292c662 --- /dev/null +++ b/compiler/docs_test.go @@ -0,0 +1,102 @@ +// The pages' own schema examples have to be schema. A declaration a reader +// copies out of docs/SPEC.md, docs/SPEC-TABLES.md or docs/USAGE.md and pastes +// into a file is the first thing they run, and one that does not parse teaches +// a syntax the language does not have. This gate names each such block by an +// anchor line, pulls the fenced block that holds it out of the page, and puts +// it through the parser — so an example cannot go stale in either direction: +// a block that stops parsing fails, and an anchor that stops existing fails +// with the page and the line to look at. +// +// Only blocks that are DECLARATIONS are listed. A block showing a construct +// the compiler does not build yet — §2.8's maps — is not one of them, and a +// block of generated C++, Go or JSON is not schema at all. +package compiler + +import ( + "fmt" + "os" + "strings" + "testing" + + "github.com/mas-bandwidth/schema/v2/internal/parser" +) + +// docExample is one fenced block, found by an anchor line and, where a page +// prints the same anchor more than once, by which occurrence. +type docExample struct { + page string // path relative to this package + anchor string // a line inside the block, compared trimmed + nth int // 1 = the first block holding it +} + +// The declaration blocks the pages print. Each is a complete unit once a +// package line is supplied; a block that carries its own `package` keeps it. +var docExamples = []docExample{ + {"../docs/SPEC.md", "// Wire.schema", 1}, // §4.9, the complete example + {"../docs/SPEC-TABLES.md", "table OpenDocument", 1}, // §2.6, union arms as tables + {"../docs/SPEC-TABLES.md", "table Node", 2}, // §3.1, framing worked + {"../docs/SPEC-TABLES.md", "package demo", 1}, // §7.3, a cook worked to the byte + {"../docs/SPEC-TABLES.md", "name string(16)", 1}, // §16.7, one declaration, two texts + {"../docs/USAGE.md", "table OpenDocument", 1}, // messages: a union whose arms are tables + {"../docs/USAGE.md", "table Node", 1}, // pointers +} + +func TestDocExamplesParse(t *testing.T) { + for _, ex := range docExamples { + t.Run(fmt.Sprintf("%s/%s/%d", ex.page, ex.anchor, ex.nth), func(t *testing.T) { + src := fencedBlockContaining(t, ex.page, ex.anchor, ex.nth) + unit := src + if !declaresPackage(unit) { + unit = "package docs\n\n" + unit + } + if _, errs := parser.Parse("Probe.schema", []byte(unit)); len(errs) > 0 { + t.Errorf("%s: the example anchored at %q does not parse: %v\n%s", + ex.page, ex.anchor, errs[0], src) + } + }) + } +} + +func declaresPackage(src string) bool { + for line := range strings.SplitSeq(src, "\n") { + if strings.HasPrefix(strings.TrimSpace(line), "package ") { + return true + } + } + return false +} + +// fencedBlockContaining returns the body of the nth ``` block in page that +// holds a line equal to anchor once trimmed. It fails the test when the page +// has no such block, because an anchor that no longer exists means this gate +// stopped covering the example it was written for. +func fencedBlockContaining(t *testing.T, page, anchor string, nth int) string { + t.Helper() + data, err := os.ReadFile(page) + if err != nil { + t.Fatal(err) + } + inside, found, seen := false, false, 0 + var body []string + for line := range strings.SplitSeq(string(data), "\n") { + if strings.HasPrefix(line, "```") { + if inside && found { + if seen++; seen == nth { + return strings.Join(body, "\n") + "\n" + } + } + inside = !inside + body, found = nil, false + continue + } + if !inside { + continue + } + body = append(body, line) + if strings.TrimSpace(line) == anchor { + found = true + } + } + t.Fatalf("%s has no fenced block number %d holding the line %q — this gate names an example that has moved", page, nth, anchor) + return "" +} diff --git a/docs/COMPARISON-TABLES.md b/docs/COMPARISON-TABLES.md index 9b496eb91..1894f7855 100644 --- a/docs/COMPARISON-TABLES.md +++ b/docs/COMPARISON-TABLES.md @@ -118,7 +118,7 @@ the other format for this one thing. | 2 | **A service or SDK that only speaks Protobuf.** | certain, for that one edge | Never removable. A foreign endpoint's format is what you use at that endpoint. Tables inside, the foreign format only at the boundary. | | 3 | **Maps with string or integer keys.** Protobuf has `map`; FlatBuffers has the sorted-vector idiom. | likely for a team raised on Protobuf | After 3.0.0, in tables only, never in types, and a map makes the table variable: it rides with the pointers, never in a fixed record ([#380](https://github.com/mas-bandwidth/schema/issues/380)). Enum keys are already better served by `[E]T` (§2.4). | | 4 | **Unbounded strings and bytes at used size.** | closed | Landed as `*bytes` and `*string`, the byte-buffer primitive (§2.5, [#259](https://github.com/mas-bandwidth/schema/issues/259)): an image lives inside a table at its own size and a mapped cook points at it with no copy. C++ and the tool carry it; the ports follow as a row on [#366](https://github.com/mas-bandwidth/schema/issues/366). | -| 5 | **A verifier for untrusted bytes.** FlatBuffers ships a separate `Verifier` pass, in C++ and Swift, with a basic one in C, of its thirteen languages. | possible | Table reads are untrusted: they arrive over the network and carry the type wire's security posture; only the cook open is trusted. So the tolerant read IS the verifier, in every language, and [#391](https://github.com/mas-bandwidth/schema/issues/391) is the fuzzer with an independent oracle that proves it in CI. Before 3.0.0, since the release makes the claim. | +| 5 | **A verifier for untrusted bytes.** FlatBuffers ships a separate `Verifier` pass, in C++, C and Swift of its thirteen languages (FB-support). | possible | Table reads are untrusted: they arrive over the network and carry the type wire's security posture; only the cook open is trusted. So the tolerant read IS the verifier, in every language, and [#391](https://github.com/mas-bandwidth/schema/issues/391) is the fuzzer with an independent oracle that proves it in CI. Before 3.0.0, since the release makes the claim. | | 6 | **Every type in tables.** Nothing here is ahead of schema: `fixed`, `ufixed` and the 128-bit integers ride in a table under kinds of their own (§3), which is what a deterministic simulation's save holds. | closed | Landed in the C++ reference and the tool ([#390](https://github.com/mas-bandwidth/schema/issues/390)); the ports follow as a row on [#366](https://github.com/mas-bandwidth/schema/issues/366). | | 7 | **Sorted lookup inside a buffer.** FlatBuffers' `key` attribute and `LookupByKey`. | possible, for a large catalog opened by id | Library-side, after 3.0.0: a sort the cook writer holds and a generated `Find` over the cooked rows. Never stored semantics. | | 8 | **Reflection in more languages.** | possible, for an editor in Python or C# | Descriptors are built into every table's generated code with no schema files and no RTTI (§8). They reach every language with the parity matrix on [#366](https://github.com/mas-bandwidth/schema/issues/366). | @@ -160,8 +160,8 @@ Each with the section that defines it and the reason a game cares. | **Enum-keyed arrays that ride by name** and refuse a bad key in every build. | §2.4, §3.2 | Per-ship-type config survives inserting a ship type in the middle. | | **Evolution by name.** Fields, enum variants and union arms are identified by their name's hash. Add anywhere, remove, reorder. `was` renames; a collision is refused at compile time. | §5 | No append-only rule and no numbers to assign by hand. | | **A read report instead of pass or fail.** `unknown`, `kind_mismatch`, `clamped`, `duplicate`, `malformed`; never fatal on data from another build; a damaged level stops only itself. | §4 | Tools surface it, games set policy, and a corrupt sub-table does not kill the file. | -| **The silent-edit class enumerated: exactly three**, with a compile-time baseline that refuses them and keeps a reasoned history. | §4.1, §18 | Saves from years ago read right, and when one does not the history says why. | -| **A build version computed by the compiler.** Every fact a cook's bytes depend on, digested; `(asset hash, build version)` is a build-cache key. Split from the protocol id, so a table edit never forces a lockstep redeploy. | §10, §20 | Distributed cooking with no version numbers by hand; ship tools and game on different days. | +| **The silent-edit class enumerated: exactly four**, with a compile-time baseline that refuses them and keeps a reasoned history. | §4.1, §18 | Saves from years ago read right, and when one does not the history says why. | +| **A build version computed by the compiler.** Every fact a cook's bytes depend on, digested; `(asset hash, build version, byte order)` is a build-cache key. Split from the protocol id, so a table edit never forces a lockstep redeploy. | §10, §20 | Distributed cooking with no version numbers by hand; ship tools and game on different days. | | **The cook.** `Open` is a header match and a pointer: O(1), mmap-friendly, byte order settled offline, attribution separable. In every port, not only C++. | §7 | "Don't parse, just point" at a gigabyte catalog. | | **The block form.** A third projection of a fixed table: rows at a pitch the compiler computes and both languages' generated code asserts, filled wide by many threads by obligation. | §19, §12.1 | Render data C++ writes and C# reads at 60 Hz with no marshalling. This is the render-data case FlatBuffers was tried on first and replaced (§12.1). | | **Optional by value.** `?T` costs a bool and changes nothing else; `T` and `?T` share one framing. | §2.3 | An optional settings block is not a pointer. | @@ -182,7 +182,7 @@ Each with the section that defines it and the reason a game cares. | Extensions and custom options | both | The attribute vocabulary is closed on purpose (SPEC §4.2); type tags are the claiming mechanism when one is wanted. | | Schema-less values (FlexBuffers, `Struct`) | both | A schema language's users have schemas; a JSON text in a `bytes` field is the escape. | | Well-known types | Protobuf | A `type Timestamp` is three lines and the language pre-defines nothing. | -| Required fields | FlatBuffers | Both competitors' own guidance says not to; every field optional with a declared default is the rule. | +| Required fields | FlatBuffers | Protobuf's own guidance says not to — "Never add a required field" (PB-dos) — while FlatBuffers describes `required` neutrally and verifies it. Here every field is optional with a declared default, which is the rule. | | Explicit or sparse enum values | both | On the table wire a variant rides by name, so its number is invisible (SPEC §9). | | Varint compaction | Protobuf | The type wire is the compact wire; tables trade bytes for tolerance by design, and the ladder states the price. | | Deprecation markers | both | Removal is free and reported, which is what deprecation exists to fake elsewhere. | @@ -262,7 +262,7 @@ the source list at the end. | Rename | `was` (§5) | free (FB-evolution) | free on binary; reserve for JSON (PB-proto3) | | Change a type | `kind_mismatch`: skipped, counted, never misdecoded (§4); same-kind respellings are the silent class the baseline refuses (§4.1) | only at identical width, "with careful handling" (FB-evolution) | a fixed list of compatible pairs; anything else misdecodes silently (PB-proto3) | | Change a default | silent on the wire; the baseline refuses it until moved with a reason (§18) | "don't" (FB-evolution) | proto3 has none; proto2 reader-side (PB-proto2) | -| Enum evolution | add anywhere, remove, reorder; unknown reads as `None` and counts (§5) | append or explicit values; code handles unknowns itself (FB-schema) | add values; open enums keep the int, closed enums move it to unknown fields (PB-enum) | +| Enum evolution | add anywhere, remove, reorder; unknown reads as `None` and counts (§5) | append or explicit values; code handles unknowns itself (FB-schema) | add values; open enums keep the int, closed enums move it to unknown fields — and protobuf.dev's own nonconformance list says C#, Go, JSPB and Ruby treat every enum as open while Dart treats every enum as closed, so the answer depends on the language you generate (PB-enum) | | Union evolution | arms by name; add anywhere, remove, reorder (§2.6, §5) | append or explicit discriminant (FB-evolution) | adding is fine; moving an existing field into a oneof is unsafe (PB-editions) | | Flags evolution | append only, retire in place; the baseline refuses the rest (§4.1) | explicit values, any order (FB-schema) | — | | Array bound change | prefix kept, `clamped` counted; a short array fills with defaults (§4) | unbounded | unbounded | @@ -270,8 +270,8 @@ the source list at the end. | Unknown fields on read | skipped by length, counted (§3, §4) | ignored (FB-evolution) | retained in the unknown set (PB-proto3) | | Unknown fields on rewrite | dropped and counted, by design; the writer has a schema | the buffer keeps them if forwarded whole (FB-evolution) | preserved and re-serialized (PB-proto3) | | Read report | `unknown`, `kind_mismatch`, `clamped`, `duplicate`, `malformed`; nothing fatal from another build (§4) | verifier pass or fail (FB-cpp) | success or failure, plus the unknown set (PB-message) | -| Silent edits enumerated | exactly three, each with its answer (§4.1) | scattered warnings (FB-evolution) | scattered (PB-dos) | -| Compile-time guard | `tables.baseline`: refuses the silent three plus kind, spelling and key changes; `--update --reason` keeps a dated history (§18) | `flatc --conform` (FB-flatc) | `buf breaking`, third party (buf) | +| Silent edits enumerated | exactly four, each with its answer (§4.1) | scattered warnings (FB-evolution) | scattered (PB-dos) | +| Compile-time guard | `tables.baseline`: refuses the silent four plus kind, spelling and key changes; `--update --reason` keeps a dated history (§18) | `flatc --conform` (FB-flatc) | `buf breaking`, third party (buf) | | Same-build fast forms | cook and block are same-build by construction: build version match or refuse (§7, §19.4, §20) | the buffer is always the evolvable form | — | | Version identity | protocol id and build version, both computed by the compiler; a table edit moves only the latter (§10, §20) | `file_identifier`, by hand (FB-schema) | none; editions version the language (PB-ed-overview) | @@ -288,7 +288,7 @@ the source list at the end. | Multi-threaded build | thread-local arenas; block fill by N workers is an obligation checked under TSan (§6.4, §19.5) | one builder per thread | — | | Relocatable | a fixed struct is memcpy-able; a locked region relocates by memcpy with self-relative references (§6.3, §9) | position-independent (FB-internals) | bytes | | Cross-language rows at a pitch | the block form: layout computed by the compiler, asserted by generated code on both sides, refused at open on mismatch (§19, §19.3) | a vector of structs is inline, but the schema does not assert both sides' native layout (FB-internals) | — | -| Offline cook for a build | `schema cook` and generated `Cook`; `(asset hash, build version)` is the cache tuple; attribution separable (§7, §7.6, §20.6) | the buffer is the cooked form, little-endian, build-independent (FB-internals) | — | +| Offline cook for a build | `schema cook` and generated `Cook`; `(asset hash, build version, byte order)` is the cache tuple; attribution separable (§7, §7.6, §20.6) | the buffer is the cooked form, little-endian, build-independent (FB-internals) | — | | Classes | fixed and variable derived from the declaration, held by a gate (§2.2) | struct and table declared; a table is always by offset (FB-schema) | one | | Performance obligations | the ladder: a fixed table beside its type on the ledger; the block matches hand-written scatter both sides; unexplained slowness is a defect (§12.1) | a benchmarks page | "fast parsing" (PB-overview) | | Big-endian targets | C++ proves wire, cook and block on an emulated target; Go, Rust, Java checked | accessors swap (FB-cpp) | neutral by varints (PB-encoding) | @@ -298,7 +298,7 @@ the source list at the end. | feature | schema tables | FlatBuffers | Protocol Buffers | |---|---|---|---| -| Untrusted bytes on the tolerant wire | the read is the validator: every length bounds-checked against its body, counts against the body, ranges clamped, a bad level stops itself; `LoadMeasure` lets the caller refuse a region size before allocating (§4, §6.5). The independent-oracle fuzzer that proves it in every language is #391 | a separate `Verifier`, required before access, in C++ and Swift with a basic one in C (FB-support); `max_depth` 64, `max_tables` 1M (FB-cpp) | the parser validates structure; recursion limit 100; UTF-8 verify (PB-features) | +| Untrusted bytes on the tolerant wire | the read is the validator: every length bounds-checked against its body, counts against the body, ranges clamped, a bad level stops itself; `LoadMeasure` lets the caller refuse a region size before allocating (§4, §6.5). The independent-oracle fuzzer that proves it in every language is #391 | a separate `Verifier`, required before access, in C++, C and Swift (FB-support); `max_depth` 64, `max_tables` 1M (FB-cpp) | the parser validates structure; recursion limit 100; UTF-8 verify (PB-features) | | Trusted fast path | cook and block are trusted by design; `Open` checks identity, not hostility; a signature over the file is the integrity answer (§7, §13.4) | skip the verifier for trusted buffers (FB-cpp) | — | | Value-range enforcement | clamp and count on the table wire; reject on the type wire (§4, SPEC §5) | — | — | | Depth and DoS bounds | by-value depth is fixed by the schema; a pointer graph is flat, so a chain is not a depth (§3.1) | verifier caps (FB-cpp) | recursion depth 100 (PB-limits) | @@ -350,7 +350,7 @@ pages on 2026-09-03. | PB-wkt | https://protobuf.dev/reference/protobuf/google.protobuf/ | | PB-enum | https://protobuf.dev/programming-guides/enum/ | | PB-limits | https://protobuf.dev/programming-guides/proto-limits/ | -| PB-dos | https://protobuf.dev/programming-guides/dos-donts/ | +| PB-dos | https://protobuf.dev/best-practices/dos-donts/ | | PB-message | https://protobuf.dev/reference/cpp/api-docs/google.protobuf.message/ | | PB-overview | https://protobuf.dev/overview/ | | PB-3p | https://github.com/protocolbuffers/protobuf/blob/main/docs/third_party.md | diff --git a/docs/FAQ.md b/docs/FAQ.md index a777fa93a..b4c9e017f 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -36,9 +36,14 @@ reads but cannot validate. For a 60 Hz gameplay packet where you decode the whole thing anyway, that trade runs strongly one way. For a memory-mapped asset you want to touch three fields -of, it runs the other — and that case is deliberately not schema's job. schema -serves the realtime wire; generated storage is relocatable and memcpy-able -(see [USAGE.md](USAGE.md#the-wire)), but the wire is one encoding with one purpose. +of, it runs the other — and schema answers that case on its **other wire**: +declare a `table`, and `schema cook` produces a build-locked region whose +`Open` is a header match and a pointer, with no per-node validation and no +fix-up at load, so you mmap the file and read the three fields at their +offsets (SPEC-TABLES.md §7). The tolerant table wire stays the format of +record beneath it, and the type wire above serves the realtime packet; +generated storage on both is relocatable and memcpy-able +(see [USAGE.md](USAGE.md#the-wire)). ## Isn't this just Protobuf? @@ -49,7 +54,7 @@ carries a tag, so old readers skip fields they do not know and missing fields fall back to defaults. That is why Protobuf is the right answer for service APIs that version independently over years. -**The schema wire has no field numbers, no tags and no evolution +**The schema TYPE wire has no field numbers, no tags and no evolution machinery at all.** Versioning is a **protocol id** — a hash of the schema itself, checked once at connect time. Two peers on the same id speak identical bits; two peers on different ids should not talk. That is an intentionally @@ -57,16 +62,20 @@ harsher contract, and it buys the thing Protobuf cannot give you: nothing on the wire identifies a field, so nothing on the wire is spent identifying one. If your client and server ship independently and must interoperate across -versions, **use Protobuf** — schema's wire will fight you. If they ship +versions over one connection, **use Protobuf** — the type wire will fight +you. If they ship together, which is the normal case for a game client and its dedicated server, the tags were pure overhead and the protocol id is the honest statement of what was always true. -That narrowness is the design, stated plainly: **schema is deliberately not -an evolution system.** Hardcoded structs, one protocol id, same-or-refuse. -Data that genuinely outlives builds and must survive schema drift wants an -evolution-tolerant format, and Protobuf is a fine one — schema does not -compete for that job. +That narrowness is one of TWO wires, and the other one is the evolution +answer: declare a `table` and its fields ride under the hash of their NAMES, +so a reader takes any data a writer ever wrote — unknown fields skipped and +counted, absent fields defaulted, changed kinds skipped rather than +misdecoded, out-of-range values clamped, every event in a read report and +only structural damage fatal (SPEC-TABLES.md). Save games, config, asset +archives and tool output belong there; packets belong on the type wire. What +schema does not offer is Protobuf's model of one wire that does both. ## Isn't this just Cap'n Proto? @@ -166,9 +175,13 @@ The protocol id changes, and peers on the old id will refuse the new one. That is the design: the id is a hash of the schema, so a changed format is a changed identity. -Practically this means **client and server deploy together** for wire -changes. If that is unacceptable for your deployment model, schema is the -wrong tool and you want an evolution system like Protobuf. +Practically this means **client and server deploy together** for type-wire +changes. Data that cannot deploy together goes in a `table` instead: editing +a table moves the build version and never the protocol id, so a save game or +an asset written by an older build still loads, with the differences counted +in a read report (SPEC-TABLES.md). If your PACKETS have to interoperate +across independently deployed versions, that is the case this design does not +serve, and Protobuf does. The id hashes a **wire shape projection**, not the source text, so an edit that moves no bytes does not move the id: a comment, a blank line, a renamed @@ -271,33 +284,51 @@ than incidental. Worth knowing before you adopt rather than after: -- **No maps.** Use a counted array of key/value pairs. +- **No maps on the packet wire.** Use a counted array of key/value pairs. + Tables have them — `map[K]V` (SPEC-TABLES.md §2.8), landing in the C++ + reference first ([#380](https://github.com/mas-bandwidth/schema/issues/380)). - **No recursive types.** `type Node { children [..4]Node }` is rejected as a composition cycle — generated storage is by value, with no pointers, which is - what makes it relocatable and memcpy-able. -- **No optional / nullable**, except the honest version: a bool and a branch, - or a `union` whose first variant is the absence. -- **No schema evolution.** No field numbers, no unknown-field skipping, no - cross-version bridging — one protocol id, same-or-refuse, on purpose. -- **No zero-copy access.** Reads decode into your struct. - -Some of these are scope (a game's packet does not need maps), some are -consequences of relocatable storage (recursion), and one is the design's -spine (evolution). The list is here so you can tell which of your + what makes it relocatable and memcpy-able. A table may point at a table, + itself included, which is what a scene graph or a linked list wants + (SPEC-TABLES.md §2.1). +- **No optional / nullable on the packet wire**, except the honest version: a + bool and a branch, or a `union` whose first variant is the absence. In a + TABLE body it is spelled `?T` — the value plus a generated presence bool, + fixed size, no allocation — and a `type` body refuses one by name. +- **No schema evolution on the packet wire.** No field numbers, no + unknown-field skipping, no cross-version bridging — one protocol id, + same-or-refuse, on purpose. Evolution is the table wire's whole subject. +- **No zero-copy access on the packet wire.** Reads decode into your struct. + A cooked table is the zero-copy form: `Open` matches a header and points + (SPEC-TABLES.md §7). + +Most of these are the PACKET wire's scope — a 60 Hz packet needs none of it, +and the table wire is where the same questions get a yes. Recursion in a +`type` is the one that is a consequence of relocatable storage rather than a +choice. The list is here so you can tell which of your requirements are unmet before you find out the hard way. ## Do I need the serialize runtimes? -Yes — generated code targets a small runtime per language +In six of the nine languages, yes; in three, no. + +C, C++, C#, Go, JavaScript and Rust target a small runtime per language ([serialize](https://github.com/mas-bandwidth/serialize), [serialize.c](https://github.com/mas-bandwidth/serialize.c), [serialize.cs](https://github.com/mas-bandwidth/serialize.cs), [serialize.go](https://github.com/mas-bandwidth/serialize.go), [serialize.js](https://github.com/mas-bandwidth/serialize.js), [serialize.rs](https://github.com/mas-bandwidth/serialize.rs)). They are small, -open source, and doing the bit-level stream work the generated code calls into -(generated JavaScript never imports its runtime — every wire call is a method -on the stream you pass in, and the stream comes from serialize.js). +open source, and doing the bit-level stream work the generated code calls +into. **Dart, Elixir and Java need nothing but their own toolchain** — those +three carry the bit reader and writer in the generated code itself, and are +held to the same goldens as the other six. + +JavaScript is the one that goes both ways: generated JS never IMPORTS a +runtime — every wire call in the runtime tier is a method on the stream you +pass in, and that stream comes from serialize.js — while the flat tier beside +it (`Flat.js`) inlines the bitpacker and needs nothing at all. ## How do I get out if I regret it? diff --git a/docs/SPEC-TABLES.md b/docs/SPEC-TABLES.md index 020ffb48c..0980f609a 100644 --- a/docs/SPEC-TABLES.md +++ b/docs/SPEC-TABLES.md @@ -161,9 +161,10 @@ carry both classes; C#, Dart, Go, Rust, Java, JavaScript and Elixir carry the fixed class (§6.1) — optionals, enum-keyed arrays, the text form (§16) and all — and each refuses a unit whose closure declares a pointer, naming its variable class as a -follow-on. Every other backend refuses a unit that declares tables at all, by -name, with this document cited. The remaining per-language backends are named -follow-ons (§15). +follow-on. **There is no tenth target and no backend left out**: every +language schema generates for carries the table wire, and what a fixed-class +port refuses is a pointer in the closure, by name, with this document cited +(§11) — never the `table` declaration itself. **ELIXIR IS THE READING TIER, and the tier is a property of the LANGUAGE rather than of the port.** A BEAM term has no layout a producer could write, so this @@ -543,8 +544,8 @@ closure — and neither does a table whose closure carries a UNION, because §19.3 pins the C# side to Sequential with generated padding and Sequential cannot overlay arms. Each says so in the Block header rather than going missing — and in the JavaScript module, which says the same thing in its own -file. Every other backend emits no Block file at all, as it emits no Table -file; those are the same named follow-on (§15). +file. No backend is without a Block file to say it in: all nine emit one, as +all nine emit a Table file. **The VIEW's type half and unit registry (§8.2–§8.7) are specified and unimplemented.** What ships today is §8.1: a table's descriptors, built in, @@ -1052,8 +1053,9 @@ enum is keyed. forward pass works, and an STL algorithm does not, because the header includes no `` (§13.9). - **Held by test**: every keyed array in the corpus is iterated in both - backends and every walk yields `E.Max` entries whose keys run `1 .. E.Max`; + **Held by test**: every keyed array in the corpus is iterated in the C++ + reference and in C#, and every walk yields `E.Max` entries whose keys run + `1 .. E.Max`; one negative control moves `begin()` off the first stored slot and another restores the `None` slot — storage `E.Max + 1` with no shift — and the tables suite, the layout gate and the `sizeof` assertion go red. **The @@ -1191,8 +1193,17 @@ Inside a TABLE closure a union's arm may name a `table`, not only a declared `type`: ``` -table OpenDocument { path string(256), line uint32 } -table SaveDocument { path string(256), force bool } +table OpenDocument +{ + path string(256) + line uint32 +} + +table SaveDocument +{ + path string(256) + force bool +} union ToolBody { @@ -1290,8 +1301,13 @@ the pitch is, what it costs and what it refuses — is §19. ### 2.8 Maps: `ships map[string(32)]ShipConfig` ``` -table ShipConfig { name string(64) health int32 } -table Item { count int32 } +table ShipConfig +{ + name string(64) + health int32 +} + +table Item { count int32 } table Fleet { @@ -1417,7 +1433,8 @@ table FleetShipsEntry // generated; never spelled in a schema touches no byte anyone wrote. - **Its field ids are two CONSTANTS**: `key` is `0xA079` and `value` is `0x9194`, the fold (§5) of those two names, and they never move. That is what - makes a user's own `table Pair { key string(32) value ShipConfig }` under + makes a user's own `table Pair` — a `key string(32)` and a + `value ShipConfig` — under `[..N]Pair` the SAME BYTES as the map — the migration path from the table-of-pairs idiom a schema used before maps existed, and the proof that a map is an array of tables rather than a new thing. @@ -2442,8 +2459,19 @@ does not spend. ``` table Palette { id int32 } -table Node { value int32 next *Node palette *Palette } -table Scene { head *Node palette *Palette } + +table Node +{ + value int32 + next *Node + palette *Palette +} + +table Scene +{ + head *Node + palette *Palette +} ``` with `scene.head = A`, `A.next = B`, and `A.palette`, `B.palette` and @@ -2803,7 +2831,8 @@ only. | a field moved between `T` and `?T` | silent — no byte moves | passes | **moves** — the presence companion is storage | | a field moved to or from `*T` | `kind_mismatch` | passes | **moves** | | an `if` GUARD added or removed | silent, and the read is faithful; the cost is the next WRITE | passes | no | -| a DECLARATION renamed — a `type` or a table | silent: a declaration name is not on the wire | **warns** when a table closure reaches it, naming what carries its contents on and how many identities that candidate carries (§18.3) | **moves** | +| a DECLARATION renamed — a `type`, or a table held BY VALUE | silent: a name held by value is not on the wire | **warns** when a table closure reaches it, naming what carries its contents on and how many identities that candidate carries (§18.3) | **moves** | +| a TABLE renamed where it is a POINTER TARGET | **not silent**: a table's own name is its node's type id on the wire (§5), so every node of the old name is unnameable — skipped by its length and counted `unknown`, with every pointer to it reading null (§3.1) | as the row above | **moves** | | a `type`'s FIELD renamed, where `was` is refused (SPEC.md §4.2) | `unknown` on the table wire, whose field id is the name's hash | passes in silence | **moves**, and through the protocol id as well (SPEC.md §3.1) | ### 4.2 The read is the verifier: the wire fuzzer @@ -2952,7 +2981,8 @@ whose id does is refused naming the field. **A MAP's entry carries two ids that are CONSTANTS of this vocabulary** (§2.8): `key` is `0xA079` and `value` is `0x9194`, the same fold over those two names, fixed for every map in every unit — which is what lets a user's own -`table Pair { key K value V }` under `[..N]Pair` be the map's bytes. The +`table Pair`, a `key K` beside a `value V`, under `[..N]Pair` be the map's +bytes. The entry's generated NAME, `Entry`, is a table name in the closure and is claimed as `_present` is; it never reaches the wire, because an entry is by value inside its holder and takes no type id. @@ -3433,9 +3463,11 @@ classes are built (schema#251).** `schema cook`, `schema cook-check` and orders, over the same IR the emitters consume — and `wire → cook → wire` is byte-identical over the corpus, which is what proves the accelerator loses no fact. Every port emits the entry point for EVERY TABLE (below) — a root is any -table — in its own idiom: the C++ backend `Open`, the C# backend -`Cook.Open`, the Go backend `Open`, the Java backend -`Cook.open`, the JavaScript backend `Cook.Open`, and the Elixir +table — in its own idiom, and the list is all nine: the C++ backend +`Open`, the C backend `_open`, the C# backend `Cook.Open`, +the Dart backend `Cook.open`, the Go backend `Open`, the Java +backend `Cook.open`, the JavaScript backend `Cook.Open`, the Rust +backend `Cook::open`, and the Elixir backend `cook_open_`, which takes a `lead` beside the bytes for the base-alignment check a BEAM binary cannot carry (§7.1's alignment word, and the backend status in §2). A game points at a cook the tooling produced, whichever @@ -3475,20 +3507,31 @@ format. the byte order of the build it is cooked for, so the fixing happens where the target is known — offline, once, on the writing side — and never on the reading side, which is what makes `Open` a match and a point rather than a -pass over the region (below). The byte order is a fact of the build version -(§20.1), so a cook for a foreign order is not this build's file and refuses. - -**A cooked artifact is CONTENT-ADDRESSED by a pair — the hash of its source -asset, and the unit's BUILD VERSION (§20)** — and that pair is the tuple the +pass over the region (below). **The byte order is a fact of the TARGET, not of +the build version**: §20.1 digests `byteorder` as a generation input and it is +`little` for every target schema generates for today, so two builds of one +schema for two orders emit the same id. What refuses a cook for a foreign +order is the header — the magic read bytewise, and the `byte_order` word +beside it (§7.1) — and `Open` does that in O(1) like every other check it +makes. + +**A cooked artifact is CONTENT-ADDRESSED by a TRIPLE — the hash of its source +asset, the unit's BUILD VERSION (§20), and the target's BYTE ORDER** — and +that triple is the tuple the runtime searches for, the tuple a distributed build cache produces under and -serves from. It is why the cooking side is a build cost rather than a runtime -one: the work happens offline, once per (asset, build version), and the game -does a lookup. That is the fact the performance ladder cites when it calls the -wire and the cook read-hot and write-cold. +serves from. The byte order is a coordinate rather than a digest input +because the build version is target-neutral by design: one id shared by every +target of one game, and one axis beside it for the fact that differs. A cache +keyed by the pair on a big-endian target would never serve wrong bytes to a +reader — the header still refuses — but it would collide across orders and +miss forever. It is why the cooking side is a build cost rather than a runtime +one: the work happens offline, once per (asset hash, build version, byte +order), and the game does a lookup. That is the fact the performance ladder +cites when it calls the wire and the cook read-hot and write-cold. **The ASSET HASH is the hash of the WIRE FILE the cook was produced from.** The wire is the format of record and a cook is produced beside one (below, -§17), so naming the wire file is what makes the pair well defined: a +§17), so naming the wire file is what makes the triple well defined: a pipeline that ran a text tree through `schema pack` and then cooked in one step still keys on the bytes the cook actually read, and an edit upstream of those bytes reaches the key through them. @@ -3863,10 +3906,10 @@ the wire, and keeps the flexibility that comes with it. - **Alignment.** The header pads the data part to the region's alignment, so a base the allocator or `mmap` gave you is already aligned; `mmap` gives page alignment for free. -- **Endianness is part of the COOK, not of `Open`** (above): a matching - build version already means a matching byte order, so `Open` never fixes - anything up. Cooking for a foreign target is where a byte swap would live - if one is ever wanted (§15). +- **Endianness is part of the COOK, not of `Open`** (above): the header's + magic and `byte_order` word refuse a foreign order outright, so `Open` + never fixes anything up. Cooking for a foreign target is where a byte swap + would live if one is ever wanted (§15). Prior art gets one sentence, and it is the contrast: systems that made pointed-at access their ONLY wire coupled access to evolution and paid @@ -4042,8 +4085,8 @@ fields would get wrong: padding, a string's or `bytes`' unused tail, the bytes of a union outside its set arm, and the slack between the last node and the rounded `data_length`. It is not tidiness: a cooked artifact is CONTENT-ADDRESSED by (asset hash, -build version) (§7), so two cooks of one wire have to be one artifact, and one -uninitialized pad byte would make them two. +build version, byte order) (§7), so two cooks of one wire for one target have +to be one artifact, and one uninitialized pad byte would make them two. ### 7.3 A cook, worked to the byte @@ -4053,8 +4096,19 @@ Every number below derives from a rule on this page; none of it is declared. package demo table Palette { id int32 } -table Node { value int32 next *Node } -table Scene { name string(4) head *Node palette *Palette } + +table Node +{ + value int32 + next *Node +} + +table Scene +{ + name string(4) + head *Node + palette *Palette +} ``` with `scene.name = "hi"`, `scene.head = A`, `A.next = B`, `A.value = 1`, @@ -4227,15 +4281,17 @@ every clause above by construction. what has to be O(1), and a tool that walks a directory is measuring its own scan. It is held over the C++ `Open`, below. -- **THE CROSS-IMPLEMENTATION LOCK, and it is what makes THREE implementations - of one page worth having.** The tool writes a cook in Go, the C++ `Open` +- **THE CROSS-IMPLEMENTATION LOCK, and it is what makes independent + implementations of one page worth having.** The tool writes a cook in Go, + the C++ `Open` points at it and the C# `Open` points at the very same bytes, and none of the - three was written from either of the others. (A FOURTH reader now points at - the same bytes: the JavaScript one, whose canonical node dump the conformance - harness byte-compares against the pinned C++ walk over all six of its - fixtures. It carries the DUMP half of this lock and not the directory half — - a reading-tier backend gets its fixtures from the harness, which holds the - dump and not the attribution part.) The lock is the ATTRIBUTION part: + three was written from either of the others. (Every other port's `Open` + now points at the same bytes too — the entry point is emitted by all nine + (§7) — and a reading-tier backend such as the JavaScript one carries the + DUMP half of this lock and not the directory half: its canonical node dump + is byte-compared against the pinned C++ walk over each of its fixtures, and + it gets those fixtures from the harness, which holds the dump and not the + attribution part.) The lock is the ATTRIBUTION part: every node a reader reaches by following its OWN derefs, through its own record layouts, must be a node the directory names, at that offset, with that type id — and the two SETS must be equal, so an edge the reader stops @@ -4420,9 +4476,9 @@ every clause above by construction. the reference and stays it: a generated writer is held to the tool's output BYTE FOR BYTE, in both byte orders, over every instance the conformance harness carries. That is the whole contract, and it is not a courtesy — a cooked -artifact is CONTENT-ADDRESSED by (asset hash, build version) (§7), so two -writers of one instance must produce ONE artifact or the pair addresses -nothing. +artifact is CONTENT-ADDRESSED by (asset hash, build version, byte order) +(§7), so two writers of one instance must produce ONE artifact or the triple +addresses nothing. **Why a runtime writes one at all**: tooling is written in whatever language the tool is written in, and a game's runtime in another. An editor, an importer @@ -4474,7 +4530,8 @@ bool SettingsCook( const Settings & value, void * out, uint64_t capacity, Tab follow-on here (§15), not a parameter on this call. - **IT IS WRITE-COLD, AND THE GENERATED CODE SAYS SO**: the writer is ordinary `inline` rather than the force-inlined shape the wire codecs take (§9), - because a cook is produced offline once per (asset, build version) and read + because a cook is produced offline once per (asset hash, build version, + byte order) and read every time a build starts. The performance ladder puts the two halves in different places and the emitter follows it. @@ -5213,14 +5270,18 @@ in build version (§20.5). - **`| max = K` headroom on an enum in a table closure** — a headroom value has no name, and the table wire identifies a variant by name (§5). Key enums are in scope on the same terms. -- Tables under a backend that carries none (status, above) — refused with the - follow-on named, never silently ignored. -- **A VARIABLE-LENGTH table's WIRE SURFACE under the C#, Go, Rust, Java, - JavaScript and Elixir backends** — every port carries the fixed class on the +- Tables under a backend that carries none — **no target is in that state** + (status, above): all nine carry the table wire, so a `table` declaration + alone is refused by nobody. What a port refuses is a CONSTRUCT it lacks, + each named below and each naming its follow-on, never silently ignored. +- **A VARIABLE-LENGTH table's WIRE SURFACE under the C#, Dart, Elixir, Go, + Java, JavaScript and Rust backends** — every port carries the fixed class on + the wire; their variable class there (the arena, the builder, the region, the node-table codec) is a named follow-on, and a pointered unit gets no - `Table.cs`, no `Table.go`, no `_table.rs`, no - `Table.java`, no `Table.js` and no `Table.ex` at all, with + `Table.cs`, no `Table.dart`, no `Table.ex`, no + `Table.go`, no `Table.java`, no `Table.js` and no + `_table.rs` at all, with the refusal NAMED in every source the unit does emit rather than left as a missing symbol — in JavaScript, a banner at the head of every `Block.js` and `Cook.js` the unit gets. @@ -5230,11 +5291,13 @@ in build version (§20.5). not parsed: a block (§19) and a cook (§7) are blittable records plus a header match, and neither needs one line of the codec the variable class is missing. So a pointered unit's block and cook sources ARE emitted in every port — - `Block.cs` and `Cook.cs`, `Block.go` and `Cook.go`, - `_block.rs` and `_cook.rs`, Java's `
Block.java`, + `Block.cs` and `Cook.cs`, `Block.dart` and + `Cook.dart`, `Block.ex` and `Cook.ex`, `Block.go` + and `Cook.go`, Java's `
Block.java`, `
Cook.java` and `Row.java`, `Block.js` and - `Cook.js`, and Elixir's `Block.ex` and `Cook.ex`; its - `Cook.Open`, its `Open`, its `Cook.open` and its + `Cook.js`, and `_block.rs` and `_cook.rs`; its + `Cook.Open`, its `Cook.open`, its `Open`, its + `Cook::open` and its `cook_open_` open its cooked assets in full, and what a consumer cannot do in any of those languages is `Measure`, `Save` and `Load` over the tolerant wire. @@ -6488,13 +6551,16 @@ inspects everything in the schema built: does, and it is not stable before JDK 22 where this backend compiles at `--release 17`; both `open`s already take a `long` length, which is the seat that overload takes when the floor moves. -- **Per-language backends beyond C, C++, C#, Go, Java, JavaScript and Rust** - (the refusal in §11 names this). +- **The VARIABLE class on the WIRE in every port but C++ and C** (the refusal + in §11 names this). All nine targets — C, C++, C#, Dart, Elixir, Go, Java, + JavaScript and Rust — carry a table backend, so no language is waiting for + one at all; what the other seven are waiting for is the variable class, and + each is listed below on its own terms. C# came first, because the dogfood's game engine reads the same config and asset bytes the C++ tools write (§12); Rust, Go, C and Java followed; and JavaScript is the first of the READING TIER — a backend with no struct layout at all, which is what proves the two accelerators can be READ by a - language that could never produce one. The FIXED class is what those need: + language that could never produce one. The FIXED class is what a port needs: storage structs, measure/save/load over caller-owned buffers, the report, the reflection descriptors, `?T`, `[E]T`, name-hashed vocabularies and the text form (§16) — the variable class is still ahead of it. A port mirrors @@ -6864,11 +6930,13 @@ its storage comes from (§6.5): SceneFromJson( builder, text, text_bytes, &report ); ``` -**Backend status for this section: the FIXED class in C++, C, C#, Go, Rust, -Java, JavaScript and Elixir; the VARIABLE class in C++ (§16.7).** A pointered +**Backend status for this section: the FIXED class in all nine — C++, C, C#, +Dart, Elixir, Go, Java, JavaScript and Rust — and the VARIABLE class in C++ +(§16.7).** A pointered unit's text form is the C++ reference's, through the builder, and carrying it -to the other backends is schema#349's row beside the wire. In C#, Go, Rust, -Java, JavaScript and Elixir the absence is already made one level up: a +to the other backends is schema#349's row beside the wire. In C#, Dart, +Elixir, Go, Java, JavaScript and Rust the absence is already made one level +up: a pointered unit gets no table source at all (§11), so it has no text form for the same reason it has no wire codec; the C port has the wire's earlier form (§3.1) and its text form follows it. @@ -7016,7 +7084,7 @@ Per kind: | `[..N]T` bounded array | array | count = length; more than N are dropped, counted | | `[E]T` enum-keyed array | object keyed by VARIANT NAME | `{ "Fighter": {...}, "Bomber": {...} }`; an absent key keeps that slot's defaults; a **repeated variant key is last-wins and counted**, as any duplicate key is; an unknown key is skipped and counted, and **`"None"` is such a key** — it names no slot (§2.4) | | nested `type` / `table` | object | the same walk, recursively | -| `?T` optional | the value, or the key absent | **presence of the KEY is presence**: a key present sets the field present, whatever its value; an absent key leaves it absent. `ToJson` writes present optionals only. An optional ARRAY (§2.3) is this row over the array's: the key present with `[]` is present-and-empty, and `ToJson` writes a present empty array as `[]` | +| `?T` optional | the value, the key absent, or `null` | **presence of the KEY is presence, with `null` the one exception**: a key present sets the field present whatever its value, EXCEPT `null`, which is the absence itself and puts the field back at its declared default (below); an absent key leaves it absent. `ToJson` writes present optionals only. An optional ARRAY (§2.3) is this row over the array's: the key present with `[]` is present-and-empty, and `ToJson` writes a present empty array as `[]` | | union | object with ONE key, the arm name | `{ "buff": { "multiplier": 2.0 } }`; `None` writes as `{}`; `{}` or absent reads as None; two keys is malformed. A `table` arm (§2.6) is the same object form. An ARRAY of unions (§2.6) is an array of this row, a `None` element as `{}` in its place | | pointer `*T` | object, or `null` | the pointee's object in place; `null` is a null pointer. A node named MORE THAN ONCE is defined once under `&node`, with its fields, and named by `&node` alone after — §16.7's one construct, and the only thing this form adds for the variable class. An ARRAY of pointers (§2.1) is an array of this row, and a slot may define or name a node any other slot or field does | | `*bytes` | string, base64, or `null` | the blob in place, as `bytes(N)` spells its bytes, with NO bound to clamp against; `""` is a present blob of length zero and `null` is a null reference (§2.5). A body that is not base64 is `kind_mismatch`, the reference left null | @@ -7214,8 +7282,20 @@ added. ``` table Palette { id int32 } -table Node { value int32 next *Node palette *Palette } -table Scene { name string(16) head *Node palette *Palette } + +table Node +{ + value int32 + next *Node + palette *Palette +} + +table Scene +{ + name string(16) + head *Node + palette *Palette +} ``` A TREE — `scene.head = A`, `A.next = B`, and each of the three naming a @@ -8710,18 +8790,25 @@ loudly, naming the type and the field; §20.3 states the asserts in full. the projection of §20.2, both pinned as goldens over the corpus. The BLOCK backends emit the constant and stamp it into every block's prologue (§19.1), and `BlockOpen` compares it; `schema cook` stamps it into the cooked header, `schema -cook-check` reads it back, and every port's cook entry point compares it (§7) — -the C++ `Open`, the C# `Cook.Open`, the Go `Open`, the Java -`Cook.open`, the JavaScript `Cook.Open`. What remains owed, largest first: +cook-check` reads it back, and every port's cook entry point compares it (§7) +— the C++ `Open`, the C `_open`, the C# `Cook.Open`, the +Dart `Cook.open`, the Elixir `cook_open_`, the Go `Open`, +the Java `Cook.open`, the JavaScript `Cook.Open` and the Rust +`Cook::open`, all nine. What remains owed, largest first: 1. **The constant rides in the TABLE-bearing sources only.** §20.7 asks for one beside `ProtocolId` in every backend; today the block backends emit - it into `Block.h` / `Block.cs`, the C++ table backend emits it - into every `Table.h` — where the cook's reader is — the Java backend - gives it a package-level file of its own, `BuildVersion.java`, emitted for - any unit with a table and for no other, the C# cook emits - it into `Cook.cs` when the unit has no block form to carry it already, - and the seven backends that carry no table emit none. The C# Table sources + it into `Block.h` / `Block.cs`, the C and C++ table backends + emit it + into every `Table.h` — where the cook's reader is — the Java and + Elixir backends + give it a package-level file of its own, `BuildVersion.java` and + `BuildVersion.ex`, and Rust a module, `build_version.rs`, each emitted for + any unit with a table and for no other; the C# cook emits + it into `Cook.cs` when the unit has no block form to carry it already. + **A TABLE-FREE unit gets none in any target** — that, and not a missing + port, is the only case where the constant is absent, since all nine + backends carry the table wire (status, §2). The C# Table sources carry none, which is the zero-cost gate (§2.2) rather than an omission: the C# cook reader is in the accelerator's own file. **In C# exactly one accelerator defines it** — `Schema` is one partial class across a unit's @@ -8787,9 +8874,12 @@ the type wire, on what the region looks like, and on what a load puts in it. **And the BYTE ORDER**, one line, because a cook is produced in the byte order of the build it is cooked for (§7): two builds alike in every other fact produce different cook bytes, and a tuple that addresses two different -artifacts is a defect in the tuple. It is a generation input, `little` for -every target schema generates for today; a big-endian cook is the cross-endian -question §15 owns. +artifacts is a defect in the tuple. It is a GENERATION input and `little` for +every target schema generates for today, so the token never varies and the id +this projection digests is target-neutral in effect; what distinguishes two +orders is the third coordinate of §7's content address — `(asset hash, build +version, byte order)` — and the cook header's own magic and `byte_order` word. +A big-endian cook is the cross-endian question §15 owns. **The set is closed, and the table below is the proof.** Every declaration-side fact this language has appears in it exactly once, assigned @@ -9132,8 +9222,8 @@ wrong fails to build instead of degrading. cook's. A reader who has just read §4.1 will look for this row, so it is here; - **the `json` key** (§16.4) and anything else the TEXT form owns. A cook is - produced from the WIRE file, and §7 defines the tuple's other half as that - file's hash; + produced from the WIRE file, and §7 defines the tuple's asset coordinate as + that file's hash; - **baseline-only facts** (§18): whether a `tables.baseline` exists at all, its recorded history, a `--reason`, its rendering version. §18 is untouched by this section and untouched by the build version; @@ -9161,10 +9251,13 @@ version. ### 20.6 The tuple -**`(asset hash, build version)` and nothing finer.** Tooling produces a cook -under it, the store is indexed by it, and the game asks for it. §7 defines the +**`(asset hash, build version, byte order)` and nothing finer.** Tooling +produces a cook under it, the store is indexed by it, and the game asks for +it. §7 defines the asset hash as the hash of the WIRE file the cook was produced from, which is -what makes the pair well defined. +what makes the triple well defined. The build version is TARGET-NEUTRAL — one +id shared by every target of one game — so the byte order rides beside it as +its own coordinate rather than inside it (§7, §20.1). **A new build version is a new cook, and that is the model rather than a cost.** The work is offline and the store absorbs it: a build cache exists to @@ -9241,7 +9334,10 @@ a second digest. nothing. - **The inclusions the sort order could hide**: a record renamed, added or - removed moves it; the target byte order moves it. + removed moves it. The `byteorder` token rides the projection beside them + (§20.1) and no target varies it, so the id it digests is target-neutral in + effect and there is no edit to pin: §7's content address carries the byte + order as its own coordinate instead. - **The worked example of §20.2 is a golden**, projection text and digest both, so a port reproduces the text and not only the number. - **Goldens over the corpus** (SPEC.md §7.2 gate 2's sibling): `schema diff --git a/docs/SPEC.md b/docs/SPEC.md index 3c9f3f40c..bd054429e 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -65,10 +65,13 @@ conventions. Delta serialization is out of scope for v1. ### Non-goals (v1) -- **No wire-format versioning, anywhere.** Versioning is by protocol id, - deliberately (§3): hardcoded structs, one id, same-or-refuse. schema is - not an evolution system, and data that must outlive builds is out of its - scope entirely. +- **No wire-format versioning on the TYPE wire.** Versioning here is by + protocol id, deliberately (§3): hardcoded structs, one id, same-or-refuse. + Data that must outlive the build that wrote it belongs to the other wire — + `table` declarations, versioned in-wire by field id, where any reader reads + any data and the differences are reported rather than fatal + (SPEC-TABLES.md). This document specifies the type wire; the two share one + language, one unit and one compiler, and nothing else. - **No unbounded collections.** Everything on the wire has a declared bound, as everywhere in the serialize family. - **No annotation of existing hand-written types.** schema owns the types it @@ -227,8 +230,10 @@ from this projection rather than carried in it. literals** (decimal, with optional fraction and exponent) appear in float constants and as float attribute values (`min`/`max`/`resolution` on `float32`). -- **Punctuation and operators:** `{ } ( ) [ ] , : = ! . .. | + - * / %` - (maximal munch: `..` wins over `.`). `|` opens a line's qualification +- **Punctuation and operators:** `{ } ( ) [ ] , : = ! ? . .. | + - * / %` + (maximal munch: `..` wins over `.`). `?` is the OPTIONAL type prefix + (§4.2's grammar), and it is accepted in a table body only — a `type` body + refuses one by name (SPEC-TABLES.md §2.3). `|` opens a line's qualification section (§4.2, Attributes) and claims the rest of the line — the newline or a `//` comment terminates it, and no newline suppression applies after `|` or after a `,` inside a qualification. `<=` is not in the language: a count @@ -1212,16 +1217,27 @@ defined behaviors: the spec owes a conforming writer exact bytes and owes a misbehaving writer nothing. The read side is untouched by this doctrine — readers face untrusted data and keep every mandated check above. -Within that doctrine, misuse surfaces by each target's own convention, and the -list is exhaustive at all nine: C, C++, Dart and Java debug-assert (unchecked -in release — C and C++ through `serialize_assert`, Dart and Java through the -language's own `assert`, live under `--enable-asserts` and `-ea`); Go and Rust -panic, C# throws and Elixir raises `ArgumentError`, on misuse in all build -modes; generated JavaScript carries no write-side check of its own, so misuse -reaches the runtime's stream methods. The generated write code's job is to +Within that doctrine, misuse surfaces by each target's own convention — a +language verifies correctness the way that language verifies correctness — +and the list is exhaustive at all nine. **Three debug-assert**, unchecked in +release: C++ through `serialize_assert`, Dart and Java through the language's +own `assert`, live under `--enable-asserts` and `-ea` (Java's contracts ride +one `check` predicate call, so a dormant assert costs the JIT one +inlining slot rather than a body). **One raises in every build**: Elixir's +`ArgumentError`, the BEAM having no dormant assert to compile out. +**Five return failure from the write** rather than invent an assert their +language does not have: C `0`, C# and JavaScript `false`, Go +`serialize.ErrValueOutOfRange`, Rust `Err(serialize::Error::ValueOutOfRange)` +— and JavaScript's flat tier, which forks checked/production at module load +(§6.1), refuses with `-1` on the checked side and trusts the caller on the +production one. **No target panics and none throws**: Elixir's raise is the +only unwinding path in the nine, and it is the BEAM's own. The generated +write code's job is to make misuse impossible by construction — bounds come from the schema. Costlier contracts -assert in DEBUG ONLY everywhere (§4.7's UTF-8 well-formedness contract is the -type case: an O(n) check no release path should carry). Ranges are trusted +assert in DEBUG ONLY, and only where a target carries one at all (§4.7's +UTF-8 well-formedness contract is the type case: an O(n) check no release +path should carry, asserted by C, C++ and Rust and absent from the other +six). Ranges are trusted inputs everywhere: generated code never feeds attacker-influenced values as min/max. @@ -1353,7 +1369,10 @@ gates it is held to, are SPEC-TABLES.md §8. **Output layout.** Each target emits one generated file per schema file — `examples/Constants.schema` → `generated/cpp/Constants.h` — so the generated -tree mirrors the schema tree a person navigates. +tree mirrors the schema tree a person navigates. What a target adds to that +one file is its own, and the bullets below state it in full: a wire header +beside a data header, a table-side file per surface, a runtime home per unit, +or a file per declaration where the language demands one. - **C++: the header splits into a data/wire pair.** `.h` is the DATA header (constants, enums, flags, structs and MaxBits/MaxBytes bounds) @@ -1372,10 +1391,17 @@ tree mirrors the schema tree a person navigates. stays so by ruling ("I think it's OK for types to remain header only"): a type is a struct and its codec, both of which a compiler folds into the caller, so there is nothing to compile once and link. A unit that declares - TABLES emits one further pair — `Table.h` and `Table.cpp` — - because the table wire carries a RUNTIME the type wire has no equivalent - of (SPEC-TABLES.md §6.1, §13.5); it is a table-side file and adds nothing - to a table-free unit. **Every unit emits one further pair per UNIT** — + TABLES emits two further pairs per schema file. `Table.h` / + `Table.cpp` carries the table wire's codecs, its reflection + descriptors, its TEXT FORM and the cooked form's `Open` + (SPEC-TABLES.md §7) — a pair rather than a header because the table wire + has a RUNTIME the type wire has no equivalent of (SPEC-TABLES.md §6.1, + §13.5). Beside it, `Block.h` / + `Block.cpp` carries the block form, which nothing declares and every + fixed table has (SPEC-TABLES.md §19): include the header and compile the + source beside it only if you use the form, and `Table.h` carries not + one symbol of it. They are table-side files and add nothing to a table-free + unit. **Every unit emits one further pair per UNIT** — `View.h` and `View.cpp` — carrying the unit registry and the reflection descriptors of every declaration (SPEC-TABLES.md §8.3, §8.5). It is per unit rather than per schema file because the registry is @@ -1417,14 +1443,23 @@ tree mirrors the schema tree a person navigates. spelling is one of the macros the generated sources define. - **Go:** one `.go` file per schema file, all in `package ` — Go packages are order-free across files, so there is no topo sort and no - include graph to refuse. + include graph to refuse. A unit that declares TABLES grows three further + files per schema file — `Table.go` with the table wire's codecs and + its reflection descriptors, `Block.go` (SPEC-TABLES.md §19) and + `Cook.go` (SPEC-TABLES.md §7) for the two accelerators — plus one + `TableJson.go` per unit carrying the TEXT FORM's single generic walk + over those descriptors (SPEC-TABLES.md §16). A Go package compiles whole, + so the separate file does not let a consumer leave the walk out of the + build; the LINKER drops what nothing calls, and the file is what keeps that + legible. A table-free unit grows none of it. - **Rust:** one module per schema file (lowercased basename) plus a generated `lib.rs` declaring and glob re-exporting them. A unit that declares TABLES grows three per-file modules and three per-UNIT runtimes, all declared by that same crate root: `_table.rs` with the table wire's codecs, its reflection descriptors and its TEXT FORM (SPEC-TABLES.md §16); - `_cook.rs` with the cooked form's blittable `Row` records and - their layout contract as const asserts (§7, §20.3); `_block.rs` with + `_cook.rs` with the cooked form's `Cook` handles, their open + paths and the layout const asserts those rest on (§7, §20.3); + `_block.rs` with the block form's projection and open path (§19); and beside them `table_runtime.rs`, `cook_runtime.rs` and `block_runtime.rs`, which carry each surface's shared runtime and the text form's one generic walk. **The @@ -1444,14 +1479,18 @@ tree mirrors the schema tree a person navigates. are byte-identical either way. - **C#:** one `.cs` file per schema file, types at namespace level and every function and constant on `public static partial class Schema`, in - `namespace `. A unit that declares TABLES emits one further file - per schema file, `Table.cs`, carrying the table wire's codecs, its - reflection descriptors and its TEXT FORM (SPEC-TABLES.md §16) — the C++ - pair's single-file twin, and single because a unit's C# files compile into - one assembly, so the shared runtime and the text form's generic walk are - emitted once per unit rather than once per translation unit behind a - guard. Every unit emits one further file per unit, `View.cs`, on - the same terms as the C++ pair above. + `namespace `. A unit that declares TABLES emits three further + files per schema file — `Table.cs` carrying the table wire's codecs, + its reflection descriptors and its TEXT FORM (SPEC-TABLES.md §16), and + `Block.cs` and `Cook.cs` for the two accelerators (§19, §7) — + plus one RUNTIME HOME per unit and per surface, `Table.cs`, + `Block.cs` and `Cook.cs`, where everything shared lands. + Each of the three is one file rather than the C++ header/source pair + because a unit's C# files compile into one assembly, so the shared runtime + and the text form's generic walk are emitted once per unit rather than once + per translation unit behind a guard; the home is named for the PACKAGE on + §19.2's rule for every port. Every unit emits one further file per unit, + `View.cs`, on the same terms as the C++ pair above. - **Dart:** one library per schema file, cross-file `import`s derived from actual references, with `show` clauses naming exactly the symbols used. A unit that declares TABLES grows three further libraries per schema file — @@ -1468,6 +1507,21 @@ tree mirrors the schema tree a person navigates. generated `_foo` at top level would be a collision no registry covers (SPEC-TABLES.md §11). A table-free unit grows none of it, and its packet libraries are byte-identical either way. +- **Java:** one `.java` file per schema file for the packet half, in + `package `. A unit that declares TABLES emits `Table.java` + per schema file with the table wire's codecs and its reflection + descriptors, and then FANS OUT: **a public Java type lives in a file of its + own name**, so the two accelerators are one file per declaration rather + than one per schema file — `Block.java` (SPEC-TABLES.md §19) and + `Cook.java` (§7) per table, and `Row.java` for every blittable + record in the closure, plain `type` members included (§20.3). The shared + runtime fans out the same way, one file per runtime type + (`TableReader.java`, `TableReport.java`, `TableJson.java` and the rest), + which is file-order independent by construction rather than by a rule and + is why this port needs no named home; each of those spellings is claimed + for every backend (SPEC-TABLES.md §11). `BuildVersion.java` is always + emitted beside them and belongs to neither accelerator (§20). A table-free + unit grows none of it. - **Elixir:** one `.ex` file per schema file, carrying one `defmodule` per declaration under the unit's own namespace plus the file-scope module `.` for constants, flags masks and the file's codecs. A unit that @@ -1492,9 +1546,17 @@ tree mirrors the schema tree a person navigates. in declaration order (specified defaults live in construction; `ZeroX` is the §5 zero form). **Generated JS never imports the serialize runtime** — every wire call is a method on the stream parameter, so no wiring file - exists and the checked/production fork stays where it lives, in the - runtime's own load-time mode selection (generated code never reads - `NODE_ENV`). A unit that declares TABLES emits three further modules per + exists and this tier's checked/production fork stays where it lives, in the + runtime's own load-time mode selection: the runtime-tier module reads no + `NODE_ENV` of its own. Beside it, **every schema file that declares types + also emits `Flat.js`** — the FLAT tier, a single-word bitpacker + inlined at every field with no runtime import at all, held byte-identical + to the runtime tier by a standing gate. That tier owns the fork itself: + `NODE_ENV` is read ONCE at module load, exactly as `serialize.js`'s own + `src/mode.js` does, and whole write variants are selected at export, so a + bundler that statically replaces `NODE_ENV` tree-shakes the checked writers + out. The READ side is never configurable in either tier. A unit that + declares TABLES emits three further modules per schema file — `Table.js` for the table wire's codecs, its reflection descriptors and its TEXT FORM (SPEC-TABLES.md §16), and `Block.js` and `Cook.js` for the two accelerators' READ side (SPEC-TABLES.md @@ -1580,20 +1642,43 @@ Go, zero third-party dependencies, one static binary: `schema`. schema check [--verbose] [dir|files...] // parse + typecheck; exit code for CI schema generate [--lang c|cpp|cs|dart|elixir|go|java|js|rust] [--out ] [--verbose] [dir|files...] -schema id | dir|files... // print the protocol id -schema projection | dir|files... // print the wire shape projection (§3.1) +schema id [dir|files...] // print the protocol id +schema projection [dir|files...] // print the wire shape projection (§3.1) +schema build-version [--facts] [dir|files...] // the cook/block id, and the text it digests (SPEC-TABLES.md §20) +schema tables-baseline [--update --reason "..."] [--verbose] [dir|files...] + // the table wire's evolution gate (SPEC-TABLES.md §18) schema fmt [--verbose] [dir|files...] // the canonical formatter, standalone (editors, hooks) +schema pack --root
--out [--tolerate] [--verbose] [dir|files...] +schema unpack --root
--in [--one-file] [--tolerate] [--verbose] [dir|files...] + // a text tree to a table-wire file, and back (SPEC-TABLES.md §17) +schema cook --root
--in --out + [--byte-order little|big] [--attribution ] [--tolerate] [--verbose] [dir|files...] +schema cook-check [--root
] [--attribution ] [--verbose] [dir|files...] +schema uncook --root
--in --out [--attribution ] [--verbose] [dir|files...] + // the cooked form: produce, validate, and back to the wire (SPEC-TABLES.md §7) schema version +schema help ``` +**Thirteen commands, plus `help`, which prints exactly the thirteen.** Six +serve the type wire this document specifies; the other seven belong to the +table wire and are specified in SPEC-TABLES.md, because one binary reads one +set of declarations and both wires are declared in it. + Success is silent. Commands whose printed output is their answer (`id`, -`projection`, `version`) print it; everything else prints nothing unless -`--verbose` asks for the per-file report — the files `generate` -wrote, the files the formatter rewrote, `check`'s ok line. Errors and +`projection`, `build-version`, `version`) print it; everything else prints +nothing unless `--verbose` asks for the per-file report — the files +`generate` wrote, the files the formatter rewrote, `check`'s ok line, the +header facts a cook was written with. Errors and diagnostics always reach stderr, and exit codes do not depend on verbosity. +`pack` and `unpack` exit nonzero when their read report is not silent, and +`--tolerate` accepts it. **Every command formats the unit's schema files in place before processing -them.** One style, no options, no separate binary; a file already in format +them**, `pack` and `unpack` excepted: those two are pointed at a config tree +and only READ the declarations, and a verb that assembles a wire file +rewriting the schema sources beside it is a surprise nobody asked for. One +style, no options, no separate binary; a file already in format is never touched. The formatter carries two built-in refusers: it re-parses its own output and structurally compares the AST against the input's, refusing to write on any difference — a formatter must never change @@ -1733,8 +1818,12 @@ Rules: dividers (`// ---- name ----`), and doc comments stay attached to what they precede. 9. **schemafmt never reorders declarations** — a formatter formats; it does - not move code. (Tags are sorted by name (§4.8), so ordering carries no - meaning — the aspect layout (§2) stays a convention.) + not move code. (Declaration order at file scope carries no wire meaning + (§3.1), so the aspect layout (§2) stays a convention. The order of + variants INSIDE an enum, a `flags` or a union is the opposite — declared + order is the wire, an enum's ordinal and a `flags` bit position and a + union's tag alike (§4.2, §4.8) — and that is the second reason a + formatter never sorts.) ## 8. Repository layout @@ -1755,10 +1844,10 @@ internal/check/ resolver, constant folding, shape checks, dominance rule, internal/format/ schemafmt internal/codegen/ c/ cpp/ csharp/ dart/ elixir/ golang/ java/ js/ rust/ — registered on the driver through the public - generator interface; cpptable/, cstable/, gotable/, - jstable/, rusttable/ and elixirtable/ are the table - emitters six of those backends carry - (SPEC-TABLES.md) + generator interface; ctable/, cpptable/, cstable/, + darttable/, elixirtable/, gotable/, javatable/, + jstable/ and rusttable/ are the table emitters, one + per backend, all nine (SPEC-TABLES.md) internal/fuzz/ compiler fuzzing (gate 6) internal/publicapi/ the acceptance gate: an external module, public API only examples/ the corpus — always compiles under this spec as written diff --git a/docs/USAGE.md b/docs/USAGE.md index a0d871b45..9823970c2 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -332,6 +332,12 @@ Signed fixed point: `I` integer bits (the sign bit counts), `F` fractional bits, stored as a raw scaled integer of exactly `I + F` bits. Bounds are in **whole units**. +**`I + F` must equal a storage width — 8, 16, 32, 64 or 128.** The raw scaled +value IS the storage, so a sum that names no integer type is refused at +compile time (`ufixed(16, 8)`: "I + F = 24 must equal a storage width"). +Split the width you want between the two: `fixed(24, 8)` and `fixed(16, 16)` +are both 32 bits, and which one you pick is where you want the point. + Fixed point is what you use when a value must be **bit-identical across machines**. Floating point is not: the same expression can differ by an ulp between compilers and architectures, which is fatal for lockstep simulation @@ -351,8 +357,9 @@ The unsigned sibling: no sign bit, whole-unit domain `[0, 2^I)`, same `F` semantics, stored as an **unsigned** integer of exactly `I + F` bits. Use it when the value cannot be negative and you want the storage type to say so — unsigned Q8.8 reaches 255 whole units where signed Q8.8 tops out at 127. -Everything else — required whole-unit bounds, exact round trips, defaults, -degenerate ranges — works exactly as for `fixed`. +Everything else — the `I + F` storage-width rule, required whole-unit bounds, +exact round trips, defaults, degenerate ranges — works exactly as for +`fixed`. ### Strings and bytes @@ -372,6 +379,18 @@ int32_t name_length = 0; The write refuses embedded NULs and any length past the maximum; the read validates both. +**`string(N)` also carries a UTF-8 contract, and it is the WRITER's.** The +wire is byte-identical to `bytes(N)`; what the `string` spelling adds is the +obligation that the payload is well-formed UTF-8 — never a reader's check, +so a reader must accept whatever a conforming writer produced. Because the +check is O(n), it is a DEBUG-only assert and only in the targets that carry +one: C and C++ assert through a generated validator, Rust through +`debug_assert!`, and C#, Dart, Elixir, Go, Java and JavaScript assert +nothing. So Latin-1 bytes in a `string(32)` fire an assert in a C++ debug +build and pass in a release one — if your payload is genuinely arbitrary +bytes, declare `bytes(N)`, which is the same wire with no encoding contract +(SPEC.md §4.7). + ### Arrays ``` @@ -451,13 +470,18 @@ compiler refuses one that would silently round. ## The wire -**The wire** is bit-packed and decided at compile time. Nothing on it +**The TYPE wire** is bit-packed and decided at compile time. Nothing on it identifies fields — both sides know the layout because they were generated from the same schema. That is what makes it small and fast, and why versioning is by [protocol id](#the-protocol-id): one id, same-or-refuse, -with no evolution machinery anywhere. schema is deliberately not an -evolution system; data that must survive schema drift wants a different -tool. +with no evolution machinery anywhere. That is a deliberate choice for data +whose writer and reader ship together, and it is one of TWO wires this +language has. Data that has to survive schema drift is the other one's job: +declare a `table` and it rides the tolerant wire, where fields carry their +name's hash, any reader reads any data, and every difference is counted in a +read report rather than being fatal (docs/SPEC-TABLES.md, and +[Tables](#tables-data-that-outlives-builds) below). Save games, config and +asset archives belong there; packets belong here. Generated storage is **relocatable by construction** — trivially copyable, standard layout, no pointers — so instances can be memcpy'd, memory-mapped, @@ -587,10 +611,11 @@ allocates, a fixed table WITH a union may allocate for the arm in a language that has no native union, and a variable-length table allocates by nature — in C++ the caller owns it. -A table lives on its own wire — evolution-tolerant TLV, carried by C++ and C -(both classes) and by C#, Go, Rust, Java and Elixir (the fixed class, wire and -text form both; the pointer surface ON THE WIRE is a follow-on in those five — -their cook and block accelerators read a pointered unit today). Field +A table lives on its own wire — evolution-tolerant TLV, carried by **all nine +targets**: C++ and C take both classes, and C#, Dart, Elixir, Go, Java, +JavaScript and Rust take the fixed class, wire and +text form both; the pointer surface ON THE WIRE is a follow-on in those seven, +whose cook and block accelerators read a pointered unit today. Field identity is a hash of the field NAME, so any reader takes any data, both directions: unknown fields are skipped, absent fields take their declared defaults, a field whose type changed is skipped rather than misdecoded, @@ -920,6 +945,10 @@ if !ship_config_load(&mut loaded, &buffer, &mut report) { // framing damage: report.malformed is set, the good prefix is kept } if report.unknown != 0 || report.kind_mismatch != 0 || report.clamped != 0 { + // the data came from a different schema generation — loaded is still + // fully usable; log the counts so drift is visible +} +``` **The Go surface is the same three functions again**, name first at package scope over a `*T` the caller owns. Storage is a plain struct — the Go packet @@ -1467,8 +1496,17 @@ Inside a table closure a union's arm may name a `table`, which is what makes a tool message set evolve safely: ``` -table OpenDocument { path string(256), line uint32 } -table SaveDocument { path string(256), force bool } +table OpenDocument +{ + path string(256) + line uint32 +} + +table SaveDocument +{ + path string(256) + force bool +} union ToolBody { @@ -2089,8 +2127,9 @@ unsafe { ``` A cooked file is an ACCELERATOR, not an archive: it is build-locked by a -build version that covers the schema's layout, its meaning facts and your -target's byte order, so it refuses the moment any of it moves and you +build version that covers the schema's layout and its meaning facts, and +target-locked by the byte order its header carries, so it refuses the moment +any of it moves and you regenerate it. The tolerant wire stays the format of record. `Open` checks the header and points — the magic, the byte order it @@ -2201,20 +2240,28 @@ puts the two back together when you want to check one. ### The build version: what a cooked asset is stored under *`schema build-version [--facts]` prints the id and the projection it digests, -both pinned as goldens; the C++, C#, C, Go, Rust and Java block backends emit -`BuildVersion` and stamp it into every block's prologue, and the JavaScript one -emits it to compare against; `schema cook` stamps the same id into every cooked -header, and `cook-check`, the C++ `Open`, the C# `Cook.Open`, the -Go `Open`, the Java `Cook.open` and the JavaScript `Cook.Open` -each read it back and compare. What is still owed is SPEC-TABLES.md §20's -status list.* +both pinned as goldens; every backend emits it — as a constant beside the +block form, or as a file of its own where the language wants one +(`BuildVersion.java`, `BuildVersion.ex`, `build_version.rs`) — and the +producing backends stamp it into every block's prologue while the reading +ones emit it to compare against. `schema cook` stamps the same id into every +cooked header, and `cook-check` and each port's cook entry point — the C++ +`Open`, the C `_open`, the C# `Cook.Open`, the Dart +`Cook.open`, the Elixir `cook_open_`, the Go `Open`, the +Java `Cook.open`, the JavaScript `Cook.Open` and the Rust +`Cook::open` — read it back and compare. What is still owed is +SPEC-TABLES.md §20's status list.* A cook is only ever produced for one build, so something has to name which build. That is the **build version**: one digest over everything a cook's bytes depend on — your protocol id, every record's layout as the compiler computes it, and the declaration facts that decide what a load puts in a slot (a specified default, a declared range, an enum's variant order, a union's arm -order) — plus the target's byte order. +order). **It is TARGET-NEUTRAL**: the byte order rides its projection as a +generation input and is `little` for every target today, so one id is shared +by every target of one game and a `--byte-order big` cook of the same tree +stamps the same number as the little one. What tells two orders apart is the +cook's own header, and the third coordinate of the key below. ``` $ schema build-version tables/block/ @@ -2226,12 +2273,16 @@ $ schema build-version tables/block/ printf( "%016llx\n", (unsigned long long) blockdemo::BuildVersion ); ``` -Your tools cook asset X to build version Y and write `(X, Y)` into the store; -your game asks the store for `(X, Y)`. That is the whole protocol. You never +Your tools cook asset X to build version Y for byte order Z and write +`(X, Y, Z)` into the store; +your game asks the store for `(X, Y, Z)`. That is the whole protocol. You never have to reason about which edits invalidate what — anything that would change a cook's bytes moves Y, so the key moves with it, and a new Y is simply a new -cook the build cache absorbs. The asset hash is the hash of the WIRE file you -cooked from. +cook the build cache absorbs. Z is there because the build version is +target-neutral: without it a store shared by a little-endian and a big-endian +target would key two different artifacts the same way, never serving wrong +bytes — the header still refuses — but missing forever. The asset hash is the +hash of the WIRE file you cooked from. It is settled by the **compiler**, not by your C++ compiler, which is what lets tooling cook before any game binary exists. The layout half comes from @@ -2290,12 +2341,17 @@ keep reading. Think of a save game. A player's file was written two years ago by a build nobody has any more, and today's build has to read it. Almost every schema edit since is safe by construction — fields came and went, an enum grew, -bounds moved — and the wire reports whatever it cannot use. **Exactly three +bounds moved — and the wire reports whatever it cannot use. **Exactly four edits are different**: they change what an OLD file MEANS, and nothing on the wire can tell you. Two are below; the third is a field's REFERENT dropped or swapped for one that cannot stand in for it — an enum-typed field respelled as its raw `uint16`, say, which rides under the same kind either way — and it -is the one this file's whole job is (SPEC-TABLES.md §4.1). +is the one this file's whole job is; the fourth is a `fixed` field's `F` +moved, where `fixed(16, 16)` and `fixed(8, 24)` ride under one kind and a +stored raw value reads back at the new scale (SPEC-TABLES.md §4.1). A fifth +edit belongs to the class and the baseline cannot see it yet: REUSING a name +you retired, where a re-added field takes the wire id of the one it replaced +— the retired-names ledger is schema#441. ``` table ShipConfig @@ -2458,10 +2514,10 @@ pointer-free, arrays inline with their `_count`/`_length` companions — so a value can be memcpy'd, mmap'd or shared across processes and still walked through descriptor offsets. Generated `static_assert`s enforce it. -Tables are generated for `--lang cpp` and `--lang cs` today — C++ carries -both classes, C# the fixed class, and a pointered unit is refused by name -under C#; every other target refuses a unit that declares tables at all, by -name. Every scalar the type wire carries rides in a table: `fixed`/`ufixed` +Tables are generated for every `--lang` today — C++ and C carry both +classes, the other seven the fixed class, and a pointered unit's WIRE +surface is refused by name in those seven while their two accelerators read +one in full. Every scalar the type wire carries rides in a table: `fixed`/`ufixed` as their raw scaled integer under a fixed kind of their storage width, with the whole-unit bounds clamping on the raw scale and the text in whole units (`1.5`), and `int128`/`uint128` under kinds of their own, sixteen bytes low From 0c18d99631432ce5821b727ade925b13097a16d4 Mon Sep 17 00:00:00 2001 From: Glenn Fiedler Date: Fri, 4 Sep 2026 12:26:14 +1000 Subject: [PATCH 2/2] =?UTF-8?q?docs:=20four=20more=20from=20the=20competit?= =?UTF-8?q?ion=20page's=20rework=20=E2=80=94=20the=20block=20form's=20prod?= =?UTF-8?q?ucers,=20and=20three=20competitor=20cells=20(#460)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **§19's block-form status said seven backends carry it live.** Only `cpptable/block.go` and `ctable/block.go` carry the builder half; every other block emitter's own header says the READ half or the READ side, and §19's Elixir paragraph already said that backend never produces a block or a cook. The sentence now says what the tree does: built by C++ and C, read by the other seven, and none of those seven emits a fill path. **Three COMPARISON-TABLES cells cited more than their sources say.** "Generated code dependencies" called FlatBuffers a header-only runtime, which no FlatBuffers page claims — FB-cpp has generated code needing `flatbuffers/flatbuffers.h` on the include path and the text and schema parsers linking further sources — so the cell states that instead. "Text form" said ProtoJSON is in every runtime; protobuf.dev's JSON page names its own nonconformant implementations, and the cell now carries them. "Doc comments" attributed source-info preservation to nothing on protobuf.dev; it now names `SourceCodeInfo` and its comment fields, with `descriptor.proto` added to the source list as PB-descriptor. The provenance line says which rows were re-read on 2026-09-04. Co-Authored-By: Claude Fable 5.1 --- docs/COMPARISON-TABLES.md | 10 ++++++---- docs/SPEC-TABLES.md | 6 ++++-- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/COMPARISON-TABLES.md b/docs/COMPARISON-TABLES.md index 1894f7855..968d34c3c 100644 --- a/docs/COMPARISON-TABLES.md +++ b/docs/COMPARISON-TABLES.md @@ -221,7 +221,7 @@ the source list at the end. | Conditional groups | `if guard { }` elided when false (SPEC §4.4, §3) | — | — | | Units and includes | one `package` per unit, all files compiled together, order-free; cross-file table references form a DAG (SPEC §3.2, §11) | `include`, nested `namespace` (FB-schema) | `import`, `import public`, `package` (PB-editions) | | Attributes | closed typed vocabulary right of `\|`; unknown is a compile error; type tags inert until claimed (SPEC §4.2) | user attributes declarable, read via reflection (FB-schema) | custom options with retention and targets (PB-editions) | -| Doc comments | deferred, design pinned (SPEC §4.1) | `///` into generated code and the binary schema (FB-flatc) | preserved in descriptors' source info | +| Doc comments | deferred, design pinned (SPEC §4.1) | `///` into generated code and the binary schema (FB-flatc) | the descriptor carries source info — `SourceCodeInfo`, whose `leading_comments` and `trailing_comments` a generator reads (PB-descriptor) | | Reserved names | after 3.0.0, a retired list in the baseline | — ; never remove, deprecate instead (FB-evolution) | `reserved` numbers and names (PB-proto3) | | Deprecation | — ; removal is free and counted | `(deprecated)`: accessors dropped, slot kept (FB-schema) | `[deprecated = true]` (PB-proto3) | | Required | — ; every field optional with a default (§4) | `(required)`, verifier-checked (FB-schema) | removed; `LEGACY_REQUIRED` only (PB-ed-overview) | @@ -292,7 +292,7 @@ the source list at the end. | Classes | fixed and variable derived from the declaration, held by a gate (§2.2) | struct and table declared; a table is always by offset (FB-schema) | one | | Performance obligations | the ladder: a fixed table beside its type on the ledger; the block matches hand-written scatter both sides; unexplained slowness is a defect (§12.1) | a benchmarks page | "fast parsing" (PB-overview) | | Big-endian targets | C++ proves wire, cook and block on an emulated target; Go, Rust, Java checked | accessors swap (FB-cpp) | neutral by varints (PB-encoding) | -| Generated code dependencies | C++ tables: C-like, no STL, C headers, hooks for assert, fatal, allocate, release (§13.9) | header-only runtime; STL in the object API (FB-cpp) | `libprotobuf` (PB-message) | +| Generated code dependencies | C++ tables: C-like, no STL, C headers, hooks for assert, fatal, allocate, release (§13.9) | `flatbuffers/flatbuffers.h` on the include path for generated code; the text and schema parsers link further runtime sources, and the object API uses the STL (FB-cpp) | `libprotobuf` (PB-message) | ### Validation of untrusted data @@ -310,7 +310,7 @@ the source list at the end. |---|---|---|---| | Runtime reflection | descriptors in every table's generated header: name, kind, id, offset, bounds, guards, nesting; no schema files, no RTTI (§8.1); the type view and registry specified (§8) | binary schema plus `reflection.h` in C++, basic in C; mini-reflection (FB-IR, FB-support) | descriptors, `Reflection`, `DynamicMessage` (PB-message) | | Reflection cost | on the side; a game build never compiles the view (§8.4) | 2 to 6 bytes per field for mini-reflection (FB-cpp) | in the full runtime always | -| Text form | JSON in and out by one walk, mapping pinned per kind, `\| json = "key"`, report counters; a shared node labeled `&node` (§16.7, landing with [#388](https://github.com/mas-bandwidth/schema/pull/388)) | JSON in `flatc`; parsing in C++ and C (FB-support) | ProtoJSON in every runtime; text format (PB-json, PB-text) | +| Text form | JSON in and out by one walk, mapping pinned per kind, `\| json = "key"`, report counters; a shared node labeled `&node` (§16.7, landing with [#388](https://github.com/mas-bandwidth/schema/pull/388)) | JSON in `flatc`; parsing in C++ and C (FB-support) | ProtoJSON, whose own page lists the implementations that do not conform to it — C++, Java and Python as of v25.x; text format (PB-json, PB-text) | | Whole-tree packing | `schema pack` and `unpack`: a directory mirrors the root; keyed arrays as one file per variant; byte-stable both ways (§17) | `flatc -b` (FB-flatc) | `protoc --encode` and `--decode` | | Dump, diff, check | `cook-check`, `uncook`, `build-version --facts`, `projection` (§7, §20.7); generic dump and diff over the registry a follow-on (§15) | `flatc --json`, `FlatBufferToString` (FB-flatc) | `DebugString`, third-party explorers (PB-3p) | | Lint, breaking, registry | the baseline and the checker; no registry by design (§18) | `--conform` (FB-flatc) | `buf lint`, `buf breaking`, the BSR (buf) | @@ -323,7 +323,8 @@ the source list at the end. ## Sources Every FlatBuffers and Protocol Buffers claim above was read from one of these -pages on 2026-09-03. +pages on 2026-09-03, and the rows citing PB-dos, PB-enum, PB-json, +PB-descriptor, FB-support and FB-cpp were re-read on 2026-09-04. | short | page | |---|---| @@ -352,6 +353,7 @@ pages on 2026-09-03. | PB-limits | https://protobuf.dev/programming-guides/proto-limits/ | | PB-dos | https://protobuf.dev/best-practices/dos-donts/ | | PB-message | https://protobuf.dev/reference/cpp/api-docs/google.protobuf.message/ | +| PB-descriptor | https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/descriptor.proto (`SourceCodeInfo`) | | PB-overview | https://protobuf.dev/overview/ | | PB-3p | https://github.com/protocolbuffers/protobuf/blob/main/docs/third_party.md | | buf | https://buf.build/docs/breaking/ | diff --git a/docs/SPEC-TABLES.md b/docs/SPEC-TABLES.md index 0980f609a..7ac72d39f 100644 --- a/docs/SPEC-TABLES.md +++ b/docs/SPEC-TABLES.md @@ -506,8 +506,10 @@ order-word check (`test/js-tables/main.mjs`): a file whose magic is intact and whose order word records the other order, which is exactly the file a reader leaning on one check would open. -**The BLOCK FORM (§2.7, §19) is live in C++, C, C#, Go, Rust, Java and Elixir, -and READ by JavaScript and Dart**, and it took C++ and C# TOGETHER to land, because the +**The BLOCK FORM (§2.7, §19) is BUILT by C++ and C, and READ by the other +seven** — C#, Dart, Elixir, Go, Java, JavaScript and Rust each emit the open +path, the projection and the accessors, and none emits a fill path — and it +took C++ and C# TOGETHER to land, because the form is an ABI between two languages and one language alone cannot hold the gate it exists for (§12.1). C++ emits `Block.h` (the projection, the generated layout asserts, the fill path