Verified structures: catalog, gates, transactional attach, and provider export - #150
Open
LiangSu8899 wants to merge 214 commits into
Open
Verified structures: catalog, gates, transactional attach, and provider export#150LiangSu8899 wants to merge 214 commits into
LiangSu8899 wants to merge 214 commits into
Conversation
Introduce flash_rt.structures: a registry over versioned structure specifications (boundary tensors, framework-neutral weight slots, reference implementation, qualification gates). First catalog entry is decoder_ffn, covering the gelu/silu activation, offset/direct norm weight, and none/ada_ln conditioning variants shared across the supported decoder families.
Structure-agnostic harness: derives the calling convention from the specification, resolves symbolic dims from actual tensors with consistency checks, judges an implementation against the reference (cosine / max-abs / p99 metrics vs explicit thresholds), and emits a machine-readable record keyed by a plan digest over spec content, variant, workload, implementation identity, and environment.
Bind-time weight packing and static-scale calibration feeding the fused FP8 gate/up -> activation -> down block from flashrt/flashrt-fp8-swiglu-ffn (gelu and silu entrypoints). Enforces the support envelope and non-empty calibration inputs at bind. The parity gate gains a bound-callable mode for implementations whose weights and variant are baked at bind time.
Maps the transformers-convention checkpoint layout (out-in linear weights, AdaRMS conditional norm with no learned weight, time-embedding modulation projected to scale/shift/gate, gated residual) onto the decoder_ffn slots and conditioning inputs.
Add the optional cond_gate boundary input: under the ada_ln conditioning variant the output residual becomes x + ffn_out * gate, matching the AdaRMS decoder families whose norm projection emits scale/shift/gate. The reference and the fp8_static implementation both honor it.
attach() swaps host modules atomically: all staged paths are resolved and validated before the first swap, any failure rolls back completely, and the returned handle restores the exact originals idempotently. fp8_static gains bind_mlp_seam for hosts whose replaceable boundary is the MLP module: shared packing and calibration with the full-structure bind, the host keeping its own norm, AdaLN gate, and residual. Parity metrics are exported for host-side acceptance checks.
Weight packing and calibration are setup-time operations; without no_grad the quantization chain recorded autograd history on the packed FP8 tensors, keeping every fp32 intermediate alive for the lifetime of the bound implementation (~1.7 GB per trunk-sized layer).
Second binding of the same structure: the PaliGemma trunk MLPs (standard learned-weight RMSNorm, no conditioning, plain residual) at prefix token counts. Together with the pi05 action-expert binding this covers both M regimes of the model with one structure definition.
Third binding of the same structure, first outside the pi05 family: Qwen2.5-1.5B-Instruct under a plain transformers host (silu activation, direct norm-weight convention, no conditioning).
torch.quantile rejects large inputs (LLM-logits-sized qualification outputs exceed its limit); kthvalue is exact and unbounded.
Second catalog structure: the non-gated LayerNorm + fc1/GELU/fc2 vision FFN block shared by the SigLIP towers of the supported VLA families, implemented over the fused FP8 GELU MLP Hub kernel (checkpoint-native weight layout, biases included).
Introduce the first schedule-layer structure. stage_pipeline specs declare a stage graph with cadence attributes instead of a tensor boundary; their parity ground truth is the host's own eager path under an explicit noise window, so the registry now carries kind/family/stages/conformance fields and region entries are unchanged. vla_tick_pipeline (cond_iter_pipeline family, tick specialization): obs_encode at observation cadence feeding a fixed-shape K-step denoise loop at tick cadence, noise as a SWAP window, condition buffers read-only. Includes the GR00T N1.6 binding with its capture rulings (eager backbone, shallow-copy mutation guard, hoisted noise site).
Same structure declaration as the GR00T N1.6 binding. obs_encode ruled eager: SmolVLM vision embeddings use a boolean-mask scatter that is illegal on a capturing stream (same blocker class as SigLIP2 CPU indexing), so the tick form is eager prefill into static KV buffers plus a captured denoise loop; noise uses the host's native injectable window.
Consumption now mirrors kernels.get_kernel: a single call discovers structure seams in the host (pattern-matched from the module tree, no hand-written binding required), calibrates on the caller's real forward passes, runs per-layer parity gates and family-level accuracy plus net-win gates, transactionally activates only what earns, and returns a Plan with a receipt and exact detach. Calibration entries: calibration="auto" captures during the provided forward(s); a file path loads a prior capture or saves this one, so dataset-scale calibration is a path string. Multi-frame via a sequence of forwards or frames=N. The net-win gate requires a margin (min_speedup, default 1.02x) so measurement noise can never activate a family. The swap machinery module is renamed attach.py -> swap.py to free the attach name for the front door; no behavior change.
The single-site counterpart of attach, with get_kernel ergonomics: pull one structure, bind it to the module you point at, plug it in yourself. bind extracts weights from the module, calibrates on caller-provided real inputs, and self-checks the replacement against the original before handing it back; a part that misses the parity gate raises GateRefused instead of being returned. Pass residual= to gate at the declared structure boundary (residual included) — the same measurement attach uses; without it the check runs on the bare seam, which is strictly harsher. The returned module carries a certification record (worst cos, gate boundary, dims, m profile, variant).
structures.capture(fn, windows=..., reference=...) is the schedule- structure counterpart of the region doors: it warms and records the hot stage into a CUDA graph on a side stream and returns a replayable CapturedStage. Declared windows are the tensors the caller may rewrite between replays (noise, observations, condition buffers); replay reads their current contents in place. Parity against the host's eager path under the same window contents and a margin-gated net-win check run inside the call; a stage that fails either raises CaptureRefused. Replay timing records its events on the replay stream — default-stream events only measure enqueue time and overstate the win by orders of magnitude.
A calibrated single projection (x[M,K] -> y[M,N], optional bias) with epilogue variants (gelu+quant, residual add) and a negotiable input dtype: fp8_static marks a producer-negotiated seam where an upstream norm/adaln producer emits fp8+scale and this region skips its own input quantization — decided at plan level and re-certified at the composed boundary. Elementwise work exists only as epilogue variants here, never as standalone regions. Discovery is qualified (weight-size floor, sibling q/k/v/o grouped into one family), not a blind scan of every nn.Linear.
linear_proj gains its first implementation: the fused BF16-entry FP8 projection (FP8 weights, static per-tensor activation scale, fused input quantization), with work-based qualification measured from standalone preflight — projections whose GEMM cannot amortize the fixed quantization cost are refused at bind time and the host keeps its Linear. Per-calibrated-M buffers are pre-allocated so the hot path is allocation-free under compile and graph capture. Discovery matches sibling q/k/v/o(out) projections as one family per attention block behind a weight-size candidacy floor — never a blind scan of every nn.Linear. The front door records bind-time refusals per layer, and impls now share one process-wide hub loader: importing the same kernel repo twice re-registers its fake ops and raises.
A captured stage's declared windows become frt_model_runtime_v1 boundary windows directly; ports reference them by name. This closes the absorption edge: attach + capture + export takes a torch host to a runtime the FlashRT serving mechanisms (Nexus tick, capsule snapshot/restore) consume exactly like a whitebox-produced one.
Calling the hub loader from forward makes dynamo trace through kernels.get_kernel's version resolution (network calls, inspect.Signature) — 26 graph breaks that fragment the surrounding compiled region, and a fragment boundary can drop host-side constant construction into eager code that is illegal under CUDA graph capture. Binding the op function once at bind time keeps the module a single traceable custom-op call, which is why the FFN implementations (which already did this) never fragmented.
…ication lifecycle
The routed forward captured module.o_proj into its closure at adapter time — before attach installs anything — so a structure seated at o_proj afterwards was never called. The ledger showed it plainly: sixteen bound projection seats, sixteen entries in seams_never_called, on both the automatic and explicit arms. The route now resolves self.o_proj at each call, so whatever is seated at that path when the call happens — the host Linear or an attached structure — is what runs, and detach restores the host path with nothing pinned.
The first explicit build stopped at 216 seats and lost 9% to the automatic path at the captured form. A kernel-bucket profile of both captured graphs in one process located every missing millisecond: 1.8 ms of norm/rope kernels (the per-head qk-norm+rope composition was never declared), 1.0 ms of stray fused elementwise (the projection, adaptive-norm and patch seats the discovery claims and the seat tables did not). The book now matches, seat for seat: the language o_proj and vl-self-attention output projections take FP8 seats; the cross blocks stop being refused whole — the producer still emits FP8 at the query projection's scale and to_q consumes it directly (the fp8-in form has no work-band floor), while to_k/to_v take plain seats that the cadence banks then absorb; the vl-self-attention subtree gets its ffn/pack/projection seats; the patch projection binds from the flattened checkpoint weight; and the per-head qk-norm-rope composition is engaged through the same adapter entry the automatic path uses, on top of the packs it requires. Measured (RTX 5090, captured form, production wiring): official host explicit 15.26 ms (1.533x) vs automatic 15.27 (1.532x); LeRobot explicit 17.08 (1.458x, zero run-time refusals) vs automatic 17.09 (1.457x, 96 guarded refusals from the vision rope family this host's loading recipe can never satisfy). Same 285-seat book, same number, bucket-for-bucket kernel profile within 0.02 ms — the two paths differ in who writes the seat book, not in what the seats can do. Teardown is now a gate: every adapter's undo rides extras["revert"] into attach, and detach restores the host bit-for-bit, measured by exact comparison of the untouched eager pass after detach.
…der proof Both GR00T hosts pay an eighty-six-gigaflop projection into the full vocabulary on every forward — a feature-extraction pipeline never reads those logits. Whether that dead code actually costs milliseconds depended, until now, on the host's wrapper form: one generation's graph lets the compiler dead-code-eliminate it, the other's lets the logits escape and burns 0.86 ms per call in a poorly-partitioned template. A cross-host kernel census against identical eager shape multisets is what separated "extra work" from "the same work, differently saved". The family settles it with a receipt instead of compiler luck: the recording pass keeps the pipeline's baseline output, pins the head to a zero-width return, re-runs the recorded request, and holds the pin only if the output is bit-identical. Any mismatch or error restores the head alone — the rest of the family's pins stand. Measured: the LeRobot captured form drops 16.78 to 14.85 ms and the two hosts land within 0.05 ms of each other — the cross-host gap was never semantics, and with this pin it is no longer compiler fortune either.
…book The vision qkv and output projections are the one large-M band both seat books left to the compiler, and the compiler's partitioning of them turned out to be host-form dependent: the same matrix product landed in templates three times apart under the two transformers generations. Forty-eight linear_proj seats (24 blocks x qkv+proj) take the question away from the compiler on every host — and take 0.3 ms with it on each. This is the explicit tier doing what it exists for: the author pushing the book past what automatic qualification claims, and answering to the same parity gate.
The measured section now carries the end state: at the matched book the arms meet within 0.01 ms; with the vision seats and the proven dead-head pin the explicit assembly lands at 14.90 / 14.85 ms on the two hosts — a 0.05 ms cross-host spread from one unchanged build() — while the automatic arm keeps its per-host character (15.16 clean, 16.20 with 96 guarded refusals). The cross-host paragraph records what the gap actually was: compiler fortune in unclaimed territory, not semantics.
Same code, both arms, any named precision scheme: the tier question
("which band does this device want?") becomes one flag on the same
harness instead of a second harness. First use measured the recorded
W4 recipe against the FP8-negotiated default on both hosts — and both
lost end-to-end (5090 19.2 vs 15.2, Thor 45.8 vs 44.5), which is the
receipt that full-band tiers are the wrong shape: the small-M denoise
band wants FP4, the large-M prefill wants FP8, and the split is a
shape fact, not a device fact.
The consumer half of the FP4 wire has existed since the W4 band landed — a pack that takes packed uint8 plus swizzled scale factors and skips its own quantization. This adds the producer half and the decision procedure that connects them. The producer's layer form gains nvfp4 emission: the fused kernel norms, modulates and quantizes into preallocated packed/SFA buffers, so a downstream pack takes the scale factors once at bind time and every call — eager, compiled, captured — reads the same storage. The packed tensor rides the host's own glue at half feature width; the wire-fed pack's stash tails admit it as the negotiated form (the per-call dtype contract belongs to the fallback path). Which chain is faster — fp8 producer feeding an fp8 pack, or nvfp4 producer feeding the wire — is a property of the device's GEMM bands at the calibrated row count, so the negotiation measures it in place: both chains built on the real conditioning and shape, timed, winner seated, decision recorded in plan.notes["format_race"]. A candidate that cannot build or run loses by default and the fp8 chain stands. First measurement, RTX 5090, GR00T N1.7 DiT (rows=41, dim=1536): the wire wins all sixteen races by ~6% per chain and the end-to-end capture holds its band at 0.9994 parity — the small-M denoise band is bandwidth-bound enough that half-width weights win even where FP8 was never the bottleneck.
…nd out projections at the denoise rows
The mechanism's fifth layer: a seat family declares its form candidates; qualification predicates read device-neutral facts (shape, calibration, package availability); the qualified candidates are timed on the calibrated shapes and the measurement seats the winner, with every outcome — unqualified, refused, timed — in the receipt. Precision precedes speed twice: a form enters only above the 0.99 parity floor, and within the win margin the higher-precision form takes the seat, so measurement noise never flips one. The explicit assembly's shape-conditional FP4 branches are gone: the ffn, the adaln chain and the output projections now declare both forms and let the adjudicator seat them — one book, and each device measures its own optimum into it.
…he v2 cache The seat-level micro-race was refuted in both directions by the production form: on the 5090 it seated FP4 everywhere and lost 2 ms end-to-end; on Thor it seated FP8 back and lost 6 ms. A kernel timed outside its compiled fusion context is not a verdict. Until the band-level adjudicator with its decision cache lands (design doc, records repo), the explicit band is an author pin backed by the two captured-form receipts we hold: FP4 for the denoise band where it measured 37.10 ms, FP8 where it measured 14.90 — selected by FRT_DIT_BAND, defaulting to the FP4 pin, recorded as such.
The H3 lesson, mechanized. Seat scratch — sibling stashes, producer residual workspaces, the FP4 wire buffers — lives one layer's forward at a time, so sequential layers now share pooled buffers keyed by (shape, dtype, tag): the memory bill drops from layers x tokens to one layer's worth, and capture stays sound for the same reason memory pools are capture-sound (fixed pointers, non-overlapping same-stream lifetimes). The pool reports bytes held per tag — the receipt's memory column. Binding is budgeted: below 512 MiB free VRAM a seat refuses with insufficient_vram(free, headroom) instead of eating the remainder and converting into an unattributable OOM at the first treated forward. And reversibility becomes a tier: attach keeps the originals (the development default), and after the parity gate handle.finalize() frees them — bytes in the receipt, seams flipped from fall-back to refuse (there is nothing left to fall back to), detach forbidden with the reason.
The production-form band measurements are the only verdicts that survived; this is where they live. A band run records the winner per (device, band) into a transportable cache file; the explicit assembly reads it as its default band (author pin still outranks it, an empty cache falls to the precision-order default), and the automatic path routes its small-M ffn seats through the same entry — never a shape rule alone, never a device name in code.
…n interface, routed by the registry the pins already resolve
…ed by the measured band decision
The adapter refused every masked site at the door, which orphaned the DiT cross-attention band on a family that could already serve it: the fa4 variant packs a constant mask pattern into ranges at bind time and the masked-mha variant exists for exactly this column form. Masked captures now reach the family — pattern constancy is the variant's own qualification — and the routed processor accepts a live mask only when its core baked this site's ranges; a maskless core still keeps the host path rather than reuse a frozen mask.
…sks — two was a guess, the copy loop is linear in segments
The multi-segment recognizer let the FA4 form bind the DiT cross sites, and the captured-form number went twenty milliseconds the wrong way: availability plus precision order decided who got weighed, and nothing decided whether the winner actually served. Now every bound variant races the host's own attention on the captured shape before it seats; one that measures slower steps aside with the reason in the trail, and the host keeps the site. Bands are measured, not conceded — in this family too.
…als in the GEMM epilogues, norms emitting FP4, per-step modulators from bind-time tables
… the mask disappear and SDPA keeps its fused backend
…s the reference; the mechanism form is a composite family, designed before built
…ool — call-lifetime buffers were paying per-layer rent (~900 MiB a layer on a 19k-token host)
…sumed originals live in the checkpoint file or a host-RAM spill, fallback and detach restore from the store, and serving retention is a declaration
…ore anything binds, consumption follows attach, and the report carries per-phase peak memory
…rm from its first run, and a local measurement outranks an imported one
…r originals off-device as they land, and the bind peak stays near one model
…stages record through the bound seats, so shape probes stay true and the model stays runnable mid-bind
…t CUDA consumers see the space, and the receipt says how much
…rbed by a bind-time channel permutation, and the call-scoped outputs share one pooled lane per shape
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Structures as specs with references and gates, qualified implementations over Hub kernels, one-call assembly onto a host, and an export path that packages a qualified host as a standard model runtime. Tracks #149.
Usage
One call
forwardis any callable that runs the host once. Nothing else is asked of the host — no module paths, no hooks, no scale plumbing.attachgates before it commits, unit by unit, where a unit is a structure — except that a negotiated FP8 chain is one unit, since the producer emits under a scale the consumer was bound for. Each unit is judged on accuracy, on whether its seams actually ran, and on a net win timed by alternating both arms every round. The union is re-checked before commit; a refusal names the metric or the shape it was measured at, and a whole-host refusal leaves nothing behind.To build a plan without judging it — development, or when you own the gate:
plan.swapsis{module_path: replacement}.plan.notescarries the receipt: what was refused and why, which seams were negotiated as a chain, how calibration was obtained, and anything discovery had to assume.Runtime contract
A bound structure is calibrated for one executable form: a device, an input dtype, a width, and where buffers were preallocated, a row count. Called outside it, a seam runs the retained host module instead and records that it did.
The ledger exists because falling back is numerically exact, so a seam that quietly reverted is invisible to a parity check and to a passing test — it shows up only as latency that is not there. The first fallback per seam warns; a seam that falls back on 32 consecutive calls restores the host module permanently rather than staying on as a claim it no longer meets. Counts are eager-only: inside a compiled or captured region the kernel runs without re-entering Python, which is also why the check costs nothing there.
Refused rather than approximated: training mode, device or dtype migration while attached, loading a state dict into a model whose packed weights came from the old one, and a second thread entering one seam (its stash and scratch buffers are shared across that seam's calls by design).
state_dict()delegates to the retained host module, so saving while attached yields the same schema and the same bytes as the unattached model.Calibration
Three ways in, one axis.
forwardis always "run the host once".samplesis any iterable; with it,forwardtakes one sample.framesdefaults to one, and to the whole ofsampleswhen a sample source is given.The statistic is this repo's own. Within one sample: the max over every call a seam sees — one forward already covers every step of an iterative host. Across samples:
flash_rt/core/calibration.py'saccumulate_amaxat percentile 99.9, so one outlier sample cannot inflate every scale. What is held per sample is one float32 vector of per-point amaxes, never activations — 16 samples cost +0.11 GiB peak on Pi0.5 — so a larger sample source costs time rather than memory. The same helper's dispersion summary and scale-ceiling check run on every calibration and name the layers whose scales sit far above the median.plan.notes["calibration"]records{frames, source, stat, keep_samples}.Precision profiles
scheme=is the one precision entry, on both doors. A profile is aregistered quantisation scheme selected by name:
"auto"(the default)resolves to the fastest profile the device can execute — static FP8 on
FP8-capable hardware,
"none"elsewhere;"none"is the explicitoff-switch, under which fusion structures still attach and every
quantised seam stays at host precision;
"w8a16_decode"and"w4a16_decode"are weight-only INT8 / NVFP4 on the FFN decode band.Quantisation happens at attach time from the host's own weights —
scales and packed formats are derived at bind, activation statistics
come from running the host's forward — and detach restores the original
bit-exactly. Hardware support is not declared in this layer: the loader
reads the arch list each kernel package ships in its own metadata and
refuses a device outside it, by name, before the kernel can fail
somewhere less legible.
Single site, visible
bindextracts weights, calibrates, builds the replacement and self-checks it against the original on those samples. Missing the parity gate raisesGateRefusedrather than returning a bad part.Graph boundary
Every varying input must be a declared window; in-graph RNG never matches eager, so noise is a window rather than a call.
Development bench
Same-process A/B with a baseline re-time. A lever that does not both stay accurate and win latency is refused and recorded.
Catalog
decoder_ffnvision_ffnlinear_projbias/no_bias/fp8_in, each with its own measured work bandqkv_packadaln_producernorm_fusedattention_coredecoder_blockcadence_staticvla_tick_pipelineEach entry is
catalog/<name>/structure.yaml(boundary, weight slots, variants, qualification, gates, evidence) plus a plain-torch reference used as the gate's ground truth.Code
catalog/registry.pydiscover.pyautobuild.pyfrontdoor.pyguard.pyimpls/adapters/swap.pyhandle.pyget(name).bind(...)explicit doorstages.pycapture(...)graph doorrecipe.pyprovider.pyfrt_model_runtime_v1bindings/beta/Hosts
Four independent host stacks, no shared integration path between them:
lerobot/smolvla_basenvidia/GR00T-N1.6-3BtransformersResults
Two numbers per host: how much faster, and how close the output stays. The
baseline says which form of the host it is — eager, or
torch.compile.Output match is same input, compared output, against that host's own
unmodified form.
torch.compile+ CUDA graphgeneratetorch.compilegeneratetorch.compilehostgenerateOutput match is same input, compared output, against that host's own
unmodified form, on twelve inputs that were not in the calibration set.
Median first, then the worst of the twelve. The Pi0.5 latency is given as a
range because single-arm timing on this machine drifts several percent
between runs; a firm number needs the paired method.
Pi0.5 is in the band this repo's own native path measures against an FP32
reference — 0.9996-0.9998 (
docs/calibration.md§10). Getting there neededa defect fixed, and it is the kind this layer exists to surface. The
attention core keeps the prefix — the vision and language keys and values —
packed and rewrites only the suffix per denoise step. Correct within one
observation, and that is exactly what the bind-time check proves. Not
correct across observations: a new image means a new prefix while the packed
region still held whatever calibration captured, so later observations had
attention computed against the wrong frame. That seam is now offered only
when the caller declares it will run the refresh at the observation cadence
(
prefix_cadence=True), which the tick pipeline does; unbound is theaccurate default and costs about 0.09× of the speedup.
The Pi0.5 figure is the assembled form itself — structures, then
torch.compile, then one captured graph with observations enteringthrough static windows — measured on held-out frames. The graph matches
the compiled form bit-for-bit, and a frozen-window negative control
degrades to 0.73, so the windows are complete. The 0.59 worst frame is
not corruption: on that frame the host's own two samples (same frame,
two noise seeds) agree only to 0.78, and the frames where the quantised
arm strays are exactly the frames with the widest natural spread — a
quantisation error far below the mode distance can tip a flow sample
into another mode. Same-input cosine on a multimodal frame measures
which mode was sampled, not fidelity; the arbiter there is task-level
evaluation. Widening calibration noise coverage (one seed to four) does
not move it, and on unimodal frames the worst stays at 0.999.
SmolVLA and Qwen3-8B first measured far lower (0.979 worst 0.854; 82%
worst 63%), and that was the measurement's data, not the mechanism.
SmolVLA — trained on SO100-format robot data — was being calibrated and
judged on frames from a different robot suite through a hand-assembled
camera and state mapping; Qwen3 on short fragments repeated and padded
out to length. Re-measured on each host's real inference distribution —
training-domain SO100 episodes through the checkpoint's own preprocessing
pipeline, natural-corpus token windows with no padding — with a bit-exact
null check before attaching and a clean ledger, Qwen3-8B lands as
tabled and SmolVLA's quantised-seam arm at 0.9987 (worst 0.9935).
The
w4a16_decoderow is the gated form: the per-seam gate admits theseven middle FFN layers whose 4-bit parity clears the value band, and
the end-to-end grade is judged as a language model — teacher-forced
same-token rate, since free-running generations stop being comparable
at the first token that differs. The ledger for that row shows zero
fallbacks, 987 decode calls on the kernel path and 14 prefill calls
dispatched to the host by the declared band. Admitting all 36 layers is
a caller's choice via
floors=: it measures 1.34× at a 93.8%teacher-forced same-token rate, which is the warn band, and the receipt
records it as such.
Calibration and parity data must come from the host's real inference
distribution; out-of-distribution input alone cost 0.02 of cosine and 13
points of token agreement here. The remaining gap to Pi0.5's band is what
static per-tensor W8A8 costs over 180 seams; lifting it is a
quantisation-scheme question (per-channel or mixed width), not a
calibration-set one.
GR00T's figure is measured over held-out episodes with the row-locked
structures (packed QKV, block assembly) left unbound: its prompts are
natively variable-length, so on any length other than the calibrated one
those seams fall back by contract — numerically exact, and the ledger is
what separates that arm from a clean measurement. The full
assembled recipe measured the same way lands at 0.9994 (worst 0.9988) —
with the caveat that on this demo data two different frames' reference
actions already agree at about 0.9996, so the dataset itself sets the
measurement floor there.
The tabled SmolVLA figure is the tick form the latency column describes.
On this host the net-win gate refuses the FP8 region (0.93×), so the
tick form carries no quantised seams and its match is capture
numerics — measured over held-out episodes by refreshing the static
prefix buffers per observation and replaying, with a frozen-buffer
negative control (0.9937, worst 0.739) proving the check can see a stale
prefix. The same-observation replay is bit-identical, which is the replan row;
both latencies are from one run on the same held-out episode.
Figures in earlier revisions of this description were higher because they
were measured on the same input the scales were calibrated on, which reports
how well a fit fits itself.
GR00T's 5.10× is two levers multiplied: the schedule structure takes 57.4
to 15.42 ms, and the region structures take that to 11.26.
The GR00T row is also the reuse statement: the same
vision_ffndefinitionqualifies both a SigLIP encoder tower and a DiT diffusion action head, on a
host stack that is neither transformers nor lerobot.
Earlier region-only results on the LLM hosts, kept as recorded:
decoder_ffn×28decoder_ffn×36decoder_ffn+vision_ffnStructure vs standalone kernel swap
Same Hub kernels, same calibrated scales; the only variable is the composition.
torch.compilemax-autotuneChoosing the parity metric for a host
Whole-tensor cosine over logits is the wrong measure for a language host, and measuring it exposed why. Same bindings, same weights, two prompt lengths:
decoder_ffn×36decoder_ffn×36The two metrics move in opposite directions with length: the aggregate cosine falls while token agreement rises. Aggregated over every position it is dominated by positions that never drive a decision, so it tracks sequence length more than output fidelity. The generation-relevant quantities — the last position, top-1 agreement, per-token KL — hold across both lengths and across structure sets.
The gate therefore selects its metric from the host's output type, read off what the host returns: token agreement and last-position fidelity for a distribution, cosine for a value. Bands are per output kind: value outputs
passat cosine 0.999 /warnfrom 0.995; distribution outputs are judged on token agreement, where a clean static W8A8 with every per-seam gate passing measures 0.95–0.98 on real text — the grade of the quantisation, not damage — sopassfrom 0.95 /warnfrom 0.85. None refuses;floors={...}is the caller turning a number into a hard requirement. The older LLM rows above were scored the aggregate way and are therefore length-dependent; they are kept as recorded rather than restated.Parity belongs to a workload, and has to be measured held out
Static per-tensor scales are calibrated from data, so a parity figure belongs to a workload rather than to a host — and a figure measured on the frame it calibrated on is measuring its own fit. Pi0.5 on LIBERO, calibrated on 8 episodes and evaluated on 12 different ones (action cosine against the unmodified host on each frame, same noise, ledger clean):
The in-sample figure is optimistic by about 0.0026, so held out is the number that means anything. These rows were measured before the attention-prefix cadence fix; the current held-out figure is the Results table's. Parity figures elsewhere in this description that were measured on the calibration frame are labelled as such.
Calibration-set size is settled: redone with the repo's reducer and its stratified sampler at 1, 8 and 64 samples, the held-out figure moves by under 0.0001 cosine and the max deviation by about 2%. What does move it is the data's distribution — the SmolVLA and Qwen3-8B story under Results.
Validation
attach/detachis transactional in both directions: failed resolution leaves the model untouched, anddetachrestores the module tree, any routing an adapter patched, and bit-exact output.provider.pyand passes the same serving adoption/tick acceptance as the native pipelines.tests/test_structures_guard.py).Boundaries
eval()mode. Training, sharded parameters, concurrent use of one attachment, and migration while attached are refused rather than approximated.