Skip to content

feat(txfeed): time Alpenglow block footers; txfeed-sub -footers (BLO-49345) - #195

Merged
kkroo merged 1 commit into
mainfrom
kkroo/footer-sampler-blo49345
Oct 27, 2026
Merged

kkroo merged 1 commit into
mainfrom
kkroo/footer-sampler-blo49345

Conversation

@kkroo

@kkroo kkroo commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

What

txfeed.FooterTimes reports each Alpenglow block footer once, with:

  • its producer time, bank hash and client;
  • the arrival of the frame after which this receiver held every byte of it.

txfeed-sub -footers joins the shred union (native SSM, through an AMT relay, or from -listen). It prints one JSON line per block and an exit summary with n, p50, p95 and p99.

Why

BLO-49345 measures leader-to-receiver latency at sfo12, London and Tokyo. Prometheus has no temporal shred metric. The only footer numbers so far came from a sampler deleted on 10-23, and they turn out to be leader-to-forwarder times.

How

  • Where the footer is read: a footer opens an FEC set of its own. It is read from that set's first data shred as soon as the shred is received or rebuilt, and from the Assembler's batch when that completes first. The 10-22 capture holds no shred of the sets before its footers, so the batch path alone never decodes one from it.
  • Once per block: one report per (slot, bank hash).
  • Frames skipped:
    • frames the Assembler refused;
    • a set whose previous shred is held without DATA_COMPLETE, because that set continues an entry batch whose bytes could frame a footer.
  • Long footers: a footer longer than one shred is joined once, and only when its held shreds, with no gap, reach the length its framing states. One stating more than its set holds is never parsed and costs no allocation.
  • Clock: latency is the arrival minus the producer time, with no clock correction.
    • -clock-source and -clock-offset are required, and both are recorded on each line, so no line reads as a measured zero offset.
    • The offset is this host's clock minus its source's, so timedatectl timesync-status's Offset negated.
  • What latency means: Agave 4.3.2 stamps the footer at the end of the block and clamps it to at least its parent's stamp plus 1 ns. So latency is the end-of-block leader-to-receiver delay plus the clock offsets. After a parent stamped by a fast clock it can read short.

Tests

  • TestFooterTimes replays txfeed/testdata/alpenglow-sets-2026-10-22.bin.gz (327 frames).
    • The footers' own shreds give 93.885486 and 96.819720 ms. These are the 10-23 sampler's 93.9 and 96.8 ms, which were leader-to-forwarder times.
    • Without those shreds, the 32nd shard of each set gives 107.333486 and 180.231720 ms.
    • Both bank hashes are pinned from the capture's bytes.
  • Edge cases covered:
    • a refused frame;
    • a footer framing in the middle of an entry batch, with a positive control;
    • a 3-shred footer with its middle shred missing;
    • a forged 0xffff body length, which allocates nothing;
    • the JSON field mapping and units;
    • the flag refusals.
  • Mutation-checked: removing any guard fails a test.
  • go test -race -tags purego passes, and gofmt and vet are clean. There are no new dependencies: go.mod and go.sum are unchanged.

Deploy

None. This is new code in package txfeed that shred-txfeed does not call, and nothing publishes on merge, since release.yml is tag-only. The measurement runs are BLO-49345 step 2.

…49345)

txfeed.FooterTimes takes union frames with their arrival times and reports
each block footer once, with its producer time, bank hash and client, the
arrival of the frame after which this receiver held every byte of it, and
that frame's send_ts_us. Latency is the arrival minus the producer time.
Agave 4.3.2 stamps a footer as it builds it, at the end of the block, so this
is the leader to receiver delay at the block's end, plus how far the
receiver's clock is ahead of the leader's. Agave clamps the stamp to at least
its parent's plus 1ns, so after a parent stamped by a fast clock the latency
can read short.

A footer opens an FEC set of its own, padded to 32 data shreds, so it is read
from the set's first data shred once that shred is received or Reed-Solomon
rebuilds it, and from the Assembler's batch when the same frame completed and
released the batch. The batch alone waits for the padding and for the
previous set's DATA_COMPLETE shred: the 10-22 capture holds no shred of the
sets before its footers, so no footer batch decodes from it at all. A set's
first data shred is not read while the shred before it is held without
DATA_COMPLETE: it then continues an entry batch, whose bytes there can frame
a footer. A footer longer than one shred is read once its shreds held, with
no gap from the first, reach the length its framing states, joined once; one
stating more than its set holds is never parsed, and costs no allocation.

TestFooterTimes replays the 10-22 capture, each frame arriving at its own
send_ts_us. The footers' own shreds give 93.885486 and 96.819720 ms: the 93.9
and 96.8 ms of the sampler deleted on 10-23, leader to forwarder times.
Without those shreds, the 32nd shard of each set gives 107.333486 and
180.231720 ms, read from the set, or from the batch once the previous set's
DATA_COMPLETE shred is primed. It pins both bank hashes, as read by hand from
the capture's bytes.

txfeed-sub -footers joins the union with -ssm, natively or through -relay, or
reads it from -listen, and prints a JSON line a block, -n of them at most. It
requires a non-blank -clock-source and -clock-offset, this host's clock minus
that source's (timedatectl timesync-status's Offset negated), and records
them on each line as clock_source and clock_offset_us; no offset is applied.
fixtureFrames takes the fixture's name and frame count.
@allyblockcast

allyblockcast Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

🔗 Paperclip issue: BLO-49345

@allyblockcast allyblockcast Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex (applied directly in-pod; no nested CLI).
Reviewed head: cf6f84f

Looks good. I checked FooterTimes.Add against Assembler internals in txfeed/assemble.go:

  • slots, sets, payload, data.has and data.complete;
  • the release of payloads on batch decode;
  • the v3/v4 header offsets (both 28-byte WireHeaderSize).

Checks I ran:

  • go test -race -tags purego ./txfeed/ ./cmd/txfeed-sub/ passes at this head.
  • go vet and gofmt -l are clean.

Critical Issues (0)

Important Issues (0)

Suggestions (2)

  • [native-codex] txfeed/footer.go:115-134: after a multi-shred footer is reported, each later frame of its set re-joins the shreds into a fresh copy and re-runs ParseMarker, until the set's batch decodes and releases payload[fec]. Only seen drops the result. The cost is bounded (≤64 frames a set, ≤32-shred copy), so this is not a blocker. A per-(slot, fec) "already reported" bit checked before the join would skip it.
  • [pr-review-toolkit:errors] txfeed/footer.go:85-87: the "refused frame" guard only catches frames that created no set, such as a non-multiple-of-32 fec. A dup or size-mismatched frame on an existing set still falls through and re-reads the footer. This is harmless today: the refused frame changes no state, and seen dedups. But the comment overstates the guard. Comparing asm.Stats().Frames-Dups-Bad before and after asm.Add would make it exact if that ever matters.

Strengths

  • The footer is read from its own FEC set before the Assembler's batch, with dedup per (slot, bank hash). This is exactly what makes the 10-22 capture measurable. The test pins both latency paths (93.885486 / 96.819720 ms received, 107.333486 / 180.231720 ms recovered), as well as the batch path.
  • Long-footer join safety:
    • it stops at the first gap and counts only contiguous held shreds;
    • it is capped at the set;
    • a forged 0xffff length allocates nothing, and TestFooterTimesBoundedWork asserts this.
  • The mid-entry-batch false positive is guarded. The remaining case (previous shred not yet held) is called out with a ponytail: note, so the corner is documented, not hidden.
  • -clock-source / -clock-offset are mandatory and checked with fs.Visit, so an unset offset can never read as a measured 0, and each line records both. The -n cap inside a multi-footer frame is handled.

Recommended Action

  1. No blocking changes requested.
  2. Merge once the remaining required CI checks finish green.

@kkroo
kkroo merged commit 1433e8d into main Oct 27, 2026
24 checks passed
@kkroo
kkroo deleted the kkroo/footer-sampler-blo49345 branch October 27, 2026 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant