Repository navigation
feat(txfeed): time Alpenglow block footers; txfeed-sub -footers (BLO-49345) - #195
Merged
Merged
Conversation
…49345) txfeed.FooterTimes takes union frames with their arrival times and reports each block footer once, with its producer time, bank hash and client, the arrival of the frame after which this receiver held every byte of it, and that frame's send_ts_us. Latency is the arrival minus the producer time. Agave 4.3.2 stamps a footer as it builds it, at the end of the block, so this is the leader to receiver delay at the block's end, plus how far the receiver's clock is ahead of the leader's. Agave clamps the stamp to at least its parent's plus 1ns, so after a parent stamped by a fast clock the latency can read short. A footer opens an FEC set of its own, padded to 32 data shreds, so it is read from the set's first data shred once that shred is received or Reed-Solomon rebuilds it, and from the Assembler's batch when the same frame completed and released the batch. The batch alone waits for the padding and for the previous set's DATA_COMPLETE shred: the 10-22 capture holds no shred of the sets before its footers, so no footer batch decodes from it at all. A set's first data shred is not read while the shred before it is held without DATA_COMPLETE: it then continues an entry batch, whose bytes there can frame a footer. A footer longer than one shred is read once its shreds held, with no gap from the first, reach the length its framing states, joined once; one stating more than its set holds is never parsed, and costs no allocation. TestFooterTimes replays the 10-22 capture, each frame arriving at its own send_ts_us. The footers' own shreds give 93.885486 and 96.819720 ms: the 93.9 and 96.8 ms of the sampler deleted on 10-23, leader to forwarder times. Without those shreds, the 32nd shard of each set gives 107.333486 and 180.231720 ms, read from the set, or from the batch once the previous set's DATA_COMPLETE shred is primed. It pins both bank hashes, as read by hand from the capture's bytes. txfeed-sub -footers joins the union with -ssm, natively or through -relay, or reads it from -listen, and prints a JSON line a block, -n of them at most. It requires a non-blank -clock-source and -clock-offset, this host's clock minus that source's (timedatectl timesync-status's Offset negated), and records them on each line as clock_source and clock_offset_us; no offset is applied. fixtureFrames takes the fixture's name and frame count.
Contributor
|
🔗 Paperclip issue: BLO-49345 |
Contributor
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex (applied directly in-pod; no nested CLI).
Reviewed head: cf6f84f
Looks good. I checked FooterTimes.Add against Assembler internals in txfeed/assemble.go:
slots,sets,payload,data.hasanddata.complete;- the release of payloads on batch decode;
- the v3/v4 header offsets (both 28-byte
WireHeaderSize).
Checks I ran:
go test -race -tags purego ./txfeed/ ./cmd/txfeed-sub/passes at this head.go vetandgofmt -lare clean.
Critical Issues (0)
Important Issues (0)
Suggestions (2)
- [native-codex]
txfeed/footer.go:115-134: after a multi-shred footer is reported, each later frame of its set re-joins the shreds into a fresh copy and re-runsParseMarker, until the set's batch decodes and releasespayload[fec]. Onlyseendrops the result. The cost is bounded (≤64 frames a set, ≤32-shred copy), so this is not a blocker. A per-(slot, fec) "already reported" bit checked before the join would skip it. - [pr-review-toolkit:errors]
txfeed/footer.go:85-87: the "refused frame" guard only catches frames that created no set, such as a non-multiple-of-32fec. A dup or size-mismatched frame on an existing set still falls through and re-reads the footer. This is harmless today: the refused frame changes no state, andseendedups. But the comment overstates the guard. Comparingasm.Stats().Frames-Dups-Badbefore and afterasm.Addwould make it exact if that ever matters.
Strengths
- The footer is read from its own FEC set before the Assembler's batch, with dedup per (slot, bank hash). This is exactly what makes the 10-22 capture measurable. The test pins both latency paths (93.885486 / 96.819720 ms received, 107.333486 / 180.231720 ms recovered), as well as the batch path.
- Long-footer join safety:
- it stops at the first gap and counts only contiguous held shreds;
- it is capped at the set;
- a forged
0xfffflength allocates nothing, andTestFooterTimesBoundedWorkasserts this.
- The mid-entry-batch false positive is guarded. The remaining case (previous shred not yet held) is called out with a
ponytail:note, so the corner is documented, not hidden. -clock-source/-clock-offsetare mandatory and checked withfs.Visit, so an unset offset can never read as a measured 0, and each line records both. The-ncap inside a multi-footer frame is handled.
Recommended Action
- No blocking changes requested.
- Merge once the remaining required CI checks finish green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
txfeed.FooterTimesreports each Alpenglow block footer once, with:txfeed-sub -footersjoins the shred union (native SSM, through an AMT relay, or from-listen). It prints one JSON line per block and an exit summary with n, p50, p95 and p99.Why
BLO-49345 measures leader-to-receiver latency at sfo12, London and Tokyo. Prometheus has no temporal shred metric. The only footer numbers so far came from a sampler deleted on 10-23, and they turn out to be leader-to-forwarder times.
How
-clock-sourceand-clock-offsetare required, and both are recorded on each line, so no line reads as a measured zero offset.timedatectl timesync-status's Offset negated.Tests
TestFooterTimesreplaystxfeed/testdata/alpenglow-sets-2026-10-22.bin.gz(327 frames).go test -race -tags puregopasses, and gofmt and vet are clean. There are no new dependencies: go.mod and go.sum are unchanged.Deploy
None. This is new code in package
txfeedthatshred-txfeeddoes not call, and nothing publishes on merge, sincerelease.ymlis tag-only. The measurement runs are BLO-49345 step 2.