fix(runtime)!: recover scoped observation streams after loss - #212
Closed
danielkov wants to merge 3 commits into
Closed
fix(runtime)!: recover scoped observation streams after loss#212danielkov wants to merge 3 commits into
danielkov wants to merge 3 commits into
Conversation
Contributor
|
Review paused before completion. Rerun the review to try again. |
BREAKING CHANGE: Runtime side-channel events now use source-scoped epoch envelopes. Older readers cannot render these payloads; mixed-version loss remains fail-closed.
Contributor
Author
|
Reworked subtask state to ride ACP tool call events and removed the problematic code path. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Recover runtime observation after publisher queue loss and nested-stream lease expiry without reopening invalidated epochs. Increase publication capacity from 256 to 1,024 frames and preserve unknown-state reporting when fresh observations resume.
Motivation
A temporary backlog previously disabled authoritative publication for the worker's lifetime. Nested stderr forwarding also permanently rejected runtime events after loss, including later healthy heartbeats.
Impact
Runtime events now use source-scoped epoch envelopes and explicit open, heartbeat, and loss boundaries. This is a breaking private wire-format change: older readers cannot render the new payloads, and ambiguous legacy loss remains fail-closed. ACP itself is unchanged.
Recovery admits future observations; it does not reconstruct lost snapshots. The TUI retains invalidation tombstones, session filtering, and an incomplete-status warning. Maximum queued frame payload increases from approximately 4 MiB to 16 MiB per initialized transport.
Technical details
Ordered publisher recovery
Generation-tagged, bounded admission separates recoverable loss from terminal writer failure. The sole writer discards invalidated queued frames and completes an ordered loss/open boundary before reopening admission. Nonblocking publication claims coordinate periodic healthy epoch rotation, allowing recovery from receiver-only lease expiry without a reverse channel.
Source-specific nested authority
Forwarding preserves the chain of containing publishers. Receivers track source identities, epochs, and leases, reject invalidated epochs, and require newer explicit opens after loss. A healthy ancestor cannot reopen a failed descendant.
Guarantees and limits
Recovery establishes ordering, not absolute wall-clock freshness: newer boundaries may themselves be buffered. Sustained overload or blocked IO can prevent recovery. Authority trackers retain at most 4,096 identities without evicting tombstones; unseen identities then fail closed until stream replacement. See
docs/runtime-authority.mdfor the protocol, compatibility, and failure contract.