Skip to content

fix(runtime)!: recover scoped observation streams after loss - #212

Closed
danielkov wants to merge 3 commits into
mainfrom
fix/runtime-queue-capacity
Closed

fix(runtime)!: recover scoped observation streams after loss#212
danielkov wants to merge 3 commits into
mainfrom
fix/runtime-queue-capacity

Conversation

@danielkov

@danielkov danielkov commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Recover runtime observation after publisher queue loss and nested-stream lease expiry without reopening invalidated epochs. Increase publication capacity from 256 to 1,024 frames and preserve unknown-state reporting when fresh observations resume.

Motivation

A temporary backlog previously disabled authoritative publication for the worker's lifetime. Nested stderr forwarding also permanently rejected runtime events after loss, including later healthy heartbeats.

Impact

Runtime events now use source-scoped epoch envelopes and explicit open, heartbeat, and loss boundaries. This is a breaking private wire-format change: older readers cannot render the new payloads, and ambiguous legacy loss remains fail-closed. ACP itself is unchanged.

Recovery admits future observations; it does not reconstruct lost snapshots. The TUI retains invalidation tombstones, session filtering, and an incomplete-status warning. Maximum queued frame payload increases from approximately 4 MiB to 16 MiB per initialized transport.

Technical details

Ordered publisher recovery

Generation-tagged, bounded admission separates recoverable loss from terminal writer failure. The sole writer discards invalidated queued frames and completes an ordered loss/open boundary before reopening admission. Nonblocking publication claims coordinate periodic healthy epoch rotation, allowing recovery from receiver-only lease expiry without a reverse channel.

Source-specific nested authority

Forwarding preserves the chain of containing publishers. Receivers track source identities, epochs, and leases, reject invalidated epochs, and require newer explicit opens after loss. A healthy ancestor cannot reopen a failed descendant.

Guarantees and limits

Recovery establishes ordering, not absolute wall-clock freshness: newer boundaries may themselves be buffered. Sustained overload or blocked IO can prevent recovery. Authority trackers retain at most 4,096 identities without evicting tombstones; unseen identities then fail closed until stream replacement. See docs/runtime-authority.md for the protocol, compatibility, and failure contract.

@kit-code-agent

kit-code-agent Bot commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Review paused before completion. Rerun the review to try again.

BREAKING CHANGE: Runtime side-channel events now use source-scoped epoch envelopes. Older readers cannot render these payloads; mixed-version loss remains fail-closed.
@danielkov danielkov changed the title fix(runtime): increase publication queue capacity fix(runtime)!: recover scoped observation streams after loss Sep 13, 2026
@danielkov

Copy link
Copy Markdown
Contributor Author

Reworked subtask state to ride ACP tool call events and removed the problematic code path.

@danielkov danielkov closed this Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant