Skip to content

feat(server)!: stream everything a mirror sends the server over one stream - #97

Merged
jvmakine merged 5 commits into
mainfrom
juho/update-stream
Nov 4, 2026
Merged

jvmakine merged 5 commits into
mainfrom
juho/update-stream

Conversation

@jvmakine

@jvmakine jvmakine commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

A mirror sent its whole state to the server every 30s with a unary Update, so the server learned of a quarantine only when the next call arrived. Upcoming pod traffic, such as captures, would have needed its own calls. Now everything a mirror sends the server travels over one ordered Update stream, as agreed in the review of #96.

The reporter keeps the mirror's one stream open:

  • Every stream it opens begins with the mirror's whole state, then sends each quarantine as it begins, so a stream can fail at any time without losing state. A quarantine whose send fails is in the next stream's state.
  • It sends a keepalive when the stream is quiet for --server-keepalive-interval (30s). This keeps the stream inside the 60s idle limit Caddy puts on request streams (found in staging, reproduced locally) and the server's 90s state TTL, and a failed keepalive is how a mirror notices a dead stream.
  • A failed send means the server ended the stream. The reporter closes the stream to learn why, logs it, and reconnects with jittered backoff from 1s to 1m.
  • It replaces each stream after --server-stream-max-age (10m, with jitter), so streams spread across server replicas after a deploy and never depend on mesh connection limits.
  • On the server, a middleware lets a handler end a stream early. Without it, Go's HTTP/1 server reads the rest of the request before replying, and a mirror that keeps sending never learns the stream ended.

The stream is client streaming rather than bidirectional, so it works over HTTP/1.1 through the mesh egress socket. A mirror's messages are a model.Update sum type of the whole state or a quarantine. Captures will add a variant. Keepalives belong to the stream rather than the update type. The reporter is the only stream for now; when captures need the same stream, we can pull out a shared abstraction with two real callers to shape it.

flowchart LR
  Q[proxy quarantines] --> R[reporter]
  R -- "Update(stream UpdateRequest): whole state, then quarantines" --> S[SPECTRE server]
Loading

Breaking changes:

  • Update is now a client-streaming RPC whose first message is the mirror's whole state.
  • --heartbeat-interval is replaced by --server-keepalive-interval and --server-stream-max-age.

🤖 Generated with Claude Code

jvmakine and others added 4 commits November 4, 2026 14:35
…tream

A mirror's state and its changes now travel over one ordered stream, so the
server learns of a quarantine as it happens from the same place as everything
else, and later traffic such as captures can share it instead of opening
its own connection. The reconnecting, resyncing and keepalives are generic, so
the next kind of message only adds a variant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keeping a quiet stream alive is the stream's concern, so every kind of
message a stream carries would otherwise need its own keepalive variant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OutChannel says what it is: the outgoing half of a channel to the server,
which keeps its stream open, rather than a mailbox messages wait in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In staging, Caddy ended request streams that were quiet for 60s, so a
longer keepalive interval would drop every quiet mirror's stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jvmakine
jvmakine marked this pull request as ready for review November 4, 2026 04:08
@jvmakine
jvmakine requested a review from alecthomas as a code owner November 4, 2026 04:08
Comment thread internal/stream/outchannel.go Resolved
The reporter is the only stream a mirror opens, so a generic channel
abstracted over one caller. Captures can pull out an abstraction once they
are a second caller that shows what it needs.

Each source sends its quarantine once, so the queue never fills and the
reporter's run loop owns the quarantines without a lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jvmakine
jvmakine merged commit 828a585 into main Nov 4, 2026
6 checks passed
@jvmakine
jvmakine deleted the juho/update-stream branch November 4, 2026 04:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants