feat: XIP-83 bidirectional Subscribe RPC on MlsApi#337
Merged
Conversation
Contributor
ApprovabilityVerdict: Needs human review This PR introduces a significant new bidirectional streaming RPC with complex protocol semantics (XIP-83). Additionally, there's an unresolved design comment questioning the cursor structure for V4 migration compatibility that should be addressed before merging. You can customize Macroscope's approvability policy. Learn more. |
insipx
reviewed
Jun 12, 2026
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
tylerhawkes
force-pushed
the
tyler/xip-83-subscribe-proto
branch
2 times, most recently
from
1a738fe to
902b69d
Compare
…ed topics; mutate_id-correlated waves; Started/CatchupComplete; ping/pong; TopicsLive; history_only) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
neekolas
approved these changes
Jun 15, 2026
| // WelcomeMetadata.message_cursor so a new membership does not refetch | ||
| // pre-join history it cannot decrypt; for a new installation's welcome | ||
| // topic, 0 is how pending welcomes are collected. | ||
| uint64 id_cursor = 2; |
Collaborator
There was a problem hiding this comment.
Do we not want this to be a multi-cursor to make it easier to migrate to V4?
Contributor
Author
There was a problem hiding this comment.
I think that's something that each client can handle since we have a v3/d14n separation already.
tylerhawkes
force-pushed
the
tyler/xip-83-subscribe-proto
branch
from
902b69d to
3988a8d
Compare
insipx
approved these changes
Jun 15, 2026
insipx
approved these changes
Jun 16, 2026
tylerhawkes
added a commit
to xmtp/libxmtp
that referenced
this pull request
Jun 16, 2026
…#3769) **Stack 1/4** of the XIP-83 bidi client lane: #3769 → #3770 → #3771 → #3772. Regenerated `xmtp.mls.api.v1` from xmtp/proto#337: the bidirectional `Subscribe` RPC with versioned `SubscribeRequest`/`SubscribeResponse`, id-based `Mutate` (cursors, `history_only`), `Ping`/`Pong`, `TopicsLive`, `CATCHUP_COMPLETE`, and STARTED `capabilities`. Purely additive (+1,896 generated lines); `proto_version` pinned to that branch's sha — draft until the proto PR merges.
tylerhawkes
added a commit
to xmtp/xmtp-node-go
that referenced
this pull request
Jun 16, 2026
… ping/pong liveness) (#562) Server side of XIP-83: one bidirectional `Subscribe` stream on the v3 MLS API that replaces repeated server-streaming subscriptions. **Depends on xmtp/proto#337** (regenerated `pkg/proto/mls/api/v1` comes from that branch) — draft until it merges. ## What's in here - **Single-writer handler** (`pkg/mls/api/v1/subscribe.go`): the select loop is the sole owner of all stream state and the sole caller of `stream.Send` — no mutexes; catch-up fetchers and the frame reader are pure producers over channels. - **Mutate-in-place**: id-based add/remove with per-topic cursors; `(*Subscription).Add/Remove` are O(1) under the dispatch lock (`pkg/subscriptions`). - **Batched catch-up**: chunked (256) bounded-pool (4) `unnest(...) CROSS JOIN LATERAL` queries with per-group pagination (`pkg/mls/store/readStore.go`); 2MB frame splitting; 64MB pending-buffer cap. - **Ordering guarantees**: live-gate before dispatcher Add + per-topic high-water mark ⇒ history-before-live, no duplicates, no gaps. - **Live-boundary signals**: `TopicsLive` after each topic's history (including the drained pending buffer); one `CATCHUP_COMPLETE` per adding Mutate (wave), after the wave's last marker. - **Bounded catch-up**: `history_only` Mutates never register with the dispatcher; client half-close drains in-flight waves then the server closes `OK` (no pings post-half-close). - **Liveness**: idle-triggered `Ping` (≤30s, resets on traffic), reap on missed `Pong` (`DeadlineExceeded`), answers client pings. ## Testing 8 handler tests against real Postgres via a hand-written `MlsApi_SubscribeServer` fake: catch-up-then-live no-dupes, mutate-remove, ping/pong keepalive, reap-on-missed-pong, multi-identity multiplexing, TopicsLive boundary ordering, history-only non-delivery, half-close drain. All race-clean across 3 consecutive `-race` runs; `golangci-lint` 0 issues.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the XIP-83 bidirectional, mutable subscription stream to the v3 MLS API: one long-lived
Subscribe(stream SubscribeRequest) returns (stream SubscribeResponse)RPC.Mutate{ adds: [{topic, id_cursor}], removes: [topics] }. One shape extends to future streamable kinds (key packages, identity updates) with no proto changes; v3 keeps its singleuint64cursor (0= from the beginning; joiners SHOULD seed from the welcome's encryptedWelcomeMetadata.message_cursor). No reconnect on membership change.mutate_id, echoed on that wave'sCatchupComplete, so completions stay attributable when waves overlap.Started{keepalive_interval_ms, capabilities}/CatchupComplete{mutate_id}instead of a status enum with side fields — frame metadata is unambiguous by construction (no proto3 presence games).TopicsLive{topics}after each topic's history.history_onlyMutates: bounded catch-up with no live registration — combined with client half-close this is the stream-native sync flow (server drains the wave, then closes).Ping/Pong; version pinned per stream (V1 requests ⇒ V1 responses only).Spec: xmtp/XIPs#139. Server implementation: xmtp/xmtp-node-go#562. Client: xmtp/libxmtp#3769–#3772.
🤖 Generated with Claude Code
Note
Add bidirectional
SubscribeRPC toMlsApifor mutable topic subscriptionsMlsApi.Subscribein mls.proto that accepts a stream ofSubscribeRequestand returns a stream ofSubscribeResponse.SubscribeRequestsupports atomic topic mutations (add/remove withhistory_onlymode andmutate_idcorrelation), plusPing/Pongliveness messages.SubscribeResponsecarries batched group and welcome messages, aStartedhandshake with keepalive interval,TopicsLivetransition events,CatchupCompleteacknowledgments, andPing/Pongliveness.Macroscope summarized 3988a8d.