Repository navigation
feat: one consumer group per coordinator schema - #152
Merged
Merged
Conversation
hasyimibhar
force-pushed
the
feat/one-consumer-group
branch
from
November 17, 2026 05:06
c94b1fb to
2f3b115
Compare
The staged log, the tables' snapshot status and the log's truncation all assume one reader, but nothing stopped a second consumer group from reading a schema. One that started late found the log truncated past what it needed and read on, losing what was gone; it never got the initial snapshot (written straight to one catalog) and applied another group's backfill rows as ordinary inserts. A group abandoned, or renamed, kept its cursors holding the log forever. Every process that reads or moves cursors (`run`, `stream-only`, `run --once`, `materializer-only`, `snapshot`) now claims the schema for its `state.group` at startup (`Coordinator::claim_group`): the first records it in `pipeline_meta.consumer_group`; another group is refused with `GroupMismatch`, as is a schema that predates the record and holds another group's cursors. `materializer-only` migrates the schema first, so it can start before the process that replicates. `state.group` can't be empty or contain '#'. Truncation now goes by the group's own cursors. And a materializer that finds its log starting past its cursor stops with `LogGap` (the table needs a new snapshot) rather than read on, unless the cursor has moved past the gap since: another worker of the group applied and truncated it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hasyimibhar
force-pushed
the
feat/one-consumer-group
branch
from
November 17, 2026 05:06
2f3b115 to
8b26a18
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The staged log, the tables' snapshot status and, since #150, the log's truncation all assume one reader per coordinator schema. Nothing enforced that, and a second consumer group (
state.group) on the same schema was already broken in several ways:#snapshotcursor exists to prevent.mat_cursorand hold back truncation of that table's log forever.Multiple materializers in one group (distributed workers) are unaffected: they share one cursor per table.