Skip to content

feat: one consumer group per coordinator schema - #152

Merged
hasyimibhar merged 1 commit into
mainfrom
feat/one-consumer-group
Nov 17, 2026
Merged

hasyimibhar merged 1 commit into
mainfrom
feat/one-consumer-group

Conversation

@hasyimibhar

@hasyimibhar hasyimibhar commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

The staged log, the tables' snapshot status and, since #150, the log's truncation all assume one reader per coordinator schema. Nothing enforced that, and a second consumer group (state.group) on the same schema was already broken in several ways:

  • A group that starts late: truncation has moved the log past what it needs, and it read on from what's left without noticing, silently losing everything before. It also never gets the initial snapshot, which goes straight into one catalog (feat: skip staging for initial snapshot #141).
  • Another group's backfill: a group without a backfill of its own applies the other group's snapshot rows as ordinary inserts, in log order. That's the stale-copy bug the per-group #snapshot cursor exists to prevent.
  • An abandoned or renamed group: its cursors stay in mat_cursor and hold back truncation of that table's log forever.

Multiple materializers in one group (distributed workers) are unaffected: they share one cursor per table.

@hasyimibhar
hasyimibhar force-pushed the feat/one-consumer-group branch from c94b1fb to 2f3b115 Compare November 17, 2026 05:06
@hasyimibhar
hasyimibhar changed the base branch from fix/truncate-staged-log to main November 17, 2026 05:06
The staged log, the tables' snapshot status and the log's truncation
all assume one reader, but nothing stopped a second consumer group
from reading a schema. One that started late found the log truncated
past what it needed and read on, losing what was gone; it never got
the initial snapshot (written straight to one catalog) and applied
another group's backfill rows as ordinary inserts. A group abandoned,
or renamed, kept its cursors holding the log forever.

Every process that reads or moves cursors (`run`, `stream-only`,
`run --once`, `materializer-only`, `snapshot`) now claims the schema
for its `state.group` at startup (`Coordinator::claim_group`): the
first records it in `pipeline_meta.consumer_group`; another group is
refused with `GroupMismatch`, as is a schema that predates the record
and holds another group's cursors. `materializer-only` migrates the
schema first, so it can start before the process that replicates.
`state.group` can't be empty or contain '#'.

Truncation now goes by the group's own cursors. And a materializer
that finds its log starting past its cursor stops with `LogGap` (the
table needs a new snapshot) rather than read on, unless the cursor
has moved past the gap since: another worker of the group applied and
truncated it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hasyimibhar
hasyimibhar force-pushed the feat/one-consumer-group branch from 2f3b115 to 8b26a18 Compare November 17, 2026 05:06
@hasyimibhar
hasyimibhar merged commit 726e863 into main Nov 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant