Conversation
…ished table The third way a stream arrives on a box. `new` refuses a published location that already holds a table, and `restore` rebuilds the log from a replicated WAL -- which a producer that ran with `wal_replication` off never shipped. `adopt` is for that producer's table once the producer is stopped for good: the table is the whole history, published in full, and the stream carries on from it in `migrate`'s shape. The table joins the metadata as a retired log, `[start, end)` read from the bucket; `<name>-v2` is created at exactly `end`, so the offsets stay one dense sequence and nothing the old producer issued is reissued; the metadata is written beside the tables. Snapshots and `connect(catch_up=True)` read both sides; the server replays the new log. A stream already migrated once is adopted from its published metadata. `end_offset` is what the old producer said its log ended at: a table that disagrees is published short (or long) and nothing is adopted. Refused when the box already holds the stream, when nothing is published at the location, and when a column's type differs from the table's. No manifest for the adopted table: its statistics were never read on this box, so a snapshot opens it unpruned. Tests: local `file://` round trips (seam offsets, snapshot across the seam, a subscriber catching up from offset 1, re-opened by `migrate` at the next start, the four refusals) and, on the replication tier, the same against S3 plus a stream adopted twice. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXyESUWnfYF3jXPHp7pz12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The third way a stream arrives on a box, beside
newandrestore.newrefuses a published location that already holds a table, andrestorerebuilds the log from a replicated WAL, which a producer that ran withwal_replicationoff never shipped.adoptis for that producer's table once the producer is stopped for good: the table is the whole history, published in full, and the stream carries on from it.In
migrate's shape. The published table joins the stream's metadata as a retired log,[start, end)read from the bucket. A new log,<name>-v2(then-v3, …), is created at exactlyend, so the offsets stay one dense sequence across the seam and nothing the old producer issued is reissued. The metadata is written beside the tables. Snapshots andconnect(catch_up=True)read both sides through the metadata; the server replays the new log and refuses a cursor below the seam withevicted. A stream already migrated once is adopted from its published metadata: its current log's table is sealed at its end and the next name follows.end_offsetis what the old producer said its log ended at. A table that disagrees is published short (or long) and nothing is adopted: the rows it holds would be lost at the seam, and a short table is the one thing nobody notices later.Refused when the box already holds the stream (
migrateopens it), when nothing is published at the location, and when a column's type differs from what the table had (check_types, as inmigrate).No manifest for the adopted table: its statistics were never read on this box, so a snapshot opens it unpruned, as it does a live log.
Tests (
tests/test_adopt.py): localfile://round trips (seam offsets, a snapshot across the seam, a subscriber catching up from offset 1 through the retired table and into the live log, re-opened bymigrateat the next start, the four refusals), and on the replication tier the same against S3 plus a stream adopted twice. Suite: 677 passed with rustfs up (-m "not slow"); lint, format andtyclean.🤖 Generated with Claude Code
https://claude.ai/code/session_01PXyESUWnfYF3jXPHp7pz12