Repository navigation
Conversation
38c4afa to
7374a75
Compare
7374a75 to
891fed6
Compare
891fed6 to
b195945
Compare
A table that becomes an async-replication replica for the second time keeps the deduplication offsets of the earlier stream. The worker names its changefeed source by the bare partition index, so the datashard recognises the new stream's records as ones it has already applied and drops them: the consumer's committed offset advances, the lag is zero and the rows are silently absent. TRemoteTopicReaderSettings.SourceIdPrefix qualifies the source name with the stream the records come from. Whether a target does so is decided when the target is created -- from the replication's INITIAL_SCAN, since only an attach-mode target meets a table that may already carry offsets -- and is persisted with it in Targets.QualifiedSourceId. The choice is deliberately not recomputed from the configuration: a target created before this column existed loads with the flag unset and keeps the bare partition index its destination offsets were written under, so an upgrade never changes the naming of a replication that is already running, whatever the cluster's replication defaults say.
ALTER ASYNC REPLICATION ... SET (STATE = 'DONE', FAILOVER_MODE = 'CONSISTENT') stops the workers at once. For a GLOBAL replication that is not consistent: writes of the current commit interval have been assigned a transaction id and staged on the replica but are committed only at the next heartbeat, so a DONE in between detaches the tables with those rows staged and invisible, and they are lost. Behind EnableConsistentReplicationDone, such a DONE is deferred while the controller still holds assigned, pending or committing transaction ids, and resumes from the heartbeat that drains the last one. The decision is persisted (Replications.ConsistentDone), so a controller restart in between does not turn it back into an immediate DONE.
b195945 to
cf50725
Compare
|
⚪ ⚪
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
✅ Documentation buildRevision built successfully |
How a replication's destination gets its initial contents was encoded in
INITIAL_SCAN, which says whether a scan happens but not who creates the
destination. Name the choice instead:
BOOTSTRAP = 'SCAN' the controller creates the tables and copies
the rows (INITIAL_SCAN = TRUE)
BOOTSTRAP = 'IMPORT' the controller creates placeholders; the
operator fills them and hands over with
STANDBY (INITIAL_SCAN = FALSE)
BOOTSTRAP = 'SWITCHOVER' the destination tables already exist and
already hold the data, as the old primary's
do after a planned role switchover; the
controller demotes them to read-only replicas
in place and copies nothing
SWITCHOVER is new. The demote was reachable only through a ModifyScheme
with Internal set, which no public API carries, so an operator had to
hand-craft one per table and per synchronous or unique index
implementation table and push it through the legacy
`ydbd db schema exec`.
The controller demotes each destination with the same internal alter it
issues in the opposite direction when a replication reaches DONE, taking
the mode, consistency level and ImportState from the replication's own
configuration so they cannot disagree with it. Index implementation
tables are adopted by their own targets. Unlike other scanless
replications, a switchover one starts in Ready rather than waiting
paused for a hand-over.
INITIAL_SCAN keeps working and is derived from BOOTSTRAP; contradictory
values are an error. SWITCHOVER is refused for a transfer, not accepted
in ALTER, and admitted only from a real cluster administrator, since
demoting a table bypasses every protection a replica table has.
RFC: ydb-platform/ydb-rfc#389
One command per step of a switchover between two clusters linked by async replication, resumable from a state file, plus a rehearsal that flips back and forth under writes on a test pair and checks that nothing is lost. The tool does not close writes: an external fence does, and the write outage runs from there until the reverse replication reports running -- not until the promote. A changefeed captures only changes made after it is created and a destination bootstrapped by SWITCHOVER takes no initial scan, so a write accepted on the new primary before the reverse replication exists reaches the new replica by no path at all. `create-reverse` reports the outage it has measured so far, and the rehearsal records it as `outage_s` with the handover time separately. RFC: ydb-platform/ydb-rfc#389
cf50725 to
da3acbc
Compare
|
🔴 Unable to merge your PR into the base branch. Please rebase or merge it with the base branch. |
Rebased onto current main. The three commits below it are #59277, #59279 and #59328; review those first — this PR is only the tool on top of them. Procedure: ydb-platform/ydb-rfc#389.
The procedure.
preflight→ (close writes with an external fence) →drain→promote→cleanup-changefeeds→create-reverse→probe→ (reopen writes) →drop-old. Every step records itself in a state file and is safe to repeat, so an interrupted flip is continued by re-runningflip.Where the outage really ends. The tool does not close writes — an external fence does — and the write outage runs from there until the reverse replication reports running, not until the promote. A changefeed captures only changes made after it is created, and a
BOOTSTRAP = 'SWITCHOVER'destination takes no initial scan, so a write accepted on the new primary before the reverse replication exists reaches the new replica by no path at all.create-reversecannot be moved earlier either: a changefeed cannot be created on a table that is still a replica. The rehearsal reportsoutage_s(fence → reverse running) andhandover_s(fence → promote) separately; on a sandbox pair they differ by 0.5-0.7 s, which is exactly the window the distinction protects.What makes it safe, and what does not.
DONEwaits for nothing by itself, so the drain is the only evidence that the replica has everything the old primary acknowledged: committed changefeed offsets for ROW, plus equal and stable row digests for GLOBAL. Theproberuns before writes reopen, because a replication can report running with zero lag while delivering nothing.An open intermittent failure, recorded rather than hidden. In one rehearsal, flip 2's probe failed and the divergence was real: 316 rows on the new primary, 315 on the new replica, a fresh write never arriving, while the replication reported
runningwith lag0s. The changefeed had captured the records (end=2) and the worker had committed them (committed=2) without applying them. It has not reproduced in the 20 flips since, on the same configuration. The controller-side diagnostic added in #59277 and #59328 (qualifiedSourceIdandbootstrapon the "Add target" log) ruled out the deduplication path: both are correct for switchover-bootstrapped targets. The leading remaining explanation is the reverse consumer's read position being established after records are already in the feed, which would meanrunningis not by itself a sufficient condition for reopening writes. Worth noting thatprobeis what caught it — without that step the tool would have declared a successful switchover over a silently diverged replica.Limits. Row tables; synchronous, unique and async indexes are fine; TTL and vector indexes are refused. No DDL during a flip. The fence is per cluster and external, so nothing in the engine enforces it. Unplanned failover is a separate path (
failover,reseed): no fence, no drain, you accept the loss, and the lost side must be reseeded.Rehearsed on sandbox pairs built from this branch: 7 runs, 23 flips total (2 tables, synchronous index, 2 partitions, writers running until each fence), 22 clean with every acknowledged row present on the new primary and the reverse direction probed each time.