Skip to content

feat: BOOTSTRAP mode for CREATE ASYNC REPLICATION - #59328

Merged
Enjection merged 1 commit into
ydb-platform:mainfrom
Enjection:feat/async-replication-adopt-existing
Nov 18, 2026
Merged

Enjection merged 1 commit into
ydb-platform:mainfrom
Enjection:feat/async-replication-adopt-existing

Conversation

@Enjection

@Enjection Enjection commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Implements ydb-platform/ydb-rfc#389.

Names how a replication's destination gets its initial contents, instead of encoding it in INITIAL_SCAN:

CREATE ASYNC REPLICATION `R` FOR `src` AS `dst`
    WITH (CONNECTION_STRING = '...', BOOTSTRAP = 'SCAN' | 'IMPORT' | 'SWITCHOVER', CONSISTENCY_LEVEL = 'ROW');

SCAN and IMPORT are today's INITIAL_SCAN = TRUE/FALSE. SWITCHOVER is new: the destination tables already exist as ordinary writable tables holding the data — as the old primary's do after a planned role switchover — and the controller converts them into read-only replicas in place, copying nothing. The demote was previously reachable only through an Internal ModifyScheme via the legacy MsgBus service, per table and per index implementation table.

The controller demotes each destination with the same internal alter it issues in the opposite direction when a replication reaches DONE, taking the mode, consistency level and ImportState from the replication's own configuration so they cannot disagree with it. Index implementation tables are adopted by their own targets. Unlike other scanless replications, a switchover one starts in Ready rather than waiting paused for a hand-over — that mapping is InitialReplicationState() with its own test, because getting it wrong creates every target and then never starts a worker.

INITIAL_SCAN keeps working and is derived from BOOTSTRAP; contradictory values are an error. SWITCHOVER is refused for a transfer, not accepted in ALTER, and admitted only from a real cluster administrator — IsAdministrator() with a non-empty AdministrationAllowedSIDs, the rule index implementation table alters already use.

Note on vocabulary: the mode is named for the scenario, the controller code for the action it performs — the predicate is IsSwitchover(), while the functions that demote a table in place keep adopt in their names.

Tests: 3 parser cases; DstCreator::SwitchoverAdoptsTableInPlace and WithoutSwitchoverAPlainTableIsRefused; Replication::InitialStateFollowsInitialScanAndSwitchover. All seven replication controller suites pass. Rehearsed on real cluster pairs: 23 switchover flips, 22 clean (see #59171 for the one failure and what it showed).

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

CyberROFL

This comment was marked as resolved.

@Enjection
Enjection force-pushed the feat/async-replication-adopt-existing branch from dd23ea4 to 7107d9a Compare November 16, 2026 13:39
@Enjection Enjection changed the title feat: CREATE ASYNC REPLICATION ... ADOPT_EXISTING to reuse existing tables feat: BOOTSTRAP mode for CREATE ASYNC REPLICATION Nov 16, 2026
@Enjection
Enjection force-pushed the feat/async-replication-adopt-existing branch 2 times, most recently from 2117602 to a8bd205 Compare November 17, 2026 21:02
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@Enjection
Enjection requested a review from CyberROFL November 17, 2026 21:11
CyberROFL

This comment was marked as resolved.

@Enjection
Enjection force-pushed the feat/async-replication-adopt-existing branch from a8bd205 to eab881a Compare November 18, 2026 09:54
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-18 09:57:11 UTC Pre-commit check linux-x86_64-relwithdebinfo for 92661bc has started.
⚪ 2026-11-18 10:00:57 UTC Artifacts will be uploaded here
⚪ 2026-11-18 10:02:33 UTC ya make is running...
🟡 2026-11-18 12:05:01 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
63102 61266 0 8 1816 12

⚪ 2026-11-18 12:05:20 UTC ya make is running... (failed tests rerun, try 2)
🟡 2026-11-18 12:07:05 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
462 (only retried tests) 460 0 1 0 1

⚪ 2026-11-18 12:07:09 UTC ya make is running... (failed tests rerun, try 3)
🟢 2026-11-18 12:08:07 UTC Tests successful.

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
402 (only retried tests) 401 0 0 0 1

🟢 2026-11-18 12:08:11 UTC Build successful.
🟡 2026-11-18 12:08:36 UTC ydbd size 2.9 GiB changed* by +27.5 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 7c1bcec merge: 92661bc diff diff %
ydbd size 3 045 473 272 Bytes 3 074 343 472 Bytes +27.5 MiB +0.948%
ydbd stripped size 643 672 192 Bytes 647 018 112 Bytes +3.2 MiB +0.520%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions

This comment has been minimized.

@Enjection
Enjection requested a review from CyberROFL November 18, 2026 11:00

@nikvas0 nikvas0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kqp ok

How a replication's destination gets its initial contents was encoded in
INITIAL_SCAN, which says whether a scan happens but not who creates the
destination. Name the choice instead:

  BOOTSTRAP = 'SCAN'        the controller creates the tables and copies
                            the rows (INITIAL_SCAN = TRUE)
  BOOTSTRAP = 'IMPORT'      the controller creates placeholders; the
                            operator fills them and hands over with
                            STANDBY (INITIAL_SCAN = FALSE)
  BOOTSTRAP = 'SWITCHOVER'  the destination tables already exist and
                            already hold the data, as the old primary's
                            do after a planned role switchover; the
                            controller demotes them to read-only replicas
                            in place and copies nothing

SWITCHOVER is new. The demote was reachable only through a ModifyScheme
with Internal set, which no public API carries, so an operator had to
hand-craft one per table and per synchronous or unique index
implementation table and push it through the legacy
`ydbd db schema exec`.

The controller demotes each destination with the same internal alter it
issues in the opposite direction when a replication reaches DONE, taking
the mode, consistency level and ImportState from the replication's own
configuration so they cannot disagree with it. Index implementation
tables are adopted by their own targets. Unlike other scanless
replications, a switchover one starts in Ready rather than waiting
paused for a hand-over.

INITIAL_SCAN keeps working and is derived from BOOTSTRAP; contradictory
values are an error. SWITCHOVER is refused for a transfer, not accepted
in ALTER, and admitted only from a real cluster administrator, since
demoting a table bypasses every protection a replica table has.

RFC: ydb-platform/ydb-rfc#389
@Enjection
Enjection force-pushed the feat/async-replication-adopt-existing branch from eab881a to d5caf07 Compare November 18, 2026 14:55
@Enjection
Enjection removed the request for review from a team November 18, 2026 14:57
@Enjection
Enjection enabled auto-merge (squash) November 18, 2026 14:57
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-18 14:58:25 UTC Pre-commit check linux-x86_64-relwithdebinfo for f161b15 has started.
⚪ 2026-11-18 14:59:19 UTC Artifacts will be uploaded here
⚪ 2026-11-18 15:00:48 UTC ya make is running...
🟡 2026-11-18 17:01:38 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
62999 61193 0 4 1793 9

⚪ 2026-11-18 17:01:57 UTC ya make is running... (failed tests rerun, try 2)
🟢 2026-11-18 17:03:10 UTC Tests successful.

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
174 (only retried tests) 173 0 0 0 1

🟢 2026-11-18 17:03:15 UTC Build successful.
🟡 2026-11-18 17:03:35 UTC ydbd size 2.9 GiB changed* by +29.9 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 7c1bcec merge: f161b15 diff diff %
ydbd size 3 045 473 272 Bytes 3 076 830 064 Bytes +29.9 MiB +1.030%
ydbd stripped size 643 672 192 Bytes 647 250 048 Bytes +3.4 MiB +0.556%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions

Copy link
Copy Markdown
Contributor

✅ Documentation build

Revision built successfully
Revision preview link

@Enjection
Enjection merged commit 16c0664 into ydb-platform:main Nov 18, 2026
16 of 23 checks passed
@ydbot

ydbot commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Backport

To backport this PR, click the button next to the target branch and then click "Run workflow" in the Run Actions UI.

Branch Run
stable-26-3-1, stable-26-4-1 ▶  Backport
stable-26-4-1 ▶  Backport

▶  Backport manual

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants