Skip to content

WriteSeqNum chain and gap detection for uncommitted writes - #48729

Merged
azevaykin merged 21 commits into
ydb-platform:mainfrom
azevaykin:PipelinedUncommittedWrites
Sep 3, 2026
Merged

azevaykin merged 21 commits into
ydb-platform:mainfrom
azevaykin:PipelinedUncommittedWrites

Conversation

@azevaykin

@azevaykin azevaykin commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Number uncommitted TEvWrite operations with a per-lock WriteSeqNum (1-based, per writer, per shard). DataShard persists the seq num with the lock, rejects gaps in and between batches, treats a resend of the last batch as IsDuplicate, and compares the echoed seq num at commit in ValidateLocks.

KQP allocates seq nums, reuses them on resend, and copies them through lock echo into the prepare/commit TKqpLocks.
This does not acknowledge uncommitted writes before the tablet log commit. Flushes wait for persistence as they do today.

@azevaykin
azevaykin force-pushed the PipelinedUncommittedWrites branch from b094d9c to 25d4bd5 Compare August 3, 2026 07:46
@ydb-platform ydb-platform deleted a comment from github-actions Bot Aug 3, 2026
@ydb-platform ydb-platform deleted a comment from ydbot Aug 3, 2026
@azevaykin
azevaykin requested a review from Copilot August 3, 2026 07:46
@azevaykin azevaykin linked an issue Aug 3, 2026 that may be closed by this pull request

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements “pipelined uncommitted writes” for DataShard by adding an explicit per-writer write index to lock state, allowing the shard to acknowledge uncommitted write flushes earlier (before they are fully durable) while still detecting lost/unordered flushes at commit time.

Changes:

  • Extend lock persistence and lock wire format to carry (WriterIndex, WriteIndex) and validate echoed write indexes on commit.
  • Add a new STATUS_ALREADY_APPLIED path for duplicate flush delivery and propagate it through DataShard/KQP.
  • Update KQP write actor + transaction manager to allocate/track write indexes per shard and add comprehensive UT coverage.

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
ydb/core/tx/locks/sys_tables.h Add in-memory lock fields for writer/write index reporting in sys locks.
ydb/core/tx/locks/locks.h Add write-index persistence API and plumb write index through lock update flow.
ydb/core/tx/locks/locks.cpp Persist write index during ApplyLocks and include it in reported locks.
ydb/core/tx/locks/locks_db.h Read/write new lock DB columns and persist write-index updates.
ydb/core/tx/datashard/ut_common/datashard_ut_common.h Add test helpers for uncommitted writes with write index and commit.
ydb/core/tx/datashard/ut_common/datashard_ut_common.cpp Implement new test helpers for write index operations.
ydb/core/tx/datashard/finish_propose_write_unit.cpp Allow early result send for pipelined uncommitted writes.
ydb/core/tx/datashard/execute_write_unit.cpp Validate write-index chain, handle duplicates, and relax wait-completion for pipelined writes.
ydb/core/tx/datashard/datashard_write_operation.h Store writer/write index on validated write tx and add pipelined flag.
ydb/core/tx/datashard/datashard_write_operation.cpp Extract write index from incoming write request into validated tx.
ydb/core/tx/datashard/datashard_ut_write.cpp Add extensive UT suite for pipelined uncommitted write semantics and failure modes.
ydb/core/tx/datashard/datashard_pipeline.cpp Validate WriteIndex request constraints and enable pipelining under feature flag.
ydb/core/tx/datashard/datashard_kqp.cpp Validate echoed write indexes against shard state during commit validation.
ydb/core/tx/datashard/datashard_impl.h Extend DataShard lock table schema with write-index columns.
ydb/core/tx/datashard/datashard__write.cpp Map STATUS_ALREADY_APPLIED to COMPLETE in propose result conversion.
ydb/core/tx/data_events/events.h Add BuildAlreadyApplied, extend AddTxLock to include write indexes, and adjust IsError.
ydb/core/tx/data_events/common/error_codes.cpp Treat STATUS_ALREADY_APPLIED as non-translatable to YDB status (like COMPLETED/PREPARED).
ydb/core/tx/columnshard/operations/manager.cpp Explicitly reject WriteIndex for ColumnShard.
ydb/core/tx/columnshard/columnshard_schema.h Extend ColumnShard lock table schema with write-index columns.
ydb/core/protos/feature_flags.proto Add feature flag EnableDataShardPipelinedUncommittedWrites.
ydb/core/protos/data_events.proto Add TWriteIndex, extend TLock, add TEvWrite.WriteIndex, and add STATUS_ALREADY_APPLIED.
ydb/core/kqp/ut/tx/kqp_sink_tx_ut.cpp Add KQP UTs to compare behavior with pipelining enabled/disabled and OLAP path.
ydb/core/kqp/runtime/kqp_write_actor.cpp Allocate/write WriteIndex per shard and handle STATUS_ALREADY_APPLIED.
ydb/core/kqp/common/kqp_tx_manager.h Add NextWriteIndex API for write-index allocation.
ydb/core/kqp/common/kqp_tx_manager.cpp Track per-writer write indexes per shard and merge echoed write indexes per writer.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread ydb/core/kqp/common/kqp_tx_manager.cpp Resolved
Comment thread ydb/core/protos/feature_flags.proto Resolved

@robot-vibe-db robot-vibe-db Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review Summary

Verdict: ❌ 1 critical issue(s) found

Critical issues

  • Critical | Medium: Feature flag EnableDataShardPipelinedUncommittedWrites defaults to true, enabling a new persistence-changing behavior automatically on upgrade. All surrounding flags default to false. — ydb/core/protos/feature_flags.proto:363

Other findings

  • Minor | High: ColumnShard schema adds unused WriteIndex/WriterIndex columns while the ColumnShard operations manager explicitly rejects WriteIndex. Adds schema complexity with no functional benefit. — ydb/core/tx/columnshard/columnshard_schema.h:615-622
  • Minor | Medium: static constexpr WriterIndex = 0 means all write actors in a transaction share the same uncommitted write chain per shard; if future changes allow concurrent write actors targeting the same shard, write index ordering could break silently. — ydb/core/kqp/runtime/kqp_write_actor.cpp:1693
  • Nit | Low: BuildAlreadyApplied doesn't populate TxStats, so duplicate write acknowledgements don't contribute to performance metrics. — ydb/core/tx/data_events/events.h:164-170

This review was generated automatically. Critical issues require attention; other findings are advisory.
If this comment was useful, please give it a 👍 — it helps us improve the review bot.

Comment thread ydb/core/protos/feature_flags.proto Resolved
Comment thread ydb/core/tx/data_events/events.h Resolved
@robot-vibe-db

robot-vibe-db Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Full analysis log

Analysis performed by claude, claude-opus-4-6.

@azevaykin
azevaykin force-pushed the PipelinedUncommittedWrites branch from 25d4bd5 to e75db57 Compare August 3, 2026 08:58
@ydb-platform ydb-platform deleted a comment from github-actions Bot Aug 3, 2026
@ydb-platform ydb-platform deleted a comment from github-actions Bot Aug 3, 2026
@ydb-platform ydb-platform deleted a comment from github-actions Bot Aug 3, 2026
@azevaykin
azevaykin force-pushed the PipelinedUncommittedWrites branch from e75db57 to 8ef12fb Compare August 3, 2026 09:24
@ydb-platform ydb-platform deleted a comment from github-actions Bot Aug 3, 2026
@ydb-platform ydb-platform deleted a comment from github-actions Bot Aug 3, 2026
@ydb-platform ydb-platform deleted a comment from github-actions Bot Aug 3, 2026
@ydb-platform ydb-platform deleted a comment from github-actions Bot Aug 3, 2026
@ydb-platform ydb-platform deleted a comment from github-actions Bot Aug 3, 2026
@github-actions

github-actions Bot commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-08-03 09:30:43 UTC Pre-commit check linux-x86_64-release-asan for 72a861b has started.
⚪ 2026-08-03 09:31:23 UTC Artifacts will be uploaded here
⚪ 2026-08-03 09:33:27 UTC ya make is running...
🟡 2026-08-03 11:50:17 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
27573 27279 0 15 216 63

⚪ 2026-08-03 11:50:28 UTC ya make is running... (failed tests rerun, try 2)
🟡 2026-08-03 12:01:39 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
434 (only retried tests) 386 0 10 37 1

⚪ 2026-08-03 12:01:43 UTC ya make is running... (failed tests rerun, try 3)
🔴 2026-08-03 12:04:21 UTC Some tests failed, follow the links below.

Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
154 (only retried tests) 145 0 9 0 0

🟢 2026-08-03 12:04:24 UTC Build successful.
🟡 2026-08-03 12:07:16 UTC ydbd size 4.3 GiB changed* by +123.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: c0db7a1 merge: 72a861b diff diff %
ydbd size 4 572 274 840 Bytes 4 572 401 320 Bytes +123.5 KiB +0.003%
ydbd stripped size 1 692 399 432 Bytes 1 692 449 096 Bytes +48.5 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions

github-actions Bot commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-08-03 09:33:02 UTC Pre-commit check linux-x86_64-relwithdebinfo for 72a861b has started.
⚪ 2026-08-03 09:33:09 UTC Artifacts will be uploaded here
⚪ 2026-08-03 09:34:39 UTC ya make is running...
🟡 2026-08-03 12:31:17 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
55742 54005 0 12 1723 2

⚪ 2026-08-03 12:31:35 UTC ya make is running... (failed tests rerun, try 2)
🟡 2026-08-03 12:34:20 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
202 (only retried tests) 193 0 9 0 0

⚪ 2026-08-03 12:34:23 UTC ya make is running... (failed tests rerun, try 3)
🔴 2026-08-03 12:35:09 UTC Some tests failed, follow the links below.

Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
115 (only retried tests) 106 0 9 0 0

🟢 2026-08-03 12:35:12 UTC Build successful.
🟢 2026-08-03 12:35:34 UTC ydbd size 2.6 GiB changed* by +84.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: c0db7a1 merge: 72a861b diff diff %
ydbd size 2 824 254 680 Bytes 2 824 341 344 Bytes +84.6 KiB +0.003%
ydbd stripped size 590 776 568 Bytes 590 796 280 Bytes +19.2 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

nikvas0
nikvas0 previously approved these changes Sep 2, 2026
ztlpn
ztlpn previously approved these changes Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-09-02 14:06:38 UTC Pre-commit check linux-x86_64-release-asan for 7e92efb has started.
⚪ 2026-09-02 14:07:28 UTC Artifacts will be uploaded here
⚪ 2026-09-02 14:09:53 UTC ya make is running...
🟡 2026-09-02 17:26:36 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
29620 29274 0 3 285 58

⚪ 2026-09-02 17:26:51 UTC ya make is running... (failed tests rerun, try 2)
🟢 2026-09-02 17:37:57 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
323 (only retried tests) 280 0 0 42 1

🟢 2026-09-02 17:38:01 UTC Build successful.
🟡 2026-09-02 17:38:37 UTC ydbd size 4.3 GiB changed* by +407.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 6b1d806 merge: 7e92efb diff diff %
ydbd size 4 595 280 456 Bytes 4 595 697 784 Bytes +407.5 KiB +0.009%
ydbd stripped size 1 701 570 312 Bytes 1 701 726 088 Bytes +152.1 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions

github-actions Bot commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-09-02 14:08:43 UTC Pre-commit check linux-x86_64-relwithdebinfo for 7e92efb has started.
⚪ 2026-09-02 14:09:03 UTC Artifacts will be uploaded here
⚪ 2026-09-02 14:10:43 UTC ya make is running...
🟡 2026-09-02 18:20:46 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
58198 56336 0 10 1842 10

⚪ 2026-09-02 18:21:05 UTC ya make is running... (failed tests rerun, try 2)
🟢 2026-09-02 18:23:17 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
68 (only retried tests) 68 0 0 0 0

🟢 2026-09-02 18:23:20 UTC Build successful.
🟡 2026-09-02 18:23:42 UTC ydbd size 2.7 GiB changed* by +258.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: c601187 merge: 7e92efb diff diff %
ydbd size 2 851 194 312 Bytes 2 851 459 288 Bytes +258.8 KiB +0.009%
ydbd stripped size 596 851 872 Bytes 596 909 728 Bytes +56.5 KiB +0.010%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@dorooleg

dorooleg commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

@kirillvasilenko посмотри пожалуйста

@azevaykin
azevaykin dismissed stale reviews from ztlpn and nikvas0 via 5c474ac September 3, 2026 10:55
@github-actions

github-actions Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-09-03 10:56:54 UTC Pre-commit check linux-x86_64-release-asan for f96fd63 has started.
⚪ 2026-09-03 10:57:14 UTC Artifacts will be uploaded here
⚪ 2026-09-03 10:58:42 UTC ya make is running...
⚫ 2026-09-03 11:29:13 UTC Check cancelled

kirillvasilenko
kirillvasilenko previously approved these changes Sep 3, 2026
@github-actions

github-actions Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-09-03 10:57:38 UTC Pre-commit check linux-x86_64-relwithdebinfo for f96fd63 has started.
⚪ 2026-09-03 10:57:58 UTC Artifacts will be uploaded here
⚪ 2026-09-03 10:59:28 UTC ya make is running...
⚫ 2026-09-03 11:29:15 UTC Check cancelled

nikvas0
nikvas0 previously approved these changes Sep 3, 2026
@azevaykin
azevaykin dismissed stale reviews from nikvas0 and kirillvasilenko via 64188d8 September 3, 2026 11:28
@github-actions

github-actions Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-09-03 11:30:49 UTC Pre-commit check linux-x86_64-relwithdebinfo for 70dc826 has started.
⚪ 2026-09-03 11:31:12 UTC Artifacts will be uploaded here
⚪ 2026-09-03 11:32:44 UTC ya make is running...
🟡 2026-09-03 15:39:10 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
58280 56452 0 2 1820 6

⚪ 2026-09-03 15:39:32 UTC ya make is running... (failed tests rerun, try 2)
🟢 2026-09-03 15:42:45 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
81 (only retried tests) 81 0 0 0 0

🟢 2026-09-03 15:42:49 UTC Build successful.
🟡 2026-09-03 15:43:10 UTC ydbd size 2.7 GiB changed* by +215.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: dabb463 merge: 70dc826 diff diff %
ydbd size 2 852 425 248 Bytes 2 852 645 616 Bytes +215.2 KiB +0.008%
ydbd stripped size 597 119 680 Bytes 597 170 368 Bytes +49.5 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions

github-actions Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-09-03 11:34:08 UTC Pre-commit check linux-x86_64-release-asan for 70dc826 has started.
⚪ 2026-09-03 11:34:29 UTC Artifacts will be uploaded here
⚪ 2026-09-03 11:35:55 UTC ya make is running...
🟡 2026-09-03 14:51:37 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
29675 29358 0 3 259 55

⚪ 2026-09-03 14:51:50 UTC ya make is running... (failed tests rerun, try 2)
🟢 2026-09-03 15:02:57 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
285 (only retried tests) 241 0 0 41 3

🟢 2026-09-03 15:03:01 UTC Build successful.
🟡 2026-09-03 15:03:39 UTC ydbd size 4.3 GiB changed* by +311.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: dabb463 merge: 70dc826 diff diff %
ydbd size 4 597 427 392 Bytes 4 597 746 736 Bytes +311.9 KiB +0.007%
ydbd stripped size 1 702 355 432 Bytes 1 702 472 808 Bytes +114.6 KiB +0.007%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@azevaykin
azevaykin merged commit ba886d0 into ydb-platform:main Sep 3, 2026
12 checks passed
@azevaykin
azevaykin deleted the PipelinedUncommittedWrites branch September 3, 2026 15:44
@ydbot

ydbot commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Backport

To backport this PR, click the button next to the target branch and then click "Run workflow" in the Run Actions UI.

Branch Run
stable-26-2-1, stable-26-3-1 ▶  Backport
stable-26-3-1 ▶  Backport

▶  Backport manual

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DataShard: support faster uncommitted writes

8 participants