Repository navigation
WriteSeqNum chain and gap detection for uncommitted writes - #48729
Conversation
b094d9c to
25d4bd5
Compare
There was a problem hiding this comment.
Pull request overview
This PR implements “pipelined uncommitted writes” for DataShard by adding an explicit per-writer write index to lock state, allowing the shard to acknowledge uncommitted write flushes earlier (before they are fully durable) while still detecting lost/unordered flushes at commit time.
Changes:
- Extend lock persistence and lock wire format to carry
(WriterIndex, WriteIndex)and validate echoed write indexes on commit. - Add a new
STATUS_ALREADY_APPLIEDpath for duplicate flush delivery and propagate it through DataShard/KQP. - Update KQP write actor + transaction manager to allocate/track write indexes per shard and add comprehensive UT coverage.
Reviewed changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| ydb/core/tx/locks/sys_tables.h | Add in-memory lock fields for writer/write index reporting in sys locks. |
| ydb/core/tx/locks/locks.h | Add write-index persistence API and plumb write index through lock update flow. |
| ydb/core/tx/locks/locks.cpp | Persist write index during ApplyLocks and include it in reported locks. |
| ydb/core/tx/locks/locks_db.h | Read/write new lock DB columns and persist write-index updates. |
| ydb/core/tx/datashard/ut_common/datashard_ut_common.h | Add test helpers for uncommitted writes with write index and commit. |
| ydb/core/tx/datashard/ut_common/datashard_ut_common.cpp | Implement new test helpers for write index operations. |
| ydb/core/tx/datashard/finish_propose_write_unit.cpp | Allow early result send for pipelined uncommitted writes. |
| ydb/core/tx/datashard/execute_write_unit.cpp | Validate write-index chain, handle duplicates, and relax wait-completion for pipelined writes. |
| ydb/core/tx/datashard/datashard_write_operation.h | Store writer/write index on validated write tx and add pipelined flag. |
| ydb/core/tx/datashard/datashard_write_operation.cpp | Extract write index from incoming write request into validated tx. |
| ydb/core/tx/datashard/datashard_ut_write.cpp | Add extensive UT suite for pipelined uncommitted write semantics and failure modes. |
| ydb/core/tx/datashard/datashard_pipeline.cpp | Validate WriteIndex request constraints and enable pipelining under feature flag. |
| ydb/core/tx/datashard/datashard_kqp.cpp | Validate echoed write indexes against shard state during commit validation. |
| ydb/core/tx/datashard/datashard_impl.h | Extend DataShard lock table schema with write-index columns. |
| ydb/core/tx/datashard/datashard__write.cpp | Map STATUS_ALREADY_APPLIED to COMPLETE in propose result conversion. |
| ydb/core/tx/data_events/events.h | Add BuildAlreadyApplied, extend AddTxLock to include write indexes, and adjust IsError. |
| ydb/core/tx/data_events/common/error_codes.cpp | Treat STATUS_ALREADY_APPLIED as non-translatable to YDB status (like COMPLETED/PREPARED). |
| ydb/core/tx/columnshard/operations/manager.cpp | Explicitly reject WriteIndex for ColumnShard. |
| ydb/core/tx/columnshard/columnshard_schema.h | Extend ColumnShard lock table schema with write-index columns. |
| ydb/core/protos/feature_flags.proto | Add feature flag EnableDataShardPipelinedUncommittedWrites. |
| ydb/core/protos/data_events.proto | Add TWriteIndex, extend TLock, add TEvWrite.WriteIndex, and add STATUS_ALREADY_APPLIED. |
| ydb/core/kqp/ut/tx/kqp_sink_tx_ut.cpp | Add KQP UTs to compare behavior with pipelining enabled/disabled and OLAP path. |
| ydb/core/kqp/runtime/kqp_write_actor.cpp | Allocate/write WriteIndex per shard and handle STATUS_ALREADY_APPLIED. |
| ydb/core/kqp/common/kqp_tx_manager.h | Add NextWriteIndex API for write-index allocation. |
| ydb/core/kqp/common/kqp_tx_manager.cpp | Track per-writer write indexes per shard and merge echoed write indexes per writer. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
AI Review Summary
Verdict: ❌ 1 critical issue(s) found
Critical issues
- Critical | Medium: Feature flag
EnableDataShardPipelinedUncommittedWritesdefaults totrue, enabling a new persistence-changing behavior automatically on upgrade. All surrounding flags default tofalse. —ydb/core/protos/feature_flags.proto:363
Other findings
- Minor | High: ColumnShard schema adds unused
WriteIndex/WriterIndexcolumns while the ColumnShard operations manager explicitly rejectsWriteIndex. Adds schema complexity with no functional benefit. —ydb/core/tx/columnshard/columnshard_schema.h:615-622 - Minor | Medium:
static constexpr WriterIndex = 0means all write actors in a transaction share the same uncommitted write chain per shard; if future changes allow concurrent write actors targeting the same shard, write index ordering could break silently. —ydb/core/kqp/runtime/kqp_write_actor.cpp:1693 - Nit | Low:
BuildAlreadyApplieddoesn't populateTxStats, so duplicate write acknowledgements don't contribute to performance metrics. —ydb/core/tx/data_events/events.h:164-170
This review was generated automatically. Critical issues require attention; other findings are advisory.
If this comment was useful, please give it a 👍 — it helps us improve the review bot.
|
Analysis performed by claude, claude-opus-4-6. |
25d4bd5 to
e75db57
Compare
e75db57 to
8ef12fb
Compare
|
⚪ ⚪ DetailsYa make output | Test bloat | Test bloat
⚪ Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ ⚪ DetailsYa make output | Test bloat | Test bloat
⚪ Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
d408c30 to
e0a805d
Compare
|
⚪ ⚪ Ya make output | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ ⚪ Ya make output | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
@kirillvasilenko посмотри пожалуйста |
64188d8
|
⚪ ⚪ Ya make output | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ ⚪ Ya make output | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
Number uncommitted TEvWrite operations with a per-lock WriteSeqNum (1-based, per writer, per shard). DataShard persists the seq num with the lock, rejects gaps in and between batches, treats a resend of the last batch as IsDuplicate, and compares the echoed seq num at commit in ValidateLocks.
KQP allocates seq nums, reuses them on resend, and copies them through lock echo into the prepare/commit TKqpLocks.
This does not acknowledge uncommitted writes before the tablet log commit. Flushes wait for persistence as they do today.