Skip to content

[Fix] Harden split LOAD recovery, deletion dispatch and memory bounds - #18930

Open
Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:codex/fix-load-recovery
Open

Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:codex/fix-load-recovery

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Split LOAD can recover a staged file under the wrong time partition, accept incomplete data during PREPARE, or lose a completed PREPARE across restart. Its deletion and eviction paths also duplicate mods and bypass the intended buffering bounds.

  • Parse the signed partition start using the known database/region prefix, including negative starts and Long.MIN_VALUE.
  • Validate every staged and coordinator-announced partition before sealing any file. Missing partitions and files without durable chunk records must reject PREPARE and remain retryable.
  • Preserve a prepared TsFile's footer and serialized resource on recovery, so the coordinator can proceed directly to COMMIT with the persisted progress and pipe marks.
  • Stage all chunks before sending each deletion once per region. The receiver broadcasts that deletion to its staged partitions, avoiding duplicate mods when several partitions share a region.
  • Stop splitting when the deletion consumer rejects a record, and dispatch buffered deletions when their memory budget is exceeded.
  • Keep one ordered eviction entry per nonempty partition, remove entries when pieces are flushed, and retain a failed eviction for retry. Tracking no longer grows with the number of chunks already processed.

The branch is based on master after #18927. Its IoTConsensusV2 progress assignment at COMMIT is unchanged by this diff.

Validation

  • All 10 added regression tests fail against unmodified master (d5a2da0b228) and pass with the fixes. They verify readable recovered data, retryable PREPARE rejection, preserved prepared bytes and resource marks, exact mods contents across multiple regions/partitions, deletion rejection, memory backpressure, bounded eviction tracking, and failed-eviction retry.
  • 162 targeted LOAD tests passed in 22 isolated JVMs, including the IoTConsensusV2 regressions from [Fix] Preserve IoTConsensusV2 replication for split LOAD TsFiles #18927, staging/recovery, deletion, split coverage, table schemas, snapshots, progress, cleanup, chunk serialization, routing, and two-phase failure handling. Changed production sources and all selected test sources were freshly compiled with JDK 17 before execution.
  • Full Maven test lifecycle attempts were interrupted by Windows committed-memory exhaustion. The targeted tests were therefore compiled and run directly with bounded JVMs and the project's Java 17 test options.
  • Spotless, Checkstyle, and git diff --check passed.

@Caideyipi
Caideyipi force-pushed the codex/fix-load-recovery branch from cba841e to 81b5ae4 Compare October 20, 2026 07:17
@Caideyipi Caideyipi changed the title [Fix] Preserve negative LOAD partitions and reject incomplete PREPARE [Fix] Harden split LOAD recovery, deletion dispatch and memory bounds Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant