Skip to content

[Fix] Preserve absent local search index during subscription WAL replay - #18805

Merged
jt2594838 merged 8 commits into
apache:masterfrom
Caideyipi:fix/subscription-wal-gap-loss-resubmitted
Oct 10, 2026
Merged

jt2594838 merged 8 commits into
apache:masterfrom
Caideyipi:fix/subscription-wal-gap-loss-resubmitted

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Description

A replicated WAL request with searchIndex=-1 could acquire a fabricated positive local index during subscription replay. The metadata fallback uses firstSearchIndex + currentEntryIndex, but that offset also counts fragments of the same local request. The queue could then advance its cursor past real local requests even while their WAL remains available.

Preserve the explicit absent local search index in the WAL body, while keeping the metadata fallback for a body too short to contain that field. Update the English and Chinese gap warning to describe an undetermined cause. Existing gap and seek behavior otherwise stays unchanged.

Regression coverage

Add a retained-WAL test with 550 fragments of local request 1877, one replicated request from another writer, and local request 1878. It verifies fragment grouping, the replicated request's absent local index and independent writer progress, and the following local request during both full replay and a seek to 1878. The old fallback returns 2427 for the replicated request, reproducing a false [1878, 2427) gap of 549 indexes. Also update the start-filtering fixture to use explicit local indexes in the WAL body.

Validation

  • Fresh reactor build and ProgressWALIteratorTest: 16 tests, 0 failures, 0 errors; 1 opt-in benchmark skipped.
  • The new regression fails with the previous iterator (expected -1, actual 2427) and passes with this PR's iterator.
  • mvn spotless:apply -pl iotdb-core/datanode and git diff --check pass.

@Caideyipi Caideyipi changed the title [Fix] Prevent silent subscription data loss on WAL replay gaps after seek [Fix] Preserve subscription WAL replay across rotation gaps and seek Oct 9, 2026
@Caideyipi Caideyipi changed the title [Fix] Preserve subscription WAL replay across rotation gaps and seek [Fix] Retry subscription WAL replay gaps across seek Oct 9, 2026
@Caideyipi Caideyipi changed the title [Fix] Retry subscription WAL replay gaps across seek [Fix] Prevent false subscription WAL gaps and preserve seek replay Oct 9, 2026
@Caideyipi Caideyipi changed the title [Fix] Prevent false subscription WAL gaps and preserve seek replay [Fix] Avoid false subscription WAL gaps and correct seek boundaries Oct 9, 2026
@Caideyipi Caideyipi changed the title [Fix] Avoid false subscription WAL gaps and correct seek boundaries [Fix] Preserve absent local search index during subscription WAL replay Oct 9, 2026
@jt2594838
jt2594838 merged commit 99a3e06 into apache:master Oct 10, 2026
36 of 40 checks passed
@jt2594838
jt2594838 deleted the fix/subscription-wal-gap-loss-resubmitted branch October 10, 2026 02:35
VGalaxies pushed a commit to VGalaxies/iotdb that referenced this pull request Oct 26, 2026
…ay (apache#18805)

* Fix subscription WAL replay gaps across seek

* Fix subscription replay across transient WAL rotation visibility

* Hide rotated WAL under a non-WAL filename in regression test

* Fix subscription WAL gap recovery behavior

* Preserve absent local index in subscription WAL replay

* Remove speculative WAL gap recovery from subscription fix

* Remove unproven seek boundary changes from subscription fix

* test: cover replicated WAL index after fragmented local requests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants