Skip to content

[To dev/1.3] [Pipe] Preserve historical TsFile mods after compaction (#18904) - #18923

Open
Caideyipi wants to merge 1 commit into
apache:dev/1.3from
Caideyipi:codex/backport-18904-dev-1.3
Open

Caideyipi wants to merge 1 commit into
apache:dev/1.3from
Caideyipi:codex/backport-18904-dev-1.3

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

When compaction removes an original TsFile and its mods after historical selection, a pipe can lose deletion filtering or fail to acquire the event's file references. Backport the historical TsFile/mod preservation fix from #18904 (10c5f89) to dev/1.3.

  • Resolve historical event files and mods from the pinned pipe directory instead of checking the removed originals.
  • Remember whether historical selection actually acquired a mod-file reference, so unpin releases exactly that reference and preserves mods acquired later by an event.
  • Use the same pipe-name/creation-time directory during historical selection and event creation. This is required on dev/1.3 for the event to find the pinned files after compaction.

Backport scope and adaptation

  • Adapted the fix to PipeHistoricalDataRegionTsFileSource and the legacy ModificationFile / .mods API.
  • Added regression coverage for historical source-to-event ownership, original-file removal with and without mods, disabled mod transfer, and mods created after pinning.
  • Subscription changes are excluded. The original PR's authentication-injection fix and empty-DataRegion auto-drop test change do not apply to the dev/1.3 implementation.

Validation

  • mvn -nsu spotless:apply -pl iotdb-core/datanode passed.
  • mvn -nsu test -pl iotdb-core/datanode -am -Dtest=PipeTsFileInsertionEventTest,PipeTsFileResourceManagerTest,PipeHistoricalDataRegionTsFileSourceTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Dcheckstyle.skip=true passed, including the dependency reactor build.
  • 25 tests passed; 0 failures, 0 errors, 0 skipped.
  • git diff --check passed.

This PR has:

  • been self-reviewed.
  • added comments explaining non-obvious ownership and compaction behavior.
  • added unit tests covering the affected code paths.

Key changed classes
  • PipeTsFileInsertionEvent
  • PipeTsFileResourceManager
  • PipeHistoricalDataRegionTsFileSource
  • Their corresponding unit tests.

…pache#18904)

Backport only the historical TsFile/mod reference fix. Adapt to the 1.3 ModificationFile API and use the same creation-time-qualified pipe directory for historical selection and events.

(cherry picked from commit 10c5f89)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant