Skip to content

[To dev/1.3] [Pipe] Avoid quadratic aligned page memory estimation (#18917) - #18931

Open
Caideyipi wants to merge 2 commits into
dev/1.3from
cp-8a8a6da-dev-1.3
Open

Caideyipi wants to merge 2 commits into
dev/1.3from
cp-8a8a6da-dev-1.3

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Backport #18917 (8a8a6da) to dev/1.3.

Use an incremental estimator for aligned page memory so adding a value chunk does not repeatedly scan all previously cached chunks. Reuse the accumulated estimates when constructing the aligned chunk reader.

Adapt the implementation to the container/scan package and the TsFile APIs on dev/1.3. The regression test uses AlignedChunkMetadata, List<MeasurementSchema>, the single-argument getAlignedChunkMetadata method, and an explicit Tablet.rowSize.

Validation

  • mvn -o -Ddevelocity.off=true spotless:apply -pl iotdb-core/datanode: passed.
  • Compiled the two reader implementations and the regression test from source with javac --release 17, using the resolved dev/1.3 dependencies (TsFile 1.1.4-260130-SNAPSHOT). Ran AlignedPageMemoryUsageEstimatorTest with JUnitCore: passed, 1 test.
  • The full Maven test lifecycle could not complete locally because installed Thrift artifacts expose unrelated RPC methods absent from this branch (getAllActivationStatus, getLicenseInfo, and fetchIoTConsensusProgress).

This PR has:

  • been self-reviewed.
  • added and adapted a regression test for multi-page aligned chunks.

@codecov

codecov Bot commented Oct 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.45794% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.52%. Comparing base (fc8525d) to head (28b1677).
⚠️ Report is 1 commits behind head on dev/1.3.

Files with missing lines Patch % Lines
...tainer/scan/AlignedSinglePageWholeChunkReader.java 95.52% 3 Missing ⚠️
...ntainer/scan/TsFileInsertionScanDataContainer.java 88.88% 3 Missing ⚠️
...ile/container/scan/SinglePageWholeChunkReader.java 92.30% 1 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             dev/1.3   #18931      +/-   ##
=============================================
+ Coverage      45.51%   45.52%   +0.01%     
  Complexity       198      198              
=============================================
  Files           3667     3667              
  Lines         251270   251296      +26     
  Branches       31130    31128       -2     
=============================================
+ Hits          114361   114403      +42     
+ Misses        136909   136893      -16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant