Skip to content

[Pipe] Avoid quadratic aligned page memory estimation - #18917

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:codex/fix-aligned-page-memory-estimation
Open

Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:codex/fix-aligned-page-memory-estimation

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Problem

Aligned TsFile page-memory estimation reparsed every already cached value chunk whenever a new value chunk was considered. For wide aligned devices this made the scan parser's work quadratic in the number of value chunks and could leave pipe history threads at 100% CPU for hours.

Changes

  • Cache per-page time sizes and row counts once per pending aligned chunk group.
  • Incrementally add each value chunk's page sizes and memory contribution, preserving the existing suffix-maximum semantics.
  • Reuse the incremental estimates when constructing the memory-controlled reader.
  • Add a regression test comparing incremental estimates with the full calculation across multiple pages and value chunks.

Validation

  • mvn spotless:check -pl iotdb-core/datanode
  • Direct compilation with the existing DataNode classpath and JUnitCore: OK (1 test)

The full Maven test invocation was attempted but could not complete because the local machine was already under heavy Java-process memory pressure; the JVM first failed to reserve memory and the reduced-heap retry remained in the existing module compilation for several minutes without producing a result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant