Skip to content

[Pipe] Unify TsFile parser allocation and resize quotas - #18984

Merged
jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:codex/fix-pipe-parser-resize-quota
Oct 22, 2026
Merged

jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:codex/fix-pipe-parser-resize-quota

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Historical tablet-format synchronization of very wide aligned TsFiles can repeatedly fail parser resizes while much of the global Pipe memory pool remains available. Parser working memory was charged to the tablet-only subquota, and initial allocations checked the existing usage while resizes checked usage including the requested growth.

Introduce a TsFile parser memory block charged to TsFile usage. Allocate it through the same resize path used for later growth, enforcing 0.95 * (tablet threshold + TsFile threshold) against post-allocation combined usage (57% with the default thresholds). This lets parsing use the shared budget while generated tablet events can still allocate memory to make downstream progress. Ordinary tablet/TsFile limits and the global pool capacity remain enforced.

Both parser and memory-manager resize exceptions report global usage, tablet/TsFile usage, their individual limits, the combined limit, the protected parser limit, and the requested increment. The new message is available in English and Chinese.

Validation

  • Full reactor compilation passed for the final commit in both locales: mvn test-compile -DskipTests -Dcheckstyle.skip=true and the same command with -P with-zh-locale. Checkstyle was checked separately for the changed production files to stay within the local JVM memory limit.
  • PipeMemoryManagerResizeTest and PipeMemoryManagerTest: 24 tests passed per locale, using JDK 17 javac/JUnit with Maven's resolved dependency classpath.
  • Regression coverage includes zero-sized block growth, allocation/resize boundary parity, downstream tablet progress, mixed tablet/TsFile usage, global exhaustion, parent release, failed-allocation cleanup, and concurrent allocations.
  • Spotless passed; Checkstyle passed for changed production files.
  • English/Chinese message key and format-specifier parity checked across all 1,217 Pipe message constants.

The original 200 GB, two-cluster workload has not been rerun; its end-to-end retry count and synchronization duration still need confirmation with this patch.

This PR has:

  • been self-reviewed, including concurrent allocation and release accounting.
  • added unit tests and Javadocs for the changed behavior.

@jt2594838
jt2594838 merged commit b626c60 into apache:master Oct 22, 2026
39 checks passed
@jt2594838
jt2594838 deleted the codex/fix-pipe-parser-resize-quota branch October 22, 2026 08:29
VGalaxies pushed a commit to VGalaxies/iotdb that referenced this pull request Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants