Skip to content

Optimize subscription WAL acknowledgement path - #18845

Open
Caideyipi wants to merge 3 commits into
masterfrom
fix/subscription-wal-prefetch-hotspots
Open

Caideyipi wants to merge 3 commits into
masterfrom
fix/subscription-wal-prefetch-hotspots

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Subscription ACKs committed writer progress and refreshed WAL retention inside an inFlightEvents.compute callback. CPU profiles and thread dumps showed WAL metadata scanning under the ConcurrentHashMap node lock blocking subscription prefetch threads.

Move the commit and retention refresh outside that callback, then remove the exact event after a successful commit. A failed commit leaves the event available for retry. Pre-size the per-entry writer progress lists when deserializing WAL metadata to reduce repeated ArrayList growth.

Validation

  • mvn spotless:apply -pl iotdb-core/datanode -DskipTests
  • mvn -pl iotdb-core/datanode -DskipTests compile
  • mvn -pl iotdb-core/datanode -Dtest=ConsensusPrefetchingQueueTest test — 46 tests passed.

Cluster performance measurements have not been rerun after this change.

Key changed classes

  • ConsensusPrefetchingQueue
  • WALMetaData

@codecov

codecov Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 16.32653% with 41 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.63%. Comparing base (d12bb33) to head (abef0a8).
⚠️ Report is 18 commits behind head on master.

Files with missing lines Patch % Lines
...on/broker/consensus/ConsensusPrefetchingQueue.java 0.00% 41 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18845      +/-   ##
============================================
+ Coverage     47.58%   47.63%   +0.04%     
  Complexity      806      806              
============================================
  Files          5562     5562              
  Lines        408293   408329      +36     
  Branches      53591    53598       +7     
============================================
+ Hits         194273   194488     +215     
+ Misses       214020   213841     -179     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…l-prefetch-hotspots

# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java
@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Self-review passed. I fixed V2 WAL metadata deserialization preallocation and corrected ACK cleanup ordering so event cleanup follows the in-flight map lock order, while the potentially slow WAL retention refresh remains outside the map callback. Targeted tests passed: ConsensusPrefetchingQueueTest (53) and WALMetaDataV3CompatibilityTest (6).

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
E Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant