Skip to content

Fix same-region flush stalls during write admission - #18985

Merged
jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:fix/flush-selected-memtables-before-admission-wait
Oct 22, 2026
Merged

jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:fix/flush-selected-memtables-before-admission-wait

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

A rejected writer can hold its DataRegion write lock while waiting for memory from a larger, already-selected MemTable in the same region. The asynchronous flush submitter needs that same lock, so the selected flush can remain blocked until the write times out with 606.

Let the waiting writer submit other already-marked MemTables in its own region through the existing flush entry point. Keep the current processor out of this path because it still owns the pending write and memory reservations. This keeps the existing largest-MemTable selection, rejection thresholds, flush policy, and duplicate-task checks; it does not mark additional MemTables or force the current small MemTable into a new file.

Add real row and tablet regressions that retain the region lock throughout admission, cross a primitive-array allocation boundary, and verify that only the selected processor flushes. Both smaller MemTables are reused, reservations are settled, and all 515 timestamp/value pairs remain readable from exactly three TsFiles after closing.

Validation

  • Before the fix, both new regressions fail after the configured five-second admission wait; the tablet result contains code 606.
  • Both new regressions pass with the fix.
  • TsFileProcessorTest and SystemInfoTest: 40 tests passed, covering admission rollback, hard-limit rejection, largest-MemTable selection, and duplicate submission prevention.
  • Maven reactor compilation, Checkstyle, Spotless, and forbidden-API checks passed for the DataNode and its dependencies.
  • The expanded DataRegionTest run was not clean on this Windows host: a WAL direct-buffer allocation failed with OutOfMemoryError, followed by file-cleanup errors and a timed-flush MemTable-count assertion. The new regressions passed in that run and also passed in isolation. No cluster-scale throughput benchmark was run.

This PR has:

  • been self-reviewed for lock ordering and pending-write ownership.
  • added comments explaining the lock dependency.
  • added row and tablet regression coverage.

@jt2594838
jt2594838 merged commit 592d663 into apache:master Oct 22, 2026
39 checks passed
@jt2594838
jt2594838 deleted the fix/flush-selected-memtables-before-admission-wait branch October 22, 2026 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants