Repository navigation
io_queue: checkpoint class queue lengths on change, not on every poll - #330
Open
travisdowns wants to merge 1 commit into
Open
travisdowns wants to merge 1 commit into
travisdowns wants to merge 1 commit into
Conversation
poll_io_queue() walked every priority class the queue has ever seen, twice per poll, to checkpoint the integrated_length counters behind the integrated_queue_length and integrated_disk_queue_length metrics. The cost grows with the number of classes, almost all of them idle at any moment: with Redpanda's ~20 scheduling groups it is about 2.5k instructions per produce request, 1.6% of the request. And because classes are created lazily, the cost differs from run to run. Checkpoint each counter right before it changes instead, and once more when its metric is read. That makes the integral exact (every value times exactly the time it was held), where the per-poll checkpoints multiplied the value at poll time by the whole interval since the previous poll, and it no longer needs the queue to be polled at all. The metrics keep their names and units. The new test holds requests queued, and then dispatched, without polling in between, and checks both integrals. Without this change the queued integral stays 0.
There was a problem hiding this comment.
🟡 Changes recommended
The timing checks use a fixed upper bound that can reject correct integrals when the test thread resumes late on loaded CI workers, so expectations should use actual elapsed time or a controllable clock.
1 open issue
What changed in this PR
This PR removes per-poll priority-class scans from the I/O queue and makes queue-length integrals reflect time spent at each value.
Changes:
- Checkpoint counters before queue-length changes and when metrics are read.
- Preserve metric names and units while removing checkpoint loops from polling.
- Add a regression test for queued and dispatched requests held without polling.
| File | Description |
|---|---|
tests/unit/io_queue_test.cc |
Tests both queue-length integrals during holds without polling. |
src/core/io_queue.cc |
Moves checkpointing to counter changes and metric reads. |
Review effort: Balanced
| constexpr auto hold = std::chrono::milliseconds(300); | ||
| // nr requests held for `hold` integrate to nr * 0.3 = 6 request-seconds; | ||
| // the metrics truncate to whole seconds and the clock is a lowres one | ||
| auto expected_lo = 5u, expected_hi = 7u; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

poll_io_queue() walked every priority class the queue has ever seen,
twice per poll, to checkpoint the integrated_length counters behind the
integrated_queue_length and integrated_disk_queue_length metrics. The
cost grows with the number of classes, almost all of them idle at any
moment: with Redpanda's ~20 scheduling groups it is about 2.5k
instructions per produce request, 1.6% of the request. And because
classes are created lazily, the cost differs from run to run.
Checkpoint each counter right before it changes instead, and once more
when its metric is read. That makes the integral exact (every value
times exactly the time it was held), where the per-poll checkpoints
multiplied the value at poll time by the whole interval since the
previous poll, and it no longer needs the queue to be polled at all.
The metrics keep their names and units.
The new test holds requests queued, and then dispatched, without
polling in between, and checks both integrals. Without this change the
queued integral stays 0.
Additional details
Jira: CORE-18383
Measured with the DynamoRIO produce-request footprint test (one request in flight, the partition leader traced): with Redpanda's separate scheduling groups, removing the per-poll walk lowers the cross-run medoid difference from 0.05% to 0.01% and the per-request instruction count by about 2.5k.
Testing:
io_queue_test(dev mode), including the newtest_integrated_queue_length_metrics, which fails without the change (the queued integral stays 0) and passes with it.Follow-ups:
bazel/repositories.bzl) after this merges.