Skip to content

io_queue: checkpoint class queue lengths on change, not on every poll - #330

Open
travisdowns wants to merge 1 commit into
redpanda-data:v26.3.xfrom
travisdowns:td-ioq-checkpoint-on-change
Open

travisdowns wants to merge 1 commit into
redpanda-data:v26.3.xfrom
travisdowns:td-ioq-checkpoint-on-change

Conversation

@travisdowns

@travisdowns travisdowns commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

poll_io_queue() walked every priority class the queue has ever seen,
twice per poll, to checkpoint the integrated_length counters behind the
integrated_queue_length and integrated_disk_queue_length metrics. The
cost grows with the number of classes, almost all of them idle at any
moment: with Redpanda's ~20 scheduling groups it is about 2.5k
instructions per produce request, 1.6% of the request. And because
classes are created lazily, the cost differs from run to run.

Checkpoint each counter right before it changes instead, and once more
when its metric is read. That makes the integral exact (every value
times exactly the time it was held), where the per-poll checkpoints
multiplied the value at poll time by the whole interval since the
previous poll, and it no longer needs the queue to be polled at all.
The metrics keep their names and units.

The new test holds requests queued, and then dispatched, without
polling in between, and checks both integrals. Without this change the
queued integral stays 0.

Additional details

Jira: CORE-18383

Measured with the DynamoRIO produce-request footprint test (one request in flight, the partition leader traced): with Redpanda's separate scheduling groups, removing the per-poll walk lowers the cross-run medoid difference from 0.05% to 0.01% and the per-request instruction count by about 2.5k.

Testing: io_queue_test (dev mode), including the new test_integrated_queue_length_metrics, which fails without the change (the queued integral stays 0) and passes with it.

Follow-ups:

  • Redpanda needs a seastar pin bump (bazel/repositories.bzl) after this merges.
  • scylladb/seastar master has the same per-poll loops, but this commit conflicts there after upstream's priority class refactor, so an upstream PR needs a rebase onto master.

poll_io_queue() walked every priority class the queue has ever seen,
twice per poll, to checkpoint the integrated_length counters behind the
integrated_queue_length and integrated_disk_queue_length metrics. The
cost grows with the number of classes, almost all of them idle at any
moment: with Redpanda's ~20 scheduling groups it is about 2.5k
instructions per produce request, 1.6% of the request. And because
classes are created lazily, the cost differs from run to run.

Checkpoint each counter right before it changes instead, and once more
when its metric is read. That makes the integral exact (every value
times exactly the time it was held), where the per-poll checkpoints
multiplied the value at poll time by the whole interval since the
previous poll, and it no longer needs the queue to be polled at all.
The metrics keep their names and units.

The new test holds requests queued, and then dispatched, without
polling in between, and checks both integrals. Without this change the
queued integral stays 0.
Copilot AI balanced review requested due to automatic review settings November 19, 2026 01:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The timing checks use a fixed upper bound that can reject correct integrals when the test thread resumes late on loaded CI workers, so expectations should use actual elapsed time or a controllable clock.

1 open issue
What changed in this PR

This PR removes per-poll priority-class scans from the I/O queue and makes queue-length integrals reflect time spent at each value.

Changes:

  • Checkpoint counters before queue-length changes and when metrics are read.
  • Preserve metric names and units while removing checkpoint loops from polling.
  • Add a regression test for queued and dispatched requests held without polling.
File Description
tests/​unit/​io_queue_test.cc Tests both queue-length integrals during holds without polling.
src/​core/​io_queue.cc Moves checkpointing to counter changes and metric reads.

Review effort: Balanced

constexpr auto hold = std::chrono::milliseconds(300);
// nr requests held for `hold` integrate to nr * 0.3 = 6 request-seconds;
// the metrics truncate to whole seconds and the clock is a lowres one
auto expected_lo = 5u, expected_hi = 7u;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants