Repository navigation
Treat consumer pause and resume as the desired state - #102
Merged
Merged
Conversation
### Problem statement After a channel reset the consumer is `NOT_CONSUMING` until the reopened channel receives `basic.qos-ok`. A `pause()` in that window moved it to `PAUSING` and created a cancel future, but no `basic.cancel` was sent, so no `basic.cancel-ok` could ever resolve it (#100). With host health monitoring, a host going unhealthy and the connection dropping tend to happen together, so this is easy to hit. As a result: - every later `resume()` was refused ("pause already in flight") until the next reset; - `cancel()`, and so `drain()`, hung on the same future, and a cancel before `basic.qos-ok` was overwritten back to paused; - a pause then resume before `basic.qos-ok` left the channel never ready, so the hung-channel timer reset the whole connection after 60s. Separately, `basic.qos-ok` marked the consumer paused unconditionally. A resume accepted while the QoS was in flight has already sent `basic.consume`; if a pause followed, the consumer reported paused while the broker delivered, and the pause never completed. More generally, `pause()` and `resume()` refused requests they could have carried out later, such as a resume while the channel was reopening or while a pause was in flight, leaving the consumer waiting for the next health poll. ### Proposed changes `pause()` and `resume()` now set the desired state, and the channel gets there at the next opportunity. A request is only refused if it repeats the state already reached, or can't be carried out: no consumer, a cancelled consumer (cancel is final), or a channel that is permanently closing. Repeating a request still in flight returns the same future. - A pause with nothing consuming on the broker completes immediately. - A resume while the channel is closed or reopening is carried out when it reopens; a resume while a pause is in flight sends `basic.consume` as soon as the `basic.cancel-ok` arrives. - `basic.qos-ok` only pauses or restarts a consumer that is not consuming, leaving one with a consume already on the wire to its reply. - A request overtaken by the opposite one fails as superseded: a pending resume as soon as a pause or cancel arrives, a pending pause at its `basic.cancel-ok`. A `basic.consume-ok` only completes a resume if the consumer is then consuming. - Pausing a paused consumer is now refused, like resuming a consuming one. - Resume count and latency metrics only count resumes that undo a pause, and stale pause/resume start times are cleared. - Add `docs/consumer-pause-resume.md` describing these rules. ### Testing New and updated `ReceiveChannelTests` cover pause, resume and cancel in each phase of a reopen, while QoS is in flight, while the opposite request is in flight, on a permanently closing channel, and the metrics. Each fix's tests failed before the change. Signed-off-by: Will Hoy <whoy2@bloomberg.net>
michalovits
approved these changes
Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #100
Problem statement
After a channel reset the consumer is
NOT_CONSUMINGuntil the reopened channel receivesbasic.qos-ok. Apause()in that window moved it toPAUSINGand created a cancel future, but nobasic.cancelwas sent, so nobasic.cancel-okcould ever resolve it. With host health monitoring, a host going unhealthy and the connection dropping tend to happen together, so this is easy to hit. As a result:resume()was refused ("pause already in flight") until the next reset;cancel(), and sodrain(), hung on the same future, and a cancel beforebasic.qos-okwas overwritten back to paused;basic.qos-okleft the channel never ready, so the hung-channel timer reset the whole connection after 60s.Resolving the pending future in the paused branch of the
basic.qos-okhandler, as suggested in #100, fixes the first symptom but not the pause-then-resume case, which never takes that branch.Separately,
basic.qos-okmarked the consumer paused unconditionally. A resume accepted while the QoS was in flight has already sentbasic.consume; if a pause followed, the consumer reported paused while the broker delivered, and the pause never completed.More generally,
pause()andresume()refused requests they could have carried out later, such as a resume while the channel was reopening or while a pause was in flight, leaving the consumer waiting for the next health poll.Proposed changes
pause()andresume()now set the desired state, and the channel gets there at the next opportunity. A request is only refused if it repeats the state already reached, or can't be carried out: no consumer, a cancelled consumer (cancel is final), or a channel that is permanently closing. Repeating a request still in flight returns the same future.basic.consumeas soon as thebasic.cancel-okarrives.basic.qos-okonly pauses or restarts a consumer that is not consuming, leaving one with a consume already on the wire to its reply.basic.cancel-ok. Abasic.consume-okonly completes a resume if the consumer is then consuming.ChannelMap, the only caller, ignores these futures.docs/consumer-pause-resume.mddescribing these rules.Thanks to @SrinathhSatuluri for the report and analysis in #100.
Remaining work
ReceiveChannelTestscover pause, resume and cancel in each phase of a reopen, while QoS is in flight, while the opposite request is in flight, on a permanently closing channel, and the metrics. Each fix's tests failed before the change.main(0e745528): rmqamqp 269, rmqa 137, rmqamqpt 149, rmqio 75, rmqt 71, rmqtestmocks 1, rmqtestutil 4.