Skip to content

Treat consumer pause and resume as the desired state - #102

Merged
willhoy merged 1 commit into
mainfrom
consumer-pause-resume-desired-state
Oct 29, 2026
Merged

willhoy merged 1 commit into
mainfrom
consumer-pause-resume-desired-state

Conversation

@willhoy

@willhoy willhoy commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Fixes #100

Problem statement

After a channel reset the consumer is NOT_CONSUMING until the reopened channel receives basic.qos-ok. A pause() in that window moved it to PAUSING and created a cancel future, but no basic.cancel was sent, so no basic.cancel-ok could ever resolve it. With host health monitoring, a host going unhealthy and the connection dropping tend to happen together, so this is easy to hit. As a result:

  • every later resume() was refused ("pause already in flight") until the next reset;
  • cancel(), and so drain(), hung on the same future, and a cancel before basic.qos-ok was overwritten back to paused;
  • a pause then resume before basic.qos-ok left the channel never ready, so the hung-channel timer reset the whole connection after 60s.

Resolving the pending future in the paused branch of the basic.qos-ok handler, as suggested in #100, fixes the first symptom but not the pause-then-resume case, which never takes that branch.

Separately, basic.qos-ok marked the consumer paused unconditionally. A resume accepted while the QoS was in flight has already sent basic.consume; if a pause followed, the consumer reported paused while the broker delivered, and the pause never completed.

More generally, pause() and resume() refused requests they could have carried out later, such as a resume while the channel was reopening or while a pause was in flight, leaving the consumer waiting for the next health poll.

Proposed changes

pause() and resume() now set the desired state, and the channel gets there at the next opportunity. A request is only refused if it repeats the state already reached, or can't be carried out: no consumer, a cancelled consumer (cancel is final), or a channel that is permanently closing. Repeating a request still in flight returns the same future.

  • A pause with nothing consuming on the broker completes immediately.
  • A resume while the channel is closed or reopening is carried out when it reopens; a resume while a pause is in flight sends basic.consume as soon as the basic.cancel-ok arrives.
  • basic.qos-ok only pauses or restarts a consumer that is not consuming, leaving one with a consume already on the wire to its reply.
  • A request overtaken by the opposite one fails as superseded: a pending resume as soon as a pause or cancel arrives, a pending pause at its basic.cancel-ok. A basic.consume-ok only completes a resume if the consumer is then consuming.
  • Behaviour change: pausing a paused consumer is now refused, like resuming a consuming one. ChannelMap, the only caller, ignores these futures.
  • Resume count and latency metrics only count resumes that undo a pause, and stale pause/resume start times are cleared.
  • Add docs/consumer-pause-resume.md describing these rules.

Thanks to @SrinathhSatuluri for the report and analysis in #100.

Remaining work

  • Unit Tests: new and updated ReceiveChannelTests cover pause, resume and cancel in each phase of a reopen, while QoS is in flight, while the opposite request is in flight, on a permanently closing channel, and the metrics. Each fix's tests failed before the change.
  • All unit test suites pass on top of main (0e745528): rmqamqp 269, rmqa 137, rmqamqpt 149, rmqio 75, rmqt 71, rmqtestmocks 1, rmqtestutil 4.
  • Integration Tests
  • Documentation

### Problem statement
After a channel reset the consumer is `NOT_CONSUMING` until the reopened
channel receives `basic.qos-ok`. A `pause()` in that window moved it to
`PAUSING` and created a cancel future, but no `basic.cancel` was sent, so
no `basic.cancel-ok` could ever resolve it (#100). With host health
monitoring, a host going unhealthy and the connection dropping tend to
happen together, so this is easy to hit. As a result:

- every later `resume()` was refused ("pause already in flight") until
  the next reset;
- `cancel()`, and so `drain()`, hung on the same future, and a cancel
  before `basic.qos-ok` was overwritten back to paused;
- a pause then resume before `basic.qos-ok` left the channel never ready,
  so the hung-channel timer reset the whole connection after 60s.

Separately, `basic.qos-ok` marked the consumer paused unconditionally. A
resume accepted while the QoS was in flight has already sent
`basic.consume`; if a pause followed, the consumer reported paused while
the broker delivered, and the pause never completed.

More generally, `pause()` and `resume()` refused requests they could have
carried out later, such as a resume while the channel was reopening or
while a pause was in flight, leaving the consumer waiting for the next
health poll.

### Proposed changes
`pause()` and `resume()` now set the desired state, and the channel gets
there at the next opportunity. A request is only refused if it repeats
the state already reached, or can't be carried out: no consumer, a
cancelled consumer (cancel is final), or a channel that is permanently
closing. Repeating a request still in flight returns the same future.

- A pause with nothing consuming on the broker completes immediately.
- A resume while the channel is closed or reopening is carried out when
  it reopens; a resume while a pause is in flight sends `basic.consume` as
  soon as the `basic.cancel-ok` arrives.
- `basic.qos-ok` only pauses or restarts a consumer that is not
  consuming, leaving one with a consume already on the wire to its reply.
- A request overtaken by the opposite one fails as superseded: a pending
  resume as soon as a pause or cancel arrives, a pending pause at its
  `basic.cancel-ok`. A `basic.consume-ok` only completes a resume if the
  consumer is then consuming.
- Pausing a paused consumer is now refused, like resuming a consuming one.
- Resume count and latency metrics only count resumes that undo a pause,
  and stale pause/resume start times are cleared.
- Add `docs/consumer-pause-resume.md` describing these rules.

### Testing
New and updated `ReceiveChannelTests` cover pause, resume and cancel in
each phase of a reopen, while QoS is in flight, while the opposite request
is in flight, on a permanently closing channel, and the metrics. Each
fix's tests failed before the change.

Signed-off-by: Will Hoy <whoy2@bloomberg.net>
@willhoy
willhoy merged commit 6127c7f into main Oct 29, 2026
7 checks passed
@willhoy
willhoy deleted the consumer-pause-resume-desired-state branch October 29, 2026 12:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ReceiveChannel::pause() during channel reopen never completes, blocking resume

2 participants