Skip to content

fix: bound the async admission call so a wedged worker cannot hang callers - #185

Merged
puzza007 merged 1 commit into
masterfrom
bounded-admission-timeout
Jul 18, 2026
Merged

puzza007 merged 1 commit into
masterfrom
bounded-admission-timeout

Conversation

@puzza007

Copy link
Copy Markdown
Owner

The admission handshake from the exactly-once dispatch work (#178) used an infinity gen_server:call, so a wedged worker — typically its C port stopped draining the pipe and port_command blocked the worker mid-callback — silently absorbed async_req callers with no bound at all. This was the one regression surface of making dispatch a synchronous call.

Admission is a port write plus a map insert, microseconds when healthy, so the call is now bounded at 5s: a worker that cannot answer in time yields an immediate {error, #{code => admission_timeout}} and the caller can retry — re-rolling onto a healthy random worker — instead of blocking indefinitely. A request the worker admits after the caller gave up is cleaned up by its own request timer. The fail-fast contract is uniform: a timeout during overload-spillover also returns immediately, so an admission waits at most one bound, never one per candidate.

The bound applies only to the async admission ack, derived from the message kind in one place (admission_bound/1): a synchronous request's single call carries the whole response — the worker replies when the transfer or its request timer completes — so it stays unbounded at the call layer and is governed by the request timeout as before. (The first cut bounded both paths and the suite caught it immediately: 120 sync tests truncated at 5s.) attempt/2 guards its timeout mapping on the bound, since on an unbounded call a {timeout, _} exit can only be a worker that died with that exit reason — without the guard, sync callers could see the async-only error code.

Deterministic coverage in katipo_async_failure_SUITE: a suspended worker (the messages-queue-but-nothing-runs shape of the production wedge) must produce admission_timeout within the bound. 199 tests, dialyzer, xref, and lint pass.

Top of the stack: #180 → #182 → #183 → #184 → this. Merge bottom-up; I'll rebase each layer as the one below lands.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Vf2gxZLpxT4HvBfZKCgP3g

@codecov

codecov Bot commented Jul 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.30769% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 78.23%. Comparing base (046fce5) to head (bec37ad).

Files with missing lines Patch % Lines
src/katipo.erl 92.30% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #185      +/-   ##
==========================================
- Coverage   78.31%   78.23%   -0.09%     
==========================================
  Files           9        9              
  Lines        1365     1369       +4     
==========================================
+ Hits         1069     1071       +2     
- Misses        296      298       +2     
Flag Coverage Δ
c 78.23% <92.30%> (-0.09%) ⬇️
erlang 78.23% <92.30%> (-0.09%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@puzza007
puzza007 force-pushed the request-backpressure branch from 098536b to d82a779 Compare July 18, 2026 08:48
@puzza007
puzza007 force-pushed the bounded-admission-timeout branch from a83b638 to 47b9148 Compare July 18, 2026 08:49
@puzza007
puzza007 force-pushed the request-backpressure branch from d82a779 to cc2c140 Compare July 18, 2026 08:59
@puzza007
puzza007 force-pushed the bounded-admission-timeout branch from 47b9148 to ba12838 Compare July 18, 2026 08:59
@puzza007
puzza007 force-pushed the request-backpressure branch from cc2c140 to 6178814 Compare July 18, 2026 09:05
@puzza007
puzza007 force-pushed the bounded-admission-timeout branch from ba12838 to 170d439 Compare July 18, 2026 09:05
@puzza007
puzza007 force-pushed the request-backpressure branch from 6178814 to 6770c62 Compare July 18, 2026 09:09
@puzza007
puzza007 force-pushed the bounded-admission-timeout branch from 170d439 to 9261944 Compare July 18, 2026 09:09
@puzza007
puzza007 force-pushed the request-backpressure branch 3 times, most recently from 7326048 to 2244db0 Compare July 18, 2026 09:34
Base automatically changed from request-backpressure to master July 18, 2026 09:37
…llers

The admission handshake introduced with the exactly-once dispatch work
used an infinity gen_server:call, so a wedged worker -- typically its C
port stopped draining the pipe and port_command blocked the worker
mid-callback -- silently absorbed async_req callers with no bound at
all. Admission is a port write plus a map insert (microseconds when
healthy), so bound the call at 5s: a worker that cannot answer in time
yields an immediate {error, #{code => admission_timeout}} and the caller
can retry, re-rolling onto a healthy random worker, instead of blocking
indefinitely. A request the worker admits after the caller gave up is
cleaned up by its own request timer.

The bound applies only to the async admission ack and is derived from
the message kind in one place (admission_bound/1): a synchronous
request's single call carries the whole response -- the worker replies
when the transfer or its request timer completes -- so it stays
unbounded at the call layer and is governed by the request timeout as
before. attempt/2 guards its timeout mapping on the bound, since on an
unbounded call a {timeout, _} exit can only be a worker that died with
that exit reason. The fail-fast contract is uniform: a timeout during
overload-spillover also returns immediately, so an admission waits at
most one bound, never one per candidate.

Deterministic coverage in katipo_async_failure_SUITE: a suspended
worker (the messages-queue-but-nothing-runs shape of the production
wedge) must produce admission_timeout within the bound. 199 tests,
dialyzer, xref, and lint pass.
@puzza007
puzza007 force-pushed the bounded-admission-timeout branch from 9261944 to bec37ad Compare July 18, 2026 09:37
@puzza007
puzza007 merged commit 95e1b11 into master Jul 18, 2026
12 checks passed
@puzza007
puzza007 deleted the bounded-admission-timeout branch July 18, 2026 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant