Repository navigation
fix: bound the async admission call so a wedged worker cannot hang callers - #185
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #185 +/- ##
==========================================
- Coverage 78.31% 78.23% -0.09%
==========================================
Files 9 9
Lines 1365 1369 +4
==========================================
+ Hits 1069 1071 +2
- Misses 296 298 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
puzza007
force-pushed
the
request-backpressure
branch
from
July 18, 2026 08:48
098536b to
d82a779
Compare
puzza007
force-pushed
the
bounded-admission-timeout
branch
from
July 18, 2026 08:49
a83b638 to
47b9148
Compare
puzza007
force-pushed
the
request-backpressure
branch
from
July 18, 2026 08:59
d82a779 to
cc2c140
Compare
puzza007
force-pushed
the
bounded-admission-timeout
branch
from
July 18, 2026 08:59
47b9148 to
ba12838
Compare
puzza007
force-pushed
the
request-backpressure
branch
from
July 18, 2026 09:05
cc2c140 to
6178814
Compare
puzza007
force-pushed
the
bounded-admission-timeout
branch
from
July 18, 2026 09:05
ba12838 to
170d439
Compare
puzza007
force-pushed
the
request-backpressure
branch
from
July 18, 2026 09:09
6178814 to
6770c62
Compare
puzza007
force-pushed
the
bounded-admission-timeout
branch
from
July 18, 2026 09:09
170d439 to
9261944
Compare
puzza007
force-pushed
the
request-backpressure
branch
3 times, most recently
from
July 18, 2026 09:34
7326048 to
2244db0
Compare
…llers
The admission handshake introduced with the exactly-once dispatch work
used an infinity gen_server:call, so a wedged worker -- typically its C
port stopped draining the pipe and port_command blocked the worker
mid-callback -- silently absorbed async_req callers with no bound at
all. Admission is a port write plus a map insert (microseconds when
healthy), so bound the call at 5s: a worker that cannot answer in time
yields an immediate {error, #{code => admission_timeout}} and the caller
can retry, re-rolling onto a healthy random worker, instead of blocking
indefinitely. A request the worker admits after the caller gave up is
cleaned up by its own request timer.
The bound applies only to the async admission ack and is derived from
the message kind in one place (admission_bound/1): a synchronous
request's single call carries the whole response -- the worker replies
when the transfer or its request timer completes -- so it stays
unbounded at the call layer and is governed by the request timeout as
before. attempt/2 guards its timeout mapping on the bound, since on an
unbounded call a {timeout, _} exit can only be a worker that died with
that exit reason. The fail-fast contract is uniform: a timeout during
overload-spillover also returns immediately, so an admission waits at
most one bound, never one per candidate.
Deterministic coverage in katipo_async_failure_SUITE: a suspended
worker (the messages-queue-but-nothing-runs shape of the production
wedge) must produce admission_timeout within the bound. 199 tests,
dialyzer, xref, and lint pass.
puzza007
force-pushed
the
bounded-admission-timeout
branch
from
July 18, 2026 09:37
9261944 to
bec37ad
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The admission handshake from the exactly-once dispatch work (#178) used an infinity gen_server:call, so a wedged worker — typically its C port stopped draining the pipe and port_command blocked the worker mid-callback — silently absorbed async_req callers with no bound at all. This was the one regression surface of making dispatch a synchronous call.
Admission is a port write plus a map insert, microseconds when healthy, so the call is now bounded at 5s: a worker that cannot answer in time yields an immediate {error, #{code => admission_timeout}} and the caller can retry — re-rolling onto a healthy random worker — instead of blocking indefinitely. A request the worker admits after the caller gave up is cleaned up by its own request timer. The fail-fast contract is uniform: a timeout during overload-spillover also returns immediately, so an admission waits at most one bound, never one per candidate.
The bound applies only to the async admission ack, derived from the message kind in one place (admission_bound/1): a synchronous request's single call carries the whole response — the worker replies when the transfer or its request timer completes — so it stays unbounded at the call layer and is governed by the request timeout as before. (The first cut bounded both paths and the suite caught it immediately: 120 sync tests truncated at 5s.) attempt/2 guards its timeout mapping on the bound, since on an unbounded call a {timeout, _} exit can only be a worker that died with that exit reason — without the guard, sync callers could see the async-only error code.
Deterministic coverage in katipo_async_failure_SUITE: a suspended worker (the messages-queue-but-nothing-runs shape of the production wedge) must produce admission_timeout within the bound. 199 tests, dialyzer, xref, and lint pass.
Top of the stack: #180 → #182 → #183 → #184 → this. Merge bottom-up; I'll rebase each layer as the one below lands.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Vf2gxZLpxT4HvBfZKCgP3g