classifySpawnFailure's fallthrough is its most aggressive schedule — an unrecognised provider outage gets the hardest retry

[lilyshen0722]

Summary

classifySpawnFailure falls through to RUNTIME, and RUNTIME is the most aggressive schedule in the table. So an error the classifier does not recognise gets the treatment reserved for "transient local fault, probe hard" — the opposite of what an unrecognised provider-side failure deserves.

This is the layer above #995, which is correct and should land as written. #995 adds one word to a vocabulary; this issue is about which way that vocabulary fails when a word is missing.

Measured

Against the module the fleet runs (cli/src/lib/spawn-retry.js, live/main-tracking @ 0cd2bdc9), intervalMs: 5000, jitter 0, * = circuit open:

class=quota          n=1,2,3 ->  900s*  900s*  900s*
class=configuration  n=1,2,3 ->  900s*  900s*  900s*
class=rate_limit     n=1,2,3 ->   60s*  120s*  240s*
class=runtime        n=1,2,3 ->    5s     10s    60s*

RUNTIME is 180× faster than QUOTA on the first retry and is the only class that does not open the circuit at n=1. It is also the value returned by the final return of classifySpawnFailure — i.e. the default for anything unmatched.

It has now happened twice, to two providers

The file records the first instance in a comment above QUOTA_RE:

out of credits is codex's exact wording for an exhausted workspace balance … Without it that outage classified as RUNTIME and drew the shortest backoff — observed live on 2026-08-03 before this pattern was added.

The second was last night: Claude's You've hit your session limit · resets 4am is not in QUOTA_RE (which does contain usage limit), so nine seats probed an account that could not answer until a fixed reset time, on a 5s/10s/60s ladder. #995 adds the string.

Two incidents, two providers, one remedy applied twice. The vocabulary will keep drifting — provider error wording is not an API and changes without notice — so each future reword silently re-selects the most aggressive branch, and the only signal is a fleet-wide stall that someone has to notice by hand.

Suggested fix

Separate "we recognise this as a transient local fault" from "we do not recognise this at all", and give the second the conservative branch:

• add SPAWN_FAILURE_CLASS.UNKNOWN (or have the fallthrough return QUOTA-equivalent timing), scheduled like CONFIGURATION — long ceiling, circuit open immediately;
• keep RUNTIME for failures that are positively identified as local and transient (non-zero exit with no recognised provider text, spawn ENOENT of the wrapper itself, etc.);
• log the class alongside the delay, so an unclassified provider outage is visible as unknown in the seat log instead of presenting as an ordinary runtime blip.

The cost of getting it wrong in the conservative direction is a genuinely transient fault waiting 15 minutes. The cost in the current direction is a fleet hammering a provider through an outage it cannot shorten — and, per #993, burning deliveries that the GC then destroys.

Related pattern

Second classifier found in the same session whose unrecognised verdict resolves toward doing more work:

• Cascade governor fails open on a backlogged seat: classifyTrigger returns 'unknown' past the 10-message snapshot #992 — classifyTrigger returns 'unknown' when the trigger message is outside a 10-message window; 'unknown' is admitted unconditionally by the cascade governor and never counted toward the cap. Fails open exactly when a seat is backlogged.
• this — classifySpawnFailure returns RUNTIME when no pattern matches; RUNTIME retries fastest and delays the circuit. Fails open exactly when a provider is down.

Both defaults are individually defensible and both bite hardest under precisely the load they exist to damp. Worth a line in docs/development/review-checklist.md: when a classifier feeds a limiter, its unknown verdict must select the conservative branch, and the reviewer should ask what produces unknown under load.

[lilyshen0722]

Precision on "most aggressive": where the fallthrough actually costs

A correction was offered in-pod that the RUNTIME fallthrough "climbs to the same 15-minute ceiling by the third probe," making the excess just two probes. Measured, it is six, and the ceiling arrives at the seventh:

n=1  delay=   5s  open=false   cumulative=    5s (0.1min)
n=2  delay=  10s  open=false   cumulative=   15s (0.3min)
n=3  delay=  60s  open=true    cumulative=   75s (1.3min)
n=4  delay= 120s  open=true    cumulative=  195s (3.3min)
n=5  delay= 240s  open=true    cumulative=  435s (7.3min)
n=6  delay= 480s  open=true    cumulative=  915s (15.3min)
n=7  delay= 900s  open=true    cumulative= 1815s (30.3min)

The shape is 60s × 2^(n − SPAWN_CIRCUIT_THRESHOLD) capped at SPAWN_RETRY_MAX_MS, so n=3 is 60s — one fifteenth of the ceiling, not the ceiling.

So the cost of a misclassification is six extra probes spread over the first ~15 minutes, against a provider that by construction cannot answer. That is bounded and it is not catastrophic; it is also about five times the two-probe version, and it is the figure this issue should carry.

Two things this sharpens rather than changes:

• The n=1 framing in the original report stands as written — 180× on the first retry, and the only class that does not open the circuit at n=1. Those are claims about the first probe and they are exact.
• The argument for a conservative fallthrough does not rest on the tail. It rests on the first two probes landing 5s and 10s apart while the circuit stays closed, and on that being what an unrecognised error selects. Whether the ladder converges at n=3 or n=7 changes the magnitude, not the direction.

Worth recording because the correction was offered in good faith and in the humble direction — it made the finding smaller, which is exactly the kind of amendment that gets accepted without being checked.