-
Notifications
You must be signed in to change notification settings - Fork 782
docs(devlog): record the Wave 3/4 second pass, audit corrections and deferrals #1865
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -48,3 +48,61 @@ Cover BOTH paths explicitly: | |
| - A candidate with unknown modality support is skipped for an image request but still usable for a text one. | ||
| - No compatible candidate produces a typed failure naming the constraint, not a silent truncation. | ||
| - Cooldown and prior-attempt exclusion still apply unchanged. | ||
|
|
||
| --- | ||
|
|
||
| ## Implementation outcome (verified at `812e7c40b`) | ||
|
|
||
| The plan above proposed two mechanisms. Tracing the actual code changed what was needed. | ||
|
|
||
| ### What shipped | ||
|
|
||
| A distinct `input_admission_refused` code (`src/server/responses/core.ts:1979`) that is | ||
| `hop`-eligible in `comboFailureDecision` (`src/combos/failover.ts:132`), while upstream | ||
| `context_length_exceeded` still stops (`:124`). | ||
|
|
||
| This covers BOTH fallback paths, which the plan assumed needed separate fixes: | ||
|
|
||
| - **Combo fallback** consults `comboFailureDecision` directly. | ||
| - **Policy fallback** consults the SAME function — `shouldHopPolicyCandidate` | ||
| (`src/server/responses/policy-fallback.ts:72`) delegates to it, which the plan missed | ||
| when it proposed re-evaluating candidates inside `policy-fallback.ts:153`. | ||
|
|
||
| So the context-window half of #1524 is closed by one change rather than two. | ||
|
|
||
| ### What the plan got wrong | ||
|
|
||
| - **"Fallback reuses the frozen verdict, so a candidate is never re-checked" understates what | ||
| already worked.** Every concrete retry re-enters `handleResponses`, which runs | ||
| `checkInputAdmission` against the NEW candidate's own ceiling before upstream I/O | ||
| (`core.ts:1970`). The candidate was always checked; the defect was that its refusal | ||
| TERMINATED the chain instead of advancing it. | ||
| - **Modality was already filtered.** `evidenceFromBody` sets `imageInputRequired` | ||
| (`src/routing/request-evidence.ts:43`), and the evaluator turns it into a | ||
| `request-image-input` requirement per candidate (`src/routing/evaluator.ts:209`), producing | ||
| a `capability-unsatisfied` exclusion. `rankPolicyFallbackCandidates` only considers | ||
| candidates with `eligible === true` and zero exclusions | ||
| (`policy-fallback.ts:32`), so an image request can never hop onto a text-only candidate. | ||
| The plan's proposed image tests would have passed before any change. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This evidence path applies only to policy profiles. Combo selection still uses Useful? React with 👍 / 👎. |
||
| - **Unknown capability is already operator-controlled, not silently permissive.** | ||
| `excludedByUnknown` (`evaluator.ts:291`) excludes on unknown evidence when the profile sets | ||
| `unknownEvidence.capability = "exclude"`. The plan asked to make conservative-unknown | ||
| unconditional; doing so would change routing for every profile that deliberately allows | ||
| unknown evidence, which is a behavior change the issue does not ask for. | ||
|
|
||
| ### Remaining gap | ||
|
|
||
| Request context size is still `unknown` in `PolicyRequestEvidence`, so the INITIAL policy | ||
| evaluation cannot pre-exclude an oversized candidate — it is discovered at admission and then | ||
| hopped. That is correct but wasteful: the chain walks candidates one refusal at a time instead | ||
| of ranking only those that fit. | ||
|
|
||
| Closing that needs a model-independent size estimate computed once and compared against each | ||
| candidate's ceiling at the same `ADMISSION_TOLERANCE = 2.5` the admission gate uses | ||
| (`src/server/responses/input-admission.ts:32`). Using a stricter threshold at evaluation time | ||
| would refuse candidates that admission would have accepted — an outage in the name of a fix. | ||
|
|
||
| This is an optimization of an already-correct chain, not the reported defect. #1524's | ||
| acceptance behavior ("reject candidates that cannot accept the request before retrying") holds | ||
| today for both context and modality. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| # 130 — #1795: undeclared tool calls from a routed provider | ||
|
|
||
| ## Verified state (at `812e7c40b`) | ||
|
|
||
| The guard is real and deliberate. Both bridge paths refuse an undeclared tool name: | ||
| streaming at `src/bridge.ts:1006` and non-streaming at `:1715`, each emitting a 502 | ||
| `upstream_error`. Pinned by `tests/bridge.test.ts:345` and | ||
| `tests/responses-stream-tool-events.test.ts:30`. | ||
|
|
||
| So the reported behavior is not a defect in the sense of "code doing something nobody | ||
| intended". It is the designed fail-closed contract, and the reporter is asking for that | ||
| contract to be relaxed. | ||
|
|
||
| ## Why this is NEEDS_HUMAN rather than a fix | ||
|
|
||
| The request — "tolerate undeclared tool calls, drop them with a warning" — changes a | ||
| safety boundary, and the failure modes on the other side are not obviously smaller than | ||
| the one being reported. | ||
|
|
||
| An `exec` call the client never declared is, by construction, a request the client has no | ||
| handler for. Dropping it silently means the model believes it ran a command and receives | ||
| either nothing or a fabricated absence, and the turn continues on that false premise. For | ||
| `exec` specifically the current 502 is the honest outcome: the turn genuinely cannot be | ||
| completed as the model intended. | ||
|
|
||
| There is also a real question of WHERE the tolerance belongs. Three candidate answers, | ||
| with materially different blast radii: | ||
|
|
||
| 1. **Global tolerance.** Every routed provider may emit any tool name and have it dropped. | ||
| Largest blast radius; removes the guard for cases it was written for. | ||
| 2. **Per-provider opt-in.** A provider config flag marks a known-noisy upstream. Contained, | ||
| but requires the operator to know which providers need it. | ||
| 3. **Subagent-scope only.** The reporter's actual case is a shadow/subagent call whose | ||
| system prompt describes capabilities the request's tool set does not include. Narrowest, | ||
| and arguably addresses the root cause on the PROMPT side rather than the response side. | ||
|
|
||
| Option 3 suggests the defect may not be in the bridge at all: if a subagent request ships | ||
| a system prompt advertising `exec` while declaring a tool set without it, the request is | ||
| internally inconsistent before the provider ever answers. That is worth checking before | ||
| loosening a validator. | ||
|
|
||
| ## Missing evidence | ||
|
|
||
| No live SenseNova/Kimi reproduction was available in this loop. Without one, two things | ||
| cannot be established: | ||
|
|
||
| - whether the hallucination is provider-specific or a general small-model behavior under | ||
| a capability-describing system prompt; | ||
| - whether the subagent request actually advertises `exec` in its prompt while omitting it | ||
| from `tools` — which would make this a request-construction defect with a different fix. | ||
|
|
||
| ## Disposition | ||
|
|
||
| **NEEDS_HUMAN.** The change is a deliberate loosening of a safety contract whose scope is | ||
| a product decision, and the evidence needed to choose the scope correctly is not available | ||
| without a live reproduction. Recorded here rather than guessed at; #1795 stays OPEN. | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the real local 413 response, the message contains "context window", so
classifyErrorreturnscontext_length_exceededandcomboFailureDecisionstops atsrc/combos/failover.ts:124before reaching theinput_admission_refusedcheck at line 132. The newly added120_outcome.md:100-105records this same ordering defect and leaves PR #1864 open, so this section incorrectly describes both fallback paths as fixed; mark the first change incomplete until the admission check is ordered before the generic stop list.Useful? React with 👍 / 👎.