Repository navigation
fix(server): refuse what a chat backend cannot serve in the gateway's words - #823
Merged
Merged
Conversation
… words sie-server's /v1/chat/completions, which its /v1/messages serves through, now refuses before the engine runs, with the gateway's status, envelope and wording: tools on a model that does not declare tool calling, and an image the serving path cannot read. Only the SGLang child on CUDA reads images; the MLX child reads text only and adapter-owned chat carries no media. Tool calling is checked first, as on the gateway. The Messages route turns these into "tools: ..." and the path of the first image block, the refusal Claude Code recovers from. The fake model declares the tool calling its scripted chat does. The MLX backend enforces the declared window, as SGLang's --context-length does: the prompt is counted as mlx_lm.server renders it, with the repo's tokenizer and chat template, and a prompt that does not fit with its max_tokens is refused with the worker's wording. The fake engine's window refusal carries its counts in the same wording, so both reach Messages clients as "prompt is too long: T tokens > C maximum". New shared vectors in conformance/anthropic_messages/capability_vectors.json pin both chat routes' refusals and the Messages answers. The gateway's and the server's tests replay each vector through both routes.
…X run The Messages conformance check against Qwen/Qwen3.5-4B:long-context on Apple Silicon now records the 64K-plus prompt as refused with Claude Code's compaction wording, and the limitations name the backends whose window refusal still carries no counts.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Priority: ⬇️ Low Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage |  |
Docstring coverage is 19.61% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 51 functions across 9 files. (5 skipped: … | Write docstrings for the functions missing them to satisfy the coverage threshold. |
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title accurately summarizes the main change: the server now refuses unsupported chat capabilities using gateway-aligned errors and wording. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 19.61% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 51 functions across 9 files. (5 skipped: 5 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
- Fix all pre-merge checks with AI
✨ Finishing Touches: 💡 1
-
📝 Generate docstrings: 💡
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three gaps on
sie-server'sPOST /v1/messages(#815), all in the chat route it serves through.Tool and image refusals in the gateway's words.
/v1/chat/completionsonsie-servernow refuses, before the engine runs and with the gateway's status, envelope and wording:toolson a model whose config does not declaretasks.generate.capabilities.tools:400 {"message": "Model '<id>' does not support tool calling", "type": "invalid_request_error", "param": "tools", "code": "unsupported_field"};image_urlorinput_imagepart the serving path cannot read:400 {"message": "Model '<id>' does not support image input", ..., "param": "messages", "code": "unsupported_field"}.Tool calling is checked first, as on the gateway.
/v1/messagesturns these intotools: Model '<id>' does not support tool callingand<path of the first image block>: Model '<id>' does not support image input, the refusal Claude Code recovers from by replacing the image with a note. Before, a text-only model answeredmessages: remote chat does not support media messageson the fake or a remote upstream, and the MLX and SGLang children were left to refuse it themselves.The image rule is the gateway's (
inputs.imageand a generate task), narrowed by what this server can deliver: only the SGLang child on CUDA reads images. The MLX child reads text only, and adapter-owned chat (the fake, remote upstreams) carries no media, so an image there is refused the same way even when the model declares image input. Video keeps its existing device check.sie-fakenow declarescapabilities.tools: true, since its scripted chat makes tool calls.Prompt-too-long with token counts on the fake. The fake engine refuses a prompt that leaves no room in its window with the worker's wording,
prompt_tokens (601) exceeds context_length (512) for model 'sie-fake'. So/v1/messagesanswersprompt is too long: 601 tokens > 512 maximum, the form Claude Code's compaction matches, instead ofprompt is too longalone.The MLX backend enforces the declared window.
mlx_lm.serverserves any prompt length, so a 74,847-token prompt ran on the 32,768-tokenQwen/Qwen3.5-4B:long-contextprofile.MLXGenerationAdapter.check_chat_windowcounts the prompt the child renders: the repo's tokenizer and chat template, with text parts joined and tool-call arguments parsed asmlx_lm.serverdoes. A prompt that does not fit with itsmax_tokensis refused, as SGLang's--context-lengthrefuses it, with the worker's wordingprompt_tokens (P) + max_new_tokens (N) = T exceeds context_length (C) for model '<id>'. A prompt the tokenizer or template cannot count is left to the child.Shared vectors
conformance/anthropic_messages/capability_vectors.jsonholds 7 vectors. Each has a Messages request to a model with the declared capabilities, the chat route's answer to its translation, and the Messages answer. The gateway (handlers::messages::tests::capability_vectors_are_refused_by_both_routes_before_dispatch) and the server (test_a_capability_vector_is_refused_by_both_routes_before_the_engine) replay every vector through both routes and check that nothing reached the engine. The gateway's wording was pinned only by its own unit tests before. It passes the vectors unchanged.Validation
mise run lint,mise run typecheck: clean. Gateway:cargo fmt --checkandclippy --all-targets -D warningsclean,handlers::messagestests 24 passed (1 new).packages/sie_server/tests, offline as in CI: 14,180 passed, 605 skipped. The 6 failures are the serving-artifact materialization tests, which fail on macOS file-system semantics (renaming a read-only tree) and do not touch this change.conformance.pyandconformance.py --api messagesagainst a realsie-serveron the fake engine.conformance.py --api messagesagainst a realsie-serveron the fake: 14 pass, 1 fail (parallel_tool_use, one call per turn), 3 recorded, as before. On the same server, an image onsie-fake:chatis refused asmessages.0.content.1.image: Model 'sie-fake:chat' does not support image input, and 600 words onsie-fakeasprompt is too long: 601 tokens > 512 maximum, buffered and streamed.mise run serve -- -b sglang,Qwen/Qwen3.5-4B:long-context:conformance.py --api messages: 13 pass, 0 fail, 5 recorded.prompt_above_64k_tokensis now recorded as refused withprompt is too long: 78943 tokens > 32768 maximum(74,847 prompt tokens and the 4,096-token cap).conformance.py: 9 pass, 0 fail, 3 recorded, withprompt_tokens (74847) + max_new_tokens (4096) = 78943 exceeds context_length (32768).usage.prompt_tokenswas refused atmax_tokens2,499 asprompt_tokens (30270) + max_new_tokens (2499) = 32769 exceeds context_length (32768), and through/v1/messagesasprompt is too long: 32769 tokens > 32768 maximum. The 74,847 is also the count the backend reported when it served that prompt before this change.messages.2.content.0.tool_result.content.1.image: Model 'Qwen/Qwen3.5-4B:long-context' does not support image input.examples/coding-agent-claude-code/README.md.input_imageparts not counted; the image refusal naming another param; the fake refusal without counts; the MLX window not checked; refusing at equality; ignoringmax_tokens; counting without tools; without chat template kwargs; without parsed tool-call arguments; without joined text parts; failing closed on an uncountable prompt; no undeclared-window guard.Not changed
upstream generation request failed, without counts. A remote upstream's refusal readsprompt is too longwithout counts. The example's limitations say so.Anthropic does not support Claude Code with models other than Claude. Nothing here is supported or endorsed by Anthropic.
Summary by CodeRabbit