Repository navigation
router: answer 503 with retry hints while an instance loads - #66
Merged
Merged
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Active routing issues can return non-retryable 404s for valid models during backend failures, unapplied profile edits, and startup races.
3 open issues
What changed in this PR
This PR improves retry behavior in llm-helper’s router during model loading and backend outages, partially addressing #5 without changing Coder’s retry UI.
Changes:
- Returns OpenAI-style 503 errors with retry headers for loading models and unreachable instance or pool backends.
- Passes configured model IDs into the router to distinguish loading models from unknown ones.
- Adds regression tests and documents the new responses.
| File | Description |
|---|---|
internal/router/poolrouter.go |
Returns retryable JSON errors for missing or unreachable backends. |
internal/router/poolrouter_internal_test.go |
Tests the pool backend-down response. |
internal/router/models.go |
Tracks loading models and adds shared 503 responses with retry hints. |
internal/router/models_internal_test.go |
Tests loading, renamed model IDs, and backend failures. |
internal/cli/router.go |
Supplies configured model IDs alongside instance URLs. |
internal/cli/router_internal_test.go |
Updates router construction for backend metadata. |
docs/multi-instance.md |
Documents loading and unknown-model responses. |
CHANGELOG.md |
Records the user-visible retry behavior. |
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Refs #5 (partial).
What changed
503,Retry-After: 10andRetry-After-Ms: 10000, and an OpenAI-style body:{"error":{"code":503,"message":"<model> is still loading; try again shortly","type":"unavailable_error"}}. This is the same shape llama.cpp uses while it loads. Before this change the router answered404 model_not_found./v1/modelsdoesn't answer yet and no other instance serves that model, the model is treated as starting.404 model_not_found.503plus retry hints, instead of502. So do pool requests when there are no backends or a backend fails; those used to get a plain-text502.docs/multi-instance.mdare updated.Why 503 plus
Retry-After: the OpenAI SDKs retry 5xx and honourretry-after-msandRetry-After. Coder's chatd does the same (coderd/x/chatd/chaterror). Chatd treats "model not found" as a config error and does not retry it.Verified against a running Coder
The test Coder server was built from source (
coder/coder0fbe705) and ran locally with embedded Postgres. It had an openai-compat provider pointing atllm-helper router serve. The model was registered and enabled withcoder-sync. The backend was a fake OpenAI server, started about 36 s after the chat was sent.running. Coder retried at about 0.3 s, 10.5 s, 20.7 s, 31.0 s and 41.2 s. Each retry wasstatus_code: 503,kind: timeout, withdelay_ms: 10000from our hint (the fifth was 16000, from Coder's own backoff). Once the backend was up, the reply streamed in and the chat endedwaitingwith no error.mainerrorafter 0.2 s:status_code: 404,retryable: false, detailno running instance serves the model "fake/loader".Limitations
go test -race ./...has one failure,internal/detect TestProbeReportsThisMachine(BudgetBytes = 0 on a 16 GB machine). It is unrelated to this change. It depends on the hardware of the box I ran the tests on and fails onmainthere too. Every other package passes, along withgo vet,gofmt, prettier andtools/recipediff.