Skip to content

router: answer 503 with retry hints while an instance loads - #66

Merged
DevelopmentCats merged 2 commits into
mainfrom
devcats/loading-503
Nov 10, 2026
Merged

DevelopmentCats merged 2 commits into
mainfrom
devcats/loading-503

Conversation

@DevelopmentCats

Copy link
Copy Markdown
Collaborator

Refs #5 (partial).

What changed

  • While an instance's model is still loading, the router answers requests for it with 503, Retry-After: 10 and Retry-After-Ms: 10000, and an OpenAI-style body: {"error":{"code":503,"message":"<model> is still loading; try again shortly","type":"unavailable_error"}}. This is the same shape llama.cpp uses while it loads. Before this change the router answered 404 model_not_found.
  • The router knows a model is starting because each instance's configured model comes from its profile. If an instance's /v1/models doesn't answer yet and no other instance serves that model, the model is treated as starting.
  • A model no instance is configured for still gets 404 model_not_found.
  • An instance that doesn't answer a proxied request (crashed, restarting) gets the same 503 plus retry hints, instead of 502. So do pool requests when there are no backends or a backend fails; those used to get a plain-text 502.
  • Tests cover a starting instance, a profile whose model id differs from the served id, the backend-down body, and the pool backend-down case. The CHANGELOG and docs/multi-instance.md are updated.

Why 503 plus Retry-After: the OpenAI SDKs retry 5xx and honour retry-after-ms and Retry-After. Coder's chatd does the same (coderd/x/chatd/chaterror). Chatd treats "model not found" as a config error and does not retry it.

Verified against a running Coder

The test Coder server was built from source (coder/coder 0fbe705) and ran locally with embedded Postgres. It had an openai-compat provider pointing at llm-helper router serve. The model was registered and enabled with coder-sync. The backend was a fake OpenAI server, started about 36 s after the chat was sent.

Build Result
This branch Chat stayed running. Coder retried at about 0.3 s, 10.5 s, 20.7 s, 31.0 s and 41.2 s. Each retry was status_code: 503, kind: timeout, with delay_ms: 10000 from our hint (the fifth was 16000, from Coder's own backoff). Once the backend was up, the reply streamed in and the chat ended waiting with no error.
main Chat went to error after 0.2 s: status_code: 404, retryable: false, detail no running instance serves the model "fake/loader".

Limitations

  • Coder's retry event and UI show a normalized message ("OpenAI Compatible is temporarily unavailable."), not our "still loading" text. Showing the loading detail during retries would need a change in Coder.
  • go test -race ./... has one failure, internal/detect TestProbeReportsThisMachine (BudgetBytes = 0 on a 16 GB machine). It is unrelated to this change. It depends on the hardware of the box I ran the tests on and fails on main there too. Every other package passes, along with go vet, gofmt, prettier and tools/recipediff.

Copilot AI balanced review requested due to automatic review settings November 10, 2026 21:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Active routing issues can return non-retryable 404s for valid models during backend failures, unapplied profile edits, and startup races.

3 open issues
What changed in this PR

This PR improves retry behavior in llm-helper’s router during model loading and backend outages, partially addressing #5 without changing Coder’s retry UI.

Changes:

  • Returns OpenAI-style 503 errors with retry headers for loading models and unreachable instance or pool backends.
  • Passes configured model IDs into the router to distinguish loading models from unknown ones.
  • Adds regression tests and documents the new responses.
File Description
internal/​router/​poolrouter.go Returns retryable JSON errors for missing or unreachable backends.
internal/​router/​poolrouter_internal_test.go Tests the pool backend-down response.
internal/​router/​models.go Tracks loading models and adds shared 503 responses with retry hints.
internal/​router/​models_internal_test.go Tests loading, renamed model IDs, and backend failures.
internal/​cli/​router.go Supplies configured model IDs alongside instance URLs.
internal/​cli/​router_internal_test.go Updates router construction for backend metadata.
docs/​multi-instance.md Documents loading and unknown-model responses.
CHANGELOG.md Records the user-visible retry behavior.

Review effort: Balanced

Comment thread internal/cli/router.go Resolved
Comment thread internal/router/models.go Resolved
Comment thread internal/router/models.go Resolved
@DevelopmentCats
DevelopmentCats merged commit 8c283e2 into main Nov 10, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants