feat(models): enrich provider model list from models.dev - #40
Conversation
d1a1f0c to
0f307dd
Compare
Provider /models endpoints rarely return the metadata a picker needs —
Anthropic hard-coded 200000 for every model (lied for Claude Sonnet 4.6,
whose real window is 1M), Gemini returned no capability flags, and most
providers report zero pricing. The dashboard's Models page, the composer
gauge, and the TUI's picker therefore either showed wrong numbers or
none at all.
This ships a hybrid catalogue:
1. Bundled snapshot (compact JSON via //go:embed) — offline-first,
answers every lookup immediately, works on corporate networks that
block outbound HTTPS.
2. Background refresh — one hour after boot and hourly thereafter, a
goroutine pulls models.dev/api.json, hot-swaps the in-memory
table via atomic.Pointer, and persists the normalised bytes to
$XDG_CACHE_HOME/antares/models.json so warm boots skip the wait.
3. Env kill-switch — ANTARES_DISABLE_MODELS_FETCH=1 for airgapped
hosts, ANTARES_MODELS_URL for a mirror, ANTARES_MODELS_CACHE for
tests.
Live provider values still win when the enrichment cascade runs; the
snapshot is metadata-only, never a source of "what models exist".
The snapshot ships as a JSON asset consumed via //go:embed rather than
as a generated Go source file full of struct literals. Data as data,
not code: the bundled asset is a single compact line (~285 KB) that
GitHub collapses in review, git history stays lean, and the runtime
cost is one json.Unmarshal at init.
Cascade for context window (unchanged from the earlier iteration):
1. Per-provider user override in cfg.Providers[…].ModelMeta (kept
from upstream; the Providers page still writes here).
2. Strict providers.MetaByProvider — exact provider/id, with slug
variant fallback (Anthropic ships claude-opus-4.7 while models.dev
catalogues claude-opus-4-7 — same model, different spelling).
3. Proxy fallback via MetaByAnyProvider — only for openai-compatible
and custom providers (EnxAPI, LiteLLM, self-hosted endpoints)
that re-serve official ids. First-party providers that genuinely
do not carry a model never inherit some other provider's entry.
4. cfg.Model.ContextWindow global fallback.
5. 128000 safety net.
The earlier cascade iterated every provider in the config and returned
the first hit, which caused MetaByProvider's loose fallback to attribute
the wrong entry to unrelated providers — e.g. reading Anthropic's row
for a Kimi request. The new candidateProviders walks only providers
that actually claim the id in their Models list or ModelMeta keys.
Adapter cleanup:
- internal/llm/anthropic.go: drop the hard-coded 200000/64000/true/true
block; emit id+name only and let the server layer enrich.
- internal/llm/gemini.go: drop `strings.Contains(id, "2.5")` reasoning
heuristic (was wrong for every non-Gemini id a proxy re-serves).
Keep the real InputTokenLimit/OutputTokenLimit the API returns.
Composer gauge:
- ChatPage refetches /context-window whenever activeModel changes,
passing the qualified provider/model id so the server can honour the
caller's provider hint.
- ModelPicker emits an `antares:model-changed` event on successful
/model/set; StatusPill listens so the "Connected · <model>" label
refreshes at once instead of waiting up to 10 s for the next poll.
TUI:
- New providers.FetchModelInfos preserves the full ModelInfo from the
live probe (FetchModels still returns []string for the CLI).
- Model picker's right column now shows "1000K ctx · anthropic" when
the cascade knows the window, falling back to the provider name.
- Header adds "145K/1000K (14%)" alongside the ↑↓ counters; a Context
section joins the Tokens section in the sidebar. /status prints the
same ratio. Both fall back to the metadata cascade so a fresh
session shows the model's budget before the first turn.
- The agent-side contextWindowFor picks up the new cascade, so
compaction now triggers at the model's real window (Claude Sonnet 4.6
compacts at 800k instead of 160k) even for TUI-only users.
Regeneration: `make sync-models` fetches models.dev/api.json and writes
internal/providers/models_generated.json (marked linguist-generated).
The 3-entry hand-curated contextWindows map stays as an escape hatch
for cases where upstream regresses.
0f307dd to
e42dc5a
Compare
|
Fixed and merged with corrective commit 7b3b5d8. Review found four reproducible issues: runtime refresh omitted seven providers present in the bundled snapshot; empty refresh data erased the catalogue; Attachment incorrectly enabled Vision for non-image models; and equal-ranked proxy candidates produced different context windows across repeated lookups. The fix shares the provider allowlist between runtime and generator, retains last-known-good data on empty refresh/cache input, derives Vision only from image capability, and applies deterministic provider ranking and tie-breaking. Added regression coverage for these cases. Local provider/server/agent/LLM/TUI tests, go vet, integrated frontend typecheck/tests/build, and GitHub go/web/smoke checks passed. Closed by merging the corrected PR, not by discarding it. |
What
Bundle a snapshot of the models.dev community catalogue (930 models, 19 providers, ~400 KB JSON) and use it to enrich
llm.ModelInfowhen the provider's own/modelsendpoint leaves fields blank.Live provider values always win; the snapshot is metadata-only (context window, pricing, capability flags) — never a source for "what models exist".
The snapshot ships as a JSON asset consumed via
//go:embed, not as a generated Go source file full of struct literals. Data as data, not code: review diff stays small, GitHub auto-collapses vialinguist-generated, and the runtime cost is onejson.Unmarshalat init (~ms for 900 entries).Regenerate with
make sync-models.Why
Provider
/modelsendpoints rarely carry the metadata a picker needs:ContextWindow: 200000, MaxOutput: 64000, Vision: true, Tools: true, Reasoning: truefor every model. That lied for Claude Sonnet 4.6 (real: 1M) and every future model with different capabilities.InputTokenLimit/OutputTokenLimitbut tagged reasoning withstrings.Contains(id, "2.5")— wrong for every non-Gemini id a proxy happens to re-serve.The dashboard's Models page, the composer gauge, and the TUI's picker therefore either showed wrong numbers or none at all. Users who picked Claude Sonnet 4.6 saw a 200k gauge and Antares would trigger compaction at ~160k when the real window was 800k unused.
Cascade
Context window and enrichment walk the same order:
cfg.Providers[…].ModelMeta(kept from upstream).providers.MetaByProvider— exactprovider/id, with slug variant fallback (Anthropic shipsclaude-opus-4.7while models.dev cataloguesclaude-opus-4-7— same model, different spelling).MetaByAnyProvider— only foropenai-compatibleandcustomproviders (EnxAPI, LiteLLM, self-hosted endpoints) that re-serve official ids. First-party providers that genuinely do not carry a model never inherit some other provider's entry.cfg.Model.ContextWindowglobal fallback.The earlier cascade iterated every provider in the config and returned the first hit, which caused
MetaByProvider's loose fallback to attribute the wrong entry to unrelated providers — e.g. reading Anthropic's row for a Kimi request. The newcandidateProviderswalks only providers that actually claim the id in theirModelslist orModelMetakeys.Adapter cleanup
internal/llm/anthropic.go: drop the hard-coded metadata block; emit id+name only.internal/llm/gemini.go: drop the reasoning heuristic. Keep the real token limits the API returns.Composer gauge
ChatPagerefetches/context-windowwheneveractiveModelchanges, passing the qualifiedprovider/modelid so the server honours the caller's provider hint.ModelPickeremits anantares:model-changedevent on successful/model/set;StatusPilllistens so the "Connected · " label refreshes at once instead of waiting up to 10 s for the next poll.TUI
providers.FetchModelInfospreserves the fullModelInfofrom the live probe.FetchModelsstill returns[]stringforcmd/antares/main.go's model-listing subcommand.1000K ctx · anthropicwhen the cascade knows the window, falling back to the provider name.145K/1000K (14%)alongside the ↑↓ counters; a Context section joins the Tokens section in the sidebar./statusprints the same ratio. Both fall back to the metadata cascade for the active model so a fresh session shows the model's budget before the first turn fires the live usage event.contextWindowForalready picks up the new cascade, so compaction now triggers at the model's real window (Claude Sonnet 4.6 compacts at 800k instead of 160k) even for TUI-only users.Files
Handwritten changes total ~1.2K lines. The bundled snapshot (
models_generated.json, ~400 KB) islinguist-generatedso GitHub collapses it in reviews.internal/providers/metadata.go(+264)ModelMeta, cascade helpers, slug variantsinternal/providers/metadata_test.go(+157)internal/providers/generated_models.go(+59)//go:embedloader — 60 lines, not 12kinternal/providers/models_generated.json(+15,399)scripts/sync-models-dev.go(+349)internal/providers/catalog.go,models.go(+44/-8)ContextWindow()cascade,FetchModelInfoshelperinternal/agent/compact.go(+77/-8)contextWindowForuses new cascadeinternal/llm/anthropic.go,gemini.go(+24/-5)internal/server/handlers_config.go,handlers_providers.go(+215/-14)internal/tui/*(+117/-13)/statuslineweb/src/pages/ChatPage.tsx,components/chat/ModelPicker.tsx,components/layout/StatusPill.tsx(+25/-5)Makefile,.gitattributes(+7/-0)sync-modelstarget + linguist tagNot breaking
ModelMetastruct gainsjson:tags (additive) with no runtime behaviour change.contextWindowsmap stays as an escape hatch.net/http+encoding/json.FetchModelsAPI unchanged;FetchModelInfosis additive.Validation
GOTOOLCHAIN=go1.26.3 go test ./internal/providers/... ./internal/agent/... ./internal/server/... ./internal/config/... ./internal/llm/... ./internal/tui/...— all pass, 11 new subtests cover cascade layers.GOTOOLCHAIN=go1.26.3 go vet ./...clean.bun run typecheckinweb/clean.enxproxy → 262k (from opencode entry), unknown model → 128k safety net (never fabricates).go run ./cmd/antares tuishows the sidebar Context section and header ratio for the active model on first launch.