Skip to content

feat(models): enrich provider model list from models.dev - #40

Merged
enowdev merged 2 commits into
enowdev:mainfrom
hshinosa:feat/model-metadata-sync
Sep 15, 2026
Merged

enowdev merged 2 commits into
enowdev:mainfrom
hshinosa:feat/model-metadata-sync

Conversation

@hshinosa

@hshinosa hshinosa commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

What

Bundle a snapshot of the models.dev community catalogue (930 models, 19 providers, ~400 KB JSON) and use it to enrich llm.ModelInfo when the provider's own /models endpoint leaves fields blank.

Live provider values always win; the snapshot is metadata-only (context window, pricing, capability flags) — never a source for "what models exist".

The snapshot ships as a JSON asset consumed via //go:embed, not as a generated Go source file full of struct literals. Data as data, not code: review diff stays small, GitHub auto-collapses via linguist-generated, and the runtime cost is one json.Unmarshal at init (~ms for 900 entries).

Regenerate with make sync-models.

Why

Provider /models endpoints rarely carry the metadata a picker needs:

  • Anthropic hard-coded ContextWindow: 200000, MaxOutput: 64000, Vision: true, Tools: true, Reasoning: true for every model. That lied for Claude Sonnet 4.6 (real: 1M) and every future model with different capabilities.
  • Gemini exposed real InputTokenLimit/OutputTokenLimit but tagged reasoning with strings.Contains(id, "2.5") — wrong for every non-Gemini id a proxy happens to re-serve.
  • Most providers report zero pricing.

The dashboard's Models page, the composer gauge, and the TUI's picker therefore either showed wrong numbers or none at all. Users who picked Claude Sonnet 4.6 saw a 200k gauge and Antares would trigger compaction at ~160k when the real window was 800k unused.

Cascade

Context window and enrichment walk the same order:

  1. Per-provider user override in cfg.Providers[…].ModelMeta (kept from upstream).
  2. Strict providers.MetaByProvider — exact provider/id, with slug variant fallback (Anthropic ships claude-opus-4.7 while models.dev catalogues claude-opus-4-7 — same model, different spelling).
  3. Proxy fallback via MetaByAnyProvideronly for openai-compatible and custom providers (EnxAPI, LiteLLM, self-hosted endpoints) that re-serve official ids. First-party providers that genuinely do not carry a model never inherit some other provider's entry.
  4. cfg.Model.ContextWindow global fallback.
  5. 128000 safety net.

The earlier cascade iterated every provider in the config and returned the first hit, which caused MetaByProvider's loose fallback to attribute the wrong entry to unrelated providers — e.g. reading Anthropic's row for a Kimi request. The new candidateProviders walks only providers that actually claim the id in their Models list or ModelMeta keys.

Adapter cleanup

  • internal/llm/anthropic.go: drop the hard-coded metadata block; emit id+name only.
  • internal/llm/gemini.go: drop the reasoning heuristic. Keep the real token limits the API returns.

Composer gauge

  • ChatPage refetches /context-window whenever activeModel changes, passing the qualified provider/model id so the server honours the caller's provider hint.
  • ModelPicker emits an antares:model-changed event on successful /model/set; StatusPill listens so the "Connected · " label refreshes at once instead of waiting up to 10 s for the next poll.

TUI

  • New providers.FetchModelInfos preserves the full ModelInfo from the live probe. FetchModels still returns []string for cmd/antares/main.go's model-listing subcommand.
  • Model picker's right column now shows 1000K ctx · anthropic when the cascade knows the window, falling back to the provider name.
  • Header adds 145K/1000K (14%) alongside the ↑↓ counters; a Context section joins the Tokens section in the sidebar. /status prints the same ratio. Both fall back to the metadata cascade for the active model so a fresh session shows the model's budget before the first turn fires the live usage event.
  • The agent-side contextWindowFor already picks up the new cascade, so compaction now triggers at the model's real window (Claude Sonnet 4.6 compacts at 800k instead of 160k) even for TUI-only users.

Files

Handwritten changes total ~1.2K lines. The bundled snapshot (models_generated.json, ~400 KB) is linguist-generated so GitHub collapses it in reviews.

File Purpose
internal/providers/metadata.go (+264) ModelMeta, cascade helpers, slug variants
internal/providers/metadata_test.go (+157) 11 subtests covering cascade layers
internal/providers/generated_models.go (+59) Tiny //go:embed loader — 60 lines, not 12k
internal/providers/models_generated.json (+15,399) Bundled snapshot, collapsed in review
scripts/sync-models-dev.go (+349) Regenerator
internal/providers/catalog.go, models.go (+44/-8) ContextWindow() cascade, FetchModelInfos helper
internal/agent/compact.go (+77/-8) contextWindowFor uses new cascade
internal/llm/anthropic.go, gemini.go (+24/-5) Drop hard-coded metadata + heuristic
internal/server/handlers_config.go, handlers_providers.go (+215/-14) User-override plumbing + enrichment endpoint
internal/tui/* (+117/-13) Context header, sidebar section, picker badge, /status line
web/src/pages/ChatPage.tsx, components/chat/ModelPicker.tsx, components/layout/StatusPill.tsx (+25/-5) Composer gauge refetch + model-changed event
Makefile, .gitattributes (+7/-0) sync-models target + linguist tag

Not breaking

  • Existing configs untouched; ModelMeta struct gains json: tags (additive) with no runtime behaviour change.
  • Live provider values still win when non-zero — snapshot only fills blanks.
  • The 3-entry hand-curated contextWindows map stays as an escape hatch.
  • No new runtime dependencies. Sync script uses stdlib net/http + encoding/json.
  • FetchModels API unchanged; FetchModelInfos is additive.

Validation

  • GOTOOLCHAIN=go1.26.3 go test ./internal/providers/... ./internal/agent/... ./internal/server/... ./internal/config/... ./internal/llm/... ./internal/tui/... — all pass, 11 new subtests cover cascade layers.
  • GOTOOLCHAIN=go1.26.3 go vet ./... clean.
  • bun run typecheck in web/ clean.
  • Smoke: bundled snapshot resolves Claude Sonnet 4.6 → 1M (was 200k), GPT-5.4 → 1.05M, Kimi K2.5 via enx proxy → 262k (from opencode entry), unknown model → 128k safety net (never fabricates).
  • Live TUI smoke: go run ./cmd/antares tui shows the sidebar Context section and header ratio for the active model on first launch.

@hshinosa
hshinosa force-pushed the feat/model-metadata-sync branch from d1a1f0c to 0f307dd Compare September 15, 2026 09:50
Provider /models endpoints rarely return the metadata a picker needs —
Anthropic hard-coded 200000 for every model (lied for Claude Sonnet 4.6,
whose real window is 1M), Gemini returned no capability flags, and most
providers report zero pricing. The dashboard's Models page, the composer
gauge, and the TUI's picker therefore either showed wrong numbers or
none at all.

This ships a hybrid catalogue:

  1. Bundled snapshot (compact JSON via //go:embed) — offline-first,
     answers every lookup immediately, works on corporate networks that
     block outbound HTTPS.
  2. Background refresh — one hour after boot and hourly thereafter, a
     goroutine pulls models.dev/api.json, hot-swaps the in-memory
     table via atomic.Pointer, and persists the normalised bytes to
     $XDG_CACHE_HOME/antares/models.json so warm boots skip the wait.
  3. Env kill-switch — ANTARES_DISABLE_MODELS_FETCH=1 for airgapped
     hosts, ANTARES_MODELS_URL for a mirror, ANTARES_MODELS_CACHE for
     tests.

Live provider values still win when the enrichment cascade runs; the
snapshot is metadata-only, never a source of "what models exist".

The snapshot ships as a JSON asset consumed via //go:embed rather than
as a generated Go source file full of struct literals. Data as data,
not code: the bundled asset is a single compact line (~285 KB) that
GitHub collapses in review, git history stays lean, and the runtime
cost is one json.Unmarshal at init.

Cascade for context window (unchanged from the earlier iteration):

  1. Per-provider user override in cfg.Providers[…].ModelMeta (kept
     from upstream; the Providers page still writes here).
  2. Strict providers.MetaByProvider — exact provider/id, with slug
     variant fallback (Anthropic ships claude-opus-4.7 while models.dev
     catalogues claude-opus-4-7 — same model, different spelling).
  3. Proxy fallback via MetaByAnyProvider — only for openai-compatible
     and custom providers (EnxAPI, LiteLLM, self-hosted endpoints)
     that re-serve official ids. First-party providers that genuinely
     do not carry a model never inherit some other provider's entry.
  4. cfg.Model.ContextWindow global fallback.
  5. 128000 safety net.

The earlier cascade iterated every provider in the config and returned
the first hit, which caused MetaByProvider's loose fallback to attribute
the wrong entry to unrelated providers — e.g. reading Anthropic's row
for a Kimi request. The new candidateProviders walks only providers
that actually claim the id in their Models list or ModelMeta keys.

Adapter cleanup:
- internal/llm/anthropic.go: drop the hard-coded 200000/64000/true/true
  block; emit id+name only and let the server layer enrich.
- internal/llm/gemini.go: drop `strings.Contains(id, "2.5")` reasoning
  heuristic (was wrong for every non-Gemini id a proxy re-serves).
  Keep the real InputTokenLimit/OutputTokenLimit the API returns.

Composer gauge:
- ChatPage refetches /context-window whenever activeModel changes,
  passing the qualified provider/model id so the server can honour the
  caller's provider hint.
- ModelPicker emits an `antares:model-changed` event on successful
  /model/set; StatusPill listens so the "Connected · <model>" label
  refreshes at once instead of waiting up to 10 s for the next poll.

TUI:
- New providers.FetchModelInfos preserves the full ModelInfo from the
  live probe (FetchModels still returns []string for the CLI).
- Model picker's right column now shows "1000K ctx · anthropic" when
  the cascade knows the window, falling back to the provider name.
- Header adds "145K/1000K (14%)" alongside the ↑↓ counters; a Context
  section joins the Tokens section in the sidebar. /status prints the
  same ratio. Both fall back to the metadata cascade so a fresh
  session shows the model's budget before the first turn.
- The agent-side contextWindowFor picks up the new cascade, so
  compaction now triggers at the model's real window (Claude Sonnet 4.6
  compacts at 800k instead of 160k) even for TUI-only users.

Regeneration: `make sync-models` fetches models.dev/api.json and writes
internal/providers/models_generated.json (marked linguist-generated).
The 3-entry hand-curated contextWindows map stays as an escape hatch
for cases where upstream regresses.
@hshinosa
hshinosa force-pushed the feat/model-metadata-sync branch from 0f307dd to e42dc5a Compare September 15, 2026 10:11
@enowdev
enowdev merged commit 4a2399f into enowdev:main Sep 15, 2026
3 checks passed
@enowdev

enowdev commented Sep 15, 2026

Copy link
Copy Markdown
Owner

Fixed and merged with corrective commit 7b3b5d8. Review found four reproducible issues: runtime refresh omitted seven providers present in the bundled snapshot; empty refresh data erased the catalogue; Attachment incorrectly enabled Vision for non-image models; and equal-ranked proxy candidates produced different context windows across repeated lookups. The fix shares the provider allowlist between runtime and generator, retains last-known-good data on empty refresh/cache input, derives Vision only from image capability, and applies deterministic provider ranking and tie-breaking. Added regression coverage for these cases. Local provider/server/agent/LLM/TUI tests, go vet, integrated frontend typecheck/tests/build, and GitHub go/web/smoke checks passed. Closed by merging the corrected PR, not by discarding it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants