llama.cpp: multimodal capability and dual-envelope /v1/models metadata are not ingested

[lidge-jun]

Provider

Other (self-hosted llama.cpp server behind the openai-chat adapter)

Endpoint or base URL

http://<host>:<port>/v1 (llama.cpp llama-server, OpenAI-compatible mode)

What does not work

A llama.cpp server truthfully advertises both multimodality and its served context length, but neither signal reaches routing evidence, so the model is treated as image-blind with an unknown context window.

Two independent gaps cause this.

1. The capability token multimodal is not recognized.

modelInputModalities() (src/codex/catalog/provider-fetch.ts:991) accepts capabilities.vision === true and the capability strings vision, image-input, image_input. llama.cpp and Ollama-compatible servers report vision as multimodal, which is in none of those sets, so the row yields undefined.

2. The image and context signals live in different envelopes.

The server returns a dual-shape body — an Ollama-style models[] array alongside an OpenAI-style data[] array. The multimodal token is in models[]; meta.n_ctx is in data[].

extractProviderModelItems() (src/providers/model-discovery.ts:337) deliberately reads only data envelopes or top-level arrays, and its comment says a stray models key on openai-chat responses must not be trusted. That conservatism looks correct in general, but it means the two halves of this server's metadata can never meet: the surviving data[] item has no capability list at all.

Observed response

{ "models": [ { "name": "<model>", "model": "<model>",
                "capabilities": ["completion", "multimodal"],
                "details": { "format": "gguf" } } ],
  "object": "list",
  "data": [ { "id": "<model>", "object": "model", "owned_by": "llamacpp",
              "meta": { "n_ctx": 262144, "n_ctx_train": 262144,
                        "n_vocab": 248320, "n_embd": 5120 } } ] }

Feeding the surviving data[] item to catalogHintsFromModelsApiItem() returns {} — no context, no modalities. Hand-merging the capability list in still returns no inputModalities, because of gap 1:

catalogHintsFromModelsApiItem("<provider>", {
  meta: { n_ctx: 262144 }, capabilities: ["completion", "multimodal"] })
=> { "capabilities": ["completion", "multimodal"] }

Expected behavior

A server that advertises multimodal should produce inputModalities: ["text", "image"], and meta.n_ctx should be usable as a context source — preferring the served n_ctx over the trained n_ctx_train, since routing must not promise a window the running server will refuse.

Version

2.21.0

Additional context

Two constraints any fix must respect:

• input_modalities is a closed enum of text | image | audio. One out-of-enum value makes Codex reject the entire catalog file, taking down plugins, apps, and MCP servers — the existing normalization comment documents this after a real incident with a provider advertising video. Mapping multimodal must therefore emit ["text", "image"], never the raw token.
• Any models[] to data[] join needs identity-safe matching by model id and bounded input, precisely because the current code refuses to trust a stray models key.

Context-source ingestion (meta.n_ctx) is being handled separately as a pure addition to the existing precedence list; this issue tracks the two harder halves: the multimodal mapping and the cross-envelope join.

Related: #1796 (routing discards catalog rows entirely, which masks this end to end).

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[coderabbitai]

🔗 Related PRs

#478 - fix(xai): declare grok image input so combos are not advertised text-only [merged]
#639 - fix(catalog): accept Together top-level /models arrays [merged]
#653 - feat(providers): add Baseten Model APIs preset [merged]
#963 - fix(catalog): preserve provider capabilities on custom models [open]
#1799 - fix(routing): source capability evidence from explicit catalog provenance [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[lidge-jun]

Fixed on dev as 681dd0ecd (PR #1807).

Both halves resolve now. End to end on the verbatim payload from this report:

{"id":"qwen3.8-27b-nvfp4","provider":"llama","owned_by":"llamacpp",
 "contextWindow":262144,"inputModalities":["text","image"],
 "capabilities":["completion","multimodal"]}

multimodal is mapped to the closed text|image enum rather than passed through, so the catalog-rejection hazard noted in the original report is avoided. The models[] to data[] join is identity-safe: membership stays with data[], matching is exact id equality, and an id claimed twice is skipped rather than guessed.

Worth recording what review caught, because the first implementation was unsafe: enriching before admission filtering let a models[] sibling supply the exact field a provider filter required — reproduced against the real Chutes policy, where a row lacking supported_features: ["tools"] was admitted once a sibling provided it. Enrichment now runs after admission and copies only capabilities, modalities, input_modalities, so it can change what is known about a model but never which models are published.

Also fixed by #1799 earlier in this chain: meta.n_ctx is read as a context source, preferring the served value over n_ctx_train.