Skip to content

inspect: print the engine's note on hybrid models - #68

Merged
DevelopmentCats merged 2 commits into
mainfrom
devcats/inspect-hybrid-note
Nov 12, 2026
Merged

DevelopmentCats merged 2 commits into
mainfrom
devcats/inspect-hybrid-note

Conversation

@DevelopmentCats

Copy link
Copy Markdown
Collaborator

Part of #65.

On hybrid models (linear-attention or Mamba layers), llama.cpp reuses a cached prompt only up to the start of a user message, so the first turn of a chat re-reads everything before it unless that part matches an earlier chat. inspect now says so.

  • hf.Arch.Hybrid() reads layer_types: a model is hybrid when some layers are not attention.
  • resolve.Detail gets hybrid and hybrid_note. ForProvider fills the note from the serving engine.
  • Engines can carry an optional notes.hybrid sentence. Only catalogue/engines/llamacpp.json sets one, so the wording is data, not Go.
  • Documented in catalogue/engines/README.md and the engine fields in docs/providers.md. CHANGELOG entry added.

The warm-up side of #65 is #67.

Copilot AI balanced review requested due to automatic review settings November 10, 2026 23:54

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the hybrid-model inspection guidance change.

No open issues 🎉

What changed in this PR

Adds engine-specific guidance to inspect for hybrid models, addressing the explanatory part of #65 without changing prompt-cache behavior or adding warm-up logic.

Changes:

  • Detects hybrid models from architecture layer_types and exposes hybrid details during inspection.
  • Adds optional engine notes and displays llama.cpp’s guidance on keeping system prompts and tools identical between chats.
  • Adds detection and provider-note tests, documentation, and a changelog entry.
File Description
internal/​resolve/​resolve.go Carries hybrid detection and the selected engine’s note.
internal/​resolve/​resolve_internal_test.go Tests when provider-specific hybrid notes are populated.
internal/​hf/​kv_internal_test.go Tests hybrid detection, including nested text configuration.
internal/​hf/​arch.go Detects hybrid architectures from layer types.
internal/​engine/​registry.go Exposes hybrid notes through the provider interface.
internal/​engine/​engine.go Adds engine note fields and an accessor.
internal/​cli/​search.go Prints the hybrid note in inspection output.
docs/​providers.md Documents optional engine notes.
CHANGELOG.md Records the new inspection guidance.
catalogue/​engines/​README.md Explains how engine definitions supply hybrid notes.
catalogue/​engines/​llamacpp.json Supplies llama.cpp’s hybrid prompt-cache guidance.

Review effort: Balanced

@DevelopmentCats

Copy link
Copy Markdown
Collaborator Author

Tested inspect against real Hugging Face repos with PROVIDER=llamacpp.

Repo hybrid Note shown
Qwen3.5-35B-A3B, Qwen3.5-0.8B true yes
GLM-5.3-Flash true yes
LFM2-8B-A1B true yes
granite-4.0-h-small true yes
unsloth GGUFs of Qwen3.5-35B-A3B and GLM-5.3-Flash true yes
Qwen3-Next-80B-A3B-Instruct true (since 667bd65) yes
NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 true (since 667bd65) yes
Qwen3-30B-A3B and its GGUF, Qwen3-8B false no

667bd65 adds the configs that name no layer_types: Qwen3-Next sets full_attention_interval and linear_* fields, and Nemotron-H sets hybrid_override_pattern, where M is a Mamba layer.
These are config field names, not model names.
On the default vLLM provider, --json still reports hybrid: true with no note, since vLLM's engine file has none.

@DevelopmentCats
DevelopmentCats merged commit 592def7 into main Nov 12, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants