Skip to content

fix: make local AI providers model-agnostic#270

Merged
deucebucket merged 2 commits into
mainfrom
fix/openai-compatible-provider
Jul 15, 2026
Merged

fix: make local AI providers model-agnostic#270
deucebucket merged 2 commits into
mainfrom
fix/openai-compatible-provider

Conversation

@deucebucket

@deucebucket deucebucket commented Jul 15, 2026

Copy link
Copy Markdown
Owner

Summary

  • add a generic OpenAI-compatible provider for llama.cpp, LM Studio, vLLM, LocalAI, and similar servers
  • discover live model IDs from the selected endpoint and auto-select only when exactly one model is available
  • remove hardcoded Ollama model fallbacks, filter embedding-only models, and bound structured generation
  • enforce an Ollama array schema for multi-book batches so later items are not silently dropped
  • route text verification, localization, and transcript identification through the configured provider chain
  • pass transcript-only Skaldleita results to the selected AI fallback without retranscribing audio
  • keep Skaldleita first in the audio chain while placing the selected local provider first for text
  • keep stored tokens out of Settings HTML and config.json, and enforce 0600 on secrets.json
  • correct narrow-screen Settings overflow found during browser validation

Root Cause

Library Manager exposed only Gemini, OpenRouter, and Ollama. The Ollama UI and runtime supplied specific fallback model IDs, while model parsing accepted only one response shape. Skaldleita transcript-only results returned before a configured AI fallback could consume them. Native Ollama generation was also unbounded, and JSON object mode allowed a multi-book prompt to stop after ITEM_1.

User Impact

Users can point Library Manager at their own llama.cpp or other OpenAI-compatible server, load that server's actual model list, select any completion model, and use it without replacing Skaldleita. Skaldleita remains the hosted metadata/audio service and can hand its transcript to the user's selected local AI.

Stored API tokens remain server-side: blank Settings fields preserve existing values, explicit removal still works, raw values are not rendered into HTML, and the secrets file is owner-only.

Validation

  • python test-env/test-naming-issues.py (290 passed)
  • python test-env/test-ai-providers.py (11 passed)
  • python test-env/test-ai-provider-integration.py (4 passed)
  • python test-env/test-ui-features.py (22 passed)
  • python test-env/test-audio-tagging.py (5 passed)
  • critical Python compile checks, targeted Ruff, and git diff --check
  • real Bonsai object inference through both native Ollama and OpenAI-compatible paths
  • real raw llama-server discovery and chat-completion inference
  • real mixed Ollama discovery filtering two embedding models while retaining the completion model
  • real two-book call_ai batch returning both ITEM_1 and ITEM_2 through native Ollama
  • real Skaldleita transcript handoff to Bonsai with zero local retranscription calls
  • Playwright desktop/mobile Settings discovery, connection test, save/reload, chain ordering, and no-overflow checks
  • Playwright fresh-install wizard from path selection through saved provider configuration
  • live secret upgrade/resave checks: absent from HTML and config.json, preserved server-side, mode 0600

@deucebucket
deucebucket marked this pull request as ready for review July 15, 2026 11:02
@deucebucket
deucebucket merged commit 893aa8b into main Jul 15, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant