Ollama Smart Router

Local OpenAI-compatible model routing for Ollama, Hermes Agent, OpenWebUI, and other chat-completions clients.

Ollama Smart Router sits between your client and Ollama, scores the current request, chooses the best configured local model, forwards the request, and can unload old models so low-VRAM systems do not keep several models resident at once.

Status

This project is suitable for local use and public alpha testing. It is not meant to be exposed directly to the public internet. If you bind outside 127.0.0.1, configure an API key and put it behind your own network controls.

Hermes Smart Router is source-available for noncommercial use. Commercial or for-profit use is not permitted without separate permission from the copyright holder.

Features

• OpenAI-compatible POST /v1/chat/completions
• OpenAI-compatible GET /v1/models
• Hermes Agent custom endpoint support
• Weighted model routing with manual override tags
• Recent-conversation routing context for follow-up turns
• Async proxying, request timeouts, and streaming cleanup
• Safe model unload tracking for concurrent requests
• Optional API-key authentication
• Prompt text logging disabled by default
• Live and readiness health checks
• Pytest coverage for routing, validation, proxying, auth, health, and unload behavior

Install

For local development:

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

On Windows PowerShell:

py -3 -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e ".[dev]"

When published as a package, the intended install shape is:

pip install ollama-smart-router

Configure

Copy the example config:

cp config.example.yaml config.yaml

For Ollama running on the same machine:

ollama_base_url: "http://localhost:11434"

For WSL talking to Windows Ollama, use your Windows WSL host IP, for example:

ollama_base_url: "http://172.29.112.1:11434"

Optional API keys:

api_keys:
  - "replace-with-a-long-random-secret"

You can also set keys with environment variables:

export OLLAMA_SMART_ROUTER_API_KEY="replace-with-a-long-random-secret"

or:

export OLLAMA_SMART_ROUTER_API_KEYS="key-one,key-two"

If api_keys is empty and no environment key is set, the router runs unauthenticated. That is acceptable for loopback-only local use, but not for LAN exposure.

Run

Recommended local-only run:

ollama-smart-router --config config.yaml --host 127.0.0.1 --port 8088

If you bind to all interfaces, configure an API key first:

ollama-smart-router --config config.yaml --host 0.0.0.0 --port 8088

The CLI logs a warning when binding to 0.0.0.0 or :: without API keys.

Hermes Agent

Run:

hermes setup model

Use:

API base URL: http://localhost:8088/v1
API mode: Chat Completions
Model name: ollama-smart-router
API key: your configured key, or any placeholder if auth is disabled

Hermes may still display the front-door model name. The router logs show the backend model selected for each request.

Routing

Default routes:

Task type	Default model
General assistant / agent	hermes3:8b
Code, scripts, troubleshooting	qwen2.5-coder:7b
RAG, documents, summaries, reports	gemma4:e2b

Manual tags override scoring:

[code] write a bash script to monitor GPU usage
[rag] summarize this technical document
[general] help me plan my AI workflow

Weighted routing can be tuned in config.yaml:

routing_context_messages: 6
routing_score_threshold: 1

routing:
  code_keywords:
    - python
    - powershell

  code_weights:
    traceback: 3
    "write a script": 4

  rag_keywords:
    - pdf
    - uploaded file

  rag_weights:
    pdf: 4
    "uploaded file": 4

Health Checks

GET /health
GET /health/live
GET /health/ready

/health/live only confirms the router process is alive.

/health/ready checks Ollama reachability and, by default, verifies that configured models appear in Ollama's /api/tags.

By default, health responses avoid exposing local model/config details. For local debugging:

expose_health_details: true

Security Defaults

• Prompt text is not logged unless log_prompt_text: true.
• API-key auth is optional but strongly recommended outside loopback-only use.
• API keys are accepted through Authorization: Bearer <key> or X-API-Key.
• config.yaml is ignored by git.
• Do not expose this directly to the internet.

Development

pip install -e ".[dev]"
python -m pytest -q
python -m compileall src

API

GET  /health
GET  /health/live
GET  /health/ready
GET  /v1/models
GET  /v1/models/{model_id}
POST /v1/chat/completions

License

PolyForm Noncommercial License 1.0.0. See LICENSE and NOTICE.

This is a source-available noncommercial license, not an OSI-approved open source license. Commercial or for-profit use requires separate permission.