Skip to content

fix(doctor): validate providers with real test calls, not /models (#48) - #62

Merged
savioruz merged 1 commit into
mainfrom
fix/doctor-proxy-false-positives
Aug 25, 2026
Merged

savioruz merged 1 commit into
mainfrom
fix/doctor-proxy-false-positives

Conversation

@savioruz

Copy link
Copy Markdown
Owner

The doctor diagnostic warned/failed when a provider's configured model name wasn't advertised in GET /models. Proxy gateways (e.g. 9Router) that don't expose /models, or expose different model naming, produced false positives even though the provider was fully functional.

Replace the /models-list check with the same real test-call logic the setup/dashboard flows already use:

  • HttpEmbedderProvider::probe() issues a real POST /embeddings and reports the returned dimension.
  • HttpLlmProvider::probe() issues a real POST /chat/completions.
  • doctor now reports success when the real call succeeds, and still fails on 401/5xx/unreachable/parse errors.

Regression tests use a mock with no /models endpoint at all (404) but working /embeddings and /chat/completions, covering both raw (embedder) and llm modes, plus a 500 misconfigured-provider case.

The doctor diagnostic warned/failed when a provider's configured model
name wasn't advertised in GET /models. Proxy gateways (e.g. 9Router)
that don't expose /models, or expose different model naming, produced
false positives even though the provider was fully functional.

Replace the /models-list check with the same real test-call logic the
setup/dashboard flows already use:

- HttpEmbedderProvider::probe() issues a real POST /embeddings and
  reports the returned dimension.
- HttpLlmProvider::probe() issues a real POST /chat/completions.
- doctor now reports success when the real call succeeds, and still
  fails on 401/5xx/unreachable/parse errors.

Regression tests use a mock with no /models endpoint at all (404) but
working /embeddings and /chat/completions, covering both raw (embedder)
and llm modes, plus a 500 misconfigured-provider case.
@savioruz
savioruz marked this pull request as ready for review August 25, 2026 04:33
@savioruz savioruz added bug Something isn't working core memayu-core domain logic api HTTP API transport/handlers infra CI/release/distribution self-hosted Relates to self-hosted / local-first usage priority-medium Medium priority — important but not blocking labels Aug 25, 2026
@savioruz savioruz added this to the v0.1.0 milestone Aug 25, 2026
@savioruz

Copy link
Copy Markdown
Owner Author

Review of PR #62 at f86f287033163a673d413a522a9a424432ccfa8a (base main = b1841db, includes #41–#61 already merged), limited to origin/main...HEAD net = 4 files (doctor.rs, tests/doctor.rs, embedder.rs, llm.rs, +190/-94). Implements issue #48 only (doctor proxy false-positives). Issues #52 (MCP camelCase) and #53 (batch) are NOT part of this PR — they live in already-merged PRs on main.

🔴 Blocking

None. CI green: cargo build (49s), clippy, fmt, test (54s), audit (2m38s).

🟡 Warning #1 — LLM probe() consumes one completion per doctor run

check_provider now issues a real, minimal completion (LLM) and a real embedding call (embedder) instead of GET /models. In extract_mode = raw the LLM probe is correctly skipped, but every doctor run in llm mode burns one upstream token request. Expected for a diagnostics command (infrequent), but worth a doc note that doctor is not free — not a bug.

🟡 Warning #2 — check_models / ModelsCheck are now unused by doctor

PR #62 replaces the GET /models probe with a real test call, but memayu-llm-client still exports check_models, ModelsCheck, and per-provider check_models() methods (still pub). Nothing inside the workspace calls them now (bin/memayu doctor no longer references them). They are not dead-code errors (they're pub), so CI is clean, but they are unused public API that can silently rot. Recommend a follow-up to gate/remove them (or #[deprecated]) once external consumers are checked.

Validation

#48 — doctor probe via real test calls (net new in this PR)

  • crates/memayu-llm-client/src/llm.rs: adds HttpLlmProvider::probe() — a single minimal System/User completion ("reply with exactly the JSON …") whose Ok(()) proves the endpoint is reachable, the key is accepted, and the model answers.
  • crates/memayu-llm-client/src/embedder.rs: adds HttpEmbedderProvider::probe() -> Result<usize, String> — a real "dimension probe" embedding, returning the vector dim on success.
  • bin/memayu/src/doctor.rs: check_provider(name, cfg, ProviderKind) replaces the old check_models(base_url, api_key, model) path; prints "reachable, key accepted, model … answered a test completion" / "… returned a {dim}-dim embedding". Local backend is unaffected (cache-dir only, no network). Result: 401/5xx/timeout → fail; success → pass. Exit 0/1 preserved.
  • Targeted test: doctor_proxy_without_models_endpoint_is_healthy in bin/memayu/tests/doctor.rs — the HTTP mock has no /models endpoint at all (the 9Router/forward-proxy scenario), and doctor still exits 0 (the real test call succeeds). Plus doctor_provider_unauthorized_exits_one + doctor_server_error_exits_one for the failure side.
  • Doc comment + .env.example note updated ("providers are probed with a real test call rather than GET /models").

Transitive base already reviewed (#59–#61 live in main)

Migration / breaking

Checklist

Note: PR title has a redundant "PR:" prefix. PR description was the auto-generated commit body — edited pre-review to clarify this PR is #48-only (not #52/#53, which are already merged on main). Comment updated in place per ZeroClaw convention; no new comment.

@savioruz
savioruz merged commit e2e7a15 into main Aug 25, 2026
5 checks passed
@savioruz
savioruz deleted the fix/doctor-proxy-false-positives branch August 25, 2026 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api HTTP API transport/handlers bug Something isn't working core memayu-core domain logic infra CI/release/distribution priority-medium Medium priority — important but not blocking self-hosted Relates to self-hosted / local-first usage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant