Feature: add a deterministic retrieval-quality evaluation and tuning cookbook

[sagarbpatel31]

Problem Statement

HydraDB's v2 documentation explains query parameters such as query_by, mode, alpha, graph_context, and max_results, but it does not give developers a repeatable way to measure whether a retrieval change actually improves results.

Confusing developer journey this fixes:

Ingest content → wait for indexing → query → results look wrong
→ measure retrieval → choose a tuning path → prove improvement → gate the release

The docs cover the individual controls, but not this end-to-end path from an uncertain recall result to an evidence-backed configuration.

Today a developer tuning retrieval has to answer these questions by inspection:

• Did the expected source appear in the first K results?
• Did duplicated chunks from one source inflate the apparent quality?
• Did a new query profile improve semantic queries while regressing literal lookups?
• Did latency change enough to matter?
• Can the chosen quality floor be enforced before release?

Without a small golden-query workflow, retrieval tuning is subjective and regressions can ship unnoticed.

Proposed Solution

Add a v2 cookbook, Evaluate and tune retrieval quality, plus a dependency-free Node.js evaluator and deterministic fixture.

The contribution would:

1. Define a small fictional corpus and golden queries with expected source IDs.
2. Provide explicit validate, seed, and run commands:
   • validate checks the fixture completely offline.
   • seed explicitly creates or reuses a named sandbox database, ingests fixed-ID fixture sources, and waits for every returned ID to finish indexing.
   • run is read-only and compares query profiles against an existing database.
3. Score source-level Hit@K, Recall@K, and MRR@K by deduplicating returned chunks on their source id.
4. Report p50/p95 observed latency and stable Markdown or JSON output.
5. Compare public v2 retrieval settings such as text/hybrid and fast/thinking without treating raw relevance scores as comparable across profiles.
6. Support optional per-profile quality gates with non-zero exit codes.
7. Include credential-free Node built-in tests for scoring, empty results, malformed API envelopes, missing/failed indexing statuses, timeouts, thresholds, and stable output.
8. Register the cookbook in Mintlify navigation and run the offline evaluator tests in CI.

Live evaluation would be opt-in, use only HYDRA_DB_API_KEY from the environment, never print or persist credentials, never delete data automatically, and never call internal endpoints.

This evaluates HydraDB runtime retrieval quality. It does not overlap with #168 / PR #169, which evaluate whether coding agents can understand the documentation itself.

Alternatives Considered

Add more query-parameter prose

The current query documentation already explains the controls. More prose does not make quality measurable or prevent regressions.

Use an LLM-as-judge

An LLM judge can evaluate answer quality, but it adds model credentials, cost, nondeterminism, and prompt sensitivity. A source-ID retrieval eval is deterministic, inexpensive, and suitable for CI.

Compare raw relevance scores

Scores are useful within one result set but are not a stable cross-profile quality metric. Golden source IDs and rank-based metrics provide a clearer contract.

Build a hosted dashboard

A dashboard would be useful later, but it is too much infrastructure for a first reproducible evaluation loop. A CLI and JSON output are portable and composable.

Additional Context

Planned scope is one logical documentation contribution:

• a v2 Mintlify cookbook and navigation entry;
• one no-dependency Node 20+ evaluator;
• one versioned fictional fixture;
• offline unit tests and a CI test step.

All API behavior will be derived from the repository's public v2 OpenAPI specification. No production data, secrets, internal architecture, or security configuration will be included.

I can implement this immediately after maintainer acknowledgment.

[sagarbpatel31]

@usecortex/hydradb-maintainers Could you please acknowledge whether this scope is appropriate for the HydraDB docs hackathon? I will wait for confirmation before implementation, per CONTRIBUTING.md. The proposed change is limited to the public v2 cookbook, a dependency-free evaluator, a fictional fixture, offline tests, and a CI test step; it will not document internal endpoints or include credentials.