docs(notes): the agreed design for moving search_docs onto this ranker

Decisions only, nothing implemented. Written down because the implementation
happens in the MCP server repo, and the reasoning that rules out the obvious
approaches does not survive a repo boundary.

The load-bearing ones: section boundaries are derived and published as a
build-time map rather than authored as markers, because a `#` opening a line
inside a bash fence is indistinguishable from a heading and because
re-deriving slugs downstream would be the third copy of md-slug.js. And
get_doc's URL fragment becomes the switch between whole page and section,
which needs no schema change — so the bump stays a minor and the OpenAI
listing review is left alone.

Also records what was deferred and why, since caller-supplied constants and
human/assistant profiles will look obviously right again later.

Author: Mikhus

notes/mcp-search-docs-plan.md

@@ -0,0 +1,153 @@
+# Plan: rework MCP `search_docs` onto the site's ranker
+
+Agreed 2026-08-05, not started. Design decisions only — no code has been written for any of
+this. Implementation spans two repos: this one (publish what the server needs) and the
+**@imqueue MCP server** (consume it).
+
+## Why
+
+There are two rankers today. The website's lives in `src/_shared/js/search.js` and is measured
+by `npm run kpi:search`. The MCP server's is separate: `search_docs` fetches imqueue.org and
+reads the purpose-built feeds `/api/search-index.json` (349 symbol entries, see
+`src/org/api/search-index.11ty.js`) and `/blog/search-index.json`.
+
+So MCP relevance is **unmeasured**. "Did we improve `search_docs`?" has no answer. Sharing one
+ranker collapses two surfaces into one that the KPI already covers, and that is the main win —
+bigger than any constant tuning.
+
+It is also nearly free: `search.js` exports itself when `typeof document === "undefined"`,
+which is exactly how the KPI drives it 12,281 times per run. The server-side path is already
+exercised harder than the browser one.
+
+Secondary win: the site ranker returns **section-level anchors** (719 sections vs page-level
+feeds). An agent handed `/page/#specific-heading` stops burning context pulling whole pages —
+which is what raises every question below.
+
+## Decisions
+
+### 1. Share the ranker, don't fork it
+
+One relevance implementation, one KPI, one place a fix lands for both surfaces. The ~1 MB of
+index (`search-index.json` 326 KB + `search-text.json` 663 KB) is a browser cost, not a server
+one: fetch once, cache, invalidate on deploy.
+
+### 2. Section boundaries are DERIVED and PUBLISHED — never authored
+
+No begin/end markers in the markdown. The boundaries already exist three times over:
+`markdown-it-anchor` assigns the ids, `scripts/lib/md-slug.js` replicates that slug algorithm,
+and `scripts/lib/search-corpus.js` already splits every page into the shipped sections. A
+section is *this heading until the next heading of equal-or-higher level, descendants
+included*.
+
+Hand-placed markers would be a fourth source of truth, invisible to authors, rotting the first
+time someone adds a heading — the same silent drift as the duplicated CLI catalog.
+
+Instead this repo emits a **build-time section map** (anchor → line/byte range) alongside the
+markdown mirrors. Two reasons it must be build-time and not computed in the server:
+
+- **Regex slicing is unsafe here.** A `#` starting a line inside a fenced code block is
+  indistinguishable from a heading, and this site is full of bash blocks with `# comment`.
+  Slicing by regex will eventually cut inside a fence and emit an unbalanced fence. Ranges
+  derived from the parsed AST cannot.
+- **It avoids a third copy of the slug algorithm.** `md-slug.js` exists *because* this already
+  had to be replicated once. A third copy diverges on unicode, duplicate-heading `-1`
+  suffixes, or `markdown-it-attrs` id overrides. The site is the only thing that knows the real
+  ids, so it publishes them.
+
+### 3. The URL fragment is the switch for `get_doc`
+
+- `get_doc("/page/")` → whole document (today's behaviour, unchanged)
+- `get_doc("/page/#heading")` → that section
+
+Chosen over adding a parameter because it needs **no schema change** — which keeps the version
+bump a minor and keeps the pending listing review undisturbed (see Constraints). It is
+self-describing, backward compatible, and it makes the anchor *mean* something instead of being
+stripped. Stripping it would throw away the precision `search_docs` just worked to find.
+
+It also puts the choice with the party that knows: summarising a tutorial chapter wants the
+page, checking one API caveat wants the section.
+
+### 4. A slice must carry its context
+
+The failure mode of slicing is not tokens, it is **a confidently wrong answer read out of
+context**. Every slice includes:
+
+- the **ancestor heading chain** — `### Retry` under `## @imqueue/job` and under
+  `## @imqueue/rpc` mean different things
+- the page title and URL
+- an explicit **"this is a section of N, full page at <url>"** notice
+
+The notice matters most: an agent that knows it got a slice can escalate; one that doesn't will
+answer from a fragment and never know what it missed.
+
+### 5. Fall back to the whole page, don't error
+
+- **Anchors that aren't headings.** `markdown-it-attrs` allows `{#id}` on a paragraph or table,
+  where no section boundary exists → return the whole page.
+- **Small pages.** Below some size the scaffolding costs more than the slice saves → ignore the
+  fragment. Keeps the common case boring.
+- **Sections that don't stand alone** ("in the example above", a code block defined under a
+  previous heading). No mechanism fixes this; it is why decision 4 exists.
+
+## Constraints
+
+**The OpenAI MCP listing is in review.** Freeze the *declared surface* until it lands: don't
+rename the tool, don't change `limit`'s meaning or bounds (1–20, default 6), don't add or
+remove output fields. Internals are safe to change — and notably the egress story does not
+move, because `search_docs` already fetches imqueue.org and is host-locked
+(`src/org/mcp/security.md`); reading different files on the same host is not a new permission.
+
+Two lesser caveats: any example outputs in the submission go stale (cosmetic, and in the good
+direction), and a reviewer may test a build newer than the one submitted (fine while the output
+*shape* is identical).
+
+**Version bump: minor.** A pure ranking change is technically a patch, but for an agent-facing
+tool the returned set *is* the product, and "patch" would falsely say "nothing you'd notice".
+It becomes minor on the merits anyway once section anchors are added (additive, backward
+compatible). It would be **major** if `limit` semantics change, an output field is
+removed/renamed, or the returned URLs become unusable by existing consumers. A major bump is
+also likelier to force the directory entry to be revisited — another reason to stay minor.
+Unrelated to the MCP protocol version, which is negotiated separately.
+
+## Verify before implementing
+
+1. **Does `get_doc` accept a fragment today?** It resolves URLs against the markdown mirrors,
+   emitted as `/page/index.md` (`src/md-mirror.liquid`) and `/page.md`
+   (`src/md-mirror-flat.liquid`). If it maps URL → mirror path by appending, a fragment lands
+   on a 404. This determines whether decision 3 is additive or a fix.
+2. **Do any current callers pass fragments?** Today's feeds are page-level, so probably not —
+   which is what makes fragment-triggered slicing purely additive.
+3. Whether the MCP server can consume the site's index directly or needs a narrower published
+   artifact.
+
+## Measurement
+
+"It feels leaner" is not a result. Two numbers:
+
+- **recall@6** on an agent-shaped query slice — `api-symbol` + `api-member` + `api-path` +
+  `keywords` + `heading` + `title` from the artificial set, ~3,400 queries that are by
+  construction "vocabulary the agent has already read". Position decay is the wrong metric for
+  MCP: an agent reads all 6 results, so whether the page is *in the set* is what matters, not
+  whether it is first.
+- **tokens spent to reach the right content**, comparing `get_doc(page)` against
+  `get_doc(#section)` — *with* a check that the answer-bearing text is actually inside the
+  slice. Without that second half the metric rewards truncating everything to nothing.
+
+For recall, the lever is not IDF — it is the eviction machinery (`GROUP_FLOOR` 0.3,
+`MIN_SCORE` 60, `MAX.perPage` 2). For a human, eviction is a kindness that stops near-misses
+filling the group under the answer. For an agent with 6 slots a dropped candidate is
+unrecoverable — and eviction is exactly what deleted the right page for
+"idempotency microservices" rather than merely ranking it low.
+
+## Explicitly deferred
+
+**Per-caller tuning constants, and human/assistant profiles.** Judged premature. Caller-supplied
+constants would destroy the KPI (no single ranker left to measure) and hand tuning to an LLM
+that has no feedback signal. Profiles are defensible in principle but must be scoped to
+retrieval/eviction/output shape — never to the scoring weights, which encode what a page is
+*about* and are a property of the corpus, not the reader.
+
+Revisit only after the shared ranker ships and one *measured* difference justifies the
+abstraction: run MCP with a single deviation (eviction off, `limit`-sized result set) and see
+whether recall@6 moves. If profiles ever land, the gate is non-negotiable — no profile without
+a KPI baseline, or the second one rots like the duplicated CLI catalog.