Skip to content

BAI-1732 keep tool display titles in Mongo only and pre-warm them from the tool catalog - #174

Merged
imranq2 merged 2 commits into
mainfrom
IQ-BAI-1732
Nov 12, 2026
Merged

imranq2 merged 2 commits into
mainfrom
IQ-BAI-1732

Conversation

@imranq2

@imranq2 imranq2 commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

BAI-1732

Summary

baileyai runs 10 to 100 pods, and the decision is no in-process caching: everything cached lives in Mongo. This removes the in-process title caches and adds a warmer that fills the Mongo title store from the tool catalog's list_tool_titles tool. That tool is what gives tools called directly through call_tool (load_skill, search_skills) a stored title, so their start line stops showing "Load Skill".

  • Removes LearnedTitleCache and the store's _misses and _recent_writes. Titles live in the per-request mapper's own dict. A call_tool start with no title reads Mongo once (bounded timeout) and holds the result for that request only. The process-wide singleton is never mutated.
  • The 30s failure cool-down stays as a circuit breaker: one state flag, no data.
  • Adds replace_titles(titles, prune) and clear() to the store (new LearnedTitleBulkStoreProtocol). Write the new set, then delete names the previous warm wrote, so readers never see an empty window. Warmed entries have no TTL; search_tools-learned entries keep the 24h TTL.
  • Adds ToolTitleWarmer (Protocol and implementation). It calls list_tool_titles through call_mcp_tool_raw with the caller's connection config. It has a bounded timeout and bounded size, never raises, returns ToolTitleWarmResult, and logs counts only.
  • New env vars, all defaulted: LEARNED_TITLE_STORE_BULK_TIMEOUT_SECONDS (10), TOOL_TITLE_WARM_TIMEOUT_SECONDS (10), TOOL_TITLE_WARM_MAX_TITLES (5000).
  • ADR: adrs/0011-mongo-only-tool-titles-and-catalog-warm.md.

Scope boundary

Library change only. The catalog tool list_tool_titles is built in baileyai-skills-service (BAI-1732). Calling warm on reload and bumping the pin are in baileyai. Not covered here: any change to those repos.

Impact assessment

  • Consumers of language-model-common: baileyai, baileyai-skills-service, language-model-gateway, mcp-fhir-agent.
  • Removed public names: LearnedTitleCache, the ToolDisplayNameMapper(learned_titles=) argument and remember_shared_title. A search of local checkouts under ~/git found no use outside this repo. Treated as Minor; flagged because it is technically a removal.
  • Behavior: titles are no longer shared across requests in-process. Each new request reads Mongo once per untitled call_tool target. search_tools results rewrite titles every time (no write de-duplication).
  • Catalog contract depended on: list_tool_titles returns {"titles": {name: template}, "unresolved_servers": [...]}. Unverified against the live catalog until the baileyai-skills-service PR lands. The response is validated defensively, and a bad or partial response never deletes stored titles.
  • Tenancy and PHI: titles are catalog metadata (tool name and title template), with no tenant, user or PHI data.

Tests

Parametrized tests for store replace and clear, the request-scoped lookup, no shared state between mappers, warmer success, partial and invalid responses, timeouts and Mongo failures. Full suite: the only failure is tests/mcp/test_mcp_session_max_sse_event_size.py, which also fails on clean main.

@imranq2 imranq2 added Minor Increment the Minor version risk:deep-review High: changes behavior consumers (baileyai etc.) depend on type:feature PR type: feature labels Nov 12, 2026
@imranq2

imranq2 commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Adversarial review (PR #174)

Summary

  • Total findings: 5. Critical 0, High 0, Medium 2, Low 3.
  • Checked and clear: tenant isolation and PHI (titles are catalog metadata, logs carry counts and exception type only), credential logging, singleton mutation (regression tests cover it), unbounded growth (no in-process data left, manifest bounded by TOOL_TITLE_WARM_MAX_TITLES).

Findings

MEDIUM: Concurrent warms can lose names from the manifest, leaving undeletable no-TTL entries

  • Location: learned_title_store.py _replace (read manifest, write, write manifest).
  • Trigger: baileyai reloads on every pod or on repeated admin reloads, so two replace_titles run at once. B reads the manifest before A writes its own; B then overwrites the manifest with only its set. A name that A wrote and B did not is in neither manifest, so no later reload deletes it, and it has no TTL.
  • Impact: a removed tool keeps a stale title forever (cosmetic only).
  • Fix: serialize reloads in baileyai (one writer per reload, which is the intended flow), or add a periodic clear() plus warm. Confidence: likely, low blast radius.

MEDIUM: Partial catalog response never prunes, so one bad entry freezes pruning

  • Location: tool_title_warmer.py partial = unresolved_count > 0 or len(titles) < len(titles_raw).
  • Trigger: one catalog entry stays over the 256-char cap or is non-string; every reload becomes upsert-only, so removed tools are never deleted.
  • Impact: stale titles accumulate. The result reports partial=True, so callers can alert.
  • Fix: acceptable as the safe choice; callers should log or alert on partial. Confidence: certain, by design.

LOW: Singleton fallback re-reads Mongo on every start event

  • Location: tool_event_handlers.py _hydrate_call_tool_target_title.
  • Trigger: a caller that sets no per-request mapper. The scoped copy is discarded, so the miss is not remembered.
  • Impact: one bounded read per call_tool start; the circuit breaker limits outage cost. baileyai uses with_tools(), so this is not the normal path.

LOW: put_titles now does a manifest read per search_tools result and rewrites every title

  • Impact: extra small Mongo reads and writes per search (no write de-duplication, by the no-cache decision). It is inside the existing timeout and breaker.

LOW: Response size cap counts characters, not bytes

  • Location: _payload (len(json.dumps(...)), len(block.text)).
  • Impact: non-ASCII titles can exceed the byte cap by up to about 4x. The title-count and length caps still bound it (5000 x 256 chars).

If I Had to Break This...

Run two pods' reloads at once while the catalog drops a tool: the manifest race leaves an orphan title that never expires. It is cosmetic, but permanent until the collection entry is removed by hand.

Grudging Respect

Write-then-delete ordering, rejecting truncated responses (so a prune never runs on a partial set), and the singleton-never-mutated regression tests.

@imranq2

imranq2 commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

EA review (PR #174)

Labels: risk:deep-review, Minor, type:feature were already present (applied by the author, not by this run). CI status: pending at review time (informational).

Findings

Rule 04, API changes additive (should-fix). Removes LearnedTitleCache, the ToolDisplayNameMapper(learned_titles=) argument and remember_shared_title. The PR discloses this and a search of local checkouts found no users, but a removal is not additive. Suggestion: maintainers confirm no external consumer, or keep learned_titles as an ignored deprecated argument for one release.

Rule 05, inject interfaces (should-fix). ToolTitleWarmer calls the module function call_mcp_tool_raw directly, so tests monkeypatch it. Suggestion: inject a small catalog-call callable or protocol in the constructor, defaulting to call_mcp_tool_raw.

Rule 28, instrumentation (should-fix). A warm logs counts, but there is no metric or span for success rate or freshness (last successful warm time). Suggestion: add an OpenTelemetry counter and a span in ToolTitleWarmer.warm, or have baileyai record them where it calls warm.

Other rules

No issue: 01, 02, 03, 06 (no secrets; credentials never logged; input validated with size and length caps), 07 (failures are returned as ToolTitleWarmResult and logged, never swallowed silently), 08 (ADR 0011 included), 09 to 13, 17, 19, 20 (synthetic data only), 23 to 27, 29 (impact assessment present). Not applicable: 14, 15, 16, 18, 21, 22.

Merge Readiness: READY_WITH_FIXES

@gecko-security gecko-security Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gecko PR Review

No vulnerabilities found, LGTM.

Summary

This PR moves tool display-title caching out of process memory and into Mongo, per ADR 0011, and the highest-risk changes are in languagemodelcommon/utilities/learned_title_store.py and languagemodelcommon/converters/tool_event_handlers.py, which sit on the streaming call_tool start path. The ADR says the store's _misses and _recent_writes dicts and the LearnedTitleCache LRU are removed, so reviewers should confirm no process-local state remains in these paths. The 30-second circuit breaker and the LEARNED_TITLE_STORE_TIMEOUT_SECONDS (0.5s default) bound must keep store failures from stalling streams. The diff excerpt is truncated, so the code-level behavior of the changed files is not fully verified here.

Important Files Changed

5 important files out of 12 changed

File Path Why it matters
languagemodelcommon/utilities/learned_title_store.py Per the ADR, this file drops the in-process _misses and _recent_writes dicts and adds replace_titles and clear, so any leftover module-level state here would violate the no-in-process-cache decision across 10 to 100 pods.
languagemodelcommon/converters/tool_event_handlers.py The call_tool start handler must fall back to humanized names when get_title times out or errors, and must keep the circuit-breaker skip so a degraded Mongo does not add latency to every stream start event.
languagemodelcommon/utilities/tool_display_name_mapper.py Per the ADR, the per-request mapper now holds titles and the store-miss marker, and the singleton must never be mutated, so a regression here could leak titles between requests or cause repeated store reads.
languagemodelcommon/utilities/tool_title_warmer.py This pre-warms Mongo from the tool catalog, and it must not raise into startup or bypass the is_storable_title length caps (128-char name, 256-char title) that the ADR defines.
languagemodelcommon/container/container_factory.py Container wiring decides which store implementation and timeout settings are injected, so a miswiring could silently disable the timeout or the circuit breaker.

Omitted 7 lower-signal changed files.

Sequence Diagram

sequenceDiagram
    participant Stream as Streaming request
    participant Handler as tool_event_handlers call_tool start
    participant Mapper as ToolDisplayNameMapper request mapper
    participant Store as LearnedTitleStore
    participant Mongo as Mongo py-key-value-aio
    Stream->>Handler: call_tool start event
    Handler->>Mapper: resolve title
    alt title found in request mapper
        Mapper-->>Handler: title
    else no title and breaker open
        Mapper-->>Handler: humanized fallback
    else no title
        Handler->>Store: get_title with timeout
        Store->>Mongo: keyed read
        alt hit
            Mongo-->>Store: title
            Store-->>Handler: title
            Handler->>Mapper: register hit
        else miss or error
            Store-->>Handler: none or error
            Handler->>Mapper: note_store_lookup_miss
            Mapper-->>Handler: humanized fallback
        end
    end
    Handler-->>Stream: start line
Loading

Edit Gecko PR Settings | Reviewed commit: 4cc9ecf

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Additional findings (outside the current diff — GitHub can't attach inline comments there):

  • 🟡 languagemodelcommon/utilities/learned_title_store.py — Every search_tools result now reads the whole warm manifest from Mongo under the 0.5s stream timeout. A slow read trips the 30s breaker and turns off title lookups for every stream on the pod. _put_learned calls _read_manifest on every learned write. That document lists up to TOOL_TITLE_WARM_MAX_TITLES (5000) names of up to 128 chars each. Fix: get the warmed-name check off the per-event path. For example, write learned entries with a conditional put that never overwrites a no-TTL entry, so no manifest read happens per search_tools event. [also at: languagemodelcommon/utilities/learned_title_store.py:122 - Every search_tools result now reads the whole warm manifest from Mongo, up to 5000 names, inside the 0.5s learn-path timeout.]

    Why this was flagged

    The trigger is any search_tools result with titles. It reaches put_titles through _learn_runtime_tool_titles on the streaming path. learned_title_store.py:313-314 runs _read_manifest(), which fetches the single __warmed_titles_manifest__ document holding every warmed name. That can be up to 5000 x 128 chars, about 640KB. The fetch runs inside asyncio.wait_for(..., timeout=self._timeout_seconds), which defaults to 0.5s via LEARNED_TITLE_STORE_TIMEOUT_SECONDS. If the read and puts exceed that, _mark_unavailable() (line 301) sets the 30s cool-down. get_title then returns None for every call_tool start on the pod, which falls back to "Load Skill". On base, put_titles only issued the puts and did no extra read per event. Across 10 to 100 pods, every search_tools event now also pulls the full manifest from Mongo.

    Verification: Severity: nit. put_titles wraps self._put_learned(pending=pending) in asyncio.wait_for(..., timeout=self._timeout_seconds), with a default of 0.5s. _put_learned first runs warmed = await self._read_manifest(). Any timeout calls self._mark_unavailable(), a 30s cool-down. On base, there was no manifest read.

  • 🟡 languagemodelcommon/utilities/learned_title_store.py — If a warm's delete step fails or times out, titles of removed tools stay in Mongo forever, with no TTL. Nothing can ever delete them. _replace rewrites the manifest without the stale names (line 397) before delete_many runs (line 405). Once the manifest drops them, no later replace_titles or clear() sees them. Fix: drop a name from the manifest only after its delete succeeds. One way is to delete stale names first, or to write the manifest as kept | previous and then rewrite it to kept once delete_many returns. [also at: languagemodelcommon/utilities/learned_title_store.py:210 - Warmed titles can end up stored with no TTL and missing from the manifest. Later warms and clear() never delete them. _replace writes entries with ttl=None via put_many before it…]

    Why this was flagged

    replace_titles puts the whole _replace under one asyncio.wait_for(..., timeout=self._bulk_timeout_seconds) (line 375-378). That covers put_many of up to 5000 titles, the manifest put and delete_many. Say delete_many raises or the bulk timeout cancels it after the manifest put at line 397 has finished. The manifest now lists only kept, but the stale entries are still stored with ttl=None. The next warm reads previous = kept (line 388), so previous - kept never includes them. _clear deletes only names in the manifest (line 426-431), so it skips them too. put_titles no longer treats them as warmed, but it rewrites a name only when search_tools returns it, and removed tools are never returned. The docstring says "Only entries written by a previous warm are deleted" and that clear() removes "every warmed entry". Both claims fail after one partial failure. Because warmed entries have no TTL, those titles never expire.

    Verification: Triggering condition: a reload warm whose delete_many raises, or is cancelled by the bulk timeout, after the manifest put has finished. It writes the manifest with ttl=None at lines 204-209. Only after that does it run delete_many at lines 210-215. _clear (lines 232-240) deletes only sorted(previous) taken from the manifest, so it never removes them either. The base branch has no warm or manifest, so this behavior is new.

@imranq2

imranq2 commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Manifest-read finding (put_titles read the whole warm manifest per search_tools result) is fixed in 052e1af: learned writes no longer touch the manifest.

@alvinhenrick alvinhenrick left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 052e1af. I read the full diff: 12 files, including the ADR, the store, the mapper, the warmer, the handler, the container wiring and the tests. Approving. The remaining items below affect which title is displayed, not correctness or data. None of them blocks a merge.

Checked and correct

  • No process-local data left. LearnedTitleCache, _misses, _recent_writes and the threading/OrderedDict imports are gone. _unavailable_until is the only remaining state, and it is a breaker flag. _hydrate_call_tool_target_title never mutates the singleton. When the resolved mapper is self._tool_display_name_mapper, it registers into a with_tools(tools=()) copy. test_process_wide_singleton_is_never_mutated_by_a_store_hit pins this.
  • The manifest-read finding is fixed at this head. _put_learned now runs get_many only on the names in pending and skips entries carrying warmed: True. The manifest key is never read on the learn path, and TestLearnedWriteDoesNotReadManifest asserts that. The 0.5s stream timeout now covers one small batched read plus the puts, not a 5000-name document.
  • The manifest key can't be reached from model output. get_title and put_titles both reject __warmed_titles_manifest__, and there is a test for it.
  • The warmer fails closed.
    • An oversized response, titles that isn't a dict, too_many_titles and no_valid_titles all return ok=False and write nothing.
    • Any unresolved server or dropped entry sets prune=False, so a partial catalog never deletes titles.
    • Exceptions are logged by type only, and test_does_not_log_credentials covers that.
  • Removed public names. I ran an org-wide code search for LearnedTitleCache, remember_shared_title and learned_titles=. The only hit is this repo's own base. The Minor label is defensible.
  • Tests run against py-key-value's real MemoryStore. The failing and slow stores are fakes only at the storage boundary, and the warmer tests patch only call_mcp_tool_raw, the MCP boundary. Nothing internal is mocked.
  • Tenancy and PHI. Titles are catalog metadata, and the warm log carries counts and a reason only.
  • Commits. Both commits start with BAI-1732, are authored by Imran, and have no AI-attribution trailer.

Still open (non-blocking, worth a follow-up ticket)

  1. Orphaned no-TTL titles after a failed delete (claude[bot]'s second finding, not addressed). _replace writes the manifest as kept and only then calls delete_many(stale). If delete_many raises, or the 10s bulk wait_for cancels it, the stale names have already left the manifest. They stay stored with ttl=None and warmed: True. No later replace_titles or clear() can see them, and put_titles won't overwrite them either. The same end state comes from the concurrent-warm race in the adversarial review. Fix: write the manifest as kept | previous, run delete_many, then rewrite it as kept. Add a test whose store raises on delete_many after the manifest put.
  2. TOCTOU in _put_learned. A warm that lands between the get_many and the put gets its no-TTL entry replaced by a 24h-TTL entry without the warmed flag. The flag narrows this window but doesn't close it. The next warm re-writes the entry, so this only matters if warms are rare. A conditional write would close it, if the store supports one.
  3. Unverified MongoDBStore semantics. claude[bot] asked whether a ttl=None upsert over a learned entry clears the existing expires_at. That case comes up on the first warm after deploy, for every recently searched tool. The tests use MemoryStore, which replaces whole entries, so they can't show it. A one-off check against the dev Mongo collection would settle it.
  4. Timeout mismatch with baileyai-skills-service#403. The warmer's TOOL_TITLE_WARM_TIMEOUT_SECONDS defaults to 10s. list_tool_titles resolves each cold server with a 20s timeout, 8 at a time, and has no overall deadline. One slow or cold downstream server makes the whole warm time out with nothing written. That PR's skip_unmatched_oauth=False also means any OAuth server the warm caller isn't logged into shows up in unresolved_servers, so partial will almost always be True and pruning will rarely run. Either raise this default above the catalog's worst case, or add an overall deadline on the catalog side that returns partial results. I've raised the same point on #403.
  5. EA rule 28. There is still no metric for warm success or freshness. A counter on ToolTitleWarmResult.ok and partial, plus a "last successful warm" gauge, would tell you when pruning stops.

ADR 0011 is still Status: proposed. Flip it when the CTO decision it cites is recorded.

I can't verify CI status through this integration, because the combined-status and check-run endpoints return 403. This approval rests on the code at 052e1af, not on a green build. The description says test_mcp_session_max_sse_event_size.py also fails on main. Please confirm that's the only red test before merging.

@imranq2
imranq2 merged commit ea92a90 into main Nov 12, 2026
4 checks passed
@imranq2
imranq2 deleted the IQ-BAI-1732 branch November 12, 2026 04:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Minor Increment the Minor version risk:deep-review High: changes behavior consumers (baileyai etc.) depend on type:feature PR type: feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants