Repository navigation
BAI-1732 keep tool display titles in Mongo only and pre-warm them from the tool catalog - #174
Conversation
…m the tool catalog
Adversarial review (PR #174)Summary
FindingsMEDIUM: Concurrent warms can lose names from the manifest, leaving undeletable no-TTL entries
MEDIUM: Partial catalog response never prunes, so one bad entry freezes pruning
LOW: Singleton fallback re-reads Mongo on every start event
LOW:
|
EA review (PR #174)Labels: FindingsRule 04, API changes additive (should-fix). Removes Rule 05, inject interfaces (should-fix). Rule 28, instrumentation (should-fix). A warm logs counts, but there is no metric or span for success rate or freshness (last successful warm time). Suggestion: add an OpenTelemetry counter and a span in Other rulesNo issue: 01, 02, 03, 06 (no secrets; credentials never logged; input validated with size and length caps), 07 (failures are returned as Merge Readiness: READY_WITH_FIXES |
There was a problem hiding this comment.
Gecko PR Review
No vulnerabilities found, LGTM.
Summary
This PR moves tool display-title caching out of process memory and into Mongo, per ADR 0011, and the highest-risk changes are in languagemodelcommon/utilities/learned_title_store.py and languagemodelcommon/converters/tool_event_handlers.py, which sit on the streaming call_tool start path. The ADR says the store's _misses and _recent_writes dicts and the LearnedTitleCache LRU are removed, so reviewers should confirm no process-local state remains in these paths. The 30-second circuit breaker and the LEARNED_TITLE_STORE_TIMEOUT_SECONDS (0.5s default) bound must keep store failures from stalling streams. The diff excerpt is truncated, so the code-level behavior of the changed files is not fully verified here.
Important Files Changed
5 important files out of 12 changed
| File Path | Why it matters |
|---|---|
languagemodelcommon/utilities/learned_title_store.py |
Per the ADR, this file drops the in-process _misses and _recent_writes dicts and adds replace_titles and clear, so any leftover module-level state here would violate the no-in-process-cache decision across 10 to 100 pods. |
languagemodelcommon/converters/tool_event_handlers.py |
The call_tool start handler must fall back to humanized names when get_title times out or errors, and must keep the circuit-breaker skip so a degraded Mongo does not add latency to every stream start event. |
languagemodelcommon/utilities/tool_display_name_mapper.py |
Per the ADR, the per-request mapper now holds titles and the store-miss marker, and the singleton must never be mutated, so a regression here could leak titles between requests or cause repeated store reads. |
languagemodelcommon/utilities/tool_title_warmer.py |
This pre-warms Mongo from the tool catalog, and it must not raise into startup or bypass the is_storable_title length caps (128-char name, 256-char title) that the ADR defines. |
languagemodelcommon/container/container_factory.py |
Container wiring decides which store implementation and timeout settings are injected, so a miswiring could silently disable the timeout or the circuit breaker. |
Omitted 7 lower-signal changed files.
Sequence Diagram
sequenceDiagram
participant Stream as Streaming request
participant Handler as tool_event_handlers call_tool start
participant Mapper as ToolDisplayNameMapper request mapper
participant Store as LearnedTitleStore
participant Mongo as Mongo py-key-value-aio
Stream->>Handler: call_tool start event
Handler->>Mapper: resolve title
alt title found in request mapper
Mapper-->>Handler: title
else no title and breaker open
Mapper-->>Handler: humanized fallback
else no title
Handler->>Store: get_title with timeout
Store->>Mongo: keyed read
alt hit
Mongo-->>Store: title
Store-->>Handler: title
Handler->>Mapper: register hit
else miss or error
Store-->>Handler: none or error
Handler->>Mapper: note_store_lookup_miss
Mapper-->>Handler: humanized fallback
end
end
Handler-->>Stream: start line
Edit Gecko PR Settings | Reviewed commit: 4cc9ecf
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🟡
languagemodelcommon/utilities/learned_title_store.py— Everysearch_toolsresult now reads the whole warm manifest from Mongo under the 0.5s stream timeout. A slow read trips the 30s breaker and turns off title lookups for every stream on the pod._put_learnedcalls_read_manifeston every learned write. That document lists up toTOOL_TITLE_WARM_MAX_TITLES(5000) names of up to 128 chars each. Fix: get the warmed-name check off the per-event path. For example, write learned entries with a conditional put that never overwrites a no-TTL entry, so no manifest read happens persearch_toolsevent. [also at: languagemodelcommon/utilities/learned_title_store.py:122 - Every search_tools result now reads the whole warm manifest from Mongo, up to 5000 names, inside the 0.5s learn-path timeout.]Why this was flagged
The trigger is any
search_toolsresult with titles. It reachesput_titlesthrough_learn_runtime_tool_titleson the streaming path. learned_title_store.py:313-314 runs_read_manifest(), which fetches the single__warmed_titles_manifest__document holding every warmed name. That can be up to 5000 x 128 chars, about 640KB. The fetch runs insideasyncio.wait_for(..., timeout=self._timeout_seconds), which defaults to 0.5s viaLEARNED_TITLE_STORE_TIMEOUT_SECONDS. If the read and puts exceed that,_mark_unavailable()(line 301) sets the 30s cool-down.get_titlethen returns None for everycall_toolstart on the pod, which falls back to "Load Skill". On base,put_titlesonly issued the puts and did no extra read per event. Across 10 to 100 pods, everysearch_toolsevent now also pulls the full manifest from Mongo.Verification: Severity: nit.
put_titleswrapsself._put_learned(pending=pending)inasyncio.wait_for(..., timeout=self._timeout_seconds), with a default of 0.5s._put_learnedfirst runswarmed = await self._read_manifest(). Any timeout callsself._mark_unavailable(), a 30s cool-down. On base, there was no manifest read. -
🟡
languagemodelcommon/utilities/learned_title_store.py— If a warm's delete step fails or times out, titles of removed tools stay in Mongo forever, with no TTL. Nothing can ever delete them._replacerewrites the manifest without the stale names (line 397) beforedelete_manyruns (line 405). Once the manifest drops them, no laterreplace_titlesorclear()sees them. Fix: drop a name from the manifest only after its delete succeeds. One way is to delete stale names first, or to write the manifest askept | previousand then rewrite it tokeptoncedelete_manyreturns. [also at: languagemodelcommon/utilities/learned_title_store.py:210 - Warmed titles can end up stored with no TTL and missing from the manifest. Later warms andclear()never delete them._replacewrites entries withttl=Nonevia put_many before it…]Why this was flagged
replace_titlesputs the whole_replaceunder oneasyncio.wait_for(..., timeout=self._bulk_timeout_seconds)(line 375-378). That coversput_manyof up to 5000 titles, the manifestputanddelete_many. Saydelete_manyraises or the bulk timeout cancels it after the manifestputat line 397 has finished. The manifest now lists onlykept, but the stale entries are still stored withttl=None. The next warm readsprevious = kept(line 388), soprevious - keptnever includes them._cleardeletes only names in the manifest (line 426-431), so it skips them too.put_titlesno longer treats them as warmed, but it rewrites a name only whensearch_toolsreturns it, and removed tools are never returned. The docstring says "Only entries written by a previous warm are deleted" and thatclear()removes "every warmed entry". Both claims fail after one partial failure. Because warmed entries have no TTL, those titles never expire.Verification: Triggering condition: a reload warm whose
delete_manyraises, or is cancelled by the bulk timeout, after the manifestputhas finished. It writes the manifest withttl=Noneat lines 204-209. Only after that does it rundelete_manyat lines 210-215._clear(lines 232-240) deletes onlysorted(previous)taken from the manifest, so it never removes them either. The base branch has no warm or manifest, so this behavior is new.
…fest, clearer scoped mapper name
|
Manifest-read finding (put_titles read the whole warm manifest per search_tools result) is fixed in 052e1af: learned writes no longer touch the manifest. |
alvinhenrick
left a comment
There was a problem hiding this comment.
Reviewed at 052e1af. I read the full diff: 12 files, including the ADR, the store, the mapper, the warmer, the handler, the container wiring and the tests. Approving. The remaining items below affect which title is displayed, not correctness or data. None of them blocks a merge.
Checked and correct
- No process-local data left.
LearnedTitleCache,_misses,_recent_writesand the threading/OrderedDict imports are gone._unavailable_untilis the only remaining state, and it is a breaker flag._hydrate_call_tool_target_titlenever mutates the singleton. When the resolved mapper isself._tool_display_name_mapper, it registers into awith_tools(tools=())copy.test_process_wide_singleton_is_never_mutated_by_a_store_hitpins this. - The manifest-read finding is fixed at this head.
_put_learnednow runsget_manyonly on the names inpendingand skips entries carryingwarmed: True. The manifest key is never read on the learn path, andTestLearnedWriteDoesNotReadManifestasserts that. The 0.5s stream timeout now covers one small batched read plus the puts, not a 5000-name document. - The manifest key can't be reached from model output.
get_titleandput_titlesboth reject__warmed_titles_manifest__, and there is a test for it. - The warmer fails closed.
- An oversized response,
titlesthat isn't a dict,too_many_titlesandno_valid_titlesall returnok=Falseand write nothing. - Any unresolved server or dropped entry sets
prune=False, so a partial catalog never deletes titles. - Exceptions are logged by type only, and
test_does_not_log_credentialscovers that.
- An oversized response,
- Removed public names. I ran an org-wide code search for
LearnedTitleCache,remember_shared_titleandlearned_titles=. The only hit is this repo's own base. TheMinorlabel is defensible. - Tests run against py-key-value's real
MemoryStore. The failing and slow stores are fakes only at the storage boundary, and the warmer tests patch onlycall_mcp_tool_raw, the MCP boundary. Nothing internal is mocked. - Tenancy and PHI. Titles are catalog metadata, and the warm log carries counts and a reason only.
- Commits. Both commits start with
BAI-1732, are authored by Imran, and have no AI-attribution trailer.
Still open (non-blocking, worth a follow-up ticket)
- Orphaned no-TTL titles after a failed delete (claude[bot]'s second finding, not addressed).
_replacewrites the manifest askeptand only then callsdelete_many(stale). Ifdelete_manyraises, or the 10s bulkwait_forcancels it, the stale names have already left the manifest. They stay stored withttl=Noneandwarmed: True. No laterreplace_titlesorclear()can see them, andput_titleswon't overwrite them either. The same end state comes from the concurrent-warm race in the adversarial review. Fix: write the manifest askept | previous, rundelete_many, then rewrite it askept. Add a test whose store raises ondelete_manyafter the manifest put. - TOCTOU in
_put_learned. A warm that lands between theget_manyand theputgets its no-TTL entry replaced by a 24h-TTL entry without thewarmedflag. The flag narrows this window but doesn't close it. The next warm re-writes the entry, so this only matters if warms are rare. A conditional write would close it, if the store supports one. - Unverified
MongoDBStoresemantics. claude[bot] asked whether attl=Noneupsert over a learned entry clears the existingexpires_at. That case comes up on the first warm after deploy, for every recently searched tool. The tests useMemoryStore, which replaces whole entries, so they can't show it. A one-off check against the dev Mongo collection would settle it. - Timeout mismatch with baileyai-skills-service#403. The warmer's
TOOL_TITLE_WARM_TIMEOUT_SECONDSdefaults to 10s.list_tool_titlesresolves each cold server with a 20s timeout, 8 at a time, and has no overall deadline. One slow or cold downstream server makes the whole warm time out with nothing written. That PR'sskip_unmatched_oauth=Falsealso means any OAuth server the warm caller isn't logged into shows up inunresolved_servers, sopartialwill almost always be True and pruning will rarely run. Either raise this default above the catalog's worst case, or add an overall deadline on the catalog side that returns partial results. I've raised the same point on #403. - EA rule 28. There is still no metric for warm success or freshness. A counter on
ToolTitleWarmResult.okandpartial, plus a "last successful warm" gauge, would tell you when pruning stops.
ADR 0011 is still Status: proposed. Flip it when the CTO decision it cites is recorded.
I can't verify CI status through this integration, because the combined-status and check-run endpoints return 403. This approval rests on the code at 052e1af, not on a green build. The description says test_mcp_session_max_sse_event_size.py also fails on main. Please confirm that's the only red test before merging.
BAI-1732
Summary
baileyai runs 10 to 100 pods, and the decision is no in-process caching: everything cached lives in Mongo. This removes the in-process title caches and adds a warmer that fills the Mongo title store from the tool catalog's
list_tool_titlestool. That tool is what gives tools called directly throughcall_tool(load_skill,search_skills) a stored title, so their start line stops showing "Load Skill".LearnedTitleCacheand the store's_missesand_recent_writes. Titles live in the per-request mapper's own dict. Acall_toolstart with no title reads Mongo once (bounded timeout) and holds the result for that request only. The process-wide singleton is never mutated.replace_titles(titles, prune)andclear()to the store (newLearnedTitleBulkStoreProtocol). Write the new set, then delete names the previous warm wrote, so readers never see an empty window. Warmed entries have no TTL;search_tools-learned entries keep the 24h TTL.ToolTitleWarmer(Protocol and implementation). It callslist_tool_titlesthroughcall_mcp_tool_rawwith the caller's connection config. It has a bounded timeout and bounded size, never raises, returnsToolTitleWarmResult, and logs counts only.LEARNED_TITLE_STORE_BULK_TIMEOUT_SECONDS(10),TOOL_TITLE_WARM_TIMEOUT_SECONDS(10),TOOL_TITLE_WARM_MAX_TITLES(5000).adrs/0011-mongo-only-tool-titles-and-catalog-warm.md.Scope boundary
Library change only. The catalog tool
list_tool_titlesis built in baileyai-skills-service (BAI-1732). Callingwarmon reload and bumping the pin are in baileyai. Not covered here: any change to those repos.Impact assessment
language-model-common: baileyai, baileyai-skills-service, language-model-gateway, mcp-fhir-agent.LearnedTitleCache, theToolDisplayNameMapper(learned_titles=)argument andremember_shared_title. A search of local checkouts under~/gitfound no use outside this repo. Treated as Minor; flagged because it is technically a removal.call_tooltarget.search_toolsresults rewrite titles every time (no write de-duplication).list_tool_titlesreturns{"titles": {name: template}, "unresolved_servers": [...]}. Unverified against the live catalog until the baileyai-skills-service PR lands. The response is validated defensively, and a bad or partial response never deletes stored titles.Tests
Parametrized tests for store replace and clear, the request-scoped lookup, no shared state between mappers, warmer success, partial and invalid responses, timeouts and Mongo failures. Full suite: the only failure is
tests/mcp/test_mcp_session_max_sse_event_size.py, which also fails on clean main.