feat(ai): Ai model registry + fake()/record() testing DSL (fluent record + LLM-as-judge)

[tmgbedu]

Summary

This branch has grown through several rounds of direct revision requests; this description reflects the current state of the diff, not the original scope.

Ai — provider/model resolution + fake-model registry

Ai (renamed from ModelBuilder) resolves the chat model for an agent and holds the fake-model registry as class-level state:

• Ai.fake(agent, messages) registers a GenericFakeChatModel replaying a fixed, ordered list of responses for an agent (by class name).
• Ai().get_model_for(agent, ...) returns the registered fake if one exists, otherwise builds a real provider model exactly as before.
• Agent._build_model() routes through get_model_for(), so a faked agent runs the same message-building / pipeline / tool-execution path as a real one — only the model at the bottom is swapped.

Agent.fake() — rebuilt around model-level fakes

Agent.fake(responses) now takes a fixed, ordered list of replies (plain strings or LangChain messages) instead of the old glob-pattern dict[pattern, reply] API. It swaps in a deterministic chat model via Ai.fake() — same execution path as a real call, only the model is faked. Agent.record()/RecordingAgent/AgentBinding (task #327) are otherwise unchanged by this part.

Agent.record() — fluent testing DSL

Bind record() with as agent for a fluent handle: a synchronous prompt() (no async/await) plus assertions against the current turn:

• assert_text_response(), assert_tool_called(name, predicate=None), assert_tool_not_called(names), assert_response_time_lt(seconds)
• assert_response_judged(model=..., expectation=...) — LLM-as-judge; the verdict is cached in the cassette next to the recorded response, so replays after the first run need no live call
• record(cassette, messages=[...]) seeds a session's prior history (dicts or LangChain messages)
• Cassette keys now fold in the conversation history so far, not just the literal message text, so a seeded session doesn't collide with an unseeded one sharing the same follow-up wording

The pre-existing bare context-manager usage (with Agent.record(cassette): await SomeAgent().prompt(...), task #327) is unchanged.

Removed: TestJudgeAgent / evals.py

The deterministic structural trajectory-match evaluator added earlier in this branch is removed — it had no usages outside its own tests anywhere in the repo, and assert_response_judged now covers the response-judging use case this branch needed.

Docs

Companion docs PR: fastapi-startkit/fastapi-startkit.github.io#65 (fluent record() DSL + messages= seeding).

Test plan

• Full suite: uv run pytest --ignore=tests/masoniteorm/postgres → 1867 passed, 7 skipped, 4 subtests.
• uv run ruff check + ruff format --check — clean.
• New coverage: tests/ai/test_ai_fake.py, tests/ai/test_agent_fake.py (rebuilt for list-based fake()), tests/ai/test_agent_record_fluent.py (24 tests for the fluent DSL, history-collision regression test, judge caching).
• example/agents/tests/units/agents/test_router_agent.py fixed to valid, API-accurate Python (was a broken sketch with invalid syntax); requires live API keys to actually execute (record() cassettes + live judge call), same as the app's other record()-based example tests.

Note for the Reviewer (task #1128): given the scope has expanded twice since the original evals-only brief, a fresh full read-through is probably warranted rather than an incremental diff review.

[tmgbedu]

Code Review Verdict: APPROVE ✅ (posting as comment — GH blocks self-approval; a few non-blocking notes)

Reviewed all 6 files, with particular scrutiny on the unplanned ModelBuilder.fake() / _build_model() scope addition. All engineer claims verified independently.

Correctness — evaluator (evals.py)

• Pure, deterministic, no live LLM. The 4 modes match LangChain agentevals semantics and the spec: strict (per-position role + tool-call multiset, content ignored), unordered (all tool calls, multiset-equal), subset (outputs' tool calls ⊆ reference), superset (reference ⊆ outputs). _tool_calls_equal = multiset equality via mutual greedy superset — correct.
• Unknown mode → ValueError ✓. .record() yields a callable evaluator; result is subscriptable with key/score (bool)/comment ✓. Accepts LangChain messages, role/tool_calls dicts, or a {"messages": [...]} dict ✓.
• __test__ = False on TestJudgeAgent/TrajectoryEvaluator correctly prevents pytest from collecting them as test classes. Nice.

Unplanned scope: ModelBuilder.fake() + _build_model() routing — verified SAFE

• Existing Agent.fake() is unchanged, literally and behaviorally. Traced the path: Agent.fake() → AgentBinding + FakeAgent binds a response-level stand-in into the DI container and bypasses _build_model() entirely (prompt() exits at the stand-in call, never reaching _build_model). The three _build_model() callers (prompt/_run/_stream) only run on the real path.
• The one-line change build() → get_model_for() is inert when no fake is registered: get_model_for() → has_fake_model_for() is False → build(...), identical to before. Directly covered by test_falls_back_to_build_when_no_fake_is_registered, and the end-to-end test_prompt_runs_a_faked_tool_call_end_to_end proves a faked agent runs the full message/pipeline/tool path.

Claims verified

• ruff check + format clean; no type: ignore/pyright: ignore added.
• pyright unchanged at 659 (production files add 0 errors — confirmed in isolation and via authoritative --outputjson; a transient 668 on one run was a stale post-checkout cache read, stable at 659 on re-run and on main).
• Tests: test_evals.py = 24, test_model_builder_fake.py = 10, all pass; full suite 1867 passed / 7 skipped (excluding Postgres integration tests, which fail only for lack of local DB creds — environmental, identical on main).

Non-blocking recommendations

1. Split suggestion: evals.py (the actual task-#1127 deliverable) is fully self-contained and does not depend on ModelBuilder.fake() — only the tests use it. Since it touches shared model-building code, it would ideally have been its own PR for tighter scope/history. It's inert-by-default and well-tested, so shipping together is acceptable, but noting the coupling.
2. Dead code: ModelBuilder.fake_responses is declared and cleared in reset_fakes() but never otherwise used ("Not yet wired up"). Suggest removing it until the follow-up actually needs it (YAGNI) — avoid shipping unused shared state.
3. Global-state hygiene: fake_models is process-global class state; isolation relies on each test calling addCleanup(ModelBuilder.reset_fakes). The new test file does this correctly, but consider an autouse fixture in tests/ai/conftest.py to auto-reset after each test as defense-in-depth against future cross-file leaks.
4. GenericFakeChatModel(messages=iter(turns)) uses a single-use iterator — a fake exhausts after its turns; fine for single-run tests, worth a docstring note for multi-invocation cases.

Functionally correct, safe, well-tested (TDD evident). Not merging per instructions — PM to merge.

[tmgbedu]

Code Review Verdict: REQUEST CHANGES 🔧 (one small, well-scoped cleanup — everything else passes)

Full fresh pass on the revised branch (ModelBuilder→Ai rename + breaking Agent.fake() redesign to an ordered-list, model-level fake). This is solid work — thoroughly tested, clean migration almost everywhere. One loose end blocks a clean approval.

✅ What checks out

• (1) Agent.record() / RecordingAgent / AgentBinding are genuinely untouched. Diff-hunk analysis of testing.py shows only FakeAgent→AgentModelFake (+ removal of NoFakeResponse/_reply_text/_word_chunks) changed; RecordingAgent/AgentBinding bodies appear only as unchanged context. Behaviorally confirmed by 7 passing TestAgentRecord cassette tests.
• (2) New Ai class + registries are sound. build() logic is behaviorally identical (agent threaded as a param instead of self._agent). get_model_for() is inert when no fake is registered (falls through to build()). The new forget() + AgentModelFake context manager give proper scoped cleanup (__exit__ → Ai.forget(...)), which resolves the global-state-leak concern from the first version. Iterator-exhaustion is now a tested contract (test_fake_raises_once_responses_are_exhausted).
• Agent.fake() redesign is well-built: list-based, usable as both context manager and decorator, faked tool calls execute through the real pipeline (end-to-end tested). Old fnmatch/_match_fake/_fakes path fully removed — no lingering references to FakeAgent/NoFakeResponse/ModelBuilder (the only _fakes/.fake({...}) hits elsewhere are the unrelated Process class).
• Example call site migrated correctly: @RouterAgent.fake([...]). Docs (in the separate fastapi_startkit.github.io repo) already describe the new list API with no stale old-API examples.
• Verification: ruff check+format clean; no type: ignore/pyright: ignore added; pyright 659 = baseline; full suite 1867 passed / 7 skipped; AI-module coverage 89% (evals.py 100%) — exceeds the 80% claim.

🔧 Requested change (single, small)

1. AgentSnapshot is left orphaned by this PR — a dangling reference to the removed dict-pattern fake() API (criterion #3).
   • src/fastapi_startkit/ai/response.py still defines AgentSnapshot with full resolve()/load()/save(), and it's still exported in ai/__init__.py __all__, but its only entry point was the old agent.fake({"*pattern*": AgentSnapshot(...)}) dict path, which this PR deleted (_match_fake + the AgentSnapshot-resolution branches in prompt()/stream() are gone). There are now zero callers of AgentSnapshot.resolve() (coverage on response.py dropped to 69% — lines 84–93 unreached).
   • Its class docstring (response.py:51) still advertises the removed API verbatim: agent.fake({"*analyze*": AgentSnapshot(path=...)}). A user following it today would hit a failure, since fake() now requires a list.
   • Please resolve one way or the other: either (a) drop AgentSnapshot along with its __init__ export and the stale docstring example if it's intentionally superseded by record(), or (b) re-wire it into the new fake/record path and fix the docstring to the new API. Right now it ships as a public, exported, non-functional class documenting a removed API.

Minor (non-blocking, carried over)

2. Ai.fake_agent_responses is still declared and cleared in reset_fakes() but never otherwise used ("Not yet wired up"). Recommend dropping it until a follow-up needs it (YAGNI).

On criterion (4) — breaking-change consistency

The diff is internally consistent with a deliberate breaking change: in-repo call sites (example + all tests) use the new list API; the only inconsistency is the orphaned AgentSnapshot above. I cannot verify the user sign-off itself (PM confirming separately) — flagging only that the diff otherwise matches an intentional break.

Not merging per instructions. Happy to re-review quickly once the AgentSnapshot loose end is resolved.

[tmgbedu]

Updated Verdict (current HEAD 6195956f): REQUEST CHANGES 🔧 — 2 items

Re-reviewed the full branch at current HEAD. Since my last pass there are three new commits, not just the async fix: rename model_builder.py→ai.py, a new fluent record() DSL (browser-style assertions + cached LLM-judge), and the async fix. Reviewed all three.

✅ The async fix (6195956f) is correct

RecordingAgent.prompt() previously wrapped the real async call in asyncio.run(), which raises RuntimeError inside an already-running event loop (e.g. async tests / ASGI handlers). It's now async def prompt(...), matching the async _run()/_stream() underneath; timing bookkeeping (last_elapsed/last_response) is preserved and agent.py now awaits stand_in.prompt(...) against the new keyword-only signature. No stray asyncio.run()/_prompt_async left; example + fluent tests correctly moved to IsolatedAsyncioTestCase + await. Good fix.

✅ Also verified

• Rename model_builder.py→ai.py is clean — zero dangling model_builder references; agent.py imports from .ai import Ai.
• record()/RecordingAgent/AgentBinding core is intact; fluent DSL (24 tests in test_agent_record_fluent.py) all pass.
• ruff clean; full suite 1867 passed / 7 skipped.

🔧 Blocking items

1. AgentSnapshot still orphaned (carried over from my last review — unaddressed). response.py still defines it with full resolve()/load()/save(), it's still exported in ai/__init__.py, but it has zero callers (its only entry point, the old dict agent.fake({...}), was removed). Its docstring (response.py:51) still advertises the removed API verbatim: agent.fake({"*analyze*": AgentSnapshot(path=...)}). Please remove it (+ export + docstring) or re-wire + fix the docstring.

2. New pyright error + latent bug in the fluent judge path — this is why pyright is now 660, not the claimed 659:

   • testing.py:283 — return self._parse_verdict(result.content). chat_model.invoke(prompt).content is typed str | list[str | dict] (LangChain message content can be a list of content blocks), but _parse_verdict(raw: str) expects str and immediately does re.search(r"\{.*\}", raw, ...).
   • Static: reported by pyright (reportArgumentType), the sole +1 over baseline.
   • Runtime: latent TypeError if any provider/model returns list-style content (tests pass today only because the models used return plain-string content). Please normalize to str before parsing (e.g. coerce/join list content, or widen _parse_verdict to accept str | list and handle it).

Note for scope awareness (not blocking)

The branch has now grown well past the original task-#1127 brief (trajectory evaluator) — it also carries the ModelBuilder→Ai fake registry, the breaking Agent.fake() redesign, and now a fluent record() DSL with a live/cached LLM-as-judge. That's a lot of independent surface in one PR; flagging for a scope/split decision at the PM's discretion.

Minor (non-blocking, carried)

3. Ai.fake_agent_responses is still declared + cleared but never used ("not yet wired up") — drop until needed.

Not merging. Fixes for (1) and (2) are both small; happy to re-review fast.