feat(ai): Ai model registry + fluent record() testing DSL#184
Merged
Conversation
…r fake registry
Add a deterministic evals harness for testing AI agents, modeled on
LangChain's agentevals create_trajectory_match_evaluator.
TestJudgeAgent(trajectory_match_mode=...).record() yields a callable
evaluator that compares an actual message trajectory against a reference
trajectory with no live LLM call — pure structural comparison, typically
driven by Agent.fake() output:
with TestJudgeAgent(trajectory_match_mode="strict").record() as evaluator:
evaluation = evaluator(outputs=response["messages"], reference_outputs=reference)
assert evaluation["score"] is True
Supported modes mirror agentevals semantics:
- strict: same messages in order, same role and tool calls per position
(message content is not compared)
- unordered: same set of tool calls, any order
- subset: output tool calls all appear in the reference
- superset: output tool calls cover all reference tool calls (extras ok)
The evaluator returns a subscriptable result with key/score/comment,
following the LangSmith evaluator-result convention.
Also add a model-level fake registry to ModelBuilder: ModelBuilder.fake()
registers a GenericFakeChatModel for an agent (by class name or instance),
and get_model_for() returns it when present, otherwise builds a real
provider model as before. Agent._build_model() now routes through
get_model_for(), so a faked agent runs the same message-building /
pipeline / tool-execution path as a real one with only the underlying
model swapped. The existing Agent.fake()/FakeAgent/AgentBinding path is
unchanged.
…nd model-level fakes Ai (formerly ModelBuilder) drops the per-instance agent binding: fake_agent_models/ fake_agent_responses are class-level registries keyed by agent class name, and get_model_for()/build() take the agent as an argument instead of storing it in the constructor. Agent.fake() now registers a fixed, ordered list of replies as a deterministic chat model (via Ai.fake()) instead of binding a whole FakeAgent stand-in into the container. Replies flow through the real message-building/pipeline/tool- execution path, so faked tool calls actually execute — only the model at the bottom is swapped. Agent.record()/RecordingAgent/AgentBinding are unchanged. Drops the dead _match_fake()/self._fakes machinery on Agent (never populated by any code path) along with the now-unreachable FakeAgent/NoFakeResponse testing helpers.
Bind record() with `as agent` to get a synchronous prompt() plus assert_text_response(), assert_tool_called()/assert_tool_not_called(), assert_response_time_lt(), and assert_response_judged() (LLM-as-judge, verdict cached in the cassette) that all judge the most recent turn. record(cassette, messages=...) seeds a session's prior history; cassette keys now fold in the conversation so far (not just the literal message text) so two sessions with different histories but the same follow-up text don't collide. The pre-existing bare context-manager usage from task #327 is unchanged. Removes TestJudgeAgent/evals.py: its deterministic structural trajectory matching is unused anywhere outside its own tests, and this DSL now covers response judging via assert_response_judged.
RecordingAgent.prompt() no longer wraps the real async call with asyncio.run() — it's now the same async method the bare context-manager path already awaited internally. Wrapping every call in a fresh event loop was unnecessary and would break inside any already-running loop. Fluent tests now use IsolatedAsyncioTestCase + await agent.prompt(...).
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
AgentBinding only ever wraps a RecordingAgent (from record()), which always implements stream() — the hasattr(swapped, "stream") check and its buffered-response fallback could never actually run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #183 (squash-merged): lands the commits pushed after that merge — model_builder.py -> ai.py rename, the fluent record() testing DSL (assert_text_response/assert_tool_called/assert_tool_not_called/assert_response_time_lt/assert_response_judged), and the async prompt() fix. TestJudgeAgent/evals.py removed (unused outside its own tests).