Ai package fix#190
Open
tmgbedu wants to merge 9 commits into
Open
Conversation
* feat(ai): add TestJudgeAgent trajectory-match evaluator + ModelBuilder fake registry
Add a deterministic evals harness for testing AI agents, modeled on
LangChain's agentevals create_trajectory_match_evaluator.
TestJudgeAgent(trajectory_match_mode=...).record() yields a callable
evaluator that compares an actual message trajectory against a reference
trajectory with no live LLM call — pure structural comparison, typically
driven by Agent.fake() output:
with TestJudgeAgent(trajectory_match_mode="strict").record() as evaluator:
evaluation = evaluator(outputs=response["messages"], reference_outputs=reference)
assert evaluation["score"] is True
Supported modes mirror agentevals semantics:
- strict: same messages in order, same role and tool calls per position
(message content is not compared)
- unordered: same set of tool calls, any order
- subset: output tool calls all appear in the reference
- superset: output tool calls cover all reference tool calls (extras ok)
The evaluator returns a subscriptable result with key/score/comment,
following the LangSmith evaluator-result convention.
Also add a model-level fake registry to ModelBuilder: ModelBuilder.fake()
registers a GenericFakeChatModel for an agent (by class name or instance),
and get_model_for() returns it when present, otherwise builds a real
provider model as before. Agent._build_model() now routes through
get_model_for(), so a faked agent runs the same message-building /
pipeline / tool-execution path as a real one with only the underlying
model swapped. The existing Agent.fake()/FakeAgent/AgentBinding path is
unchanged.
* refactor(ai): rename ModelBuilder to Ai, rebuild fake()/record() around model-level fakes
Ai (formerly ModelBuilder) drops the per-instance agent binding: fake_agent_models/
fake_agent_responses are class-level registries keyed by agent class name, and
get_model_for()/build() take the agent as an argument instead of storing it in
the constructor.
Agent.fake() now registers a fixed, ordered list of replies as a deterministic
chat model (via Ai.fake()) instead of binding a whole FakeAgent stand-in into
the container. Replies flow through the real message-building/pipeline/tool-
execution path, so faked tool calls actually execute — only the model at the
bottom is swapped. Agent.record()/RecordingAgent/AgentBinding are unchanged.
Drops the dead _match_fake()/self._fakes machinery on Agent (never populated
by any code path) along with the now-unreachable FakeAgent/NoFakeResponse
testing helpers.
* feat(ai): add TestJudgeAgent trajectory-match evaluator + ModelBuilder fake registry
Add a deterministic evals harness for testing AI agents, modeled on
LangChain's agentevals create_trajectory_match_evaluator.
TestJudgeAgent(trajectory_match_mode=...).record() yields a callable
evaluator that compares an actual message trajectory against a reference
trajectory with no live LLM call — pure structural comparison, typically
driven by Agent.fake() output:
with TestJudgeAgent(trajectory_match_mode="strict").record() as evaluator:
evaluation = evaluator(outputs=response["messages"], reference_outputs=reference)
assert evaluation["score"] is True
Supported modes mirror agentevals semantics:
- strict: same messages in order, same role and tool calls per position
(message content is not compared)
- unordered: same set of tool calls, any order
- subset: output tool calls all appear in the reference
- superset: output tool calls cover all reference tool calls (extras ok)
The evaluator returns a subscriptable result with key/score/comment,
following the LangSmith evaluator-result convention.
Also add a model-level fake registry to ModelBuilder: ModelBuilder.fake()
registers a GenericFakeChatModel for an agent (by class name or instance),
and get_model_for() returns it when present, otherwise builds a real
provider model as before. Agent._build_model() now routes through
get_model_for(), so a faked agent runs the same message-building /
pipeline / tool-execution path as a real one with only the underlying
model swapped. The existing Agent.fake()/FakeAgent/AgentBinding path is
unchanged.
* refactor(ai): rename ModelBuilder to Ai, rebuild fake()/record() around model-level fakes
Ai (formerly ModelBuilder) drops the per-instance agent binding: fake_agent_models/
fake_agent_responses are class-level registries keyed by agent class name, and
get_model_for()/build() take the agent as an argument instead of storing it in
the constructor.
Agent.fake() now registers a fixed, ordered list of replies as a deterministic
chat model (via Ai.fake()) instead of binding a whole FakeAgent stand-in into
the container. Replies flow through the real message-building/pipeline/tool-
execution path, so faked tool calls actually execute — only the model at the
bottom is swapped. Agent.record()/RecordingAgent/AgentBinding are unchanged.
Drops the dead _match_fake()/self._fakes machinery on Agent (never populated
by any code path) along with the now-unreachable FakeAgent/NoFakeResponse
testing helpers.
* refactor(ai): rename model_builder.py to ai.py
* feat(ai): fluent Agent.record() testing DSL
Bind record() with `as agent` to get a synchronous prompt() plus
assert_text_response(), assert_tool_called()/assert_tool_not_called(),
assert_response_time_lt(), and assert_response_judged() (LLM-as-judge,
verdict cached in the cassette) that all judge the most recent turn.
record(cassette, messages=...) seeds a session's prior history; cassette
keys now fold in the conversation so far (not just the literal message
text) so two sessions with different histories but the same follow-up
text don't collide. The pre-existing bare context-manager usage from
task #327 is unchanged.
Removes TestJudgeAgent/evals.py: its deterministic structural trajectory
matching is unused anywhere outside its own tests, and this DSL now
covers response judging via assert_response_judged.
* fix(ai): fluent record() prompt() is async, not sync
RecordingAgent.prompt() no longer wraps the real async call with
asyncio.run() — it's now the same async method the bare context-manager
path already awaited internally. Wrapping every call in a fresh event
loop was unnecessary and would break inside any already-running loop.
Fluent tests now use IsolatedAsyncioTestCase + await agent.prompt(...).
* refactor(ai): drop dead non-streaming fallback in Agent.stream()
AgentBinding only ever wraps a RecordingAgent (from record()), which
always implements stream() — the hasattr(swapped, "stream") check and
its buffered-response fallback could never actually run.
* style(ai): remove stale comments on Ai's fake registries
* refactor(ai): back assert_response_judged with a JudgeAgent Replace RecordingAgent._judge_live's hand-rolled init_chat_model()/invoke() call with JudgeAgent, an Agent subclass. The judge now goes through the same provider/model resolution pipeline as any other agent, and is fakeable via JudgeAgent.fake() and replayable via JudgeAgent.record() instead of only being testable by mocking langchain directly. assert_response_judged() is now async (the judge call is a real Agent.prompt() under the hood) and accepts an optional provider kwarg, forwarded through to JudgeAgent and folded into the cached verdict's cache key. * refactor(ai): drop JudgeAgent's constructor, use plain Agent attributes model/provider are never set via constructor args anywhere else in the framework -- always class attributes or the @model()/@Provider() decorators. Match that: JudgeAgent has no __init__ override, and _judge_live() sets .model/.provider directly on the instance, same as any other Agent. * refactor(ai): parse JudgeAgent verdicts via schema(), not hand-rolled JSON Give JudgeAgent a Verdict pydantic schema() and put the grading rubric in instructions() -- the JSON reply is now turned into a typed result through the same structured-output path any Agent gets from schema()/response.parsed. Drops the bespoke _build_prompt()/_parse_verdict() helpers. * feat(ai): enforce schema() via with_structured_output on real model calls When an Agent declares schema() and has no tools, build the model with chat_model.with_structured_output(schema, include_raw=True) so the provider enforces the shape, instead of relying on prompt instructions + post-hoc JSON parsing. include_raw keeps the raw message so content/usage/cassettes still work; the Runner passes the structured result through without executing the synthetic tool call, and _to_agent_response unwraps it into response.parsed. Streaming opts out (needs raw token chunks), tools take precedence over a schema in a single call, and the fake/record paths are unchanged (they keep exercising the JSON-string parse path for deterministic replay). Also drops two stale docstrings from Ai. * feat(ai): pass tools and schema in one payload; model picks per turn When an agent declares both tools() and schema(), bind them together via bind_tools([*tools, schema]) so a single model call offers both. The model returns either a real tool call (which the Runner executes) or the schema as its structured answer (which the Runner parses into response.parsed). Schema without tools still uses with_structured_output() for enforcement. _apply_schema no longer coerces content when the agent has tools, so a tool's plain-text output is not force-parsed into the schema. * refactor(ai): always bind schema as a tool, drop with_structured_output branch schema() is just appended to tools() and the whole set is bound in one call. This collapses the schema-only special case: with_structured_output only added tool_choice="any" enforcement, which is redundant now that _apply_schema parses plain JSON text for tool-less agents -- so a schema-only agent gets its parsed result whether the model emits the schema tool call or replies in JSON text. Also drops the now-dead structured-dict passthrough in Runner.run. * refactor(ai): use with_structured_output for schema; leave tool binding as-is Bind tools exactly as before, then wrap the model with with_structured_output(schema, include_raw=True) when a schema is declared. The wrapped model returns {raw, parsed, parsing_error}; the Runner passes it through and _to_agent_response unwraps it into response.parsed. Reverts the schema-as-a-tool detection and the _apply_schema tools guard.
utils/structures.py was used (Configuration.data(), Loader.load()), so per the foundation/support consolidation it belongs in support/ alongside the rest of the framework's internal helpers. The utils/ directory had no __init__.py and nothing else referenced it, so it's gone entirely now that its one file has moved. Updated the two import sites accordingly.
…ures-1133 refactor: move utils/structures.py into support/
…kes directly Agent.fake()/record() now return AgentFake/AgentRecordFake directly instead of wrapping the latter in an AgentBinding indirection layer. AgentRecordFake gains the container-binding, cassette-resolution, and decorator behavior AgentBinding used to provide, so `with Agent.record(...) as agent:` and the `@Agent.record(...)` decorator form both keep working unchanged. AgentBinding is removed entirely, along with every reference to it (imports, __init__ exports, type hints, tests).
Two of the example/agents record() cassettes were recorded against message text that no longer matches the tests' current wording, so their cache keys never hit and every run fell through to a live Gemini call. The new units/agents/record_stream.json fixture had the same problem in the other direction: it only captured the first turn, so the second prompt() call and the assert_response_judged() verdict both missed the cassette too. Re-recorded all three cassettes against the real code path (mocking only the network boundary and the judge, same seam test_agent_record_fluent.py uses) so every prompt/stream/judge call in test_router_agent.py and test_chat_controller.py now replays from disk with no live model calls. uv.lock refreshes the editable fastapi-startkit metadata, which was stale at 0.45.0 against the package's actual 0.50.0.
…1141 refactor(ai): simplify Agent.fake()/record(), remove AgentBinding
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.