Skip to content

feat(testing): run the golden cookbook suite without an API key - #201

Open
s-alexey wants to merge 1 commit into
cifrom
golden_tests
Open

feat(testing): run the golden cookbook suite without an API key#201
s-alexey wants to merge 1 commit into
cifrom
golden_tests

Conversation

@s-alexey

@s-alexey s-alexey commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

The golden tests were the only end-to-end coverage of the harness — tasks, schema parsing, tool loops, chatrooms, assertions — and every one of them needed a live model. They could not run on a branch, so that coverage never gated a change, and each task definition was locked inside the single 1500-line test module.

Add a public ScriptedLLM that replays canned responses at the invoke() boundary. Because it plugs in below respond(), a scripted run still goes through the real path: schema parsing, assertions, chat history and serialization all behave as they would against a live backend, just offline and deterministically. tests/mocks.py's MockedChat becomes a thin alias.

Split the cookbook into one file per theme. Each file holds its tasks, and under each task the two tests that exercise it: a scripted one that runs anywhere, and a live one parametrized over a model pool. Tests asserting a failure are scripted only — a real model may legitimately answer correctly. Scenarios whose input needs the network, or that verify provider wiring, stay live-only: faking those would assert nothing.

Model pools come from a table where each model is named once alongside what it can be used for, replacing per-feature name sets that had drifted out of sync. A pool is empty when no provider is configured, so live tests skip on an empty parameter set and the suite stays green without credentials — no marker or skip machinery.

Wire the suite into CI, and run it in TESTING mode: the default client serializes every task and run to JSON in the working directory, which left 183 files behind and dominated the runtime.

Offline: 43 tests, no credentials. Live selection is unchanged from the previous suite at 476 model-parametrized tests.

@s-alexey
s-alexey requested a review from dolaameng September 1, 2026 13:13
The golden tests were the only end-to-end coverage of the harness — tasks,
schema parsing, tool loops, chatrooms, assertions — and every one of them
needed a live model. They could not run on a branch, so that coverage never
gated a change, and each task definition was locked inside the single
1500-line test module.

Add a public ScriptedLLM that replays canned responses at the invoke()
boundary. Because it plugs in below respond(), a scripted run still goes
through the real path: schema parsing, assertions, chat history and
serialization all behave as they would against a live backend, just offline
and deterministically. tests/mocks.py's MockedChat becomes a thin alias.

Split the cookbook into one file per theme. Each file holds its tasks, and
under each task the two tests that exercise it: a scripted one that runs
anywhere, and a live one parametrized over a model pool. Tests asserting a
failure are scripted only — a real model may legitimately answer correctly.
Scenarios whose input needs the network, or that verify provider wiring,
stay live-only: faking those would assert nothing.

Model pools come from a table where each model is named once alongside what
it can be used for, replacing per-feature name sets that had drifted out of
sync. A pool is empty when no provider is configured, so live tests skip on
an empty parameter set and the suite stays green without credentials — no
marker or skip machinery.

Wire the suite into CI, and run it in TESTING mode: the default client
serializes every task and run to JSON in the working directory, which left
183 files behind and dominated the runtime.

Offline: 43 tests, no credentials. Live selection is unchanged from the
previous suite at 476 model-parametrized tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant