Move the lab harness to AI SDK v7 - #144
Merged
Merged
Conversation
Supersedes the three separate Dependabot PRs for `ai`, `@ai-sdk/anthropic` and
`@ai-sdk/google`. They cannot land one at a time: `ai` v7 speaks
LanguageModelV4 and the v2 providers do not, so each PR alone fails while all
three together pass.
ai ^5.0.228 -> ^7.0.88 (#140)
@ai-sdk/anthropic ^2.0.93 -> ^4.0.46 (#142)
@ai-sdk/google ^2.0.86 -> ^4.0.60 (#143)
Four breaking changes, in two layers.
The harness reads usage. v7 moved the cache breakdown into `inputTokenDetails`
and redefined `inputTokens` as the *total* prompt count, cached reads included.
TokenUsage documents inputTokens and cachedInputTokens as disjoint and
priceUsage adds them, so a mechanical rename would have billed every cache read
twice — quietly, in a tool whose output is cost. `noCacheTokens` is the figure
v5's `inputTokens` used to carry; where a provider omits it we fall back to
total-minus-cached rather than to zero, so the invariant holds either way.
The test's scripted model needed three: MockLanguageModelV2 is now
MockLanguageModelV4 (the providers implement V4, so the mock should exercise the
same path), usage is two nested breakdowns rather than flat counters, and
finishReason is `{ unified, raw }` rather than a bare string. That last one is
why the first fix attempt still failed with no error: the model returned
successfully, the string finish reason simply meant no tool calls were seen, and
every conformance clause about tool calls went quiet rather than red.
`npm run check` passes: 290 files, 7554 tests.
Contributor
🛫 PR Readiness Check
PR Scope: 🔧 Infrastructure
|
| Status | Check | Details |
|---|---|---|
| ✅ | Single commit | 1 commit — clean history |
| ✅ | Not in draft | Ready for review |
| ✅ | Branch up to date | Up to date with dev |
| ❌ | Copilot review | No Copilot review yet — it may still be processing |
| ✅ | Changeset present | No source files changed — changeset not required |
| ✅ | Scope clean | No .squad/ or docs/proposals/ files |
| ✅ | No merge conflicts | No merge conflicts |
| ✅ | Copilot threads resolved | No Copilot review threads |
| ✅ | CI passing | All checks passing |
Files Changed (4 files, +83 −87)
| File | +/− |
|---|---|
package-lock.json |
+58 −76 |
packages/squad-lab/package.json |
+3 −3 |
packages/squad-lab/src/harnesses/ai-sdk.ts |
+11 −3 |
test/lab-ai-sdk-harness.test.ts |
+11 −5 |
Total: +83 −87
This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.
Contributor
🟡 Impact Analysis — PR #144Risk tier: 🟡 MEDIUM 📊 Summary
🎯 Risk Factors
📦 Modules Affectedroot (3 files)
tests (1 file)
|
Contributor
🔒 Security Review🔒 Security review: 1 info.
Automated security review — informational only. |
This was referenced Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #140, #142 and #143, which cannot land separately:
aiv7 speaksLanguageModelV4and the v2 providers do not, so each PR on its own fails while all three together pass. #140 already hadtestred for exactly this reason.ai@ai-sdk/anthropic@ai-sdk/googleThe one that matters: cost accounting
v7 moved the cache breakdown into
inputTokenDetailsand redefinedinputTokensas the total prompt count, cache reads included.TokenUsageinpricing.tsdocuments the opposite:and
priceUsageadds the two terms. So the mechanical fix — swapusage.cachedInputTokensfor the new path, leaveinputTokensalone — would have billed every cache read twice. Silently, in a tool whose entire output is cost comparison, on the token class that dominates a long agent turn.The harness now takes
inputTokenDetails.noCacheTokens, which is the figure v5'sinputTokenscarried. Where a provider omits it, it falls back to total-minus-cached rather than to zero, so the disjointness invariant holds either way instead of trading a double-count for an under-count.Three more in the test's scripted model
MockLanguageModelV2→MockLanguageModelV4. BothV3andV4exist;@ai-sdk/anthropicv4 implementsLanguageModelV4, so the mock should exercise the path the real providers take.inputTokens: { total, noCache, cacheRead, cacheWrite }) rather than flat counters, and top-leveltotalTokensis gone.finishReasonis{ unified, raw }, not a bare string.That last one is worth calling out, because it failed invisibly. With a string finish reason the model still returned successfully — the SDK simply saw no tool calls, so every conformance clause about tool calls reported "saw 0" rather than throwing. Three separate assertions went quiet instead of red, and the first fix attempt looked like it had achieved nothing.
Validation
npm run check→ exit 0, 290 files / 7554 tests, from a cleannpm ci.One process note worth recording: an incremental
npm installon this branch produced 257 failures across 17 files, nearly all of them Ink terminal-rendering tests with nothing to do with this change. A cleannpm cion the identical tree gives 4 failures, all in the harness test. The 257 were an artifact of a half-updated workspace, and anyone bisecting them would have spent the afternoon on ghosts.npm ci, notnpm install, before believing a red suite.Not included
The upgrade also exposes
inputTokenDetails.cacheWriteTokens, whichTokenUsageandpriceUsagealready support but the harness has never populated — cache writes are currently unbilled here. Wiring it up would change reported costs, so it belongs in its own change rather than riding along with a dependency bump.