Skip to content

Move the lab harness to AI SDK v7 - #144

Merged
DeDuva merged 1 commit into
devfrom
deps/ai-sdk-v7
Sep 1, 2026
Merged

Move the lab harness to AI SDK v7#144
DeDuva merged 1 commit into
devfrom
deps/ai-sdk-v7

Conversation

@DeDuva

@DeDuva DeDuva commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Supersedes #140, #142 and #143, which cannot land separately: ai v7 speaks LanguageModelV4 and the v2 providers do not, so each PR on its own fails while all three together pass. #140 already had test red for exactly this reason.

package from to PR
ai ^5.0.228 ^7.0.88 #140
@ai-sdk/anthropic ^2.0.93 ^4.0.46 #142
@ai-sdk/google ^2.0.86 ^4.0.60 #143

The one that matters: cost accounting

v7 moved the cache breakdown into inputTokenDetails and redefined inputTokens as the total prompt count, cache reads included. TokenUsage in pricing.ts documents the opposite:

/** Cache reads. Assumed **disjoint** from `inputTokens`, as both SDKs report. */
cachedInputTokens?: number;

and priceUsage adds the two terms. So the mechanical fix — swap usage.cachedInputTokens for the new path, leave inputTokens alone — would have billed every cache read twice. Silently, in a tool whose entire output is cost comparison, on the token class that dominates a long agent turn.

The harness now takes inputTokenDetails.noCacheTokens, which is the figure v5's inputTokens carried. Where a provider omits it, it falls back to total-minus-cached rather than to zero, so the disjointness invariant holds either way instead of trading a double-count for an under-count.

Three more in the test's scripted model

  • MockLanguageModelV2MockLanguageModelV4. Both V3 and V4 exist; @ai-sdk/anthropic v4 implements LanguageModelV4, so the mock should exercise the path the real providers take.
  • Usage is two nested breakdowns (inputTokens: { total, noCache, cacheRead, cacheWrite }) rather than flat counters, and top-level totalTokens is gone.
  • finishReason is { unified, raw }, not a bare string.

That last one is worth calling out, because it failed invisibly. With a string finish reason the model still returned successfully — the SDK simply saw no tool calls, so every conformance clause about tool calls reported "saw 0" rather than throwing. Three separate assertions went quiet instead of red, and the first fix attempt looked like it had achieved nothing.

Validation

npm run check → exit 0, 290 files / 7554 tests, from a clean npm ci.

One process note worth recording: an incremental npm install on this branch produced 257 failures across 17 files, nearly all of them Ink terminal-rendering tests with nothing to do with this change. A clean npm ci on the identical tree gives 4 failures, all in the harness test. The 257 were an artifact of a half-updated workspace, and anyone bisecting them would have spent the afternoon on ghosts. npm ci, not npm install, before believing a red suite.

Not included

The upgrade also exposes inputTokenDetails.cacheWriteTokens, which TokenUsage and priceUsage already support but the harness has never populated — cache writes are currently unbilled here. Wiring it up would change reported costs, so it belongs in its own change rather than riding along with a dependency bump.

Supersedes the three separate Dependabot PRs for `ai`, `@ai-sdk/anthropic` and
`@ai-sdk/google`. They cannot land one at a time: `ai` v7 speaks
LanguageModelV4 and the v2 providers do not, so each PR alone fails while all
three together pass.

    ai                  ^5.0.228 -> ^7.0.88   (#140)
    @ai-sdk/anthropic    ^2.0.93 -> ^4.0.46   (#142)
    @ai-sdk/google       ^2.0.86 -> ^4.0.60   (#143)

Four breaking changes, in two layers.

The harness reads usage. v7 moved the cache breakdown into `inputTokenDetails`
and redefined `inputTokens` as the *total* prompt count, cached reads included.
TokenUsage documents inputTokens and cachedInputTokens as disjoint and
priceUsage adds them, so a mechanical rename would have billed every cache read
twice — quietly, in a tool whose output is cost. `noCacheTokens` is the figure
v5's `inputTokens` used to carry; where a provider omits it we fall back to
total-minus-cached rather than to zero, so the invariant holds either way.

The test's scripted model needed three: MockLanguageModelV2 is now
MockLanguageModelV4 (the providers implement V4, so the mock should exercise the
same path), usage is two nested breakdowns rather than flat counters, and
finishReason is `{ unified, raw }` rather than a bare string. That last one is
why the first fix attempt still failed with no error: the model returned
successfully, the string finish reason simply meant no tool calls were seen, and
every conformance clause about tool calls went quiet rather than red.

`npm run check` passes: 290 files, 7554 tests.
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

🛫 PR Readiness Check

ℹ️ This comment updates on each push. Last checked: commit 83471ec

PR Scope: 🔧 Infrastructure

⚠️ 1 item(s) to address before review

Status Check Details
Single commit 1 commit — clean history
Not in draft Ready for review
Branch up to date Up to date with dev
Copilot review No Copilot review yet — it may still be processing
Changeset present No source files changed — changeset not required
Scope clean No .squad/ or docs/proposals/ files
No merge conflicts No merge conflicts
Copilot threads resolved No Copilot review threads
CI passing All checks passing

Files Changed (4 files, +83 −87)

File +/−
package-lock.json +58 −76
packages/squad-lab/package.json +3 −3
packages/squad-lab/src/harnesses/ai-sdk.ts +11 −3
test/lab-ai-sdk-harness.test.ts +11 −5

Total: +83 −87


This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

🟡 Impact Analysis — PR #144

Risk tier: 🟡 MEDIUM

📊 Summary

Metric Count
Files changed 4
Files added 0
Files modified 4
Files deleted 0
Modules touched 2
Critical files 1

🎯 Risk Factors

  • 4 files changed (≤5 → LOW)
  • 2 modules touched (2-4 → MEDIUM)
  • Critical files touched: packages/squad-lab/package.json

📦 Modules Affected

root (3 files)
  • package-lock.json
  • packages/squad-lab/package.json
  • packages/squad-lab/src/harnesses/ai-sdk.ts
tests (1 file)
  • test/lab-ai-sdk-harness.test.ts

⚠️ Critical Files

  • packages/squad-lab/package.json

This report is generated automatically for every PR. See #733 for details.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

🔒 Security Review

🔒 Security review: 1 info.

Severity Category Finding Location
ℹ️ info new-dependency 3 new/changed dependency version(s) in packages/squad-lab/package.json. Verify these packages are trusted and necessary. packages/squad-lab/package.json:40

Automated security review — informational only.

@DeDuva
DeDuva merged commit f2a730b into dev Sep 1, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant