Skip to content

test(ai): move Baseten tests off Kimi-K2.6, which models.dev removed - #3816

Merged
flora131 merged 2 commits into
mainfrom
fix/ai-tests-baseten-kimi
Oct 21, 2026
Merged

flora131 merged 2 commits into
mainfrom
fix/ai-tests-baseten-kimi

Conversation

@flora131

@flora131 flora131 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

models.dev removed moonshotai/Kimi-K2.6 from Baseten (Kimi-K3 replaces it), so two packages/ai tests no longer typecheck on a fresh build: on main and on every branch. This updates them, following pi's practice of updating tests when a model leaves the catalog (pi e86b0547, 99fe1489), so drift stays visible. pi still has the K2.6 test.

Changes

  • image-tool-result.test.ts: the live Baseten provider block (runs only with BASETEN_API_KEY) uses Kimi-K3, which also accepts image input.

  • baseten-models.test.ts: the toggle-only reasoning test used K2.6 as its example. No current Baseten model is toggle-only, but the path is still live in the generator and runtime, so the test builds a toggle-only model on Kimi-K3 and keeps its runtime assertions (levels off/high, chat_template_args.enable_thinking: true, no reasoning_effort).

  • baseten-qwen-provider.test.ts (coding-agent): models.dev removed ten more Baseten models after the first commit (DeepSeek-V4-Pro, Kimi-K2.5/K2.6/K2.7-Code, Nemotron-120B-A12B, both Inkling models, GLM-4.7, GLM-5, GLM-5.1). The exact-list assertion now matches the current 14-model catalog.

Notes

  • The first CI run also failed codemode-worker-admission "holds shared capacity until actual worker exit after timeout" on Windows. That test is load-sensitive and unrelated to this change (it passes in isolation). It is tracked separately.
  • The two assertions that only restated the fixture were dropped, so the generator's toggle-only mapping has no live model to check until Baseten lists one again.
  • The Baseten default (zai-org/GLM-5.3) is still in the catalog; no product change is needed.

Assistant-model: Claude Opus 5.5
Assistant-workflow: inline
Assistant-duration: 30m converged, estimated 15m
Assistant-verification: bun typecheck failed before and passed after: packages/ai typecheck on origin/main 35bee21 with a freshly generated catalog (2 TS2345 -> 0)
Assistant-verification: vitest packages/ai passed: baseten-models (5 tests)
Assistant-verification: npm run check passed: biome, typecheck, shrinkwrap
Assistant-verification: npm run test --workspace=@bastani/atomic passed except 1 load-sensitive codemode timing test, which passed alone twice (codemode-worker-admission, 8/8)
User-preference: when models.dev removes a model, update the tests that name it rather than decoupling tests from live IDs, so drift stays visible (matches pi)
Co-authored-by: Norin Lavaee nlavaee@umich.edu

RetriggerView in GreptileConfidence Score: 5/5

Safe to merge based on the reviewed changes: they update test model references and catalog expectations without changing product behavior.

Summary

Updates Baseten tests after models.dev removed older models.

  • Baseten AI tests use Kimi-K3 while keeping toggle-only reasoning covered.
  • The coding-agent test expects only Baseten model IDs still listed.

No actionable issues or changed-line rule violations were found. No merge-blocking issues remain.

Reviews (2) · Last reviewed commit: "test(coding-agent): match the Baseten ca..." · Reviewed by Greptile

models.dev dropped moonshotai/Kimi-K2.6 from Baseten (Kimi-K3 replaces it), so
two packages/ai tests no longer typecheck on a fresh build.

- image-tool-result.test.ts: the live Baseten provider block (runs only with
  BASETEN_API_KEY) now uses Kimi-K3, which also accepts image input.
- baseten-models.test.ts: the toggle-only reasoning test used K2.6 as its
  example. No current Baseten model is toggle-only, but the path is still live
  in the generator and the runtime. The test now builds a toggle-only model on
  Kimi-K3 and keeps its runtime assertions: supported levels off/high,
  chat_template_args enable_thinking true, and no reasoning_effort. The two
  assertions that only restated the fixture were dropped, so the generator's
  toggle-only mapping has no live model to check until Baseten lists one again.

This follows pi's practice of updating tests when a model leaves the catalog
(pi e86b0547, 99fe1489), so that drift stays visible. pi still has the K2.6 test.

Assistant-model: Claude Opus 5.5
Assistant-workflow: inline
Assistant-duration: 30m converged, estimated 15m
Assistant-verification: bun typecheck failed before and passed after: packages/ai typecheck on origin/main 35bee21 with a freshly generated catalog (2 TS2345 -> 0)
Assistant-verification: vitest packages/ai passed: baseten-models (5 tests)
Assistant-verification: npm run check passed: biome, typecheck, shrinkwrap
Assistant-verification: npm run test --workspace=@bastani/atomic passed except 1 load-sensitive codemode timing test, which passed alone twice (codemode-worker-admission, 8/8)
User-preference: when models.dev removes a model, update the tests that name it rather than decoupling tests from live IDs, so drift stays visible (matches pi)
Co-authored-by: Norin Lavaee <nlavaee@umich.edu>
greptile-apps[bot]
greptile-apps Bot previously approved these changes Oct 21, 2026
…vals

models.dev removed ten Baseten models in the hours after the previous commit:
DeepSeek-V4-Pro, Kimi-K2.5, Kimi-K2.6, Kimi-K2.7-Code, Nemotron-120B-A12B,
inkling, inkling-small, GLM-4.7, GLM-5 and GLM-5.1. The provider test asserts
the exact Baseten model list, so it failed in CI. The expected list now matches
the current catalog. The Baseten default (zai-org/GLM-5.3) and Kimi-K3 are
still listed.

Assistant-model: Claude Opus 5.5
Assistant-workflow: inline
Assistant-verification: vitest coding-agent failed in CI and passed locally after: baseten-qwen-provider (6 tests) against a catalog regenerated after the removal
Assistant-verification: npm run check passed: pre-commit hook
Co-authored-by: Norin Lavaee <nlavaee@umich.edu>
@greptile-apps
greptile-apps Bot dismissed their stale review October 21, 2026 05:28

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

@flora131
flora131 merged commit 2e217d5 into main Oct 21, 2026
25 of 27 checks passed
flora131 added a commit that referenced this pull request Oct 21, 2026
…artupFailure's public signature

`isExtensionStartupFailure` is exported from @bastani/atomic, and its signature
exposed the internal `seen` set used to stop cycles in nested AggregateErrors.
The recursion now lives in a private helper, and the public function takes only
the error. Behaviour is unchanged.

Assistant-model: Claude Opus 5.5
Assistant-workflow: inline (review follow-up for goal run 38cbac46-c203-49d9-9675-ad67a3613264)
Assistant-verification: vitest unit passed: workflow-stage-extension-startup-fallback, stage-runner-thrown-retry, model-fallback-classifier-conformance (44 tests)
Assistant-verification: npm run check passed: pre-commit hook after rebase onto #3816
Co-authored-by: Norin Lavaee <nlavaee@umich.edu>
flora131 added a commit that referenced this pull request Oct 21, 2026
…artupFailure's public signature

`isExtensionStartupFailure` is exported from @bastani/atomic, and its signature
exposed the internal `seen` set used to stop cycles in nested AggregateErrors.
The recursion now lives in a private helper, and the public function takes only
the error. Behaviour is unchanged.

Assistant-model: Claude Opus 5.5
Assistant-workflow: inline (review follow-up for goal run 38cbac46-c203-49d9-9675-ad67a3613264)
Assistant-verification: vitest unit passed: workflow-stage-extension-startup-fallback, stage-runner-thrown-retry, model-fallback-classifier-conformance (44 tests)
Assistant-verification: npm run check passed: pre-commit hook after rebase onto #3816
Co-authored-by: Norin Lavaee <nlavaee@umich.edu>
flora131 added a commit that referenced this pull request Oct 21, 2026
…e caller timed out

A refresh's caller stops waiting after OAUTH_REFRESH_TIMEOUT_MS (request auth
after REQUEST_AUTH_PREPARATION_TIMEOUT_MS, 15 s), while the token exchange,
its commit and the late persist kept running untracked. A process that then
exited through process.exit (RPC shutdown, which the interactive engine and
subagent runners use; `atomic auth`; package commands; the interactive and
print-mode signal handlers) dropped a rotation the provider had already
completed, leaving the replaced refresh token in auth.json, and the next
process got invalid_grant. With a token endpoint that never answered,
`atomic -p` did not exit at all: the exchange, which Atomic no longer aborts,
kept the event loop alive (still running 176 s after its request failed).

- pi-ai tracks every started token exchange until its commit or late persist
  settles. New export drainPendingOAuthRotations(timeoutMs =
  OAUTH_SHUTDOWN_DRAIN_MS) waits for them, at most OAUTH_SHUTDOWN_DRAIN_MS
  (3 s), resolves true when none is left and false when the bound passes,
  and never rejects. Abrupt termination (SIGKILL, a crash, power loss) is out
  of scope and documented at the constant: /login <provider> recovers.
- AgentSessionRuntime.dispose(), which interactive quit and signals, RPC
  shutdown and print mode (end and SIGTERM/SIGHUP) all run before exiting,
  drains within the bound measured from the start of disposal, so an
  interactive engine stays inside the host's 5 s ENGINE_SHUTDOWN_GRACE_MS.
  `atomic auth` and package commands drain before their process.exit. After
  print mode, an exchange still running past the drain is hung, so the
  process exits instead of waiting on it.
- Not covered on purpose: crash and dead-terminal exits (uncaughtCrash,
  emergencyTerminalExit, handleFatalRuntimeError), a repeated signal during
  RPC shutdown, startup-error exits before a runtime exists, and SIGINT in
  print and RPC mode, which Atomic does not handle. `atomic config` and
  `atomic mcp` end by natural exit.
- Docs: the extension API reference, custom-provider OAuth guide and the
  ExtensionOAuthConfig.refreshToken JSDoc no longer describe a 60-second
  rotation timeout signal. The signal is never aborted on a timeout, a late
  result is saved, a normal exit waits briefly, and refreshers must not add
  their own abort. providers.md (Token Refresh) and sdk.md
  (runtime.dispose) describe the exit wait and the /login recovery.

Assistant-workflow: goal (run c69320d9-174e-43d4-bab7-e8841600cd91)
Assistant-model: Claude Opus 5.5
Assistant-verification: npx vitest --run test/auth-storage-oauth-rotation-recovery.test.ts -t "graceful shutdown|hung token exchange" (packages/coding-agent) against af26e9d66 sources (dist rebuilt, OAUTH_SHUTDOWN_DRAIN_MS pinned to 3000 in the test for that run only) failed: 2 of 2 (real `atomic --mode rpc` child, stdin closed after "Request authentication timed out", stored 'fake-refresh-1' while the server had rotated to 'fake-refresh-2'; `atomic -p` still running 18000 ms after its request failed)
Assistant-verification: npx vitest --run test/oauth-shutdown-drain.test.ts (packages/ai) against af26e9d66 sources failed: 2 of 2 (drainPendingOAuthRotations is not a function)
Assistant-verification: the same tests passed after the fix on 3 consecutive runs: packages/coding-agent recovery and provider-lock-wait files 8 of 8; packages/ai oauth-shutdown-drain, models-runtime, anthropic-oauth, oauth-refresh-built-in-providers and kimi-coding-oauth 80 of 80
Assistant-verification: multi-process harness `bun run run.ts timeout` (LINGER_MS=0, child exits through drainPendingOAuthRotations as Atomic's exit paths do) CONSISTENT: late rotation persisted, later process got fake-refresh-2; EXIT_PATH=abrupt control (immediate process.exit, the documented SIGKILL/crash limit) ROTATION LOST with later invalid_grant
Assistant-verification: npm run check steps passed on the pinned 18:59 catalog (restored from the main checkout after the worktree's catalog was found regenerated at 21:54): biome check --error-on-warnings (3,532 files, 8 pre-existing infos), build:offline, pi-ai typecheck (448 files), alias-pi-ai, root bun typecheck (2,818 files), @bastani/atomic typecheck (825 and 832 files), check:shrinkwrap
Assistant-verification: npm run test:unit passed: 959 files, 11224 tests (24 skipped); an earlier run failed 2 tests in test/unit/interactive-engine-extension-default-fallback.test.ts with "(project settings) Lock file is already being held" (repo-root cwd shared with parallel tests), which passed alone twice and in the full rerun
Assistant-verification: npm run test --workspace=@bastani/atomic passed: 745 files, 6547 tests (52 skipped)
Assistant-verification: npm run test --workspace=@bastani/pi-ai passed: 192 files, 1991 tests (863 skipped)
Assistant-verification: rebased onto origin/main 2e217d5 (includes #3815 and #3816); one conflict in packages/coding-agent/CHANGELOG.md resolved by keeping main's Unreleased entries and appending this branch's under the single ### Fixed
Assistant-verification: npm run build passed (live models.dev catalog generated 2026-10-21T05:52:31Z)
Assistant-verification: after the rebase, npm run test:unit passed: 961 files, 11237 tests (24 skipped)
Assistant-verification: after the rebase, npm run test --workspace=@bastani/atomic passed: 747 files, 6567 tests (52 skipped)
Assistant-verification: npm run check passed after rebase onto origin/main 48f6fe2 (#3819 Together drift fix): biome 3,539 files, typecheck 448/2,823/825/832 files, shrinkwrap up to date, live models.dev catalog
Assistant-verification: vitest suites passed after rebase: npm run test:unit 961 files 11237 tests (24 skipped); npm run test --workspace=@bastani/atomic 747 files 6567 tests (52 skipped); npm run test --workspace=@bastani/pi-ai 192 files 1991 tests (863 skipped)
Assistant-duration: 3h53m converged, estimated 3h
Co-authored-by: Norin Lavaee <nlavaee@umich.edu>
flora131 added a commit that referenced this pull request Oct 21, 2026
settleAgentRun released the open-run marker only after awaiting the agent
event queue and the custom-message flush. When either rejected (for example a
session listener throwing on agent_end), the marker, the active prompt count,
and the deferred settled actions all leaked. Every later interactive message
was then shown as queued but never delivered.

The run is now always closed and settled in a finally block. Queued-input
delivery also tolerates an event failure already reported by the earlier run,
so a deferred message is no longer silently dropped.

Assistant-model: Claude Opus 5.5 (debugger subagent; the requested gpt-6.1-sol-ultrafast override was not applied)
Assistant-workflow: inline (debugger subagent; goal run 29e672f9 stopped by the user)
Assistant-duration: 6h55m converged (goal run 6h30m until stopped by the user, then 24m inline), estimated 2h
Assistant-verification: new settle-failure tests in interactive-busy-submission.test.ts fail before the fix (P2 stranded in steering) and pass after (16/16)
Assistant-verification: vitest interactive-busy-submission, interactive-busy-submission-record, suite/agent-session-queue: 140/140 pass
Assistant-verification: npm run check passes; npm run test --workspace=@bastani/atomic: 6724 pass, 0 fail
Assistant-verification: branch rebased onto origin/main (picks up model-catalog fixes #3815, #3816, #3819); CHANGELOG conflicts merged into one section per heading; non-CHANGELOG branch diff unchanged
Co-authored-by: Norin Lavaee <nlavaee@umich.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant