Repository navigation
test(ai): move Baseten tests off Kimi-K2.6, which models.dev removed - #3816
Merged
Merged
Conversation
models.dev dropped moonshotai/Kimi-K2.6 from Baseten (Kimi-K3 replaces it), so two packages/ai tests no longer typecheck on a fresh build. - image-tool-result.test.ts: the live Baseten provider block (runs only with BASETEN_API_KEY) now uses Kimi-K3, which also accepts image input. - baseten-models.test.ts: the toggle-only reasoning test used K2.6 as its example. No current Baseten model is toggle-only, but the path is still live in the generator and the runtime. The test now builds a toggle-only model on Kimi-K3 and keeps its runtime assertions: supported levels off/high, chat_template_args enable_thinking true, and no reasoning_effort. The two assertions that only restated the fixture were dropped, so the generator's toggle-only mapping has no live model to check until Baseten lists one again. This follows pi's practice of updating tests when a model leaves the catalog (pi e86b0547, 99fe1489), so that drift stays visible. pi still has the K2.6 test. Assistant-model: Claude Opus 5.5 Assistant-workflow: inline Assistant-duration: 30m converged, estimated 15m Assistant-verification: bun typecheck failed before and passed after: packages/ai typecheck on origin/main 35bee21 with a freshly generated catalog (2 TS2345 -> 0) Assistant-verification: vitest packages/ai passed: baseten-models (5 tests) Assistant-verification: npm run check passed: biome, typecheck, shrinkwrap Assistant-verification: npm run test --workspace=@bastani/atomic passed except 1 load-sensitive codemode timing test, which passed alone twice (codemode-worker-admission, 8/8) User-preference: when models.dev removes a model, update the tests that name it rather than decoupling tests from live IDs, so drift stays visible (matches pi) Co-authored-by: Norin Lavaee <nlavaee@umich.edu>
…vals models.dev removed ten Baseten models in the hours after the previous commit: DeepSeek-V4-Pro, Kimi-K2.5, Kimi-K2.6, Kimi-K2.7-Code, Nemotron-120B-A12B, inkling, inkling-small, GLM-4.7, GLM-5 and GLM-5.1. The provider test asserts the exact Baseten model list, so it failed in CI. The expected list now matches the current catalog. The Baseten default (zai-org/GLM-5.3) and Kimi-K3 are still listed. Assistant-model: Claude Opus 5.5 Assistant-workflow: inline Assistant-verification: vitest coding-agent failed in CI and passed locally after: baseten-qwen-provider (6 tests) against a catalog regenerated after the removal Assistant-verification: npm run check passed: pre-commit hook Co-authored-by: Norin Lavaee <nlavaee@umich.edu>
greptile-apps
Bot
dismissed
their stale review
October 21, 2026 05:28
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
This was referenced Oct 21, 2026
flora131
added a commit
that referenced
this pull request
Oct 21, 2026
…artupFailure's public signature `isExtensionStartupFailure` is exported from @bastani/atomic, and its signature exposed the internal `seen` set used to stop cycles in nested AggregateErrors. The recursion now lives in a private helper, and the public function takes only the error. Behaviour is unchanged. Assistant-model: Claude Opus 5.5 Assistant-workflow: inline (review follow-up for goal run 38cbac46-c203-49d9-9675-ad67a3613264) Assistant-verification: vitest unit passed: workflow-stage-extension-startup-fallback, stage-runner-thrown-retry, model-fallback-classifier-conformance (44 tests) Assistant-verification: npm run check passed: pre-commit hook after rebase onto #3816 Co-authored-by: Norin Lavaee <nlavaee@umich.edu>
flora131
added a commit
that referenced
this pull request
Oct 21, 2026
…artupFailure's public signature `isExtensionStartupFailure` is exported from @bastani/atomic, and its signature exposed the internal `seen` set used to stop cycles in nested AggregateErrors. The recursion now lives in a private helper, and the public function takes only the error. Behaviour is unchanged. Assistant-model: Claude Opus 5.5 Assistant-workflow: inline (review follow-up for goal run 38cbac46-c203-49d9-9675-ad67a3613264) Assistant-verification: vitest unit passed: workflow-stage-extension-startup-fallback, stage-runner-thrown-retry, model-fallback-classifier-conformance (44 tests) Assistant-verification: npm run check passed: pre-commit hook after rebase onto #3816 Co-authored-by: Norin Lavaee <nlavaee@umich.edu>
flora131
added a commit
that referenced
this pull request
Oct 21, 2026
…e caller timed out A refresh's caller stops waiting after OAUTH_REFRESH_TIMEOUT_MS (request auth after REQUEST_AUTH_PREPARATION_TIMEOUT_MS, 15 s), while the token exchange, its commit and the late persist kept running untracked. A process that then exited through process.exit (RPC shutdown, which the interactive engine and subagent runners use; `atomic auth`; package commands; the interactive and print-mode signal handlers) dropped a rotation the provider had already completed, leaving the replaced refresh token in auth.json, and the next process got invalid_grant. With a token endpoint that never answered, `atomic -p` did not exit at all: the exchange, which Atomic no longer aborts, kept the event loop alive (still running 176 s after its request failed). - pi-ai tracks every started token exchange until its commit or late persist settles. New export drainPendingOAuthRotations(timeoutMs = OAUTH_SHUTDOWN_DRAIN_MS) waits for them, at most OAUTH_SHUTDOWN_DRAIN_MS (3 s), resolves true when none is left and false when the bound passes, and never rejects. Abrupt termination (SIGKILL, a crash, power loss) is out of scope and documented at the constant: /login <provider> recovers. - AgentSessionRuntime.dispose(), which interactive quit and signals, RPC shutdown and print mode (end and SIGTERM/SIGHUP) all run before exiting, drains within the bound measured from the start of disposal, so an interactive engine stays inside the host's 5 s ENGINE_SHUTDOWN_GRACE_MS. `atomic auth` and package commands drain before their process.exit. After print mode, an exchange still running past the drain is hung, so the process exits instead of waiting on it. - Not covered on purpose: crash and dead-terminal exits (uncaughtCrash, emergencyTerminalExit, handleFatalRuntimeError), a repeated signal during RPC shutdown, startup-error exits before a runtime exists, and SIGINT in print and RPC mode, which Atomic does not handle. `atomic config` and `atomic mcp` end by natural exit. - Docs: the extension API reference, custom-provider OAuth guide and the ExtensionOAuthConfig.refreshToken JSDoc no longer describe a 60-second rotation timeout signal. The signal is never aborted on a timeout, a late result is saved, a normal exit waits briefly, and refreshers must not add their own abort. providers.md (Token Refresh) and sdk.md (runtime.dispose) describe the exit wait and the /login recovery. Assistant-workflow: goal (run c69320d9-174e-43d4-bab7-e8841600cd91) Assistant-model: Claude Opus 5.5 Assistant-verification: npx vitest --run test/auth-storage-oauth-rotation-recovery.test.ts -t "graceful shutdown|hung token exchange" (packages/coding-agent) against af26e9d66 sources (dist rebuilt, OAUTH_SHUTDOWN_DRAIN_MS pinned to 3000 in the test for that run only) failed: 2 of 2 (real `atomic --mode rpc` child, stdin closed after "Request authentication timed out", stored 'fake-refresh-1' while the server had rotated to 'fake-refresh-2'; `atomic -p` still running 18000 ms after its request failed) Assistant-verification: npx vitest --run test/oauth-shutdown-drain.test.ts (packages/ai) against af26e9d66 sources failed: 2 of 2 (drainPendingOAuthRotations is not a function) Assistant-verification: the same tests passed after the fix on 3 consecutive runs: packages/coding-agent recovery and provider-lock-wait files 8 of 8; packages/ai oauth-shutdown-drain, models-runtime, anthropic-oauth, oauth-refresh-built-in-providers and kimi-coding-oauth 80 of 80 Assistant-verification: multi-process harness `bun run run.ts timeout` (LINGER_MS=0, child exits through drainPendingOAuthRotations as Atomic's exit paths do) CONSISTENT: late rotation persisted, later process got fake-refresh-2; EXIT_PATH=abrupt control (immediate process.exit, the documented SIGKILL/crash limit) ROTATION LOST with later invalid_grant Assistant-verification: npm run check steps passed on the pinned 18:59 catalog (restored from the main checkout after the worktree's catalog was found regenerated at 21:54): biome check --error-on-warnings (3,532 files, 8 pre-existing infos), build:offline, pi-ai typecheck (448 files), alias-pi-ai, root bun typecheck (2,818 files), @bastani/atomic typecheck (825 and 832 files), check:shrinkwrap Assistant-verification: npm run test:unit passed: 959 files, 11224 tests (24 skipped); an earlier run failed 2 tests in test/unit/interactive-engine-extension-default-fallback.test.ts with "(project settings) Lock file is already being held" (repo-root cwd shared with parallel tests), which passed alone twice and in the full rerun Assistant-verification: npm run test --workspace=@bastani/atomic passed: 745 files, 6547 tests (52 skipped) Assistant-verification: npm run test --workspace=@bastani/pi-ai passed: 192 files, 1991 tests (863 skipped) Assistant-verification: rebased onto origin/main 2e217d5 (includes #3815 and #3816); one conflict in packages/coding-agent/CHANGELOG.md resolved by keeping main's Unreleased entries and appending this branch's under the single ### Fixed Assistant-verification: npm run build passed (live models.dev catalog generated 2026-10-21T05:52:31Z) Assistant-verification: after the rebase, npm run test:unit passed: 961 files, 11237 tests (24 skipped) Assistant-verification: after the rebase, npm run test --workspace=@bastani/atomic passed: 747 files, 6567 tests (52 skipped) Assistant-verification: npm run check passed after rebase onto origin/main 48f6fe2 (#3819 Together drift fix): biome 3,539 files, typecheck 448/2,823/825/832 files, shrinkwrap up to date, live models.dev catalog Assistant-verification: vitest suites passed after rebase: npm run test:unit 961 files 11237 tests (24 skipped); npm run test --workspace=@bastani/atomic 747 files 6567 tests (52 skipped); npm run test --workspace=@bastani/pi-ai 192 files 1991 tests (863 skipped) Assistant-duration: 3h53m converged, estimated 3h Co-authored-by: Norin Lavaee <nlavaee@umich.edu>
This was referenced Oct 21, 2026
flora131
added a commit
that referenced
this pull request
Oct 21, 2026
settleAgentRun released the open-run marker only after awaiting the agent event queue and the custom-message flush. When either rejected (for example a session listener throwing on agent_end), the marker, the active prompt count, and the deferred settled actions all leaked. Every later interactive message was then shown as queued but never delivered. The run is now always closed and settled in a finally block. Queued-input delivery also tolerates an event failure already reported by the earlier run, so a deferred message is no longer silently dropped. Assistant-model: Claude Opus 5.5 (debugger subagent; the requested gpt-6.1-sol-ultrafast override was not applied) Assistant-workflow: inline (debugger subagent; goal run 29e672f9 stopped by the user) Assistant-duration: 6h55m converged (goal run 6h30m until stopped by the user, then 24m inline), estimated 2h Assistant-verification: new settle-failure tests in interactive-busy-submission.test.ts fail before the fix (P2 stranded in steering) and pass after (16/16) Assistant-verification: vitest interactive-busy-submission, interactive-busy-submission-record, suite/agent-session-queue: 140/140 pass Assistant-verification: npm run check passes; npm run test --workspace=@bastani/atomic: 6724 pass, 0 fail Assistant-verification: branch rebased onto origin/main (picks up model-catalog fixes #3815, #3816, #3819); CHANGELOG conflicts merged into one section per heading; non-CHANGELOG branch diff unchanged Co-authored-by: Norin Lavaee <nlavaee@umich.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
models.dev removed
moonshotai/Kimi-K2.6from Baseten (Kimi-K3 replaces it), so twopackages/aitests no longer typecheck on a fresh build: onmainand on every branch. This updates them, following pi's practice of updating tests when a model leaves the catalog (pie86b0547,99fe1489), so drift stays visible. pi still has the K2.6 test.Changes
image-tool-result.test.ts: the live Baseten provider block (runs only withBASETEN_API_KEY) uses Kimi-K3, which also accepts image input.baseten-models.test.ts: the toggle-only reasoning test used K2.6 as its example. No current Baseten model is toggle-only, but the path is still live in the generator and runtime, so the test builds a toggle-only model on Kimi-K3 and keeps its runtime assertions (levelsoff/high,chat_template_args.enable_thinking: true, noreasoning_effort).baseten-qwen-provider.test.ts(coding-agent): models.dev removed ten more Baseten models after the first commit (DeepSeek-V4-Pro, Kimi-K2.5/K2.6/K2.7-Code, Nemotron-120B-A12B, both Inkling models, GLM-4.7, GLM-5, GLM-5.1). The exact-list assertion now matches the current 14-model catalog.Notes
codemode-worker-admission"holds shared capacity until actual worker exit after timeout" on Windows. That test is load-sensitive and unrelated to this change (it passes in isolation). It is tracked separately.zai-org/GLM-5.3) is still in the catalog; no product change is needed.Assistant-model: Claude Opus 5.5
Assistant-workflow: inline
Assistant-duration: 30m converged, estimated 15m
Assistant-verification: bun typecheck failed before and passed after: packages/ai typecheck on origin/main 35bee21 with a freshly generated catalog (2 TS2345 -> 0)
Assistant-verification: vitest packages/ai passed: baseten-models (5 tests)
Assistant-verification: npm run check passed: biome, typecheck, shrinkwrap
Assistant-verification: npm run test --workspace=@bastani/atomic passed except 1 load-sensitive codemode timing test, which passed alone twice (codemode-worker-admission, 8/8)
User-preference: when models.dev removes a model, update the tests that name it rather than decoupling tests from live IDs, so drift stays visible (matches pi)
Co-authored-by: Norin Lavaee nlavaee@umich.edu


Confidence Score: 5/5
Safe to merge based on the reviewed changes: they update test model references and catalog expectations without changing product behavior.Summary
Updates Baseten tests after models.dev removed older models.
No actionable issues or changed-line rule violations were found. No merge-blocking issues remain.
Reviews (2) · Last reviewed commit: "test(coding-agent): match the Baseten ca..." · Reviewed by Greptile