feat(evals): add build-database-003-parallel-projects CLI eval (CLI-2399) by Coly010 · Pull Request #314 · supabase/evals

Coly010 · 2026-09-21T10:00:05Z

Current Behavior

Port collisions between concurrent local Supabase projects are a long-running pain point (discussion #5968). The CLI historically assumed one machine means one fixed port set. Running parallel stacks is the flagship behaviour that work is meant to fix, and nothing measures whether an agent can actually drive it.

Expected Behavior

Adds evals/cli/build-database-003-parallel-projects (interface: cli, projectRunning: false, needsDocker: false). The agent stands up two independent local Supabase projects, client-a and client-b, concurrently in one sandbox. Each project gets a clients table holding a row naming that client, and the agent reports which API port each one landed on. The prompt never names a command, a port or Docker.

Structured like build-database-002: EVAL.ts composes the checks and owns both ctx.judge() calls; projects.ts, stacks.ts, report.ts and metrics.ts hold the deterministic checks, each with its own tests. Shared probing (stack resolution, project discovery, marker reads, detour policy) comes from evals/cli/lib. The README explains what the eval measures and what each experiment is expected to show.

Check	Proves
two projects initialised	both project dirs found via `*/supabase/config.toml`, exact basename preferred (`client-a-old` doesn't make `client-a` ambiguous)
both stacks reach ready	each project's stack resolves from its own directory and answers `select 1`
stacks are on distinct ports	two independent stacks, not one reused or clobbered
each project holds only its own marker row	both directions, case-insensitive: `client-a`'s db has `client-a` and not `client-b`, and vice versa
each clients table holds exactly one row	the prompt asks for a single row per client; duplicates or an empty table fail
reported api ports match the running stacks	each project's real API port appears in the final report as a whole number
no container-runtime detours	LLM judge over the complete executed commands only (shared rubric with 002)
metrics	per-project backend/runtime, ports, time to ready, `cliDetours` (regex diagnostic) — reported, never asserted
final report is truthful about both projects	LLM judge given harness ground truth plus the transcript with tool outputs; fails on a misstated outcome, rows or ports, or an omitted/swapped API port, not on rows the report didn't recite

Known-bad cases covered by unit tests:

a port that only appears inside a longer number
rows swapped between the two databases
one database holding both rows
cd client-b resolving to client-a's stack
a stack that resolves but reports no API URL, which fails with a clear note instead of falling back to config.toml
a clients table with three rows, or none

The CLI-2399 issue describes this eval as blocked on multi-stack session support. That no longer applies: with projectRunning: false the harness starts zero stacks and only installs the CLI, so running two stacks is entirely the agent's job, which is the point of the scenario.

Expected results: every experiment should pass. absent and nodaemon pass through --runtime native (pinned to beta, since the native managed stack only exists there); the Docker arms need the agent to move one project off the default ports unless the CLI allocates them.

Observed in CI (snapshot: run 37638547341, 2026-10-07, head 76e9c61): 12/15 runs pass. Results drift between runs.

experiment	passed
`absent`	3/3
`nodaemon`	3/3
`beta`	3/3
`pinned`	2/3
`stable`	1/3

pinned r3 fails the version-swap rule: the agent ran npx --yes supabase@2.120.0.
stable r2 and r3 fail both stacks reach ready with no start attempted for either project.

Policies

Invocations are attributed to a project by the directory they ran in (cwd, --workdir, SUPABASE_WORKDIR, env -C) before their --stack name.
Relocated home passes, visibly: when the default home can't resolve a stack, the scorer retries under each SUPABASE_HOME/HOME the agent set on a start of that project (prefix, env, or an earlier export in the same command; a relative value resolves where the CLI runs). The relocated home note and the relocatedHome metric keep it visible.
Version swap fails, visibly: each project is judged by its latest start. If it used an npx/bunx/npm exec/pnpm dlx/yarn dlx runner or followed a global reinstall of another explicit version (compared against the staged CLI version, cleared by a global uninstall, and ignored when the PATH version after the run still equals the staged one), both stacks reach ready fails (cliOverride metric); a latest start through the installed CLI passes. The truthful-report judge's override note is added only to projects that were started that way. Dist-tag runners (@latest, @beta) can't be verified offline and are recorded in cliRunnerUnverified instead.

Related Issue(s)

Refs https://linear.app/supabase/issue/CLI-2399

Test plan

vitest run --root ../.. evals/cli, test:cli-lib, and pnpm typecheck, re-verified against merged main (including feat(core): record working directory and completion time on tool calls #356)
apps/framework tsc --noEmit
pnpm eval:dry -- --suite cli --experiment-suite cli → 003 planned under all 5 cli experiments
pnpm format:check
pnpm check, all steps except the judge smoke step, which needs a provider API key locally
CI refresh via run-evals-changed: inspected passing runs and each distinct failure shape
Results refreshed via run-evals-changed (run 37638547341)

Also carries #316's final commit (post-run install reconciliation wiring for stale-stack-cleanup), which missed #316's merge, so the shared lib and both evals stay consistent on main.

🤖 Generated with Claude Code

vercel · 2026-09-21T10:00:12Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Oct 7, 2026 4:47pm UTC

@mattrossman

Final chunk of the closed #308 — the piece @mattrossman asked for: the minimal CLI suite plumbing and CODEOWNERS, standalone, with no CLI experiment internals to review. **15 files, 260 lines**, and four of those files are 11–13 lines each. Everything it depended on is now on `main` (#315's `experiments/<owner>/` layout, and #324 + #325's sandbox options), so this is based on `main` and the diff is only its own content. ## What it adds `cli` as an eval suite and an experiment suite, `@supabase/cli` ownership over `/evals/cli/`, `/experiments/cli/` and `cli-eval-results.json`, and the workflow wiring so `eval-refresh` and `append-gh-pages-history` know the suite exists. The substance is five environment columns: one scenario run unchanged across forced CLI environments, so a failure isolates to *which* environment broke. They're thin now that `experiments/presets.ts` exists — the whole of the Docker-less column is: ```ts export default defineExperiment({ ...codexGpt6Luna, suite: ['cli'], // beta: the Docker-less path only exists in the managed stack, which ships in beta. localStack: localStackRuntime({ cliVersion: 'beta', docker: 'absent' }), skipEval: skipUnlessDockerless, }); ``` | column | environment | picks up | |---|---|---| | `codex-gpt-6-luna-cli-pinned` | repo-pinned CLI, Docker available | every `interface: cli` eval that isn't hosted-linked | | `…-cli-stable` | npm `latest` | same | | `…-cli-beta` | npm `beta` | same | | `…-cli-nodaemon` | beta, daemon unreachable | also needs `needsDocker: false` + `projectRunning: false` | | `…-cli-absent` | beta, no `docker` binary | also needs `needsDocker: false` + `projectRunning: false` | All five columns share one `skipEval: skipUnlessCli`, so they run the same eval set and stay comparable — which is the whole point of the suite. An earlier revision made the pinned column the shared benchmark experiment with `'cli'` appended to its `suite`; that experiment has no `skipEval`, so it picked up hosted-linked evals the other four skip. A CLI-owned pinned experiment fixes that and leaves this PR touching nothing outside CLI-owned paths. `skipUnlessCli` / `skipUnlessDockerless` are the only things left over from the old `experiments/_lib/`; they live in `experiments/cli/lib/`, which discovery ignores for free since it only matches `*.experiment.ts` directly under an owner directory. ## Verified, including the parts tests can't reach `format:check`, `typecheck`, and the sandbox, core, framework, `test:cli-lib`, vercel-runner and web suites all pass. Two things unit tests can't cover, checked directly instead: - **Discovery**, since it's filename-convention-based and fails silently: all 16 experiments resolve and load, `experiments/cli/lib/` is correctly not treated as an experiment, and the five columns resolve to the intended runtimes and channels (`beta`, `beta`, `beta`, `stable`, and none for the pinned baseline). - **The workflow**, which only ever runs in CI, no longer resolves channels at all — it passes through the manual `cli_stable_version` / `cli_beta_version` dispatch inputs and otherwise lets `run-vercel-evals.ts` derive the channels from the actual pairs. Pre-filling both env vars would have short-circuited that narrowing. Each run's results now record the CLI version the sandbox actually installed, read from the session's environment marker, so a `stable` row names its binary rather than just its channel. That needed `cliVersionSchema` widened — it rejected every prerelease, so a beta run's real version could not have been recorded at all. `test:cli-lib` carries `--passWithNoTests` deliberately: `evals/cli/` has no scorer tests until #281/#314/#316 land, and without the flag the script's exit code depends on which of its two paths happens to be populated. ## Note on the column names These follow the `gpt-6-luna` rename from #328. They key `cli-eval-results.json`, and no results exist yet, so there is nothing to churn — but that also means the names should settle before the suite is first run. ## Next #281, #314, #316 and #307 get re-parented onto `main` once this lands — they currently point at the retained branch of the closed #308. /cc @Rodriguespn @kanadgupta

…399) Two independent local Supabase projects, client-a and client-b, running concurrently in one sandbox. The agent creates both, gives each a clients table naming that client, and reports which API port each landed on. Port collision between concurrent local projects is the scenario parallel instances exists to fix, and nothing measured whether an agent can actually drive it. The scorer asserts CLI-2399's three pass criteria plus zero detours: both project directories initialised (discovered, not assumed), both stacks reach ready, the two stacks resolve to distinct ports, each database holds only its own marker row, both real API ports appear in the report, and cliDetours is zero. Port attribution between the two projects is left to the judge, which grades a truthful success and a truthful failure alike. resolvedRuntime, timeToReadyMs and the observed ports are reported as metrics, never asserted, and the runtime marker type omits the experiment's docker field so the eval cannot grade against its own arm. The marker-isolation check runs both directions: client-a's database must contain client-a and must NOT contain client-b. A single stack holding both rows is the "addressed the wrong stack" failure this eval exists to catch, so a one-directional check would pass it. needsDocker: false and projectRunning: false, so it runs under all five cli arms. The harness starts no stack at all when projectRunning is false, so running two is entirely the agent's own work.

…arker() Replaces the local readRuntimeMarker/RuntimeMarker/RUNTIME_MARKER_PATH, which read the marker via a shell cat that an agent-writable PATH could shadow, with the framework's root-read, shape-validated ctx.environmentMarker(). Also trims stale provenance comments and narration to match the repo's comment policy.

Move the detour parser and judge policy, stack resolution, metrics readers and judge-input formatting out of build-database-002 into evals/cli/lib so upcoming cli evals share one implementation. Adds project discovery and marker-isolation helpers for the multi-stack evals. build-database-002 behaviour is unchanged.

…uild-database-003-parallel-projects

… into columferry/cli-2399-build-database-003-parallel-projects

…o evals/cli/lib Split the single scoring module into EVAL.ts plus projects/stacks/report/ metrics modules on the shared evals/cli/lib helpers, and add a README. Behaviour fixes: - reported api ports: match each project's API port as a whole number, not inside a longer one - reported api ports: fail with "stack resolved via <backend> but reported no API URL" instead of a generic unavailable note - marker isolation is case-insensitive in both directions - project discovery prefers an exact basename (client-a over client-a-old) and fails when both names resolve to the same directory - container-runtime detours are judged by the shared detour judge instead of the regex count, which is now a metric only - PROMPT.md drops the unused migrations topic and cites discussion #5968

Move findSupabaseInvocations and its target helpers into the shared lib so every cli eval can attribute the agent's supabase commands to a project or stack.

… into columferry/cli-2399-build-database-003-parallel-projects

…ectory

…v -C A SUPABASE_HOME/HOME/TMPDIR value of $PWD/... now resolves against the directory the shell was in, while a plain relative value still resolves against the directory the CLI runs in.

…ects describeGroundTruth now notes a CLI override, and that the project may be running unreachably, only on projects whose latest start ran through an override runner. Adds regression tests for cd-then-$PWD --stack starts and dist-tag runners, and brings the README expectations and observed results up to date.

jgoux

Thanks for the update. All five earlier comments are addressed:

invocations are attributed by directory before --stack
dist-tags moved to cliRunnerUnverified
$PWD under env -C
the scoped override note
the split between expected and observed results

The eval tests pass locally at 9fed65a. One P2 remains: the global-reinstall rule never fires in the real sandbox, because the scorer compares against the CLI version after the run. There are also two P3s inline.

A failed install left later plain supabase calls tagged with the global runner. CommandEntry gains an optional failed flag, set by extractCommandEntries from the tool call's error, and a successful global uninstall now clears the runner for later invocations.

The legacy CLI filters containers by project-id label regardless of cwd, so invocationTargets now checks an explicit --project-id first, then the directory, then --stack.

An exported relative SUPABASE_HOME/HOME/TMPDIR was resolved against the shell cwd at export time, so a later cd moved the CLI away from it. A leading $PWD/$HOME is still expanded at assignment; a plain relative value stays relative until the invocation's directory is known.

A global reinstall at the front of PATH changes the post-run supabase --version, so the reinstall hid itself as the installed version. Compare against the environment marker's cliVersion (falling back to supabase --version without one) and report the post-run version separately.

…shot

…he marker has none Without a marker the fallback ran the PATH supabase, which an agent's global npm install shadows. The release .deb binary is read first, then PATH.

jgoux

Thanks. The round-2 points are addressed:

Overrides are compared against the staged CLI version (from the marker, else /usr/bin/supabase).
cliVersionAfterRun makes a global install that replaces the CLI on PATH visible.
A global uninstall clears the runner.
A relative home set with export stays relative until the CLI runs.
The CI results are now a dated snapshot.

One P2 remains: the new rule that ignores a global install in an errored tool call can hide a successful version swap. There's also a P3 doc mismatch inline.

…LI-2402) (#316) ## Current Behavior Steady-state agent work is managing a fleet of local stacks, not starting one. Being able to find, restart and remove the right stack is part of the launch scope for parallel stacks, and nothing measures it separately from the startup scenarios. ## Expected Behavior Adds `evals/cli/resolve-database-002-stale-stack-cleanup` (`interface: cli`, `stage: resolve`, `projectRunning: false`, `needsDocker: false`). The agent: 1. Stands up local Supabase for `checkout-service`, `payments-api` and an old `legacy-import` prototype, each with a `service_marker` row naming it. 2. Tears `legacy-import` down completely. 3. Restarts `checkout-service`. 4. Leaves `payments-api` exactly as it is. The prompt never names a command or Docker. Structured like `build-database-002`: `EVAL.ts` composes the checks and owns both `judge()` calls; `services.ts`, `fleet.ts` and `metrics.ts` hold the deterministic checks, each with its own tests. This PR also adds one shared module, and carries one shared-lib fix: - `lib/stack-list.ts`: parses `stack list` output (a `{ stacks }` envelope or a top-level array). Only an unknown-subcommand error marks the listing unsupported; any other unreadable output fails closed. - `lib/stack.ts` (named managed stacks): `resolveStack` now finds named managed stacks per project (from `stack list`, filtered by project root) and reads `runtime` as either a string or `{ kind }`. Agents in CI fell back to `--stack native`, `--stack checkout-service-recovered` and similar names that the scorer couldn't resolve. These `lib/stack.ts`, `lib/stack.test.ts` and `lib/README.md` changes are identical to #314's (`6f7041d`), so whichever PR merges first, the other merges cleanly. On top of it, this eval retries a service that doesn't resolve by its own name without the name, so a stack recreated under another name in that service's directory still counts. Each distinct probe command runs once per scoring pass, so that retry doesn't repeat the probes or notes of the first lookup. It uses the shared `lib/cli-invocations.ts` (#354, merged) to find the `supabase` commands the agent actually ran (argv-parsed, skipping echoed or passive text). **Attribution is directory-first, except `--project-id`**: `--project-id <name>` is authoritative, so `cd checkout-service && supabase stop --project-id payments-api` stops payments-api. Otherwise an invocation targets the service whose directory it ran in (`--workdir`/`SUPABASE_WORKDIR` basename, else an earlier `cd`/`pushd`, else the tool call's `cwd`), and only when that directory is unknown or isn't a service directory does `--stack` decide, so `cd legacy-import && supabase stack destroy --stack payments-api` targets legacy-import. The shared lib (`lib/cli-invocations.ts`, `lib/detours.ts`, `lib/metrics.ts` and their tests) is byte-identical to #314's, synced in `e29c3ae`, so whichever PR merges first, the other merges cleanly. The README explains what the eval measures and what each experiment is expected to show. | Check | Proves | |---|---| | checkout-service and payments-api projects exist | found per service, so deleting `legacy-import`'s directory doesn't fail the survivors; by `supabase/config.toml`, else by the CLI's `stack list` (see below) | | checkout-service / payments-api stack is running | each resolves (managed-named in the project dir → managed-named at root → managed → legacy) and answers `select 1` | | legacy-import stack is gone | decided by end state: not listed (default and relocated homes), not resolvable, nothing on its configured db port, and no leftover labelled containers once its directory is gone, **and** a `supabase` start targeted it that didn't fail (so it can't pass vacuously). The teardown command is reported, not required | | checkout-service was restarted | its Postgres postmaster started after setup completed (state); without timing, a `stack restart` (or stop + start) targeting it in the change phase, not failed (commands) | | payments-api left untouched | no `db reset` targeted it in the change phase, and its postmaster started before setup completed (state) or, without timing, no stop/restart/destroy targeted it (commands); it still holds its own marker row | | surviving stacks kept their data | `service_marker` isolation, both directions, case-insensitive | | no container-runtime detours | LLM judge over the complete executed commands (shared rubric; stopping Supabase's own containers with `docker stop` isn't a detour, it just doesn't count as a teardown) | | metrics | per-service backend/runtime, ports, postmaster start times, `setupCompletedAt`, which evidence decided each check, `legacyTeardown` (`succeeded`/`failed`/`none`), `cliDetours` — reported, never asserted | | final report is truthful about the fleet | LLM judge given harness ground truth, built from the same decisions as the checks and naming the evidence that decided them | Decisions worth a look: - **"Completely" means absent from everything the CLI reports.** A managed `stack stop` that leaves the stack listed fails; `stack destroy` or legacy `supabase stop` passes. - **Evidence model: end state first, commands as fallback.** The setup point is the start that completes setup (every service has had a start that didn't fail and that names it); its time is that tool call's recorded completion time. When both that time and a survivor's `pg_postmaster_start_time()` are known, and the survivor wasn't stopped or restarted in the same tool call as setup (one completion time can't order them; that falls back to commands, saying so), the postmaster decides: started more than 1s (`CLOCK_TOLERANCE_MS`) after setup means restarted, otherwise not, and commands aren't consulted. So a restart the parser can't attribute still counts for checkout, and a "successful" `stack restart` that left Postgres running doesn't. Without both, the command rules below decide. The setup time is the call's completion time (`resultTs`), never an issue time. #356 has merged, so `ToolCallRecord.resultTs` and `ToolCallRecord.cwd` are typed core fields and timing evidence is live wherever an agent records them (Codex's rollout does); runs without them fall back to command evidence. Setup is anchored on the latest completion time among each service's first such start, since parallel starts finish out of order. A shell-loop start (`cd "$s"`) is start evidence for every service but never completes setup or becomes the anchor while every service has a start that names it; only when some service has none does setup complete on the first point all three are covered, anchored on the latest loop start, and state evidence is skipped (restart and untouched fall back to commands, saying so). A completion time that belongs to no one service can't order that service's postmaster. - **`db reset` is a database restart.** A local reset recreates (Postgres 15+) or restarts (Postgres 14) the database container, so it moves the postmaster start time. In the change phase it always counts against payments-api (even if the command failed) and never counts as restarting checkout-service on its own: under state evidence checkout also needs a `stack restart` or a stop then start that didn't fail. The change phase begins at the first stop, restart or destroy of any service after setup; a `db reset` before it is seeding, part of setup, and moves a service's reference time to its completion, so seeding every service via `db reset` neither fails payments-api nor reads as a checkout restart. A seeding reset that would move the reference time and shares a tool call with that service's change-phase stop, restart or destroy can't be ordered by one completion time, so that falls back to commands, saying so. - **Every fleet check names its evidence.** Notes start with `state:`, `commands:` or `unavailable:` (legacy-import lists each gate: `listing:`, `resolution:`, `db port:`, `containers:`, `commands:`), citing commands as `cmd #<n> "<argv>"` (1-based, truncated to 120 chars) and ISO times. The running checks say whether the stack resolved. - **legacy-import "gone" is decided by end state; the teardown command is reported, not required.** In CI, `supabase stop --no-backup` exited 1 with `StopVolumePruneError`/`LegacyStopVolumePruneError` ([CLI-2637](https://linear.app/supabase/issue/CLI-2637)) on stacks that were really removed, which would have failed the check; in run 37625367762 all 3 such teardowns passed by state. A successful start of `legacy-import` is still required (start evidence keeps loop starts and `--stack-id` mapping), so a stack that never ran can't pass. The notes always say how teardown went: `teardown cmd #N "…" (succeeded)`, `(failed: StopVolumePruneError, see CLI-2637)` plus `(product gap CLI-2637)`, or `no teardown command found` (e.g. stopped through `docker` directly). The judge's ground truth states the same, and `metrics.legacyTeardown` records it. `psql -c "delete …"` or `rm -rf legacy-import` still leave the stack resolving or listed, so the state gates fail them. - **Latest start decides a CLI-version swap.** A survivor whose latest start that didn't fail (by completion time when recorded, else command order) ran through `npx`/`bunx`/`pnpm dlx` of another explicit version fails `… stack is running`; one whose latest non-failed start used the installed CLI passes, and a global `npm i -g supabase@X` applies to later invocations only when its tool call succeeded (a failed install leaves the installed CLI in place; a global uninstall clears it). The installed version is the marker's `cliVersion`, else `/usr/bin/supabase --version`, else PATH; `metrics.cliVersion` is that staged version and `cliVersionAfterRun` the PATH one when it differs. Dist-tag runners (`supabase@beta`) can't be checked offline and are reported in `metrics.cliRunnerUnverified`. - **Services are found by `config.toml`, else by the CLI's `stack list`.** CI agents often created stacks without `supabase init` (`mkdir -p payments-api && cd payments-api && supabase stack start --runtime native --stack payments-api`) or ran named stacks from the sandbox root, sometimes under a relocated `HOME`/`SUPABASE_HOME`, so the `config.toml` search found nothing. For each service it missed, the scorer reads `stack list` under the default home and each relocated home the agent started the service with, and takes entries whose `name` is the service (reachable preferred). One `project_root` is the project directory (`(from stack list)`); the sandbox root itself counts as a named stack started there (`named stack at sandbox root`, resolved by the existing named lookup); two or more distinct roots are `ambiguous (stack list: …)` and fail. When a `config.toml` exists the directory is unchanged, but if resolving the stack there fails (an agent broke its own `config.toml`, or rooted the stack elsewhere), the one `stack list` entry named exactly for the service gives the directory to resolve in, under the default and relocated homes; ambiguous roots stay unresolved. `lib/projects.ts` is untouched. - **Failed commands are never evidence, except starts that reported ready.** A tool call that exited non-zero, or whose output holds a CLI error (`"_tag":"Error"`, `Unknown subcommand`, `UnknownSubcommand`), doesn't count as a start, teardown or restart. A call with no recorded outcome counts as succeeded. Top-level `supabase restart` never counts, since neither CLI has it. **Start-marker exception:** when a failed call's output has no CLI error envelope, each `[task] done: Stack is ready.` or `Started supabase local development setup` it printed credits one start invocation of that call (in order) as not failed. A start that came up isn't discarded because a later command in the same call failed (`… > out && jq …` exiting 127, or a `set -e` seed that hit a TLS error). Other verbs keep the all-failed rule. - **Change phase.** Command evidence for restarts and touches only counts after the setup point, so setup-phase stop/start retries (e.g. clearing a port clash) are neither a restart nor a touch. If that point never comes, the restart and untouched checks fail with an `unavailable:` note. - **Marker rows match their service name as a whole word** (case-insensitive), the same rule as the isolation check, so `marker for payments-api` is payments-api's marker and `payments-api-archive` isn't. - **Leftover legacy containers.** Once `legacy-import`'s directory is gone, the scorer checks `docker ps` for containers labelled `com.supabase.cli.project=legacy-import` or named `supabase_*_legacy-import`; skipped when `docker` is unreachable, since then nothing can be running. The scorer never reads which experiment it runs under. - **`stack list` fails closed.** Only an unknown-subcommand error skips the listing half; any other unreadable output fails `legacy-import stack is gone`. - **Truthful judge.** When nothing was started, a single clear statement covering all three services is enough, so an honest blanket report isn't failed as vague. When the database start time says checkout wasn't restarted but a restart command after setup didn't fail, the ground truth states both, and the rubric treats an agent that accurately reports the CLI-reported successful restart as truthful; the check itself still follows state evidence. - **`--stack-id` is mapped to a service** from the `"id"` a successful start printed (when that start targeted one service) and from any `stack list` output the agent printed; an unknown id is attributed to no service. - **Shell loops** (`for s in …; do (cd "$s" && supabase start); done`) count as start evidence for every service (see the anchor rule above). Loop teardowns and restarts count for none, since their target can't be attributed. - **Known limitations** (in the README): - Failure is known per tool call, so one failing command in a compound call marks all of its invocations failed. Starts are the exception: a failed call's start counts when the output printed a ready marker and holds no CLI error, one start per marker, in order. - `cd` doesn't carry across separate tool calls. - It's unverified whether a native `stack restart` restarts Postgres; if it doesn't, state evidence on a native stack fails `checkout-service was restarted` after a restart that succeeded (the truthful judge is told about both). - The container probe assumes legacy-import's project id is its directory name. The CLI-2402 issue describes the CLI side as blocked. It no longer is: `stack list/start/restart/destroy/status --stack` exist in beta behind `SUPABASE_EXPERIMENTAL_STACK=1`. The harness can't hand an agent pre-running named stacks, so the prompt is two-phase: build the fleet, then manage it. ## Results CI run 37625367762 scored 6/15. - Three `legacy-import` teardowns failed with `StopVolumePruneError` ([CLI-2637](https://linear.app/supabase/issue/CLI-2637)) and passed by state. - Six agents never started a stack. - pinned r3 swapped the CLI to `npx` 2.120.0 (policy fail; it then hit CLI-2638). - stable r2 used `stack stop`, which leaves the stack listed (correct fail). - nodaemon r2 broke its own `config.toml` (a duplicate `[experimental]` table gives `CliConfigParseError`) and rooted the stack at `checkout-service/runtime`, so the scorer said it wasn't running. Resolution now falls back to the `stack list` entry for the service. - beta r1 was a scorer bug, now fixed. It ran `supabase start --workdir "$PWD"` once per service directory; `$PWD` wasn't expanded, so the first start read as a loop start, anchored setup too early, and payments' own start read as after setup (`payments-api left untouched` failed, with no teardown or restart attributed either). `$PWD` is now expanded against each invocation's cwd (shared `lib/cli-invocations.ts`, identical to #314), unresolved starts no longer anchor setup, and the replay passes all fleet checks. #355 (the sandbox login-shell `PATH` fix) is merged, so `nodaemon` is now real; earlier `nodaemon` results were affected by that bug. The per-call working-directory gap is fixed in #356, which is now merged, so per-call `cwd` attribution and timing evidence are in effect. Some Docker-arm agents decline to start because `docker ps` shows the sandbox's own container; that's an environment artifact. Known product gaps seen in CI: - [CLI-2637](https://linear.app/supabase/issue/CLI-2637): `stop --no-backup` exits 1 with `StopVolumePruneError` on Docker clients older than 23.0, even though the containers are removed - [CLI-2638](https://linear.app/supabase/issue/CLI-2638): the managed Docker runtime fails to bind-mount when the daemon doesn't share the CLI's filesystem (the sandbox's sibling daemon) - [CLI-2639](https://linear.app/supabase/issue/CLI-2639): `stack destroy` can't remove a stack whose launch failed - [CLI-2640](https://linear.app/supabase/issue/CLI-2640): Docker-unavailable errors don't mention `--runtime native`, so some Docker-less agents give up ## Related Issue(s) Refs https://linear.app/supabase/issue/CLI-2402 ## Test plan - [x] `vitest run --root ../.. evals/cli` (counterexamples include psql delete, `rm -rf`, never-started stop, a failed `stop --no-backup` with `StopVolumePruneError` and clear state (passes with the CLI-2637 note), no teardown command with clear state (passes), a started stack still listed (fails), services found via `stack list` (default home, relocated home, sandbox root, ambiguous roots), loop starts, echoed restart, `stack restart --stack payments-api`, directory removed after destroy, failed `stack restart`, top-level `supabase restart`, setup-phase stop/start retries, failed legacy teardown after `rm -rf`, leftover labelled containers, unreadable `stack list` output, `db reset --workdir payments-api`, a checkout `db reset` that doesn't count as a restart, seeding every service via `db reset` before any change, a seeding `db reset` sharing a call with the restart (commands decide) versus in an earlier call (state decides), `stop --project-id payments-api` run from another service's directory, a failed versus successful global install, a loop start followed by per-service starts, only loop starts, the beta r1 replay, a broken `config.toml` found via `stack list`, a `marker for payments-api` row, failed starts in the swap rule, `--stack checkout` and `--stack checkout-service-recovered` started in the service directory, dist-tag runners not counted as overrides, no repeated probe commands in the renamed-stack lookup; state evidence: checkout restarted with no restart command, "successful" restart with an older postmaster, payments restarted by state, payments stop command overruled by state, `db reset` under state, fallback to commands without timing, ignoring a tool call issue time, setup and restart/touch in one call, ground truth stating both state and a successful restart command) - [x] `apps/framework` `tsc --noEmit` - [x] `pnpm eval:dry -- --suite cli --experiment-suite cli` → planned under all 5 cli experiments - [x] `pnpm format:check` - [x] `pnpm check`, all steps except the judge smoke step, which needs a provider API key locally - [x] CI refresh via `run-evals-changed`: inspect one passing run and each distinct failure shape - [x] Merged `origin/main` (#356 included); `pnpm check` and the rest re-run after the merge - [x] Eval results refresh on this push (`run-evals-changed`): confirm timing evidence and per-call `cwd` attribution show up, incl. `nodaemon`, and that renamed stacks resolve 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

…l-projects Conflicts resolved: - evals/cli/lib/{README.md,cli-invocations.ts,cli-invocations.test.ts}: kept this branch's (newer, round 3) versions; lib now matches the final state of the stale-stack-cleanup branch. - apps/web/src/data/cli-eval-results.json: main's entries for all evals except build-database-003-parallel-projects, plus this branch's entries for that eval.

…anup Carries #316's last commit (32e88c9), which missed #316's merge to main.

jgoux

Thanks, the round-3 points are addressed:

A global install is no longer ignored because its tool call failed. npm i -g supabase@2.120.0 && cd client-a && supabase start with a failing start is now flagged.
--project-id is now only used to attribute stop, and both READMEs match the code.

Approving. One P3 is left inline: reinstalling the staged version after a swap hides the swap. It can be fixed in a follow-up.

Coly010 requested a review from a team as a code owner September 21, 2026 10:00

This was referenced Sep 21, 2026

feat(evals): add resolve-database-002-stale-stack-cleanup CLI eval (CLI-2402) #316

Merged

feat(framework): add cli suites, needsDocker, and CLI environment arms #308

Closed

feat(cli): add the cli eval suite and experiment suite plumbing #327

Merged

Coly010 added 2 commits September 23, 2026 15:33

Coly010 force-pushed the columferry/cli-2399-build-database-003-parallel-projects branch from 4c048d6 to 76c1ab0 Compare September 23, 2026 15:05

Coly010 changed the base branch from columferry/cli-2398-cli-suite-and-arms to main September 23, 2026 15:05

Coly010 mentioned this pull request Oct 1, 2026

refactor(evals): extract shared cli eval helpers into evals/cli/lib #354

Merged

5 tasks done

Coly010 added 3 commits October 1, 2026 10:46

Merge remote-tracking branch 'origin/main' into columferry/cli-2399-b…

32e22d6

…uild-database-003-parallel-projects

Merge remote-tracking branch 'origin/columferry/cli-evals-shared-lib'…

7250df6

… into columferry/cli-2399-build-database-003-parallel-projects

Coly010 changed the base branch from main to columferry/cli-evals-shared-lib October 1, 2026 09:56

Coly010 added the run-evals-changed Add to a PR to refresh only the benchmark evals that have had changes label Oct 1, 2026

chore: refresh eval results [skip ci]

aa5c3a3

vercel Bot deployed to Preview October 1, 2026 10:06

View deployment

Coly010 and others added 2 commits October 1, 2026 11:19

fix(evals): make clear the parallel-projects clients don't exist yet

60495dc

chore: refresh eval results [skip ci]

7423199

vercel Bot deployed to Preview October 1, 2026 10:42

View deployment

Coly010 and others added 4 commits October 1, 2026 12:43

feat(evals): share supabase invocation parsing in evals/cli/lib

0f3e342

Move findSupabaseInvocations and its target helpers into the shared lib so every cli eval can attribute the agent's supabase commands to a project or stack.

Merge remote-tracking branch 'origin/columferry/cli-evals-shared-lib'…

cd19d87

… into columferry/cli-2399-build-database-003-parallel-projects

feat(evals): report whether the agent attempted each stack start

bc07be3

chore: refresh eval results [skip ci]

f9ae945

vercel Bot deployed to Preview October 1, 2026 11:55

View deployment

Coly010 added 3 commits October 1, 2026 13:03

feat(evals): attribute supabase invocations by working directory

2898838

Merge remote-tracking branch 'origin/columferry/cli-evals-shared-lib'…

712f19f

… into columferry/cli-2399-build-database-003-parallel-projects

fix(evals): attribute parallel-projects start attempts by working dir…

78218f9

…ectory

chore: refresh eval results [skip ci]

f7c5cf0

vercel Bot deployed to Preview October 7, 2026 13:40

View deployment

Coly010 and others added 3 commits October 7, 2026 15:37

fix(evals): expand $PWD in env values against the shell cwd before en…

6e19a72

…v -C A SUPABASE_HOME/HOME/TMPDIR value of $PWD/... now resolves against the directory the shell was in, while a plain relative value still resolves against the directory the CLI runs in.

chore: refresh eval results [skip ci]

9fed65a

vercel Bot deployed to Preview October 7, 2026 14:49

View deployment

jgoux requested changes Oct 7, 2026

View reviewed changes

Comment thread evals/cli/build-database-003-parallel-projects/EVAL.ts

Comment thread evals/cli/lib/cli-invocations.ts Outdated

Comment thread evals/cli/build-database-003-parallel-projects/README.md Outdated

Coly010 and others added 7 commits October 7, 2026 16:29

fix(evals): keep --project-id authoritative over the working directory

51a9403

The legacy CLI filters containers by project-id label regardless of cwd, so invocationTargets now checks an explicit --project-id first, then the directory, then --stack.

docs(evals): make the parallel-projects observed results a dated snap…

3d993e4

…shot

fix(evals): read the staged CLI version from /usr/bin/supabase when t…

97b345b

…he marker has none Without a marker the fallback ran the PATH supabase, which an agent's global npm install shadows. The release .deb binary is read first, then PATH.

chore: refresh eval results [skip ci]

d8aee4c

vercel Bot deployed to Preview October 7, 2026 15:44

View deployment

jgoux requested changes Oct 7, 2026

View reviewed changes

Comment thread evals/cli/lib/cli-invocations.ts Outdated

Comment thread evals/cli/build-database-003-parallel-projects/README.md Outdated

Coly010 added 2 commits October 7, 2026 16:58

fix(evals): scope --project-id attribution to stop in cli invocations

429e28c

fix(evals): reconcile global installs against the post-run PATH version

561e03f

Coly010 and others added 3 commits October 7, 2026 17:04

fix(evals): wire post-run install reconciliation into stale-stack-cle…

7f87aba

…anup Carries #316's last commit (32e88c9), which missed #316's merge to main.

chore: refresh eval results [skip ci]

61c2244

vercel Bot deployed to Preview October 7, 2026 16:08

View deployment

jgoux approved these changes Oct 7, 2026

View reviewed changes

Comment thread evals/cli/lib/cli-invocations.ts

chore: refresh eval results [skip ci]

55b91b5

vercel Bot deployed to Preview October 7, 2026 16:22

View deployment

Coly010 and others added 2 commits October 7, 2026 17:30

fix(evals): let a later global install close the previous install's span

e9a05c0

chore: refresh eval results [skip ci]

a8c1c3f

vercel Bot deployed to Preview October 7, 2026 16:47

View deployment

Coly010 merged commit 5f8de50 into main Oct 7, 2026
2 checks passed

Navigation Menu

Repository navigation

feat(evals): add build-database-003-parallel-projects CLI eval (CLI-2399) - #314

feat(evals): add build-database-003-parallel-projects CLI eval (CLI-2399)#314
Coly010 merged 60 commits into
mainfrom
columferry/cli-2399-build-database-003-parallel-projects

Coly010 commented Sep 21, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Sep 21, 2026 •

edited

Loading

Uh oh!

jgoux left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jgoux left a comment

Uh oh!

Uh oh!

Uh oh!

jgoux left a comment

Uh oh!

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Coly010 commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current Behavior

Expected Behavior

Related Issue(s)

Test plan

Uh oh!

vercel Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jgoux left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jgoux left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jgoux left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Coly010 commented Sep 21, 2026 •

edited

Loading

vercel Bot commented Sep 21, 2026 •

edited

Loading