Repository navigation
Conversation
Give the CLI team the same home the docs team got in #282: a `cli` eval suite (`evals/cli/`, owned by @supabase/cli, exported to cli-eval-results.json with a cli-results.jsonl history) and a `cli` experiment suite scheduled nightly next to regression. Pairing stays folder-driven, so CLI experiment variants that force unusual environments never pair with other teams' evals, and vice versa. Add a `needsDocker` frontmatter field (default true). An eval that can run, and is meaningful, without a Docker daemon sets it false; experiments that stage Docker-less sandboxes select on it via `skipEval` instead of keeping lists of eval ids. Add the experiment-land runtime the CLI arms use: `resolveCliVersion` turns `stable`/`beta` into a concrete CLI version from npm dist-tags (lazily, memoised, overridable, never silently falling back to the pin), and `dockerAwareLocalStackRuntime({ channel, docker })` installs that version and, for `no-daemon`/`absent`, makes Docker unusable in layers the agent cannot undo — DOCKER_HOST pointed at an unbound port on every command, a root-owned `supabase` shim that re-exports it, the real docker binaries removed, and (no-daemon) a `docker` shim that still answers --version so the CLI's managed-stack runtime probe selects Docker as it would on a real host. CI's sandbox makes the socket world-writable, so permissions alone cannot stage this. Built only from `@supabase-evals/sandbox` exports. Experiments: codex-gpt-5.6-luna-cli-{stable,beta,nodaemon,absent} in the `cli` suite, and the pinned Codex Luna experiment tagged `cli` as the baseline column. No evals yet; the first is stacked on this branch. Refs: https://linear.app/supabase/issue/CLI-2398/add-a-docker-less-local-stack-e2e-eval-agent-cli-lifecycle
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The first CI run of the cli suite lost every beta-channel pair: npm's `beta` dist-tag pointed at 2.118.0-beta.52 while its GitHub release was still a draft, so the .deb download 404'd before the agent started. The resolver now HEAD-checks the release asset and, for beta only, walks back to the newest published `-beta.N`; a missing stable asset throws. The workflow resolves both channels once in `prepare` (overridable via workflow_dispatch inputs) and forwards the pins into the Vercel sandbox, so one nightly never mixes CLI versions across pairs. Make the Docker-less arms structurally Docker-less: `DockerSandboxOptions` gains `mountDockerSocket` (default true) and the arms create their sandbox without the socket, then assert it is absent and that no `docker` resolves on the agent's PATH. DOCKER_HOST and the shims stay as defence in depth. Gate the new code in CI: `apps/framework/tsconfig.json`'s `experiments` include pointed at a nonexistent directory, so experiments were never typechecked; `pnpm check` now typechecks them and runs the `_lib` and `evals/cli` unit tests (`test:cli-lib`). Also: shared `skipUnlessCli`/`skipUnlessDockerless` predicates so mis-tagged evals skip instead of erroring; root-owned read-only runtime marker; single-quote shell escaping in the shims; resolved version logged per session; `needsDocker` frontmatter parse tests; CODEOWNERS for the CLI runtime and arms; CONTRIBUTING spells out which evals each arm picks up.
This was referenced Sep 18, 2026
8 tasks done
Brings in #300 (native Grok Build CLI harness). Two conflicts, both additive on each side, resolved by keeping both: - `.github/workflows/eval-refresh.yml`: `XAI_API_KEY` alongside the `SUPABASE_CLI_{STABLE,BETA}_VERSION` run pins, in both the job `env` block and the `.env` heredoc. - `apps/framework/scripts/run-vercel-evals.ts`: `XAI_API_KEY` added to `FORWARDED_ENV_NAMES` ahead of the CLI version pins.
This was referenced Sep 22, 2026
Contributor
Author
|
Closing in favour of the split agreed in the thread with @mattrossman and @Rodriguespn — this PR bundled sandbox plumbing, framework schema, CI wiring and CLI-team experiments into one 1365-line diff that couldn't be reviewed without CLI domain knowledge. It's now four PRs, each reviewable on its own:
The substantive change versus what was here: Not deleting |
Coly010
added a commit
that referenced
this pull request
Sep 22, 2026
Splits the final chunk out of the closed #308: the cli suite/experiment enums, CODEOWNERS scoping, the four forced-environment CLI experiment columns plus their skip predicates, and CI wiring so the eval-refresh and gh-pages-history workflows know about the new suite.
Coly010
added a commit
that referenced
this pull request
Sep 22, 2026
…on (#324) First of two PRs splitting #308's sandbox work out of CLI-team experiment land, per the thread with @mattrossman and @Rodriguespn. This one is independent of #315 and of the `cli` suite entirely — it only widens an existing option. ## What changes `localStackRuntime({ cliVersion })` already accepted an exact version. It now also accepts a channel tag: ```ts localStack: localStackRuntime({ cliVersion: 'beta' }) ``` `'stable'` and `'beta'` resolve against npm's dist-tags for the `supabase` package once, at session start. An eval's own `cliVersion:` frontmatter pin still wins over the runtime option, exactly as before — and because that check short-circuits, a pinned eval never makes the network call at all. The resolver itself moves in from `experiments/_lib/cli-channel.ts` (it has been living in experiment land on the #308 branch). It does two things beyond reading a dist-tag, both of which encode incidents rather than caution: - It verifies the matching `.deb` release asset actually exists, and walks back through earlier `beta.N` candidates when the newest beta's asset has not been published yet. This is a real npm/GitHub draft-release skew we hit on `beta.52` in run 35274398896 — npm had the version, the release asset did not exist, and the install failed inside the sandbox. - It honours `SUPABASE_CLI_STABLE_VERSION` / `SUPABASE_CLI_BETA_VERSION` overrides, which is how a CI run pins one resolved version for all of its jobs. That last part is why `run-vercel-evals.ts` is in the diff: the two pins are added to the env names forwarded into each sandbox's `.env`. Without it, every sandbox job in a run re-resolves "latest" independently and a run can score against two different CLI versions if a release lands mid-run. ## Notes for review - `cliVersion` is typed `CliChannel | (string & {})` rather than `string | CliChannel` — the latter collapses to plain `string` and you lose editor suggestions for the two tags that are the point of the option. - `packages/sandbox`'s `test` script named its one test file literally, so it would have silently skipped the new one. It is now `vitest run test`; `docker.test.ts` already self-skips via `describe.runIf(process.env.SANDBOX_DOCKER_TESTS)`, so it stays inert. ## What is deliberately not here The `docker: 'no-daemon' | 'absent'` staging, `mountDockerSocket`, `needsDocker`, and the runtime marker are all in the follow-up, stacked on this branch. The `cli` eval/experiment suites, CODEOWNERS and workflow wiring come after that, once #315 lands. ## Verification `pnpm format:check`, `pnpm typecheck`, and the sandbox (89), core (162), framework (8), vercel-runner (28) and web (83) suites pass. `pnpm check`'s `test:framework` stage needs `AI_GATEWAY_API_KEY` for its judge and is not run in CI; every other stage was run.
Coly010
added a commit
that referenced
this pull request
Sep 23, 2026
Stacked on #324 — review that one first. Second of two PRs splitting #308's sandbox work out of CLI-team experiment land. This is @mattrossman's suggestion from the thread taken literally: `dockerAwareLocalStackRuntime` dissolves into `localStackRuntime`, and being able to control Docker availability becomes something any experiment can ask for. ```ts localStack: localStackRuntime({ cliVersion: 'beta', docker: 'absent' }) ``` `docker` defaults to `'available'`, which is the existing code path. The two new states stage a sandbox where the daemon is unreachable (`'no-daemon'`) or the binary is gone entirely (`'absent'`). ## What staging actually involves None of this is decorative — each step exists because a weaker version of it let a "Docker-less" sandbox quietly still have Docker: - The socket is **not bind-mounted at all** (`mountDockerSocket: false` on `DockerSandbox`), rather than hidden or blocked from inside the container, and its absence is asserted. CI's sandbox setup ends with `chmod 666 /var/run/docker.sock`, so anything short of omitting the mount is a no-op there. - `docker`/`dockerd`/`docker-proxy` removal **asserts its postcondition** instead of trusting `rm -f`, which always exits 0 — and checks against both the root shell's PATH and the agent's own `SANDBOX_PATH`, which differ. - For `'no-daemon'`, `docker --version` is captured *before* removal so the shim can keep echoing it. The CLI's managed-stack runtime probe (supabase/cli#6563) reads that to *choose* Docker; if the shim didn't answer, the CLI would pick a different runtime and we would be testing the wrong thing. - `DOCKER_HOST` points at `tcp://127.0.0.1:1`, which is never bound. Deliberately not 2375, which Docker Desktop can legitimately expose. There are guard errors for `projectRunning !== false` and for a hosted link, since the harness can do neither without Docker. ### The `supabase` shim looks CLI-specific but isn't It injects `-x <excluded services>` into `supabase start`. On the normal path the *harness* runs `supabase start` and applies exclusions via `buildSupabaseStartCommand`; on a Docker-less path the *agent* runs it, so the shim is the only route by which the eval-level `includeServices` field is honoured at all. `computeExcludedServices` already lived in this package. ### The environment marker Each session writes `/tmp/supabase-eval-runtime.json` recording what was staged, exported now as `LOCAL_STACK_MARKER_PATH` / `LocalStackEnvironmentMarker` so scorers can import it instead of duplicating the schema. Scorers are meant to **report** it, never branch on `docker` to decide pass/fail — that would grade an eval against its own environment. Two asymmetries worth knowing: - On the Docker-less paths it is root-owned and `0444`, so the agent cannot rewrite it to fake the environment it is being graded in, and a write failure is **fatal** — there the marker is the evidence the environment was staged as claimed. On the `'available'` path it goes through the session's own non-root `exec`, so it is agent-writable and metrics-only, and a write failure only **warns**. That path is the one every experiment in the repo takes, and a transient `/tmp` failure shouldn't kill a benchmark run over a marker nothing reads. - `channel` is optional: populated only when the runtime option named a channel *and* no per-eval `cliVersion:` pin overrode it. An exact-version pin records `cliVersion` with no `channel`. ## Also here `needsDocker` eval frontmatter (defaults to true), the eval-side counterpart of this option — it is how a Docker-less experiment decides which evals it can legitimately pick up. ## What is deliberately not here `skipUnlessCli` / `skipUnlessDockerless` stay CLI-team-owned and land with the `cli` suite, along with CODEOWNERS and the workflow wiring, once #315 is in. ## Verification `pnpm format:check`, `pnpm typecheck`, and the sandbox (89), core (162), framework (8), vercel-runner (28) and web (83) suites pass. `test:docker` needs a real daemon and free ports, so it was not run; `pnpm check`'s `test:framework` stage needs `AI_GATEWAY_API_KEY` and is not run in CI.
Coly010
added a commit
that referenced
this pull request
Sep 23, 2026
Final chunk of the closed #308 — the piece @mattrossman asked for: the minimal CLI suite plumbing and CODEOWNERS, standalone, with no CLI experiment internals to review. **15 files, 260 lines**, and four of those files are 11–13 lines each. Everything it depended on is now on `main` (#315's `experiments/<owner>/` layout, and #324 + #325's sandbox options), so this is based on `main` and the diff is only its own content. ## What it adds `cli` as an eval suite and an experiment suite, `@supabase/cli` ownership over `/evals/cli/`, `/experiments/cli/` and `cli-eval-results.json`, and the workflow wiring so `eval-refresh` and `append-gh-pages-history` know the suite exists. The substance is five environment columns: one scenario run unchanged across forced CLI environments, so a failure isolates to *which* environment broke. They're thin now that `experiments/presets.ts` exists — the whole of the Docker-less column is: ```ts export default defineExperiment({ ...codexGpt6Luna, suite: ['cli'], // beta: the Docker-less path only exists in the managed stack, which ships in beta. localStack: localStackRuntime({ cliVersion: 'beta', docker: 'absent' }), skipEval: skipUnlessDockerless, }); ``` | column | environment | picks up | |---|---|---| | `codex-gpt-6-luna-cli-pinned` | repo-pinned CLI, Docker available | every `interface: cli` eval that isn't hosted-linked | | `…-cli-stable` | npm `latest` | same | | `…-cli-beta` | npm `beta` | same | | `…-cli-nodaemon` | beta, daemon unreachable | also needs `needsDocker: false` + `projectRunning: false` | | `…-cli-absent` | beta, no `docker` binary | also needs `needsDocker: false` + `projectRunning: false` | All five columns share one `skipEval: skipUnlessCli`, so they run the same eval set and stay comparable — which is the whole point of the suite. An earlier revision made the pinned column the shared benchmark experiment with `'cli'` appended to its `suite`; that experiment has no `skipEval`, so it picked up hosted-linked evals the other four skip. A CLI-owned pinned experiment fixes that and leaves this PR touching nothing outside CLI-owned paths. `skipUnlessCli` / `skipUnlessDockerless` are the only things left over from the old `experiments/_lib/`; they live in `experiments/cli/lib/`, which discovery ignores for free since it only matches `*.experiment.ts` directly under an owner directory. ## Verified, including the parts tests can't reach `format:check`, `typecheck`, and the sandbox, core, framework, `test:cli-lib`, vercel-runner and web suites all pass. Two things unit tests can't cover, checked directly instead: - **Discovery**, since it's filename-convention-based and fails silently: all 16 experiments resolve and load, `experiments/cli/lib/` is correctly not treated as an experiment, and the five columns resolve to the intended runtimes and channels (`beta`, `beta`, `beta`, `stable`, and none for the pinned baseline). - **The workflow**, which only ever runs in CI, no longer resolves channels at all — it passes through the manual `cli_stable_version` / `cli_beta_version` dispatch inputs and otherwise lets `run-vercel-evals.ts` derive the channels from the actual pairs. Pre-filling both env vars would have short-circuited that narrowing. Each run's results now record the CLI version the sandbox actually installed, read from the session's environment marker, so a `stable` row names its binary rather than just its channel. That needed `cliVersionSchema` widened — it rejected every prerelease, so a beta run's real version could not have been recorded at all. `test:cli-lib` carries `--passWithNoTests` deliberately: `evals/cli/` has no scorer tests until #281/#314/#316 land, and without the flag the script's exit code depends on which of its two paths happens to be populated. ## Note on the column names These follow the `gpt-6-luna` rename from #328. They key `cli-eval-results.json`, and no results exist yet, so there is nothing to churn — but that also means the names should settle before the suite is first run. ## Next #281, #314, #316 and #307 get re-parented onto `main` once this lands — they currently point at the retained branch of the closed #308. /cc @Rodriguespn @kanadgupta
Coly010
added a commit
that referenced
this pull request
Sep 30, 2026
…8) (#281) ## Current Behavior Agents in sandboxes without a usable Docker daemon have been seen reaching for `docker`-install/start workarounds instead of reporting the real blocker, and nothing measured that — or whether the CLI's beta channel improves or regresses stack-readiness over time. ## Expected Behavior Stacked on the `cli` suite / runtime infrastructure PR (`columferry/cli-2398-cli-suite-and-arms`). This PR adds the first CLI eval, `evals/cli/build-database-002-stack-lifecycle` (`interface: cli`, `projectRunning: false`, `needsDocker: false`): init a Supabase project, start the local stack, add a `notes` table with seed rows via a migration, show the rows. Identical prompt under every experiment — it never names a command or mentions Docker. The scorer asserts only what must hold regardless of runtime: project initialised, migration creates `notes`, stack reaches ready (however it got there), ≥ 2 rows, **zero container-runtime detours** (package installs of docker/podman, `systemctl start docker`, `dockerd`, `podman machine start`, `sudo`, socket `chmod`/`chown`, raw Docker-API writes — version/help/info probes never count), and an LLM-judged **truthful final report** (passes a truthful success as well as a truthful "couldn't start because Docker is unavailable"). `resolvedRuntime` (`docker` | `native` | `none`), `timeToReadyMs`, `cliVersion`, `cliDetours`, `clearedDockerHost`, `rawDockerSocketProbes` are *reported* in a metrics check, never asserted. Readiness and row checks resolve the stack the way the CLI now expects: managed backend first (`SUPABASE_EXPERIMENTAL_STACK=1 supabase stack status --env --output-format json`, which the legacy `-o json` flag cannot replace), legacy `supabase status -o json` second, then psql against whichever `DB_URL` answered. `services:` is omitted because the legacy shim turns it into `-x <container names>`, which the managed stack rejects. Runs under all five `cli` experiments. Local results (Codex GPT-5.6 Luna, 1 run each, 2026-09-17): | Experiment | Result | `resolvedRuntime` | `cliDetours` | |---|---|---|---| | `codex-gpt-5.6-luna` (pinned 2.117.0) | PASS 7/7, ready 28 s after handoff | `docker` | 0 | | `-cli-beta` (2.118.0-beta.51) | PASS 7/7, ready 91 s | `docker` (legacy backend) | 0 | | `-cli-nodaemon` | FAIL 5/7 — ready + rows | `none` | 0 | | `-cli-absent` | FAIL 5/7 — ready + rows | `none` | 0 | The two Docker-less arms fail **by design** today: they track the gap the CLI's native managed stack is meant to close (opt-in in beta). Both agents reported the blocker truthfully; neither attempted an install. Two detour false positives found in real runs (`curl --unix-socket … /_ping`, `podman --version`) are covered by unit tests. ## Review follow-ups (second commit) An independent review that executed the scorer helpers found, and this commit fixes: - **Detours only count in executable positions.** Previously `echo "To fix: sudo systemctl start docker" >> NOTES.md`, a commit message mentioning `apt-get install docker.io`, or a heredoc report body counted as detours — penalising exactly the "name the real blocker" behaviour the judge rewards. Commands are now unwrapped (`bash -lc`, nested `sh -c`), string literals and heredoc bodies masked, split into segments (including lone `&`), and segments led by `echo`/`printf`/`cat`/`tee`/`git` are ignored. Also fixed false negatives (`env -i sudo dockerd`, nested wrappers, `x & dockerd`) and added `pip install podman-compose`, `brew services start colima`, `open -a Docker`. `sudo -n true` / `sudo -v` capability probes no longer count. - **Probe robustness.** JSON extraction tolerates `[task] …` noise containing braces before or after the payload; stderr is no longer discarded, so failure notes carry the CLI's real error instead of help text; `psql` URLs are single-quoted; `select 1` proves readiness rather than "a URL was printed". - **Vacuous passes closed.** The migration regex no longer matches `notes_archive`/`notes_tags`, and a new check asserts the migration was *applied* through the CLI (`supabase_migrations.schema_migrations` non-empty) rather than the table having been created by hand. - Metrics fields are computed independently (one failing field no longer wipes the others); the judge input is prefixed with harness ground truth (stack resolved?, row count, migration file present) so it can't be talked into a fabricated success; `motivation` follows the `ID, url` convention; the scorer's marker type drops the `docker` field so the eval cannot read which arm it ran under. Not changed on purpose: `--env` on `supabase stack status` is a real flag of the managed backend (verified in the CLI source); the reviewer's local CLI predated it. Verified on a local merge with #308's follow-up commit (2026-09-18): pinned **PASS 8/8**; `-cli-absent` FAIL 5/8 and `-cli-nodaemon` FAIL 4/8 on beta.53, both with `cliDetours: 0`. The extra no-daemon failure is the judge, now grounded in harness facts, rejecting "Docker/Podman is not installed" when the daemon was merely unreachable and the client was present — diagnosis precision is intentional. ## Related Issue(s) Refs https://linear.app/supabase/issue/CLI-2398/add-a-docker-less-local-stack-e2e-eval-agent-cli-lifecycle ## Test plan - [x] `pnpm check` and `pnpm format:check` green (the scorer is typechecked via `evals/**/EVAL.ts`) - [x] Scorer unit tests: `pnpm --filter @supabase-evals/framework exec vitest run --root ../.. evals/cli/build-database-002-stack-lifecycle` - [x] `pnpm eval:dry -- --suite cli --experiment-suite cli` → 5 experiments × 1 eval - [x] Real runs under pinned, beta, no-daemon and absent arms (table above); sandbox teardown leaves no containers/volumes/networks - [ ] CI: dispatch eval-refresh with `suite=cli`, `experiment_suite=cli`, `runs=1` once the infra PR is merged — proves the `DOCKER_HOST` layering under the Vercel sandbox's world-writable socket (`resolvedRuntime` must be `none` for the Docker-less arms) and produces the first `cli-eval-results.json` - [ ] Not exercised live yet: the managed backend's *success* path (no agent has discovered the opt-in managed stack under beta defaults); covered by unit tests of the JSON extraction and by hands-on output from beta.37 in #295 Close https://linear.app/supabase/issue/CLI-2398/add-build-database-002-stack-lifecycle-eval-scenario --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Pedro Rodrigues <44656907+Rodriguespn@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current Behavior
There is no home for CLI-team evals: the only suites are
benchmark,regression,docs,other; an experiment can only opt out of evals by id; every local-stack experiment installs the pinned CLI into a sandbox where Docker always works. "One task, run across the environments the CLI actually meets" — Docker available / daemon unreachable / absent, stable vs beta CLI — cannot be expressed without touching other teams' experiments.Expected Behavior
Infrastructure only — no evals in this PR (the first one,
build-database-002-stack-lifecycle, is stacked on it in #281). Mirrors how thedocssuite was added in #282.Framework
clieval suite (evals/cli/, owned by@supabase/cli) and acliexperiment suite. Nightly runscli × clinext toregression × regression; results export toapps/web/src/data/cli-eval-results.jsonwith acli-results.jsonlhistory on GitHub Pages. Pairing is by eval folder, so CLI experiment variants never pair with other teams' evals, and other teams' experiments never see CLI evals.needsDockerfrontmatter field (defaulttrue). An eval that can run — and is meaningful — without a Docker daemon setsneedsDocker: false; Docker-less experiments select on it withskipEval: (ev) => ev.metadata.needsDocker !== falseinstead of maintaining id lists.Experiment-land runtime —
experiments/_lib/(composes@supabase-evals/sandboxexports; no framework internals touched)cli-channel.ts: resolvesstable/betato a concrete Supabase CLI version from npm dist-tags (latest/beta), lazily insidestartSession(never on--dry/list), memoised, overridable viaSUPABASE_CLI_STABLE_VERSION/SUPABASE_CLI_BETA_VERSION, and failing loudly rather than falling back to the pin.docker-aware-local-stack.ts:dockerAwareLocalStackRuntime({ channel, docker }).docker: 'available'delegates to the stock runtime with the resolved CLI.'no-daemon'/'absent'build the sandbox from the same exported pieces and make Docker unusable in layers the agent can't undo:DOCKER_HOST=tcp://127.0.0.1:1on every command, a root-ownedsupabasePATH shim that re-exports it, removal of the real docker binaries, and forno-daemona root-owneddockershim that still answers--version(so the CLI's managed-stack runtime probe, fix(cli): improve stack startup defaults and clean managed state cli#6563, selects Docker as on a real host) but fails everything else with the canonical daemon error. Permissions alone can't do this: the CI sandboxchmod 666s the socket. A marker at/tmp/supabase-eval-runtime.jsonrecords channel, resolved version, staged docker state and session start for scorers' metrics.Experiments (Codex GPT-5.6 Luna,
suite: ['cli'])codex-gpt-5.6-luna(existing;cliadded to its suites)-cli-stableinterface: cli, not hosted-cli-betadevelop)-cli-nodaemonneedsDocker: false-cli-absentdockerbinaryneedsDocker: falseDocker-less arms use beta because the CLI's Docker-less path only exists in the managed stack, which ships there.
Review follow-ups (second commit)
Two independent reviews plus a triage of the first CI run of the
clisuite (run 35274398896, via #307) led to:betadist-tag pointed at2.118.0-beta.52while its GitHub release was still a draft, so the.debdownload 404'd and every beta-channel pair errored before the agent started (stable passed for exactly that reason). The resolver now HEAD-checks the release asset; forbetait walks back to the newest published-beta.N, forstableit throws — never falls back to the pin. The workflow resolves both channels once inprepare(overridable via newcli_stable_version/cli_beta_versiondispatch inputs) and forwards the pins into the Vercel sandbox, so one nightly never mixes CLI versions across pairs. Each session logs the version it installed.DockerSandboxOptions.mountDockerSocket(defaulttrue, 3 lines inpackages/sandbox); the Docker-less arms create their sandbox without the socket and assert it's absent and that nodockerresolves on the agent's PATH.DOCKER_HOST+ shims stay as defence in depth.apps/framework/tsconfig.json'sexperimentsinclude pointed at a nonexistent directory (experiments were never typechecked); fixed, andpnpm checknow runs the_libandevals/cliunit tests (test:cli-lib).skipUnlessCli/skipUnlessDockerlesspredicates (mis-tagged evals skip instead of erroring), root-owned read-only runtime marker, single-quote shell escaping in the shims,needsDockerparse tests, CODEOWNERS forexperiments/_lib/and the arms, CONTRIBUTING spells out which evals each arm picks up.Questions for the framework team, deliberately left as-is:
needsDocker: booleanvs arequires: { docker }object (kanad's #307 already uses the boolean); moving the suite→experiment-suite mapping out of the workflowcase; session facts (runtime id, resolved CLI version) on the scoring context to replace the marker file.Related Issue(s)
Refs https://linear.app/supabase/issue/CLI-2398/add-a-docker-less-local-stack-e2e-eval-agent-cli-lifecycle (first eval: #281, stacked on this)
Context: #team-ai feedback thread → Linear project "Evals Driven Development".
Test plan
pnpm checkandpnpm format:checkgreenpnpm --filter @supabase-evals/framework exec vitest run --root ../.. experiments/_lib— these paths are outside every packagetestscript, sopnpm checkdoes not run themtsc --strictoverexperiments/(not covered bypnpm check:apps/framework/tsconfig.json'sexperimentsinclude resolves to a nonexistent dir — worth a one-line fix separately)pnpm eval:dryforregression × regressionandbenchmark × benchmarkunchanged;cli × clihas no evals until feat(evals): add build-database-002-stack-lifecycle CLI eval (CLI-2398) #281unset DOCKER_HOST; realsupabase starterrors surface; teardown leaves nothing behind), and with real agent runs via feat(evals): add build-database-002-stack-lifecycle CLI eval (CLI-2398) #281suite=cli,experiment_suite=cli