feat(evals): add build-cli-004-worktree-stacks (CLI-2400)

evals/regression/build-cli-004-worktree-stacks/EVAL.ts

@@ -0,0 +1 @@
+export { worktreeStacksScorer as default } from './scoring.js';

evals/regression/build-cli-004-worktree-stacks/PROMPT.md

@@ -0,0 +1,17 @@
+---
+stage: build
+interface: cli
+product:
+  - database
+topic:
+  - migrations
+projectRunning: false
+motivation: https://linear.app/supabase/issue/CLI-2400/build-cli-007-worktree-stacks-one-stack-per-git-worktree-schemadata, https://linear.app/supabase/issue/FDBKIN-20391/support-running-multiple-isolated-supabase-stacks-locally-for-parallel
+---
+
+I'm about to have three coding agents work on features in parallel, each in
+its own git worktree. Set this repo up with worktrees `feature-a`, `feature-b`,
+and `feature-c`, and get a local Supabase stack running in each one.
+`feature-a` needs a `widgets` table, `feature-b` needs `gadgets`, and
+`feature-c` needs `gizmos`. Drop one sample row in each and tell me how to
+reach each stack.

evals/regression/build-cli-004-worktree-stacks/README.md

@@ -0,0 +1,63 @@
+# build-cli-004-worktree-stacks
+
+One human plus N coding agents, each in its own git worktree, each needing its
+own isolated local Supabase stack. The agent creates a repo with three
+worktrees (`feature-a`, `feature-b`, `feature-c`), starts a stack in each, and
+applies a different migration plus one seed row per worktree. The scorer then
+proves the three stacks are really three: distinct database endpoints, and
+each table present in exactly one of them.
+
+Tracks [CLI-2400](https://linear.app/supabase/issue/CLI-2400) under the
+[Slim CLI evals RFC](https://linear.app/supabase/issue/CLI-2393).
+
+## What the scorer checks
+
+All checks inspect end state through `ctx.exec`. The harness's built-in
+`ctx.query` / `ctx.stackStatus` resolve a single stack from the workspace root,
+so this eval addresses each stack itself by `cd`-ing into its worktree.
+
+1. Three real git worktrees named `feature-a/b/c`, each on its own branch
+   (`git worktree list --porcelain`). Three plain directories do not count.
+2. One live stack per worktree, with three distinct `host:port` database
+   endpoints. Two worktrees reporting the same endpoint means the CLI aliased
+   or reused a stack.
+3. Schema isolation, one check per table: `to_regclass('public.<table>')` is
+   non-null only against the home worktree's stack.
+4. Data isolation: at least one row in each table in its home stack.
+5. Each table is created by a migration file in its worktree, not ad hoc SQL.
+6. A `metrics` check that always passes and reports fleet wall-clock (latest
+   `pg_postmaster_start_time()` across the three stacks minus session start),
+   the CLI version and channel, the backend and runtime each stack resolved to,
+   and how many `supabase start` invocations the agent needed.
+
+## How stacks are resolved
+
+The scorer asks the managed stack first
+(`SUPABASE_EXPERIMENTAL_STACK=1 supabase stack status --env --output-format json`)
+and falls back to the legacy backend
+(`SUPABASE_EXPERIMENTAL_STACK=0 supabase status -o json`). Both are forced via
+the env var so the result does not depend on how the agent enabled the flag.
+The managed backend rejects the legacy `-o json` flag, and the legacy backend
+knows nothing about managed stacks, so neither alone is enough.
+
+## Deliberate choices
+
+- **No `services:` frontmatter.** With it set, the sandbox shim appends
+  `-x gotrue,kong,...` to every `supabase start`; the managed backend only
+  accepts capability names (`rest`, `auth`, ...) and rejects the command
+  outright. Omitting `services` keeps the managed path reachable. The agent
+  gets the full stack per `config.toml` and may trim it itself.
+- **Nothing tells the agent about the managed stack, the feature flag, or
+  port collisions.** Worktree isolation only exists on the managed stack,
+  which is gated behind `experimental.stack` / `SUPABASE_EXPERIMENTAL_STACK`
+  and ships in the beta channel. Whether agents can discover that is part of
+  what's being measured. Expect every arm to fail until agents find the flag
+  or the CLI flips the default.
+- **Known footgun, left in.** `supabase init` writes explicit ports into
+  `config.toml`. The managed stack treats those as exact intents, so three
+  worktrees sharing the committed config collide on the second start
+  ("Persisted database port is unavailable") until the ports are removed and
+  become automatic.
+- **Environment-agnostic**, like `build-database-002-stack-lifecycle`: the
+  same eval runs under the pinned experiment and every `-cli-*` arm, and the
+  scorer never branches on which one it is in.

evals/regression/build-cli-004-worktree-stacks/scoring.test.ts

@@ -0,0 +1,199 @@
+// Run: pnpm --filter @supabase-evals/framework exec vitest run --root ../.. evals/regression/build-cli-004-worktree-stacks
+import { describe, expect, it } from 'vitest';
+import {
+  countSupabaseStarts,
+  endpointKey,
+  matchWorktrees,
+  migrationCreatesTable,
+  parseJsonObject,
+  parseWorktreeList,
+} from './scoring.js';
+
+// Captured from `git worktree list --porcelain` on a repo with three linked
+// worktrees (paths shortened).
+const PORCELAIN = `worktree /tmp/sandbox-1234/repo
+HEAD 14b5b37788863c554f10ecf818490f56f2d51e4b
+branch refs/heads/main
+
+worktree /tmp/sandbox-1234/feature-a
+HEAD 14b5b37788863c554f10ecf818490f56f2d51e4b
+branch refs/heads/feature-a
+
+worktree /tmp/sandbox-1234/feature-b
+HEAD 14b5b37788863c554f10ecf818490f56f2d51e4b
+branch refs/heads/feature-b
+
+worktree /tmp/sandbox-1234/feature-c
+HEAD 14b5b37788863c554f10ecf818490f56f2d51e4b
+branch refs/heads/feature-c
+`;
+
+const NAMES = ['feature-a', 'feature-b', 'feature-c'];
+
+describe('parseWorktreeList', () => {
+  it('parses one entry per worktree with its branch', () => {
+    const entries = parseWorktreeList(PORCELAIN);
+    expect(entries.map((e) => [e.path, e.branch])).toEqual([
+      ['/tmp/sandbox-1234/repo', 'refs/heads/main'],
+      ['/tmp/sandbox-1234/feature-a', 'refs/heads/feature-a'],
+      ['/tmp/sandbox-1234/feature-b', 'refs/heads/feature-b'],
+      ['/tmp/sandbox-1234/feature-c', 'refs/heads/feature-c'],
+    ]);
+    expect(entries.every((e) => !e.detached && !e.bare)).toBe(true);
+  });
+
+  it('marks detached and bare worktrees', () => {
+    const entries = parseWorktreeList(
+      'worktree /r\nbare\n\nworktree /r/x\nHEAD abc\ndetached\n'
+    );
+    expect(entries[0]).toMatchObject({ path: '/r', bare: true });
+    expect(entries[1]).toMatchObject({ path: '/r/x', detached: true });
+    expect(entries[1]?.branch).toBeUndefined();
+  });
+
+  it('returns nothing for empty or non-porcelain output', () => {
+    expect(parseWorktreeList('')).toEqual([]);
+    expect(parseWorktreeList('fatal: not a git repository')).toEqual([]);
+  });
+});
+
+describe('matchWorktrees', () => {
+  it('accepts three worktrees on three distinct branches', () => {
+    const { matched, problems } = matchWorktrees(
+      parseWorktreeList(PORCELAIN),
+      NAMES
+    );
+    expect(problems).toEqual([]);
+    expect(matched['feature-b']?.branch).toBe('refs/heads/feature-b');
+  });
+
+  it('reports a missing worktree', () => {
+    const entries = parseWorktreeList(PORCELAIN).filter(
+      (e) => !e.path.endsWith('feature-c')
+    );
+    expect(matchWorktrees(entries, NAMES).problems).toEqual([
+      'no worktree named feature-c',
+    ]);
+  });
+
+  it('rejects two worktrees on the same branch', () => {
+    const entries = parseWorktreeList(PORCELAIN).map((e) =>
+      e.path.endsWith('feature-c')
+        ? { ...e, branch: 'refs/heads/feature-b' }
+        : e
+    );
+    expect(matchWorktrees(entries, NAMES).problems).toEqual([
+      'more than one worktree is on refs/heads/feature-b',
+    ]);
+  });
+
+  it('rejects a detached worktree', () => {
+    const entries = parseWorktreeList(PORCELAIN).map((e) =>
+      e.path.endsWith('feature-a')
+        ? { ...e, branch: undefined, detached: true }
+        : e
+    );
+    expect(matchWorktrees(entries, NAMES).problems).toEqual([
+      'feature-a is not checked out on a branch',
+    ]);
+  });
+
+  it('matches by basename, so a nested repo layout still works', () => {
+    const entries = parseWorktreeList(
+      PORCELAIN.replaceAll(
+        '/tmp/sandbox-1234/',
+        '/tmp/sandbox-1234/repo/.worktrees/'
+      )
+    );
+    expect(matchWorktrees(entries, NAMES).problems).toEqual([]);
+  });
+});
+
+describe('parseJsonObject', () => {
+  it('skips the CLI task progress lines around the JSON payload', () => {
+    const stdout = `[task] start: Starting local Supabase stack...
+[task] done: Stack is ready.
+{"id":"963a","lifecycle":"running","runtime":{"kind":"native"}}
+`;
+    expect(parseJsonObject(stdout)).toEqual({
+      id: '963a',
+      lifecycle: 'running',
+      runtime: { kind: 'native' },
+    });
+  });
+
+  it('returns the error envelope the managed backend prints for a missing stack', () => {
+    const stdout =
+      '{"_tag":"Error","error":{"code":"ExperimentalStackStatusError","message":"No managed stack exists for the selected project."}}';
+    expect(parseJsonObject(stdout)?._tag).toBe('Error');
+  });
+
+  it('is undefined for non-JSON output', () => {
+    expect(parseJsonObject('')).toBeUndefined();
+    expect(
+      parseJsonObject('The legacy -o/--output flag is not supported here')
+    ).toBeUndefined();
+    expect(parseJsonObject('{not json}')).toBeUndefined();
+  });
+});
+
+describe('endpointKey', () => {
+  it('reduces a connection URL to host:port, ignoring credentials', () => {
+    expect(
+      endpointKey(
+        'postgresql://postgres:ed941619-34a3-4772-bf78-9d9c855ecb64@127.0.0.1:25304/postgres'
+      )
+    ).toBe('127.0.0.1:25304');
+    expect(
+      endpointKey('postgresql://postgres:postgres@127.0.0.1:54322/postgres')
+    ).toBe('127.0.0.1:54322');
+  });
+
+  it('tells two stacks with different passwords but the same port apart as the same endpoint', () => {
+    const a = endpointKey('postgresql://postgres:aaa@127.0.0.1:54322/postgres');
+    const b = endpointKey('postgresql://postgres:bbb@127.0.0.1:54322/postgres');
+    expect(a).toBe(b);
+  });
+
+  it('is undefined for garbage', () => {
+    expect(endpointKey('not a url')).toBeUndefined();
+  });
+});
+
+describe('migrationCreatesTable', () => {
+  it.each([
+    'create table widgets (id serial primary key);',
+    'CREATE TABLE IF NOT EXISTS public.widgets (id int);',
+    'create table "public"."widgets" ("id" int);',
+    'create table\n  public.widgets (\n  id int\n);',
+  ])('matches: %s', (sql) => {
+    expect(migrationCreatesTable(sql, 'widgets')).toBe(true);
+  });
+
+  it.each([
+    'create table gadgets (id int);',
+    'create table widgets_archive (id int);',
+    "insert into widgets values ('w1');",
+  ])('does not match: %s', (sql) => {
+    expect(migrationCreatesTable(sql, 'widgets')).toBe(false);
+  });
+});
+
+describe('countSupabaseStarts', () => {
+  it('counts legacy and managed start invocations across commands', () => {
+    expect(
+      countSupabaseStarts([
+        'cd feature-a && supabase start',
+        `bash -lc 'cd feature-b && SUPABASE_EXPERIMENTAL_STACK=1 supabase stack start --runtime native'`,
+        'supabase status',
+        'cd feature-c && supabase start -x studio && supabase start',
+      ])
+    ).toBe(4);
+  });
+
+  it('ignores unrelated mentions', () => {
+    expect(
+      countSupabaseStarts(['echo "run supabase start later"', 'git status'])
+    ).toBe(1);
+  });
+});