Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
eef60ee
feat(evals): add resolve-database-002-stale-stack-cleanup CLI eval (C…
Coly010 Sep 21, 2026
1b1ae55
refactor(evals): read the local-stack environment marker through ctx.…
Coly010 Sep 23, 2026
b22d5ee
refactor(evals): extract shared cli eval helpers into evals/cli/lib
Coly010 Oct 1, 2026
7a6ecb0
Merge remote-tracking branch 'origin/main' into columferry/cli-2402-r…
Coly010 Oct 1, 2026
098158d
Merge remote-tracking branch 'origin/columferry/cli-evals-shared-lib'…
Coly010 Oct 1, 2026
fd88032
refactor(evals): restructure resolve-database-002-stale-stack-cleanup…
Coly010 Oct 1, 2026
6246e66
chore: refresh eval results [skip ci]
github-actions[bot] Oct 1, 2026
cc4d596
fix(evals): disambiguate stale-stack-cleanup prompt and ground truth
Coly010 Oct 1, 2026
85de4c9
chore: refresh eval results [skip ci]
github-actions[bot] Oct 1, 2026
0f3e342
feat(evals): share supabase invocation parsing in evals/cli/lib
Coly010 Oct 1, 2026
be6e01b
Merge remote-tracking branch 'origin/columferry/cli-evals-shared-lib'…
Coly010 Oct 1, 2026
9f357a1
feat(evals): report whether the agent attempted each stack start
Coly010 Oct 1, 2026
2c597fb
chore: refresh eval results [skip ci]
github-actions[bot] Oct 1, 2026
2898838
feat(evals): attribute supabase invocations by working directory
Coly010 Oct 1, 2026
d831c80
Merge remote-tracking branch 'origin/columferry/cli-evals-shared-lib'…
Coly010 Oct 1, 2026
927ea9a
fix(evals): attribute stale-stack fleet commands by working directory
Coly010 Oct 1, 2026
01c7230
chore: refresh eval results [skip ci]
github-actions[bot] Oct 1, 2026
a657656
fix(evals): attribute more invocation shapes and resolve named stacks…
Coly010 Oct 1, 2026
72bc83a
Merge remote-tracking branch 'origin/columferry/cli-evals-shared-lib'…
Coly010 Oct 1, 2026
803aa1f
fix(evals): harden stale-stack fleet checks against failed and setup-…
Coly010 Oct 1, 2026
fa2b2f4
chore: refresh eval results [skip ci]
github-actions[bot] Oct 1, 2026
681a3e7
fix(evals): address shared cli lib review findings
Coly010 Oct 1, 2026
fd28fea
feat(evals): carry tool call times onto supabase invocations
Coly010 Oct 1, 2026
01734d7
Merge remote-tracking branch 'origin/columferry/cli-evals-shared-lib'…
Coly010 Oct 1, 2026
187245d
feat(evals): judge stale-stack restarts from database start times
Coly010 Oct 1, 2026
6ff1958
fix(evals): take invocation times only from recorded completion times
Coly010 Oct 1, 2026
083f793
Merge remote-tracking branch 'origin/columferry/cli-evals-shared-lib'…
Coly010 Oct 1, 2026
a4c7a82
fix(evals): fall back to commands when setup and a restart share a call
Coly010 Oct 1, 2026
99f6a1e
chore: refresh eval results [skip ci]
github-actions[bot] Oct 1, 2026
f08b14c
Merge origin/main into columferry/cli-2402-resolve-cli-001-stale-stac…
Coly010 Oct 7, 2026
317c210
fix(evals): align stale-stack-cleanup with the merged shared lib
Coly010 Oct 7, 2026
ffc2ecc
chore: refresh eval results [skip ci]
github-actions[bot] Oct 7, 2026
324adbb
fix(evals): attribute stack ids and anchor setup on completion time
Coly010 Oct 7, 2026
b2d9a3a
chore: refresh eval results [skip ci]
github-actions[bot] Oct 7, 2026
560c799
Merge remote-tracking branch 'origin/main' into columferry/cli-2402-r…
Coly010 Oct 7, 2026
9fdb684
fix(evals): resolve named managed stacks per project
Coly010 Oct 7, 2026
f03de9d
chore(evals): align stale-stack-cleanup with main after tool call cwd…
Coly010 Oct 7, 2026
631643b
chore: refresh eval results [skip ci]
github-actions[bot] Oct 7, 2026
645d94b
fix(evals): resolve stacks under an agent-relocated CLI home
Coly010 Oct 7, 2026
bbfd353
fix(evals): handle relocated homes and swapped CLI versions in stale-…
Coly010 Oct 7, 2026
186831a
chore: refresh eval results [skip ci]
github-actions[bot] Oct 7, 2026
2009254
fix(evals): decide legacy-import teardown from end state
Coly010 Oct 7, 2026
2141d26
fix(evals): find stale-stack services via the CLI's stack list
Coly010 Oct 7, 2026
ce0819d
chore: refresh eval results [skip ci]
github-actions[bot] Oct 7, 2026
62da11e
fix(evals): attribute supabase invocations by directory before stack …
Coly010 Oct 7, 2026
d5dd12e
fix(evals): use directory attribution and latest-start swap rule in s…
Coly010 Oct 7, 2026
5bca1de
fix(evals): credit starts that reported ready inside a failed call
Coly010 Oct 7, 2026
f4b3c32
chore: refresh eval results [skip ci]
github-actions[bot] Oct 7, 2026
0577256
fix(evals): expand $PWD in supabase --workdir values
Coly010 Oct 7, 2026
31c4d95
fix(evals): keep unresolved starts from anchoring setup
Coly010 Oct 7, 2026
e3daa80
fix(evals): treat db reset as a database restart
Coly010 Oct 7, 2026
6e64920
fix(evals): find stacks via stack list when config.toml resolution fails
Coly010 Oct 7, 2026
2c3e1de
docs(evals): bring stale-stack-cleanup results up to date
Coly010 Oct 7, 2026
12a3334
chore: refresh eval results [skip ci]
github-actions[bot] Oct 7, 2026
12dee3b
fix(evals): address stale-stack-cleanup review (swap rule, marker mat…
Coly010 Oct 7, 2026
96ae774
fix(evals): expand $PWD against the shell cwd before env -C
Coly010 Oct 7, 2026
459b8a1
chore: refresh eval results [skip ci]
github-actions[bot] Oct 7, 2026
e29c3ae
fix(evals): sync shared cli lib from the parallel-projects eval
Coly010 Oct 7, 2026
72aa278
fix(evals): order a seeding reset sharing a call with a restart by co…
Coly010 Oct 7, 2026
88a8bb0
chore: refresh eval results [skip ci]
github-actions[bot] Oct 7, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5,720 changes: 4,626 additions & 1,094 deletions apps/web/src/data/cli-eval-results.json

Large diffs are not rendered by default.

16 changes: 16 additions & 0 deletions evals/cli/lib/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,22 @@ What stays in each eval's `EVAL.ts`: check composition, every `ctx.judge()`
call, its scenario-specific rubric text, and `export default`. Scenario-only
helpers with a single consumer stay colocated with that eval.

Project stack targets (`{ kind: 'project', dir }` without a stack name) also
discover named managed stacks started from that directory via
`supabase stack list`, before falling back to the legacy backend.

`resolveStackWithAgentHomes` retries a failed resolution under each
`SUPABASE_HOME`/`HOME` the agent started the target with (`cli-invocations.ts`
records them per invocation, including an earlier `export` in the same
command) and marks the result with `relocatedHome`; `listCliOverrides` reports
`npx`/`bunx`/`npm exec`/`pnpm dlx`/`yarn dlx` runs and global installs of
another explicit CLI version, and `listUnverifiedRunners` the dist-tag ones.

`invocationTargets` attributes an invocation to the directory it ran in
before its `--stack`/`--project-id` name; pass `knownTargets` (the eval's
sibling project or stack names) so a name given inside another known
target's directory isn't credited to the named one.

Invocation attribution (`cli-invocations.ts`) starts from a tool call's `cwd`
only when the agent parser records one (Codex, OpenCode); a `cd` persisting
across separate tool calls in a persistent shell (e.g. Claude Code) is not
Expand Down
Loading