[aboucaud]

Acknowledgements

This PR is built on lightcone-bench and brings its core into this repository:

• @dkn16 (Kangning) designed and built lightcone-bench, and this workflow takes several pieces from it:

  • running Harbor across harnesses;
  • the oracle gate;
  • the matrix that skips a leg when its secret is missing;
  • the ARG LIGHTCONE_COMMIT= contract for pinning the stack under test;
  • how skill bundles are found in agent-skills;
  • baselines taken from main;
  • the two-stage trace analysis: deterministic evidence first, then an LLM report.

  The Harbor task we run, lightcone-cli/snae-build in lightcone-evals, is also his port of this repository's earlier eval.

• @cailmdaley (Cail) contributed the trajectory metrics (lightcone-bench#1): agent steps, tool calls, tokens, cost and wall time, read from Harbor's ATIF trajectory. The agent-step count this workflow gates on is his metric.

The difference from the bench is packaging: everything lives in one workflow file that runs inside the repository under test, with no cross-repo dispatch, watcher scripts or bot commits.

What

eval.yml runs the eval on Harbor as one self-contained workflow file. agent-skills reuses the same file through workflow_call; see LightconeResearch/agent-skills#56.

• prepare:

  • Checks the legs input.
  • Fetches lightcone-cli/snae-build from lightcone-evals (at tasks_ref).
  • Rewrites the task's ARG LIGHTCONE_COMMIT= to the lightcone-cli commit under test.
  • Collects the built agent-skills bundles.

  The prompt is the one line a user would type, so how to drive lc and astra has to come from the skills and --help.

• oracle requires the task's scripted solution to score 1.0. No LLM is involved.

• agents runs each leg of legs, a JSON list of {name, agent, model, secret, env?}:

  • agent is any Harbor agent (claude-code, codex, opencode, …).
  • secret names the secret the leg needs, exported under its own name.
  • env adds extra variables.

  Here the default is one claude-code leg on claude-sonnet-5-5. Skills are installed through Harbor's --skill, and a leg whose secret is missing is skipped with a notice.

• report:

  • Reads each trial's result.json and ATIF trajectory with jq: reward, exception, agent steps, and whether a skill was loaded.
  • Fails on any of:
    • an oracle failure;
    • a reward below 1.0;
    • an errored trial;
    • a missing trace analysis;
    • with require_skills: true, a leg that never loaded a skill.
  • Off main, also fails when a leg's agent steps are more than 25% and at least 3 steps over the median of its last main runs, once three are on record. The history and the trajectories travel in the newest successful main run's eval-results artifact, so each calling repository keeps its own baseline.
  • Analysis: runs one claude -p per leg, limited to Read and Grep. It reads this run's trajectory, main's trajectory, the verifier output and the PR's commits, then explains what went wrong and where any extra steps came from. The result is posted as a sticky PR comment.

• Reuse: whichever repository triggered the run is checked out at its commit, and the other at its input ref (stack_ref / skills_ref). A caller picks its own legs and require_skills; the header comment has an example. Because everything the workflow runs is in the file, a caller can't get a script that doesn't match its workflow.

• Plugins and hooks: Harbor 0.22 has no route for them. --skill copies SKILL.md bundles, the agents expose no plugin or setup hook, and only a custom agent class could add one. Plugin installs therefore stay with agent-skills' e2e-hooks.yml.

• Removed: .github/scripts/trace_digest.py, evals/prompt.md and evals/tasks/, with no replacement outside eval.yml.

Needs

• REPO_READ_TOKEN: a secret with contents:read on lightcone-evals, needed while that repository is private. Without it prepare fails. Once lightcone-evals is public, the token: line and this requirement go away.

Verified

• First end-to-end run, on an earlier revision with the task vendored (1db117b): oracle 1.0, and claude-code reward 1.00 in 10 steps with skills loaded. The analysis was posted.

• Report steps: taken out of the workflow and run on these cases:

  • pass;
  • step regression;
  • no history;
  • errored with a low reward;
  • oracle failure;
  • a main run appending history;
  • require_skills on a leg with a skill call and on one without.

  On lightcone-bench's real ATIF fixture they count 59 agent steps, the same as the bench's Python metric.

• Legs: the default, a custom leg with env, and the three-leg list in agent-skills#56 pass. Malformed lists (missing keys, not JSON, empty) are refused in prepare.

• Analysis: one real claude -p --allowedTools Read Grep run read the raw trajectory and produced a step-cited analysis.

• Checks: actionlint is clean, and Tests is green on the head.

Open

• Which agent-skills version the eval should install is still up for discussion.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y9FLtiiMxVhhkvZjd9SQp4

[aboucaud]

test (ubuntu-latest, 3.13) is red for a reason outside this PR.

All 7 failures are in tests/test_container_smoke.py. Each image build stops at:

RUN UV_PYTHON_INSTALL_DIR=/opt/python uv python install 3.13.17
error: No download found for request: cpython-3.13.17-linux-x86_64-gnu

The runner's interpreter is now CPython 3.13.17. That is the .python-version lc init writes, and so the version the image installs. The image's pinned uv (image.UV_IMAGE, ghcr.io/astral-sh/uv:0.12.23) predates 3.13.17 and has no download for it. main was last green on Oct 15, before the runner moved, and would fail the same way today. This PR touches no src/ or tests/ file.

No open PR fixes it. The fix is the #248 shape: bump UV_IMAGE in src/lightcone/engine/image.py to a uv that knows 3.13.17 (the current release, uv 0.13.5, lists cpython-3.13.17-linux-x86_64-gnu), with its manifest-list digest. That moves the identity document, so containerized outputs read behind. That is an engine change of its own, so I'm proposing it as a separate PR rather than folding it in here. A re-run would hit the same missing download, so I haven't triggered one.

---

Generated by Claude Code

[github-actions]

✅ Eval

stack: LightconeResearch/lightcone-cli@1db117b · skills: LightconeResearch/agent-skills@278ecdb · skill_set: astra,lightcone · sha: 1db117b985f7a82a01e7dad5fb9a5966155c67b0

Leg	Reward	Steps	main median	Tool calls	Cost	Agent min	Skills
claude-code	1.00	10	—	11	$0.19	1.4	yes
codex	skipped: OPENAI_API_KEY not set

• ℹ️ claude-code: NEW — no main history to compare steps against

claude-code

Confusion & pain points

• Scaffold was not self-consistent (step 4). lc status --json failed with Error: 3 output(s) declare no format: best_fit, hubble_diagram, residuals. The seeded astra.yaml is incomplete, so the agent's first status probe was wasted. The error message is clear (it suggests format: png) and the agent fixed it later. If the seed is meant to be a starting point, the skill could say that lc status can fail on a half-written spec. This cost one step and caused no real harm.
• One over-stuffed exploratory command (step 4). The agent bundled lc status, several ls calls on nonexistent dirs (scripts, src), cat AGENTS.md (absent), and cat index.md pyproject.toml into one call. That produced the two ls errors and the AGENTS.md error. This is agent probing, not a product gap. A one-line statement of the project layout in the skill or the instruction would remove the guessing.
• Python environment check failed (step 5). The final .venv/bin/python -c "import numpy,scipy,matplotlib" exited 1, so the dependencies were not preinstalled. The agent then ran uv add numpy scipy matplotlib (step 6). That worked, but the Dockerfile or skill could say the deps need adding. This is a small detour.
• Looking for authorship/licence guidance (steps 5–7). The agent grepped the astra SKILL for "author|license", then tried uvx astra-tools@0.2.18 spec authors, which returned Unknown term: authors. It then ran spec Analysis | grep, which found nothing. Neither the astra skill nor the spec CLI says where authors or licence are declared. The lc materialize warning "no [project].license in pyproject.toml" is what exposes it. The skill should say that authorship and licence live in pyproject.toml. Source: the agent's own step 12 summary says the licence was "my choice", which suggests it guessed.
• Late spec validation, minor noise (steps 8, 11). The build went through in one big heredoc at step 8, and validation passed. At step 11 sed: couldn't edit /dev/null: not a regular file appeared, and lc materialize reported nothing to do. Both are harmless. The run was otherwise clean: reward 1.00, 10 steps, no repeated commands, and the skills were loaded and used (Skill lightcone at step 2, then the publishing.md and reporting.md references).

What changed vs main

No baseline trace is available, so no comparison is possible. This PR adds the Harbor eval harness itself (evals/snae-build/*, eval.yml, eval_report.py), so there is no main run on the same leg to compare against.

Evidence (deterministic)

Evidence — claude-code

Tool calls by phase (mean per trial):

phase	this run	main	Δ
build	2.0	—	—
compute	2.0	—	—
skills	4.0	—	—
spec	3.0	—	—

Skills loaded: yes

Failed or erroring tool calls (2):

• task__eempNRd step 4 Bash lc status --json; ls universes data results scripts src 2>&1; cat universes/*; head -5 data/*.txt; cat AGENTS.md 2>&1 | head -30; cat index.md pyproject.toml

  Error: 3 output(s) declare no `format:`: best_fit, hubble_diagram, residuals. lc names each output's file from it, so there is nowhere to write them — add the artifact's file extension, e.g. `format: png`.
  ls: cannot access 'scripts': No such file or directory
  ls: cannot access 'src': No such file or directory
  data:
  README.md
  SCPUnion2.1_mu_vs_z.txt
  
  results:
  README.md
  
  universes:
  baseline.yaml
  id: baseline
  description: Default configuration using standard practices
  decisions:
    optimizer: nelder_mead
    error_model: stat_and_sys
    redshift_cut: z_min_001
  # alpha 0.121851859725
  # beta 2.4656927… [2172 more chars]
  

• task__eempNRd step 5 Bash cat data/README.md; sed -n 1,25p data/SCPUnion2.1_mu_vs_z.txt; wc -l data/*.txt; grep -n -i -A15 "author\|license" /logs/agent/sessions/skills/astra/SKILL.md | head -60; ls /logs/agent/sessions/skills… [60 more chars]

  Exit code 1
  # data/
  
  Declared inputs live here, and `astra.yaml` references them by path:
  
      inputs:
        catalog:
          source: data/catalog.fits
  
  git-annex holds the bytes and git holds a pointer, so the repository
  records exactly which data produced a result without carrying its size.
  # alpha 0.121851859725
  # beta 2.46569277393
  # delta -0.0363405630486
  # M(h=0.7, statistical only) -19.3182761161
  # M(h=0.7, with systematics) -19.3081547178
  1993ah	0.028488	35.3465833928	0.223905932998	0.128418942246
  1993ag	0.050043	36.6823679154	0.166828851413	0.128418942246
  1993o	0.052926	36.8176912545	… [4496 more chars]
  

Commands run more than once:

• none

Verifier output:

Validating astra.yaml...
✓ Schema validation passed
✓ Semantic validation passed

Validation successful!
  mode:    direct
  sandbox: landlock (fs: declared, network: allowed)
  crate:   up to date with the outputs

  · current  baseline/best_fit        e9c7d74
  · current  baseline/hubble_diagram  e9c7d74
  · current  baseline/residuals       e9c7d74

3 current
{"reward": 1.0000, "spec_valid": 1.0, "outputs_current": 1.0000, "publication_ready": 1.0}

Run: https://github.com/LightconeResearch/lightcone-cli/actions/runs/39736199038

[aboucaud]

Eval / prepare is red: the REPO_READ_TOKEN secret is missing.

Since d27a8a5 the eval fetches its tasks from LightconeResearch/lightcone-evals. That repository is private, so the workflow needs a token that can read it. The step fails on purpose, with REPO_READ_TOKEN is not set — it needs contents:read on LightconeResearch/lightcone-evals.

Fix: add a repository secret REPO_READ_TOKEN here (Settings → Secrets and variables → Actions). It should be a fine-grained PAT, or an org GitHub App token, with Contents: read on LightconeResearch/lightcone-evals. lightcone-bench uses the same secret name. agent-skills will need the same secret once it calls this workflow.

Making lightcone-evals public would remove the need for the token. Nothing in this PR can supply the token, so I'll re-run once it's in place.

---

Generated by Claude Code

[aboucaud]

Which agent-skills version should the eval install?

This is open for discussion; nothing in the PR changes because of it yet.

Today: prepare installs agent-skills at skills_ref, which defaults to main. It copies the built plugins/*/skills/*/ bundles, the same ones the marketplace serves. An agent-skills PR (as a workflow_call caller) always uses its own head commit. The exact commit is recorded in refs.json and in the report header (agent-skills@278ecdb on this PR's first run).

What "published" means for agent-skills

• Both marketplace manifests point at local paths ("source": "./plugins/lightcone"). claude plugin marketplace add LightconeResearch/agent-skills therefore installs whatever the default branch holds. There is no separate release artifact.
• The plugin version in skills.config.json (0.0.6 now) is what tells an existing install to update. It doesn't point to a copy of the files.
• There are no release tags to pin to. The only tag, v0.0.2, predates the lightcone plugin.

Today main (278ecdb) is the 0.0.6 release commit, so main and "latest published" coincide. They split as soon as a skill change merges without a version bump. It is then live for new installs, while existing users don't see it until the next bump.

The weakness of following main as it moves

A lightcone-cli PR's agent steps are compared with the median of recent lightcone-cli main runs. Each of those runs, and the PR run, used whatever agent-skills main was at the time. A skill rewrite merged in between would then show up as a step change blamed on the lightcone-cli PR.

Options

	Skills used by lightcone-cli runs	For	Against
A. main (current)	Newest agent-skills commit	Matches a fresh install	Skill changes move the lightcone-cli baseline, and the PR gets blamed
B. Last release	The commit that set the current plugin version	Matches what existing users run; skill changes reach the eval only when released	Needs agent-skills to tag releases (e.g. lightcone-v0.0.6 in the release PR), or the workflow to search history for the bump
C. The baseline's commit	The agent-skills commit the latest successful lightcone-cli main run used, read from its eval-results	PR and baseline always run identical skills, so a difference is the lightcone-cli change; small, all in this repo	Skill updates reach lightcone-cli PRs only after the next lightcone-cli main run

None of these affect an agent-skills PR, which always runs its own head.

Proposal

C for PR runs, with main runs choosing the skills. Under the proposal, main runs use agent-skills main today. Once agent-skills tags each plugin release, they would use the latest release tag (B), so the baseline tracks what users have rather than unreleased edits.

The step history should also start fresh, or be marked, whenever the skills commit changes between main runs. Otherwise a release that legitimately changes step counts either fails PRs or gets absorbed slowly by the median.

The lightcone-cli side is the prepare step plus a few lines in eval_report.py. The tagging would live in agent-skills' release process.

---

Generated by Claude Code

[aboucaud]

@EiffL for the agent-skill question above I would go with option C which ensures that a PR eval check is doing its job by comparing results with what's on main

Do you agree ?

[aboucaud]

This is what it allows now that everything has been folded in a single workflow.