feat(eval): measure every domain a model can propose in - #60
Merged
Conversation
The corpus spans three domains and the benchmark measured one. eval hardcoded the network prompt and the network adapter, so `--dir scenarios/kubernetes` would have handed Kubernetes snapshots to an incident-bundle schema. The AI layer already had the right seam — ANALYSIS_DOMAINS, whose doc comment explains that terraform is absent because a plan already *is* the proposal. This registers kubernetes there and gives each entry its prompt and gate adapter, so eval resolves a domain instead of knowing one. Terraform stays out by design and says so: `eval --domain terraform` explains why and exits 2 rather than producing a number for a question nobody asked a model. The Kubernetes prompt mirrors the network one's posture — trusted instructions here, snapshot inside untrusted delimiters, every rule independently enforced afterwards — and states the domain's own trap: an operation carries a whole resource, so an omitted field is a deleted field. Its cross-check deliberately judges only forward `replace` against the snapshot. An `add` names a resource that is supposed not to exist yet, and a rollback `remove` undoes an add from the same proposal; checking either would reject correct proposals, as scenario-z-orphaned-canary-service demonstrates and now tests. Report schema goes to 3 with corpus.domain recorded. Two domains are two benchmarks: different prompts, corpora, and domain policies, so the scores must not be averaged, and a report that did not say which domain it measured would invite exactly that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c677d22041
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…tion gaps Addresses PR #60 review feedback: - eval's kubernetes domain now normalizes raw, collector-shaped input (normalizeSnapshot) instead of parsing it directly against the strict, already-normalized KubernetesSnapshotSchema. Every bundled Kubernetes scenario previously failed this parse before ever reaching the provider. Since normalizeSnapshot is not idempotent, parseKubernetesInput tries the strict parse first and only normalizes on failure, so both a raw snapshot (eval reading scenario fixtures off disk) and an already-normalized one (a caller driving the domain from the scenario registry) work. - describeValidIdentifiers no longer copies podLabels/selector values into the trusted preamble that precedes <untrusted_snapshot_data>. Those are Record<string,string> with no charset restriction beyond length, so an instruction-like value there bypassed the untrusted-content boundary the system prompt describes. The full snapshot already carries them inside the untrusted block via canonicalize(snapshot); nothing is lost. - crossCheck now validates rollback replace targets against the snapshot and only accepts a rollback remove when it undoes a forward add. Before, a rollback reference to a nonexistent resource passed local validation and probeProposal counted it "accepted", inflating eval's grounded-output metric for a proposal PATCH_SCHEMA would later block anyway. - defineAnalysisDomain binds a prompt and adapter under one generic TInput, so pairing a Kubernetes prompt with the network adapter (or any other mismatch) fails to typecheck instead of compiling via two independent `as unknown as never` casts. Full gate green: lint, typecheck, build, build:cli, 1178 tests, scenario check, gallery. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
9 tasks
wonkwonlee
added a commit
that referenced
this pull request
Aug 7, 2026
* chore(release): prepare v0.5.0 Bumps all nine workspace packages and every internal @changesafe/* range from 0.4.1 to 0.5.0. The currently-published 0.4.1 core carries a real skip-legitimacy gap (PR #61) and a broken Kubernetes eval path (PR #60); this is not a routine version bump but the release that gets those fixes into published hands. No breaking changes: policyCoverage is an additive optional receipt field, Terraform's new PLAN_CONTEXT_REQUIRED policy replaces a skip that already existed, and Kubernetes eval support only adds a domain --domain kubernetes could not previously use. CORE_POLICY_VERSION and TERRAFORM_POLICY_VERSION move independently of the package semver, as always. This prepares the release; it does not publish. Publishing still requires an explicit owner-authorized tag and GitHub Release. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(release): keep v0.5.0 pinned references at v0.4.1 until actually published Fixes review feedback on the v0.5.0 prep PR: the example GitHub Action workflow, README, and docs/LAUNCH.md all pointed wonkwonlee/ChangeSafe@v0.5.0 before that tag exists, so a workflow copied from the example would fail to resolve the action rather than run the gate. Reverted those to v0.4.1 (still published and valid) and reworded "canonical" claims to say v0.5.0 is prepared but not yet published, matching what MEMORY.md's release-status section already said correctly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
changesafe evalno longer hardcodes the network domain: it resolves anAnalysisDomainfrom--domain(defaultnetwork), and Kubernetes is now registered inANALYSIS_DOMAINSwith its own prompt (packages/ai/src/prompts/kubernetes.ts).--dirnow follows--domain(scenarios/${domain}), and the report schema bumps to version 3, recordingcorpus.domainso a report is only ever compared against another report of the same domain.main(v0.4.1) after this branch fell behind the v0.4.0/v0.4.1 release-prep commits; merged cleanly with the deadline/timeout work that landed on main in the meantime (packages/ai/src/provider.ts,--timeoutineval/main.ts).Test plan
npm run lintnpm run typechecknpm run buildnpm run build:clinpm test— 1174 passed, 4 skippednode packages/cli/dist/changesafe.js scenario checknode packages/cli/dist/changesafe.js scenario gallery --check🤖 Generated with Claude Code