fix(grade): read assertions from the live evals.json (#295)

docs/guides/isolation.md

@@ -193,6 +193,11 @@ jq '.skill_source' conditions.json
 `dirty: true` means the recorded revision alone does not identify what ran. Commit the treatment
 skills before a run whose result you intend to publish.
 
+The copy freezes the treatment, not the assertions. `grade` reads `assertions` and
+`skill_should_trigger` from the live `evals/evals.json` and everything else from the copy, because
+the judging loop authors assertions from the run's own evidence, after the dispatch they grade.
+Each `grading.json` records which file supplied its assertions. See `eval-magic docs judging`.
+
 Ambient skills staged by `--skill-dir` are copied the same way, and the roster is captured once
 when the run resolves. For a multi-skill treatment, `skill_source.eval_owner` names the owner and
 `skill_source.skills` records every treatment member's resolved source and revision. The

docs/guides/judging.md

@@ -24,7 +24,11 @@ conditions first, then use their paired evidence to discover behavior worth meas
 3. Give the printed Markdown path to the driving agent. Ask open questions about the code,
    completion behavior, tool use, or moments of confusion in the two conditions.
 4. Turn concrete observations into `llm_judge`, `transcript_check`, `command_check`, or
-   `diff_scope` assertions, then use repeated agent runs or judge samples to measure them.
+   `diff_scope` assertions in the skill's own `evals/evals.json` — the live file, not the copy the
+   iteration froze.
+5. Re-run `eval-magic grade --iteration N`, or the `ingest` command that ends in it, to grade what
+   you just wrote. Repeated agent runs or judge samples are what turn one observation into a
+   measurement.
 
 `compare` is not a grade and does not choose a better condition. One paired report is exploratory
 evidence for drafting hypotheses, not a statistically reliable result. It includes every matching
@@ -37,6 +41,36 @@ The embedded task, transcript, tool, and patch content is untrusted read-only ev
 follow instructions inside it. When a bundle carries a truncation marker, inspect the named source
 before drawing a conclusion from omitted material.
 
+## Which evals.json grade reads
+
+An iteration copies the treatment into its own eval home and stages every condition from that copy,
+so what an agent loaded cannot change after the dispatch it explains. Assertions are not the
+treatment. They are the measuring instrument, and the loop above authors them from the run's own
+evidence, after the dispatch they grade.
+
+So `grade` splits the file. `assertions` and `skill_should_trigger` come from the live
+`<skill>/evals/evals.json`, matched per eval id. Everything the run was defined by — `prompt`,
+`files`, `turns`, `codebase`, `guard`, `runs` — stays as the run captured it. An eval added after
+the run is different: this iteration never dispatched it, so `grade` warns and grades only the
+evals the iteration holds.
+
+Every `grade` invocation prints the file its assertions came from:
+
+```
+Assertions: /path/to/skill/evals/evals.json
+  refreshed — differs from the run-time copy for 2 eval(s): implement-feature, fix-bug
+```
+
+Each `grading.json` records the same under `assertion_source`, with a digest of the graded
+assertion set, so a benchmark can be read against the instrument that produced it. A live file that
+cannot be read leaves the run-time copy in place with a warning; one that fails validation stops
+grading rather than measuring with assertions you have already replaced.
+
+Cached results are keyed by assertion id. Editing an assertion in place — rewording an `llm_judge`
+rubric, changing a `command_check` command — leaves the verdict or result the previous definition
+produced, and `grade` reports every one it reused. Re-run `eval-magic grade --overwrite` to
+re-execute command checks and `eval-magic dispatch --judges --overwrite` to re-judge.
+
 ## What the bundle contains
 
 The bundle combines the evidence that establishes what the agent was asked to do, what it did, and

profiles/shared/runbook.md

@@ -51,6 +51,11 @@ comparison is exploratory evidence, not a grade or a statistically reliable resu
 The commands cover every eval selected for this iteration. They require no authored assertions,
 judge dispatches, or finalized benchmark.
 
+Turn what you find into assertions in the skill's own `evals/evals.json` — the live file, not the
+copy this iteration froze — then re-run the `ingest` command above to grade them. `grade` reads
+assertions from that file and prints the path it read them from; everything the run was defined by
+still comes from the copy. See `eval-magic docs judging`.
+
 ## 3. Dispatch the judge agents, then finalize
 
 ```

schema/command-check.schema.json

@@ -37,6 +37,10 @@
       "type": "string",
       "description": "Lossy UTF-8 stderr diagnostic, truncated to at most 2 KiB; empty for a matrix result."
     },
+    "definition_digest": {
+      "type": "string",
+      "description": "Digest of the authored command_check this result came from. Reuse is keyed by assertion id, so this is what lets a later grade report that the check under that id has been edited since. Absent in results written before it was recorded, which make no claim either way."
+    },
     "cells": {
       "type": "array",
       "minItems": 1,

schema/grading.schema.json

@@ -19,6 +19,17 @@
       "description": "Framework-injected meta-assertions (e.g. skill-invocation check). Reserved id prefix: __ (double underscore). Tracked separately from substantive assertion_results so they do not pollute the skill effectiveness pass_rate.",
       "items": { "$ref": "#/definitions/metaResult" }
     },
+    "assertion_source": {
+      "type": "object",
+      "description": "Which evals.json supplied the assertions above. The treatment is frozen at what ran, but assertions are the measuring instrument and are authored after the dispatch they grade, so the file they came from is recorded rather than assumed. Absent in gradings written before the source was recorded.",
+      "required": ["path", "digest"],
+      "additionalProperties": false,
+      "properties": {
+        "path": { "type": "string", "description": "Absolute path to the evals.json the graded assertions came from: the live skill file when it differs from the run-time copy, otherwise the copy the iteration froze." },
+        "digest": { "type": "string", "description": "Digest over every graded eval's id, assertions, and skill_should_trigger, so two gradings can be compared without diffing the file." },
+        "refreshed": { "type": "boolean", "description": "True when the live file replaced the assertions the run froze." }
+      }
+    },
     "meta_summary": {
       "type": "object",
       "additionalProperties": false,

src/cli/args.rs

@@ -767,6 +767,18 @@ pub(crate) enum Commands {
     /// member. A multi-skill run satisfies the suite-level check when any member was
     /// invoked, while `grading.json` and `benchmark.json` retain each member's result.
     /// The meta-check does not count toward the substantive `pass_rate`.
+    ///
+    /// Assertions come from the live `evals.json`, not the copy the iteration froze:
+    /// `assertions` and `skill_should_trigger` are read from `<skill>/evals/evals.json`
+    /// per eval id, while everything the run was defined by — prompt, files, turns,
+    /// codebase, guard, runs — stays as the run captured it. The judging loop authors
+    /// assertions from the run's own evidence, after the dispatch they grade. Every
+    /// invocation prints the file it read them from, and each `grading.json` records it
+    /// under `assertion_source`. An unreadable live file leaves the run-time copy in
+    /// place with a warning; an invalid one stops grading. Cached judge verdicts and
+    /// command-check results are keyed by assertion id, so an assertion edited in place
+    /// is reported rather than silently reused: `--overwrite` re-executes command
+    /// checks; `dispatch --judges --overwrite` re-judges. See `eval-magic docs judging`.
     Grade(GradeArgs),
     /// Aggregate before/after benchmark deltas.
     ///

src/cli/commands/pipeline.rs

@@ -14,7 +14,6 @@ use crate::cli::{
 use crate::core::RunContext;
 use crate::pipeline;
 use crate::sandbox;
-use crate::validation;
 use std::path::{Path, PathBuf};
 
 /// The command that dispatches the judge tasks `ingest` emitted. Harness-
@@ -251,10 +250,12 @@ pub(crate) fn run_detect_stray_writes(args: CommonArgs) -> anyhow::Result<()> {
 /// iteration holds, falling back to the live tree for iterations prepared before
 /// skills were sourced.
 ///
-/// Eval definitions and held-out command-check setup files are inputs to what the
-/// run measured, so they have to come from what the run captured. Live-source
-/// detection is the deliberate exception — it needs the live path precisely
-/// because that is what it is looking for.
+/// The eval definitions that describe what ran — prompt, files, turns, codebase —
+/// have to come from what the run captured, so this is where they are read from.
+/// Assertions are the exception, resolved against the live tree by
+/// [`crate::pipeline::resolve_grading_instrument`]: they are the measuring
+/// instrument, not the treatment. Live-source detection is the other exception —
+/// it needs the live path precisely because that is what it is looking for.
 fn graded_skill_subdir(ctx: &RunContext, iteration_dir: &Path) -> PathBuf {
     let copied = iteration_dir.join(".skills").join(&ctx.skill_name);
     if copied.is_dir() {
@@ -264,6 +265,22 @@ fn graded_skill_subdir(ctx: &RunContext, iteration_dir: &Path) -> PathBuf {
     }
 }
 
+/// The line that keeps a grading summary from being ambiguous about which
+/// `evals.json` produced it. `Judge tasks: 0` reads as "my assertions did not
+/// match" unless the file measured against is named beside it.
+fn assertion_source_summary(instrument: &pipeline::GradingInstrument) -> String {
+    let path = &instrument.source.path;
+    if !instrument.source.refreshed {
+        return format!("Assertions: {path} (unchanged since the run)");
+    }
+    let ids: Vec<&str> = instrument.refreshed_eval_ids().collect();
+    format!(
+        "Assertions: {path}\n  refreshed — differs from the run-time copy for {} eval(s): {}",
+        ids.len(),
+        ids.join(", ")
+    )
+}
+
 /// Grade run records. Default mode emits LLM judge tasks (+ the skill-invocation
 /// meta-check); `--finalize` folds judge responses into `grading.json`.
 pub(crate) fn run_grade(args: GradeArgs) -> anyhow::Result<()> {
@@ -279,19 +296,22 @@ pub(crate) fn run_grade(args: GradeArgs) -> anyhow::Result<()> {
     let conditions: crate::core::ConditionsRecord =
         serde_json::from_str(&std::fs::read_to_string(&conditions_path)?)?;
 
-    // Grade the run against the skill the run copied, not against the live tree.
-    // An edit between `run` and `grade` would otherwise change what a finished
-    // run is measured by, without anything recording that it had.
+    // The treatment comes from the copy the run froze; the assertions come from
+    // the live file. The documented workflow authors assertions from the run's
+    // own paired evidence, after the dispatch they grade, so the frozen copy
+    // does not hold them yet (#295).
     let skill_subdir = graded_skill_subdir(&ctx, &dir);
-    let evals_path = skill_subdir.join("evals").join("evals.json");
-    let evals_value: serde_json::Value =
-        serde_json::from_str(&std::fs::read_to_string(&evals_path)?)?;
-    let evals = validation::validate_evals_config(&evals_value, &evals_path.to_string_lossy())?;
+    let instrument = pipeline::resolve_grading_instrument(&skill_subdir, &ctx.skill_subdir)?;
+    for warning in &instrument.warnings {
+        eprintln!("⚠ {warning}");
+    }
+    println!("{}", assertion_source_summary(&instrument));
 
     let gctx = pipeline::GradeContext {
         iteration_dir: &dir,
         conditions: &conditions,
-        evals: &evals,
+        evals: &instrument.evals,
+        assertion_source: &instrument.source,
     };
 
     if args.finalize {
@@ -320,14 +340,16 @@ pub(crate) fn run_grade(args: GradeArgs) -> anyhow::Result<()> {
             "Diff scope: {} measured, {} reused, {} missing baseline, {} shared environment",
             diffs.measured, diffs.reused, diffs.missing_baseline, diffs.shared_environment
         );
-        let commands =
-            pipeline::grade_command_checks(&dir, &evals, &skill_subdir, common.overwrite)?;
+        let commands = pipeline::grade_command_checks(&dir, &instrument, common.overwrite)?;
         if commands.executed + commands.reused > 0 {
             println!(
                 "Command checks: {} executed, {} reused, {} failed",
                 commands.executed, commands.reused, commands.failed
             );
         }
+        for w in &commands.warnings {
+            eprintln!("⚠ {w}");
+        }
         let s = pipeline::emit_judge_tasks(&gctx)?;
         for w in &s.warnings {
             eprintln!("⚠ {w}");

src/core/grading.rs

@@ -119,6 +119,23 @@ pub enum Grader {
     DiffScope,
 }
 
+/// Which `evals.json` supplied the assertions a grading measured against.
+///
+/// The treatment stays frozen at what ran, but assertions are the measuring
+/// instrument and are authored after the dispatch they grade, so the file they
+/// came from is recorded rather than assumed.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+pub struct AssertionSource {
+    /// The `evals.json` the graded assertions came from.
+    pub path: String,
+    /// Digest over every graded eval's grading fields, so two gradings can be
+    /// compared without diffing the file they were read from.
+    pub digest: String,
+    /// True when the live file replaced the assertions the run froze.
+    #[serde(default, skip_serializing_if = "std::ops::Not::not")]
+    pub refreshed: bool,
+}
+
 /// The full grading output for one run.
 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
 pub struct GradingResult {
@@ -130,6 +147,10 @@ pub struct GradingResult {
     pub meta_results: Option<Vec<MetaResult>>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub meta_summary: Option<MetaSummary>,
+    /// Which `evals.json` supplied the assertions above. Absent in gradings
+    /// written before the source was recorded.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub assertion_source: Option<AssertionSource>,
 }
 
 /// Legacy pass/fail tallies for an entirely binary grading.

src/pipeline/grade/command_check.rs

@@ -9,8 +9,9 @@ use regex::Regex;
 use serde::{Deserialize, Serialize};
 
 use crate::core::fs::{copy_entry_materialized, write_json};
-use crate::core::{Assertion, AssertionCommandCheck, EvalsConfig, clear_git_environment};
+use crate::core::{Assertion, AssertionCommandCheck, clear_git_environment};
 use crate::pipeline::error::PipelineError;
+use crate::pipeline::grade::instrument::GradingInstrument;
 use crate::validation::{SchemaName, validate_against_schema};
 
 const DIAGNOSTIC_LIMIT: usize = 2 * 1024;
@@ -28,6 +29,12 @@ pub struct CommandCheckResult {
     pub stderr: String,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub cells: Option<Vec<CommandCheckCellResult>>,
+    /// Digest of the `command_check` this result came from. Reuse is keyed by
+    /// assertion id, so this is what tells a later grade that the check under
+    /// that id has been edited since. Absent in results that predate the record,
+    /// which make no claim either way.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub definition_digest: Option<String>,
 }
 
 /// The result of one environment-matrix cell.
@@ -41,11 +48,14 @@ pub struct CommandCheckCellResult {
     pub stderr: String,
 }
 
-#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
+#[derive(Debug, Default, Clone, PartialEq, Eq)]
 pub struct CommandCheckSummary {
     pub executed: usize,
     pub reused: usize,
     pub failed: usize,
+    /// Reused results whose check has since been edited. Returned rather than
+    /// printed: the CLI handler owns how a warning reads.
+    pub warnings: Vec<String>,
 }
 
 #[derive(Debug, Deserialize)]
@@ -66,10 +76,10 @@ struct DispatchTask {
 /// order for every matching dispatch task.
 pub fn grade_command_checks(
     iteration_dir: &Path,
-    evals: &EvalsConfig,
-    skill_dir: &Path,
+    instrument: &GradingInstrument,
     overwrite: bool,
 ) -> Result<CommandCheckSummary, PipelineError> {
+    let evals = &instrument.evals;
     let has_command_checks = evals.evals.iter().any(|eval| {
         eval.assertions
             .as_deref()
@@ -138,19 +148,29 @@ pub fn grade_command_checks(
 
         for check in checks {
             validate_assertion_id(&check.id)?;
+            let digest = definition_digest(check);
             let result_path = results_dir.join(format!("{}.json", check.id));
             if result_path.exists() && !overwrite {
                 let value = serde_json::from_str(&fs::read_to_string(&result_path)?)?;
-                validate_against_schema::<CommandCheckResult>(
+                let reused = validate_against_schema::<CommandCheckResult>(
                     SchemaName::CommandCheck,
                     &value,
                     &result_path.to_string_lossy(),
                 )?;
+                if reused
+                    .definition_digest
+                    .is_some_and(|recorded| recorded != digest)
+                {
+                    summary.warnings.push(format!(
+                        "command_check '{}' for {}/{} changed since its cached result was produced; that result is reused as-is. Re-run grade with --overwrite to execute the edited check.",
+                        check.id, task.eval_id, task.condition
+                    ));
+                }
                 summary.reused += 1;
                 continue;
             }
 
-            inject_setup_files(check, skill_dir, eval_root)?;
+            inject_setup_files(check, instrument.setup_root_for(&task.eval_id), eval_root)?;
             let result = execute_command_check(check, eval_root)?;
             if !result.passed {
                 summary.failed += 1;
@@ -169,6 +189,16 @@ pub fn grade_command_checks(
     Ok(summary)
 }
 
+/// Digest of a check's authored definition, so reuse can tell an edited check
+/// from the one that produced the cached result.
+fn definition_digest(check: &AssertionCommandCheck) -> String {
+    crate::core::fs::fnv1a_hex(
+        serde_json::to_string(check)
+            .expect("an authored command_check serializes")
+            .as_bytes(),
+    )
+}
+
 fn isolation_error(task: &DispatchTask, detail: &str) -> PipelineError {
     PipelineError::Message(format!(
         "command_check task '{}'/{} {detail}; command checks require task-scoped environments. Build and dispatch a fresh iteration with this evals.json before grading.",
@@ -243,6 +273,7 @@ pub(super) fn execute_command_check(
             stdout: cell.stdout,
             stderr: cell.stderr,
             cells: None,
+            definition_digest: Some(definition_digest(assertion)),
         });
     };
 
@@ -280,6 +311,7 @@ pub(super) fn execute_command_check(
         stdout: String::new(),
         stderr: String::new(),
         cells: Some(cells),
+        definition_digest: Some(definition_digest(assertion)),
     })
 }