Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion docs/guides/byoh.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,12 @@ For a descriptor that should not live in the project, pass it directly:
eval-magic run --harness-file ./cool-custom-harness.toml
```

Its `label` becomes the default harness for that invocation.
Its `label` becomes the default harness for that invocation. Every follow-up command `run`
generates — the printed Next: steps and each `eval-magic …` line in RUNBOOK.md — re-emits
`--harness-file`, because the descriptor can decide whether a comparison is valid (dispatch
templates, shadow isolation). Follow the generated commands verbatim: dropping the flag resolves a
different descriptor, and `dispatch`/`ingest` compare the resolved descriptor against the digest
`run` records in `conditions.json` and warn when the two differ.

## Add a dispatch command first

Expand Down
7 changes: 7 additions & 0 deletions docs/guides/isolation.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,13 @@ intrinsic severity as provenance. `run` presents operator-environment findings a
and `aggregate` omits those warnings only while no transcript evidence contradicts the declaration.
Codebase-sourced findings remain warnings regardless of this descriptor setting.

The declaration is written from the descriptor resolved at prep time, so the remedy must ride along
with every follow-up. When the descriptor arrives via `--harness-file`, the generated RUNBOOK.md
and Next: commands re-emit the flag — follow them verbatim. Dropping it silently reverts to the
un-overlaid descriptor while the declaration stands, the comparison-invalid state this guide exists
to prevent. `dispatch` and `ingest` compare the resolved descriptor against the digest `run`
records in `conditions.json` and warn when the two differ.

Do not set it when:

- Any reported source remains discoverable.
Expand Down
28 changes: 27 additions & 1 deletion src/adapters/registry.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
//! behind the artifact `Deserialize`), [`Harness::known`] enumerates the
//! entries, and [`adapter_for`] serves each handle's [`HarnessAdapter`].

use std::path::Path;
use std::path::{Path, PathBuf};
use std::sync::{LazyLock, OnceLock};

use crate::core::Harness;
Expand Down Expand Up @@ -79,6 +79,11 @@ pub enum RegistryInitError {
/// via `--harness`.
static SESSION_DEFAULT_HARNESS: OnceLock<&'static str> = OnceLock::new();

/// When `--harness-file` is in play, its absolutized path — consulted by the
/// CLI when it renders generated follow-up commands, which must re-emit the
/// flag or they silently resolve a different descriptor (#294).
static SESSION_HARNESS_FILE: OnceLock<PathBuf> = OnceLock::new();

/// Initialize the registry from every descriptor layer: embedded built-ins →
/// user-global (`<config-root>/harnesses/*.toml`) → project-local
/// (`<cwd>/.eval-magic/harnesses/*.toml`) → the optional one-off
Expand All @@ -100,6 +105,12 @@ pub fn init_registry(harness_file: Option<&Path>) -> Result<(), RegistryInitErro
for warning in io_warnings.iter().chain(&built.warnings) {
eprintln!("⚠ {warning}");
}
if let Some(file) = harness_file {
// The file was read during discovery, so it exists; fall back to the
// passed spelling only if resolution itself fails.
let resolved = crate::core::fs::real_path(file).unwrap_or_else(|_| file.to_path_buf());
let _ = SESSION_HARNESS_FILE.set(resolved);
}
if harness_file.is_some()
&& let Some(entry) = built
.entries
Expand Down Expand Up @@ -337,6 +348,21 @@ pub fn descriptor_value_for(harness: Harness) -> &'static serde_json::Value {
.value
}

/// The absolutized `--harness-file` path this session loaded, if any.
pub fn session_harness_file() -> Option<&'static Path> {
SESSION_HARNESS_FILE.get().map(PathBuf::as_path)
}

/// A stable digest of the fully-resolved (layer-merged) descriptor behind
/// `harness`, over its canonical JSON form. A run records it at prep time so
/// a later invocation can tell whether it resolved the same descriptor the
/// run was prepared with (#294).
pub fn descriptor_digest(harness: Harness) -> String {
let canonical = serde_json::to_string(descriptor_value_for(harness))
.expect("a descriptor value serializes");
crate::core::fs::fnv1a_hex(canonical.as_bytes())
}

/// True when `harness` has an embedded built-in descriptor among its sources
/// — i.e. it is not defined by user-supplied descriptor files alone. Preflight
/// uses this to hard-reject `--guard` on user-only harnesses (the write guard
Expand Down
7 changes: 7 additions & 0 deletions src/cli/args.rs
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,13 @@ pub(crate) struct Cli {
/// label becomes the invocation's default harness. Unlike discovered
/// descriptor files (skipped with a warning when broken), errors in this
/// explicitly named file are fatal.
///
/// Every command `run` generates — the printed Next: steps and the
/// RUNBOOK.md — re-emits this flag, because the descriptor it loads can
/// decide whether a comparison is valid (dispatch templates, shadow
/// isolation). Follow those commands verbatim: `run` records the resolved
/// descriptor's digest in conditions.json, and `dispatch`/`ingest` warn
/// when a follow-up resolves a different one.
#[arg(long, global = true, value_name = "PATH")]
pub harness_file: Option<String>,
#[command(subcommand)]
Expand Down
18 changes: 12 additions & 6 deletions src/cli/commands/pipeline.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,10 @@ use anyhow::bail;
use crate::cli::args::{CommonArgs, GradeArgs};
use crate::cli::command_target_args;
use crate::cli::run;
use crate::cli::{iteration_dir, resolve_iteration, run_context_from, staged_env_roots};
use crate::cli::{
harness_descriptor_drift_warning, iteration_dir, resolve_iteration, run_context_from,
staged_env_roots,
};
use crate::core::RunContext;
use crate::pipeline;
use crate::sandbox;
Expand Down Expand Up @@ -59,6 +62,13 @@ fn run_step(step: &run::steps::StepCommand) -> anyhow::Result<()> {
pub(crate) fn run_ingest(args: CommonArgs) -> anyhow::Result<()> {
let ctx = run_context_from(&args)?;
let iteration = resolve_iteration(&ctx, args.iteration)?;
let dir = ctx
.workspace_root
.join(&ctx.skill_name)
.join(format!("iteration-{iteration}"));
if let Some(warning) = harness_descriptor_drift_warning(&ctx, &dir) {
eprintln!("⚠ {warning}");
}

let steps = run::steps::build_ingest_commands(&run::steps::StepParams {
skill_dir: args.skill_dir.as_deref(),
Expand All @@ -73,11 +83,7 @@ pub(crate) fn run_ingest(args: CommonArgs) -> anyhow::Result<()> {
);
}

let judge_path = ctx
.workspace_root
.join(&ctx.skill_name)
.join(format!("iteration-{iteration}"))
.join("judge-tasks.json");
let judge_path = dir.join("judge-tasks.json");
let total_tasks = std::fs::read_to_string(&judge_path)
.ok()
.and_then(|s| serde_json::from_str::<serde_json::Value>(&s).ok())
Expand Down
3 changes: 3 additions & 0 deletions src/cli/commands/run.rs
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,9 @@ pub(crate) fn run_run(args: RunArgs) -> anyhow::Result<()> {
pub(crate) fn run_dispatch(args: DispatchArgs) -> anyhow::Result<()> {
let ctx = run_context_from(&args.common)?;
let iteration_dir = iteration_dir(&ctx, args.common.iteration)?;
if let Some(warning) = crate::cli::harness_descriptor_drift_warning(&ctx, &iteration_dir) {
eprintln!("⚠ {warning}");
}
// `--timeout 0` means "no deadline", which is the only way to say it with a
// plain seconds flag.
let timeout = (args.timeout > 0).then(|| std::time::Duration::from_secs(args.timeout));
Expand Down
157 changes: 132 additions & 25 deletions src/cli/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ pub(crate) fn run_context_with_bootstrap(
bootstrap,
workspace_dir: args.workspace_dir.clone(),
harness,
harness_file: crate::adapters::registry::session_harness_file().map(Path::to_path_buf),
cwd: None,
})?;
// `core` returns warnings rather than printing them; this is the one place
Expand All @@ -175,21 +176,69 @@ pub(crate) fn parse_id_list(v: Option<&str>) -> Option<Vec<String>> {

/// Render a fully self-sufficient target selector for the current run context.
///
/// Always names `--skill-dir`, `--skill`, and `--workspace-dir` (all three are
/// always populated in [`RunContext`] and always re-resolve), so the printed
/// "Next:" commands are copy-pasteable from any cwd — not just the one `run`
/// happened to start in. The absolute `--workspace-dir` is what lets the human
/// run `ingest`/`finalize` from a per-`(group, condition)` env dir: without it,
/// `workspace_root` would fall back to the derived default (`detect_run_context`),
/// which is keyed on the skill directory rather than on the cwd, and the
/// iteration tree above the env would not resolve.
/// The selector reproduces the invocation that built the context, so a printed
/// "Next:" command is copy-pasteable from any cwd AND re-runs the same
/// experiment (#294):
///
/// * `--skill-dir … --skill …` only when the invocation used `--skill-dir`
/// (`stage_siblings`); otherwise `--skill` names the absolute skill subdir.
/// Inventing a `--skill-dir` would stage every sibling skill ambiently — a
/// different experiment from the one just prepared.
/// * an absolute `--workspace-dir`, so the human can run `ingest`/`finalize`
/// from a per-`(group, condition)` env dir: without it, `workspace_root`
/// would fall back to the derived default (`detect_run_context`), which is
/// keyed on the skill directory rather than on the cwd, and the iteration
/// tree above the env would not resolve.
/// * `--harness-file …` when the invocation loaded one; dropping it silently
/// resolves a different descriptor than the run was prepared with.
pub(crate) fn command_target_args(ctx: &RunContext) -> String {
format!(
" --skill-dir {} --skill {} --workspace-dir {}",
artifact_path(&ctx.skill_dir),
ctx.skill_name,
artifact_path(&ctx.workspace_root),
)
let mut args = String::new();
if ctx.stage_siblings {
args.push_str(&format!(
" --skill-dir {} --skill {}",
artifact_path(&ctx.skill_dir),
ctx.skill_name
));
} else {
args.push_str(&format!(" --skill {}", artifact_path(&ctx.skill_subdir)));
}
args.push_str(&format!(
" --workspace-dir {}",
artifact_path(&ctx.workspace_root)
));
if let Some(file) = &ctx.harness_file {
args.push_str(&format!(" --harness-file {}", artifact_path(file)));
}
args
}

/// The warn-loudly backstop for #294. Post-prep stages resolve the harness
/// descriptor by label, so a follow-up that drops `--harness-file` (or runs
/// where project descriptor layers differ) silently switches descriptors
/// mid-campaign while the iteration's artifacts keep the prep-time
/// declarations. Returns the warning when this invocation's resolved
/// descriptor differs from the one the iteration was prepared with; `None`
/// when they match, or when the iteration predates descriptor provenance.
pub(crate) fn harness_descriptor_drift_warning(
ctx: &RunContext,
iteration_dir: &Path,
) -> Option<String> {
let raw = std::fs::read_to_string(iteration_dir.join("conditions.json")).ok()?;
let conditions: crate::core::ConditionsRecord = serde_json::from_str(&raw).ok()?;
let prepared = conditions.harness_descriptor_digest?;
let current = crate::adapters::registry::descriptor_digest(ctx.harness);
if prepared == current {
return None;
}
let label = ctx.harness.name();
Some(match (&conditions.harness_file, &ctx.harness_file) {
(Some(file), None) => format!(
"harness descriptor drift: this iteration was prepared with --harness-file {file} (descriptor digest {prepared}), but this invocation resolved '{label}' as digest {current}. Re-run with --harness-file {file}: the iteration's dispatch templates and shadow declarations came from that descriptor."
),
_ => format!(
"harness descriptor drift: '{label}' resolves to digest {current}, but this iteration was prepared with digest {prepared} — descriptor files changed since prep. Stages resolve the descriptor by label, so continuing mixes two descriptors in one comparison."
),
})
}

/// Resolve the explicit iteration, or default to the latest existing
Expand Down Expand Up @@ -279,10 +328,13 @@ mod tests {
subdir
}

/// The selector must be copy-pasteable: even when `run` was invoked from
/// inside the skill dir (the case that used to render an empty selector), it
/// must name both `--skill-dir` and `--skill`, and re-resolve to the same
/// skill from an unrelated cwd.
/// The selector must be copy-pasteable *and* behavior-preserving (#294):
/// even when `run` was invoked from inside the skill dir (the case that
/// used to render an empty selector), it names `--skill` as an absolute
/// path and re-resolves to the same skill from an unrelated cwd. It must
/// NOT invent a `--skill-dir` the invocation never used — `--skill-dir`
/// sets `stage_siblings`, so re-running from that selector would stage
/// every sibling skill ambiently and run a different experiment.
#[test]
fn target_args_are_self_sufficient_when_run_from_inside_skill_dir() {
let tmp = TempDir::new().unwrap();
Expand All @@ -298,26 +350,81 @@ mod tests {

let args = command_target_args(&ctx);
assert!(
args.contains("--skill-dir"),
"selector names --skill-dir: {args}"
!args.contains("--skill-dir"),
"an invocation without --skill-dir must not gain one: {args}"
);
assert!(
args.contains("--skill mr-review"),
"selector names --skill: {args}"
args.contains(&format!("--skill {}", artifact_path(&skill_subdir))),
"selector names --skill as an absolute path: {args}"
);

// Round-trip: feeding the rendered selector back from an unrelated cwd
// resolves the same skill.
// resolves the same skill with the same staging behavior.
let other = root.join("elsewhere");
fs::create_dir_all(&other).unwrap();
let resolved = detect_run_context(DetectInput {
skill_dir: Some(ctx.skill_dir.display().to_string()),
skill: Some(ctx.skill_name.clone()),
skill: Some(ctx.skill_subdir.display().to_string()),
cwd: Some(other),
..Default::default()
})
.unwrap();
assert_eq!(resolved.skill_subdir, ctx.skill_subdir);
assert!(!resolved.stage_siblings);
}

/// `--skill-dir <dir> --skill <name>` is the sibling-staging form, so when
/// the invocation used it the selector reproduces it exactly.
#[test]
fn target_args_keep_skill_dir_when_the_invocation_used_it() {
let tmp = TempDir::new().unwrap();
let root = fs::canonicalize(tmp.path()).unwrap();
let skill_subdir = make_skill(&root, "skills", "mr-review");
let skill_dir = skill_subdir.parent().unwrap().to_path_buf();

let ctx = detect_run_context(DetectInput {
skill_dir: Some(skill_dir.display().to_string()),
skill: Some("mr-review".to_string()),
cwd: Some(root.clone()),
..Default::default()
})
.unwrap();

let args = command_target_args(&ctx);
assert!(
args.contains(&format!("--skill-dir {}", artifact_path(&skill_dir))),
"selector keeps --skill-dir: {args}"
);
assert!(args.contains("--skill mr-review"), "{args}");
}

/// A run prepared with `--harness-file` must re-emit the flag in every
/// generated follow-up command, or the follow-up silently resolves a
/// different descriptor (#294).
#[test]
fn target_args_reemit_harness_file() {
let tmp = TempDir::new().unwrap();
let root = fs::canonicalize(tmp.path()).unwrap();
let skill_subdir = make_skill(&root, "skills", "mr-review");
let harness_file = root.join("overlay.toml");
fs::write(
&harness_file,
r#"label = "claude-code"
"#,
)
.unwrap();

let ctx = detect_run_context(DetectInput {
cwd: Some(skill_subdir),
harness_file: Some(harness_file.clone()),
..Default::default()
})
.unwrap();

let args = command_target_args(&ctx);
assert!(
args.contains(&format!("--harness-file {}", artifact_path(&harness_file))),
"selector re-emits --harness-file: {args}"
);
}

/// The human runs `ingest`/`finalize` from a per-`(group, condition)` env dir.
Expand Down
5 changes: 5 additions & 0 deletions src/cli/run/orchestrate/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,11 @@ pub(super) fn write_dispatch(
label: opts.label.map(str::to_owned),
codebases: r.codebases.iter().map(super::RunCodebase::usage).collect(),
skill_source: Some(r.skill.record()),
// Prep-time descriptor provenance (#294): the digest lets a later
// stage detect that it resolved a different descriptor than this run
// was prepared with; the path makes the remedy copy-pasteable.
harness_file: ctx.harness_file.as_deref().map(artifact_path),
harness_descriptor_digest: Some(crate::adapters::registry::descriptor_digest(ctx.harness)),
};
write_json(&r.iteration_dir.join("conditions.json"), &conditions)?;
// One record, cloned into every task: grading reads `run.json` alone, so a
Expand Down
Loading
Loading