Keep verifier outcomes, uncertainty, and execution failures auditable

packages/evals/docs/verifier-gates.md

@@ -0,0 +1,29 @@
+# Verifier evidence gates
+
+External harnesses use the existing V3 verifier with the task's precomputed rubric when available. A requested verification that errors or returns a verifier-uncertainty sentinel fails closed: `_success` is false, `verifierError` explains the failure, and `agentReportedSuccess` preserves the original self-report. Such rows are ungraded and must not be presented as verified benchmark outcomes.
+
+For completed grades, the raw judge verdict is retained under `judge` in `scores/result.json`. The top-level result, `task_data.json`, and the run row carry the same adjusted outcome. Failed or uncertain verification instead persists `{ graded: false, verifierError, judge? }`, without top-level outcome or process scores; `judge` is present only if the verifier returned a response. `scores/gates.json` records diagnostics for completed grades when trajectory persistence is enabled.
+
+## Outcome
+
+A judge pass is rejected when the final answer is empty or, if a mounted-tool matcher is available, the trajectory has no browser tool calls. Grounding checks are advisory by default. `EVAL_REQUIRE_GROUNDING=1` additionally rejects an answer whose checked numeric findings all lack matching observations from a known non-search page. Captured step and terminal accessibility trees are included; unknown-page text cannot establish grounding. A terminal match is recorded as `groundedAtFinalObservation`, while untrusted matches can be recorded as `seenOnUnknownPage`. The check is a text heuristic: images, paraphrases and valid snippet sources can escape its matching, so it is not a replacement for rubric verification.
+
+Execution state remains separate. When populated by the runner or adapter, `harnessStatus`, `harnessStopReason` and `terminationReason` describe completion, budget exhaustion, abort, SDK error and browser loss. The verifier does not synthesize missing lifecycle metadata; its absence means unavailable. A supported task completion can still pass after a disconnect; execution error alone does not erase earlier evidence. An unfinished task must fail its rubric.
+
+## Process
+
+`processScoreStrict` recomputes the weighted process score with explicit `evidenceInsufficient` criteria earning zero while retaining their maximum points in the denominator. `processScore` uses this score; `processScoreLenient` preserves the judge's aggregate. Not-applicable criteria are excluded. Without a criterion breakdown, the judge's aggregate is retained and `scoringIncomplete` flags a short result against the rubric.
+
+Blocker wording is recorded as `blockerMentioned` on criterion diagnostics. It never changes points by itself: permitted fallback and stop-boundary explanations can correctly mention a blocker. The rubric and observed evidence determine whether the requirement was satisfied. This replaces the campaign's overbroad blocker substring heuristic.
+
+## Reporting
+
+These fields depend on the producing runner; this verifier layer forwards them but does not make every harness emit them:
+
+- Where supplied, `facade_tool_calls`, `facade_tool_call_failures` and `facade_tool_calls_after_session_lost` distinguish attempted browser work from repeated terminal failures. Missing counters are unknown, not measured zero. A graded pass with an explicit zero browser-call count is shown in the batch summary; with `EVAL_MAX_UNVERIFIABLE_CRITERIA` enabled, it fails the batch gate.
+- Separate agent, evidence-capture and verifier wall times are available only when recorded by the producer.
+- Usage must be interpreted with the producer's presence marker and cache convention. Legacy runners may supply zero placeholders; without an explicit presence marker, zero does not establish measured usage. Historical Cursor CLI usage remains unreported.
+- Costs reported by the harness can be retained. A `cost_source` field, when supplied by a producer, distinguishes reported dollars from a catalog estimate (`computed`). This verifier layer does not compute estimates. Without provenance, cost origin is unavailable; unknown or subscription costs must not be inferred as zero.
+- `harnessImplementation` records adapter and SDK versions when supplied. Its absence means unknown implementation; historical labels are preserved.
+
+Use `VERIFIER_PERSIST_TRAJECTORIES=1` for reviewable evidence. HardBench's compatibility gate rejects verifier errors, uncertainty sentinels, missing criteria and self-report fallbacks before accepting a result. Offline transport checks establish integration compatibility; live rubric accuracy still requires the separately recorded live fixtures.

packages/evals/framework/harnesses/persistTrajectory.ts

@@ -7,13 +7,15 @@ import {
   resolveTrajectoryRoot,
   writeTrajectoryMetadata,
 } from "../trajectoryGroup.js";
-import type { EvaluationResult, TaskSpec, Trajectory } from "stagehand-v3";
+import type { EvaluationResult, TaskSpec } from "stagehand-v3";
+import type { HarnessTrajectory } from "./trajectoryAdapter.js";
+import type { UngradedVerifierResult } from "../verifierAdapter.js";
 
 export interface PersistAdapterTrajectoryOptions {
-  trajectory: Trajectory;
+  trajectory: HarnessTrajectory;
   taskSpec: TaskSpec;
-  /** EvaluationResult from V3Evaluator.verify(). Written to scores/result.json. */
-  evaluationResult?: EvaluationResult;
+  /** Accepted grade or explicit ungraded audit record. Written to scores/result.json. */
+  evaluationResult?: EvaluationResult | UngradedVerifierResult;
   /**
    * Output directory root. Final layout lives at
    * `<outputRoot>/<group>/<task.id>/<runId>/`. Entrypoints normally generate
@@ -73,6 +75,16 @@ export async function persistAdapterTrajectory(
     runDir: path.basename(directory),
     attempt,
     status: opts.trajectory.status,
+    ...(opts.trajectory.cost_pricing && { cost_pricing: opts.trajectory.cost_pricing }),
+    ...(opts.trajectory.terminationReason && {
+      terminationReason: opts.trajectory.terminationReason,
+    }),
+    ...(opts.trajectory.harnessImplementation && {
+      harnessImplementation: opts.trajectory.harnessImplementation,
+    }),
+    ...(opts.trajectory.harnessConfiguration && {
+      harnessConfiguration: opts.trajectory.harnessConfiguration,
+    }),
   });
 
   if (opts.evaluationResult) {
@@ -86,6 +98,7 @@ export async function persistAdapterTrajectory(
         {
           task: opts.trajectory.task,
           status: opts.trajectory.status,
+          ...(opts.trajectory.cost_pricing && { cost_pricing: opts.trajectory.cost_pricing }),
           finalAnswer: opts.trajectory.finalAnswer ?? null,
           result: opts.evaluationResult,
         },

packages/evals/framework/harnesses/trajectoryAdapter.ts

@@ -7,6 +7,26 @@ import type {
   TrajectoryStep,
 } from "stagehand-v3";
 
+/**
+ * Why a run stopped. `status` on the Trajectory only says whether it ended
+ * cleanly; this says what ended it, so a budget-exhausted run can be told
+ * apart from a crash offline.
+ */
+export type TerminationReason =
+  | "completed"
+  | "step_budget"
+  | "browser_session_lost"
+  | "sdk_error"
+  | "aborted";
+
+/** A Trajectory as external harnesses persist it. */
+export type HarnessTrajectory = Trajectory & {
+  terminationReason?: TerminationReason;
+  harnessImplementation?: { name: string; version: number; sdkVersion?: string };
+  harnessConfiguration?: Record<string, unknown>;
+  cost_pricing?: { as_of: string; model: string; source: string };
+};
+
 /**
  * Pure converter from a harness-specific result to a verifier Trajectory.
  * Implementations must be deterministic (no I/O, no mutation of input).