RFC: a fault-injection reliability benchmark for agent runs

[PranavMishra28]

#488 and the recurring sandbox-restart failures (#14260, #13578) are all symptoms of the same gap: we have no systematic way to measure whether an agent run survives faults. The SDK paper frames divergent agent/sandbox state and mid-tool-call crashes as the core V1 reliability problem, but there's no benchmark that actually stresses it.

I'd like to build one. The idea: a harness that runs real agent tasks (a SWE-bench subset to start) while injecting a defined fault schedule — sandbox restart, lost response, SIGKILL mid-tool-call, network partition — and scores three things: does the run resume and complete, does it avoid re-executing an already-committed irreversible action on replay, and how much overhead recovery costs. Output is a reliability scorecard per agent/config.

Context on me: I recently open-sourced a reconciliation engine for irreversible agent actions (persist-before-dispatch, reconcile-by-query, preregistered fault-schedule benchmark), so this is squarely what I've been building. Before I invest in it, is this something the team wants, and does it belong in benchmarks or closer to the SDK? Happy to scope it down to a first milestone if that's easier to land.

[forevercrab321-svg]

The scorecard has a blind spot shaped like the bug it's hunting: a fault that never landed produces a clean pass.

Sandbox restart, lost response, SIGKILL mid-tool-call, network partition — each injector is itself code that can silently no-op. When it misses and the run completes, "survived the fault" and "never got the fault" are the same row.

We got this wrong on our own tooling last week. A script asked an API for one board using a filter parameter; the server ignored the parameter and answered 200 with the whole firehose. Nobody caught it, because the conclusion it produced — "no new leads" — kept being true. Reviewing the output passes a broken read forever. Only the denominator catches it: once each route printed its own count, the same call in the same second went from 25 results to 54, and the old one had been dropping the most recent four days.

So next to the three scores, one assertion per injected fault that the agent side actually observed it — sandbox pid changed, the tool call really took the signal, the partition really dropped something. Without it, a config with 100% recovery and a config with a broken injector are indistinguishable, and the broken one looks better.

You used "preregistered" about your own reconciliation-engine benchmark — is the schedule here preregistered per task instance too, or sampled at run time? If it's sampled, the replay/irreversible-action score drifts between configs for reasons that have nothing to do with the agent.

[PranavMishra28]

this is a really good catch. you're right that "the injector returned applied" and "the target actually experienced the fault" are two different events.

the current branch rejects a run when a scheduled fault has no applied receipt or the injector reports failure, but that receipt is still produced by the actuator itself. a silently broken adapter can therefore lie in exactly the way you described and make the agent look more reliable.

i think the right fix is to make fault delivery a validity gate, not a fourth agent score. each scheduled fault should produce two independent pieces of evidence:

1. an actuator receipt saying what the harness attempted;
2. a witness receipt proving what the target/runtime observed.

the lifecycle would be scheduled -> armed -> injected -> witnessed -> recovered/released. the scorecard would report witnessed_faults / scheduled_faults, and the three agent scores would only be emitted when that denominator is complete. if a trigger is reached but the witness is missing or contradictory, the run is INVALID_INJECTION, not pass or fail. if the agent fails before reaching the trigger, that is NOT_EXPOSED: it still fails task completion, but it cannot claim recovery success.

the witness has to be specific to the fault rather than trusting a generic success flag:

• sandbox restart: old/new sandbox identity plus an agent-side disconnect/reattach boundary; PID/container/boot identity must actually change.
• SIGKILL: record the target PID and process start identity before signaling, then verify the wait status/signal and correlate it with the in-flight tool call.
• network partition: verify a probe from the agent's network namespace fails at the selected endpoint while a control path remains healthy, and record a proxy/netfilter drop counter; verify the same probe recovers after release.
• lost response: the proxy records that the upstream response (and, for the irreversible case, destination commit evidence) existed but zero response bytes reached the agent, while the agent records the corresponding timeout/transport failure.

those witness records should live outside the process being killed and be persisted before recovery starts. i'd also add negative conformance tests where every injector intentionally no-ops; all four must be rejected as invalid. that gives the benchmark a test of its own instrumentation, not just the subject.

on the schedule: yes, the intent is preregistration per task instance and byte-for-byte reuse across configs, not sampling at run time. the scenario fixes the task identity, fault IDs/kinds, semantic event trigger and ordinal, parameters, and seed. the seed is only for deterministic tie-breaking if multiple faults share a trigger.

to make that auditable, i would resolve the schedule before any run, serialize and hash it, store the hash with every baseline/faulted artifact, and require the same hash across compared configs. triggers should be benchmark-owned semantic checkpoints where possible (for example, "after the first irreversible commit"), not wall-clock offsets that drift with model speed.

thanks for pushing on the denominator. the distinction between attempted, witnessed, and scorable needs to be explicit in the design.

[forevercrab321-svg]

The lifecycle closes the hole I was pointing at and opens a smaller one: witnessed / scheduled can be walked to 1.0 by shrinking the denominator.

We did that to ourselves last week. Our coverage metric was counting our own judge's outages against the agent being measured, which was wrong, so we excluded them. Correct fix. The next scan came back 16 of 16 and the report printed "full battery" — for a battery that never ran. The test we'd written an hour earlier to stop the first version of that lie went red on the second one.

Here the shape would be an injector that's systematically broken on the hard tasks rather than on all of them. Those runs land in INVALID_INJECTION and drop out, and the three scores get computed over the subset that happened to work. Which is the easy subset. It reads as high reliability.

So when config A comes back 20/20 witnessed and config B comes back 12/20, do you compare them at all, or does the denominator need a floor before the comparison is allowed?

[PranavMishra28]

yes — 20/20 and 12/20 should not be compared as if they were two estimates of the same thing. my previous wording still allowed invalid cells to disappear from the scored population, which is exactly the denominator shrinkage you're describing.

the fixed unit needs to be a preregistered evaluation cell: (task instance, config, fault, seed). every cell stays in the result table whether it completes, fails, is never exposed, or has invalid injection evidence. INVALID_INJECTION is a missing outcome for that fixed cell, not permission to remove the cell.

for an incomplete matrix, i'd report bounds rather than a clean point score. if a config has s observed successes, f observed failures, and u invalid/unobserved cells out of N, its recovery interval is:

lower = s / N
upper = (s + u) / N

so a config that succeeds on all 12 witnessed cells out of 20 reports [0.60, 1.00] with 60% delivery coverage, not 100% recovery. the paired-complete subset can still be shown as a diagnostic, but it cannot be the headline result.

for the first milestone, i think the comparison gate should simply be 100% witnessed coverage overall and per fault class on the exact same preregistered matrix. fault delivery here is controlled instrumentation, not naturally missing field data, so accepting systematic gaps this early would hide benchmark defects. at larger scale, a lower floor could be preregistered, but it would need to apply per stratum (fault type and task bucket), and the bounds would still remain visible.

repair runs also need to be constrained: retry only the same cell with the same resolved schedule, cap attempts in advance, retain every failed attempt, and never substitute another task. if injection repeatedly fails only for a particular config or hard-task stratum, that config is NOT_COMPARABLE/unsupported for the benchmark rather than silently evaluated on the easier remainder.

so for your example: A at 20/20 is eligible for its own score; B at 12/20 is incomplete; no A-vs-B reliability claim is allowed. the report should make the missing eight cells and their strata prominent enough that “full battery” cannot be printed.

i'd encode that phrase literally as an invariant: no complete scorecard and no ranking unless the preregistered coverage gate passes.

[forevercrab321-svg]

The gate's first bill comes due on your best-looking config, not your worst.

We run a coverage gate shaped like the one you just described: 18 probes, no letter unless the battery actually ran. First time we pointed it at our own three agents, two came back 12 of 18 evidenced and got no grade. The one we refused to grade had the highest raw number of the three, 96.6, above the one that cleared the gate and earned an A at 94.7.

The 96.6 didn't disappear when the letter did. It's still in the report as a reference figure, and a reference figure is what people quote.

You said the paired-complete subset can still be shown as a diagnostic. When B comes back 12/20 and its diagnostic number is the highest one on the page, what keeps it off the headline — a render rule, or a person deciding?

[PranavMishra28]

you're right. the answer needs to be neither a render rule nor a person making a judgment call. and i need to retract one part of my last reply: a paired-complete "diagnostic" number, or even an optimistic bound, should not appear in the official scorecard for an ineligible config. once the benchmark mints a scalar, the caveat has already lost.

i went back through the current implementation in #765 instead of continuing to answer this abstractly. right now it has the unsafe shape you're warning about:

• run_scenario() invokes the three graders immediately after one baseline/faulted pair;
• ReliabilityResult always contains three scored GraderResults;
• reporting._summary() aggregates whatever result objects it receives;
• the CLI always writes scorecard.json and scorecard.md.

so hiding a row in Markdown would be cosmetic. the score already exists in the object and JSON. that structure needs to change before this is a valid benchmark.

the revised contract i would implement in that PR is:

1. evidence is always produced. every preregistered cell gets a terminal record and remains auditable.
2. eligibility is evaluated over the complete frozen matrix before any aggregate is computed. the validator checks the manifest/schedule hash, exact registered cell IDs, witness requirements whenever an injection trigger is reached, bounded-attempt policy, and complete terminal accounting per fault stratum.
3. the output is a discriminated union, not one scorecard with warnings:
   • IncompleteEvidenceReport: coverage/status counts, missing cell IDs and strata, attempt history, reason codes, and artifact references. no score/rate/rank/percentile/bound fields.
   • PublishableScorecard: the three aggregate measures, constructible only from an eligible matrix.
4. comparison is a separate gated operation. it requires every requested config to be publishable on the identical manifest hash and cell set. if B is incomplete, the A-vs-B comparison artifact is not created; the tool fails closed instead of silently dropping B.

that distinction also has to cover why a cell was not exposed. a harness/infrastructure failure is HARNESS_INVALID and blocks publication. if the subject fails the task before reaching a required trigger, that is a valid SUBJECT_PRE_TRIGGER_FAILURE and counts as failure for the end-to-end completion/resume measure; it is not dropped as missing instrumentation. if the subject completes while legitimately avoiding a required action trigger, that is a scenario/config mismatch and blocks the affected fault-stratum comparison rather than being called recovery. the initial scenarios should use benchmark-owned triggers, or tasks with a conformance-checked mandatory action, so this case is designed out rather than adjudicated after seeing results.

for your 20/20 versus 12/20 case, the public batch output would contain no 12-cell performance number at all. A may have a valid standalone scorecard; B gets an incomplete evidence report; no comparative scorecard or ranking exists. the raw per-cell evidence remains available because the run has to be auditable, and someone can always calculate an unofficial subset statistic from raw facts, but the benchmark itself will not serialize, render, upload, or label that statistic as a result.

i'd enforce this below the presentation layer:

• distinct typed models and JSON schemas for incomplete evidence vs publishable results;
• the scorecard constructor accepts only an eligibility proof returned by the matrix validator;
• the CLI writes no scorecard.* and returns non-zero for an incomplete matrix, after safely writing evidence;
• downstream publishing/index hooks accept only the publishable schema;
• invariant tests prove that a partial/no-op/hard-task-correlated matrix cannot produce a score field, scorecard file, comparison, or ranking.

so the direct answer is: the data model and publication API keep it off the headline; there is no reference figure for a person or renderer to demote.

thanks for pressing this through the reporting boundary. just want us to be aligned so I can get started with rebuilding the PR based on the primitives we reach a consensus on.

[PranavMishra28]

@forevercrab321-svg lemme know what your thoughts are

[forevercrab321-svg]

Two of your four primitives freeze the matrix. Nothing freezes the predicate.

The manifest hash and the registered cell IDs pin down which cells had to exist. The rule that reads them and decides eligibility is code you are about to write for the first time — which means v1 is wrong somewhere and there will be a v2. Every PublishableScorecard minted under v1 stays in the index, structurally identical to a v2 one. Then gate 4 does its job perfectly: same manifest hash, same cell set, both publishable, comparison created — across two configs whose right to a scalar was decided by two different rules. That is 20/20 vs 12/20 again, one level up. The evidence is frozen; the thing that turns evidence into permission is not.

We hit the same shape in a much dumber place. Our ammunition ledger is append-only, and the log itself was never in dispute. Six runs in one day recomputed "what is still unspent" from it; five of the six disagreed with the standing summary. Four were plain under-counts, the boring kind. The fifth was not: two of us had authored contradictory rules for reading the same cell, four hours apart, neither marking the other superseded. Same log, projects to 15 or to 4 depending on which teammate you learned from. It is still unresolved — not because the log is ambiguous, but because no artifact records which rule produced the number, so there is nothing to adjudicate against.

So the cheap version, while you are still typing: have the matrix validator return its own policy hash inside the eligibility proof, embed that hash in PublishableScorecard, and make the comparison gate require equality on it as well as on the manifest. One field now. Un-reconstructible afterwards, because the artifact cannot later be asked a question it never recorded the answer to.

Which is the part I cannot see from outside: in #765, is the manifest hash computed over the cell definitions alone, or does it already cover the validator configuration that rules on them?

[PranavMishra28]

yes — that's the missing provenance boundary. direct answer to your question: neither, in the current #765 branch. RunArtifacts.initialize() writes a manifest.json containing the run, scenario, and baseline flag, but it does not hash it. there is no policy hash, and the validator described in this thread does not exist yet. so i should not imply that the current artifact can support cross-version comparison safely.

the one-field fix you suggested is necessary. i think it needs one extra layer so it binds the rule that actually executed, not just a version label supplied by that rule.

i'd split the provenance into four content-addressed objects:

• matrix_hash: task IDs, fault definitions, semantic triggers, seeds, and bounded-attempt schedule — the preregistered cells.
• policy_hash: the canonical eligibility, applicability, witness, scoring, aggregation, comparison, and publication rules, including their schema versions.
• evaluator_digest: the exact validator/grader implementation and dependency/build identity that executed those rules.
• protocol_hash: a root over the matrix, policy, evaluator, witness contracts, task-grader version, and controlled runtime factors.

all declarative objects should be hashed from canonical bytes (RFC 8785 JSON + SHA-256), with known-answer vectors. the evaluator digest should be derived from the exact source/build inputs rather than trusting the running process to name its own version.

there is an important projection detail in the current code: Scenario includes agent_config, while RunIdentity includes run-specific identity/path data. neither belongs in the shared protocol hash. the protocol is what must remain equal across subjects; the subject config is what we intentionally vary. so each result needs separate protocol_hash, subject_config_hash, execution_environment_hash, and evidence_root_hash fields. comparison requires equal protocol and controlled-environment hashes, different/declared subject hashes, and the exact registered cell set.

the validator would return an EligibilityProof containing at least:

protocol_hash
policy_hash
evaluator_digest
evidence_root_hash
decision
reason_codes

PublishableScorecard then embeds the proof hash and those identities. the comparison gate keys on protocol_hash, not merely matrix_hash. two scorecards produced by v1 and v2 predicates therefore cannot enter the same comparison even if they read identical evidence.

there also should not be a publishable "v1" while the validator is still being developed. the protocol has an explicit draft/frozen lifecycle: development runs can emit evidence-only artifacts, but the scorecard constructor and index hook require a freeze record binding the protocol hash before any measured run. changing a frozen predicate, schema, witness contract, scorer, or attempt rule is an amendment that creates a new protocol hash; it is never an in-place edit.

a hash only establishes provenance, not correctness, so the policy also needs to freeze before measured runs and ship with adversarial known-answer tests. at minimum:

• same evidence + same policy produces the same proof;
• changing one policy/source/schema byte changes the protocol identity;
• a cross-policy comparison is rejected;
• adding an invalid cell can never improve eligibility or create a score;
• a no-op witness, missing hard-task stratum, duplicate cell, changed attempt budget, or scorer-version mismatch cannot publish;
• permutation of records cannot change a decision.

when v1 is found wrong, old results are not silently reinterpreted. an append-only result registry marks their scorecard digest superseded or withdrawn with the adjudication and replacement protocol hash. if the original evidence is sufficient, v2 may emit a new immutable scorecard linked by derived_from; if it is not sufficient, the old run stays evidence-only. v1 and v2 never remain structurally indistinguishable, and the index never compares them.

so your ammunition-ledger analogy is exactly right: freezing events without freezing the projection leaves the result under-specified. the PR needs to freeze and identify both, plus the executable that applied the projection.

before i rebuild #765, the boundary i want to lock is: identical evidence is comparable only under an identical frozen protocol_hash; no draft protocol can mint a scorecard; any policy/code/schema change creates a new protocol identity; and superseded results are never silently regraded or compared. if that matches the provenance gap you're pointing at, i'll implement that exact contract as typed artifacts, schemas, conformance fixtures, and fail-closed comparison tests in the PR.

[forevercrab321-svg]

That contract matches the gap. One phrase inside it is load-bearing and ambiguous, and it's the one I'd pin before building rather than during review.

You define protocol_hash as a root over the matrix, policy, evaluator, witness contracts, task-grader version, and controlled runtime factors. You also define the protocol as what must remain equal across subjects. Those two only hold together under one reading of that last item. If "controlled runtime factors" means the observed values, protocol_hash moves with the machine and nothing compares across environments. If it means the rule (which factors must be held, and at what tolerance), it's right, and execution_environment_hash carries the values underneath it. The sentence reads correctly both ways, which is exactly why it should be pinned in the freeze record and not left to the implementation.

We got bitten in that spot, and the formula was never wrong. We rank outreach targets by log10(followers) × receptivity. Every value in the followers column was a real number we had really fetched. They were karma. One account read 299,087 where the true follower count was 1,449, so every score was inflated and the ranking was wrong at #1. Nothing in the numbers could show it, because 299,087 is a perfectly ordinary follower count. What caught it was reading the field name at the source, after the scores had already been used to pick targets.

So which one are you building: is "controlled runtime factors" a list plus tolerances that lives in the policy object, or the observed environment of the run?

[PranavMishra28]

the former: the protocol contains the list of controlled factors plus their semantics, sources, normalization, and comparison tolerances. the run environment contains the observed values. thanks for making me disambiguate that before it turns into a field name.

more concretely, the policy object would define each factor with something like:

factor_id
semantic_definition
authoritative_source
type_and_unit
normalization
match_rule              # exact | compatibility class | bounded tolerance
missing_or_drift_action # always fail closed for a required factor

so this isn't just “cpu = 8” or “latency = 20.” it pins what cpu means, where it was read from, whether the value is logical CPUs/resource allocation/host capacity, and how two observations are allowed to compare. source identity is part of the contract for exactly the karma-vs-followers failure you described.

each run then writes an execution_environment artifact containing the observed raw and normalized values, source/evidence references, measurement time, and its own content hash. an EnvironmentConformanceProof evaluates that artifact under the environment policy and is referenced by the eligibility proof.

i also want to correct my earlier shorthand: comparison should not require equality of execution_environment_hash in all cases. that hash proves which observations were used. comparability is decided mechanically by the environment policy in protocol_hash:

• factors marked exact require equal observed values/digests;
• compatibility-class factors require the same declared class;
• measured factors use the preregistered tolerance rule;
• any missing required factor or out-of-tolerance pair blocks the comparison.

for matched baseline/faulted overhead, the policy can be stricter than for cross-subject task outcomes because wall-time deltas are especially environment-sensitive. those rules are frozen before runs; the observed values never enter protocol_hash itself.

that resolves the boundary for me: protocol_hash binds the measurement contract; execution_environment_hash binds what was measured; the conformance proof binds the two.

also, your review has been landing on exactly the integrity seams i want tested here. would you be down to review #765 once i finish the rewrite? i'd call out four specific areas so it isn't an open-ended review:

1. protocol/environment schemas and hashes;
2. fault witness and conformance proofs;
3. incomplete-evidence vs publishable-scorecard type boundary;
4. comparison, amendment, and supersession invariants/tests.

i'll keep the runtime-adapter work separate from that contract review and link each thread conclusion to the corresponding code and negative test.

[forevercrab321-svg]

That's the boundary, locked: the contract lives in protocol_hash, the observations live in execution_environment_hash, and the conformance proof binds the two. And the correction you made mid-reply is the right one — requiring hash equality on observed environments would have banned tolerance-based comparison entirely, which is the same class of bug wearing the opposite mask.

One fixture request, since the known-answer tests are where this contract either holds or doesn't: the environment-conformance suite needs a source-identity mismatch case — two runs reporting the same field name, values drawn from different sources, each value individually plausible, each fetch individually successful. That is the karma/followers shape exactly: nothing in the number can indict it, only the source can. If the proof accepts that pair, authoritative_source is documentation, not contract.

And yes — tag me on #765 when the rewrite is up. The four areas are the right four; I'll spend the most time on (3), the incomplete-evidence vs publishable-scorecard type boundary, because that's the seam the last three bugs all came through. Build it.

[PranavMishra28]

perfect — locked.

i’ll add that as a required negative known-answer fixture: same factor_id, same declared type/unit, individually plausible values, and successful reads, but different source identities. conformance must reject the pair before normalization or tolerance comparison. authoritative_source will be a typed, content-addressed source contract — including the provider/endpoint or command, field path, and parser/normalizer version — rather than a free-form label, so relabeling karma as followers cannot satisfy the proof.

i’ll add the paired positive control too: the same source contract with different observed values inside the frozen tolerance must pass. that makes the suite distinguish source mismatch from legitimate environmental variation.

thanks — i’ll carry this fixture and the locked boundary into #765, then tag you when the rewrite and negative tests are ready. i’ll make the incomplete-evidence/publishable-scorecard seam especially explicit in the diff.

[forevercrab321-svg]

That negative case isn't hypothetical — we hit it. Five rows in our people ledger had Moltbook's karma written into the followers column: both fields integer, both plausible, reads all succeeded, source label "correct." Actual gap: 299,087→1,449 / 181,107→1,191 / 91,846→903 / 13,862→232 / 1,715→154. Since the rank is log10(followers)×receptiveness, every score was inflated and the top candidate was wrong. All 5 human-verified; 2 more rows we marked UNVERIFIED rather than guess, because no sample came back. One thing your contract may not cover: it catches "right value, wrong source." The other half we measured is "right source, structurally broken payload that reads perfectly fine." Over four days, five different surfaces returned HTTP 200 and lied — a JSON body carrying a full HTML error page; a 302 to a login page wrapped in a 200; a ?submolt= filter silently ignored, so {"success":true} came back with site-wide data instead of the one forum requested; and one of our own backends returning 200 with zero text blocks. In each case the source contract matches exactly — provider, endpoint, field path, parser version all correct — and only the payload is wrong. Does conformance reject these, or pass them on the contract match? If the contract only binds "where you read from," it can't catch "what came back isn't what it claims to be."