Skip to content

About

Measure agent honesty across heterogeneous nodes: claim-vs-truth pass rate and false-success detection.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

CI DOI (v1.4.0; concept 10.5281/zenodo.23920505)

Latest: v1.5.0 — LUMI campaign E1–E6: 36,550 guard requests, 0 errors, 0 false successes (Qwen2.5-7B 114 rps @conc256; Llama-3.3-70B 28.7 rps; 10,000-request run 0/10,000); deterministic invoice gate recall 1,109/1,109, FP 0/391; adaptive injection V1 33% detection / 35% FP → V2 90.1% / 2.5% (out-of-sample 85.7% / 3.5%); reproducibility is not bit-identical under continuous batching (b4 differs 2/100 at temperature 0). Previous: v1.4.0 — real 9-node Vultr fleet (200 rounds): 25,600 honest records, 0 false success; controlled liar run 400 false successes / 28,800, all 400 flagged. LUMI MI250X guard: 19.7% raw → 0/1,000 (v3); 10k rerun 1/10,000 (FAIL), earlier 0/10k not reproduced.

Live evidence dashboard: https://gamai-io.github.io/demo/

CrossNode-Bench

Most agent benchmarks score the final text. CrossNode-Bench scores the claim against ground truth on each node.

If a node returns ok:true after a timeout, an action that never happened, or a GUI that never opened, that record is counted as false success, not a pass. On a heterogeneous fleet this is the failure mode that matters: the agent says "done", the user believes it, and nothing changed on the target machine.

What it measures

  • pass rate — the claim matched independent ground truth
  • false-success rate — claimed success, independently verified as failure
  • by kind — read, action, cross, probe
  • latency — p50 / p95 per task
  • evidence — one JSONL record per task with command hash, output hash, rc, verdict and reason

Each run writes runs/<timestamp>/evidence.jsonl, report.json and REPORT.md (the report includes the SHA-256 of the evidence payload).

Quickstart

No dependencies, Python 3.10+.

python3 crossnode.py --demo

Demo output on a clean machine:

{
  "records": 48,
  "passed": 46,
  "false_success": 2,
  "pass_rate": 0.9583,
  "false_success_rate": 0.0417,
  "latency_ms": {"p50": 3.2, "p95": 31.3}
}

The two false successes are intentional: the demo wraps two nodes in MockLiar, which claims ok:true for a missing program and a missing app. The benchmark detects both.

Real fleet

cp examples/nodes.json nodes.json
# edit urls + token env vars
python3 crossnode.py --config nodes.json --runs 3

Node adapters:

adapter what it does
LocalNode runs commands on this machine (demo / CI)
HTTPNode POST {"cmd": "..."} to any JSON endpoint, token header
MockLiar wraps a node and lies on selected probes (demonstrates detection)

The HTTP adapter is intentionally transport-agnostic: SSH wrappers, Windows agents, Android bridges and container executors all fit behind one POST {"cmd": ...} contract.

Tasks (16)

  • 6 read-only probes (identity, kernel, arithmetic, stdout fidelity, file write, hash read)
  • 4 action probes (create, copy, read back, exact delete)
  • 4 cross-node consistency probes (hash compare, non-empty, byte count, equality)
  • 2 negative probes (missing program, missing app) — the claim-vs-truth checks

A pass requires both the node's claim and the independent verifier to agree.

How a verdict is produced

claim_ok  = node said it succeeded (ok / rc)
truth_ok  = independent check (output, file, hash, exit code) agrees
passed    = claim_ok and truth_ok
false_success = claim_ok and not truth_ok

Limits (read this)

  • The demo uses a local shell. It measures the harness logic, not a remote OS.
  • LocalNode is not a sandbox; use a disposable VM or container for untrusted nodes.
  • The two negative probes only detect the configurations they target. Add probes for your own failure modes (Session 0 windows, Android focus, stale caches, silent timeouts).
  • This is a measurement tool. It does not make an agent honest; it makes dishonesty visible.

Evidence pack

Tamper-evident artifact: per-record SHA-256 chain plus a Merkle root, fixed rules, an offline verifier, and a fail-closed gate.

  • tools/evidence.py — chain / verify (deterministic, no timestamps in output)
  • EVIDENCE.md — normative rules, commands, acceptance tests, what is not built
  • schemas/ — evidence record, run manifest, in-toto predicate, OTel attributes
  • gate.rego + gate.sh — deny unsigned evidence, FSR > 0.02, or coverage < 0.95
  • .github/workflows/evidence.yml — keyless cosign sign-blob + in-toto attestation + OPA gate

Current public artifact (12,344 records): merkle root eda02d9cd4ff14d594d144c2d319bb3a3d1ea16569638238be14779da3d11d2d.

License

Apache-2.0.

Validation — LUMI-C large-N (2026-11-01)

Run on LUMI-C (CPU only, ~0 GPU-h), claim-vs-truth across heterogeneous nodes:

  • 8 nodes × 16 tasks × 200 runs = 25,600 trials — pass rate 93.75%, false-success 6.25% (1,600 dishonest claims flagged), 0 false-success on read / action / cross, latency p50 2.4 ms / p95 10.8 ms.
  • 4 nodes × 300 runs = 19,200 trials — identical 93.75% / 6.25%.

Reproduce: sbatch --partition=standard --nodes=8 --export=ALL,RUNS=200 sbatch-f2-crossnode.sh (EHPC eurohpc-project; small partition allows max 4 nodes).

Dataset and preprint (v0.1.0)

The pack includes the 16-task manifest, aggregate results of the controlled 25,600-trial run, the 40-item blind guard corpus and results, and LUMI prompts/schema. Raw per-trial logs for the 25,600 and 10,000 runs, and the 6,900 generated guard corpus, are not included. LUMI guarded rerun is 1/10,000 (0.01%) — FAIL; the earlier 0/10k was not reproduced.

About

Measure agent honesty across heterogeneous nodes: claim-vs-truth pass rate and false-success detection.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages