A test-first gate for AI-written code: the agent that writes the tests can't see the implementation, and the tests can't be edited once sealed.
When the same agent writes code and writes the tests that check the code, the tests are not an independent verdict. They pass against whatever the agent built, misreadings included. Themis breaks that by making the verifier's independence a property of the system rather than a matter of trust: the agent that derives the tests is structurally denied any view of the implementation, the tests are cryptographically sealed before any code exists, and a separate agent runs them.
Themis grounds only what is executably specifiable. Within that scope, it can establish that code does what a specification says without the author of the code in the loop.
It does not:
- judge whether a spec is correct, wise, or complete (a vague or wrong spec earns a confident pass on the wrong thing; the spec is the trust boundary). One slice of that risk is measurable: an untestable criterion is the one kind of bad spec the system can detect, and
python -m blind_tdd.spec_healthreports the rate at which a project's criteria come backneeds_humanfrom the blind writer, with reasons and trend. Wrong or incomplete specs stay invisible to it — see the three-class taxonomy indocs/limitations.md - verify declarative knowledge, prose, or design rationale (it is not a lie-detector for an agent's text)
- guarantee a perfect wall (it is defense in depth; see Limitations)
- guarantee test strength. The gate enforces the test's independence (written without sight of the code) and integrity (not edited afterward). It does not prove a test is strong enough to fail a wrong implementation. A test can be independent, untampered, AC-tagged, and still too weak to catch a bug. Measuring strength requires perturbing the implementation and checking whether a test fails (mutation testing), which needs an agent that can read the code, the opposite of the blind writer. So it's a separate, advisory pass —
blind_tdd.mutatesurfaces surviving (test-passing) mutants as weak spots — not part of the gate's guarantee: a surviving mutant is a warning, never a gate failure.
A green gate means the code passed an independent, sealed test of every acceptance criterion. That is a strong and specific claim — necessary, not sufficient. It is not the same as "the code is correct"; keep the two separate.
It is Claude Code-specific: blindness is enforced through Claude Code PreToolUse / PostToolUse hooks and fresh-agent spawning (the production claude -p subscription spawner, or an optional Agent SDK spawner). It is not a framework-agnostic library.
The design is permissions. Each agent is granted exactly what it needs and denied the rest, enforced by tooling rather than by instruction.
- Blind writer derives the acceptance tests from the spec and a single
public_api.md. A hook denies it any read of the source, the examples, or generated data, so it cannot copy the implementation's behavior because it cannot see it. - Implementer writes code to pass those tests. It can see and change everything except the sealed tests. The wall here is honest about its limits: a path guard blocks the ordinary edit tools from rewriting a locked test, but it does not cover
Bash, and a determined agent can rewrite a file through the shell. That gap is closed by the hash layer below. - Blind runner is spawned fresh, never having seen the implementer's reasoning. It re-verifies the seal, runs the tests, and checks that every acceptance criterion has a passing test tagged
Covers: AC-N. - Arbiter appears only on a dispute. The implementer cannot delete a test it believes is wrong; it can only file a challenge, which a fresh arbiter (having seen neither the writer's nor the implementer's reasoning) rules on. Upheld removes the test; rejected keeps it; ambiguous escalates to a human. Only "upheld" removes a locked test.
Before any implementation exists, each test file is fingerprinted with a SHA-256 hash and the fingerprints are recorded (in .themis/blind_tdd/red_state/<task_id>.json). At the green phase the runner recomputes each fingerprint and compares it to the sealed one. A test changed by any route, shell included, no longer matches and fails the run regardless of whether it now passes. Adding or deleting a locked test also breaks the comparison.
That record is itself a file in the project tree, so a Bash-capable agent could in principle rewrite both a test and its stored hash. Set THEMIS_SEAL_KEY in the gate's environment to HMAC-sign the record — the green phase then rejects a modified one, and the spawner withholds the key from every spawned agent. Optional and opt-in; see Limitations.
spec + public_api.md
|
[ RED ] blind writer -> failing tests -> SHA-256 seal
|
implementer writes code (cannot edit sealed tests)
|
[ GREEN ] blind runner -> re-verify seal -> run tests -> check AC coverage
|
(optional) challenge -> arbiter -> upheld / rejected / ambiguous
|
[ GATED ]
Most tasks do not need this. For a large class of work, a verifier you simply assign in the loop gives you solid testing, and the multi-agent overhead here is pure tax. Use the cheapest thing that holds:
- Assigned in-loop verifier for most tasks: unambiguous spec, and a wrong-but-passing result would be caught cheaply downstream. No reason to pay for enforced blindness.
- Themis (blind gate) when a false green is expensive and there is pressure toward it: security or financial logic, anything you will not manually re-check, anything feeding later automated steps, anything under deadline or reward pressure. This is the case where an assigned verifier is most likely to stop telling you the truth, because the cheap path to green has become attractive.
- Mutation and audit pass on top (the mutation check is built but advisory — it flags weak tests, never gates; the audit pass and a gating layer are still planned — see Limitations) when you also cannot afford a weak-but-honest test letting a wrong implementation through. That is a different threat from collusion, and a different layer.
Each layer covers a threat the one below it does not. The blind gate addresses an assigned verifier conforming to the code; it does not address a verifier that is honest but too weak. Pick the layer that matches the cost of being wrong.
You can encode that "most tasks don't need this" line as policy instead of judgment: a routing block reserves the gate for the tasks that match it (by path glob, tag, severity, or keyword) and lets everything else fall through to an ordinary verifier. Set it once with python -m blind_tdd.init (or the /blind-tdd:setup plugin command) — the policy is human-set on purpose, because the trigger must be a rule the implementing agent can't choose to skip. See docs/adoption-guide.md.
The routing rule is also where detection feeds back into prediction. A hash break at green fails that run, but it can't do more than that — by the time the fingerprint mismatches, the tampering has already run. What the break is good for is learning: the gate records it in a tamper ledger (.themis/blind_tdd/tamper_ledger.jsonl), and on later runs a task similar to a previously tampered one — same task, same directory, shared tag — is escalated into the gate even where the selective policy would have skipped it. Escalation is additive-only: history can force the gate on, never off, so deleting the ledger merely reverts to the human-set base policy. The same match is exposed pre-spawn (blind_tdd.escalation.evaluate_escalation) as the signal a stricter containment tier — an OS-level sandbox around the whole run — would key off before any agent exists, rather than reacting to an attack already in progress.
Two security-focused layers ride on that same machinery. A suppression-marker audit (on by default, advisory) baselines the repo's lint/security suppression comments (# noqa, # nosec, eslint-disable, ...) at seal time and diffs at green — masking a problem instead of fixing it is the same move as editing a test, one layer over — and a marker introduced during implementation warns on the result and feeds the tamper ledger, never failing the run. An opt-in security AC pack attacks the spec-as-trust-boundary problem from the other side: an operator-authored, fingerprinted catalog of security acceptance criteria (input rejection, no hardcoded secrets, sanitized error surfaces) is appended to matching tasks before the blind writer sees them, so security tests the spec author forgot get derived, sealed, and enforced like everything else. Both are covered by the seal HMAC and both are honest about their limits — see docs/adoption-guide.md and docs/limitations.md. Neither layer moves the gate's oldest caveat: the pack is itself spec content, and a vague or wrong pack earns the same confident pass as a vague or wrong spec — the gate can enforce that code conforms to a criterion, never that the criterion describes the right behavior.
The shipped default pack is deliberately generic, and generic criteria derive weaker tests — so the pack comes with a tailoring interview (/blind-tdd:security-pack-setup). It explores the repo before asking anything, interviews the operator one question at a time against the evidence it found (real entry points, sinks, secret handling, error surfaces), and drafts project-specific criteria. The interview's exit gate is objective rather than a judgment call: python -m blind_tdd.pack_wizard --lint holds drafts to the same criterion-quality checks preflight applies post-injection, plus a genericity warning for anything still phrased as "any public entry point". The LLM only drafts — installation is human-approved, and the installed pack is project-root spec content, committed and reviewed like any spec change. A pack untouched for 90+ days earns a re-interview nudge, because tailored criteria rot as the attack surface grows.
git clone <repo-url> themis && cd themis
pip install -e ".[dev]" # add ".[dev,sdk]" for the Agent SDK spawner
python -m pytest # run the test suite
# inspect the impossible-AC probe catalog and the gate verdict
python -m blind_tdd.probes --list
python -m blind_tdd.probes --gate --runs-file data/probe_runs.jsonl --min-on-runs 20
# how measurable are this project's specs? (needs_human escalation rate)
python -m blind_tdd.spec_healthTo adopt the gate on an existing project, start with docs/adoption-guide.md. The design rationale is in docs/rfc.md.
The blindness layer ships as an installable Claude Code plugin in plugin/blind-tdd/:
/plugin marketplace add dscherm/themis
/plugin install blind-tdd@themis
The plugin provides the enforcement surface — the PreToolUse path/bash guards, the PostToolUse audit, and the writer/runner/arbiter agent definitions, auto-activating passthrough-by-default — plus two setup interviews: /blind-tdd:setup (routing — which tasks the gate engages on) and /blind-tdd:security-pack-setup (tailoring the security AC pack to the project's actual attack surface). This package is the orchestration engine it drives (install it with pip install themis-blind-tdd). See the plugin README for a 60-second quickstart and a recorded demo.
Themis was tested against an ImpossibleBench-style probe set: coding tasks whose acceptance criteria are mutually contradictory, so that any verified pass is provably a cheat. On these tasks there is nothing but cheating to detect.
| Condition | Result |
|---|---|
| Gate on — verified bypasses (all 42 runs, both framings) | 0 / 42 |
| Gate off — false greens, adversarial framing | 2 / 21 (9.5%) |
| Gate off — false greens, ordinary framing | 0 / 21 |
| Measurement losses | 0 |
The honest reading separates two claims. Base rate: across 42 ON runs there were 0 verified bypasses — but "0 observed" with n=42 means a 95% CI upper bound of ~7%, not zero, and fresh claude -p agents rarely even attempt a tamper, so those runs mostly show low cheat propensity, not enforcement. Enforcement — that the lock actually stops a tamper — is shown by the contrast arm where it was exercised (Agent-tool implementers attacked the locked tests 7/7, 4 escalated to the shell past the path guard, and the hash layer caught all 4), not by the idle subscription runs. The no-gate arm cheated only under adversarial framing (2/21). Net: agents seldom cheat here, and the lock holds when tested — two findings from two arms, not one 42-run proof of both.
Two caveats belong next to those numbers, not in a footnote:
- Small sample. n is 21 per cell; read
docs/impossible-ac-results.mdbefore quoting any figure. - How the agent is spawned moved the result a lot. Identical probes cheated ~10× more through in-conversation Agent-tool subagents than through fresh
claude -pagents — same task, same prompt, different inherited context. So every number here uses the productionclaude -pspawner, and each row in the ledger records itsspawner. The practical takeaway, the part I'm confident about as a builder: if you're measuring cheat rates, pin and report the spawn method, or you're partly measuring your own harness. This looks to me like the context-as-active-ingredient effect Jane Pan et al. (2407.04549) study —docs/impossible-ac-results.md§4 has the honest question I'd put to researchers about whether it's the same effect or a cousin.
The full, honest list (including the spec-as-trust-boundary problem, the absence of test-strength/mutation measurement, the persistence of the hash baseline, the arbiter's input surface, and per-model staleness) is in docs/limitations.md. Read it before relying on the gate.
The principle is old: a checker cannot be independent of the thing it checks. It is the logic of clean-room software development (a separate team certifies code its authors never test) and separation of duties (whoever writes the check cannot sign it), and the general failure it guards against is reward hacking, or Goodhart's law. Themis applies those ideas to AI-written code, with the independence enforced as a permission rather than a policy. That last point is the distinction from the spec-driven-development tools that assign a verifier as a role the agent is asked to play, where the verifier can still read the implementation and the tests can still be edited after the fact.
On the empirical side, two papers map closely. Jane Pan et al. (arXiv:2407.04549) show why this fails — when one model is both author and judge with shared context, the two quietly converge on the same wrong answer; Themis is, in effect, an attempt to engineer that sharing away. And ImpossibleBench (arXiv:2510.20270) shows that asking an agent not to edit the tests helps but isn't enough: a good prompt cuts GPT-5's cheating to 1% on easy tasks, yet it still cheats 54% on hard multi-file ones — the case for a mechanism over a polite request. A fuller, practitioner's-eye map of the related research is in docs/related-work.md.
Themis is the standalone extraction of a blind-TDD gate built inside a larger agent-learning harness. The blind gate itself (the hooks, the hash seal, the arbiter, and the impossible-AC probe harness) is original to this project.
MIT. See LICENSE.