[Proposal] Ship negative conformance vectors in v0.2

[rocklambros]

Negative vectors belong in v0.2.

That answers the go/no-go @tyche-dev asked for on #29 on August 21, which @avp9-nexus held implementation waiting on. Sixteen days is too long for a scope question. There's an item below aimed at stopping that from recurring.

Why v0.2

The MUST list is still moving. #21 is open and relaxes MODIFY, system/ping, and wrapped MCP to SHOULD in docs/spec/conformance.md. #33 flags the same inconsistency as unresolved. Falsification needs something settled to falsify, so any must-reject vector written this month gets rewritten when #21 lands.

Two smaller reasons back that up. We already told readers this was v0.2 work in docs/topics/faq.md:340, so shipping a conformance/ directory at 0.1.1 contradicts our own site. The other reason is that a partial suite feeds the problem #19 was filed about: "19/19" reads stronger than "self-attested" while resting on identical trust, and a number stops people asking what it covers.

Work that shouldn't wait

The two-verdict idea belongs in the spec. A refusal because evidence was measured and found wrong differs from a refusal because evidence couldn't be measured, and the operator does something different in each case. @avp9-nexus and @astrogilda arrived at this separately from opposite directions. ACS already has most of it in DEFER, which carries resolution_timeout_ms, required_context, and a timeout_decision whose enum has no allow path. The job is to write that down in normative text, where it costs nothing to maintain and survives every revision.

The six classes from #29 should anchor the v0.2 corpus, since they outlast spec changes: valid-but-out-of-mandate, replayed authorization, expired or revoked mandate, wrong-context approval, malformed evidence with a fail-open consumer, and stripped observability.

An earlier version of this issue recommended adopting #35's SHA-256 manifest pinning and its capability profiles. A premortem of #35 has since falsified both recommendations, and the detail is in the first comment below. In short: the manifest pins itself, so a hostile PR that edits a vector and regenerates the digests passes every CI step; and an empty capability profile scores zero vectors and exits 0. Neither mechanism should be copied as-is. What survives from #35 is the shape of the questions it asks, not its answers.

What the v0.2 suite has to get right

These came out of a premortem on #30, run against the branch rather than read off it. Writing them down so the next attempt starts here.

1. A published corpus can't prove enforcement. An adapter whose first statement discards the input scored 19/19 and exited 0, because the vector id encodes the verdict class. Redacting the id doesn't fix it. We emptied the question to {}, handing the adapter nothing whatsoever, and a seven-line sys._getframe walk still passed every vector. Only a held-out corpus tells a real implementation from a lookup table, and holding vectors back conflicts with publishing them. So the suite has to describe itself as a self-certification aid, the way MLPerf and SPEC do.

2. The gate needs negative controls of its own. Mutating the reference adapter measures the vectors. The runner goes untested. Nineteen runner mutants found five guards with nothing behind them, two exploitable, one of which silently restores a defect a reviewer had already caught and the author had already fixed.

3. Structural rules need their mirror. Add conformance/negative/: vectors, failure codes, and runner (closes the suite gap for the enforcement layer) #30 requires a positive control in every category, which is correct. Nothing requires a negative one, so a positives-only corpus certifies an adapter that allows everything, exit 0.

4. Adapters can't own the runner's process. sys.exit(0) inside an adapter gave exit 0 with no output at all, since except Exception misses SystemExit. Adapters need to run out of process, one child per run. Per-vector children break any implementation with real state, which includes anything that actually detects replay.

5. A constant can't be measured. All 19 vectors asserted signature: "VALID", so an implementation with no signature verification passed everything. Signatures are the one thing ACS-Core requires unconditionally.

6. Positive controls need tighter review than negatives. A positive control is a mandatory allow. One contributed vector can quietly demand a bypass, at which point the runner flags the honest implementation as the broken one.

7. Assert codes, never prose. Substring matching against operator messages failed correct implementations that worded things differently and passed lies naming the wrong cause. specification.md:129 already tells implementers to switch on codes.

8. Every vector needs a requirement ID and a spec version. Nothing in Add conformance/negative/: vectors, failure codes, and runner (closes the suite gap for the enforcement layer) #30 ties a vector to any ACS requirement, which means it can't drift. It stays quietly wrong. This is the argument for building the v0.2 suite the way [Proposal] Milestone: full ACS-Core conformance (requirement ledger + behavioral tests, beyond adapter emission) #33 scopes it, as a ledger over the MUSTs where CI fails when an applicable MUST has no falsifying test. Then a spec change tells you which test it broke.

Before the OWASP relaunch

Separate from the suite, cheaper, and aimed at what the relaunch actually exposes.

• Add the self-attestation paragraph to README.md and the top of docs/spec/conformance.md. [Bug] v0.1.0 ships without conformance test suite, registry, or steward, leaving the label unfalsifiable #19 called this v0.1.0 minimum hygiene. It was never written, and a grep for "self-attest" returns nothing.
• Rescope the Discussion gate in CONTRIBUTING.md from file paths to effect. Today it covers acs_schema.json, hooks, and events. Add conformance/negative/: vectors, failure codes, and runner (closes the suite gap for the enforcement layer) #30 touched none of them, declared its type accurately, complied honestly, and still landed a failure-code registry. Creating a top-level directory, publishing an enumeration, or using "conformance" in a path should all trigger a Discussion plus a maintainer answer inside a stated time.
• Add a catch-all row to LICENSING.md. Nothing governs conformance/** and there's no default, so a new top-level directory arrives licensed under nothing. Add adapters/ with Claude Code, Cursor, and NAT reference implementations #22 has the identical gap at adapters/**.
• Add /conformance/ to .github/CODEOWNERS and give it an owning workstream in GOVERNANCE.md. None of the five covers conformance or testing.

Related: #19, #21, #22, #29, #30, #33, #35, #52

[rocklambros]

Premortem of #35, and a correction to what I wrote above

Since #35 is getting the same decision as #30, we ran the same premortem against it. Four perspectives, run against the branch at c957fdf rather than read off it. This is worth recording because #35 is a genuinely independent second attempt at the same problem by a different contributor, so the places where the two converge tell us something the first premortem couldn't.

First, the correction. I wrote above that #35's manifest "makes a run checkable by someone who didn't produce it." That's false, and the way it fails is instructive.

MANIFEST.json carries a SHA-256 per file, and the README instructs contributors to "regenerate the manifest deliberately, in the same commit as the change it covers." Following that instruction literally: replace the adapter with one that allows everything, change return EXIT_FAILURES if failures else EXIT_OK to return EXIT_OK, change the selftest's return the same way, regenerate all six digests. Every CI step exits 0, including the manifest step, which reports 6/6 files match their pinned digest. The digests ship in the same commit as the files they pin, verified by a runner in the same commit, so the trust root is the pull request. Three narrower gaps compound it: the check iterates the manifest's own list rather than the directory, so an added file is invisible; it keys on basename rather than path; and it runs last, after the contributed adapter has already executed.

The manifest is worth keeping for what it does buy, which is catching accidental drift between a vector edit and its digest, and catching a runner pointed at a stale local copy. Neither is a security property. The generalizable version is requirement 12 below.

What both attempts got wrong independently

Six defects appear in both PRs, written by different people who reviewed each other. That makes them properties of the design pattern rather than of either author:

• The vector id encodes the verdict class, and both runners hand it to the adapter on the same line. Add capability-scoped revocation conformance vectors #35's runner.py:111 is byte-identical to Add conformance/negative/: vectors, failure codes, and runner (closes the suite gap for the enforcement layer) #30's runner.py:93. An adapter that discards the input scores 9/9 on Add capability-scoped revocation conformance vectors #35, output byte-identical to the reference adapter's.
• except Exception misses SystemExit, so sys.exit(0) in an adapter gives a clean exit with no output.
• The structural gate has no mirror, so a positives-only corpus certifies an allow-everything adapter.
• reason_must_mention is unanchored substring matching. One eight-word constant satisfies every reason check in Add capability-scoped revocation conformance vectors #35's corpus.
• The self-test hardcodes the reference adapter, so a submitted adapter never faces a single negative control.
• Nothing binds any vector to a spec version.

Convergence on this many points suggests the next attempt will reproduce them too unless the requirements below are written down first.

What #35 found that #30 could not

The capability profile is the interesting one, because it's a mechanism #30 doesn't have and it fails in a way nobody predicted. --profile "" or CAPABILITIES = [] resolves to an empty profile, every vector falls out of scope, and the runner prints 0/0 vectors conform and exits 0. The README states "the flag is not a way to opt out of being measured," and the asymmetry is what makes it dangerous: a partial opt-out fails loudly with SUITE INVALID: capabilities declared with no positive control, while a total opt-out passes silently.

Worse, the mechanism creates an incentive gradient. Tested against a corpus edited into the shape v0.2 would take, with two vectors requiring a second capability: the honest declaration naming both capabilities gets SUITE INVALID, exit 2. The narrow declaration naming one gets 7/7 vectors conform, exit 0. The documented remedy for that error message is to narrow the profile. Every implementer who hits it learns to under-declare.

Two more, both new:

• An in-process adapter can rewrite the runner's own exit constants. It walks sys.modules, finds the module whose __file__ ends in runner.py, and sets EXIT_FAILURES = 0. The report is completely honest, printing 3/9 vectors conform and six FAIL lines, and the gate is green. This defeats any defense based on scanning the adapter for suspicious strings.
• The CI job asserts on exit code and never reads the report. That's the amplifier: in both the exit-constant case and the hostile-manifest case, six FAIL lines printed under a green check. The selftest already does the stronger check internally, asserting f"{total}/{total} vectors conform" in out, and the workflow doesn't carry it across.

Additional requirements for the v0.2 suite

Continuing the numbering from the issue body.

9. A run that scores nothing must not exit 0. Add conformance/negative/: vectors, failure codes, and runner (closes the suite gap for the enforcement layer) #30 had an empty-suite guard. Add capability-scoped revocation conformance vectors #35 dropped it while adding the flag that opens the hole wider. Exit non-zero when zero vectors ran, and make it a negative control.

10. Self-declared scope needs a floor. A capability model where the implementation chooses which slice of the corpus it faces is self-attestation one layer down. Either mark a baseline set of capabilities as non-optional, or run everything and report a three-valued per-capability matrix where "not implemented" is derived from observed behavior rather than declared. Deciding the floor is a spec question, not a code question.

11. CI asserts on the report, not the exit code. Two lines of shell, and it's the cheapest fix on this list by a wide margin.

12. Digests get published outside the artifact they pin, and the integrity check runs first. A digest file living in the commit it protects is a drift detector, not an integrity control. Say which one it is in the README.

13. One runner. Suites are data. Add capability-scoped revocation conformance vectors #35's runner is 66% shared skeleton with Add conformance/negative/: vectors, failure codes, and runner (closes the suite gap for the enforcement layer) #30's, and the genuinely suite-specific logic in each is one line, the default vectors path. The copy also regressed two bugs Add conformance/negative/: vectors, failure codes, and runner (closes the suite gap for the enforcement layer) #30 had already fixed, including one with a three-line comment above it explaining the fix. Two suites means two copies of every future fix.

14. A conformance suite ships with a pytest entry point or it doesn't ship. A twelve-line tests/test_conformance_revocation.py shelling out to the two scripts lands the suite inside the test job, which is a required check, for about 0.4 seconds of CI time. Add capability-scoped revocation conformance vectors #35's 74-line workflow buys strictly less, because the ruleset requires only the test and build contexts and the new conformance context is advisory.

15. A workflow that executes contributed code is pull_request only, SHA-pinned, installed from the lockfile, time-bounded, and branch-filtered. Add capability-scoped revocation conformance vectors #35 gets the first right and declares contents: read, which is the dangerous variant correctly avoided. It misses the rest: actions/checkout@v4 and actions/setup-python@v5 are mutable tags in the only workflow in the repo that doesn't SHA-pin, pip install --quiet jsonschema bypasses a lockfile that already pins jsonschema 4.26.0, there's no timeout-minutes, and on: push carries no branches: filter, so a branch push runs a workflow definition that CODEOWNERS never reviewed. The hardening commit that established this pattern is an ancestor of the PR branch, so staleness doesn't explain it.

16. Never mark a paths-filtered workflow as a required check. A workflow-level path filter that doesn't match produces no check run at all, rather than a skipped one, so a required context sits waiting forever and blocks every unrelated PR. This is the trap waiting for whoever notices requirement 14 and fixes it the obvious way.

Two repo-level causes worth fixing regardless

Neither is about the suite, and both explain why this happened twice.

CONTRIBUTING.md never mentions pytest, tests/, or uv run pytest. It documents how to preview the docs and says nothing about the gate that blocks every merge. Two contributors independently built test suites outside the only directory pytest collects. Neither was careless, and the third will do it again.

The main ruleset sets strict_required_status_checks_policy: false. PR #30 and PR #35 share the same merge-base at c7ad162 and the same 36-commit gap, which is not a coincidence between two unrelated contributors. CI runs against the merge result while the human reviewer reads a diff against a six-week-old tree, and what the gap hid here was 97 files including the whole tests/ and tools/ trees and three of the five workflows. Both reviewers approved a workflow without being able to see the workflows it duplicates.

Where #35 stands on its merits

Two things it does that #30 does not, worth carrying forward as questions even though the answers need rework: it asks what makes a run checkable by a third party, and it asks how an implementation declares what it can legitimately not support. Both are the right questions. It also correctly chose pull_request over pull_request_target, which is the single decision that keeps a code-executing workflow from becoming a repository compromise.

One thing to settle before any of this returns: ACS has no revocation model. A grep across docs/ and specification/ returns sixteen hits and zero normative constructs, and docs/identity/overview.md:121 lists revocation propagation as Pending. docs/identity/overview.md:124 marks last-mile enforcement "partially specified" and records that the spec's fail-open default is in tension with default-deny, as an open work item. #35 hard-codes the refusal. Making that a merge gate would freeze an undecided working-group question by way of a contributor PR, which is a good argument for binding a suite to a spec version before binding it to CI.

cc @avp9-nexus @astrogilda, since both of you asked for the technical detail rather than the summary.

[avp9-nexus]

Thank you for this review. It is the most useful the branch has had, and the three repository fixes that landed on main the same evening say more about how this project treats contributions than the closure does. Contributing here was worth it and remains so.

Reproduced before replying, same as on #30, on a fresh clone of the branch at 00c024d in an offline sandbox. Six claims, six results.

• Published corpus: an adapter whose first statement is del q["input"] and which looks the id up in negative_vectors.json scores 19/19, exit 0, output byte for byte the reference adapter's. The sys._getframe walk does the same without reading the question at all.
• Runner never mutated: nine single-line weakenings of runner.py against selftest.py. Seven go red. Two stay at 24/24 green: dropping the verdict comparison, and reverting is_positive_control to the declared flag alone, which is the exact regression review 4996153628 caught and b3ef6a13 fixed. Your 5 of 19 and my 2 of 9 are different mutant sets; it is the same finding.
• Positives-only corpus: the six PASS vectors plus an allow-everything adapter, 6/6 exit 0. The same adapter on the full corpus fails 6/19, exit 1, so the suite discriminates only because negatives happen to be there.
• sys.exit(0) in the adapter: exit 0, no output.
• Signatures: 16 of the 19 vectors carry signature: "VALID", three category-4 vectors carry no signature field at all. Your point stands unchanged: no vector carries an invalid one, so an implementation that never verifies a signature passes everything.
• runner.py:93 hands the adapter the id, as you say.

Nothing you wrote about this branch is wrong, and the two runner guards with nothing behind them are the sentence about the gate nobody mutated, one level further in than I looked.

The deferral and its reason are accepted. Vectors written against a MUST list that #21 is about to change would be rewritten on landing.

For v0.2 I take four things from #53 as written: the suite describes itself as a self-certification aid, since a published corpus cannot separate enforcement from a lookup table; adapters run out of process, one child per run; the runner ships with its own negative controls; and every vector carries a requirement id and a spec version, which is the #33 shape. One concrete offer, for whenever you want it and not before: the normative wording of the REJECT versus UNMEASURABLE split against DEFER, since timeout_decision already has no allow path and you said that is where the distinction goes. I will hold the branch until v0.2 opens and pick #53 up there.

[rocklambros]

Triage note, ahead of the OWASP relaunch. This is now labelled scope:deferred, which means tracked and scheduled rather than rejected. The project has published a Current Priority Scope committing the next ninety days to one outcome, a runnable Guardian reference implementation benchmarked against Microsoft AGT. Work that does not feed that outcome lands in the v0.2.0 window instead of being closed.

You called this one yourself: the issue is titled for v0.2 and argues the MUST list is still moving while #21 is open. Triage agrees, and this is the record of that agreement. It also settles the go/no-go that #29 was waiting on.

Nothing here needs re-filing. It stays open, keeps its labels, and moves to the backlog when the v0.2.0 window opens. If you think it belongs in the ninety-day window instead, say so on this thread and a maintainer will re-scope it.

[astrogilda]

Six findings, all six reproduce here, in your order.

The digests ship in the same commit as the files they pin, verified by a runner in the same commit, so the trust root is the pull request.

Right: my README taught the attack. The manifest becomes a drift detector labelled as one, with digests published outside what they pin and the check running first over the directory.

#35's runner.py:111 is byte-identical to #30's runner.py:93.

That line was copied from #30, and redaction does not fix it, so the verdict class comes out of the vector id altogether, as avp9-nexus already did at e70f677.

--profile "" or CAPABILITIES = [] resolves to an empty profile, every vector falls out of scope, and the runner prints 0/0 vectors conform and exits 0.

Mine twice over: #30 had the guard I dropped while widening the hole, so a run that scores nothing exits non-zero, with a negative control.

Every implementer who hits it learns to under-declare.

I would have missed this one, having written the error message that teaches it; the floor belongs to the working group, not to a runner inferring one.

An in-process adapter can rewrite the runner's own exit constants.

No string scan reaches that, so adapters run out of process, one child per run, as your requirement 4 says.

The CI job asserts on exit code and never reads the report.

Two lines, and the selftest already carries the assertion the workflow does not.

Next on v0.2, concretely: one runner with the suites as data, a twelve-line pytest entry point under tests/ rather than a paths-filtered required check, and a requirement id plus spec version on every vector, so the corpus binds to the text before CI.

[avp9-nexus]

On the v0.2 list above: the corpus from #30 now exists in that shape, on a branch of its own. Every vector carries the spec version it was written against, the profile, the anchor in docs/spec/conformance.md, the clause and the sentence it quotes; ledger_id is present and null on all of them, since the ledger proposed in #33 does not exist yet and an invented identifier would look like a binding.

Three defects it shipped with are fixed and each has a control that fails without the fix: the runner resolves entry_point instead of comparing it as a string, the schema that shipped beside the vectors is executed at load, and a RESULT line closes the output so the headline and the exit code cannot disagree. The selftest holds 28 negative controls where it held 24.

It is here: https://github.com/avp9-nexus/agent-control-standard/tree/negative-vectors-v0.2 The branch of #30 is untouched, so the review on that thread keeps pointing at the code it reviewed, and the details are in a comment there. Nothing is asked here; if the v0.2 window wants this corpus, it can be taken as it stands.

Avp9.

[astrogilda]

That corpus is published, so a real implementation and a lookup table score the same against it. A design at https://github.com/astrogilda/agent-evidence-vectors/blob/main/docs/HELD-OUT-CONFORMANCE.md records the measurement: over 272 vectors a classifier predicted the verdict from the path alone. Separability was 1.0000 against a permutation null of 0.5879, and 0.5075 once each vector is named after a digest of its bytes. Rename the v0.2 vectors by digest and I will post the same measurement over the branch.