fp-check - merge with concept-prover and online-triage, add evals

[GrosQuildu]

Rewrites fp-check into workflows, merges it with concept-prover and online-triage (as optional steps); adds tests and evals.

Based on measurements it works as good as the old fp-check and concept-prover run together, but saves tokens and provides single unified skill.

What it does

Decides whether a suspected security bug is real, and says why. You hand it a
finding; it returns TRUE POSITIVE, FALSE POSITIVE or NEEDS MORE INFO
with the evidence behind the answer.

Three stages, in order:

Stage	When	What it does
0 Batch	optional	derives the shared context once, runs Stage 1 per finding with it, accounts for every finding by id, and then checks the pairs that are only exploitable together
1 Static	always	traces the path from entry point to sink, one agent per validation layer, then checks recovery, upstream fixes, impact and severity, and finally six gates
2 Online	on request	the project's published policy, bounty scope, past reports and duplicates. Refuses to answer offline rather than guessing from memory
3 PoC	on request	builds a real exploit, runs it, then five agents that did not build it try to reject it

Stage 1 always reaches a verdict on its own. Stages 2 and 3 can only narrow or
correct it.

The design point: every gate is a pure function in the workflow script, not an
instruction an agent is asked to follow. The script returns a status you cannot
argue with. That distinction is the whole reason the plugin exists — the same
rules, written as prose in a prompt, were not obeyed.

Two examples of what that buys: a finding already fixed upstream is retracted and
must cite a commit, and an integration or external root cause is capped at Medium
by arithmetic rather than by asking the model to be modest.

Stop Hooks Remove

The two Stop/SubagentStop prompt hooks were dropped rather than fixed because the completeness check they performed is now enforced deterministically by the workflow gates, while every observation we have of the hooks themselves is negative — invalid JSON on every session end, firing in sessions that never used the plugin, blocking correct standard-path output.

What was measured

Question: does loading the plugin produce better security conclusions than the
same prompt with no plugin at all?

Method: claude plugin eval with an ablation baseline — the identical prompt
is run twice, once with the plugin loaded and once with no plugin at all, and the
difference between the two scores is the delta. The two configurations are called
the with-arm and the no-plugin arm.

Because a language model gives a different answer each time, every case is run
3 times per arm — that is what n=3 means. So: 7 cases × 3 runs × 2 arms =
42 runs, and a case's score is the mean of its 3 runs. A case scores 1.000 only
if all three runs got it right.

Each case scaffolds a small realistic codebase and reports a finding the way a user
would; some reports are correct, some are wrong in a specific way. Every case is
graded by an LLM judge reading the final answer, plus at least one deterministic
regex or tool-use check — never an LLM grader alone, because it will happily pass a
run that described doing the work instead of doing it. Both arms and the judge
run on Sonnet.

The seven cases

Each one is a finding reported to the tool, and each has a specific right answer:

Case	The report	The right answer
integration-cap	a real ledger bug, reported as Critical	real, but the attacker does not control the upstream value, so severity caps at Medium
blocked-attack-path	SQL built by string concatenation	false positive — two validators reject the metacharacters injection needs
already-fixed	a timing oracle reported against v1.4.0	retracted — the fix landed one layer up in the caller; only git history shows it
dead-route	shell injection in a handler	false positive — no route in the routing table reaches that handler
inflated-impact	a panic claimed as a remote process crash	real but over-stated — Go's net/http recovers per connection, so one connection drops and the server keeps serving
wrong-parameter	shell=True and attacker input in one file	false positive — there is no path between the two; the report joined the wrong sink to the wrong source
should-not-fire	"explain what this function does, I am writing API docs"	do nothing — this is not a security request, and the tool must not activate

Which stages were measured

Only Stage 1 is measured across the board. All seven prompts pin "work
offline", so:

Stage	Coverage
1 Static	measured on all 7 cases, both arms — this is what +0.281 describes
3 PoC	six prompts ask for a PoC, but Stage 3 only runs on a TRUE_POSITIVE, and only integration-cap reaches one. So it is exercised by one case
2 Online	never measured. Its one case does not discriminate — a plain session scores 1.000 on it — and it needs a suite built on real public advisories, which does not exist yet

So this is strong evidence for the static path, thin evidence for the PoC path, and
no evidence for the online path.

Results

v2.0.0 — 42/42 runs clean, $29.47, 2h26m

	with plugin	no plugin	delta
Mean over 7 cases	1.000	0.719	+0.281

Per case:

Case	with	without	delta
integration-cap	1.000	0.333	+0.667
blocked-attack-path	1.000	0.400	+0.600
already-fixed	1.000	0.500	+0.500
dead-route	1.000	0.800	+0.200
inflated-impact	1.000	1.000	0.000
should-not-fire	1.000	1.000	0.000
wrong-parameter	1.000	1.000	0.000

Free test layers: 358 node, 382 pytest, 44 bats, validator clean. Mutation gate
146 mutations run, 0 survived — each covered behaviour is broken in a sandbox
and the suite is required to go red, so the tests are known to test something.

Conclusions

It works, and the effect is concentrated. Four of seven cases carry the whole
delta; three contribute nothing because a plain session already scores 1.000 on
them. The wins are exactly where gates are enforced in code: rejecting an
unreachable sink, retracting an already-fixed bug, capping an inflated severity.

Judge progress by the with-arm score, not by the delta. The delta is a
subtraction: with − without. Both halves move between sweeps, so the delta can
rise because the plugin improved or because the no-plugin baseline happened to
score worse that day — and you cannot tell which from the delta alone.

Across the four sweeps the no-plugin baseline scored 0.656, 0.700, 0.652, 0.719.
Nothing about the plugin changed those; that spread is just run-to-run variation at
3 runs per case. The with-arm over the same four sweeps went 0.868 → 0.908 → 0.919
→ 1.000 — that one only moves when the plugin changes, so it is the number that
actually tracks progress. It is also a more modest claim than the deltas suggest.

The suite is now saturated, and that is the binding limit. All 21 with-arm runs
scored exactly 1.000. There is no headroom left to measure an improvement in, so
the next change to this plugin cannot be evaluated by this suite — it will report
roughly +0.28 whatever the change does. A harder case is owed before another sweep
is worth paying for.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ GrosQuildu
❌ t

---

t seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[claude]

🟡 poc-lint.sh's multi-line docstring/comment tracker (awk pass, ~L351-378) uses index($0, closer) to find the closing """, which matches ANY occurrence of that 3-char sequence regardless of escaping. If a docstring body line contains a raw """ (e.g. an escaped triple-quote or prose mentioning Python's triple-quote syntax), tracking closes one line early, the real closing line re-opens a bogus docstring, and the following line (often the function body) is swallowed into comment mode — hiding stub/reimplementation/ellipsis violations from rules 3, 6, and 7. Verified by execution: a PoC with a stub pass body plus such a docstring line lints clean and exits 0; removing that one line correctly flags stub-body and exits 1. Fix the closer-matching to account for escaped/nested triple-quotes (e.g. track raw string state or check for an odd number of preceding backslashes) and add a bats fixture for it.

Extended reasoning...

The bug. The awk pass that tracks multi-line docstrings/block comments uses a single closer variable and closes it with a bare substring test:

closer != "" { if (index($0, closer)) closer = ""; next }
...
if (index(substr(body, length(opener) + 1), terminator) == 0) closer = terminator

index() reports the position of the FIRST occurrence of the 3-character string """ anywhere on the line, with no regard for escaping, nesting, or whether it's inside a different quoting context. It cannot distinguish an actual closing triple-quote from an escaped one (\""") or from three quote characters that just happen to appear together in prose.

The trigger. Consider:

def exploit():
    """
    Some text with \""" escaped inline, not actually closing.
    Real content: after that.
    """
    pass

Line 1 (""") sets closer = '"""'. Line 3 contains the literal 3-byte sequence """ (as part of \"""), so index($0, closer) finds it and clears closer — one line early. Line 4 (Real content...) is now read as ordinary awk-processed code (harmless here, but this is where a real function body would slip through). Line 5, the REAL closing """, is then evaluated fresh against the opener test (index(body, TQD) == 1), matches, and re-opens a brand-new docstring. That newly-opened (bogus) docstring then swallows line 6 — the pass stub body — into comment mode, so it never reaches the awk rules that check for stub bodies (rule 7), reimplementation (rule 6), or ellipsis placeholders (rule 3).

Why nothing else catches it. poc-lint.sh is the sole mechanical gate for these rules; there's no secondary check. index() is a plain substring search with no awareness of Python string-escaping semantics, so any of the following independently trigger the desync: an escaped triple-quote in prose, a docstring that describes Python's """ syntax, or simply three quote characters landing adjacent for unrelated reasons.

Impact. triage-poc.js's isAcceptableBuild gate trusts lintPassed, and the independent artifact-check agent re-runs this same script and gates on lintExitZero. Both trust the exit code as ground truth. A PoC whose docstring happens to contain a raw """ sequence, with an actual stub (pass/...) or reimplemented body, would lint clean and pass both the build gate and the supposedly-independent re-check — exactly the "checker that silently passes is worse than none" failure class this repo's own guidance calls out as the most expensive kind of bug.

Step-by-step proof (independently reproduced by three verifiers via direct execution):

1. Write a Python file with def exploit(): whose docstring's second line contains Some text with \""" escaped inline... and whose body is just pass.
2. Run bash poc-lint.sh --symbol target_app.ledger.transfer_balance file.py.
3. Output: 1 file(s) clean, exit code 0 — the stub body was never inspected.
4. Delete only the line containing the embedded \""", keeping the docstring otherwise intact and the pass body unchanged.
5. Re-run the identical command: now it reports stub-body ... A definition whose whole body is a placeholder and exits 1.
6. The only variable changed between steps 3 and 5 is the presence of the embedded triple-quote — which conclusively demonstrates the tracking desync, not a difference in the actual code being linted.

The fix. The closer-matching needs to account for escaping — e.g. treat a """ preceded by an odd number of backslashes as not a real terminator, or more robustly, scan character-by-character tracking raw-string/escape state rather than doing a bare index() substring search. A bats fixture reproducing this exact case (stub body + docstring containing an embedded triple-quote) should be added to poc-lint.bats to lock in the fix, since this file's own stated design principle is that "a checker that inspects nothing must fail, not pass" — the same standard applies to a checker whose internal state can be silently desynced by attacker-controllable (or just incidental) input.

[claude]

🟡 citedReference() in workflows/triage-static.js (duplicated verbatim in triage-poc.js) uses the regex [0-9a-f]{7,40} to detect a commit-sha citation, but since decimal digits are a subset of hex, any bare 7-40 digit number (a date like 20240115, a unix timestamp, a large ticket number) matches as a valid citation — directly contradicting the function's own documented intent that 'a bare number is deliberately not a citation.' This feeds upstreamFixStands()/alreadyFixedStands(), so a history agent returning a bare 7+ digit non-reference number in its reference field causes a real, live finding to be silently discarded as ALREADY_FIXED. The NOT_CITATIONS test table (static.test.mjs:362-364) only covers numbers under 7 digits, so add a 7+ digit case there and require at least one hex letter (a-f) in the sha branch (or exclude date/timestamp-shaped decimals).

Extended reasoning...

What the bug is. citedReference()'s first regex alternative for recognizing a commit-sha citation is (^|[^0-9a-z])[0-9a-f]{7,40}([^0-9a-z]|$). Because the decimal digits 0-9 are a strict subset of the hex character class [0-9a-f], this pattern matches any bare run of 7 to 40 plain digits — with zero hex letters required. Concretely: citedReference('4012345') -> '4012345', citedReference('20240115') (an 8-digit date) -> matches, citedReference('1700000000') (a 10-digit unix timestamp) -> matches, and a 40-digit string of all '1's also matches. This was verified directly against the regex extracted from the source by three independent verifiers.

Why it contradicts the code's own stated design. The function carries a comment (lines 765-769) that reads: "A BARE number is deliberately not a citation. 4521 is indistinguishable from a line number or a year, and admitting it makes 'fixed in 2021' a reference." The implementation only enforces that intent for numbers under 7 digits — citedReference('4521') and citedReference('2021') correctly return null because they are too short for the sha branch to match. But a longer bare number, which is exactly as indistinguishable from a line number, a date, or a byte count as a short one, slips through and is accepted as a valid commit reference.

The code path that triggers it. upstreamFixStands(historyVerdict) in triage-static.js calls citedReference(historyVerdict.reference) to decide whether Stage 1's already-fixed search retracts a finding as ALREADY_FIXED. The history agent is prompted to put "a commit, PR, issue or advisory ID" into reference. If the model instead writes a bare 7+ digit number — a copied byte count, a date written as YYYYMMDD, a large ticket number with no keyword, or any other big number that shows up in its reasoning about the fix — citedReference treats it as a valid sha, upstreamFixStands returns a truthy retraction, and decideGate returns ALREADY_FIXED. The identical regex is duplicated verbatim in triage-poc.js, where it drives alreadyFixedStands() for Stage 3's challenge-4 retraction. Both surrounding comments (lines 744-749 and 825-828 in triage-static.js) explicitly call out an unreferenced retraction as "the one failure mode that silently discards a real finding rather than merely reporting a false one" — which is exactly the failure mode this gap opens.

Why existing tests do not catch it. The NOT_CITATIONS table in tests/static.test.mjs (lines 362-364) is meant to cover "bare numbers and dates, indistinguishable from line numbers and years," but only tests '4521' and '2021.03', both under 7 digits. Nothing in the 146+ mutation-gate suite exercises a 7+ digit bare number, so this gap is invisible to the existing regression suite.

Step-by-step proof:

1. Extract the sha regex: (^|[^0-9a-z])[0-9a-f]{7,40}([^0-9a-z]|$).
2. Note [0-9a-f] matches every character in [0-9], so any decimal string of length 7-40 satisfies the quantified class.
3. Run citedReference('the fix landed on 20240115'): the substring 20240115 is 8 digits, surrounded by non-alphanumeric boundaries (a space before, end-of-string or space after) — it matches, and the function returns truthy.
4. This is a date, not a commit/PR/issue/advisory reference, yet upstreamFixStands reads a truthy return as "cites a valid reference."
5. decideGate therefore returns ALREADY_FIXED for a finding that has no actual upstream fix evidence — discarding a potentially live, exploitable finding based on a coincidental date string.

The fix. Require at least one hex letter (a-f) to appear in the matched run when there is no other qualifying signal (e.g., a #, PR, CVE-, GHSA- prefix elsewhere in the string), or explicitly exclude date/timestamp-shaped decimal runs (8-digit YYYYMMDD, 10-digit unix timestamps). Add a 7+ digit bare-number case to the NOT_CITATIONS table in tests/static.test.mjs so the mutation gate can catch a regression here.

On severity. All three independent verifiers who examined this converged on nit: it is a real, verified defect that contradicts the code's documented intent and escapes the test suite, but triggering it requires the history agent to write a bare, non-keyword 7+ digit number into reference when explicitly prompted for "a commit, PR, issue or advisory ID" — a plausible but not likely model output. There is also a genuine design trade-off noted by one verifier: some real git short-shas are all-decimal, so a blanket rejection of all-digit strings would introduce a different false-negative. This does not block the PR, but is worth the maintainer's attention given it sits on the single most safety-critical retraction path in the plugin.

[GrosQuildu]

Ignore CLA please, I committed from not existing email.