mcp-cve-bench

A ground-truth benchmark for MCP and agent-stack security scanners, built from real, patched, publicly disclosed CVEs rather than synthetic attack traffic.

Status: pre-alpha. There are no results here yet, and there will be no detection-rate claims until the reproduction work below is done.

The gap this is for

MCP security scanning has roughly 776 public repositories and is backed by NVIDIA, Tencent, Cisco, Snyk, Uber and Stacklok. Practitioner surveys of that market concede the same thing: no MCP scanner offers a verified, independently audited detection rate. Every tool in the category reports a capability no third party can check.

Meanwhile the CVE corpus keeps growing on its own. All figures below are reproducible with scripts/pull_nvd.py; the recorded pull was 2026-09-23.

• 133 CVEs match "model context protocol" in NVD: 23 in 2025, 110 in 2026
• 24 at CVSS >= 9.0, 90 at >= 7.0
• the three largest classes are missing-or-unenforced authentication (30), credential and secret exposure (30) and transport hardening such as DNS rebinding, Host/Origin validation and 0.0.0.0 binds (28)
• 68 of 133 are observable from config, metadata or a transport probe alone -- no source code, and for most of them no container either
• all six official MCP language SDKs shipped a DNS-rebinding-by-default CVE
• CVE-2025-66401 (9.8) was command injection inside another project's MCPScanner class, which is why this repository treats scanner input as hostile by default

Neighbouring benchmarks are not answering this question. AgentDojo measures whether an agent falls for an attack. AgentDefense-Bench is a synthetic attack suite. Corvus and mcpscan are scanners scoring themselves. Nobody measures whether the scanners catch what actually shipped.

Method, stated before the results

The scoring rules are written down in docs/METHODOLOGY.md before any vendor is run against the corpus, and published alongside the numbers. Naming commercial products' failure rates invites dispute, so the only defence is that a critic can re-run every claim from a clean clone.

Two constraints are not negotiable:

1. Patched-and-disclosed only. Every fixture names the version that fixed it. Reproduction fixtures for unpatched software would double as a target list, and a measured 41.6% of live MCP servers vanish within three days -- the long tail is exactly where an unpatched corpus would do damage.
2. Every fixture is a vulnerable/patched pair, so a "detection" is scored against a stated expectation rather than judgement.

Layout

data/raw/       verbatim NVD responses, with the retrieval timestamp
data/triage/    per-CVE class labels and the config-visibility split
fixtures/       vulnerable/patched pairs, one directory per CVE
src/mcp_cve_bench/taxonomy.py   label vocabulary, with its provenance
tests/                          the executable spec for what counts as a fixture

Contributing

The corpus triage is the bottleneck and the most useful place to start: 38 of the 133 records still have no class label, and tests/test_corpus_triage_integrity.py fails until that is worked off. Read one CVE, label it, send a pull request. Label disputes are settled by the record, not by opinion.

License

Code is MIT. The corpus is CC-BY-4.0 so that results published against it can cite their ground truth.