quietbench

We-run benchmarks of security tools on fixed fixtures, scored on noise, speed, flake, and actionability.

v0: secrets + SCA + SAST · Python + JS/TS · tools: gitleaks, trufflehog, osv-scanner, trivy, semgrep, bandit.

Results

Draft table: RESULTS.md. Scoring is still heuristic (no fingerprint-matched P/R yet). Flake not measured.

Methodology

See METHODOLOGY.md for scope, metrics, process, and pin/integrity rules.

Planted CVE pins

Fixtures intentionally pin known-vulnerable dependency versions so SCA tools have labeled ground-truth issues to find. That is test data for the benchmark, not a claim about production apps.

Fixture	Pin	CVE
py-app	PyYAML==5.1	CVE-2019-20477
js-app	lodash@4.17.15	CVE-2019-10744

Secrets in fixtures are synthetic (fake Stripe shapes). Do not use real tokens.

Layout

fixtures/py-app/     planted Python issues (secret, SCA, SAST)
fixtures/js-app/     planted JS issues (secret, SCA, SAST)
ground-truth/        schema.md + labels.yml
runners/             invoke tools → results/*.json (stub if binary missing)
results/             per-run JSON (gitignored regenerable outputs; .gitkeep kept)
scripts/             install-tools.ps1, gen_results_md.py
tools.yml            locked tool slate + versions + checksums

Result file scheme

results/<tool>-<fixture>.json — e.g. results/gitleaks-py-app.json.

Shared shape:

{
  "tool": "gitleaks",
  "fixture": "fixtures/py-app",
  "status": "ok|stub|error",
  "started_at": "…",
  "duration_ms": 0,
  "findings": [],
  "raw_note": "optional",
  "pinned": {
    "version": "8.30.1",
    "source": "winget:Gitleaks.Gitleaks",
    "sha256": "…",
    "from": "tools.yml"
  }
}

pinned always comes from tools.yml, not from whatever binary happens to be on PATH. Missing binaries produce status: "stub" with findings: [] — the run does not abort.

Reproduce

Pins live in tools.yml. Install (idempotent, refuses latest):

# from the quietbench repo root
powershell -ExecutionPolicy Bypass -File scripts\install-tools.ps1

The install script puts tool binaries on your user PATH (including %USERPROFILE%\bin and the Python --user Scripts directory when needed). Freshness: manual until first public table; then weekly pin review. Details in METHODOLOGY.md.

Run the suite:

# from the quietbench repo root; ensure tool binaries are on PATH
python runners\run.py --fixture both --tool all
python scripts\gen_results_md.py

Use a real Python 3.12+ install (python on PATH). On Windows Store alias installs, point at the full python.exe under %LOCALAPPDATA%\Programs\Python\ instead.