From ac451434db025edda60d780835993edd81cb526c Mon Sep 17 00:00:00 2001 From: Synvoya Date: Fri, 17 Jul 2026 23:23:17 +1000 Subject: [PATCH] docs: add reproducible scan proof --- README.md | 31 ++++++----- examples/reports/ai-chatbot-rag.md | 55 -------------------- examples/reports/nextjs-supabase.md | 59 --------------------- examples/reports/node-react.md | 56 -------------------- examples/reports/vulnerable-app-v0.3.1.md | 63 +++++++++++++++++++++++ 5 files changed, 82 insertions(+), 182 deletions(-) delete mode 100644 examples/reports/ai-chatbot-rag.md delete mode 100644 examples/reports/nextjs-supabase.md delete mode 100644 examples/reports/node-react.md create mode 100644 examples/reports/vulnerable-app-v0.3.1.md diff --git a/README.md b/README.md index 1b8f69d..7febf58 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,7 @@ ![Local-first](https://img.shields.io/badge/local--first-yes-brightgreen.svg) ![No telemetry](https://img.shields.io/badge/telemetry-none-brightgreen.svg) [![codeinspectus MCP server](https://glama.ai/mcp/servers/Synvoya/codeinspectus/badges/score.svg)](https://glama.ai/mcp/servers/Synvoya/codeinspectus) +[![GitHub stars](https://img.shields.io/github/stars/Synvoya/codeinspectus?style=social)](https://github.com/Synvoya/codeinspectus) **A local-first, privacy-preserving security MCP server.** Any AI coding agent (Claude Code, Cursor, Codex, Windsurf, Cline, Aider) can invoke CodeInspectus to @@ -16,6 +17,16 @@ egress at scan time**. ![CodeInspectus demo](assets/codeinspectus-demo.gif) +**Reproduce the proof:** the shipped intentionally vulnerable fixture produces +**18 normalized findings across all four engines** with v0.3.1 (4 critical, 8 high, +5 medium, 1 low). Inspect the [fixture](fixtures/vulnerable-app), read the +[full scanner-derived report](examples/reports/vulnerable-app-v0.3.1.md), or run the +[17-case eval suite](evals/run-evals.ts). Dependency findings can change as the local +Trivy database updates; the report records the exact engine and database versions used. + +If CodeInspectus is useful, [star the repository](https://github.com/Synvoya/codeinspectus) +so other AI-app builders can find it. + CodeInspectus orchestrates three best-in-class OSS engines behind one normalized, CWE-keyed schema, and adds its own **AI-code-specific checks** that generic scanners miss: @@ -223,9 +234,7 @@ ALL LOCAL. NO NETWORK EGRESS AT SCAN TIME. ## Example reports -- [Next.js + Supabase SaaS app](examples/reports/nextjs-supabase.md) -- [AI chatbot / RAG app](examples/reports/ai-chatbot-rag.md) -- [Node/React app](examples/reports/node-react.md) +- [Reproducible v0.3.1 scan of the shipped vulnerable fixture](examples/reports/vulnerable-app-v0.3.1.md) ## Trademark @@ -282,15 +291,13 @@ practitioner)* ## Good first contributions -- Add a fixture for the unsafe raw inner-HTML sink (`ci-ai-llm-output-dangerous-html`) from component props. -- Improve detection for Supabase `user_metadata.role` authorization checks. -- Add detection for exposed `NEXT_PUBLIC_OPENAI_API_KEY` and similar client-side AI keys. -- Add a rule for user-controlled URLs passed into server-side `fetch()`. -- Add a rule for model output passed into `eval`, `Function`, shell commands, or unsafe tool calls. -- Add a check for missing auth guards in Next.js `/api/admin/*` routes. -- Verify one CWE to OWASP Top 10 mapping. -- Verify one CWE to SOC 2 / ISO 27001 mapping. -- Add a vulnerable fixture and expected finding snapshot for an existing rule. +- [Trace tainted component props into `dangerouslySetInnerHTML`](https://github.com/Synvoya/codeinspectus/issues/1) +- [Detect model output passed to dynamic execution or shell sinks](https://github.com/Synvoya/codeinspectus/issues/2) +- [Detect unguarded Next.js admin API routes](https://github.com/Synvoya/codeinspectus/issues/3) +- [Community-verify one CWE to OWASP Top 10 mapping](https://github.com/Synvoya/codeinspectus/issues/4) +- [Community-verify one CWE to SOC 2 or ISO 27001](https://github.com/Synvoya/codeinspectus/issues/5) + +[Browse all good first issues](https://github.com/Synvoya/codeinspectus/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22). ## Changelog diff --git a/examples/reports/ai-chatbot-rag.md b/examples/reports/ai-chatbot-rag.md deleted file mode 100644 index 13baf7a..0000000 --- a/examples/reports/ai-chatbot-rag.md +++ /dev/null @@ -1,55 +0,0 @@ -# Example report — AI chatbot / RAG app - -> **This is an illustrative example/demo report, not a real audit.** The findings, -> paths, and values below are synthetic and exist only to show the *shape* of -> CodeInspectus output. Run `codeinspectus_scan` on your own repo for real results. -> Prompt-injection detection is heuristic and immature — such findings are worded -> "potential …" and marked medium confidence. - -**Target:** `example-chatbot/` (synthetic) · **Profile:** Next.js + OpenAI + vector store -**Scan:** local · zero network egress at scan time - -## Summary - -| Severity | Count | -|----------|-------| -| High | 2 | -| Medium | 2 | - -## Findings (CWE-keyed) - -### 1. Model output rendered as raw HTML — **high** -- **Rule:** `ci-ai-llm-output-dangerous-html` · **CWE-79/116** · OWASP **LLM05** · confidence: medium -- **Where:** `components/ChatMessage.tsx:18` — model output passed to React's raw inner-HTML prop -- **What:** LLM/model output is rendered as raw HTML without sanitization — a direct XSS sink. -- **Fix:** Wrap the value in `DOMPurify.sanitize(...)`, or render as text/markdown with a safe renderer. - -### 2. Client-exposed AI provider key — **high** -- **Rule:** `ci-ai-client-secret-exposure` · **CWE-522** · confidence: high -- **Where:** `.env.local` — `NEXT_PUBLIC_OPENAI_API_KEY=` -- **What:** The `NEXT_PUBLIC_` prefix ships this key to the browser, where anyone can read and abuse it. -- **Fix:** Remove the prefix; call the provider from a server route and keep the key server-side. - -### 3. Potential prompt-injection sink into a tool call — **medium** -- **Rule:** `ci-ai-prompt-injection-sink` · **CWE-77** · confidence: medium (heuristic) -- **Where:** `app/api/agent/route.ts:41` -- **What:** Retrieved document text is concatenated into a system prompt that can trigger tool - execution, with no delimiter/whitelist between untrusted context and instructions. -- **Fix:** Isolate untrusted context from instructions; constrain tools; validate tool arguments. - -### 4. Model output passed into a dynamic code evaluator — **medium** -- **Rule:** `security-baseline/no-model-output-eval` · **CWE-94** · confidence: medium -- **Where:** `lib/formula.ts:12` — model reply passed into a dynamic code-generation constructor -- **What:** Model-generated text is compiled and executed at runtime — arbitrary code execution risk. -- **Fix:** Never execute model output. Parse to a constrained AST or use a sandboxed interpreter. - -## Compliance (code-visible subset — not certification) - -These findings touch code-visible controls under OWASP LLM Top 10 (2025) and OWASP Top 10 -(2021) A03. CodeInspectus reports **code-level control coverage only** — never a -"% compliant" verdict. See [`docs/COMPLIANCE-RATIONALE.md`](../../docs/COMPLIANCE-RATIONALE.md). - -## Next step - -Apply fixes, then run `codeinspectus_rescan` to diff against this scan -(resolved / remaining / introduced). diff --git a/examples/reports/nextjs-supabase.md b/examples/reports/nextjs-supabase.md deleted file mode 100644 index 1c4afe6..0000000 --- a/examples/reports/nextjs-supabase.md +++ /dev/null @@ -1,59 +0,0 @@ -# Example report — Next.js + Supabase SaaS app - -> **This is an illustrative example/demo report, not a real audit.** The findings, -> paths, and values below are synthetic and exist only to show the *shape* of -> CodeInspectus output. Run `codeinspectus_scan` on your own repo for real results. -> Secret values are always redacted in real output. - -**Target:** `example-saas/` (synthetic) · **Profile:** Next.js (App Router) + Supabase -**Scan:** local · zero network egress at scan time - -## Summary - -| Severity | Count | -|----------|-------| -| Critical | 1 | -| High | 2 | -| Medium | 1 | - -## Findings (CWE-keyed) - -### 1. `service_role` key reachable from client code — **critical** -- **Rule:** `ci-ai-client-secret-exposure` · **CWE-798** · confidence: high -- **Where:** `lib/supabaseAdmin.ts:4` -- **What:** Supabase `service_role` key value present in a module imported by client - components. This key bypasses Row Level Security. -- **Fix:** Move all `service_role` usage to server-only code (Route Handlers / Server - Actions); never import it into a client bundle. Rotate the leaked key. - -### 2. Authorization decision trusts client-writable `user_metadata` — **high** -- **Rule:** `ci-ai-client-metadata-authz` · **CWE-639** · confidence: medium -- **Where:** `middleware.ts:22` — `if (user.user_metadata.role === 'admin')` -- **What:** `user_metadata` is editable by the signed-in user via Supabase's - `/auth/v1/user` endpoint, so anyone can self-assign `role: 'admin'`. -- **Fix:** Gate privileged logic on server-controlled `app_metadata.role` instead. - -### 3. `service_role` key behind a client-exposed env prefix — **high** -- **Rule:** `ci-ai-client-secret-exposure` · **CWE-522** · confidence: high -- **Where:** `.env.local` — `NEXT_PUBLIC_SUPABASE_SERVICE_ROLE_KEY=` -- **What:** The `NEXT_PUBLIC_` prefix inlines this value into the browser bundle. -- **Fix:** Remove the prefix; keep the service key server-side only. - -### 4. Supabase RLS not enabled on a user-data table — **medium** -- **Rule:** `ci-ai-supabase-rls` · **CWE-284** · confidence: medium -- **Where:** `supabase/migrations/0002_profiles.sql:1` -- **What:** `profiles` table has no `enable row level security` — reads/writes are - unrestricted with the anon key. -- **Fix:** `alter table profiles enable row level security;` and add explicit policies. - -## Compliance (code-visible subset — not certification) - -Findings above touch code-visible controls under OWASP Top 10 (2021) A01/A05/A07 and -map to related CWE→control entries for SOC 2 and ISO/IEC 27001:2022. CodeInspectus -reports **code-level control coverage only** — never a "% compliant" or pass/fail -verdict. See [`docs/COMPLIANCE-RATIONALE.md`](../../docs/COMPLIANCE-RATIONALE.md). - -## Next step - -Apply fixes, then run `codeinspectus_rescan` to diff against this scan -(resolved / remaining / introduced). diff --git a/examples/reports/node-react.md b/examples/reports/node-react.md deleted file mode 100644 index 47b4765..0000000 --- a/examples/reports/node-react.md +++ /dev/null @@ -1,56 +0,0 @@ -# Example report — Node/React app - -> **This is an illustrative example/demo report, not a real audit.** The findings, -> paths, and values below are synthetic and exist only to show the *shape* of -> CodeInspectus output. Run `codeinspectus_scan` on your own repo for real results. -> Secret values are always redacted in real output. - -**Target:** `example-app/` (synthetic) · **Profile:** Node/Express API + React SPA -**Scan:** local · zero network egress at scan time - -## Summary - -| Severity | Count | -|----------|-------| -| High | 2 | -| Medium | 1 | -| Low | 1 | - -## Findings (CWE-keyed) - -### 1. Untrusted value rendered as raw HTML — **high** -- **Rule:** `ci-ai-llm-output-dangerous-html` · **CWE-79** · confidence: medium -- **Where:** `src/components/Bio.jsx:27` — a component prop rendered via React's raw inner-HTML prop -- **What:** User-supplied `bio` text is rendered as raw HTML without sanitization — an XSS sink. -- **Fix:** Sanitize with `DOMPurify.sanitize(...)` before rendering, or render as plain text. - -### 2. Server-side request to a user-controlled URL — **high** -- **Rule:** `security-baseline/ssrf-user-url` · **CWE-918** · confidence: medium -- **Where:** `server/routes/proxy.js:14` — `fetch(req.query.url)` -- **What:** A request parameter is passed directly into a server-side `fetch()` — SSRF: an attacker - can reach internal services or cloud metadata endpoints. -- **Fix:** Allow-list destinations; reject private/link-local address ranges; never fetch raw user input. - -### 3. Hard-coded credential in source — **medium** -- **Rule:** Gitleaks · **CWE-798** · confidence: high -- **Where:** `server/db.js:3` — `const dbPassword = ""` -- **What:** A database password is committed in source. Secret value is redacted in output. -- **Fix:** Move to an environment variable / secret manager; rotate the exposed credential. - -### 4. Vulnerable dependency — **low** -- **Rule:** Trivy (SCA) · **CVE-EXAMPLE-0000** · confidence: high -- **Where:** `package-lock.json` — `example-lib@1.2.3` -- **What:** A transitive dependency has a known advisory with a fixed version available. -- **Fix:** Upgrade to the patched release; re-run the scan to confirm resolution. - -## Compliance (code-visible subset — not certification) - -These findings touch code-visible controls under OWASP Top 10 (2021) A03/A06/A10 and map to -related CWE→control entries for SOC 2 and ISO/IEC 27001:2022. CodeInspectus reports -**code-level control coverage only** — never a "% compliant" verdict. -See [`docs/COMPLIANCE-RATIONALE.md`](../../docs/COMPLIANCE-RATIONALE.md). - -## Next step - -Apply fixes, then run `codeinspectus_rescan` to diff against this scan -(resolved / remaining / introduced). diff --git a/examples/reports/vulnerable-app-v0.3.1.md b/examples/reports/vulnerable-app-v0.3.1.md new file mode 100644 index 0000000..1dc6550 --- /dev/null +++ b/examples/reports/vulnerable-app-v0.3.1.md @@ -0,0 +1,63 @@ +# Reproducible v0.3.1 fixture scan + +This is a scanner-derived report from CodeInspectus v0.3.1 running against the +repository's intentionally vulnerable [`fixtures/vulnerable-app`](../../fixtures/vulnerable-app) +on 17 July 2026. It replaces illustrative reports whose findings could not be +reproduced from a committed fixture. + +## Reproduce it + +```bash +npm install +npm run build +npx tsx scripts/dev-scan.ts fixtures/vulnerable-app +``` + +The engine binaries must already be installed and verified. See the main README for +`install-engines` and `verify-engines`. + +## Recorded environment + +| Component | Version or value | +| --- | --- | +| CodeInspectus package | 0.3.1 | +| Opengrep | 1.23.0 | +| Gitleaks | 8.30.1 | +| Trivy | 0.71.2 | +| CodeInspectus AI checks | 1.0.0 | +| Trivy database | 2026-07-15T23:36:30.755301Z | + +## Result + +CodeInspectus normalized 21 raw engine results into 18 findings: **4 critical, +8 high, 5 medium, and 1 low**. The lower normalized count is expected because the +same hard-coded client secret was detected by Gitleaks, Trivy, and the CodeInspectus +AI checks, then deduplicated into one finding with all three engines retained as evidence. + +| Severity | Rule | Location | Engine evidence | +| --- | --- | --- | --- | +| Critical | `CVE-2019-10744` | `package-lock.json:15` | Trivy | +| Critical | `CVE-2021-44906` | `package-lock.json:20` | Trivy | +| Critical | `ci-ai-client-hardcoded-secret` | `src/config.ts:5` | Gitleaks + Trivy + CodeInspectus AI | +| Critical | `ci-ai-rls-using-true` | `supabase/migrations/0001_init.sql:18` | CodeInspectus AI | +| High | `CVE-2018-16487` | `package-lock.json:15` | Trivy | +| High | `CVE-2020-8203` | `package-lock.json:15` | Trivy | +| High | `CVE-2021-23337` | `package-lock.json:15` | Trivy | +| High | `CVE-2026-4800` | `package-lock.json:15` | Trivy | +| High | `ci-ai-public-env-secret` | `src/components/PaymentForm.tsx:6` | CodeInspectus AI | +| High | `ci-baseline-sql-injection-string-build` | `src/db.ts:11` | Opengrep | +| High | `ci-ai-prompt-injection-sink` | `src/llm.ts:11` | CodeInspectus AI | +| High | `ci-ai-rls-missing` | `supabase/migrations/0001_init.sql:22` | CodeInspectus AI | +| Medium | `CVE-2019-1010266` | `package-lock.json:15` | Trivy | +| Medium | `CVE-2020-28500` | `package-lock.json:15` | Trivy | +| Medium | `CVE-2025-13465` | `package-lock.json:15` | Trivy | +| Medium | `CVE-2026-2950` | `package-lock.json:15` | Trivy | +| Medium | `CVE-2020-7598` | `package-lock.json:20` | Trivy | +| Low | `CVE-2018-3721` | `package-lock.json:15` | Trivy | + +## What is stable and what can drift + +The committed fixture, CodeInspectus rule IDs, file locations, and engine versions above +make this run auditable. CVE findings are database-dependent: a newer local Trivy database +may add, remove, or reclassify dependency findings. That is expected and should not be +hidden behind a permanently frozen marketing number.