Feature/add scan for litllm

README.md

@@ -9,11 +9,11 @@
 [![Node](https://img.shields.io/badge/node-%3E%3D22.12-brightgreen)](https://nodejs.org)
 [![OWASP](https://img.shields.io/badge/OWASP-LLM%20%C2%B7%20ASI%20%C2%B7%20MCP%20Top%2010-000000)](#rules)
 
-**The AI security scanner that proves its findings.**
+**Offline CLI that scans TypeScript, JavaScript, and Python for LLM, MCP, Agent Skill, and RAG risks — import-resolved dataflow evidence, zero default false positives, mapped to OWASP LLM/ASI/MCP Top 10.**
 
-SecureAI-Scan finds LLM, MCP, Agent Skill, and RAG vulnerabilities in **TypeScript, JavaScript, and Python** — and shows you the evidence: the exact source → flow → sink path for every dataflow finding, resolved through real imports, not keyword matching.
+Most scanners in this space pattern-match a keyword and call it a finding. SecureAI-Scan traces the actual source → flow → sink path through real, import-resolved code — and a default scan shows you only what it can prove. No account, no cloud upload, nothing leaves your machine.
 
-It provides launch-week support for the official [OWASP Top 10 for LLM Applications 2026](https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/), alongside the [Top 10 for Agentic Applications (2026)](https://genai.owasp.org/) and the [MCP Top 10](https://owasp.org/www-project-mcp-top-10/). Every threat model distinguishes static coverage from runtime concerns.
+Covers the official [OWASP Top 10 for LLM Applications 2026](https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/), [Top 10 for Agentic Applications (2026)](https://genai.owasp.org/), and the [MCP Top 10](https://owasp.org/www-project-mcp-top-10/) from launch week.
 
 ## Get started in 30 seconds
 
@@ -38,6 +38,8 @@ No account, cloud upload, Python interpreter, or configuration required. TypeScr
 
 **Is this for you?** SecureAI-Scan is scoped deliberately to LLM, MCP, and RAG/agent risks — prompt injection, tool poisoning, unsafe output handling, vector-store access control, agent-skill poisoning. It is not a general SAST or secrets scanner, and doesn't try to be one; a known-malicious package with no LLM-shaped payload (e.g. a hardcoded exfiltration address in an email API call) is caught by the offline advisory list (`DEP003`), not a pattern rule. If your codebase talks to an LLM, an MCP server, a vector store, or ships Agent Skills, this is built for you.
 
+<sub>New: static config scanning for LiteLLM Proxy (`config.yaml`) — hardcoded secrets, plaintext provider endpoints, missing guardrails. See [Rules](#rules) (LLC001–LLC003).</sub>
+
 ## Contents
 
 - [Why this scanner is different](#why-this-scanner-is-different)
@@ -60,7 +62,7 @@ No account, cloud upload, Python interpreter, or configuration required. TypeScr
 
 - **Evidence tiers, not noise.** Every finding is `proven` (traced dataflow or parsed config fact), `likely` (resolved sink, one heuristic hop), or `heuristic`. **A default scan shows only proven + likely.** Heuristics are opt-in via `--paranoid`.
 - **Import-resolved detection.** A call is only an "LLM call" if it resolves to a real SDK import (`openai`, `@anthropic-ai/sdk`, `ai`, `@google/genai`, LangChain, Bedrock, …). Your Google Maps client will never be flagged as an LLM again.
-- **Precision-gated, and benchmarked against real repos.** The test suite asserts every vulnerable fixture fires *and* every safe fixture stays clean — a false positive on the safe corpus fails the build. Beyond that, `npm run regression` scans real public repos (OpenAI/Anthropic/Vercel AI SDKs, official MCP servers, LlamaIndex) against a committed, hand-reviewed baseline and **fails on any new `proven`/`likely` finding**. See [Testing & benchmarking](#testing--benchmarking) for the actual before/after numbers, or [What we found scanning real repos](docs/RealWorldFindings.md) for the story behind them — a 6/6 catch rate on a labeled malicious-skill corpus, and why we're *not* calling llama_index "vulnerable" over an honest library-level finding.
+- **Precision-gated, and benchmarked against real repos.** The test suite asserts every vulnerable fixture fires *and* every safe fixture stays clean — a false positive on the safe corpus fails the build. Beyond that, `npm run regression` scans real public repos (OpenAI/Anthropic/Vercel AI SDKs, official MCP servers, LlamaIndex) against a committed, hand-reviewed baseline and **fails on any new `proven`/`likely` finding**. See [Testing & benchmarking](#testing--benchmarking) for the actual before/after numbers, or [What we found scanning real repos](docs/RealWorldFindings.md) for the story behind them — a 6/6 catch rate on a labeled malicious-skill corpus, and why we're *not* calling llama_index "vulnerable" over an honest library-level finding. [Discussion write-up →](https://github.com/akanthed/SecureAI-Scan/discussions/19)
 - **SARIF for GitHub code scanning.** `--output report.sarif` puts findings inline on pull requests and in the Security tab.
 - **AI-BOM.** `secureai-scan bom .` builds a syntax-derived inventory of SDKs, model IDs, vector stores, agent frameworks, and MCP servers, mapped to OWASP LLM Top 10 / EU AI Act documentation needs.
 - **MCP config scanning.** Parses `.mcp.json`, `claude_desktop_config.json`, `.cursor/mcp.json`: unpinned `npx -y` servers, inline secrets, plaintext HTTP transports.
@@ -205,7 +207,7 @@ Scanning clean? Add the badge to your own README:
 
 ## Rules
 
-**39 rules**, mapped to the official OWASP Top 10 for LLM Applications (2026) — plus, where applicable, the OWASP Top 10 for Agentic Applications (2026, ASI), the OWASP MCP Top 10 (2025), and an EU AI Act article. See the [versioned 2026 coverage and limits](docs/OWASP2026.md); `threat-model` renders the matrix for each scanned project.
+**42 rules**, mapped to the official OWASP Top 10 for LLM Applications (2026) — plus, where applicable, the OWASP Top 10 for Agentic Applications (2026, ASI), the OWASP MCP Top 10 (2025), and an EU AI Act article. See the [versioned 2026 coverage and limits](docs/OWASP2026.md); `threat-model` renders the matrix for each scanned project.
 
 | Rule | What it proves | OWASP |
 |------|----------------|-------|
@@ -248,6 +250,9 @@ Scanning clean? Add the badge to your own README:
 | DEP001 | Dependency name not found in the registry (opt-in `--check-dependencies`) | LLM04 |
 | DEP002 | Dependency name one edit away from a popular package (opt-in) | LLM04 |
 | DEP003 | Dependency with a documented malicious release or critical CVE — checked offline on every scan, version-range aware (postmark-mcp, mcp-remote CVE-2025-6514, …) | LLM04 · MCP04 |
+| LLC001 | Hardcoded secret in a LiteLLM proxy `config.yaml` | LLM02 |
+| LLC002 | LiteLLM proxy `api_base` reachable over plaintext HTTP | LLM04 |
+| LLC003 | LiteLLM proxy config has no `guardrails:` section (heuristic, `--paranoid` only) | LLM03 |
 
 `secureai-scan explain <RULE_ID>` gives the exploit walkthrough and a before/after code example for any rule.
 

docs/RealWorldFindings.md

@@ -38,6 +38,26 @@ The same regression scan is also how we caught our own bugs. An earlier run agai
 
 All three were fixed at the root cause, not patched at the call site, and pinned as permanent fixtures so they can't regress silently. Full numbers, plus the same before/after treatment for `vercel/ai`, `openai-node`, `anthropic-sdk-typescript`, and `modelcontextprotocol/typescript-sdk`, are in the [Testing & benchmarking](../README.md#testing--benchmarking) section of the README.
 
+## A live example: adding LiteLLM to the regression set found three bugs before it found one real issue
+
+When we added static `config.yaml` scanning for LiteLLM Proxy (`LLC001`–`LLC003`: hardcoded secrets, plaintext provider endpoints, missing guardrails), none of the repos already in the regression set exercise those rules — none of them ship a LiteLLM proxy config. So we added [BerriAI/litellm](https://github.com/BerriAI/litellm) itself, the official repo, specifically to get real coverage. It's a large, real production monorepo (6,978 TS/JS files, thousands more in Python) — a genuinely harder target than our own fixtures.
+
+First pass surfaced two new false-positive classes and one unrelated but serious robustness bug, in that order:
+
+1. **Line misattribution.** `LLC001` correctly detected that *some* entry in a `model_list` had a hardcoded secret, but anchored the finding to the first line in the file containing the string `api_key` — not the line that actually held the offending value. In a config with dozens of `api_key:` entries, that meant a reported finding could point straight at an `os.environ/...` reference and contradict its own evidence. Fixed by anchoring on the flagged *value* instead of the key name (unique per credential, unlike the key).
+2. **Placeholder-value false positives.** LiteLLM's own docs and tests inline dummy values like `fake-key`, `my-fake-key`, `sk-lar1-demo` to demonstrate config shape — not real secrets. `LLC001` initially flagged all of them. Fixed with a placeholder-word check plus a "does this look like a random credential blob or a human-typed phrase" heuristic (longest unbroken alphanumeric run < 12 chars ⇒ not credential-shaped) rather than guessing at a denylist of exact strings.
+3. **A rule crash was silently killing the entire scan.** Unrelated to LiteLLM's config files — a ts-morph type-checker failure on one file in the repo's Next.js admin dashboard (a large monorepo with multiple independent `tsconfig.json` files merged into one project) took down the *whole* scan, discarding every rule's findings, not just the one that crashed. This is worse than any false positive: a scan that silently reports nothing instead of erroring looks identical to "clean." Fixed by isolating each rule's `run()` — one rule failing now logs a warning and the rest of the scan continues.
+
+Two more, smaller false positives surfaced once the scan could actually complete against the full repo:
+
+4. `MCP001` (Python) matched the phrase "system prompt" inside an *admin-UI settings description* — plain English describing an unrelated caching feature, in a module-level dict inside a 17,000-line file. The scoping guard meant to require real MCP-listing context fell back to the entire file when a match wasn't inside a function, so "the file mentions MCP somewhere" (true of nearly any file that size in this codebase) satisfied it. Capped the fallback to a small line window instead of the whole module.
+5. `MCP002` (TypeScript) flagged a pure URL-parsing utility (`extractMCPToken(url: string)`) as "MCP server URL from user input" for no reason other than "url" being the name of one of its own parameters — a blanket rule that treated *every* function parameter as request-tainted regardless of whether the function had anything to do with handling a request. The known-vulnerable fixture never needed this: it matches `req.body.serverUrl` directly. Removed the blanket taint.
+6. `VEC001` matched Python's stdlib `re.search(r"/vector_stores/([^/]+)/", path)` — ordinary URL-path parsing — as a vector-store similarity search, purely because the regex *pattern string* contained the substring "vector" and the call syntactically looked like `.search(...vector...)`. Added an exclusion for `re.search`/`regex.search`.
+
+After all six fixes: **zero LLC001/LLC002 false positives, one confirmed-real `LLC002` finding** (a proxy config routing to an internal `vllm-command` host over plain `http://`, in `litellm/proxy/_super_secret_config.yaml`), and the rest of the rule set continuing to run clean against the same repo. Every fix shipped with a permanent fixture under `test-fixtures/safe/`, named for the pattern, so none of these six can regress silently.
+
+This is what "zero tolerance for false positives" costs in practice: not zero bugs, but a standing habit of reading every new finding against its source line before trusting it, on code we didn't write.
+
 ## Run it yourself
 
 ```bash

docs/index.html

@@ -506,7 +506,7 @@ <h3>Known-bad packages</h3>
   </div>
 
   <footer>
-    <p>For your own repository, run the full scanner: <code class="cli">npx secureai-scan@0.9.0 scan .</code> — traces prompt injection, RAG poisoning, and agent-privilege issues across your codebase, not just what you paste here.</p>
+    <p>For your own repository, run the full scanner: <code class="cli">npx secureai-scan@0.9.0 scan .</code> — traces prompt injection, RAG poisoning, and agent-privilege issues across your codebase, not just what you paste here. Read more: <a href="mcp-tool-poisoning.html">how to detect MCP tool poisoning</a> · <a href="prompt-injection-detection.html">how to detect prompt injection</a>.</p>
     <a href="https://github.com/akanthed/SecureAI-Scan" target="_blank" rel="noopener">SecureAI-Scan on GitHub</a>
   </footer>
 

docs/llms.txt

@@ -9,6 +9,10 @@ The TypeScript/JavaScript engine uses ts-morph AST analysis. The Python engine u
 Project: https://github.com/akanthed/SecureAI-Scan
 Package: https://www.npmjs.com/package/secureai-scan
 Browser MCP scanner: https://akanthed.github.io/SecureAI-Scan/
+How to detect MCP tool poisoning: https://akanthed.github.io/SecureAI-Scan/mcp-tool-poisoning.html
+How to detect prompt injection in code: https://akanthed.github.io/SecureAI-Scan/prompt-injection-detection.html
+Real-world findings (regression scan against public repos): https://github.com/akanthed/SecureAI-Scan/blob/main/docs/RealWorldFindings.md
+FAQ: https://github.com/akanthed/SecureAI-Scan/blob/main/docs/FAQ.md
 Release assurance: https://github.com/akanthed/SecureAI-Scan/blob/main/docs/ReleaseAssurance.md
 Latest benchmark: https://github.com/akanthed/SecureAI-Scan/blob/main/docs/benchmarks/v0.9.0.json
 Security policy: https://github.com/akanthed/SecureAI-Scan/blob/main/SECURITY.md