From 487e0362a414c4f5250e59f8457f851b47f94956 Mon Sep 17 00:00:00 2001 From: Eric Sognefest Date: Wed, 29 Jul 2026 14:06:46 -0400 Subject: [PATCH] llm: align dependency review with CVSS v3.0 and dependency scope Specify CVSS v3.0 as the version used for Bitwarden's vulnerability reporting and triage in the reviewing-dependencies skill, and explain how to reach a v3.0 score from the v3.1 vectors GitHub publishes. Resolve dependency scope before fix availability when deciding on action. A build-time-only finding confirmed absent from every shipped artifact is declared Not Affected / Informational and tracked with the next routine parent-tooling upgrade, rather than being routed to "update immediately" by the fix-availability table. Fix defects found while testing the skill against live Dependabot data: - The alert query returned only the first page, silently dropping alerts beyond 100. Page order is not severity-ranked, so a critical finding could be dropped. - The severity sort was claimed in a comment but never performed, and with --paginate alone it would have sorted per page rather than globally. - cvss_v3.score reads 0 rather than null when an advisory carries no v3 vector, making high-severity findings appear to score zero. - Grype: --only-fixed filters by fix availability rather than severity, and --fail-on sets the exit code without filtering output. The table column is FIXED IN, the default set also includes EPSS and RISK, and FIXED IN is omitted entirely when no finding has a fix (verified against grype 0.116.1). Surface scope, manifest, vulnerable_range, and first_patched in the gathering query, since the assessment steps ask about them. --- .claude-plugin/marketplace.json | 2 +- .cspell.json | 1 + README.md | 2 +- .../.claude-plugin/plugin.json | 2 +- .../bitwarden-security-engineer/CHANGELOG.md | 21 +++++ .../skills/reviewing-dependencies/SKILL.md | 83 +++++++++++++++---- 6 files changed, 90 insertions(+), 21 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index feda5ee0..3b0effd5 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -54,7 +54,7 @@ { "name": "bitwarden-security-engineer", "source": "./plugins/bitwarden-security-engineer", - "version": "1.3.0", + "version": "1.3.1", "description": "Application security engineering assistant for vulnerability triage, threat modeling, and secure code analysis." }, { diff --git a/.cspell.json b/.cspell.json index cec7852e..1707f1d5 100644 --- a/.cspell.json +++ b/.cspell.json @@ -38,6 +38,7 @@ "ECDH", "ECIES", "editui", + "epss", "ESL", "evals", "excalidraw", diff --git a/README.md b/README.md index a53b17ee..4edf8209 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ A curated collection of plugins for AI-assisted development at Bitwarden. Enable | [bitwarden-devops-engineer](plugins/bitwarden-devops-engineer/) | 0.1.4 | DevOps engineering assistant: workflow compliance linting, action security auditing, and org-wide CI/CD remediation | | [bitwarden-init](plugins/bitwarden-init/) | 1.2.0 | Initialize and enhance CLAUDE.md files with Bitwarden's standardized template format | | [bitwarden-product-analyst](plugins/bitwarden-product-analyst/) | 0.1.5 | Product analyst agent for creating comprehensive Bitwarden requirements documents from multiple sources | -| [bitwarden-security-engineer](plugins/bitwarden-security-engineer/) | 1.3.0 | Application security engineering: vulnerability triage, threat modeling, and secure code analysis | +| [bitwarden-security-engineer](plugins/bitwarden-security-engineer/) | 1.3.1 | Application security engineering: vulnerability triage, threat modeling, and secure code analysis | | [bitwarden-software-engineer](plugins/bitwarden-software-engineer/) | 1.0.0 | Software engineer agent for a Bitwarden product team. Implements stories, tasks, and bugs with code quality, performance, security, and team comms in mind. | | [claude-config-validator](plugins/claude-config-validator/) | 1.1.1 | Validates Claude Code configuration files for security, structure, and quality | | [claude-retrospective](plugins/claude-retrospective/) | 1.1.1 | Analyze Claude Code sessions to identify successful patterns and improvement opportunities | diff --git a/plugins/bitwarden-security-engineer/.claude-plugin/plugin.json b/plugins/bitwarden-security-engineer/.claude-plugin/plugin.json index 9f31fc9d..541a3e00 100644 --- a/plugins/bitwarden-security-engineer/.claude-plugin/plugin.json +++ b/plugins/bitwarden-security-engineer/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "bitwarden-security-engineer", - "version": "1.3.0", + "version": "1.3.1", "description": "Application security engineering assistant for vulnerability triage, threat modeling, and secure code analysis at Bitwarden.", "author": { "name": "Bitwarden", diff --git a/plugins/bitwarden-security-engineer/CHANGELOG.md b/plugins/bitwarden-security-engineer/CHANGELOG.md index 03d20678..cb046b62 100644 --- a/plugins/bitwarden-security-engineer/CHANGELOG.md +++ b/plugins/bitwarden-security-engineer/CHANGELOG.md @@ -5,6 +5,27 @@ All notable changes to the `bitwarden-security-engineer` plugin will be document The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [1.3.1] - 2026-07-29 + +### Added + +- `reviewing-dependencies` skill: impact assessment now starts by asking whether the dependency ships or is build-time only, using the alert's `scope` field, and requires confirming absence from shipped artifacts before relying on it. +- `reviewing-dependencies` skill: the action decision now resolves dependency scope first — a `development` finding confirmed absent from every shipped artifact is declared Not Affected / Informational and tracked with the next routine parent-tooling upgrade, rather than being routed to "update immediately" by the fix-availability table. +- `reviewing-dependencies` skill: added a decision row distinguishing transitive findings whose fix is installable from those where a parent pins the vulnerable version, where overriding the pin risks breaking the parent. + +### Changed + +- `reviewing-dependencies` skill: the CVSS impact-assessment step now specifies CVSS v3.0 as the version used for Bitwarden's vulnerability reporting and triage, and explains how to get there from the v3.1 vectors GitHub publishes. +- `reviewing-dependencies` skill: the Dependabot alert-gathering query now emits the CVSS v3 vector needed for v3.0 scoring, plus `scope`, `manifest`, `vulnerable_range`, and `first_patched` — the fields the assessment steps ask about. + +### Fixed + +- `reviewing-dependencies` skill: the alert-gathering query returned only the first page of results, silently discarding alerts beyond 100. Page order is not severity-ranked, so a critical finding could be dropped. Now uses `--paginate --slurp` so the severity sort is global rather than per-page. +- `reviewing-dependencies` skill: the alert-gathering query claimed to sort by severity but performed no sort. +- `reviewing-dependencies` skill: guarded the CVSS v3 score read, which returns `0` rather than `null` for advisories carrying only a v4.0 vector, making high-severity alerts appear to score zero. +- `reviewing-dependencies` skill: corrected the Grype examples — `--only-fixed` filters by fix availability rather than severity, and `--fail-on` sets the exit code without filtering output. +- `reviewing-dependencies` skill: corrected the documented Grype table columns against grype 0.116.1 — the column is `FIXED IN`, the default set also includes `EPSS` and `RISK`, and `FIXED IN` is dropped entirely when no finding has a fix. Added the `fix.state` JSON field as the reliable alternative to parsing the table. + ## [1.3.0] - 2026-07-21 ### Added diff --git a/plugins/bitwarden-security-engineer/skills/reviewing-dependencies/SKILL.md b/plugins/bitwarden-security-engineer/skills/reviewing-dependencies/SKILL.md index d77b6b60..b6622610 100644 --- a/plugins/bitwarden-security-engineer/skills/reviewing-dependencies/SKILL.md +++ b/plugins/bitwarden-security-engineer/skills/reviewing-dependencies/SKILL.md @@ -8,8 +8,37 @@ description: This skill should be used when the user asks to "review Dependabot ### Step 1: Gather Alerts ```bash -# List all open Dependabot alerts sorted by severity -gh api /repos/{owner}/{repo}/dependabot/alerts --jq '.[] | select(.state == "open") | {number, severity: .security_vulnerability.severity, package: .security_vulnerability.package.name, ecosystem: .security_vulnerability.package.ecosystem, summary: .security_advisory.summary}' +# List ALL open Dependabot alerts, sorted by severity. Notes on why this is +# shaped the way it is — each of these is a trap worth avoiding: +# - --paginate is required: a single page caps at 100 and silently drops the +# rest. Page order is not severity-ranked, so a critical can be on page 2. +# - --slurp merges pages into one array so the sort is global, not per-page. +# gh rejects --slurp together with --jq, so pipe to jq instead. +# - The API cannot sort by severity (sort= takes only created, updated, +# epss_percentage), so sort in jq. +# - scope tells you development (build/CI only) vs runtime (ships to users). +# This usually decides the triage outcome — see Step 2 and Step 3. +# - cvss_v3.score returns 0 rather than null when an advisory carries no v3 +# vector, so it is guarded; unguarded it reads 0 on high-severity findings. +gh api --paginate --slurp "/repos/{owner}/{repo}/dependabot/alerts?state=open&per_page=100" \ +| jq ' + [ .[][] + | (.security_advisory.cvss_severities // {}) as $c + | { number, + severity: .security_vulnerability.severity, + package: .dependency.package.name, + ecosystem: .dependency.package.ecosystem, + scope: .dependency.scope, + manifest: .dependency.manifest_path, + vulnerable_range: .security_vulnerability.vulnerable_version_range, + first_patched: .security_vulnerability.first_patched_version.identifier, + cvss_v3_vector: $c.cvss_v3.vector_string, + cvss_v3_score: (if $c.cvss_v3.vector_string then $c.cvss_v3.score else null end), + cvss_v4_vector: $c.cvss_v4.vector_string, + summary: .security_advisory.summary } + ] + | sort_by({ critical: 0, high: 1, medium: 2, low: 3 }[.severity]) + | .[]' # Filter by severity gh api "/repos/{owner}/{repo}/dependabot/alerts?severity=critical&state=open" @@ -22,20 +51,34 @@ gh api /repos/{owner}/{repo}/dependabot/alerts/{alert_number} For each alert, determine: -1. **Is the vulnerable code path reachable?** — Does the application actually use the vulnerable function/feature of the dependency? -2. **Is it a direct or transitive dependency?** — Transitive vulnerabilities may be harder to fix but still pose real risk. -3. **What is the CVSS score and exploit availability?** — A high CVSS with a public exploit needs immediate action. A medium CVSS with no known exploit can be scheduled. -4. **What versions are affected and what versions fix it?** — Check if updating is a minor bump or a breaking change. +1. **Does the dependency ship, or is it build-time only?** — Start here; it usually decides the outcome. `scope` on the alert reports `development` or `runtime`. Treat `development` as a strong signal, not proof: confirm the package is absent from every shipped artifact (extension, desktop, web vault, CLI, server image) before relying on it, because some build-time packages inline code into published output. A build-time-only package has no live process in production for an attacker to reach. +2. **Is the vulnerable code path reachable?** — Does the application actually use the vulnerable function/feature of the dependency? +3. **Is it a direct or transitive dependency?** — Transitive vulnerabilities may be harder to fix but still pose real risk. Identify the parent that pins it (`npm ls `), because when a parent pins a vulnerable version the fix may not be available to you directly. +4. **What is the CVSS score and exploit availability?** — Score in **CVSS v3.0**, Bitwarden's standard for reporting and triage of vulnerabilities. GitHub publishes v3.1 vectors for recent advisories (v3.0 appears only on older imported CVEs), so transcribe `cvss_severities.cvss_v3.vector_string` into a v3.0 calculator — v3.0 and v3.1 share the same base metrics, so the vector maps directly. Some advisories carry a v4.0 vector only, or no vector at all; v4.0 metrics do not map onto v3.0, so score those by hand from the advisory details rather than reusing the v4.0 number. A high CVSS with a public exploit needs immediate action. A medium CVSS with no known exploit can be scheduled. +5. **What versions are affected and what versions fix it?** — `vulnerable_range` and `first_patched` on the alert give both. Check whether reaching `first_patched` is a minor bump, a breaking change, or blocked behind a parent package that pins the vulnerable version. ### Step 3: Decide on Action -| Situation | Action | -| ----------------------------------------- | ---------------------------------------------- | -| Fix available, minor version bump | Update immediately | -| Fix available, major version bump | Evaluate breaking changes, schedule update | -| No fix available, code path reachable | Implement workaround or replace dependency | -| No fix available, code path not reachable | Document and monitor, set review date | -| Vulnerability in transitive dependency | Use overrides/resolutions to pin fixed version | +Resolve scope first. A package that never reaches a shipped artifact cannot be attacked in production, and treating it as though it can produces pointless urgency and risky churn. + +| Scope | Action | +| ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `development`, confirmed absent from every shipped artifact | Declare **Not Affected / Informational**. Record the reachability reasoning explicitly, then track the bump with the next routine upgrade of the parent tooling rather than as a hotfix | +| `development`, but the package or its output lands in a shipped artifact | Treat as `runtime` and use the table below | +| `runtime` | Use the table below | + +A `development` finding still needs the declaration written down. "It's a devDependency" is not by itself an assessment — state which artifacts were checked and why the code path cannot be reached in production. + +For `runtime` findings, and for `development` findings that turned out to ship: + +| Situation | Action | +| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Fix available, minor version bump | Update immediately | +| Fix available, major version bump | Evaluate breaking changes, schedule update | +| No fix available, code path reachable | Implement workaround or replace dependency | +| No fix available, code path not reachable | Document and monitor, set review date | +| Transitive, and the fixed version is installable | Use overrides/resolutions to pin the fixed version | +| Transitive, but a parent pins the vulnerable version | Overriding the parent's pin can break it. Prefer waiting for a parent release that bumps its own pin, and track it; override only when the finding is runtime-reachable and the risk justifies it | ## Transitive Dependency Risk @@ -90,15 +133,19 @@ grype dir:/path/to/project # Output as JSON for programmatic processing grype -o json -# Filter by severity -grype --only-fixed --fail-on high +# Show only vulnerabilities that have a fix available +grype --only-fixed + +# Exit non-zero (code 2) if anything at or above the given severity is found. +# This sets the exit code for CI gating; it does not filter the output. +grype --fail-on high ``` **Interpreting Grype output:** -- Each finding includes: CVE ID, severity, package name, installed version, fixed version -- `Fixed` column indicates whether an update is available -- Use `--only-fixed` to focus on actionable items (vulnerabilities with available fixes) +- Table columns (grype 0.116.x) are `NAME`, `INSTALLED`, `FIXED IN`, `TYPE`, `VULNERABILITY`, `SEVERITY`, `EPSS`, `RISK` +- The column set is dynamic: `FIXED IN` is omitted entirely when no finding has a fix available. When present, a blank cell means that particular vulnerability has no fix +- Use `--only-fixed` to focus on actionable items (vulnerabilities with available fixes). For scripting, read `.matches[].vulnerability.fix.state` from `-o json` (`fixed`, `not-fixed`, `wont-fix`, `unknown`) rather than parsing the table ## Platform-Specific Guidance