From f2a8c9b46116dc95c86760c4c03c2900fd88176b Mon Sep 17 00:00:00 2001 From: Ahmed ElMallah Date: Thu, 30 Jul 2026 01:31:36 -0700 Subject: [PATCH 1/5] docs: correct how diff --fail-on gating is described MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Writing the tutorial exposed that three pages claimed --fail-on gates only on introduced findings. The implementation is deliberate and different (internal/cli/diff_cmd.go: auditBlockingFindings): the diff audit inspects only packages the change touched — that scoping, not a persisted-bucket exemption, is what keeps pre-existing debt out of PR reviews — and within that scope introduced AND persisted findings both gate, because persisted means the changed package still ships a known issue at its new version. AUDITORS (via its generator), the diff command page, and the USE_CASES PR-gate recipe (whose sample-output conclusion asserted a passing run that actually exits 2) now describe the real semantics, verified against the binary. Co-Authored-By: Claude Fable 5 --- docs/AUDITORS.md | 2 +- docs/USE_CASES.md | 8 ++++---- docs/commands/diff.md | 2 +- internal/support/component_docs.go | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/AUDITORS.md b/docs/AUDITORS.md index cb62dc31..7f541d53 100644 --- a/docs/AUDITORS.md +++ b/docs/AUDITORS.md @@ -137,7 +137,7 @@ See [EXIT_CODES.md](EXIT_CODES.md) for the full table. - **Resolved** — present in base, absent in head - **Persisted** — present in both -Combine with `--fail-on` to fail PRs that introduce new high-severity findings without complaining about pre-existing ones: +The diff audit only inspects packages the change touched, so findings on unchanged packages never appear — that is how pre-existing debt stays out of PR reviews. Both remaining buckets gate: `--fail-on` fails on introduced findings and on persisted ones, because a persisted finding means the changed package still ships a known issue at its new version: ```bash bomly diff --base main --head HEAD --enrich --audit --fail-on high diff --git a/docs/USE_CASES.md b/docs/USE_CASES.md index 1fe20f03..41383ed5 100644 --- a/docs/USE_CASES.md +++ b/docs/USE_CASES.md @@ -4,13 +4,13 @@ Recipes for the jobs people actually use Bomly for. Each one is a goal, the comm ## Gate a pull request on new vulnerabilities -**Goal:** fail a PR when it *introduces* a high-severity vulnerability, without nagging about pre-existing ones. +**Goal:** fail a PR when its dependency changes carry a high-severity vulnerability, without nagging about debt the PR didn't touch. ```bash bomly diff --base main --head HEAD --enrich --audit --fail-on high ``` -`diff` classifies findings as introduced / resolved / persisted and only `--fail-on` matches against the introduced set, so a clean PR passes even on a repo with known debt. Exit code `2` means a new high finding; see [Exit codes](EXIT_CODES.md). +`diff` audits only the packages the change touched and classifies their findings as introduced / resolved / persisted, so untouched debt elsewhere in the repo never blocks a PR. Introduced and persisted findings both gate — persisted means the changed package still ships a known issue at its new version. Exit code `2` means the change carries a gating finding; see [Exit codes](EXIT_CODES.md).
Sample output — an express upgrade that drags in one new advisory @@ -39,7 +39,7 @@ Version changed (9) Run again with --format json to see remediation details. ``` -With `--fail-on high`, this run passes: the only *introduced* finding is medium, and the persisted high is pre-existing debt. +With `--fail-on high`, this run exits `2`: the introduced finding is only medium, but the persisted high on `path-to-regexp@0.1.12` means the upgrade still ships a known high-severity issue. The gate holds until a bump reaches a fixed version — or the finding is accepted into a committed [baseline](BASELINES.md).
@@ -149,7 +149,7 @@ bomly diff --sbom --base old.spdx.json --head new.spdx.json bomly diff --image ghcr.io/example/app --base 1.4.0 --head 1.5.0 --enrich --audit --fail-on high ``` -You get added, removed, and updated dependencies, plus introduced/resolved findings when `--audit` is set. Great for release notes, upgrade reviews, and catching what a base-image bump dragged in. +You get added, removed, and updated dependencies, plus introduced, resolved, and persisted findings when `--audit` is set. Great for release notes, upgrade reviews, and catching what a base-image bump dragged in. ## Understand why a dependency is there diff --git a/docs/commands/diff.md b/docs/commands/diff.md index db806d71..deceabf5 100644 --- a/docs/commands/diff.md +++ b/docs/commands/diff.md @@ -49,7 +49,7 @@ Add `--enrich --audit` to classify findings between the two states into three bu bomly diff --base main --head HEAD --enrich --audit --fail-on high ``` -`--fail-on` gates on the *introduced* bucket, so this is the PR-gate form: fail the build when the change introduces a matching finding, without complaining about pre-existing ones. See [Auditors → Diff and auditing](../AUDITORS.md#diff-and-auditing). This is exactly what [Bomly Guard](../BOMLY_GUARD.md) runs on pull requests, with the result rendered as a PR comment. +The audit only inspects packages the change touched, so unrelated pre-existing debt never appears. Within that scope, `--fail-on` gates on *introduced* **and** *persisted* findings — a persisted finding means the changed package still carries a known issue at its new version, so the gate holds until the bump reaches a fixed version (or the finding is [baselined](../BASELINES.md)). See [Auditors → Diff and auditing](../AUDITORS.md#diff-and-auditing). This is exactly what [Bomly Guard](../BOMLY_GUARD.md) runs on pull requests, with the result rendered as a PR comment. ```bash bomly diff --base main --head HEAD --json # structured delta for tooling diff --git a/internal/support/component_docs.go b/internal/support/component_docs.go index bb7628c1..c12eea6d 100644 --- a/internal/support/component_docs.go +++ b/internal/support/component_docs.go @@ -483,7 +483,7 @@ See [EXIT_CODES.md](EXIT_CODES.md) for the full table. - **Resolved** — present in base, absent in head - **Persisted** — present in both -Combine with `+"`--fail-on`"+` to fail PRs that introduce new high-severity findings without complaining about pre-existing ones: +The diff audit only inspects packages the change touched, so findings on unchanged packages never appear — that is how pre-existing debt stays out of PR reviews. Both remaining buckets gate: `+"`--fail-on`"+` fails on introduced findings and on persisted ones, because a persisted finding means the changed package still ships a known issue at its new version: `+"```bash"+` bomly diff --base main --head HEAD --enrich --audit --fail-on high From 8fc60a9a18ec44eba87fcf1def2dd75ba6c9bb4b Mon Sep 17 00:00:00 2001 From: Ahmed ElMallah Date: Thu, 30 Jul 2026 01:31:36 -0700 Subject: [PATCH 2/5] docs: add the end-to-end tutorial From first scan to a CI gate on the pinned example-javascript-npm repo, with real captured output for every step: inventory, explain, enrich, audit (31 findings, exit 2), reachability triage (31 -> 16, with a dev-only-vs-runtime minimist contrast), baseline acceptance (green re-run), the tag-to-tag audited diff teaching changed-package scoping and persisted gating, and the Guard/CLI CI wiring. Co-Authored-By: Claude Fable 5 --- README.md | 1 + docs/README.md | 1 + docs/TUTORIAL.md | 220 +++++++++++++++++++++++++++++++++++++++++++++ docs/manifest.json | 6 ++ 4 files changed, 228 insertions(+) create mode 100644 docs/TUTORIAL.md diff --git a/README.md b/README.md index db88662c..d14dd86e 100644 --- a/README.md +++ b/README.md @@ -214,6 +214,7 @@ See [Plugins](docs/PLUGINS.md) for install, trust, and authoring guidance. ## Documentation - [Getting Started](docs/GETTING_STARTED.md) - install Bomly and run your first scan +- [Tutorial](docs/TUTORIAL.md) - from first scan to a CI gate on a real project - [FAQ](https://bomly.dev/faq) - quick answers on privacy, accounts, and tool differences - [Installation](docs/INSTALLATION.md) - install methods, checksums, upgrades, uninstall - [Use Cases](docs/USE_CASES.md) - practical recipes for PR gates, SBOMs, triage, and offline scans diff --git a/docs/README.md b/docs/README.md index b04315d0..08e34d13 100644 --- a/docs/README.md +++ b/docs/README.md @@ -7,6 +7,7 @@ Start with [Getting Started](GETTING_STARTED.md) if you're new. Otherwise pick t Install Bomly and run your first scan. - [Getting Started](GETTING_STARTED.md) — first scan, enrich, audit, diff +- [Tutorial](TUTORIAL.md) — from first scan to a CI gate on a real project, with the output of every step - [Installation](INSTALLATION.md) — install methods, `bomly` vs `bomly-lite`, checksum verification, uninstall - [Use Cases](USE_CASES.md) — recipes for PR gates, SBOMs, triage, license and offline scans - [Scan Targets](SCAN_TARGETS.md) — directories, Git repos, containers, SBOMs diff --git a/docs/TUTORIAL.md b/docs/TUTORIAL.md new file mode 100644 index 00000000..fd5b8414 --- /dev/null +++ b/docs/TUTORIAL.md @@ -0,0 +1,220 @@ +# Tutorial: From First Scan to a CI Gate + +A start-to-finish walkthrough on a real project: scan it, understand a finding, cut the list down to what matters, accept the existing debt, and wire the gate into CI. Every command below was run against the pinned repository shown, and the output you see is what it printed. + +Where [Getting Started](GETTING_STARTED.md) introduces each feature on its own, this page chains them into the workflow you'd actually deploy. + +## Prerequisites + +- `bomly` on your `PATH` ([Installation](INSTALLATION.md)) — verify with `bomly version` +- `git` +- Network access for the enrichment steps (`--enrich` is the explicit opt-in — see [Network and Privacy](NETWORK.md)) + +We'll use a deliberately vulnerable demo project maintained by the Bomly org, pinned to a tag so your results match this page: + +```bash +git clone --depth 1 --branch v1.0.0 https://github.com/bomly-dev/example-javascript-npm +cd example-javascript-npm +``` + +## Step 1 — Inventory the project + +```bash +bomly scan +``` + +```text +✓ 136 packages in 1 manifest (14 direct, 122 transitive · runtime 43, dev 93) + +Top-level dependencies + NAME VERSION LICENSE SCOPE VULNS + algo-httpserv 1.1.1 - runtime - + js-yaml 3.13.0 - runtime - + larvitbase 3.1.3 - runtime - + larvitfs 2.3.1 - runtime - + larvitreqparser 0.2.1 - runtime - + larvitrouter 3.0.2 - runtime - + larvitutils 2.3.0 - runtime - + lodash 4.17.15 - runtime - + marked 0.3.19 - runtime - + mocha 7.2.0 - development - + node-yaml-config 0.0.5 - runtime - + semver 5.7.1 - runtime - + to 0.2.9 - runtime - + url 0.11.0 - runtime - +``` + +Fourteen direct dependencies fan out into 136 packages — the other 122 are transitive, and they're where most surprises live. No network call happened yet: the license and vulnerability columns stay empty until you enrich. + +## Step 2 — Ask why a package is there + +Pick a package you never installed on purpose. `minimist` is a classic: + +```bash +bomly explain minimist +``` + +```text +ecosystem npm +manager npm +package minimist@0.0.10 +scope runtime +direct no +introduced by: + example-javascript-vulnerable-methods@0.0.1 + └─ to@0.2.9 + └─ optimist@0.6.1 + └─ minimist@0.0.10 [analyzed] (transitive) +``` + +The chain reads bottom-up: removing `minimist@0.0.10` means dealing with `optimist`, which means dealing with the direct dependency `to`. That's the fact you need before filing any upgrade PR. ([explain reference](commands/explain.md)) + +## Step 3 — Enrich with vulnerability and license data + +```bash +bomly scan --enrich +``` + +```text +✓ 136 packages in 1 manifest (14 direct, 122 transitive · runtime 43, dev 93) + +✓ Enriched via Grype, deps.dev License Matcher + +✓ 22 fix suggestions for 22 of 23 vulnerable packages. + Run again with --format json to see remediation details. + +Top-level dependencies + NAME VERSION LICENSE SCOPE VULNS + algo-httpserv 1.1.1 MIT runtime 1H + js-yaml 3.13.0 MIT runtime 1H 1M + ... +``` + +Twenty-three vulnerable packages, and a fix exists for all but one. Enrichment sends package coordinates — never your code — to the services listed in [Matchers](MATCHERS.md). + +## Step 4 — Turn data into a gate + +```bash +bomly scan --enrich --audit --fail-on high +echo $? +``` + +
+Findings (31 total — first ten shown) + +```text +Findings + [CRITICAL] GHSA-cf4h-3jhx-xvhq underscore@1.9.2 + [CRITICAL] GHSA-xvch-5gv4-984h minimist@0.0.10 + [CRITICAL] GHSA-xvch-5gv4-984h minimist@1.2.5 + [HIGH] GHSA-23c5-xmqv-rm74 minimatch@3.0.4 + [HIGH] GHSA-35jh-r3h4-6jhm lodash@4.17.15 + [HIGH] GHSA-3ppc-4f35-3m26 minimatch@3.0.4 + [HIGH] GHSA-5v2h-r2cx-5xgj marked@0.3.19 + [HIGH] GHSA-8j8c-7jfh-h6hx js-yaml@3.13.0 + [HIGH] GHSA-cgjv-rghq-qhgp algo-httpserv@1.1.1 + [HIGH] GHSA-p6mc-m468-83gw lodash@4.17.15 +``` + +
+ +The exit code is `2`: at least one finding matched `--fail-on high`. On this project that's 31 findings — too many to fix in one sitting, which is exactly the situation the next two steps are for. ([Auditors](AUDITORS.md), [Exit Codes](EXIT_CODES.md)) + +## Step 5 — Triage by reachability (experimental) + +Which of those 31 findings live in code this app actually imports? + +```bash +bomly scan --enrich --audit --analyze --fail-on high --fail-on reachable +``` + +The two `--fail-on` constraints AND together: a finding now gates only when it is high-or-above **and** reachable. On this project that cuts 31 findings to 16 — for example, `minimist@1.2.5` (pulled in by the dev-only `mocha` toolchain) drops out, while `minimist@0.0.10` on the runtime path stays. + +Reachability is experimental, and at package precision "unreachable" means "no import path found", not "safe" — read [Reachability](REACHABILITY.md) before making this your only gate. + +## Step 6 — Accept the existing debt + +A gate you can't turn on because of old findings protects nothing. Baselines record today's findings so the gate fails only on *new* ones: + +```bash +bomly baseline create --enrich +bomly scan --enrich --audit --fail-on high +echo $? # 0 +``` + +The findings are still reported — their policy status becomes `suppressed` instead of gating. Commit `.bomly/baseline.json` so CI and teammates share the same accepted set, and use `bomly baseline update` / `prune` as debt gets paid down. ([Finding Baselines](BASELINES.md)) + +## Step 7 — Review changes, not snapshots + +For pull requests, compare two states and audit only what changed. The demo repo has two tags to compare: + +```bash +bomly diff --url https://github.com/bomly-dev/example-javascript-npm \ + --base v0.9.0 --head v1.0.0 --enrich --audit --fail-on high +echo $? # 2 +``` + +```text +Version changed (1) + ~ lodash 4.17.14 → 4.17.15 + +6 finding(s) persisted. + persisted [HIGH] GHSA-35jh-r3h4-6jhm lodash@4.17.15 + persisted [HIGH] GHSA-p6mc-m468-83gw lodash@4.17.15 + persisted [HIGH] GHSA-r5fr-rjxr-66jc lodash@4.17.15 + persisted [MEDIUM] GHSA-29mw-wpgm-hmr9 lodash@4.17.15 + persisted [MEDIUM] GHSA-f23m-r3pf-42rh lodash@4.17.15 + persisted [MEDIUM] GHSA-xxjr-mmjv-4gpg lodash@4.17.15 + +✓ 1 fix suggestion for 1 of 1 vulnerable package. + Run again with --format json to see remediation details. +``` + +Read this carefully, because it shows both halves of how diff gating works: + +- The other 21 vulnerable packages from Step 4 are absent. Diff audits **only the packages the change touched** — untouched debt elsewhere in the repo never blocks a PR. +- The exit code is still `2`. The lodash bump *persisted* three high-severity findings: 4.17.15 still ships them, so the gate correctly refuses to call this upgrade done. The way out is the bump the fix suggestion points at — or an accepted baseline. +- The baseline from Step 6 didn't apply here because diff materializes each side from git: every side uses the baseline **committed at that ref**. Commit `.bomly/baseline.json` and PRs that leave accepted findings untouched pass. + +([diff reference](commands/diff.md)) + +## Step 8 — Put it in CI + +The turnkey form for GitHub pull requests is the Guard action — it runs the diff from Step 7 against the PR's merge base and posts the result as a check and an updatable comment: + +```yaml +# .github/workflows/bomly.yml +on: pull_request +jobs: + bomly: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + with: + fetch-depth: 0 + - uses: bomly-dev/bomly-guard@v1 + with: + fail-on: high +``` + +Prefer the CLI directly (any CI system), with SARIF for the security tab: + +```yaml +- name: Bomly gate + run: bomly diff --base origin/main --head HEAD --enrich --audit --fail-on high -o sarif=bomly.sarif +- name: Upload SARIF + if: success() || failure() + uses: github/codeql-action/upload-sarif@v3 + with: + sarif_file: bomly.sarif +``` + +Recipes for GitLab, Jenkins, Azure DevOps, and CircleCI live in [CI Integration](CI_INTEGRATION.md); Guard's full input reference in [Bomly Guard](BOMLY_GUARD.md). + +## What you built + +An inventory you can query (`scan`, `explain`), an enrichment-backed policy gate with a reachability-aware triage path, a committed baseline that separates old debt from new risk, and a PR-level diff gate in CI. From here: + +- [Use Cases](USE_CASES.md) — more goal-shaped recipes (licenses, typosquats, containers, SBOMs) +- [Output Formats](OUTPUT_FORMATS.md) — JSON, SARIF, and SBOM artifacts from the same runs +- [Interactive TUI](TUI.md) — explore the same graph with `bomly scan --interactive` diff --git a/docs/manifest.json b/docs/manifest.json index 00fb87d5..68278ecf 100644 --- a/docs/manifest.json +++ b/docs/manifest.json @@ -33,6 +33,12 @@ "description": "First scan, enrich, audit, diff — all in five minutes.", "group": "start" }, + { + "slug": "tutorial", + "title": "Tutorial", + "description": "From first scan to a CI gate on a real project, with the output of every step.", + "group": "start" + }, { "slug": "installation", "title": "Installation", From 0a3fe85288c45f8fbb22fc9bd331124e85727f98 Mon Sep 17 00:00:00 2001 From: Ahmed ElMallah Date: Thu, 30 Jul 2026 01:33:05 -0700 Subject: [PATCH 3/5] docs: show verification output and current tags on the installation page The checksum and cosign examples referenced v0.14.2 and showed no expected result. They now use the current release tag (with a substitute-your-tag note), correct GoReleaser asset names, and the captured output of both verifications run against the real v0.20.2 assets. Co-Authored-By: Claude Fable 5 --- docs/INSTALLATION.md | 44 ++++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/docs/INSTALLATION.md b/docs/INSTALLATION.md index 56874814..689754b1 100644 --- a/docs/INSTALLATION.md +++ b/docs/INSTALLATION.md @@ -94,12 +94,12 @@ irm https://bomly.dev/install.ps1 | iex Pin a version or install the lite binary: ```bash -curl -fsSL https://bomly.dev/install.sh | BOMLY_VERSION=v0.14.2 sh +curl -fsSL https://bomly.dev/install.sh | BOMLY_VERSION=v0.20.2 sh curl -fsSL https://bomly.dev/install.sh | BOMLY_BINARY=bomly-lite sh ``` ```powershell -$env:BOMLY_VERSION = "v0.14.2"; irm https://bomly.dev/install.ps1 | iex +$env:BOMLY_VERSION = "v0.20.2"; irm https://bomly.dev/install.ps1 | iex $env:BOMLY_BINARY = "bomly-lite"; irm https://bomly.dev/install.ps1 | iex ``` @@ -143,19 +143,19 @@ Archive naming: Manual Linux / macOS install: ```bash -curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.14.2/bomly_v0.14.2_linux_amd64.tar.gz -curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.14.2/SHA256SUMS +curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.20.2/bomly_0.20.2_linux_amd64.tar.gz +curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.20.2/SHA256SUMS sha256sum --check SHA256SUMS --ignore-missing -tar -xzf bomly_v0.14.2_linux_amd64.tar.gz +tar -xzf bomly_0.20.2_linux_amd64.tar.gz sudo install -m 0755 bomly /usr/local/bin/ ``` Manual Windows install: ```powershell -$archive = "bomly_v0.14.2_windows_amd64.zip" -Invoke-WebRequest -Uri "https://github.com/bomly-dev/bomly-cli/releases/download/v0.14.2/$archive" -OutFile $archive -Invoke-WebRequest -Uri "https://github.com/bomly-dev/bomly-cli/releases/download/v0.14.2/SHA256SUMS" -OutFile SHA256SUMS +$archive = "bomly_0.20.2_windows_amd64.zip" +Invoke-WebRequest -Uri "https://github.com/bomly-dev/bomly-cli/releases/download/v0.20.2/$archive" -OutFile $archive +Invoke-WebRequest -Uri "https://github.com/bomly-dev/bomly-cli/releases/download/v0.20.2/SHA256SUMS" -OutFile SHA256SUMS Get-FileHash .\$archive -Algorithm SHA256 Expand-Archive -Path $archive -DestinationPath . # Move bomly.exe somewhere on your PATH. @@ -191,19 +191,25 @@ curl -sSfL https://get.anchore.io/grype | sh -s -- -b /usr/local/bin ## Verify release checksums -Releases include `SHA256SUMS` alongside every archive and package. +Releases include `SHA256SUMS` alongside every archive and package. The commands below use `v0.20.2` — substitute the release you are verifying. Linux and macOS: ```bash -curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.14.2/SHA256SUMS +curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.20.2/SHA256SUMS sha256sum --check SHA256SUMS --ignore-missing ``` +Each downloaded archive in the directory prints one `OK` line, and the command exits `0`: + +```text +bomly_0.20.2_darwin_arm64.tar.gz: OK +``` + PowerShell: ```powershell -Get-FileHash .\bomly_v0.14.2_windows_amd64.zip -Algorithm SHA256 +Get-FileHash .\bomly_0.20.2_windows_amd64.zip -Algorithm SHA256 # Compare the printed hash against the matching line in SHA256SUMS. ``` @@ -212,7 +218,7 @@ Get-FileHash .\bomly_v0.14.2_windows_amd64.zip -Algorithm SHA256 `SHA256SUMS` is itself signed keylessly with [cosign](https://docs.sigstore.dev/cosign/signing/overview/), tying the release to the exact GitHub Actions workflow run that built it: ```bash -curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.14.2/SHA256SUMS.sigstore.json +curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.20.2/SHA256SUMS.sigstore.json cosign verify-blob \ --bundle SHA256SUMS.sigstore.json \ --certificate-identity-regexp "^https://github.com/bomly-dev/bomly-cli/.github/workflows/release.yml@.*$" \ @@ -220,17 +226,23 @@ cosign verify-blob \ SHA256SUMS ``` +A valid signature prints exactly: + +```text +Verified OK +``` + ### Verify SLSA provenance Each release also publishes a single `multiple.intoto.jsonl` SLSA Build Level 3 provenance file covering every release artifact, attesting which source commit and workflow produced them: ```bash -curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.14.2/bomly_v0.14.2_linux_amd64.tar.gz -curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.14.2/multiple.intoto.jsonl -slsa-verifier verify-artifact bomly_v0.14.2_linux_amd64.tar.gz \ +curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.20.2/bomly_0.20.2_linux_amd64.tar.gz +curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.20.2/multiple.intoto.jsonl +slsa-verifier verify-artifact bomly_0.20.2_linux_amd64.tar.gz \ --provenance-path multiple.intoto.jsonl \ --source-uri github.com/bomly-dev/bomly-cli \ - --source-tag v0.14.2 + --source-tag v0.20.2 ``` `slsa-verifier` is available from the [slsa-framework/slsa-verifier releases](https://github.com/slsa-framework/slsa-verifier/releases). From 4520f82c496ac1843414d4eb76fed0a57bc19ff6 Mon Sep 17 00:00:00 2001 From: Ahmed ElMallah Date: Thu, 30 Jul 2026 02:35:05 -0700 Subject: [PATCH 4/5] docs: address tutorial and verification review feedback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All five findings reproduced and confirmed before fixing: - Propagate the diff-gating correction to the remaining CI surfaces (GETTING_STARTED, CI_INTEGRATION x2, BOMLY_GUARD) that still described introduced-only gating or unconditional ignoring of pre-existing findings. - Tutorial: pin the capture context (Bomly 0.20.2, 2026-07-30) and stop promising identical results for live enrichment data; account findings precisely (25 gating vulnerabilities + 6 non-gating license warnings = 31 rows; reachability cuts gating findings 25 -> 10, shown as 16 rows); replace the Step 7 diff block with the output actually produced under --fail-on high (3 HIGH rows — the threshold filters rendering) and correct the absent-package count to 22. - Installation: the checksum flow now downloads an explicit artifact and splits per platform — GNU sha256sum on Linux, line-selected shasum -a 256 -c on macOS (verified against the live v0.20.2 assets). Co-Authored-By: Claude Fable 5 --- docs/BOMLY_GUARD.md | 2 +- docs/CI_INTEGRATION.md | 4 ++-- docs/GETTING_STARTED.md | 2 +- docs/INSTALLATION.md | 15 ++++++++++++--- docs/TUTORIAL.md | 19 +++++++++---------- 5 files changed, 25 insertions(+), 17 deletions(-) diff --git a/docs/BOMLY_GUARD.md b/docs/BOMLY_GUARD.md index e9e9284a..7b8501c2 100644 --- a/docs/BOMLY_GUARD.md +++ b/docs/BOMLY_GUARD.md @@ -43,7 +43,7 @@ Pin to a major tag (`@v1`) for automatic patch and minor updates, or to an exact On a `pull_request` event the action: 1. Installs the Bomly CLI (`version` input, defaults to `latest`). -2. Resolves the base and head refs. Pull requests default to the PR merge base for the base and the PR head SHA for the head, so review focuses only on what the PR changes — pre-existing findings on the target branch are ignored. +2. Resolves the base and head refs. Pull requests default to the PR merge base for the base and the PR head SHA for the head, so review focuses only on what the PR changes. Packages the PR doesn't touch are never audited — which is how pre-existing target-branch findings stay out of the review; findings that persist on a package the PR did change still count. 3. Runs `bomly diff --format json` with the enrich/audit/analyze and policy flags mapped from the action inputs. 4. Renders the Markdown summary and SARIF side outputs. 5. Writes the job summary, optionally posts (or updates) a PR comment, and optionally uploads SARIF. diff --git a/docs/CI_INTEGRATION.md b/docs/CI_INTEGRATION.md index 17eb4525..4dcb5c78 100644 --- a/docs/CI_INTEGRATION.md +++ b/docs/CI_INTEGRATION.md @@ -71,7 +71,7 @@ jobs: --json > bomly-diff.json ``` -This fails only when the PR introduces a new high finding, ignoring pre-existing ones. +The diff audits only the packages the PR touched, so findings on untouched packages never fail the job. Within that scope, introduced and persisted high findings both gate — persisted means a changed package still ships a known issue at its new version. ### Cache matcher data @@ -254,7 +254,7 @@ Tune `--fail-on` to taste. `pre-push` keeps commits fast and only runs on push. - Pin the Bomly version in CI. Use a tagged release URL or package-manager version, not `latest`. - Cache `~/.bomly/cache` across runs. Matcher TTLs make this safe. - Always upload the SBOM even when the scan fails. The SBOM is a release artifact in its own right. -- Use `bomly diff` on PRs to avoid penalizing PRs for pre-existing findings. +- Use `bomly diff` on PRs so pre-existing findings on untouched packages don't penalize the job — only the packages the PR changes are audited. - Pre-warm enrichment on `main` with a scheduled nightly run so PR jobs start with a warm cache. ## See also diff --git a/docs/GETTING_STARTED.md b/docs/GETTING_STARTED.md index 6ba4cf06..d8ac4e65 100644 --- a/docs/GETTING_STARTED.md +++ b/docs/GETTING_STARTED.md @@ -179,7 +179,7 @@ Or two SBOM files: bomly diff --sbom --base ./old.spdx.json --head ./new.spdx.json --json ``` -Add `--audit --fail-on high` to fail PRs that introduce new high-severity findings without complaining about pre-existing ones. +Add `--audit --fail-on high` to gate PRs. The diff audits only the packages the change touched — untouched debt elsewhere never blocks a PR — and within that scope it fails when the change introduces a matching finding or keeps one alive at the changed package's new version (see [diff](commands/diff.md)). ## Use Bomly with an AI agent diff --git a/docs/INSTALLATION.md b/docs/INSTALLATION.md index 689754b1..29ae66af 100644 --- a/docs/INSTALLATION.md +++ b/docs/INSTALLATION.md @@ -191,16 +191,25 @@ curl -sSfL https://get.anchore.io/grype | sh -s -- -b /usr/local/bin ## Verify release checksums -Releases include `SHA256SUMS` alongside every archive and package. The commands below use `v0.20.2` — substitute the release you are verifying. +Releases include `SHA256SUMS` alongside every archive and package. The commands below use `v0.20.2` and one archive per platform — substitute the release and artifact you are verifying, and download both the archive and the checksum file first. -Linux and macOS: +Linux (GNU coreutils): ```bash +curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.20.2/bomly_0.20.2_linux_amd64.tar.gz curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.20.2/SHA256SUMS sha256sum --check SHA256SUMS --ignore-missing ``` -Each downloaded archive in the directory prints one `OK` line, and the command exits `0`: +macOS (the system `shasum`; select the artifact's line first): + +```bash +curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.20.2/bomly_0.20.2_darwin_arm64.tar.gz +curl -L -O https://github.com/bomly-dev/bomly-cli/releases/download/v0.20.2/SHA256SUMS +grep " bomly_0.20.2_darwin_arm64.tar.gz$" SHA256SUMS | shasum -a 256 -c +``` + +Either way, each verified archive prints one `OK` line and the command exits `0`: ```text bomly_0.20.2_darwin_arm64.tar.gz: OK diff --git a/docs/TUTORIAL.md b/docs/TUTORIAL.md index fd5b8414..f12bdea3 100644 --- a/docs/TUTORIAL.md +++ b/docs/TUTORIAL.md @@ -10,7 +10,7 @@ Where [Getting Started](GETTING_STARTED.md) introduces each feature on its own, - `git` - Network access for the enrichment steps (`--enrich` is the explicit opt-in — see [Network and Privacy](NETWORK.md)) -We'll use a deliberately vulnerable demo project maintained by the Bomly org, pinned to a tag so your results match this page: +We'll use a deliberately vulnerable demo project maintained by the Bomly org, pinned to a tag so the *dependency graph* you scan matches this page. The outputs below were captured with Bomly 0.20.2 on 2026-07-30; graph numbers are stable, but advisory counts and IDs come from live enrichment data and may drift as services add or withdraw records: ```bash git clone --depth 1 --branch v1.0.0 https://github.com/bomly-dev/example-javascript-npm @@ -100,7 +100,7 @@ echo $? ```
-Findings (31 total — first ten shown) +Findings (31 rows: 25 gating vulnerabilities + 6 license warnings — first ten shown) ```text Findings @@ -118,17 +118,17 @@ Findings
-The exit code is `2`: at least one finding matched `--fail-on high`. On this project that's 31 findings — too many to fix in one sitting, which is exactly the situation the next two steps are for. ([Auditors](AUDITORS.md), [Exit Codes](EXIT_CODES.md)) +The exit code is `2`: at least one finding matched `--fail-on high`. The report lists 31 findings, of two kinds: 25 gating vulnerability findings, plus 6 license *warnings* (the `[WARNING] UNKNOWN-…` rows — packages whose license couldn't be determined) that inform but never gate. Twenty-five gating findings is too many to fix in one sitting, which is exactly the situation the next two steps are for. ([Auditors](AUDITORS.md), [Exit Codes](EXIT_CODES.md)) ## Step 5 — Triage by reachability (experimental) -Which of those 31 findings live in code this app actually imports? +Which of those 25 gating findings live in code this app actually imports? ```bash bomly scan --enrich --audit --analyze --fail-on high --fail-on reachable ``` -The two `--fail-on` constraints AND together: a finding now gates only when it is high-or-above **and** reachable. On this project that cuts 31 findings to 16 — for example, `minimist@1.2.5` (pulled in by the dev-only `mocha` toolchain) drops out, while `minimist@0.0.10` on the runtime path stays. +The two `--fail-on` constraints AND together: a vulnerability finding now gates only when it is high-or-above **and** reachable. On this project that cuts the gating vulnerability findings from 25 to 10 (the report shows 16 rows — the 6 license warnings are unaffected by reachability). For example, `minimist@1.2.5` (pulled in by the dev-only `mocha` toolchain) drops out, while `minimist@0.0.10` on the runtime path stays. Reachability is experimental, and at package precision "unreachable" means "no import path found", not "safe" — read [Reachability](REACHABILITY.md) before making this your only gate. @@ -158,21 +158,20 @@ echo $? # 2 Version changed (1) ~ lodash 4.17.14 → 4.17.15 -6 finding(s) persisted. +3 finding(s) persisted. persisted [HIGH] GHSA-35jh-r3h4-6jhm lodash@4.17.15 persisted [HIGH] GHSA-p6mc-m468-83gw lodash@4.17.15 persisted [HIGH] GHSA-r5fr-rjxr-66jc lodash@4.17.15 - persisted [MEDIUM] GHSA-29mw-wpgm-hmr9 lodash@4.17.15 - persisted [MEDIUM] GHSA-f23m-r3pf-42rh lodash@4.17.15 - persisted [MEDIUM] GHSA-xxjr-mmjv-4gpg lodash@4.17.15 ✓ 1 fix suggestion for 1 of 1 vulnerable package. Run again with --format json to see remediation details. ``` +(The `--fail-on high` threshold filters the rendered findings too — drop it to also see this bump's three medium-severity persisted advisories.) + Read this carefully, because it shows both halves of how diff gating works: -- The other 21 vulnerable packages from Step 4 are absent. Diff audits **only the packages the change touched** — untouched debt elsewhere in the repo never blocks a PR. +- The other 22 vulnerable packages from Step 3 are absent. Diff audits **only the packages the change touched** — untouched debt elsewhere in the repo never blocks a PR. - The exit code is still `2`. The lodash bump *persisted* three high-severity findings: 4.17.15 still ships them, so the gate correctly refuses to call this upgrade done. The way out is the bump the fix suggestion points at — or an accepted baseline. - The baseline from Step 6 didn't apply here because diff materializes each side from git: every side uses the baseline **committed at that ref**. Commit `.bomly/baseline.json` and PRs that leave accepted findings untouched pass. From 11bda963f6c1f6936be1b6b9f60dc9a03e572647 Mon Sep 17 00:00:00 2001 From: Ahmed ElMallah Date: Thu, 30 Jul 2026 02:46:26 -0700 Subject: [PATCH 5/5] docs: address second-round review feedback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - GETTING_STARTED: the diff gate sentence now includes --enrich; the CLI rejects --audit without it (verified: exit 4). - USE_CASES: rename the PR-gate section to cover persisted findings, and fix the pre-existing typosquat recipe that claimed a fully offline run — as written it exits 4 on the same validation; the recipe now includes --enrich and states the check is name-based while the flag pairing still contacts enrichment services. - BOMLY_GUARD: persisted findings count only when they match the configured policy. - Tutorial descriptions in the docs index and manifest promise representative output, not output for every step. Co-Authored-By: Claude Fable 5 --- docs/BOMLY_GUARD.md | 2 +- docs/GETTING_STARTED.md | 2 +- docs/README.md | 2 +- docs/USE_CASES.md | 6 +++--- docs/manifest.json | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/BOMLY_GUARD.md b/docs/BOMLY_GUARD.md index 7b8501c2..c55be04b 100644 --- a/docs/BOMLY_GUARD.md +++ b/docs/BOMLY_GUARD.md @@ -43,7 +43,7 @@ Pin to a major tag (`@v1`) for automatic patch and minor updates, or to an exact On a `pull_request` event the action: 1. Installs the Bomly CLI (`version` input, defaults to `latest`). -2. Resolves the base and head refs. Pull requests default to the PR merge base for the base and the PR head SHA for the head, so review focuses only on what the PR changes. Packages the PR doesn't touch are never audited — which is how pre-existing target-branch findings stay out of the review; findings that persist on a package the PR did change still count. +2. Resolves the base and head refs. Pull requests default to the PR merge base for the base and the PR head SHA for the head, so review focuses only on what the PR changes. Packages the PR doesn't touch are never audited — which is how pre-existing target-branch findings stay out of the review; persisted findings on a package the PR did change still count when they match the configured policy (here, `fail-on: high`). 3. Runs `bomly diff --format json` with the enrich/audit/analyze and policy flags mapped from the action inputs. 4. Renders the Markdown summary and SARIF side outputs. 5. Writes the job summary, optionally posts (or updates) a PR comment, and optionally uploads SARIF. diff --git a/docs/GETTING_STARTED.md b/docs/GETTING_STARTED.md index d8ac4e65..10d0463c 100644 --- a/docs/GETTING_STARTED.md +++ b/docs/GETTING_STARTED.md @@ -179,7 +179,7 @@ Or two SBOM files: bomly diff --sbom --base ./old.spdx.json --head ./new.spdx.json --json ``` -Add `--audit --fail-on high` to gate PRs. The diff audits only the packages the change touched — untouched debt elsewhere never blocks a PR — and within that scope it fails when the change introduces a matching finding or keeps one alive at the changed package's new version (see [diff](commands/diff.md)). +Add `--enrich --audit --fail-on high` to gate PRs (auditing requires enrichment). The diff audits only the packages the change touched — untouched debt elsewhere never blocks a PR — and within that scope it fails when the change introduces a matching finding or keeps one alive at the changed package's new version (see [diff](commands/diff.md)). ## Use Bomly with an AI agent diff --git a/docs/README.md b/docs/README.md index 08e34d13..ab1a5d98 100644 --- a/docs/README.md +++ b/docs/README.md @@ -7,7 +7,7 @@ Start with [Getting Started](GETTING_STARTED.md) if you're new. Otherwise pick t Install Bomly and run your first scan. - [Getting Started](GETTING_STARTED.md) — first scan, enrich, audit, diff -- [Tutorial](TUTORIAL.md) — from first scan to a CI gate on a real project, with the output of every step +- [Tutorial](TUTORIAL.md) — from first scan to a CI gate on a real project, with representative output from the workflow - [Installation](INSTALLATION.md) — install methods, `bomly` vs `bomly-lite`, checksum verification, uninstall - [Use Cases](USE_CASES.md) — recipes for PR gates, SBOMs, triage, license and offline scans - [Scan Targets](SCAN_TARGETS.md) — directories, Git repos, containers, SBOMs diff --git a/docs/USE_CASES.md b/docs/USE_CASES.md index 41383ed5..6c973b15 100644 --- a/docs/USE_CASES.md +++ b/docs/USE_CASES.md @@ -2,7 +2,7 @@ Recipes for the jobs people actually use Bomly for. Each one is a goal, the command that does it, what you get back, and where to go deeper. New to Bomly? Start with [Getting Started](GETTING_STARTED.md) first. -## Gate a pull request on new vulnerabilities +## Gate a pull request on dependency vulnerabilities **Goal:** fail a PR when its dependency changes carry a high-severity vulnerability, without nagging about debt the PR didn't touch. @@ -83,14 +83,14 @@ Licenses are matched as SPDX expressions. Use `--deny-license` to block specific **Goal:** flag dependency names that impersonate packages you trust, or that you've banned outright. ```bash -bomly scan --audit \ +bomly scan --enrich --audit \ --protected-package react --protected-package lodash \ --typosquat-threshold 0.85 \ --deny-package event-stream \ --fail-on any ``` -This check is name-based and needs no enrichment, so it runs fully offline. See the [package auditor](auditors/package.md). +The check itself is name-based — no enrichment data feeds it — but the CLI currently requires `--enrich` alongside `--audit`, so this run does contact the enrichment services. See the [package auditor](auditors/package.md). ## Scan offline / air-gapped diff --git a/docs/manifest.json b/docs/manifest.json index 68278ecf..0b08bab2 100644 --- a/docs/manifest.json +++ b/docs/manifest.json @@ -36,7 +36,7 @@ { "slug": "tutorial", "title": "Tutorial", - "description": "From first scan to a CI gate on a real project, with the output of every step.", + "description": "From first scan to a CI gate on a real project, with representative output from the workflow.", "group": "start" }, {