Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ EXE_SUFFIX=$(if $(filter Windows_NT,$(OS)),.exe,)
GOLANGCI_LINT=$(GOPATH_BIN)/golangci-lint$(EXE_SUFFIX)
FUZZTIME?=60s

.PHONY: build build-full build-lite fmt fmt-check lint install-hooks test fuzz run generate docs-config docs-schema docs-schema-md docs-support-matrix docs-components smoke benchmark benchmark-samples benchmark-report licenses
.PHONY: build build-full build-lite fmt fmt-check lint install-hooks test fuzz run generate docs-config docs-schema docs-schema-md docs-support-matrix docs-components smoke evidence benchmark benchmark-samples benchmark-report licenses

build: build-full build-lite

Expand Down Expand Up @@ -41,6 +41,9 @@ fuzz:
smoke:
go test -tags "smoke" ./test/smoke/ -v -count=1 -timeout 15m $(if $(ARGS),$(ARGS),)

evidence:
go run ./internal/tools/publicevidence $(if $(CASE),-case $(CASE),)

benchmark: build-full
bin/$(BINARY_NAME)$(EXE_SUFFIX) benchmark $(if $(ARGS),$(ARGS),)

Expand Down
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,10 @@ Bomly reads manifests, lockfiles, package-manager output, container layers, or e
| Can CI fail on high-severity findings? | `bomly scan --enrich --audit --fail-on high --format sarif` |
| Can I triage reachable findings first? | `bomly scan --enrich --audit --analyze --fail-on high --fail-on reachable` |

For more recipes, see [Getting Started](docs/GETTING_STARTED.md) and [Use Cases](docs/USE_CASES.md).
For more recipes, see [Getting Started](docs/GETTING_STARTED.md) and
[Use Cases](docs/USE_CASES.md). To review the public inputs, commands, expected
results, and limitations behind important behavior claims, see
[Reproducible Evidence](docs/EVIDENCE.md).

## Explore Interactively

Expand Down
92 changes: 92 additions & 0 deletions docs/EVIDENCE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Reproducible evidence

Bomly publishes the inputs, commands, expected artifacts, and limitations
behind important behavior claims. The goal is to make a claim reviewable
without private infrastructure or a one-off demonstration.

The machine-readable catalog is
[`test/evidence/cases.json`](../test/evidence/cases.json). Check it from a
repository checkout:

```sh
make evidence
```

To inspect one case:

```sh
make evidence CASE=graph-npm
```

This verifies the recorded checksums and prints the exact command to reproduce
the case.

## Evidence levels

| Level | Meaning |
| --- | --- |
| Deterministic | Uses checked-in inputs or local services and compares a stable normalized result |
| Pinned input | Uses a public repository at a recorded commit; local tools or artifact registries can still affect build-tool-backed resolution |
| Snapshot | Records the normalized result of an input that can move, such as a container tag |
| Live service | Uses a pinned project with current advisory data; the result is a dated observation |
| Manual assurance | Runs a separately started GitHub Actions workflow and saves its detailed report as an artifact |

Each catalog case must state both what it proves and what it does not prove.
Remote Git inputs include a full commit revision. Fixtures, workflows, and
expected results include SHA-256 checksums.

## Case studies

- [Dependency graph evidence](evidence/DEPENDENCY_GRAPHS.md) covers npm, pnpm,
Yarn, Bun, Go, Python, and Maven graphs. A separate deterministic
`degraded-detector-fallback` case covers visible fallback orchestration.
- [Policy and vulnerability-guidance evidence](evidence/POLICY_AND_GUIDANCE.md)
covers vulnerability and SPDX policy, baselines, source changes, persisted
findings, reachability tiers, and read-only remediation suggestions.
- [Targets and operational assurance](evidence/TARGETS_AND_OPERATIONS.md)
covers local and Git projects, containers, SBOM ingestion and validation,
repeated unit tests across supported systems, release builds, and repeatable
performance measurements.

## Dated workflow evidence

The
[SBOM interoperability run from July 24, 2026](https://github.com/bomly-dev/bomly-cli/actions/runs/30057587653)
completed successfully at commit
`9530b9f3bfcb3fe1d2748fa2bcfadb5e53e3346c`. The workflow generated canonical
SPDX 2.3 and CycloneDX 1.6 documents and checked them with its recorded
checksum-pinned validators.

The
[portable stability run from July 24, 2026](https://github.com/bomly-dev/bomly-cli/actions/runs/30065452505)
completed successfully at commit
`e6bc5235f85dc909b6bc73f6ba9eb82c22c44ac4`. It repeated Go unit tests on
Linux, macOS, and Windows, repeated the Java-related and full Linux unit-test
suites, and built every release target. It did not run remote smoke tests.

Workflow summaries explain what ran, the result, and where to inspect failures.
Their downloadable artifacts retain detailed commands, versions, diagnostics,
and hashes where the workflow produces a run manifest.

## How to read the results

- A checked golden proves the normalized result for its recorded input. It is
not a promise that every project in that ecosystem has the same fidelity.
- A live enrichment golden can change when advisory services add, correct, or
withdraw records. Bomly does not claim an immutable offline advisory view.
- A package count alone is not graph proof. Review identities, versions,
relationships, scopes, sources, and occurrence paths.
- A fallback warning means useful evidence may still exist, but coverage can
be lower than the preferred detector.
- `unreachable` does not mean safe. The analyzer and tier define what was
checked.
- Remediation output is read-only guidance. It does not apply or validate a
package change.
- SBOM validation proves acceptance by the named validator versions for the
canonical fixtures, not lossless conversion for every producer or consumer.

## Provenance

The tracked cases use Bomly-owned fixtures, Bomly-owned example repositories,
or complete public input repositories. The descriptions and test structure
are written for Bomly's own graph, pipeline, and output contracts.
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ Generated from code. Treat as authoritative.
- [Matcher Reference](matchers/) — per-matcher behavior, cache, output
- [Auditor Reference](auditors/) — per-auditor options, examples, limitations
- [JSON Schemas](SCHEMAS.md) — scan, explain, diff output shapes
- [Reproducible Evidence](EVIDENCE.md) — public inputs, commands, results, and limitations behind important behavior claims

## Project

Expand Down
86 changes: 86 additions & 0 deletions docs/evidence/DEPENDENCY_GRAPHS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Dependency graph evidence

Bomly's findings depend on the graph produced before enrichment or audit.
These public cases check the graph first: package identity, version,
relationship, scope, source, and the manifest that owns each occurrence.

The cases use complete public example repositories at recorded Git commits.
Their normalized JSON results are checked into `test/smoke/testdata/golden`.
The evidence catalog records a SHA-256 checksum for every result, so an
expected change must also update the public evidence record.

## What is covered

| Case | Input evidence | Detector path | What the case checks |
| --- | --- | --- | --- |
| `graph-npm` | `package-lock.json` | npm lockfile detector | Package inventory, versions, scopes, and placement |
| `graph-pnpm` | `pnpm-lock.yaml` | pnpm lockfile detector | Package inventory and placement |
| `graph-yarn` | `yarn.lock` | Yarn lockfile detector | Package inventory and placement |
| `graph-bun` | `bun.lock` | Native Bun detector | Package inventory and placement |
| `graph-go` | `go.mod` and Go tool output | Go detector | Build-tool-backed module graph |
| `graph-python` | Pinned requirements lock | pip detector | Resolved Python package graph |
| `graph-maven` | `pom.xml` and Maven output | Maven detector | Build-tool-backed JVM graph |

This is placement evidence, not just a list of package names. The Node cases
retain duplicate versions as separate package identities and keep occurrences
attached to the dependency paths represented by each lockfile.

## Reproduce a case

From a Bomly CLI checkout:

```sh
make evidence CASE=graph-npm
```

The command verifies the catalog and prints the exact focused smoke command:

```sh
go test -tags smoke ./test/smoke/ -v -count=1 -timeout 15m \
-run 'TestScan$/scan-npm$'
```

The smoke test builds Bomly, clones the recorded public input, selects the
named detector, normalizes only documented volatile fields, and compares the
result with the checked-in JSON.

## Example review workflow

A maintainer updates a lockfile and wants to confirm that Bomly still sees the
same dependency structure:

1. Run the matching graph case before changing the detector.
2. Make the detector change.
3. Run the case again.
4. Review every golden change as a package, relationship, scope, source, or
manifest-placement change.
5. Update the catalog checksum only when the new graph is intentional.

This makes a changed count a starting point for review, not the conclusion.
The actual package and occurrence records explain what changed.

## Degraded and unknown evidence

Bomly can fall back when a preferred detector cannot run. A fallback is not
silently treated as equivalent: the result records which detector failed,
which detector ran, and whether coverage was reduced. Unresolved package
placement remains `unknown`; a synthetic link to a manifest helps navigation
but does not become a direct or transitive parent claim.

The graph cases above pin their detector selector. If that detector is not
ready or fails, the smoke test fails instead of accepting a differently shaped
fallback result.

## Limits

- A case proves the checked repository and lockfile shape, not every format
version ever produced by that package manager.
- Build-tool-backed cases depend on compatible local tools and may need
registry access to resolve artifacts.
- A fallback inventory can contain useful packages while carrying less
relationship or source evidence than a native graph.
- Project roots, workspaces, local paths, Git dependencies, and arbitrary URL
dependencies stay in the graph but are not queried as published registry
packages.
- Graph evidence does not prove advisory accuracy. Matching is a later step
with separate evidence and limits.
117 changes: 117 additions & 0 deletions docs/evidence/POLICY_AND_GUIDANCE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Policy and vulnerability-guidance evidence

Bomly separates observed evidence from policy decisions:

- enrichment attaches vulnerability and license information to packages;
- analysis can add reachability evidence;
- audit evaluates the available evidence against the selected policy;
- read-only remediation guidance is derived from vulnerability and dependency
evidence during enrichment.

That separation matters when reviewing results. A vulnerability can be
present but warning-only, suppressed by a baseline, or failing policy.
Reachability is another property of the same vulnerability; it does not
replace severity or policy status.

## Policy cases

| Case | What it checks | Expected outcome |
| --- | --- | --- |
| `vulnerability-policy` | Severity, reachability, known exploitation, advisory allowlists, and repeated constraints | Only findings that satisfy the selected constraints fail |
| `license-policy` | SPDX `AND`, `OR`, nesting, exceptions, custom references, and invalid expressions | Valid expressions keep their Boolean meaning; invalid expressions remain visible and fail |
| `baseline-policy` | Create and automatically use a project finding baseline | The finding remains present with suppressed policy status |
| `source-change-policy` | Same-version move from a registry package to a Git source | Diff requests review; `--fail-on source-change` can fail the audited change |
| `persisted-risk` | Same finding remains across a package version change | The finding is reported as persisted, not hidden as one resolved and one new finding |

Run any case through the public catalog:

```sh
make evidence CASE=license-policy
make evidence CASE=source-change-policy
make evidence CASE=persisted-risk
```

Each command prints the focused test command, the recorded inputs, and the
case limitation.

## Example policy workflow

A team wants high-severity vulnerabilities and risky source changes to block
a pull request while keeping accepted findings visible:

```sh
bomly diff \
--base main \
--head HEAD \
--enrich \
--audit \
--fail-on high \
--fail-on source-change
```

If a package changed from a known registry source to Git or an arbitrary URL,
Bomly explains that registry-based vulnerability matching may no longer cover
it. That is a review signal, not a claim that the change is malicious. Source
formats that cannot prove origin remain unknown.

A finding baseline does not delete matching findings. It changes their policy
status to `suppressed`, so JSON, SARIF, Guard, MCP, and human output can still
show the accepted risk.

## Reachability cases

The catalog includes `reachability-go`, `reachability-node`,
`reachability-python`, and `reachability-java`.

- The Go case records the analyzer and the tier actually returned for each
vulnerability.
- The JavaScript, Python, and Java cases prove package-tier reachable and
package-tier unreachable branches.
- Every case keeps the analyzer name, tier, status, and plain-language reason
with the vulnerability.

These cases use current advisory services, so the checked golden is a dated
observation. Re-running later may legitimately discover added, changed, or
withdrawn advisories.

`unreachable` never means `safe`. Static analysis can miss dynamic loading,
reflection, generated code, tests, build tags, or code paths outside the
analyzer's model. Use reachability to prioritize review, not to erase the
underlying vulnerability.

## Read-only remediation guidance

The `remediation-read-only` case exercises the central remediation component.
For each vulnerable package, it derives:

- whether the available evidence describes a complete fix, partial fix, no
upstream fix, or an unknown fix state;
- a recommended version only when all vulnerability evidence supports one;
- an occurrence-specific suggested action;
- package-manager advice supplied by the detector when available.

The guidance is derived during enrichment and applies only to vulnerabilities.
License and package-policy findings stay audit results because their meaning
depends on project policy.

Suggestions do not edit manifests, run package managers, or claim that a
change is guaranteed to work. Unknown parents and non-registry occurrences
receive manual-review guidance instead of a guessed parent upgrade.

Reproduce the deterministic derivation matrix:

```sh
make evidence CASE=remediation-read-only
```

## Limits

- `--audit` still requires `--enrich` by default. This avoids an accidental
audit with fewer findings because matching was forgotten.
- Policy cannot make missing evidence complete. Unknown source, license,
reachability, or fix data remains unknown.
- Live advisory services can change independently. Bomly does not ship an
immutable offline advisory snapshot.
- Source changes, reachability, and remediation are different signals. None
of them alone proves that a dependency is safe or malicious.
- Bomly provides read-only remediation suggestions; it does not apply fixes.
Loading