Skip to content

fix: an unreadable scan report is not a clean scan (sable-fgk7) - #224

Merged
Rome-1 merged 1 commit into
mainfrom
fix/sable-fgk7-unreadable-report-is-not-clean
Sep 2, 2026
Merged

fix: an unreadable scan report is not a clean scan (sable-fgk7)#224
Rome-1 merged 1 commit into
mainfrom
fix/sable-fgk7-unreadable-report-is-not-clean

Conversation

@Rome-1

@Rome-1 Rome-1 commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

A scan report the tooling cannot read is not a clean scan. Three consumers of the /api/static/scan payload coerced "could not read the report" into "zero findings"; this PR makes each of them fail loudly instead, with the same regression coverage discipline as #220.

The composite action (github-action/action.yml, "Fetch results"): every count was jq ... 2>/dev/null || echo "0". A body that was not JSON, a 200 carrying an error object, or a parseable payload with no vulnerabilities key produced findings_count=0 for every severity, passed every severity-threshold including critical, and rendered the PR comment :white_check_mark: No security findings detected. The step now validates the shape first (vulnerabilities must be an array of objects); if it does not hold, the step exits 1 with status=unreadable, an actionable message naming the scan id, the dashboard and rafter get <id>, and no count outputs at all. Once the shape holds the count expressions cannot fail, so the fallbacks are gone rather than moved.

rafter issues create from-scan --scan-id (Node and Python): data.vulnerabilities || [] turned a still-processing scan, a failed scan, an error object, or a keyless payload into "No findings to create issues for" and exit 0. Both runtimes now refuse with exit 1 and a message that names the status when the scan has not completed. An empty array remains a legitimate clean result. Python additionally no longer tracebacks on a non-JSON body.

Bead: sable-fgk7. Specimen of sable-d2x2 (vacuous checks): the error path and the good-news path produced the same value.

Why the customer incident makes this P1

The AppSumo customer in sable-l10k reported two things: a 500 polling in CI, and "no errors on legacy PHP". The dashboard renders an unreadable report as No security vulnerabilities were detected (filed as se-3lf6 in secbolt), and until this PR the action rendered an unparseable one the same way. A scanner cannot report zero findings and "I could not read my results" with the same integer.

Tests

  • Two new end-to-end CI jobs drive the real action against the mock backend (test-results-unreadable-is-not-clean, a 3-way matrix over missing-key / not-json / error-object, and test-results-counts-exact). The first asserts the build fails, status=unreadable, and findings-count is empty — the floor: a count never computed must be absent, not 0. The second asserts every count is exactly the report's (3/1/1/0/1) and the build passes, so a "validation" that rejected everything would not land green either.
  • mock-rafter-api.py gains RESULTS_SHAPE / SHAPE_FROM; the poll loop sees one healthy completed before the results fetch gets the shaped body, so only the step under test is exercised. md/sarif fetches are never shaped.
  • Drift detector: three new assertions (shape validation present; zero || echo "0" fallbacks; the new branch records status=unreadable and exits 1).
  • Node: 8 tests on the real vulnerabilitiesFromPayload (imported from src, not mirrored). Python: 8 on vulnerabilities_from_payload plus 2 through the from_scan command surface.

Mutation-verified by hand, per sable-d2x2 rule A:

  • Restoring one || echo "0" → drift check 15 fails. Deleting the jq -e line → check 14 fails. Dropping status=unreadable from the new branch → check 16 fails. Unmutated: 0 failures.
  • Reverting vulnerabilitiesFromPayload to the old || [] → 6 Node tests fail. Same for Python → 7 fail.

Local: drift detector 0 failures; both YAML files parse; tsc --noEmit clean; issues.test.ts 109 pass; test_issues.py 107 pass; mock shapes verified over HTTP.

Docs

CLI_SPEC.md: one bullet under the composite action's retry contract, one paragraph under rafter issues create from-scan. github-action/README.md: findings-count is empty, never 0, when the report could not be read; status values enumerated.

The composite action's results step coerced every jq failure into
findings_count=0: a body that was not JSON, a 200 carrying an error object,
or a parseable payload with no vulnerabilities key passed every severity
threshold and rendered "No security findings detected". Validate the shape
first; on failure exit 1 with status=unreadable, an actionable message, and
no count outputs at all. Once the shape holds the counts cannot fail, so the
fallbacks are removed rather than moved.

rafter issues create from-scan (Node and Python) had the same shape:
data.vulnerabilities || [] turned a processing or failed scan, an error
object, or a keyless payload into "No findings to create issues for". Both
now refuse with exit 1 and name the scan status. An empty array is still a
clean result.

Regression coverage: two end-to-end CI jobs drive the real action against
the mock backend (a 3-way matrix of unreadable shapes asserting the build
fails with status=unreadable and an EMPTY findings-count, plus an exact
counts job asserting 3/1/1/0/1); three drift assertions; unit tests on the
real functions in both runtimes. Every new guard was mutation-verified by
hand: restoring the old behaviour fails the tests that exist to catch it.
@Rome-1
Rome-1 merged commit 3856a9d into main Sep 2, 2026
21 checks passed
@Rome-1
Rome-1 deleted the fix/sable-fgk7-unreadable-report-is-not-clean branch September 2, 2026 19:43
@Rome-1 Rome-1 mentioned this pull request Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant