fix: an unreadable scan report is not a clean scan (sable-fgk7) - #224
Merged
Conversation
The composite action's results step coerced every jq failure into findings_count=0: a body that was not JSON, a 200 carrying an error object, or a parseable payload with no vulnerabilities key passed every severity threshold and rendered "No security findings detected". Validate the shape first; on failure exit 1 with status=unreadable, an actionable message, and no count outputs at all. Once the shape holds the counts cannot fail, so the fallbacks are removed rather than moved. rafter issues create from-scan (Node and Python) had the same shape: data.vulnerabilities || [] turned a processing or failed scan, an error object, or a keyless payload into "No findings to create issues for". Both now refuse with exit 1 and name the scan status. An empty array is still a clean result. Regression coverage: two end-to-end CI jobs drive the real action against the mock backend (a 3-way matrix of unreadable shapes asserting the build fails with status=unreadable and an EMPTY findings-count, plus an exact counts job asserting 3/1/1/0/1); three drift assertions; unit tests on the real functions in both runtimes. Every new guard was mutation-verified by hand: restoring the old behaviour fails the tests that exist to catch it.
This was referenced Sep 2, 2026
This was referenced Sep 2, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A scan report the tooling cannot read is not a clean scan. Three consumers of the
/api/static/scanpayload coerced "could not read the report" into "zero findings"; this PR makes each of them fail loudly instead, with the same regression coverage discipline as #220.The composite action (
github-action/action.yml, "Fetch results"): every count wasjq ... 2>/dev/null || echo "0". A body that was not JSON, a200carrying an error object, or a parseable payload with novulnerabilitieskey producedfindings_count=0for every severity, passed everyseverity-thresholdincludingcritical, and rendered the PR comment:white_check_mark: No security findings detected. The step now validates the shape first (vulnerabilitiesmust be an array of objects); if it does not hold, the step exits 1 withstatus=unreadable, an actionable message naming the scan id, the dashboard andrafter get <id>, and no count outputs at all. Once the shape holds the count expressions cannot fail, so the fallbacks are gone rather than moved.rafter issues create from-scan --scan-id(Node and Python):data.vulnerabilities || []turned a still-processingscan, afailedscan, an error object, or a keyless payload into "No findings to create issues for" and exit 0. Both runtimes now refuse with exit 1 and a message that names the status when the scan has not completed. An empty array remains a legitimate clean result. Python additionally no longer tracebacks on a non-JSON body.Bead: sable-fgk7. Specimen of sable-d2x2 (vacuous checks): the error path and the good-news path produced the same value.
Why the customer incident makes this P1
The AppSumo customer in sable-l10k reported two things: a
500polling in CI, and "no errors on legacy PHP". The dashboard renders an unreadable report as No security vulnerabilities were detected (filed as se-3lf6 in secbolt), and until this PR the action rendered an unparseable one the same way. A scanner cannot report zero findings and "I could not read my results" with the same integer.Tests
test-results-unreadable-is-not-clean, a 3-way matrix overmissing-key/not-json/error-object, andtest-results-counts-exact). The first asserts the build fails,status=unreadable, andfindings-countis empty — the floor: a count never computed must be absent, not0. The second asserts every count is exactly the report's (3/1/1/0/1) and the build passes, so a "validation" that rejected everything would not land green either.mock-rafter-api.pygainsRESULTS_SHAPE/SHAPE_FROM; the poll loop sees one healthycompletedbefore the results fetch gets the shaped body, so only the step under test is exercised. md/sarif fetches are never shaped.|| echo "0"fallbacks; the new branch recordsstatus=unreadableand exits 1).vulnerabilitiesFromPayload(imported fromsrc, not mirrored). Python: 8 onvulnerabilities_from_payloadplus 2 through thefrom_scancommand surface.Mutation-verified by hand, per sable-d2x2 rule A:
|| echo "0"→ drift check 15 fails. Deleting thejq -eline → check 14 fails. Droppingstatus=unreadablefrom the new branch → check 16 fails. Unmutated: 0 failures.vulnerabilitiesFromPayloadto the old|| []→ 6 Node tests fail. Same for Python → 7 fail.Local: drift detector 0 failures; both YAML files parse;
tsc --noEmitclean;issues.test.ts109 pass;test_issues.py107 pass; mock shapes verified over HTTP.Docs
CLI_SPEC.md: one bullet under the composite action's retry contract, one paragraph underrafter issues create from-scan.github-action/README.md:findings-countis empty, never0, when the report could not be read;statusvalues enumerated.