feat(engine): add rule evaluation coverage contract (#263) - #321
Draft
dipeshrayg wants to merge 2 commits into
Draft
feat(engine): add rule evaluation coverage contract (#263)#321dipeshrayg wants to merge 2 commits into
dipeshrayg wants to merge 2 commits into
Conversation
scan() only ever reports violations, so a scan with no findings for a rule is indistinguishable from compliant, not-applicable, and never evaluated. get_compliance_score() inferred PASS from the absence of a finding, silently treating errored or unmigrated rules as passing. Rules can now additionally expose evaluate(azure_client, subscription_id) -> List[RuleEvaluation], reporting a PASS/FAIL/ UNKNOWN/ERROR/NOT_APPLICABLE status per resource instead of only per violation. This is additive: scan() is unchanged, and a rule without evaluate() still runs, its coverage recorded as UNKNOWN/ LEGACY_RULE_NOT_MIGRATED rather than assumed to be a pass. - New rule_evaluations table (migration chained after openshield-org#308's head), with CHECK constraints on the five statuses, a non-empty canonical resource_id, and a required reason_code for UNKNOWN/ERROR/ NOT_APPLICABLE. FAIL evaluations get a nullable finding_id FK, backfilled in the same transaction as the finding insert. - Engine wiring: evaluator exceptions produce an ERROR at a canonical rule/subscription scope rather than vanishing; a FAIL evaluation contributes its own finding only when scan() hasn't already reported the same (rule_id, resource_id), so a rule implementing both never double-counts. - get_compliance_score() now derives PASS/FAIL/UNKNOWN/ERROR/ NOT_APPLICABLE from rule_evaluations instead of inferring PASS from missing findings, and returns evaluated/passed/failed/unknown/error/ not_applicable counts separately. UNKNOWN and ERROR never improve the score. - AZ-KV-006 migrated as the reference evaluate() implementation. Single Alembic head and migration-chain checks, plus populated-database CHECK constraint tests gated on DATABASE_URL (CI already runs these against a live migrated Postgres). Signed-off-by: Dipesh Ray <dipesh.ray.g@gmail.com>
- Join the two-line CHECK constraint SQL string in the rule_evaluations migration onto one line, under the 120-char limit. - Add the blank line ruff format wants before a top-level def in the evaluate() code sample, and drop an em dash from the surrounding text. Signed-off-by: Dipesh Ray <dipesh.ray.g@gmail.com>
Collaborator
|
@dipeshrayg, this remains the canonical review track for #263’s evaluation semantics: status vocabulary, per-resource evidence rows, reason-code constraints, finding linkage, aggregation, and the reference-rule migration. Please keep it draft while we align its migration and persistence API with #325. Do not widen it into leases, admission, enrichment, or finding lifecycle; those belong to separate layers. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds the rule evaluation coverage contract from #263: rules can now report a status for every resource they looked at (PASS included), so a scan that never ran, errored, or came from an unmigrated rule can no longer be silently read as a pass.
Type of change
Testing
alembic headsresolves to exactly one head)Related issue
Addresses #263
What changed
scanner/evaluation.py:EvaluationStatus(PASS/FAIL/UNKNOWN/ERROR/NOT_APPLICABLE),RuleEvaluationdataclass,subscription_scope_id()for canonical non-empty scope identifiers, andaggregate_status()implementing the conservative FAIL > ERROR > UNKNOWN > PASS > NOT_APPLICABLE roll-up order.alembic/versions/3f59f83a5253_rule_evaluations.py, chained onto fix(core): enforce severity contract v1 #308's head (d8e4f6a1b2c3).rule_evaluationstable with CHECK constraints on the five statuses, a non-emptyresource_id, and a requiredreason_codefor UNKNOWN/ERROR/NOT_APPLICABLE.finding_idis a nullable FK, populated only for FAIL evaluations.scanner/engine.py: callsrule.evaluate()when a rule exposes it. A rule withoutevaluate()gets one UNKNOWN/LEGACY_RULE_NOT_MIGRATED row instead of silently having no coverage. Anevaluate()exception (or a non-list return) produces one ERROR at the canonical rule/subscription scope. A FAIL evaluation contributes its own finding only whenscan()hasn't already reported the same(rule_id, resource_id), so a rule implementing both never double-counts.api/models/finding.py:save_scan()persistsrule_evaluationsin the same transaction as findings and backfillsfinding_idfor FAIL rows via a join on(scan_id, rule_id, resource_id).get_compliance_score()now derives each control's status fromrule_evaluations(never from finding absence), returnsevaluated/passed/failed/unknown/error/not_applicablecounts separately, and excludes UNKNOWN/ERROR from the numerator so they can never improve the score. Response carriescontract_version: "2".scanner/rules/az_kv_006.py: migrated as the one reference rule.evaluate()reports PASS/FAIL per vault, UNKNOWN for a vault missing its properties payload (previously silently skipped byscan()), and NOT_APPLICABLE when the vault list is empty (sinceAzureClient.get_key_vaults()can't distinguish "no vaults" from "the list call failed"; closing that gap is the deferred ARG cross-check).docs/adding-a-rule.md: short opt-in section documenting theevaluate()contract for future rule migrations.Regression coverage
evaluate()) is recorded as UNKNOWN/LEGACY_RULE_NOT_MIGRATED, never PASS.evaluate()exception produces ERROR at canonical scope instead of vanishing.evaluate()returning a non-list is treated as an error, not silently accepted.scan()didn't already report it.scan()already reported.rule_evaluationsrows persist in the same transaction as findings, withfinding_idcorrectly linked (FAIL) or left null (UNKNOWN/ERROR/PASS/NOT_APPLICABLE).rule_evaluationsrows atomically, same as findings.aggregate_status()conservative ordering (FAIL beats ERROR beats UNKNOWN beats PASS beats NOT_APPLICABLE) under every input order.get_compliance_score(): a clean scan with real PASS evaluation rows shows PASS; a clean scan with zero evaluation rows shows UNKNOWN, not PASS (the bug this PR fixes); a remediated rule with a new PASS row shows PASS; worst-severity aggregation across findings is unchanged.evaluate(): compliant vault -> PASS, non-compliant -> FAIL with the finding embedded, missing properties -> UNKNOWN, empty vault list -> NOT_APPLICABLE, mixed vaults report one status each.resource_id, reject a NULLreason_codeon UNKNOWN, and accept a valid PASS row withfinding_idleft null.d8e4f6a1b2c3.Scope
Matches what was agreed on #263: the contract, persistence/aggregation, and one end-to-end reference rule (AZ-KV-006). The Azure Resource Graph completeness cross-check stays a separate follow-up, as discussed.
Checklist
Signed-off-bytrailer