Skip to content

feat(engine): add rule evaluation coverage contract (#263) - #321

Draft
dipeshrayg wants to merge 2 commits into
openshield-org:devfrom
dipeshrayg:feat/263-rule-evaluation-coverage
Draft

feat(engine): add rule evaluation coverage contract (#263)#321
dipeshrayg wants to merge 2 commits into
openshield-org:devfrom
dipeshrayg:feat/263-rule-evaluation-coverage

Conversation

@dipeshrayg

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds the rule evaluation coverage contract from #263: rules can now report a status for every resource they looked at (PASS included), so a scan that never ran, errored, or came from an unmigrated rule can no longer be silently read as a pass.

Type of change

  • Bug fix
  • API endpoint
  • Documentation

Testing

  • Full test suite passes locally (821 passed, 9 skipped: the 4 populated-database tests below are skipped without a live Postgres and run for real in CI)
  • All seven CI-equivalent checks reproduced locally and passing (rule structure, credential scan, playbook existence/syntax, compliance JSON validity, API syntax, compliance cross-reference, alembic heads resolves to exactly one head)
  • No hardcoded credentials or secrets

Related issue

Addresses #263

What changed

  • scanner/evaluation.py: EvaluationStatus (PASS/FAIL/UNKNOWN/ERROR/NOT_APPLICABLE), RuleEvaluation dataclass, subscription_scope_id() for canonical non-empty scope identifiers, and aggregate_status() implementing the conservative FAIL > ERROR > UNKNOWN > PASS > NOT_APPLICABLE roll-up order.
  • New migration alembic/versions/3f59f83a5253_rule_evaluations.py, chained onto fix(core): enforce severity contract v1 #308's head (d8e4f6a1b2c3). rule_evaluations table with CHECK constraints on the five statuses, a non-empty resource_id, and a required reason_code for UNKNOWN/ERROR/NOT_APPLICABLE. finding_id is a nullable FK, populated only for FAIL evaluations.
  • scanner/engine.py: calls rule.evaluate() when a rule exposes it. A rule without evaluate() gets one UNKNOWN/LEGACY_RULE_NOT_MIGRATED row instead of silently having no coverage. An evaluate() exception (or a non-list return) produces one ERROR at the canonical rule/subscription scope. A FAIL evaluation contributes its own finding only when scan() hasn't already reported the same (rule_id, resource_id), so a rule implementing both never double-counts.
  • api/models/finding.py: save_scan() persists rule_evaluations in the same transaction as findings and backfills finding_id for FAIL rows via a join on (scan_id, rule_id, resource_id). get_compliance_score() now derives each control's status from rule_evaluations (never from finding absence), returns evaluated/passed/failed/unknown/error/not_applicable counts separately, and excludes UNKNOWN/ERROR from the numerator so they can never improve the score. Response carries contract_version: "2".
  • scanner/rules/az_kv_006.py: migrated as the one reference rule. evaluate() reports PASS/FAIL per vault, UNKNOWN for a vault missing its properties payload (previously silently skipped by scan()), and NOT_APPLICABLE when the vault list is empty (since AzureClient.get_key_vaults() can't distinguish "no vaults" from "the list call failed"; closing that gap is the deferred ARG cross-check).
  • docs/adding-a-rule.md: short opt-in section documenting the evaluate() contract for future rule migrations.

Regression coverage

  • Legacy rule (no evaluate()) is recorded as UNKNOWN/LEGACY_RULE_NOT_MIGRATED, never PASS.
  • evaluate() exception produces ERROR at canonical scope instead of vanishing.
  • evaluate() returning a non-list is treated as an error, not silently accepted.
  • A FAIL evaluation contributes its own finding when scan() didn't already report it.
  • A FAIL evaluation does not duplicate a finding scan() already reported.
  • rule_evaluations rows persist in the same transaction as findings, with finding_id correctly linked (FAIL) or left null (UNKNOWN/ERROR/PASS/NOT_APPLICABLE).
  • A retried scan replaces prior rule_evaluations rows atomically, same as findings.
  • aggregate_status() conservative ordering (FAIL beats ERROR beats UNKNOWN beats PASS beats NOT_APPLICABLE) under every input order.
  • get_compliance_score(): a clean scan with real PASS evaluation rows shows PASS; a clean scan with zero evaluation rows shows UNKNOWN, not PASS (the bug this PR fixes); a remediated rule with a new PASS row shows PASS; worst-severity aggregation across findings is unchanged.
  • AZ-KV-006 evaluate(): compliant vault -> PASS, non-compliant -> FAIL with the finding embedded, missing properties -> UNKNOWN, empty vault list -> NOT_APPLICABLE, mixed vaults report one status each.
  • CHECK constraints (populated-database, run against CI's Postgres): reject an unsupported status, reject an empty resource_id, reject a NULL reason_code on UNKNOWN, and accept a valid PASS row with finding_id left null.
  • Single Alembic head and correct chain onto d8e4f6a1b2c3.

Scope

Matches what was agreed on #263: the contract, persistence/aggregation, and one end-to-end reference rule (AZ-KV-006). The Azure Resource Graph completeness cross-check stays a separate follow-up, as discussed.

Checklist

  • Every commit includes a DCO Signed-off-by trailer
  • My code follows the rule template in CONTRIBUTING.md
  • I have not committed any real Azure credentials
  • My branch name follows the convention: feat/description

scan() only ever reports violations, so a scan with no findings for a
rule is indistinguishable from compliant, not-applicable, and never
evaluated. get_compliance_score() inferred PASS from the absence of a
finding, silently treating errored or unmigrated rules as passing.

Rules can now additionally expose evaluate(azure_client,
subscription_id) -> List[RuleEvaluation], reporting a PASS/FAIL/
UNKNOWN/ERROR/NOT_APPLICABLE status per resource instead of only per
violation. This is additive: scan() is unchanged, and a rule without
evaluate() still runs, its coverage recorded as UNKNOWN/
LEGACY_RULE_NOT_MIGRATED rather than assumed to be a pass.

- New rule_evaluations table (migration chained after openshield-org#308's head),
  with CHECK constraints on the five statuses, a non-empty canonical
  resource_id, and a required reason_code for UNKNOWN/ERROR/
  NOT_APPLICABLE. FAIL evaluations get a nullable finding_id FK,
  backfilled in the same transaction as the finding insert.
- Engine wiring: evaluator exceptions produce an ERROR at a canonical
  rule/subscription scope rather than vanishing; a FAIL evaluation
  contributes its own finding only when scan() hasn't already reported
  the same (rule_id, resource_id), so a rule implementing both never
  double-counts.
- get_compliance_score() now derives PASS/FAIL/UNKNOWN/ERROR/
  NOT_APPLICABLE from rule_evaluations instead of inferring PASS from
  missing findings, and returns evaluated/passed/failed/unknown/error/
  not_applicable counts separately. UNKNOWN and ERROR never improve
  the score.
- AZ-KV-006 migrated as the reference evaluate() implementation.

Single Alembic head and migration-chain checks, plus populated-database
CHECK constraint tests gated on DATABASE_URL (CI already runs these
against a live migrated Postgres).

Signed-off-by: Dipesh Ray <dipesh.ray.g@gmail.com>
- Join the two-line CHECK constraint SQL string in the rule_evaluations
  migration onto one line, under the 120-char limit.
- Add the blank line ruff format wants before a top-level def in the
  evaluate() code sample, and drop an em dash from the surrounding text.

Signed-off-by: Dipesh Ray <dipesh.ray.g@gmail.com>
@m-khan-97

Copy link
Copy Markdown
Collaborator

@dipeshrayg, this remains the canonical review track for #263’s evaluation semantics: status vocabulary, per-resource evidence rows, reason-code constraints, finding linkage, aggregation, and the reference-rule migration. Please keep it draft while we align its migration and persistence API with #325. Do not widen it into leases, admission, enrichment, or finding lifecycle; those belong to separate layers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants