Skip to content

bug(compliance): make framework reports current, evidence-based, and non-certifying #302

Description

@m-khan-97

What is wrong

Current reports can overstate assurance:

  • A missing finding is counted as PASS, including failed/unrun rules.
  • Every rule is forced into CIS, NIST, ISO and SOC 2 files even where the mapping is N/A, weak or inapplicable.
  • Several SOC 2 descriptions are swapped or describe a different rule.
  • The repository uses CIS Azure 2.0.0, NIST CSF 1.1 and ISO 27001:2013 metadata.
  • Historical scans are reinterpreted using whatever mapping files happen to be deployed now.
  • A technical cloud scanner cannot establish all organizational/process controls needed for certification.

Evidence: api/models/finding.py:535-595, compliance/frameworks/*.json, and .github/workflows/ci.yml:312-367.

Required work

Treat output as versioned technical evidence coverage, not certification. Build framework packs with explicit direct/supporting/unmapped rationale and independent review.

Acceptance criteria

  • Depends on feat: persist PASS/FAIL/ERROR/NOT_APPLICABLE per rule per resource, fix compliance score #263: PASS is emitted only from an explicit successful evaluation.
  • Framework name, edition, mapping-pack version and source are persisted with every scan/report.
  • Current supported framework editions are documented; legacy packs are clearly versioned.
  • N/A and organizational controls are excluded from technical pass-rate denominators.
  • Every mapping has rationale, evidence type, primary source, owner and review date.
  • CI validates mapping semantics/fixtures, not only file presence.
  • Public copy no longer claims full compliance or certification.
  • A sample mapping set receives independent security/compliance review before release.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingcoreCore team ownership not for studentspriority: criticalMust be fixed immediately, breaks core functionality

Type

No type

Projects

Status
🔨 In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions