feat(agent): durable evidence and finding lifecycle [Automation PR 1/5]

[TFT444]

Objective

Turn repeated scan observations into stable, tenant-isolated, auditable finding lifecycles and three deterministic pattern types. This is PR 1 of 5 and the evidence foundation for all later automation.

Scope

Scan completeness contract

Record a versioned outcome for every scheduled rule and required collector:

• SUCCESS, EMPTY_SUCCESS, PERMISSION_DENIED, TIMEOUT, FAILED, or NOT_APPLICABLE.
• Store rule version, collector version, inventory boundary, tenant, subscription, timestamps, and error category.
• A scan may be operationally completed while individual rules are incomplete, but incomplete rules must never resolve findings.
• A finding may become RESOLVED only when its rule and required collectors succeeded over the same authorised inventory boundary.

Stable finding fingerprint

Derive the fingerprint from tenant ID, subscription ID, normalized Azure resource ID, rule ID, and a versioned rule-owned evidence key. Define normalization and fingerprint schema versions. Never use mutable display names or finding prose.

Persistence

Add reversible Alembic migrations for:

• scan_rule_outcomes and collector outcomes.
• finding_lifecycles containing fingerprint, first/last seen, occurrence count, consecutive successful-scan count, state, reopen count, and optimistic/row-locking metadata.
• Pattern records and pattern-to-scan/finding evidence references.
• An idempotency record preventing a completed scan from being applied twice.

Lifecycle states: OPEN, RESOLVED, ACCEPTED, SUPPRESSED, and REOPENED. Preserve raw finding instances and append-only transition history.

Deterministic patterns

Implement only persistent finding, cross-resource recurrence, and reopened finding. Store threshold values and algorithm version with every published pattern. No LLM creates or publishes patterns.

APIs

Use the repository's explicitly documented API-versioning convention:

• GET /api/v1/patterns
• GET /api/v1/patterns/{pattern_id}

Enforce server-side tenant and subscription authorization on every query. Keep existing APIs backward compatible.

Required implementation boundaries

• Routes perform authentication, authorization, schema parsing, and status mapping only.
• Services own lifecycle and pattern transitions.
• Repositories own transactional PostgreSQL operations.
• Lifecycle update and completed-scan persistence occur in one transaction.
• Concurrent processing uses row locks or equivalent conflict protection.
• Failed, partial, timed-out, or permission-incomplete evidence fails closed.

Measurable acceptance criteria

• Replaying the same scan 100 times produces exactly one lifecycle application and no duplicate transition.
• Two concurrent processors applying the same scan produce the same result as one processor.
• An absent finding resolves only after a successful relevant rule/collector outcome.
• PERMISSION_DENIED, TIMEOUT, FAILED, missing outcome, or incomplete inventory never resolves a finding.
• A resolved fingerprint seen again becomes REOPENED and increments reopen_count exactly once.
• Tenant A cannot read Tenant B patterns; cross-subscription data cannot combine unless explicitly authorised.
• Every pattern cites existing scan and finding IDs and exposes threshold and algorithm versions.
• Upgrade from current schema and downgrade back succeed against a populated test database.
• Positive, negative, boundary, retry, concurrency, authorization, migration, and failure-path tests pass.
• Existing API and scanner regression suites pass unchanged.

Delivery evidence attached to PR

• Threat scenarios and abuse cases.
• Schema and transition diagram.
• API examples and authorization matrix.
• CI-equivalent test output, migration output, and security-scan output.
• Security-focused human review.
• Read-only deployment telemetry showing no false resolutions.

Deferred

Deployment-wave analysis, statistical attribution, organization dashboards, and LLM-generated patterns.

Dependencies and handoff

No dependency. #312 begins only after this migration and lifecycle contract are merged and observed in read-only mode.

---

Mandatory safety and automation checklist

Evidence integrity

• Fingerprint normalization is versioned and covered by fixed test vectors.
• Tenant ID, subscription ID, normalized resource ID, rule ID, and evidence key are included in every fingerprint.
• Mutable names, descriptions, AI text, and timestamps cannot change identity.
• Raw findings and lifecycle transition history remain immutable/auditable.
• Every scheduled rule and required collector records an explicit completeness outcome.
• Partial pagination, throttling exhaustion, timeout, permission denial, and missing outcomes fail closed.
• Only successful relevant rule and collector outcomes may resolve a finding.

Transactions and retries

• Completed-scan persistence and lifecycle application use one transaction.
• A unique idempotency constraint prevents applying a scan twice.
• Concurrent processors use row locks, optimistic versioning, or equivalent conflict protection.
• Worker crash and retry cannot duplicate counts, reopen events, resolutions, or patterns.
• Database errors roll back the complete lifecycle operation.

Isolation and authorization

• Every new record contains tenant and subscription boundaries.
• Every endpoint applies server-controlled tenant/subscription authorization.
• Cross-tenant and unauthorized cross-subscription queries return no records.
• Public-demo mode cannot expose real lifecycle or pattern data.

Tests and proof

• Positive, negative, boundary, retry, concurrency, authorization, and failure-path tests pass.
• Replaying one completed scan 100 times produces one lifecycle application.
• Failed/incomplete scan fixtures demonstrate zero false resolutions.
• Alembic upgrade and downgrade pass against a populated database.
• Existing scanner and API regression tests pass.
• Threat model, state diagram, schema versions, API examples, and permission behavior are documented.
• CI-equivalent validation and security scanning pass with evidence attached to the PR.
• Security-focused human review is approved.
• Read-only deployment telemetry is observed before feat(agent): evidence-backed attack-path MVP [Automation PR 2/5] #312 begins.

Exit gate

• No known critical/high security defect remains open.
• No lifecycle transition depends on LLM output.
• Maintainer confirms feat(agent): durable evidence and finding lifecycle [Automation PR 1/5] #311 acceptance criteria and this checklist are complete before merge.