Skip to content

feat(security-agent): add audit reports and refresh management UX#4069

Closed
jeanduplessis wants to merge 13 commits into
mainfrom
sa-audit-report
Closed

feat(security-agent): add audit reports and refresh management UX#4069
jeanduplessis wants to merge 13 commits into
mainfrom
sa-audit-report

Conversation

@jeanduplessis

Copy link
Copy Markdown
Contributor

Summary

Security Agent now records durable Security Finding activity, provides owner-scoped Audit Reports, and guides users through analysis, dismissal, and remediation workflows.

Why this change is needed

Security owners need period-bounded evidence of material Security Finding actions and outcomes for investigation and compliance work without implying complete legacy reconstruction or aggregate historical SLA compliance. Existing management surfaces also obscure action eligibility, stale-analysis decisions, and remediation policy rejections, making safe next steps harder to understand.

How this is addressed

  • Extends the canonical audit ledger with owner-scoped, idempotent, deletion-safe Security Finding Activity Events emitted across sync, analysis, dismissal, remediation, and deletion flows.
  • Adds personal and organization Audit Reports with UTC date ranges, deterministic finding timelines, evidence-based filters, privacy controls, and fail-closed query budgets.
  • Preserves valid analysis across unchanged syncs, returns typed remediation rejection reasons, supports analysis restart, and derives dismissal defaults from completed analysis.
  • Refreshes dashboard, finding list, detail, settings, navigation, help, telemetry, documentation, and Storybook coverage around clearer user actions and outcomes.

Human Verification

  • Reviewed all eight branch commits and the final 121-file diff against current origin/main.
  • Checked Security Agent terminology and Audit Report guarantees against .specs/security-agent.md and CONTEXT.md.
  • Inspected automated test and Storybook coverage; no browser or runtime verification was performed during PR preparation.

Reviewer Notes

Human Reviewer Flags

  • This branch expands the draft Security Agent spec and context contract with Audit Report evidence, access, privacy, deletion, and availability guarantees.
  • Current UI appears to require an active integration and configuration and does not render the returned reliable-coverage fields. Confirm whether this intentionally deviates from the new historical-access and coverage-display guarantees before merge.
  • Schema-first deployment is required because authoritative finding transitions now write audit evidence in the same transaction.
  • Root typography, global theme tokens, and shared UI primitives change alongside Security Agent surfaces, increasing visual regression scope.
  • Synchronous report assembly intentionally returns no partial data when its 10,000-event, 8 MiB, or timeout budgets are exceeded.

Code Reviewer Agent

Code Reviewer Notes
  • Audit events use owner-partitioned idempotency, compact deletion-safe snapshots, strict payload validation, and event-time actor attribution.
  • Report assembly uses owner-scoped keyset pagination, deterministic grouping, safe link projection, and explicit legacy-evidence handling.
  • Test coverage is strongest around event validation, lifecycle atomicity, analysis freshness, remediation admission, and report transformation.
  • Focus review on route authorization, real-database multi-page report behavior, deleted-finding end-to-end evidence, and migration behavior on populated audit data.

Comment thread services/security-auto-analysis/src/remediation.ts Outdated
Comment thread services/security-auto-analysis/src/auto-dismiss.ts
@kilo-code-bot

kilo-code-bot Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: 3 Issues Found | Recommendation: Address before merge

Executive Summary

The security_audit_log migration creates 4 indexes on a populated production table without CONCURRENTLY, which will lock concurrent reads/writes during deployment. Two previous WARNINGs from the prior review (remediation audit swallowing and undefined analysisIdentity) have been resolved.

Overview

Severity Count
CRITICAL 1
WARNING 2
SUGGESTION 0
Issue Details (click to expand)

CRITICAL

File Line Issue
packages/db/src/migrations/0166_easy_iceman.sql 10 CREATE UNIQUE INDEX/CREATE INDEX without CONCURRENTLY on populated security_audit_log table

WARNING

File Line Issue
apps/web/src/lib/security-agent/services/auto-dismiss-service.ts 558 Unvalidated triage.confidence from DB JSONB can produce NaN counters
services/security-auto-analysis/src/auto-dismiss.ts 205 Sandbox analysis gating now blocks triage-based auto-dismiss as fallback
Resolved from Prior Review
File Issue Status
services/security-auto-analysis/src/remediation.ts recordRemediationAudit silently swallowed audit write failures ✅ Fixed — callers now catch and log at error level
services/security-auto-analysis/src/auto-dismiss.ts analysisIdentity could fall through to undefined producing malformed event keys ✅ Fixed — explicit throw added for nullish identity
Files Reviewed (66 files)
  • packages/db/src/migrations/0166_easy_iceman.sql — 1 issue
  • packages/db/src/schema.ts — 0 issues
  • packages/db/src/schema-types.ts — 0 issues
  • packages/worker-utils/src/security-finding-audit.ts — 0 issues
  • packages/worker-utils/src/security-remediation-policy.ts — 0 issues
  • packages/worker-utils/src/index.ts — 0 issues
  • services/security-auto-analysis/src/remediation.ts — 0 issues (prior fixed)
  • services/security-auto-analysis/src/auto-dismiss.ts — 1 issue
  • services/security-auto-analysis/src/analysis-start-lifecycle.ts — 0 issues
  • services/security-auto-analysis/src/db/queries.ts — 0 issues
  • services/security-sync/src/dismiss.ts — 0 issues
  • services/security-sync/src/index.ts — 0 issues
  • services/security-sync/src/notifications/sweep.ts — 0 issues
  • apps/web/src/lib/security-agent/db/security-audit-report.ts — 0 issues
  • apps/web/src/lib/security-agent/db/security-findings.ts — 0 issues
  • apps/web/src/lib/security-agent/db/security-analysis.ts — 0 issues
  • apps/web/src/lib/security-agent/router/shared-handlers.ts — 0 issues
  • apps/web/src/lib/security-agent/services/auto-dismiss-service.ts — 1 issue
  • apps/web/src/lib/security-agent/services/analysis-service.ts — 0 issues
  • apps/web/src/lib/security-agent/services/manual-dismiss-client.ts — 0 issues
  • apps/web/src/lib/security-agent/services/manual-analysis-client.ts — 0 issues
  • apps/web/src/lib/security-agent/services/manual-remediation-client.ts — 0 issues
  • apps/web/src/lib/security-agent/posthog-tracking.ts — 0 issues
  • apps/web/src/routers/security-agent-router.ts — 0 issues
  • apps/web/src/routers/organizations/organization-security-agent-router.ts — 0 issues
  • .specs/security-agent.md — 0 issues
  • CONTEXT.md — 0 issues
  • 39 UI + remaining files — 0 issues

Fix these issues in Kilo Cloud

Previous Review Summaries (3 snapshots, latest commit 985b43f)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 985b43f)

Status: 2 Issues Found | Recommendation: Address before merge

Executive Summary

Two audit trail integrity concerns from the initial review persist: silently swallowed audit write failures in the remediation service and a latent undefined-in-event-key path in auto-dismiss. The incremental diff (commit baa0f39 → HEAD, 31 files) adds typed actor classification to the audit ledger with strong validation and DB-sourced actor identity — no new issues introduced.

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
services/security-auto-analysis/src/remediation.ts 672 recordRemediationAudit silently swallows audit write failures
services/security-auto-analysis/src/auto-dismiss.ts 126 analysisIdentity can fall through to undefined, producing malformed "undefined" event keys
Incremental Review (baa0f39 → HEAD)

31 files changed. Key changes:

  • Migration 0164: Renamed and regenerated; adds nullable actor_type column + CHECK constraint; removed stale backfill UPDATE.
  • Schema types: New SecurityAuditLogActorType enum (customer_user, kilo_admin, system).
  • security-finding-audit.ts: actor is now a required typed union (SecurityFindingAuditActor). System actor has no PII fields. Human actors carry id/email/name. buildSecurityFindingAuditHumanActor constructs from DB-sourced data with isAdmin flag.
  • Audit report: assembleSecurityAgentAuditReport wraps count + scan in a repeatable-read transaction with row-count consistency check. resolveSecurityAgentAuditReliableCoverageStart reads from SECURITY_AGENT_AUDIT_RELIABLE_COVERAGE_START env var.
  • Dismissal (security-sync/dismiss.ts): Actor is now fetched from kilocode_users table instead of trusting client-provided identity. Manual dismiss client sends only {id}.
  • Remediation: Automated paths use SECURITY_FINDING_AUDIT_SYSTEM_ACTOR; human paths construct typed actors from DB query (getAnalysisActorById returns is_admin). cancelRemediation hard-asserts actor exists before interrupt.
  • Auto-dismiss (web): System actor used for sandbox/triage auto-dismiss audit events.
  • Tests: Extended throughout for actor classification, admin/customer masking in reports, pagination edge cases, and idempotency.

No new issues found.

Files Reviewed (31 files)
  • apps/web/src/lib/security-agent/db/security-analysis.ts — 0 issues
  • apps/web/src/lib/security-agent/db/security-audit-report.test.ts — 0 issues
  • apps/web/src/lib/security-agent/db/security-audit-report.ts — 0 issues
  • apps/web/src/lib/security-agent/db/security-findings.ts — 0 issues
  • apps/web/src/lib/security-agent/router/shared-handlers.test.ts — 0 issues
  • apps/web/src/lib/security-agent/router/shared-handlers.ts — 0 issues
  • apps/web/src/lib/security-agent/services/auto-dismiss-service.test.ts — 0 issues
  • apps/web/src/lib/security-agent/services/auto-dismiss-service.ts — 0 issues
  • apps/web/src/lib/security-agent/services/manual-dismiss-client.test.ts — 0 issues
  • apps/web/src/lib/security-agent/services/manual-dismiss-client.ts — 0 issues
  • apps/web/src/lib/user/index.test.ts — 0 issues
  • packages/db/src/migrations/0164_famous_meteorite.sql — 0 issues
  • packages/db/src/migrations/meta/0164_snapshot.json — 0 issues (generated)
  • packages/db/src/migrations/meta/_journal.json — 0 issues (generated)
  • packages/db/src/schema-types.ts — 0 issues
  • packages/db/src/schema.test.ts — 0 issues
  • packages/db/src/schema.ts — 0 issues
  • packages/worker-utils/src/index.ts — 0 issues
  • packages/worker-utils/src/security-finding-audit.test.ts — 0 issues
  • packages/worker-utils/src/security-finding-audit.ts — 0 issues
  • services/security-auto-analysis/src/analysis-start-lifecycle.ts — 0 issues
  • services/security-auto-analysis/src/auto-dismiss.ts — 1 issue (carried forward)
  • services/security-auto-analysis/src/db/queries.ts — 0 issues
  • services/security-auto-analysis/src/remediation-admission.integration.test.ts — 0 issues
  • services/security-auto-analysis/src/remediation.ts — 1 issue (carried forward)
  • services/security-sync/src/dismiss.test.ts — 0 issues
  • services/security-sync/src/dismiss.ts — 0 issues
  • services/security-sync/src/index.test.ts — 0 issues
  • services/security-sync/src/index.ts — 0 issues
  • services/security-sync/src/notifications/sweep.ts — 0 issues
  • services/security-sync/src/sync.ts — 0 issues

Fix these issues in Kilo Cloud

Previous review (commit baa0f39)

Status: 2 Issues Found | Recommendation: Address before merge

Executive Summary

Two audit trail integrity concerns in the remediation and auto-dismiss service workers: silently swallowed audit write failures and a latent undefined-in-event-key path. No new issues found in the incremental diff (commit baa0f39).

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
services/security-auto-analysis/src/remediation.ts 664 recordRemediationAudit silently swallows audit write failures
services/security-auto-analysis/src/auto-dismiss.ts 125 analysisIdentity can fall through to undefined, producing malformed "undefined" event keys
Incremental Review (commit baa0f39 vs f4e1e3b)

15 files changed in this increment. Key changes:

  • SecurityAuditReportPage.tsx: Removes redirect-to-config gate, always renders audit report regardless of setup state. Adds provenance section with coverage warnings and stale repository filter normalization.
  • SecurityAgentLayout.tsx: Extracts getSecurityAgentNavItems, includes audit report in setup-only navigation.
  • Storybook files: Removes font decorator from preview config, deletes SecuritySettings stories, adds provenance tests.
  • No new issues found.
Files Reviewed (12 key files + 15 incremental)
  • .specs/security-agent.md — 0 issues
  • CONTEXT.md — 0 issues
  • packages/db/src/schema.ts — 0 issues
  • packages/db/src/migrations/0164_curvy_rocket_racer.sql — 0 issues
  • packages/worker-utils/src/security-finding-audit.ts — 0 issues
  • packages/worker-utils/src/security-remediation-policy.ts — 0 issues
  • apps/web/src/lib/security-agent/db/security-audit-report.ts — 0 issues
  • apps/web/src/lib/security-agent/router/shared-handlers.ts — 0 issues
  • apps/web/src/lib/security-agent/services/analysis-service.ts — 0 issues
  • apps/web/src/lib/security-agent/services/auto-dismiss-service.ts — 0 issues
  • services/security-auto-analysis/src/auto-dismiss.ts — 1 issue
  • services/security-auto-analysis/src/remediation.ts — 1 issue

Fix these issues in Kilo Cloud

Previous review (commit f4e1e3b)

Status: 2 Issues Found | Recommendation: Address before merge

Executive Summary

Two audit trail integrity concerns in the remediation and auto-dismiss service workers: silently swallowed audit write failures and a latent undefined-in-event-key path.

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
services/security-auto-analysis/src/remediation.ts 664 recordRemediationAudit silently swallows audit write failures
services/security-auto-analysis/src/auto-dismiss.ts 125 analysisIdentity can fall through to undefined, producing malformed "undefined" event keys
Files Reviewed (12 key files)
  • .specs/security-agent.md — 0 issues
  • CONTEXT.md — 0 issues
  • packages/db/src/schema.ts — 0 issues
  • packages/db/src/migrations/0164_curvy_rocket_racer.sql — 0 issues
  • packages/worker-utils/src/security-finding-audit.ts — 0 issues
  • packages/worker-utils/src/security-remediation-policy.ts — 0 issues
  • apps/web/src/lib/security-agent/db/security-audit-report.ts — 0 issues
  • apps/web/src/lib/security-agent/router/shared-handlers.ts — 0 issues
  • apps/web/src/lib/security-agent/services/analysis-service.ts — 0 issues
  • apps/web/src/lib/security-agent/services/auto-dismiss-service.ts — 0 issues
  • services/security-auto-analysis/src/auto-dismiss.ts — 1 issue
  • services/security-auto-analysis/src/remediation.ts — 1 issue

Fix these issues in Kilo Cloud


Reviewed by deepseek-v4-pro-20260423 · 1,092,096 tokens

Review guidance: REVIEW.md from base branch main

ALTER TABLE "security_audit_log" ADD COLUMN "schema_version" smallint;--> statement-breakpoint
ALTER TABLE "security_audit_log" ADD COLUMN "finding_snapshot" jsonb;--> statement-breakpoint
ALTER TABLE "security_audit_log" ADD COLUMN "source_context" text;--> statement-breakpoint
CREATE UNIQUE INDEX "UQ_security_audit_log_org_event_key" ON "security_audit_log" USING btree ("owned_by_organization_id","event_key") WHERE "security_audit_log"."owned_by_organization_id" IS NOT NULL AND "security_audit_log"."event_key" IS NOT NULL;--> statement-breakpoint

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CRITICAL: CREATE UNIQUE INDEX on populated security_audit_log table without CONCURRENTLY

The security_audit_log table has existed since migration 0027 and is populated in production. Creating unique indexes (lines 10-11) and composite indexes (lines 12-13) without CONCURRENTLY acquires an AccessExclusiveLock, blocking all concurrent reads and writes for the duration of each index build. Use CREATE UNIQUE INDEX CONCURRENTLY / CREATE INDEX CONCURRENTLY for these.


Reply with @kilocode-bot fix it to have Kilo Code address this issue.

}

eligible++;
byConfidence[triage.confidence]++;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: Unvalidated triage.confidence from DB JSONB can produce NaN counters

The old code if (confidence && confidence in byConfidence) guarded against unexpected confidence values before incrementing. The new code byConfidence[triage.confidence]++ increments without validation. If the DB-stored JSON has a confidence value outside 'high' | 'medium' | 'low' (e.g., from a legacy version or corrupted data), byConfidence[someValue] is undefined, and undefined++ yields NaN, propagating corrupt count data to the UI. Consider restoring the in byConfidence guard.

Suggested change
byConfidence[triage.confidence]++;
byConfidence[triage.confidence] = (byConfidence[triage.confidence] ?? 0) + 1;

Reply with @kilocode-bot fix it to have Kilo Code address this issue.

})
.where(eq(security_findings.id, params.findingId));
if (sandbox) {
if (sandbox.isExploitable !== false || sandbox.suggestedAction !== 'dismiss') return;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: Sandbox analysis gating now blocks triage-based auto-dismiss as fallback

The new sandbox gate (lines 204-222) returns early for ALL sandbox-ful findings that don't meet the stricter dual condition isExploitable === false AND suggestedAction === 'dismiss'. Previously, findings with sandbox analysis that was not isExploitable === false would fall through to triage evaluation (line 225). Now, if sandbox ran but didn't explicitly recommend dismissal, triage is never evaluated as a fallback — the function returns without auto-dismissing, even if triage independently recommended dismissal. The auto-dismiss-service.ts comment block explicitly documents this as intentional ("After Tier 2 sandbox: dismiss only when analysis says not exploitable and recommends dismissal"), but the behavioral change from the old worker code path should be confirmed intentional.


Reply with @kilocode-bot fix it to have Kilo Code address this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant