feat: make SEL rotation and retention limits env-tunable (#4993) - #4997
feat: make SEL rotation and retention limits env-tunable (#4993)#4997patrigao wants to merge 1 commit into
Conversation
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsIntent: let an operator size the SEL audit log's rotation and retention for their disk or compliance window — an ADDITION.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 7e3ee6e |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Raise-only is the right call; but the floor marker lives in a sandbox-VISIBLE directory, so its anti-erasure guarantee is tool-gate-only, weaker than documented. Watch
Suggestions
[DESIGN-REVIEWED] 7e3ee6e |
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/sel.py:810 -- Retention floor trusts a sandbox-writable marker Adjudication (Opus 4.8) — is blocking on each finding proportionate?The adjudicable block is empty; F1 is fenced (annotate-only). F1 — Harm rung: UNBOUNDED (destruction of tamper-evident audit history / operator-set retention ceiling). The floor marker's entire defense rests on the claim (sel.py:316-317) that the segment directory "sits on the SEL sensitive-path floor, so the audited agent's file tools cannot lower the marker." But that only fences the file-tool gate. At the OS sandbox layer [ADJUDICATION] 7e3ee6e total=0 uphold=0 downgrade=0 False positive or not applicable? A repository writer can comment: |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
7a1091f to
c71355f
Compare
Disposition: GPT blocking finding on
|
c71355f to
75d296f
Compare
|
Review dispositions for head
Same-span ledger — span Also carried in |
75d296f to
5a06de6
Compare
|
Pre-push review round 4 (local dual-model gate on the restructure), dispositions — new head Local Opus: PASS, 0 blockers, 2 advisories (both taken). Local GPT: 4 blockers, split verdict resolved finding-by-finding:
Span ledger Gates on |
|
5a06de6 to
281b0e4
Compare
|
Round-6 server GPT blocker — span This finding ( Per the same-span stall protocol (this span is at 5 consecutive server rounds: Everything outside this span on head |
Status: green except the escalated ruling — and Design Review dispositionsBoard on head Design Review (🟡 CONCERNS on
Every other concern on this PR has a prior written disposition (round 1–5 comments). Nothing further is actionable by the automation; handing the PR to the maintainer. |
|
Heads-up on your Design Review disposition #2: I closed #5032 as not planned, so the line reading "fixed. Filed #5032" now points at a closed tracker. The accurate record is that the lowering direction is an accepted scope cut ruled won't-fix, not a tracked follow-up -- worth editing that line so a later reader does not chase a dead reference. Reasoning is on #5032. Short version: the raise direction has a named beneficiary (compliance retention), the lowering direction does not, and serving it safely needs a keystone leaf plus a Settings surface plus an owner-session-authenticated writer -- disproportionate to a bound an operator can already move in source. Also relevant: the rotation product is a bound, not an allocation, so an install that never generates that volume never pays it. This does not block your PR. The raise-only design stands as shipped; only the disposition's wording is stale. One thing that may be useful while your escalation is still open, in case the ruling on Finding A goes toward an authenticated channel: a single operator-written file read from a fixed path by every SEL writer has no per-process divergence to defend against, so it removes the need for |
|
🤖 Kiro Crew [operator: bolichen97#bb3ad1ca]: Triage disposition — routing to The only red on head No automation will touch this PR while |
|
Concrete overlap note: #2497 and this PR both edit #2497 is being closed because its locking behavior is fully covered by merged #6081. Please rebase this branch onto current main/#6081, retain only the retention/segment configuration and pruning behavior, and replay its rotation tests against the merged lock path so pruning cannot bypass or split the chain-lock critical section. |
|
🤖 Kiro Crew [operator: dwu96#8c08bee0]: Triage re-assessment — this PR re-entered the triage queue today, so I re-verified it independently and reached the same conclusion as the standing disposition: it needs the author's and a maintainer's input, not an automated drive session. Moving What I verified on head
Decisions needed before this can move:
Order matters: if a maintainer intends to grant the override, that comment should land before the conflict resolution, because the override is pinned to a head that a rebase replaces. Add |
The segment size cap (32 MiB), keep-count (7), and retention window (365 days) were hardcoded module constants, fixing the audit-log disk ceiling at ~256 MiB for every install: a small-disk container could not lower it, and a compliance install could not raise the keep-count to match the stated retention window. Read three overrides once at SecurityEventLog construction — KIROCREW_SEL_MAX_BYTES, KIROCREW_SEL_KEEP, KIROCREW_SEL_RETENTION_DAYS — via a fail-soft _env_int() helper. Malformed, zero, or negative values clamp to the default with a warning that names the variable only (never the rejected value, which could be a mis-pasted secret). Plain os.environ reads on purpose: kiro_crew.config would be an import cycle. The module constants stay the single source of the defaults, and the rotation/prune sites now read the instance attributes. Closes #4993
821696d to
7e3ee6e
Compare
|
Rebased onto main Conflicts and resolutions:
Gates run locally on changed files: isort, flake8 clean; Please review the |
Problem / Motivation
SEL segment rotation limits are hardcoded source constants (
_SEGMENT_MAX_BYTES = 32 MiB,_SEGMENT_KEEP = 7), fixing the audit-log disk ceiling at ~256 MiB for every install with no runtime override (zeroenviron/getenvreads insel.py). This bites in both directions: a small-disk or container install cannot lower the ceiling, and a compliance-retention install cannot raise the keep-count to match_RETENTION_DAYS = 365— the count bound can silently discard history well inside the stated retention window.Why it matters
The audit log is the tamper-evident record of every tool invocation and permission decision. An operator who cannot size it for their disk either loses the machine to log growth (container installs) or loses audit history their retention policy requires them to keep (compliance installs). Both are operational failures of the one subsystem whose whole job is being reliably present.
What changed (motivation → approach → change)
Goal: operator tunability without a config-package integration (the
security.*config section was previously ruled out to avoid an import cycle; environment variables sidestep that entirely, and the issue explicitly scoped the request that way).Approach: three environment variables read once at
SecurityEventLogconstruction —KIROCREW_SEL_MAX_BYTES(default 33554432),KIROCREW_SEL_KEEP(default 7),KIROCREW_SEL_RETENTION_DAYS(default 365) — via a module-level, fail-soft_env_int()helper. Fail-soft because an audit logger that refuses to start converts a typo into lost audit coverage. The fallback/clamp warnings name the variable only, never the rejected value: a mis-pasted deployment value can contain a secret, and the SEL process log is itself a small exfiltration surface. Zero/negative values clamp to the default — treating0as never-rotate would remove the disk bound this subsystem exists to enforce.Change: the module constants stay the single source of the defaults; the rotation gate, the rotation size re-check, the segment count-retention sweep, and
prune()'s default window now read instance attributes. The overrides are raise-only and bounded (the erasure hazard and the enumeration bound were both found by pre-push/CI model review, see Review dispositions below):32intended as MiB, read as bytes, rotating per append on the hot path).KIROCREW_SEL_KEEPis additionally capped at 4092 (_SEGMENT_SCAN_CAP - 4) — segment enumeration examines at most 4096 directory entries, so a keep-count the cap cannot admit would make count retention silently stop; −4 reserves the transient segment, the rotation lock file, the retention-floor marker, and the floor lock file.KEEP,RETENTION_DAYS) are additionally process-shared — raise-only per process is not enough on a shared directory, because a child process with a scrubbed environment would enforce the compiled defaults and delete segments a sibling's raised policy protects. They are backed by a high-water marker (security_events.d/retention_floor.json) that is monotone by construction: the single write site re-reads the stored floor and writesmax(stored, requested)per field under the existing cross-process rotation lock — the same mutex every deletion sweep holds — acquired with a non-blocking try: a contended stamp defers outright and retries at the next construction, rotation, or sweep, so nothing on the construction path can ever block and a deferred stamp never degrades to an unlocked write. Concurrent processes raised via different variables therefore can never publish a lower field over each other's raise. One narrow window is deliberately accepted rather than closed with a blocking acquire: a contention-deferred first stamp is not yet visible to siblings until its retry lands, and the raised process itself never deletes below its own bounds. Construction takesmax(own, persisted)and eagerly creates the fenced segment directory when raised above the compiled defaults (so the stamp lands before any sibling can prune); both deletion sites re-read the marker before deleting (_enforce_segment_retention_locked,prune()'s default path), covering processes constructed before the raise, andprune()runs its floor refresh and both delete phases under that same rotation lock — when the lock is unusable on a degraded filesystem the sweep is skipped entirely, deferring deletion (safe direction; the next healthy sweep catches up). Reads are nofollow-fenced (a pre-planted link at the directory, marker, or lock path is never trusted or followed) and fd-fenced: only a regular, single-link file within a 4 KiB cap is trusted, opened non-blocking, so a planted FIFO, hard-linked alias, or oversized decoy reads as no floor and cannot stall an event-loop-thread construction. Max-bytes needs no marker — deletion is governed solely by keep/retention, so a divergent smaller size cap only rotates earlier, never deletes more.prune()guards its cutoff computation againstOverflowError— a retention window past the representable date range means "keep everything", so the daily sweep prunes nothing (with a warning) instead of failing.Scope consequence: of the issue's two directions, this PR serves the widen direction (compliance retention) fully. The narrow direction (small-disk installs lowering the ceiling) is deliberately not servable via environment variables at all — any lowerable env knob reintroduces the audit-erasure channel above. Narrowing today means editing the source constants (an operator-authenticated act); a config-file key would be the follow-up shape if wanted, and needs the known import cycle resolved first.
Docs updated in the same commit per the spec-management rule:
docs/system-specs/modules/sel.md(bounds, fail-soft contract, per-process semantics) and the operator-facing env-var table indocs/guides/install.md.Review dispositions (pre-push dual model review
gpt-5.6-sol+claude-opus-5verified by a focused checker, plus CI review rounds):KIROCREW_SEL_KEEP=1plus a smallMAX_BYTESon any child CLI had the trusted writer rotate the shared log early and unlink protected segments, with no forensic marker → all three overrides made raise-only (floor = compiled default). Regression test reproduces the exact PoC env and asserts every lowering attempt clamps up with a visible warning. The demanded full revert was disproportional: the erasure channel exists only for lowering, and raise-only preserves the issue's compliance use case._SEGMENT_KEEP_MAXreserve widened.prune()fences refresh-and-delete under that same lock, and the scan-cap reserve reverted to-3(keep cap 4093).O_NONBLOCK|O_NOFOLLOWopen, fstat gate: regular file,st_nlink == 1, 4 KiB cap; three regression tests). GPT's other two rebutted on the PR: extending the floor to max-bytes conflates rotation cadence with deletion bounds, and "defer construction until the floor is durably visible" contradicts the round-3 no-blocking invariant and the module's fail-soft construction contract — the window is documented as deliberately accepted instead._SEGMENT_KEEP_MAXclamp._SEGMENT_MIN_BYTESremoved.timedeltain the daily prune →OverflowErrorguard returning 0.using the defaultsuffix.install.mdenvironment-variable table.sel.mdstates the per-process semantics and the marker's guarantees.Note for maintainers: issue #4992 is being fixed concurrently in the same file (
fix/sel-nofollow-open-4992); this change is confined to the constants region, constructor, rotation/prune sites, and the SEL test module to minimize the conflict surface. Whichever PR lands second must rebase onto main and re-run gates.Tests
All in
test/test_sel.py::TestEnvOverrides(plus one adapted existing test):test_defaults_unchanged_when_vars_unset— behavior at defaults is byte-identical to the constants.test_lowering_attempts_clamp_to_the_compiled_defaults— the review PoC (MAX_BYTES=65536 KEEP=1 RETENTION_DAYS=1): all three clamp up to the defaults with visible warnings.test_max_bytes_raise_reaches_the_rotation_gate— an env raise governs the actual rotation trigger (proven against a patched-down default so the raise is genuine).test_keep_raise_controls_retained_segment_count— a raised keep-count retains exactly that many closed segments.test_retention_days_raise_reaches_prune—prune()with no args picks up the widened window (an entry inside the raise but outside the default survives).test_explicit_keep_days_still_wins_over_the_override— an explicit argument beats the env var.test_malformed_values_fall_back_with_a_redacting_warning(parametrized: non-integer,0,-5,1.5) — default kept, warning present, rejected value never echoed.test_each_variable_falls_back_independently— one bad variable does not poison the others.test_keep_count_clamps_below_the_scan_cap— an over-cap keep clamps below the enumeration scan cap with a warning.test_scrubbed_env_child_cannot_lower_a_raised_deletion_floor— the round-2 PoC verbatim: operator raises KEEP=100/RETENTION=400, a scrubbed-env process on the same directory inherits both from the marker.test_a_child_constructed_before_the_raise_prunes_with_the_raised_floor— the reverse ordering: a process constructed before the raise picks the floor up at its next sweep (deletion-time re-read).test_the_floor_merge_is_per_field_monotone— differently-raised concurrent writers cannot clobber each other's field; a no-raise persist never rewrites the marker.test_a_planted_marker_link_is_not_trusted_and_is_replaced— a pre-planted marker symlink is ignored on read (no value inflation, no stamp suppression) and replaced by a real file; the link target is untouched.test_a_damaged_floor_marker_fails_soft— a corrupt marker never stops construction; defaults apply.test_a_contended_stamp_defers_instead_of_writing— the round-3 contract: while another process holds the rotation lock the stamp returns without touching the marker (no fail-open), and lands after release.test_a_planted_fifo_marker_neither_blocks_nor_is_trusted— a FIFO at the marker path returns no floor promptly (thread-bounded; a blocking open would hang it).test_an_oversized_marker_decoy_is_ignored— a marker past the 4 KiB cap is rejected instead of parsed.test_a_hard_linked_marker_is_not_trusted—st_nlink != 1reads as no floor.test_a_retention_window_past_the_date_range_prunes_nothing— 10⁹ days returns 0 instead of raising.The root
conftest.pyhost floor clears the three variables suite-wide so a developer-exported override cannot move the limits the SEL tests assert against.Manual verification
N/A — unit coverage sufficient: the overrides are pure construction-time reads exercised end-to-end (rotation, retention, prune) by the tests above; there is no UI or external-service path.
Screenshots / video
Why no screenshot: backend-only change (audit-log rotation limits); no rendered surface is touched.
Related Issues
Closes #4993
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)