Skip to content

supervisor: watch ktimerd and edr_daemon, and restore priority on restart - #123

Closed
douglasmun wants to merge 1 commit into
fix/double-fault-task-gatefrom
feat/supervisor-watch-edr-ktimerd
Closed

supervisor: watch ktimerd and edr_daemon, and restore priority on restart#123
douglasmun wants to merge 1 commit into
fix/double-fault-task-gatefrom
feat/supervisor-watch-edr-ktimerd

Conversation

@douglasmun

Copy link
Copy Markdown
Owner

Stacked on #122. Base is fix/double-fault-task-gate, so this diff shows only the supervisor work. Merge #122 first; this PR will retarget to main automatically.

What

supervisor_watch() covered knetd only. ktimerd and the EDR daemon were unsupervised, and neither failure is visible: timer_softirq_run() drives the TCP timers, the DHCP renewal, the EDR periodic hooks and the CSPRNG reseed, so a dead ktimerd degrades all four silently. Capacity raised 4 → 8.

The latent bug this surfaced

supervisor_restart() never restored priority. task_create_kernel() assigns PRIORITY_NORMAL unconditionally, so a restarted task came back demoted.

This was invisible while knetd was the only watched task — knetd is PRIORITY_NORMAL, so restoring its priority and failing to restore it produce identical output. Both newly-watched tasks are PRIORITY_HIGH, so the latent bug would have started biting immediately: the task is alive and the restart counter rises, so every status surface reports a healthy recovery, and the only symptom is the timer bottom-half running behind interactive work.

Fixed by recording the priority in supervisor_watch() and restoring it in supervisor_restart() before scheduler_add_task(). Set inside the restart function rather than at the call site, for the same reason the rate limit lives there: so no future restart path can skip it.

edr_daemon_main() loses static (the supervisor stores a void(*)(void)); kernel.c still starts the daemon via edr_daemon_start(). It is watched after edr_daemon_start() returns rather than beside knetd/ktimerd, because the daemon does not exist until then.

verify-supervisor.sh step 6 was a FALSE PASS

The new leg indexed the last sample with "${KT_PRI_SAMPLES[-1]}". macOS ships bash 3.2, which has no negative array subscripting, and under set -u that raises bad array subscript / unbound variable rather than yielding empty. That aborted the two assignments, leaving the operands unset; the -ne comparisons then failed on stderr without stopping the script, so the step printed its OK line and the harness printed RESULT: PASS while grading nothing.

The kernel was correct throughout — the serial log showed ktimerd restarting at priority 3 as intended — so the only evidence was a stderr line nobody read. That is the exact false-pass shape this suite exists to prevent.

Fixed by computing the last index explicitly and guarding the parse: all four fields must be non-empty and numeric before any is compared, because -ne treats an empty operand as a syntax error and continues.

Negative-controlled, since a fix that isn't falsified is how the false pass shipped

Replayed against the real serial log:

Mutation Result
simulated demotion (post pri 3→2) FAIL DEMOTED 3 -> 2
ktimerd never restarted (same PID) FAIL not restarted (vacuous)
broken ps -l column layout FAIL expected 2 rows, got 1
wrong baseline (pre pri 3→2) FAIL baseline not 3
unmodified correct-kernel log OK

Fault injection uses ktimerd, not knetd, for the reason above: knetd's PRIORITY_NORMAL cannot witness a demotion. The victim clears its own CAP_UNKILLABLE and calls task_terminate(self_pid) — not task_exit(), which is inert for scheduler-run tasks — so the capability check itself stays unmodified.

Verified

  • make clean + rebuild: 0 warnings under -Werror
  • Full verify-supervisor.sh run PASSes steps 0–6, step 6 reporting pre PID=17356 pri=3 -> post PID=47438 pri=3 — restart proven by the PID change, priority proven unchanged

🤖 Generated with Claude Code

…tart

supervisor_watch() covered knetd only. ktimerd and the EDR daemon were
unsupervised, and neither failure is visible: timer_softirq_run() drives the
TCP timers, the DHCP renewal, the EDR periodic hooks and the CSPRNG reseed, so
a dead ktimerd degrades all four silently. Capacity raised 4 -> 8.

Found while wiring it: supervisor_restart() never restored priority.

task_create_kernel() assigns PRIORITY_NORMAL unconditionally, so a restarted
task came back demoted. This was invisible while knetd was the only watched
task -- knetd IS PRIORITY_NORMAL, so restoring its priority and failing to
restore it produce identical output. Both newly-watched tasks are
PRIORITY_HIGH, so the latent bug would have started biting immediately: the
task is alive and the restart counter rises, so every status surface reports a
healthy recovery, and the only symptom is the timer bottom-half running behind
interactive work.

Fixed by recording the priority in supervisor_watch() and restoring it in
supervisor_restart() before scheduler_add_task(). Set inside the restart
function rather than at the call site, for the same reason the rate limit lives
there: so no future restart path can skip it.

edr_daemon_main() loses `static` (the supervisor stores a void(*)(void));
kernel.c still starts the daemon via edr_daemon_start(). It is watched after
edr_daemon_start() returns rather than beside knetd/ktimerd, because the daemon
does not exist until then.

verify-supervisor.sh step 6: was a FALSE PASS, now grades.

The new leg indexed the last sample with "${KT_PRI_SAMPLES[-1]}". macOS ships
bash 3.2, which has no negative array subscripting, and under `set -u` that
raises "bad array subscript" / "unbound variable" rather than yielding empty.
That aborted the two assignments, leaving the operands unset; the `-ne`
comparisons then failed on stderr WITHOUT stopping the script, so the step
printed its "OK" line and the harness printed RESULT: PASS while grading
nothing.

The kernel was correct throughout -- the serial log showed ktimerd restarting
at priority 3 as intended -- so the only evidence was a stderr line nobody read.
That is the exact false-pass shape this suite exists to prevent.

Fixed by computing the last index explicitly and guarding the parse: all four
fields must be non-empty and numeric before any is compared, because `-ne`
treats an empty operand as a syntax error and CONTINUES.

Negative-controlled against the real serial log, since a fix that is not
falsified is how the false pass shipped:
  - simulated demotion (post pri 3->2)      -> FAIL "DEMOTED 3 -> 2"
  - ktimerd never restarted (same PID)      -> FAIL "not restarted (vacuous)"
  - broken `ps -l` column layout            -> FAIL "expected 2 rows, got 1"
  - wrong baseline (pre pri 3->2)           -> FAIL "baseline not 3"
  - unmodified correct-kernel log           -> OK

Fault injection uses ktimerd, not knetd, for the reason above: knetd's
PRIORITY_NORMAL cannot witness a demotion. The victim clears its OWN
CAP_UNKILLABLE and calls task_terminate(self_pid) -- not task_exit(), which is
inert for scheduler-run tasks -- so the capability check itself stays unmodified.

Verified: make clean + rebuild 0 warnings under -Werror; full
verify-supervisor.sh run PASSes all of steps 0-6, with step 6 reporting
pre PID=17356 pri=3 -> post PID=47438 pri=3 (restart proven by the PID change,
priority proven unchanged).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CEkhAhgTxbE5TgifyYf8v4
@douglasmun

Copy link
Copy Markdown
Owner Author

Superseded by #129 — same commit rebased onto main. GitHub auto-closed this PR when its base branch fix/double-fault-task-gate was deleted on merging #122, and a closed PR's base cannot be retargeted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant