verify: keep a failing run's serial log instead of deleting it - #131
Merged
Conversation
verify-dns-noprivinsn.sh, verify-dns-rx-counters.sh and verify-netd-arbitration.sh put SERIAL="$WORK/serial.log" inside a mktemp -d dir and rm -rf'd $WORK from an EXIT trap. That trap fires on FAILURE too, so the one artifact that explains a failure was destroyed before it could be read. This is not hypothetical. Issue #126 is a boot panic that reproduces only under some harnesses, and PR #128 added exactly the diagnostics needed to identify it -- faulting PID/name, kernel ESP, stack words -- printed to the serial log these traps deleted. A harness that discards its own evidence turns a diagnosable panic back into a mystery. Adds verify/preserve-serial.sh: on a non-zero exit it copies the log to $TMPDIR/tinyos-failed-<harness>.serial.log and says where; on success it does nothing, so passing runs still leave no litter. Keyed on the EXIT STATUS rather than a parsed verdict, because a run killed by set -e, a QEMU timeout or a typist TIMEOUT never prints a verdict at all -- and those are precisely the runs worth keeping. One shared file rather than three copies, for the reason verify/edr-rejoin.sh documents. TRAP $? CLOBBER: an EXIT trap's $? survives only until the trap's first command runs. All three call sites run cleanup_qemu first, which SUCCEEDS and resets $? to 0, so a helper reading $? itself preserved nothing on exactly the failing runs that needed it. The status is now captured as the trap's first action and passed in explicitly. My unit tests all passed against this bug because they called the helper first in the trap -- the one position where $? is intact; only an end-to-end test in the real trap shape caught it. verify/preserve-serial-test.sh covers 9 cases (pure shell, no guest): preserve on failure, exit code passthrough, preserved content, nothing on success, empty log skipped, missing source not aborting the trap, and test 5 as a NEGATIVE CONTROL asserting the $? clobber is real -- without it test 1 could silently stop testing anything. Verified to bite: reverting the helper to the $?-reading form fails 2 of 9; restored, 9/9. verify-dns-noprivinsn.sh re-run end to end with the change: RESULT: PASS. Gated in CI alongside edr-rejoin-test.sh. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CEkhAhgTxbE5TgifyYf8v4
This was referenced Aug 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three QEMU harnesses (
verify-dns-noprivinsn.sh,verify-dns-rx-counters.sh,verify-netd-arbitration.sh) putSERIAL="$WORK/serial.log"inside amktemp -ddir andrm -rf'd$WORKfrom an EXIT trap. That trap fires on failure too, so the one artifact explaining a failure was destroyed before anyone could read it.Not hypothetical: issue #126 is a boot panic reproducing only under some harnesses, and PR #128 added exactly the diagnostics needed to identify it — faulting PID/name, kernel ESP, stack words — printed to the serial log these traps deleted. A harness that discards its own evidence turns a diagnosable panic back into a mystery.
The fix
verify/preserve-serial.sh: on non-zero exit, copy the log to$TMPDIR/tinyos-failed-<harness>.serial.logand say where; on success, do nothing (passing runs leave no litter).Keyed on the exit status, not a parsed verdict — a run killed by
set -e, a QEMU timeout, or a typist TIMEOUT never prints a verdict, and those are precisely the runs worth keeping. One shared file rather than three copies, for the reasonverify/edr-rejoin.shdocuments.The trap
$?clobber (worth reading)An EXIT trap's
$?survives only until the trap's first command runs. All three call sites runcleanup_qemufirst, which succeeds and resets$?to 0 — so a helper reading$?itself preserved nothing on exactly the failing runs that needed it, silently.The status is now captured as the trap's first action and passed in:
My unit tests all passed against this bug, because they called the helper first in the trap — the one position where
$?is intact. Only an end-to-end test in the real trap shape caught it.Verification
verify/preserve-serial-test.sh— 9 cases, pure shell, no guest. Test 5 is a negative control asserting the$?clobber is real; without it, test 1 could silently stop testing anything.$?-reading form fails 2 of 9; restored, 9/9.verify-dns-noprivinsn.shre-run end to end with the change: RESULT: PASS.edr-rejoin-test.sh.🤖 Generated with Claude Code
https://claude.ai/code/session_01CEkhAhgTxbE5TgifyYf8v4