Timeouts reach worker threads and async tasks; fix AsyncCallbackPool; revive stale unit tests - #260
Merged
b-macker merged 1 commit intoSep 28, 2026
Conversation
… revive stale unit tests --timeout on worker threads (async fn). The Python interrupt was a CPython pending call, which runs on the main thread only. Python inside an async fn (on another thread, VM) ran 20.8s against --timeout 3, and 40s when the loop caught every Exception. Threads running Python now register with a sequence number; a worker is sent PyThreadState_SetAsyncExc, re-sent until the execution that was running when the deadline fired leaves Python. The exception is naab.ExecutionTimeout, a BaseException subclass: an async exception lands at the next eval-breaker check, which inside `try: <loop> except Exception: pass` is almost always inside the try. (The recorded limit named the wrong path: tree-walker parallel groups run Python on the MAIN thread, and a probe on that path passes on old builds.) Script-wide vs local timeouts. Every timer set the process-wide flag and every arm/clear reset it, so a per-task timeout on a pool worker would stop the whole script and a worker arming one could erase the script's timeout after it fired. Only the outermost timeout of a run is script-wide now; nested and ScopedTimeout(ms, Local) set only their thread's flag. Async executors honour `timeout`: each pool task runs under a local ScopedTimeout (ShellWithTimeout: 52ms against 50ms). AsyncCallbackPool: std::launch::deferred ran callbacks only on .get(), so a full pool deadlocked and races never finished; lambdas captured `this`, so the pool freed wrappers mid-return and a timed-out detached thread kept running a member of a freed wrapper. State now lives in a shared_ptr held by every thread, and work starts eagerly. Pool tests: 30/30 repeated runs. Unit tests: 14 stale expectations rewritten to the rule each encodes (DIV-001, mandatory catch, ISS-036 first-definition-wins, math.abs float, newline and |> tokens, module list resolves). 580 pass; exclusions 30 -> 9. REST: F28 had no regression test; test_rest_hard_block_survives.sh pins it (reintroducing _exit(3) fails all three arms). Writing it found that builtin print() wrote to std::cout in both engines, bypassing the per-request capture stream: clients got empty output and prints landed in the server log. CLAUDE.md: F28/F31/F32 were still marked "live"; all three were fixed (#223, #218, #241). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
NAAb Governance Report
All governance checks passed! Generated by NAAb Governance Engine v4.0 |
Owner
Author
|
Local results on
Generated by Claude Code |
b-macker
marked this pull request as ready for review
September 28, 2026 08:28
b-macker
deleted the
claude/naab-inadmissible-action-prevention-4cmn1m
branch
September 28, 2026 08:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR covers the remaining open items from #259: timeouts in threads other than the main one, the async-callback defects, and the out-of-date unit tests.
It also adds a regression test for F28 (the REST server exiting on a HARD block, fixed in #223 but never tested). Writing that test surfaced a new REST bug:
print()output went to the server's log instead of the response.The three items listed as "live" in CLAUDE.md, F28, F31 and F32, are all already fixed on master (#223, #218, #241). Their CLAUDE.md entries are corrected here.
Changes
--timeout.async fn.--timeout 3: anasync fnPython busy loop on the VM ran its full 20 s, and 40 s when the loop catches every exception.PyThreadState_SetAsyncExc), re-sent until the run that was active when the deadline fired has left Python.naab.ExecutionTimeout, which subclassesBaseExceptionlikeKeyboardInterruptdoes. With a normalTimeoutError, the injected exception almost always landed inside the loop'stry, soexcept Exceptionkept swallowing it.ScopedTimeout(ms, Scope::Local), set only their own thread's flag.timeoutargument. Each pool task runs under a local timeout, so an overrun stops only that task.ShellWithTimeoutnow stops at 52 ms against a 50 ms budget.AsyncCallbackPooldeadlock and use-after-free fixed.std::launch::deferredran callbacks only when.get()was called, so a full pool deadlocked andexecuteRacenever finished.this, so the pool could free a wrapper while its callback was still returning, and a timed-out detached thread kept running a member of a freed wrapper.shared_ptrheld by every thread that runs it, and work starts immediately.print()fix. The built-inprint()wrote straight tostd::coutin both engines, bypassing the per-request capture thatio.*uses. Clients got emptyoutput, and every request's prints ended up in the server log, mixed across concurrent requests.catchis mandatory, re-registering a struct keeps the first definition (ISS-036),math.absreturns a float, the newline and|>token names, and every listed stdlib module resolves (the old test hard-coded a count of 13).known_failures.txtgoes from 30 entries to 9, with no hangs left.printnote) anddocs/unit-test-findings.md(new items 2e and 2f, section 3, and the CI section).Still a limit: Python blocked inside a C call (
time.sleep, a socket read) is only interrupted when that call returns. A bareexcept:can still catch the timeout, the same as withKeyboardInterrupt.Test Plan
tests/security/test_timeout_reach.sh, now 13 checks, all pass.async fn, with and without a loop that catches every exception.tests/api/test_rest_hard_block_survives.sh, 3 checks:printoutput._exit(3)put back temporarily, all three fail.run_unit_tests.sh: 580 pass, and all 9 remaining exclusions still fail for their recorded reasons. The previously hanging or crashing pool tests passed 30 of 30 repeated runs.🤖 Generated with Claude Code
https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
Generated by Claude Code