Skip to content

Make --timeout reach Python, http and nested blocks; run unit tests in CI - #259

Merged
b-macker merged 4 commits into
masterfrom
claude/naab-inadmissible-action-prevention-4cmn1m
Sep 28, 2026
Merged

b-macker merged 4 commits into
masterfrom
claude/naab-inadmissible-action-prevention-4cmn1m

Conversation

@b-macker

Copy link
Copy Markdown
Owner

Summary

--timeout is a flag the interpreter checks between steps. Work that runs inside the process without checking it only noticed the timeout after it finished on its own. This PR fixes three ways a script could run past --timeout. All three were measured with --timeout 3.

Case Before After
<<python>> busy loop (20 s) 20.1 s 3.06 s, both engines
Same loop wrapped in try: ... except Exception: pass ran until killed 3 s
while true after one <<javascript>> expression never stopped (killed from outside), both engines 3 s
while true after codegen.run(...) never stopped 3 s
http.get(url, {}, 0) to a host that never answers still connecting at 20 s 3 s

The JavaScript/codegen case was found while fixing the Python one. It is the most serious of the three: any script that ran a single JavaScript block had no time limit for the rest of the run.

This PR also runs the GoogleTest unit tests in CI, which was the rest of item 8.

Changes

  • Timeouts nest (resource_limits.h/.cpp):
    • The problem:
      • Each thread has one timer. The CLI and REST server wrap the whole run in a ScopedTimeout, and the JS, shell, subprocess and C++ executors wrap each block in another one.
      • Each inner scope replaced the timer with its own budget, and its destructor cleared the timer on exit.
      • The counter used to cancel timers was shared by the whole process. So a scope on another thread cancelled the main thread's timer: the tree-walker runs this JavaScript off the main thread, and on the REST server one request could cancel another request's timeout.
    • The fix:
      • ScopedTimeout now records the deadline already in force. It can only shorten it, never extend it, and restores it on exit.
      • The cancel counter is now per thread.
    • 0 now means "no limit of its own". Before, it set a zero-second timer that killed every subprocess as soon as it started under the UNRESTRICTED sandbox preset (defect Add tab completion to REPL #4 in docs/unit-test-findings.md).
    • codegen.run set and cleared the timeout by hand; it now uses ScopedTimeout.
  • Embedded Python (python_c_wrapper.c, python_interpreter_manager.cpp):
    • When the timeout fires, the timer thread schedules a callback inside CPython (Py_AddPendingCall) that raises TimeoutError in the running code.
    • The callback re-schedules itself while the timeout stands, so catching the exception doesn't keep the loop alive.
    • Before raising, it checks that the timeout is still in force, so a late callback can't hit a later, unrelated block.
    • On CPython 3.11, scheduling the callback alone did nothing (measured): Python doesn't notice a callback scheduled from another thread. The timer thread now briefly takes the Python lock (GIL), which makes the running thread check for it.
    • That lock step is skipped on Android, where taking the GIL from an outside thread is the crash this file already avoids.
  • http (http_impl.cpp):
    • A curl progress callback stops the transfer once the timeout fires. curl calls it at least once a second, including while connecting.
    • A timeout_ms of 0 or less (libcurl's "never time out") now uses the 30 s default.
  • Unit tests in CI (ci.yml, tests/unit/run_unit_tests.sh, tests/unit/known_failures.txt):
    • 559 tests pass.
    • The 30 known failures are listed, each with its reason from docs/unit-test-findings.md.
    • The runner fails if an unlisted test fails, if a listed test doesn't exist, or if a listed test starts passing, so the list has to shrink as fixes land.
    • Three tests that hang or crash (the AsyncCallbackPool deadlocks and use-after-free) are not run at all.
    • Two shell unit tests now print the error message when they fail.
  • Docs: docs/unit-test-findings.md (items 2a, 2c, 2d, and the CI section) and a CLAUDE.md note on nesting timeouts.

Known limits, recorded but not fixed:

  • Python code blocked inside a C call such as time.sleep isn't interrupted, because CPython only runs the callback between bytecodes. time.sleep(15) took 15 s on both the old and new code.
  • Python running on a worker thread isn't interrupted, because CPython only runs these callbacks on its main thread.

Test Plan

  • tests/security/test_timeout_reach.sh, 9 checks, all passing, measured on elapsed time rather than error text:
    • P-01/P-03: Python loops. P-04 is the control that Python blocks still finish normally under a timeout.
    • H-01: http. Skipped as unmeasurable if nothing on the network makes a connection hang.
    • N-js/N-cg: the loop after a JavaScript block or a codegen.run call must still be stopped.
    • On the old code, the control passes and the other 8 fail.
  • run_unit_tests.sh reports PASS locally in about 4 s. I checked all three ways it can fail by editing the list: dropping an entry, adding a test that passes, and adding a typo. Each made it fail.
  • The full suite, the error-message leak check and the VM/tree-walker differential run are still going locally; I'll post the results in a comment.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC


Generated by Claude Code

…n CI

--timeout is a flag the interpreter polls. Three ways a script outran it,
all measured with --timeout 3:

- A <<python>> busy loop ran to completion (20.1s). Embedded CPython had
  no interrupt path. The timer thread now queues a pending call that raises
  TimeoutError and re-queues itself while the timeout stands, so a broad
  except cannot swallow it. On CPython 3.11 queuing alone did nothing
  (Py_AddPendingCall from a non-main thread does not flag the eval loop);
  a brief GIL acquire makes the running thread recompute it. Not on
  Android (PyGILState_Ensure on a foreign thread is the CFI crash).
- One <<javascript>> expression, or codegen.run, removed --timeout for the
  rest of the script: every executor's ScopedTimeout re-armed the single
  per-thread timer and cleared it on exit, and the cancel counter was
  process-wide, so a scope on another thread cancelled the main thread's
  timer (and one REST request could cancel another's). ScopedTimeout now
  nests: it only tightens the deadline in force and restores it on exit.
  0 means "no limit of its own" -- it armed a zero-second timer that
  killed every subprocess under the UNRESTRICTED preset.
- http.get(url, {}, 0) to a SYN-dropping host was still connecting at 20s.
  A curl progress callback now aborts on timeout, and timeout_ms <= 0
  falls back to the 30s default.

Known limits, recorded not fixed: Python blocked in C (time.sleep) and
Python on worker threads are not interrupted.

tests/security/test_timeout_reach.sh: 9 arms. On the old code the control
passes and the other 8 fail.

naab_unit_tests now runs in CI via tests/unit/run_unit_tests.sh. 559 pass;
30 known failures are listed with their reasons in known_failures.txt. The
runner fails on an unlisted failure, on a listed test that does not exist,
and on a listed test that starts passing, so the list has to shrink as
fixes land. All three failure modes were checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
@github-actions

Copy link
Copy Markdown

NAAb Governance Report

Metric Count
Files checked 16
Passed 16
Failed 0

All governance checks passed!

Generated by NAAb Governance Engine v4.0

build-windows: H-01 stopped at 10s, not 3s. The progress-callback abort
was not reached during connect on the Windows curl, so the request ran to
curl's 10s connect timeout. Capping CURLOPT_TIMEOUT_MS (and through it the
connect timeout) at the time left before the ScopedTimeout deadline is
exact on every platform; the callback stays as the second line. With the
callback disabled locally, the cap alone stops H-01 at 3s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
…he sandbox

From building repo-sentinel in NAAb with a governed agent pipeline:

- dict.get() miss hints (F-002): a counting loop over 50 file paths
  printed 49 hint lines, most a "did you mean" pointing at a sibling path.
  The three copies of the hint code are now one helper that stops after 3
  hints per run and then says how to mark an expected miss (a default
  argument or d.has()), which already prints nothing.
- Ternary hint (F-004): it existed but fired only when '?' started an
  expression. In a dict literal, parentheses or a call argument, expect()
  reported "Expected '}'" (plus a missing-brace diagnosis) or "Expected
  ')'" instead. expect() now gives the if-expression hint for any stray '?'.
- Node under the sandbox (found checking F-005, whose argv report is node's
  own `--` handling, not NAAb): children got RLIMIT_AS at the memory
  budget, and node 22 needs 512-768 MB of address space just to start, so
  process.run("node", ...) died with "Failed to reserve virtual memory"
  under `elevated`. The budget now applies to RLIMIT_DATA, with an
  RLIMIT_AS ceiling (4x, at least 2 GB) as the backstop for shared
  anonymous memory, which RLIMIT_DATA does not count.

tests/parser/test_dogfood_hints.sh (10 arms: old build fails 5, its 5
controls pass) and tests/security/test_child_memory_limit.sh (4 arms: old
build fails M-01 only -- M-02/M-03 assert the limit still bounds private
and shared memory, M-04 is their control). Both pass on the new build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC

Copy link
Copy Markdown
Owner Author

Local results on the head commit 9ce76ce (Linux, Debug):

  • Full suite: 381 of 447 pass. The 2 unexpected failures are known issues in this container, caused by a stray /govern.json at its filesystem root: test_require_governance_gov007.sh and test_platform_fixes.sh. Both pass on CI.
  • Error-message leak check: 874/874 pass.
  • Engine comparison (VM vs tree-walker): 56/56 match.
  • Unit tests (run_unit_tests.sh): 559 pass, and all 27 known failures still fail for their recorded reasons.

Since the PR was opened, two commits were added from the repo-sentinel dogfooding findings:

  • Commit 306bd18:
    • Noisy dict.get hints are now capped per run.
    • The ternary hint now appears in every position, including inside dict literals, parentheses and call arguments.
    • Child processes are now limited on the memory they actually allocate instead of on reserved address space, with a 4× address-space backstop (minimum 2 GB). Before this, process.run("node", ...) could not start under elevated.
  • Commit 9ce76ce: registers the two new test suites so CI runs them.

On an earlier local run, test_validation_signal.sh G-03 failed once. That run shared the machine with a parallel compile. The suite then passed 29/29 twice in isolation, and it passed in this run and on CI, so I'm treating it as load-sensitive rather than broken.


Generated by Claude Code

@b-macker
b-macker marked this pull request as ready for review September 28, 2026 01:57
@b-macker
b-macker merged commit 1e8a7fe into master Sep 28, 2026
23 checks passed
@b-macker
b-macker deleted the claude/naab-inadmissible-action-prevention-4cmn1m branch September 28, 2026 01:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants