Skip to content

Revive naab_unit_tests; fix SafeRegex timeout and nested-quantifier bypass - #257

Merged
b-macker merged 4 commits into
masterfrom
claude/naab-inadmissible-action-prevention-4cmn1m
Sep 26, 2026
Merged

b-macker merged 4 commits into
masterfrom
claude/naab-inadmissible-action-prevention-4cmn1m

Conversation

@b-macker

@b-macker b-macker commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Summary

naab_unit_tests (GoogleTest, 19 files, 589 tests) was declared in CMake but no CI job or script ever built it, and 5 of its files had quietly stopped compiling. Getting it running again surfaced one hang. Following that hang into the real naab-lang binary showed that a single regex.matches call can run past every configured time limit. This PR revives the suite, fixes that defect, and writes up every other failure in docs/unit-test-findings.md.

Changes

  • SafeRegex now stops at its time budget (src/utils/safe_regex.cpp).
    • Before: executeWithTimeout() threw once wait_for expired, but a std::async future's destructor blocks until the task finishes, so the throw only happened after the runaway regex completed. On a 28-character input that was 30 s against a 1 s budget; --timeout 5 also fired only at 30.5 s.
    • Now the work runs on a detached thread that owns copies of its inputs. The old lambdas captured their inputs by reference, so detaching those would have been a use-after-free.
    • A regex can't be interrupted, only abandoned, so at most 4 timed-out workers can be left running. Past that cap, new regex calls are refused with an error.
  • Nested-quantifier detection is a group-tracking scanner. The old single regex check was defeated by redundant parentheses: ((a+))+b passed validation. Scored on an 18-pattern table, the old check got 10/18 (5 catastrophic patterns missed, and 3 ordinary ones like (?:ab)+ wrongly refused). The new scanner gets 18/18.
  • Unit tests compile again. Five files are ported to the NaabVal API through small adapters; every assertion is unchanged. Details:
    • rust_ffi_test.cpp had stale hand-written prototypes. ffiToValue differed from the real function only in return type, which isn't part of the mangled name, so it linked and would have read a NaabVal as a shared_ptr.
    • The polyglot fixture now sets up a sandbox policy. Without one, the runtime correctly refuses every block (fail-closed).
    • Parser and interpreter test sources are wrapped in main {}. All 96 of those failures were the top-level parse error. With the wrap, 4 remain, and all 4 are language rules rather than bugs.
  • New suite tests/security/test_regex_timeout_bound.sh, registered in run-all-tests.sh.
  • docs/unit-test-findings.md: every failure traced to a cause, with reachability checked against the real binary.

Test Plan

  • test_regex_timeout_bound.sh passes 7/7. It checks elapsed time, not just the error text, because the broken build printed the same timeout message 29 s late. Run against the old implementation, 4 arms fail and the 3 controls pass.
  • ((a+))+b is now rejected immediately. (a|a)+b on a 31-character input stops at 1.0 s; raw std::regex needs minutes on that input.
  • No regressions from the stricter validator: 40 regex-using .naab tests and 8 governance/security suites that use regex all pass.
  • Unit binary: 558 of 587 pass, the whole run takes 2.4 s, and the only hang left is the known AsyncCallbackPool deadlock (2 tests, excluded from that run).
  • Error-message leak check 874/0; coverage-visibility gate 4/4.

Related Issues

Portability item 8. Still open, all recorded in the findings doc:

  • Embedded Python ignores --timeout; a <<python>> busy loop ran its full 20 s against --timeout 3.
  • CI doesn't run the unit binary yet; that will need an exclusion list.
  • About 108 stale tests still need rewriting.
  • Lower priority: the async executors ignore their timeout argument, AsyncCallbackPool deadlocks, a zero-second timer in the unrestricted preset, and naab::Context has no polyglot executors registered.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC

Nothing built naab_unit_tests, so five of its nineteen files had silently
stopped compiling after the shared_ptr<Value> -> NaabVal migration. Ported
with small boundary adapters that leave every assertion as written:

- stdlib_test: callModule() converts args to NaabVal and the result back;
  makeArray builds the vector<NaabVal> list alternative Value now holds.
- interpreter_test / struct_test: .toLegacy() at getResult()/getField().
- ffi_async_callback_test: callbacks return NaabVal.
- polyglot_async_test: block arg vectors are vector<NaabVal>; the fixture
  now establishes a sandbox policy. Without one, effectiveConfig() denies
  (fail-closed, correct), which is why 12 of its tests could not pass.
- rust_ffi_test: the hand-written prototypes named the pre-NaabVal
  signatures. valueToFfi failed to link; ffiToValue differs only in return
  type, which a free function's mangled name does not encode, so it would
  have linked and returned a NaabVal read as a shared_ptr.

The binary builds and ~510 of 632 tests pass. The rest are classified
(stale vs real defect) in the PR; wiring into CI is a separate step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
@github-actions

Copy link
Copy Markdown

NAAb Governance Report

Metric Count
Files checked 16
Passed 16
Failed 0

All governance checks passed!

Generated by NAAb Governance Engine v4.0

Found by following the one hanging unit test into the real binary. Two
defects combined:

1. executeWithTimeout() threw after wait_for() timed out, but a std::async
   future's destructor blocks until the task finishes, so the throw waited
   for the runaway regex. Measured: 30s against a 1s budget on a 28-char
   input, and --timeout 5 fired at 30.5s -- it could not preempt it either.
   The work now runs on a detached thread that owns copies of its inputs
   (the old lambdas captured by reference, so abandoning them would have
   been a use-after-free), and the caller returns at the deadline.
   std::regex cannot be interrupted, only abandoned, so timed-out workers
   are capped at 4 and further regex work is refused beyond that.

2. hasNestedQuantifiers() was one regex that required the quantified
   group's ')' to follow its inner quantifier directly, so ((a+))+b passed
   validation. Replaced with a scanner that tracks groups, skipping escapes
   and character classes. On an 18-pattern table the old check scored
   10/18: 5 catastrophic patterns missed and 3 ordinary ones refused,
   including (?:ab)+.

tests/security/test_regex_timeout_bound.sh asserts wall time, not error
text (the broken build produced the same text 29s late). Against the old
implementation 4 of 7 arms fail; its 3 controls pass on both builds.

Also wraps the parser/interpreter unit-test sources in main {}: all 96 of
those failures were the top-level parse gate, and rerunning behind it left
4 (language rules, not bugs). Unit binary: 558/587 pass, no hangs besides
the known AsyncCallbackPool deadlock. Findings, reachability and the
remaining open items: docs/unit-test-findings.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
@b-macker b-macker changed the title Make naab_unit_tests compile again (item 8, part 1) Revive naab_unit_tests; fix SafeRegex timeout and nested-quantifier bypass Sep 26, 2026
build-windows stalled in "CLI tests - shell suites" for 50+ minutes: no
step timeout fired and no logs were uploaded, the failure mode windows.yml
documents for a wedged runner. It is the second stall in that phase, and
this suite is new in it.

std::regex cannot be interrupted, so a timed-out worker is abandoned, not
killed. The budget arms used a 31-char input -- minutes to hours of work
per worker, five workers across B-01 and C-01. On Linux they die with the
process; if MinGW instead holds the process open for running threads,
several 100%-CPU workers per process would starve the runner, matching
every symptom. Unproven without logs, but no test should leave that much
work behind on any platform.

The budget arms now use n=25: several times the 1s budget even when
optimised (raw -O0: n=24 takes 6.1s), but finite in seconds. B-01's bound
tightens from 10s to 4s so it still discriminates: the old implementation
takes 13s at this size and fails, as does C-01 (38s); the three controls
pass on both builds. If exit does wait for abandoned workers on Windows,
B-01 now fails with a log instead of wedging the runner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
build-linux on 0c212fd: C-01 measured 6 timeouts and 0 refusals in 6s.
At n=25 the optimised CI build finishes each abandoned search in under
~4s, so with one call per second the stuck-worker count never reached the
cap of 4 and the refusal path was never exercised -- a test-sizing error,
not a product one (the cap engaged on the slower local build).

C-01 now uses n=28, ~8x the work: roughly 8-32s per abandoned worker on
that build, comfortably alive across the 6 calls, still finite. B-01 keeps
n=25, where the aim is the opposite -- abandoned work that ends quickly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
@b-macker
b-macker marked this pull request as ready for review September 26, 2026 22:52
@b-macker
b-macker merged commit 298c9d7 into master Sep 26, 2026
24 checks passed
@b-macker
b-macker deleted the claude/naab-inadmissible-action-prevention-4cmn1m branch September 26, 2026 22:52
@b-macker
b-macker restored the claude/naab-inadmissible-action-prevention-4cmn1m branch September 26, 2026 23:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants