Revive naab_unit_tests; fix SafeRegex timeout and nested-quantifier bypass - #257
Merged
b-macker merged 4 commits intoSep 26, 2026
Merged
Conversation
Nothing built naab_unit_tests, so five of its nineteen files had silently stopped compiling after the shared_ptr<Value> -> NaabVal migration. Ported with small boundary adapters that leave every assertion as written: - stdlib_test: callModule() converts args to NaabVal and the result back; makeArray builds the vector<NaabVal> list alternative Value now holds. - interpreter_test / struct_test: .toLegacy() at getResult()/getField(). - ffi_async_callback_test: callbacks return NaabVal. - polyglot_async_test: block arg vectors are vector<NaabVal>; the fixture now establishes a sandbox policy. Without one, effectiveConfig() denies (fail-closed, correct), which is why 12 of its tests could not pass. - rust_ffi_test: the hand-written prototypes named the pre-NaabVal signatures. valueToFfi failed to link; ffiToValue differs only in return type, which a free function's mangled name does not encode, so it would have linked and returned a NaabVal read as a shared_ptr. The binary builds and ~510 of 632 tests pass. The rest are classified (stale vs real defect) in the PR; wiring into CI is a separate step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
NAAb Governance Report
All governance checks passed! Generated by NAAb Governance Engine v4.0 |
Found by following the one hanging unit test into the real binary. Two
defects combined:
1. executeWithTimeout() threw after wait_for() timed out, but a std::async
future's destructor blocks until the task finishes, so the throw waited
for the runaway regex. Measured: 30s against a 1s budget on a 28-char
input, and --timeout 5 fired at 30.5s -- it could not preempt it either.
The work now runs on a detached thread that owns copies of its inputs
(the old lambdas captured by reference, so abandoning them would have
been a use-after-free), and the caller returns at the deadline.
std::regex cannot be interrupted, only abandoned, so timed-out workers
are capped at 4 and further regex work is refused beyond that.
2. hasNestedQuantifiers() was one regex that required the quantified
group's ')' to follow its inner quantifier directly, so ((a+))+b passed
validation. Replaced with a scanner that tracks groups, skipping escapes
and character classes. On an 18-pattern table the old check scored
10/18: 5 catastrophic patterns missed and 3 ordinary ones refused,
including (?:ab)+.
tests/security/test_regex_timeout_bound.sh asserts wall time, not error
text (the broken build produced the same text 29s late). Against the old
implementation 4 of 7 arms fail; its 3 controls pass on both builds.
Also wraps the parser/interpreter unit-test sources in main {}: all 96 of
those failures were the top-level parse gate, and rerunning behind it left
4 (language rules, not bugs). Unit binary: 558/587 pass, no hangs besides
the known AsyncCallbackPool deadlock. Findings, reachability and the
remaining open items: docs/unit-test-findings.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
build-windows stalled in "CLI tests - shell suites" for 50+ minutes: no step timeout fired and no logs were uploaded, the failure mode windows.yml documents for a wedged runner. It is the second stall in that phase, and this suite is new in it. std::regex cannot be interrupted, so a timed-out worker is abandoned, not killed. The budget arms used a 31-char input -- minutes to hours of work per worker, five workers across B-01 and C-01. On Linux they die with the process; if MinGW instead holds the process open for running threads, several 100%-CPU workers per process would starve the runner, matching every symptom. Unproven without logs, but no test should leave that much work behind on any platform. The budget arms now use n=25: several times the 1s budget even when optimised (raw -O0: n=24 takes 6.1s), but finite in seconds. B-01's bound tightens from 10s to 4s so it still discriminates: the old implementation takes 13s at this size and fails, as does C-01 (38s); the three controls pass on both builds. If exit does wait for abandoned workers on Windows, B-01 now fails with a log instead of wedging the runner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
build-linux on 0c212fd: C-01 measured 6 timeouts and 0 refusals in 6s. At n=25 the optimised CI build finishes each abandoned search in under ~4s, so with one call per second the stuck-worker count never reached the cap of 4 and the refusal path was never exercised -- a test-sizing error, not a product one (the cap engaged on the slower local build). C-01 now uses n=28, ~8x the work: roughly 8-32s per abandoned worker on that build, comfortably alive across the 6 calls, still finite. B-01 keeps n=25, where the aim is the opposite -- abandoned work that ends quickly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
b-macker
marked this pull request as ready for review
September 26, 2026 22:52
b-macker
deleted the
claude/naab-inadmissible-action-prevention-4cmn1m
branch
September 26, 2026 22:52
b-macker
restored the
claude/naab-inadmissible-action-prevention-4cmn1m
branch
September 26, 2026 23:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
naab_unit_tests(GoogleTest, 19 files, 589 tests) was declared in CMake but no CI job or script ever built it, and 5 of its files had quietly stopped compiling. Getting it running again surfaced one hang. Following that hang into the realnaab-langbinary showed that a singleregex.matchescall can run past every configured time limit. This PR revives the suite, fixes that defect, and writes up every other failure indocs/unit-test-findings.md.Changes
src/utils/safe_regex.cpp).executeWithTimeout()threw oncewait_forexpired, but astd::asyncfuture's destructor blocks until the task finishes, so the throw only happened after the runaway regex completed. On a 28-character input that was 30 s against a 1 s budget;--timeout 5also fired only at 30.5 s.((a+))+bpassed validation. Scored on an 18-pattern table, the old check got 10/18 (5 catastrophic patterns missed, and 3 ordinary ones like(?:ab)+wrongly refused). The new scanner gets 18/18.NaabValAPI through small adapters; every assertion is unchanged. Details:rust_ffi_test.cpphad stale hand-written prototypes.ffiToValuediffered from the real function only in return type, which isn't part of the mangled name, so it linked and would have read aNaabValas ashared_ptr.main {}. All 96 of those failures were the top-level parse error. With the wrap, 4 remain, and all 4 are language rules rather than bugs.tests/security/test_regex_timeout_bound.sh, registered inrun-all-tests.sh.docs/unit-test-findings.md: every failure traced to a cause, with reachability checked against the real binary.Test Plan
test_regex_timeout_bound.shpasses 7/7. It checks elapsed time, not just the error text, because the broken build printed the same timeout message 29 s late. Run against the old implementation, 4 arms fail and the 3 controls pass.((a+))+bis now rejected immediately.(a|a)+bon a 31-character input stops at 1.0 s; rawstd::regexneeds minutes on that input..naabtests and 8 governance/security suites that use regex all pass.AsyncCallbackPooldeadlock (2 tests, excluded from that run).Related Issues
Portability item 8. Still open, all recorded in the findings doc:
--timeout; a<<python>>busy loop ran its full 20 s against--timeout 3.timeoutargument,AsyncCallbackPooldeadlocks, a zero-second timer in the unrestricted preset, andnaab::Contexthas no polyglot executors registered.🤖 Generated with Claude Code
https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC