diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 50ff34d14..005bc3b99 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -46,6 +46,16 @@ jobs: # binary and reports UNMEASURABLE rather than passing without it. ninja -C build naab-lang libnaab naab-verify-audit naab-gov -j$(nproc) + # naab_unit_tests (GoogleTest) was the same kind of unbuilt target: five + # of its nineteen files had stopped compiling before anyone ran it. + # Exclusions are in tests/unit/known_failures.txt, each tied to + # docs/unit-test-findings.md; the runner also fails if an excluded test + # starts passing, so the list cannot outlive its reasons. + - name: Unit tests + run: | + ninja -C build naab_unit_tests -j$(nproc) + bash tests/unit/run_unit_tests.sh + - name: Run tests run: bash run-all-tests.sh diff --git a/CLAUDE.md b/CLAUDE.md index e4db2238b..7a4fc6350 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,6 +25,9 @@ bash run-all-tests.sh # from the repo root # Security leak check — 874 checks, 0 failures bash tests/security/test_error_msg_leaks.sh + +# GoogleTest unit tests (also in CI) — exclusions in tests/unit/known_failures.txt +cd build && make naab_unit_tests -j4 && cd .. && bash tests/unit/run_unit_tests.sh ``` Test categories in `tests/` (non-exhaustive — 40+ directories total): governance_v4, security, stdlib, vm, cli, e2e, integration, bugs, gorilla, scanner, formatter, lsp, platform, chaos, robustness, agent, unit, property. @@ -250,7 +253,7 @@ include/naab/ All headers - `src/runtime/language_registry.cpp` — executor registration - `<>` syntax — `>>` must be at line start to close block - Executor base: `executeWithReturn()`/`callFunction()` use NaabVal -- **Subprocess containment** (`src/runtime/subprocess_helpers.h/cpp`): `SubprocessContainment` struct applied to all polyglot child processes via `execute_subprocess_with_pipes()`. 5-layer defense: RLIMIT_NPROC=0 (blocks fork/subprocess), PATH restriction, env scrubbing, timeout (SIGKILL), allow_exec/allow_fork flags. `SubprocessContainment::fromCurrentSandbox()` reads current sandbox level: `standard` → fork+exec blocked, `elevated` → fork allowed, `unrestricted` → no restrictions. Pre-execution source scanning (governance checks) catches static patterns; containment catches runtime-constructed commands that evade static analysis. +- **Subprocess containment** (`src/runtime/subprocess_helpers.h/cpp`): `SubprocessContainment` struct applied to all polyglot child processes via `execute_subprocess_with_pipes()`. The memory budget is `RLIMIT_DATA` (private writable memory), with `RLIMIT_AS` only as a backstop ceiling (4x the budget, at least 2 GB) for shared anonymous mappings, which `RLIMIT_DATA` does not count. It used to be `RLIMIT_AS` at the budget, which counts untouched reservations: node 22 needs 512-768 MB of address space just to start, so `process.run("node", ...)` died with "Failed to reserve virtual memory" under `elevated`. Test: `tests/security/test_child_memory_limit.sh` (M-02/M-03 are why the fix is not "drop the limit"). 5-layer defense: RLIMIT_NPROC=0 (blocks fork/subprocess), PATH restriction, env scrubbing, timeout (SIGKILL), allow_exec/allow_fork flags. `SubprocessContainment::fromCurrentSandbox()` reads current sandbox level: `standard` → fork+exec blocked, `elevated` → fork allowed, `unrestricted` → no restrictions. Pre-execution source scanning (governance checks) catches static patterns; containment catches runtime-constructed commands that evade static analysis. ### Stdlib Notable - `array.sort()` mutates in place, `array.sorted()` returns a new sorted array (non-mutating) @@ -412,4 +415,5 @@ Always run `bash tests/security/test_error_msg_leaks.sh` after changing any erro - **Governance is discovered PER PROCESS from the script's own directory, so a spawned interpreter is governed by whatever config sits next to the CHILD -- or by none.** `capabilities.shell.enabled: true` therefore means project governance is ADVISORY for anything that can spawn: a program writes a `.naab` file somewhere no `govern.json` is discoverable, runs `naab-lang --no-governance` on it, and the child reads what the parent is HARD-blocked from. Measured on `62244b3` with a **validly signed** parent config, `sandbox_level: elevated`: the parent's own `file.read("secret.txt")` is blocked at exit 3 (the control), while the spawned child prints `CHILD_READ:POLICY_SECRET` and the parent exits 0. **Signing does not help** -- the implicit bypass-flag block at `main.cpp:1583` fires only for a config the process actually discovers, and the child discovers none; `integrity.blocked_flags` is inert for the same reason. `--no-governance` cannot simply be removed either: `global_require_governance` defaults **true** (`main.cpp:560`), so waiving the "no govern.json found" error is the one job nothing else does. **`blocked_commands: ["naab-lang"]` is NOT containment** -- it matches the command line, so a renamed copy of the same binary walks straight past it (measured: literal name blocked, `cp naab-lang mytool` ESCAPED). A substring match on a command line looks like containment in a table and is not; same shape as `shell_allowed`'s content scan versus its execution gate, which this file already separates for exactly this reason. The containment that works is `capabilities.shell.enabled: false`, because it removes SYS_EXEC at the SANDBOX layer rather than matching a string -- verified to stop the renamed binary too (`SANDBOX VIOLATION`, exit 1, both spellings). This is the logical extreme of the `process.run` note above: that pipeline keeps taint and BSD; this one keeps nothing, because the child is a fresh process with a fresh (empty) policy. Do NOT "fix" it by having the parent pass its config path down -- a child chooses its own arguments, so that is a request, not a gate. - **A polyglot block's stdout is captured, not forwarded — assert on side effects.** A test that runs `<>` and greps the parent's stdout for `TOKEN` sees nothing even when the block ran perfectly. The first draft of `test_polyglot_gate_coverage.sh` did exactly that and reported fifteen of nineteen languages UNMEASURABLE, including several already measured as running. Have the block WRITE A FILE and assert on the file: it is observable, and under a restricted sandbox it is also a write the configuration does not permit, which is the stronger claim anyway. - **The filesystem gate is a hand-written module allowlist, and it has been short three times.** `GovernanceEngine::filesystemAccessMode(module, method)` is the ONLY thing that decides whether a stdlib call gets `checkFilesystemAllowed()` + `checkPathAccess()`; both engines (`vm.cpp` OP_CALL_METHOD, `call_dispatch.cpp`) skip the check when it returns `""`. It shipped knowing only `file`, gained `csv`/`log`, and was still missing `io`, `env` and `path` — so `io.read_file`/`write_file`/`exists`/`list_dir`, `env.load_dotenv` and `path.exists`/`path.resolve` reached the disk with no `blocked_paths`/`allowed_paths` check at all. Their implementations call `Sandbox::canRead`/`canWrite` and stop, and **the sandbox is an allowlist with no concept of `blocked_paths`** (`sandbox.cpp` `canRead`/`canWrite`), so the project path policy was simply not in the path. Measured on `fb1e4bd` with `filesystem.mode:"write"` + `blocked_paths:["vault/"]`: `file.read` exit 3, `io.read_file` exit 0 returning the contents, on BOTH engines — and `io.write_file("govern.json", ...)` overwrote the policy file in its own run under `Governance: PASS`, because `addGovernanceProtectedPaths()` only protects what reaches `checkPathAccess`. Note `filesystem.mode:"none"` DID stop these, but via the sandbox (exit 1, catchable), not governance (exit 3) — which is why the gap survives any audit that only tests the off switch. **`io` is why this must stay an explicit per-method allowlist and never gain a write-by-default fallback like `file` has**: `io.write`/`output`/`write_error`/`print`/`println`/`log` are CONSOLE calls, and a fallback would gate every print statement in the language. When adding a stdlib module that opens a file, add it here or it is ungoverned. A default argument is invisible to this gate — it classifies the path the caller WROTE, so `env.load_dotenv()`'s `.env` default is resolved and checked inside `env_impl.cpp` instead. The same rule settled `path`: `exists` stats and `resolve` calls `fs::canonical` (disclosing symlink targets), while `join`/`dirname`/`basename`/`normalize` are lexical and must not be gated. Test: `tests/governance_v4/test_io_env_fs_gate.sh` (IE-06 and IE-12 are load-bearing — without it a blanket deny on `module=="io"` passes every other arm; IE-09 isolates the default-argument half). +- **Timeouts nest through `ScopedTimeout` — never call `setExecutionTimeout()`/`clearTimeout()` around a block.** There is one timer per thread; the CLI and REST wrap the whole run in a `ScopedTimeout`, and executors wrap each block in another. Before the fix every inner scope re-armed that timer and CLEARED it on exit, so one `<>` expression (or `codegen.run`) removed `--timeout` for the rest of the script — a `while true` after it ran until killed from outside, both engines. `ScopedTimeout` now reads the outer deadline (`ResourceLimiter::currentDeadline()`), only ever TIGHTENS it, and restores it on exit; `0` means "no limit of its own" (it used to arm a zero-second timer). The cancel counter is per thread, because a process-wide one let a scope on another thread cancel the main thread's timer. Anything that runs in-process without polling `isTimeoutTriggered()` needs its own interrupt: QuickJS has an interrupt handler, embedded CPython gets a pending call queued from the timer thread via `setTimeoutInterruptHook()` (on 3.11 this also needs a brief GIL acquire, because `Py_AddPendingCall` from a non-main thread does not flag the eval loop), and `http.*` aborts through a curl progress callback. Known limits: Python blocked inside C (`time.sleep`) and Python on worker threads are not interrupted. Test: `tests/security/test_timeout_reach.sh` (P-04 is the control; the N arms assert on the loop AFTER the block). - **GovernanceHardError catch-and-rethrow required** — any new `catch (const std::exception&)` or `catch (const std::runtime_error&)` in interpreter.cpp or vm.cpp that could intercept governance exceptions MUST have a preceding `catch (const governance::GovernanceHardError&) { throw; }`. Without this, HARD governance violations become catchable by NAAb try/catch. diff --git a/docs/unit-test-findings.md b/docs/unit-test-findings.md index 61ded831e..6b1f88698 100644 --- a/docs/unit-test-findings.md +++ b/docs/unit-test-findings.md @@ -118,7 +118,57 @@ stall is most likely the runner wedge `windows.yml` documents. Measured on that run: `test_r22_fixes.sh` alone took about 6 of the 9m39s shell phase (the slow `naab-gov scan` of an 11 MB file). -**Open (2a):** embedded Python. +**Fixed (2a):** the timeout's timer thread now queues a CPython pending call +that raises `TimeoutError` in the running block, and re-queues itself while the +timeout stands, so `except Exception: pass` in a loop cannot swallow it. +Queuing alone was measured to do nothing on CPython 3.11: `Py_AddPendingCall` +computes the eval breaker on the *calling* thread, where "can handle pending +calls" is false, so the loop is never told. A brief GIL acquire from the timer +thread makes the running thread re-take the GIL and recompute the breaker on +its own thread (skipped on Android, where `PyGILState_Ensure` on a foreign +thread is the bionic CFI crash). Measured: the 20 s busy loop stops at 3.06 s +in both engines. **Limits:** a block blocked inside C (`time.sleep`, a socket +read) is not interrupted, since CPython only runs pending calls between +bytecodes (`time.sleep(15)` took 15 s on both builds), and Python on a worker +thread is not interrupted, since CPython runs pending calls on its main thread +only. + +### 2c. One block cancelled the script's `--timeout` (found while fixing 2a) + +There is one timer per thread. The CLI and REST wrap the run in a +`ScopedTimeout`, and the JS, shell, subprocess and C++ executors wrap every +block in another. Each scope re-armed the timer with its own budget and its +destructor **cleared** it, so after a single `<>` expression the +script had no timeout at all. Measured, `--timeout 3`: a `while true` after +`let v = <>` ran until killed from outside, in both +engines. `codegen.run` did the same by calling `setExecutionTimeout` / +`clearTimeout` directly. A second layer: the cancel counter was process-wide, +so a scope on another thread (the tree-walker runs this JS off the main +thread) cancelled the main thread's timer, and a REST request could cancel a +concurrent request's. + +**Fixed:** `ScopedTimeout` nests. A scope can only tighten the deadline in +force, never extend it, and restores the outer deadline on exit. The cancel +counter is per thread. `0` now means "no limit of its own", which also fixes +the zero-second timer in section 3 (it killed every subprocess on start under +the `UNRESTRICTED` preset). + +### 2d. `http.*` outlived `--timeout` (was section 5, unmeasured) + +Measured once an address that drops SYNs was found (`8.8.8.8:81`): +`http.get(url, {}, 0)` was still connecting 20 s into a `--timeout 3` run. +curl never returns to the interpreter mid-transfer, and `timeout_ms = 0` is +libcurl's "never". **Fixed:** a progress callback aborts the transfer when the +timeout fires (curl calls it at least once a second, including while +connecting), a non-positive `timeout_ms` falls back to the 30 s default, and +curl's own timeouts are capped at the time left before the script's deadline. +The cap is what holds on Windows: there the progress callback was not called +during connect, and the first CI run stopped at curl's 10 s connect timeout +instead of 3 s. With the callback disabled locally, the cap alone stops the +request at 3 s. + +Regression suite for 2a, 2c and 2d: `tests/security/test_timeout_reach.sh`, +9 arms. Against the old code, all 8 non-control arms fail and the control passes. ## 3. Real but not reachable today @@ -138,11 +188,11 @@ run: `test_r22_fixes.sh` alone took about 6 of the 9m39s shell phase (the slow runs on whichever thread calls `.get()`, while another thread's `submit()` runs `cleanupCompleted()` and erases wrappers it sees as done -- a wrapper can be freed while its callback is still returning through it. -- **Zero means two things.** `PermissionLevel::UNRESTRICTED` sets - `max_cpu_seconds = 0` ("no limit"); the shell, JS, generic-subprocess and - persistent-process executors pass it to `ScopedTimeout(0)`, whose timer fires - immediately. The CLI and REST API always overwrite the value from - `--timeout`, so no measured path reaches it. Clamp at `ScopedTimeout` anyway. +- **Zero means two things** (fixed with 2c). `PermissionLevel::UNRESTRICTED` + sets `max_cpu_seconds = 0` ("no limit"); the shell, JS and generic-subprocess + executors passed it to `ScopedTimeout(0)`, whose timer fired immediately. + `ScopedTimeout(0)` now arms nothing. The persistent-process executor + converts it to a millisecond budget of its own and is unchanged. - **`naab::Context` cannot run polyglot.** Executor registration (`initialize_executors()`) lives in `src/cli/main.cpp`, not in `libnaab`, so an embedder gets "No executor found" for every polyglot block. An API gap rather @@ -170,13 +220,19 @@ run: `test_r22_fixes.sh` alone took about 6 of the 9m39s shell phase (the slow - **Other `std::async` sites** (`vm.cpp`, `call_dispatch.cpp`) never abandon their futures on a timeout, so they do not share SafeRegex's defect. -## 5. Unmeasured - -- **`http.*` with `timeout_ms = 0`.** The script-supplied value goes straight - to `CURLOPT_TIMEOUT_MS`, which libcurl documents as "never time out", and - nothing clamps it to `--timeout`. Could not be measured here: SSRF protection - refuses loopback, and no external slow endpoint was available. *Code*, not - *measured*. +## 5. CI + +`naab_unit_tests` now runs in CI (`ci.yml`, Build & Test) through +`tests/unit/run_unit_tests.sh`. The 30 known failures are listed in +`tests/unit/known_failures.txt`, each with its reason from this document. The +runner fails if an unlisted test fails, if a listed test no longer exists, or +if a listed `fails` entry starts passing, so the list has to shrink when +something is fixed. Three entries are `hang` (the `AsyncCallbackPool` +deadlocks and the use-after-free) and are not run. Two shell tests changed +status with 2c: `ShellWithTimeout` used to pass only because the zero-second +timer killed the command; it now shows the async-executor timeout defect. The +two `executeBlocking` shell tests are refused by the fail-closed sandbox, +because the callback thread has none (a fixture issue). ## 6. Lessons that generalise diff --git a/include/naab/error_helpers.h b/include/naab/error_helpers.h index 59ae945d2..b53a2c201 100644 --- a/include/naab/error_helpers.h +++ b/include/naab/error_helpers.h @@ -58,6 +58,15 @@ std::string suggestDictKey( const std::string& requested_key, const std::vector& actual_keys); +// Print the "[hint] dict.get(...) returned null" diagnostic for a miss with +// no default. Shared by both engines (three call sites). Capped per run: the +// hint exists for typos, but building a map from data -- counting files from +// git log -- misses on every new key, and one hint per distinct key buried +// real output under hundreds of lines. Thread-safe. +void hintDictGetMiss( + const std::string& requested_key, + const std::vector& actual_keys); + } // namespace error } // namespace naab diff --git a/include/naab/python_c_wrapper.h b/include/naab/python_c_wrapper.h index 17ff8c3a3..dfe3f2d86 100644 --- a/include/naab/python_c_wrapper.h +++ b/include/naab/python_c_wrapper.h @@ -184,6 +184,27 @@ char* python_c_object_to_string(void* obj); */ void python_c_warmup(void); +/** + * Execution-timeout interrupt. + * + * `--timeout` is a flag the NAAb interpreter polls; code running inside + * CPython never polls it, so a <> busy loop ran to completion however + * long it took. python_c_request_interrupt() queues a CPython pending call + * that raises TimeoutError in the running code. It needs neither a thread + * state nor the GIL (so no PyGILState_Ensure on a foreign thread), and is + * called from the timeout's timer thread. + * + * The pending call re-checks `check` when it runs, so a stale request cannot + * fire into a later, unrelated block; while the timeout stands it re-queues + * itself, so `except Exception: pass` in a loop cannot swallow it. + * + * Limit: CPython runs pending calls on the MAIN thread only, so Python running + * on a worker thread (parallel polyglot groups) is not interrupted. + */ +typedef int (*NaabPyTimeoutCheckFn)(void); +void python_c_set_timeout_check(NaabPyTimeoutCheckFn check); +void python_c_request_interrupt(void); + /** * Shutdown Python interpreter (call once from main thread) * diff --git a/include/naab/resource_limits.h b/include/naab/resource_limits.h index 4816c7486..125564f07 100644 --- a/include/naab/resource_limits.h +++ b/include/naab/resource_limits.h @@ -1,6 +1,7 @@ #pragma once #include +#include #include #include #include @@ -23,6 +24,12 @@ class ResourceLimiter { // Throws ResourceLimitException when timeout expires static void setExecutionTimeout(unsigned int seconds); + // Arm the timer for an absolute deadline (setExecutionTimeout is this with + // now + seconds). currentDeadline() reports this thread's active deadline, + // if any; ScopedTimeout uses the pair to nest. + static void setDeadline(std::chrono::steady_clock::time_point deadline); + static bool currentDeadline(std::chrono::steady_clock::time_point& out); + // Clear the current timeout static void clearTimeout(); @@ -48,6 +55,13 @@ class ResourceLimiter { global_shutdown_.store(true, std::memory_order_relaxed); } + // Called from the timer thread (never from a signal handler) right after + // an execution timeout fires. An in-process runtime that cannot poll + // isTimeoutTriggered() itself -- embedded CPython -- registers one to be + // interrupted. QuickJS needs none: its interrupt handler polls the flag. + using TimeoutInterruptHook = void (*)(); + static void setTimeoutInterruptHook(TimeoutInterruptHook hook); + // Check if timeout has been triggered on this thread OR process-wide. // thread_local flag: set when this specific thread received SIGALRM. // global_shutdown_: set by the signal handler; visible to ALL threads, @@ -79,20 +93,46 @@ class ResourceLimiter { #endif }; -// RAII helper for automatic timeout cleanup +// RAII timeout that NESTS. The script's run is wrapped in one (CLI, REST), +// and executors wrap each block in another. There is one timer per thread, so +// these used to clobber each other: the block's scope re-armed the timer with +// its own budget and its destructor CLEARED it, so one <> +// expression silently removed --timeout for the rest of the script (measured: +// a `while true` after it ran until killed from outside). +// +// Now a scope can only TIGHTEN the deadline in force, never extend it, and on +// exit restores the outer deadline instead of clearing it. `seconds == 0` +// means "no limit of its own" (the UNRESTRICTED sandbox preset); it used to arm +// a zero-second timer that killed every subprocess on start. class ScopedTimeout { public: explicit ScopedTimeout(unsigned int seconds) { - ResourceLimiter::setExecutionTimeout(seconds); + had_outer_ = ResourceLimiter::currentDeadline(outer_); + if (seconds == 0) return; + auto mine = std::chrono::steady_clock::now() + std::chrono::seconds(seconds); + if (had_outer_ && outer_ <= mine) return; // the outer limit is tighter + ResourceLimiter::setDeadline(mine); + armed_ = true; } ~ScopedTimeout() { - ResourceLimiter::clearTimeout(); + if (!armed_) return; + if (had_outer_) { + // Re-arms for the remaining time, or fires at once if it has passed. + ResourceLimiter::setDeadline(outer_); + } else { + ResourceLimiter::clearTimeout(); + } } // Prevent copying ScopedTimeout(const ScopedTimeout&) = delete; ScopedTimeout& operator=(const ScopedTimeout&) = delete; + +private: + std::chrono::steady_clock::time_point outer_{}; + bool had_outer_ = false; + bool armed_ = false; }; } // namespace security diff --git a/include/naab/subprocess_helpers.h b/include/naab/subprocess_helpers.h index 6cd873c2a..baad617e7 100644 --- a/include/naab/subprocess_helpers.h +++ b/include/naab/subprocess_helpers.h @@ -54,7 +54,7 @@ struct SubprocessContainment { bool block_fork = false; // L2: RLIMIT_NPROC=0 / ACTIVE_PROCESS=1 size_t max_fsize_bytes = 0; // L3: RLIMIT_FSIZE (0 = no limit) size_t max_nofile = 0; // L3: RLIMIT_NOFILE (0 = no limit) - size_t max_memory_bytes = 0; // L3/L7: Memory limit (RLIMIT_AS / Job memory) + size_t max_memory_bytes = 0; // L3/L7: Memory limit (RLIMIT_DATA + RLIMIT_AS ceiling / Job memory) size_t max_cpu_ms = 0; // L3/L8: CPU time limit (RLIMIT_CPU / Job CPU time) bool no_new_privs = false; // L4: prctl(PR_SET_NO_NEW_PRIVS) diff --git a/run-all-tests.sh b/run-all-tests.sh index c4b06f99c..383bb980f 100755 --- a/run-all-tests.sh +++ b/run-all-tests.sh @@ -3297,6 +3297,45 @@ else echo " test_regex_timeout_bound.sh: not found, skipping" fi +PARSER_DOGFOOD_HINTS_SCRIPT="tests/parser/test_dogfood_hints.sh" +if [ -f "$PARSER_DOGFOOD_HINTS_SCRIPT" ]; then + if run_shell_test "$PARSER_DOGFOOD_HINTS_SCRIPT" 2>&1; then + echo " test_dogfood_hints.sh: ALL PASSED" + else + echo " test_dogfood_hints.sh: FAILURE(S)" + FAILED=$((FAILED + 1)) + FAILED_TESTS+=("test_dogfood_hints.sh") + fi +else + echo " test_dogfood_hints.sh: not found, skipping" +fi + +SEC_CHILD_MEMORY_LIMIT_SCRIPT="tests/security/test_child_memory_limit.sh" +if [ -f "$SEC_CHILD_MEMORY_LIMIT_SCRIPT" ]; then + if run_shell_test "$SEC_CHILD_MEMORY_LIMIT_SCRIPT" 2>&1; then + echo " test_child_memory_limit.sh: ALL PASSED" + else + echo " test_child_memory_limit.sh: FAILURE(S)" + FAILED=$((FAILED + 1)) + FAILED_TESTS+=("test_child_memory_limit.sh") + fi +else + echo " test_child_memory_limit.sh: not found, skipping" +fi + +SEC_TIMEOUT_REACH_SCRIPT="tests/security/test_timeout_reach.sh" +if [ -f "$SEC_TIMEOUT_REACH_SCRIPT" ]; then + if run_shell_test "$SEC_TIMEOUT_REACH_SCRIPT" 2>&1; then + echo " test_timeout_reach.sh: ALL PASSED" + else + echo " test_timeout_reach.sh: FAILURE(S)" + FAILED=$((FAILED + 1)) + FAILED_TESTS+=("test_timeout_reach.sh") + fi +else + echo " test_timeout_reach.sh: not found, skipping" +fi + PARSER_STRING_INTERP_ESCAPE_SCRIPT="tests/parser/test_string_interp_escape.sh" if [ -f "$PARSER_STRING_INTERP_ESCAPE_SCRIPT" ]; then if run_shell_test "$PARSER_STRING_INTERP_ESCAPE_SCRIPT" 2>&1; then diff --git a/src/interpreter/call_dispatch.cpp b/src/interpreter/call_dispatch.cpp index 774ba83ca..fc79afac2 100644 --- a/src/interpreter/call_dispatch.cpp +++ b/src/interpreter/call_dispatch.cpp @@ -647,33 +647,11 @@ void Interpreter::visit(ast::CallExpr& node) { } else if (args.size() >= 2) { result_ = args[1]; } else { - // "Did you mean?" hint when key not found and similar keys exist - // Deduplicate: only show each (key, suggestion) pair once per execution if (!dict.empty()) { std::vector keys; keys.reserve(dict.size()); for (const auto& [k, v] : dict) { (void)v; keys.push_back(k); } - auto suggestion = naab::error::suggestDictKey(key, keys); - if (!suggestion.empty()) { - static std::unordered_set seen_hints; - auto hint_key = key + "\xe2\x86\x92" + suggestion; - if (seen_hints.insert(hint_key).second) { - fprintf(stderr, "[hint] dict.get(\"%s\") returned null — did you mean \"%s\"?\n", - key.c_str(), suggestion.c_str()); - } - } else if (keys.size() <= 8) { - static std::unordered_set seen_avail; - auto avail_key = key + "\xe2\x86\x92?"; - if (seen_avail.insert(avail_key).second) { - std::string avail; - for (size_t j = 0; j < keys.size(); ++j) { - if (j > 0) avail += ", "; - avail += "\"" + keys[j] + "\""; - } - fprintf(stderr, "[hint] dict.get(\"%s\") returned null — available keys: %s\n", - key.c_str(), avail.c_str()); - } - } + naab::error::hintDictGetMiss(key, keys); } result_ = NaabVal::makeNull(); } @@ -1605,33 +1583,11 @@ void Interpreter::visit(ast::CallExpr& node) { } else if (args.size() >= 2) { result_ = args[1]; // default value } else { - // "Did you mean?" hint when key not found and similar keys exist - // Deduplicate: only show each (key, suggestion) pair once per execution if (!dict.empty()) { std::vector keys; keys.reserve(dict.size()); for (const auto& [k, v] : dict) { (void)v; keys.push_back(k); } - auto suggestion = naab::error::suggestDictKey(key, keys); - if (!suggestion.empty()) { - static std::unordered_set seen_hints; - auto hint_key = key + "\xe2\x86\x92" + suggestion; - if (seen_hints.insert(hint_key).second) { - fprintf(stderr, "[hint] dict.get(\"%s\") returned null — did you mean \"%s\"?\n", - key.c_str(), suggestion.c_str()); - } - } else if (keys.size() <= 8) { - static std::unordered_set seen_avail; - auto avail_key = key + "\xe2\x86\x92?"; - if (seen_avail.insert(avail_key).second) { - std::string avail; - for (size_t j = 0; j < keys.size(); ++j) { - if (j > 0) avail += ", "; - avail += "\"" + keys[j] + "\""; - } - fprintf(stderr, "[hint] dict.get(\"%s\") returned null — available keys: %s\n", - key.c_str(), avail.c_str()); - } - } + naab::error::hintDictGetMiss(key, keys); } result_ = NaabVal::makeNull(); } diff --git a/src/parser/parser.cpp b/src/parser/parser.cpp index 25a222cee..25c03b987 100644 --- a/src/parser/parser.cpp +++ b/src/parser/parser.cpp @@ -310,6 +310,21 @@ bool Parser::check(lexer::TokenType type) const { return current().type == type; } +// A '?' outside a type annotation is almost always a C/JS ternary. It reaches +// the parser in two ways: as the start of an expression (parsePrimary), or -- +// far more often -- right after a complete operand inside a dict literal, a +// call or parentheses, where expect() used to report "Expected '}'" and, for a +// brace, a missing-brace diagnosis that had nothing to do with it. +static std::string ternaryHint() { + return "\n\n NAAb does not support the C/JS ternary operator (cond ? a : b).\n" + " Use NAAb's if-expression instead:\n" + " \xE2\x9C\x97 Wrong: condition ? value_a : value_b\n" + " \xE2\x9C\x93 Right: if condition { value_a } else { value_b }\n\n" + " The if-expression IS a value \xe2\x80\x94 assign it directly:\n" + " let result = if x > 0 { x } else { 0 }\n\n" + " Note: '?' is only valid in type annotations (e.g., string? for nullable).\n"; +} + const lexer::Token& Parser::expect(lexer::TokenType type, const std::string& msg) { if (check(type)) { // Track brace positions for better error messages @@ -325,6 +340,9 @@ const lexer::Token& Parser::expect(lexer::TokenType type, const std::string& msg // Phase 2.1: Use enhanced error hints for better error messages const auto& token = current(); + if (token.type == lexer::TokenType::QUESTION && type != lexer::TokenType::QUESTION) { + throw ParseError(formatError(msg, token) + ternaryHint()); + } error_reporter_.error(msg, token.line, token.column); // Get context-aware hints @@ -3196,13 +3214,7 @@ std::unique_ptr Parser::parsePrimary() { } } else if (tok.type == lexer::TokenType::QUESTION) { - hint = "\n\n NAAb does not support the C/JS ternary operator (cond ? a : b).\n" - " Use NAAb's if-expression instead:\n" - " \xE2\x9C\x97 Wrong: condition ? value_a : value_b\n" - " \xE2\x9C\x93 Right: if condition { value_a } else { value_b }\n\n" - " The if-expression IS a value \xe2\x80\x94 assign it directly:\n" - " let result = if x > 0 { x } else { 0 }\n\n" - " Note: '?' is only valid in type annotations (e.g., string? for nullable).\n"; + hint = ternaryHint(); } throw ParseError(formatError("Unexpected token in expression", tok) + hint); diff --git a/src/runtime/python_c_wrapper.c b/src/runtime/python_c_wrapper.c index 12d9554b3..c8aa05a6f 100644 --- a/src/runtime/python_c_wrapper.c +++ b/src/runtime/python_c_wrapper.c @@ -29,6 +29,40 @@ static __thread PyThreadState* worker_tstate = NULL; // Sentinel value: python_c_gil_acquire returns -1 when using pre-created state #define GIL_HANDLE_PRECREATED (-1) +// --- Execution-timeout interrupt (see python_c_wrapper.h) ------------------ +static NaabPyTimeoutCheckFn timeout_check = NULL; + +void python_c_set_timeout_check(NaabPyTimeoutCheckFn check) { + timeout_check = check; +} + +static int naab_timeout_pending_call(void* arg) { + (void)arg; + if (!timeout_check || !timeout_check()) return 0; // stale: timeout cleared + PyErr_SetString(PyExc_TimeoutError, + "execution timeout exceeded while running Python code"); + // Re-arm: the next eval-breaker check raises again, so a broad except + // clause cannot keep the block alive past the timeout. + Py_AddPendingCall(naab_timeout_pending_call, NULL); + return -1; +} + +void python_c_request_interrupt(void) { + if (!Py_IsInitialized()) return; + Py_AddPendingCall(naab_timeout_pending_call, NULL); +#if !defined(__ANDROID__) + // Queuing is not enough on CPython 3.11: Py_AddPendingCall computes the + // eval breaker on the CALLING thread, and "can handle pending calls" is + // false off the main thread, so the running loop is never told. Measured: + // the call was queued and never ran. Taking the GIL briefly makes the + // running thread drop it; when it re-takes it, it recomputes the breaker on + // its own thread and runs the call. Not on Android, where PyGILState_Ensure + // on a foreign thread is the bionic CFI crash this file avoids elsewhere. + PyGILState_STATE g = PyGILState_Ensure(); + PyGILState_Release(g); +#endif +} + // --- Sandbox audit hook (see python_c_wrapper.h) --------------------------- // Policy lives in C++ (it owns the sandbox); this side only decodes CPython's // audit arguments and recognises the interpreter loading its own modules. diff --git a/src/runtime/python_interpreter_manager.cpp b/src/runtime/python_interpreter_manager.cpp index c65f85476..c6286455a 100644 --- a/src/runtime/python_interpreter_manager.cpp +++ b/src/runtime/python_interpreter_manager.cpp @@ -2,6 +2,7 @@ #include "naab/python_interpreter_manager.h" #include "naab/python_c_wrapper.h" +#include "naab/resource_limits.h" #include #include @@ -39,6 +40,13 @@ PythonInterpreterManager::PythonInterpreterManager() if (python_c_init() != 0) { throw std::runtime_error("Failed to initialize Python interpreter"); } + + // Let --timeout interrupt Python code: the timer thread queues a pending + // call that raises TimeoutError inside the running block. + python_c_set_timeout_check([]() -> int { + return naab::security::ResourceLimiter::isTimeoutTriggered() ? 1 : 0; + }); + naab::security::ResourceLimiter::setTimeoutInterruptHook(&python_c_request_interrupt); } PythonInterpreterManager::~PythonInterpreterManager() { diff --git a/src/runtime/resource_limits.cpp b/src/runtime/resource_limits.cpp index cce7cb166..e78e2072a 100644 --- a/src/runtime/resource_limits.cpp +++ b/src/runtime/resource_limits.cpp @@ -1,5 +1,6 @@ #include "naab/resource_limits.h" #include +#include #include #include @@ -33,18 +34,39 @@ thread_local volatile bool ResourceLimiter::timeout_triggered_ = false; // threads — including ThreadPool workers that never receive SIGALRM directly. // Cleared by setExecutionTimeout() (new request) and clearTimeout() (RAII cleanup). std::atomic ResourceLimiter::global_shutdown_{false}; +// Generation counter for cancellable timer threads: each arm bumps it and the +// timer fires only if its captured value still matches, so a stale timer from +// execution N cannot poison execution N+1 (R1). File-local to avoid a header +// ABI change. +// +// PER THREAD: a process-wide counter let one thread's clearTimeout() cancel +// another thread's timer -- a block run off the main thread removed the +// script's --timeout, and one REST request could cancel a concurrent +// request's. The timer thread holds a reference, so the counter outlives the +// thread that armed it. +static const std::shared_ptr>& timerGeneration() { + static thread_local const std::shared_ptr> gen = + std::make_shared>(0); + return gen; +} #ifdef _WIN32 std::atomic ResourceLimiter::win_timer_cancel_{false}; -// R1 fix: generation counter for cancellable Windows timer threads. -// File-local to avoid a header ABI change. -static std::atomic g_win_timer_generation{0}; #else // V-RT-007: cancel flag for the POSIX timer thread (set by clearTimeout()). std::atomic ResourceLimiter::posix_timer_cancel_{false}; -// Generation counter for cancellable POSIX timer threads (matches Windows pattern). -static std::atomic g_posix_timer_generation{0}; #endif +// File-local to avoid a header ABI change (same reason as the generation counters). +static std::atomic g_timeout_interrupt_hook{nullptr}; + +void ResourceLimiter::setTimeoutInterruptHook(TimeoutInterruptHook hook) { + g_timeout_interrupt_hook.store(hook, std::memory_order_release); +} + +static void fireTimeoutInterruptHook() { + if (auto hook = g_timeout_interrupt_hook.load(std::memory_order_acquire)) hook(); +} + void ResourceLimiter::installSignalHandlers() { if (initialized_) { return; @@ -81,10 +103,26 @@ bool ResourceLimiter::isInitialized() { return initialized_; } +// Deadline of the innermost active timeout on this thread. ScopedTimeout +// reads it to nest: see resource_limits.h. +static thread_local bool t_has_deadline = false; +static thread_local std::chrono::steady_clock::time_point t_deadline; + +bool ResourceLimiter::currentDeadline(std::chrono::steady_clock::time_point& out) { + if (t_has_deadline) out = t_deadline; + return t_has_deadline; +} + void ResourceLimiter::setExecutionTimeout(unsigned int seconds) { + setDeadline(std::chrono::steady_clock::now() + std::chrono::seconds(seconds)); +} + +void ResourceLimiter::setDeadline(std::chrono::steady_clock::time_point deadline) { if (!initialized_) { installSignalHandlers(); } + t_has_deadline = true; + t_deadline = deadline; // V-ASYNC-001: reset both flags at the start of each new execution budget. global_shutdown_.store(false, std::memory_order_relaxed); @@ -94,46 +132,48 @@ void ResourceLimiter::setExecutionTimeout(unsigned int seconds) { // V-RT-007: capture the calling thread's id by value so the timer thread // can call pthread_kill() on the exact thread, not a random one in the pool. // tid is NOT stored as a static — it lives in the lambda closure so each - // concurrent call to setExecutionTimeout() has its own independent timer. + // concurrent call to setDeadline() has its own independent timer. pthread_t tid = pthread_self(); posix_timer_cancel_.store(false, std::memory_order_relaxed); - uint64_t my_gen = ++g_posix_timer_generation; - std::thread([seconds, tid, my_gen]() { + auto gen = timerGeneration(); + uint64_t my_gen = ++*gen; + std::thread([deadline, tid, gen, my_gen]() { using clock = std::chrono::steady_clock; - auto deadline = clock::now() + std::chrono::seconds(seconds); while (clock::now() < deadline) { - if (g_posix_timer_generation.load(std::memory_order_relaxed) != my_gen) return; + if (gen->load(std::memory_order_relaxed) != my_gen) return; std::this_thread::sleep_for(std::chrono::milliseconds(50)); } - if (g_posix_timer_generation.load(std::memory_order_relaxed) == my_gen) { + if (gen->load(std::memory_order_relaxed) == my_gen) { // Set global_shutdown_ first so isTimeoutTriggered() returns true // even if the signal is not delivered immediately (e.g. tight loops // on Android/Termux where SIGALRM may stay pending). ResourceLimiter::global_shutdown_.store(true, std::memory_order_relaxed); pthread_kill(tid, SIGALRM); + fireTimeoutInterruptHook(); } }).detach(); alarm(0); // cancel any prior system-level alarm #else // Windows has no alarm(). Spawn a detached timer thread that sets - // global_shutdown_ after the deadline. + // global_shutdown_ at the deadline. // // R1 fix: generation counter prevents a stale timer from execution N - // from poisoning execution N+1. Each new setExecutionTimeout() bumps the + // from poisoning execution N+1. Each new setDeadline() bumps the // counter; the timer thread captures the pre-bump value; before setting // global_shutdown_ it verifies the counter still matches. clearTimeout() // also bumps, so normal completion invalidates the in-flight timer. - uint64_t my_gen = ++g_win_timer_generation; + auto gen = timerGeneration(); + uint64_t my_gen = ++*gen; win_timer_cancel_.store(false, std::memory_order_relaxed); // kept for compat - std::thread([seconds, my_gen]() { + std::thread([deadline, gen, my_gen]() { using clock = std::chrono::steady_clock; - auto deadline = clock::now() + std::chrono::seconds(seconds); while (clock::now() < deadline) { - if (g_win_timer_generation.load(std::memory_order_relaxed) != my_gen) return; + if (gen->load(std::memory_order_relaxed) != my_gen) return; std::this_thread::sleep_for(std::chrono::milliseconds(50)); } - if (g_win_timer_generation.load(std::memory_order_relaxed) == my_gen) { + if (gen->load(std::memory_order_relaxed) == my_gen) { ResourceLimiter::global_shutdown_.store(true, std::memory_order_relaxed); + fireTimeoutInterruptHook(); } }).detach(); #endif @@ -144,18 +184,19 @@ void ResourceLimiter::clearTimeout() { // V-RT-007: cancel the posix timer thread and any residual system alarm. // Bump generation counter to invalidate any in-flight timer thread // (matches Windows pattern — stale timer sees mismatched generation and exits). - ++g_posix_timer_generation; + ++*timerGeneration(); posix_timer_cancel_.store(true, std::memory_order_relaxed); alarm(0); #else // R1 fix: bumping the generation counter invalidates any in-flight timer // thread — when it wakes, its captured my_gen no longer matches and it // returns without touching global_shutdown_. - ++g_win_timer_generation; + ++*timerGeneration(); win_timer_cancel_.store(true, std::memory_order_relaxed); // kept for compat #endif timeout_triggered_ = false; global_shutdown_.store(false, std::memory_order_relaxed); // V-ASYNC-001 + t_has_deadline = false; } void ResourceLimiter::setMemoryLimit(size_t megabytes) { diff --git a/src/runtime/subprocess_helpers.cpp b/src/runtime/subprocess_helpers.cpp index 8939342cb..d897ab0b5 100644 --- a/src/runtime/subprocess_helpers.cpp +++ b/src/runtime/subprocess_helpers.cpp @@ -147,10 +147,28 @@ static void apply_posix_containment(const SubprocessContainment& c) { // space even for trivial programs like /usr/bin/echo. RLIMIT_AS < 12 GB // causes every child exec to SIGABRT (dynamic linker OOM). Actual RSS // remains small — the virtual size is just address-space reservation. + // + // The memory budget is enforced on the DATA segment (private writable + // memory: heap, anonymous mmap, thread stacks), not on address space. + // RLIMIT_AS at the budget counted reservations that are never touched, and + // modern runtimes reserve far more than they use: node 22 needs 512-768 MB + // of address space just to START (V8's code range and pointer cage), so + // under the 512 MB budget process.run("node", ...) died with "Failed to + // reserve virtual memory" before running a line. Measured: RLIMIT_DATA at + // 256 MB lets node start and still stops it at 192 MB of real allocation. + // + // RLIMIT_DATA does not count SHARED anonymous mappings, so an address-space + // ceiling stays as a backstop against that route: 4x the budget, at least + // 2 GB -- loose enough for runtime reservations, still finite. #ifndef __ANDROID__ if (c.max_memory_bytes > 0) { - struct rlimit rl = {(rlim_t)c.max_memory_bytes, (rlim_t)c.max_memory_bytes}; - setrlimit(RLIMIT_AS, &rl); + struct rlimit data = {(rlim_t)c.max_memory_bytes, (rlim_t)c.max_memory_bytes}; + setrlimit(RLIMIT_DATA, &data); + const uint64_t kMinAsCeiling = 2048ULL * 1024ULL * 1024ULL; + uint64_t as_ceiling = c.max_memory_bytes * 4ULL; + if (as_ceiling < kMinAsCeiling) as_ceiling = kMinAsCeiling; + struct rlimit as = {(rlim_t)as_ceiling, (rlim_t)as_ceiling}; + setrlimit(RLIMIT_AS, &as); } #endif diff --git a/src/semantic/error_helpers.cpp b/src/semantic/error_helpers.cpp index 0c87aaea5..bda71d595 100644 --- a/src/semantic/error_helpers.cpp +++ b/src/semantic/error_helpers.cpp @@ -6,6 +6,8 @@ #include #include #include +#include +#include #include namespace naab { @@ -293,5 +295,48 @@ std::string suggestDictKey( return ""; } +void hintDictGetMiss( + const std::string& requested_key, + const std::vector& actual_keys) { + + if (actual_keys.empty()) return; + // Small enough to show every key only; otherwise a "did you mean" or nothing. + std::string suggestion = suggestDictKey(requested_key, actual_keys); + if (suggestion.empty() && actual_keys.size() > 8) return; + + static constexpr int kMaxHints = 3; + static std::mutex mu; + static std::unordered_set seen; + static int shown = 0; + static bool capped = false; + std::lock_guard lock(mu); + + // Each (key, suggestion) pair once per run, as before. + if (!seen.insert(requested_key + "\xe2\x86\x92" + suggestion).second) return; + if (shown >= kMaxHints) { + if (!capped) { + capped = true; + std::fprintf(stderr, + "[hint] more dict.get() misses; further hints suppressed. If a miss is\n" + " expected, say so: d.get(key, 0) or d.get(key, null) with a default,\n" + " or check d.has(key) first -- neither prints a hint.\n"); + } + return; + } + ++shown; + if (!suggestion.empty()) { + std::fprintf(stderr, "[hint] dict.get(\"%s\") returned null \xe2\x80\x94 did you mean \"%s\"?\n", + requested_key.c_str(), suggestion.c_str()); + } else { + std::string avail; + for (size_t j = 0; j < actual_keys.size(); ++j) { + if (j > 0) avail += ", "; + avail += "\"" + actual_keys[j] + "\""; + } + std::fprintf(stderr, "[hint] dict.get(\"%s\") returned null \xe2\x80\x94 available keys: %s\n", + requested_key.c_str(), avail.c_str()); + } +} + } // namespace error } // namespace naab diff --git a/src/stdlib/codegen_impl.cpp b/src/stdlib/codegen_impl.cpp index 1600423f6..85056a19a 100644 --- a/src/stdlib/codegen_impl.cpp +++ b/src/stdlib/codegen_impl.cpp @@ -496,9 +496,9 @@ interpreter::NaabVal CodegenModule::call( auto exec_start = std::chrono::steady_clock::now(); try { - security::ResourceLimiter::setExecutionTimeout(static_cast(timeout)); + // Nested, so it cannot extend or cancel the script's own --timeout. + security::ScopedTimeout codegen_timeout(static_cast(timeout)); interpreter::NaabVal result = executor->executeWithReturn(final_code); - security::ResourceLimiter::clearTimeout(); output = executor->getCapturedOutput(); exit_code = executor->getLastExitCode(); @@ -508,10 +508,8 @@ interpreter::NaabVal CodegenModule::call( output = result.asString(); } } catch (const governance::GovernanceHardError&) { - security::ResourceLimiter::clearTimeout(); throw; // V-CG-001: HARD blocks propagate without suppression } catch (const std::exception& e) { - security::ResourceLimiter::clearTimeout(); exit_code = 1; stderr_output = e.what(); } diff --git a/src/stdlib/http_impl.cpp b/src/stdlib/http_impl.cpp index 8f1171311..98d254531 100644 --- a/src/stdlib/http_impl.cpp +++ b/src/stdlib/http_impl.cpp @@ -5,10 +5,12 @@ #include "naab/governance.h" #include "naab/interpreter.h" #include "naab/sandbox.h" +#include "naab/resource_limits.h" #include "naab/utils/string_utils.h" #include #include #include +#include #include #ifdef _WIN32 #include @@ -124,6 +126,16 @@ struct BoundedResponseSink { size_t max_size; }; +// --timeout is a flag the interpreter polls, and curl_easy_perform() never +// returns to the interpreter mid-transfer, so a request ran for its own +// timeout_ms however long that was -- and timeout_ms = 0 is libcurl's "never". +// Measured: http.get(url, {}, 0) to an address that drops SYNs was still +// connecting 20s into a --timeout 3 run. curl calls this at least once a +// second, including while connecting; a non-zero return aborts the transfer. +static int TimeoutProgressCallback(void*, curl_off_t, curl_off_t, curl_off_t, curl_off_t) { + return naab::security::ResourceLimiter::isTimeoutTriggered() ? 1 : 0; +} + static size_t WriteCallback(void* contents, size_t size, size_t nmemb, void* userp) { size_t total_size = size * nmemb; auto* sink = static_cast(userp); @@ -312,10 +324,29 @@ interpreter::NaabVal performRequest( curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, HeaderCallback); curl_easy_setopt(curl, CURLOPT_HEADERDATA, &response_headers); - // Set timeout (in milliseconds) + // Set timeout (in milliseconds). 0 or negative is libcurl's "no timeout"; + // a script cannot opt out of a bound, so it gets the default instead. + if (timeout_ms <= 0) timeout_ms = 30000; + // Never let a request outlast the script's own deadline. The progress + // callback below also aborts on timeout, but curl does not reliably call + // it while connecting on every platform: on the Windows runner a connect + // to a SYN-dropping host ran to the 10s connect timeout. Capping curl's + // own timeouts at the time remaining is exact everywhere. + { + std::chrono::steady_clock::time_point deadline; + if (naab::security::ResourceLimiter::currentDeadline(deadline)) { + auto left = std::chrono::duration_cast( + deadline - std::chrono::steady_clock::now()).count(); + if (left < 1) left = 1; + if (left < timeout_ms) timeout_ms = static_cast(left); + } + } curl_easy_setopt(curl, CURLOPT_TIMEOUT_MS, static_cast(timeout_ms)); curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT_MS, std::min(static_cast(timeout_ms), 10000L)); curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L); + // Stop the transfer when --timeout fires (see TimeoutProgressCallback). + curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, TimeoutProgressCallback); + curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L); // Follow redirects. Safe only because naabCurlOpenSocketCallback below // re-adjudicates every hop -- the URL-level isPrivateHost() check never @@ -362,6 +393,15 @@ interpreter::NaabVal performRequest( } // Check for errors (curl_guard handles cleanup on any exit path) + if (res == CURLE_ABORTED_BY_CALLBACK || + (res == CURLE_OPERATION_TIMEDOUT && + naab::security::ResourceLimiter::isTimeoutTriggered())) { + throw std::runtime_error( + "HTTP request stopped: the script's execution time limit was reached\n\n" + " Help:\n" + " - The request was still in progress when the run's time limit expired\n" + " - Pass a smaller timeout_ms to http.* so a slow server fails fast\n"); + } if (res != CURLE_OK) { throw std::runtime_error(fmt::format( "HTTP request failed: {} ({})", diff --git a/src/vm/vm.cpp b/src/vm/vm.cpp index 2763ec019..7528363d8 100644 --- a/src/vm/vm.cpp +++ b/src/vm/vm.cpp @@ -4649,46 +4649,11 @@ interpreter::NaabVal VM::callBuiltinMethod(interpreter::NaabVal& obj, const std: auto it = dict.find(key); if (it != dict.end()) return it->second; if (argc >= 2) return args[1]; // default value - // "Did you mean?" hint when key not found and similar keys exist - // Deduplicate: only show each (key, suggestion) pair once per execution if (!dict.empty()) { std::vector keys; keys.reserve(dict.size()); for (const auto& [k, v] : dict) { (void)v; keys.push_back(k); } - auto suggestion = naab::error::suggestDictKey(key, keys); - if (!suggestion.empty()) { - // H5 fix: protect static set from concurrent async VM access - static std::mutex hints_mutex; - static std::unordered_set seen_hints; - auto hint_key = key + "\xe2\x86\x92" + suggestion; - bool is_new; - { - std::lock_guard lock(hints_mutex); - is_new = seen_hints.insert(hint_key).second; - } - if (is_new) { - fprintf(stderr, "[hint] dict.get(\"%s\") returned null — did you mean \"%s\"?\n", - key.c_str(), suggestion.c_str()); - } - } else if (keys.size() <= 8) { - static std::mutex avail_mutex; - static std::unordered_set seen_avail; - auto avail_key = key + "\xe2\x86\x92?"; - bool is_new; - { - std::lock_guard lock(avail_mutex); - is_new = seen_avail.insert(avail_key).second; - } - if (is_new) { - std::string avail; - for (size_t j = 0; j < keys.size(); ++j) { - if (j > 0) avail += ", "; - avail += "\"" + keys[j] + "\""; - } - fprintf(stderr, "[hint] dict.get(\"%s\") returned null — available keys: %s\n", - key.c_str(), avail.c_str()); - } - } + naab::error::hintDictGetMiss(key, keys); } return interpreter::NaabVal::makeNull(); } diff --git a/tests/parser/test_dogfood_hints.sh b/tests/parser/test_dogfood_hints.sh new file mode 100755 index 000000000..d423a7687 --- /dev/null +++ b/tests/parser/test_dogfood_hints.sh @@ -0,0 +1,108 @@ +#!/usr/bin/env bash +# test_dogfood_hints.sh -- diagnostics found noisy or missing while building a +# real project (repo-sentinel) in NAAb. +# +# Group D: dict.get() miss hints. The hint is for typos, but building a map +# from data (counting files from `git log`) misses on every new key, +# and one hint per distinct key printed hundreds of lines over the +# program's own output. Now capped per run, with one line saying how +# to mark an expected miss. D-02 is the control that a lone typo still +# gets its "did you mean" -- a cap of zero would pass D-01 alone. +# Group T: the ternary hint. It existed but fired only when '?' started an +# expression; inside a dict literal or parentheses expect() reported +# "Expected '}'" (plus a missing-brace diagnosis) or "Expected ')'" +# instead. T-01 is the path that always worked, kept as the control. + +set -uo pipefail +PASS=0 +FAIL=0 +REPO="$(cd "$(dirname "$0")/../.." && pwd)" +NAAB="$REPO/build/naab-lang" +WORK="$(mktemp -d "${TMPDIR:-/tmp}/naab_hints.XXXXXX")" +[ -n "$WORK" ] && [ -d "$WORK" ] || { echo "FATAL: could not create work dir" >&2; exit 1; } +source "$REPO/tests/helpers/trust_setup.sh" +setup_isolated_trust +trap 'rm -rf "$WORK"; teardown_isolated_trust' EXIT + +if [ ! -x "$NAAB" ]; then + echo "FAIL: naab-lang not built at $NAAB (UNMEASURABLE, not a pass)" + exit 1 +fi +echo '{ "version": "4.0", "mode": "off" }' > "$WORK/govern.json" + +pass() { echo " PASS [$1] $2"; PASS=$((PASS+1)); } +fail() { echo " FAIL [$1] $2"; [ -n "${3:-}" ] && echo " $3"; FAIL=$((FAIL+1)); } + +run() { # run FLAG FILE -> stdout+stderr + (cd "$WORK" && timeout 30 "$NAAB" $1 "$WORK/$2" 2>&1) +} + +echo "=== D: dict.get miss hints ===" +cat > "$WORK/count.naab" <<'EOF' +main { + let counts = {} + let i = 0 + while i < 200 { + let f = "src/dir/file_" + string(i % 50) + ".cpp" + let n = counts.get(f) + if n == null { counts[f] = 1 } else { counts[f] = n + 1 } + i = i + 1 + } + print("FILES:" + string(counts.size())) +} +EOF +cat > "$WORK/typo.naab" <<'EOF' +main { + let cfg = {"service": "api", "port": 8080} + let s = cfg.get("servce") + print("DONE") +} +EOF +cat > "$WORK/default.naab" <<'EOF' +main { + let counts = {"a.cpp": 1, "b.cpp": 2} + let n = counts.get("c.cpp", 0) + print("N:" + string(n)) +} +EOF +for eng in "" "--tree-walk"; do + tag="${eng:-vm}"; tag="${tag#--}" + out="$(run "$eng" count.naab)" + hints=$(grep -c '^\[hint\] dict.get' <<<"$out") + if [[ "$out" == *"FILES:50"* ]] && [ "$hints" -le 3 ] && [[ "$out" == *"further hints suppressed"* ]]; then + pass "D-01/$tag" "50 distinct expected misses: $hints hints, then one suppression line" + else + fail "D-01/$tag" "miss hints not capped ($hints hint lines)" "$(grep '^\[hint\]' <<<"$out" | head -2)" + fi + out="$(run "$eng" typo.naab)" + case "$out" in + *'did you mean "service"'*) pass "D-02/$tag" "control: a lone typo still gets its did-you-mean" ;; + *) fail "D-02/$tag" "typo hint lost" "$(head -3 <<<"$out")" ;; + esac + out="$(run "$eng" default.naab)" + if [[ "$out" == *"N:0"* ]] && ! grep -q '^\[hint\]' <<<"$out"; then + pass "D-03/$tag" "a miss with a default prints no hint" + else + fail "D-03/$tag" "default did not silence the hint" "$(head -3 <<<"$out")" + fi +done + +echo "=== T: ternary hint ===" +printf 'main {\n let c = true\n let x = c ? "a" : "b"\n print(x)\n}\n' > "$WORK/t_let.naab" +printf 'main {\n let n = 3\n let d = {"k": n > 2 ? "big" : "small"}\n print(d)\n}\n' > "$WORK/t_dict.naab" +printf 'main {\n let n = 3\n print("v=" + (n > 2 ? "big" : "small"))\n}\n' > "$WORK/t_paren.naab" +printf 'fn f(n) {\n return g(n > 2 ? 1 : 0)\n}\nfn g(x) { return x }\nmain { print(f(3)) }\n' > "$WORK/t_call.naab" +for case_ in "T-01:t_let:let (control)" "T-02:t_dict:dict literal value" "T-03:t_paren:parenthesised" "T-04:t_call:call argument"; do + id="${case_%%:*}"; rest="${case_#*:}"; f="${rest%%:*}"; label="${rest#*:}" + out="$(run "" "$f.naab")" + if [[ "$out" == *"ternary operator"* ]] && [[ "$out" == *"if condition { value_a } else { value_b }"* ]] \ + && [[ "$out" != *"missing"*"closing"* ]]; then + pass "$id" "ternary in a $label: hint shown" + else + fail "$id" "ternary in a $label: no hint" "$(head -3 <<<"$out")" + fi +done + +echo "" +echo "Results: $PASS passed, $FAIL failed" +[ "$FAIL" -eq 0 ] diff --git a/tests/security/test_child_memory_limit.sh b/tests/security/test_child_memory_limit.sh new file mode 100755 index 000000000..ab101f1f3 --- /dev/null +++ b/tests/security/test_child_memory_limit.sh @@ -0,0 +1,100 @@ +#!/usr/bin/env bash +# test_child_memory_limit.sh -- the child memory budget must bound real memory +# without refusing runtimes that merely RESERVE address space. +# +# Children of process.run / polyglot subprocesses used to get RLIMIT_AS at the +# memory budget. Address space counts reservations that are never touched, and +# node 22 needs 512-768 MB of it just to start, so under `sandbox_level: +# elevated` process.run("node", ...) died with "Fatal process out of memory: +# Failed to reserve virtual memory" before running a line (found while +# building a real project, repo-sentinel). The budget now applies to +# RLIMIT_DATA, with an address-space ceiling (4x, at least 2 GB) kept as a +# backstop for shared anonymous memory, which RLIMIT_DATA does not count. +# +# M-01 is the fix. M-02 and M-03 are why the fix is not "drop the limit": +# real private allocation over the budget, and a shared anonymous mapping +# past the ceiling, must both still fail. M-04 is their control -- an +# allocation well under the budget succeeds, so M-02/M-03 cannot pass +# because the child failed for some other reason. +# Linux only: the rlimit semantics are Linux's, and Windows children are +# contained by a job object instead. + +set -uo pipefail +PASS=0 +FAIL=0 +SKIP=0 +REPO="$(cd "$(dirname "$0")/../.." && pwd)" +NAAB="$REPO/build/naab-lang" +WORK="$(mktemp -d "${TMPDIR:-/tmp}/naab_cmem.XXXXXX")" +[ -n "$WORK" ] && [ -d "$WORK" ] || { echo "FATAL: could not create work dir" >&2; exit 1; } +source "$REPO/tests/helpers/trust_setup.sh" +setup_isolated_trust +trap 'rm -rf "$WORK"; teardown_isolated_trust' EXIT + +if [ ! -x "$NAAB" ]; then + echo "FAIL: naab-lang not built at $NAAB (UNMEASURABLE, not a pass)" + exit 1 +fi +if [ "$(uname -s)" != "Linux" ]; then + echo "SKIP: rlimit containment is Linux-specific (UNMEASURABLE here)" + exit 0 +fi + +pass() { echo " PASS [$1] $2"; PASS=$((PASS+1)); } +fail() { echo " FAIL [$1] $2"; [ -n "${3:-}" ] && echo " $3"; FAIL=$((FAIL+1)); } +skip() { echo " SKIP [$1] $2 (UNMEASURABLE)"; SKIP=$((SKIP+1)); } + +echo '{ "version": "4.0", "mode": "enforce", "security": { "sandbox_level": "elevated" } }' > "$WORK/govern.json" + +# child ID CMD ARGS_NAAB_ARRAY -> OUT (the child's stdout+stderr, via process.run) +child() { + cat > "$WORK/$1.naab" <&1)" +} + +echo "=== M: child memory containment ===" +if command -v node >/dev/null 2>&1; then + child m01 node '["-e", "console.log(\"NODE_OK\")"]' + case "$OUT" in + *STDOUT:NODE_OK*) pass M-01 "node starts under the elevated memory budget" ;; + *) fail M-01 "node could not start under the budget" "$(grep -m1 -i 'memory\|STDERR' <<<"$OUT")" ;; + esac +else + skip M-01 "node not installed" +fi + +if command -v python3 >/dev/null 2>&1; then + child m04 python3 '["-c", "a = bytearray(64 * 1024 * 1024); print(\"SMALL_OK\")"]' + if [[ "$OUT" != *STDOUT:SMALL_OK* ]]; then + skip M-02 "python3 cannot run as a child here (control M-04 failed)" + skip M-03 "python3 cannot run as a child here (control M-04 failed)" + fail M-04 "a 64 MB allocation failed -- the budget is too tight or python3 is broken" "$(head -3 <<<"$OUT")" + else + pass M-04 "control: a 64 MB allocation succeeds" + child m02 python3 '["-c", "a = bytearray(3 * 1024 * 1024 * 1024); a[-1] = 1; print(\"BIG_OK\")"]' + if [[ "$OUT" == *STDOUT:BIG_OK* ]]; then + fail M-02 "a 3 GB private allocation succeeded -- the budget no longer bounds memory" + else + pass M-02 "a 3 GB private allocation is refused" + fi + child m03 python3 '["-c", "import mmap; m = mmap.mmap(-1, 5 * 1024 * 1024 * 1024); print(\"SHARED_OK\")"]' + if [[ "$OUT" == *STDOUT:SHARED_OK* ]]; then + fail M-03 "a 5 GB shared anonymous mapping succeeded -- no address-space backstop" + else + pass M-03 "a 5 GB shared anonymous mapping is refused by the ceiling" + fi + fi +else + skip M-02 "python3 not installed"; skip M-03 "python3 not installed"; skip M-04 "python3 not installed" +fi + +echo "" +echo "Results: $PASS passed, $FAIL failed, $SKIP unmeasurable" +[ "$FAIL" -eq 0 ] diff --git a/tests/security/test_timeout_reach.sh b/tests/security/test_timeout_reach.sh new file mode 100755 index 000000000..33ced39b7 --- /dev/null +++ b/tests/security/test_timeout_reach.sh @@ -0,0 +1,210 @@ +#!/usr/bin/env bash +# test_timeout_reach.sh -- --timeout must stop work that never returns to NAAb. +# +# The execution timeout is a flag the interpreter POLLS. Work that runs inside +# the process without polling it only noticed the timeout once it returned by +# itself. Two such paths, both measured before this suite with --timeout 3: +# +# <> busy loop (20s) ran 20.1s -- embedded CPython had no +# interrupt path (QuickJS has one) +# http.get(url, {}, 0), SYN-dropping host still connecting at 20s -- +# timeout_ms=0 is libcurl's "never", +# and curl never returns mid-transfer +# +# Group P: Python. Asserted on WALL TIME, not the error text: the old build +# also ended with a timeout error, 17 seconds late. +# P-03 catches every Exception in a loop -- the interrupt must re-arm, +# or one `except Exception: pass` keeps the block alive forever. +# P-04 is the control that a Python block still COMPLETES under a +# timeout, so P-01..03 cannot pass by refusing every block. +# Group H: http. Needs an address where a connect attempt hangs; the viability +# probe checks that first and reports UNMEASURABLE otherwise, since on +# a network that refuses the connection instantly H-01 would pass for +# free. +# +# Group N: nesting. One timer per thread, and every executor wrapped its block +# in its own ScopedTimeout, which re-armed that timer and CLEARED it on +# exit -- so after a single <> expression the script's +# --timeout was gone (measured: `while true` ran until killed from +# outside, both engines). codegen.run did the same by hand. The loop +# AFTER the block is what must be stopped. +# +# Known limit, not asserted: a Python block blocked inside C (time.sleep, a +# socket read) is not interrupted -- CPython runs the interrupt only between +# bytecodes. Python on worker threads (parallel polyglot groups) is not either: +# CPython runs pending calls on its main thread only. + +set -uo pipefail +PASS=0 +FAIL=0 +SKIP=0 +REPO="$(cd "$(dirname "$0")/../.." && pwd)" +NAAB="$REPO/build/naab-lang" +WORK="$(mktemp -d "${TMPDIR:-/tmp}/naab_tmo.XXXXXX")" +[ -n "$WORK" ] && [ -d "$WORK" ] || { echo "FATAL: could not create work dir" >&2; exit 1; } +source "$REPO/tests/helpers/trust_setup.sh" +setup_isolated_trust +trap 'rm -rf "$WORK"; teardown_isolated_trust' EXIT + +if [ ! -x "$NAAB" ]; then + echo "FAIL: naab-lang not built at $NAAB (UNMEASURABLE, not a pass)" + exit 1 +fi + +pass() { echo " PASS [$1] $2"; PASS=$((PASS+1)); } +fail() { echo " FAIL [$1] $2"; [ -n "${3:-}" ] && echo " $3"; FAIL=$((FAIL+1)); } +skip() { echo " SKIP [$1] $2 (UNMEASURABLE)"; SKIP=$((SKIP+1)); } + +# run DIR FILE [flags...] -> OUT, RC, ELAPSED (whole seconds). The outer +# timeout is well past the old failure time, so an unfixed build is reported +# as slow rather than killed silently. +run() { + local dir="$1" file="$2"; shift 2 + local start=$SECONDS + OUT="$(cd "$dir" && timeout 40 "$NAAB" "$@" "$file" 2>&1)" + RC=$? + ELAPSED=$((SECONDS - start)) +} + +mkdir -p "$WORK/p" "$WORK/h" +echo '{ "version": "4.0", "mode": "off" }' > "$WORK/p/govern.json" + +echo "=== P: embedded Python ===" +cat > "$WORK/p/ctl.naab" <<'EOF' +main { + let a = <> + let b = <> + print("CTL:" + string(a) + "," + string(b)) +} +EOF +run "$WORK/p" ctl.naab --timeout 5 +if [[ "$OUT" != *"CTL:499500,3"* ]]; then + for id in P-01/vm P-01/tree-walk P-03 P-04; do + skip "$id" "embedded Python executor not available in this build" + done +else + pass P-04 "control: Python blocks still complete under --timeout (${ELAPSED}s)" + + cat > "$WORK/p/busy.naab" <<'EOF' +main { + <> + print("AFTER") +} +EOF + for eng in "" "--tree-walk"; do + tag="${eng:-vm}"; tag="${tag#--}" + run "$WORK/p" busy.naab $eng --timeout 3 + if [ "$ELAPSED" -le 8 ] && [ "$RC" -ne 0 ] && [[ "$OUT" != *AFTER* ]]; then + pass "P-01/$tag" "20s Python loop stopped at the 3s limit (${ELAPSED}s, rc=$RC)" + else + fail "P-01/$tag" "Python loop not stopped (took ${ELAPSED}s, rc=$RC)" "$(tail -2 <<<"$OUT")" + fi + done + + cat > "$WORK/p/swallow.naab" <<'EOF' +main { + <> + print("AFTER") +} +EOF + run "$WORK/p" swallow.naab --timeout 3 + if [ "$ELAPSED" -le 8 ] && [[ "$OUT" != *AFTER* ]]; then + pass P-03 "a loop that catches every Exception is still stopped (${ELAPSED}s)" + else + fail P-03 "the interrupt was swallowed (took ${ELAPSED}s)" "$(tail -2 <<<"$OUT")" + fi +fi + +echo "=== H: http ===" +echo '{ "version": "4.0", "mode": "enforce", "security": { "sandbox_level": "elevated" }, "capabilities": { "network": { "enabled": true } } }' > "$WORK/h/govern.json" +HANG_URL="http://8.8.8.8:81/" # port 81 is not served; SYNs are dropped +cat > "$WORK/h/probe.naab" < "$WORK/h/zero.naab" < "$WORK/n/govern.json" +cat > "$WORK/n/js.naab" <<'EOF' +main { + let v = <> + print("JS:" + string(v)) + let i = 0 + while true { i = i + 1 } +} +EOF +cat > "$WORK/n/cg.naab" <<'EOF' +use codegen +main { + let r = codegen.run("javascript", "1 + 1") + print("CG:" + string(r["exit_code"])) + let i = 0 + while true { i = i + 1 } +} +EOF +for prog in js:JS:2 cg:CG:0; do + f="${prog%%:*}"; marker="${prog#*:}" + for eng in "" "--tree-walk"; do + tag="${eng:-vm}"; tag="${tag#--}" + run "$WORK/n" "$f.naab" $eng --timeout 3 + # Slow is a failure whatever the output says: a run killed from outside + # loses its buffered stdout, so the marker cannot be required first. + if [ "$ELAPSED" -gt 8 ]; then + fail "N-$f/$tag" "the $f block cancelled --timeout (took ${ELAPSED}s)" + elif [[ "$OUT" != *"$marker"* ]]; then + # The block itself did not run, so the loop after it proves nothing. + skip "N-$f/$tag" "the $f block did not run here" + else + pass "N-$f/$tag" "--timeout still applies after a $f block (${ELAPSED}s)" + fi + done +done + +echo "" +echo "Results: $PASS passed, $FAIL failed, $SKIP unmeasurable" +[ "$FAIL" -eq 0 ] diff --git a/tests/unit/known_failures.txt b/tests/unit/known_failures.txt new file mode 100644 index 000000000..069329fa8 --- /dev/null +++ b/tests/unit/known_failures.txt @@ -0,0 +1,43 @@ +# Unit tests excluded from tests/unit/run_unit_tests.sh. +# +# Every entry is a test that fails for a reason recorded in +# docs/unit-test-findings.md -- nothing here is excluded because it was +# inconvenient. Format: Suite.Test +# +# fails -- run separately; the runner FAILS if it starts passing, so a fix +# must also remove the entry (the list cannot quietly go stale). +# hang -- not run at all: it deadlocks or crashes the process. +# +# Stale tests: the code changed on purpose (section 1). +LexerTest.MultipleStatementsOnOneLine fails stale: newline tokens are now emitted +LexerTest.PipeOperator fails stale: |> is PIPELINE, not PIPE +LexerTest.SingleLineComment fails stale: newline tokens are now emitted +LexerTest.Whitespace fails stale: newline tokens are now emitted +ParserTest.DictLiteral fails stale: `{` at statement start is a block +ParserTest.TryFinallyBlock fails stale: try requires catch +InterpreterTest.Division fails stale: DIV-001, division is always double +InterpreterTest.FinallyBlock fails stale: try requires catch +StdLibTest.AllModulesAvailable fails stale: 13 modules became 25 +MathModuleTest.Abs fails stale: math.abs returns float +MathModuleTest.Ceil fails stale: math.ceil returns int +MathModuleTest.Floor fails stale: math.floor returns int +MathModuleTest.Round fails stale: math.round returns int +StructRegistryTest.DuplicateThrows fails stale: ISS-036 made identical re-registration idempotent +PolyglotAsyncTest.CppSimpleExecution fails stale: C++ snippet uses the pre-NaabVal ABI +PolyglotAsyncTest.CppBlockingExecution fails stale: C++ snippet uses the pre-NaabVal ABI +PolyglotAsyncTest.ShellBlockingExecution fails fixture: callback thread has no sandbox, so fail-closed denies it +PolyglotAsyncTest.ShellConcurrentExecutions fails fixture: callback thread has no sandbox, so fail-closed denies it +# +# Real defects, not reachable from the CLI today (section 3). +PolyglotAsyncTest.PythonTimeout fails async executors ignore timeout +PolyglotAsyncTest.ShellWithTimeout fails async executors ignore timeout +FFICallbackValidatorTest.GetTypeNameReturnsCorrectNames fails FFI callback validator is a stub +FFICallbackValidatorTest.GuardDetectsSignatureMismatch fails FFI callback validator is a stub +FFICallbackValidatorTest.RejectsIncorrectReturnType fails FFI callback validator is a stub +FFICallbackValidatorTest.RejectsSignatureMismatch fails FFI callback validator is a stub +FFICallbackValidatorTest.RejectsTypeMismatch fails FFI callback validator is a stub +FFIAsyncCallbackTest.CancelDuringExecution fails AsyncCallbackPool / deferred launch +FFIAsyncCallbackTest.ExecuteRaceFirstWins fails AsyncCallbackPool / deferred launch +FFIAsyncCallbackTest.PoolConcurrencyLimit hang AsyncCallbackPool deadlock (deferred launch) +FFIAsyncCallbackTest.PoolCancelAll hang AsyncCallbackPool deadlock (deferred launch) +FFIAsyncCallbackTest.PoolThreadSafety hang AsyncCallbackPool use-after-free (segfaults) diff --git a/tests/unit/polyglot_async_test.cpp b/tests/unit/polyglot_async_test.cpp index ea5815936..b9a0909fb 100644 --- a/tests/unit/polyglot_async_test.cpp +++ b/tests/unit/polyglot_async_test.cpp @@ -494,7 +494,7 @@ TEST_F(PolyglotAsyncTest, ShellBlockingExecution) { std::string command = "echo 42"; auto result = executor.executeBlocking(command, {}); - EXPECT_TRUE(result.success); + EXPECT_TRUE(result.success) << "Error: " << result.error_message; } TEST_F(PolyglotAsyncTest, ShellConcurrentExecutions) { @@ -518,7 +518,7 @@ TEST_F(PolyglotAsyncTest, ShellConcurrentExecutions) { // Verify all succeeded for (int i = 0; i < num_threads; ++i) { - EXPECT_TRUE(results[i].success); + EXPECT_TRUE(results[i].success) << "Error: " << results[i].error_message; } } diff --git a/tests/unit/run_unit_tests.sh b/tests/unit/run_unit_tests.sh new file mode 100755 index 000000000..b09bdb48e --- /dev/null +++ b/tests/unit/run_unit_tests.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# run_unit_tests.sh -- run naab_unit_tests minus the documented exclusions. +# +# naab_unit_tests was built and run by nothing for long enough that five of its +# nineteen files stopped compiling unnoticed. This is what CI runs so that +# cannot happen again. Exclusions live in known_failures.txt, each tied to a +# finding in docs/unit-test-findings.md. +# +# 1. Every test NOT in the list must pass. +# 2. Every `fails` entry is run on its own and must still FAIL: a test that +# starts passing means something was fixed and the entry must go, so the +# list cannot silently outlive its reasons. +# 3. `hang` entries are not run (they deadlock or crash the process). +# +# Usage: bash tests/unit/run_unit_tests.sh [path/to/naab_unit_tests] + +set -uo pipefail +REPO="$(cd "$(dirname "$0")/../.." && pwd)" +BIN="${1:-$REPO/build/naab_unit_tests}" +LIST="$REPO/tests/unit/known_failures.txt" + +if [ ! -x "$BIN" ]; then + echo "FAIL: $BIN not built (UNMEASURABLE, not a pass)" + exit 1 +fi + +# Run from the repo root: several tests use repo-relative fixtures. The +# tamper-evident logger tests leave their logs in the working directory. +cd "$REPO" || exit 1 +trap 'rm -f "$REPO"/test_tamper_evident_*.log "$REPO"/test_tamper_evident_*.log.tamper_evident' EXIT + +all=() +fails=() +while read -r name kind _; do + [[ -z "$name" || "$name" == \#* ]] && continue + case "$kind" in + fails) fails+=("$name") ;; + hang) ;; + *) echo "FAIL: bad kind '$kind' for $name in $LIST"; exit 1 ;; + esac + all+=("$name") +done < "$LIST" + +# Guard against a typo that excludes nothing: every entry must name a real test. +listed="$("$BIN" --gtest_list_tests | awk '/^[^ ]/{s=$1} /^ /{print s $1}')" +bad=0 +for t in "${all[@]}"; do + if ! grep -qxF "$t" <<<"$listed"; then + echo "FAIL: known_failures.txt names a test that does not exist: $t" + bad=1 + fi +done +[ "$bad" -eq 0 ] || exit 1 + +filter="-$(IFS=:; echo "${all[*]}")" +echo "=== naab_unit_tests, ${#all[@]} documented exclusions ===" +timeout 900 "$BIN" --gtest_filter="$filter" --gtest_brief=1 +rc=$? +if [ "$rc" -ne 0 ]; then + echo "FAIL: naab_unit_tests exited $rc" + exit 1 +fi + +echo "=== excluded tests must still fail ===" +fixed=0 +for t in "${fails[@]}"; do + if timeout 120 "$BIN" --gtest_filter="$t" >/dev/null 2>&1; then + echo " NOW PASSES: $t -- remove it from known_failures.txt" + fixed=1 + fi +done +if [ "$fixed" -ne 0 ]; then + echo "FAIL: known_failures.txt lists tests that now pass" + exit 1 +fi +echo " all ${#fails[@]} still fail for their recorded reason" +echo "PASS"