Repository navigation
Add optional NOOA investigations and Responses support - #184
Conversation
Let xLens investigate recorded evidence through an opt-in NOOA runtime. Keep its dependencies outside base, GPU and Dragon installations, and bound generated Python in an isolated worker. Add explicit Responses API selection while preserving tool reasoning between calls. Document optional Rosalind configuration and the model access and compatibility requirements. Signed-off-by: Trent Nelson <trentn@nvidia.com>
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one): 1
Please check the settings in the CodeRabbit UI or the
|
melo-gonzo
left a comment
There was a problem hiding this comment.
Thanks for this. The sandbox command, the frame budgets and the API key handling look careful. Existing Chat Completions configs send the same requests as on main.
The main blocker is CI. In the new test-nooa job, bwrap fails on hosted Ubuntu 24.04, and ci-required now depends on that job. The inline comments cover that, what users see when the worker cannot start, three transport points, two tests and the docs.
Test gaps that do not need their own threads:
- No test sends two tool calls in one model turn. Adding
pending.clear()afterpending.remove(call_id)in_responses.pyrejects the second tool output, and all 16test_responses.pytests still pass. test_parent_broker_with_stdio_workeris the only NOOA protocol test that runs on a base install. There, 22 of 26 mutants of the parent broker innooa_backend.pysurvivetest_nooa_backend.py. They include the byte limits on frames, model requests and answers, the domain read caps and the tool-count limits. One stdio probe per check would cover most of them.- No test passes a
domain_querytoInvestigator.investigate, sodiagnostic()(inference.py:474-501) never runs. The NOOA branch ofinvestigate(inference.py:514-558) runs only through the real worker. A fakerun_nooawould let both run on a base install.
Also: the CHANGELOG entry says "Restore NOOA" and "supersedes the earlier development removal". No release shipped NOOA: #135 added it and #171 removed it before v0.2.0. For release-note readers this is a new feature. The entry also does not name the new api setting.
Mutation runs on a base install (the real-worker tests skip)
test_responses.py, unmodified -> 16 passed
pending.clear() after pending.remove(call_id) -> 16 passed
test_nooa_backend.py, unmodified -> 6 passed, 7 skipped
killed (4): tool allowlist off, iteration limit off by one,
findings bound 21, domain allowlist off
survived (22): payload key check, query/final/model tool limits,
domain total and per-result caps, domain sample check,
compare bound 9, bool limit, evidence references,
answer type, answer/model response/model payload byte
limits, frame limit default, unknown query, spawn env,
status extra args, tool fields, error frame, round trip,
first call only
Check bubblewrap namespace support before enabling investigation and keep worker failures distinct from endpoint failures. Accept nullable tool calls and validate model arguments consistently across both APIs. Exercise broker limits and cleanup without optional dependencies, and configure hosted Ubuntu workers for unprivileged namespaces. Describe NOOA as a new optional feature without unreleased development history. Signed-off-by: Trent Nelson <trentn@nvidia.com>
melo-gonzo
left a comment
There was a problem hiding this comment.
Thanks for the fixes. I re-ran our checks at eac5e9f, and all nine points are fixed.
- On a host with the stock Ubuntu 24.04 AppArmor setting,
check_available()now fails before a report is created. It names bwrap's own message. - An early worker exit now gives the same worker-stopped error in all 60 runs, with no "started" event.
"tool_calls": nullcompletes. Malformed or non-object tool arguments fail the same way under both APIs, with "model returned invalid tool arguments".- Without HTTPX, the NOOA backend names
cuphoton[xlens-nooa]. On macOS, the platform error comes first. - Both hosted NOOA jobs pass 141 tests with none skipped, so the real sandbox tests ran.
- The new tests catch each regression we tried: removing the kill, the namespace check, the null handling, the
ConnectionErrormapping or the argument check, and emitting "started" early.
Two hardening points for a follow-up. Neither blocks this PR.
_command()passes no--seccompfilter. Inside the sandbox, namespaces,--cap-drop ALLand--disable-usernsare the only limits on kernel calls. A deny list for calls the worker never needs, passed with--seccomp FD, would add a layer behind them.- I could not check this one, because bwrap cannot start where I test. Inside the sandbox,
/proc/self/mountinfoprobably lists the host source path of each bind mount: the interpreter, the package directories and_nooa_worker.py. That would show host paths, such as the checkout or home directory, to model-generated code, which can pass them to the model. Reading/proc/self/mountinfointest_sandbox_hides_host_files_environment_and_networkand checking for the checkout path would settle it.
Reproducers at eac5e9f
The fake-worker cases replace httpx with an in-memory stub and use a protocol-only worker. The availability check fakes nooa and httpx; bwrap is real.
availability, stock Ubuntu 24.04 setting:
NooaError | NOOA isolation unavailable: bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted
missing HTTPX, backend="nooa":
InferenceError | Install cuphoton[xlens-nooa] for investigation.
macOS, backend="nooa":
NooaError: NOOA requires Linux and Python 3.12 or 3.13
worker = /usr/bin/false, 15 rounds of 4 concurrent investigations:
60 ('failed', 'NOOA worker stopped; verify bubblewrap user namespace support and the optional installation', 0)
tool_calls omitted -> completed | null -> completed | [] -> completed
args '{"code": "print(1)"' chat: failed model returned invalid tool arguments | responses: same
args '["print(1)"]' chat: failed model returned invalid tool arguments | responses: same
tests/xlens (base install, unmodified): 84 passed, 13 skipped
mutants (test_nooa_backend.py, plus test_inference.py or test_responses.py where relevant):
killpg removed -> 32 failed (no hang)
namespace probe removed -> 1 failed
null tool_calls rejected -> 2 failed
ConnectionError unmapped -> 2 failed
"started" before frame -> 1 failed
argument type check off -> 2 failed
Remove procfs from the worker so mount sources and launcher arguments cannot expose host paths to generated code. Build a native syscall deny list with system libseccomp and pass it to bubblewrap with scoped file descriptor ownership. Keep libseccomp an external requirement of the optional NOOA runtime, and document its use alongside bubblewrap. Signed-off-by: Trent Nelson <trentn@nvidia.com>
|
Addressing the syscall-filter follow-up in your review: NOOA now builds a native-ABI deny list with system libseccomp and passes it to bubblewrap for both the availability probe and the actual worker. Missing filtering disables NOOA; filter descriptors are closed on success and failure. The system library is required only for NOOA and is documented in the installation guide and third-party notices. Implemented in b83d600c. Real worker tests verify filter mode and EPERM for personality and mount-metadata calls while normal evidence tools still execute. All 145 xLens tests pass without skips on x86-64 Python 3.12, x86-64 Python 3.13, and ARM64 Python 3.13. Removing only the actual worker filter makes the regression test fail. |
|
Addressing the mount-metadata follow-up in your review: confirmed. Mount tables exposed bind-source paths, and the bubblewrap guardian's Fixed in b83d600c by giving the worker an empty, read-only |
melo-gonzo
left a comment
There was a problem hiding this comment.
Thanks for taking both follow-ups. I re-reviewed b83d600, and the filter and the empty /proc look right. A plain child process that loads the exported filter gets these results. i386 int 0x80 and x32 calls are killed. Each listed call returns EPERM. clone3 stays allowed, and the process limit and dropped capabilities still block it. The parent fails closed when libseccomp or the filter is unavailable. The filter descriptor closes after success, after an error and after cancellation.
Considerations, none blocking:
- Host paths still reach the worker without procfs.
/runtime/libis the interpreter's wholelibdirectory, so_sysconfigdata_*.pycontains the install prefix. For a uv-managed or pyenv Python, that prefix is under the user's home. The.pycfiles in the mounted packages record absolute source paths inco_filename. The worker also keeps the host's hostname, because bwrap sets one only with--hostname. The sentence atdocs/components/xlens.md:361-362claims more than this. Passing--hostnameand narrowing that sentence would cover it. - The worker test checks 3 of the 37 rules, so dropping
bpfor any other unchecked rule still passes. A unit test could load the exported filter in a forked child and check every rule plus the i386 kill. It would run on any Linux runner, without bwrap. - Socket families are not filtered. The network namespace covers the IP families, but some families, such as AF_VSOCK on some kernels, do not follow network namespaces. Allowing only the families the worker needs would close that. This point comes from code reading.
Probes at b83d600
The filter was exported from the head module and loaded in a plain child after PR_SET_NO_NEW_PRIVS. bwrap cannot start on the test host.
call no filter filter
PR_GET_SECCOMP 0 2
personality(-1) 0 EPERM
statmount, listmount EINVAL EPERM
unshare, bpf, io_uring_setup EINVAL/E2BIG/EFAULT EPERM
clone3(0, 0) EINVAL EINVAL (allowed)
i386 int 0x80 returns pid killed by signal 31
x32 getpid ENOSYS killed by signal 31
The worker test's seccomp assertions were run in a forked child against filter mutants. Removing the filter, personality, or statmount/listmount, or changing EPERM to ENOSYS, fails them. Removing bpf passes.
Host files a uv-managed Python exposes through the mounts:
interpreter base under home: True
sysconfigdata files: 1 | home path occurrences: 27
site-packages .pyc files: 3866 | first co_filename under home: True
Allow Unix-domain sockets for local event-loop control while denying other socket families that may not follow network namespaces. Give the worker a fixed hostname and document paths retained in runtime metadata. Exercise each denied syscall with independent policy expectations and a distinct denial errno so host permissions cannot hide a missing rule. Cover foreign-ABI rejection and preserve normal Unix socketpair use. Signed-off-by: Trent Nelson <trentn@nvidia.com>
|
Addressed the host-metadata point from your review in 606083c2. The worker now uses the fixed hostname |
|
Addressed the test-coverage point from your review in 606083c2. A plain child process now loads the exported filter and checks an independent list of all 37 denied syscalls, plus i386/x32 rejection. The probes use harmless invalid arguments and a distinctive filter errno so host permission errors cannot hide a missing rule. Removing each rule individually made its test fail; omitting the filter failed all 47 negative checks. The 52-test filter suite passes on Python 3.12/3.13 and in the base environment without NOOA or HTTPX. |
|
Addressed the socket-family point from your review in 606083c2. Both |
melo-gonzo
left a comment
There was a problem hiding this comment.
Thanks, all three points are covered at 606083c. The AF_UNIX rule compares the whole argument, so extra high bits cannot pass another family, and io_uring_setup stays denied, so io_uring cannot create sockets around it. test_nooa_seccomp.py passes 52 tests and catches each change I tried.
Mutants of _nooa_seccomp.py, run with tests/xlens/test_nooa_seccomp.py
unmodified -> 52 passed
socket/socketpair rule removed -> 8 failed, 44 passed
"bpf" removed from the deny list -> 1 failed, 51 passed
allowed family AF_UNIX -> AF_INET -> 3 failed, 49 passed
Add optional NOOA investigations so xLens can query recorded evidence and run isolated Python across multiple model turns. Users opt in with
cuphoton[xlens-nooa]; basic inference remains the default. NOOA requires Linux, Python 3.12/3.13, bubblewrap, libseccomp and a non-root user, with namespace support and syscall filtering checked before enabling CLI or session investigation.Add
api = "responses"alongside Chat Completions, preserving provider reasoning across tool calls. Document optionalgpt-6-rosalindconfiguration for users with API access. Live Rosalind compatibility remains unqualified. This change contains no EDP2 integration.The worker prevents child/thread creation and confines temporary writes to a bounded workspace. Procfs is unavailable, and a syscall deny list adds restrictions before Python starts. NOOA is selected only by its extra, and upstream dependencies are not bundled into cuPhoton artifacts. The changelog describes the new feature without unreleased development history.
Validation
Repository hygiene