Skip to content

Add optional NOOA investigations and Responses support - #184

Merged
tpn merged 4 commits into
mainfrom
codex/restore-optional-nooa
Nov 12, 2026
Merged

tpn merged 4 commits into
mainfrom
codex/restore-optional-nooa

Conversation

@tpn

@tpn tpn commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Add optional NOOA investigations so xLens can query recorded evidence and run isolated Python across multiple model turns. Users opt in with cuphoton[xlens-nooa]; basic inference remains the default. NOOA requires Linux, Python 3.12/3.13, bubblewrap, libseccomp and a non-root user, with namespace support and syscall filtering checked before enabling CLI or session investigation.

Add api = "responses" alongside Chat Completions, preserving provider reasoning across tool calls. Document optional gpt-6-rosalind configuration for users with API access. Live Rosalind compatibility remains unqualified. This change contains no EDP2 integration.

The worker prevents child/thread creation and confines temporary writes to a bounded workspace. Procfs is unavailable, and a syscall deny list adds restrictions before Python starts. NOOA is selected only by its extra, and upstream dependencies are not bundled into cuPhoton artifacts. The changelog describes the new feature without unreleased development history.

Validation

  • Full xLens and package-layout suites on Python 3.12/3.13, including real isolated workers and concurrent HTTP tool loops for both APIs.
  • Base-install protocol tests exercise worker startup failures, evidence and tool budgets, malformed arguments, and diagnostic callbacks. Mutation checks confirm multiple tool outputs and missing worker termination are detected.
  • Basic Python 3.14 checks, repository hooks, typing and lock consistency.
  • Clean base and optional-extra wheel installation checks; no upstream dependency trees are bundled.
  • Hosted NOOA jobs load a bubblewrap-specific AppArmor profile and probe namespace support before running real worker tests on x86-64 and AArch64. Regression tests check inaccessible process metadata, kernel-enforced syscall denial and filter descriptor cleanup.

Repository hygiene

  • No credentials, private paths, local datasets, notebook outputs, or generated run artifacts were added.
  • Large binary artifacts are excluded or intentionally tracked through an approved storage plan.
  • Security-sensitive information is not included.
  • User-visible behavior changes are documented.
  • New dependencies and their licenses are documented.

Let xLens investigate recorded evidence through an opt-in NOOA runtime.
Keep its dependencies outside base, GPU and Dragon installations, and
bound generated Python in an isolated worker.

Add explicit Responses API selection while preserving tool reasoning
between calls. Document optional Rosalind configuration and the model
access and compatibility requirements.

Signed-off-by: Trent Nelson <trentn@nvidia.com>
@tpn
tpn requested a review from melo-gonzo as a code owner November 11, 2026 20:58
@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one): 1

  • ai-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

@melo-gonzo melo-gonzo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The sandbox command, the frame budgets and the API key handling look careful. Existing Chat Completions configs send the same requests as on main.

The main blocker is CI. In the new test-nooa job, bwrap fails on hosted Ubuntu 24.04, and ci-required now depends on that job. The inline comments cover that, what users see when the worker cannot start, three transport points, two tests and the docs.

Test gaps that do not need their own threads:

  • No test sends two tool calls in one model turn. Adding pending.clear() after pending.remove(call_id) in _responses.py rejects the second tool output, and all 16 test_responses.py tests still pass.
  • test_parent_broker_with_stdio_worker is the only NOOA protocol test that runs on a base install. There, 22 of 26 mutants of the parent broker in nooa_backend.py survive test_nooa_backend.py. They include the byte limits on frames, model requests and answers, the domain read caps and the tool-count limits. One stdio probe per check would cover most of them.
  • No test passes a domain_query to Investigator.investigate, so diagnostic() (inference.py:474-501) never runs. The NOOA branch of investigate (inference.py:514-558) runs only through the real worker. A fake run_nooa would let both run on a base install.

Also: the CHANGELOG entry says "Restore NOOA" and "supersedes the earlier development removal". No release shipped NOOA: #135 added it and #171 removed it before v0.2.0. For release-note readers this is a new feature. The entry also does not name the new api setting.

Mutation runs on a base install (the real-worker tests skip)
test_responses.py, unmodified                         -> 16 passed
pending.clear() after pending.remove(call_id)          -> 16 passed

test_nooa_backend.py, unmodified                       -> 6 passed, 7 skipped
killed (4):   tool allowlist off, iteration limit off by one,
              findings bound 21, domain allowlist off
survived (22): payload key check, query/final/model tool limits,
              domain total and per-result caps, domain sample check,
              compare bound 9, bool limit, evidence references,
              answer type, answer/model response/model payload byte
              limits, frame limit default, unknown query, spawn env,
              status extra args, tool fields, error frame, round trip,
              first call only

Comment thread .github/workflows/ci.yml Resolved
Comment thread src/cuphoton/xlens/nooa_backend.py Resolved
Comment thread src/cuphoton/xlens/nooa_backend.py Resolved
Comment thread src/cuphoton/xlens/nooa_backend.py Resolved
Comment thread src/cuphoton/xlens/_responses.py Resolved
Comment thread src/cuphoton/xlens/inference.py Resolved
Comment thread docs/components/xlens.md Resolved
Comment thread tests/xlens/test_nooa_backend.py Resolved
Comment thread tests/xlens/test_nooa_backend.py Resolved
Check bubblewrap namespace support before enabling investigation and
keep worker failures distinct from endpoint failures. Accept nullable
tool calls and validate model arguments consistently across both APIs.

Exercise broker limits and cleanup without optional dependencies, and
configure hosted Ubuntu workers for unprivileged namespaces. Describe
NOOA as a new optional feature without unreleased development history.

Signed-off-by: Trent Nelson <trentn@nvidia.com>
@tpn tpn changed the title Restore optional NOOA investigations and Responses support Add optional NOOA investigations and Responses support Nov 12, 2026

@melo-gonzo melo-gonzo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fixes. I re-ran our checks at eac5e9f, and all nine points are fixed.

  • On a host with the stock Ubuntu 24.04 AppArmor setting, check_available() now fails before a report is created. It names bwrap's own message.
  • An early worker exit now gives the same worker-stopped error in all 60 runs, with no "started" event.
  • "tool_calls": null completes. Malformed or non-object tool arguments fail the same way under both APIs, with "model returned invalid tool arguments".
  • Without HTTPX, the NOOA backend names cuphoton[xlens-nooa]. On macOS, the platform error comes first.
  • Both hosted NOOA jobs pass 141 tests with none skipped, so the real sandbox tests ran.
  • The new tests catch each regression we tried: removing the kill, the namespace check, the null handling, the ConnectionError mapping or the argument check, and emitting "started" early.

Two hardening points for a follow-up. Neither blocks this PR.

  • _command() passes no --seccomp filter. Inside the sandbox, namespaces, --cap-drop ALL and --disable-userns are the only limits on kernel calls. A deny list for calls the worker never needs, passed with --seccomp FD, would add a layer behind them.
  • I could not check this one, because bwrap cannot start where I test. Inside the sandbox, /proc/self/mountinfo probably lists the host source path of each bind mount: the interpreter, the package directories and _nooa_worker.py. That would show host paths, such as the checkout or home directory, to model-generated code, which can pass them to the model. Reading /proc/self/mountinfo in test_sandbox_hides_host_files_environment_and_network and checking for the checkout path would settle it.
Reproducers at eac5e9f

The fake-worker cases replace httpx with an in-memory stub and use a protocol-only worker. The availability check fakes nooa and httpx; bwrap is real.

availability, stock Ubuntu 24.04 setting:
  NooaError | NOOA isolation unavailable: bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted
missing HTTPX, backend="nooa":
  InferenceError | Install cuphoton[xlens-nooa] for investigation.
macOS, backend="nooa":
  NooaError: NOOA requires Linux and Python 3.12 or 3.13

worker = /usr/bin/false, 15 rounds of 4 concurrent investigations:
  60 ('failed', 'NOOA worker stopped; verify bubblewrap user namespace support and the optional installation', 0)

tool_calls omitted -> completed | null -> completed | [] -> completed
args '{"code": "print(1)"'  chat: failed model returned invalid tool arguments | responses: same
args '["print(1)"]'         chat: failed model returned invalid tool arguments | responses: same

tests/xlens (base install, unmodified): 84 passed, 13 skipped
mutants (test_nooa_backend.py, plus test_inference.py or test_responses.py where relevant):
  killpg removed            -> 32 failed (no hang)
  namespace probe removed   -> 1 failed
  null tool_calls rejected  -> 2 failed
  ConnectionError unmapped  -> 2 failed
  "started" before frame    -> 1 failed
  argument type check off   -> 2 failed

Remove procfs from the worker so mount sources and launcher arguments
cannot expose host paths to generated code. Build a native syscall deny
list with system libseccomp and pass it to bubblewrap with scoped file
descriptor ownership.

Keep libseccomp an external requirement of the optional NOOA runtime,
and document its use alongside bubblewrap.

Signed-off-by: Trent Nelson <trentn@nvidia.com>
@tpn

tpn commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

Addressing the syscall-filter follow-up in your review: NOOA now builds a native-ABI deny list with system libseccomp and passes it to bubblewrap for both the availability probe and the actual worker. Missing filtering disables NOOA; filter descriptors are closed on success and failure. The system library is required only for NOOA and is documented in the installation guide and third-party notices.

Implemented in b83d600c. Real worker tests verify filter mode and EPERM for personality and mount-metadata calls while normal evidence tools still execute. All 145 xLens tests pass without skips on x86-64 Python 3.12, x86-64 Python 3.13, and ARM64 Python 3.13. Removing only the actual worker filter makes the regression test fail.

@tpn

tpn commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

Addressing the mount-metadata follow-up in your review: confirmed. Mount tables exposed bind-source paths, and the bubblewrap guardian's /proc/1/cmdline exposed launch arguments containing host paths.

Fixed in b83d600c by giving the worker an empty, read-only /proc. The filter also denies statmount and listmount. The regression checks self, thread-self, numeric-PID and task aliases, plus host-file, environment and loopback isolation. Restoring procfs makes it fail. Normal NOOA and concurrent HTTP tool loops pass on both architectures; the built wheel also passes all 49 worker/filter/HTTP checks.

@melo-gonzo melo-gonzo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking both follow-ups. I re-reviewed b83d600, and the filter and the empty /proc look right. A plain child process that loads the exported filter gets these results. i386 int 0x80 and x32 calls are killed. Each listed call returns EPERM. clone3 stays allowed, and the process limit and dropped capabilities still block it. The parent fails closed when libseccomp or the filter is unavailable. The filter descriptor closes after success, after an error and after cancellation.

Considerations, none blocking:

  • Host paths still reach the worker without procfs. /runtime/lib is the interpreter's whole lib directory, so _sysconfigdata_*.py contains the install prefix. For a uv-managed or pyenv Python, that prefix is under the user's home. The .pyc files in the mounted packages record absolute source paths in co_filename. The worker also keeps the host's hostname, because bwrap sets one only with --hostname. The sentence at docs/components/xlens.md:361-362 claims more than this. Passing --hostname and narrowing that sentence would cover it.
  • The worker test checks 3 of the 37 rules, so dropping bpf or any other unchecked rule still passes. A unit test could load the exported filter in a forked child and check every rule plus the i386 kill. It would run on any Linux runner, without bwrap.
  • Socket families are not filtered. The network namespace covers the IP families, but some families, such as AF_VSOCK on some kernels, do not follow network namespaces. Allowing only the families the worker needs would close that. This point comes from code reading.
Probes at b83d600

The filter was exported from the head module and loaded in a plain child after PR_SET_NO_NEW_PRIVS. bwrap cannot start on the test host.

call                              no filter            filter
PR_GET_SECCOMP                    0                    2
personality(-1)                   0                    EPERM
statmount, listmount              EINVAL               EPERM
unshare, bpf, io_uring_setup      EINVAL/E2BIG/EFAULT  EPERM
clone3(0, 0)                      EINVAL               EINVAL (allowed)
i386 int 0x80                     returns pid          killed by signal 31
x32 getpid                        ENOSYS               killed by signal 31

The worker test's seccomp assertions were run in a forked child against filter mutants. Removing the filter, personality, or statmount/listmount, or changing EPERM to ENOSYS, fails them. Removing bpf passes.

Host files a uv-managed Python exposes through the mounts:

interpreter base under home: True
sysconfigdata files: 1 | home path occurrences: 27
site-packages .pyc files: 3866 | first co_filename under home: True

Allow Unix-domain sockets for local event-loop control while denying
other socket families that may not follow network namespaces. Give the
worker a fixed hostname and document paths retained in runtime metadata.

Exercise each denied syscall with independent policy expectations and
a distinct denial errno so host permissions cannot hide a missing rule.
Cover foreign-ABI rejection and preserve normal Unix socketpair use.

Signed-off-by: Trent Nelson <trentn@nvidia.com>
@tpn

tpn commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the host-metadata point from your review in 606083c2. The worker now uses the fixed hostname xlens-nooa. The documentation limits the procfs claim to command lines and mount tables, and explicitly says interpreter/dependency files can retain installation paths and build metadata. The real sandbox test asserts the hostname and unavailable procfs entries; it passes on Python 3.12 and 3.13.

@tpn

tpn commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the test-coverage point from your review in 606083c2. A plain child process now loads the exported filter and checks an independent list of all 37 denied syscalls, plus i386/x32 rejection. The probes use harmless invalid arguments and a distinctive filter errno so host permission errors cannot hide a missing rule. Removing each rule individually made its test fail; omitting the filter failed all 47 negative checks. The 52-test filter suite passes on Python 3.12/3.13 and in the base environment without NOOA or HTTPX.

@tpn

tpn commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the socket-family point from your review in 606083c2. Both socket and socketpair now permit only AF_UNIX; IP, netlink, VSOCK and other families are denied by the filter. Tests verify those denials and successful Unix socketpair data transfer. Both real NOOA HTTP tool loops still pass, and the full xLens/package suite passes all 215 tests on Python 3.12 and 3.13. Hosted checks for this commit are running.

@melo-gonzo melo-gonzo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, all three points are covered at 606083c. The AF_UNIX rule compares the whole argument, so extra high bits cannot pass another family, and io_uring_setup stays denied, so io_uring cannot create sockets around it. test_nooa_seccomp.py passes 52 tests and catches each change I tried.

Mutants of _nooa_seccomp.py, run with tests/xlens/test_nooa_seccomp.py
unmodified                          -> 52 passed
socket/socketpair rule removed      -> 8 failed, 44 passed
"bpf" removed from the deny list    -> 1 failed, 51 passed
allowed family AF_UNIX -> AF_INET   -> 3 failed, 49 passed

@tpn
tpn merged commit de0264c into main Nov 12, 2026
57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants