feat(onboard): preserve Jetson GPU device groups

[cjagwani]

Summary

This draft preserves Jetson GPU policy, device-group, container-lifecycle, and validation changes after #8875 merged the CPU-only E2E path. Issue #7610 still blocks the non-root OpenShell CUDA acceptance test and the product-scope decision. Maintainers must keep this PR as a draft until those conditions are resolved.

Related Issue

Related to #7610. This PR does not resolve or close that issue.

Changes

• Detect existing, non-symbolic-link Jetson GPU character devices before adding filesystem policy entries.
• Add read-only access to injected NVIDIA libraries and read-write access to detected Jetson GPU devices.
• Preserve detected device-group IDs during compatibility and managed container recreation.
• Add a root-owned bootstrap helper that validates group IDs, updates the sandbox account, verifies memberships, and starts only the OpenShell supervisor.
• Add unit, integration, image-layout, workflow, and source-watch coverage for the new behavior.
• Document the proposed controls and the CUDA claim that they do not establish.

Draft Exit Conditions

Maintainers must keep this PR as a draft until all of these conditions are true:

1. Maintainers accept the Jetson GPU product scope and define ownership, lifecycle, compatibility, security, and validation requirements.
2. Issue [Jetson][Sandbox] Sandbox GPU passthrough proof fails for the non-root sandbox user on JetPack 6.2 IGX Orin — onboarding aborts #7610 no longer blocks the non-root OpenShell CUDA acceptance test.
3. A Jetson hardware run verifies successful cuInit(0) through the supported OpenShell sandbox path.
4. Documentation review, sensitive-path review, and applicable GitHub checks pass for the commit under review.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Required before this draft can leave draft status.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: blocked
• Evidence: docs/reference/troubleshooting.mdx, test/e2e/README.md, and test/e2e/docs/README.md. The changed documentation passes writing, terminology, claim-accuracy, and agent-variant review. The managed-bootstrap conflict resolution preserves the bounded supervisor reconnect timeout and the exact Jetson helper command boundary. Maintainer product-scope acceptance and IGX Orin OpenShell cuInit(0)=0 evidence remain required.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — 180 tests passed and 2 tests skipped across 14 affected test files.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — deferred while issue [Jetson][Sandbox] Sandbox GPU passthrough proof fails for the non-root sandbox user on JetPack 6.2 IGX Orin — onboarding aborts #7610 blocks the acceptance contract.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only) — build passed with 0 errors and 2 existing Fern warnings.
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only) — no new documentation pages.

---

Signed-off-by: Charan Jagwani cjagwani@nvidia.com
Signed-off-by: San Dang sdang@nvidia.com

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-8910.docs.buildwithfern.com/nemoclaw

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall line coverage in commit aa0a5a2 in the codex/jetson-gpu-fol... branch remains at 96%, unchanged from commit b7e4400 in the main branch.

---

_{Updated August 24, 2026 12:05 UTC}

[github-actions]

PR Review Advisor — Blocking findings reported

Advisor assessment: Blockers require maintainer review
Next action: Review the blockers below.
Findings: 2 blockers · 0 warnings · 0 suggestions
Synthesis status: Completed · high confidence · 2 blockers · 0 warnings · 0 suggestions

E2E guidance

Advisory only. A maintainer can dispatch the default E2E suite for the commit under review.

Recommended E2E: managed-image-protected-runtime

Manual-only E2E: cloud-onboard, managed-image-multiarch-startup, onboard-repair, onboard-resume, ubuntu-repo-docker-post-reboot-recovery, full-e2e
The manual PR workflow does not run these selectors for the commit under review. Run them from reviewed code on main.

Blockers

PRA-1 Blocker — Require a valid nvmap device before forwarding Jetson groups

• Location: src/lib/onboard/docker-gpu-jetson-groups.ts:109
• Category: security
• Problem: The Jetson group probe can forward a DRI render-device group when no valid /dev/nvmap device establishes that the host is a Jetson GPU host. Ordinary and managed Docker replacement consumers then add that host group to the sandbox.
• Impact: A sandbox user can receive a host supplementary group that is not required by verified Jetson GPU devices. That group can authorize access to host resources that use the same numeric GID.
• Fix: Return no device-group GIDs unless /dev/nvmap is a non-symlink character device. Preserve render-device GIDs only after that Jetson anchor succeeds.
• Verification: Read detectTegraDeviceGroupGids with a path list containing only /dev/dri/renderD128. The result must be empty. Then verify that valid /dev/nvmap and render-device inputs retain both required groups.
• Test coverage: Add a detectTegraDeviceGroupGids test with a qualifying render device and no valid /dev/nvmap that expects no GIDs. Add a clone-level assertion that this input produces no --group-add.
• Evidence: src/lib/onboard/docker-gpu-jetson-groups.ts:109-145 collects qualifying GIDs from every supplied device path without requiring /dev/nvmap. src/lib/onboard/docker-gpu-jetson-groups.test.ts:111-160 accepts a render-device GID without an nvmap input. src/lib/onboard/docker-gpu-patch-recreate.ts:267-294 and src/lib/onboard/managed-bootstrap/docker-runtime.ts:66-84 forward returned GIDs into replacement configuration.

PRA-2 Blocker — Describe sequential bootstrap group mutation accurately

• Location: scripts/jetson-device-group-bootstrap.sh:40
• Category: correctness
• Problem: The helper mutates group state while it resolves each GID, but troubleshooting guidance states that complete-list validation occurs before sandbox-account mutation. A malformed later group can fail after an earlier group is created or appended.
• Impact: A failed bootstrap can leave group state that the documented recovery contract says was not changed. Operators and recovery code can misinterpret the sandbox security state after failure.
• Fix: Either preflight every group record before the first group creation or membership update, or revise the troubleshooting guidance to state that group resolution and mutation occur sequentially after input-list validation.
• Verification: Inspect the helper with two GIDs where the first is absent and the later existing group record is malformed. Confirm whether the first group is created or appended before failure.
• Test coverage: Add a two-GID helper test with an earlier mutable GID and a malformed later record. Assert either no mutation after a complete preflight or the documented permitted partial mutation.
• Evidence: scripts/jetson-device-group-bootstrap.sh:27-38 validates GID syntax, range, and uniqueness before the mutation loop. scripts/jetson-device-group-bootstrap.sh:40-51 resolves, creates, and appends each group in sequence. docs/reference/troubleshooting.mdx:3475-3477 says the helper validates the complete detected group-ID list before it changes the sandbox account group database. test/jetson-device-group-bootstrap.test.ts:287-390 covers one malformed group record but not a malformed later record after an earlier mutation.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[prekshivyas]

Reviewed exact draft head fff5a76.

The proposed implementation is appropriately fail-closed for its draft purpose: it derives supplementary groups only from existing non-symlink Jetson character devices, validates numeric group IDs, keeps the helper root-owned, limits the root action to group setup plus the OpenShell supervisor handoff, and adds focused policy/recreation/bootstrap tests. The documentation correctly avoids claiming CUDA success.

This cannot be approved or leave draft state. The repository product-scope gate is still unmet, #7610 remains open, and the only reproduced IGX Orin result is that non-root cuInit(0) fails even with the existing groups. The PR therefore has no evidence that these broader preserved groups establish the supported sandbox-user CUDA contract. GitHub also reports the branch CONFLICTING/DIRTY against current main, and the documentation review is explicitly blocked. Resolve the product decision first, reconcile with current main, then obtain exact-head Jetson hardware evidence through the supported OpenShell path proving cuInit(0)=0, lifecycle/recreation behavior, and the least-privilege device/group set; rerun docs and sensitive-path review afterward.

Security review:

• Input validation: PASS — device paths, symlink state, character-device type, and numeric group IDs are bounded and validated.
• Authentication and authorization: WARNING — supplementary group membership expands device authority and still lacks accepted least-privilege hardware proof.
• Secrets and sensitive data: PASS — no credentials or sensitive payloads are introduced.
• Injection risks: PASS — group identifiers are numeric and the root helper does not evaluate caller-supplied shell fragments.
• Data exposure and privacy: PASS — no new data egress or logging surface.
• Cryptography: PASS — no cryptographic behavior changes.
• Dependencies and supply chain: PASS — no dependency or artifact-source changes.
• System security: FAIL for approval — the core non-root CUDA security/compatibility postcondition remains unproven and product ownership is undefined.
• Testing and verification: FAIL for approval — #7610 is unresolved, exact-head Jetson hardware evidence is absent, docs review is blocked, and the branch conflicts with current main.

Files reviewed: all 40 changed Jetson device/group policy, clone/recreate/create, startup/bootstrap, managed-runtime, Dockerfile/build-context, workflow, documentation, and test files; linked issue #7610 and automated review evidence.

[prekshivyas]

Reviewed commit fff5a76e22c1c3c57731a82c09d9a2bd7155a0eb as a security-critical Jetson policy and root-helper change.

I did not find an additional correctness or security defect in the current diff. The helper validates the complete bounded GID set before mutation, requires root, accepts only the fixed OpenShell supervisor handoff, verifies the resulting membership, and fails closed. The policy path is anchored on a real non-symlink /dev/nvmap character device and grants only the enumerated device paths.

Security review: secrets PASS; input validation PASS; authentication/authorization PASS; dependencies PASS; error handling PASS; cryptography N/A; configuration/environment PASS; security tests PASS for the implemented boundary; system security BLOCKED for merge.

This is not an approval. The PR is correctly still draft and currently conflicts with the base branch. Its stated merge gates remain material: maintainer product-scope acceptance, resolution of #7610, real supported-path Jetson evidence showing cuInit(0), and completion of the sensitive-path/broad verification gates.

[prekshivyas]

Refreshed the draft through current main (b4b66806c, including #9347 and the latest test-loop conversion) at head 15330cddd.

The only overlap was initial-policy.test.ts; the resolution preserves all Jetson policy regressions and the upstream table-driven Station boundary test. Exact-head validation: 3 focused files / 76 tests passed, test-loop scan is back at the current-main 743-loop baseline, repository/pre-commit hooks passed, and pre-push CLI/plugin type checks passed.

The branch is conflict-free again. Draft exit conditions remain unchanged: maintainer product-scope acceptance, #7610 resolution, and supported-path physical IGX Orin cuInit(0)=0 evidence.

[prekshivyas]

Reviewed exact draft head 15330cddd313eda2d9d18aa9feba9b245d96c085.

I found no additional code-level blocker in this revision. The bootstrap boundary is fixed to the OpenShell supervisor, device/group inputs are bounded and validated, Tegra discovery rejects symlinks and non-character devices, failures are fail-closed, and the exact-head required checks are green.

This is not an approval. The PR's own draft-exit conditions remain unmet: #7610 is still open, the expected non-root cuInit(0) = 0 result has not been demonstrated on supported physical Jetson/IGX hardware, and maintainer acceptance of the product scope is not recorded. Please keep this draft until those conditions are satisfied.

Security review:

1. Secrets/credentials — PASS: no credential material or secret-handling path is introduced.
2. Input validation — PASS: GIDs, device paths, argument shape, helper path, and supervisor target are constrained.
3. Authentication/authorization — BLOCKED FOR APPROVAL: supplemental device-group membership expands sandbox device authority and still needs the required least-privilege hardware validation.
4. Dependencies — PASS: no new third-party dependency or supply-chain source is added.
5. Error handling/logging — PASS: invalid or incomplete device/group state fails closed with bounded diagnostics.
6. Cryptography/data protection — N/A: no cryptographic or protected-data path changes.
7. Configuration/security headers — PASS: the managed bootstrap and fixed supervisor boundary are explicit; no web security-header surface changes.
8. Security testing — PASS for the simulated implementation tests; BLOCKED FOR ACCEPTANCE on the required physical Jetson/IGX test.
9. System security — BLOCKED FOR APPROVAL pending the real-hardware cuInit(0) proof and product-scope acceptance.

I also completed the cross-issue sweep for findings from this review and found no separate, actionable issue candidate beyond the tracked #7610 prerequisite.

[jyaunches]

LOC Reduction / Codebase Simplicity Review

Why this blocks

• preserveJetsonDeviceGroupMembership appears 28 times across 14 production files.
• It is derived only from agent identity, then threaded through generic create, GPU, patch, and managed-bootstrap interfaces.
• Direct and managed paths independently wrap argv, and managed bootstrap imports its bootstrap string from the clone implementation.

Refactor direction

• Give docker-gpu-jetson-groups.ts one normalized resolver and renderer that returns either null or a bootstrap plan with detected GIDs and wrapped argv.
• Derive eligibility at the leaf from existing agent identity.
• Have both recreation paths consume the same wrapper helper.

Expected result

• Remove the optional flag from generic interfaces and calls.
• Centralize bootstrap and argv ownership so direct and managed implementations cannot drift.

[cv]

Draft status refresh at commit 15330cddd313eda2d9d18aa9feba9b245d96c085:

• Both automated review lanes reported no findings on this commit, and no unresolved review threads remain.
• The branch is still based on an older main; current required checks are not present, so the displayed green state is not current merge evidence.
• This change spans Docker image contents, generated startup commands, managed bootstrap, host device-group detection, policy, documentation, and platform CI. It also introduces a 415-line shell-boundary integration test.

Before moving out of draft, update from current main and rerun the focused Jetson group detector, clone/recreate, startup-command, managed-bootstrap, initial-policy, bootstrap helper, build-context, workflow, watch-trigger, typecheck, repository, and documentation checks. The PR also needs reviewable Jetson hardware evidence for the actual supplemental-group handoff across the OpenShell supervisor boundary; deterministic mocks establish command construction but not host kernel/device behavior. Keep the scope tied to the accepted Jetson issue and record the tested Jetson model, software image, commit, outcome, and supporting run evidence.

[cv]

Blocking issues remain at exact head 94e6e89c74f588e9b16a722d3d12a8722ffb93b4:

1. Product and hardware acceptance needs a maintainer decision, not a mechanical repair. The PR description correctly says that #7610 remains open, maintainers have not accepted the Jetson GPU product scope, and the supported OpenShell path has not demonstrated non-root cuInit(0)=0. The issue evidence still reports direct Docker success but OpenShell execution returning 801. Because this change expands sandbox device policy and supplementary-group authority, deterministic command tests cannot establish the least-privilege or compatibility contract. Please record the product ownership/lifecycle/security decision and provide exact-commit physical IGX Orin evidence through the supported OpenShell path before approval.

2. The requested design simplification is still outstanding. preserveJetsonDeviceGroupMembership remains threaded through generic startup, sandbox-create, GPU-flow, patch, recreation, and managed-bootstrap contracts (49 source/test occurrences). Eligibility is derived from agent identity, while direct and managed paths separately wrap the startup command. Resolve the existing review by centralizing Jetson group detection and argv rendering at the leaf, and remove the optional flag from generic interfaces so the paths cannot drift.

3. Current CI is red. cli-test-shards (11) fails the growth guard because src/lib/onboard/sandbox-gpu-create-flow.test.ts is 1,514 lines against its 1,500-line budget. static-checks, cli-tests, and the aggregate checks job consequently fail. Split or relocate the added cases rather than weakening the budget.

The earlier PR Review Advisor security and documentation findings are addressed in this head: /dev/nvmap now anchors group discovery, and the helper preflights the complete group database before mutation. The latest advisor run did not produce a new assessment because its specialist sessions were unavailable, so it does not replace the older applicable findings or this review.

Security rubric: secrets PASS; input validation PASS; injection PASS; dependencies PASS; data exposure PASS; cryptography N/A; error handling PASS; authentication/authorization and system security BLOCKED pending least-privilege hardware evidence and product-scope acceptance; verification FAIL while required CI is red and the supported-path acceptance result is missing.