Skip to content

test(e2e): verify OpenClaw recreation and restore - #14387

Open
jyaunches wants to merge 1 commit into
run-brev-v1-e2e/fabric-fork-main-20261110from
codex/openclaw-recreate-lease-regression
Open

jyaunches wants to merge 1 commit into
run-brev-v1-e2e/fabric-fork-main-20261110from
codex/openclaw-recreate-lease-regression

Conversation

@jyaunches

Copy link
Copy Markdown
Contributor

Extend the existing hosted OpenClaw E2E case to require a usable native runtime after recreation, a tool configuration change, and restore on retained volumes.

Problem

The hosted case stopped after its first reply and destroy, so it could pass while the replacement Gateway could not start. Retaining OpenClaw's native state can also retain an unexpired ownership lease that blocks startup in a new container.

Change

  • Mount caller-owned workspace and artifacts volumes, provisioned through the existing volume helper.
  • Drive the intermediate destroy/recreate and configuration changes through an SDK probe. Require fresh replies, native tool-call records and filesystem effects, retained conversation records, and ownership handoff from each replacement.
  • Keep the existing hosted case, unchanged-plan checks, and final destroy. Generated-skill retention stays separate from basic readiness.

Stacked on #14362 for the retained-state implementation. This PR adds regression coverage; it does not change runtime shutdown or ownership protection.

Beyond CI

The lifecycle probe reproduced the failure on Brev Linux AMD64 with NemoClaw 4645db0ca99195047d056011b58d08eb5e1e033f, Fabric 1b5c8ce5057f846dd86db6788e0be38c6b3675af, OpenClaw 2026.9.34, and OpenRouter openai/gpt-4.1-mini. Initial inference and native tools passed; recreate failed on an unexpired Gateway ownership lease. After natural lease expiry, a configuration change and fresh inference/file operations passed; restore reproduced the lease failure. No lease or persistent state was repaired to obtain those results.

The final HCL wiring and NVIDIA-hosted model variant were not rerun live after folding the probe into this existing case. The Brev deployment was destroyed and its instance stopped and deleted.

@jyaunches

Copy link
Copy Markdown
Contributor Author
Authoring session — opening
Field Status Value Evidence or limit
Harness recorded Codex Desktop; CLI runtime 0.166.0-alpha.5.1 Persisted session metadata; matched the recorded worktree creation and regression-test edits.
Skills partial Complete reads: brev-cli; test-driven-development; write-tests; rust-lsp; small-fix Full contents matched the authoring record. The verify-fabric-boundary and validate-change read outputs were truncated or partial, so they are excluded from the complete-read inventory.
Instruction files partial 8 complete guidance files: NemoClaw AGENTS.md and WRITING.md at the original and tested checkouts; parent workspace AGENTS.md; Fabric AGENTS.md; scope.md; brev-openrouter-target.md Full contents matched the authoring record. OpenClaw AGENTS.md was read only in part and is excluded. Skill catalogs and source-code reads are not counted as complete guidance files.
Tool calls partial Outer: exec 159; wait 7; request_user_input_async 1; sleep 10. Observable nested lower bounds: exec_command 223; write_stdin 76; apply_patch 24; Rust LSP workspace_symbols 1 Attempted calls include failures. Wrapper and nested counts are separate. One exec was terminated during LSP startup; its later exec_command and write_stdin were not counted. Tool-internal calls are not exposed; shell subprocesses are outside the counted API layer.
Session window recorded 2026-11-10T21:55:23.876Z to 2026-11-11T02:57:13.605Z; 5 h 1 min 49.729 s, including idle time Whole authoring session through the completed existing-case change; includes investigation and Brev reproduction. Publication activity after this cutoff is excluded. No forked authoring history.
Compactions recorded 0 No compaction events in the persisted record from session creation through the cutoff.
Models recorded gpt-6.1-sol Initial and subsequent authoring turn contexts; no recorded model changes through the cutoff.
Reasoning levels recorded medium Initial and subsequent authoring turn contexts; no recorded effort changes through the cutoff.

The Brev probe reached the native runtime through the production SDK and CLI paths. Recreate and restore both failed on the retained Gateway lease. The new case wiring passed local compilation, Clippy, OpenTofu validation and formatting, case selection, and 16 offline tests. These checks do not establish a passing hosted lifecycle. Reviewed reproduction evidence is retained locally; no raw session records, credentials, or private evidence paths are published here.

@jyaunches

Copy link
Copy Markdown
Contributor Author

Awaiting Fabric fix: NVIDIA/NeMo-Fabric#626, tracked in epic NVIDIA/NeMo-Fabric#577 and required for the retained-state acceptance in NVIDIA/NeMo-Fabric#578.

The new regression coverage is red at the first destroy/recreate transition: initial deployment and native inference/tool checks pass, then reapply fails with Fabric start_failed. The red Test / linux_amd64 check aggregates that same failure. The configuration-change and restore stages are not reached in this CI run.

The Brev reproduction found the previous Gateway's unexpired ownership lease in retained state. Fabric #626 now documents the shutdown mechanism: OpenClaw's spawn-broker process-group cleanup encounters EPERM, and Fabric's default five-second stop allowance force-kills the Gateway before lease release. Its longer-timeout control releases ownership and permits replacement, but still reports a broker-cleanup error, so that control does not establish clean shutdown.

Before this PR is ready: land the Fabric correction, consume the corrected Fabric revision in NemoClaw, and rerun the existing hosted OpenClaw case through recreation, configuration change, and restore. Each replacement must start promptly on retained state and pass the fresh inference, native tool, retained-data/history, and ownership checks. This PR adds coverage only; it does not implement the shutdown correction.

Failing E2E job: https://github.com/NVIDIA/NemoClaw/actions/runs/42952200933/job/130307269526

@wscurran wscurran added area: e2e End-to-end tests, nightly failures, or validation infrastructure area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery area: security Security controls, permissions, secrets, or hardening chore Build, CI, dependency, or tooling maintenance integration: openclaw OpenClaw integration behavior labels Nov 12, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e End-to-end tests, nightly failures, or validation infrastructure area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery area: security Security controls, permissions, secrets, or hardening chore Build, CI, dependency, or tooling maintenance integration: openclaw OpenClaw integration behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants