Skip to content

fix(runtime): Put hull on Linux sandboxes on a network gateway - #667

Merged
cmainas merged 2 commits into
mainfrom
fix/hull-linux-agetway
Nov 18, 2026
Merged

cmainas merged 2 commits into
mainfrom
fix/hull-linux-agetway

Conversation

@cmainas

@cmainas cmainas commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

hull on Linux put every networked sandbox on its CNI bridge, one L2 segment for all of them, so a sandbox reached another's listener and could answer ARP for it whatever posture either asked for. isolated was refused there, and with no posture named brig fell back to shared, so every sandbox was on the bridge by default.

hull now runs network-gateway on Linux and urunc joins hvi to it, so a sandbox on hull on Linux joins a gateway as an hvi sandbox does on macOS: one of its own when isolated, the shared one otherwise. isolated now runs there and is the default, as on nerdctl. The bridge is never used. A networked run without a gateway, or on a monitor other than hvi, is refused. A policy, a divert, brig.internal and a published port are still refused, now with the real reason.

brig rm --all stops the shared gateway once nothing is on it. On Linux the process table cannot show the guests, because urunc binds the socket into hvi's root, so brig asks the gateway's GET /members and checks that no instance hull inspect reports is on it. The second check covers a guest whose hvi has not connected yet, which the gateway does not list. A gateway that cannot answer (a hull older than /members) or that brig cannot check is kept.

brig policy check now refuses an egress-only policy on a run path that cannot enforce it, with exit 7, as brig run does. It printed its general note and exited 0, and the run then failed.

The sandbox-to-sandbox behaviour was measured on a real hull on Linux on 2026-11-16 (hull 28f4238, urunc 5c6bc70, the hvi of brig-standalone-linux v0.1.0-rc21): on shared one microVM reached the other's listener, and with each on its own gateway the same request timed out while outbound connections worked. That was before the shared-gateway pruning and the policy check were added, and neither has been run against a real hull. go test -race ./... and make all pass except the internal/secret keyring integration tests, which need an unlocked desktop keyring and fail the same way on main on this host.

This PR needs the removes-tests label: TestProfileIsolationOnLinuxHullNamesNerdctlNotHvi is renamed to TestProfileIsolationRunsOnLinuxHull, because it now asserts the opposite.

Related issues

Refs brig-sh/hull#105

Changes

  • hull on Linux joins a networked sandbox to a hull network gateway: its own when isolated, the shared one when shared. An unset posture is isolated.
  • A networked run on hull on Linux without a gateway, or on a monitor other than hvi, is refused. The refusals of a policy, a divert, brig.internal and a published port name the real reason.
  • brig info and the stale-network check read a hull on Linux sandbox's network from hull inspect (network, gatewaySocket).
  • brig rm --all stops the shared gateway on hull on Linux only when GET /members lists no guest and no instance hull lists, other than a stopped one, is on it. Any doubt keeps it: no /members, a failed answer, no API, or a hull ps or hull inspect that fails.
  • brig policy check asks the egress answer the run reads, on the run path the run takes, and refuses with exit 7 where the run would. An offline run passes.
  • Docs: docs/security.md says isolated is provided and the default on hull on Linux, with the measured sandbox-to-sandbox row. docs/runtimes.md covers the gateway on Linux and when the shared one is stopped. docs/policies.md says check refuses what the run refuses.

Checklist

  • make all passes (vet, test, build)
  • script/smoke.sh passes
  • I have added or updated tests covering the change
  • I have run go test ./... -race, if the change touches concurrency, subprocesses or the daemon
  • I have exercised the change against a real runtime (brig run <agent>), if it touches the run, exec or credential path
  • I have updated the affected docs (see docs/README.md for the map)

Credentials and the sandbox boundary

Neither promise moves: each VM gets only the directories and credentials named for its session, before and after. What changes is the network posture on hull on Linux, which docs/security.md now states: isolated is provided and the default, where before it was refused and sandboxes shared one bridge.

Negative tests:

  • Two isolated sandboxes on hull on Linux are each on a gateway of their own, on networks of their own, and neither is on the shared gateway.
  • A networked run on hull on Linux on a monitor other than hvi, and a policy on every monitor, are refused.
  • brig rm --all keeps the shared gateway while it lists a guest, while it cannot answer /members (404, 503, no API), while a boot holds a lease, and while hull lists a running instance on it or cannot be asked.
  • brig policy check refuses an egress-only policy on hull on vz or qemu and on docker, with exit 7, and the egress answer it reads refuses hull on Linux on every monitor.

@cmainas

cmainas commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

@claude-fleet review

@claude-fleet

claude-fleet Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Review task queued.

Task: 82fd3011-c49a-4313-80d9-8dbb58e557b9
Commit: e212bda
Model: claude-opus-5 (effort medium)
Executor: pimox5
Status: done. Review posted.

@cmainas cmainas added the removes-tests Removes or renames a test on purpose; skips the tests-kept check label Nov 17, 2026
@ananos
ananos marked this pull request as ready for review November 17, 2026 14:25

@claude-fleet claude-fleet Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: two should_fix findings, both stale claims the diff leaves behind. The code itself holds up — I found no correctness defect in the gateway join, the prune guard or the new policy check refusal, and each new test fails when its fix is reverted.

Findings:

  • should_fix, docs/policies.md:994 — "check does not look at the runtime" is now false
  • should_fix, cmd/brig/doctor_host.go:414 — the cni row still says the plugins are for "a run on the shared network"
  • nit, docs/cli.md:1316 — "Exit 7 cases" names only brig run and brig sh for an egress policy

Verified:

  • joinGateway is reached by both hulls and picks the sandbox's own gateway when isolatedNet, the shared one otherwise — internal/runtime/hull.go:621-646, 674-704; runLinux joins before linuxRunArgs — hull.go:755-767
  • linuxRunArgs refuses a networked run with no gateway, so hull's bare --net shared (the CNI bridge) is never reached — hull.go:1161-1165, 1201-1204
  • linuxSupports still refuses a policy, a divert, brig.internal and a published port, and now refuses a networked generic boot off hvi — hull.go:857-909
  • PruneSharedNetwork keeps the shared gateway unless /members answers a list and it is empty and no non-stopped instance hull inspect places on that socket; every doubt (no /members, 404, 503, no API, ps or inspect failing, an id mismatch, an unknown label) returns true — hull.go:1755-1760, 1781-1821; gatewayMembers reads null, {} and a non-200 as unknown — gatewayapi.go:128-140
  • ensureSharedGateway(bin, false) on Linux never stops an API-less old gateway — gateway.go:279-296
  • wrap.EgressServed mirrors RouteModeServed exactly (same env, same isolated-unless-offline posture, same RoutePath), so check and run read one answer — internal/wrap/routemode.go:141-172; requireEgress ignores the posture except for none — internal/runtime/capability.go:176-186
  • linuxHullSandboxNetwork reads a gateway label through the gateway's own socket, and refuses a gateway with no socket — networkinspect.go:131-155
  • NetworkStale now runs on hull on Linux whatever BRIG_HYPERVISOR names — hull.go:1638-1646
  • The PR's removes-tests claim is exact: script/check-tests-kept.sh 87bb90c HEAD reports only TestProfileIsolationOnLinuxHullNamesNerdctlNotHvi

Checks run:

  • go test ./internal/runtime/ ./internal/wrap/ ./cmd/brig/ → ok (3 packages)
  • Reverted the wrap.EgressServed block at cmd/brig/policy.go:1294-1301 → TestCheckRefusesEgressTheRunCannotEnforce fails on hull on vz, hull on qemu and docker, each "exit 0, want exit 7"; restored
  • Reverted the --gateway-sock/--gateway-cidr append at hull.go:1201-1204 → TestTwoIsolatedSandboxesOnLinuxHullShareNoGateway fails ("the run line joins no gateway") and TestLinuxHullRunArgsNetwork fails for "", shared and isolated; restored
  • Reverted the h.linux block at hull.go:1755-1760 → TestLinuxHullKeepsTheSharedGatewayAGuestMayBeOn fails on /members fails and no API, and TestLinuxHullKeepsTheSharedGatewayASandboxIsOn fails on all five keep cases; restored
  • script/smoke.sh → PASS (608 ok), the hull-on-Linux section included: both isolated runs join a gw/sandbox-*.sock, the plain run carries [--gateway-sock][.../gw/gateway-
  • gofmt -l . → nothing; go vet ./... → clean
  • script/check-retired-spellings.sh → rc 0; script/check-claims.sh → rc 0

Smaller things:

  • cmd/brig/policy.go:1278 and :1295 both call detectRuntimeFor(runtime.Preference{Bin: p.RuntimeBin}). The route block is conditional on there being modes, so the second call is not dead, but one detection hoisted above both would spare a second runtime probe on every policy check.
  • gatewayMember.Transport and JoinedAt (gatewayapi.go:120-123) are decoded and never read — only len() is. Fine as documentation of the payload; worth a word saying so, since the sweep for write-only fields will keep finding them.
  • The console example at docs/policies.md:975-980 shows check printing the note and exiting 0 for locked-down, an egress-only policy. On vz, qemu or docker that example now exits 7. The first example at line 44 got the new "on Linux with nerdctl, or on macOS with BRIG_HYPERVISOR=hvi" qualifier; this one did not.
  • script/smoke.sh no longer exercises a refusal on hull on Linux for a networked run on a non-hvi monitor. TestLinuxHullRefusesWhatItCannotRunYet covers it, so this is only a note that the binary-level path is unexercised.
  • onSharedGateway skips only State == "stopped". Any other word hull might print for a dead instance (exited, created) keeps the gateway. That is the safe direction, and the comment says doubt keeps it, so no change is needed — but it does mean a gateway can be kept indefinitely by a stale listing.

Not checked:

  • hull's side of the new contract: that hull run --net shared --gateway-sock X is recorded by hull inspect as network: "gateway" with gatewaySocket as the host path, and that GET /members exists and lists a guest from the moment its VMM connects. onSharedGateway (hull.go:1812-1814) treats network: "shared" as "on no gateway"; if hull instead labels a gateway run by the --net word it was given, then during the window where hvi has not yet connected — the very window the instance check exists to cover — brig rm --all would stop the shared gateway under a booting guest. The PR states the pruning and the policy check were never run against a real hull, so this is the one place I would want that run before merge.
  • The 2026-11-16 sandbox-to-sandbox measurement in docs/security.md:1470, and the hull/urunc/hvi revisions it names.
  • macOS-only paths: supports, the vz/qemu rows, publication reconciliation, and internal/secret's keychain tests.

Not checked:

  • hull's own behaviour: that hull inspect on Linux reports network: "gateway" with a host-side gatewaySocket for a run given --net shared --gateway-sock, and that GET /members exists and lists a guest as soon as its VMM connects. If hull labels a gateway run shared, internal/runtime/hull.go:1812-1814 reads it as on no gateway and brig rm --all could stop the shared gateway under a guest whose hvi has not connected yet.
  • The 2026-11-16 sandbox-to-sandbox measurement in docs/security.md:1470 and the hull 28f4238 / urunc 5c6bc70 / hvi rc21 revisions it names.
  • macOS-only paths: supports, the vz and qemu capability rows, publication reconciliation, and the internal/secret keychain tests.

Findings outside the diff:

  • docs/policies.md:994 (should_fix): This paragraph is now false for the behaviour this PR adds: check does look at the runtime for a policy without routes, and refuses it with exit 7.
    Failure: a user on vz reads "check does not look at the runtime, so it cannot tell you whether this host will boot the run or refuse it", then runs brig policy check claude-code and gets exit 7 naming vz. The numbered list above at line 983 ("check does three checks") has the same gap -- the egress check is a fourth.
    Evidence: cmd/brig/policy.go:1294-1301 calls wrap.EgressServed whenever a runtime is detected, and checkPolicy returns a *runtime.CapabilityError that cmd/brig/exit.go:38 maps to 7; the opposite statement is already in this same file at line 44-46. TestCheckRefusesEgressTheRunCannotEnforce pins it for vz, qemu and docker.
  • cmd/brig/doctor_host.go:414 (should_fix): The cni row for hull on Linux still says the plugins are needed for "a run on the shared network", which this PR says is no longer a thing Brig does.
    Failure: on a Linux host with hull and no CNI plugins, brig doctor prints cni fail: no bridge, firewall, ... plugin in /opt/cni/bin, and a run on the shared network needs them and tells the user to run install.sh -- for a bridge docs/security.md:1445 now states Brig never puts a sandbox on. Either the row is stale and should name whatever hull still needs the plugins for (the container side of a gateway run, if anything), or the "never" in security.md is too strong. The row does not change doctor's exit status, so the cost is a wrong instruction rather than a wrong verdict.
    Evidence: cmd/brig/doctor_host.go:412-414 and :84 (cni = hullCNICheck); docs/security.md:1445 "Brig never puts a sandbox on hull's own CNI bridge"; internal/runtime/hull.go:1161-1165 refuses a networked run that has no gateway, so no brig run reaches --net shared alone.
  • docs/cli.md:1316 (nit): "Exit 7 cases" still names only brig run and brig sh as the commands that refuse an egress policy with 7.
    Failure: someone scripting against the exit-code table -- which docs/stability.md calls stable to script against -- sees brig policy check change from 0 to 7 on vz, qemu and docker with nothing in this list to explain it. Adding brig policy check to the sentence at line 1316 covers it.
    Evidence: cmd/brig/policy.go:1294-1301; docs/cli.md:1303 points at this section for every exit 7.

Task 82fd3011-c49a-4313-80d9-8dbb58e557b9 on pimox5, model claude-opus-5, effort medium.

@cmainas
cmainas force-pushed the fix/hull-linux-agetway branch from e212bda to 02bbb6e Compare November 17, 2026 14:41
@cmainas

cmainas commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

@claude-fleet review

@claude-fleet

claude-fleet Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Review task queued.

Task: bd9e76bb-b2e9-4b81-87bf-d0288970cb6c
Commit: 02bbb6e
Model: claude-opus-5 (effort medium)
Executor: ravel-clickbench-ubuntu-2
Status: done. Review posted.

@claude-fleet claude-fleet Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previous round at e212bda: 3 findings.

Previous finding Classification Cited line
docs/policies.md:994 (should_fix): This paragraph is now false for the behaviour this PR adds: check does look at the runtime for a policy without routes, and refuses it with exit 7. Failure: a user on vz reads "check does not look at the runtime, so it cannot tell you whether this host will boot the run or refuse it", then runs brig policy check claude-code and gets exit 7 naming vz. The numbered list above at line 983 ("check does three checks") has the same gap -- the egress check is a fourth. Evidence: cmd/brig/policy.go:1294-1301 calls wrap.EgressServed whenever a runtime is detected, and checkPolicy returns a *runtime.CapabilityError that cmd/brig/exit.go:38 maps to 7; the opposite statement is already in this same file at line 44-46. TestCheckRefusesEgressTheRunCannotEnforce pins it for vz, qemu and docker. fixed docs/policies.md:1000
cmd/brig/doctor_host.go:414 (should_fix): The cni row for hull on Linux still says the plugins are needed for "a run on the shared network", which this PR says is no longer a thing Brig does. Failure: on a Linux host with hull and no CNI plugins, brig doctor prints cni fail: no bridge, firewall, ... plugin in /opt/cni/bin, and a run on the shared network needs them and tells the user to run install.sh -- for a bridge docs/security.md:1445 now states Brig never puts a sandbox on. Either the row is stale and should name whatever hull still needs the plugins for (the container side of a gateway run, if anything), or the "never" in security.md is too strong. The row does not change doctor's exit status, so the cost is a wrong instruction rather than a wrong verdict. Evidence: cmd/brig/doctor_host.go:412-414 and :84 (cni = hullCNICheck); docs/security.md:1445 "Brig never puts a sandbox on hull's own CNI bridge"; internal/runtime/hull.go:1161-1165 refuses a networked run that has no gateway, so no brig run reaches --net shared alone. fixed cmd/brig/doctor_host.go:85
docs/cli.md:1316 (nit): "Exit 7 cases" still names only brig run and brig sh as the commands that refuse an egress policy with 7. Failure: someone scripting against the exit-code table -- which docs/stability.md calls stable to script against -- sees brig policy check change from 0 to 7 on vz, qemu and docker with nothing in this list to explain it. Adding brig policy check to the sentence at line 1316 covers it. Evidence: cmd/brig/policy.go:1294-1301; docs/cli.md:1303 points at this section for every exit 7. fixed docs/cli.md:1325

Previous round's three findings are all fixed. Re-reviewed the whole diff at the current head.

Approve: no blocking or should_fix finding.

Findings:

  • nit, internal/runtime/hull.go:844 — the "cannot enforce" reason for hull on Linux describes plumbing that already exists

Verified:

  • Previous #1: docs/policies.md:984-1002 now lists the egress check as the fourth check and replaces the "check does not look at the runtime" paragraph with what check does and does not ask.
  • Previous #2: the cni row is gone for hull on Linux (cmd/brig/doctor_host.go:81-86), hullCNIDirs/hullCNICheck removed, and TestDoctorAsksHullOnLinuxForNoCNI pins that nerdctl and a host with no runtime keep the row.
  • Previous #3: docs/cli.md:1325-1328 adds brig policy check to the exit-7 section, and says it does not run the boot probes.
  • One detection for both runtime checks in checkPolicy — cmd/brig/policy.go:1277-1300; rt is used only under detectErr == nil.
  • gatewayMembers now counts without a typed struct, and still reads only a non-null JSON list as known — internal/runtime/gatewayapi.go:119-139, pinned by TestGatewayMembersReadsOnlyAListAsKnown.
  • onSharedGateway compares absolute paths and fails closed on filepath.Abs error, hull ps/inspect failure, an id mismatch, an unknown network label, and an empty gatewaySocket — internal/runtime/hull.go:1789-1825, pinned by TestLinuxHullKeepsTheSharedGatewayASandboxIsOn and ...ItNamesRelatively.
  • A stopped sandbox is never restarted in place (Runtime.Remove then a fresh boot — internal/wrap/run.go:323-326), so pruning the shared gateway cannot strand a stopped guest's network.
  • isolated and a published port, a policy, a divert and brig.internal stay refused on hull on Linux through linuxSupports (internal/runtime/hull.go:852-910), and a networked generic boot off hvi is refused at :899-907.
  • The macOS path is unchanged: ensureGateway keeps membersSeen true (internal/runtime/gateway.go:273), and PruneSharedNetwork's new checks are under h.linux.

Checks run:

  • go test ./internal/runtime/ ./internal/wrap/ ./cmd/brig/ → ok
  • go build ./... && go vet ./... → clean; gofmt -l . → nothing
  • script/check-retired-spellings.sh, script/check-claims.sh → pass
  • Reverted the new egress block in cmd/brig/policy.go:1297 (if false && detectErr == nil) → TestCheckRefusesEgressTheRunCannotEnforce fails on hull/vz, hull/qemu and docker with "exit 0, want exit 7"; file restored, git status --porcelain clean.

Smaller things:

  • TestLinuxHullNeverStopsTheSharedGateway (internal/runtime/hulllinuxgateway_test.go:112) now sits beside TestLinuxHullStopsTheSharedGatewayNoGuestIsOn; the name reads as the old unconditional behaviour. It tests the no-/members case, so something like ...NeverStopsASharedGatewayItCannotAsk would say what it pins.
  • The "urunc binds the socket into hvi's root, so the process table cannot show the guests" explanation now exists in three code copies (internal/runtime/gateway.go:277, internal/runtime/hull.go:1730, internal/runtime/hulllinuxgateway_test.go:109) plus docs/runtimes.md:393-396. One of them (PruneSharedNetwork) could hold it with pointers from the others.

Not checked:

  • Whether hull on Linux records gatewaySocket in hull inspect, lists an instance from the moment hull run --detach returns, serves GET /members, needs no CNI plugin for --net shared --gateway-sock, and refuses or reroutes a networked non-generic boot (GenericBoot: false, no --hypervisor passed) onto hvi. All four are claims about brig-sh/hull, which is not in this clone; the doctor cni row removal and the monitor refusal being generic-boot-only both rest on them.
  • brig run against a real hull on Linux for the shared-gateway pruning and the policy check — the PR says neither has been run there, and the stub smoke path cannot answer /members.

Not checked:

  • brig-sh/hull behaviour the change rests on: whether hull inspect on Linux prints gatewaySocket, whether hull ps lists an instance as soon as hull run --detach returns, whether the gateway serves GET /members, and whether --net shared --gateway-sock needs no CNI plugin (the basis for removing doctor's cni row)
  • A networked non-generic boot (GenericBoot false): linuxSupports' hvi-only refusal is inside the GenericBoot branch and no --hypervisor is passed, so the claim that hull boots such an image on hvi or refuses it was not verified
  • Any run against a real hull on Linux; the PR states the shared-gateway pruning and the policy check have not been run there

Task bd9e76bb-b2e9-4b81-87bf-d0288970cb6c on ravel-clickbench-ubuntu-2, model claude-opus-5, effort medium.

Comment thread internal/runtime/hull.go Outdated
const (
linuxHullNoNetworkWhy = "hull on Linux gives a sandbox no network of its own yet"
linuxHullRemedy = "Use nerdctl (BRIG_RUNTIME=nerdctl), or detach the policy"
linuxHullNoPolicyWhy = "brig does not yet hand a policy's rules to the network gateway " +

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: linuxHullNoPolicyWhy states a gap that the code does not have: brig does hand a policy's rules to the gateway on hull on Linux.
Failure: a user on hull on Linux attaching an egress policy reads "brig does not yet hand a policy's rules to the network gateway of a sandbox on hull on Linux" from brig run, brig policy check, docs/policies.md:1266, docs/runtimes.md:90 and docs/security.md, and a maintainer goes looking for plumbing that is already there. The refusal stands only because the capability table says so, which makes the reason circular.
Evidence: internal/runtime/hull.go:1159 (joinGateway) passes spec.Egress to ensureIsolatedGateway on both hulls, and internal/runtime/gateway.go:558 hands that policy to startGateway; the only thing stopping a Linux policy run is requireEgress(spec, linuxHullPath) at internal/runtime/hull.go:857 reading the row at internal/runtime/capability.go:83. The honest reason looks like "brig has not verified that the gateway enforces the rules for a urunc guest" -- same for linuxHullNoRouteWhy at :846, whose divert flags travel the same path.

@asapranidis asapranidis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on Linux with a real hull, on a rootless install of the rc21 runtime bundle.

Setup

  • brig: this PR at e212bda. The findings below are re-checked against 02bbb6e: List() and the List() call in onSharedGateway are unchanged, and the rebase does not touch the run path tested here.
  • hull: 4bf76a6
  • runtime bundle: brig-standalone-linux v0.1.0-rc21 (rootless), hvi 0.1.0-rc.2
  • urunc: 3c00b69 (rc21), then 5c6bc70 built from source
  • host: Linux 7.2.6, x86_64

Results

Case urunc Result
--network offline 3c00b69 Boots. Guest kernel 6.18.59, host kernel 7.2.6. The guest has lo and sit0, both down.
default posture (isolated) 3c00b69 brig starts the sandbox's own gateway and passes --gateway-sock. hull refuses the start: urunc started hvi without --net-gateway, so the guest has no network: the shim's urunc does not read the gateway's annotations, which takes urunc-dev/urunc@5c6bc70. The instance is removed. Exit 1.
isolated and --network shared 5c6bc70 Task creation fails: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: mkdir /sys/fs/cgroup/brig: permission denied.
brig rm --all, no instances n/a The shared gateway is not stopped. See the inline comment on onSharedGateway.

Sandbox-to-sandbox reachability and egress were not tested. On a rootless install of the rc21 bundle, which is what BRIG_INSTALL_ROOTLESS=1 sh install.sh produces, no networked guest starts: with rc21's urunc hull refuses the start (row 2), and with urunc 5c6bc70 swapped in, task creation fails (row 3).

Findings 1 and 2 do not depend on the host. Finding 1 is in how List() parses hull's output, and finding 2 is in the rc21 release artifacts.

To reproduce row 3:

  1. BRIG_INSTALL_ROOTLESS=1 sh install.sh
  2. Build urunc and containerd-shim-urunc-v2 at 5c6bc70 (CGO_ENABLED=1, -tags netgo,osusergo, static) and replace the two binaries in ~/.local/share/brig/data/bin.
  3. BRIG_RUNTIME=hull BRIG_RUNTIME_BIN=<hull 4bf76a6> brig run -d claude-code@iso1 --no-project

Findings

  1. brig rm --all never stops the shared gateway on hull on Linux: List() parses hull's No instances found line as an instance. Inline at internal/runtime/hull.go:1797.
  2. A networked run on hull on Linux needs urunc 5c6bc70, and the rc21 bundle pinned in install.sh:77 ships 3c00b69. Inline at internal/runtime/hull.go:1205.

The cgroup failure with urunc 5c6bc70 on a rootless install happens in urunc's task creation, after brig and hull have built the run. On the same host, urunc 3c00b69 passes that step with the same brig and hull.

Comment thread internal/runtime/hull.go
if err != nil {
return true
}
list, err := h.List()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a host with no instances, hull ps -a prints No instances found. List() (hull.go:594-605, unchanged by this PR) keeps every line with two or more fields, so it returns {Name: "No", State: "instances"}. onSharedGateway then runs hull inspect No, which fails, and returns true, so PruneSharedNetwork keeps the gateway.

Failure: after every sandbox is removed, brig rm --all leaves the shared gateway running. Reproduced twice: GET /members answered [] with HTTP 200, hull ps -a printed No instances found, and the gateway process was still running after brig rm --all. strace of brig rm --all shows execve("hull", ["hull", "inspect", "No"]).

hull prints that line from writeInstanceTable (cmd/hull/instancelist.go:30) on macOS and Linux. noInstancesHull in hulllinuxmembers_test.go fakes an empty host as a NAME STATE header with no rows, which is not what hull prints, so the tests do not cover this.

Fix: skip hull's empty-list line in List(), and make noInstancesHull print No instances found.

Comment thread internal/runtime/hull.go
// hull refuses one without the other: the socket says which network to
// join, the CIDR says which address to take on it.
if net != "none" {
args = append(args, "--gateway-sock", gatewaySock, "--gateway-cidr", gatewayCidr)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this change every networked run on hull on Linux joins a gateway, which needs urunc 5c6bc70 to read com.urunc.unikernel.netSocket, netAddress and netGateway. install.sh:77 pins brig-standalone-linux v0.1.0-rc21. Its pins.env has URUNC_REF=3c00b69, and its urunc and containerd-shim-urunc-v2 contain none of the three annotation names.

Failure: on an rc21 install, brig run on hull on Linux fails at start for the default posture and for --network shared. Before this PR, --network shared booted on hull's CNI bridge. hull refuses the start and removes the instance, so no guest runs without a network.

Fix: move the install.sh pin to a bundle that ships urunc 5c6bc70 or later, or state the requirement in docs/runtimes.md and in the PR description until that bundle is released.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be handled by NOFireAI/brig-standalone-linux#40

@ananos ananos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @cmainas, this is the step that makes isolated real on hull on Linux, and the shape is right: one join path for both hulls, refusals with the real reason, pruning through /members. Two things to fix before it lands, two to word or decide, one question.

Must fix

  1. List() reads hull's empty-store line No instances found as an instance named No (hull prints it from cmd/hull/instancelist.go:30). onSharedGateway then runs hull inspect No, the inspect fails, and the doubt keeps the gateway, so brig rm --all never prunes the shared gateway on an empty host. This is the case the pruning exists for. Treat that line as an empty list, and have the fake ps in hulllinuxmembers_test.go print the real text instead of a NAME STATE header hull never prints for an empty store. Alexandros reproduced it; I confirmed it in the code.
  2. Non-root images. isolated is now the default, and every networked run joins a gateway. hvi connects to the socket as the image's user; the socket is created by net.Listen under brig's umask with no chmod, and the community images end in USER ${AGENT_USER}. hull prints a warning and exits 0; brig's narrate holds stderr and shows it only on a failed boot, so the sandbox comes up with no network and nothing says why. Before this PR the same run had the bridge. I do not think brig should fix the permission itself, that belongs to hull/urunc (an open design point: group-writable socket, mode set by the gateway, or a connected fd handed to hvi), but this PR should at least surface hull's warning on a successful boot and state the limit in docs/runtimes.md and docs/security.md. If you prefer to refuse a non-root image on a gateway until that lands, also fine by me.

Wording
3. linuxHullNoPolicyWhy says brig does not hand the rules to the gateway, but joinGateway passes spec.Egress to ensureIsolatedGateway on both hulls. The refusal is right as a conservative gate; the reason should say the path is unproven on a real hull on Linux, or the row flips once it is.

Ordering
4. The pinned bundle (rc21, urunc 3c00b69) ignores the annotations, so on a stock install every networked run on hull now fails where it used to get the bridge, and brig doctor says nothing because the CNI row went and nothing replaced it. rc22 with urunc 5c6bc70, hull 4bf76a6, hull-assets 0.1.12 and hvi rc.3 is tagged and building; the install.sh bump follows as its own PR. A doctor row that checks the bundle's urunc reads netSocket would close the gap for anyone who does not reinstall.

Question
5. The rootless mkdir /sys/fs/cgroup/brig: permission denied with urunc 5c6bc70: brig sets no cgroup path, hull sets none, and the 3c00b69..5c6bc70 diff touches no cgroup code, so I cannot place it in this PR. @asapranidis was the 5c6bc70 urunc built with the same flags and installed the same way as the bundle's? Worth a urunc-side look before rc22 goes into install.sh.

Branch is on current main with no merge commits, CI green.

Comment thread internal/runtime/hull.go
if err != nil {
return true
}
list, err := h.List()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hull ps -a on an empty store prints No instances found, which passes List()'s two-field test and becomes {Name: No, State: instances}; the inspect of No fails and the gateway is kept. Treat that line as an empty list, and have the fake ps in the test print it.

Comment thread internal/runtime/hull.go Outdated
const (
linuxHullNoNetworkWhy = "hull on Linux gives a sandbox no network of its own yet"
linuxHullRemedy = "Use nerdctl (BRIG_RUNTIME=nerdctl), or detach the policy"
linuxHullNoPolicyWhy = "brig does not yet hand a policy's rules to the network gateway " +

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

joinGateway hands spec.Egress to the gateway on Linux too (line 685), so this reason is circular. "Unproven against a real hull on Linux" is the true one.

Comment thread internal/runtime/hull.go
if net != "none" {
var release func()
var err error
gatewaySock, gatewayCidr, usedIsolated, release, err = h.joinGateway(spec.Name, net, spec.Egress)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hvi joins as the image's user; with a non-root USER the socket is not writable and hull only warns, which narrate drops on a successful boot. Surface the warning here, or refuse, and state the limit in the docs.

hull on Linux put every networked sandbox on its CNI bridge, one L2
segment for all of them. A sandbox reached another's listener, and could
answer ARP for it, whatever posture either asked for. So isolated was
refused, and brig fell back to shared when no posture was named, which
put every sandbox on the bridge by default.

hull now runs network-gateway on Linux, and urunc joins hvi to it, so a
sandbox there joins a gateway as an hvi sandbox does on macOS: one of
its own when it is isolated, and the shared one otherwise. The bridge is
never used, and a networked run without a gateway is refused. Isolated,
the shipped profiles' network: isolated and an unset posture now run,
and the unset posture is isolated, as on nerdctl. Only hvi joins a
gateway, so a networked run on another monitor is refused.

On Linux the process table cannot show which guests are on the shared
gateway: urunc binds the socket into hvi's root, so no host process names
it. So brig rm --all stops that gateway there only when the gateway's
GET /members lists no guest and hull lists no instance on it. hull
prints "No instances found" for an empty store, and brig read that line
as an instance named No, which hull could not inspect, so the doubt kept
the gateway up for good. That line now reads as no instance. The
instances cover a guest whose hvi has not connected yet, which the
gateway does not list, after its boot has let go of its lease. A gateway
that cannot answer, from a hull older than /members, is kept, as is one
brig cannot check, and a boot keeps an old one rather than replace it.
Stopping one that looks unused would cut the network of every shared
sandbox still running.

A policy, a divert, brig.internal and a published port are still
refused. Their refusals now give the real reason. brig hands the
gateway the rules and the diverts, as on macOS, but no run on a real
hull on Linux has shown that they hold for a urunc guest.

hvi joins the gateway as the image's user, and only the user who runs
brig may write to its socket, so a guest whose image user maps to
another host user boots with no network: on a rootless install, any
image with a non-root USER. hull warns and exits 0, and brig held that
warning with the rest of hull's output, so nothing said why. brig now
prints hull's warnings on a boot that worked. Letting that user join is
hull's and urunc's to change, and runtimes.md and security.md state the
limit until then.

urunc joins hvi to the gateway through the netSocket, netAddress and
netGateway annotations, from urunc 5c6bc70 on. The urunc of the rc21
runtime bundle ignores them, and hull refuses the start, so on that
bundle no networked sandbox boots on hull. runtimes.md says so.

brig doctor no longer gives hull on Linux a cni row. No run brig starts
there reads a CNI plugin, and the row told a host without them to
install them.

TestProfileIsolationOnLinuxHullNamesNerdctlNotHvi is renamed to
TestProfileIsolationRunsOnLinuxHull: it now asserts the opposite.
TestCNICheckForHullOnLinuxLooksWhereHullDoes goes with the row it
tested, and TestDoctorAsksHullOnLinuxForNoCNI asserts that the row is
gone. The fake hull of the pruning tests prints what hull prints for an
empty store, where it printed a bare header that hull never does.

This changes what security.md claims for the isolated posture on hull
on Linux: it is now provided, and the default.

Signed-off-by: Charalampos Mainas <cmainas@nofire.ai>
brig policy check asked the runtime about a policy's routes, through the
check a boot makes, but not about its egress rules. For an egress-only
policy it printed its general note and exited 0 on every runtime, and
brig run then exited 7 on hull on vz or qemu, on hull on Linux and on
docker. On hull on Linux the note also reads as covering the hvi gateway
a networked sandbox now joins, which brig does not yet start with
the rules.

check now asks the egress answer the run's CanRun reads, on the run path
the run takes, and refuses with the same capability error and exit 7. An
offline run passes, as at boot. With no runtime on the host there is no
run path to ask, and the note stands. Like the route check, it reads the
table and not the probes a boot runs.

Signed-off-by: Charalampos Mainas <cmainas@nofire.ai>
@cmainas
cmainas force-pushed the fix/hull-linux-agetway branch from 02bbb6e to 505524f Compare November 18, 2026 08:30
@cmainas

cmainas commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Hello @ananos

1. `List()` reads hull's empty-store line `No instances found` as an instance named `No` (hull prints it from `cmd/hull/instancelist.go:30`). `onSharedGateway` then runs `hull inspect No`, the inspect fails, and the doubt keeps the gateway, so `brig rm --all` never prunes the shared gateway on an empty host. This is the case the pruning exists for. Treat that line as an empty list, and have the fake `ps` in `hulllinuxmembers_test.go` print the real text instead of a `NAME STATE` header hull never prints for an empty store. Alexandros reproduced it; I confirmed it in the code.

This is pre-existing and not directly connected with this PR, but sure let me include it.

2. Non-root images. `isolated` is now the default, and every networked run joins a gateway. hvi connects to the socket as the image's user; the socket is created by `net.Listen` under brig's umask with no chmod, and the community images end in `USER ${AGENT_USER}`. hull prints a warning and exits 0; brig's `narrate` holds stderr and shows it only on a failed boot, so the sandbox comes up with no network and nothing says why. Before this PR the same run had the bridge. I do not think brig should fix the permission itself, that belongs to hull/urunc (an open design point: group-writable socket, mode set by the gateway, or a connected fd handed to hvi), but this PR should at least surface hull's warning on a successful boot and state the limit in `docs/runtimes.md` and `docs/security.md`. If you prefer to refuse a non-root image on a gateway until that lands, also fine by me.

Well, the VMM must start as the container user and therefore, it must have access to the socket. Now, we either allow everyone to have access to the socket or brig/hull must find the container user to set the appropriate permissions. This is not the job of urunc. In any case, I updated the PR.

Wording 3. linuxHullNoPolicyWhy says brig does not hand the rules to the gateway, but joinGateway passes spec.Egress to ensureIsolatedGateway on both hulls. The refusal is right as a conservative gate; the reason should say the path is unproven on a real hull on Linux, or the row flips once it is.

Addressed.

Ordering 4. The pinned bundle (rc21, urunc 3c00b69) ignores the annotations, so on a stock install every networked run on hull now fails where it used to get the bridge, and brig doctor says nothing because the CNI row went and nothing replaced it. rc22 with urunc 5c6bc70, hull 4bf76a6, hull-assets 0.1.12 and hvi rc.3 is tagged and building; the install.sh bump follows as its own PR. A doctor row that checks the bundle's urunc reads netSocket would close the gap for anyone who does not reinstall.

Yet another time, this is handled by NOFireAI/brig-standalone-linux#40

Question 5. The rootless mkdir /sys/fs/cgroup/brig: permission denied with urunc 5c6bc70: brig sets no cgroup path, hull sets none, and the 3c00b69..5c6bc70 diff touches no cgroup code, so I cannot place it in this PR. @asapranidis was the 5c6bc70 urunc built with the same flags and installed the same way as the bundle's? Worth a urunc-side look before rc22 goes into install.sh.

Please instruct agents to use the makefile targets to build urunc, otherwise the shim will never call urunc, but runc.

@cmainas
cmainas requested a review from ananos November 18, 2026 08:37
@asapranidis

Copy link
Copy Markdown
Member

Correction to my review: the cgroup failure in row 3 came from my urunc build, not from urunc. I built the shim with go build instead of the Makefile target, which changes DefaultCommand = "runc" to "urunc" in the vendored go-runc, so that shim called runc. I rebuilt with make urunc_static shim_static at 5c6bc70 and re-tested this PR at 505524f. The setup is otherwise the same: hull 4bf76a6 and the rootless rc21 bundle with hvi 0.1.0-rc.2.

Case Result
Boot iso1, iso2 (default isolated), sh1, sh2 (--network shared) All four boot, exit 0. Guest kernel 6.18.59.
Guest addresses iso1 198.18.1.2/30, iso2 198.18.1.6/30, sh1 198.18.0.2/24, sh2 198.18.0.3/24. Each guest's resolver is its own gateway.
DNS and egress, no policy example.com resolves. https://example.com and https://api.github.com return 200 from every guest.
brig.internal Does not resolve. No route is attached.
sh2 → sh1:8080 Returns sh1's marker file.
iso2 → iso1, iso1 → sh1, sh2 → iso1, iso2 → sh1 (port 8080) Connection timed out after 5 s. Each listener answers on 127.0.0.1.
brig info NETWORK isolated and NETWORK shared. hull inspect reports network: "gateway" with the matching gatewaySocket.
brig rm of iso1 Its gateway stops.
brig rm of sh1, with sh2 running The shared gateway stays up. GET /members lists 1 guest.
brig rm --all All gateways stop, and no gateway sockets remain.

Not covered: an image with a non-root USER (ghcr.io/brig-sh/claude-code-stock:root runs as root), and policies and routes, which this path refuses.

Finding 1 is fixed in 505524f, and the brig rm --all row confirms it. Finding 2 is tracked in NOFireAI/brig-standalone-linux#40.

@ananos ananos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @cmainas. Three nits are open and nothing blocks the merge. The three items from last round are settled.

Verified

  • List() skips the No instances found line before the two-field test: internal/runtime/hull.go:596-601, with tests at hulllinuxmembers_test.go:255 and :26.
  • brig rm --all keeps the shared gateway when /members is unknown, lists a guest, a boot holds a lease, or hull lists or fails to inspect an instance on it: internal/runtime/hull.go:1769-1780 and 1799-1846, gatewayMembers in gatewayapi.go.
  • A relative BRIG_GATEWAY_SOCK is compared as an absolute path: internal/runtime/hull.go:1807-1811, TestLinuxHullKeepsTheSharedGatewayItNamesRelatively.
  • Isolated is no longer refused on hull on Linux, and an unset posture is no longer forced to shared: linuxSupports in internal/runtime/hull.go, internal/wrap/network.go:83.
  • A networked run on a monitor other than hvi is refused, and linuxRunArgs refuses a networked run with no gateway: internal/runtime/hull.go around 913-926.
  • brig policy check exits 7 on an egress-only policy where the run refuses it, and passes offline: cmd/brig/policy.go:1294-1302, EgressServed in internal/wrap/routemode.go, TestCheckRefusesEgressTheRunCannotEnforce.
  • go test -race ./internal/runtime/ ./internal/wrap/ ./cmd/brig/ passes in all three packages.
  • gofmt -l . prints nothing.

Findings

  • [nit] internal/runtime/hull.go:1776: On hull on Linux, PruneSharedNetwork calls onSharedGateway, and so h.List() (hull ps -a, no deadline), while it holds the gateway's flock. A hull that hangs blocks every leaseSharedGateway until it returns. Run List with a context deadline, or list before taking the lock and re-check under it.
  • [nit] internal/wrap/routemode.go:167: EgressServed reads offline from BRIG_NETWORK and the profile only. A flagless brig run also reuses the recorded session posture (internal/wrap/config.go:643), so brig policy check can exit 7 for an offline-recorded session whose run boots. RouteModeServed has the same pattern.
  • [nit] cmd/brig/doctor_host.go:78: The PR description does not mention that TestCNICheckForHullOnLinuxLooksWhereHullDoes, hullCNIDirs and hullCNICheck are deleted, or that brig doctor on hull on Linux loses its cni row. It names only the renamed TestProfileIsolationOnLinuxHullNamesNerdctlNotHvi, so the removes-tests request covers less than the diff removes.

Dropped since last round

  • internal/runtime/hull.go:599: fixed. List() skips the exact line No instances found, so no bogus instance is produced.
  • internal/runtime/hull.go:879: fixed. linuxSupports refuses a policy before joinGateway runs, and its doc comment gives the real reason.
  • internal/runtime/hull.go:744: fixed. boot() prints the narration warnings through noticef after a successful boot. They are skipped when spec.Progress is set.

Comment thread internal/runtime/hull.go
if guests, known := gatewayMembers(sock); !known || guests > 0 {
return false
}
if h.onSharedGateway(sock) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit]

onSharedGateway calls h.List() here while the flock on the gateway socket is held, and List() runs hull ps -a with no deadline. If hull hangs during brig rm --all, a concurrent brig run --network shared waits in leaseSharedGateway for the same lock. A deadline on List, or listing before the lock and re-checking under it, bounds the wait.

func EgressServed(p profile.Profile, rt runtime.Runtime, bound []policy.Policy) error {
env := NewEnv(p.Name, nil)
net := NetIsolated.RuntimeNet()
if env.String("NETWORK", p.Network) == string(NetOffline) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit]

EgressServed decides offline from BRIG_NETWORK and the profile, while the run also reuses the posture recorded for the session. For a session recorded offline with an egress-only policy attached, brig policy check exits 7 on hull/vz and the run boots. Reading the recorded posture here, or saying in the message that check assumes isolated, would remove the mismatch.

Comment thread cmd/brig/doctor_host.go
@@ -78,12 +78,13 @@ func linuxHostRows(rt runtime.Runtime, hypervisor string) []check {
rows = append(rows, subIDCheck("/etc/subuid", "/etc/subgid"), userNSCheck(rt), uidmapCheck())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit]

The hull-on-Linux cni row is gone from brig doctor, and TestCNICheckForHullOnLinuxLooksWhereHullDoes is deleted with it. The PR description mentions neither, so the removes-tests request names one test while the diff removes two. Adding both to the description fixes it.

@cmainas
cmainas merged commit cf80f90 into main Nov 18, 2026
9 checks passed
@cmainas
cmainas deleted the fix/hull-linux-agetway branch November 18, 2026 11:07

@ananos ananos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked 505524f against my round: List() skips No instances found and the fake ps prints the real text; boot() prints hull's warning: lines after a boot that worked; the policy reason now says the path is unproven on a real hull on Linux; runtimes.md and security.md state the non-root socket limit. Thanks @cmainas!

Branch is on current main with no merge commits, CI green. The install pin moves to rc22 in #670 once this lands; the rootless cgroup question to @asapranidis stays open on the side.

ananos added a commit that referenced this pull request Nov 18, 2026
…1-18

Brig puts a hull on Linux sandbox on a network gateway since #667, and
refuses a policy, a divert and a published port there because no run on
a real hull had shown they work. This records one, on a fresh rootless
install of brig-standalone-linux v0.1.0-rc22 with Brig from main: the
postures, sandbox-to-sandbox reachability, rm --all, and the three
refused capabilities driven at hull level with the flags and API calls
Brig uses on macOS. All hold. The one refusal is hull's own: an image
whose USER is a name.

Refs: brig-sh/hull#105
Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
ananos added a commit that referenced this pull request Nov 19, 2026
…1-18

Brig puts a hull on Linux sandbox on a network gateway since #667, and
refuses a policy, a divert and a published port there because no run on
a real hull had shown they work. This records one, on a fresh rootless
install of brig-standalone-linux v0.1.0-rc22 with Brig from main: the
postures, sandbox-to-sandbox reachability, rm --all, and the three
refused capabilities driven at hull level with the flags and API calls
Brig uses on macOS. All hold. The one refusal is hull's own: an image
whose USER is a name.

Refs: brig-sh/hull#105
Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

removes-tests Removes or renames a test on purpose; skips the tests-kept check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants