Repository navigation
fix(runtime): Put hull on Linux sandboxes on a network gateway - #667
Conversation
|
@claude-fleet review |
|
Review task queued. Task: 82fd3011-c49a-4313-80d9-8dbb58e557b9 |
There was a problem hiding this comment.
Request changes: two should_fix findings, both stale claims the diff leaves behind. The code itself holds up — I found no correctness defect in the gateway join, the prune guard or the new policy check refusal, and each new test fails when its fix is reverted.
Findings:
- should_fix, docs/policies.md:994 — "
checkdoes not look at the runtime" is now false - should_fix, cmd/brig/doctor_host.go:414 — the
cnirow still says the plugins are for "a run on the shared network" - nit, docs/cli.md:1316 — "Exit 7 cases" names only
brig runandbrig shfor an egress policy
Verified:
joinGatewayis reached by both hulls and picks the sandbox's own gateway whenisolatedNet, the shared one otherwise — internal/runtime/hull.go:621-646, 674-704;runLinuxjoins beforelinuxRunArgs— hull.go:755-767linuxRunArgsrefuses a networked run with no gateway, so hull's bare--net shared(the CNI bridge) is never reached — hull.go:1161-1165, 1201-1204linuxSupportsstill refuses a policy, a divert,brig.internaland a published port, and now refuses a networked generic boot offhvi— hull.go:857-909PruneSharedNetworkkeeps the shared gateway unless/membersanswers a list and it is empty and no non-stopped instancehull inspectplaces on that socket; every doubt (no/members, 404, 503, no API,psorinspectfailing, an id mismatch, an unknown label) returnstrue— hull.go:1755-1760, 1781-1821;gatewayMembersreadsnull,{}and a non-200 as unknown — gatewayapi.go:128-140ensureSharedGateway(bin, false)on Linux never stops an API-less old gateway — gateway.go:279-296wrap.EgressServedmirrorsRouteModeServedexactly (same env, same isolated-unless-offline posture, sameRoutePath), so check and run read one answer — internal/wrap/routemode.go:141-172;requireEgressignores the posture except fornone— internal/runtime/capability.go:176-186linuxHullSandboxNetworkreads agatewaylabel through the gateway's own socket, and refuses a gateway with no socket — networkinspect.go:131-155NetworkStalenow runs on hull on Linux whateverBRIG_HYPERVISORnames — hull.go:1638-1646- The PR's
removes-testsclaim is exact:script/check-tests-kept.sh 87bb90c HEADreports onlyTestProfileIsolationOnLinuxHullNamesNerdctlNotHvi
Checks run:
go test ./internal/runtime/ ./internal/wrap/ ./cmd/brig/→ ok (3 packages)- Reverted the
wrap.EgressServedblock at cmd/brig/policy.go:1294-1301 →TestCheckRefusesEgressTheRunCannotEnforcefails onhull on vz,hull on qemuanddocker, each "exit 0, want exit 7"; restored - Reverted the
--gateway-sock/--gateway-cidrappend at hull.go:1201-1204 →TestTwoIsolatedSandboxesOnLinuxHullShareNoGatewayfails ("the run line joins no gateway") andTestLinuxHullRunArgsNetworkfails for"",sharedandisolated; restored - Reverted the
h.linuxblock at hull.go:1755-1760 →TestLinuxHullKeepsTheSharedGatewayAGuestMayBeOnfails on/members failsandno API, andTestLinuxHullKeepsTheSharedGatewayASandboxIsOnfails on all five keep cases; restored script/smoke.sh→ PASS (608 ok), the hull-on-Linux section included: both isolated runs join agw/sandbox-*.sock, the plain run carries[--gateway-sock][.../gw/gateway-gofmt -l .→ nothing;go vet ./...→ cleanscript/check-retired-spellings.sh→ rc 0;script/check-claims.sh→ rc 0
Smaller things:
- cmd/brig/policy.go:1278 and :1295 both call
detectRuntimeFor(runtime.Preference{Bin: p.RuntimeBin}). The route block is conditional on there being modes, so the second call is not dead, but one detection hoisted above both would spare a second runtime probe on everypolicy check. gatewayMember.TransportandJoinedAt(gatewayapi.go:120-123) are decoded and never read — onlylen()is. Fine as documentation of the payload; worth a word saying so, since the sweep for write-only fields will keep finding them.- The console example at docs/policies.md:975-980 shows
checkprinting the note and exiting 0 forlocked-down, an egress-only policy. Onvz,qemuor docker that example now exits 7. The first example at line 44 got the new "on Linux with nerdctl, or on macOS withBRIG_HYPERVISOR=hvi" qualifier; this one did not. - script/smoke.sh no longer exercises a refusal on hull on Linux for a networked run on a non-
hvimonitor.TestLinuxHullRefusesWhatItCannotRunYetcovers it, so this is only a note that the binary-level path is unexercised. onSharedGatewayskips onlyState == "stopped". Any other word hull might print for a dead instance (exited,created) keeps the gateway. That is the safe direction, and the comment says doubt keeps it, so no change is needed — but it does mean a gateway can be kept indefinitely by a stale listing.
Not checked:
- hull's side of the new contract: that
hull run --net shared --gateway-sock Xis recorded byhull inspectasnetwork: "gateway"withgatewaySocketas the host path, and thatGET /membersexists and lists a guest from the moment its VMM connects.onSharedGateway(hull.go:1812-1814) treatsnetwork: "shared"as "on no gateway"; if hull instead labels a gateway run by the--netword it was given, then during the window where hvi has not yet connected — the very window the instance check exists to cover —brig rm --allwould stop the shared gateway under a booting guest. The PR states the pruning and the policy check were never run against a real hull, so this is the one place I would want that run before merge. - The 2026-11-16 sandbox-to-sandbox measurement in docs/security.md:1470, and the hull/urunc/hvi revisions it names.
- macOS-only paths:
supports, thevz/qemurows, publication reconciliation, andinternal/secret's keychain tests.
Not checked:
- hull's own behaviour: that
hull inspecton Linux reportsnetwork: "gateway"with a host-sidegatewaySocketfor a run given--net shared --gateway-sock, and thatGET /membersexists and lists a guest as soon as its VMM connects. If hull labels a gateway runshared, internal/runtime/hull.go:1812-1814 reads it as on no gateway andbrig rm --allcould stop the shared gateway under a guest whose hvi has not connected yet. - The 2026-11-16 sandbox-to-sandbox measurement in docs/security.md:1470 and the hull 28f4238 / urunc 5c6bc70 / hvi rc21 revisions it names.
- macOS-only paths:
supports, the vz and qemu capability rows, publication reconciliation, and the internal/secret keychain tests.
Findings outside the diff:
- docs/policies.md:994 (should_fix): This paragraph is now false for the behaviour this PR adds:
checkdoes look at the runtime for a policy without routes, and refuses it with exit 7.
Failure: a user on vz reads "check does not look at the runtime, so it cannot tell you whether this host will boot the run or refuse it", then runsbrig policy check claude-codeand gets exit 7 naming vz. The numbered list above at line 983 ("check does three checks") has the same gap -- the egress check is a fourth.
Evidence: cmd/brig/policy.go:1294-1301 callswrap.EgressServedwhenever a runtime is detected, andcheckPolicyreturns a*runtime.CapabilityErrorthat cmd/brig/exit.go:38 maps to 7; the opposite statement is already in this same file at line 44-46.TestCheckRefusesEgressTheRunCannotEnforcepins it for vz, qemu and docker. - cmd/brig/doctor_host.go:414 (should_fix): The
cnirow for hull on Linux still says the plugins are needed for "a run on the shared network", which this PR says is no longer a thing Brig does.
Failure: on a Linux host with hull and no CNI plugins,brig doctorprintscni fail: no bridge, firewall, ... plugin in /opt/cni/bin, and a run on the shared network needs themand tells the user to run install.sh -- for a bridge docs/security.md:1445 now states Brig never puts a sandbox on. Either the row is stale and should name whatever hull still needs the plugins for (the container side of a gateway run, if anything), or the "never" in security.md is too strong. The row does not change doctor's exit status, so the cost is a wrong instruction rather than a wrong verdict.
Evidence: cmd/brig/doctor_host.go:412-414 and :84 (cni = hullCNICheck); docs/security.md:1445 "Brig never puts a sandbox on hull's own CNI bridge"; internal/runtime/hull.go:1161-1165 refuses a networked run that has no gateway, so no brig run reaches--net sharedalone. - docs/cli.md:1316 (nit): "Exit 7 cases" still names only
brig runandbrig shas the commands that refuse an egress policy with 7.
Failure: someone scripting against the exit-code table -- which docs/stability.md calls stable to script against -- seesbrig policy checkchange from 0 to 7 on vz, qemu and docker with nothing in this list to explain it. Addingbrig policy checkto the sentence at line 1316 covers it.
Evidence: cmd/brig/policy.go:1294-1301; docs/cli.md:1303 points at this section for every exit 7.
Task 82fd3011-c49a-4313-80d9-8dbb58e557b9 on pimox5, model claude-opus-5, effort medium.
e212bda to
02bbb6e
Compare
|
@claude-fleet review |
|
Review task queued. Task: bd9e76bb-b2e9-4b81-87bf-d0288970cb6c |
There was a problem hiding this comment.
Previous round at e212bda: 3 findings.
| Previous finding | Classification | Cited line |
|---|---|---|
docs/policies.md:994 (should_fix): This paragraph is now false for the behaviour this PR adds: check does look at the runtime for a policy without routes, and refuses it with exit 7. Failure: a user on vz reads "check does not look at the runtime, so it cannot tell you whether this host will boot the run or refuse it", then runs brig policy check claude-code and gets exit 7 naming vz. The numbered list above at line 983 ("check does three checks") has the same gap -- the egress check is a fourth. Evidence: cmd/brig/policy.go:1294-1301 calls wrap.EgressServed whenever a runtime is detected, and checkPolicy returns a *runtime.CapabilityError that cmd/brig/exit.go:38 maps to 7; the opposite statement is already in this same file at line 44-46. TestCheckRefusesEgressTheRunCannotEnforce pins it for vz, qemu and docker. |
fixed | docs/policies.md:1000 |
cmd/brig/doctor_host.go:414 (should_fix): The cni row for hull on Linux still says the plugins are needed for "a run on the shared network", which this PR says is no longer a thing Brig does. Failure: on a Linux host with hull and no CNI plugins, brig doctor prints cni fail: no bridge, firewall, ... plugin in /opt/cni/bin, and a run on the shared network needs them and tells the user to run install.sh -- for a bridge docs/security.md:1445 now states Brig never puts a sandbox on. Either the row is stale and should name whatever hull still needs the plugins for (the container side of a gateway run, if anything), or the "never" in security.md is too strong. The row does not change doctor's exit status, so the cost is a wrong instruction rather than a wrong verdict. Evidence: cmd/brig/doctor_host.go:412-414 and :84 (cni = hullCNICheck); docs/security.md:1445 "Brig never puts a sandbox on hull's own CNI bridge"; internal/runtime/hull.go:1161-1165 refuses a networked run that has no gateway, so no brig run reaches --net shared alone. |
fixed | cmd/brig/doctor_host.go:85 |
docs/cli.md:1316 (nit): "Exit 7 cases" still names only brig run and brig sh as the commands that refuse an egress policy with 7. Failure: someone scripting against the exit-code table -- which docs/stability.md calls stable to script against -- sees brig policy check change from 0 to 7 on vz, qemu and docker with nothing in this list to explain it. Adding brig policy check to the sentence at line 1316 covers it. Evidence: cmd/brig/policy.go:1294-1301; docs/cli.md:1303 points at this section for every exit 7. |
fixed | docs/cli.md:1325 |
Previous round's three findings are all fixed. Re-reviewed the whole diff at the current head.
Approve: no blocking or should_fix finding.
Findings:
- nit, internal/runtime/hull.go:844 — the "cannot enforce" reason for hull on Linux describes plumbing that already exists
Verified:
- Previous #1: docs/policies.md:984-1002 now lists the egress check as the fourth check and replaces the "check does not look at the runtime" paragraph with what
checkdoes and does not ask. - Previous #2: the
cnirow is gone for hull on Linux (cmd/brig/doctor_host.go:81-86),hullCNIDirs/hullCNICheckremoved, andTestDoctorAsksHullOnLinuxForNoCNIpins that nerdctl and a host with no runtime keep the row. - Previous #3: docs/cli.md:1325-1328 adds
brig policy checkto the exit-7 section, and says it does not run the boot probes. - One detection for both runtime checks in
checkPolicy— cmd/brig/policy.go:1277-1300;rtis used only underdetectErr == nil. gatewayMembersnow counts without a typed struct, and still reads only a non-null JSON list as known — internal/runtime/gatewayapi.go:119-139, pinned byTestGatewayMembersReadsOnlyAListAsKnown.onSharedGatewaycompares absolute paths and fails closed onfilepath.Abserror,hull ps/inspectfailure, an id mismatch, an unknown network label, and an emptygatewaySocket— internal/runtime/hull.go:1789-1825, pinned byTestLinuxHullKeepsTheSharedGatewayASandboxIsOnand...ItNamesRelatively.- A stopped sandbox is never restarted in place (
Runtime.Removethen a fresh boot — internal/wrap/run.go:323-326), so pruning the shared gateway cannot strand a stopped guest's network. isolatedand a published port, a policy, a divert andbrig.internalstay refused on hull on Linux throughlinuxSupports(internal/runtime/hull.go:852-910), and a networked generic boot offhviis refused at :899-907.- The macOS path is unchanged:
ensureGatewaykeepsmembersSeentrue (internal/runtime/gateway.go:273), andPruneSharedNetwork's new checks are underh.linux.
Checks run:
go test ./internal/runtime/ ./internal/wrap/ ./cmd/brig/→ okgo build ./... && go vet ./...→ clean;gofmt -l .→ nothingscript/check-retired-spellings.sh,script/check-claims.sh→ pass- Reverted the new egress block in cmd/brig/policy.go:1297 (
if false && detectErr == nil) →TestCheckRefusesEgressTheRunCannotEnforcefails on hull/vz, hull/qemu and docker with "exit 0, want exit 7"; file restored,git status --porcelainclean.
Smaller things:
TestLinuxHullNeverStopsTheSharedGateway(internal/runtime/hulllinuxgateway_test.go:112) now sits besideTestLinuxHullStopsTheSharedGatewayNoGuestIsOn; the name reads as the old unconditional behaviour. It tests the no-/memberscase, so something like...NeverStopsASharedGatewayItCannotAskwould say what it pins.- The "urunc binds the socket into hvi's root, so the process table cannot show the guests" explanation now exists in three code copies (internal/runtime/gateway.go:277, internal/runtime/hull.go:1730, internal/runtime/hulllinuxgateway_test.go:109) plus docs/runtimes.md:393-396. One of them (PruneSharedNetwork) could hold it with pointers from the others.
Not checked:
- Whether hull on Linux records
gatewaySocketinhull inspect, lists an instance from the momenthull run --detachreturns, servesGET /members, needs no CNI plugin for--net shared --gateway-sock, and refuses or reroutes a networked non-generic boot (GenericBoot: false, no--hypervisorpassed) ontohvi. All four are claims about brig-sh/hull, which is not in this clone; the doctorcnirow removal and the monitor refusal being generic-boot-only both rest on them. brig runagainst a real hull on Linux for the shared-gateway pruning and the policy check — the PR says neither has been run there, and the stub smoke path cannot answer/members.
Not checked:
- brig-sh/hull behaviour the change rests on: whether
hull inspecton Linux printsgatewaySocket, whetherhull pslists an instance as soon ashull run --detachreturns, whether the gateway serves GET /members, and whether--net shared --gateway-sockneeds no CNI plugin (the basis for removing doctor's cni row) - A networked non-generic boot (GenericBoot false): linuxSupports' hvi-only refusal is inside the GenericBoot branch and no --hypervisor is passed, so the claim that hull boots such an image on hvi or refuses it was not verified
- Any run against a real hull on Linux; the PR states the shared-gateway pruning and the policy check have not been run there
Task bd9e76bb-b2e9-4b81-87bf-d0288970cb6c on ravel-clickbench-ubuntu-2, model claude-opus-5, effort medium.
| const ( | ||
| linuxHullNoNetworkWhy = "hull on Linux gives a sandbox no network of its own yet" | ||
| linuxHullRemedy = "Use nerdctl (BRIG_RUNTIME=nerdctl), or detach the policy" | ||
| linuxHullNoPolicyWhy = "brig does not yet hand a policy's rules to the network gateway " + |
There was a problem hiding this comment.
nit: linuxHullNoPolicyWhy states a gap that the code does not have: brig does hand a policy's rules to the gateway on hull on Linux.
Failure: a user on hull on Linux attaching an egress policy reads "brig does not yet hand a policy's rules to the network gateway of a sandbox on hull on Linux" from brig run, brig policy check, docs/policies.md:1266, docs/runtimes.md:90 and docs/security.md, and a maintainer goes looking for plumbing that is already there. The refusal stands only because the capability table says so, which makes the reason circular.
Evidence: internal/runtime/hull.go:1159 (joinGateway) passes spec.Egress to ensureIsolatedGateway on both hulls, and internal/runtime/gateway.go:558 hands that policy to startGateway; the only thing stopping a Linux policy run is requireEgress(spec, linuxHullPath) at internal/runtime/hull.go:857 reading the row at internal/runtime/capability.go:83. The honest reason looks like "brig has not verified that the gateway enforces the rules for a urunc guest" -- same for linuxHullNoRouteWhy at :846, whose divert flags travel the same path.
asapranidis
left a comment
There was a problem hiding this comment.
Tested on Linux with a real hull, on a rootless install of the rc21 runtime bundle.
Setup
- brig: this PR at e212bda. The findings below are re-checked against 02bbb6e:
List()and theList()call inonSharedGatewayare unchanged, and the rebase does not touch the run path tested here. - hull: 4bf76a6
- runtime bundle: brig-standalone-linux v0.1.0-rc21 (rootless), hvi 0.1.0-rc.2
- urunc: 3c00b69 (rc21), then 5c6bc70 built from source
- host: Linux 7.2.6, x86_64
Results
| Case | urunc | Result |
|---|---|---|
--network offline |
3c00b69 | Boots. Guest kernel 6.18.59, host kernel 7.2.6. The guest has lo and sit0, both down. |
default posture (isolated) |
3c00b69 | brig starts the sandbox's own gateway and passes --gateway-sock. hull refuses the start: urunc started hvi without --net-gateway, so the guest has no network: the shim's urunc does not read the gateway's annotations, which takes urunc-dev/urunc@5c6bc70. The instance is removed. Exit 1. |
isolated and --network shared |
5c6bc70 | Task creation fails: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: mkdir /sys/fs/cgroup/brig: permission denied. |
brig rm --all, no instances |
n/a | The shared gateway is not stopped. See the inline comment on onSharedGateway. |
Sandbox-to-sandbox reachability and egress were not tested. On a rootless install of the rc21 bundle, which is what BRIG_INSTALL_ROOTLESS=1 sh install.sh produces, no networked guest starts: with rc21's urunc hull refuses the start (row 2), and with urunc 5c6bc70 swapped in, task creation fails (row 3).
Findings 1 and 2 do not depend on the host. Finding 1 is in how List() parses hull's output, and finding 2 is in the rc21 release artifacts.
To reproduce row 3:
BRIG_INSTALL_ROOTLESS=1 sh install.sh- Build
uruncandcontainerd-shim-urunc-v2at 5c6bc70 (CGO_ENABLED=1,-tags netgo,osusergo, static) and replace the two binaries in~/.local/share/brig/data/bin. BRIG_RUNTIME=hull BRIG_RUNTIME_BIN=<hull 4bf76a6> brig run -d claude-code@iso1 --no-project
Findings
brig rm --allnever stops the shared gateway on hull on Linux:List()parses hull'sNo instances foundline as an instance. Inline atinternal/runtime/hull.go:1797.- A networked run on hull on Linux needs urunc 5c6bc70, and the rc21 bundle pinned in
install.sh:77ships 3c00b69. Inline atinternal/runtime/hull.go:1205.
The cgroup failure with urunc 5c6bc70 on a rootless install happens in urunc's task creation, after brig and hull have built the run. On the same host, urunc 3c00b69 passes that step with the same brig and hull.
| if err != nil { | ||
| return true | ||
| } | ||
| list, err := h.List() |
There was a problem hiding this comment.
On a host with no instances, hull ps -a prints No instances found. List() (hull.go:594-605, unchanged by this PR) keeps every line with two or more fields, so it returns {Name: "No", State: "instances"}. onSharedGateway then runs hull inspect No, which fails, and returns true, so PruneSharedNetwork keeps the gateway.
Failure: after every sandbox is removed, brig rm --all leaves the shared gateway running. Reproduced twice: GET /members answered [] with HTTP 200, hull ps -a printed No instances found, and the gateway process was still running after brig rm --all. strace of brig rm --all shows execve("hull", ["hull", "inspect", "No"]).
hull prints that line from writeInstanceTable (cmd/hull/instancelist.go:30) on macOS and Linux. noInstancesHull in hulllinuxmembers_test.go fakes an empty host as a NAME STATE header with no rows, which is not what hull prints, so the tests do not cover this.
Fix: skip hull's empty-list line in List(), and make noInstancesHull print No instances found.
| // hull refuses one without the other: the socket says which network to | ||
| // join, the CIDR says which address to take on it. | ||
| if net != "none" { | ||
| args = append(args, "--gateway-sock", gatewaySock, "--gateway-cidr", gatewayCidr) |
There was a problem hiding this comment.
With this change every networked run on hull on Linux joins a gateway, which needs urunc 5c6bc70 to read com.urunc.unikernel.netSocket, netAddress and netGateway. install.sh:77 pins brig-standalone-linux v0.1.0-rc21. Its pins.env has URUNC_REF=3c00b69, and its urunc and containerd-shim-urunc-v2 contain none of the three annotation names.
Failure: on an rc21 install, brig run on hull on Linux fails at start for the default posture and for --network shared. Before this PR, --network shared booted on hull's CNI bridge. hull refuses the start and removes the instance, so no guest runs without a network.
Fix: move the install.sh pin to a bundle that ships urunc 5c6bc70 or later, or state the requirement in docs/runtimes.md and in the PR description until that bundle is released.
There was a problem hiding this comment.
This will be handled by NOFireAI/brig-standalone-linux#40
ananos
left a comment
There was a problem hiding this comment.
Thanks @cmainas, this is the step that makes isolated real on hull on Linux, and the shape is right: one join path for both hulls, refusals with the real reason, pruning through /members. Two things to fix before it lands, two to word or decide, one question.
Must fix
List()reads hull's empty-store lineNo instances foundas an instance namedNo(hull prints it fromcmd/hull/instancelist.go:30).onSharedGatewaythen runshull inspect No, the inspect fails, and the doubt keeps the gateway, sobrig rm --allnever prunes the shared gateway on an empty host. This is the case the pruning exists for. Treat that line as an empty list, and have the fakepsinhulllinuxmembers_test.goprint the real text instead of aNAME STATEheader hull never prints for an empty store. Alexandros reproduced it; I confirmed it in the code.- Non-root images.
isolatedis now the default, and every networked run joins a gateway. hvi connects to the socket as the image's user; the socket is created bynet.Listenunder brig's umask with no chmod, and the community images end inUSER ${AGENT_USER}. hull prints a warning and exits 0; brig'snarrateholds stderr and shows it only on a failed boot, so the sandbox comes up with no network and nothing says why. Before this PR the same run had the bridge. I do not think brig should fix the permission itself, that belongs to hull/urunc (an open design point: group-writable socket, mode set by the gateway, or a connected fd handed to hvi), but this PR should at least surface hull's warning on a successful boot and state the limit indocs/runtimes.mdanddocs/security.md. If you prefer to refuse a non-root image on a gateway until that lands, also fine by me.
Wording
3. linuxHullNoPolicyWhy says brig does not hand the rules to the gateway, but joinGateway passes spec.Egress to ensureIsolatedGateway on both hulls. The refusal is right as a conservative gate; the reason should say the path is unproven on a real hull on Linux, or the row flips once it is.
Ordering
4. The pinned bundle (rc21, urunc 3c00b69) ignores the annotations, so on a stock install every networked run on hull now fails where it used to get the bridge, and brig doctor says nothing because the CNI row went and nothing replaced it. rc22 with urunc 5c6bc70, hull 4bf76a6, hull-assets 0.1.12 and hvi rc.3 is tagged and building; the install.sh bump follows as its own PR. A doctor row that checks the bundle's urunc reads netSocket would close the gap for anyone who does not reinstall.
Question
5. The rootless mkdir /sys/fs/cgroup/brig: permission denied with urunc 5c6bc70: brig sets no cgroup path, hull sets none, and the 3c00b69..5c6bc70 diff touches no cgroup code, so I cannot place it in this PR. @asapranidis was the 5c6bc70 urunc built with the same flags and installed the same way as the bundle's? Worth a urunc-side look before rc22 goes into install.sh.
Branch is on current main with no merge commits, CI green.
| if err != nil { | ||
| return true | ||
| } | ||
| list, err := h.List() |
There was a problem hiding this comment.
hull ps -a on an empty store prints No instances found, which passes List()'s two-field test and becomes {Name: No, State: instances}; the inspect of No fails and the gateway is kept. Treat that line as an empty list, and have the fake ps in the test print it.
| const ( | ||
| linuxHullNoNetworkWhy = "hull on Linux gives a sandbox no network of its own yet" | ||
| linuxHullRemedy = "Use nerdctl (BRIG_RUNTIME=nerdctl), or detach the policy" | ||
| linuxHullNoPolicyWhy = "brig does not yet hand a policy's rules to the network gateway " + |
There was a problem hiding this comment.
joinGateway hands spec.Egress to the gateway on Linux too (line 685), so this reason is circular. "Unproven against a real hull on Linux" is the true one.
| if net != "none" { | ||
| var release func() | ||
| var err error | ||
| gatewaySock, gatewayCidr, usedIsolated, release, err = h.joinGateway(spec.Name, net, spec.Egress) |
There was a problem hiding this comment.
hvi joins as the image's user; with a non-root USER the socket is not writable and hull only warns, which narrate drops on a successful boot. Surface the warning here, or refuse, and state the limit in the docs.
hull on Linux put every networked sandbox on its CNI bridge, one L2 segment for all of them. A sandbox reached another's listener, and could answer ARP for it, whatever posture either asked for. So isolated was refused, and brig fell back to shared when no posture was named, which put every sandbox on the bridge by default. hull now runs network-gateway on Linux, and urunc joins hvi to it, so a sandbox there joins a gateway as an hvi sandbox does on macOS: one of its own when it is isolated, and the shared one otherwise. The bridge is never used, and a networked run without a gateway is refused. Isolated, the shipped profiles' network: isolated and an unset posture now run, and the unset posture is isolated, as on nerdctl. Only hvi joins a gateway, so a networked run on another monitor is refused. On Linux the process table cannot show which guests are on the shared gateway: urunc binds the socket into hvi's root, so no host process names it. So brig rm --all stops that gateway there only when the gateway's GET /members lists no guest and hull lists no instance on it. hull prints "No instances found" for an empty store, and brig read that line as an instance named No, which hull could not inspect, so the doubt kept the gateway up for good. That line now reads as no instance. The instances cover a guest whose hvi has not connected yet, which the gateway does not list, after its boot has let go of its lease. A gateway that cannot answer, from a hull older than /members, is kept, as is one brig cannot check, and a boot keeps an old one rather than replace it. Stopping one that looks unused would cut the network of every shared sandbox still running. A policy, a divert, brig.internal and a published port are still refused. Their refusals now give the real reason. brig hands the gateway the rules and the diverts, as on macOS, but no run on a real hull on Linux has shown that they hold for a urunc guest. hvi joins the gateway as the image's user, and only the user who runs brig may write to its socket, so a guest whose image user maps to another host user boots with no network: on a rootless install, any image with a non-root USER. hull warns and exits 0, and brig held that warning with the rest of hull's output, so nothing said why. brig now prints hull's warnings on a boot that worked. Letting that user join is hull's and urunc's to change, and runtimes.md and security.md state the limit until then. urunc joins hvi to the gateway through the netSocket, netAddress and netGateway annotations, from urunc 5c6bc70 on. The urunc of the rc21 runtime bundle ignores them, and hull refuses the start, so on that bundle no networked sandbox boots on hull. runtimes.md says so. brig doctor no longer gives hull on Linux a cni row. No run brig starts there reads a CNI plugin, and the row told a host without them to install them. TestProfileIsolationOnLinuxHullNamesNerdctlNotHvi is renamed to TestProfileIsolationRunsOnLinuxHull: it now asserts the opposite. TestCNICheckForHullOnLinuxLooksWhereHullDoes goes with the row it tested, and TestDoctorAsksHullOnLinuxForNoCNI asserts that the row is gone. The fake hull of the pruning tests prints what hull prints for an empty store, where it printed a bare header that hull never does. This changes what security.md claims for the isolated posture on hull on Linux: it is now provided, and the default. Signed-off-by: Charalampos Mainas <cmainas@nofire.ai>
brig policy check asked the runtime about a policy's routes, through the check a boot makes, but not about its egress rules. For an egress-only policy it printed its general note and exited 0 on every runtime, and brig run then exited 7 on hull on vz or qemu, on hull on Linux and on docker. On hull on Linux the note also reads as covering the hvi gateway a networked sandbox now joins, which brig does not yet start with the rules. check now asks the egress answer the run's CanRun reads, on the run path the run takes, and refuses with the same capability error and exit 7. An offline run passes, as at boot. With no runtime on the host there is no run path to ask, and the note stands. Like the route check, it reads the table and not the probes a boot runs. Signed-off-by: Charalampos Mainas <cmainas@nofire.ai>
02bbb6e to
505524f
Compare
|
Hello @ananos
This is pre-existing and not directly connected with this PR, but sure let me include it.
Well, the VMM must start as the container user and therefore, it must have access to the socket. Now, we either allow everyone to have access to the socket or brig/hull must find the container user to set the appropriate permissions. This is not the job of urunc. In any case, I updated the PR.
Addressed.
Yet another time, this is handled by NOFireAI/brig-standalone-linux#40
Please instruct agents to use the makefile targets to build urunc, otherwise the shim will never call urunc, but runc. |
|
Correction to my review: the cgroup failure in row 3 came from my urunc build, not from urunc. I built the shim with
Not covered: an image with a non-root Finding 1 is fixed in 505524f, and the |
ananos
left a comment
There was a problem hiding this comment.
Thanks @cmainas. Three nits are open and nothing blocks the merge. The three items from last round are settled.
Verified
List()skips theNo instances foundline before the two-field test:internal/runtime/hull.go:596-601, with tests athulllinuxmembers_test.go:255and:26.brig rm --allkeeps the shared gateway when/membersis unknown, lists a guest, a boot holds a lease, or hull lists or fails to inspect an instance on it:internal/runtime/hull.go:1769-1780and1799-1846,gatewayMembersingatewayapi.go.- A relative
BRIG_GATEWAY_SOCKis compared as an absolute path:internal/runtime/hull.go:1807-1811,TestLinuxHullKeepsTheSharedGatewayItNamesRelatively. - Isolated is no longer refused on hull on Linux, and an unset posture is no longer forced to shared:
linuxSupportsininternal/runtime/hull.go,internal/wrap/network.go:83. - A networked run on a monitor other than hvi is refused, and
linuxRunArgsrefuses a networked run with no gateway:internal/runtime/hull.goaround913-926. brig policy checkexits 7 on an egress-only policy where the run refuses it, and passes offline:cmd/brig/policy.go:1294-1302,EgressServedininternal/wrap/routemode.go,TestCheckRefusesEgressTheRunCannotEnforce.go test -race ./internal/runtime/ ./internal/wrap/ ./cmd/brig/passes in all three packages.gofmt -l .prints nothing.
Findings
- [nit]
internal/runtime/hull.go:1776: On hull on Linux,PruneSharedNetworkcallsonSharedGateway, and soh.List()(hull ps -a, no deadline), while it holds the gateway's flock. A hull that hangs blocks everyleaseSharedGatewayuntil it returns. RunListwith a context deadline, or list before taking the lock and re-check under it. - [nit]
internal/wrap/routemode.go:167:EgressServedreads offline fromBRIG_NETWORKand the profile only. A flaglessbrig runalso reuses the recorded session posture (internal/wrap/config.go:643), sobrig policy checkcan exit 7 for an offline-recorded session whose run boots.RouteModeServedhas the same pattern. - [nit]
cmd/brig/doctor_host.go:78: The PR description does not mention thatTestCNICheckForHullOnLinuxLooksWhereHullDoes,hullCNIDirsandhullCNICheckare deleted, or thatbrig doctoron hull on Linux loses itscnirow. It names only the renamedTestProfileIsolationOnLinuxHullNamesNerdctlNotHvi, so theremoves-testsrequest covers less than the diff removes.
Dropped since last round
internal/runtime/hull.go:599: fixed.List()skips the exact lineNo instances found, so no bogus instance is produced.internal/runtime/hull.go:879: fixed.linuxSupportsrefuses a policy beforejoinGatewayruns, and its doc comment gives the real reason.internal/runtime/hull.go:744: fixed.boot()prints the narration warnings throughnoticefafter a successful boot. They are skipped whenspec.Progressis set.
| if guests, known := gatewayMembers(sock); !known || guests > 0 { | ||
| return false | ||
| } | ||
| if h.onSharedGateway(sock) { |
There was a problem hiding this comment.
[nit]
onSharedGateway calls h.List() here while the flock on the gateway socket is held, and List() runs hull ps -a with no deadline. If hull hangs during brig rm --all, a concurrent brig run --network shared waits in leaseSharedGateway for the same lock. A deadline on List, or listing before the lock and re-checking under it, bounds the wait.
| func EgressServed(p profile.Profile, rt runtime.Runtime, bound []policy.Policy) error { | ||
| env := NewEnv(p.Name, nil) | ||
| net := NetIsolated.RuntimeNet() | ||
| if env.String("NETWORK", p.Network) == string(NetOffline) { |
There was a problem hiding this comment.
[nit]
EgressServed decides offline from BRIG_NETWORK and the profile, while the run also reuses the posture recorded for the session. For a session recorded offline with an egress-only policy attached, brig policy check exits 7 on hull/vz and the run boots. Reading the recorded posture here, or saying in the message that check assumes isolated, would remove the mismatch.
| @@ -78,12 +78,13 @@ func linuxHostRows(rt runtime.Runtime, hypervisor string) []check { | |||
| rows = append(rows, subIDCheck("/etc/subuid", "/etc/subgid"), userNSCheck(rt), uidmapCheck()) | |||
There was a problem hiding this comment.
[nit]
The hull-on-Linux cni row is gone from brig doctor, and TestCNICheckForHullOnLinuxLooksWhereHullDoes is deleted with it. The PR description mentions neither, so the removes-tests request names one test while the diff removes two. Adding both to the description fixes it.
ananos
left a comment
There was a problem hiding this comment.
Checked 505524f against my round: List() skips No instances found and the fake ps prints the real text; boot() prints hull's warning: lines after a boot that worked; the policy reason now says the path is unproven on a real hull on Linux; runtimes.md and security.md state the non-root socket limit. Thanks @cmainas!
Branch is on current main with no merge commits, CI green. The install pin moves to rc22 in #670 once this lands; the rootless cgroup question to @asapranidis stays open on the side.
…1-18 Brig puts a hull on Linux sandbox on a network gateway since #667, and refuses a policy, a divert and a published port there because no run on a real hull had shown they work. This records one, on a fresh rootless install of brig-standalone-linux v0.1.0-rc22 with Brig from main: the postures, sandbox-to-sandbox reachability, rm --all, and the three refused capabilities driven at hull level with the flags and API calls Brig uses on macOS. All hold. The one refusal is hull's own: an image whose USER is a name. Refs: brig-sh/hull#105 Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
…1-18 Brig puts a hull on Linux sandbox on a network gateway since #667, and refuses a policy, a divert and a published port there because no run on a real hull had shown they work. This records one, on a fresh rootless install of brig-standalone-linux v0.1.0-rc22 with Brig from main: the postures, sandbox-to-sandbox reachability, rm --all, and the three refused capabilities driven at hull level with the flags and API calls Brig uses on macOS. All hold. The one refusal is hull's own: an image whose USER is a name. Refs: brig-sh/hull#105 Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
Summary
hull on Linux put every networked sandbox on its CNI bridge, one L2 segment for all of them, so a sandbox reached another's listener and could answer ARP for it whatever posture either asked for.
isolatedwas refused there, and with no posture named brig fell back toshared, so every sandbox was on the bridge by default.hull now runs
network-gatewayon Linux and urunc joins hvi to it, so a sandbox on hull on Linux joins a gateway as an hvi sandbox does on macOS: one of its own when isolated, the shared one otherwise.isolatednow runs there and is the default, as on nerdctl. The bridge is never used. A networked run without a gateway, or on a monitor other than hvi, is refused. A policy, a divert,brig.internaland a published port are still refused, now with the real reason.brig rm --allstops the shared gateway once nothing is on it. On Linux the process table cannot show the guests, because urunc binds the socket into hvi's root, so brig asks the gateway'sGET /membersand checks that no instancehull inspectreports is on it. The second check covers a guest whose hvi has not connected yet, which the gateway does not list. A gateway that cannot answer (a hull older than/members) or that brig cannot check is kept.brig policy checknow refuses an egress-only policy on a run path that cannot enforce it, with exit 7, asbrig rundoes. It printed its general note and exited 0, and the run then failed.The sandbox-to-sandbox behaviour was measured on a real hull on Linux on 2026-11-16 (hull
28f4238, urunc5c6bc70, the hvi of brig-standalone-linux v0.1.0-rc21): onsharedone microVM reached the other's listener, and with each on its own gateway the same request timed out while outbound connections worked. That was before the shared-gateway pruning and the policy check were added, and neither has been run against a real hull.go test -race ./...andmake allpass except theinternal/secretkeyring integration tests, which need an unlocked desktop keyring and fail the same way on main on this host.This PR needs the
removes-testslabel:TestProfileIsolationOnLinuxHullNamesNerdctlNotHviis renamed toTestProfileIsolationRunsOnLinuxHull, because it now asserts the opposite.Related issues
Refs brig-sh/hull#105
Changes
isolated, the shared one whenshared. An unset posture isisolated.brig.internaland a published port name the real reason.brig infoand the stale-network check read a hull on Linux sandbox's network fromhull inspect(network,gatewaySocket).brig rm --allstops the shared gateway on hull on Linux only whenGET /memberslists no guest and no instance hull lists, other than a stopped one, is on it. Any doubt keeps it: no/members, a failed answer, no API, or ahull psorhull inspectthat fails.brig policy checkasks the egress answer the run reads, on the run path the run takes, and refuses with exit 7 where the run would. An offline run passes.docs/security.mdsaysisolatedis provided and the default on hull on Linux, with the measured sandbox-to-sandbox row.docs/runtimes.mdcovers the gateway on Linux and when the shared one is stopped.docs/policies.mdsays check refuses what the run refuses.Checklist
make allpasses (vet, test, build)script/smoke.shpassesgo test ./... -race, if the change touches concurrency, subprocesses or the daemonbrig run <agent>), if it touches the run, exec or credential pathCredentials and the sandbox boundary
Neither promise moves: each VM gets only the directories and credentials named for its session, before and after. What changes is the network posture on hull on Linux, which
docs/security.mdnow states:isolatedis provided and the default, where before it was refused and sandboxes shared one bridge.Negative tests:
brig rm --allkeeps the shared gateway while it lists a guest, while it cannot answer/members(404, 503, no API), while a boot holds a lease, and while hull lists a running instance on it or cannot be asked.brig policy checkrefuses an egress-only policy on hull on vz or qemu and on docker, with exit 7, and the egress answer it reads refuses hull on Linux on every monitor.