Repository navigation
feat(runtime): Route and publish on hull on Linux - #682
Conversation
cacfe5a to
cf02d3c
Compare
f8f0184 to
36ca557
Compare
36ca557 to
3085061
Compare
| // shares. With no way to tell an unused gateway from one with guests, an old | ||
| // one is reused rather than stopped. It cannot publish a port, and hull on | ||
| // Linux publishes none. | ||
| // one is reused rather than stopped. An old one reused there may serve no |
There was a problem hiding this comment.
This PR lets hull on Linux publish (runLinux now calls reconcilePublications, hull.go:782), so a gateway that cannot publish is now refused there through unpublishable. Its advice is the macOS one, and on Linux none of it works:
- "brig replaces such a gateway at the next boot that finds no sandbox on it" (
gatewayapi.go:334, anddocs/runtimes.md:419): on LinuxensureSharedGatewayis called withmembersSeenfalse and reuses any gateway that answers, so stopping every sandbox and running again finds the same one. - "Upgrade the runtime": the new hull leaves the running gateway in place, and nothing replaces it.
brig rm --alldoes not stop it either: on LinuxPruneSharedNetworkstops the gateway only whenGET /membersanswers an empty list, which a gateway with no API socket, or one too old for/members, never does.
The new comment here says the refusal "names the upgrade", which on Linux does not help. The "older brig" cause it names cannot happen on Linux (brig has passed --api since 7d3c349, before Linux had a gateway); what reaches it is a gateway from a hull without /forwards, or one whose API socket is gone. Give Linux its own remedy (stop the gateway process), or let PruneSharedNetwork stop a gateway whose API is gone when hull lists no instance on it, and correct the comment.
There was a problem hiding this comment.
right, none of the macOS advice worked on Linux. Fixed in 19a9844 by making the remedy real, not only the wording: on Linux PruneSharedNetwork now stops a shared gateway that will never answer GET /members (API socket gone, or a 404 from a hull older than /members) once hull lists no instance on it, so brig rm --all clears it and the next brig run --publish starts one that publishes. It still keeps a gateway that hull lists an instance on, one whose hull cannot list its instances, and one that fails any other way. Linux has its own refusal now (linuxUnpublishable, also used by brig network publish on a running sandbox): it no longer names an older brig, says a boot reuses the shared gateway, and names brig rm --all; for an isolated sandbox it says brig stop takes the gateway with it while it runs, and names the pid file otherwise. The comments in gateway.go and gatewayapi.go and the paragraph in runtimes.md are corrected too.
| // twice" as a hull that takes a link-local divert does. runExit is what | ||
| // `hull run` exits with, so a test can fail the boot after the forwards were | ||
| // installed and see them withdrawn. | ||
| func newRoutingHullDouble(t *testing.T, linux bool, runExit string) (*hull, *hullDouble) { |
There was a problem hiding this comment.
Nothing tests the new publish path on hull on Linux. newRoutingHullDouble takes a runExit so a test can fail the boot, but its only caller (:438) passes "0". No test drives runLinux with spec.Publish: not the reconcile before the boot, and not the withdrawal on a failed boot. The smoke test has no Linux publish case either. Add a runLinux test with Publish and runExit "1" that asserts the forwards were withdrawn, and a negative case for an instance that is still running
There was a problem hiding this comment.
added in 73be080: TestLinuxHullPublishesAPortOnTheGateway drives runLinux with Publish and checks the forward reaches the guest's address; TestALinuxHullBootThatFailsReleasesItsHostPorts fails the boot (runExit "1") and checks this sandbox's forward is withdrawn while another sandbox's forward on the same gateway is not; TestALinuxHullRunBehindAGatewayThatCannotPublishIsToldWhatWorks checks the refusal and that following it (the prune) stops the gateway; and TestLinuxHullNeverStopsASharedGatewayItCannotAsk keeps the three kinds of gateway up while hull lists an instance on them. Each fails with its fix reverted. The smoke test has a Linux publish case too, which passes on a Linux host.
The divert and link-local probes named hull on hvi in every refusal, so a hull on Linux with a gateway too old for a route would be told the macOS path. They now take the path, as the probe for the rules does, and the old names stay as the macOS forms. The route mode row of hull on Linux is enforced over brig.internal. A run on 2026-11-18 (brig-standalone-linux v0.1.0-rc22: hull 4bf76a6, urunc 5c6bc70, hvi 0.1.0-rc.3, rootless amd64) showed the sandbox's gateway answering brig.internal with its own address, diverting port 443 of it and 169.254.170.2:80 to their sockets, and refusing a port with no divert. A hull double that takes the diverts, answers the link-local probe and can fail its boot lets a test drive a routed run and a failed one. Refs: brig-sh/hull#105 Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
hull on Linux refused a divert, brig.internal and a published port, because brig had not shown that the sandbox's gateway serves them there. The run on 2026-11-18 did: the gateway answered brig.internal with its own address, diverted port 443 of it and 169.254.170.2:80 to their sockets, and refused a port with no divert. The gateway brig starts on Linux already carries the diverts and the --host record, so nothing was missing but the refusals. linuxSupports now accepts a divert and brig.internal and refuses both on an offline run, as the macOS hull does, naming hull on io.containerd.urunc.v2. A published port is accepted on a networked run, and runLinux installs it on the sandbox's gateway before the boot, as Run does on hvi, and withdraws it when the run fails after that: a forward left for a guest that never booted would hold the host port on the shared gateway until it restarts. Refs: brig-sh/hull#105 Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
The runtime now serves a divert, brig.internal and a published port on hull on Linux, so the pages that said it refuses them were wrong. They now put hull on Linux beside hvi: the same gateway, the same diverts and the same API for forwards. The measured run is the record of 2026-11-18, which #674 adds; that pull request merges first, since these pages link its page. Refs: brig-sh/hull#105 Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
A route rides the sandbox's gateway as the egress rules do, and urunc joins only hvi to a gateway. brig policy check answered a route mode as enforced on hull on Linux on any monitor, while the run refused a networked sandbox on another monitor with a plain error. linuxHullOnHvi now holds the refusal for any property, and RequireGatewayMonitor asks it for a run on a given runtime. The run refuses brig.internal and a route's divert off hvi with it, and brig policy check asks it before it answers a route mode, so both exit 7. The route mode rows in the docs say hvi, with a row for another monitor. Refs: brig-sh/hull#105 Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
3085061 to
f2a0b0d
Compare
A run on hull on Linux that publishes through a gateway without the /forwards API, or whose API socket is gone, was refused with the macOS advice. None of it works there. A boot on Linux reuses any shared gateway that answers, so stopping every sandbox and running again finds the same one. An upgrade of hull leaves the running gateway in place. And brig rm --all kept it, because it stopped a Linux gateway only when GET /members answered an empty list, which such a gateway never does. The gateway stayed for good, and no command stopped it. PruneSharedNetwork now decides a gateway that will never answer GET /members, with no API socket or from a hull older than /members, by hull's instances alone. It stops it once hull lists no instance on it, as it stops one that lists no guest. A gateway with an instance on it, one hull cannot list instances for, and one whose answer fails for another reason are kept. What this gives up is a guest that hull does not list, which only /members would show. hull on Linux gets its own refusal, linuxUnpublishable. For the shared gateway it names brig rm --all, which now stops it. For a gateway from a hull without /forwards it also says to upgrade hull, since the upgrade alone leaves the gateway running. For an isolated sandbox it names brig stop while the sandbox runs, and the gateway's pid file while it does not: a boot reaches the refusal with no sandbox to stop. It drops the older brig as a cause, which cannot happen on Linux. brig network publish on a running sandbox gets the same words. The comments on ensureSharedGateway and PruneSharedNetwork, and docs/runtimes.md, said what holds on macOS only. They now say what holds on Linux. TestLinuxHullNeverStopsASharedGatewayItCannotAsk now holds such a gateway with an instance on it, or with hull unable to list instances. Its cases with no instance on it move to TestLinuxHullStopsASharedGatewayThatCannotListItsGuests, which asserts the stop. TestLinuxHullKeepsTheSharedGatewayAGuestMayBeOn keeps its listed guest and failed answer cases and drops the two that moved. Refs: brig-sh/hull#105 Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
Nothing ran runLinux with a published port. newRoutingHullDouble takes the exit of hull run so a test can fail the boot, but its one caller passed 0. So neither the reconcile before the boot nor the withdrawal after a failed one was tested on hull on Linux. The tests run it against the shared gateway the publish tests fake. A boot that works leaves the forward on the gateway, to the address the run line hands the guest. A boot that fails takes its forward off again, and leaves the forward of another sandbox on the same gateway alone, since the withdrawal goes by this sandbox's address. A run behind a gateway with no API socket is refused before hull run with the Linux remedy, and brig rm --all, which that remedy names, then stops the gateway. The smoke test's hull on Linux section runs the same refusal through the binary: the stub's gateway opens no API socket, so a run with --publish is refused naming brig rm --all, nothing reaches hull run, and the rm stops that gateway. That section runs only on Linux. Refs: brig-sh/hull#105 Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
Summary
hull on Linux refused a divert,
brig.internaland a published port, because brig had not shown that the sandbox's gateway serves them there. The run recorded in #674 did: the gateway answeredbrig.internalwith its own address, diverted its port 443 and169.254.170.2:80to their sockets, refused a gateway port with no divert, and installed and withdrew a forward through its API. The gateway brig starts on Linux already carried the diverts and the--hostrecord, so nothing was missing but the refusals.Stacked on #675 (egress policy on hull on Linux). #674 merges first, since the docs here link its page.
Related issues
Refs: brig-sh/hull#105
Changes
enforcedoverbrig.internal. The divert and link-local probes take the run path, so a refusal nameshull on io.containerd.urunc.v2, as the rules probe does since feat(runtime): Enforce an egress policy on hull on Linux #675.linuxSupportsaccepts a divert,brig.internaland a published port on a networked run, and refuses them offline, as the macOS hull does.runLinuxinstalls the published ports on the sandbox's gateway before the boot, asRundoes on hvi, and withdraws them when the run fails after that.brig network publishworks on a running sandbox here), security, broker and policies put hull on Linux beside hvi.Checklist
make allpasses (vet, test, build)script/smoke.shpassesgo test ./... -race, if the change touches concurrency, subprocesses or the daemonbrig run <agent>), if it touches the run, exec or credential pathReal runtime, 2026-11-19, rootless brig-standalone-linux v0.1.0-rc22 on amd64, brig built from this branch with #679 on top (without #679 hull refuses every boot from a bundle brig fetched):
brig run -d --publish 127.0.0.1:18080:8080 ubuntu@…, a server on the guest's 8080curl 127.0.0.1:18080returns the guest's pagebrig network publish ubuntu@… 127.0.0.1:18081:8080on the running sandboxnetwork lsshows bothopen; 18081 serves the pagebrig network unpublish ubuntu@… 127.0.0.1:18080default: deny, allowexample.com, oncodex@…--egress-default deny --egress-allow host=example.com; example.com 200, github.com and one.one.one.one exit 6echo(upstream httpbin.org)--divert <gw>:443=<broker divert socket> --host brig.internal=<gw>; in the guestbrig.internalresolves to the gateway, andhttps://brig.internal/<sandbox>/r/echo/headersreaches the broker over TLS under the brig CA (401, no session token underbrig sh); httpbin.org direct is refused (exit 6)brig rmof eachNot shown: a full upstream round trip through the route, which needs the agent's session token;
brig shdoes not carry it.