Skip to content

feat(runtime): Route and publish on hull on Linux - #682

Merged
ananos merged 6 commits into
mainfrom
feat/hull-linux-routes-publish
Nov 19, 2026
Merged

ananos merged 6 commits into
mainfrom
feat/hull-linux-routes-publish

Conversation

@ananos

@ananos ananos commented Nov 19, 2026

Copy link
Copy Markdown
Member

Summary

hull on Linux refused a divert, brig.internal and a published port, because brig had not shown that the sandbox's gateway serves them there. The run recorded in #674 did: the gateway answered brig.internal with its own address, diverted its port 443 and 169.254.170.2:80 to their sockets, refused a gateway port with no divert, and installed and withdrew a forward through its API. The gateway brig starts on Linux already carried the diverts and the --host record, so nothing was missing but the refusals.

Stacked on #675 (egress policy on hull on Linux). #674 merges first, since the docs here link its page.

Related issues

Refs: brig-sh/hull#105

Changes

  • The route mode row of hull on Linux is enforced over brig.internal. The divert and link-local probes take the run path, so a refusal names hull on io.containerd.urunc.v2, as the rules probe does since feat(runtime): Enforce an egress policy on hull on Linux #675.
  • linuxSupports accepts a divert, brig.internal and a published port on a networked run, and refuses them offline, as the macOS hull does.
  • runLinux installs the published ports on the sandbox's gateway before the boot, as Run does on hvi, and withdraws them when the run fails after that.
  • Tests: the old refusals now assert the new answers (served networked, refused offline); a hull double that takes the diverts and can fail its boot drives a routed run and a failed one.
  • Docs: runtimes, cli (brig network publish works on a running sandbox here), security, broker and policies put hull on Linux beside hvi.

Checklist

  • make all passes (vet, test, build)
  • script/smoke.sh passes
  • I have added or updated tests covering the change
  • I have run go test ./... -race, if the change touches concurrency, subprocesses or the daemon
  • I have exercised the change against a real runtime (brig run <agent>), if it touches the run, exec or credential path
  • I have updated the affected docs (see docs/README.md for the map)

Real runtime, 2026-11-19, rootless brig-standalone-linux v0.1.0-rc22 on amd64, brig built from this branch with #679 on top (without #679 hull refuses every boot from a bundle brig fetched):

Check Result
brig run -d --publish 127.0.0.1:18080:8080 ubuntu@…, a server on the guest's 8080 host curl 127.0.0.1:18080 returns the guest's page
brig network publish ubuntu@… 127.0.0.1:18081:8080 on the running sandbox network ls shows both open; 18081 serves the page
brig network unpublish ubuntu@… 127.0.0.1:18080 18080 refused (curl exit 7), 18081 still open
policy default: deny, allow example.com, on codex@… gateway started with --egress-default deny --egress-allow host=example.com; example.com 200, github.com and one.one.one.one exit 6
policy with a route echo (upstream httpbin.org) gateway started with --divert <gw>:443=<broker divert socket> --host brig.internal=<gw>; in the guest brig.internal resolves to the gateway, and https://brig.internal/<sandbox>/r/echo/headers reaches the broker over TLS under the brig CA (401, no session token under brig sh); httpbin.org direct is refused (exit 6)
brig rm of each no sandbox and no gateway left

Not shown: a full upstream round trip through the route, which needs the agent's session token; brig sh does not carry it.

@ananos
ananos force-pushed the feat/hull-linux-routes-publish branch from cacfe5a to cf02d3c Compare November 19, 2026 08:24
@ananos
ananos marked this pull request as ready for review November 19, 2026 08:36
@ananos
ananos force-pushed the feat/hull-linux-routes-publish branch 2 times, most recently from f8f0184 to 36ca557 Compare November 19, 2026 10:43
@ananos
ananos changed the base branch from feat/hull-linux-egress-policy to main November 19, 2026 12:15
@ananos
ananos force-pushed the feat/hull-linux-routes-publish branch from 36ca557 to 3085061 Compare November 19, 2026 12:22

@cmainas cmainas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @ananos for the PR, just a couple of small comments.

Comment thread internal/runtime/gateway.go Outdated
// shares. With no way to tell an unused gateway from one with guests, an old
// one is reused rather than stopped. It cannot publish a port, and hull on
// Linux publishes none.
// one is reused rather than stopped. An old one reused there may serve no

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR lets hull on Linux publish (runLinux now calls reconcilePublications, hull.go:782), so a gateway that cannot publish is now refused there through unpublishable. Its advice is the macOS one, and on Linux none of it works:

  • "brig replaces such a gateway at the next boot that finds no sandbox on it" (gatewayapi.go:334, and docs/runtimes.md:419): on Linux ensureSharedGateway is called with membersSeen false and reuses any gateway that answers, so stopping every sandbox and running again finds the same one.
  • "Upgrade the runtime": the new hull leaves the running gateway in place, and nothing replaces it.
  • brig rm --all does not stop it either: on Linux PruneSharedNetwork stops the gateway only when GET /members answers an empty list, which a gateway with no API socket, or one too old for /members, never does.

The new comment here says the refusal "names the upgrade", which on Linux does not help. The "older brig" cause it names cannot happen on Linux (brig has passed --api since 7d3c349, before Linux had a gateway); what reaches it is a gateway from a hull without /forwards, or one whose API socket is gone. Give Linux its own remedy (stop the gateway process), or let PruneSharedNetwork stop a gateway whose API is gone when hull lists no instance on it, and correct the comment.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, none of the macOS advice worked on Linux. Fixed in 19a9844 by making the remedy real, not only the wording: on Linux PruneSharedNetwork now stops a shared gateway that will never answer GET /members (API socket gone, or a 404 from a hull older than /members) once hull lists no instance on it, so brig rm --all clears it and the next brig run --publish starts one that publishes. It still keeps a gateway that hull lists an instance on, one whose hull cannot list its instances, and one that fails any other way. Linux has its own refusal now (linuxUnpublishable, also used by brig network publish on a running sandbox): it no longer names an older brig, says a boot reuses the shared gateway, and names brig rm --all; for an isolated sandbox it says brig stop takes the gateway with it while it runs, and names the pid file otherwise. The comments in gateway.go and gatewayapi.go and the paragraph in runtimes.md are corrected too.

// twice" as a hull that takes a link-local divert does. runExit is what
// `hull run` exits with, so a test can fail the boot after the forwards were
// installed and see them withdrawn.
func newRoutingHullDouble(t *testing.T, linux bool, runExit string) (*hull, *hullDouble) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing tests the new publish path on hull on Linux. newRoutingHullDouble takes a runExit so a test can fail the boot, but its only caller (:438) passes "0". No test drives runLinux with spec.Publish: not the reconcile before the boot, and not the withdrawal on a failed boot. The smoke test has no Linux publish case either. Add a runLinux test with Publish and runExit "1" that asserts the forwards were withdrawn, and a negative case for an instance that is still running

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added in 73be080: TestLinuxHullPublishesAPortOnTheGateway drives runLinux with Publish and checks the forward reaches the guest's address; TestALinuxHullBootThatFailsReleasesItsHostPorts fails the boot (runExit "1") and checks this sandbox's forward is withdrawn while another sandbox's forward on the same gateway is not; TestALinuxHullRunBehindAGatewayThatCannotPublishIsToldWhatWorks checks the refusal and that following it (the prune) stops the gateway; and TestLinuxHullNeverStopsASharedGatewayItCannotAsk keeps the three kinds of gateway up while hull lists an instance on them. Each fails with its fix reverted. The smoke test has a Linux publish case too, which passes on a Linux host.

The divert and link-local probes named hull on hvi in every refusal, so a
hull on Linux with a gateway too old for a route would be told the macOS
path. They now take the path, as the probe for the rules does, and the old
names stay as the macOS forms.

The route mode row of hull on Linux is enforced over brig.internal. A run
on 2026-11-18 (brig-standalone-linux v0.1.0-rc22: hull 4bf76a6, urunc
5c6bc70, hvi 0.1.0-rc.3, rootless amd64) showed the sandbox's gateway
answering brig.internal with its own address, diverting port 443 of it
and 169.254.170.2:80 to their sockets, and refusing a port with no divert.

A hull double that takes the diverts, answers the link-local probe and can
fail its boot lets a test drive a routed run and a failed one.

Refs: brig-sh/hull#105
Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
hull on Linux refused a divert, brig.internal and a published port,
because brig had not shown that the sandbox's gateway serves them there.
The run on 2026-11-18 did: the gateway answered brig.internal with its own
address, diverted port 443 of it and 169.254.170.2:80 to their sockets,
and refused a port with no divert. The gateway brig starts on Linux
already carries the diverts and the --host record, so nothing was missing
but the refusals.

linuxSupports now accepts a divert and brig.internal and refuses both on
an offline run, as the macOS hull does, naming hull on
io.containerd.urunc.v2. A published port is accepted on a networked run,
and runLinux installs it on the sandbox's gateway before the boot, as Run
does on hvi, and withdraws it when the run fails after that: a forward
left for a guest that never booted would hold the host port on the shared
gateway until it restarts.

Refs: brig-sh/hull#105
Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
The runtime now serves a divert, brig.internal and a published port on
hull on Linux, so the pages that said it refuses them were wrong. They
now put hull on Linux beside hvi: the same gateway, the same diverts and
the same API for forwards. The measured run is the record of 2026-11-18,
which #674 adds; that pull request merges first, since these pages link
its page.

Refs: brig-sh/hull#105
Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
A route rides the sandbox's gateway as the egress rules do, and urunc
joins only hvi to a gateway. brig policy check answered a route mode as
enforced on hull on Linux on any monitor, while the run refused a
networked sandbox on another monitor with a plain error.

linuxHullOnHvi now holds the refusal for any property, and
RequireGatewayMonitor asks it for a run on a given runtime. The run
refuses brig.internal and a route's divert off hvi with it, and brig
policy check asks it before it answers a route mode, so both exit 7. The
route mode rows in the docs say hvi, with a row for another monitor.

Refs: brig-sh/hull#105
Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
@ananos
ananos force-pushed the feat/hull-linux-routes-publish branch from 3085061 to f2a0b0d Compare November 19, 2026 13:00
@ananos
ananos added this pull request to stack #685 November 19, 2026 13:02
A run on hull on Linux that publishes through a gateway without the
/forwards API, or whose API socket is gone, was refused with the macOS
advice. None of it works there. A boot on Linux reuses any shared
gateway that answers, so stopping every sandbox and running again finds
the same one. An upgrade of hull leaves the running gateway in place. And
brig rm --all kept it, because it stopped a Linux gateway only when
GET /members answered an empty list, which such a gateway never does.
The gateway stayed for good, and no command stopped it.

PruneSharedNetwork now decides a gateway that will never answer
GET /members, with no API socket or from a hull older than /members, by
hull's instances alone. It stops it once hull lists no instance on it,
as it stops one that lists no guest. A gateway with an instance on it,
one hull cannot list instances for, and one whose answer fails for
another reason are kept. What this gives up is a guest that hull does not
list, which only /members would show.

hull on Linux gets its own refusal, linuxUnpublishable. For the shared
gateway it names brig rm --all, which now stops it. For a gateway from a
hull without /forwards it also says to upgrade hull, since the upgrade
alone leaves the gateway running. For an isolated sandbox it names brig
stop while the sandbox runs, and the gateway's pid file while it does
not: a boot reaches the refusal with no sandbox to stop. It drops the
older brig as a cause, which cannot happen on Linux. brig network publish
on a running sandbox gets the same words.

The comments on ensureSharedGateway and PruneSharedNetwork, and
docs/runtimes.md, said what holds on macOS only. They now say what holds
on Linux.

TestLinuxHullNeverStopsASharedGatewayItCannotAsk now holds such a
gateway with an instance on it, or with hull unable to list instances.
Its cases with no instance on it move to
TestLinuxHullStopsASharedGatewayThatCannotListItsGuests, which asserts
the stop. TestLinuxHullKeepsTheSharedGatewayAGuestMayBeOn keeps its
listed guest and failed answer cases and drops the two that moved.

Refs: brig-sh/hull#105
Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
Nothing ran runLinux with a published port. newRoutingHullDouble takes
the exit of hull run so a test can fail the boot, but its one caller
passed 0. So neither the reconcile before the boot nor the withdrawal
after a failed one was tested on hull on Linux.

The tests run it against the shared gateway the publish tests fake. A
boot that works leaves the forward on the gateway, to the address the
run line hands the guest. A boot that fails takes its forward off again,
and leaves the forward of another sandbox on the same gateway alone,
since the withdrawal goes by this sandbox's address. A run behind a
gateway with no API socket is refused before hull run with the Linux
remedy, and brig rm --all, which that remedy names, then stops the
gateway.

The smoke test's hull on Linux section runs the same refusal through the
binary: the stub's gateway opens no API socket, so a run with --publish
is refused naming brig rm --all, nothing reaches hull run, and the rm
stops that gateway. That section runs only on Linux.

Refs: brig-sh/hull#105
Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
@ananos
ananos requested a review from cmainas November 19, 2026 14:36
@ananos
ananos merged commit 8810a25 into main Nov 19, 2026
9 checks passed
@ananos
ananos deleted the feat/hull-linux-routes-publish branch November 19, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants