Skip to content

Latest commit

 

History

History
1037 lines (811 loc) · 51 KB

federated-fleet.md

File metadata and controls

1037 lines (811 loc) · 51 KB

Federated Forge

A Kenn Forge fleet gives several development machines one shared control point without moving their repositories, worktrees, or agent sessions to a central server.

Why use a fleet

Use a fleet when one machine is not the right place for every task. For example, you may keep routine work on a laptop, run large builds on a workstation, and use a machine with a special operating system or GPU when a task needs it.

  • Open any machine's Forge to see workspaces from every machine and the node that owns each one.
  • Create a workspace on any writable machine from any machine's Forge, and choose which machine new workspaces use by default.
  • Sync pull requests, issues, and activity once instead of spending provider API budget on every machine.
  • Keep each repository, worktree, process, and tmux session on its execution machine.

If one machine already handles all your work, keep Forge standalone. A fleet adds value when the execution location matters.

The mental model

One hub owns provider synchronization, provider changes, and the fleet-wide view. Each spoke owns its local repositories, workspaces, Git commands, processes, and tmux sessions. The hub can also own local workspaces, so it is an execution machine as well as the coordinator.

<title id="fleet-topology-title">Forge hub-and-spoke topology</title> A browser can open the hub or either spoke. The hub talks to the Git provider for synchronized provider data and sends workspace or terminal operations to the spoke that owns the work. Each spoke keeps its own repositories, workspaces, and agents. Your browser Open any Forge directly
<rect class="fleet-diagram__node" x="675" y="45" width="230" height="90" rx="14" />
<text class="fleet-diagram__title" x="790" y="82" text-anchor="middle">Git provider</text>
<text class="fleet-diagram__body" x="790" y="111" text-anchor="middle">Pull requests, issues, Git</text>

<rect class="fleet-diagram__node fleet-diagram__node--hub" x="330" y="210" width="300" height="130" rx="14" />
<text class="fleet-diagram__eyebrow" x="480" y="245" text-anchor="middle">HUB</text>
<text class="fleet-diagram__title" x="480" y="276" text-anchor="middle">Shared provider data + fleet view</text>
<text class="fleet-diagram__body" x="480" y="307" text-anchor="middle">Also owns its local workspaces</text>

<rect class="fleet-diagram__node" x="55" y="425" width="330" height="100" rx="14" />
<text class="fleet-diagram__eyebrow" x="220" y="458" text-anchor="middle">SPOKE A</text>
<text class="fleet-diagram__title" x="220" y="489" text-anchor="middle">Local repos, workspaces, agents</text>

<rect class="fleet-diagram__node" x="575" y="425" width="330" height="100" rx="14" />
<text class="fleet-diagram__eyebrow" x="740" y="458" text-anchor="middle">SPOKE B</text>
<text class="fleet-diagram__title" x="740" y="489" text-anchor="middle">Local repos, workspaces, agents</text>

<path class="fleet-diagram__line fleet-diagram__line--secondary" d="M282 115 C340 145 380 165 415 210" marker-end="url(#fleet-arrow)" />
<path class="fleet-diagram__line fleet-diagram__line--secondary" d="M145 135 C110 245 120 340 185 425" marker-end="url(#fleet-arrow)" />
<path class="fleet-diagram__line fleet-diagram__line--secondary" d="M215 135 C420 165 845 140 830 425" marker-end="url(#fleet-arrow)" />
<text class="fleet-diagram__label" x="177" y="220">private HTTPS</text>

<path class="fleet-diagram__line" d="M630 250 C690 225 745 185 775 136" marker-start="url(#fleet-arrow-start)" marker-end="url(#fleet-arrow)" />
<text class="fleet-diagram__label" x="700" y="214">provider API</text>

<path class="fleet-diagram__line" d="M405 340 C360 375 315 397 270 425" marker-start="url(#fleet-arrow-start)" marker-end="url(#fleet-arrow)" />
<path class="fleet-diagram__line" d="M555 340 C600 375 645 397 690 425" marker-start="url(#fleet-arrow-start)" marker-end="url(#fleet-arrow)" />
<text class="fleet-diagram__label" x="480" y="389" text-anchor="middle">authenticated Forge API</text>
The hub coordinates the fleet. It does not become the filesystem or process owner for a spoke.

Every machine runs the same kenn-forge binary and keeps its own directly accessible UI. Forge-to-Forge requests and terminal WebSockets use authenticated HTTPS. The hub does not use SSH to start a spoke or attach to its tmux server; start and supervise each daemon on its own machine.

Fleet rules at a glance

Question Hub Spoke
Which workspaces appear? Workspaces from the full fleet, each with an execution-node badge The same
Where can New workspace run? Any writable fleet member selected in Run on The same; the spoke sends the request through the hub
Where do pull-request, issue, and quick-action workspaces run? On this Forge's default machine The same
Who runs Git and agent processes? The machine that owns the workspace The machine that owns the workspace
Who syncs provider data and applies provider changes? The hub The spoke asks the hub

Each Forge has its own default machine. Without one, new workspaces run on the Forge whose UI you opened.

This guide uses the following example topology. Each name is a private HTTPS origin reachable from every fleet machine:

https://forge-hub.example.test  hub and local execution
https://build-a.example.test            spoke and local execution
https://build-b.example.test            spoke and local execution

Tailscale Serve is a first-class way to provide those origins, but it is not a fleet requirement. A private LAN, UniFi network, VPN, or custom reverse proxy works through the same federation protocol.

Understand the trust boundary

Use federation only inside one operator-controlled private network. Keep every Forge listener behind that private ingress.

Private network access is necessary, but it is not sufficient. Forge also requires HTTPS, local browser authentication, and separate credentials created during enrollment. An activated hub and its spokes form one administrative trust domain. Someone who controls an active peer should be treated as controlling fleet workspace and terminal operations.

An active spoke can create, inspect, remove, and attach to workspaces on every other member. It sends those requests to the hub, which forwards them with its own credential for the owning member. Spokes never hold credentials for each other.

Pending spokes have narrower access. They can resolve only the provider facts needed for preparation, hand state to the hub, and use enrollment routes. They cannot browse general provider data or perform provider writes. Those preparation routes are limited by operation, not by repository. Creating and transferring the one-time token approves that machine to become a full fleet member, so give a token only to a machine you trust with the fleet's provider data.

Before you start

On every machine:

  1. Install the same kenn-forge build on every machine.
  2. Install Git, tmux, your agent CLIs, and the language tools used by local workspaces. Authenticate the agents as the user that will run Forge.
  3. Log in to the Git host used for local clone, fetch, and push operations. For GitHub, gh auth login is the normal route.
  4. Choose one stable private HTTPS origin for that machine.
  5. Confirm every spoke can reach the hub. The hub reaches a spoke through a connection the spoke opens, so the hub does not need its own route to the spoke. See Spokes the hub cannot reach.

fleet setup configures Forge and its service. It does not provision the operating system, install development tools, clone your repositories, or authenticate agents. SSH access is useful for provisioning and upgrades; federation itself uses HTTPS and does not require reciprocal SSH keys.

The examples below use the default config location. If your service uses a custom config, pass that same --config /path/to/config.toml to every fleet command, including setup, token creation, join, preparation, and revocation. These commands contact the local daemon selected by that config; the hub URL does not select your local instance.

Do not copy a Forge data directory between machines. Each directory receives a stable random node ID on first start.

A federation origin contains only a scheme and authority, such as https://build-a.example.test. Forge rejects cleartext HTTP origins, URL paths, query strings, fragments, and embedded credentials. Explicit ports are allowed; default :443 is normalized away.

Federation also requires Forge to use the root base path (base_path = "/"). Non-root UI prefixes are not part of the federation protocol.

Setup at a glance

Set up the hub and spoke services first. Then enroll one spoke at a time. A spoke stays standalone until preparation has safely moved provider ownership to the hub and a service restart activates the enrollment.

<title id="fleet-enrollment-title">Four steps to enroll one Forge spoke</title> Set up a hub and standalone spoke, create a one-time token on the hub and join from the spoke, prepare the spoke so provider state moves to the hub, then restart and verify the active spoke.
<circle class="fleet-diagram__step" cx="75" cy="42" r="23" />
<text class="fleet-diagram__step-number" x="75" y="49" text-anchor="middle">1</text>
<rect class="fleet-diagram__node" x="35" y="78" width="200" height="125" rx="14" />
<text class="fleet-diagram__title" x="135" y="115" text-anchor="middle">Set up services</text>
<text class="fleet-diagram__body" x="135" y="146" text-anchor="middle">Hub + standalone spoke</text>
<text class="fleet-diagram__command" x="135" y="176" text-anchor="middle">fleet setup</text>

<circle class="fleet-diagram__step" cx="310" cy="42" r="23" />
<text class="fleet-diagram__step-number" x="310" y="49" text-anchor="middle">2</text>
<rect class="fleet-diagram__node" x="270" y="78" width="200" height="125" rx="14" />
<text class="fleet-diagram__title" x="370" y="115" text-anchor="middle">Join</text>
<text class="fleet-diagram__body" x="370" y="146" text-anchor="middle">One-time hub token</text>
<text class="fleet-diagram__command" x="370" y="176" text-anchor="middle">fleet join</text>

<circle class="fleet-diagram__step" cx="545" cy="42" r="23" />
<text class="fleet-diagram__step-number" x="545" y="49" text-anchor="middle">3</text>
<rect class="fleet-diagram__node" x="505" y="78" width="200" height="125" rx="14" />
<text class="fleet-diagram__title" x="605" y="115" text-anchor="middle">Prepare</text>
<text class="fleet-diagram__body" x="605" y="146" text-anchor="middle">Drain + hand off state</text>
<text class="fleet-diagram__command" x="605" y="176" text-anchor="middle">fleet prepare-spoke</text>

<circle class="fleet-diagram__step" cx="780" cy="42" r="23" />
<text class="fleet-diagram__step-number" x="780" y="49" text-anchor="middle">4</text>
<rect class="fleet-diagram__node fleet-diagram__node--hub" x="740" y="78" width="185" height="125" rx="14" />
<text class="fleet-diagram__title" x="832" y="115" text-anchor="middle">Activate</text>
<text class="fleet-diagram__body" x="832" y="146" text-anchor="middle">Restart + verify</text>
<text class="fleet-diagram__command" x="832" y="176" text-anchor="middle">active spoke</text>

<path class="fleet-diagram__line" d="M235 141 H265" marker-end="url(#fleet-flow-arrow)" />
<path class="fleet-diagram__line" d="M470 141 H500" marker-end="url(#fleet-flow-arrow)" />
<path class="fleet-diagram__line" d="M705 141 H735" marker-end="url(#fleet-flow-arrow)" />
Complete all four steps for one spoke and verify it before enrolling the next.

Set up the hub

Run setup as the operating-system user that will run Forge. The normal Tailscale path discovers the machine's certificate name and current Tailscale login, publishes Forge with Tailscale Serve, installs a per-user service, and checks the protected HTTPS API.

For the Tailscale path, first join the machine to your tailnet and enable HTTPS certificates. On Linux, an administrator can let the Forge user manage Tailscale without running Forge as root:

sudo tailscale set --operator="$USER"

This grants the current user control of the local Tailscale daemon; see the Tailscale CLI reference. Then run Forge setup as that user:

kenn-forge fleet setup hub --tailscale

On macOS, setup also finds the command-line client inside the standard Tailscale app when tailscale is not on PATH.

Review the displayed plan and confirm it. For unattended setup, inspect the same plan with --dry-run, then repeat with --yes.

With --tailscale, anyone whose Tailscale login is on the allowlist opens the UI without a token. Setup allowlists this device's login and keeps any logins you allowed before. If your phone or another machine signs in to Tailscale with a different login, for example because the device is shared from another tailnet, add it:

kenn-forge fleet setup hub --tailscale \
  --tailscale-login you@example.com \
  --tailscale-login you@work.example

Passing --tailscale-login replaces the device's own login in the new allowlist, so list every login you use. To remove a login, delete it from [api.tailscale_serve] allowed_users in the config and restart Forge.

Serve publishes on HTTPS port 443 by default. To use another port, such as https://build-a.example.ts.net:8091, pass --tailscale-https-port 8091. Running setup again keeps the port of the existing origin.

MCP access through Serve requires explicit [mcp].enabled = true. The same logins can then reach it at the Serve origin without a token, for example https://build-a.example.ts.net/mcp. The omitted default enables only the direct loopback companion.

If your LAN, UniFi network, VPN, or reverse proxy already provides private HTTPS, give Forge its canonical origin instead:

kenn-forge fleet setup hub \
  --origin https://forge-hub.example.test

The --origin path never calls Tailscale. It configures and supervises the same loopback Forge service, while your ingress owns DNS and TLS. It retains Forge's normal bearer and browser-cookie authentication.

Setup requires exactly one of --tailscale and --origin. It does not expose the Forge listener directly, weaken API authentication, or make the service depend on a particular network product.

The hub also needs its own provider credentials. Open Settings → Repositories on the hub and add the repositories it should synchronize. Cloning a repository on disk does not add it to this list. On a fresh hub, allow the initial sync to populate provider data before moving existing spokes.

Set up each spoke

Run the matching setup command on each spoke:

# Tailscale Serve
kenn-forge fleet setup spoke --tailscale

# Operator-managed private HTTPS
kenn-forge fleet setup spoke --origin https://build-a.example.test

Spoke setup deliberately leaves the daemon in standalone mode. Do not set fleet.role = "spoke" by hand for a spoke you enroll with fleet join. Enrollment preparation persists that role only after provider writes drain and state handoff completes. A spoke whose membership you declare in config sets the role in config instead, and Forge runs the same preparation when it starts.

On Linux, setup installs a systemd user service and enables lingering so Forge survives logout. On macOS, it installs a LaunchAgent under the selected user. Both service definitions execute the installed binary directly and preserve the setup user's HOME and PATH. They do not start an interactive shell or copy arbitrary exported credentials. Configure any additional credential environment through your service manager.

Run setup with a stable PATH that includes your agent CLIs and tool-manager shims. A minimal SSH command environment can omit tools that work in your interactive terminal. Verify by launching an agent from a Forge workspace after the service restarts.

Tailscale identity mode treats local processes on the Forge host as trusted, because Tailscale Serve forwards identity headers over loopback. Use it on a single-user or otherwise trusted machine. On a multi-user host, use an external origin and Forge's bearer/cookie authentication instead.

Adopt an existing service

If Forge already runs under a service you maintain, inspect fleet setup ... --dry-run first. Setup refuses to overwrite a service definition it does not own. For an existing federation-ready daemon with working private HTTPS, keep its service and continue at Enroll one spoke at a time. Re-enrollment does not require reinstalling the service or moving its data.

If you want setup to own the service instead, back up the current definition and configuration, stop it, and explicitly move the old definition aside before running setup. Keep its data directory and installed binary path consistent with the displayed plan.

Spokes the hub cannot reach

Every active spoke keeps one tunnel open to its hub. The spoke opens it over the hub's HTTPS origin, next to its provider event stream. While the tunnel is connected, the hub sends all of its requests to that spoke through it: workspace snapshots, remote workspace operations, terminals, browser login, and revocation. A spoke that can reach the hub, but that the hub cannot reach, therefore appears in the hub's workspace view like any other spoke.

When the tunnel is down, the hub dials the spoke's enrolled origin directly. If the hub can never reach that origin, add outbound_disabled = true to the spoke's existing [[fleet.members]] entry in the hub's config. Identify the entry by its node_id. The change reloads without a restart.

An outbound-disabled spoke is reached only through its tunnel. While the tunnel is down, the hub skips its snapshot requests, excludes it from the workspace host selector, and refuses remote operations targeting it, without waiting for a direct request to time out. It remains listed in Fleet settings and does not produce an unreachable-host warning in the workspace view.

Revoking an outbound-disabled spoke whose tunnel is down does not call the spoke; it observes the revocation when it next checks its activation with the hub.

Keep credentials separate

Four credential types serve different purposes:

Credential Stored on Purpose
Browser/API session Each daemon Authorizes a person using that daemon's UI or local API
Federation credential Hub and enrolled spoke Authorizes one exact Forge-to-Forge direction and route set
Provider API credential Hub Synchronizes provider data and performs provider mutations
Git credential Each execution machine Clones, fetches, pulls, and pushes directly against the Git host

Do not copy browser cookies or API tokens between machines. Enrollment creates a different machine credential in each direction. Forge stores inbound credentials as digests and keeps credential files private to the daemon account.

A spoke needs its own Git credential even though it does not synchronize provider data. Preparation verifies the exact Git credential route before it stores a workspace launch specification.

The hub may have more than one credential source. User-attributed provider mutations use a user credential. Forge-managed Git follows normal credential priority and can use an App credential for repositories covered by that App. Verify the route that matters to your deployment instead of assuming all hub Git traffic uses one credential.

The hub strips browser authorization, cookies, origin, and forwarding headers before it calls a spoke. It adds only the federation credential enrolled for that exact HTTPS origin and does not follow redirects.

Enroll one spoke at a time

Finish and verify one spoke before enrolling another. This keeps state handoff and rollback bounded to one machine.

1. Create a one-time token

On the hub:

umask 077
kenn-forge fleet enrollment-token \
  --base-url https://forge-hub.example.test \
  --name "Forge hub" \
  --ttl 10m > ./forge-enrollment-token

The token is printed once, expires, and can create one enrollment. Creating and transferring it approves that spoke to finish preparation and activate. Treat it as a membership credential.

Transfer the file through an approved secret-sharing channel. Do not put the token in a command-line argument or config file.

2. Join from the spoke

On build-a:

kenn-forge fleet join https://forge-hub.example.test \
  --base-url https://build-a.example.test \
  --name "Build spoke A" \
  --token-file ./forge-enrollment-token
rm ./forge-enrollment-token

For automation, pass the token on standard input instead of creating a spoke-side file:

secret-command | kenn-forge fleet join \
  https://forge-hub.example.test \
  --base-url https://build-a.example.test \
  --name "Build spoke A"

Interactive terminals use a hidden token prompt when neither input method is provided.

Joining records a pending enrollment. It does not change the spoke role or restart the daemon. Record the printed enrollment ID with the host and origin in your private operations inventory; revocation uses that ID. Remove any remaining token copies after a successful join.

3. Prepare the spoke

Run on the spoke:

kenn-forge fleet prepare-spoke

Preparation stops new provider writes, waits for admitted writes and deferred merges, drains notification acknowledgements, refreshes workspace launch information, hands review drafts and user-authored workflows to the hub, and seals local provider writes. If it reports concrete remaining work, resolve that work and run the command again.

This handoff does not copy the entire provider database or move repositories, worktrees, or agent sessions. The hub must already track the repositories needed by existing workspaces, and the spoke must be able to use their Git credentials.

The token's original deadline no longer applies after preparation starts.

4. Restart and verify activation

When preparation reports completion, restart Forge through the spoke's service manager. Activation happens during startup. Verify the spoke is active in the hub's Fleet settings and in the spoke's direct UI.

For services installed by fleet setup, run the command for your platform as the service user:

# Linux
systemctl --user restart kenn-forge.service
systemctl --user status kenn-forge.service --no-pager

# macOS, in the user's GUI login session
launchctl kickstart -k "gui/$(id -u)/io.kenn.forge"

For an existing custom service, use its actual unit or LaunchAgent label. A running process alone does not prove enrollment activated.

Do not enroll the next spoke yet. Complete the checks in Verify the fleet for this spoke first.

Declare membership in config

Instead of running the four enrollment steps, you can declare the hub and its spokes in each daemon's config. This suits fleets deployed from a configuration management tool. A declared spoke joins, prepares, and activates on its first start, with no CLI step and no second restart.

Each spoke gets its own enrollment secret: at least 32 characters with no whitespace, such as the output of openssl rand -hex 32. Put the same secret in a file on the hub and on that spoke, readable only by the Forge service user. Secret managers that publish files as symlinks work. Never put the secret itself in config.toml.

On the hub, list each spoke by name and origin:

[fleet]
enabled = true
base_url = "https://forge-hub.example.test"

[[fleet.declared_spokes]]
name = "build-a"
base_url = "https://build-a.example.test"
enrollment_secret_file = "/run/secrets/forge-fleet-build-a"

On the spoke, set the role and name the hub:

[fleet]
enabled = true
role = "spoke"
base_url = "https://build-a.example.test"

[fleet.hub]
base_url = "https://forge-hub.example.test"
enrollment_secret_file = "/run/secrets/forge-fleet-build-a"

Declared membership and a default machine combine. A spoke declared this way can also set workspaces.default_execution_target to another member's origin, which configuration management knows before any machine starts; see Choose where new workspaces run.

Forge records what it learns in config.toml: the hub's node ID in fleet.hub.node_id on the spoke and each active spoke in fleet.members on the hub. The Forge service user must be able to write the file. If your configuration tool rewrites it without these entries, Forge restores them at the next start or config reload.

Deploy the hub first, then start the spoke. On startup the spoke:

  1. Joins the hub with the secret. The hub admits the join only from the origin it declares for that secret.
  2. Runs the same preparation as fleet prepare-spoke: it drains provider writes and hands review drafts and user-authored workflows to the hub.
  3. Activates, as a restart after manual preparation would.

Until the spoke has activated for the first time, Forge exits with an error when it cannot read its secret file, write its config, or reach the hub. Its service manager restarts it until all three work. After that, the spoke behaves like any enrolled spoke: restarts keep the same enrollment, and a hub outage pauses only provider work. If the hub rejects the join or preparation finds work it cannot finish, the spoke starts with local execution only and logs the reason with the next step. Fix the cause and restart the spoke.

If the hub stays unreachable for more than 24 hours after the spoke joins and before preparation starts, the hub drops the pending enrollment. Write a new secret on both machines; the spoke joins again when it next starts.

A spoke cannot send notification read marks to the provider, so a standalone daemon with read marks still waiting to be sent cannot finish preparation after you declare it a spoke. Convert such a daemon with fleet join and fleet prepare-spoke while it runs standalone, then add the declared config.

Each secret starts one enrollment, like a one-time token. A spoke that restarts with a joined enrollment does not use its secret again. Rotate the secret on both machines when you:

  • reinstall a spoke with a new data directory. The hub revokes the enrollment of the machine previously at that origin and enrolls the new one.
  • see a join rejected because the secret was consumed. This happens when the hub accepted a join but the spoke never saw the reply.

Removing a spoke from fleet.declared_spokes revokes its enrollment when the hub reloads its config or restarts. The hub tells the spoke if it can reach it. If the spoke is offline, it learns of the removal the next time it renews its activation lease, within 6 hours of coming back, or the next time it starts. The hub logs revoke fleet spoke removed from fleet.declared_spokes when it removes the spoke. Once the spoke has learned, it starts with local execution only and a reason saying the hub revoked its enrollment, and kenn-forge fleet leave accepts it. A removed spoke starts with local execution only until you return it to standalone operation with Return a spoke to standalone operation, or write a new secret on the hub and the spoke and declare it again. It does so after you delete its secret file too. If the hub refuses a new secret, the spoke keeps its revoked enrollment, and fleet leave still accepts it.

Declared and CLI-enrolled spokes can share a hub. When a spoke enrolled with fleet join sits at an origin you later declare, the hub adopts it as declared membership, so an existing fleet can move to declared config without re-enrolling. Removing declarations never revokes spokes you enrolled with the CLI and did not declare.

Abort or revoke an enrollment

Before activation, restore standalone operation from the spoke:

kenn-forge fleet abort-preparation

This revokes the pending enrollment and reopens the durable provider-write gate. On a standalone process, provider writes resume immediately. If the daemon already restarted in spoke mode, the command reports that another restart is required.

If the hub is unavailable, --force performs local recovery and prints the enrollment ID that must be revoked later. The spoke retains only a narrow revocation credential so that later hub cleanup can still complete.

To remove a pending or active enrollment, run on the hub:

kenn-forge fleet revoke ENROLLMENT_ID

The first attempt requires the spoke to be reachable so it can invalidate its local side of the relationship. Once the spoke acknowledges revocation, retries can finish hub cleanup without contacting it again. Revoke before removing a spoke from configuration. After activation, restoring a former standalone spoke requires its complete pre-enrollment unit: config, database, federation store, credential store, and binary. Do not restore only one file from that set.

Verify the fleet

Configuration checks are not enough. Exercise the protected paths and the behavior an operator depends on.

Transport and identity

  • Open every direct HTTPS UI and establish its own authenticated browser session.
  • Confirm every HTTPS certificate validates normally.
  • Confirm the HTTPS ingress is reachable only through the intended private network.
  • Confirm every daemon reports a unique node ID.
  • Restart each daemon and confirm its node ID and enrollment persist.

Fleet view and local ownership

  • Confirm every UI shows the same fleet host set and provider data.
  • Create a disposable workspace on each execution host.
  • Confirm every UI lists each workspace under its owning machine.
  • Start a tmux-backed session and attach through the spoke's direct UI.
  • Attach to the same session through the hub's WebSocket proxy and through another spoke's UI.
  • From one spoke's UI, create a workspace on another spoke, then remove it and confirm the owning spoke performs the deletion.

Provider and Git ownership

  • Observe provider synchronization on the hub.
  • Confirm spokes run no provider synchronization worker.
  • Start one safe provider mutation from a spoke UI and observe it complete through the hub.
  • Clone or fetch a private repository on every spoke and confirm that machine's Git credential route is used.
  • Confirm provider API request budget is not independently consumed by every spoke.

Failure behavior

  • Stop one spoke. Healthy machines and their workspaces should remain visible; only the stopped spoke should become unavailable.
  • Stop the hub. Each spoke should retain local workspace authority, lose access to other members' workspaces, keep provider data already loaded in that browser tab visible behind a stale-data banner, disable Sync and pull-request Refresh, and mark aggregate data incomplete.
  • Reload a spoke UI while the hub remains stopped. The new page has no persistent provider cache, so provider pages cannot repopulate; local workspaces must remain usable.
  • Restart the hub. Spokes should reconnect, replay events, reconcile provider state, and restore the fleet view without a browser reload.
  • Restart each spoke. Its direct UI, local workspaces, enrollment, and remote terminal attachment should recover.

Use the fleet

Use the Forge selector in the top bar to open any fleet member directly. Choosing the hub from a spoke, or a spoke from the hub, signs you in to that Forge automatically and opens the same page there. From a spoke, other spokes open as ordinary links and use their own sign-in, such as Tailscale Serve. Browser modifiers open the plain link in another tab or window. Each origin keeps its own browser state, filters, searches, terminal connections, and event cursors; changing one Forge tab does not retarget another.

Every member's UI shows the fleet-wide workspace view, and each workspace row shows its execution-node badge. Supported actions call the owning machine's HTTPS API, directly from the hub and through the hub from a spoke:

  • create a repository, pull-request, or issue workspace on any writable member;
  • inspect, refresh, or remove a workspace on its owning machine;
  • launch or stop a runtime session, including quick actions;
  • inspect local Git state;
  • open a terminal through the WebSocket bridge.

The Run on selector in New workspace starts on this Forge's default machine and keeps unavailable machines visible with their reason.

No Forge creates a local proxy process for a remote terminal. Input, output, resize messages, and close events pass between WebSocket connections: one hop from the hub, two from a spoke.

If the hub is down, a spoke remains useful for its own workspaces but cannot reach other members' workspaces or provide current hub-owned provider data.

Choose where new workspaces run

Set workspaces.default_execution_target in a Forge's config.toml to the fleet member where new workspaces from that Forge's UI should run. Name the member by its HTTPS origin or its node ID:

[workspaces]
default_execution_target = "https://build-b.example.test"

For example, set this on the build-a spoke to drive from its UI while every new workspace runs on build-b. The hub keeps syncing the provider and applying provider changes as before. The value also accepts devbox:<id> for a devbox on a hub or standalone Forge. Leave it empty, or name this Forge itself, to create workspaces locally, so configuration management can write the same value on every machine.

Settings → Workspaces lists the fleet members as well and saves the same key. The default applies to the pull-request and issue workspace buttons, quick actions, onboarding, and the Run on selector's starting choice.

When the default machine is offline, Forge does not fall back to this Forge. The workspace button is disabled, the page names the unavailable machine and why, and it offers a Create on button for each machine that can take the workspace now. A quick action run on selected items reports the unavailable machine for each item and creates nothing.

A spoke reaches other members only through an active hub. After upgrading, upgrade the hub first, then restart each spoke; the hub grants the spoke this access when the spoke next activates.

Understand Sync, Refresh, and hub outages

Provider controls keep the same owner regardless of which Forge UI you opened:

Action in a spoke UI What Forge does
Top-bar Sync Asks the hub to sync its configured repositories. The spoke does not run a second provider sync.
Sync current repo Asks the hub to sync only the repository selected by the route or repository filter. Use this when one repository needs attention without spending a full-fleet sync budget.
Pull-request Refresh Asks the hub to fetch that pull request, then returns the refreshed detail to the spoke.
Start a shell or agent Starts the session on the spoke from the existing local workspace. It does not contact the hub.
Provider mutation Sends the mutation to the hub, which applies it with the hub's provider credential.

Opening a pull request normally reads the hub's stored detail first. If the repository is tracked but that pull request is missing from the stored projection, Forge performs one targeted pull-request sync automatically. This repairs a cold or partially completed repository sync without starting another full sync. If the repository itself is not configured, Forge directs you to Settings → Repositories; add it there, then use Sync current repo or reopen the pull request.

When a spoke loses its hub connection, it keeps provider data already loaded in that browser tab visible and labels it as stale. Sync and pull-request Refresh are disabled, provider changes are unavailable, and local workspace and terminal operations continue. New shell and agent sessions in an existing workspace also remain local; generated agent context uses the last validated workspace facts. A fresh browser load has no durable provider cache, so it cannot reconstruct provider pages until the hub returns.

Reconnection is automatic. The spoke replays hub events and reconciles the visible provider projection before removing the stale-data banner. Do not reload merely to recover a connection.

Installing a provider API credential on a spoke does not create automatic failover. In spoke role, Forge uses local credentials for Git clone, fetch, pull, and push, but it does not construct an independent provider syncer. Automatic multi-writer provider failover is not currently supported.

Advanced: publish with your own reverse proxy

The setup command always binds Forge to loopback and validates the public authority with trust_reverse_proxy = false. Your proxy must preserve the incoming Host header and forward to the configured loopback port. It must not rewrite Host to the upstream address.

For example:

https://build-a.example.test {
    bind 192.0.2.20
    reverse_proxy 127.0.0.1:8091
}

Use your real private address and a certificate trusted by every fleet machine. Verify the HTTPS URL with the normal operating-system trust store. Do not use curl -k; it hides certificate and hostname mistakes that also break federation.

--origin does not manage this proxy, its certificate, DNS, or browser login. That separation is intentional: the federation contract is canonical HTTPS, not Tailscale or Caddy.

Replace the hub

Do not replace the hub by changing DNS or copying one credential. An enrollment is bound to the hub's stable node ID and canonical origin. The hub also stores provider state alongside its own local execution state.

Choose the recovery you need before changing enrollment:

Goal Approach
Restore the same hub after an OS reinstall Restore its complete matching backup, including identity, configuration, database, and credentials. Restore its original HTTPS origin and verify existing spokes.
Start with a fresh hub and retain local workspaces on other machines Create a new hub identity, configure its repositories, then revoke and re-enroll each old member using the procedure below.
Move the complete provider archive to a different hub while leaving the old hub's execution state behind Forge does not provide an automated selective migration for this. Retain the old backup; do not transplant its database or individual credentials into a new hub.

Move to a fresh hub

This procedure rebuilds provider data through synchronization. It preserves each retained machine's local identity and workspaces; it does not promise to reproduce the old hub's full archive. Preparation transfers review drafts and user-authored workflows when each machine joins.

The old hub and spokes must remain reachable for revocation. If the old hub was lost in a reinstall, restore it from its matching backup first. A replacement at the same hostname with an empty data directory is still a different hub.

  1. Record and back up the old fleet. Record each machine's node ID, origin, enrollment ID, config and data paths, service definition, binary version, and local workspace list. Pause fleet changes. Stop the Forge services while taking a consistent backup of their databases and state, including any SQLite WAL files, enrollment and credential stores, configuration, binaries, and service definitions. Keep these as matching rollback sets, with access restricted to the operator. Back up local repositories and worktrees separately if they are outside those paths. Restart the old fleet and verify it before proceeding.
  2. Prepare the new hub. Use a new data directory and its own private HTTPS origin. Follow Set up the hub, authenticate its provider and Git access, and configure its repository list. Check actual repository and pull-request reads, plus any local agent tools the hub will run.
  3. Move one spoke. On the old hub, run kenn-forge fleet revoke ENROLLMENT_ID using that spoke's old enrollment ID. Then create a new token on the new hub and follow join, prepare, restart, and verify on the spoke. Use the new hub origin and retain the spoke's existing origin, config, and data directory. Expect provider access to pause between revocation and activation. Confirm the spoke's node ID and local workspace list are unchanged, and that the new hub can reach its workspaces and terminals, before moving the next spoke.
  4. Move the old hub last, if it will become a spoke. Revoke all its old members first; a hub with members cannot join another hub. Keep its existing service, origin, and data directory, then run the same join, prepare, and restart sequence against the new hub. Its local workspaces remain on that machine. Do not set its role by hand or run spoke setup merely to change the role. If retiring the old hub instead, stop its service after the cutover.
  5. Verify and record the final fleet. Check every direct HTTPS UI and the new hub's aggregate workspace view. Exercise a local agent and a terminal through the hub on each execution machine. Record the new enrollment IDs, service restart commands, and verification results. Keep the pre-cutover backups until the replacement has passed your recovery checks.

Roll back a cutover

Rollback restores a coordinated fleet state, not just the old hub's database. Stop the new hub and affected Forge services first. Preserve current state and any work created since the cutover, then restore the old hub and affected members from their matching pre-cutover sets. Restore the service definitions and original ingress bindings along with state. Start the old hub, then its spokes, and verify their original identities, enrollment, and workspace access. Keep the replacement hub stopped while this restored fleet is authoritative.

Database rollback can discard changes made after the backup. Do not overwrite new worktree contents as part of restoring Forge state. Revoking a new enrollment alone does not restore the old enrollment or provider ownership.

Return a spoke to standalone operation

Keep the spoke running and connected until revocation finishes. An outbound-disabled spoke whose tunnel is down is revoked on the hub without a callback. On the hub, run:

kenn-forge fleet revoke ENROLLMENT_ID --config /path/to/hub/config.toml

Stop the spoke's Forge service, leaving its worktrees and tmux sessions in place. Retain a matching backup of its current data directory, configuration, binary, and service definition, as described in Move to a fresh hub. Then, on the stopped spoke, run:

kenn-forge fleet leave --config /path/to/spoke/config.toml

This disables federation and removes the old hub's credentials. It preserves the current node identity, local database, workspaces, and session records; repeating it after an interruption is supported. It does not restore an old database or copy the hub's provider archive, review drafts, or credentials.

Configure this machine's own provider access and repository list before starting Forge. Sync its repositories, then check its local workspaces and terminals. Provider state created on the hub stays there. Pending enrollments use fleet abort-preparation, not fleet leave.

After abort-preparation --force, finish pending revocation on the hub before disabling federation. leave must not remove the credential needed for that cleanup; an already-disabled standalone instance is left unchanged.

Upgrade federation protocol 3 to 4

A protocol-3 fleet needs a one-time data migration before protocol-4 binaries can use its enrollment. Keep a matching rollback backup of every host's database, enrollment and credential stores, configuration, and binary.

  1. Stop the Forge service on every fleet member. Keep its worktrees and tmux sessions in place.

  2. On each host, run the new binary against that host's existing configuration:

    /path/to/new/kenn-forge fleet migrate-protocol --config ~/.kenn/forge/config.toml
  3. Install the new binary, start the hub, then start its spokes. Verify the fleet workspace list and terminal access on each member.

The migration retains node identities, enrollment, credentials, preparation seals, local workspaces, and provider data. It does not enable communication between protocol-3 and protocol-4 peers. Repeating the command after an interruption resumes the migration; already migrated and unenrolled hosts need no changes. To roll back, stop the affected services and restore every host's matching pre-migration state and binary together.

Replace an older fleet

Older key-based and shell-relay entries are not accepted. Before starting the new binary:

  1. Remove the old fleet entries from every config file.
  2. Enable API authentication and select the hub.
  3. Start each daemon with a reachable HTTPS origin.
  4. Enroll every spoke again with a new one-time token.

Do not copy old tokens, node IDs, relay sockets, or daemon data directories. There is no automatic translation because the older entries did not establish the identity and credential pair required by federation.

Troubleshooting

Setup fails its HTTPS check

Read the setup error and any rollback errors before retrying. Setup attempts to restore its previous config, service, and managed publication when a later step fails. Confirm what is actually running; do not assume a failed setup left a usable service behind.

For Tailscale, check that the device is logged in, the Forge user can manage Serve, and HTTPS certificates are enabled. First-time certificate issuance can outlast Forge's readiness check. Inspect Tailscale's certificate and Serve status, resolve the reported problem, then rerun the setup plan. For external ingress, check the proxy route, preserved Host header, and certificate trust.

Agents work in SSH but not in Forge

Check the service's PATH and the service user's agent authentication. Shell startup files are not loaded by the generated service. For a service owned by setup, rerun the appropriate setup command with the intended stable tool paths; for a custom service, update its environment through your service manager. Restart and launch an agent from a workspace to verify the result.

A fresh hub is missing repositories or pull requests

Check Settings → Repositories on the new hub first. Enrollment does not import the old hub's repository selection. Then inspect sync status: a large first sync can consume the available request budget before all repositories finish. Let the budget recover and use Sync current repo for the repository you need, rather than repeatedly starting a full sync. Verify a real repository or pull-request read separately from fleet reachability.

A spoke is unreachable

A running spoke that can reach its hub keeps a tunnel open, and the hub uses it. First confirm that the spoke daemon is running and can reach the hub.

Without a tunnel, the hub dials the spoke directly. From the hub machine, verify the spoke's HTTPS origin and certificate. Confirm the URL exactly matches the enrolled origin. A changed hostname, port, or certificate route requires deliberate re-enrollment rather than a redirect.

If the daemon is down, restart it through its local service manager. The hub does not log in to start it.

A request is forbidden

Read the full degraded-host message in the hub. Forge preserves the peer's authorization reason instead of reducing it to an HTTP status.

If the message says fleet authorization is inactive, inspect the spoke service's startup log and confirm it can resolve and reach the hub. An active spoke revalidates its sealed enrollment with the hub at startup and every six hours. If the hub is temporarily unavailable, the spoke can use its last confirmed activation lease for up to 24 hours. Once that lease expires, hub requests are rejected until the spoke reaches the hub and renews it.

The lease handshake is versioned separately from fleet snapshots. During a rolling upgrade, upgrade the hub first, then each spoke. A lease-unaware spoke is isolated from fleet reads and writes until it is upgraded and restarted; its existing enrollment is reused, so you do not need to re-enroll it. If a spoke was upgraded before its hub, upgrade the hub and restart that spoke so it can complete the lease handshake.

If the peer rejects the hub identity, check that the enrollment is active and has not been revoked. Re-enroll instead of widening or copying a token by hand. A valid credential can still be forbidden when its direction does not grant the requested operation.

A terminal does not open

Confirm the spoke is reachable and the local session still exists. The hub attaches to the spoke's terminal WebSocket; it does not attach to the spoke's tmux server directly.

A config no longer loads

Remove old fleet keys and relay entries, then use the enrollment workflow or declare membership in config. Forge fails closed instead of guessing how an old host label maps to a stable spoke identity.