Skip to content

chore(release): staged fleet rollout of the epic #98 release without disruption #115

Closed

Description

@entelecheia

Context

Epic #98 changes many command and module paths. The owner requires that updating the managed fleet from v0.18.1 to the next release causes no disruption. This issue is the release and rollout procedure. The release itself and every fleet mutation still need the owner's explicit go-ahead (release guard in AGENTS.md).

Facts from the 2026-11-10 review (verified against main 79f1dc5 and the open PRs #101, #105, #114):

Decisions

  • Merge all epic PRs first and cut one release. No partial fleet rollout of an intermediate version.
  • Canary first, then one host at a time. Rocky hosts and the tenant host get their own steps.
  • Stop condition: any host where fleet check after fleet update is not clean, or where doctor gets worse than before. Investigate before continuing; never fleet apply to make an unexpected drift go away.
  • The operator Mac's controller binary is updated to the same release before the fleet steps.

Design

  1. Release candidate: all epic PRs merged, CI green, release notes compiled from the PR bodies (### Action needed must be empty for fleet hosts, or each item must be handled below).
  2. Pre-flight (read-only): fleet status --all, fleet check --all, fleet doctor --all on the current release; save the JSON as the baseline.
  3. Controller: update the Mac binary; re-run fleet status --all to confirm the new controller reads the old hosts.
  4. Canary (one GPU host with monitoring and no tenants): fleet update --host <canary> --version vX --timeout 25m, then fleet check and fleet doctor on it. Expected: clean, no apply-pending. Only if a planned change shows, review it, then fleet apply --host <canary> --dry-run before a real apply.
  5. Remaining Ubuntu GPU and studio hosts, one at a time with the same check.
  6. Rocky hosts, one at a time (different package, SSH Include and SELinux paths).
  7. Tenant host last: also confirm running GPU containers still see their GPUs (nvidia-smi -L inside each) after the update.
  8. Post: fleet check --all and fleet doctor --all clean; record versions and backup IDs in the private fleet record.

Changes

  • No code. Release notes docs/releases/vX.Y.Z.md in the release PR.

Acceptance criteria

  • AC1: every epic chore: 2026-11-09 audit plan (commands, modules, docs, specs) #98 PR is merged and CI is green on the release commit.
  • AC2: canary shows clean check and unchanged-or-better doctor after fleet update, with no fleet apply needed.
  • AC3: all hosts updated; fleet check --all and fleet doctor --all clean afterwards.
  • AC4: no service restart caused by the update other than those listed in the release notes (sshd, fail2ban, journald, cloudflared, docker untouched on converged hosts).

Tasks

  • T1: release PR with notes (owner authorizes the tag)
  • T2: baseline JSON (read-only)
  • T3: controller update
  • T4: canary update and verification
  • T5: serial rollout (Ubuntu, then Rocky, then tenant host)
  • T6: final verification and record

Out of scope

  • Changing host site configs. Hosts keep their current configs.

Activity

  1. entelecheia commented on Nov 10, 2026

    @entelecheia
    MemberAuthor

    v0.19.0 is released (82d8f2d, release run green; checksum and provenance attestation verified) and rolled out to all 11 managed hosts with no fleet apply.

    • AC1: every epic chore: 2026-11-09 audit plan (commands, modules, docs, specs) #98 PR merged; CI green on the release commit.
    • AC2: the canary showed check satisfied, the applied-config digest unchanged, doctor with 0 failures and the same findings as the v0.18.1 baseline.
    • AC3: serial rollout, in order: canary, 7 Ubuntu hosts, 2 Rocky hosts, then the tenant host. Each host had to match the baseline before the next one started. The final fleet check --all and fleet doctor --all match the baseline on every host.
    • AC4: no restart of sshd, fail2ban, journald, Docker, containerd or the exporter firewall on any host. On the tenant host, every container kept its start time and GPU count, and a hand-set cpuset stayed in place.

    Release authorization: the agent-hub guard cannot read a version from a repo without package.json (tracked in STAIxBWLB/agent-hub#1381), so the owner pushed the tag directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions