Skip to content

Omarchy Mac Lab, built from the build plan - #1

Merged
maralcbr merged 676 commits into
mainfrom
build
Nov 16, 2026
Merged

maralcbr merged 676 commits into
mainfrom
build

Conversation

@maralcbr

Copy link
Copy Markdown
Member

Summary

This branch builds Omarchy Mac Lab from the build plan, one slice at a time, on top of main (the public history of maclab and omarchy-m-testing). It holds slices 01 to 57 except 50: one Go server (labd) with two stores, the report pages ported from the Rails site, sign-in, Leases, jobs and builds, Volunteers' Macs, the mini's scripts and the cutover's tests. All of it is tested locally against stand-ins; none of it has met the mini, real Macs, Cloudflare or GitHub's runners yet.

What is on the branch

Repository and CI

  • 01 The repository with both histories: both histories joined on main by a repeatable script; module path github.com/omacom/omarchy-maclab.
  • 02 Layout, licence and green CI: the spec's layout, container files gone, both copyright lines, tests.yml, scripts/ci-local.
  • 03 Release workflow and the first Release candidate: build, sign, verify and check scripts and release.yml; every file signed.

Server and stores

  • 05 Listeners, host check, caller log and the Preview switch: loopback port and tunnel socket; only the socket trusts Cloudflare's headers; the Alias redirects.
  • 07 Two stores: labd.db and reports.db, WAL with full sync, strict numbered migrations.
  • 26 Held responses: Keepalives for lab exec and freeze entry through the tunnel; capped job waits.
  • 27 Uploads in parts and resumable downloads: 32 MiB parts, SHA-256 checked on close, resume.
  • 43 Cache policy and the refused-credential limiter: public answers cached at the edge, session answers private, no-store.

Sign-in and people

  • 08 People and browser sign-in: GitHub sign-in, People by account id, hashed sessions, CSRF checks.
  • 09 Interface shell, /terms and /privacy: one header and footer by role, both texts word for word.
  • 10 Lab tokens, the three cuts and break-glass: hashed lab_ tokens with expiry, /me, Sign out everywhere; admin.token retired.
  • 12 Roles on People: M team, leave to lend, Maintainer, Blocked with a reason.
  • 15 Mac sign-in and Verified runs: device-flow sign-in; Verified only for an M team member's Mac.
  • 41 Forgetting, and Blocked in full: ask to be forgotten, Forget on People, erasure across both stores.

Reports and public pages

  • 13 Report rules: the Rails upload's parsing, schema, signature and machine id rules, ported.
  • 14 Upload pipeline and deletion links: the old order, statuses and words; rate limits; deletion links.
  • 16 Public pages I: home, Matrix, Models, Features, with the old ids and classes.
  • 17 Public pages II: Kernel gaps, Aurora, Benchmarks, Candidates, Reports and the exports at the old paths.
  • 18 Reports for Maintainers, /install, the parity gate: Hide, Show, Delete; /install; scripts/parity-gate. Partial: site/ stays until the gate is green.
  • 19 The import of the old site's reports: labd import dump|load|check and its runner.
  • 32 Publishing a lab run: a Known-good lab run published through the same rules, signed by the Mac.
  • 34 The Releases page and the verdict on Candidates: coverage, history, verdict form.

The CLI: lab, omarchy-m-test, lab-agent

  • 11 lab: login and identity: one saved login, the token never an argument, every call names itself.
  • 20 omarchy-m-test 0.2.0: the Lab address, --upload, Verified wording, the report kept while Dark.
  • 36 lab-agent setup at the Mac: pre-checks that change nothing, then the set-up.
  • 39 The lab's own software on a Mac: signed lab-agent from labd; lab-agent update checks the pinned key.
  • 42 lab and lab mcp: the sweep, and the guides: commands match the spec's table; one MCP tool per command.

Leases, jobs, builds, exec

  • 22 Leases I: who takes a Lease, on which Macs, for how long, one per Configuration.
  • 23 Leases II: the Lease record, what a Lease fences, what each ending cancels.
  • 24 A Mac's page and the live views: Overview, Live (screen, logs) and Records.
  • 25 Jobs, Builds and "Boot a kernel": /jobs, Cancel and Retry, /boot.
  • 28 lab exec: your own Lease first, the Exec record, the serial console off.
  • 29 The Out-of-band controller on the mini: oobd, status and reset per port, one lock.
  • 30 Sighting, Proving reset and the recovery ladder: Macs matched to ports; Hard reset only when sighted.
  • 31 The Builder: one credential per builder, one build at a time, source rules.
  • 33 Releases, verdicts and Promotion: Releases from the freeze, four verdict states, Promotion.

Macs joining and Volunteers

  • 35 Leave to lend and the Join command: "Lend a Mac", requests, a cap of 5, a one-time Join command.
  • 37 A Volunteer's Mac: arrives closed, first baseline queued, Pause, Limit, Leave.
  • 38 My Macs: a card per Mac with state, window and controls.
  • 40 Notices and a change of terms: one way to make a Notice, five wordings linking to My Macs, the lab-wide ntfy topic gone.

The mini: set-up, deploy, backups, alerts

  • 04 labd under launchd on the mini: scripts/mini/setup and deploy (tags only, signature checked).
  • 06 The tunnel and the zone: zone, tunnel and record scripts (dry run, --check, --apply).
  • 21 The Signal sender, the Lab group and "Agents": every line recorded and sent at most once.
  • 44 The mini's set-up in full: the whole machine in one run: tailscaled, keys-only SSH, updates, power.
  • 45 Backups and the owner's copy: nightly snapshots, restic to two targets, Litestream.
  • 46 Restore drill and Dead-man switch: monthly restore and check from each target.
  • 47 /status, operations alerts and the weekly digest: labd watches its parts and tells "Agents".
  • 48 Deploy in full and the Maintenance run: drain, switch, check, and the way back by itself.

Cutover and front-door tests

  • 49 The old lab's data and the rehearsal import: the old lab's records upgraded with their ids; Macs Retired.
  • 51 Front door tests I: FD 1, 2, 6 to 13, and the runner of all fourteen.
  • 52 Front door tests II: FD 4, 5, 15, 16: resumed upload, outside transfers, performance.
  • 53 The Alias rehearsal and the Dark recordings: Alias and rehearsal-record scripts.
  • 54 Out-of-band wiring's eight tests: one script per test, stopping where a hand is needed.
  • 55 Hardening's fourteen tests: H 1 to 14 against a local lab.
  • 56 The Builder's gate test: builds at a set core count, judged on page times and volume sizes.
  • 57 The day's scripts and the opening switch: labd's opening, day, back, confirmed, ready.

Not on the branch

  • 50 The Lab-owned Macs Join and are accepted: not built; its checks run on the M1 Pro and M2 Max with the owner there.
  • 58 The Release candidate that passes Ready: comes after this merge: the last candidate, tagged on main, with every gate test run again.

How it is tested

  • scripts/ci-local runs what tests.yml runs: go build, go vet (also for linux/arm64), go test ./..., make all, omarchy-m-test against a real labd, the labd smoke, and the Python suites for cli/ (3.11 and 3.13) and builder/, plus installer, catalogue and schema checks. On macOS seven steps skip (Linux-only tools, TLA+, GPU helper, Rails reference); the rest pass.
  • Tests were written red first where they could fail first; many guards were checked by turning them off.
  • Outside services are stand-ins: tools/fake-github, fake Cloudflare, Porkbun and old site, lab-agent sim, a fake controller, a made-up mini.
  • Mini, cloud and cutover scripts each have a dry run, a local mode and a real mode; tests cover the first two.
  • Load flakes fixed, two of them real races in labd. One remains: TestRebootFailingAfterTheAckIsAnInfraError can fail under load.
  • Mini and account defaults are not in the repository: the scripts read the mini's address and the old lab's host from deploy/mini.conf (see deploy/mini.conf.example).
  • tests.yml runs on GitHub for the first time on this pull request. release.yml runs only on the owner's v* tags.

Not proved here

  • GitHub's runners: Linux, -race, the GPU helper and the arm64 release build have never run.
  • Rails parity: Rails has never judged the corpus or rendered a page; the gate's Rails part is "not judged".
  • Postgres: the import was proved from a dump file; no real Postgres was read.
  • Hardware: no reset sent, no real Mac ran lab-agent; how the plug count moves across a Hard reset is unknown.
  • The mini: nothing ran under launchd or as root; brew, container, cloudflared, restic, Litestream and signal-cli never ran there.
  • Outside services: Cloudflare, Porkbun, GitHub OAuth, Signal, ntfy and healthchecks.io met only stand-ins.
  • Gate tests: run locally only; their recorded runs in docs/acceptance/ are the owner's.
  • Browsers: some pages were checked by markup tests, not in a browser.
  • Known gaps: a source URL's ?access_token= is dropped, not refused (H 5); the serial console stream can go silent through the tunnel; the Python tools read a login file lab no longer writes; lab-agent uninstall leaves the program file.

The owner's part

  • GitHub: repository settings, main protection, the v* ruleset, the release environment, the OAuth apps.
  • Secrets: the release signing key, the machine-id key, OAuth secrets, backup and alert credentials.
  • Accounts: Cloudflare, Porkbun, a Signal number, healthchecks.io, R2, Tailscale, Bitwarden.
  • DNS: the tunnel and record, zone settings, DNSSEC, the Alias rehearsal, the old name's redirect.
  • The mini: the first set-up, every sudo run, Deploys, restarts with Unlock, the Builder, oobd.
  • Hardware: cables, the declined lab-agent setup run on both test Macs, then slice 50.
  • Gate runs: front door, out-of-band, hardening, the Builder's gate, the restore drill, the rehearsal import.
  • Look once: the Releases page, two privacy wording points, the deletion confirmation in three browsers.
  • After merge: the first v0.2.0-rc tag, slice 58, deleting site/ once the gate is green, and the day.

@greptile-apps

greptile-apps Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 0/5

[Critical impact] The PR establishes a lab service that controls access to Macs and runs work on them.

This PR is not ready to merge while the exposed notice topic and the existing command, freeze, reset, and setup failures remain unresolved.

Findings

  1. P1 Expired leases accept commands ▶
  2. P1 Forget breaks concurrent freezes ▶
  3. P1 Jobs can use wrong builds ▶
  4. P1 Setup skips the SSH change ▶
  5. P1 A local Pause can disappear ▶
  6. P1 The wrong Mac restarts ▶
  7. P1 Security The notice secret is exposed ▶
  8. P1 New archives can be ignored ▶
  9. P1 Image updates cancel running builds ▶
  10. P2 Operations messages can repeat ▶
  11. P2 Script usage text disappears ▶
  12. P2 Working fallback gets removed ▶

Summary

This PR brings the lab server, public reports, sign-in, Mac jobs, releases, and mini operations into one repository.

  • People and Mac agents can use one server to run lab work.
  • People can sign in, lend Macs, and reserve them for lab work.
  • The lab can publish reports and track Release verdicts.
  • Operators can set up the mini and deploy signed lab releases.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Browser[Browser] --> Labd[labd]
  CLI[lab and omarchy-m-test] --> Labd
  Labd --> LabDB[labd.db]
  Labd --> ReportsDB[reports.db]
  Labd --> Agent[lab-agent on Macs]
  Labd --> Builder[lab-builder]
  Labd --> OOB[oobd on the mini]
Loading

Reviews (73) · Last reviewed commit: "Merge main into build: the Asahi attribu..." · Reviewed by Greptile

Comment thread internal/server/forgetting.go Resolved
Comment thread internal/server/exec.go
Comment on lines +74 to +76
case l != nil && leaseIsOf(l, p) && l.Extended:
return errNotSent{refuse(409, "%s, so this command was not sent", extendedOnly(l, name, fenceExec.what))}
case l == nil || !leaseIsOf(l, p) || session.Context().Err() != nil:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Expired leases accept commands

stillLeased checks Extended, not the lease's expiry time. After Expires passes, liveLeaseLocked still returns the lease while its job runs. Until the once-per-second keeper sets Extended, a new lab exec passes both checks and is queued. Cutting its response on the next tick does not stop that command on the Mac.

Check the expiry time when allowing new work, even before the keeper records the extension.

Comment thread scripts/cutover/owner.sh Resolved
Comment thread internal/server/runner.go Resolved
…e making them, so a "no" leaves the lab as it was
…t starts, so a run cut half way leaves no old pass
…in, makes a Notice for the redirect, and names the topic to put back however it ends
…ed, and proves with a Mac of no topic of its own that a Notice does go out
…ts processes read, so an empty list is never a pass
…percentile without every page, and the gate counts such a reading as a missed target
…en the run ends before it, failed or stopped
…e that fails its checksum is no build failure to the gate
Comment thread internal/server/forgetting.go Resolved
… too, following 3999676: the constant it read is gone, so the page parity test stopped on a NameError, and the recording it is compared with carries no note
…s port counts plugs at every boot (the M2 Max's went from 21 to 25 across a job), so labd keeps a Sighting whose plug count rose while the Mac comes back on a new boot or stays silent, ends it once the Mac is heard again on the boot it had, and sends no Hard reset meanwhile while it is up; before, every job ended it and a Mac placed by a Proving reset had no Hard reset after its first job
Comment on lines +541 to +543
case portRestarted:
sg.seen.PlugCount, sg.boot, sg.rose = p.PlugCount, m.boot, time.Time{}
s.cables.keep(name, sg)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 The wrong Mac restarts

hardReset can reset Mac B when someone asks to reset Mac A. If A reboots and its cable is swapped between port checks, against treats the higher plug count as A's restart. This happens even when the port explicitly shows B's name: hardReset ignores that name and sends B's current count to the controller, so its final check passes. B then restarts, interrupting its work.

Reject a conflicting port name before accepting the count change or sending the reset.

…6, 7, 9, 11, 13 and 14 pass, 3 fails for want of an installed Release on m2-max
Comment on lines +17 to +18
-- put back: topic https://ntfy.sh/maclab-53689934430104571482000252004488, job notices off
Notices for this Mac now go to https://ntfy.sh/maclab-53689934430104571482000252004488 once it next checks in with labd.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 security The notice secret is exposed

This record publishes the Mac's real ntfy topic and says it was restored after the test. The topic name is its only secret, so anyone reading the repository can read recent notices and send fake ones to its subscribers. Replace the topic with a placeholder and rotate the exposed topic; removing it from this file alone leaves it in Git history.

How this was verified: The record restores the disclosed topic, and the notice sender posts to it without a separate credential.

…backup guest and the Dead-man switch firing and clearing, on 2026-11-16
…ook at the ports a restarted labd keeps each one whose port is still connected with the same product id and logs it (sighting kept across the restart: m2-max on middle), and the usual rules go on from there; before, every Deploy left a Mac placed by a Proving reset with no Hard reset until a Maintainer proved its port again
Comment thread internal/server/sighting.go Resolved
…d to docs/acceptance, not that it holds none: the records of the real runs are committed there now, and since 57be751 every push failed on them
…ing machine has too, such as the bridge Apple's container gives every Mac: it reached the asker itself, and on 2026-11-16 its own 7780 failed the test
…in, where lab-builder has put a build's input since 4338ad1; it still looked in /io, so every kernel build failed after compiling with module-build: No such file or directory: '/io/config'
…me limit on git ls-remote ends git's https helper too, which a stalled GitHub had kept waiting 126 s after git was killed, and asking for a build has 80 s in all, a Mac's config included
… a request with a Lab token or the session cookie from the edge cache, as FD 13 found through the real edge on 2026-11-16
…ook finds it, still connected with the same product id, so it holds across a restart of the mini too, whose plug counts start again: before, a mini restart that brought a port back to its old count kept it and one that did not dropped it, and OOB 7 failed by turns; OOB 7 now checks every Sighting is kept across labd alone, an Unlock and a planned restart
…so the close of a 1 GiB upload takes about 1 s on the mini instead of 5 s: each sync there is a full flush of the drive, and FD 16's upload spent a tenth of its time on them
…oy whose builder/ changed has the set-up build the new images and start the Builder with them, keeping the last ones as the way back, where before a mini kept running an image older than the Deployed build until it was deleted by hand
… with the images it already names, as it did before the images were tagged
Comment thread scripts/mini/lib.sh
Comment on lines +647 to +651
cd "$build/$dir" || exit 1
find . -type f ! -path './pkg/*' | LC_ALL=C sort | while IFS= read -r file; do
echo "$(shasum -a 256 < "$file" | cut -c1-64) $file"
done
[[ -z $rootfs ]] || echo "$(shasum -a 256 < "$rootfs" | cut -c1-64) $BUILDER_ROOTFS"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 New archives can be ignored

A relative --builder-rootfs path loses its archive hash. setup accepts and copies the file from the caller's directory, but builder_image changes into builder/pkg before reading that same path. The failed read is hidden by echo, so different archives produce the same tag. A later run can copy a new archive and still reuse the package image built from the old one.

Hash the copied archive by its absolute path, or resolve the supplied path before changing directories.

Comment thread scripts/mini/setup
change "remove the Builder's image $old, which the set-up made before the one kept as the way back" forget_image "$old"
done <<<"$(older_images "$tag")"
done
if rendered "$BUILDER_LABEL" "$user" "$cpus" "$image" "$pkg_image"; then BUILDER_AGAIN=1; fi

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Image updates cancel running builds

Updating an image now restarts lab-builder without first stopping new work or waiting for its current build. A Deploy allows builds again before telling the owner to run setup, and building the new images can take a while. If a build starts during that time, the final start_rendered stops its Builder and cancels its VM.

Keep builds drained through the image switch and restart.

Comment thread scripts/mini/setup
tag="$(builder_image "$name" "$dir" "$LAB_CURRENT" "${builder_rootfs:-$rootfs}")"
[[ -n $tag ]] || fail "the Builder's image $name could not be named from $LAB_CURRENT/$dir"
if [[ $option == --image ]]; then image="$tag"; else pkg_image="$tag"; fi
"$watch" as "$user" image inspect "$tag" >/dev/null 2>&1 && continue

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Working fallback gets removed

Pruning keeps the last image built, not the image being replaced. After building images A then B, rolling back to A reuses it without changing the history. Building C then deletes A and keeps B—the image just rolled back from—as the way back. Going back to the working image then requires rebuilding it.

Record successful image switches, including reused images, and preserve the image being replaced before pruning.

…ab/packages and each package build logs its gcc, glibc and linux-api-headers, since build b1116-183947-7af0 failed on Arch Linux ARM's linux-api-headers 7.3 meeting linux-aurora's 7.1 tree, not on a partial upgrade
…old kernel build, a warm rebuild and a package build, labd's pages at 1 ms
…a bad signature, and an older one, are refused
…xd_device, no longer ends, moves or places a Sighting, so a Mac hung there still gets its Hard reset
…l runs check that they add no record to docs/acceptance, not that it holds none, as 6b56a89 did for hardening and out-of-band: the real runs' records are committed there now
@maralcbr
maralcbr merged commit 837e1a8 into main Nov 16, 2026
25 checks passed
@maralcbr
maralcbr deleted the build branch November 16, 2026 12:28

This branch was successfully deployed

1 active deployment
release — 837e1a83 Deployed Nov 16, 2026 by maralcbr via Sign and publish #1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant