Repository navigation
Conversation
[Critical impact] The PR establishes a lab service that controls access to Macs and runs work on them. This PR is not ready to merge while the exposed notice topic and the existing command, freeze, reset, and setup failures remain unresolved. Findings
SummaryThis PR brings the lab server, public reports, sign-in, Mac jobs, releases, and mini operations into one repository.
|
| case l != nil && leaseIsOf(l, p) && l.Extended: | ||
| return errNotSent{refuse(409, "%s, so this command was not sent", extendedOnly(l, name, fenceExec.what))} | ||
| case l == nil || !leaseIsOf(l, p) || session.Context().Err() != nil: |
There was a problem hiding this comment.
Expired leases accept commands
stillLeased checks Extended, not the lease's expiry time. After Expires passes, liveLeaseLocked still returns the lease while its job runs. Until the once-per-second keeper sets Extended, a new lab exec passes both checks and is queued. Cutting its response on the next tick does not stop that command on the Mac.
Check the expiry time when allowing new work, even before the keeper records the extension.
…t lands meanwhile holds
…ner" ended it, never nothing, so it still leaves
…tting it short at that line
…e it cannot sweep, so it can be done again
…d every name kept beside their account id
…pt without the handle
…lment ended or a Lease cut, since a forgetting never reaches it
…, so an ended Mac is never unknown with nothing to tell it why
… done, and a Forget then changes nothing
…e making them, so a "no" leaves the lab as it was
…its Lease before it takes them
… when the hold ran out or broke
…t starts, so a run cut half way leaves no old pass
… is never in the process list
…in, makes a Notice for the redirect, and names the topic to put back however it ends
…ed, and proves with a Mac of no topic of its own that a Notice does go out
…ts processes read, so an empty list is never a pass
…rwise skip unread
…ommand's token from what the lab answered
… that never answers does not hold a test
…percentile without every page, and the gate counts such a reading as a missed target
…en the run ends before it, failed or stopped
…ile lab hashes the file before sending it
…e that fails its checksum is no build failure to the gate
… too, following 3999676: the constant it read is gone, so the page parity test stopped on a NameError, and the recording it is compared with carries no note
…s port counts plugs at every boot (the M2 Max's went from 21 to 25 across a job), so labd keeps a Sighting whose plug count rose while the Mac comes back on a new boot or stays silent, ends it once the Mac is heard again on the boot it had, and sends no Hard reset meanwhile while it is up; before, every job ended it and a Mac placed by a Proving reset had no Hard reset after its first job
| case portRestarted: | ||
| sg.seen.PlugCount, sg.boot, sg.rose = p.PlugCount, m.boot, time.Time{} | ||
| s.cables.keep(name, sg) |
There was a problem hiding this comment.
hardReset can reset Mac B when someone asks to reset Mac A. If A reboots and its cable is swapped between port checks, against treats the higher plug count as A's restart. This happens even when the port explicitly shows B's name: hardReset ignores that name and sends B's current count to the controller, so its final check passes. B then restarts, interrupting its work.
Reject a conflicting port name before accepting the count change or sending the reset.
…6, 7, 9, 11, 13 and 14 pass, 3 fails for want of an installed Release on m2-max
| -- put back: topic https://ntfy.sh/maclab-53689934430104571482000252004488, job notices off | ||
| Notices for this Mac now go to https://ntfy.sh/maclab-53689934430104571482000252004488 once it next checks in with labd. |
There was a problem hiding this comment.
This record publishes the Mac's real ntfy topic and says it was restored after the test. The topic name is its only secret, so anyone reading the repository can read recent notices and send fake ones to its subscribers. Replace the topic with a placeholder and rotate the exposed topic; removing it from this file alone leaves it in Git history.
How this was verified: The record restores the disclosed topic, and the notice sender posts to it without a separate credential.
…backup guest and the Dead-man switch firing and clearing, on 2026-11-16
…ook at the ports a restarted labd keeps each one whose port is still connected with the same product id and logs it (sighting kept across the restart: m2-max on middle), and the usual rules go on from there; before, every Deploy left a Mac placed by a Proving reset with no Hard reset until a Maintainer proved its port again
…r on 2026-11-16, seen in the Lab group
…d to docs/acceptance, not that it holds none: the records of the real runs are committed there now, and since 57be751 every push failed on them
…ing machine has too, such as the bridge Apple's container gives every Mac: it reached the asker itself, and on 2026-11-16 its own 7780 failed the test
…in, where lab-builder has put a build's input since 4338ad1; it still looked in /io, so every kernel build failed after compiling with module-build: No such file or directory: '/io/config'
…me limit on git ls-remote ends git's https helper too, which a stalled GitHub had kept waiting 126 s after git was killed, and asking for a build has 80 s in all, a Mac's config included
… a request with a Lab token or the session cookie from the edge cache, as FD 13 found through the real edge on 2026-11-16
…ook finds it, still connected with the same product id, so it holds across a restart of the mini too, whose plug counts start again: before, a mini restart that brought a port back to its old count kept it and one that did not dropped it, and OOB 7 failed by turns; OOB 7 now checks every Sighting is kept across labd alone, an Unlock and a planned restart
…so the close of a 1 GiB upload takes about 1 s on the mini instead of 5 s: each sync there is a full flush of the drive, and FD 16's upload spent a tenth of its time on them
…oy whose builder/ changed has the set-up build the new images and start the Builder with them, keeping the last ones as the way back, where before a mini kept running an image older than the Deployed build until it was deleted by hand
… with the images it already names, as it did before the images were tagged
| cd "$build/$dir" || exit 1 | ||
| find . -type f ! -path './pkg/*' | LC_ALL=C sort | while IFS= read -r file; do | ||
| echo "$(shasum -a 256 < "$file" | cut -c1-64) $file" | ||
| done | ||
| [[ -z $rootfs ]] || echo "$(shasum -a 256 < "$rootfs" | cut -c1-64) $BUILDER_ROOTFS" |
There was a problem hiding this comment.
A relative --builder-rootfs path loses its archive hash. setup accepts and copies the file from the caller's directory, but builder_image changes into builder/pkg before reading that same path. The failed read is hidden by echo, so different archives produce the same tag. A later run can copy a new archive and still reuse the package image built from the old one.
Hash the copied archive by its absolute path, or resolve the supplied path before changing directories.
| change "remove the Builder's image $old, which the set-up made before the one kept as the way back" forget_image "$old" | ||
| done <<<"$(older_images "$tag")" | ||
| done | ||
| if rendered "$BUILDER_LABEL" "$user" "$cpus" "$image" "$pkg_image"; then BUILDER_AGAIN=1; fi |
There was a problem hiding this comment.
Image updates cancel running builds
Updating an image now restarts lab-builder without first stopping new work or waiting for its current build. A Deploy allows builds again before telling the owner to run setup, and building the new images can take a while. If a build starts during that time, the final start_rendered stops its Builder and cancels its VM.
Keep builds drained through the image switch and restart.
| tag="$(builder_image "$name" "$dir" "$LAB_CURRENT" "${builder_rootfs:-$rootfs}")" | ||
| [[ -n $tag ]] || fail "the Builder's image $name could not be named from $LAB_CURRENT/$dir" | ||
| if [[ $option == --image ]]; then image="$tag"; else pkg_image="$tag"; fi | ||
| "$watch" as "$user" image inspect "$tag" >/dev/null 2>&1 && continue |
There was a problem hiding this comment.
Pruning keeps the last image built, not the image being replaced. After building images A then B, rolling back to A reuses it without changing the history. Building C then deletes A and keeps B—the image just rolled back from—as the way back. Going back to the working image then requires rebuilding it.
Record successful image switches, including reused images, and preserve the image being replaced before pruning.
…ab/packages and each package build logs its gcc, glibc and linux-api-headers, since build b1116-183947-7af0 failed on Arch Linux ARM's linux-api-headers 7.3 meeting linux-aurora's 7.1 tree, not on a partial upgrade
…old kernel build, a warm rebuild and a package build, labd's pages at 1 ms
…a bad signature, and an older one, are refused
…xd_device, no longer ends, moves or places a Sighting, so a Mac hung there still gets its Hard reset
…l runs check that they add no record to docs/acceptance, not that it holds none, as 6b56a89 did for hardening and out-of-band: the real runs' records are committed there now
…ch build already carries as 3999676
Summary
This branch builds Omarchy Mac Lab from the build plan, one slice at a time, on top of
main(the public history of maclab and omarchy-m-testing). It holds slices 01 to 57 except 50: one Go server (labd) with two stores, the report pages ported from the Rails site, sign-in, Leases, jobs and builds, Volunteers' Macs, the mini's scripts and the cutover's tests. All of it is tested locally against stand-ins; none of it has met the mini, real Macs, Cloudflare or GitHub's runners yet.What is on the branch
Repository and CI
mainby a repeatable script; module pathgithub.com/omacom/omarchy-maclab.tests.yml,scripts/ci-local.release.yml; every file signed.Server and stores
labd.dbandreports.db, WAL with full sync, strict numbered migrations.lab execand freeze entry through the tunnel; capped job waits.private, no-store.Sign-in and people
/termsand/privacy: one header and footer by role, both texts word for word.lab_tokens with expiry,/me, Sign out everywhere;admin.tokenretired.Reports and public pages
/install, the parity gate: Hide, Show, Delete;/install;scripts/parity-gate. Partial:site/stays until the gate is green.labd import dump|load|checkand its runner.The CLI:
lab,omarchy-m-test,lab-agentlab: login and identity: one saved login, the token never an argument, every call names itself.omarchy-m-test0.2.0: the Lab address,--upload, Verified wording, the report kept while Dark.lab-agent setupat the Mac: pre-checks that change nothing, then the set-up.lab-agentfrom labd;lab-agent updatechecks the pinned key.labandlab mcp: the sweep, and the guides: commands match the spec's table; one MCP tool per command.Leases, jobs, builds, exec
/jobs, Cancel and Retry,/boot.lab exec: your own Lease first, the Exec record, the serial console off.oobd, status and reset per port, one lock.Macs joining and Volunteers
The mini: set-up, deploy, backups, alerts
scripts/mini/setupanddeploy(tags only, signature checked).--check,--apply).tailscaled, keys-only SSH, updates, power./status, operations alerts and the weekly digest: labd watches its parts and tells "Agents".Cutover and front-door tests
day,back,confirmed,ready.Not on the branch
main, with every gate test run again.How it is tested
scripts/ci-localruns whattests.ymlruns:go build,go vet(also for linux/arm64),go test ./...,make all,omarchy-m-testagainst a reallabd, the labd smoke, and the Python suites forcli/(3.11 and 3.13) andbuilder/, plus installer, catalogue and schema checks. On macOS seven steps skip (Linux-only tools, TLA+, GPU helper, Rails reference); the rest pass.tools/fake-github, fake Cloudflare, Porkbun and old site,lab-agent sim, a fake controller, a made-up mini.TestRebootFailingAfterTheAckIsAnInfraErrorcan fail under load.deploy/mini.conf(seedeploy/mini.conf.example).tests.ymlruns on GitHub for the first time on this pull request.release.ymlruns only on the owner'sv*tags.Not proved here
-race, the GPU helper and the arm64 release build have never run.lab-agent; how the plug count moves across a Hard reset is unknown.brew,container,cloudflared, restic, Litestream andsignal-clinever ran there.docs/acceptance/are the owner's.?access_token=is dropped, not refused (H 5); the serial console stream can go silent through the tunnel; the Python tools read a login filelabno longer writes;lab-agent uninstallleaves the program file.The owner's part
mainprotection, thev*ruleset, thereleaseenvironment, the OAuth apps.sudorun, Deploys, restarts with Unlock, the Builder,oobd.lab-agent setuprun on both test Macs, then slice 50.v0.2.0-rctag, slice 58, deletingsite/once the gate is green, and the day.