Repository navigation
feat: build and boot what the tool generates for every base, two a night - #759
Conversation
Nothing checked that the projects the tool writes still build into images that boot. The bases change upstream daily, so a breakage was first found by a user (#756). variant_boot_test.py generates a project the way "Test build locally" does (seed_project_template, write_project_files) with a fixed test configuration (tmux, podman.socket), builds it with podman, and boots it with bcvk ephemeral. The guest reports whether boot settled, whether the package is installed and the service enabled and active, and which units failed; units that fail only in an ephemeral VM are allowlisted with their reason. A failed boot is retried once, then the unmodified base is booted the same way to say whose failure it is, never to pass one. variant-boot-test.yml runs two bases a night on ubuntu-26.04 (all 13 a week; Sunday's second slot builds one base with the BlueBuild method) and keeps one issue per failing base, closed only when that base passes again. It is exempt from auto-issues.yml, whose per-workflow issue would be closed by the next night's unrelated bases. Two things bcvk needs from a desktop base are injected into every boot, generated and unmodified alike: /var/roothome before tmpfiles runs, so the SSH key credential lands, and sshd.service, since the desktops ship it disabled. Builds use --no-cache: podman keyed the build.sh RUN step on its command line, not the bind-mounted script, and reused a stale layer. Closes #756
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
danathar-atomic-hive
Bot
added
author/trusted
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 220e06f43c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
220e06f
|
danathar-atomic-hive
Bot
added
sentinel-alert
and removed
author/trusted
…gnature, ship the check script as a file A slot could run 90 minutes of build and three 15-minute boots, longer than the job's 120-minute timeout, so the slowest failures were cancelled before their logs and issue were written. Build and boot limits are now 45 and 12 minutes, and a test holds the job timeout above the worst case. Two boot failures were blamed on upstream whenever both were 'boot', even when one never answered SSH and the other panicked. They now have to match. The guest script went over SSH base64-encoded, which hid what ran. It is now a file on a read-only virtiofs mount, kept in the run's artifact.
The hosted runs failed before the VM started (virtiofsd exited), and both slots were labelled upstream because an unmodified base that also never starts "failed the same way". A boot with no guest journal is now stage "host": no upstream comparison, and, like a job that died in setup, it goes to one shared runner issue instead of opening an issue against each base. Any passing slot closes that issue. Also: the issue's log carries the guest console beside bcvk's output, run refuses a --work directory that already holds a project instead of deleting it, and the mcelog allowlist reason says only what was observed.
…nd run slots one at a time An empty guest journal alone also fits a guest that died before its journal reached the host, which would have filed a base's failure under the runner issue. A failure is now the runner's only when bcvk's output names a host-side error (virtiofsd, QEMU or the container failing to start) and the guest wrote nothing; a silent boot stays the base's. The matrix runs one slot at a time: both slots can write the shared runner issue, and find-then-create is not atomic, so two failing together could open it twice or one's pass could close it under the other's failure.
… feat/756-variant-boot-test
…rs past terminal queries The hosted scratch run with bcvk's own CI host setup (bootc-dev/actions bootc-host-setup at a pinned commit) booted both bases: silverblue passed every check, and bazzite booted and passed its package and service checks. The workflow now runs that setup first. bazzite still came out red for two reasons fixed here: - its login shell prints terminal colour and cursor queries on the same line as the first marker, so the systemd state was never read and the boot was reported as never settling. Markers are now found anywhere in a line. - greenboot-healthcheck.service and bazzite-hardware-setup.service fail on every ephemeral boot because rpm-ostree refuses a system not booted via libostree. Both are allowlisted with that reason. Replaying both runs' captured output through the new parser and allowlist gives a pass for every boot.
|
Hosted runs. A scratch branch (now deleted) ran this workflow on
Also since the first push:
|
…on command A workstation run cannot reproduce a runner that failed to set up or to start bcvk, so the shared runner issue names the setup steps to read instead of the per-base reproduce command.
danathar-atomic-hive
Bot
added
the
author/trusted
🔒 Hold-gate integrity: branch changed while hold-gated — fresh review requiredThis PR's branch moved while it sat hold-gated: the head recorded when the hold was applied was Commits not present in the hold-time snapshot (8):
Authors not in the hold-time author set: A human should review the FULL diff at the current head — a rebase or force-push renames every commit, so everything above needs eyes even if it looks familiar. Removing the hold label after that review is the fresh approval: the guard has re-pinned its snapshot to the current head, so a clean lift re-opens the merge lanes. |
danathar-atomic-hive
Bot
added
hold
and removed
author/trusted
48543ae
|
Two paths still turned a runner failure into a verdict about an image:
- When bcvk never started the unmodified base's VM, a generated image that
failed to boot was blamed on the tool ("does not fail this way"), though
the base was never compared. That comparison now reports Unknown.
- When the first boot reached a real failure and the retry hit a host
failure, the retry was reported: the base's failure went to the shared
runner issue, which the next passing slot closes. The first attempt's
failure now stands.
The runbook also says how to re-check a "(BlueBuild)" issue: only a
BlueBuild pass of that base closes it, and the rotation reaches a base
that way once in as many weeks as there are bases.
9bc571e
|
danathar-atomic-hive
Bot
added
author/clanker
The boot job holds issues: write and then runs bootc-dev/actions' host setup scripts. actions/checkout persists its token for later git commands by default, and nothing in this job runs git with credentials: only the report step needs a token, and it gets one through GH_TOKEN. Signed-off-by: Douglas Baggett <doug.baggett@gmail.com>
Nothing checked that the projects the tool generates still build into images that boot, so an upstream change that broke them was first found by a user. This adds a nightly job that generates a project for two bases, builds it, boots it in a VM, and keeps one issue per failing base. All 13 bases are covered each week.
What changed
variant_boot_test.py:seed_project_template, thenwrite_project_files), withtmuxandpodman.socketas the test package and service.podman build --no-cacheand boots it withbcvk ephemeral run-ssh.variant-boot-test.yml: runs the rotation from Nothing checks that the images the tool generates still build and boot on each supported base #756 onubuntu-26.04at 02:41 UTC, plusworkflow_dispatchfor one base. Issues are opened, commented on or closed only frommain..github/actionlint.yamldeclaresubuntu-26.04, because actionlint v1.7.12 predates that label.Why it is shaped this way
auto-issues.yml: that workflow's per-workflow issue would be closed by the next night's unrelated bases. The exemption is explicit intests/test_auto_issues_workflow.py.sshd.servicedisabled, and an ephemeral/varlacks/var/roothome, so bcvk never got SSH. Both are injected into the generated image and the unmodified base alike, so they cannot be the difference between them. For the same reason,sshdis not the test service.--no-cache: podman reused a stalebuild.shlayer locally, and that booted an image without the test service.Closes #756
How it was checked
python3 -m unittest discover -s tests(2574 tests),ruff check,actionlint,python3 maintenance_audit.py --skip-upstream,git diff --check.run --base sway-atomicpassed in 84 s with no cache hits. An earlier stale-cache run exercised the failure path: it failed the service check, re-booted, booted upstream, and blamed the tool.test/756-hosted, which adds a push trigger and is not for merge) for bazzite and silverblue. Results follow in a comment.