diff --git a/DESIGN.md b/DESIGN.md index 68a65b06..441c1940 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -494,7 +494,7 @@ The non-`profiles.toml` configs (`packages.toml`, `toolchain.toml`, `kernel.toml `sync-config` is an **add-only**, comment-preserving merge from `etc/sysforge/*.toml` into the live config dir (`$SYSFORGE_CONFIG_DIR` itself, else `/etc/sysforge`, or `--target DIR`): it injects keys, tables, and their leading comment blocks the live file is missing, and never overwrites a value the live file already sets (even if the shipped default changed). Arrays-of-tables (`[[package]]`) are user content and are left untouched. Bare keys are spliced before the first table header (TOML adjacency rule); a new table is spliced **after its nearest shipped-order predecessor that the live file actually has** — falling back through the shipped order, and to EOF only when there is no such predecessor — so the live file keeps shipped section order. Order matters even though TOML ignores it (`3.1.0-B9`): appending at EOF reorders the live file a little further with every sync, and the `.sfnew` adoption step below is a line-based diff with no move detection, which renders a merely relocated section as a deletion in one hunk and an addition far away in another — the reading that invites an operator to hand-merge away a section that was never gone. A run of consecutive new tables anchors on each other, staying in shipped order; insertion lands immediately after the anchor's own entry, before any standalone comment run tomlkit holds for the following table, so those comments stay attached to what they document. Pre-existing order drift is **not** rewritten — the tool never does a wholesale rewrite of a live file it did not lay out. The merge is **key-anchored**, so it can only carry comments that lead an active key it injects — pure documentation comments and *commented-out example* settings (`# interactive = true`) have no key to anchor to. When a shipped file gains such content the live file lacks, the shipped file is written verbatim beside the target as `.sfnew` (pacnew-style) for the operator to diff and adopt; a stale `.sfnew` is removed once the drift is resolved. Drift detection is line-level and value-aware: a commented example the live file has **already adopted** by uncommenting it into an *identical* active line (`# interactive = true` shipped vs `interactive = true` live) is not drift and does not spill a `.sfnew`, but a differing value or a trailing inline note still does. One drift class is **structural** rather than textual and blocks the merge outright: a section header that is active in the shipped file but only present *commented out* in the live file (a pre-`3.0.0-STD1` vintage still carrying `# [build]`). A commented header does not disable its section — it reassigns every key beneath it to the preceding table or to the top level, so the settings stay syntactically valid while being read from the wrong place. The add-only merge cannot flip an existing line from commented to active, and worse, it would see the table as absent and append a *second* copy holding the shipped defaults, superseding the operator's orphaned value on reparse. Header liveness is therefore compared in **both** directions on the pre-merge text (the comment-signature subtraction cannot see this, being one-directional — activating a header removes its commented form from shipped); on a hit the file reports status `needs merge`, the write is **skipped entirely** rather than merged into a structure the tool has misread, and the `.sfnew` companion is spilled pointing at `sysforge config merge`. A header commented out in the *shipped* file is an example block (`#[group.cosmic]`), not a live section, and is excluded. `tomlkit` is a **dev-only** dependency (ephemeral `uv run --no-sync --with tomlkit`), never added to `pyproject.toml`. `--dry-run` reports without writing. `bootstrap.toml` is excluded (per-host, no live counterpart). -**Adopting the `.sfnew` residue — `sysforge config merge`.** Hand-merging the `.sfnew` companions is the only remaining manual step, so `sysforge config merge` (verb `config-merge`, `config_cmd.py`) is a pacdiff-style interactive driver over them. It scans the resolved config dir (`--config-dir` override, else `paths.CONFIG_DIR`) for `*.sfnew` — plus pacman's own `*.pacnew`/`*.pacsave` for sysforge config files on a packaged install — and for each presents: `[v]iew` (a `difflib` unified diff through `$PAGER` via `maybe_pager`, prefixed by a **relocated-section banner** naming any header the diff shows as both removed and added — a header on both a `-` and a `+` line is present in both files, so the pair is a move, not a deletion; `difflib` has no move detection of its own), `[m]erge` (launch the resolved diff/merge tool with `live new`, then re-loop), `[s]kip`, `[r]emove` (drop the companion once the live file is satisfactory), `[o]verwrite` (copy the companion over the live file verbatim — guarded by a confirm and a "discards your local values" warning, never the default, intended for the `.pacnew` accept-maintainer case), and a[b]ort. Because a `.sfnew` is the *verbatim shipped file*, there is **no blind "accept theirs"** as a primary action — the safe path is merge-then-remove. The verb edits config files in place but never builds/installs, so it carries **no sentinel**. The diff/merge tool resolves through one home shared with the reconfigure editor chain — `primitives/editor.resolve_merge_tool` (`SYSFORGE_MERGE` > `sysforge.toml [ui].merge` > `$DIFFPROG` > `vimdiff`) launched via `run_tty_argv` (the `/dev/tty` passthrough, also used by `resolve_editor`'s callers, which additionally runs the child inside a `"child"` `yield_terminal` so an active progress bar's reserved bottom row is released before a full-screen editor takes the terminal). `--dry-run` reports the companion→target pairs without prompting (scripting/CI). +**Adopting the `.sfnew` residue — `sysforge config merge`.** Hand-merging the `.sfnew` companions is the only remaining manual step, so `sysforge config merge` (verb `config-merge`, `config_cmd.py`) is a pacdiff-style interactive driver over them. It scans the resolved config dir (`--config-dir` override, else `paths.CONFIG_DIR`) for `*.sfnew` — plus pacman's own `*.pacnew`/`*.pacsave` for sysforge config files on a packaged install — and for each presents: `[v]iew` (a `difflib` unified diff through `$PAGER` via `maybe_pager`, prefixed by a **relocated-section banner** naming any header the diff shows as both removed and added — a header on both a `-` and a `+` line is present in both files, so the pair is a move, not a deletion; `difflib` has no move detection of its own), `[m]erge` (launch the resolved diff/merge tool with `live new`, then re-loop), `[s]kip`, `[r]emove` (drop the companion once the live file is satisfactory), `[o]verwrite` (copy the companion over the live file verbatim — guarded by a confirm and a "discards your local values" warning, never the default, intended for the `.pacnew` accept-maintainer case), and a[b]ort. Because a `.sfnew` is the *verbatim shipped file*, there is **no blind "accept theirs"** as a primary action — the safe path is merge-then-remove. The verb edits config files in place but never builds/installs, so it carries **no sentinel**. The diff/merge tool resolves through one home shared with the reconfigure editor chain — `primitives/editor.resolve_merge_tool` (`SYSFORGE_MERGE` > `sysforge.toml [ui].merge` > `$DIFFPROG` > `vimdiff`) launched via `run_tty_argv` (the `/dev/tty` passthrough, also used by `resolve_editor`'s callers, which additionally runs the child inside a `"child"` `yield_terminal` so an active progress bar's reserved bottom row is released before a full-screen editor takes the terminal). `--dry-run` reports the companion→target pairs without prompting (scripting/CI). The live dir is normally root's `/etc/sysforge`, so the two writes the verb makes itself escalate (`4.0.0-B3`): `[r]emove` unlinks directly and falls back to `run_privileged(["rm", "-f", …])`, and `[o]verwrite` goes through `atomic_write.replace_file` (keeping the live file's mode and owner) before that removal. The merge tool runs as the user; saving into a root-owned file is the tool's own business (`sudoedit`-style), as with `pacdiff`. `tools/e2e/scenarios/75-config-merge.sh` runs the whole loop against `/etc/sysforge`. ### Checking and repairing a live config diff --git a/ROADMAP.md b/ROADMAP.md index 274776c5..9500e9cf 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -112,10 +112,7 @@ canonical ordering. | ID | Item | Priority | Effort | Bump | |----|------|----------|--------|------| -| `3.5.0-DEV25` | an end-to-end scenario for the toolchain → kernel handoff | med | small | patch | -| `3.5.0-DEV28` | an end-to-end scenario for a stock package installed over a sysforge build | med | small | patch | -| `3.5.0-DEV30` | an end-to-end scenario for config merge | med | small | patch | -| `3.5.0-DEV31` | an end-to-end scenario for build --interactive and update --interactive | med | small | patch | +| `4.0.0-B5` | a real drop-in in a batch auto-confirms every other conflict in it | med | small | patch | | `3.5.0-DEV29` | the build sandbox end to end, in the VM tier | med | large | patch | | `3.5.0-DEV33` | an end-to-end scenario for unattended runs | low | small | patch | | `3.5.0-DEV34` | an end-to-end scenario for the --frozen and --thaw flags | low | small | patch | @@ -156,28 +153,33 @@ canonical ordering. ### Bugs -### Developer tooling - -- **`3.5.0-DEV25` — an end-to-end scenario for the toolchain → kernel handoff.** `95-stage-dry-run` - previews `run toolchain` and `run kernel` separately, so nothing checks what one leaves for the - other: the compiler, its paths and its variant in `pipeline_state.toml`, which the kernel stage - (and `packages`) read. A scenario under `tools/e2e/scenarios/` runs a real `run toolchain` with - `toolchain.toml compiler = "llvm"` and `skip_build = true` (it registers the container's clang - without building, in seconds), asserts the stage result it saved, then asserts `run kernel - --dry-run` plans with that compiler (`compiler: llvm (from …)`, `cc=/usr/bin/clang`, - `variant: stock_llvm`), and again after switching back to `compiler = "gcc"`. A real kernel build - in the container is a follow-up only if a spike shows it fits: a `tinyconfig` build is minutes, - but the stage's boot-readiness check expects mkinitcpio and a bootloader the container lacks - (`state boot-entries` is already excused in `tools/e2e/uncovered.toml` for that reason). Tier - rubric row "state one command leaves for the next" (§Release Process). - *Priority: med · Effort: small · Bump: patch* — the cross-stage contract behind every LLVM kernel - build is untested end to end; the scenario is seconds long. +- **`4.0.0-B5` — a real drop-in in a batch auto-confirms every other conflict in it.** + `pacman.batch_install_pkgs` adds `--ask=4` when any built package supersedes an installed one + (`pkg_supersedes_installed`: `replaces`, or `conflicts` plus `provides` naming it). pacman + applies that answer to every `ALPM_QUESTION_CONFLICT_PKG` in the transaction, not only the + drop-in's own. So an `update` that rebuilds `wayland-git` over `wayland` in the same batch as a + package that merely conflicts with something installed removes that package without asking, + where the conflict alone would have stopped the install. `4.0.0-B4` closed the case where the + "drop-in" was a package naming itself; this is the general one. The fix decides before the + call: collect every conflict the batch would raise (each built package's `conflict` against the + installed set, and installed packages' conflicts naming a batch member, minus each package's own + name and the names the batch reinstalls) and pass `--ask=4` only when all of them are in the + superseded set. Otherwise leave it off, so pacman answers N and the install stops as it would + with no drop-in present, naming the unexpected conflict. Splitting the batch (drop-ins in their own + transaction) was the alternative, but the toolchain rollback relies on + `batch_install_pkgs` being all-or-nothing. Unit tests over a mixed batch; the e2e tier needs + a stand-in pair that conflicts and provides, beside `e2e-hello-alt` in `45-interactive`. + *Priority: med · Effort: small · Bump: patch* — silently removes a user's package, but only when + a drop-in and an unrelated conflict land in one batch; the check is local to one function. --- +### Developer tooling + - **`3.5.0-DEV26` — a release-time run of the real toolchain → kernel chain on real hardware.** - `DEV25` checks the handoff with a registered clang; what only a real build shows is that the kernel - is compiled by the toolchain sysforge just built and is recorded that way. A script under `tools/` + `96-toolchain-kernel-handoff` checks the handoff with a registered clang; what only a real build + shows is that the kernel is compiled by the toolchain sysforge just built and is recorded that + way. A script under `tools/` (and a Make target, with a step in `docs/RELEASE-CHECKLIST.md` for minor and major releases) runs `run toolchain` then `run kernel` on the workstation, and asserts on provenance, never on speed: the built `vmlinux`'s `.comment` section names the clang version `build_state.toml` recorded for @@ -194,20 +196,6 @@ canonical ordering. --- -- **`3.5.0-DEV28` — an end-to-end scenario for a stock package installed over a sysforge build.** A - user's own `pacman -S` of a package sysforge built from source should demote its record to a plain - `pacman` marker on the next `update` (`primitives/install_reconcile.py`, through the - `sysforge-buildstate.hook` sentinel), and a toolchain package changed outside sysforge should leave - the toolchain reminder (`sysforge-toolchain.hook`). The hooks run inside every scenario, but nothing - asserts their effect, and the ordering between the hook's events and sysforge's own installs has - needed two fixes (`3.5.0-B64`, `3.5.0-B120`). The scenario builds `pkgs/tree` from source, runs - `sudo pacman -S tree`, then asserts `update` demotes it and does not rebuild it; then `sudo pacman -S - lld` and asserts the reminder appears once and the sentinel is consumed. Needs the mirrors, as - `60-revert-to-stock` already does. - *Priority: med · Effort: small · Bump: patch* — a documented behaviour with a bug history and no end-to-end check; one short scenario. - ---- - - **`3.5.0-DEV29` — the build sandbox end to end, in the VM tier.** `[security] sandbox_builds = true` (clean chroot, local repo) is how every build on the maintainer's system runs, about ten 3.5.0 fixes were sandbox bugs, and no scenario enables it. It cannot run in the container tier: @@ -221,28 +209,6 @@ canonical ordering. --- -- **`3.5.0-DEV30` — an end-to-end scenario for `config merge`.** `config merge` is excused in - `tools/e2e/uncovered.toml` because it "needs a .sfnew and an interactive merge tool", but the tool is - `sysforge.toml [ui] merge`, which a scenario can set to a script. The scenario leaves a `.sfnew` - beside an edited config (as a package upgrade would), sets `merge` to a script that takes the new - file's added key and keeps the user's edit, runs `config merge`, and asserts the merged file, that - the `.sfnew` is gone and that `config check` is clean. The excusal comes out of `uncovered.toml` in - the same change. - *Priority: med · Effort: small · Bump: patch* — a command the maintainer runs after upgrades, excused for a reason that does not hold. - ---- - -- **`3.5.0-DEV31` — an end-to-end scenario for `build --interactive` and `update --interactive`.** `--interactive` - runs makepkg without `--noconfirm` and hands it the terminal, so pacman's own questions (a package - conflict, a provider choice) reach the user instead of failing under `--noconfirm` (`3.3.0-B6`). No - scenario passes it, and prompts and terminal handling account for about seventeen 3.5.0 fixes. A - stand-in package that conflicts with an installed one, built with `--interactive` and the conflict - question answered through `tools/e2e/answer.py`, asserts the swap happened; the same without the - flag asserts the build stops and installs nothing. - *Priority: med · Effort: small · Bump: patch* — a frequently used mode whose bug class is the largest after the two stages. - ---- - - **`3.5.0-DEV32` — an end-to-end scenario for package PGO (`build --pgo`).** `docs/guides/pgo.md` documents an instrumented build, a training run and a rebuild with the profile; the e2e tier runs none of it. A stand-in C package whose program has a hot path the training run exercises: `build diff --git a/docs/design/06-config-layer.md b/docs/design/06-config-layer.md index fe34d7fe..7b4b3536 100644 --- a/docs/design/06-config-layer.md +++ b/docs/design/06-config-layer.md @@ -19,7 +19,7 @@ The non-`profiles.toml` configs (`packages.toml`, `toolchain.toml`, `kernel.toml `sync-config` is an **add-only**, comment-preserving merge from `etc/sysforge/*.toml` into the live config dir (`$SYSFORGE_CONFIG_DIR` itself, else `/etc/sysforge`, or `--target DIR`): it injects keys, tables, and their leading comment blocks the live file is missing, and never overwrites a value the live file already sets (even if the shipped default changed). Arrays-of-tables (`[[package]]`) are user content and are left untouched. Bare keys are spliced before the first table header (TOML adjacency rule); a new table is spliced **after its nearest shipped-order predecessor that the live file actually has** — falling back through the shipped order, and to EOF only when there is no such predecessor — so the live file keeps shipped section order. Order matters even though TOML ignores it (`3.1.0-B9`): appending at EOF reorders the live file a little further with every sync, and the `.sfnew` adoption step below is a line-based diff with no move detection, which renders a merely relocated section as a deletion in one hunk and an addition far away in another — the reading that invites an operator to hand-merge away a section that was never gone. A run of consecutive new tables anchors on each other, staying in shipped order; insertion lands immediately after the anchor's own entry, before any standalone comment run tomlkit holds for the following table, so those comments stay attached to what they document. Pre-existing order drift is **not** rewritten — the tool never does a wholesale rewrite of a live file it did not lay out. The merge is **key-anchored**, so it can only carry comments that lead an active key it injects — pure documentation comments and *commented-out example* settings (`# interactive = true`) have no key to anchor to. When a shipped file gains such content the live file lacks, the shipped file is written verbatim beside the target as `.sfnew` (pacnew-style) for the operator to diff and adopt; a stale `.sfnew` is removed once the drift is resolved. Drift detection is line-level and value-aware: a commented example the live file has **already adopted** by uncommenting it into an *identical* active line (`# interactive = true` shipped vs `interactive = true` live) is not drift and does not spill a `.sfnew`, but a differing value or a trailing inline note still does. One drift class is **structural** rather than textual and blocks the merge outright: a section header that is active in the shipped file but only present *commented out* in the live file (a pre-`3.0.0-STD1` vintage still carrying `# [build]`). A commented header does not disable its section — it reassigns every key beneath it to the preceding table or to the top level, so the settings stay syntactically valid while being read from the wrong place. The add-only merge cannot flip an existing line from commented to active, and worse, it would see the table as absent and append a *second* copy holding the shipped defaults, superseding the operator's orphaned value on reparse. Header liveness is therefore compared in **both** directions on the pre-merge text (the comment-signature subtraction cannot see this, being one-directional — activating a header removes its commented form from shipped); on a hit the file reports status `needs merge`, the write is **skipped entirely** rather than merged into a structure the tool has misread, and the `.sfnew` companion is spilled pointing at `sysforge config merge`. A header commented out in the *shipped* file is an example block (`#[group.cosmic]`), not a live section, and is excluded. `tomlkit` is a **dev-only** dependency (ephemeral `uv run --no-sync --with tomlkit`), never added to `pyproject.toml`. `--dry-run` reports without writing. `bootstrap.toml` is excluded (per-host, no live counterpart). -**Adopting the `.sfnew` residue — `sysforge config merge`.** Hand-merging the `.sfnew` companions is the only remaining manual step, so `sysforge config merge` (verb `config-merge`, `config_cmd.py`) is a pacdiff-style interactive driver over them. It scans the resolved config dir (`--config-dir` override, else `paths.CONFIG_DIR`) for `*.sfnew` — plus pacman's own `*.pacnew`/`*.pacsave` for sysforge config files on a packaged install — and for each presents: `[v]iew` (a `difflib` unified diff through `$PAGER` via `maybe_pager`, prefixed by a **relocated-section banner** naming any header the diff shows as both removed and added — a header on both a `-` and a `+` line is present in both files, so the pair is a move, not a deletion; `difflib` has no move detection of its own), `[m]erge` (launch the resolved diff/merge tool with `live new`, then re-loop), `[s]kip`, `[r]emove` (drop the companion once the live file is satisfactory), `[o]verwrite` (copy the companion over the live file verbatim — guarded by a confirm and a "discards your local values" warning, never the default, intended for the `.pacnew` accept-maintainer case), and a[b]ort. Because a `.sfnew` is the *verbatim shipped file*, there is **no blind "accept theirs"** as a primary action — the safe path is merge-then-remove. The verb edits config files in place but never builds/installs, so it carries **no sentinel**. The diff/merge tool resolves through one home shared with the reconfigure editor chain — `primitives/editor.resolve_merge_tool` (`SYSFORGE_MERGE` > `sysforge.toml [ui].merge` > `$DIFFPROG` > `vimdiff`) launched via `run_tty_argv` (the `/dev/tty` passthrough, also used by `resolve_editor`'s callers, which additionally runs the child inside a `"child"` `yield_terminal` so an active progress bar's reserved bottom row is released before a full-screen editor takes the terminal). `--dry-run` reports the companion→target pairs without prompting (scripting/CI). +**Adopting the `.sfnew` residue — `sysforge config merge`.** Hand-merging the `.sfnew` companions is the only remaining manual step, so `sysforge config merge` (verb `config-merge`, `config_cmd.py`) is a pacdiff-style interactive driver over them. It scans the resolved config dir (`--config-dir` override, else `paths.CONFIG_DIR`) for `*.sfnew` — plus pacman's own `*.pacnew`/`*.pacsave` for sysforge config files on a packaged install — and for each presents: `[v]iew` (a `difflib` unified diff through `$PAGER` via `maybe_pager`, prefixed by a **relocated-section banner** naming any header the diff shows as both removed and added — a header on both a `-` and a `+` line is present in both files, so the pair is a move, not a deletion; `difflib` has no move detection of its own), `[m]erge` (launch the resolved diff/merge tool with `live new`, then re-loop), `[s]kip`, `[r]emove` (drop the companion once the live file is satisfactory), `[o]verwrite` (copy the companion over the live file verbatim — guarded by a confirm and a "discards your local values" warning, never the default, intended for the `.pacnew` accept-maintainer case), and a[b]ort. Because a `.sfnew` is the *verbatim shipped file*, there is **no blind "accept theirs"** as a primary action — the safe path is merge-then-remove. The verb edits config files in place but never builds/installs, so it carries **no sentinel**. The diff/merge tool resolves through one home shared with the reconfigure editor chain — `primitives/editor.resolve_merge_tool` (`SYSFORGE_MERGE` > `sysforge.toml [ui].merge` > `$DIFFPROG` > `vimdiff`) launched via `run_tty_argv` (the `/dev/tty` passthrough, also used by `resolve_editor`'s callers, which additionally runs the child inside a `"child"` `yield_terminal` so an active progress bar's reserved bottom row is released before a full-screen editor takes the terminal). `--dry-run` reports the companion→target pairs without prompting (scripting/CI). The live dir is normally root's `/etc/sysforge`, so the two writes the verb makes itself escalate (`4.0.0-B3`): `[r]emove` unlinks directly and falls back to `run_privileged(["rm", "-f", …])`, and `[o]verwrite` goes through `atomic_write.replace_file` (keeping the live file's mode and owner) before that removal. The merge tool runs as the user; saving into a root-owned file is the tool's own business (`sudoedit`-style), as with `pacdiff`. `tools/e2e/scenarios/75-config-merge.sh` runs the whole loop against `/etc/sysforge`. ### Checking and repairing a live config diff --git a/docs/release-notes/unreleased.md b/docs/release-notes/unreleased.md index c4a7fe7d..90c07fae 100644 --- a/docs/release-notes/unreleased.md +++ b/docs/release-notes/unreleased.md @@ -24,6 +24,22 @@ https://keepachangelog.com/en/1.1.0/ ## Changed +- **`3.5.0-DEV25` — an end-to-end scenario for the toolchain → kernel handoff.** `96-toolchain-kernel-handoff` runs a real `run toolchain` with `skip_build = true`, which registers a compiler without building it, and checks the result saved in `pipeline_state.toml`: `cc=/usr/bin/clang`, variant `stock_llvm` for `compiler = "llvm"`, and gcc after switching back. It then checks that `run kernel --dry-run`, with no compiler in `kernel.toml`, plans with the same compiler and variant. Before this, `95-stage-dry-run` previewed the two stages separately, so nothing checked what one leaves for the other. + +--- + +- **`3.5.0-DEV28` — an end-to-end scenario for a stock package installed over a sysforge build.** `65-external-install` builds `tree` from source, reinstalls it with `sudo pacman -S tree`, and checks that the next `update` demotes its record to `pacman` without rebuilding it and consumes the hook's sentinel. It then installs `lld` and checks that `update -v` shows the toolchain reminder once and not again. The pacman hooks ran in every scenario before, but nothing checked what they did. Needs the mirrors. + +--- + +- **`3.5.0-DEV30` — an end-to-end scenario for `config merge`.** `75-config-merge` leaves a `.sfnew` beside an edited `sysforge.toml`, sets `[ui] merge` to a script that takes the new file's added key, and runs `config merge` at a terminal: `[m]erge`, then `[r]emove`. It checks that the key arrived, the user's edit survived and the `.sfnew` is gone. `config merge` is no longer listed as uncovered. The scenario found `4.0.0-B3`. + +--- + +- **`3.5.0-DEV31` — an end-to-end scenario for `build --interactive` and `update --interactive`.** `45-interactive` uses a new stand-in, `e2e-hello-alt`, which conflicts with `e2e-hello` without providing it. Without the flag, `update` and then `build` stop at pacman's conflict question and install nothing. With `--interactive`, the question shows on the terminal and a yes swaps the packages. + +--- + - **`4.0.0-DEV1` — releases go through a release PR instead of a push to `main`.** The `main` ruleset requires a pull request, so the push the release script asked for was rejected. `tools/release.sh` now cuts each release on a `release/vX.Y.Z` branch and asks for `git push -u origin release/vX.Y.Z && git push origin vX.Y.Z`. Resuming requires that branch, and the script refuses to start if it already exists. The release checklist's last stage opens the release PR and merges it with a merge commit, never a squash: a squash rewrites the signed `release:` commit, leaving the `vX.Y.Z` tag outside `main`'s history, so `git describe` in the `-git` package keeps reporting the previous version. --- @@ -37,3 +53,11 @@ https://keepachangelog.com/en/1.1.0/ --- - **`4.0.0-B2` — bare `sysforge doctor` runs the full sweep again, and `doctor system` reads your config.** The `--config` axis flag and the loaded config shared one attribute. Bare `sysforge doctor` took the loaded config for the flag and checked only the live config settings, so it printed one section and stopped. `sysforge doctor system` took the flag's unset value for a loaded config and ran every axis against an empty one, so the toolchain, cache, graphics and storage checks ignored your settings. The flag now has its own attribute; its spelling is unchanged. + +--- + +- **`4.0.0-B3` — `config merge` can remove and overwrite files in `/etc/sysforge`.** Choosing `[r]emove` or `[o]verwrite` for a `.sfnew` or `.pacnew` crashed with `PermissionError`, because sysforge wrote the root-owned directory as your user. Removing a companion now escalates through `sudo`, and overwriting a live file goes through the same atomic replace as `config clean` and `config reset`, which keeps the live file's mode and owner. + +--- + +- **`4.0.0-B4` — a non-interactive install no longer removes a package it conflicts with because another package in the batch names itself.** Some AUR packages declare `provides` and `conflicts` on their own name (`cbonsai` does). sysforge read that as a drop-in replacing the installed copy and passed `--ask=4`, which pacman applies to every conflict question in the transaction. An `update` that rebuilt `cbonsai` alongside an unrelated conflicting package then removed the installed side without asking, instead of stopping. A package no longer counts as replacing itself, so only a real drop-in (`replaces`, or `conflicts` plus `provides` naming another installed package) gets the auto-confirm. Found by `45-interactive` in CI. diff --git a/sysforge/config_cmd.py b/sysforge/config_cmd.py index 53f6df0e..a5ca3ec7 100644 --- a/sysforge/config_cmd.py +++ b/sysforge/config_cmd.py @@ -37,7 +37,6 @@ import difflib import re -import shutil import time from pathlib import Path @@ -45,6 +44,7 @@ from sysforge.primitives import atomic_write, config_schema, json_out, paths, prompt from sysforge.primitives.editor import resolve_merge_tool, run_tty_argv from sysforge.primitives.pager import maybe_pager +from sysforge.primitives.privilege import run_privileged from sysforge.primitives.prompt import prompt_choice, prompt_key from sysforge.verbs import ExecResult, PreCheckResult, Verb, VerbGroup @@ -131,6 +131,16 @@ def _confirm(msg: str) -> bool: retry_on_invalid=False) in {"y", "yes"} +def _remove_companion(path: Path) -> None: + """Delete a companion, escalating when its dir is root's, as /etc/sysforge + is (4.0.0-B3).""" + try: + path.unlink() + except PermissionError: + # mutates=False: one unlink, nothing for an interruption to leave half done. + run_privileged(["rm", "-f", str(path)], tag="CONFIG", mutates=False) + + def _merge_one(target: Path, new_path: Path, use_pager: bool) -> str: """Drive the pacdiff-style loop for one companion file. @@ -170,14 +180,15 @@ def _merge_one(target: Path, new_path: Path, use_pager: bool) -> str: return "skipped" elif ans == "r": if _confirm(f" Remove {label}? (live file is already complete) [y/N] "): - new_path.unlink() + _remove_companion(new_path) _log.ui(f" Removed {label}") return "removed" elif ans == "o": if _confirm(f" Overwrite {target.name} with {label} verbatim? " "This discards your local values. [y/N] "): - shutil.copy2(new_path, target) - new_path.unlink() + atomic_write.replace_file( + target, new_path.read_text(encoding="utf-8"), tag="CONFIG") + _remove_companion(new_path) _log.ui(f" Installed {label} over {target.name}") return "installed" elif ans == "b": diff --git a/sysforge/primitives/pacman.py b/sysforge/primitives/pacman.py index 1169a39f..ce911d2a 100644 --- a/sysforge/primitives/pacman.py +++ b/sysforge/primitives/pacman.py @@ -495,11 +495,18 @@ def pkg_supersedes_installed(path, installed: set) -> set: collision, not a substitution, and must still stop the transaction for review. Empty set on any read failure — unreadable metadata never widens what gets auto-confirmed. + + A package never supersedes itself: some AUR packages declare the pair on + their own name (cbonsai's ``provides=(cbonsai) conflicts=(cbonsai)``), and + reinstalling one is no drop-in. Counting it would put ``--ask=4`` on the + whole batch, which pacman applies to every conflict question in the + transaction (4.0.0-B4). """ - fields = _read_pkginfo_names(path, ("replaces", "conflict", "provides")) - return supersedes_names( + fields = _read_pkginfo_names( + path, ("pkgname", "replaces", "conflict", "provides")) + return (supersedes_names( fields["replaces"], fields["conflict"], fields["provides"], - ) & installed + ) - fields["pkgname"]) & installed def filter_pkgs_to_installed( diff --git a/tests/test_config_merge.py b/tests/test_config_merge.py index c4f8564d..4605234b 100644 --- a/tests/test_config_merge.py +++ b/tests/test_config_merge.py @@ -141,6 +141,66 @@ def test_overwrite_replaces_live_after_confirm(tmp_path, monkeypatch): assert target.read_text(encoding="utf-8") == "# doc comment\nkey = \"shipped\"\n" +@pytest.fixture +def _root_owned_dir(monkeypatch): + """A config dir the user cannot write, as /etc/sysforge is root's, and a + stand-in sudo: each escalated argv runs with the dir writable for it.""" + import os + import subprocess + + from sysforge.primitives import atomic_write + + if os.geteuid() == 0: + pytest.skip("root writes a read-only dir anyway") + escalated: list[list[str]] = [] + + def setup(config_dir: Path) -> list[list[str]]: + def fake_run_privileged(argv, *, tag, mutates=True, **kw): # noqa: ARG001 + escalated.append(argv) + config_dir.chmod(0o755) + try: + return subprocess.run(argv, check=True) + finally: + config_dir.chmod(0o555) + + monkeypatch.setattr(config_cmd, "run_privileged", fake_run_privileged) + monkeypatch.setattr(atomic_write, "run_privileged", fake_run_privileged) + config_dir.chmod(0o555) + return escalated + + yield setup + for p in escalated: # let tmp_path cleanup in + Path(p[-1]).parent.chmod(0o755) + + +def test_remove_escalates_in_a_root_owned_dir(tmp_path, monkeypatch, _root_owned_dir): + # 4.0.0-B3: [r]emove unlinked as the user and crashed on /etc/sysforge. + config_dir = tmp_path / "cfg" + target, companion = _make_pair(config_dir) + escalated = _root_owned_dir(config_dir) + monkeypatch.setattr(config_cmd, "prompt_key", _scripted(["r"])) + monkeypatch.setattr(config_cmd, "prompt_choice", lambda *a, **k: "y") + + assert config_cmd.cmd_config_merge(_args(config_dir)) == 0 + assert not companion.exists() + assert escalated == [["rm", "-f", str(companion)]] + assert target.read_text(encoding="utf-8") == 'key = "live"\n' + + +def test_overwrite_escalates_in_a_root_owned_dir(tmp_path, monkeypatch, _root_owned_dir): + # 4.0.0-B3: [o]verwrite copied over the live file as the user and crashed. + config_dir = tmp_path / "cfg" + target, companion = _make_pair(config_dir) + escalated = _root_owned_dir(config_dir) + monkeypatch.setattr(config_cmd, "prompt_key", _scripted(["o"])) + monkeypatch.setattr(config_cmd, "prompt_choice", lambda *a, **k: "y") + + assert config_cmd.cmd_config_merge(_args(config_dir)) == 0 + assert not companion.exists() + assert target.read_text(encoding="utf-8") == "# doc comment\nkey = \"shipped\"\n" + assert escalated[-1] == ["rm", "-f", str(companion)] + + def test_abort_stops_the_run(tmp_path, monkeypatch): config_dir = tmp_path / "cfg" _make_pair(config_dir, ".sfnew") diff --git a/tests/test_pacman.py b/tests/test_pacman.py index f642d33a..b63143a9 100644 --- a/tests/test_pacman.py +++ b/tests/test_pacman.py @@ -494,6 +494,14 @@ def test_ignores_names_that_are_not_installed(self): with self._pkginfo(info): assert pkg_supersedes_installed("/tmp/p.pkg.tar", {"mesa"}) == set() + def test_a_package_never_supersedes_itself(self): + """AUR cbonsai declares `provides=(cbonsai) conflicts=(cbonsai)`. + Reinstalling it is no drop-in, and counting it would put ``--ask=4`` on + the whole batch, confirming any other conflict in it (4.0.0-B4).""" + info = "pkgname = cbonsai\nconflict = cbonsai\nprovides = cbonsai\n" + with self._pkginfo(info): + assert pkg_supersedes_installed("/tmp/p.pkg.tar", {"cbonsai"}) == set() + @patch("sysforge.primitives.pacman.subprocess.run", side_effect=FileNotFoundError) def test_unreadable_is_empty(self, _mock_run): assert pkg_supersedes_installed("/tmp/p.pkg.tar", {"wayland"}) == set() diff --git a/tools/e2e/README.md b/tools/e2e/README.md index b8a465cb..9ba77423 100644 --- a/tools/e2e/README.md +++ b/tools/e2e/README.md @@ -43,23 +43,29 @@ sentinels (`lib.sh`), not on rendered output. Prompts are answered on a pseudo-t | `30-flag-drift` | a default-profile change reported by `update --dry-run` and rebuilt by `--rebuild-on-flag-drift` | | `35-update-devel` | `update --devel` on a `-git` package: nothing rebuilt with no new commit, the new `pkgver()` built after one, the built commit recorded | | `40-pre-install-failure` | a build failing before any install leaves no sentinel | +| `45-interactive` | `update --interactive` and `build --interactive` put pacman's conflict question to the user; without the flag the install stops and nothing changes | | `50-maintain` | `state list`, `resolve`, `log`, `packages add`/`list`/`remove`/`add-group`, `state forget` and `uninstall` on a source-built package | | `60-revert-to-stock` | a source-built repo package replaced by the stock one, its record dropped | | `70-config` | `config check`, `clean` and `reset`, and `setup --preset` undone by `config reset` | +| `65-external-install` | a source-built package reinstalled with `pacman -S` is demoted by `update`, not rebuilt; a toolchain package changed outside sysforge leaves one reminder | +| `75-config-merge` | a `.sfnew` beside an edited config merged by the `[ui] merge` tool, the user's edit kept, the `.sfnew` removed | | `80-inspect` | the read-only commands (`state` reports, `doctor`, `env`, `fetch`) change nothing | | `85-artifacts` | a user script discovered, adopted, reviewed, edited, deployed and removed | | `90-source-policy` | `[aur] enabled = false` and `freeze_sources` refuse before any checkout; `search` with the AUR off | | `95-stage-dry-run` | `run toolchain`/`run kernel --dry-run` with non-default configs change nothing, and the kernel's shows its plan | +| `96-toolchain-kernel-handoff` | the compiler `run toolchain` registers (`skip_build`, llvm then gcc) is the one `run kernel --dry-run` plans with | | `99-self-uninstall` | a repo package reverted, a local one kept, then the package, its data, group and `IgnoreGroup` entry gone; last, since nothing runs after it | Packages come from `pkgs/`, local PKGBUILDs with no network sources, so an outage can't flake the -tier. Four scenarios need the network, because it is the point: `10-quickstart` (the AUR, and -pacman for build dependencies), `60-revert-to-stock` and `99-self-uninstall` (the stock package -from the mirrors) and `90-source-policy` (the AUR lookup a frozen build makes before refusing the +tier. Five scenarios need the network, because it is the point: `10-quickstart` (the AUR, and +pacman for build dependencies), `60-revert-to-stock`, `65-external-install` and `99-self-uninstall` +(the stock packages from the mirrors) and `90-source-policy` (the AUR lookup a frozen build makes before refusing the clone). `pkgs/tree` is a stand-in named after a repo package, which puts `build` on its repo opt-in path without fetching Arch's packaging repo. `pkgs/e2e-hello-git` is a VCS package whose source is a git repository its scenario creates, so `update --devel` runs `git ls-remote` and `pkgver()` offline. +`pkgs/e2e-hello-alt` conflicts with `e2e-hello` without providing it, so installing either over +the other raises pacman's conflict question. A scenario can't catch a change in what the live repos contain, because `pkgs/` stands in for them. That is the live-mirror tier's job (`make test-live`, `tests/test_live_stock_toolchain.py`). diff --git a/tools/e2e/pkgs/e2e-hello-alt/PKGBUILD b/tools/e2e/pkgs/e2e-hello-alt/PKGBUILD new file mode 100644 index 00000000..1df2c374 --- /dev/null +++ b/tools/e2e/pkgs/e2e-hello-alt/PKGBUILD @@ -0,0 +1,20 @@ +# A local package that conflicts with e2e-hello without providing it: an +# unexpected collision, not a drop-in, so pacman's "Remove e2e-hello?" question +# is answered N under --noconfirm and reaches the user only with --interactive. +pkgname=e2e-hello-alt +pkgver=1.0 +pkgrel=1 +pkgdesc="sysforge end-to-end test package that conflicts with e2e-hello" +arch=('x86_64') +license=('MIT') +conflicts=('e2e-hello') +source=('hello.c') +sha256sums=('SKIP') + +build() { + ${CC:-cc} $CPPFLAGS $CFLAGS $LDFLAGS -o e2e-hello-alt hello.c +} + +package() { + install -Dm755 e2e-hello-alt "$pkgdir/usr/bin/e2e-hello-alt" +} diff --git a/tools/e2e/pkgs/e2e-hello-alt/hello.c b/tools/e2e/pkgs/e2e-hello-alt/hello.c new file mode 100644 index 00000000..31ef1dda --- /dev/null +++ b/tools/e2e/pkgs/e2e-hello-alt/hello.c @@ -0,0 +1,6 @@ +#include + +int main(void) { + puts("hello from sysforge's end-to-end tier"); + return 0; +} diff --git a/tools/e2e/scenarios/45-interactive.sh b/tools/e2e/scenarios/45-interactive.sh new file mode 100644 index 00000000..577a2094 --- /dev/null +++ b/tools/e2e/scenarios/45-interactive.sh @@ -0,0 +1,70 @@ +# shellcheck shell=bash +# `--interactive` hands pacman's own questions to the user (3.3.0-B6). A +# package conflict is answered N under --noconfirm, so without the flag the +# install stops and nothing changes; with it, the question shows on the +# terminal and a yes swaps the packages. `update` first, then `build`. +# start: ready + +hello=$(stage_pkg e2e-hello) +alt=$(stage_pkg e2e-hello-alt) +profiles=/etc/sysforge/profiles.toml + +# conflicts_with PKG: the installed PKG's "Conflicts With" field. +conflicts_with() { + pacman -Qi "$1" | sed -n 's/^Conflicts With *: //p' +} + +step "two packages that do not conflict yet" +sed -i 's/^conflicts=.*/conflicts=()/' "$alt/PKGBUILD" +# e2e-hello by makepkg alone: no record, so the profile change below rebuilds +# only e2e-hello-alt. pacman asks "Remove e2e-hello?" only when e2e-hello is +# installed and not itself in the transaction. Under --here 30-flag-drift +# left a record for it, which the makepkg install below does not clear, so +# forget it first or the drift rebuilds both sides and pacman never asks. +if [[ -n $(state_get e2e-hello build_mode) ]]; then + sysforge state forget e2e-hello /dev/null || true +[[ -s $sentinels/buildstate ]] || fail "sysforge-buildstate.hook left no sentinel" +packager=$(pacman -Qi tree | sed -n 's/^Packager *: //p') +[[ $packager != "Unknown Packager" ]] || fail "pacman -S did not install the repo's tree" + +step "update demotes it instead of rebuilding it" +sysforge update --yes --no-sysupgrade &1) || fail "update failed: $out" +echo "$out" +[[ $(grep -c -F "$reminder" <<<"$out") == 1 ]] || fail "update did not show the toolchain reminder once" +[[ ! -e $sentinels/toolchain ]] || fail "update left the toolchain sentinel" +out=$(sysforge update -v --yes --no-sysupgrade &1) || fail "update failed: $out" +if grep -q -F "$reminder" <<<"$out"; then + fail "the toolchain reminder showed again" +fi diff --git a/tools/e2e/scenarios/75-config-merge.sh b/tools/e2e/scenarios/75-config-merge.sh new file mode 100644 index 00000000..22d32940 --- /dev/null +++ b/tools/e2e/scenarios/75-config-merge.sh @@ -0,0 +1,44 @@ +# shellcheck shell=bash +# Adopting a shipped config change: a package upgrade leaves the new default +# file beside an edited one as a .sfnew, and `config merge` runs the merge tool +# sysforge.toml [ui] merge names on the pair, then removes the .sfnew. The +# tool here is a script that takes the new file's added key and keeps the +# user's edit, as a person in a merge tool would. +# start: ready + +conf=/etc/sysforge/sysforge.toml +tool=~/e2e/merge-tool + +step "an edited config, and a .sfnew whose shipped file gained a key" +sudo sed -i 's/^fetch_timeout = [0-9]*$/fetch_timeout = 77/' "$conf" +grep -q '^fetch_timeout = 77$' "$conf" || fail "could not edit fetch_timeout" +# The shipped file plus one key. Unknown to the schema, so `config check` +# shows whether the merge carried it into the live file; removed at the end. +sed '/^\[git\]/a e2e_merged_key = 1' "$conf" | sed 's/^fetch_timeout = 77$/fetch_timeout = 30/' \ + | sudo tee "$conf.sfnew" >/dev/null +grep -q '^e2e_merged_key = 1$' "$conf.sfnew" || fail "could not stage the .sfnew" + +step "[ui] merge names a script" +mkdir -p ~/e2e +cat >"$tool" <<'SH' +#!/usr/bin/env bash +# merge-tool LIVE NEW: copy NEW's lines missing from LIVE into it, under [git]. +set -euo pipefail +line=$(grep '^e2e_merged_key' "$2") +sudo sed -i "/^\[git\]/a $line" "$1" +SH +chmod +x "$tool" +sudo sed -i "s|^#merge = .*|merge = \"$tool\"|" "$conf" +grep -q "^merge = \"$tool\"$" "$conf" || fail "could not set [ui] merge" + +step "config merge: [m]erge, then [r]emove the .sfnew" +answer 'a\[b\]ort\? ' 'm' 'a\[b\]ort\? ' 'r' 'Remove sysforge.toml.sfnew\?' 'y\n' -- sysforge config merge --no-pager +[[ ! -e $conf.sfnew ]] || fail "config merge left the .sfnew" +grep -q '^e2e_merged_key = 1$' "$conf" || fail "the merge tool's change is not in the live file" +grep -q '^fetch_timeout = 77$' "$conf" || fail "the merge lost the user's edit" + +step "the merged file checks clean once the test key is gone" +expect_exit 1 sysforge config check &1) || fail "run kernel --dry-run failed: $out" + echo "$out" +} + +step "the toolchain registers its compiler without building; the kernel names none" +# Set outright, not from the shipped lines: under --here 95-stage-dry-run has +# already edited both files. +sudo sed -i '0,/^#\?enabled = [a-z]*$/s//enabled = true/; s/^#\?skip_build = [a-z]*$/skip_build = true/' "$tc" +sudo sed -i '0,/^#\?enabled = [a-z]*$/s//enabled = true/; s/^compiler = .*/#compiler = "gcc"/; + s/^#\?bootloader = "[a-z-]*"$/bootloader = "none"/; s/^#\?source = "[a-z]*"$/source = "local"/' "$kc" +grep -q '^skip_build = true$' "$tc" || fail "could not set toolchain.toml skip_build" +if grep -q '^compiler = ' "$kc"; then + fail "kernel.toml still names a compiler" +fi +mkdir -p ~/src/linux-sysforge +printf '%s\n' 'pkgbase=linux-sysforge' 'pkgname=(linux-sysforge linux-sysforge-headers)' \ + 'pkgver=6.1' 'pkgrel=1' 'arch=(x86_64)' 'prepare() { make olddefconfig; }' \ + >~/src/linux-sysforge/PKGBUILD + +step "run toolchain with llvm saves clang" +set_compiler llvm +sysforge run toolchain --no-update