Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -494,7 +494,7 @@ The non-`profiles.toml` configs (`packages.toml`, `toolchain.toml`, `kernel.toml

`sync-config` is an **add-only**, comment-preserving merge from `etc/sysforge/*.toml` into the live config dir (`$SYSFORGE_CONFIG_DIR` itself, else `/etc/sysforge`, or `--target DIR`): it injects keys, tables, and their leading comment blocks the live file is missing, and never overwrites a value the live file already sets (even if the shipped default changed). Arrays-of-tables (`[[package]]`) are user content and are left untouched. Bare keys are spliced before the first table header (TOML adjacency rule); a new table is spliced **after its nearest shipped-order predecessor that the live file actually has** — falling back through the shipped order, and to EOF only when there is no such predecessor — so the live file keeps shipped section order. Order matters even though TOML ignores it (`3.1.0-B9`): appending at EOF reorders the live file a little further with every sync, and the `.sfnew` adoption step below is a line-based diff with no move detection, which renders a merely relocated section as a deletion in one hunk and an addition far away in another — the reading that invites an operator to hand-merge away a section that was never gone. A run of consecutive new tables anchors on each other, staying in shipped order; insertion lands immediately after the anchor's own entry, before any standalone comment run tomlkit holds for the following table, so those comments stay attached to what they document. Pre-existing order drift is **not** rewritten — the tool never does a wholesale rewrite of a live file it did not lay out. The merge is **key-anchored**, so it can only carry comments that lead an active key it injects — pure documentation comments and *commented-out example* settings (`# interactive = true`) have no key to anchor to. When a shipped file gains such content the live file lacks, the shipped file is written verbatim beside the target as `<name>.sfnew` (pacnew-style) for the operator to diff and adopt; a stale `.sfnew` is removed once the drift is resolved. Drift detection is line-level and value-aware: a commented example the live file has **already adopted** by uncommenting it into an *identical* active line (`# interactive = true` shipped vs `interactive = true` live) is not drift and does not spill a `.sfnew`, but a differing value or a trailing inline note still does. One drift class is **structural** rather than textual and blocks the merge outright: a section header that is active in the shipped file but only present *commented out* in the live file (a pre-`3.0.0-STD1` vintage still carrying `# [build]`). A commented header does not disable its section — it reassigns every key beneath it to the preceding table or to the top level, so the settings stay syntactically valid while being read from the wrong place. The add-only merge cannot flip an existing line from commented to active, and worse, it would see the table as absent and append a *second* copy holding the shipped defaults, superseding the operator's orphaned value on reparse. Header liveness is therefore compared in **both** directions on the pre-merge text (the comment-signature subtraction cannot see this, being one-directional — activating a header removes its commented form from shipped); on a hit the file reports status `needs merge`, the write is **skipped entirely** rather than merged into a structure the tool has misread, and the `.sfnew` companion is spilled pointing at `sysforge config merge`. A header commented out in the *shipped* file is an example block (`#[group.cosmic]`), not a live section, and is excluded. `tomlkit` is a **dev-only** dependency (ephemeral `uv run --no-sync --with tomlkit`), never added to `pyproject.toml`. `--dry-run` reports without writing. `bootstrap.toml` is excluded (per-host, no live counterpart).

**Adopting the `.sfnew` residue — `sysforge config merge`.** Hand-merging the `.sfnew` companions is the only remaining manual step, so `sysforge config merge` (verb `config-merge`, `config_cmd.py`) is a pacdiff-style interactive driver over them. It scans the resolved config dir (`--config-dir` override, else `paths.CONFIG_DIR`) for `*.sfnew` — plus pacman's own `*.pacnew`/`*.pacsave` for sysforge config files on a packaged install — and for each presents: `[v]iew` (a `difflib` unified diff through `$PAGER` via `maybe_pager`, prefixed by a **relocated-section banner** naming any header the diff shows as both removed and added — a header on both a `-` and a `+` line is present in both files, so the pair is a move, not a deletion; `difflib` has no move detection of its own), `[m]erge` (launch the resolved diff/merge tool with `live new`, then re-loop), `[s]kip`, `[r]emove` (drop the companion once the live file is satisfactory), `[o]verwrite` (copy the companion over the live file verbatim — guarded by a confirm and a "discards your local values" warning, never the default, intended for the `.pacnew` accept-maintainer case), and a[b]ort. Because a `.sfnew` is the *verbatim shipped file*, there is **no blind "accept theirs"** as a primary action — the safe path is merge-then-remove. The verb edits config files in place but never builds/installs, so it carries **no sentinel**. The diff/merge tool resolves through one home shared with the reconfigure editor chain — `primitives/editor.resolve_merge_tool` (`SYSFORGE_MERGE` > `sysforge.toml [ui].merge` > `$DIFFPROG` > `vimdiff`) launched via `run_tty_argv` (the `/dev/tty` passthrough, also used by `resolve_editor`'s callers, which additionally runs the child inside a `"child"` `yield_terminal` so an active progress bar's reserved bottom row is released before a full-screen editor takes the terminal). `--dry-run` reports the companion→target pairs without prompting (scripting/CI).
**Adopting the `.sfnew` residue — `sysforge config merge`.** Hand-merging the `.sfnew` companions is the only remaining manual step, so `sysforge config merge` (verb `config-merge`, `config_cmd.py`) is a pacdiff-style interactive driver over them. It scans the resolved config dir (`--config-dir` override, else `paths.CONFIG_DIR`) for `*.sfnew` — plus pacman's own `*.pacnew`/`*.pacsave` for sysforge config files on a packaged install — and for each presents: `[v]iew` (a `difflib` unified diff through `$PAGER` via `maybe_pager`, prefixed by a **relocated-section banner** naming any header the diff shows as both removed and added — a header on both a `-` and a `+` line is present in both files, so the pair is a move, not a deletion; `difflib` has no move detection of its own), `[m]erge` (launch the resolved diff/merge tool with `live new`, then re-loop), `[s]kip`, `[r]emove` (drop the companion once the live file is satisfactory), `[o]verwrite` (copy the companion over the live file verbatim — guarded by a confirm and a "discards your local values" warning, never the default, intended for the `.pacnew` accept-maintainer case), and a[b]ort. Because a `.sfnew` is the *verbatim shipped file*, there is **no blind "accept theirs"** as a primary action — the safe path is merge-then-remove. The verb edits config files in place but never builds/installs, so it carries **no sentinel**. The diff/merge tool resolves through one home shared with the reconfigure editor chain — `primitives/editor.resolve_merge_tool` (`SYSFORGE_MERGE` > `sysforge.toml [ui].merge` > `$DIFFPROG` > `vimdiff`) launched via `run_tty_argv` (the `/dev/tty` passthrough, also used by `resolve_editor`'s callers, which additionally runs the child inside a `"child"` `yield_terminal` so an active progress bar's reserved bottom row is released before a full-screen editor takes the terminal). `--dry-run` reports the companion→target pairs without prompting (scripting/CI). The live dir is normally root's `/etc/sysforge`, so the two writes the verb makes itself escalate (`4.0.0-B3`): `[r]emove` unlinks directly and falls back to `run_privileged(["rm", "-f", …])`, and `[o]verwrite` goes through `atomic_write.replace_file` (keeping the live file's mode and owner) before that removal. The merge tool runs as the user; saving into a root-owned file is the tool's own business (`sudoedit`-style), as with `pacdiff`. `tools/e2e/scenarios/75-config-merge.sh` runs the whole loop against `/etc/sysforge`.

### Checking and repairing a live config

Expand Down
82 changes: 24 additions & 58 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,10 +112,7 @@ canonical ordering.
<!-- BEGIN roadmap-table (generated by tools/gen_roadmap_table.py — do not edit by hand; run `make roadmap-table`) -->
| ID | Item | Priority | Effort | Bump |
|----|------|----------|--------|------|
| `3.5.0-DEV25` | an end-to-end scenario for the toolchain → kernel handoff | med | small | patch |
| `3.5.0-DEV28` | an end-to-end scenario for a stock package installed over a sysforge build | med | small | patch |
| `3.5.0-DEV30` | an end-to-end scenario for config merge | med | small | patch |
| `3.5.0-DEV31` | an end-to-end scenario for build --interactive and update --interactive | med | small | patch |
| `4.0.0-B5` | a real drop-in in a batch auto-confirms every other conflict in it | med | small | patch |
| `3.5.0-DEV29` | the build sandbox end to end, in the VM tier | med | large | patch |
| `3.5.0-DEV33` | an end-to-end scenario for unattended runs | low | small | patch |
| `3.5.0-DEV34` | an end-to-end scenario for the --frozen and --thaw flags | low | small | patch |
Expand Down Expand Up @@ -156,28 +153,33 @@ canonical ordering.

### Bugs

### Developer tooling

- **`3.5.0-DEV25` — an end-to-end scenario for the toolchain → kernel handoff.** `95-stage-dry-run`
previews `run toolchain` and `run kernel` separately, so nothing checks what one leaves for the
other: the compiler, its paths and its variant in `pipeline_state.toml`, which the kernel stage
(and `packages`) read. A scenario under `tools/e2e/scenarios/` runs a real `run toolchain` with
`toolchain.toml compiler = "llvm"` and `skip_build = true` (it registers the container's clang
without building, in seconds), asserts the stage result it saved, then asserts `run kernel
--dry-run` plans with that compiler (`compiler: llvm (from …)`, `cc=/usr/bin/clang`,
`variant: stock_llvm`), and again after switching back to `compiler = "gcc"`. A real kernel build
in the container is a follow-up only if a spike shows it fits: a `tinyconfig` build is minutes,
but the stage's boot-readiness check expects mkinitcpio and a bootloader the container lacks
(`state boot-entries` is already excused in `tools/e2e/uncovered.toml` for that reason). Tier
rubric row "state one command leaves for the next" (§Release Process).
*Priority: med · Effort: small · Bump: patch* — the cross-stage contract behind every LLVM kernel
build is untested end to end; the scenario is seconds long.
- **`4.0.0-B5` — a real drop-in in a batch auto-confirms every other conflict in it.**
`pacman.batch_install_pkgs` adds `--ask=4` when any built package supersedes an installed one
(`pkg_supersedes_installed`: `replaces`, or `conflicts` plus `provides` naming it). pacman
applies that answer to every `ALPM_QUESTION_CONFLICT_PKG` in the transaction, not only the
drop-in's own. So an `update` that rebuilds `wayland-git` over `wayland` in the same batch as a
package that merely conflicts with something installed removes that package without asking,
where the conflict alone would have stopped the install. `4.0.0-B4` closed the case where the
"drop-in" was a package naming itself; this is the general one. The fix decides before the
call: collect every conflict the batch would raise (each built package's `conflict` against the
installed set, and installed packages' conflicts naming a batch member, minus each package's own
name and the names the batch reinstalls) and pass `--ask=4` only when all of them are in the
superseded set. Otherwise leave it off, so pacman answers N and the install stops as it would
with no drop-in present, naming the unexpected conflict. Splitting the batch (drop-ins in their own
transaction) was the alternative, but the toolchain rollback relies on
`batch_install_pkgs` being all-or-nothing. Unit tests over a mixed batch; the e2e tier needs
a stand-in pair that conflicts and provides, beside `e2e-hello-alt` in `45-interactive`.
*Priority: med · Effort: small · Bump: patch* — silently removes a user's package, but only when
a drop-in and an unrelated conflict land in one batch; the check is local to one function.

---

### Developer tooling

- **`3.5.0-DEV26` — a release-time run of the real toolchain → kernel chain on real hardware.**
`DEV25` checks the handoff with a registered clang; what only a real build shows is that the kernel
is compiled by the toolchain sysforge just built and is recorded that way. A script under `tools/`
`96-toolchain-kernel-handoff` checks the handoff with a registered clang; what only a real build
shows is that the kernel is compiled by the toolchain sysforge just built and is recorded that
way. A script under `tools/`
(and a Make target, with a step in `docs/RELEASE-CHECKLIST.md` for minor and major releases) runs
`run toolchain` then `run kernel` on the workstation, and asserts on provenance, never on speed:
the built `vmlinux`'s `.comment` section names the clang version `build_state.toml` recorded for
Expand All @@ -194,20 +196,6 @@ canonical ordering.

---

- **`3.5.0-DEV28` — an end-to-end scenario for a stock package installed over a sysforge build.** A
user's own `pacman -S` of a package sysforge built from source should demote its record to a plain
`pacman` marker on the next `update` (`primitives/install_reconcile.py`, through the
`sysforge-buildstate.hook` sentinel), and a toolchain package changed outside sysforge should leave
the toolchain reminder (`sysforge-toolchain.hook`). The hooks run inside every scenario, but nothing
asserts their effect, and the ordering between the hook's events and sysforge's own installs has
needed two fixes (`3.5.0-B64`, `3.5.0-B120`). The scenario builds `pkgs/tree` from source, runs
`sudo pacman -S tree`, then asserts `update` demotes it and does not rebuild it; then `sudo pacman -S
lld` and asserts the reminder appears once and the sentinel is consumed. Needs the mirrors, as
`60-revert-to-stock` already does.
*Priority: med · Effort: small · Bump: patch* — a documented behaviour with a bug history and no end-to-end check; one short scenario.

---

- **`3.5.0-DEV29` — the build sandbox end to end, in the VM tier.** `[security] sandbox_builds = true`
(clean chroot, local repo) is how every build on the maintainer's system runs, about ten 3.5.0
fixes were sandbox bugs, and no scenario enables it. It cannot run in the container tier:
Expand All @@ -221,28 +209,6 @@ canonical ordering.

---

- **`3.5.0-DEV30` — an end-to-end scenario for `config merge`.** `config merge` is excused in
`tools/e2e/uncovered.toml` because it "needs a .sfnew and an interactive merge tool", but the tool is
`sysforge.toml [ui] merge`, which a scenario can set to a script. The scenario leaves a `.sfnew`
beside an edited config (as a package upgrade would), sets `merge` to a script that takes the new
file's added key and keeps the user's edit, runs `config merge`, and asserts the merged file, that
the `.sfnew` is gone and that `config check` is clean. The excusal comes out of `uncovered.toml` in
the same change.
*Priority: med · Effort: small · Bump: patch* — a command the maintainer runs after upgrades, excused for a reason that does not hold.

---

- **`3.5.0-DEV31` — an end-to-end scenario for `build --interactive` and `update --interactive`.** `--interactive`
runs makepkg without `--noconfirm` and hands it the terminal, so pacman's own questions (a package
conflict, a provider choice) reach the user instead of failing under `--noconfirm` (`3.3.0-B6`). No
scenario passes it, and prompts and terminal handling account for about seventeen 3.5.0 fixes. A
stand-in package that conflicts with an installed one, built with `--interactive` and the conflict
question answered through `tools/e2e/answer.py`, asserts the swap happened; the same without the
flag asserts the build stops and installs nothing.
*Priority: med · Effort: small · Bump: patch* — a frequently used mode whose bug class is the largest after the two stages.

---

- **`3.5.0-DEV32` — an end-to-end scenario for package PGO (`build --pgo`).** `docs/guides/pgo.md`
documents an instrumented build, a training run and a rebuild with the profile; the e2e tier runs
none of it. A stand-in C package whose program has a hot path the training run exercises: `build
Expand Down
Loading
Loading