Skip to content

pull: merge into edited files, generate migrations, and the review fixes - #101

Open
joris-sevenlab wants to merge 63 commits into
mainfrom
fix/pull-fixes
Open

joris-sevenlab wants to merge 63 commits into
mainfrom
fix/pull-fixes

Conversation

@joris-sevenlab

@joris-sevenlab joris-sevenlab commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

What this changes

One PR for the battlestack pull work that was spread over #94, #97, #100 and #89, plus the fixes their reviews asked for. After this merges, pull merges upstream changes into files you edited, never overwrites local work, generates a migration when it changes the schema (and says so honestly when it could not), and reports what is left to do in .battlestack/pull/REPORT.md with exit code 2.

Supersedes and closes #94, #97 and #100. Closes #89.

Why

The three PRs fixed overlapping parts of pull from different angles: #100's ordering bug is already fixed at the root by #94's hash handling, and #97 sat on top of the same lines in pull.ts. Reviewing and merging them one by one meant three rounds of conflicts in the same file. Testing them together against a real scaffold also turned up bugs none of them caught on their own (see below).

What is in here

From #94 (seamless upgrade), unchanged apart from the merge with main: three-way merge into edited files, the overwrite check after the update, the clean-tree preflight, REPORT.md and exit code 2. The merge with #93 keeps the rename re-render; its warning now points at REPORT.md instead of patch files.

From #97 (migrations), reworked for the review in Slack:

  • A migration only counts as generated when a new .sql file appears. drizzle-kit exits 0 when it stops at a rename prompt without a TTY, so the old "ran db:generate" line showed up with no migration written. Scaffold's initial migration uses the same check.
  • When a schema file is staged or has conflicts, pull does not generate (the change is not on disk yet) and lists "finish the merge, then run db:generate" in REPORT.md.
  • Anything left to do goes to REPORT.md and exit code 2, instead of a warning that changed nothing.
  • drizzle-kit's own error line is quoted (not a stack frame), with a hint to run it in a terminal for a rename. "No schema changes" is treated as nothing to do, only the files drizzle.config.ts reads are hashed, and the hint names the project's package manager.

From #100: its test only. The reorder is not needed on top of #94. Separately, pull now removes files no feature ships any more only after every feature has updated, so a file moving between features is never deleted and written back in the same run, whatever the order.

From #89 (rateLimiter gate), reworked: the gate is applied through the config AST instead of a regex, so a limiter with nested options is gated too, a commented-out one is ignored, and false, an already-gated value or a variable is left alone.

Found while testing against a real nuxt4-fullstack scaffold: every nuxt4:database bump staged server/database/schema/users.ts for a manual merge, because nuxt4:storage patches in avatarKey after the database feature recorded the file. Taking the staged file and running db:generate would drop the avatar_key column. nuxt4:database now ships a users.ts variant with avatarKey when storage is on (same pattern as health/with-db), and a test pins it byte-for-byte to storage's patch.

Review follow-ups from #95, #96, #98 and #99:

  • runtimeConfig that is not an inline object gives a clear error instead of a proxy crash, skipped nested defaults are warned about, an explicit null is kept. setRuntimePublicDefault goes through the same merge.
  • A sessionHeader other than false is kept with a warning; two more e2e cases for the session header (revoked on another device, live session in the header only).
  • Icon detection covers shortcut icon, any casing, and useHead in app/app.vue and layouts; pull notes when it skips the template links.
  • CI prints server.log when the tests fail, the readiness curl has timeouts, stale comments fixed, and the ruleset file turns on strict status checks.

Also: a rename stays pending when a feature fails or its plugin does not load during the re-render (edge case from the #93 review), and .claude/worktrees/ is ignored.

Fixes from the in-depth review of this PR

  • A file moving to another feature with changed content was deleted when its old owner updated first (a regression in the deferred deletion above). The new owner now sees the old owner's queued hash.
  • Monorepos: the clean-tree check, the overwrite repair and the git-history merge base are scoped to the project folder (--relative, HEAD:./<rel>).
  • Checkout filters: HEAD and history are read with git cat-file --filters, so core.autocrlf or eol=crlf no longer make pristine files look edited. The history lookup is async and reads each blob once.
  • Files claimed with battlestack own are skipped by every recorded writer again (the Dockerfile was merged), and a feature with structural files no longer drops own entries from the manifest.
  • shared:github ignores a retired workflow path it never wrote (a project's own production.yml).
  • battlestack add puts its section above an earlier pull's REPORT.md instead of erasing it.
  • --force / --overwrite start over conflict markers; the overwrite repair keeps a .bak under --force and stays out of the way under --overwrite. Its staged fallback restores the writing feature's own hash and writes the promised .patch.
  • Report: a restored file is listed once, and feature lines count every merge, conflict and staged file.
  • Migrations: "No schema changes" is read from both streams. On a storage project an edited users.ts gets a stored merge base, and pull repairs a migration that drops users.avatar_key, which earlier pulls could have led to: it takes the drop out of the migration and drizzle's snapshots, and adds a migration (ADD COLUMN IF NOT EXISTS, made with drizzle-kit generate --custom) that puts the column back where the drop already ran. Verified with real drizzle-kit: a fresh db:generate afterwards finds no schema changes.
  • A rename stays pending while a plugin is not loaded.
  • ^0.x ranges: a higher 0.x minor is not kept over a feature's range.
  • Template favicon links an earlier pull added are removed once the project has its own (only when all three are present).
  • One core helper (trackersOf / recordedFiles) replaces four copies of the recorded-files walk; duplicated sha256 helpers are gone.

Finishing touches

  • .battlestack/base/ is a local merge cache and stays out of git: new projects ignore it, the docs and the upgrade skill no longer say to commit it, and pull lists git rm -r --cached .battlestack/base for a project that committed it.
  • A project scaffolded before the release-age policy moved to pnpm-workspace.yaml can have a lockfile pnpm then rejects. The pull that adds the policy checks it with pnpm install --frozen-lockfile --lockfile-only, and when it fails, skips lint and db:generate and lists the lockfile rebuild in REPORT.md. Rare in practice, since older projects had a policy in .npmrc that pnpm did not enforce, so only recently updated packages trip it.
  • The battlestack-upgrade skill covers migrations, set-aside schema files, --force backups, monorepos and checking that a second pull exits with code 0.
  • Run against a real client project's upgrade from wolf 2.3.9: same conflict set as the pull: merge into edited files, never overwrite local work #94 run, a clean generated migration (no drops), users.ts kept avatarKey, and the overwrite repair caught five files a feature wrote over.

Type of change

  • Bug fix
  • New feature or template
  • Refactor with no behaviour change
  • Documentation
  • Build, CI or release tooling

AI assistance

  • No AI agent involved
  • Claude Code
  • Cursor
  • GitHub Copilot
  • Codex
  • Other, named below

Agent, service and model, if you know it: Claude Code, Opus 5.5

How much of this is agent-written?

  • Whole change, reviewed by me before opening
  • Substantial parts, with my own edits on top
  • Small parts, or just the tests or docs
  • Only the commit message or this description

Feature versions

  • I changed what a feature emits, and bumped that feature's version: nuxt4:auth 1.10.8, nuxt4:essentials 1.2.4, nuxt4:database 1.7.5, nuxt4:storage 1.3.4, shared:security 1.0.5, plus pull: merge into edited files, never overwrite local work #94's own bumps
  • I changed a collectDocs or collectEnv contribution, and also bumped the feature that writes the target file
  • Not applicable

Changelog

  • I added a line under ## Unreleased saying what changes for someone using battlestack
  • Not applicable

Testing

pnpm tsc
pnpm test                                   # 105 files, 1171 passed, 1 skipped
BUMP_CHECK_BASE=origin/main pnpm bump:check # OK

Against a real nuxt4-fullstack scaffold (installed, committed), running the CLI from source:

  • plain pull with no template change: every feature up to date, exit 0, nothing in REPORT.md

  • index added to the auth sessions.ts template: generated 0001_*.sql, exit 0, SQL is the CREATE INDEX

  • column renamed: no SQL, REPORT.md quotes drizzle-kit, exit 2 (the stack-frame quoting found here is fixed)

  • sessions.ts edited by the user: a clean merge generates the migration with both edits; a conflict skips generation and lists the merge, exit 2

  • nuxt4:database bump with storage on: reproduced the users.ts staging above, now fixed

  • pnpm tsc passes

  • pnpm test passes

  • I removed the behaviour each new assertion covers, watched the test fail, and put it back

Checklist

  • Scoped to one concern. Unrelated cleanups are in their own commit or PR
  • Public-facing docs updated where behaviour changed (docs/, README.md)
  • No internal tracker numbers, post-mortem narrative, or internal infrastructure details in the diff
  • Files under packages/*/templates/** are LF
  • No secrets, tokens or real credentials, including in test fixtures

Not one concern on purpose, see Why. Each fix is its own commit, and the merges of #94, #96, #97 and #89 are kept as merge commits, so the history reads per topic.

Anything else

  • The two new auth e2e tests and the template edits are not covered by pnpm tsc; the scaffold e2e job in CI is their first run.
  • Strict status checks are only on in .github/rulesets/require-ci-to-pass.json. Someone with admin rights still needs to apply the ruleset and enable allow_update_branch (docs/repository-settings.md has the command).
  • setRuntimePublicDefault changes reach projects at the next bump of nuxt-ui, auth-verification, build-info or health; they were not bumped for it.
  • Known limits, deliberately left for later: battlestack add and remove do not generate migrations yet; --skip-install always skips db:generate; pull --dry-run lists no per-file preview; k8s files are staged, not merged.
  • Decisions from pull: merge into edited files, never overwrite local work #94 still open for the reviewer: the build-info footer separator changed from an em-dash to ·, and the runningLow strings in dashboard.json were reworded.
  • The rustfs image in CI stays on :latest: Dependabot does not update images in a workflow's services:, so a digest pin would go stale.

christian-at-sevenlab and others added 30 commits October 15, 2026 14:45
Projects scaffolded before the NUXT_RATE_LIMIT_DISABLED gate kept a bare
rateLimiter object in nuxt.config.ts, and update() never touched it, so e2e
runs from one loopback IP hit nuxt-security's 429s with no way to switch the
limiter off. update() now wraps an existing object literal in the gate,
keeping its values; an already-gated, disabled or nested value is left alone.
An upgrade of an older project overwrote local work in several ways: a
feature new to the project replaced an untracked file at the same path, the
Dockerfile and plugin-written workflows were rewritten without a drift
check, and every edited file was only staged as .new/.patch, where the
patch replaces the whole file with upstream.

Edited files are now merged three-way against the version battlestack last
wrote, with diff3 markers where both sides changed the same lines. Those
versions live in .battlestack/base/, keyed by sha256, with git history as
the source on a project's first merge. During pull and add the recorded
writers go through the same check, and after the feature pass pull restores
and merges any edited file a feature still wrote over, which also covers
plugins running their own copy of core.

pull requires a clean working tree (--allow-dirty), refuses to start while
markers from a previous pull remain, writes .battlestack/pull/REPORT.md
with what is left and how to handle it, and exits with code 2 when that is
not empty. A feature whose plugin did not load keeps its manifest record.

Also: shared:github drops records of the deploy workflows it no longer
ships, pull keeps a project's higher dependency floor, ports in committed
files depend only on the project name, a project-mode command outside a
project no longer starts a new project with that name, and generated
projects get a battlestack-upgrade skill with matching AGENTS.md and README
text.
- The overwrite check compares against HEAD, so under --allow-dirty it read
  uncommitted edits as feature output and restored HEAD over them. It now
  runs only when the tree was clean, and warns when it was skipped.
- When the check stages a file instead of merging, the feature's record goes
  back to its previous hash and no base is stored, so the next pull still
  merges against the old upstream.
- `add` reports what its merge-aware writes left (conflicts, staged files)
  in REPORT.md and exits with code 2, instead of printing plain success.
- `remove` of a feature whose plugin is not loaded drops just its record and
  leaves its files, now that pull keeps such records.
- The ledger fills in fields a plugin on another core version left out.
- The upgrade skill's marker example is indented, so a team edit to it can
  never trip the leftover-marker check, and the outside-a-project hint points
  at `battlestack migrate`, which lists the conversions on offer.
A pull that rewrites a file under server/database/schema/ wrote no migration,
and projects apply only committed SQL in production, so the change never
reached the production database (1.4.0's new index on sessions.user_id, for
one). Only scaffold ran db:generate. Pull now hashes the schema files before
updating and, when they changed, runs the same db:generate helper scaffold
uses. With --skip-install, or when it fails, it warns to run it by hand.
# Conflicts:
#	CHANGELOG.md
#	packages/cli/src/commands/pull.ts
#	packages/cli/test/pull.test.ts
…into fix/pull-fixes

# Conflicts:
#	CHANGELOG.md
#	packages/cli/src/commands/pull.ts
- Print the app's server.log whenever the job fails, not only when the
  server does not start.
- Bound each readiness probe with --connect-timeout 2 --max-time 5 so a
  hanging server cannot hold the job until the 30-minute timeout.
- Pin rustfs/rustfs to the digest that :latest resolved to on 2026-10-19.
- Rewrite the scaffold comments that predate the postgres and rustfs
  services and the e2e run.
- Turn on strict required status checks and document the
  allow_update_branch setting that goes with it. Also correct the check
  count in the docs (eight, not six).
drizzle-kit exits 0 when it stops at a rename prompt without a TTY or
cannot load the schema, so pull reported a migration that was never
written. Pull now compares the migrations folder before and after, does
not generate while a schema file waits for the user's merge, and puts
what is left to do in REPORT.md so the run exits with code 2.
…dding ours

- Match rel tokens case-insensitively, so `shortcut icon` and `Icon` count as
  the project's own branding.
- Also scan app/app.vue and app/layouts/**/*.vue for icon rels (useHead or a
  <link> tag). Pages, plugins and error.vue are not scanned.
- Report a note on pull when the template links are skipped.
- Document the detection scope in the generated README text and wireIcons JSDoc.
- Bump nuxt4:essentials to 1.2.4 (origin/main already has 1.2.3 from #96).
A file moving between features was deleted by its old owner when that
owner updated first, then written back by the new owner.
… sessionHeader

A runtimeConfig written as a variable or shorthand crashed pull with a
proxy trap error; it now fails with an error naming
nuxt.config.ts#runtimeConfig. Defaults under a non-literal nested value
(e.g. `session: baseSession`) were dropped silently; patchNuxtConfig now
warns with each path left out. An explicit `null` is kept as the
project's value, as the comment already claimed (magicast reads it back
as undefined, so presence is checked by key).

On pull, nuxt4:auth notes a `sessionHeader` that is not `false`: it is
kept, but a sealed session sent as a header skips the revocation check.
The auth e2e template now covers a revoked session and a live session
sent only as the header.
A pull note prints dimmed mid-scroll, and a report action would make
every pull of a project that keeps the header on exit as needing
attention. Also drops two JSDoc lines that restated the function.
nuxt4:storage patched users.ts after nuxt4:database recorded it, so every
database bump staged users.ts for a manual merge, and taking the staged
file dropped the avatar column. Pull also quoted a stack frame instead
of drizzle-kit's error when generation stopped.
The pre-patch users.ts storage projects recorded was never committed, so
the first pull after the variant split could only stage it.
… writeRecorded

writeGuarded classified against an empty owned set, so a file a project took over
with `battlestack own` was merged or staged on the next bump whenever its feature
wrote it via writeRecorded or copyTemplateDirRecorded instead of
updateFromTemplateDirs. It now reads the feature's `owned:` state, which pull
seeds from the manifest. Seeding also stops writeManifest from replacing a
feature's ownedByUser with just its structural files.
dropObsolete treated any existing file on a path an older shared:github shipped as
its own leftover, so a project's own production.yml, staging.yml or development.yml
was reported as edited and no longer managed. Only a path this feature recorded is
handled now. Recorded keys are matched separator-insensitively, both here and when
checking whether another feature tracks the path.
add rewrote .battlestack/pull/REPORT.md with only its own leftovers, which erased the
conflicts, staged files and migration steps a previous pull had listed. When a report
already exists, add now puts its own section above it instead of replacing it.
writeGuarded is exported, so the owned set a caller passes is now unioned with the
feature's owned state rather than dropped from the signature.
…repos

- Scope every git call to the project directory: `diff --relative`,
  `status -- .`, and `HEAD:./<rel>` reads, so a project in a subdirectory
  of a repository is repaired and preflighted against its own files.
- Read HEAD and history blobs with `cat-file --filters`, so eol and smudge
  conversion match the working-tree bytes the manifest hashes.
- findBaseInGit is async: one `git log --raw` for distinct blob ids, then
  one filtered read per blob until a hash matches. `cat-file --batch
  --filters` was not usable: its headers report the unfiltered size.
- repairOverwrites skips `--force` and `--overwrite`, and the conflict
  marker preflight no longer blocks them.
- A staged repair restores the writing feature's own start hash, resolved
  from fqid to bare id, under the key its map uses; it now writes a
  `.patch` next to the `.new`, as the report says.
- The report lists a restored file once; restores through writers whose
  report a feature discards are found by comparing against files missing
  at start. Per-feature lines include ledger merges, conflicts and staged
  files those writers recorded.
- Core: `hashContent`, `toPosixRel`, `recordedFiles(ctx)` and
  `trackersOf(ctx, rel)` replace local walks of the `files:*` state maps
  and duplicate sha256 helpers; `stageUpstream` and `isBinaryPath` are
  exported for staging outside templates.ts.
- pull.ts keeps per-run report lists in a local, not ctx.state, and drops
  a dead `--skills-only` check.
# Conflicts:
#	packages/cli/src/commands/pull.ts
…ding git

With core.autocrlf, a file battlestack wrote as LF and committed came back
as CRLF from cat-file --filters, so on Windows the git-history merge base
was never found and pristine files looked edited.
Pull restores a file the way git restore does, which is CRLF on a Windows
runner with core.autocrlf.
It is a local merge cache. New projects ignore it, and a project that
committed it gets a step in REPORT.md to stop tracking it.
…st added

A project scaffolded before the policy can lock packages younger than it
allows, and pnpm then refuses every script. Pull skips lint and
db:generate and lists the lockfile rebuild in REPORT.md.
…age project

Takes the drop out of the migration and drizzle's snapshots, and adds a
migration that puts the column back where the drop already ran, so avatars
survive wherever it has not been deployed yet.
@joris-sevenlab
joris-sevenlab requested a review from koentv October 20, 2026 07:53
landing-shell and build-info emit with `this.id`, so the template-dir
regex missed them and their changes needed no bump. landing-shell's layout
changed in this PR without one, so existing projects would never get it.

@JoeyHoutenbos JoeyHoutenbos left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 6a6bf7d. pnpm tsc, pnpm test (1179 passed) and bump:check are green, and replaying a real 1.3 → 1.4 project pull with this build keeps the moved auth e2e test and the project name. Docs, changelog and the upgrade skill match the code. Four things need fixing before this merges, though.

Blockers

1. A file moving to an earlier-stage feature is deleted and never comes back (packages/core/src/utils/templates.ts, deferredToLater in sharedRelOwnership + removeQueuedObsoleteFiles)
The mirror of the case #100 was about. Old owner A (later stage) ships moved.ts at v1; at v1.1 it moves to B (earlier stage). B still sees the rel in A's seeded files: map, so it defers to the "later owner" and neither writes nor records it. A then drops it as obsolete, nobody tracks it, and the queued deletion removes it. Pull exits 0, and a second pull does not restore it.
Repro: in packages/cli/test/pull.test.ts "a file moving to a later feature", swap the stages in setup (from = STAGE.AUTH, to = STAGE.NAMING); "keeps a pristine file and hands its record to the new owner" fails with ENOENT. Suggest: ignore a later feature's rel in deferredToLater when that feature's new version no longer ships it, and add the swapped-stage variants as tests.

2. The avatar_key repair rewrites a migration that may already be applied (packages/preset-nuxt4/src/utils/migrations.ts, repairDroppedAvatarColumn)
The project's own templates/database/tools/migrate.mjs skips a migration only when its sha256 is in __drizzle_migrations. After the repair the file has a new hash, so the next deploy runs it again. With the PR's own fixture (DROP COLUMN "avatar_key" + CREATE INDEX "users_email_idx") that is "relation already exists", rollback, failed deploy. Only a migration whose sole statement was the drop survives (SELECT 1;). Safer: leave applied SQL alone and only add the ADD COLUMN IF NOT EXISTS migration, or report the drop and let the user decide.

3. With npm, the repair overwrites a real schema migration (migrations.ts, the generateMigration(ctx, ['--custom', '--name=restore_avatar_key']) call and the write of RESTORE_SQL)
npm run db:generate --custom --name=… hands the flags to npm, not the script (no --), so drizzle-kit runs a plain generate. If the pull also changed the schema, that writes the real migration, exactly one new file appears, and it is overwritten with the avatar SQL. The snapshot already holds the real change, so the next generate says "No schema changes" and the change never reaches the database. The fallback hint gives npm users the same command. Suggest: put -- before extra args, and only write RESTORE_SQL into an empty file named *_restore_avatar_key.sql.

4. On a core.autocrlf=true checkout an untouched file becomes one whole-file conflict (templates.ts, classifyForUpdate / mergeInPlace)
The working copy is CRLF, the recorded hash is of the LF bytes, so the file counts as drifted; findBase finds the LF blob and git merge-file turns every line into one diff3 conflict, written in place. Before this PR it was staged untouched. repairOverwrites already compares the --filters form; the drift check needs the same. The test "finds a file battlestack wrote as LF under core.autocrlf" never re-checks the file out, so the working copy stays LF and this path is not exercised.

Should fix

  • Owned files get merged into via a second tracker (writeGuarded → ownedRelsFor(ctx, featureId)): battlestack own records ownership on the first tracker only, so a second feature writing the same rel (e.g. app/app.config.ts, nuxt-ui and landing-shell) merges into the owned file. Check ownership across every tracker.
  • Merging only works on the machine that ran the previous merge: after a merge the manifest records the upstream hash, but those bytes live only in .battlestack/base/, which is now gitignored. On another clone or in CI findBaseInGit finds no blob with that hash, so every later pull of an edited file stages instead of merging.
  • A later pull wipes unfinished work from REPORT.md and exits 0: pull stages a.ts (exit 2), a same-version re-run skips every feature and rewrites the report to "Nothing needs attention" while .battlestack/pull/a.ts.new is still there. Same for the --skip-install "run db:generate" action: the next pull sees no schema change and the reminder is gone. Carry unresolved items forward, or leave the report alone when nothing ran.
  • The drop match ignores the table (AVATAR_COLUMN and the statement filter): a deliberate ALTER TABLE "teams" DROP COLUMN "avatar_key" gets rewritten too, and the users.ts check is a plain includes('avatar_key'). With prefix: 'timestamp', parseInt of the file name makes the snapshot repair skip every snapshot while the SQL drop is still removed.
  • The new e2e test "a session revoked from another device is refused…" cannot fail on revocation: with sessionHeader: false every header-only request is 401, as the next test asserts. It passes with revocation broken. Send the cookie and assert the revocation check refuses it, or drop it.
  • runtimeConfig: { public: pub } aborts the feature's pull (nuxt-config.ts / ts-file.ts) with "runtimeConfig.public is not an object literal", while any other non-literal nested parent is a warning. Treat public the same, or document it.

Nits

  • An obsolete, locally edited file gives "Nothing needs attention" and exit 0 alongside a "No longer managed" section asking for a decision.
  • add's warning can read "merging into 0 existing file(s)" when only actions are left.
  • The sessionHeader warning runs on update only, not on add.
  • rateLimiter: { … } as any / satisfies X stays ungated without a note.
  • Icon detection misses :rel="'icon'" and unquoted rel=icon.
  • The lockfile check: execFile('pnpm') cannot start pnpm.cmd on Windows, and offline or ERR_PNPM_OUTDATED_LOCKFILE both read as "not blocked".
  • Migration detection reads only server/database/migrations, so a project with another out gets "wrote no migration" when one was written.
  • pull-safety.test.ts "pull with --allow-dirty skips the overwrite check" never calls repairOverwrites or pull, so it passes with the guard removed; the merge outcome of repairOverwrites has no test.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants