fix(history): harden GC and delete follow-ups, query-aware Home/End

docs/prompt-history.md

@@ -44,7 +44,9 @@ Env values are trimmed and case-insensitive. While the variable forces a
 value, the Customize rows show `env override` and the preview says the
 variable overrides the preference; a selection is still saved and takes effect
 once the variable stops forcing a value. A malformed preference file is
-reported and never rewritten by Customize: fix or remove it by hand.
+reported and never rewritten by Customize: fix or remove it by hand. The
+history selector names that file and says the preference is invalid or
+unreadable, instead of asking you to turn capture on in Customize.
 
 - The check runs per prompt: changing the preference or the variable stops or
   starts new captures immediately.
@@ -60,7 +62,10 @@ after the extension loads, or at its first delivered prompt if that comes
 first. The selector reads the store but does not initiate import.
 With capture off, both capture and the selector leave the store untouched.
 Failed migration reads can be retried on a later session; untrusted deletion
-records defer transcript bootstrap until they can be read safely.
+records defer transcript bootstrap until they can be read safely. Migration
+holds the `history-global.jsonl.migration-lock` directory while it runs; if a
+pi process dies in that window, the lock stays and migration is skipped until
+you remove that directory by hand.
 
 An import creates **new searchable copies** under `~/.pi/agent/history`. The
 source transcripts stay untouched and read-only. Turning capture off again
@@ -113,6 +118,18 @@ rm -rf ~/.pi/agent/history            # whole store
 rm -rf ~/.pi/agent/history/projects/<hash>   # one project (see registry.json)
 ```
 
+## Selector keys
+
+`Home` and `End` depend on the search box:
+
+- **Search box empty:** they move the list selection. `Home` selects the
+  newest prompt; `End` loads every remaining prompt and selects the oldest.
+- **Any text in the search box** (whitespace included): they move the search
+  caret to the start or end of the query, like the other editing keys. They
+  never move the list, and `End` does not load the remaining prompts.
+
+As with other editing keys, the list selection returns to the first match.
+
 ## Delete
 
 The selector's delete key (`ctrl+shift+backspace`) is a two-step y/n
@@ -135,7 +152,8 @@ prompt is affected — prompts that merely share a beginning stay.
    `history-global.jsonl`. Each affected file is rewritten atomically (temp
    file + rename). Files are never removed, even when they end up empty.
    Lines that another pi instance appends while a file is being rewritten
-   are carried over into the new file.
+   are carried over into the new file; if that append fails, they are kept
+   in a sibling `<name>.carry-<pid>-<ts>.jsonl` store file instead.
 2. **A tombstone is written** to `hidden.json`, so the prompt stays hidden
    everywhere the selector reads, and a later transcript bootstrap does not
    import it again. The session transcripts themselves are never modified.
@@ -175,6 +193,13 @@ Failures surface an error notification and never report a clean delete:
 - If the tombstone write fails after store copies were removed, the prompt
   may reappear from session transcripts ("Deleted from the store, but
   hiding failed — the prompt may reappear from session transcripts.").
+- If both happen — some files cannot be rewritten and the tombstone write
+  fails — a single notice says so and never claims the prompt is hidden
+  ("Some history files could not be rewritten and hiding failed — the
+  prompt may reappear from those files or from session transcripts.").
+
+The notice is chosen after the tombstone write, so it always describes the
+final state.
 
 `hidden.json` fails closed: if it exists but cannot be trusted (unreadable,
 corrupt, or not an array), history is blocked with a recovery warning
@@ -200,6 +225,10 @@ in total. Then:
   `compact-<pid>-<ts>.jsonl`, which is written completely (temp file + rename)
   before any merged file is removed. Earlier compact files are merged again
   like any other file.
+- Compaction runs only when at least **two** files can be merged. Merging a
+  single file cannot reduce the file count, so a directory that stays above
+  a threshold after compaction (for example, because compaction never
+  lowers the entry count) is not rewritten again at every shutdown.
 - Never merged: `seed.jsonl` (while it exists, the transcript import does not
   run again) and the capture file of the session that is shutting down.
   `history-global.jsonl` sits outside the project directories and is never
@@ -211,9 +240,11 @@ in total. Then:
 
 Other pi instances may still be appending to the files being merged. Each
 file is renamed to a claim name before it is read (`<name>.gc-<pid>-<ts>.jsonl`),
-so a later append by path starts a fresh file under the original name, and
-bytes written to the claimed file after it was read are moved into the
-compact file once the claim is removed.
+so a later append by path starts a fresh file under the original name.
+Complete lines written to the claimed file after it was read are appended to
+the compact file **before** the claim is removed; if that append fails, the
+claim stays on disk with every byte. The claim is read once more after its
+removal for a write that landed in between.
 
 Failures never lose prompts: a file that cannot be read is left untouched,
 and if the compact file cannot be written, the claimed files stay on disk and

extensions/history/index.ts

@@ -42,6 +42,7 @@ import { gentlePiConfigHome } from "../../lib/agent-home.ts";
 import {
   historyCaptureEnabled,
   historyCaptureEnvOverride,
+  resolveHistoryCapturePolicy,
 } from "../../lib/history-capture-policy.ts";
 import { hidePrompt } from "./hide-prompts.ts";
 import {
@@ -68,7 +69,6 @@ import {
   deleteConfirmFooterText,
   deleteConfirmStep,
   deletionActionsFor,
-  EDITOR_HIDE_FAILED_TEXT,
   filterPrompts,
   getVisiblePromptRecords,
   initialLoadedCount,
@@ -81,6 +81,7 @@ import {
   shouldGrowWindow,
   STORE_DELETE_FAILED_TEXT,
   storeDeleteFollowUp,
+  storeDeleteNotice,
   withExpandedHistoryGlobals,
   type PiHistoryGlobals,
   type PromptEntry,
@@ -151,11 +152,23 @@ export function captureEnabled(
   return historyCaptureEnabled({ env, gentlePiConfigHome: configHome });
 }
 
-/** Why capture is off, naming the control that actually decides it. */
-function captureDisabledMessage(env: NodeJS.ProcessEnv): string {
-  return historyCaptureEnvOverride(env) === "off"
-    ? "Prompt history is disabled by GENTLE_PI_HISTORY_CAPTURE, which overrides the Gentle → Customize → History preference."
-    : "Prompt history is disabled. Turn on \"Prompt history capture\" in Gentle → Customize → History, or set GENTLE_PI_HISTORY_CAPTURE=1.";
+/**
+ * Why capture is off, naming the control that actually decides it. A
+ * malformed or unreadable preference is reported as such: Customize refuses
+ * to rewrite it, so "turn it on in Customize" would not help.
+ */
+function captureDisabledMessage(
+  env: NodeJS.ProcessEnv,
+  configHome: string,
+): string {
+  if (historyCaptureEnvOverride(env) === "off") {
+    return "Prompt history is disabled by GENTLE_PI_HISTORY_CAPTURE, which overrides the Gentle → Customize → History preference.";
+  }
+  const preference = resolveHistoryCapturePolicy({ gentlePiConfigHome: configHome });
+  if (preference.malformed) {
+    return `Prompt history is disabled because the Gentle → Customize → History preference is invalid or unreadable: ${preference.globalFile}. Fix or remove that file, or set GENTLE_PI_HISTORY_CAPTURE=1.`;
+  }
+  return "Prompt history is disabled. Turn on \"Prompt history capture\" in Gentle → Customize → History, or set GENTLE_PI_HISTORY_CAPTURE=1.";
 }
 
 // ---------------------------------------------------------------------------
@@ -362,12 +375,14 @@ class PromptHistorySelector extends Container implements Focusable {
       match: (_d, kb) => kb.matches(_d, "tui.select.cancel"),
       handler: () => this.onCancel(),
     },
+    // Home/End jump the list only while the search box is empty; with any
+    // text they fall through to the search input and move its caret.
     {
-      match: (d, _kb) => matchesKey(d, "home"),
+      match: (d, _kb) => matchesKey(d, "home") && this.listOwnsHomeEnd(),
       handler: () => this.jumpToFirst(),
     },
     {
-      match: (d, _kb) => matchesKey(d, "end"),
+      match: (d, _kb) => matchesKey(d, "end") && this.listOwnsHomeEnd(),
       handler: () => this.jumpToLast(),
     },
     {
@@ -688,12 +703,12 @@ class PromptHistorySelector extends Container implements Focusable {
     // actions via the pure planner over the injected store root/cwd.
     const actions = deletionActionsFor(selected.source ?? "editor");
 
+    let sweep: SweepResult | null = null;
     if (actions.deleteFromEditorStore) {
       // Store path: physically remove EVERY copy from the JSONL store, one
       // atomic rewrite per file. A thrown store failure is contained here
       // (PR #1393): toast + abort — nothing was removed and no tombstone is
       // written, so the delete never lies about state.
-      let sweep: SweepResult;
       try {
         sweep =
           this.scope === "global"
@@ -704,9 +719,9 @@ class PromptHistorySelector extends Container implements Focusable {
         return;
       }
       // Files that could not be read or rewritten may still hold a copy:
-      // say so, and still write the tombstone that hides them.
+      // still write the tombstone that hides them; the notice waits for
+      // the hide result below.
       const followUp = storeDeleteFollowUp(sweep);
-      if (followUp.notice) this.onNotify?.(followUp.notice, "error");
       if (!followUp.proceed) return;
     }
 
@@ -721,7 +736,12 @@ class PromptHistorySelector extends Container implements Focusable {
         this.onNotify?.(hide.message, "error");
         return;
       }
-      this.onNotify?.(EDITOR_HIDE_FAILED_TEXT, "error");
+    }
+    // One notice for the editor path, stating both halves: a partial sweep
+    // only says "hidden" when the tombstone was actually written.
+    if (sweep) {
+      const notice = storeDeleteNotice(sweep, hide.status === "error");
+      if (notice) this.onNotify?.(notice, "error");
     }
     // Remove from the master records array so a subsequent filter doesn't
     // bring it back.
@@ -870,6 +890,15 @@ class PromptHistorySelector extends Container implements Focusable {
     this.rebuildPreview();
   }
 
+  /**
+   * An empty search box has no caret to move, so Home/End keep their list
+   * jumps; any text (whitespace included) routes them to the caret, and End
+   * then never loads the whole list.
+   */
+  private listOwnsHomeEnd(): boolean {
+    return this.searchInput.getValue().length === 0;
+  }
+
   private jumpToFirst(): void {
     if (this.filteredRecords.length === 0) return;
     this.selectedIndex = 0;
@@ -1130,7 +1159,7 @@ function createOpenFlow(
     // Capture gate (#1390) FIRST: with capture off the selector is a no-op —
     // no registry writes, no writer init, no store reads, no overlay.
     if (!captureEnabled(env, configHome)) {
-      ctx.ui.notify(captureDisabledMessage(env), "warning");
+      ctx.ui.notify(captureDisabledMessage(env, configHome), "warning");
       return;
     }
 

extensions/history/selector-helpers.ts

@@ -331,6 +331,13 @@ export const EDITOR_HIDE_FAILED_TEXT =
 export const STORE_DELETE_PARTIAL_TEXT =
   "Some history files could not be rewritten; the prompt is hidden, but copies may remain on disk.";
 
+/**
+ * Toast copy when some store files could not be rewritten AND the tombstone
+ * write failed: nothing hides the copies that may remain on disk.
+ */
+export const STORE_DELETE_PARTIAL_HIDE_FAILED_TEXT =
+  "Some history files could not be rewritten and hiding failed — the prompt may reappear from those files or from session transcripts.";
+
 /** The counts a scope delete reports (structural twin of store's SweepResult). */
 export interface StoreSweepCounts {
   filesAffected: number;
@@ -353,6 +360,23 @@ export function storeDeleteFollowUp(
   return { proceed: counts.removed > 0 };
 }
 
+/**
+ * The one error notice of an editor-path delete, chosen AFTER the tombstone
+ * write so it never claims the prompt is hidden when hiding failed. No
+ * notice for a clean sweep with a written tombstone.
+ */
+export function storeDeleteNotice(
+  counts: StoreSweepCounts,
+  hideFailed: boolean,
+): string | undefined {
+  if (counts.failed > 0) {
+    return hideFailed
+      ? STORE_DELETE_PARTIAL_HIDE_FAILED_TEXT
+      : STORE_DELETE_PARTIAL_TEXT;
+  }
+  return hideFailed ? EDITOR_HIDE_FAILED_TEXT : undefined;
+}
+
 export function getVisiblePromptRecords(
   records: PromptRecord[],
   selectedIndex: number,

extensions/history/store.ts

@@ -474,8 +474,11 @@ function readFrom(fd: number, position: number): Buffer {
  * the descriptor kept open on the replaced inode supplies everything
  * appended after the read — including a torn last line — which is carried
  * over verbatim into the new file after the rename. Kept lines are copied
- * byte-for-byte. On failure the tmp file is removed, the original stays in
- * place unless the rename already happened, and the error is rethrown.
+ * byte-for-byte. If that carry-over append fails, the raced bytes are
+ * written to a sibling `<file>.carry-<pid>-<ts>.jsonl` store file instead
+ * of vanishing with the replaced inode. On failure the tmp file is removed,
+ * the original stays in place unless the rename already happened, and the
+ * error is rethrown.
  */
 function sweepFile(file: string, key: string): number {
   const fd = fs.openSync(file, "r");
@@ -502,7 +505,7 @@ function sweepFile(file: string, key: string): number {
     tmp = null;
     // Lines another instance appended to the replaced inode since the read.
     const carried = readFrom(fd, complete);
-    if (carried.length > 0) fs.appendFileSync(file, carried);
+    if (carried.length > 0) carryIntoRewrite(file, carried);
     return removed;
   } catch (error) {
     if (tmp !== null) {
@@ -518,6 +521,25 @@ function sweepFile(file: string, key: string): number {
   }
 }
 
+/**
+ * Append raced bytes to the rewritten file; if that fails, keep them in a
+ * sibling store file (read like any other) so they are never lost. A torn
+ * last line is completed so the sibling stays parseable. Throws only when
+ * both writes fail.
+ */
+function carryIntoRewrite(file: string, carried: Buffer): void {
+  try {
+    fs.appendFileSync(file, carried);
+  } catch {
+    const text = carried.toString("utf8");
+    fs.writeFileSync(
+      `${file}.carry-${process.pid}-${Date.now()}.jsonl`,
+      text.endsWith("\n") ? text : `${text}\n`,
+      { flag: "wx" },
+    );
+  }
+}
+
 /**
  * Remove every line whose prompt identity matches `text` from each file in
  * `files`, one atomic rewrite per affected file (see sweepFile). Files whose
@@ -834,7 +856,9 @@ export interface GcOptions {
  * Compaction consolidates files: it merges the oldest store files of ONE
  * project into a single `compact-<pid>-<ts>.jsonl` and removes the merged
  * originals. It is not a retention limit — every visible prompt is copied;
- * only tombstoned prompts (already deleted by the user) are dropped.
+ * only tombstoned prompts (already deleted by the user) are dropped. It
+ * runs only when at least two files can be merged, so a repeated GC over
+ * an already compacted dir rewrites nothing.
  *
  * Never merged: `seed.jsonl` (its presence is the bootstrap gate, so
  * removing it would re-seed deleted prompts from transcripts), the files
@@ -881,7 +905,10 @@ export function gcProjectDir(
       .slice(opts.keepNewest ?? GC_KEEP_NEWEST)
       .reverse() // oldest first: the merged output is chronological
       .map((c) => c.file);
-    if (tail.length === 0) return none;
+    // Merging a single file cannot reduce the file count: rewriting it
+    // would repeat on every shutdown while a threshold stays exceeded
+    // (compaction never lowers the entry count), so GC stays idempotent.
+    if (tail.length < 2) return none;
     return compactFiles(dir, tail, hidden.keys);
   } catch {
     return none;
@@ -941,9 +968,11 @@ function claimFile(file: string): ClaimedFile | null {
  * atomically (tmp + rename) BEFORE any claim is removed. A failure before
  * the compact file lands leaves every claim in place (still a readable
  * store file); a claim that cannot be removed survives as a harmless
- * duplicate (drains dedupe by identity). After each removal the claim's
- * descriptor is drained once more and any late bytes are appended to the
- * compact file (or written back under the claim name if that fails).
+ * duplicate (drains dedupe by identity). Complete lines that reached a
+ * claim after its read are appended to the compact file BEFORE the claim
+ * is removed — if that fails, the claim stays with every byte. After the
+ * removal the descriptor is drained once more for a write that landed in
+ * between (written back under the claim name if its append fails).
  */
 function compactFiles(
   dir: string,
@@ -979,6 +1008,7 @@ function compactFiles(
   }
   for (const c of claimed) {
     try {
+      carryCompleteLines(c, compact, hidden);
       fs.rmSync(c.claim);
     } catch {
       // the claim keeps every byte; it is merged again by a later GC
@@ -990,7 +1020,27 @@ function compactFiles(
   return { compacted: true, merged: claimed.length };
 }
 
-/** Move bytes that reached a removed claim after it was read. */
+/**
+ * Append the complete lines that reached a still-present claim after it
+ * was read, advancing its cursor; a torn last line waits for the final
+ * drain. Throws when the append fails, so the caller keeps the claim.
+ */
+function carryCompleteLines(
+  c: ClaimedFile,
+  compact: string,
+  hidden: ReadonlySet<string>,
+): void {
+  const late = readFrom(c.fd, c.consumed);
+  const complete = late.lastIndexOf(NEWLINE) + 1;
+  if (complete === 0) return;
+  fs.appendFileSync(
+    compact,
+    keepVisibleLines(late.subarray(0, complete).toString("utf8"), hidden),
+  );
+  c.consumed += complete;
+}
+
+/** Move bytes that reached a claim between its last carry and its removal. */
 function carryOver(
   c: ClaimedFile,
   compact: string,