feat(research): multi-day runs — no turn cap without a gate, mid-send saves, dedup, smaller images, append tool

apps/docs/src/content/docs/integrations/chrome.mdx

@@ -70,6 +70,10 @@ Research sessions are meant to run for hours, so a single model hiccup doesn't e
 - **Repetition loops.** The retry is forced to make a tool call, at a higher temperature, and the repeated lines are removed from the history first. The loop budget counts only *consecutive* loops and starts over whenever a tool call succeeds. If three loops in a row get through, the context is compacted and tried once more before the session stops.
 - **"Let me record it." with no tool call.** When there's no gate, a reply that announces its next step without taking it gets a nudge to make the call. It is not treated as the final answer.
 - **Compaction.** The summary request is bounded. If it still fails, your requests are kept verbatim and the older tool output is dropped, so the context always shrinks.
+- **No turn cap.** Without a live gate a single message can run for days. Gated builds keep the 1000-turn backstop. `TSFORGE_MAX_TURNS` sets a cap anyway, or `0` for unlimited everywhere.
+- **Saved as it goes.** The session is saved every 40 turns, not only when a message finishes, so `--continue` picks up close to where a crash or reboot stopped it.
+- **Data files.** `append` adds records to a data file (`.jsonl`, CSV, logs) without an edit anchor. JSONL lines are checked, so a malformed record never gets written. Code and config files are refused.
+- **No TypeScript rules on prose.** The streaming TypeScript rules (like "no `as` cast") stay off until the workspace has code.
 
 ## Site plugins
 

apps/docs/src/content/docs/integrations/reddit.mdx

@@ -31,11 +31,13 @@ notes/guitar-wiring-issues/
   assets/<post-id>/1.jpg …   images from posts and comments
 ```
 
-`sources.md` also stops duplicate reads: a run that lasts days restarts many times and never re-reads a thread it already logged.
+`sources.md` also stops duplicate reads. `reddit_thread` refuses a thread already logged for the topic (pass `force: true` to read it again) and `reddit_search` marks it `already read`, so a run that lasts days and restarts many times never processes a thread twice.
+
+Resuming research you began another way, like an earlier crawl or a URL list? Have the agent call `reddit_mark_read` with the threads it has already covered, and they're skipped from then on.
 
 ## Images
 
-When a thread is read, its images are downloaded: the post's image or gallery (in order), and images linked or embedded in comments (`i.redd.it`, `preview.redd.it`, `external-preview.redd.it`, `i.imgur.com`). The limit is 30 per thread and 10 MB each. They appear inline in what the agent reads, for example `[image 3: notes/…/assets/abc123/3.jpg — comment by u/someone]`.
+When a thread is read, its images are downloaded: the post's image or gallery (in order), and images linked or embedded in comments (`i.redd.it`, `preview.redd.it`, `external-preview.redd.it`, `i.imgur.com`). Up to 20 post images and 10 comment images per thread are saved, 10 MB each at most, using Reddit's 1080px rendition when it has one. That's about a fifth of the size of the originals and still sharp enough for a wiring photo. Images past the caps stay as links. They appear inline in what the agent reads, for example `[image 3: notes/…/assets/abc123/3.jpg — comment by u/someone]`.
 
 The agent looks at an image with `read_image` (your vision model) only when it carries the insight, such as a wiring diagram or a photo of the problem. Video (`v.redd.it`) is linked, never downloaded.
 
@@ -46,7 +48,10 @@ The agent looks at an image with `read_image` (your vision model) only when it c
 | `reddit_subreddits` | Finds communities for a domain: name, members, description. |
 | `reddit_search` | Searches site-wide or inside one subreddit. Returns one line per post, with `after` for the next page. |
 | `reddit_listing` | Browses a subreddit: hot, new, top or rising. |
-| `reddit_thread` | Reads a whole thread in chunks, saves its images, and logs it in `sources.md`. |
+| `reddit_thread` | Reads a whole thread in chunks, saves its images, and logs it in `sources.md`. Refuses threads already logged unless `force: true`. |
+| `reddit_mark_read` | Logs threads already covered another way, so they're skipped. |
+
+For structured datasets (one JSON record per thread, CSV rows), the agent uses `append`, which adds lines to a data file without an edit anchor.
 
 ## Safety and limits
 

apps/docs/src/content/docs/reference/flags.mdx

@@ -119,6 +119,7 @@ Extra gate steps (default off; each skips cleanly when nothing applies). See [Ho
 | `TSFORGE_THINKING_BUDGET` | unset | reasoning token cap |
 | `TSFORGE_CONTEXT_WINDOW` | `32768` | context window |
 | `TSFORGE_COMPACT_AT` | `0.8` | auto-compact threshold. The summary call is bounded (≤60k chars in, 2048 tokens out, 3 min); if it still fails, tsforge keeps your requests verbatim and drops the older tool output instead of retrying forever |
+| `TSFORGE_MAX_TURNS` | unset | turn cap per message. Unset: research/chat sessions (no live gate) are uncapped, gated builds and plan mode stop at 1000. A number caps every session; `0` means unlimited |
 
 ## Judge (eval scripts)
 

packages/core/ARCHITECTURE.md

@@ -2,7 +2,7 @@
 
 <!-- Generated by `bun run arch:build` from packages/core/src. Do not edit by hand. -->
 
-Derived from `packages/core/src`: **32 subsystems**, **723 files**, **140k lines**, **153 cross-subsystem edges**.
+Derived from `packages/core/src`: **32 subsystems**, **724 files**, **141k lines**, **153 cross-subsystem edges**.
 
 
 This page is the exhaustive record: every subsystem, every cross-subsystem edge, and
@@ -13,7 +13,7 @@ inventory, see the hand-drawn map on [Internals](/internals/).
 
 | Subsystem | Purpose | Tier | Files | Lines | Fan-in | Fan-out |
 | --- | --- | --- | --- | --- | --- | --- |
-| `loop` | The drive-to-green engine: turns, tools, gate settling, steering, adapters | core | 174 | 46k | 8 | 24 |
+| `loop` | The drive-to-green engine: turns, tools, gate settling, steering, adapters | core | 175 | 47k | 8 | 24 |
 | `rule-packs` | The ESLint rule packs the gate enforces, grouped by stack | core | 191 | 23k | 3 | 3 |
 | `render` | Terminal UI — status bar, menus, wizards, markdown, diffs, spinners | core | 45 | 11k | 7 | 7 |
 | `cli` | Argument parsing, the interactive REPL, and per-mode wiring | core | 27 | 9k | 2 | 20 |
@@ -25,8 +25,8 @@ inventory, see the hand-drawn map on [Internals](/internals/).
 | `scaffold` | Stands up a new project from an archetype and configures its gate | optional | 20 | 3k | 3 | 3 |
 | `(root)` | CLI entry, model registry, session persistence — the loose files in src/ | core | 6 | 3k | 6 | 16 |
 | `gate` | Composes and runs the deterministic gate: linter, stages, tool paths | core | 16 | 3k | 5 | 8 |
+| `site-plugins` | Built-in site experts (Reddit…) that read a site's structured data through the Chrome bridge | optional | 18 | 3k | 1 | 3 |
 | `editor` | The terminal input-line editor behind the REPL prompt | core | 10 | 2k | 2 | 2 |
-| `site-plugins` | Built-in site experts (Reddit…) that read a site's structured data through the Chrome bridge | optional | 18 | 2k | 1 | 3 |
 | `reviewers` | Independent review panel that grades a change before it is trusted | optional | 9 | 2k | 1 | 3 |
 | `eval` | Run scoring, failure classification, and the quality judge | optional | 10 | 2k | 4 | 4 |
 | `lib` | Shared primitives — fs, json, guards, scope globs, SSRF checks, clipboard | core | 21 | 2k | 26 | 0 |
@@ -103,7 +103,7 @@ Async functions returning an exit code, declared under the CLI — the commands.
 | `main` | `cli.ts:799` |
 | `mapMode` | `cli.ts:518` |
 | `recipesMode` | `cli.ts:537` |
-| `repl` | `cli/repl.ts:1149` |
+| `repl` | `cli/repl.ts:1160` |
 | `reviewMode` | `cli.ts:201` |
 | `runOnce` | `cli.ts:103` |
 | `runTraceCommand` | `cli/repl-commands.ts:160` |

packages/core/src/agent/agent.constants.ts

@@ -56,7 +56,9 @@ export const TOOL_NAME = {
   redditThread: "reddit_thread",
   redditListing: "reddit_listing",
   redditSubreddits: "reddit_subreddits",
+  redditMarkRead: "reddit_mark_read",
   note: "note",
+  append: "append",
   script: "script",
   spawnAgent: "spawn_agent",
   readImage: "read_image",
@@ -161,9 +163,11 @@ export const TOOL_SPECS: Readonly<Record<ToolName, IToolSpec>> = {
   [TOOL_NAME.redditThread]: { readOnly: true, scriptExposable: false },
   [TOOL_NAME.redditListing]: { readOnly: true, scriptExposable: false },
   [TOOL_NAME.redditSubreddits]: { readOnly: true, scriptExposable: false },
+  [TOOL_NAME.redditMarkRead]: { readOnly: false, scriptExposable: false },
   // `note` appends to ./notes/<topic>.md — a disk write (withheld in plan mode),
   // but outside the code scope/write-guard: notes are research output, not code.
   [TOOL_NAME.note]: { readOnly: false, scriptExposable: false },
+  [TOOL_NAME.append]: { readOnly: false, scriptExposable: false },
   // `script` mutates (it can call edit/create) and must never call itself.
   [TOOL_NAME.script]: { readOnly: false, scriptExposable: false },
   // Delegation is itself read-only (the orchestrator only receives findings;
@@ -614,6 +618,30 @@ export const NOTE_TOOL = {
   },
 };
 
+export const APPEND_TOOL = {
+  type: "function",
+  function: {
+    name: TOOL_NAME.append,
+    description:
+      "Append lines to the END of a data file in the workspace (a .jsonl record, a CSV row, a log line) — no anchor, no rewrite. Creates the file if missing. For .jsonl every line must be valid JSON or nothing is written. Not for code or config: use edit/create for those.",
+    parameters: {
+      type: "object",
+      properties: {
+        file: {
+          type: "string",
+          description: "workspace-relative path, e.g. 'data/results.jsonl'",
+        },
+        text: {
+          type: "string",
+          description:
+            "the line(s) to append; a trailing newline is added if missing",
+        },
+      },
+      required: ["file", "text"],
+    },
+  },
+};
+
 /** A stable marker so the browser guidance is appended to the system prompt once. */
 export const BROWSER_MARKER = "## Researching in the user's browser";
 

packages/core/src/cli/logging.ts

@@ -140,6 +140,22 @@ export function makeReporter(
   return withLedger(render, logFile, runId, sessionId);
 }
 
+/** Wrap a reporter so `onCheckpoint` runs on every checkpoint event. The REPL
+ *  saves the session there: a research send can run for days, and saving only
+ *  when a send ends would lose the whole conversation to a crash or reboot. */
+export function withCheckpointHook(
+  report: Reporter,
+  onCheckpoint: () => void
+): Reporter {
+  return (event) => {
+    report(event);
+
+    if (event.kind === "checkpoint") {
+      onCheckpoint();
+    }
+  };
+}
+
 /** Resolve the run-log file when `--log` is set: an auto-named, timestamped JSONL
  *  under ~/.tsforge/logs/ (created if needed), so logs are always in one findable
  *  place and you never specify a path. Empty string = logging off. */

packages/core/src/cli/repl.ts

@@ -148,6 +148,7 @@ import {
   spinner,
   outputRouter,
   makeReporter,
+  withCheckpointHook,
   resolveLogPath,
   observeEvents,
 } from "./logging";
@@ -781,6 +782,8 @@ async function initReplSession(args: ICliArgs): Promise<{
   gateLabel: string;
   logFile: string;
   report: Reporter;
+  /** Bind what runs on each checkpoint event (the REPL saves the session). */
+  setCheckpointHandler: (fn: () => void) => void;
   resumed: ISessionRecord | null;
   files: string[];
   activeModelEntry: IModelEntry;
@@ -861,7 +864,12 @@ async function initReplSession(args: ICliArgs): Promise<{
     envNumber("TSFORGE_CONTEXT_WINDOW") ??
     (await detectContextWindow(provider.config)) ??
     32_768;
-  const report = makeReporter(logFile, id, id);
+  // Checkpoints (every N turns of a long send) save the session; the hook is
+  // bound once `persist` exists below.
+  let onCheckpoint = (): void => undefined;
+  const report = withCheckpointHook(makeReporter(logFile, id, id), () => {
+    onCheckpoint();
+  });
   const profile = resolveCliProfile(args.profile);
   const gatedBuild = autoGate !== undefined || accept.length > 0;
   const config = {
@@ -941,6 +949,9 @@ async function initReplSession(args: ICliArgs): Promise<{
     gateLabel,
     logFile,
     report,
+    setCheckpointHandler: (fn) => {
+      onCheckpoint = fn;
+    },
     resumed,
     files,
     activeModelEntry: activeModel.entry,
@@ -1163,6 +1174,7 @@ export async function repl(args: ICliArgs): Promise<number> {
     gateLabel: initialGateLabel,
     logFile,
     report,
+    setCheckpointHandler,
     resumed,
     activeModelEntry,
     autoGate,
@@ -1279,6 +1291,11 @@ export async function repl(args: ICliArgs): Promise<number> {
     }, PERSIST_DEBOUNCE_MS);
   };
 
+  // A long send checkpoints every N turns; save then, not only when it ends.
+  setCheckpointHandler(() => {
+    void persist();
+  });
+
   // "update available" notice: read from the local cache (no network on the hot
   // path) and refresh it in the background for next time. Gated to interactive,
   // non-CI sessions inside update-check, so eval/headless runs are unaffected.
@@ -2060,7 +2077,7 @@ export async function repl(args: ICliArgs): Promise<number> {
       files: session.scope,
       accept: rebound.accept,
       contextWindow,
-      report: makeReporter(logFile, id, id),
+      report,
       enableThinking: false,
       // Keep ask_user (WS-C) offered after /clear when a human is present — but gated
       // on the TTY like the init session, so a piped REPL doesn't advertise a pause

packages/core/src/config/config.constants.ts

@@ -10,6 +10,8 @@ export const ENV_FLAG = {
   browser: "TSFORGE_BROWSER",
   browserPort: "TSFORGE_BROWSER_PORT",
   browserAllowPrivate: "TSFORGE_BROWSER_ALLOW_PRIVATE",
+  // Per-send turn cap override for interactive sessions (0 = unlimited).
+  maxTurns: "TSFORGE_MAX_TURNS",
   noScriptTool: "TSFORGE_NO_SCRIPT",
   noGitTool: "TSFORGE_NO_GIT_TOOL",
   noGithub: "TSFORGE_NO_GITHUB",

packages/core/src/config/flags.ts

@@ -40,6 +40,18 @@ export const flags = {
    *  Off by default so injected page text can't aim the logged-in browser at a
    *  router, an internal service, or the bridge itself. */
   browserAllowPrivate: (): boolean => isOn(ENV_FLAG.browserAllowPrivate),
+  /** Per-send turn cap override: a positive integer caps every send, `0` means
+   *  unlimited; unset/invalid → undefined (the session picks its default). */
+  maxTurns: (): number | undefined => {
+    const raw = process.env[ENV_FLAG.maxTurns];
+    const n = raw === undefined || raw.trim() === "" ? Number.NaN : Number(raw);
+
+    if (!Number.isInteger(n) || n < 0) {
+      return undefined;
+    }
+
+    return n === 0 ? Number.POSITIVE_INFINITY : n;
+  },
   /** Programmatic Tool Calling: advertise the `script` tool. ON by default — it
    *  measurably speeds up read-dependent multi-file work (codemods) and is a no-op
    *  on simple tasks; withhold with TSFORGE_NO_SCRIPT (the A/B / kill switch). It

packages/core/src/loop/readonly-spin.ts

@@ -28,6 +28,7 @@ export const WRITE_ATTEMPT_TOOLS: ReadonlySet<string> = new Set([
   // A research session reads for many turns by design; saving notes is its
   // "write". Without this a long thread read trips the re-steer mid-page.
   TOOL_NAME.note,
+  TOOL_NAME.append,
 ]);
 
 /** Offered after a readonly re-steer — model cannot pick survey reads again.
@@ -39,6 +40,7 @@ export const WRITE_FORCE_TOOL_NAMES: ReadonlySet<string> = new Set([
   TOOL_NAME.editLines,
   TOOL_NAME.check,
   TOOL_NAME.note,
+  TOOL_NAME.append,
 ]);
 
 /** Research tools: fetching, searching, and reading pages (the web tools and