Skip to content

feat(research): multi-day runs — no turn cap without a gate, mid-send saves, dedup, smaller images, append tool - #402

Merged
agjs merged 1 commit into
mainfrom
feat/long-research-runs
Sep 26, 2026
Merged

agjs merged 1 commit into
mainfrom
feat/long-research-runs

Conversation

@agjs

@agjs agjs commented Sep 26, 2026

Copy link
Copy Markdown
Collaborator

Fixes from the first long live Reddit crawl: 5h+, 1,095 threads, ended by the 1000-turn backstop.

No turn cap for research

  • A session with no live gate (research, notes, chat) runs uncapped. It stops when the model finishes, you interrupt, or a real guard trips (repetition-loop budget, repeated timeouts, read-only spin).
  • Gated builds and plan mode keep the 1000-turn backstop.
  • TSFORGE_MAX_TURNS overrides either: a number caps every session, 0 means unlimited.
  • The cap is re-evaluated every turn (Session.turnCap()), so an auto gate that wakes mid-send brings the backstop back.

Saving during long sends

The REPL now saves the session on every checkpoint event (every 40 turns), via withCheckpointHook. Before, it only saved when a send ended, so a crash or reboot on day two lost the whole conversation for --continue.

TypeScript stream rules off in research sessions

TTSR rules ("no as cast / no as any") are skipped while build mode is suppressed (no code yet, or gate cleared). They interrupted the model three times on prose about guitar wiring.

Reddit: no duplicate reads

  • reddit_thread refuses a thread already logged in sources.md unless force: true. Within a session, any chunk (including chunk 1) is served from cache, never refetched.
  • New reddit_mark_read logs threads covered another way (earlier crawls, URL lists) so they're skipped. It's in the playbook for resuming research.
  • The run produced 64 duplicate entries (55 repeats of threads read that run, 9 from before sources.md existed).

Reddit: smaller images

  • Images now use Reddit's ≤1080px renditions (gallery media_metadata.p, post preview.resolutions) instead of full originals, about 5× smaller. The run had saved 1.3 GB.
  • Caps are now 20 post images plus 10 comment images per thread (was 30 combined). Images past the caps stay links, and files are named by render number.

New append tool

  • Adds lines to a data file (.jsonl, CSV, logs) with no edit anchor, creating the file if needed.
  • JSONL lines are validated, so nothing is written if a line is malformed.
  • Code and config files are refused, as are .. paths, absolute paths, node_modules/, .git/, .tsforge/, and symlinked escapes.
  • It replaces anchored edit appends, which failed 5 times on stale or ambiguous anchors in the run.

Tests

  • New session-long-runs.test.ts includes a real 1,005-turn no-gate send that finishes as responded, plus the TSFORGE_MAX_TURNS cap and parsing, the gated backstop, checkpoints, and TTSR off/on.
  • New append-tool.test.ts.
  • Reddit long-crawl cases: refuse/force, mark-read dedupe, comment-image cap, 1080px choice.
  • Each fix was removed in turn and its test went red.
  • bun run ci:local passes: 6191 tests, 0 failures. The docs build passes.

Not covered by tests

The REPL line that binds persist to the checkpoint hook. The hook itself and the checkpoint events are tested.

Docs: TSFORGE_MAX_TURNS in the flags reference; integrations/chrome.mdx "Long runs" (no cap, saves, append, TTSR); integrations/reddit.mdx (refuse/force, reddit_mark_read, image sizes and caps, append).

… saves, dedup, smaller images, append tool

Lessons from a 5h+/1,095-thread live crawl:
- turn cap: sessions with no live gate (research/notes/chat) run uncapped;
  gated builds and plan mode keep the 1000-turn backstop; TSFORGE_MAX_TURNS
  overrides (0 = unlimited). Re-evaluated each turn so a waking gate applies.
- the REPL saves the session on every checkpoint (every 40 turns), so a crash
  or reboot mid-send no longer loses the conversation for --continue
- TTSR stream rules (TypeScript: no-as-cast…) are off while build mode is
  suppressed — they fired on prose about guitar wiring
- reddit_thread refuses threads already logged in sources.md unless force;
  within a session any chunk is served from cache. New reddit_mark_read logs
  threads covered another way (earlier crawls). The run had 64 duplicates.
- images: Reddit's <=1080px rendition instead of originals (~5x smaller; the
  run saved 1.3 GB), 20 post + 10 comment images per thread
- append tool: add lines to a data file (jsonl/csv/log) without an edit
  anchor; JSONL validated, code/config refused. Replaces anchored edits that
  failed on stale/ambiguous anchors
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
tsforge e03168a Commit Preview URL

Branch Preview URL
Sep 26 2026, 07:37 PM

@agjs
agjs merged commit 3c80632 into main Sep 26, 2026
9 checks passed
@agjs
agjs deleted the feat/long-research-runs branch September 26, 2026 20:34
@agjs agjs mentioned this pull request Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant