feat(research): SearXNG-first search with result routing + Hacker News and Stack Exchange readers

[agjs]

Makes every hop of a research run cheap: search → source → search → source.

Spec: docs/superpowers/specs/2026-09-27-research-search-hn-se-design.md. Infra: SearXNG is exposed LAN-only at searx.lan via Programmer-Network/argocd-app-of-apps#133, plus a Pi-hole record.

Search (web_search)

• queries: up to 5 phrasings per call, merged round-robin and deduplicated.
• recency: week added.
• topic: results already logged in notes/<topic>/sources.md are marked already read.
• Every result names its reader: → reddit_thread post:"…", → hn_thread item:"…", → se_question question:"…" site:"…", or → web_fetch.
• The backend is unchanged. TSFORGE_SEARXNG_URL=http://searx.lan gives Google results (through the instance's Google CSE engine) as JSON.
• Google through the browser is deliberately not done: CAPTCHAs, account risk, and Google's terms.

web_fetch

• With topic, a Reddit / HN / SE URL is read by its site tool (the whole discussion, structured). Other pages are logged as web:<sha1-12> of the normalised URL and refused on re-read unless force.
• Without topic, it behaves as before, plus a tip naming the better tool.

Site plugins: new direct transport

• Public JSON APIs called by tsforge itself, no browser: directFetchJson checks the host allowlist, requires https, paces requests, and backs off on 429/5xx.
• ISitePluginContext.session is now optional, and stateKey keys per-session state.
• route(url) per plugin, and routeUrl in the registry.
• Direct plugins are offered whenever the session can research (web tools or the Chrome bridge). Reddit stays browser-only.

Hacker News (hn_search, hn_thread)

Uses Algolia's official API. hn_thread reads the entire comment tree in one request, rendered as nested markdown with dead or deleted leaves dropped. Reads are chunked, cached, logged, and refused on re-read unless force.

Stack Exchange (se_search, se_question)

• Official API v2.3. Any site: stackoverflow, superuser, askubuntu, or <name>.stackexchange.com.
• se_question returns the question plus answers, accepted first then by votes, with code blocks kept and entities decoded.
• Honours the API's backoff, reports low quota, and accepts an optional TSFORGE_STACKEXCHANGE_KEY (300 → 10k requests/day).

Guidance

A short "Finding and reading sources" playbook for research sessions, plus per-plugin notes. Web-only sessions get the HN/SE guidance too.

Tests

• New site-plugins-hn-se.test.ts and research-search.test.ts, with fixtures shaped from the live API responses.
• Each rule was removed in turn and its test went red: direct host allowlist, HN refuse-read, SE backoff, SE accepted-first, transport gating, URL normalisation, round-robin, search already-read, fetch delegation, fetch refuse, SE api-host rejection.
• A live smoke test from this machine passed:
  • SearXNG at searx.lan returned Google results with reader hints
  • hn_search → hn_thread read a 156-comment tree
  • se_search → web_fetch handed off to se_question
  • both reads were logged in sources.md
• bun run ci:local passes: 6219 tests, 0 failures. The docs build passes.

Docs: a new integrations/research-sources.mdx, plus updates to web-tools.mdx, the flags reference (TSFORGE_STACKEXCHANGE_KEY), and where-to-change (plugin transports).

---

Also in this PR

The TUI was broken by library output. jsdom reported page CSS problems ("Could not parse CSS stylesheet") with console.error straight onto the terminal. That scrolled the alternate screen, and the fixed rows (header, status, Tasks panel) were redrawn one line off, leaving stacked duplicates.

• Root fix: every jsdom is created through quietDocument with a silent VirtualConsole (web_fetch extraction and both page-render documents). A test reproduces the default jsdom printing the error and proves the quiet path prints nothing.
• Safety net: while the interactive UI owns the terminal, stray console.error / console.warn from any library go to the debug trace (TSFORGE_TRACE). The UI itself never uses them.

Persistent settings, instead of passing a dozen env vars at every launch.

• ~/.tsforge/config.json (and a project's tsforge.config.json) gets a settings block with readable names: browser, webTools, searxngUrl, stackExchangeKey, maxTurns, browserPort, searchBackend, compactAt, browserAllowPrivate.
• An env block passes through TSFORGE_* only; PATH and other names are ignored.
• Settings are applied at startup, before any flag is read. A real env var still wins, and the project file beats the user file.
• /config now saves Web tools, Chrome browser and a new Web search backend (SearXNG URL) field to ~/.tsforge/config.json, keeping the rest of the file (keybindings).

Tests: quiet-console.test.ts and user-settings.test.ts. Break-to-prove covers the quiet jsdom, real-env-wins, TSFORGE_*-only passthrough, project-over-user, keeping keybindings, and saving the browser toggle. ci:local passes: 6228 tests, 0 failures. The flags reference has a new "Settings file" section, and the Chrome / research-sources / web-tools pages point to it.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	tsforge	b029dd0	Commit Preview URL Branch Preview URL	Sep 27 2026, 09:56 AM