feat(research): SearXNG-first search with result routing + Hacker News and Stack Exchange readers

apps/docs/astro.config.mjs

@@ -206,6 +206,7 @@ export default defineConfig({
             { label: "Web access", link: "/integrations/web-tools/" },
             { label: "Research in your Chrome", link: "/integrations/chrome/" },
             { label: "Research on Reddit", link: "/integrations/reddit/" },
+            { label: "Research sources", link: "/integrations/research-sources/" },
             { label: "Decision memory", link: "/integrations/decision-memory/" },
           ],
         },

apps/docs/src/content/docs/integrations/chrome.mdx

@@ -39,7 +39,7 @@ Remaining risk: injected text could still ask the agent to *open* a URL that car
 3. Start tsforge with the bridge on, or flip **Chrome browser** in [`/config`](/cli/interactive/):
 
    ```bash
-   TSFORGE_BROWSER=1 tsforge
+   TSFORGE_BROWSER=1 tsforge   # once; or set "browser": true in ~/.tsforge/config.json (or toggle it in /config) and it sticks
    ```
 
    The boot chip reads `browser · waiting for the Chrome extension`.

apps/docs/src/content/docs/integrations/research-sources.mdx

@@ -0,0 +1,71 @@
+---
+title: Research sources
+description: "How a tsforge research run moves between search and sources cheaply: Google-quality results from your own SearXNG, each result routed to the reader that suits it, and built-in readers for Hacker News and Stack Exchange that read whole discussions from their official APIs."
+---
+
+A research run keeps bouncing between search and sources: it finds something on Reddit, searches the web for more, lands on a Hacker News thread or a Stack Exchange question, and goes back. tsforge makes each hop cheap.
+
+## Search: your own SearXNG
+
+`web_search` uses DuckDuckGo by default. If you run [SearXNG](https://docs.searxng.org/), point tsforge at it once. It queries Google, Bing and others and returns clean JSON, with no scraping and nothing tied to your Google account. Set it in `/config` → **Web search backend**, or in `~/.tsforge/config.json`:
+
+```json
+{ "settings": { "searxngUrl": "http://searx.lan", "webTools": true, "browser": true } }
+```
+
+(`TSFORGE_SEARXNG_URL` still works and wins for a single run. See [Settings file](/reference/flags/#settings-file-set-it-once).)
+
+In SearXNG's `settings.yml`, `search.formats` must include `json`. Keep the instance on your LAN: it has no login of its own.
+
+Searching Google through your logged-in browser is deliberately not supported. Google blocks automated searches with CAPTCHAs, it would put your own Google account at risk, and its terms forbid it. SearXNG gets the same results without those problems.
+
+Two options help research runs:
+
+- `queries`: up to 5 phrasings in one call. Results are interleaved, so every phrasing's best hits come first, and duplicates are removed.
+- `topic`: results already read for that research topic are marked `already read`.
+
+## Every result names its reader
+
+Each search result ends with the tool that reads it best:
+
+```text
+1. Technical understanding of guitar hum and grounding - Reddit
+   https://www.reddit.com/r/audioengineering/comments/169qpf3/…
+   → reddit_thread post:"169qpf3"
+2. Validating pickup wiring - Music Stack Exchange
+   https://music.stackexchange.com/questions/7263/validating-pickup-wiring
+   → se_question question:"7263" site:"music"
+3. How do I fix a 60Hz hum on my guitar? - Quora
+   → web_fetch
+```
+
+`web_fetch` routes on its own, too. Give it a Reddit, Hacker News or Stack Exchange URL together with a `topic`, and the site's reader reads the whole discussion as clean text instead of scraping the page. Without a topic it fetches normally and adds a tip.
+
+## Hacker News
+
+These use Algolia's official HN API. No browser or key is needed.
+
+| Tool | What it does |
+| --- | --- |
+| `hn_search` | Stories (or comments) by relevance or date, optionally within a time window. One line each: id, points, comments, age, title, domain. |
+| `hn_thread` | The story plus its entire comment tree in one request, as nested markdown, in chunks. |
+
+## Stack Exchange
+
+These use the official Stack Exchange API. No browser is needed. Without a key you get 300 requests a day; set `TSFORGE_STACKEXCHANGE_KEY` for 10,000.
+
+| Tool | What it does |
+| --- | --- |
+| `se_search` | Questions on any site: `stackoverflow`, `superuser`, `askubuntu`, or any `<name>.stackexchange.com` such as `electronics`, `music` or `diy`. Can filter to questions with an accepted answer. |
+| `se_question` | The question and all its answers, the accepted one first, then by votes. Code blocks are kept. |
+
+tsforge honours the API's own request to slow down (`backoff`), and warns when the day's quota runs low.
+
+## Nothing is read twice
+
+Pass the same `topic` to every reader, including `web_fetch`. Each source (thread, question or web page) is logged in `notes/<topic>/sources.md`. Readers refuse a source that's already logged unless you pass `force: true`, and search marks it `already read`, so a run that restarts many times over days never repeats itself.
+
+## When they're available
+
+- Hacker News and Stack Exchange tools come with the **Web tools** (`/config`, or `TSFORGE_WEB=1`) or the [Chrome bridge](/integrations/chrome/).
+- The [Reddit](/integrations/reddit/) tools need the Chrome bridge, because they read through your logged-in browser.

apps/docs/src/content/docs/integrations/web-tools.mdx

@@ -15,9 +15,13 @@ TSFORGE_WEB=1 tsforge "update the deprecated API call (check the library's curre
 - **`package_docs`**: read package docs without a docs API. Local `node_modules` README/package metadata/types first, then the npm registry README when needed.
 - **`web_fetch`**: retrieve a URL and return readable markdown. The page is fetched and the content extracted **on-machine** (no third-party reader API), so you get clean text without shipping the URL to a service.
 - **`web_browse`**: open a public URL in local headless Chromium via Playwright and return rendered visible text, final URL, title, and links. Use it for JavaScript-rendered docs that `web_fetch` cannot see. No hosted browser service.
-- **`web_search`**: discover sources for a query. It returns public result URLs only, supports `recency` (`day`, `month`, `year`), `domains` for official-site scoping, and `maxResults` up to 20. Defaults to DuckDuckGo's keyless HTML endpoint.
+- **`web_search`**: discover sources for a query. It returns public result URLs only, supports `queries` (up to 5 phrasings in one call), `recency` (`day`, `week`, `month`, `year`), `domains` for site scoping, `maxResults` up to 20, and `topic` to flag results already read. Each result names the tool that reads it best. Defaults to DuckDuckGo's keyless HTML endpoint.
+- **`hn_search` / `hn_thread`**: Hacker News through Algolia's official API. Search, then read a whole discussion in one request.
+- **`se_search` / `se_question`**: Stack Exchange through its official API. Search any site, then read a question with all its answers.
 
-SearXNG is **not bundled or started by tsforge**. If you run your own SearXNG service (Docker, Compose, or any deployment), point `TSFORGE_SEARXNG_URL` at it for full privacy/control:
+`web_fetch` also takes a `topic`: the page is logged in `notes/<topic>/sources.md` and not read twice, and Reddit / Hacker News / Stack Exchange URLs are handed to their site tool. How research runs use all of this together: [Research sources](/integrations/research-sources/).
+
+SearXNG is **not bundled or started by tsforge**. If you run your own SearXNG service (Docker, Compose, or any deployment), point tsforge at it once: `/config` → **Web search backend**, or `"settings": { "searxngUrl": "…" }` in `~/.tsforge/config.json` ([Settings file](/reference/flags/#settings-file-set-it-once)). For a single run the variable also works:
 
 ```bash
 TSFORGE_WEB=1 TSFORGE_SEARXNG_URL=http://localhost:8888 tsforge
@@ -38,6 +42,7 @@ The tools are read-only and offline-safe. Interactive sessions enable them by de
 | `TSFORGE_WEB` | on interactive, off one-shot/eval | force keyless research tools on (`=1`) or off (`=0`) |
 | `TSFORGE_NPM_REGISTRY` | npm registry | registry used by `package_info` / `package_docs` |
 | `TSFORGE_SEARXNG_URL` | unset | route `web_search` to a SearXNG instance you already run |
+| `TSFORGE_STACKEXCHANGE_KEY` | unset | Stack Exchange API key: 300 → 10,000 requests/day for `se_*` |
 | `TSFORGE_WEB_SEARCH_BACKEND` | auto | `duckduckgo` or `searxng`; `searxng` fails closed if no SearXNG URL is set |
 
 → [Environment variables](/reference/flags/) · [MCP servers](/integrations/mcp/) · [Permissions & policy](/guardrails/policy/)

apps/docs/src/content/docs/internals/where-to-change.mdx

@@ -40,7 +40,7 @@ Built-in only: a plugin fetches with the user's cookies, so it ships in tsforge'
 | 3. Name its tools | `TOOL_NAME` + `TOOL_SPECS` in `agent/agent.constants.ts`, `HANDLERS` via `sitePluginTool()` in `loop/tools/execute-tool.ts`, `network` in `policy/classify.ts`, `RESEARCH_TOOLS` in `loop/readonly-spin.ts` |
 | 4. Allow its hosts | `FETCH_HOSTS` in `packages/chrome-extension/src/fetch-policy.ts` (a test fails until it matches the registry) |
 
-The plugin's tools are advertised with the browser tools, and its guidance is appended once when the browser capability turns on.
+Pick the plugin's `transport`. A `browser` plugin (Reddit) reads through `page.fetch` in the user's tab and is offered only with the Chrome bridge; its hosts go in `FETCH_HOSTS`. A `direct` plugin (Hacker News, Stack Exchange) calls a public JSON API with `directFetchJson` and is offered whenever the session can research. Give it a `route(url)` so `web_search` hints and `web_fetch` delegation know which URLs it owns, and log reads through `site-plugins/reading.ts` so `sources.md` dedupes them.
 
 ### A new CLI command
 

apps/docs/src/content/docs/reference/flags.mdx

@@ -3,6 +3,37 @@ title: Environment variables
 description: Canonical list of every TSFORGE_* environment variable.
 ---
 
+## Settings file: set it once
+
+You don't have to pass these variables every time you start tsforge. Put them in **`~/.tsforge/config.json`**, which applies everywhere, or in a project's `tsforge.config.json` to override for that project:
+
+```json
+{
+  "settings": {
+    "browser": true,
+    "webTools": true,
+    "searxngUrl": "http://searx.lan",
+    "stackExchangeKey": "…",
+    "maxTurns": 0
+  },
+  "env": { "TSFORGE_TDD": "0" }
+}
+```
+
+| Setting | Variable it sets |
+| --- | --- |
+| `browser` | `TSFORGE_BROWSER` (Chrome research bridge) |
+| `browserPort` | `TSFORGE_BROWSER_PORT` |
+| `browserAllowPrivate` | `TSFORGE_BROWSER_ALLOW_PRIVATE` |
+| `webTools` | `TSFORGE_WEB` |
+| `searxngUrl` | `TSFORGE_SEARXNG_URL` |
+| `searchBackend` | `TSFORGE_WEB_SEARCH_BACKEND` |
+| `stackExchangeKey` | `TSFORGE_STACKEXCHANGE_KEY` |
+| `maxTurns` | `TSFORGE_MAX_TURNS` |
+| `compactAt` | `TSFORGE_COMPACT_AT` |
+
+`env` sets any other `TSFORGE_*` variable (other names are ignored). Order of precedence: a real environment variable beats the project file, which beats `~/.tsforge/config.json`. The `/config` toggles for web tools, Chrome and the SearXNG URL save to `~/.tsforge/config.json` for you.
+
 ## Behavior & tools: configure in `/config`
 
 Feature toggles are configured **inside the harness**, not through env vars. Run
@@ -71,7 +102,8 @@ Opt-in, free, and no required service keys. Turn on **Web tools** in [`/config`]
 | Variable | Default | Toggles |
 | --- | --- | --- |
 | `TSFORGE_NPM_REGISTRY` | npm registry | registry used by `package_info` / `package_docs` |
-| `TSFORGE_SEARXNG_URL` | unset | route `web_search` to a SearXNG instance you already run (e.g. `http://localhost:8888`) |
+| `TSFORGE_SEARXNG_URL` | unset | route `web_search` to a SearXNG instance you already run (e.g. `http://searx.lan`) |
+| `TSFORGE_STACKEXCHANGE_KEY` | unset | Stack Exchange API key for `se_search` / `se_question`: raises the daily quota from 300 to 10,000 requests |
 | `TSFORGE_WEB_SEARCH_BACKEND` | auto | `duckduckgo` or `searxng`; `searxng` fails closed if no SearXNG URL is set |
 
 ## Chrome bridge

docs/superpowers/specs/2026-09-27-research-search-hn-se-design.md

@@ -0,0 +1,114 @@
+# Research search: SearXNG, result routing, Hacker News, Stack Exchange
+
+Status: agreed in chat 2026-09-27 ("agreed with all of it … do all this in a single PR").
+Builds on: `2026-09-26-site-plugins-reddit-design.md`.
+
+## Goal
+
+A research run keeps bouncing between search and sources: it finds something
+on Reddit, searches the web for more, lands on a Hacker News or Stack Exchange
+thread, and goes back. Make every hop cheap:
+
+- **Search** returns Google-quality results as JSON from the user's own
+  SearXNG (already hosted in the home k3s, exposed LAN-only at `searx.lan` by
+  argocd-app-of-apps#133), several queries per call, with already-read results
+  flagged.
+- **Every result says which reader to use.** Reddit → `reddit_thread`, Hacker
+  News → `hn_thread`, Stack Exchange → `se_question`, anything else →
+  `web_fetch`. `web_fetch` given a Reddit / HN / SE URL hands it to the plugin.
+- **Hacker News and Stack Exchange plugins**, over their official public JSON
+  APIs. They need no browser, so they are "direct" plugins.
+- **Every source read** (threads, questions, web pages) is logged in
+  `notes/<topic>/sources.md`, so nothing is read twice across restarts.
+
+Nothing is tied to a particular topic or product.
+
+## Why not Google through the browser
+
+Google has no structured view, detects automated querying quickly ("unusual
+traffic" CAPTCHAs, which tsforge cannot and must not solve), would put the
+user's own Google account at risk, and its terms forbid it. SearXNG already
+queries Google (the home instance uses Google CSE) and returns JSON.
+
+## Plugin transport
+
+`ISitePlugin` gains `transport: "browser" | "direct"`.
+
+- `browser` (Reddit): as before — tools advertised only with the Chrome
+  bridge; requests go through `page.fetch` in the user's tab.
+- `direct` (HN, SE): tools advertised whenever the session can research (web
+  tools on, or the browser capability on). Requests are made by tsforge itself
+  with `directFetchJson(plugin, host, path)`: https only, host must be one the
+  plugin declares, per-host pacing (1 req/s) with the same 429/5xx backoff as
+  `page.fetch`, 20 s timeout, a tsforge user agent.
+- `ISitePluginContext.session` becomes optional; a `stateKey` object (the
+  browser session when there is one, else a stable per-workspace object) keys
+  per-session plugin state (chunk caches, read sets).
+
+`routeUrl(url)` on the registry returns `{ tool, args, key }` for URLs a plugin
+owns, or null. `key` is the `sources.md` key (`reddit:abc12`, `hn:4214`,
+`se:stackoverflow_123`).
+
+## Hacker News plugin (`site-plugins/hackernews/`)
+
+Host: `hn.algolia.com` (Algolia's official HN API; public, no key).
+
+| Tool | Arguments | Behaviour |
+|---|---|---|
+| `hn_search` | `query`, `sort` relevance\|date, `type` story\|comment (default story), `time` day\|week\|month\|year\|all, `limit` ≤ 50, `page?`, `topic?` | `/api/v1/search` or `/search_by_date`, `numericFilters=created_at_i>…` for time. One line per hit: id, points, comments, age, title, domain; `already read` when logged. |
+| `hn_thread` | `item` (id or `news.ycombinator.com/item?id=` URL), `topic`, `chunk?`, `force?` | `/api/v1/items/<id>` returns the WHOLE comment tree in one request. Rendered like Reddit threads (nested, author + age per comment), HTML → markdown, dead/deleted leaves dropped. Chunked + cached; logs `hn:<id>`; refuses already-read unless `force`. |
+
+## Stack Exchange plugin (`site-plugins/stackexchange/`)
+
+Host: `api.stackexchange.com` (official API v2.3; no key needed for 300
+requests/day per IP; `TSFORGE_STACKEXCHANGE_KEY` raises it to 10,000).
+
+| Tool | Arguments | Behaviour |
+|---|---|---|
+| `se_search` | `query`, `site` (default `stackoverflow`; any SE site: `electronics`, `music`, `diy`, `superuser`, `askubuntu`, …), `sort` relevance\|votes\|activity\|creation, `accepted?`, `limit` ≤ 50, `page?`, `topic?` | `/2.3/search/advanced`. One line per question: site/id, score, answers, ✓ accepted, age, title, tags; `already read` when logged. |
+| `se_question` | `question` (id or any SE question URL), `site?` (inferred from the URL), `topic`, `chunk?`, `force?` | `/2.3/questions/{id}?filter=withbody` + `/2.3/questions/{id}/answers?sort=votes&filter=withbody` (2 requests). Question, then answers by votes with the accepted one marked; bodies HTML → markdown (code blocks kept). Logs `se:<site>_<id>`. |
+
+The API's `backoff` field is honoured (the pacer waits that long), and
+`quota_remaining` is shown in the footer when low.
+
+## `web_search`
+
+- `queries: string[]` (≤ 5) alongside `query`; results merged round-robin and
+  deduplicated by URL.
+- `recency` gains `week`.
+- `topic?`: results whose source key is in `notes/<topic>/sources.md` are
+  marked `already read`.
+- Each result line ends with its reader: `→ reddit_thread post:"abc12"`,
+  `→ hn_thread item:4214`, `→ se_question stackoverflow/123`, or `→ web_fetch`.
+- Backend unchanged: SearXNG when `TSFORGE_SEARXNG_URL` is set (e.g.
+  `http://searx.lan`), else DuckDuckGo.
+
+## `web_fetch`
+
+- A URL a plugin owns: with `topic`, the plugin's reader is called directly
+  (same output as calling it); without `topic`, a one-line pointer to the
+  right tool (no fetch).
+- `topic?` for ordinary pages: the page is logged as `web:<12-hex sha1 of the
+  normalised URL>` and a re-read is refused unless `force: true`.
+
+## Guidance
+
+A short "sources" playbook is appended (once) whenever research tools are on:
+search with `web_search` (several phrasings, `topic`), follow each result's
+reader hint, prefer `hn_*` / `se_*` / `reddit_*` over generic fetching, log
+findings with `note` / `append`. The Reddit playbook stays as is.
+
+## Testing
+
+Fixtures shaped from the documented APIs (Algolia `hits` / `items` tree; SE
+`items` with `quota_remaining` / `backoff`; SearXNG JSON). Unit tests for
+routing, URL parsing (HN item URLs, every SE host shape), rendering, direct
+fetch (host allowlist, https-only, pacing/backoff, SE `backoff`), multi-query
+merge/dedupe, already-read flags, web_fetch delegation and page logging.
+Break-to-prove on the host allowlist, dedupe, routing and refusal rules.
+A live smoke test against the home SearXNG, HN and SE APIs from this machine.
+
+## Out of scope
+
+Stack Exchange comments (extra quota per question), HN user pages, Discourse
+forums (next plugin), paid search APIs.

packages/chrome-extension/tests/reddit-e2e.test.ts

@@ -222,10 +222,10 @@ describe("reddit research over the real bridge", () => {
 
     const reddit = createRedditHandlers({
       now: () => new Date(NOW_UTC * 1000),
-      fetchJson: (s, path) => pageFetchJson(s, "www.reddit.com", path),
+      fetchJson: (_s, path) => pageFetchJson(session, "www.reddit.com", path),
       download: async () => ({ ok: false, reason: "offline test" }),
     });
-    const ctx = { session, cwd, progress: () => undefined };
+    const ctx = { session, stateKey: session, cwd, progress: () => undefined };
 
     const search = await reddit.reddit_search?.(
       { query: "jimmy page wiring", topic: "wiring" },