feat: add a nightly topic map to the left rail

[dimslaev]

Left-rail panel: each recent article as a dot, grouped by meaning into labelled clusters. A nightly job on the box writes a static JSON file; the frontend fetches it directly, with no API call.

What changed

• backend/pipeline/topic_map.py (python -m pipeline.topic_map)
  • Window: one COUNT of the last 7 days. 60 or more uses 7 days, otherwise 30. Capped at the 300 highest-scored.
  • Vectors: stored embeddings, with digest._vectors as the fallback (imported, not copied). numpy only.
  • k-means++ from a fixed seed on L2-normalised vectors, k = clamp(round(sqrt(n/2)), 4, 8). Clusters under 3 fold into the nearest centroid.
  • Layout: centroid PCA scaled into [0.12, 0.88], points placed by each cluster's own PCA and sized by its spread.
  • Labels: top tags by lift, at least 2 per cluster, Tag.name, fallback "other".
  • Schema v1 written atomically (temp file + os.replace, chmod 0644) to TOPIC_MAP_PATH. Logs one line: window, count, k.
• deploy/agentique-topicmap.service + .timer (06:10), Caddyfile handle_path /data/* before the SPA fallback, deploy/README rows and box steps.
• frontend/src/components/Rails/panels/TopicMap.tsx, registered in LEFT with meta "by meaning". It uses plain fetch through react-query with a 30 min staleTime.
  • Top line: "this week · N" or "this month · N".
  • Dots are 5px squares, paper at 55%. Scores at SCORE_STANDOUT or above are signal-coloured.
  • Hovering a dot shows its title and publisher. Clicking opens the article in a new tab and tracks it with articleClickHandlers.
  • Clicking a label dims the other clusters. Clicking it again clears the dimming.
  • Accessibility: the SVG has role="img" and a summary aria-label, there is an sr-only list of clusters, and each dot has a <title>.
• Docs: glossary "Topic map", one paragraph in product.md, deploy/README.

Where I departed from the plan, and why

• Hiding the whole panel. If the component only returns null, the panel header is left above an empty body. So RailPanel now hides an open section whose body is empty, using one CSS :has() rule. RailPanelDef is unchanged. A collapsed panel stays visible because its body is empty only because it's collapsed. Expanding it with no usable file then hides it.
• Centres pushed apart (MIN_GAP 0.3). On local data the PCA alone put two or three clusters on top of each other, and their labels overlapped. The adjustment is still deterministic.
• Label placement in the panel. Labels are centred over their dots and clamped inside the map. If a larger cluster's label or another cluster's dots are in the way, a label moves below its dots or further up. A second tag is added only while the label stays at 18 characters or fewer.
• Square dots, not circles. I used squares to match the design image and the site's square corners. ink-2 (#1b1d14, from the design palette) is added as a token in index.css.
• Empty or missing data. A file with no points also hides the panel, and so does a non-JSON response (the Vite dev server answers a missing file with index.html).
• Labels don't repeat. A tag that already names a larger cluster is skipped for the others.

Verified

• Backend tests (scripts/tests-start.sh, with Postgres 16 + pgvector set up locally because the pgvector image pull is blocked here): 547 passed, 1 failed. The failure is test_read_endpoints_are_public[/search?q=agents], which also fails on master in this sandbox because the embedding model download goes through a blocked proxy.
• New test_topic_map.py (16 tests):
  • the window switch at MIN_POINTS
  • deterministic output
  • small clusters folding in
  • distinctive labels and the "other" fallback
  • the schema
  • centre separation
  • atomic write: an interrupted write keeps the last good file and leaves no temp file
  • env path
  • two DB tests dated in 2100 so seeded rows stay out of the window
• Ruff check and format are clean. ty check app scripts is clean.
• Frontend: biome is clean on the changed files and bun run build passes.
• Playwright tests/topic-map.spec.ts + articles.spec.ts: 5 passed, run with --project=chromium --no-deps. The auth setup needs mailcatcher, which can't run here, and these specs don't need a login.
  • a fixture renders 10 dots and 3 labels with the right aria summary
  • a label click dims the rest and a second click clears it
  • a 404 hides the panel
  • a 4-day-old file hides the panel
• End-to-end run: I ran the job against the local seed DB (83 articles, 7-day window, k=6), served the file through Vite and checked screenshots of the panel with hover and cluster focus.
• Nothing was run against production.

One-time box steps (manual)

• sudo install -d -o agentique -g agentique -m 755 /var/lib/agentique/public
• Add TOPIC_MAP_PATH=/var/lib/agentique/public/topic-map.json to /opt/agentique/.env
• sudo cp /opt/agentique/deploy/agentique-topicmap.{service,timer} /etc/systemd/system/ && sudo systemctl daemon-reload && sudo systemctl enable --now agentique-topicmap.timer
• sudo cp /opt/agentique/deploy/Caddyfile /etc/caddy/Caddyfile && sudo systemctl reload caddy
• First run: sudo systemctl start agentique-topicmap, then journalctl -u agentique-topicmap -n 5 and curl -sI https://agentique.ch/data/topic-map.json

Until these are done the file returns a 404 and the panel stays hidden.

Merge note

Master had no new commits when I opened this. The who-wrote and stories sessions also edit LEFT/RIGHT in panels.tsx. If one merges first, the conflict is just the array entries. Keep both, with Topic map after Who wrote.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NQZAhjzHARib3J1DyZNQAs

---

Generated by Claude Code