[recipes] Wiki synthesis + autobiography pipeline - #222
Merged
justfinethanku merged 6 commits intoApr 21, 2026
Merged
Conversation
…from thoughts Adds a new recipe that ships two Node scripts for wiki-style synthesis over the core `thoughts` table, plus optional Next.js dashboard snippets. - `scripts/synthesize-wiki.mjs` — topic-scoped synthesizer with a built-in `autobiography` mode that groups thoughts by year and asks an OpenAI-compatible Chat Completions endpoint to produce second-person biographical prose per year. Extend the catalogue to add more topics. - `scripts/backfill-gmail-wikis.mjs` — resume-safe per-thread wiki generator for Gmail-imported thoughts. Groups by `metadata.gmail.thread_id`, filters by word-count + message/atom thresholds, writes wiki thoughts with `derived_from` edges to their source atoms. Prefers an `upsert_thought` RPC when present, falls back to plain inserts. - `dashboard-snippets/` — optional Next.js components (Server Action + `/wiki` index + `/wiki/[slug]` detail) to copy into a dashboard. Generalized from ExoCortex origin: replaced the MCP edge-function client with direct PostgREST + service role, swapped the Anthropic-direct call for OpenAI-compatible Chat Completions, removed personal data references, introduced SUBJECT_NAME and SOURCE_TYPE_FILTER env knobs. Complements the in-flight entity-wiki recipe — this one does corpus-slice synthesis and only requires the core `thoughts` table. README documents the thought_edges / upsert_thought prerequisites so users know which OB1 layers they need before running the email-thread pipeline.
…tion Wrap raw thought content in <entries>...</entries> delimiters and tell the system+user prompt explicitly to ignore instructions inside it. Captured thoughts are user data and can contain adversarial text that overrides the biographer task.
Wrap raw email-thread content in <thread>...</thread> delimiters and tell the system prompt to treat it as untrusted data. Email bodies arrive from external senders and may contain adversarial instructions that override the summarization task.
…idate year
Server Action was passing the host's entire process.env to the
synthesize-wiki child process, leaking every unrelated secret. Pass only
the OB1/LLM-related vars plus the minimal system env the child needs.
Also server-side-validate scope_year against /^(19|20)\d{2}$/ (P3).
Auth guard stays as a placeholder so the snippet remains optional
reference code — the README already warns to add one.
…rrow RPC fallback
P1-2: Wrap the wiki-delete PostgREST filter id in encodeURIComponent so
non-numeric (e.g., uuid) ids with reserved chars don't break the URL.
P1-3: Coerce thrown non-Error values in the retry loop — 'throw 'msg''
used to crash the catch and skip the state-log append.
P2: Narrow the upsert_thought RPC fallback trigger to the specific
rpc/upsert_thought 404 signal, not any string with 'not found',
so auth/permission errors surface instead of silently falling
through to a direct insert. Also cap the LLM error body at 500
chars to avoid leaking long provider diagnostics to logs.
…rereqs Escape SUBJECT_NAME in autobiography frontmatter so names containing ':', quotes, or YAML-reserved chars don't break the frontmatter parse the dashboard snippet does. Expand the README's thought_edges prerequisite with column types and the UNIQUE index needed for the ignore-duplicates edge-upsert header. Also spell out what users observe when the schema is missing.
justfinethanku
merged commit Apr 21, 2026
2507c13
into
NateBJones-Projects:main
2 of 3 checks passed
This was referenced Apr 22, 2026
gleesonb
pushed a commit
to gleesonb/OB1
that referenced
this pull request
May 12, 2026
…lanshurafa/wiki-synthesis [recipes] Wiki synthesis + autobiography pipeline
lucifer
pushed a commit
to lucifer/OB1
that referenced
this pull request
Jul 3, 2026
…lanshurafa/wiki-synthesis [recipes] Wiki synthesis + autobiography pipeline
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
recipes/wiki-synthesisrecipe that turns the atomicthoughtstable into human-readable synthesis surfaces: topic-scoped wiki articles, per-thread email wikis, and a full autobiography — all via any OpenAI-compatible Chat Completions endpoint (OpenRouter, OpenAI, local LLMs). Default model isanthropic/claude-haiku-4-5; swap via env var. Ported from my ExoCortex personal memory stack after running in production for several months.What's new
scripts/synthesize-wiki.mjs— corpus-slice synthesis: tag/type/date filters overthoughts, writes wiki pages back as new thoughts.scripts/backfill-gmail-wikis.mjs— per-thread mode that usesthought_edgesto group email thoughts before synthesis.dashboard-snippets/— optional Next.js components (wiki list, detail page, Autobiography button) shipped as copy-paste snippets with explicit auth-guard placeholders, not wired-in code.upsert_thoughtRPC fallback.Dependencies
thoughtstable only for corpus mode. No schema changes required.thought_edges(see PR [docs] Add Getting Started guide — internalize setup from Substack #5 knowledge-graph-schema) — documented as a hard prereq.entity-wiki(PR [recipes] Entity wiki pages from knowledge graph #213) is one-page-per-entity and needs entity extraction; this recipe is corpus slices and works with vanilla Open Brain.upsert_thoughtRPC is optional — script falls back to plainPOST /thoughtsinsert.Review process
Pre-reviewed on my fork (alanshurafa#17) via gsd-code-reviewer + Codex cross-AI pass. Single FIXED-AND-CLEAN round landed 10 fixes across 5 commits: 2 P0 prompt-injection hardenings (autobiography + gmail-wiki prompts), 3 P1s (Server Action env leak, unencoded PostgREST filter, non-Error throw crash), 4 P2s, 1 P3. Happy to address further feedback.