[recipes] Atomizer — generic + Gmail re-atomization toolkit - #19
Open
alanshurafa wants to merge 87 commits into
Open
[recipes] Atomizer — generic + Gmail re-atomization toolkit#19alanshurafa wants to merge 87 commits into
alanshurafa wants to merge 87 commits into
Conversation
Keep the public contribution contract unchanged but make the maintainer-local execution layer legible to future agents, and stop tracking local-only overlays that have no business landing upstream. - CLAUDE.md: add Local GSD Execution Layer section pointing at .planning/ - .gitignore: add .local/, .agent/, .claude.json, __pycache__/
- dry_run now uses peekQueueItems() (read-only SELECT) instead of claimQueueItems(), so items stay "pending" during preview runs - claimQueueItems() returns only rows actually claimed via .select(), preventing race conditions where concurrent workers see stale results - markError() clears started_at and worker_version when resetting to "pending" so retryable items don't appear stale in monitoring Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Why: Schema stores thoughts.id / entity_extraction_queue.thought_id as UUID (gen_random_uuid()), not BIGINT. TypeScript types were declaring number, which is a load-bearing lie: PostgREST returns UUIDs as strings, and any arithmetic or Number-coerce on a consumer path would produce NaN. Updated claimQueueItems, peekQueueItems, markComplete, markError, linkThoughtEntity signatures to string. Entity IDs remain number (BIGSERIAL).
Why: The worker had no upper bound on LLM spend — a misconfigured cron on a large queue could mint unbounded OpenRouter/OpenAI/Anthropic cost before anyone noticed. Added ENTITY_EXTRACTION_MAX_CALLS env (default 10000, 0 = unlimited), a module-scoped llmCallCount counter, a pre-call gate that throws ExtractionCostCapError when the cap is reached, and graceful abort in the main loop that returns remaining claimed rows to 'pending' so the next invocation can resume. Summary now reports truncated / truncated_reason / llm_calls so callers can observe the cap firing.
Add entities, edges, thought_entities, entity_extraction_queue, and consolidation_log tables for automatic entity/relationship extraction from thoughts. A trigger on the thoughts table enqueues new/updated rows for an async worker (shipped separately in integrations/entity-extraction-worker/). Positioned as the extraction-side complement to recipes/ob-graph/ — the two schemas are independent; ob-graph is a manual 2-table graph, this is an automatic extraction pipeline with evidence-bearing links. Part of the OB1 alpha milestone.
Base OB1 thoughts.id is UUID (gen_random_uuid()), not BIGINT. Fixed thought_entities.thought_id, entity_extraction_queue.thought_id, consolidation_log.survivor_id, and consolidation_log.loser_id. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…mparison Why: The comparison table advertised the UUID/BIGSERIAL mismatch as a design choice, but it was the BLOCKER-1 bug -- thought_id FKs into upstream thoughts.id (UUID) could not be BIGINT. With the UUID alignment restored via 495a183 (cherry-picked), the comparison row is stale and misleading; drop it.
…correct MCP posture Why: The anon/authenticated SELECT GRANTs with no RLS exposed the entire entity graph, evidence excerpts, queue errors, and consolidation audit trail to anyone with the project URL -- same pattern as the Wave 1 PR-A BLOCKER-2. MCP tools on stock OB1 use the service-role key from an Edge Function server-side, so the "MCP needs anon read access" justification (HIGH-4) is factually wrong. Changes: - Drop all GRANT SELECT ... TO anon on the five tables. - Enable RLS on entities, edges, thought_entities, entity_extraction_queue, consolidation_log. - Add explicit service_role FOR ALL policies (legibility; role bypasses RLS anyway) and a minimum authenticated SELECT USING (true) policy as a multi-tenant scaffold. No anon policy. - Keep service_role full GRANTs and narrow authenticated to SELECT. - Rewrite README Expected Outcome to describe the actual RLS + service-role-only posture, removing the incorrect MCP claim.
…eader Why: HIGH-1 -- the trigger reads NEW.content_fingerprint unconditionally, so on a brain that skipped Step 2.6 the migration would install silently and then crash the first INSERT INTO thoughts, breaking the primary write path. Adding an information_schema precheck at the top of the file hard-fails at install time with a clear remediation pointer, which also covers MEDIUM-4 (the backfill reads the same column). INFO-1 -- the top-of-file comment was a renaming artifact; rename "Knowledge Graph Tables" to "Entity Extraction Tables" while we're editing the header block.
Why: entity_extraction_queue and consolidation_log are both unbounded with no documented retention strategy. The README positions this as a production schema for long-running brains, which need operational guidance. Add a "Pruning and Retention" section with terminal-state DELETEs for the queue (safe because the trigger re-queues on edit) and a simple age-based DELETE for the consolidation log. No schema change -- retention stays an operator choice.
Why: The security posture of this schema depends on the RLS pattern documented in primitives/rls/. Declaring the dependency in metadata makes the gate's Rule 10 linkage visible and tells downstream indexers that this schema consumes the rls primitive.
Why: Deno's global fetch has no body-read timeout on Supabase Edge. A stuck OpenRouter / OpenAI / Anthropic upstream would hang the invocation until the 150s platform wall-clock killed it, leaving claimed rows in 'processing' with no status update. Added fetchWithTimeout() helper that wraps AbortController around fetch, defaulting to 60s and overridable via FETCH_TIMEOUT_MS. All three LLM call sites in extractEntities now route through it. Timeouts surface as thrown errors, caught by the per-item try/catch, and flow through markError — so the standard retry-then-fail path handles them (reset to 'pending' on attempt < 5, 'failed' on cap).
Why: Thought content was interpolated directly into the LLM prompt, giving
any captured text (emails, browser history, Slack dumps) a direct channel
to override the extraction instructions — which would then flow unescaped
into entities.canonical_name, a TEXT column rendered by dashboards and
MCP tools (stored XSS vector). Four layered defenses:
1. Wrap content in <thought_content>...</thought_content> tags and tell the
model explicitly that content inside is untrusted data, not instructions.
2. Escape literal tag occurrences in content so an attacker can't forge a
close-tag and break out of the wrapper.
3. Enforce response_format: { type: "json_object" } on OpenRouter (OpenAI
already had it) so prose wrapping doesn't crash the JSON parser.
4. sanitizeEntityName() strips control chars and clips to 200 chars before
entities land in the DB — caps the blast radius of a surviving injection.
Why: Supabase Edge Functions hard-kill at 150s. With limit=50 and LLM calls averaging 3s each, cumulative latency alone could exceed the budget — killing the invocation mid-loop and leaving rows stuck in 'processing' with no recovery except the manual SQL from the README. Added startTime at invocation, INVOCATION_BUDGET_MS = 140000 (30s headroom), and a per-item gate that releases remaining claimed rows back to 'pending' so the next invocation picks them up. Surfaces as summary.truncated=true with truncated_reason='wall_clock_budget', plus elapsed_ms for monitoring.
Why: The knowledge-graph schema's queue_entity_extraction trigger re-queues
a thought when its content changes. The worker then re-runs extraction but
only upserted new thought_entities rows — it never deleted links from the
prior extraction. So editing a thought from {Alice, Bob, PostgreSQL} to
{Alice, Redis} ended up with the thought linked to all four entities. Over
time this silently corrupts the graph: edges.support_count inflates with
thoughts that no longer mention the underlying entity. Delete our own
prior links (scoped to source='entity_worker' so we don't clobber links
from other sources) before re-writing. Non-fatal if DELETE fails — we
still attempt the upserts, because missed extraction is worse than drift.
Why: The code fixes for BLOCKER-3 (ENTITY_EXTRACTION_MAX_CALLS), BLOCKER-4 (FETCH_TIMEOUT_MS), and WARNING-2 (wall-clock budget) added env knobs and summary fields (truncated, truncated_reason, llm_calls, elapsed_ms) that weren't surfaced anywhere users would see. Also documented the 'skipped' queue state per INFO-1 — the worker marks system-generated thoughts (metadata.generated_by) as skipped, and until now only the schema comment knew that. Added queue status reference and a short note that dry-run leaves the queue untouched.
Generates per-entity markdown wiki pages by aggregating thought_entities links and synthesizing with an LLM. Three output modes (file, entity-metadata, thought) let users choose between filesystem, graph metadata, or thought store — with the pollution trade-offs of the last documented.
Adds thought_edges table (supports, contradicts, supersedes, evolved_into, depends_on, related_to) plus valid_from/valid_until/decay_weight columns on the existing entity edges table. Documents open design questions around the supersedes overlap with provenance-chains.
Classifier that populates thought_edges: Haiku filters candidate thought pairs, Opus confirms the relation. Cost-capped, batch-processed. Documents the unresolved question of whether to mirror supersedes edges back to public.thoughts.
Also fixes P1-3: upsert dossier by entity_id, move timestamp to metadata. Dossier thoughts now (a) compute and store an embedding so they are retrievable via match_thoughts, matching the MCP capture flow in server/index.ts, and (b) dedup by metadata.wiki_entity_id instead of content fingerprint, so regenerating a wiki for the same entity refreshes the existing row in place rather than accumulating duplicates. The per-run timestamp now lives in metadata.generated_at, not the content body, so upsert_thought's content-fingerprint dedup is not defeated.
…tion prereq The README linked to schemas/entity-extraction/ and integrations/entity-extraction-worker/, neither of which is in OB1 main. Add a prominent warning at the top of the README and in the Prerequisites block noting the companion PRs that must land first, remove broken internal links to those pending paths, and add a deferred-follow-up warning about the O(N) listBatchCandidates scalability concern for large brains.
Thought content is now wrapped in <thought id="..."> fences and passed in a dedicated INPUT SNIPPETS block that the system prompt explicitly marks as untrusted. A light pre-scrub strips control chars, neutralizes literal <thought> tags that a malicious snippet could use to close the outer fence, and flags common injection phrases in place. The system prompt now tells the model to surface suspicious snippets under Open Questions rather than obey them. This is defense in depth — wikis are regenerable, but thought-mode dossiers would otherwise write a hijacked wiki back into the searchable knowledge graph.
When --semantic-expand is set, make one probe embedding call at startup and verify the returned dimension matches the expected pgvector(1536) column size. Also call match_thoughts with a dummy vector of the expected size to detect a missing or renamed RPC up front. Bails with a clear remediation message instead of N silent per-entity RPC failures mid-run. Cached per process so the overhead is a single extra call per run.
…d co-mention branch P2: --entity resolves via canonical_name + normalized_name only. The previous README and CLI help implied alias matching worked; it does not. Both now say so explicitly and point users at the SQL-then-id fallback documented in Troubleshooting. P3: fetchTypedEdges already filters co_occurs_with at the SQL layer, but buildSynthesisInput still constructed co_mention_summary and the system prompt still asked the model to render an 'Also co-occurs with' subsection. Removed all three — the branch was permanently dead code.
…odex/chatgpt-mcp-compat Improve ChatGPT MCP compatibility
…emory-openclaw-launch
Ship a community recipe for splitting compound thoughts into atomic single-topic thoughts via an LLM, plus Gmail-specific repair tooling. Components: - atomize-packs.mjs — generic pack-file atomizer with heuristic compound detection and four-provider LLM backend (Claude CLI, Codex, Anthropic, OpenRouter). - re-atomize-gmail-thought.mjs — heals Gmail imports where long bodies were stored whole; splits via the atomizer, re-inserts via upsert_thought, redirects replies_to edges, re-links correspondents. - audit-gmail-pipeline.mjs — JSON/MD report covering scale, metadata completeness, entity-graph integrity, classification distributions, and retrieval probes. - backfill-gmail-correspondents.mjs — idempotent backfill that pre-filters on author-edge presence specifically. - lib/ — shared atomize-text, entity-resolver, and Claude CLI utilities. - test-atomize.mjs — zero-setup sanity test. Ported from the author's private capture pipeline; all personal emails, internal ticket IDs, and hardcoded paths generalized. No secrets; no modifications to the core thoughts table; no DROP / TRUNCATE / unqualified DELETE. Markdownlint clean; metadata.json validates against the OB1 schema.
Codex provider was spawning `codex exec --dangerously-bypass-approvals-and-sandbox` with arbitrary user-controlled memory/email text as the prompt. A prompt injection in a hostile email body could trigger local code execution via the agent's tool access. Removed the codex provider entirely (OpenRouter, Anthropic, and claude-cli cover all use cases without tool access). Added prompt-injection hardening: wrap all user content in <INPUT>...</INPUT> delimiters with an "inert data" instruction, escape literal </INPUT> tags. Redact raw model output from error messages (gated behind ATOMIZE_DEBUG=1).
…ction atomize-packs.mjs now: - loads recipes/atomizer/.env.local resolved relative to the script (so the documented `node atomize-packs.mjs --provider=openrouter` path no longer fails with "requires OPENROUTER_API_KEY" when the key lives in .env.local) - defaults to openrouter provider (codex provider was removed) - warns when --concurrency > MAX_CONCURRENCY is clamped, instead of silent - skips memories whose memoryId matches -split-N$ or that carry metadata.atomization.parent_id, so re-runs don't double-split children - writes only a 60-char preview + fingerprint into atomization-errors.json by default (full text persists only with ATOMIZE_DEBUG_ERRORS=1) to avoid duplicating sensitive memory content
…mize - re-atomize-gmail-thought.mjs: load .env.local relative to script, add &order=id.asc, warn when hitting the default 1000-row cap on --all, wrap main in an async function with .catch() for consistent exit, document partial-failure recovery via metadata.re_atomized_from. - backfill-gmail-correspondents.mjs: script-relative .env.local; move the per-2000 progress log inside the per-thought loop so it actually fires. - audit-gmail-pipeline.mjs: script-relative .env.local; sbCount() now throws on !res.ok instead of silently returning 0 (was hiding auth/query errors as "zero findings"); use explicit jsonb aliases like `thread_id:metadata->gmail->>thread_id` so reads don't break on PostgREST version changes. - test-atomize.mjs: script-relative .env.local, drop codex provider reference, default to openrouter.
- upsertPersonByEmail orphan-adoption is now race-safe: PATCH conditionally on `canonical_email=is.null`, re-SELECT by email if the winner already adopted the orphan with a different canonical_email. Prevents two concurrent backfill workers from linking two different emails to the same entity row. - Resolver log includes only email domain by default (set ENTITY_RESOLVER_DEBUG=1 for full addresses); the 23505 fallback error drops the email and reports only the domain. - makeSbClient.call() error message strips the query string by default so PostgREST filter values (emails, thread_ids) don't leak into shared logs. Full URL available behind ENTITY_RESOLVER_DEBUG=1. - Document loadEnv constraints (UPPER_SNAKE keys, single-line values, process.env wins, caller should pass absolute script-relative path).
- Replace the "Credential Tracker" section that instructed users to paste service-role keys into a text editor with a structured table + security warning. Keeps service-role keys confined to .env.local. - Drop codex from the supported provider list + document why it was removed (prompt-injection → local-code-execution on untrusted input). - Document the 1000-row default cap on re-atomize --all, the partial- failure recovery via metadata.re_atomized_from, and the new debug env flags (ATOMIZE_DEBUG, ATOMIZE_DEBUG_ERRORS, ENTITY_RESOLVER_DEBUG). - Add recipes/atomizer/.env.example with placeholder values (uses "your-…-placeholder" strings that match the Gate's .env allowlist).
…tale refs - entity-resolver.mjs: orphan adoption now detects zero-row PATCH via Prefer: return=representation. If another worker already adopted the orphan with a DIFFERENT email, we fall through to the disambiguated insert path (case c) instead of incorrectly returning the winner's id. Extracted into tryAdoptOrDisambiguate() for clarity. - claude-cli.mjs: stderr/stdout snippets in error messages are gated behind ATOMIZE_DEBUG=1; default prints only byte counts so arbitrary user email/memory text the CLI echoed doesn't end up in logs. - re-atomize-gmail-thought.mjs: buildAtomizeOpts() pre-loads the OpenRouter key when no --provider flag is passed, because atomize-text.mjs defaults to 'openrouter'. Previously running `node re-atomize-gmail-thought.mjs --id=123` without the flag hit a spurious "opts.openrouterApiKey" error at runtime. - README.md / metadata.json: drop stale 4-provider references that still listed Codex as a supported option.
Add a non-blockquote line between two adjacent GitHub alert callouts so markdownlint stops treating them as one blockquote with a blank line inside. Restores a clean run on recipes/atomizer/README.md. The rest of the repo-wide Markdown Lint failure is pre-existing and covered by a separate cleanup PR (tracker in MEMORY, pattern matches NateBJones-Projects#161/NateBJones-Projects#215).
alanshurafa
force-pushed
the
contrib/alanshurafa/atomizer
branch
from
May 19, 2026 00:09
3290fca to
5212179
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A new community recipe under
recipes/atomizer/that splits compoundmulti-topic thoughts into atomic single-topic thoughts via an LLM, plus a
Gmail-specific repair + audit toolkit.
Source: ported from my private ExoCortex capture pipeline (generic
atomization orchestrator + Gmail re-atomize / audit / correspondent-backfill
scripts). All personal emails, internal ticket IDs, and hardcoded paths
were stripped during the port.
Files
atomize-packs.mjs— generic JSON pack-file atomizer. Heuristic compounddetection (sentence count, enumerations, semicolons, conjunction density)
followed by LLM split. Three-provider backend: OpenRouter (default),
Anthropic API, Claude CLI.
re-atomize-gmail-thought.mjs— heals whole-bodygmail_exportthoughts.Parses the
[Email from X to Y | Subject ... | date]prefix, atomizes thebody, inserts atoms via
upsert_thought, redirectsreplies_toedges,re-links correspondents, deletes the original.
audit-gmail-pipeline.mjs— JSON or markdown report: scale, metadatacompleteness, entity-graph integrity, classification distributions, top
correspondents, atom samples, retrieval probes.
backfill-gmail-correspondents.mjs— idempotent backfill withauthor-edge-specific pre-filter.
lib/— sharedatomize-text.mjs,entity-resolver.mjs,claude-cli.mjs.test-atomize.mjs— zero-setup sanity test..env.example— credential template.Requires
npm install— pure built-ins).thoughts.source_type,thoughts.metadatajsonb,entities,thought_entities,thought_edges,plus an
upsert_thought(p_content, p_payload)Postgres function. TheREADME lists the minimum columns required per table.
Tested
Live on my own Open Brain instance over a 350-thought STARRED Gmail corpus:
27% atomized, 100% author-edge coverage after backfill, 144
replies_toedges built, entity-keyed retrieval probe exact-match.
Pre-review pipeline
This branch went through an automated Phase B ultra-review (Claude
gsd-code-reviewer role + Codex
codex exec+ dedicated security pass)before opening the upstream PR. Round 1 surfaced 1 P0, 5 P1, 11 P2,
8 P3 findings across three reviewers. Round 2 surfaced 1 new P1.
Round 3 reviewed clean. All P0 / P1 findings are fixed:
codexprovider entirely — it shelled out with--dangerously-bypass-approvals-and-sandboxon arbitrary user-controlledmemory/email text, which is a prompt-injection → local-code-execution
primitive. Replaced with
<INPUT>delimiter hardening for remainingproviders.
atomize-packs.mjsnow loads.env.localscript-relative sothe documented
node atomize-packs.mjs --provider=openrouterpath works.re-atomize-gmail-thought.mjspre-loadsOPENROUTER_API_KEYfor the default-provider path, warns when hitting the implicit 1000-row
cap, wraps
main()in async try/catch.entity-resolver.mjsorphan adoption is now race-safe — PATCHwith
Prefer: return=representationdetects zero-row results, fallsthrough to the disambiguated insert path instead of returning the wrong
entity id.
(gated behind
ENTITY_RESOLVER_DEBUG=1); Claude CLI and LLM responsesnippets in errors are gated behind
ATOMIZE_DEBUG=1;atomization-errors.jsonpersists only a 60-char preview + fingerprintper failure unless
ATOMIZE_DEBUG_ERRORS=1.service-role keys into a text editor was replaced with a security table
and warning to keep keys only in
.env.local.P2/P3 follow-ups (non-blocking):
upsert_thought+ edge-redirect + delete RPC.Documented in README "Partial-failure recovery" for now.
shell: true+child.kill()can leak the underlying CLI process onWindows. Not a correctness bug; timeouts still fire.
Known architectural flag
This recipe assumes an Enhanced-Thoughts-style schema (
entities,thought_entities,thought_edgestables +upsert_thoughtRPC) that isnot yet part of upstream OB1
origin/main. This is deliberate — therecipe is a stand-alone opt-in capability, not a core-schema change. The
README documents the minimum DDL required and users without those tables
will see clear errors pointing at the prerequisites.
Note to reviewers
This is the pre-review fork PR. The upstream PR to
NateBJones-Projects/OB1will be opened in a separate Phase C step.Please do not merge this fork PR — it exists so the review loop has a
stable target.