[recipes] Gmail smart pull — sensitivity routing + contact entities + atomize fixes - #20
Closed
alanshurafa wants to merge 313 commits into
Closed
[recipes] Gmail smart pull — sensitivity routing + contact entities + atomize fixes#20alanshurafa wants to merge 313 commits into
alanshurafa wants to merge 313 commits into
Conversation
…h current generator
…ch provenance-chains contract
…test_path + traverse_graph) The previous recursive-CTE implementations enumerated every path to every reachable node, which exploded on densely connected graphs (a hub with 1k+ neighbours at depth 2 produced tens of thousands of rows) and depended on the per-path ANY(path) check to break cycles. On a cyclic graph that exceeded the statement_timeout the planner never actually pruned the walk. Replace both functions with iterative plpgsql BFS: - Global seen-set (UUID[]) so each node is visited at most once - JSONB parent-pointer map records the first (parent, relation) that reached each node; BFS's "first discovery wins" invariant is enforced with DISTINCT ON (next_id) - Shared reconstruct_bfs_path() helper walks the parent map end -> start with a safety guard against malformed maps - find_shortest_path keeps bidirectional edge traversal; traverse_graph keeps outgoing-only with optional relationship_type filter - Signatures, argument order, RETURNS shape, and language (plpgsql, no SECURITY DEFINER) match the original so this is a body-only rewrite - All queries scope by p_user_id to preserve the multi-tenant isolation the edge function relies on (service_role bypasses RLS) Update the README's "How It Works" section and function table to describe the new implementation so the docs don't contradict the SQL. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Standalone Node script that exercises traverse_graph and find_shortest_path against a live Supabase project with ob-graph installed. Picks an arbitrary edge, calls both RPCs at depth 1/2/6, and prints row counts + timings so maintainers can confirm the iterative-BFS rewrite stays inside the statement_timeout and returns sensible shapes before shipping. Reads SUPABASE_PROJECT_REF, SUPABASE_SERVICE_ROLE_KEY, and OB_GRAPH_USER_ID from recipes/ob-graph/.env.local so the service-role key never lands in repo history. Sets process.exitCode=1 on any RPC failure so CI or shell chaining picks up a non-zero exit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rsive CTE (preserve contract)
…directional shortest-path
…gth === 2 to protect bidirectional shortest-path
…allel edges produce 3 D-rows
…shortest_path + traverse_graph
…match metadata 1.0.1
…es with HTML comment
Auto-run: markdownlint-cli2 --fix --config .github/.markdownlint.jsonc Manual: 2 MD028 in life-engine/README.md (HTML comment separator) Brings `markdown-lint` CI check from 55 pre-existing errors to 0. Files: - recipes/adaptive-capture-classification/README.md - recipes/adaptive-capture-classification/classifier_prompt.md - recipes/life-engine/README.md - recipes/life-engine/life-engine-skill.md - recipes/ob-graph/README.md (same fix as PR NateBJones-Projects#210; harmless duplicate) - recipes/obsidian-vault-import/README.md - recipes/vercel-neon-telegram/README.md - schemas/workflow-status/README.md No content changes. Pure formatting: blank lines around fences/headings, HTML-comment separators between adjacent blockquotes, list-indent/ numbering normalization. Precedent: PR NateBJones-Projects#161 did the same for 15 files.
…from thoughts Adds a new recipe that ships two Node scripts for wiki-style synthesis over the core `thoughts` table, plus optional Next.js dashboard snippets. - `scripts/synthesize-wiki.mjs` — topic-scoped synthesizer with a built-in `autobiography` mode that groups thoughts by year and asks an OpenAI-compatible Chat Completions endpoint to produce second-person biographical prose per year. Extend the catalogue to add more topics. - `scripts/backfill-gmail-wikis.mjs` — resume-safe per-thread wiki generator for Gmail-imported thoughts. Groups by `metadata.gmail.thread_id`, filters by word-count + message/atom thresholds, writes wiki thoughts with `derived_from` edges to their source atoms. Prefers an `upsert_thought` RPC when present, falls back to plain inserts. - `dashboard-snippets/` — optional Next.js components (Server Action + `/wiki` index + `/wiki/[slug]` detail) to copy into a dashboard. Generalized from ExoCortex origin: replaced the MCP edge-function client with direct PostgREST + service role, swapped the Anthropic-direct call for OpenAI-compatible Chat Completions, removed personal data references, introduced SUBJECT_NAME and SOURCE_TYPE_FILTER env knobs. Complements the in-flight entity-wiki recipe — this one does corpus-slice synthesis and only requires the core `thoughts` table. README documents the thought_edges / upsert_thought prerequisites so users know which OB1 layers they need before running the email-thread pipeline.
github-actions
Bot
added
dashboard
documentation
…lanshurafa/smart-ingest [integrations] Smart ingest Edge Function
…wicegood/dashboard-open-next-cloudflare [dashboards] open-brain-dashboard-next: add Cloudflare Workers deploy support
…lanshurafa/enhanced-thoughts [schemas] Enhanced thoughts columns and utility RPCs
…rain-health-monitoring
…rain-health-monitoring
…lanshurafa/brain-backup [recipes] Brain backup and export
…rain-health-monitoring
…lanshurafa/brain-health-monitoring [recipes] Brain health monitoring views
…ns1002/per-request-mcp [integrations] Fix per-request McpServer instantiation for stable MCP connections
…xcfi-scott/delete-thought-mcp [integrations] delete_thought MCP
…er/typed-reasoning-edges-comment-syntax-fix [schemas] Fix typed-reasoning-edges COMMENT ON syntax error
…uto-capture-claude-code
…lanshurafa/auto-capture-claude-code [skills] Auto-capture Claude Code adapter
Owner
Author
|
Closing as redundant — merged upstream as NateBJones-Projects#220. The fork's |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
recipes/gmail-smart-pull/— a Gmail puller that emits an ingest-ready pack with local sensitivity routing, engagement filtering, contact-based relationship tiers, and LLM atomization of long messages. Ports the EXO-0129 through EXO-0137 work from Alan's ExoCortex second brain into a generalized OB1 recipe.Complements the existing
recipes/email-history-import/(one-email-one-thought onboarding). This recipe is for users whose mailbox is big enough that they need careful filtering, routing, and splitting before ingest.What's ported
scripts/pull-gmail.mjs): read-only Gmail API fetch, quoted-reply + signature stripping, auto-generated-noise filter, engagement gate (threads where you've replied), RFC 2822 threading headers captured at source, structured correspondents parsed once at pull time.scripts/lib/sensitivity.mjs): two pattern sets — restricted (SSN, passport, bank, API keys, passwords, credit cards) and personal (email/phone/health/financial) — tag-only, no enforcement.scripts/lib/atomize-text.mjs): long messages (>= 150 words default) split into atomic thoughts; providersanthropic/openrouter/claude-cli/codex; graceful fallback to whole-message on failure.scripts/lib/entity-resolver.mjs): pure parsing only — the pack carries{ name, email }arrays so a downstream job can upsert correspondents as first-class entities.sql/):merge_thought_metadataRPC for targeted metadata backfills, andentities.canonical_emailcolumn + indexes so the correspondents the pack carries can be upserted as entities. BothCREATE OR REPLACE/IF NOT EXISTS.Atomize fixes included
scripts/lib/atomize-text.mjscarries two fixes that surfaced during real-world use:-pcommand-line flag. Under Windowsshell:true, cmd.exe mangled multi-line prompts containing quotes/newlines so the child received a truncated string and the LLM replied conversationally ("Looks like your message got cut off..."). Same fix applied to thecodexprovider.codexprovider shells out tocodex execso users orchestrating from a Codex session can atomize without crossing streams with a nestedclaude-cli(which fails nested-process detection).Dependencies on other candidates
public.persons/person_tierstables become a natural source for the contacts-cache JSON file that powersrelationship_tier. Not a hard dependency — the README documents two other ways to build the cache (Google Contacts API dump, vCard export).What this recipe does NOT do
sensitivity+sensitiveReasons; your ingest pipeline decides what to do with restricted / personal atoms. The README spells this out explicitly because OB1 is cloud-first and "restricted stays local" isn't automatic.Pre-review status
This is the fork PR. Not pushing upstream yet — waiting on cross-AI review (
gsd-code-reviewer+codex exec) per Alan's OB1 PR protocol before opening the upstream PR toNateBJones-Projects/OB1.Test plan
node --checkpasses on all four JS files (pull-gmail.mjs,atomize-text.mjs,entity-resolver.mjs,sensitivity.mjs) — verified locallymetadata.jsonparses as valid JSON — verified locallymarkdownlint-cli2error count stays at 57 (baseline onorigin/main; this branch adds 0 new errors) — verified locallygmail.readonlyonly — verified inSCOPESconstantCREATE OR REPLACE,IF NOT EXISTS) and contain noDROP TABLE,TRUNCATE, or unqualifiedDELETE FROM— verified by inspectionprovider=anthropicon a >= 150-word synthetic email🤖 Generated with Claude Code