feat(emails): SQL thread-head winnowing — removes the 2000-row accuracy/CPU window#936
Open
seonghobae wants to merge 1 commit into
Open
feat(emails): SQL thread-head winnowing — removes the 2000-row accuracy/CPU window#936seonghobae wants to merge 1 commit into
seonghobae wants to merge 1 commit into
Conversation
…window get_emails previously fetched up to 2000 owner rows into memory and grouped threads in Python: past that window, thread heads, reply counts, and requires_reply silently went wrong, and every request paid the full grouping CPU. - Thread heads now come from SQL: row_number() over (partition by the canonical thread key, newest first) filtered to rank 1, ordered by date, LIMIT'd — accuracy no longer depends on mailbox size. Window functions keep the statement portable (no DISTINCT ON). - Reply counts, requires_reply, and the sent-folder filter are computed from a second bounded query fetching only the page's threads, preserving the existing semantics (including app-side parsed-address matching for sent). The sent folder over-fetches a bounded limit*5 of heads instead of a fixed global window. - Defensive app-side dedupe kept (no-op on PostgreSQL) so non-window mock backends stay exact. - LimitAwareMockSession now emulates the head-winnowing contract and records the head query's LIMIT. Verification: full email API suite 48 passed (incl. exact-threads-beyond- window and interleaved-ordering tests), ruff clean. The postgres smoke test (test_get_emails_reply_tracking_real_postgres_smoke) fails identically on origin/develop (pre-existing, unrelated to this change; verified by baseline run against a fresh pgvector instance). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RkKdtHRLG4wSLh6PVsp8J
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gap (reliability audit #2)
get_emailsfetched up to 2000 rows into memory and grouped threads in Python: beyond that window, thread heads,reply_count, andrequires_replysilently went wrong, and every request paid full grouping CPU.What
row_number() OVER (PARTITION BY canonical-thread-key ORDER BY date DESC)→ rank 1 → ordered + LIMIT'd. Accuracy no longer depends on mailbox size. (Window functions, notDISTINCT ON, keep it portable.)reply_count,requires_reply, sent-folder filter keep their exact semantics (incl. app-side parsed-address matching). Sent folder over-fetches a boundedlimit*5heads instead of a fixed global window.LimitAwareMockSessionupdated to emulate the head-winnowing contract.Verification (local)
exact_distinct_threads_beyond_overfetch_windowandorders_interleaved_threads_by_latest_message_datewhich encode the correctness this fixes; ruff clean.test_get_emails_reply_tracking_real_postgres_smokefails identically onorigin/develop(verified baseline against fresh pgvector:pg16) — pre-existing, unrelated; worth a separate look.Pairs with
#934 (search hybrid retrieval) — together they remove both O(N)-cliff read paths flagged in the reliability audit.
🤖 Generated with Claude Code