Skip to content

perf(identity): index pending outbound counts by agent - #1075

Merged
jiashuoz merged 3 commits into
tokencanopy:mainfrom
RauPro:perf/129-pending-count-index
Oct 20, 2026
Merged

jiashuoz merged 3 commits into
tokencanopy:mainfrom
RauPro:perf/129-pending-count-index

Conversation

@RauPro

@RauPro RauPro commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Addresses item 1 of #129: add a partial messages(agent_id) index for status = 'pending_review' AND direction = 'outbound', matching ListAgentsByUser's existing all-time pending_count subquery. DNS error visibility is handled separately in #1032; this PR does not close the whole issue.

The expiry index leads with approval_expires_at, so it cannot directly seek to one agent's holds. The new index supports that lookup without adding a cached counter or changing the query's results. It intentionally adds neither an age cutoff nor a deleted_at predicate absent from the query.

Measurement

Measured the actual SQL emitted by ListAgentsByUser on PostgreSQL 16.15, using EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) and default planner settings. Synthetic fixture: 100 agents across 5 owners, 160,000 historical sent messages, 2,000 outbound holds and 2,000 inbound holds; the selected owner has 20 agents. Rows are 14 days old and carry a 512-byte synthetic raw body.

Both sides used the same rows after VACUUM (ANALYZE); timings are medians of five warm executions after one warm-up.

Measurement Before After
Complete list-agents SQL 1.357 ms 0.201 ms
Complete query shared buffer hits 517 286
Pending-count subplan shared buffer hits 272 41

The pending subplan changed from a bitmap intersection of the agent/history and global review indexes to an index-only scan on idx_messages_agent_pending, with zero heap fetches in this vacuumed fixture. The new index occupied 40 KB. These are local synthetic results, not a production latency guarantee.

Operational risk

Migration 132 uses CREATE INDEX CONCURRENTLY and the existing e2a:no-transaction runner, avoiding a write-blocking index build. Building still consumes I/O and can wait for existing transactions; matching holds incur index-maintenance cost. The migration documents invalid-index recovery, and the runner already refuses to record an invalid build as applied.

No public API or SDK changes.

Test plan

New DB-backed tests verify the embedded migration and idempotent reapplication, index validity/readiness and definition, and pending counts across agents, owners, directions, terminal states, old holds and a transition to sent. Timing and planner-choice assertions are deliberately kept out of CI.

Verified locally with Go 1.26.9 and PostgreSQL 16.15:

  • New regression tests and existing non-transactional migration/invalid-index recovery tests
  • go test -tags integration ./internal/identity -count=1 -timeout=11m
  • go test ./...
  • make fmt-check
  • scripts/check-repository-text-integrity.sh
  • scripts/check-no-committed-credentials.sh
  • git diff --check

Add a partial agent_id index matching ListAgentsByUser's all-time
outbound pending_review count. Build it concurrently through the
existing non-transactional migration runner.

Verify index validity and idempotent reapplication, plus per-agent and
per-owner counts across directions, statuses and resolved holds.

Refs tokencanopy#129 (pending-count index; DNS handling is covered by tokencanopy#1032).

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Raul Ernesto Guillen Hernandez <37725305+RauPro@users.noreply.github.com>
@jiashuoz

Copy link
Copy Markdown
Member

Heads-up: migration slot 132 is now taken on main by 132_owner_email_proof_oidc_source.sql (#1076, shipped in v1.16.0). This PR's migrations/132_messages_agent_pending_idx.sql needs renumbering before merge. #1061 also uses 132, so whichever of the two merges first takes 133 and the other takes 134.

jiashuoz and others added 2 commits October 20, 2026 10:12
Slot 132 was taken on main by 132_owner_email_proof_oidc_source.sql
(tokencanopy#1076). Move this migration to 133 in the filename, its header line,
and the test that reads it from the embedded FS.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jiashuoz

Copy link
Copy Markdown
Member

Pushed to your branch (maintainer edit): merged main and renumbered the migration 132 → 133 (filename, header line, and the test's ReadFile name), since 132_owner_email_proof_oidc_source.sql landed on main with #1076. The uniqueness guard and both pending-count tests pass locally. Will merge once CI is green.

@jiashuoz
jiashuoz merged commit 969ff2c into tokencanopy:main Oct 20, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants