Skip to content

fix(google): make the Google AI embedding provider usable - #124

Merged
jeffgreendesign merged 10 commits into
mainfrom
claude/aug-2026-updates-verification-lifnds
Aug 2, 2026
Merged

fix(google): make the Google AI embedding provider usable#124
jeffgreendesign merged 10 commits into
mainfrom
claude/aug-2026-updates-verification-lifnds

Conversation

@jeffgreendesign

@jeffgreendesign jeffgreendesign commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Summary

An Aug 2026 maintenance pass: verify the core primitives still work, and land the updates the project had drifted behind on.

The headline finding is that the Google AI embedding provider had never worked. gemini-embedding-2 emits 3072 dimensions, but pgvector caps HNSW at 2000 for the vector type, so all four CREATE INDEX ... USING hnsw statements across the Google schemas errored out. EMBEDDING_PROVIDER=google is documented in the README, ENV docs, architecture and install guides — following those instructions could not succeed. Confirmed against a real PostgreSQL 16 + pgvector instance:

ERROR:  column cannot have more than 2000 dimensions for hnsw index

The fix requests 1536 dimensions rather than storing 3072. gemini-embedding-2 is Matryoshka-trained and auto-normalizes truncated output, so cosine distance stays valid with no client-side renormalization, and MTEB across its dimension tiers spans only 67.99–68.17. Indexing a halfvec cast (indexable to 4000) was the alternative, but that is an expression index — any later edit to hybrid_search() that dropped the cast would silently fall back to a sequential scan. At 1536 the Google schemas match the already-proven OpenAI shape and nothing sits near a cap.

Changes

  • Google provider (P0). Request 1536d via outputDimensionality; rewrite the four Google schemas to vector(1536). Migrate off @google/generative-ai, which is end-of-life (its repo is now deprecated-generative-ai-js; pnpm outdated misses it because 0.24.1 is the final frozen release) to @google/genai, which is also what exposes outputDimensionality. Default GOOGLE_EMBEDDING_MODEL moves to the GA gemini-embedding-2.
  • Self-migrating schemas. setup-db-google.sql and setup-db-insights-google.sql now detect a mismatched dimension and recreate the column and index, matching the pattern setup-db-conversation-google.sql already used. This matters because the old script left tables created but unindexed, and CREATE TABLE IF NOT EXISTS would not have narrowed that column — a re-run of even the fixed script would have failed identically.
  • Idempotent re-runs. Added DROP POLICY IF EXISTS across all four setup-db-insights*.sql and all four setup-db-conversation*.sql; a second run previously died on policy ... already exists.
  • hnsw.ef_search race. The pool's connect handler fired the GUC query and dropped the promise, so the first query on a fresh connection could run at pgvector's default ef_search=40 with iterative scans off — the exact condition that makes filtered vector searches silently under-return. Now recorded per client and awaited in pgQuery.
  • Current-generation models. INSIGHT_MODELclaude-sonnet-5, EXTRACTION_MODELclaude-haiku-4-5. Sonnet 5 runs adaptive thinking when thinking is omitted and max_tokens bounds thinking plus output, so insight synthesis is pinned to thinking: disabled with a raised ceiling — otherwise every synthesis would have silently degraded to the rule-based fallback.
  • Security. 81 advisories → 46, critical 1 → 0. Patched multer (HIGH, upload path), ws (HIGH, WebSocket API), sanitize-html; bumped stale tar and new body-parser overrides. Note hono is transitive under the MCP SDK, so the override — not the direct dependency — is what governs it.
  • Major dependency upgrades. openai 4→7, @anthropic-ai/sdk 0.71→0.115, express-rate-limit 7→8 with rate-limit-redis 4→6, google-auth-library 10→11, file-type 21→22, jsdom 29→30, redis 5→6, p-limit, commander, glob, @types/*, and TypeScript 5.7→7.0 (its own commit, independently revertable).
  • Dead code and doc accuracy. Removed audio-processor.ts/image-processor.ts (zero importers since they were added incidentally) and the unused hono dependency. The README advertised "Multimodal — images (Claude vision) and audio (Whisper)" and PPTX support; none of that has ever had a code path, so the feature list now reflects what actually ships (which was also under-selling CSV, JSON, Markdown and ZIP).

Type of Change

  • Bug fix
  • New feature
  • Documentation update
  • Refactoring
  • Other (describe): dependency upgrades, security patches

Checklist

  • I've run pnpm typecheck with no errors
  • I've tested my changes locally with pnpm dev — see the note below on what was run instead
  • I've followed the code style (ESM imports with .js, no console.log)
  • I've updated documentation if needed

On testing: pnpm verify passes end to end (478 tests, up from 444; biome, markdownlint, the security/docs/tool-sync scripts, the esbuild bundle and the docs site build), and dist/index.js was smoke-tested by importing it. Rather than pnpm dev, the database work was verified against a scratch PostgreSQL 16 + pgvector instance: the pre-fix schema was confirmed to fail; the fixed schema applies cleanly to both a fresh database and one already in the broken 3072/no-index state; three consecutive full re-runs are clean; hybrid_search returns correctly RRF-ranked results with all three filters pushed down; semantic_search returns exact 1.0/0.0 cosine similarity on orthogonal vectors; and EXPLAIN confirms Index Scan using chunks_embedding_idx.

Not verified: a live end-to-end ingest against the Google API, which needs a real GOOGLE_AI_API_KEY.

Related Issues

None — this came out of a scheduled maintenance review rather than a filed issue.

claude added 8 commits August 2, 2026 14:46
The Google provider could never complete setup. gemini-embedding-2 emits 3072
dimensions, but pgvector caps HNSW at 2000 for the `vector` type, so all four
`CREATE INDEX ... USING hnsw` statements across setup-db-google.sql,
setup-db-conversation-google.sql and setup-db-insights-google.sql errored out.
EMBEDDING_PROVIDER=google is documented in the README, ENV docs, architecture
and install guides, so following those instructions could not work.

Request 1536 dimensions instead of storing 3072. gemini-embedding-2 is
Matryoshka-trained and auto-normalizes truncated output, so cosine distance
stays valid with no client-side renormalization, and MTEB across its dimension
tiers spans only 67.99-68.17. This removes the failure mode rather than working
around it: an alternative fix is to index a halfvec cast (indexable to 4000),
but that is an expression index, so any later edit to hybrid_search() that
dropped the cast would silently fall back to a sequential scan. At 1536 the
Google schemas match the already-proven OpenAI shape and nothing is near a cap.

Also migrate off @google/generative-ai, which is end-of-life (its repo is now
google-gemini/deprecated-generative-ai-js). pnpm outdated does not flag it
because 0.24.1 is the final frozen release. The replacement @google/genai is
also what exposes outputDimensionality, so the two changes are one change.
batchEmbedContents becomes models.embedContent with an array of Content, and
readGoogleEmbeddings() validates the now-optional embeddings/values fields so a
partial response fails at the call site instead of writing undefined vectors.

Default GOOGLE_EMBEDDING_MODEL moves from gemini-embedding-2-preview to the GA
gemini-embedding-2.

Adds vector-index-dimensions.test.ts, which reads every setup-db*.sql and fails
on an HNSW index over a >2000-dim vector column — verified to flag all four
original indexes. Adds coverage for getEmbeddingDimensions(), which had none.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDjKtobTDeToJnMHRbYjGp
…hono

audio-processor.ts and image-processor.ts have had zero importers since they
were added incidentally in c311a99 (a GCS storage commit). They are not
referenced by any tool, route, doc, or roadmap item. Removing them also drops
one of the two `openai` consumers, which makes the pending v4->v7 upgrade a
single-file change.

src/tools/index.ts was missing registerPgAnalyzeTools, the only tool registrar
absent from the re-export surface. server.ts imports every tool directly, so
nothing was broken by this — the barrel was just incomplete.

hono was declared as a direct dependency but imported nowhere. Note that simply
removing it does NOT clear its advisories: hono is also transitive via
@modelcontextprotocol/sdk and @hono/node-server, so the pnpm override is what
actually governs the installed version. Bumped that override from ^4.11.7
(resolving to a vulnerable 4.12.18) to ^4.12.33, which clears the HIGH CORS
credential-reflection advisory and the hono/jsx cx() XSS advisory.

@hono/node-server's <2.0.5 advisory (Windows serve-static path traversal)
remains: it is pinned by the MCP SDK, forcing a major on it risks breaking the
SDK, and nothing here serves static files through hono.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDjKtobTDeToJnMHRbYjGp
Direct dependencies with advisories reachable from server or CLI code:

  multer      2.1.1 -> 2.2.0    HIGH, DoS via deeply nested field names,
                                reachable from src/api/upload.ts
  ws          8.20.1 -> 8.21.1  HIGH, memory-exhaustion DoS from tiny frames,
                                reachable from src/api/websocket.ts
  sanitize-html 2.17.4 -> 2.17.6 MODERATE, incomplete javascript: URI
                                validation, used by the CLI HTML normalizer

Also refreshes @modelcontextprotocol/sdk 1.29.0 -> 1.30.0 and
@modelcontextprotocol/ext-apps 1.7.1 -> 1.7.5.

Two transitive pins in pnpm.overrides were stale and are bumped:

  tar         >=7.5.7 -> >=7.5.22   clears six advisories including the only
                                    CRITICAL in the tree (decompression DoS,
                                    vulnerable <=7.5.18)
  body-parser (new) >=2.3.0         clears a low-severity DoS; transitive via
                                    express under the MCP SDK

Audit totals: 81 advisories (1 critical / 34 high / 40 moderate / 6 low) ->
57 (0 critical / 30 high / 23 moderate / 4 low).

Not fixed, deliberately: @hono/node-server <2.0.5 (Windows serve-static path
traversal) is pinned by the MCP SDK, forcing a major on it risks breaking the
SDK, and nothing here serves static files through hono. The remaining highs are
in the dashboard/desktop workspaces (next, vite, sharp, electron-updater) and
are out of scope for this pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDjKtobTDeToJnMHRbYjGp
INSIGHT_MODEL: claude-sonnet-4-6 -> claude-sonnet-5 (same $3/$15 list price).
EXTRACTION_MODEL: claude-haiku-4-5-20251001 -> claude-haiku-4-5. Haiku 4.5 is
still current, so that one is alias hygiene rather than a generation bump.

The Sonnet 5 swap is not safe as a bare model-ID change. Sonnet 5 runs adaptive
thinking when `thinking` is omitted (Sonnet 4.6 ran thinking-off), and
max_tokens bounds thinking PLUS response text. synthesizeInsights() asks for a
JSON array under max_tokens: 2000 and falls through to generateRuleBasedInsights
whenever parsing fails — so the new default would have burned the budget before
emitting JSON and silently downgraded every synthesis to the rule-based path.
memory-extraction.ts already carries a comment about exactly this failure mode
from a previous truncation bug.

Guards both ways: thinking is explicitly disabled on that call, and max_tokens
goes 2000 -> 4096 to absorb the Sonnet 5 tokenizer's ~30% higher token counts.
Neither call site passes temperature/top_p/top_k/budget_tokens, all of which now
400 on Sonnet 5, so nothing else in the request shape needed changing.

Docs also referenced `claude-sonnet-4-6-20250514` as the INSIGHT_MODEL default
in docs/ENV.md and configuration.mdx. That model ID does not exist — the Sonnet
4.6 alias carries no date suffix — so anyone copying it would have hit a 404.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDjKtobTDeToJnMHRbYjGp
The feature table advertised "Multimodal — Process images (Claude vision) and
audio (Whisper transcription) alongside documents". No such code path exists or
ever has. audio-processor.ts and image-processor.ts were added incidentally in
c311a99 and were never imported by anything; `git log -S` shows the symbols
appearing in that commit and disappearing when they were removed as dead code,
with no wiring in between. The processor registry has handlers for text, pdf,
docx, csv, xlsx, json and html only — there is no image or audio MIME type
registered anywhere, so uploading either is rejected as an unsupported type.

Multi-Format also listed PPTX. There is no pptx handler and no pptx converter;
the string does not appear anywhere in src/ or scripts/. Replaced the list with
what is actually registered, which was under-selling as much as over-selling —
CSV, JSON, Markdown/text and ZIP archives were all supported but unlisted.

Also removes images/audio from the intro and the privacy section, which
described sending them to providers that were never called.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDjKtobTDeToJnMHRbYjGp
The 'connect' handler fired the hnsw.* GUC query and dropped the promise. pg
emits 'connect' synchronously and hands the client to the waiter without
awaiting anything the handler started, so the first query on a fresh connection
could run before the settings landed — falling back to pgvector's default
ef_search=40 with iterative scans off. That combination is what makes filtered
vector searches silently under-return, which is the exact failure
hnsw.iterative_scan was added to prevent. With Neon scaling compute to zero,
fresh connections are common rather than rare.

Records the tuning promise per client in a WeakMap and awaits it in pgQuery,
which every query path funnels through. This requires checking out the client
explicitly instead of using pool.query(), since pool.query() never exposes the
client the barrier is keyed to.

Also replaces the DO ... EXCEPTION WHEN others THEN NULL wrapper with two
independent SET statements. The old block swallowed every error server-side, so
the client-side .catch() could never fire and there was no way to discover that
tuning had failed. Now a failure rejects and is logged, an older pgvector that
rejects one setting still gets the other, and the connection continues to serve
queries with server defaults.

Leaves HNSW_EF_SEARCH on the local envInt() helper rather than routing it
through the zod config: nothing under src/db/ imports src/utils/config.js, and
the value is already validated as a positive integer. Preserving that layer
boundary is worth more than the consistency.

Adds pg-client.test.ts covering the ordering guarantee, both GUCs, the
tuning-failure path, and client release on error. Verified to fail when the
barrier is removed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDjKtobTDeToJnMHRbYjGp
  openai              4.104 -> 7.3    three majors; a single call site
                                      (embeddings.ts) since audio-processor.ts
                                      was removed. Also clears the unmet
                                      zod@^3 peer warning — v7 supports zod 4.
  @anthropic-ai/sdk   0.71  -> 0.115  well behind, and directly relevant now
                                      that INSIGHT_MODEL is claude-sonnet-5
  express-rate-limit  7.5   -> 8.6    paired with rate-limit-redis 4 -> 6;
                                      the store interface changes together
  google-auth-library 10    -> 11
  file-type           21    -> 22     magic-byte sniffing in the ZIP entry path
  jsdom               29    -> 30
  redis               5     -> 6
  p-limit             5 -> 7, commander 12 -> 15, glob 10 -> 13
  @types/node 22 -> 26, @types/jsdom 21 -> 28, @types/multer 1.4 -> 2.2

Plus current-line refreshes: biome 2.5.6, vitest 4.1.10, tsx 4.23.4, jose
6.2.7, helmet 8.3.0, unpdf 1.8.0, @google-cloud/storage 7.21, tasks 6.3,
mailparser 3.9.14, @types/pg, @types/supertest.

Two follow-on source changes:

- rateLimit.ts uses `limit` instead of `max`. Both work in v8, but `max` is
  deprecated. `standardHeaders: true` is left alone deliberately — switching
  draft versions would change the response headers clients see, which is a
  behavioral change rather than an upgrade.
- Two test files are reformatted by biome 2.5.6's updated formatter. No
  behavior change.

The removed v8 options (draft_polli_ratelimit_headers, onLimitReached) were not
in use. Full `pnpm verify` passes: 478 tests, typecheck, esbuild bundle and the
docs site build.

Not upgraded: typescript 5.7 -> 7.0 is a compiler rewrite, out of scope here and
worth its own pass. Workspace packages (next, react, tailwindcss, @vercel/*)
are likewise left for a separate change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDjKtobTDeToJnMHRbYjGp
TypeScript 7 is the native compiler port, so this is a larger change than the
version bump suggests — kept as its own commit to make it independently
revertable.

Clean on this codebase: no source changes were needed. `pnpm verify` passes
end to end — typecheck, 478 tests, biome, markdownlint, the security/docs/
tool-sync scripts, the esbuild bundle, and the docs site build. The built
dist/index.js was additionally smoke-tested by importing it directly.

Incidental: the docs site build drops from ~15.5s to ~6.7s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDjKtobTDeToJnMHRbYjGp
@vercel

vercel Bot commented Aug 2, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
dashboard Building Building Preview Aug 2, 2026 8:01pm
textrawl Ready Ready Preview Aug 2, 2026 8:01pm

Request Review

@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 68e75978-a048-4975-a0e3-8cef1dc1fd59

📥 Commits

Reviewing files that changed from the base of the PR and between 3d6ef46 and 34c3f42.

📒 Files selected for processing (1)
  • src/db/__tests__/pg-client.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/db/tests/pg-client.test.ts

Walkthrough

The change migrates Google embeddings to @google/genai and gemini-embedding-2 with 1536-dimensional output. It updates embedding validation, vector database schemas, search function signatures, migration checks, and pgvector client setup. Configuration defaults and model examples are updated. Documentation reflects new dimensions, supported formats, privacy details, and provider constraints. Rate-limit options, insight generation settings, exports, dependencies, and tests are also updated.

Sequence Diagram(s)

sequenceDiagram
  participant Application
  participant EmbeddingsService
  participant GoogleGenAI
  Application->>EmbeddingsService: Request embeddings
  EmbeddingsService->>GoogleGenAI: Request 1536-dimensional vectors
  GoogleGenAI-->>EmbeddingsService: Return embedding values
  EmbeddingsService-->>Application: Return validated vectors
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely identifies the main change: making the Google AI embedding provider usable.
Description check ✅ Passed The description includes all required sections and provides detailed changes, testing results, checklist status, and issue information.
Docstring Coverage ✅ Passed Docstring coverage is 91.67% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/aug-2026-updates-verification-lifnds

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (1)
package.json (1)

116-117: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Bound the tar and body-parser overrides to compatible majors. The >= ranges admit future major releases. Use ^7.5.22 and ^2.3.0, or exact patched versions, to prevent incompatible selections during lockfile refreshes.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@package.json` around lines 116 - 117, Update the tar and body-parser
dependency overrides in package.json from unbounded minimum versions to
compatible-major ranges, using ^7.5.22 for tar and ^2.3.0 for body-parser (or
exact patched versions), so lockfile refreshes cannot select future incompatible
majors.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/research/memory-systems.md`:
- Line 292: Update the “Embeddings” paragraph in memory-systems.md to accurately
document only the existing memory schema variants: OpenAI with VECTOR(1536) and
Ollama with VECTOR(1024). Remove or qualify the claims that Google AI and Ollama
v2 schema variants exist, unless corresponding setup-db-memory SQL schemas are
added.

In `@scripts/setup-db-google.sql`:
- Around line 19-25: Update the Option B migration example to recreate
chunks_embedding_idx after adding the embedding column back as vector(1536),
using the same HNSW index definition expected by hybrid_search and
semantic_search.

In `@scripts/setup-db-insights-google.sql`:
- Line 47: Update the database setup flow for proactive_insights.embedding to
migrate existing vector(3072) columns before HNSW index creation: use an
idempotent schema change that replaces or clears the old column with
vector(1536), preserving compatibility with CREATE TABLE IF NOT EXISTS and
requiring existing insights to be re-embedded.

In `@src/db/__tests__/pg-client.test.ts`:
- Line 43: Update the query mock in the pg-client test to explicitly reject when
the SQL is `BOOM`, while preserving the existing resolution behavior for other
non-`SET` queries. In the test covering `BOOM`, assert that `inFlight` rejects
before verifying the release count.

---

Nitpick comments:
In `@package.json`:
- Around line 116-117: Update the tar and body-parser dependency overrides in
package.json from unbounded minimum versions to compatible-major ranges, using
^7.5.22 for tar and ^2.3.0 for body-parser (or exact patched versions), so
lockfile refreshes cannot select future incompatible majors.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3bac8386-dd6c-415e-a63c-ca841ee3e97a

📥 Commits

Reviewing files that changed from the base of the PR and between c6227b1 and d2f8875.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (34)
  • .env.example
  • CLAUDE.md
  • README.md
  • docs/ENV.md
  • docs/architecture/embeddings.mdx
  • docs/architecture/index.mdx
  • docs/concepts/index.mdx
  • docs/getting-started/configuration.mdx
  • docs/getting-started/installation.mdx
  • docs/research/memory-systems.md
  • package.json
  • scripts/migrations/2026-07-10-hybrid-search-filters.sql
  • scripts/setup-db-claims-google.sql
  • scripts/setup-db-claims-ollama-v2.sql
  • scripts/setup-db-claims-ollama.sql
  • scripts/setup-db-claims.sql
  • scripts/setup-db-conversation-google.sql
  • scripts/setup-db-google.sql
  • scripts/setup-db-insights-google.sql
  • src/api/__tests__/upload-process.test.ts
  • src/api/middleware/rateLimit.ts
  • src/db/__tests__/claims-schema.test.ts
  • src/db/__tests__/pg-client.test.ts
  • src/db/__tests__/vector-index-dimensions.test.ts
  • src/db/pg-client.ts
  • src/services/__tests__/embeddings.dimensions.test.ts
  • src/services/audio-processor.ts
  • src/services/embeddings.ts
  • src/services/image-processor.ts
  • src/services/insight-analysis.ts
  • src/services/processor/__tests__/archive-zip.test.ts
  • src/tools/index.ts
  • src/utils/__tests__/query-embedding-cache.test.ts
  • src/utils/config.ts
💤 Files with no reviewable changes (2)
  • src/services/image-processor.ts
  • src/services/audio-processor.ts

Comment thread docs/research/memory-systems.md Outdated
Comment thread scripts/setup-db-google.sql Outdated
Comment thread scripts/setup-db-insights-google.sql
Comment thread src/db/__tests__/pg-client.test.ts Outdated
…potent

Addresses four CodeRabbit findings on #124, all verified valid against a real
PostgreSQL 16 + pgvector instance rather than by inspection.

The premise is now confirmed empirically: applying the pre-fix
setup-db-google.sql fails with

  ERROR: column cannot have more than 2000 dimensions for hnsw index

and leaves the tables created but chunks_embedding_idx absent. That partial
state is what made CodeRabbit's migration finding important — CREATE TABLE IF
NOT EXISTS would not narrow the existing vector(3072) column, so re-running even
the fixed script would have failed at exactly the same place.

Changes:

- setup-db-google.sql and setup-db-insights-google.sql now carry an idempotent
  resize migration, matching the pattern setup-db-conversation-google.sql
  already used. Each detects a mismatched atttypmod, drops the column and its
  index, and recreates them at vector(1536); the CREATE INDEX IF NOT EXISTS
  further down rebuilds the index. Verified to take a broken 3072 database to
  1536 with the index present, in one run.
- The Option B manual migration steps in setup-db-google.sql dropped
  chunks_embedding_idx and never recreated it, which would have left
  hybrid_search and semantic_search on a silent sequential scan. Replaced with
  the automatic migration above rather than patching the instructions.
- CREATE POLICY was unguarded in all four setup-db-insights*.sql and all four
  setup-db-conversation*.sql variants, so a second run failed on "policy already
  exists". Added DROP POLICY IF EXISTS, the pattern the claims schemas already
  use. Three consecutive full re-runs are now clean.
- docs/research/memory-systems.md claimed the setup-db-memory*.sql variants
  cover all four providers. Only two exist (OpenAI 1536, Ollama 1024); corrected.
- pg-client.test.ts mocked every non-SET query as resolving, so the "releases
  the client even when the query throws" case never exercised a throw. The mock
  now rejects BOOM and the test asserts the rejection.

End-to-end functional check on the same instance: hybrid_search returns
correctly RRF-ranked results with source_type, content_type and tag filters
pushed down; semantic_search returns exact 1.0/0.0 cosine similarity on
orthogonal vectors; and EXPLAIN confirms "Index Scan using chunks_embedding_idx"
rather than a sequential scan.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDjKtobTDeToJnMHRbYjGp

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/db/__tests__/pg-client.test.ts (1)

43-43: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use async/throw for the rejection branch.

Line 43 returns Promise.reject(...). This TypeScript guideline requires async/await instead of raw promises. Mark query as async and throw new Error('query failed') for BOOM.

Proposed fix
-				query: (sql: string) => {
+				query: async (sql: string) => {
 					callLog.push(sql);
 					if (sql.startsWith('SET')) {
 						// existing tuning behavior
 					}
-					if (sql === 'BOOM') return Promise.reject(new Error('query failed'));
+					if (sql === 'BOOM') throw new Error('query failed');

As per coding guidelines, “Always use async/await over raw promises for asynchronous operations.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/db/__tests__/pg-client.test.ts` at line 43, Update the test’s query mock
to be async and, for the “BOOM” input, throw the query failure error directly
instead of returning Promise.reject; preserve the existing behavior for all
other inputs.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/db/__tests__/pg-client.test.ts`:
- Line 43: Update the test’s query mock to be async and, for the “BOOM” input,
throw the query failure error directly instead of returning Promise.reject;
preserve the existing behavior for all other inputs.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b0170dc0-a863-4d7a-8942-79563123385a

📥 Commits

Reviewing files that changed from the base of the PR and between d2f8875 and 3d6ef46.

📒 Files selected for processing (11)
  • docs/research/memory-systems.md
  • scripts/setup-db-conversation-google.sql
  • scripts/setup-db-conversation-ollama-v2.sql
  • scripts/setup-db-conversation-ollama.sql
  • scripts/setup-db-conversation.sql
  • scripts/setup-db-google.sql
  • scripts/setup-db-insights-google.sql
  • scripts/setup-db-insights-ollama-v2.sql
  • scripts/setup-db-insights-ollama.sql
  • scripts/setup-db-insights.sql
  • src/db/__tests__/pg-client.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • scripts/setup-db-conversation-google.sql

Follows the repo's TypeScript rule (.cursor/rules/typescript.mdc, "Always use
async/await over raw promises"): the mock's query() is now async and throws
instead of returning Promise.reject.

The SET branch deliberately keeps its raw `new Promise`. That promise must stay
pending until the test resolves it — it is how these tests control when the
tuning lands relative to the query — so there is nothing to await. Added a
comment so it does not read as an oversight. The executor still runs
synchronously, so pendingTuning is populated before query() returns and the
drain loop cannot deadlock.

Re-verified the barrier test still fails when `await clientTuning.get(client)`
is removed from pgQuery: making query() async delays settling by a microtask,
but callLog.push happens synchronously on call, so the assertion still catches a
missing barrier rather than passing vacuously.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PDjKtobTDeToJnMHRbYjGp
@jeffgreendesign
jeffgreendesign merged commit b8dc9ce into main Aug 2, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants