fix(retrieval): suppress weak lexical fusion hits

[777genius]

Russian and punctuation-heavy queries could promote broad substring matches into weighted RRF, displacing relevant dense candidates. This change normalizes lexical terms with Unicode word extraction and casefolding, removes a bounded RU/EN low-information word set, and requires two informative matches for multi-term keyword queries while preserving single-term and dense-only retrieval.

It also retains an explicit trigram-indexable PostgreSQL prefilter and preserves escaped punctuation-bearing signals such as 100%, so they cannot match decoys such as 1000.

Real frozen-corpus retrieval-only A/B on the same 39 paired requests:

• complete-question Recall@5: 17.86% -> 25.00% (+7.14 pp, +40% relative)
• complete-question Recall@10: 21.43% -> 39.29% (+17.86 pp, +83% relative)
• micro block Recall@10: 28.33% -> 50.00% (+21.67 pp, +76% relative)
• MRR@10: 24.67% -> 39.09% (+14.42 pp, +58% relative)
• top-10 gains/losses: 8 / 0
• candidate p95 retrieval latency: 979 ms
• candidate retrieval failures: 0 / 39

The A/B reused sealed baseline responses and invoked no answer model. A test-only adapter bypassed release-lease admission because baseline and candidate releases cannot own the same active profile simultaneously; ranking, PostgreSQL/Qdrant lanes, canonical hydration, and response bounds remained production implementations.

Validation:

• 97 focused and architecture tests passed
• 28 final targeted tests passed
• remediation regressions: 32 passed, 1 skipped
• independent hosted review of exact SHA 80ad724487e14b18d38872e795d7944d78954c74: approve, no blocking findings
• Ruff passed
• file-size boundary passed
• git diff --check passed

Summary by CodeRabbit

• Bug Fixes
  • Improved keyword search handling for punctuation, capitalization, Unicode variations, and accented text.
  • Added better support for English and Russian queries by ignoring low-information words while preserving meaningful terms.
  • Prevented weak or empty keyword searches from returning irrelevant matches.
  • Improved matching for literal percent signs and underscores.
  • Preserved dense-search results when keyword search produces no usable terms.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: a77319ca-3784-4790-a764-96cd733d816a

📥 Commits

Reviewing files that changed from the base of the PR and between cee6739 and 80ad724.

📒 Files selected for processing (5)

• packages/infinity_context_adapters/infinity_context_adapters/postgres/locator_retrieval.py
• packages/infinity_context_core/infinity_context_core/features/context_building/tests/test_locator_retrieval_scoring.py
• tests/adapters/test_locator_parent_lifecycle.py
• tests/adapters/test_locator_retrieval_adapters.py
• tests/adapters/test_postgres_canonical_keyword_trigram.py

💤 Files with no reviewable changes (1)

• tests/adapters/test_postgres_canonical_keyword_trigram.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The PostgreSQL lexical retrieval lane now normalizes Unicode and punctuation, removes low-information terms, matches canonical and folded aliases, enforces minimum relevance, and rejects empty queries. Tests cover keyword matching, dense-only retrieval, score fusion, and decoy exclusion.

Changes

Keyword retrieval

Layer / File(s)	Summary
Keyword normalization and filtering packages/infinity_context_adapters/infinity_context_adapters/postgres/locator_retrieval.py, tests/adapters/test_locator_retrieval_adapters.py	_keyword_terms extracts normalized Unicode words, creates folded aliases, removes duplicates and low-information English/Russian terms, and preserves literal percent characters.
Candidate statement matching packages/infinity_context_adapters/infinity_context_adapters/postgres/locator_retrieval.py, tests/adapters/test_locator_parent_lifecycle.py, packages/infinity_context_core/infinity_context_core/features/context_building/tests/test_locator_retrieval_scoring.py, tests/adapters/test_postgres_canonical_keyword_trigram.py	The candidate statement combines canonical and folded predicates, applies minimum relevance, rejects empty lexical terms, and preserves dense-only and fused retrieval behavior. Tests cover Unicode, weak queries, wildcard handling, and decoy exclusion.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 80ad7

The keyword retrieval changes include normalization, matching, and fusion coverage with no unresolved actionable issue identified. The change is ready to merge after normal checks.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 6.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 15 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: suppressing weak lexical fusion hits in retrieval.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/retrieval-query-normalization-r1

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{A rabbit sorts the words with care
Unicode hops through meadow air
Weak terms tumble from the trail
Strong matches leave a clear detail
Dense and lexical paths now blend
And empty searches find no end}

_{Comment @coderabbitai help to get the list of available commands.}