Skip to content

fix: apply FilterPolicy.MERGE correctly in in-memory retrievers#12066

Merged
davidsbatista merged 5 commits into
deepset-ai:mainfrom
Aarkin7:fix/filter-policy-merge-in-memory-retrievers
Jul 20, 2026
Merged

fix: apply FilterPolicy.MERGE correctly in in-memory retrievers#12066
davidsbatista merged 5 commits into
deepset-ai:mainfrom
Aarkin7:fix/filter-policy-merge-in-memory-retrievers

Conversation

@Aarkin7

@Aarkin7 Aarkin7 commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Related Issues

Proposed Changes:

InMemoryBM25Retriever and InMemoryEmbeddingRetriever both documented FilterPolicy.MERGE, but for the common comparison-filter shape they were effectively behaving like REPLACE: the runtime filter overwrote the init-time filter instead of narrowing the result set.

This PR switches both retrievers to the shared apply_filter_policy() helper in their sync and async paths, so init and runtime filters are combined correctly. It also adds focused regression tests for the broken behavior in both retrievers and includes a release note.

How did you test it?

  • Ran hatch run test:unit test/components/retrievers/test_in_memory_bm25_retriever.py test/components/retrievers/test_in_memory_embedding_retriever.py
  • Ran hatch run test:unit test/document_stores/test_filter_policy.py test/components/retrievers
  • Ran hatch run test:types
  • Manually verified the important cases against in-memory BM25 and embedding retrieval:
    • MERGE combines init and runtime filters
    • REPLACE still preserves the old override behavior
    • same-field MERGE still follows the helper's existing warning/runtime-wins behavior

Notes for the reviewer

  • The production change is intentionally small and only touches the two in-memory retrievers that implement filter_policy.
  • Wrapper retrievers were not changed because they just pass filters through to the underlying retriever.
  • The new tests cover both sync and async execution paths.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

@Aarkin7
Aarkin7 requested a review from a team as a code owner July 19, 2026 14:36
@Aarkin7
Aarkin7 requested review from davidsbatista and removed request for a team July 19, 2026 14:36
@vercel

vercel Bot commented Jul 19, 2026

Copy link
Copy Markdown

@Aarkin7 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions

github-actions Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/components/retrievers/in_memory
  bm25_retriever.py
  embedding_retriever.py
  haystack/document_stores/in_memory
  document_store.py
  haystack/document_stores/types
  filter_policy.py
Project Total  

This report was generated by python-coverage-comment-action

@vercel

vercel Bot commented Jul 20, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
haystack-docs Ready Ready Preview, Comment Jul 20, 2026 8:12am

Request Review

@davidsbatista davidsbatista left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good!

@davidsbatista
davidsbatista merged commit 20d9bc4 into deepset-ai:main Jul 20, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FilterPolicy.MERGE silently drops init filters in InMemoryBM25Retriever and InMemoryEmbeddingRetriever

2 participants