Python: Add portable vector filters and in-memory store

[Eduard van Valkenburg (eavanvalkenburg)]

Motivation & Context

Python now has core vector-store abstractions but no usable built-in store, and its experimental lambda/source-based filter model is unsuitable for safe local evaluation or consistent connector translation. This adds a dependency-free in-memory implementation for development and testing while establishing a data-only filter contract that future connectors can translate without parsing or executing Python source.

Description & Review Guide

• What are the major changes? Adds mutable Filter and FilterGroup operation inputs plus frozen Param declarations; uses validated defensive snapshots at operation boundaries; updates vector search and filtered retrieval to use portable filter trees; derives closed search-tool schemas from native Param declarations; adds InMemoryCollection and InMemoryStore with batch CRUD, filtering, paging, ordering, generated keys, and pure-Python vector search; and adds direct-filter and agent-tool samples using the Azure hotel dataset. It also aligns shared connector concepts with Microsoft.Extensions.VectorData through open provider index/distance values, copied provider annotations, selective per-field embedding generation, provider-side vectorization pass-through, and binary vector payload support.
• What is the impact of these changes? Applications get the first usable core vector store without external database dependencies. Connector authors get explicit filter semantics and extension points for provider-specific operators and field configuration. The experimental callable/string filter surface is replaced before external connectors depend on it.
• What do you want reviewers to focus on? Please focus on the Filter/FilterGroup semantics and provider-extension boundary, the model-controlled Param safety constraints in create_vector_search_tool, and the in-memory store's lifecycle, isolation, generated-key, and distance behavior.

Related Issue

Closes #4166

Contribution Checklist

• The code builds clean without any errors or warnings
• All unit tests pass, and I have added new tests where possible
• The PR follows the Contribution Guidelines
• This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
• This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

[github-actions]

MAF Automated Review — Iteration 1

Result: Findings reported
Scope: full PR (1 commit(s)): 50eba4bcb4b2
Model: gpt-5.6-sol-fast

Overview

The review found 3 verified inline finding(s).

Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
3 verified findings remained after source verification (1 high, 2 medium) across 2 files. Details are attached to the affected lines below.

Affected areas: python/packages/core/agent_framework/_in_memory.py, python/packages/core/agent_framework/_vector_filters.py

[Copilot]

🟡 Changes recommended

A critical regex CPU-exhaustion risk and multiple moderate validation, immutability, and limit issues remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds portable vector filters and a dependency-free in-memory vector store for Python.

Changes:

• Adds immutable filter trees and typed parameter schemas.
• Implements in-memory CRUD, filtering, ordering, paging, and vector search.
• Expands connector metadata, tests, documentation, and samples.

File summaries

File	Description
python/samples/AGENTS.md	Updates sample guidance.
python/samples/02-agents/vector_stores/README.md	Documents the new samples.
python/samples/02-agents/vector_stores/in_memory_search_tool.py	Demonstrates parameterized search tools.
python/samples/02-agents/vector_stores/in_memory_filters.py	Demonstrates direct filtering.
python/packages/core/tests/core/test_vectors.py	Tests filter and vector abstractions.
python/packages/core/tests/core/test_in_memory.py	Tests in-memory behavior and safety.
python/packages/core/AGENTS.md	Updates core API guidance.
python/packages/core/agent_framework/_vectors.py	Integrates filters and vector enhancements.
python/packages/core/agent_framework/_vector_filters.py	Implements filters and parameters.
python/packages/core/agent_framework/_in_memory.py	Implements the in-memory store.
python/packages/core/agent_framework/__init__.pyi	Exports new typing APIs.
python/packages/core/agent_framework/__init__.py	Exports new runtime APIs.
docs/features/vector-stores-and-embeddings/README.md	Documents the new architecture.
.github/CODEOWNERS	Assigns owners for new modules.

Review details

Suppressed comments (2)

python/packages/core/agent_framework/_in_memory.py:423

• The query vector is never checked against the selected field's declared dimensions. If both query and stored vectors are three-dimensional while the definition says two, search succeeds; an empty collection also accepts any query length. Validate the query length before scanning records so the collection definition is enforced deterministically.

        query_vector = _numeric_vector(vector, field_name="query")
        vector_field = self.definition.try_get_vector_field(vector_property_name)
        if vector_field is None:
            raise ValueError("InMemoryCollection vector search requires a vector field.")

python/packages/core/agent_framework/_vector_filters.py:400

• These limits apply per container but do not bound the total number of nodes traversed. A nested value can contain up to 256 children at every level, and this full traversal occurs before the filter validator's 64-node limit, allowing a model-controlled argument to consume excessive CPU and memory. Track and enforce a small global node count during parameter validation.

            for item in mapping.values():
                _validate_param_data(item, seen=seen, depth=depth + 1)
        finally:

• Files reviewed: 14/14 changed files
• Comments generated: 8
• Review effort level: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Six moderate validation and cycle-safety issues remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

python/packages/core/agent_framework/_vector_filters.py:288

• Numeric constraints are accepted for nonnumeric parameters. For example, Param("category", str, minimum=1) emits minimum beside a string schema, and runtime validation silently skips it, so the declared constraint is never enforced. Reject minimum/maximum unless every non-null variant is numeric (or apply them explicitly to numeric variants).

• Files reviewed: 14/14 changed files
• Comments generated: 5
• Review effort level: Balanced

[Eduard van Valkenburg (eavanvalkenburg)]

Addressed the suppressed numeric-constraint finding in 2e52f8035. minimum and maximum are now accepted only when every non-null schema variant is numeric; declarations involving strings, booleans, collections, or mixed numeric/nonnumeric unions fail at Param construction. Nullable numeric parameters remain supported and covered by regression tests.

[Copilot]

🟡 Changes recommended

Three moderate validation and comparison issues remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (6)

Previously missed (2) — in code that hasn't changed since the last review.

python/packages/core/agent_framework/_in_memory.py:429

• Unsupported provider-defined distance functions are rejected only inside _calculate_score. An empty collection, or one where every record lacks this vector, therefore returns an empty successful search instead of reporting that the distance is unsupported. Validate the selected distance before scanning records.
  python/packages/core/agent_framework/_vectors.py:289
• is_auto_generated is used by truthiness without validating its declared boolean type, so values such as 1 or "false" silently enable generated-key behavior. Reject non-booleans before applying the key-field constraint.

This issue also appears on line 586 of the same file.

python/packages/core/agent_framework/_vector_filters.py:577

• Mapping keys are not traversed, so a Param used as a key bypasses the rule that parameters must be the entire filter value. It is then absent from the generated tool schema and can reach a definition-less SupportsVectorSearch implementation unresolved. Inspect both mapping keys and values when detecting nested parameters.

        if isinstance(value, Mapping):
            return any(_value_contains_param(item, seen=seen) for item in cast(Mapping[Any, Any], value).values())

python/packages/core/agent_framework/_vector_filters.py:623

• A supplied mutable parameter value is returned by reference. Because the tool supports arbitrary SupportsVectorSearch implementations, an implementation that mutates the filter can mutate the caller's list or mapping; only defaults currently receive the documented defensive copy. Copy supplied values during substitution as well.

    if isinstance(value, Param):
        if value.name in arguments:
            return arguments[value.name]

python/packages/core/agent_framework/_vector_filters.py:543

• The depth/node limits do not bound snapshot work because the complete value is recursively traversed by _value_contains_param and then deep-copied before validate_filter applies its 8/64 limits. A deeply nested value can raise RecursionError, and an oversized value can consume CPU/memory far beyond the advertised limit before rejection. Build the defensive snapshot with the same bounded traversal, rejecting as soon as either limit is exceeded.

            value = (
                _snapshot_filter(filter_.value, active=active)
                if isinstance(filter_.value, Filter | FilterGroup)
                else deepcopy(filter_.value)
            )

python/packages/core/agent_framework/_vectors.py:588

• bytearray payloads are normalized to bytes during serialization and embedding normalization, but annotation inference declares their field type as "bytearray". Connectors advertising support for the normalized "bytes" representation will reject such a model before serialization. Infer "bytes" for either binary annotation.

        binary_candidate = next((candidate for candidate in candidates if candidate in (bytes, bytearray)), None)
        if binary_candidate is not None:
            return binary_candidate.__name__

• Files reviewed: 14/14 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

Eight unresolved moderate findings affect vector correctness, validation consistency, resource limits, and filter safety.

Review details

Suppressed comments (8)

Previously missed (8) — in code that hasn't changed since the last review.

python/packages/core/agent_framework/_in_memory.py:114

• These direct products and squared norms overflow for valid finite inputs. For example, [1e308, 0] and [-1e308, 0] produce -inf / inf == NaN, after which the clamp returns 1.0 instead of the correct cosine similarity -1.0. Scale vectors before computing cosine (and reject any non-finite final score) so large finite vectors cannot be misranked.
  python/packages/core/agent_framework/_in_memory.py:129
• Hamming distance is the number of dimensions that differ, but dividing by vector length returns normalized Hamming distance instead. For example, [0, 0] versus [1, 0] returns 0.5 rather than 1, changing scores and threshold behavior. Return the mismatch count and update the all-dimensions-different test expectation/add a partial-mismatch case.
  python/packages/core/agent_framework/_in_memory.py:199
• Only the filter operand is restricted to safe in-memory value types. Dictionary records and custom codecs can store arbitrary objects, so these comparisons can invoke actual.__eq__, rich-comparison methods, or custom collection iteration during filter evaluation. Validate stored operands recursively as exact supported data types before dispatch so filtering cannot execute arbitrary record methods.
  python/packages/core/agent_framework/_in_memory.py:430
• Unsupported provider-defined distance functions are rejected only when _calculate_score is reached. An empty collection, or one whose records lack this vector, therefore accepts an unsupported distance and returns an empty result instead of failing consistently. Validate the selected function before scanning records.
  python/packages/core/agent_framework/_vector_filters.py:620
• The depth/node limits are applied only after this snapshot has recursively copied the complete mutable tree. A filter with a very large sequence/group therefore consumes unbounded memory and CPU before the 64-node check can reject it, and sufficiently deep values can raise RecursionError during copying instead of the documented validation error. Enforce the limits while snapshotting rather than after the copy.
  python/packages/core/agent_framework/_vector_filters.py:650
• _value_contains_param only descends into Sequence values, although nested Param references are forbidden in any collection. A set, frozenset, or another non-sequence collection containing a Param therefore passes both this check and _FilterValidator, allowing an unresolved declaration to reach a provider connector. Traverse every non-string Collection (or reject unsupported collection types).
  python/packages/core/agent_framework/_vector_filters.py:675
• The common string operators never validate that their expected value is a string. For example, Filter("text", "contains_text", 1) passes shared validation, and searching an empty in-memory collection succeeds because require_filter_string is only reached while evaluating records. Validate these operators here after parameter resolution so malformed filters fail independently of collection contents.
  python/packages/core/agent_framework/_vectors.py:588
• bytearray payloads are normalized to bytes during serialization and embedding generation, but a bytearray annotation is recorded as type_="bytearray". A connector that correctly declares support for "bytes" will reject this model before receiving the normalized payload. Canonicalize both binary annotations to "bytes".

• Files reviewed: 14/14 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Eduard van Valkenburg (eavanvalkenburg)]

Follow-up to the suppressed findings in the latest review and the preceding review. Completed the agreed handling in bd63d45:

Finding	Handling
Cosine overflow	Fixed. Scale each vector independently before cosine arithmetic, including very small/subnormal values. Reject non-finite similarity before clamping and non-finite final scores for every metric. Euclidean distance uses stable math.dist; other metrics no longer compute unnecessary squares.
Hamming count versus proportion	Retained normalized Hamming deliberately. The existing scores and thresholds use the proportion of unequal dimensions, consistent with SciPy. Added explicit documentation and partial-mismatch/threshold regressions rather than silently changing units.
Stored objects invoking custom comparisons	Not reproduced through normal upsert. Dictionary inputs and custom encoder output both pass through msgspec.to_builtins. Added regressions with nested objects and list subclasses whose comparison/iteration methods raise, confirming normalization before filtering through both paths. No redundant recursive stored-operand scan was added; custom codecs and connector overrides remain trusted Python code, not sandboxed execution.
Unsupported distance on empty collections	Fixed. Reject unsupported functions before scanning, including empty, missing-vector, and fully filtered-out collections.
Snapshot resource limits	Fixed for the filter tree and supported data containers. Perform bounded structural validation before snapshot copying; bound constructor parameter inspection, group-child copying, and parameter defaults before copying. This does not introduce deep immutability or sandbox arbitrary provider-native objects/hooks.
Hidden Param in collections or mapping keys	Fixed. Inspect non-string collections, including sets/frozensets, and mapping keys. Structural validation also applies to search implementations without a collection definition. Existing operator-specific value restrictions remain intact.
Non-string string-operator operands	Fixed centrally for starts_with, ends_with, and contains_text, including after parameter substitution and on empty collections.
bytearray annotation metadata	Fixed. Both binary annotations infer "bytes", matching normalized payloads and connector capability declarations.
Non-boolean is_auto_generated	Fixed. Reject non-booleans before applying generated-key behavior.
Supplied mutable parameter aliasing	Fixed. Copy supplied values per filter invocation, including nested mappings/lists and definition-less custom search implementations, just as for defaults.

All ten items above now have either a code fix or an explicit, documented decision to retain the existing behavior.