User summaries include candidate/superseded memories and discard procedural provenance

[coding-totoro]

Problem

generate_user_summary_durable() admits candidate procedures and superseded facts into the user-profile model input, while dropping top-level procedural status, scope, and evidence fields. Both PipelineService and AsyncPipelineService are affected, for initial summaries and incremental updates.

This differs from build_procedural_context(), which restricts procedures to status='active' and non-superseded records. For example, an unaccepted assistant suggestion stored as a candidate procedure can reach the profile summarizer without the information explaining that it is tentative. The resulting profile can misattribute a suggestion or domain-specific procedure as a personal preference; the deterministic defect demonstrated below is in input preparation, not a claim about what every model will generate.

Version and source

• Package: azure-cosmos-agent-memory==0.3.0b2.
• Release and current main checked on 2026-09-08: 552ceae6253018970e4c120ed2000923a7cc08e4.
• Reproduction uses Python 3.14, standard-library mocks, and the released package. No Azure resources, credentials, model calls, embeddings, or writes are needed.
• Sync user-summary path
• Async user-summary path
• Existing active-procedure retrieval contract

The source query only restricts user, creation time for updates, and memory type. The subsequent transcript builder does not preserve top-level status, scope_type, scope_value, source_kind, source_authority, or evidence identifiers. The optional transcript metadata allow-list reads nested metadata; enabling source,category,temporal_context restores fact source labels, but does not fix either procedure eligibility or top-level provenance loss.

Offline reproduction

In a fresh virtual environment:

python -m venv .venv
.venv/bin/python -m pip install 'azure-cosmos-agent-memory==0.3.0b2'
# Save the Python block below as repro.py.
.venv/bin/python repro.py --expect affected

The script calls the real SDK generation paths and captures the actual rendered chat input and Cosmos query. Only the container I/O and chat response are mocked. The fake container deliberately returns identical rows in every case; it does not emulate Cosmos SQL. The script checks the real query independently, and checks whether the real pipeline filters and describes those rows before invoking chat. All fixtures are synthetic.

Complete repro.py

"""Offline reproduction: inspect the real SDK's user-summary model input."""
import argparse
import asyncio
import copy
import inspect
import json
from importlib.metadata import version
from unittest.mock import AsyncMock, MagicMock

from azure.cosmos.exceptions import CosmosResourceNotFoundError
from azure.cosmos.agent_memory._container_routing import ContainerKey
from azure.cosmos.agent_memory.aio.services.pipeline import AsyncPipelineService
from azure.cosmos.agent_memory.services.pipeline import PipelineService


def record(name, kind="fact", **fields):
    return dict(id=name, user_id="example-user", thread_id="example-thread",
                type=kind, role="system", content=name,
                created_at="2026-01-01T00:00:00+00:00", metadata={}, **fields)


async def observe(cls, metadata_enabled, incremental):
    rows = [
        record("USER_FACT"),
        record("AGENT_SUGGESTION"),
        record("ACTIVE_RULE", "procedural", status="active", scope_type="domain",
               scope_value="example-domain", source_kind="user_instruction",
               source_authority="high", source_fact_ids=["USER_FACT"]),
        record("CANDIDATE_RULE", "procedural", status="candidate",
               scope_type="domain", scope_value="example-domain",
               source_kind="episode_distillation", source_authority="medium"),
        record("SUPERSEDED_FACT", superseded_by="USER_FACT"),
    ]
    rows[0]["metadata"] = {"source": "user"}
    rows[1]["metadata"] = {"source": "agent"}
    memories, summaries = MagicMock(), MagicMock()
    # Deliberately return the same rows: inspect the actual query separately.
    memories.query_items.return_value = copy.deepcopy(rows)
    summaries.query_items.return_value = []
    summaries.read_item.side_effect = CosmosResourceNotFoundError(message="no prior summary")
    if incremental:
        summaries.read_item.side_effect = None
        summaries.read_item.return_value = {
            "id": "user_summary_example-user", "user_id": "example-user",
            "thread_id": "__user_summary__", "content": "Synthetic prior",
            "created_at": "2025-12-01T00:00:00+00:00",
            "updated_at": "2025-12-01T00:00:00+00:00",
            "metadata": {"structured_summary": {"key_facts": []}, "source_memory_count": 0},
        }
    chat = MagicMock()
    factory = AsyncMock if cls is AsyncPipelineService else MagicMock
    chat.generate = factory(return_value='{"key_facts":["Synthetic output"]}')
    containers = {ContainerKey.MEMORIES: memories, ContainerKey.SUMMARIES: summaries,
                  ContainerKey.TURNS: MagicMock()}
    keys = ("source", "category", "temporal_context") if metadata_enabled else None
    service = cls(MagicMock(), chat, MagicMock(), containers=containers,
                  transcript_metadata_keys=keys)
    result = service.generate_user_summary_durable("example-user", recent_k=None)
    if inspect.isawaitable(result):
        result = await result
    # Inspect only the transcript-bearing message, not the SDK instructions.
    messages = chat.generate.call_args.args[0]
    transcript = "\n".join(m["content"] for m in messages if "ACTIVE_RULE" in m["content"])
    query = memories.query_items.call_args.kwargs["query"]
    return {
        "api": cls.__name__, "metadata_enabled": metadata_enabled, "incremental": incremental,
        "query_filters_procedure_status": "c.status" in query,
        "query_filters_superseded": "superseded_by" in query,
        "candidate_reaches_model": "CANDIDATE_RULE" in transcript,
        "superseded_reaches_model": "SUPERSEDED_FACT" in transcript,
        "active_scope_reaches_model": "example-domain" in transcript,
        "evidence_ids_reach_model": "source_fact_ids" in transcript,
        "agent_source_reaches_model": '"source":"agent"' in transcript,
        "records_rendered_as_system": "[system]" in transcript,
        "source_count": result["metadata"]["source_memory_count"],
    }


async def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--expect", choices=["affected", "fixed"])
    args = parser.parse_args()
    print("azure-cosmos-agent-memory", version("azure-cosmos-agent-memory"))
    for cls in [PipelineService, AsyncPipelineService]:
        for enabled in [False, True]:
            for incremental in [False, True]:
                result = await observe(cls, enabled, incremental)
                print(json.dumps(result, sort_keys=True))
                if args.expect:
                    affected = args.expect == "affected"
                    for key in ["candidate_reaches_model", "superseded_reaches_model",
                                "records_rendered_as_system"]:
                        assert result[key] == affected, (key, result)
                    for key in ["query_filters_procedure_status", "query_filters_superseded",
                                "active_scope_reaches_model", "evidence_ids_reach_model"]:
                        assert result[key] != affected, (key, result)
                    assert result["source_count"] == (5 if affected else 3)
                    assert result["agent_source_reaches_model"] == (enabled or not affected)


if __name__ == "__main__":
    asyncio.run(main())

Actual result

The command succeeds with --expect affected for all eight cases (sync/async × initial/update × metadata disabled/enabled):

Observation	Metadata disabled	Metadata enabled
Query filters procedural status	false	false
Query excludes superseded records	false	false
Candidate reaches model input	true	true
Superseded fact reaches model input	true	true
Active procedure scope reaches model input	false	false
Evidence IDs reach model input	false	false
Fact's agent source reaches model input	false	true
Records have textual [system] labels	true	true
Source count	5	5

[system] here is a textual label inside the transcript, not an assertion that these records become separate system-role chat messages. The stubbed model response is intentionally unrelated to the inputs: the repro does not depend on model behavior.

Expected behavior / suggested correction

• Align procedure eligibility with normal procedural retrieval: active, non-superseded procedures only. Exclude superseded source records before per-thread recent_k trimming and source-count/watermark calculations.
• Preserve source attribution and procedure scope/status/evidence in the profile transcript. Distinguish user-sourced facts, assistant-sourced facts, unknown-source facts, and derived memories; a derived record's stored role='system' does not establish user endorsement.
• Preserve the existing no-source behavior: no new eligible sources should not call the model or advance an existing profile; a new user with no eligible sources should follow NoSourceMemoriesError handling.
• Apply the same behavior to sync and async paths and both initial/update templates.

A locally tested correction passes all eight cases with --expect fixed: source count 3, neither excluded record reaches chat, and scope/evidence/source fields survive. The original 1,213 upstream unit tests pass on the unmodified release; with the correction and seven added regression cases, all 1,220 pass. Existing tests therefore do not currently catch this defect.

Changing future input preparation alone does not repair an already affected prior profile; a deliberate rebuild is needed. This report does not claim to solve all evidence validation or model inference errors. It is separate from the extraction-readiness race fixed in #40/#41.