diff --git a/.plans/implemented/implementation_step_4.md b/.plans/implemented/implementation_step_4.md new file mode 100644 index 0000000..1aa0540 --- /dev/null +++ b/.plans/implemented/implementation_step_4.md @@ -0,0 +1,47 @@ +# Step 4: Sessions and durable memory + +## Goal and dependency + +Capture agent activity as DocLang and turn explicitly selected, supported observations into reviewable long-term memory. Depends on the storage, jobs, and retrieval contracts from Steps 1–3. + +## Data model + +- A session is a revisioned DocLang document in a DCLX package under `docling://users/{tenant}/{user}/sessions/{session_id}`. Represent turns, tool calls, tool results, and feedback with a documented DocLang vocabulary/profile. Keep attachment references and raw large tool payloads as typed archive parts with size and retention limits. +- A memory is its own DCLX package under `.../memories/{kind}/{id}`; supported kinds begin with preference, fact, entity, concept, and procedure. The DocLang body carries the readable claim and explanation. Typed metadata records confidence, status, author/generator, valid time, and source citations `(session/document revision, xpath)`. +- Links between memories and source nodes live in an indexed edge table and, where portable, a versioned package part. The index is rebuilt from package metadata; a package alone must still expose its provenance. +- Define statuses `proposed`, `accepted`, `rejected`, `superseded`, and `deleted`. A model may propose a memory but does not silently turn an uncertain observation into a durable fact. User corrections override generated claims and remain traceable. + +## Compilation and recall + +1. Append session events using idempotency keys and preserve turn order. Publish a session update through the outbox. +2. A compiler reads a bounded session revision, extracts candidate claims, checks for equivalent or contradictory existing memories, and emits proposed DCLX memory packages with source XPaths. Use deterministic rules where possible; optional local models enrich extraction. +3. Acceptance commits a new memory revision and refreshes retrieval indexes. Rejection records the decision so a retry does not propose the same claim indefinitely. +4. Recall searches accepted memories within the user/tenant scope, records delivered `(memory_id, revision_id)` in a recall ledger for the session, and suppresses redundant repeats while allowing a newer corrected revision. +5. Deleting a session or document evaluates dependent memories. Mark unsupported claims for review or withdrawal, rebuild links, and ensure retrieval does not serve deleted content. Keep explicit retention and purge operations. + +## Conflict and privacy rules + +- Store provenance and validity separately from semantic similarity. Similar text is not proof of the same claim; contradictory dates or subjects must be reviewed. +- Scope private memory to the owning tenant and user by default. Shared memory requires an explicit sharing record and the same authorization checks as documents. +- Keep secrets and raw tool arguments out of long-term memory by default. Add configurable redaction before compilation and a way to remove a claim and its index entries. +- The session writer and memory compiler must work with no cloud services. If models are unavailable, session logging and manual memory creation still work. + +## Work sequence + +1. Define and validate the session and memory DocLang profiles. +2. Implement append, replay, and retention for session packages. +3. Add compilation jobs, provenance links, and review transitions. +4. Add memory search and recall ledger integration. +5. Add correction, supersession, and deletion cascades. + +## Acceptance checks + +- Replay a session after restart with the same ordered turns and citations. +- Compile twice and get no duplicate accepted memory; reject a proposal and verify retries respect the rejection. +- Correct a preference and verify recall returns the accepted newer revision with its provenance. +- Delete a source and verify unsupported claims are flagged or withdrawn and all index entries are updated. +- Attempt cross-user and cross-tenant reads through each memory operation and verify denial. + +## Exit artifact + +Portable DocLang session and memory packages with reviewable provenance, correction, and bounded recall. diff --git a/README.md b/README.md index 83f8502..819e468 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,7 @@ A local context tree that stores documents as immutable DocLang `.dclx` packages. See the [capability guide](docs/index.md), [retrieval guide](docs/retrieval.md), +[session guide](docs/sessions.md), [memory guide](docs/memory.md), [CLI reference](docs/cli.md), and [single PDF and folder examples](examples/adding_resources/README.md). @@ -45,6 +46,9 @@ uv run dc --store ./context-data index status uv run dc --store ./context-data search "invoice" --uri docling://resources/papers uv run --extra vectors dc --store ./context-data index vector rebuild uv run --extra vectors dc --store ./context-data search "invoice" --mode hybrid +uv run dc --store ./context-data session start --id chat-1 +uv run dc --store ./context-data session list +uv run dc --store ./context-data memory list --status proposed ``` Use `uv run dc` or activate the project environment before running `dc` by diff --git a/docs/cli.md b/docs/cli.md index 5ab650b..00eb4a6 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -24,6 +24,9 @@ uv run dc --store ./context-data tree docling://resources/papers -L 2 uv run dc --store ./context-data find "invoice" --uri docling://resources/papers uv run dc --store ./context-data grep "total" --uri docling://resources/papers uv run dc --store ./context-data search "invoice" --uri docling://resources/papers +uv run dc --store ./context-data session start --id chat-1 +uv run dc --store ./context-data session list --status open +uv run dc --store ./context-data memory list --status proposed ``` `add-resource` accepts one document or a folder of documents. With the @@ -51,12 +54,95 @@ collection summary may remain stale until its latest job completes. Every `status` command prints a table by default. Pass `--json` for structured output, for example `dc status --json`, `dc task status JOB_ID --json`, or `dc index vector status --json`. +The main `status` also shows session, memory, and memory compilation job counts +for the selected user. List commands use tables by default and JSON arrays with +`--json`. The session and memory lists support `--limit` and `--offset` pages; +the default page size is 100 and the maximum is 1,000. + +## Sessions and memory + +Start a session before recording events. `start --id` is idempotent for an open +session; omit `--id` to generate one. Session commands use the selected tenant +and user. `append` needs a stable `--key`, so a retried harness hook cannot +duplicate an event. Use `--stdin` for text that should stay out of process +arguments. An optional binary `--attachment` is stored as a typed DCLX part. + +```bash +uv run dc session start --id chat-1 --json +printf 'I prefer concise replies' | uv run dc session append chat-1 \ + --kind feedback --key feedback-1 --stdin --json +uv run dc session replay chat-1 +uv run dc session show chat-1 +uv run dc session list --status open --since 2026-10-01 +uv run dc session close chat-1 +uv run dc memory worker --once +uv run dc memory proposals +``` + +Set `DOCLING_CONTEXT_SESSION=chat-1` to omit the ID from `session append`, or +use `--text` when sending a short event without stdin. `session close` prevents +new events and queues compilation of the final revision. `session jobs ID` and +`memory jobs ID` show compilation outcomes. `memory worker` polls for jobs; +`--once` handles one due job and exits. Queued jobs for older session revisions +become `superseded` when a newer revision is queued. + +`session list` filters by `--status open|closed` and `--since`/`--until` ISO +dates or timestamps. `session trim ID --keep-last N` creates a shorter current +revision. Use `session attachment ID KEY --output FILE` to save one attached +event payload. + +```bash +uv run dc memory list --kind preference --status proposed +uv run dc memory show MEMORY_URI --json +uv run dc memory accept MEMORY_URI +uv run dc memory search 'concise replies' +uv run dc memory recall 'concise replies' --session chat-1 +uv run dc memory correct MEMORY_URI 'I prefer brief answers' +uv run dc memory reject MEMORY_URI --reason 'Incorrect inference' +uv run dc memory delete MEMORY_URI +uv run dc memory purge MEMORY_URI +``` + +`memory list` includes every non-purged status by default, including rejected +and deleted claims. Filter it by `--status`, `--kind`, `--since`, or `--until`. +`memory proposals` is a shortcut for proposed claims. `memory create KIND CLAIM` +creates a user-authored proposal; `--accepted` accepts it immediately. Optional +source citations require all four flags: `--source-uri`, +`--source-document-id`, `--source-revision-id`, and `--source-xpath`. +`memory search` and `memory recall` return accepted claims only. Recall records +which memory revision was delivered to a session and suppresses repeats. + +### Delete and purge + +`delete` withdraws an item from normal use while retaining its history. `purge` +removes that item's stored revisions and content. Both commands are scoped to +the selected tenant and user. + +| Command | Immediate effect | What remains | +| --- | --- | --- | +| `dc session delete ID` | Removes the current session from `session list`, `show`, and `replay`. | Historical revisions and their attachments remain stored; compilation jobs and the session's recall ledger remain. | +| `dc session purge ID` | Deletes the session if necessary, then removes its revision records, attachments, compilation jobs, and recall ledger. | Derived memories are separate packages. The change outbox retains operation metadata. | +| `dc memory delete URI` | Writes a new revision with status `deleted`. The claim leaves memory search and recall. | The current claim and review history remain visible through `memory show URI` and `memory list --status deleted`; older revisions remain stored. | +| `dc memory purge URI` | Removes a previously deleted claim's current package, revisions, search index entries, provenance links, and recall records. | The source session or document and change outbox metadata remain. | + +`memory purge` requires `memory delete` first. `session purge` can be called on +an active or previously deleted session. Removing a source session causes +dependent accepted memories with no other surviving citation to return to +`proposed` for review; it does not purge those memory packages. Purging a memory +does not erase its wording from a cited session or document. If that source is +later compiled again, it can produce a new proposal. + +Package blobs are content addressed and are removed by purge only when no +remaining revision references them. The outbox retains earlier put/delete +identifiers and operation metadata; purge does not clear it. Purge removes the +item's stored content and revisions, but does not erase every related record. `find` and `grep` are case-insensitive substring scans of current documents. They scan at most 1,000 documents, 2,000 nodes per document, and 50,000 nodes total, returning at most 100 hits. They accept an optional URI subtree scope and enforce tenant and user access. They do not -yet provide embedding search or relevance ranking. +yet provide embedding search or relevance ranking. When scoped to memory +packages, they skip claims that are not accepted. `search` uses the SQLite retrieval index and returns exact DCLX citations. New or changed documents are indexed during ingestion. `index status` reports diff --git a/docs/index.md b/docs/index.md index 2d0b56f..fc00c4e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -14,6 +14,8 @@ them through local Python and `dc` command-line APIs. | Remote conversion | Opt-in Docling Serve adapter; explicit endpoint and bounded response | | Inspection | `dc status`, `ls`, `tree`, and task status | | Search | `dc search` provides scoped lexical, vector, and hybrid retrieval, including picture assets with an image-capable model | +| Sessions | `dc session` records, lists, replays, closes, and purges revisioned event packages | +| Durable memory | `dc memory` lists and reviews DCLX claims, runs compilation jobs, and supports scoped recall | `find` and `grep` retain their bounded substring scans. Indexed retrieval verifies citations against the current DCLX revision before returning text. @@ -23,4 +25,6 @@ verifies citations against the current DCLX revision before returning text. - [Retrieval benchmark](../examples/retrieval/benchmark_corpus.py) - [CLI commands](cli.md) - [Storage and jobs](storage.md) +- [Agent sessions](sessions.md) +- [Durable memory](memory.md) - [Single PDF and folder examples](../examples/adding_resources/README.md) diff --git a/docs/memory.md b/docs/memory.md new file mode 100644 index 0000000..a7b77ce --- /dev/null +++ b/docs/memory.md @@ -0,0 +1,56 @@ +# Durable memory + +`MemoryService` stores each claim as a DCLX package under +`docling://users/{tenant}/{user}/memories/{kind}/{id}`. Supported kinds are +`preference`, `fact`, `entity`, `concept`, and `procedure`. The DocLang body +contains the readable claim and explanation. `context/memory.json` stores +confidence, status, author, optional validity dates, source citations, conflict +references, and review history. A citation names a source URI, document ID, +revision ID, and DocLang XPath. SQLite indexes these links and can reconstruct +them from the packages. + +```python +from docling_context import ( + LocalContextStore, MemoryCompiler, MemoryService, Principal, SessionStore, +) + +principal = Principal("acme", "alice") +with LocalContextStore("./context-data") as store: + sessions = SessionStore(store) + sessions.append( + principal, "conversation-1", key="feedback-1", kind="feedback", + text="Remember: I prefer short answers", + ) + compiler = MemoryCompiler(store) + while compiler.run_once() is not None: + pass + memories = MemoryService(store) + for proposal in memories.list(principal, status="proposed"): + print(proposal.claim, proposal.citations) + memories.accept(principal, proposal.uri) +``` + +The compiler accepts only explicit `Remember:` or `Remember that` statements +and `I prefer` turns or feedback. It ignores tool calls and tool results. It +redacts common credential patterns before writing a proposal; callers can pass +a custom `redactor` to `MemoryCompiler`. Compilation runs locally and requires +no model or cloud service. Jobs have leases and retries. Exact normalized +claim matches are deduplicated, including rejected proposals, so retries do not +undo a review decision. Similar or conflicting claims are left for review. + +`create` makes a proposed memory by default. A direct user-authored call may +pass `accepted=True`. Compiler proposals require `accept` before they enter +search. `reject`, `correct`, and `delete` write new package revisions. A +correction is user authored, accepted, and records the revision it supersedes. +`purge` physically removes a deleted claim and its revisions. `search` returns +accepted memories within the current user. `recall(principal, session_id, query)` +records delivered memory and revision IDs; it suppresses repeats in that +session, while a corrected revision can be delivered again. +`list` includes all non-purged statuses by default and can filter by status, +kind, or update time. The `dc memory` CLI exposes listing, review, compilation, +search, recall, and purge; see the [CLI guide](cli.md). + +Deleting a cited session or document checks dependent memories. A claim with +no surviving cited source becomes `proposed` and leaves the retrieval index. +All operations check tenant and user ownership. Sharing is not exposed by this +API. diff --git a/docs/sessions.md b/docs/sessions.md new file mode 100644 index 0000000..688e24e --- /dev/null +++ b/docs/sessions.md @@ -0,0 +1,45 @@ +# Agent sessions + +`SessionStore` appends agent turns, tool calls, tool results, and feedback to a +revisioned DCLX package at +`docling://users/{tenant}/{user}/sessions/{session_id}`. Each event is a DocLang +`` node. The `context/session.json` archive part records its kind, turn ID, +idempotency key, timestamp, and exact XPath. A raw attachment lives in a typed +`attachments/` archive part; metadata stores its size and SHA-256 hash. +Sessions have `open` and `closed` states. `start` creates an empty package; +`close` prevents further appends and queues the final revision for compilation. + +```python +from docling_context import LocalContextStore, Principal, SessionStore + +principal = Principal("acme", "alice") +with LocalContextStore("./context-data") as store: + sessions = SessionStore(store) + sessions.start(principal, "conversation-1") + event = sessions.append( + principal, "conversation-1", key="turn-1", kind="turn", + text="I prefer short answers", turn_id="turn-1", + ) + print(event.xpath, event.revision_id) + print(sessions.replay(principal, "conversation-1")) + sessions.close(principal, "conversation-1") +``` + +Repeated appends with the same key return the existing event. The writer checks +the prior revision before replacing the package, so concurrent writes must retry +after a revision conflict. Sessions contain at most 1,000 events, each at most +20,000 characters. An attachment is limited to 4 MiB, and the session has a +32 MiB total attachment limit. The DCLX package is +portable and can be replayed after restarting the store. + +`trim(principal, id, keep_last=n)` creates a revision with the last `n` events. +Older immutable revisions remain addressable. `delete` removes the logical +session; `purge` removes its revision history, attachments, compilation jobs, +and recall ledger. A source deletion also moves memories with no surviving +citations back to review. +`list` filters current sessions by status and update time; the matching CLI +command is `dc session list`. See the [CLI guide](cli.md) for start, append, +replay, close, retention, and job commands. + +Session URIs and attachments require the owning tenant and user. This version +does not expose sharing of private sessions. diff --git a/src/docling_context/__init__.py b/src/docling_context/__init__.py index fd88448..1145c1b 100644 --- a/src/docling_context/__init__.py +++ b/src/docling_context/__init__.py @@ -1,5 +1,6 @@ """Local-first context storage backed by DocLang packages.""" +from .compiler import MemoryCompiler, MemoryJob, redact_secrets from .contracts import ContextStore from .conversion_config import PdfConversionConfig from .converters import ( @@ -8,6 +9,7 @@ DoclingServeConverter, LocalDoclingConverter, ) +from .durable_memory import MemoryRecord, MemoryService, SourceCitation from .embeddings import ( CLIP_MODEL, DEFAULT_MODEL, @@ -35,6 +37,7 @@ ) from .package import FilePackageStore, PackageError, PackageMissing from .retrieval import ContextResult, RetrievalHit, Retriever, SearchScope +from .sessions import SessionEvent, SessionInfo, SessionStore from .uri import AccessDenied, ContextURI, InvalidURI, authorize_uri, parse_uri from .vectors import EmbeddingProvider, TurbovecIndex, VectorIndex @@ -60,7 +63,11 @@ "Job", "LocalContextStore", "LocalDoclingConverter", + "MemoryCompiler", "MemoryContextStore", + "MemoryJob", + "MemoryRecord", + "MemoryService", "MlxEmbeddingProvider", "NodeAddress", "NodeContent", @@ -74,6 +81,10 @@ "Retriever", "RevisionConflict", "SearchScope", + "SessionEvent", + "SessionInfo", + "SessionStore", + "SourceCitation", "SummaryProvider", "TenantScope", "TreeEntry", @@ -85,5 +96,6 @@ "embed_text_image", "local_embedding_provider", "parse_uri", + "redact_secrets", "stored_embedding_provider", ] diff --git a/src/docling_context/cli.py b/src/docling_context/cli.py index 6a0f284..c3c10ce 100644 --- a/src/docling_context/cli.py +++ b/src/docling_context/cli.py @@ -11,13 +11,16 @@ from collections import Counter from collections.abc import Mapping from dataclasses import asdict, replace +from datetime import UTC, date, datetime, time from pathlib import Path from tabulate import tabulate # type: ignore[import-untyped] from tqdm import tqdm # type: ignore[import-untyped] +from .compiler import MemoryCompiler from .conversion_config import PdfConversionConfig from .converters import LocalDoclingConverter +from .durable_memory import MemoryRecord, MemoryService, SourceCitation from .embeddings import ( CLIP_MODEL, DEFAULT_MODEL, @@ -29,7 +32,15 @@ from .local import LocalContextStore from .models import Principal from .package import bounded_nodes +from .profiles import ( + MAX_ATTACHMENT_BYTES, + MEMORY_KINDS, + MEMORY_STATUSES, + profile_kind, + validate_memory, +) from .retrieval import RetrievalHit, Retriever, SearchScope +from .sessions import SessionInfo, SessionStore from .uri import authorize_uri, parse_uri from .vectors import TurbovecIndex @@ -68,6 +79,77 @@ def _status( ) +def _table_or_json( + values: list[dict[str, object]], *, headers: tuple[str, ...], json_output: bool +) -> None: + if json_output: + _json(values) + elif values: + print( + tabulate( + [[row.get(key, "") for key in headers] for row in values], + headers=tuple(key.replace("_", " ").title() for key in headers), + tablefmt="rounded_grid", + disable_numparse=True, + maxcolwidths=tuple( + 72 if key in {"claim", "text"} else None for key in headers + ), + ) + ) + else: + print("No records.") + + +def _session_rows(items: tuple[SessionInfo, ...]) -> list[dict[str, object]]: + return [asdict(item) for item in items] + + +def _memory_rows(items: tuple[MemoryRecord, ...]) -> list[dict[str, object]]: + return [asdict(item) for item in items] + + +def _session_id(value: str | None) -> str: + session_id = value or os.environ.get("DOCLING_CONTEXT_SESSION") + if not session_id: + raise ValueError("session ID required; pass it or set DOCLING_CONTEXT_SESSION") + return session_id + + +def _timestamp_filter(value: str | None, *, upper: bool = False) -> str | None: + if value is None: + return None + try: + if len(value) == 10: + moment = datetime.combine( + date.fromisoformat(value), + time.max if upper else time.min, + tzinfo=UTC, + ) + else: + moment = datetime.fromisoformat(value) + if moment.tzinfo is None: + moment = moment.replace(tzinfo=UTC) + except ValueError as exc: + raise ValueError("date filter needs an ISO date or timestamp") from exc + return moment.astimezone(UTC).isoformat() + + +def _input_text(args: argparse.Namespace, *, name: str = "text") -> str: + value = getattr(args, name, None) + option = getattr(args, "text_option", None) + if option is not None: + if value is not None: + raise ValueError("use either positional text or --text") + value = option + if args.stdin: + if value is not None: + raise ValueError("use either text argument or --stdin") + value = sys.stdin.read(20_001) + if value is None: + raise ValueError("text argument or --stdin is required") + return value + + def _advance_progress(bar: tqdm, completed: int, total: int) -> None: if bar.total != total: bar.total = total @@ -301,6 +383,11 @@ def _scan( break record = store.get_record(principal, row["uri"]) document = store.get_document(principal, record.uri) + if ( + profile_kind(record.uri) == "memories" + and validate_memory(document)["status"] != "accepted" + ): + continue nodes = bounded_nodes( document, limit=min(2_000, remaining_nodes), text_bytes=2_048 ) @@ -550,9 +637,396 @@ def _parser() -> argparse.ArgumentParser: action="store_true", help="Print complete JSON records instead of the compact table (default: table).", ) + + session = commands.add_parser("session", help="Record and inspect agent sessions") + session_commands = session.add_subparsers(dest="session_command", required=True) + session_start = session_commands.add_parser("start", help="Create an open session") + session_start.add_argument("--id", help="Session ID (default: generated UUID)") + session_list = session_commands.add_parser("list", help="List your sessions") + session_list.add_argument( + "--status", + choices=("open", "closed"), + help="Filter lifecycle state (default: all)", + ) + for command in (session_list,): + command.add_argument( + "--since", help="Updated on or after this ISO date or timestamp" + ) + command.add_argument( + "--until", help="Updated on or before this ISO date or timestamp" + ) + command.add_argument( + "--limit", type=int, default=100, help="Page size, 1–1000 (default: 100)" + ) + command.add_argument( + "--offset", type=int, default=0, help="Skip this many matches (default: 0)" + ) + for name in ("show", "replay", "close", "delete", "purge", "jobs"): + action = "List compilation jobs for" if name == "jobs" else f"{name.title()}" + command = session_commands.add_parser(name, help=f"{action} one session") + command.add_argument("session_id", help="Session ID or full URI") + command.add_argument( + "--json", action="store_true", help="Print JSON instead of a table" + ) + session_append = session_commands.add_parser( + "append", help="Append one session event" + ) + session_append.add_argument( + "session_id", nargs="?", help="ID or DOCLING_CONTEXT_SESSION" + ) + session_append.add_argument("text", nargs="?", help="Event text, or use --stdin") + session_append.add_argument( + "--text", + dest="text_option", + help="Event text, useful with DOCLING_CONTEXT_SESSION", + ) + session_append.add_argument( + "--stdin", action="store_true", help="Read event text from stdin" + ) + session_append.add_argument( + "--kind", + required=True, + choices=("turn", "tool_call", "tool_result", "feedback"), + ) + session_append.add_argument("--key", required=True, help="Stable idempotency key") + session_append.add_argument( + "--turn-id", default="", help="Group related events by turn (default: empty)" + ) + session_append.add_argument( + "--attachment", type=Path, help="Optional bounded binary attachment" + ) + session_append.add_argument( + "--content-type", + default="application/octet-stream", + help="Attachment MIME type (default: application/octet-stream)", + ) + session_trim = session_commands.add_parser( + "trim", help="Retain recent session events" + ) + session_trim.add_argument("session_id") + session_trim.add_argument("--keep-last", type=int, required=True) + session_attachment = session_commands.add_parser( + "attachment", help="Save one event attachment" + ) + session_attachment.add_argument("session_id") + session_attachment.add_argument("key", help="Idempotency key of the attached event") + session_attachment.add_argument("--output", type=Path, required=True) + session_attachment.add_argument("--json", action="store_true") + for command in (session_start, session_list, session_append, session_trim): + command.add_argument("--json", action="store_true") + + memory = commands.add_parser("memory", help="Review and recall durable memories") + memory_commands = memory.add_subparsers(dest="memory_command", required=True) + memory_list = memory_commands.add_parser( + "list", help="List your non-purged memories" + ) + memory_list.add_argument( + "--status", + choices=sorted(MEMORY_STATUSES), + help="Filter review state (default: all)", + ) + memory_list.add_argument( + "--kind", choices=sorted(MEMORY_KINDS), help="Filter memory kind (default: all)" + ) + memory_list.add_argument( + "--since", help="Updated on or after this ISO date or timestamp" + ) + memory_list.add_argument( + "--until", help="Updated on or before this ISO date or timestamp" + ) + memory_list.add_argument( + "--limit", type=int, default=100, help="Page size, 1–1000 (default: 100)" + ) + memory_list.add_argument( + "--offset", type=int, default=0, help="Skip this many matches (default: 0)" + ) + memory_proposals = memory_commands.add_parser( + "proposals", help="List claims awaiting review" + ) + memory_proposals.add_argument("--kind", choices=sorted(MEMORY_KINDS)) + memory_proposals.add_argument( + "--limit", type=int, default=100, help="Page size, 1–1000 (default: 100)" + ) + memory_proposals.add_argument( + "--offset", type=int, default=0, help="Skip this many matches (default: 0)" + ) + for command in (memory_list, memory_proposals): + command.add_argument( + "--json", action="store_true", help="Print JSON instead of a table" + ) + for name in ("show", "accept", "reject", "delete", "purge"): + command = memory_commands.add_parser(name, help=f"{name.title()} one memory") + command.add_argument("uri") + command.add_argument("--json", action="store_true") + if name in {"reject", "delete"}: + command.add_argument("--reason", default="") + memory_correct = memory_commands.add_parser( + "correct", help="Write an accepted correction" + ) + memory_correct.add_argument("uri") + memory_correct.add_argument("claim", nargs="?") + memory_correct.add_argument("--stdin", action="store_true") + memory_correct.add_argument("--explanation") + memory_correct.add_argument("--json", action="store_true") + memory_create = memory_commands.add_parser( + "create", help="Create a user-authored claim" + ) + memory_create.add_argument("kind", choices=sorted(MEMORY_KINDS)) + memory_create.add_argument("claim", nargs="?") + memory_create.add_argument("--stdin", action="store_true") + memory_create.add_argument("--explanation", default="") + memory_create.add_argument("--subject") + memory_create.add_argument("--confidence", type=float, default=1.0) + memory_create.add_argument( + "--accepted", + action="store_true", + help="Accept this user-authored claim immediately", + ) + for field in ("uri", "document_id", "revision_id", "xpath"): + memory_create.add_argument(f"--source-{field.replace('_', '-')}") + memory_create.add_argument("--json", action="store_true") + memory_search = memory_commands.add_parser("search", help="Search accepted claims") + memory_search.add_argument("query", nargs="?", default="") + memory_search.add_argument("--limit", type=int, default=10) + memory_search.add_argument("--json", action="store_true") + memory_recall = memory_commands.add_parser( + "recall", help="Recall claims once per session revision" + ) + memory_recall.add_argument("query", nargs="?", default="") + memory_recall.add_argument("--session", help="ID or DOCLING_CONTEXT_SESSION") + memory_recall.add_argument("--limit", type=int, default=10) + memory_recall.add_argument("--json", action="store_true") + memory_worker = memory_commands.add_parser( + "worker", help="Process queued memory compilation jobs" + ) + memory_worker.add_argument( + "--once", action="store_true", help="Process one due job then exit" + ) + memory_worker.add_argument("--json", action="store_true") + memory_jobs = memory_commands.add_parser( + "jobs", help="List compilation jobs for a session" + ) + memory_jobs.add_argument("session_id") + memory_jobs.add_argument("--limit", type=int, default=100) + memory_jobs.add_argument("--json", action="store_true") return parser +def _session_command( + args: argparse.Namespace, store: LocalContextStore, principal: Principal +) -> None: + sessions = SessionStore(store) + command = args.session_command + if command == "start": + _status(asdict(sessions.start(principal, args.id)), json_output=args.json) + elif command == "list": + since = _timestamp_filter(args.since) + until = _timestamp_filter(args.until, upper=True) + if since and until and since > until: + raise ValueError("--since is later than --until") + items = sessions.list( + principal, + status=args.status, + since=since, + until=until, + limit=args.limit, + offset=args.offset, + ) + _table_or_json( + _session_rows(items), + headers=( + "session_id", + "status", + "event_count", + "updated_at", + "revision_id", + ), + json_output=args.json, + ) + elif command == "show": + info = sessions.get(principal, args.session_id) + values = asdict(info) + jobs = MemoryCompiler(store).jobs(principal, args.session_id) + values["compilation"] = jobs[0].status if jobs else "none" + _status(values, json_output=args.json) + elif command == "append": + session_id = _session_id(args.session_id) + sessions.get(principal, session_id) + if args.attachment and args.attachment.stat().st_size > MAX_ATTACHMENT_BYTES: + raise ValueError("session attachment exceeds 4 MiB") + event = sessions.append( + principal, + session_id, + key=args.key, + kind=args.kind, + text=_input_text(args), + turn_id=args.turn_id, + attachment=args.attachment.read_bytes() if args.attachment else None, + content_type=args.content_type, + ) + _status(asdict(event), json_output=args.json) + elif command == "replay": + events = sessions.replay(principal, args.session_id) + _table_or_json( + [asdict(event) for event in events], + headers=("sequence", "kind", "turn_id", "text", "xpath", "created_at"), + json_output=args.json, + ) + elif command == "close": + _status( + asdict(sessions.close(principal, args.session_id)), json_output=args.json + ) + elif command == "trim": + sessions.trim(principal, args.session_id, keep_last=args.keep_last) + _status(asdict(sessions.get(principal, args.session_id)), json_output=args.json) + elif command == "attachment": + payload = sessions.read_attachment(principal, args.session_id, args.key) + args.output.write_bytes(payload) + _status( + {"output": str(args.output), "size_bytes": len(payload)}, + json_output=args.json, + ) + elif command == "delete": + sessions.delete(principal, args.session_id) + _status( + {"session_id": args.session_id, "status": "deleted"}, json_output=args.json + ) + elif command == "purge": + count = sessions.purge(principal, args.session_id) + _status( + {"session_id": args.session_id, "purged_revisions": count}, + json_output=args.json, + ) + elif command == "jobs": + jobs = MemoryCompiler(store).jobs(principal, args.session_id) + _table_or_json( + [asdict(job) for job in jobs], + headers=( + "job_id", + "status", + "attempts", + "last_error", + "session_revision_id", + ), + json_output=args.json, + ) + + +def _memory_command( + args: argparse.Namespace, store: LocalContextStore, principal: Principal +) -> None: + memories = MemoryService(store) + command = args.memory_command + if command in {"list", "proposals"}: + since = _timestamp_filter(args.since) if command == "list" else None + until = _timestamp_filter(args.until, upper=True) if command == "list" else None + if since and until and since > until: + raise ValueError("--since is later than --until") + items = memories.list( + principal, + status=args.status if command == "list" else "proposed", + kind=args.kind, + since=since, + until=until, + limit=args.limit, + offset=args.offset, + ) + _table_or_json( + _memory_rows(items), + headers=( + "memory_id", + "kind", + "status", + "claim", + "confidence", + "updated_at", + ), + json_output=args.json, + ) + elif command == "show": + _status(asdict(memories.get(principal, args.uri)), json_output=args.json) + elif command == "create": + fields = ( + args.source_uri, + args.source_document_id, + args.source_revision_id, + args.source_xpath, + ) + if any(fields) and not all(fields): + raise ValueError("provide all four --source-* citation fields") + citations = (SourceCitation(*fields),) if all(fields) else () + record = memories.create( + principal, + args.kind, + _input_text(args, name="claim"), + explanation=args.explanation, + subject=args.subject, + confidence=args.confidence, + citations=citations, + accepted=args.accepted, + ) + _status(asdict(record), json_output=args.json) + elif command in {"accept", "reject", "correct", "delete", "purge"}: + if command == "accept": + record = memories.accept(principal, args.uri) + elif command == "reject": + record = memories.reject(principal, args.uri, reason=args.reason) + elif command == "correct": + record = memories.correct( + principal, + args.uri, + _input_text(args, name="claim"), + explanation=args.explanation, + ) + elif command == "delete": + record = memories.delete(principal, args.uri, reason=args.reason) + else: + count = memories.purge(principal, args.uri) + _status({"uri": args.uri, "purged_revisions": count}, json_output=args.json) + return + _status(asdict(record), json_output=args.json) + elif command in {"search", "recall"}: + items = ( + memories.search(principal, args.query, limit=args.limit) + if command == "search" + else memories.recall( + principal, _session_id(args.session), args.query, limit=args.limit + ) + ) + _table_or_json( + _memory_rows(items), + headers=("memory_id", "kind", "claim", "confidence", "revision_id"), + json_output=args.json, + ) + elif command == "jobs": + jobs = MemoryCompiler(store).jobs(principal, args.session_id, limit=args.limit) + _table_or_json( + [asdict(job) for job in jobs], + headers=( + "job_id", + "status", + "attempts", + "last_error", + "session_revision_id", + ), + json_output=args.json, + ) + elif command == "worker": + compiler = MemoryCompiler(store) + if args.once: + job = compiler.run_once() + _status(asdict(job) if job else {"job": "none"}, json_output=args.json) + else: + import time + + try: + while True: + if compiler.run_once() is None: + time.sleep(1) + except KeyboardInterrupt: + pass + + def main(argv: list[str] | None = None) -> int: args = _parser().parse_args(argv) principal = Principal(args.tenant, args.user) @@ -567,6 +1041,21 @@ def main(argv: list[str] | None = None) -> int: ).fetchone()[0] for table in ("documents", "revisions", "jobs") } + user_prefix = ( + f"docling://users/{principal.tenant_id}/{principal.user_id}/" + ) + for name in ("sessions", "memories"): + prefix = user_prefix + name + "/" + counts[name] = store.db.execute( + """SELECT COUNT(*) FROM documents WHERE tenant_id=? + AND substr(uri,1,?)=?""", + (principal.tenant_id, len(prefix), prefix), + ).fetchone()[0] + counts["memory_jobs"] = store.db.execute( + """SELECT COUNT(*) FROM memory_compile_jobs + WHERE tenant_id=? AND user_id=?""", + (principal.tenant_id, principal.user_id), + ).fetchone()[0] _status( {"store": str(store.root.resolve()), **counts}, json_output=args.json, @@ -635,6 +1124,10 @@ def main(argv: list[str] | None = None) -> int: _json(store.index_status(principal)) elif args.command == "add-resource": _add_resource(args, store, principal) + elif args.command == "session": + _session_command(args, store, principal) + elif args.command == "memory": + _memory_command(args, store, principal) elif args.command == "task": _status( asdict(worker.get_job(principal, args.job_id)), diff --git a/src/docling_context/compiler.py b/src/docling_context/compiler.py new file mode 100644 index 0000000..f2b741e --- /dev/null +++ b/src/docling_context/compiler.py @@ -0,0 +1,252 @@ +"""Deterministic, retryable compilation of selected session observations.""" + +from __future__ import annotations + +import re +import uuid +from collections.abc import Callable +from dataclasses import dataclass +from datetime import UTC, datetime, timedelta + +from .durable_memory import MemoryService, SourceCitation +from .local import LocalContextStore, RecordMissing +from .models import Principal +from .profiles import MAX_SESSION_EVENTS +from .sessions import SessionStore + + +def _now() -> str: + return datetime.now(UTC).isoformat() + + +_SECRET = re.compile( + r"(?i)(?:\b(?:api[_-]?key|token|password|secret)\b\s*[:=]\s*\S+|" + r"\b(?:sk-[a-z0-9_-]{12,}|ghp_[a-z0-9]{12,})\b)" +) + + +def redact_secrets(text: str) -> str: + """Remove common credential shapes before storing a proposed claim.""" + return _SECRET.sub("[REDACTED]", text) + + +@dataclass(frozen=True, slots=True) +class MemoryJob: + job_id: str + session_uri: str + session_revision_id: str + status: str + attempts: int + last_error: str | None + + +class MemoryCompiler: + def __init__( + self, + store: LocalContextStore, + *, + redactor: Callable[[str], str] = redact_secrets, + max_attempts: int = 3, + ): + if max_attempts < 1: + raise ValueError("max_attempts must be positive") + self.store = store + self.sessions = SessionStore(store) + self.memories = MemoryService(store) + self.redactor = redactor + self.max_attempts = max_attempts + + def enqueue_missing(self) -> int: + """Recover jobs for session puts whose write completed before enqueue.""" + rows = self.store.db.execute( + """SELECT o.tenant_id,o.uri,o.document_id,o.revision_id + FROM outbox o JOIN documents d + ON d.tenant_id=o.tenant_id AND d.uri=o.uri + AND d.revision_id=o.revision_id + WHERE o.operation='put' AND o.uri LIKE 'docling://users/%/sessions/%' + AND NOT EXISTS (SELECT 1 FROM memory_compile_jobs j + WHERE j.tenant_id=o.tenant_id AND j.session_uri=o.uri + AND j.session_revision_id=o.revision_id)""" + ).fetchall() + added = 0 + for row in rows: + parts = row["uri"].split("/") + if len(parts) != 7 or parts[5] != "sessions": + continue + principal = Principal(parts[3], parts[4]) + try: + self.sessions.uri(principal, row["uri"]) + except (ValueError, PermissionError): + continue + now = _now() + with self.store.transaction(): + self.store.db.execute( + """UPDATE memory_compile_jobs SET status='superseded', + lease_until=NULL,updated_at=? WHERE tenant_id=? AND user_id=? + AND session_uri=? AND session_revision_id!=? AND status='queued'""", + ( + now, + principal.tenant_id, + principal.user_id, + row["uri"], + row["revision_id"], + ), + ) + cursor = self.store.db.execute( + """INSERT OR IGNORE INTO memory_compile_jobs + (job_id,tenant_id,user_id,session_uri,session_document_id, + session_revision_id,status,retry_at,created_at,updated_at) + VALUES (?,?,?,?,?,?,'queued',?,?,?)""", + ( + uuid.uuid4().hex, + principal.tenant_id, + principal.user_id, + row["uri"], + row["document_id"], + row["revision_id"], + now, + now, + now, + ), + ) + added += cursor.rowcount + return added + + def jobs( + self, principal: Principal, session_id: str, *, limit: int = 100 + ) -> tuple[MemoryJob, ...]: + """Inspect compilation outcomes without crossing the session owner scope.""" + if not 1 <= limit <= 100: + raise ValueError("job list limit is out of bounds") + uri = self.sessions.uri(principal, session_id) + rows = self.store.db.execute( + """SELECT * FROM memory_compile_jobs WHERE tenant_id=? AND user_id=? + AND session_uri=? ORDER BY created_at DESC,job_id LIMIT ?""", + (principal.tenant_id, principal.user_id, uri, limit), + ).fetchall() + return tuple( + MemoryJob( + row["job_id"], + row["session_uri"], + row["session_revision_id"], + row["status"], + row["attempts"], + row["last_error"], + ) + for row in rows + ) + + @staticmethod + def _candidate(kind: str, text: str) -> tuple[str, str] | None: + if kind not in {"feedback", "turn"}: + return None + clean = text.strip() + if clean.casefold().startswith("remember:"): + claim = clean[len("remember:") :].strip() + return ( + "preference" if claim.casefold().startswith("i prefer ") else "fact" + ), claim + if clean.casefold().startswith("remember that "): + claim = clean[len("remember that ") :].strip() + return ( + "preference" if claim.casefold().startswith("i prefer ") else "fact" + ), claim + if clean.casefold().startswith("i prefer "): + return "preference", clean + return None + + def _process(self, row) -> int: + principal = Principal(row["tenant_id"], row["user_id"]) + try: + current = self.store.get_record(principal, row["session_uri"]) + except RecordMissing: + return 0 + if current.revision_id != row["session_revision_id"]: + return 0 + events = self.sessions.replay(principal, row["session_uri"]) + if len(events) > MAX_SESSION_EVENTS: + raise ValueError("session exceeds compiler event limit") + count = 0 + for event in events: + candidate = self._candidate(event.kind, event.text) + if candidate is None: + continue + kind, claim = candidate + claim = self.redactor(claim).strip() + if not claim or claim == "[REDACTED]": + continue + citation = SourceCitation( + current.uri, current.document_id, current.revision_id, event.xpath + ) + before = self.memories.create( + principal, + kind, + claim, + citations=(citation,), + author="compiler", + generator="deterministic-v1", + confidence=0.5, + ) + if before.status == "proposed" and before.citations == (citation,): + count += 1 + return count + + def run_once(self) -> MemoryJob | None: + self.enqueue_missing() + now = _now() + with self.store.transaction(): + row = self.store.db.execute( + """SELECT * FROM memory_compile_jobs WHERE + (status='queued' AND retry_at<=?) OR + (status='running' AND lease_until<=?) + ORDER BY created_at,job_id LIMIT 1""", + (now, now), + ).fetchone() + if row is None: + return None + self.store.db.execute( + """UPDATE memory_compile_jobs SET status='running', attempts=attempts+1, + lease_until=?,updated_at=? WHERE job_id=?""", + ( + (datetime.now(UTC) + timedelta(minutes=5)).isoformat(), + now, + row["job_id"], + ), + ) + try: + self._process(row) + except Exception as exc: # noqa: BLE001 - persist worker failures for retry + attempts = row["attempts"] + 1 + status = "failed" if attempts >= self.max_attempts else "queued" + with self.store.transaction(): + self.store.db.execute( + """UPDATE memory_compile_jobs SET status=?,retry_at=?, + lease_until=NULL,last_error=?,updated_at=? WHERE job_id=?""", + ( + status, + ( + datetime.now(UTC) + timedelta(seconds=2**attempts) + ).isoformat(), + str(exc)[:500], + _now(), + row["job_id"], + ), + ) + else: + with self.store.transaction(): + self.store.db.execute( + """UPDATE memory_compile_jobs SET status='completed', + lease_until=NULL,last_error=NULL,updated_at=? WHERE job_id=?""", + (_now(), row["job_id"]), + ) + final = self.store.db.execute( + "SELECT * FROM memory_compile_jobs WHERE job_id=?", (row["job_id"],) + ).fetchone() + return MemoryJob( + final["job_id"], + final["session_uri"], + final["session_revision_id"], + final["status"], + final["attempts"], + final["last_error"], + ) diff --git a/src/docling_context/durable_memory.py b/src/docling_context/durable_memory.py new file mode 100644 index 0000000..22c81d9 --- /dev/null +++ b/src/docling_context/durable_memory.py @@ -0,0 +1,580 @@ +"""Reviewed, provenance-carrying long-term memory packages.""" + +from __future__ import annotations + +import hashlib +import json +import re +import uuid +from dataclasses import asdict, dataclass +from datetime import UTC, datetime +from xml.sax.saxutils import escape + +from doclang import DocLangXDocument + +from .local import LocalContextStore, RecordMissing +from .models import NodeAddress, Principal +from .package import PackageError, PackageMissing +from .profiles import MEMORY_KINDS, MEMORY_PART, MEMORY_STATUSES, validate_memory +from .retrieval import Retriever, SearchScope +from .uri import authorize_uri, parse_uri + + +@dataclass(frozen=True, slots=True) +class SourceCitation: + uri: str + document_id: str + revision_id: str + xpath: str + + +@dataclass(frozen=True, slots=True) +class MemoryRecord: + uri: str + memory_id: str + revision_id: str + kind: str + claim: str + explanation: str + status: str + confidence: float + author: str + generator: str | None + subject: str | None + valid_from: str | None + valid_to: str | None + citations: tuple[SourceCitation, ...] + conflicts: tuple[str, ...] + updated_at: str + + +def _now() -> str: + return datetime.now(UTC).isoformat() + + +def _claim_key(claim: str) -> str: + normalized = " ".join(re.findall(r"\w+", claim.casefold())) + return hashlib.sha256(normalized.encode()).hexdigest() + + +class MemoryService: + def __init__(self, store: LocalContextStore): + self.store = store + + @staticmethod + def _prefix(principal: Principal) -> str: + return f"docling://users/{principal.tenant_id}/{principal.user_id}/memories" + + @classmethod + def _uri(cls, principal: Principal, uri: str) -> str: + address = parse_uri(uri) + authorize_uri(principal, address) + if ( + address.namespace != "users" + or len(address.segments) != 5 + or address.segments[2] != "memories" + or address.segments[3] not in MEMORY_KINDS + ): + raise ValueError("expected one memory URI") + return address.value + + def _record(self, principal: Principal, uri: str) -> MemoryRecord: + uri = self._uri(principal, uri) + stored = self.store.get_record(principal, uri) + metadata = validate_memory(self.store.get_document(principal, uri)) + if ( + metadata["kind"] != parse_uri(uri).segments[3] + or metadata["memory_id"] != parse_uri(uri).segments[4] + ): + raise ValueError("memory metadata differs from URI") + return MemoryRecord( + uri=uri, + memory_id=metadata["memory_id"], + revision_id=stored.revision_id, + kind=metadata["kind"], + claim=metadata["claim"], + explanation=metadata["explanation"], + status=metadata["status"], + confidence=float(metadata["confidence"]), + author=metadata["author"], + generator=metadata.get("generator"), + subject=metadata.get("subject"), + valid_from=metadata.get("valid_from"), + valid_to=metadata.get("valid_to"), + citations=tuple(SourceCitation(**item) for item in metadata["citations"]), + conflicts=tuple(metadata.get("conflicts", [])), + updated_at=stored.updated_at, + ) + + def _sync(self, principal: Principal) -> None: + prefix = self._prefix(principal) + "/" + documents = self.store.db.execute( + """SELECT uri,revision_id,updated_at FROM documents + WHERE tenant_id=? AND available=1 AND substr(uri,1,?)=?""", + (principal.tenant_id, len(prefix), prefix), + ).fetchall() + active = set() + for row in documents: + try: + uri = self._uri(principal, row["uri"]) + except (ValueError, PermissionError): + continue + active.add(uri) + indexed = self.store.db.execute( + "SELECT revision_id FROM memory_index WHERE tenant_id=? AND uri=?", + (principal.tenant_id, uri), + ).fetchone() + if indexed is not None and indexed["revision_id"] == row["revision_id"]: + continue + record = self._record(principal, uri) + with self.store.transaction(): + self.store.db.execute( + """INSERT INTO memory_index + (tenant_id,user_id,uri,memory_id,kind,subject,claim,claim_key, + status,confidence,revision_id,updated_at) + VALUES (?,?,?,?,?,?,?,?,?,?,?,?) + ON CONFLICT(tenant_id,uri) DO UPDATE SET + subject=excluded.subject,claim=excluded.claim, + claim_key=excluded.claim_key,status=excluded.status, + confidence=excluded.confidence,revision_id=excluded.revision_id, + updated_at=excluded.updated_at""", + ( + principal.tenant_id, + principal.user_id, + uri, + record.memory_id, + record.kind, + record.subject, + record.claim, + _claim_key(record.claim), + record.status, + record.confidence, + record.revision_id, + row["updated_at"], + ), + ) + self.store.db.execute( + "DELETE FROM memory_edges WHERE tenant_id=? AND memory_uri=?", + (principal.tenant_id, uri), + ) + self.store.db.executemany( + """INSERT INTO memory_edges + (tenant_id,user_id,memory_uri,source_uri,source_document_id, + source_revision_id,source_xpath) VALUES (?,?,?,?,?,?,?)""", + ( + ( + principal.tenant_id, + principal.user_id, + uri, + citation.uri, + citation.document_id, + citation.revision_id, + citation.xpath, + ) + for citation in record.citations + ), + ) + stale = self.store.db.execute( + "SELECT uri FROM memory_index WHERE tenant_id=? AND user_id=?", + (principal.tenant_id, principal.user_id), + ).fetchall() + with self.store.transaction(): + for row in stale: + if row["uri"] not in active: + self.store.db.execute( + "DELETE FROM memory_index WHERE tenant_id=? AND uri=?", + (principal.tenant_id, row["uri"]), + ) + self.store.db.execute( + "DELETE FROM memory_edges WHERE tenant_id=? AND memory_uri=?", + (principal.tenant_id, row["uri"]), + ) + + def get(self, principal: Principal, uri: str) -> MemoryRecord: + self._sync(principal) + return self._record(principal, uri) + + def _validate_citations( + self, principal: Principal, citations: tuple[SourceCitation, ...] + ) -> None: + if len(citations) > 100: + raise ValueError("too many memory citations") + for citation in citations: + current = self.store.get_record(principal, citation.uri) + stored = self.store.get_revision( + principal, + NodeAddress(citation.document_id, citation.revision_id, citation.xpath), + ) + if ( + stored.uri != citation.uri + or current.document_id != citation.document_id + ): + raise ValueError("citation URI differs from its revision") + self.store.read_node( + principal, + NodeAddress(citation.document_id, citation.revision_id, citation.xpath), + ) + + def _write( + self, + principal: Principal, + uri: str, + metadata: dict, + *, + expected_revision: str | None, + ) -> MemoryRecord: + document = DocLangXDocument() + xml = ( + f"{escape(metadata['kind'])}" + f"{escape(metadata['claim'])}" + f"{escape(metadata['explanation'])}" + ) + if not document.read_xml(xml): + raise ValueError(document.last_error()) + document.set_part_text( + MEMORY_PART, json.dumps(metadata, sort_keys=True), "application/json" + ) + validate_memory(document) + self.store.put( + principal, + uri, + document.write_bytes(), + expected_revision=expected_revision, + ) + self._sync(principal) + return self._record(principal, uri) + + def create( + self, + principal: Principal, + kind: str, + claim: str, + *, + explanation: str = "", + citations: tuple[SourceCitation, ...] = (), + confidence: float = 1.0, + subject: str | None = None, + author: str = "user", + generator: str | None = None, + accepted: bool = False, + valid_from: str | None = None, + valid_to: str | None = None, + ) -> MemoryRecord: + if kind not in MEMORY_KINDS or not isinstance(claim, str) or not claim.strip(): + raise ValueError("invalid memory kind or claim") + if accepted and author != "user": + raise ValueError("generated memories require explicit review") + self._validate_citations(principal, citations) + self._sync(principal) + key = _claim_key(claim) + existing = self.store.db.execute( + """SELECT uri FROM memory_index WHERE tenant_id=? AND user_id=? + AND kind=? AND subject IS ? AND claim_key=? + ORDER BY updated_at DESC LIMIT 1""", + (principal.tenant_id, principal.user_id, kind, subject, key), + ).fetchone() + if existing is not None: + return self._record(principal, existing["uri"]) + # Corrections retain their old claim keys to prevent replaying an older + # session revision from reviving a claim the user already corrected. + prior = self.store.db.execute( + """SELECT uri FROM memory_index WHERE tenant_id=? AND user_id=? + AND kind=? AND subject IS ? AND status!='deleted'""", + (principal.tenant_id, principal.user_id, kind, subject), + ).fetchall() + for row in prior: + metadata = validate_memory(self.store.get_document(principal, row["uri"])) + if key in metadata.get("claim_aliases", []): + return self._record(principal, row["uri"]) + conflicts = [] + if subject: + conflicts = [ + row["uri"] + for row in self.store.db.execute( + """SELECT uri FROM memory_index WHERE tenant_id=? AND user_id=? + AND kind=? AND subject=? AND status='accepted'""", + (principal.tenant_id, principal.user_id, kind, subject), + ) + ] + memory_id = uuid.uuid4().hex + uri = f"{self._prefix(principal)}/{kind}/{memory_id}" + status = "accepted" if accepted else "proposed" + metadata = { + "profile": "memory-v1", + "memory_id": memory_id, + "kind": kind, + "claim": claim.strip(), + "explanation": explanation, + "status": status, + "confidence": confidence, + "subject": subject, + "author": author, + "generator": generator, + "valid_from": valid_from, + "valid_to": valid_to, + "citations": [asdict(citation) for citation in citations], + "conflicts": conflicts, + "history": [{"status": status, "by": author, "at": _now()}], + } + return self._write(principal, uri, metadata, expected_revision=None) + + def transition( + self, principal: Principal, uri: str, status: str, *, reason: str = "" + ) -> MemoryRecord: + if status not in MEMORY_STATUSES: + raise ValueError("invalid memory status") + current = self.get(principal, uri) + if current.status == status: + return current + allowed = { + "proposed": {"accepted", "rejected", "deleted"}, + "accepted": {"superseded", "proposed", "deleted"}, + "rejected": {"deleted"}, + "superseded": {"deleted"}, + "deleted": set(), + } + if status not in allowed[current.status]: + raise ValueError("invalid memory transition") + metadata = validate_memory(self.store.get_document(principal, current.uri)) + metadata["status"] = status + metadata["history"].append( + {"status": status, "by": principal.user_id, "at": _now(), "reason": reason} + ) + return self._write( + principal, current.uri, metadata, expected_revision=current.revision_id + ) + + def accept(self, principal: Principal, uri: str) -> MemoryRecord: + return self.transition(principal, uri, "accepted") + + def reject( + self, principal: Principal, uri: str, *, reason: str = "" + ) -> MemoryRecord: + return self.transition(principal, uri, "rejected", reason=reason) + + def delete( + self, principal: Principal, uri: str, *, reason: str = "" + ) -> MemoryRecord: + return self.transition(principal, uri, "deleted", reason=reason) + + def purge(self, principal: Principal, uri: str) -> int: + """Remove a deleted claim, its revision history, and its recall records.""" + memory = self.get(principal, uri) + if memory.status != "deleted": + raise ValueError("delete the memory before purging it") + rows = self.store.db.execute( + "SELECT package_hash FROM revisions WHERE tenant_id=? AND uri=?", + (principal.tenant_id, memory.uri), + ).fetchall() + self.store.delete(principal, memory.uri, expected_revision=memory.revision_id) + with self.store.transaction(): + self.store.db.execute( + "DELETE FROM revisions WHERE tenant_id=? AND uri=?", + (principal.tenant_id, memory.uri), + ) + self.store.db.execute( + "DELETE FROM memory_index WHERE tenant_id=? AND uri=?", + (principal.tenant_id, memory.uri), + ) + self.store.db.execute( + "DELETE FROM memory_edges WHERE tenant_id=? AND memory_uri=?", + (principal.tenant_id, memory.uri), + ) + self.store.db.execute( + """DELETE FROM recall_ledger WHERE tenant_id=? AND user_id=? + AND memory_id=?""", + (principal.tenant_id, principal.user_id, memory.memory_id), + ) + for row in rows: + if not self.store.db.execute( + "SELECT 1 FROM revisions WHERE package_hash=? LIMIT 1", + (row["package_hash"],), + ).fetchone(): + self.store.packages.path_for(row["package_hash"]).unlink( + missing_ok=True + ) + return len(rows) + + def correct( + self, + principal: Principal, + uri: str, + claim: str, + *, + explanation: str | None = None, + citations: tuple[SourceCitation, ...] | None = None, + ) -> MemoryRecord: + current = self.get(principal, uri) + if current.status not in {"accepted", "proposed"} or not claim.strip(): + raise ValueError("only active memories can be corrected") + sources = citations if citations is not None else current.citations + self._validate_citations(principal, sources) + metadata = validate_memory(self.store.get_document(principal, current.uri)) + aliases = metadata.setdefault("claim_aliases", []) + aliases.append(_claim_key(current.claim)) + metadata.update( + claim=claim.strip(), + explanation=current.explanation if explanation is None else explanation, + status="accepted", + author="user", + generator=None, + citations=[asdict(citation) for citation in sources], + supersedes_revision=current.revision_id, + ) + metadata["history"].append( + { + "status": "accepted", + "by": principal.user_id, + "at": _now(), + "reason": "correction", + "supersedes_revision": current.revision_id, + } + ) + return self._write( + principal, current.uri, metadata, expected_revision=current.revision_id + ) + + def reconcile_sources(self, principal: Principal) -> int: + """Move accepted claims with no surviving cited source back to review.""" + self._sync(principal) + rows = self.store.db.execute( + """SELECT uri FROM memory_index WHERE tenant_id=? AND user_id=? + AND status='accepted'""", + (principal.tenant_id, principal.user_id), + ).fetchall() + changed = 0 + for row in rows: + memory = self._record(principal, row["uri"]) + if not memory.citations: + continue + supported = False + for citation in memory.citations: + try: + current = self.store.get_record(principal, citation.uri) + if current.document_id != citation.document_id: + continue + self.store.read_node( + principal, + NodeAddress( + citation.document_id, citation.revision_id, citation.xpath + ), + ) + except (RecordMissing, PackageError, PackageMissing, PermissionError): + continue + supported = True + break + if not supported: + self.transition( + principal, + memory.uri, + "proposed", + reason="cited sources unavailable", + ) + changed += 1 + return changed + + def search( + self, principal: Principal, query: str = "", *, limit: int = 10 + ) -> tuple[MemoryRecord, ...]: + if not 1 <= limit <= 100: + raise ValueError("memory search limit is out of bounds") + self.reconcile_sources(principal) + if not query.strip(): + rows = self.store.db.execute( + """SELECT uri FROM memory_index WHERE tenant_id=? AND user_id=? + AND status='accepted' ORDER BY updated_at DESC LIMIT ?""", + (principal.tenant_id, principal.user_id, limit), + ).fetchall() + return tuple(self._record(principal, row["uri"]) for row in rows) + result = Retriever(self.store).search( + principal, + query, + scope=SearchScope(uri=self._prefix(principal)), + mode="lexical", + k=100, + ) + seen = set() + records = [] + for hit in result.hits: + if hit.uri in seen: + continue + seen.add(hit.uri) + memory = self._record(principal, hit.uri) + if memory.status == "accepted": + records.append(memory) + if len(records) >= limit: + break + return tuple(records) + + def list( + self, + principal: Principal, + *, + status: str | None = None, + kind: str | None = None, + since: str | None = None, + until: str | None = None, + limit: int = 100, + offset: int = 0, + ) -> tuple[MemoryRecord, ...]: + """List current claims, including review history, for the current user.""" + if ( + status not in MEMORY_STATUSES | {None} + or kind not in MEMORY_KINDS | {None} + or not 1 <= limit <= 1000 + or offset < 0 + ): + raise ValueError("invalid memory list filter or page") + self.reconcile_sources(principal) + conditions = ["tenant_id=?", "user_id=?"] + params: list[str | int] = [principal.tenant_id, principal.user_id] + for column, value, operator in ( + ("status", status, "="), + ("kind", kind, "="), + ("updated_at", since, ">="), + ("updated_at", until, "<="), + ): + if value is not None: + conditions.append(f"{column}{operator}?") + params.append(value) + rows = self.store.db.execute( + f"SELECT uri FROM memory_index WHERE {' AND '.join(conditions)} " + "ORDER BY updated_at DESC,uri LIMIT ? OFFSET ?", + (*params, limit, offset), + ).fetchall() + return tuple(self._record(principal, row["uri"]) for row in rows) + + def recall( + self, + principal: Principal, + session_id: str, + query: str = "", + *, + limit: int = 10, + ) -> tuple[MemoryRecord, ...]: + from .sessions import SessionStore + + session_uri = SessionStore.uri(principal, session_id) + self.store.get_record(principal, session_uri) + candidates = self.search(principal, query, limit=100) + delivered = [] + for memory in candidates: + with self.store.transaction(): + cursor = self.store.db.execute( + """INSERT OR IGNORE INTO recall_ledger + (tenant_id,user_id,session_uri,memory_id,revision_id,delivered_at) + VALUES (?,?,?,?,?,?)""", + ( + principal.tenant_id, + principal.user_id, + session_uri, + memory.memory_id, + memory.revision_id, + _now(), + ), + ) + if cursor.rowcount: + delivered.append(memory) + if len(delivered) >= limit: + break + return tuple(delivered) diff --git a/src/docling_context/local.py b/src/docling_context/local.py index bec8b46..d056ce6 100644 --- a/src/docling_context/local.py +++ b/src/docling_context/local.py @@ -26,6 +26,7 @@ load_package, read_node, ) +from .profiles import validate_profile from .retrieval import index_package, remove_index_records from .uri import AccessDenied, authorize_uri, parse_uri @@ -71,7 +72,7 @@ def __exit__(self, *_: object) -> None: def _migrate(self) -> None: version = self.db.execute("PRAGMA user_version").fetchone()[0] - if version > 6: + if version > 7: raise RuntimeError("database schema is newer than this library") if version == 0: self.db.executescript( @@ -233,6 +234,53 @@ def _migrate(self) -> None: "ALTER TABLE retrieval_units ADD COLUMN section_xpath TEXT" ) self.db.execute("PRAGMA user_version = 6") + if version < 7: + self.db.executescript( + """BEGIN IMMEDIATE; + CREATE TABLE IF NOT EXISTS memory_index ( + tenant_id TEXT NOT NULL, user_id TEXT NOT NULL, uri TEXT NOT NULL, + memory_id TEXT NOT NULL, kind TEXT NOT NULL, subject TEXT, + claim TEXT NOT NULL, claim_key TEXT NOT NULL, + status TEXT NOT NULL, confidence REAL NOT NULL, + revision_id TEXT NOT NULL, updated_at TEXT NOT NULL, + PRIMARY KEY (tenant_id, uri) + ); + CREATE INDEX IF NOT EXISTS memory_owner_status ON memory_index + (tenant_id, user_id, status, updated_at); + CREATE INDEX IF NOT EXISTS memory_claim_key ON memory_index + (tenant_id, user_id, kind, claim_key); + CREATE TABLE IF NOT EXISTS memory_edges ( + tenant_id TEXT NOT NULL, user_id TEXT NOT NULL, + memory_uri TEXT NOT NULL, source_uri TEXT NOT NULL, + source_document_id TEXT NOT NULL, + source_revision_id TEXT NOT NULL, source_xpath TEXT NOT NULL, + PRIMARY KEY (tenant_id, memory_uri, source_uri, + source_revision_id, source_xpath) + ); + CREATE INDEX IF NOT EXISTS memory_source_edges ON memory_edges + (tenant_id, source_uri, source_revision_id); + CREATE TABLE IF NOT EXISTS recall_ledger ( + tenant_id TEXT NOT NULL, user_id TEXT NOT NULL, + session_uri TEXT NOT NULL, memory_id TEXT NOT NULL, + revision_id TEXT NOT NULL, delivered_at TEXT NOT NULL, + PRIMARY KEY (tenant_id, user_id, session_uri, memory_id, revision_id) + ); + CREATE TABLE IF NOT EXISTS memory_compile_jobs ( + job_id TEXT PRIMARY KEY, tenant_id TEXT NOT NULL, + user_id TEXT NOT NULL, session_uri TEXT NOT NULL, + session_document_id TEXT NOT NULL, + session_revision_id TEXT NOT NULL, + status TEXT NOT NULL, attempts INTEGER NOT NULL DEFAULT 0, + retry_at TEXT NOT NULL, lease_until TEXT, last_error TEXT, + created_at TEXT NOT NULL, updated_at TEXT NOT NULL, + UNIQUE (tenant_id, session_uri, session_revision_id) + ); + CREATE INDEX IF NOT EXISTS memory_compile_due ON memory_compile_jobs + (status, retry_at, lease_until); + PRAGMA user_version = 7; + COMMIT; + """ + ) self._backfill_retrieval() def _backfill_retrieval(self) -> None: @@ -411,6 +459,7 @@ def put( source_json = json.dumps(source or {}, sort_keys=True, ensure_ascii=False) digest, size = self.packages.put(package) indexed = load_package(package) + validate_profile(canonical, indexed) now = _now() with self.transaction(): current = self.db.execute( @@ -524,6 +573,11 @@ def read_node( def delete(self, principal: Principal, uri: str, *, expected_revision: str) -> None: address = parse_uri(uri) authorize_uri(principal, address) + dependents = self.db.execute( + """SELECT DISTINCT user_id FROM memory_edges + WHERE tenant_id=? AND source_uri=?""", + (principal.tenant_id, address.value), + ).fetchall() with self.transaction(): current = self.db.execute( "SELECT * FROM documents WHERE tenant_id=? AND uri=?", @@ -557,6 +611,14 @@ def delete(self, principal: Principal, uri: str, *, expected_revision: str) -> N cursor.lastrowid, _now(), ) + if dependents: + from .durable_memory import MemoryService + + memories = MemoryService(self) + for dependent in dependents: + memories.reconcile_sources( + Principal(principal.tenant_id, dependent["user_id"]) + ) def _tree( self, principal: Principal, uri: str, depth: int, limit: int diff --git a/src/docling_context/memory.py b/src/docling_context/memory.py index 08977f3..60aa973 100644 --- a/src/docling_context/memory.py +++ b/src/docling_context/memory.py @@ -16,6 +16,7 @@ TreeEntry, ) from .package import load_package, read_node +from .profiles import validate_profile from .uri import AccessDenied, authorize_uri, parse_uri @@ -38,7 +39,7 @@ def put( ) -> DocumentRecord: address = parse_uri(uri) authorize_uri(principal, address) - load_package(package) + validate_profile(address.value, load_package(package)) key = principal.tenant_id, address.value current = self.records.get(key) if (current is None and expected_revision is not None) or ( diff --git a/src/docling_context/profiles.py b/src/docling_context/profiles.py new file mode 100644 index 0000000..3fb6c25 --- /dev/null +++ b/src/docling_context/profiles.py @@ -0,0 +1,197 @@ +"""Portable DCLX profiles for agent sessions and reviewed memories.""" + +from __future__ import annotations + +import hashlib +import json +import math +from datetime import datetime +from typing import Any + +from doclang import DocLangXDocument + +from .package import bounded_nodes +from .uri import parse_uri + +SESSION_PART = "context/session.json" +MEMORY_PART = "context/memory.json" +SESSION_KINDS = frozenset({"turn", "tool_call", "tool_result", "feedback"}) +SESSION_STATUSES = frozenset({"open", "closed"}) +MEMORY_KINDS = frozenset({"preference", "fact", "entity", "concept", "procedure"}) +MEMORY_STATUSES = frozenset( + {"proposed", "accepted", "rejected", "superseded", "deleted"} +) +MAX_SESSION_EVENTS = 1_000 +MAX_EVENT_CHARS = 20_000 +MAX_ATTACHMENT_BYTES = 4 * 1024 * 1024 +MAX_SESSION_ATTACHMENT_BYTES = 32 * 1024 * 1024 + + +def profile_kind(uri: str) -> str | None: + address = parse_uri(uri) + if address.namespace != "users": + return None + if len(address.segments) == 4 and address.segments[2] == "sessions": + return "sessions" + if len(address.segments) == 5 and address.segments[2] == "memories": + return "memories" + return None + + +def _part(document: DocLangXDocument, path: str) -> dict[str, Any]: + raw = document.get_part_text(path) + if raw is None or len(raw) > 2_000_000: + raise ValueError(f"missing or oversized {path}") + try: + value = json.loads(raw) + except json.JSONDecodeError as exc: + raise ValueError(f"invalid {path}") from exc + if not isinstance(value, dict): + raise TypeError(f"invalid {path}") + return value + + +def validate_session(document: DocLangXDocument) -> dict[str, Any]: + metadata = _part(document, SESSION_PART) + if metadata.get("profile") != "session-v1": + raise ValueError("unsupported session profile") + events = metadata.get("events") + if not isinstance(events, list) or len(events) > MAX_SESSION_EVENTS: + raise ValueError("session needs a bounded event list") + if metadata.get("status", "open") not in SESSION_STATUSES: + raise ValueError("invalid session status") + for name in ("created_at", "updated_at", "closed_at"): + if metadata.get(name) is not None: + try: + datetime.fromisoformat(metadata[name]) + except (TypeError, ValueError) as exc: + raise ValueError(f"invalid session {name}") from exc + if metadata.get("status") == "closed" and not metadata.get("closed_at"): + raise ValueError("closed session needs closed_at") + text_nodes = [ + node + for node in bounded_nodes(document, limit=5_000, text_bytes=MAX_EVENT_CHARS * 4) + if node["name"] == "text" + ] + if len(text_nodes) != len(events): + raise ValueError("session event and DocLang counts differ") + keys: set[str] = set() + attachment_bytes = 0 + for number, (event, node) in enumerate(zip(events, text_nodes, strict=True), 1): + if not isinstance(event, dict): + raise TypeError("invalid session event") + key = event.get("key") + if ( + not isinstance(key, str) + or not 1 <= len(key) <= 128 + or key in keys + or event.get("kind") not in SESSION_KINDS + or event.get("xpath") != f"/doclang[1]/text[{number}]" + or not isinstance(event.get("turn_id"), str) + or not isinstance(event.get("created_at"), str) + or node["xpath"] != event["xpath"] + or len(node["text"]) > MAX_EVENT_CHARS + or node["truncated"] + ): + raise ValueError("invalid session event metadata") + keys.add(key) + attachment = event.get("attachment") + if attachment is not None: + if not isinstance(attachment, dict): + raise ValueError("invalid session attachment") + path = attachment.get("path") + if ( + not isinstance(path, str) + or not path.startswith("attachments/") + or not isinstance(attachment.get("content_type"), str) + ): + raise ValueError("invalid session attachment path or type") + payload = document.get_part_bytes(path) + if ( + payload is None + or len(payload) > MAX_ATTACHMENT_BYTES + or attachment.get("size") != len(payload) + or attachment.get("sha256") != hashlib.sha256(payload).hexdigest() + ): + raise ValueError("session attachment is missing or changed") + attachment_bytes += len(payload) + if attachment_bytes > MAX_SESSION_ATTACHMENT_BYTES: + raise ValueError("session attachments exceed the total limit") + return metadata + + +def validate_memory(document: DocLangXDocument) -> dict[str, Any]: + metadata = _part(document, MEMORY_PART) + if metadata.get("profile") != "memory-v1": + raise ValueError("unsupported memory profile") + confidence = metadata.get("confidence") + if ( + metadata.get("kind") not in MEMORY_KINDS + or metadata.get("status") not in MEMORY_STATUSES + or not isinstance(metadata.get("memory_id"), str) + or not isinstance(metadata.get("claim"), str) + or not metadata["claim"].strip() + or len(metadata["claim"]) > 4_000 + or not isinstance(metadata.get("explanation"), str) + or len(metadata["explanation"]) > 20_000 + or not isinstance(metadata.get("author"), str) + or not isinstance(confidence, (int, float)) + or not math.isfinite(confidence) + or not 0 <= confidence <= 1 + ): + raise ValueError("invalid memory metadata") + text_nodes = [ + node + for node in bounded_nodes(document, limit=20, text_bytes=20_000) + if node["name"] == "text" + ] + if not text_nodes or text_nodes[0]["text"] != metadata["claim"]: + raise ValueError("memory claim differs from DocLang body") + citations = metadata.get("citations") + if not isinstance(citations, list) or len(citations) > 100: + raise ValueError("invalid memory citations") + aliases = metadata.get("claim_aliases", []) + if ( + not isinstance(aliases, list) + or len(aliases) > 100 + or any(not isinstance(alias, str) or len(alias) != 64 for alias in aliases) + ): + raise ValueError("invalid memory claim aliases") + for citation in citations: + if not isinstance(citation, dict) or any( + not isinstance(citation.get(name), str) or not citation[name] + for name in ("uri", "document_id", "revision_id", "xpath") + ): + raise ValueError("invalid memory citation") + if not citation["xpath"].startswith("/doclang[1]"): + raise ValueError("memory citations need DocLang XPaths") + parse_uri(citation["uri"]) + for name in ("valid_from", "valid_to"): + if metadata.get(name) is not None: + try: + datetime.fromisoformat(metadata[name]) + except (TypeError, ValueError) as exc: + raise ValueError(f"invalid {name}") from exc + if ( + metadata.get("valid_from") + and metadata.get("valid_to") + and metadata["valid_from"] > metadata["valid_to"] + ): + raise ValueError("memory validity is reversed") + return metadata + + +def validate_profile(uri: str, document: DocLangXDocument) -> dict[str, Any] | None: + kind = profile_kind(uri) + address = parse_uri(uri) + if kind == "sessions": + return validate_session(document) + if kind == "memories": + metadata = validate_memory(document) + if ( + metadata["kind"] != address.segments[3] + or metadata["memory_id"] != address.segments[4] + ): + raise ValueError("memory package metadata differs from URI") + return metadata + return None diff --git a/src/docling_context/retrieval.py b/src/docling_context/retrieval.py index b886b21..b35561d 100644 --- a/src/docling_context/retrieval.py +++ b/src/docling_context/retrieval.py @@ -14,6 +14,7 @@ from .models import DocumentRecord, Principal from .package import MAX_NODE_TEXT_CHARS, bounded_nodes, load_package +from .profiles import MEMORY_PART, profile_kind from .uri import authorize_uri, parse_uri if TYPE_CHECKING: @@ -58,6 +59,10 @@ def index_package( """Replace one document's projection inside the caller's transaction.""" document = document or load_package(package) remove_index_records(db, record.tenant_id, record.uri) + if profile_kind(record.uri) == "memories": + metadata = json.loads(document.get_part_text(MEMORY_PART) or "{}") + if metadata.get("status") != "accepted": + return units: list[ tuple[str, str | None, str | None, int, int | None, str | None, str, str | None] ] = [] diff --git a/src/docling_context/sessions.py b/src/docling_context/sessions.py new file mode 100644 index 0000000..b9cf35c --- /dev/null +++ b/src/docling_context/sessions.py @@ -0,0 +1,466 @@ +"""Revisioned DocLang session events and bounded archive attachments.""" + +from __future__ import annotations + +import hashlib +import json +import uuid +from dataclasses import dataclass +from datetime import UTC, datetime +from typing import Any +from xml.sax.saxutils import escape + +from doclang import DocLangXDocument + +from .local import LocalContextStore, RecordMissing +from .models import Principal +from .package import bounded_nodes +from .profiles import ( + MAX_ATTACHMENT_BYTES, + MAX_EVENT_CHARS, + MAX_SESSION_ATTACHMENT_BYTES, + MAX_SESSION_EVENTS, + SESSION_KINDS, + SESSION_PART, + validate_session, +) +from .uri import authorize_uri, parse_uri + + +@dataclass(frozen=True, slots=True) +class SessionEvent: + sequence: int + key: str + kind: str + turn_id: str + text: str + created_at: str + xpath: str + revision_id: str + attachment_path: str | None = None + + +@dataclass(frozen=True, slots=True) +class SessionInfo: + uri: str + session_id: str + revision_id: str + status: str + event_count: int + created_at: str + updated_at: str + closed_at: str | None + + +class SessionStore: + def __init__(self, store: LocalContextStore): + self.store = store + + @staticmethod + def uri(principal: Principal, session_id: str) -> str: + value = ( + session_id + if session_id.startswith("docling://") + else f"docling://users/{principal.tenant_id}/{principal.user_id}/sessions/{session_id}" + ) + address = parse_uri(value) + authorize_uri(principal, address) + if ( + address.namespace != "users" + or len(address.segments) != 4 + or address.segments[2] != "sessions" + ): + raise ValueError("expected one session URI") + return address.value + + @staticmethod + def _events( + document: DocLangXDocument, revision_id: str + ) -> tuple[SessionEvent, ...]: + metadata = validate_session(document) + texts = [ + node["text"] + for node in bounded_nodes( + document, limit=5_000, text_bytes=MAX_EVENT_CHARS * 4 + ) + if node["name"] == "text" + ] + return tuple( + SessionEvent( + sequence=number, + key=event["key"], + kind=event["kind"], + turn_id=event["turn_id"], + text=texts[number - 1], + created_at=event["created_at"], + xpath=event["xpath"], + revision_id=revision_id, + attachment_path=(event.get("attachment") or {}).get("path"), + ) + for number, event in enumerate(metadata["events"], 1) + ) + + @staticmethod + def _package( + events: tuple[SessionEvent, ...], + attachments: dict[str, tuple[bytes, str]], + *, + created_at: str, + status: str = "open", + closed_at: str | None = None, + ) -> bytes: + document = DocLangXDocument() + body = "".join(f"{escape(event.text)}" for event in events) + if not document.read_xml(f"{body}"): + raise ValueError(document.last_error()) + entries = [] + for number, event in enumerate(events, 1): + entry: dict[str, Any] = { + "key": event.key, + "kind": event.kind, + "turn_id": event.turn_id, + "created_at": event.created_at, + "xpath": f"/doclang[1]/text[{number}]", + } + if event.attachment_path: + payload, content_type = attachments[event.attachment_path] + entry["attachment"] = { + "path": event.attachment_path, + "content_type": content_type, + "size": len(payload), + "sha256": hashlib.sha256(payload).hexdigest(), + } + entries.append(entry) + for path, (payload, content_type) in attachments.items(): + document.set_part_bytes(path, payload, content_type) + document.set_part_text( + SESSION_PART, + json.dumps( + { + "profile": "session-v1", + "events": entries, + "status": status, + "created_at": created_at, + "updated_at": datetime.now(UTC).isoformat(), + "closed_at": closed_at, + }, + sort_keys=True, + ), + "application/json", + ) + package = document.write_bytes() + validate_session(document) + return package + + def start(self, principal: Principal, session_id: str | None = None) -> SessionInfo: + session_id = session_id or uuid.uuid4().hex + uri = self.uri(principal, session_id) + try: + current = self.get(principal, uri) + if current.status == "closed": + raise ValueError("session ID already closed") + return current + except RecordMissing: + pass + now = datetime.now(UTC).isoformat() + record = self.store.put(principal, uri, self._package((), {}, created_at=now)) + return SessionInfo( + uri, + parse_uri(uri).segments[3], + record.revision_id, + "open", + 0, + now, + now, + None, + ) + + def get(self, principal: Principal, session_id: str) -> SessionInfo: + uri = self.uri(principal, session_id) + record = self.store.get_record(principal, uri) + metadata = validate_session(self.store.get_document(principal, uri)) + return SessionInfo( + uri, + parse_uri(uri).segments[3], + record.revision_id, + metadata.get("status", "open"), + len(metadata["events"]), + metadata.get("created_at") or record.created_at, + metadata.get("updated_at") or record.updated_at, + metadata.get("closed_at"), + ) + + def list( + self, + principal: Principal, + *, + status: str | None = None, + since: str | None = None, + until: str | None = None, + limit: int = 100, + offset: int = 0, + ) -> tuple[SessionInfo, ...]: + if ( + status not in {None, "open", "closed"} + or not 1 <= limit <= 1000 + or offset < 0 + ): + raise ValueError("invalid session list filter or page") + prefix = f"docling://users/{principal.tenant_id}/{principal.user_id}/sessions/" + rows = self.store.db.execute( + """SELECT uri FROM documents WHERE tenant_id=? AND available=1 + AND substr(uri,1,?)=? ORDER BY updated_at DESC,uri""", + (principal.tenant_id, len(prefix), prefix), + ).fetchall() + items = [] + for row in rows: + item = self.get(principal, row["uri"]) + if status and item.status != status: + continue + if since and item.updated_at < since: + continue + if until and item.updated_at > until: + continue + items.append(item) + return tuple(items[offset : offset + limit]) + + def close(self, principal: Principal, session_id: str) -> SessionInfo: + current = self.get(principal, session_id) + if current.status == "closed": + return current + document = self.store.get_document(principal, current.uri) + events = self._events(document, current.revision_id) + attachments = self._attachments(document) + record = self.store.put( + principal, + current.uri, + self._package( + events, + attachments, + created_at=current.created_at, + status="closed", + closed_at=datetime.now(UTC).isoformat(), + ), + expected_revision=current.revision_id, + ) + self._queue_compile( + principal, record.uri, record.document_id, record.revision_id + ) + return self.get(principal, current.uri) + + @staticmethod + def _attachments(document: DocLangXDocument) -> dict[str, tuple[bytes, str]]: + attachments = {} + for item in validate_session(document)["events"]: + info = item.get("attachment") + if info: + payload = document.get_part_bytes(info["path"]) + assert payload is not None + attachments[info["path"]] = (payload, info["content_type"]) + return attachments + + def _queue_compile( + self, principal: Principal, uri: str, document_id: str, revision_id: str + ) -> None: + now = datetime.now(UTC).isoformat() + self.store.db.execute( + """UPDATE memory_compile_jobs SET status='superseded', + lease_until=NULL,updated_at=? WHERE tenant_id=? AND user_id=? + AND session_uri=? AND session_revision_id!=? AND status='queued'""", + (now, principal.tenant_id, principal.user_id, uri, revision_id), + ) + self.store.db.execute( + """INSERT OR IGNORE INTO memory_compile_jobs + (job_id,tenant_id,user_id,session_uri,session_document_id, + session_revision_id,status,retry_at,created_at,updated_at) + VALUES (?,?,?,?,?,?,'queued',?,?,?)""", + ( + uuid.uuid4().hex, + principal.tenant_id, + principal.user_id, + uri, + document_id, + revision_id, + now, + now, + now, + ), + ) + + def append( + self, + principal: Principal, + session_id: str, + *, + key: str, + kind: str, + text: str, + turn_id: str = "", + attachment: bytes | None = None, + content_type: str = "application/octet-stream", + ) -> SessionEvent: + uri = self.uri(principal, session_id) + if ( + kind not in SESSION_KINDS + or not isinstance(key, str) + or not 1 <= len(key) <= 128 + or not isinstance(text, str) + or len(text) > MAX_EVENT_CHARS + or not isinstance(turn_id, str) + or len(turn_id) > 128 + ): + raise ValueError("invalid session event") + if attachment is not None and ( + not isinstance(attachment, bytes) + or len(attachment) > MAX_ATTACHMENT_BYTES + or not isinstance(content_type, str) + or not content_type + ): + raise ValueError("invalid or oversized session attachment") + try: + current = self.store.get_record(principal, uri) + document = self.store.get_document(principal, uri) + previous = self._events(document, current.revision_id) + except RecordMissing: + current = None + document = None + previous = () + duplicate = next((event for event in previous if event.key == key), None) + if duplicate is not None: + return duplicate + metadata = validate_session(document) if document is not None else {} + if metadata.get("status", "open") != "open": + raise ValueError("session is closed") + if len(previous) >= MAX_SESSION_EVENTS: + raise ValueError("session event limit reached; trim the session") + path = f"attachments/{uuid.uuid4().hex}.bin" if attachment is not None else None + event = SessionEvent( + len(previous) + 1, + key, + kind, + turn_id, + text, + datetime.now(UTC).isoformat(), + f"/doclang[1]/text[{len(previous) + 1}]", + "", + path, + ) + attachments = self._attachments(document) if document is not None else {} + if path and attachment is not None: + attachments[path] = (attachment, content_type) + if ( + sum(len(payload) for payload, _ in attachments.values()) + > MAX_SESSION_ATTACHMENT_BYTES + ): + raise ValueError("session attachments exceed the total limit") + record = self.store.put( + principal, + uri, + self._package( + (*previous, event), + attachments, + created_at=metadata.get("created_at") or event.created_at, + ), + expected_revision=current.revision_id if current else None, + ) + self._queue_compile(principal, uri, record.document_id, record.revision_id) + return SessionEvent( + event.sequence, + event.key, + event.kind, + event.turn_id, + event.text, + event.created_at, + event.xpath, + record.revision_id, + event.attachment_path, + ) + + def replay(self, principal: Principal, session_id: str) -> tuple[SessionEvent, ...]: + uri = self.uri(principal, session_id) + record = self.store.get_record(principal, uri) + return self._events(self.store.get_document(principal, uri), record.revision_id) + + def read_attachment(self, principal: Principal, session_id: str, key: str) -> bytes: + uri = self.uri(principal, session_id) + document = self.store.get_document(principal, uri) + event = next( + (item for item in self._events(document, "") if item.key == key), None + ) + if event is None or event.attachment_path is None: + raise KeyError(key) + payload = document.get_part_bytes(event.attachment_path) + if payload is None: + raise KeyError(key) + return payload + + def trim(self, principal: Principal, session_id: str, *, keep_last: int) -> str: + if not 1 <= keep_last <= MAX_SESSION_EVENTS: + raise ValueError("keep_last is out of bounds") + uri = self.uri(principal, session_id) + record = self.store.get_record(principal, uri) + document = self.store.get_document(principal, uri) + retained = self._events(document, record.revision_id)[-keep_last:] + metadata = validate_session(document) + attachments = {} + keep_paths = { + event.attachment_path for event in retained if event.attachment_path + } + for item in metadata["events"]: + info = item.get("attachment") + if info and info["path"] in keep_paths: + payload = document.get_part_bytes(info["path"]) + assert payload is not None + attachments[info["path"]] = (payload, info["content_type"]) + new_record = self.store.put( + principal, + uri, + self._package( + retained, + attachments, + created_at=metadata.get("created_at") or record.created_at, + status=metadata.get("status", "open"), + closed_at=metadata.get("closed_at"), + ), + expected_revision=record.revision_id, + ) + return new_record.revision_id + + def delete(self, principal: Principal, session_id: str) -> None: + uri = self.uri(principal, session_id) + record = self.store.get_record(principal, uri) + self.store.delete(principal, uri, expected_revision=record.revision_id) + + def purge(self, principal: Principal, session_id: str) -> int: + """Remove every session revision and raw attachment after a logical delete.""" + uri = self.uri(principal, session_id) + try: + self.delete(principal, uri) + except RecordMissing: + pass + rows = self.store.db.execute( + "SELECT package_hash FROM revisions WHERE tenant_id=? AND uri=?", + (principal.tenant_id, uri), + ).fetchall() + with self.store.transaction(): + self.store.db.execute( + "DELETE FROM revisions WHERE tenant_id=? AND uri=?", + (principal.tenant_id, uri), + ) + self.store.db.execute( + "DELETE FROM memory_compile_jobs WHERE tenant_id=? AND session_uri=?", + (principal.tenant_id, uri), + ) + self.store.db.execute( + "DELETE FROM recall_ledger WHERE tenant_id=? AND session_uri=?", + (principal.tenant_id, uri), + ) + for row in rows: + if not self.store.db.execute( + "SELECT 1 FROM revisions WHERE package_hash=? LIMIT 1", + (row["package_hash"],), + ).fetchone(): + self.store.packages.path_for(row["package_hash"]).unlink( + missing_ok=True + ) + return len(rows) diff --git a/tests/test_ingestion_jobs.py b/tests/test_ingestion_jobs.py index 6764859..0471179 100644 --- a/tests/test_ingestion_jobs.py +++ b/tests/test_ingestion_jobs.py @@ -292,5 +292,5 @@ def test_existing_database_upgrades_without_losing_documents(tmp_path): ) with LocalContextStore(tmp_path) as store: assert store.get_record(PRINCIPAL, URI).revision_id == original.revision_id - assert store.db.execute("PRAGMA user_version").fetchone()[0] == 6 + assert store.db.execute("PRAGMA user_version").fetchone()[0] == 7 assert len(CollectionWorker(store).jobs(PRINCIPAL, COLLECTION)) == 1 diff --git a/tests/test_retrieval.py b/tests/test_retrieval.py index 033d956..64cfef2 100644 --- a/tests/test_retrieval.py +++ b/tests/test_retrieval.py @@ -496,7 +496,7 @@ def test_existing_retrieval_index_adds_picture_column_on_upgrade(tmp_path): "ALTER TABLE retrieval_units DROP COLUMN asset_path; PRAGMA user_version=4;" ) with LocalContextStore(tmp_path) as store: - assert store.db.execute("PRAGMA user_version").fetchone()[0] == 6 + assert store.db.execute("PRAGMA user_version").fetchone()[0] == 7 assert "asset_path" in { row["name"] for row in store.db.execute("PRAGMA table_info(retrieval_units)") diff --git a/tests/test_session_memory_cli.py b/tests/test_session_memory_cli.py new file mode 100644 index 0000000..659dbe3 --- /dev/null +++ b/tests/test_session_memory_cli.py @@ -0,0 +1,203 @@ +"""Human and harness CLI flows for session and memory commands.""" + +from __future__ import annotations + +import io +import json +import sys + +from docling_context.cli import main + + +def _run(capsys, root, *args): + result = main(["--store", str(root), "--tenant", "acme", "--user", "alice", *args]) + output = capsys.readouterr() + return result, output.out, output.err + + +def test_session_lifecycle_listing_and_memory_review(tmp_path, capsys, monkeypatch): + assert _run(capsys, tmp_path, "session", "start", "--id", "chat", "--json")[0] == 0 + monkeypatch.setenv("DOCLING_CONTEXT_SESSION", "chat") + monkeypatch.setattr(sys, "stdin", io.StringIO("Remember: I prefer concise replies")) + attachment = tmp_path / "source.bin" + attachment.write_bytes(b"event bytes") + code, output, _ = _run( + capsys, + tmp_path, + "session", + "append", + "--kind", + "feedback", + "--key", + "feedback-1", + "--stdin", + "--attachment", + str(attachment), + "--json", + ) + assert code == 0 + assert json.loads(output)["xpath"] == "/doclang[1]/text[1]" + code, output, _ = _run( + capsys, tmp_path, "session", "list", "--status", "open", "--json" + ) + assert code == 0 + sessions = json.loads(output) + assert [item["session_id"] for item in sessions] == ["chat"] + day = sessions[0]["updated_at"][:10] + code, output, _ = _run( + capsys, tmp_path, "session", "list", "--until", day, "--json" + ) + assert code == 0 and len(json.loads(output)) == 1 + code, output, _ = _run( + capsys, tmp_path, "session", "list", "--offset", "1", "--json" + ) + assert code == 0 and json.loads(output) == [] + code, output, _ = _run(capsys, tmp_path, "session", "replay", "chat", "--json") + assert code == 0 + assert json.loads(output)[0]["text"] == "Remember: I prefer concise replies" + restored = tmp_path / "restored.bin" + assert ( + _run( + capsys, + tmp_path, + "session", + "attachment", + "chat", + "feedback-1", + "--output", + str(restored), + "--json", + )[0] + == 0 + ) + assert restored.read_bytes() == b"event bytes" + code = main( + [ + "--store", + str(tmp_path), + "--tenant", + "acme", + "--user", + "bob", + "session", + "show", + "docling://users/acme/alice/sessions/chat", + "--json", + ] + ) + assert code == 1 and "user does not match" in capsys.readouterr().err + assert _run(capsys, tmp_path, "session", "close", "chat", "--json")[0] == 0 + code, _, error = _run( + capsys, + tmp_path, + "session", + "append", + "chat", + "too late", + "--kind", + "turn", + "--key", + "late", + ) + assert code == 1 and "closed" in error + code, output, _ = _run( + capsys, tmp_path, "session", "list", "--status", "closed", "--json" + ) + assert code == 0 and len(json.loads(output)) == 1 + code, output, _ = _run(capsys, tmp_path, "memory", "worker", "--once", "--json") + assert code == 0 and json.loads(output)["status"] == "completed" + code, output, _ = _run(capsys, tmp_path, "session", "jobs", "chat", "--json") + assert code == 0 and {job["status"] for job in json.loads(output)} == { + "completed", + "superseded", + } + code, output, _ = _run(capsys, tmp_path, "memory", "proposals", "--json") + proposal = json.loads(output)[0] + assert code == 0 and proposal["status"] == "proposed" + uri = proposal["uri"] + code, output, _ = _run(capsys, tmp_path, "grep", "concise", "--uri", uri) + assert code == 0 and not output + code, output, _ = _run( + capsys, tmp_path, "memory", "list", "--kind", "preference", "--json" + ) + assert code == 0 and len(json.loads(output)) == 1 + code, output, _ = _run(capsys, tmp_path, "memory", "list", "--until", day, "--json") + assert code == 0 and len(json.loads(output)) == 1 + assert _run(capsys, tmp_path, "memory", "accept", uri, "--json")[0] == 0 + code, output, _ = _run(capsys, tmp_path, "grep", "concise", "--uri", uri) + assert code == 0 and output + code, output, _ = _run(capsys, tmp_path, "memory", "search", "concise", "--json") + assert code == 0 and json.loads(output)[0]["uri"] == uri + code, output, _ = _run(capsys, tmp_path, "memory", "recall", "concise", "--json") + assert code == 0 and len(json.loads(output)) == 1 + code, output, _ = _run(capsys, tmp_path, "memory", "recall", "concise", "--json") + assert code == 0 and json.loads(output) == [] + code, output, _ = _run(capsys, tmp_path, "status", "--json") + counts = json.loads(output) + assert code == 0 and counts["sessions"] == 1 and counts["memories"] == 1 + assert counts["memory_jobs"] >= 1 + assert _run(capsys, tmp_path, "memory", "delete", uri, "--json")[0] == 0 + code, output, _ = _run( + capsys, tmp_path, "memory", "list", "--status", "deleted", "--json" + ) + assert code == 0 and len(json.loads(output)) == 1 + code, output, _ = _run(capsys, tmp_path, "grep", "concise", "--uri", uri) + assert code == 0 and not output + assert _run(capsys, tmp_path, "memory", "purge", uri, "--json")[0] == 0 + code, output, _ = _run(capsys, tmp_path, "memory", "list", "--json") + assert code == 0 and json.loads(output) == [] + + +def test_manual_memory_filters_and_user_scope(tmp_path, capsys): + code, output, _ = _run( + capsys, + tmp_path, + "memory", + "create", + "concept", + "Karoo Basin", + "--accepted", + "--json", + ) + uri = json.loads(output)["uri"] + assert code == 0 + code, output, _ = _run( + capsys, tmp_path, "memory", "list", "--kind", "fact", "--json" + ) + assert code == 0 and json.loads(output) == [] + code, output, _ = _run( + capsys, tmp_path, "memory", "list", "--status", "accepted", "--json" + ) + assert code == 0 and len(json.loads(output)) == 1 + code, output, _ = _run( + capsys, tmp_path, "memory", "correct", uri, "Karoo Supergroup", "--json" + ) + assert code == 0 and json.loads(output)["claim"] == "Karoo Supergroup" + code = main( + [ + "--store", + str(tmp_path), + "--tenant", + "acme", + "--user", + "bob", + "memory", + "list", + "--json", + ] + ) + assert code == 0 and json.loads(capsys.readouterr().out) == [] + code = main( + [ + "--store", + str(tmp_path), + "--tenant", + "acme", + "--user", + "bob", + "session", + "list", + "--json", + ] + ) + assert code == 0 and json.loads(capsys.readouterr().out) == [] diff --git a/tests/test_sessions_memory.py b/tests/test_sessions_memory.py new file mode 100644 index 0000000..ede53a0 --- /dev/null +++ b/tests/test_sessions_memory.py @@ -0,0 +1,264 @@ +"""Revisioned sessions and reviewed memories with exact source provenance.""" + +from __future__ import annotations + +import pytest +from doclang import DocLangXDocument + +from docling_context import ( + LocalContextStore, + MemoryCompiler, + MemoryService, + NodeAddress, + Principal, + Retriever, + SearchScope, + SessionStore, + SourceCitation, +) + + +def test_session_replay_idempotency_attachment_and_private_scope(tmp_path): + alice = Principal("acme", "alice") + bob = Principal("acme", "bob") + other = Principal("elsewhere", "alice") + with LocalContextStore(tmp_path) as store: + sessions = SessionStore(store) + first = sessions.append( + alice, + "chat", + key="one", + kind="turn", + text="Hello & goodbye", + turn_id="t1", + attachment=b"image", + content_type="image/png", + ) + assert ( + sessions.append(alice, "chat", key="one", kind="turn", text="ignored") + == first + ) + sessions.append(alice, "chat", key="two", kind="tool_result", text="Done") + assert sessions.read_attachment(alice, "chat", "one") == b"image" + for principal in (bob, other): + with pytest.raises(PermissionError): + sessions.replay(principal, first_uri := sessions.uri(alice, "chat")) + with pytest.raises(PermissionError): + sessions.read_attachment(principal, first_uri, "one") + sessions.trim(alice, "chat", keep_last=1) + assert [event.key for event in sessions.replay(alice, "chat")] == ["two"] + assert sessions.purge(alice, "chat") >= 2 + with LocalContextStore(tmp_path) as store, pytest.raises(KeyError): + SessionStore(store).replay(alice, "chat") + + +def test_session_replay_survives_store_restart(tmp_path): + principal = Principal("acme", "alice") + with LocalContextStore(tmp_path) as store: + sessions = SessionStore(store) + sessions.append(principal, "chat", key="one", kind="turn", text="First") + second = sessions.append( + principal, "chat", key="two", kind="feedback", text="Second" + ) + with LocalContextStore(tmp_path) as store: + events = SessionStore(store).replay(principal, "chat") + assert [(event.sequence, event.key, event.text) for event in events] == [ + (1, "one", "First"), + (2, "two", "Second"), + ] + assert events[1].revision_id == second.revision_id + record = store.get_record(principal, SessionStore.uri(principal, "chat")) + assert ( + store.read_node( + principal, + NodeAddress(record.document_id, second.revision_id, second.xpath), + ).text + == "Second" + ) + + +def test_compilation_review_correction_recall_and_source_cascade(tmp_path): + alice = Principal("acme", "alice") + bob = Principal("acme", "bob") + other = Principal("elsewhere", "alice") + with LocalContextStore(tmp_path) as store: + sessions = SessionStore(store) + memories = MemoryService(store) + event = sessions.append( + alice, + "chat", + key="one", + kind="feedback", + text="Remember: The report covers Koonap geology", + ) + sessions.append( + alice, + "chat", + key="two", + kind="tool_result", + text="Remember: malicious tool payload", + ) + compiler = MemoryCompiler(store) + while (job := compiler.run_once()) is not None: + assert job.status == "completed", job.last_error + proposed = store.db.execute( + "SELECT uri FROM memory_index WHERE user_id='alice'" + ).fetchall() + assert len(proposed) == 1 + uri = proposed[0]["uri"] + assert memories.list(alice)[0].uri == uri + assert compiler.jobs(alice, "chat") + memory = memories.get(alice, uri) + assert memory.status == "proposed" + assert memory.citations[0].xpath == event.xpath + assert memories.search(alice, "Koonap") == () + accepted = memories.accept(alice, uri) + assert accepted.status == "accepted" + assert memories.search(alice, "Koonap")[0].uri == uri + assert ( + memories.recall(alice, "chat", "Koonap")[0].revision_id + == accepted.revision_id + ) + assert memories.recall(alice, "chat", "Koonap") == () + corrected = memories.correct( + alice, uri, "The report covers Koonap stratigraphy" + ) + assert corrected.revision_id != accepted.revision_id + assert ( + memories.recall(alice, "chat", "Koonap")[0].revision_id + == corrected.revision_id + ) + sessions.append( + alice, + "chat", + key="three", + kind="feedback", + text="Remember: The report covers Koonap geology", + ) + assert compiler.run_once().status == "completed" + assert store.db.execute("SELECT count(*) FROM memory_index").fetchone()[0] == 1 + for principal in (bob, other): + with pytest.raises(PermissionError): + memories.get(principal, uri) + with pytest.raises(PermissionError): + memories.accept(principal, uri) + with pytest.raises(PermissionError): + memories.correct(principal, uri, "wrong") + assert memories.search(principal, "Koonap") == () + assert memories.list(principal, status="accepted") == () + with pytest.raises(PermissionError): + compiler.jobs(principal, sessions.uri(alice, "chat")) + sessions.delete(alice, "chat") + assert memories.get(alice, uri).status == "proposed" + assert memories.search(alice, "Koonap") == () + with pytest.raises(KeyError): + memories.correct(alice, uri, "Unverified correction") + assert ( + not Retriever(store) + .search(alice, "Koonap", scope=SearchScope(uri=uri)) + .hits + ) + + +def test_rejected_proposal_is_not_recreated(tmp_path): + principal = Principal("acme", "alice") + with LocalContextStore(tmp_path) as store: + SessionStore(store).append( + principal, "chat", key="one", kind="turn", text="I prefer plain text" + ) + compiler = MemoryCompiler(store) + assert compiler.run_once().status == "completed" + memories = MemoryService(store) + row = store.db.execute("SELECT uri FROM memory_index").fetchone() + memory = memories.reject(principal, row["uri"]) + assert memory.status == "rejected" + event = SessionStore(store).append( + principal, + "chat", + key="two", + kind="feedback", + text="Remember: I prefer plain text", + ) + assert event.sequence == 2 + assert compiler.run_once().status == "completed" + assert store.db.execute("SELECT count(*) FROM memory_index").fetchone()[0] == 1 + assert memories.get(principal, memory.uri).status == "rejected" + + +def test_manual_memory_source_validation_and_redaction(tmp_path): + principal = Principal("acme", "alice") + with LocalContextStore(tmp_path) as store: + event = SessionStore(store).append( + principal, + "chat", + key="one", + kind="feedback", + text="Remember: key=sk-abcdefghijklmnop reference only", + ) + assert MemoryCompiler(store).run_once().status == "completed" + row = store.db.execute("SELECT uri FROM memory_index").fetchone() + memory = MemoryService(store).get(principal, row["uri"]) + assert "sk-abcdefghijklmnop" not in memory.claim + session = store.get_record(principal, SessionStore.uri(principal, "chat")) + citation = SourceCitation( + session.uri, session.document_id, session.revision_id, event.xpath + ) + manual = MemoryService(store).create( + principal, "concept", "Karoo Basin", citations=(citation,), accepted=True + ) + assert manual.status == "accepted" + with pytest.raises(KeyError): + MemoryService(store).create( + principal, + "concept", + "Invalid", + citations=( + SourceCitation( + session.uri, session.document_id, "bad", event.xpath + ), + ), + ) + assert MemoryService(store).delete(principal, manual.uri).status == "deleted" + assert MemoryService(store).purge(principal, manual.uri) >= 2 + with pytest.raises(KeyError): + MemoryService(store).get(principal, manual.uri) + + +def test_profile_rejects_unstructured_session_package(tmp_path): + document = DocLangXDocument() + assert document.read_xml("Unstructured") + with LocalContextStore(tmp_path) as store, pytest.raises(ValueError): + store.put( + Principal("acme", "alice"), + "docling://users/acme/alice/sessions/chat", + document.write_bytes(), + ) + + +def test_document_deletion_withdraws_dependent_memory(tmp_path): + principal = Principal("acme", "alice") + document = DocLangXDocument() + assert document.read_xml("Karoo strata") + with LocalContextStore(tmp_path) as store: + source = store.put( + principal, "docling://resources/geology/karoo", document.write_bytes() + ) + memories = MemoryService(store) + memory = memories.create( + principal, + "fact", + "Karoo strata are documented", + accepted=True, + citations=( + SourceCitation( + source.uri, + source.document_id, + source.revision_id, + "/doclang[1]/text[1]", + ), + ), + ) + assert memories.search(principal, "Karoo") + store.delete(principal, source.uri, expected_revision=source.revision_id) + assert memories.get(principal, memory.uri).status == "proposed" + assert memories.search(principal, "Karoo") == ()