Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions .plans/implemented/implementation_step_4.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Step 4: Sessions and durable memory

## Goal and dependency

Capture agent activity as DocLang and turn explicitly selected, supported observations into reviewable long-term memory. Depends on the storage, jobs, and retrieval contracts from Steps 1–3.

## Data model

- A session is a revisioned DocLang document in a DCLX package under `docling://users/{tenant}/{user}/sessions/{session_id}`. Represent turns, tool calls, tool results, and feedback with a documented DocLang vocabulary/profile. Keep attachment references and raw large tool payloads as typed archive parts with size and retention limits.
- A memory is its own DCLX package under `.../memories/{kind}/{id}`; supported kinds begin with preference, fact, entity, concept, and procedure. The DocLang body carries the readable claim and explanation. Typed metadata records confidence, status, author/generator, valid time, and source citations `(session/document revision, xpath)`.
- Links between memories and source nodes live in an indexed edge table and, where portable, a versioned package part. The index is rebuilt from package metadata; a package alone must still expose its provenance.
- Define statuses `proposed`, `accepted`, `rejected`, `superseded`, and `deleted`. A model may propose a memory but does not silently turn an uncertain observation into a durable fact. User corrections override generated claims and remain traceable.

## Compilation and recall

1. Append session events using idempotency keys and preserve turn order. Publish a session update through the outbox.
2. A compiler reads a bounded session revision, extracts candidate claims, checks for equivalent or contradictory existing memories, and emits proposed DCLX memory packages with source XPaths. Use deterministic rules where possible; optional local models enrich extraction.
3. Acceptance commits a new memory revision and refreshes retrieval indexes. Rejection records the decision so a retry does not propose the same claim indefinitely.
4. Recall searches accepted memories within the user/tenant scope, records delivered `(memory_id, revision_id)` in a recall ledger for the session, and suppresses redundant repeats while allowing a newer corrected revision.
5. Deleting a session or document evaluates dependent memories. Mark unsupported claims for review or withdrawal, rebuild links, and ensure retrieval does not serve deleted content. Keep explicit retention and purge operations.

## Conflict and privacy rules

- Store provenance and validity separately from semantic similarity. Similar text is not proof of the same claim; contradictory dates or subjects must be reviewed.
- Scope private memory to the owning tenant and user by default. Shared memory requires an explicit sharing record and the same authorization checks as documents.
- Keep secrets and raw tool arguments out of long-term memory by default. Add configurable redaction before compilation and a way to remove a claim and its index entries.
- The session writer and memory compiler must work with no cloud services. If models are unavailable, session logging and manual memory creation still work.

## Work sequence

1. Define and validate the session and memory DocLang profiles.
2. Implement append, replay, and retention for session packages.
3. Add compilation jobs, provenance links, and review transitions.
4. Add memory search and recall ledger integration.
5. Add correction, supersession, and deletion cascades.

## Acceptance checks

- Replay a session after restart with the same ordered turns and citations.
- Compile twice and get no duplicate accepted memory; reject a proposal and verify retries respect the rejection.
- Correct a preference and verify recall returns the accepted newer revision with its provenance.
- Delete a source and verify unsupported claims are flagged or withdrawn and all index entries are updated.
- Attempt cross-user and cross-tenant reads through each memory operation and verify denial.

## Exit artifact

Portable DocLang session and memory packages with reviewable provenance, correction, and bounded recall.
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
A local context tree that stores documents as immutable DocLang `.dclx` packages.

See the [capability guide](docs/index.md), [retrieval guide](docs/retrieval.md),
[session guide](docs/sessions.md), [memory guide](docs/memory.md),
[CLI reference](docs/cli.md), and
[single PDF and folder examples](examples/adding_resources/README.md).

Expand Down Expand Up @@ -45,6 +46,9 @@ uv run dc --store ./context-data index status
uv run dc --store ./context-data search "invoice" --uri docling://resources/papers
uv run --extra vectors dc --store ./context-data index vector rebuild
uv run --extra vectors dc --store ./context-data search "invoice" --mode hybrid
uv run dc --store ./context-data session start --id chat-1
uv run dc --store ./context-data session list
uv run dc --store ./context-data memory list --status proposed
```

Use `uv run dc` or activate the project environment before running `dc` by
Expand Down
88 changes: 87 additions & 1 deletion docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ uv run dc --store ./context-data tree docling://resources/papers -L 2
uv run dc --store ./context-data find "invoice" --uri docling://resources/papers
uv run dc --store ./context-data grep "total" --uri docling://resources/papers
uv run dc --store ./context-data search "invoice" --uri docling://resources/papers
uv run dc --store ./context-data session start --id chat-1
uv run dc --store ./context-data session list --status open
uv run dc --store ./context-data memory list --status proposed
```

`add-resource` accepts one document or a folder of documents. With the
Expand Down Expand Up @@ -51,12 +54,95 @@ collection summary may remain stale until its latest job completes.
Every `status` command prints a table by default. Pass `--json` for structured
output, for example `dc status --json`, `dc task status JOB_ID --json`, or
`dc index vector status --json`.
The main `status` also shows session, memory, and memory compilation job counts
for the selected user. List commands use tables by default and JSON arrays with
`--json`. The session and memory lists support `--limit` and `--offset` pages;
the default page size is 100 and the maximum is 1,000.

## Sessions and memory

Start a session before recording events. `start --id` is idempotent for an open
session; omit `--id` to generate one. Session commands use the selected tenant
and user. `append` needs a stable `--key`, so a retried harness hook cannot
duplicate an event. Use `--stdin` for text that should stay out of process
arguments. An optional binary `--attachment` is stored as a typed DCLX part.

```bash
uv run dc session start --id chat-1 --json
printf 'I prefer concise replies' | uv run dc session append chat-1 \
--kind feedback --key feedback-1 --stdin --json
uv run dc session replay chat-1
uv run dc session show chat-1
uv run dc session list --status open --since 2026-10-01
uv run dc session close chat-1
uv run dc memory worker --once
uv run dc memory proposals
```

Set `DOCLING_CONTEXT_SESSION=chat-1` to omit the ID from `session append`, or
use `--text` when sending a short event without stdin. `session close` prevents
new events and queues compilation of the final revision. `session jobs ID` and
`memory jobs ID` show compilation outcomes. `memory worker` polls for jobs;
`--once` handles one due job and exits. Queued jobs for older session revisions
become `superseded` when a newer revision is queued.

`session list` filters by `--status open|closed` and `--since`/`--until` ISO
dates or timestamps. `session trim ID --keep-last N` creates a shorter current
revision. Use `session attachment ID KEY --output FILE` to save one attached
event payload.

```bash
uv run dc memory list --kind preference --status proposed
uv run dc memory show MEMORY_URI --json
uv run dc memory accept MEMORY_URI
uv run dc memory search 'concise replies'
uv run dc memory recall 'concise replies' --session chat-1
uv run dc memory correct MEMORY_URI 'I prefer brief answers'
uv run dc memory reject MEMORY_URI --reason 'Incorrect inference'
uv run dc memory delete MEMORY_URI
uv run dc memory purge MEMORY_URI
```

`memory list` includes every non-purged status by default, including rejected
and deleted claims. Filter it by `--status`, `--kind`, `--since`, or `--until`.
`memory proposals` is a shortcut for proposed claims. `memory create KIND CLAIM`
creates a user-authored proposal; `--accepted` accepts it immediately. Optional
source citations require all four flags: `--source-uri`,
`--source-document-id`, `--source-revision-id`, and `--source-xpath`.
`memory search` and `memory recall` return accepted claims only. Recall records
which memory revision was delivered to a session and suppresses repeats.

### Delete and purge

`delete` withdraws an item from normal use while retaining its history. `purge`
removes that item's stored revisions and content. Both commands are scoped to
the selected tenant and user.

| Command | Immediate effect | What remains |
| --- | --- | --- |
| `dc session delete ID` | Removes the current session from `session list`, `show`, and `replay`. | Historical revisions and their attachments remain stored; compilation jobs and the session's recall ledger remain. |
| `dc session purge ID` | Deletes the session if necessary, then removes its revision records, attachments, compilation jobs, and recall ledger. | Derived memories are separate packages. The change outbox retains operation metadata. |
| `dc memory delete URI` | Writes a new revision with status `deleted`. The claim leaves memory search and recall. | The current claim and review history remain visible through `memory show URI` and `memory list --status deleted`; older revisions remain stored. |
| `dc memory purge URI` | Removes a previously deleted claim's current package, revisions, search index entries, provenance links, and recall records. | The source session or document and change outbox metadata remain. |

`memory purge` requires `memory delete` first. `session purge` can be called on
an active or previously deleted session. Removing a source session causes
dependent accepted memories with no other surviving citation to return to
`proposed` for review; it does not purge those memory packages. Purging a memory
does not erase its wording from a cited session or document. If that source is
later compiled again, it can produce a new proposal.

Package blobs are content addressed and are removed by purge only when no
remaining revision references them. The outbox retains earlier put/delete
identifiers and operation metadata; purge does not clear it. Purge removes the
item's stored content and revisions, but does not erase every related record.

`find` and `grep` are case-insensitive substring scans of current documents.
They scan at most 1,000 documents, 2,000 nodes per document, and 50,000 nodes
total, returning at most 100 hits. They accept
an optional URI subtree scope and enforce tenant and user access. They do not
yet provide embedding search or relevance ranking.
yet provide embedding search or relevance ranking. When scoped to memory
packages, they skip claims that are not accepted.

`search` uses the SQLite retrieval index and returns exact DCLX citations.
New or changed documents are indexed during ingestion. `index status` reports
Expand Down
4 changes: 4 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ them through local Python and `dc` command-line APIs.
| Remote conversion | Opt-in Docling Serve adapter; explicit endpoint and bounded response |
| Inspection | `dc status`, `ls`, `tree`, and task status |
| Search | `dc search` provides scoped lexical, vector, and hybrid retrieval, including picture assets with an image-capable model |
| Sessions | `dc session` records, lists, replays, closes, and purges revisioned event packages |
| Durable memory | `dc memory` lists and reviews DCLX claims, runs compilation jobs, and supports scoped recall |

`find` and `grep` retain their bounded substring scans. Indexed retrieval
verifies citations against the current DCLX revision before returning text.
Expand All @@ -23,4 +25,6 @@ verifies citations against the current DCLX revision before returning text.
- [Retrieval benchmark](../examples/retrieval/benchmark_corpus.py)
- [CLI commands](cli.md)
- [Storage and jobs](storage.md)
- [Agent sessions](sessions.md)
- [Durable memory](memory.md)
- [Single PDF and folder examples](../examples/adding_resources/README.md)
56 changes: 56 additions & 0 deletions docs/memory.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Durable memory

`MemoryService` stores each claim as a DCLX package under
`docling://users/{tenant}/{user}/memories/{kind}/{id}`. Supported kinds are
`preference`, `fact`, `entity`, `concept`, and `procedure`. The DocLang body
contains the readable claim and explanation. `context/memory.json` stores
confidence, status, author, optional validity dates, source citations, conflict
references, and review history. A citation names a source URI, document ID,
revision ID, and DocLang XPath. SQLite indexes these links and can reconstruct
them from the packages.

```python
from docling_context import (
LocalContextStore, MemoryCompiler, MemoryService, Principal, SessionStore,
)

principal = Principal("acme", "alice")
with LocalContextStore("./context-data") as store:
sessions = SessionStore(store)
sessions.append(
principal, "conversation-1", key="feedback-1", kind="feedback",
text="Remember: I prefer short answers",
)
compiler = MemoryCompiler(store)
while compiler.run_once() is not None:
pass
memories = MemoryService(store)
for proposal in memories.list(principal, status="proposed"):
print(proposal.claim, proposal.citations)
memories.accept(principal, proposal.uri)
```

The compiler accepts only explicit `Remember:` or `Remember that` statements
and `I prefer` turns or feedback. It ignores tool calls and tool results. It
redacts common credential patterns before writing a proposal; callers can pass
a custom `redactor` to `MemoryCompiler`. Compilation runs locally and requires
no model or cloud service. Jobs have leases and retries. Exact normalized
claim matches are deduplicated, including rejected proposals, so retries do not
undo a review decision. Similar or conflicting claims are left for review.

`create` makes a proposed memory by default. A direct user-authored call may
pass `accepted=True`. Compiler proposals require `accept` before they enter
search. `reject`, `correct`, and `delete` write new package revisions. A
correction is user authored, accepted, and records the revision it supersedes.
`purge` physically removes a deleted claim and its revisions. `search` returns
accepted memories within the current user. `recall(principal, session_id, query)`
records delivered memory and revision IDs; it suppresses repeats in that
session, while a corrected revision can be delivered again.
`list` includes all non-purged statuses by default and can filter by status,
kind, or update time. The `dc memory` CLI exposes listing, review, compilation,
search, recall, and purge; see the [CLI guide](cli.md).

Deleting a cited session or document checks dependent memories. A claim with
no surviving cited source becomes `proposed` and leaves the retrieval index.
All operations check tenant and user ownership. Sharing is not exposed by this
API.
45 changes: 45 additions & 0 deletions docs/sessions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Agent sessions

`SessionStore` appends agent turns, tool calls, tool results, and feedback to a
revisioned DCLX package at
`docling://users/{tenant}/{user}/sessions/{session_id}`. Each event is a DocLang
`<text>` node. The `context/session.json` archive part records its kind, turn ID,
idempotency key, timestamp, and exact XPath. A raw attachment lives in a typed
`attachments/` archive part; metadata stores its size and SHA-256 hash.
Sessions have `open` and `closed` states. `start` creates an empty package;
`close` prevents further appends and queues the final revision for compilation.

```python
from docling_context import LocalContextStore, Principal, SessionStore

principal = Principal("acme", "alice")
with LocalContextStore("./context-data") as store:
sessions = SessionStore(store)
sessions.start(principal, "conversation-1")
event = sessions.append(
principal, "conversation-1", key="turn-1", kind="turn",
text="I prefer short answers", turn_id="turn-1",
)
print(event.xpath, event.revision_id)
print(sessions.replay(principal, "conversation-1"))
sessions.close(principal, "conversation-1")
```

Repeated appends with the same key return the existing event. The writer checks
the prior revision before replacing the package, so concurrent writes must retry
after a revision conflict. Sessions contain at most 1,000 events, each at most
20,000 characters. An attachment is limited to 4 MiB, and the session has a
32 MiB total attachment limit. The DCLX package is
portable and can be replayed after restarting the store.

`trim(principal, id, keep_last=n)` creates a revision with the last `n` events.
Older immutable revisions remain addressable. `delete` removes the logical
session; `purge` removes its revision history, attachments, compilation jobs,
and recall ledger. A source deletion also moves memories with no surviving
citations back to review.
`list` filters current sessions by status and update time; the matching CLI
command is `dc session list`. See the [CLI guide](cli.md) for start, append,
replay, close, retention, and job commands.

Session URIs and attachments require the owning tenant and user. This version
does not expose sharing of private sessions.
12 changes: 12 additions & 0 deletions src/docling_context/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
"""Local-first context storage backed by DocLang packages."""

from .compiler import MemoryCompiler, MemoryJob, redact_secrets
from .contracts import ContextStore
from .conversion_config import PdfConversionConfig
from .converters import (
Expand All @@ -8,6 +9,7 @@
DoclingServeConverter,
LocalDoclingConverter,
)
from .durable_memory import MemoryRecord, MemoryService, SourceCitation
from .embeddings import (
CLIP_MODEL,
DEFAULT_MODEL,
Expand Down Expand Up @@ -35,6 +37,7 @@
)
from .package import FilePackageStore, PackageError, PackageMissing
from .retrieval import ContextResult, RetrievalHit, Retriever, SearchScope
from .sessions import SessionEvent, SessionInfo, SessionStore
from .uri import AccessDenied, ContextURI, InvalidURI, authorize_uri, parse_uri
from .vectors import EmbeddingProvider, TurbovecIndex, VectorIndex

Expand All @@ -60,7 +63,11 @@
"Job",
"LocalContextStore",
"LocalDoclingConverter",
"MemoryCompiler",
"MemoryContextStore",
"MemoryJob",
"MemoryRecord",
"MemoryService",
"MlxEmbeddingProvider",
"NodeAddress",
"NodeContent",
Expand All @@ -74,6 +81,10 @@
"Retriever",
"RevisionConflict",
"SearchScope",
"SessionEvent",
"SessionInfo",
"SessionStore",
"SourceCitation",
"SummaryProvider",
"TenantScope",
"TreeEntry",
Expand All @@ -85,5 +96,6 @@
"embed_text_image",
"local_embedding_provider",
"parse_uri",
"redact_secrets",
"stored_embedding_provider",
]
Loading
Loading