Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 24 additions & 11 deletions docs/architecture/openai-agents-sessions.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,23 @@
Loom episodic memories. Existing tools can separately retain facts beyond a
conversation. The framework packages remain optional dependencies.

Each item is one `memory.set` call with its native JSON, a timestamp-and-random
key, the end user and a session derived from the user and native session id.
Point writes bypass SDK coalescing so concurrent repeated items remain separate.
Episodic rows preserve repeated items without semantic extraction or duplicate
merging. Python and TypeScript use separate derived-session prefixes because
their native item formats differ.
Each nonempty addition encodes every native item before discovery or mutation,
checks that the server advertises `memory.append_batch`, and sends one request.
The server requires the Postgres memory authority. It prepares every episodic row
and commits the whole addition only when every key is fresh under the generation
and head locks. A refused item or existing key refuses the batch. There is no
point-write or ordinary-bulk fallback; an empty addition sends nothing. The SDKs
and server come from the same checkout, as the
[installation instructions](../features/integrations/frameworks.md#openai-agents-sdk)
require.

Each item retains its native JSON, timestamp-and-random key, end user and session
derived from the user and native session id. Episodic rows preserve repeated
values as distinct occurrences without semantic extraction or duplicate merging.
Python and TypeScript use separate derived-session prefixes because their native
item formats differ. A committed batch can report `PUBLICATION_PENDING`; resolve
an uncertain outcome with the original request's idempotency key, without a fresh
addition or compensating deletion.

Reads page through `memory.list`, deduplicate overlapping keys and hydrate missing
or truncated values with `memory.get`. Items are returned in key order, with a
Expand All @@ -19,8 +30,8 @@ snapshots the validated user at construction. Its native image and file byte
payloads use JSON arrays in storage and return as `Uint8Array`; ordinary JSON and
media information remain unchanged. Deletion carries the user and derived
session: `pop_item` deletes the latest item, and `clear_session` deletes the listed
history. Concurrent appends or clears are individual server operations, not a
transaction across the conversation.
history. Each addition is atomic; concurrent history paging, pops and clears do
not form a transaction across the conversation.
If an item disappears before hydration, the read fails instead of returning an
incomplete native item. Keys use writer clocks; concurrent writers do not share a
server sequence.
Expand All @@ -36,9 +47,11 @@ conversation.
Native history reads and additions run as automatic work under all retained
contributors. A marked reply protects every contributor before its value returns.
A complete first marked history reply may still return its items; subsequent
pagination, hydration or item writes are suppressed. A marked item write stops a
batch before the next item. Explicit pop and clear operations remain usable after
protection and keep observing replies under the retained contributors.
pagination, hydration or batch writes are suppressed. A marked discovery response
prevents the append request. A marked append acknowledgement protects the retained
contributors after the whole batch may already have committed; it does not imply
rollback. Explicit pop and clear operations remain usable after protection and
keep observing replies under the retained contributors.
Model-selected tools keep their explicit operation semantics.

## Source entry points
Expand Down
28 changes: 28 additions & 0 deletions docs/architecture/storage-authority.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,34 @@ stops serving and background writers, waits out bounded sends, and states when t
guarantee and optional settings apply. Both builds serve the same schema.
An older writer can still reproduce the unchecked expiry race during that transition.

## Atomic episodic appends

`memory.append_batch` appends 1–1,000 episodic occurrences under one identity tuple.
Every key must be distinct and absent, including from tombstoned heads. Equal values
remain separate rows. The handler requires the Postgres authority before memory or
provider mutation, then reuses bulk row preparation, blacklist checks, sensitivity,
provenance and policy admission. It runs no embedding, identity deduplication, entity
resolution or semantic supersession. Any refused entry aborts the entire batch.

The commit checks absence again while holding the authority's generation and head
locks. Heads, revision rows, hash-only audit rows, delivery obligations and the bound
idempotency result commit together. A competing writer that creates one key refuses
the complete append. Ordinary `memory.bulk_set` keeps its per-entry verdicts and
identity deduplication; its partial selection is unsuitable for native history.

Publication follows the commit. `PUBLICATION_PENDING` means the batch is committed and
its remaining delivery belongs to the delivery worker. A replay of that request's
idempotency key returns the committed rows, without appending again. A lost commit
acknowledgement remains an uncertain outcome until that same request is resolved;
clients must not compensate with deletes or invent replacement keys.

The operation uses the existing episodic row format and erasure coverage, with no
schema migration. Older servers do not register it and refuse rather than accepting
an ignored atomicity flag. The ClickHouse authority refuses it; the operation catalogue
advertises availability, not the deployment's storage readiness. Existing history
reads, pops and clears keep their row format and behavior; this append operation does
not give them cross-process linearizability.

## The tables

The schema is `schemas/pg_migrations/` ([migrations](#migrations)). Every tenant primary
Expand Down
42 changes: 37 additions & 5 deletions docs/features/integrations/frameworks.md
Original file line number Diff line number Diff line change
Expand Up @@ -306,8 +306,16 @@ line per hit. With a `SyncAgent` the tool is sync; with an `Agent` it is async,
## OpenAI Agents SDK

`loom_tools` returns Loom's agent verbs as the OpenAI Agents SDK's `FunctionTool`s, so the
model keeps and reads long-term memory in Loom during a run. Install with
`pip install 'clickhouse-loom[openai-agents]'` (`openai-agents` 0.24).
model keeps and reads long-term memory in Loom during a run. For the Python examples,
install the SDK from the root of the Loom checkout that supplies this guide:

```bash
pip install './sdk/python[openai-agents]'
```

The extra installs `openai-agents` 0.24. Native sessions also require a server built
from that same checkout with the Postgres memory authority; see
[running the server from source](../../development/local-development.md#running-the-server-from-source).

```python
from agents import Agent as OpenAIAgent, Runner
Expand Down Expand Up @@ -363,13 +371,29 @@ result = await Runner.run(assistant, "Book my usual seat to Lisbon", session=ses
| `Session` | Loom |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `get_items` | `memory.list` of the session's items, sorted by writer-generated key; `limit` (or `session_settings.limit`) keeps the latest after scanning the history; an item the list cuts at 2,048 characters, or sends without a value, is read whole with `memory.get` |
| `add_items` | one `memory.set` per item, an `episodic` memory under `<key_prefix>/<nanoseconds>-<random>`, the item's JSON its value |
| `add_items` | One `memory.append_batch` per call, each item an `episodic` row under `<key_prefix>/<nanoseconds>-<random>` with the item's JSON as its value |
| `pop_item` | the latest item, read as `get_items(limit=1)` reads it, deleted with `memory.delete` and returned |
| `clear_session` | `memory.delete` of every item of the session |

Python and TypeScript additions require a server advertising
[`memory.append_batch`](../../reference/operations/memory.md#memoryappend_batch) and using the
Postgres memory authority. A nonempty call encodes every item before sending one request
of at most 1,000 items. Equal values remain distinct occurrences. A refused entry or an
existing key refuses the whole batch before any history is appended. An empty call sends
nothing. Discovery failures, an older replica's missing operation and storage refusals
propagate without a point-write or bulk-write fallback. Existing history remains readable
on older servers. This changes the server requirement for additions in both SDKs; their supported
OpenAI Agents SDK dependency ranges are unchanged.

A committed batch whose publication is pending reports `PUBLICATION_PENDING` with
`committed=true`; it is not a rollback. Preserve the original request's idempotency key
when resolving an uncertain outcome. Do not retry as a fresh addition or delete earlier
items to compensate. These append guarantees do not make concurrent history paging,
`pop_item` or `clear_session` atomic.

An `episodic` memory is not embedded, so an item is never merged with or judged against
another row, and a repeated item stays two items (one `memory.set` per item, bypassing SDK
coalescing: a batch would fold a value identical to a live row). A read fails if an item
another row. Repeated values remain separate occurrences in both SDKs' batches.
A read fails if an item
disappears before its full value can be fetched. Items are stored under a Loom session derived from
the user and the session id (`oai-py-` and a digest of the pair in Python, `oai-ts-`
in TypeScript). Keys follow writer clocks; independent writers do not share a server
Expand All @@ -388,6 +412,14 @@ established beyond its history, give the model `loom_remember`, or send the turn
| `key_prefix` | `openai-agents` | the items' key prefix |
| `session_settings` | the SDK's | `limit`: the default number of latest items `get_items` returns |

Build and install the TypeScript SDK from that same checkout using the
[SDK source instructions](../../../sdk/typescript/README.md). In your application,
install its supported native peer:

```bash
npm install '@openai/agents@>=0.22.0 <0.23.0'
```

In TypeScript, `loomTools(agent, scope)` from `@clickhouse/loom/integrations/openai-agents`
returns the same tools built with `tool()` from `@openai/agents`, an optional peer
dependency:
Expand Down
11 changes: 6 additions & 5 deletions docs/reference/http-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,13 +159,14 @@ curl -sX POST "$LOOM/v1/memory.set" \
blank or `"-"` org is the token's own. On a single-user server, a call with no token
that names no `org`, or a blank or `"-"` one, runs in the install's org
(`LOOM_SELF_SIGNUP_ORG`), the one MCP and the dashboard use; a code or tracker op
keeps its plane org. A call with no token to `memory.set`, `memory.bulk_set`,
`memory.get`, `memory.delete`, `memory.list`, `memory.history`, `memory.search`,
keeps its plane org. A call with no token to `memory.set`, `memory.append_batch`,
`memory.bulk_set`, `memory.get`, `memory.delete`, `memory.list`, `memory.history`,
`memory.search`,
`memory.search_trace`, `memory.ask`, `memory.update`, `search.query` or
`knowledge.search` may also leave out `namespace` and `agent`: it runs in the
install's namespace (`LOOM_DEFAULT_NAMESPACE`). With no agent, `memory.set` and
`memory.bulk_set` write as agent `http`; `memory.get` and `memory.delete` try the
key under agent `http`, then under any agent of the same user; `memory.list` reads
install's namespace (`LOOM_DEFAULT_NAMESPACE`). With no agent, `memory.set`,
`memory.append_batch` and `memory.bulk_set` write as agent `http`; `memory.get` and
`memory.delete` try the key under agent `http`, then under any agent of the same user; `memory.list` reads
every agent's rows of the call's user and session (`-` when it names none) and
`memory.history` every agent's, user's and session's versions of the key, as on
MCP; a `memory.history` that names a `user_id` or `session_id` names its agent
Expand Down
3 changes: 2 additions & 1 deletion docs/reference/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ python scripts/codegen/docs_reference.py --check # exit 1 when a region is stal
<!-- Generated by `python scripts/codegen/docs_reference.py`; edit the source, not this region. -->
<!-- prettier-ignore-start -->

375 operations in 43 groups. Every op is served at `POST /v1/<op>` and as the MCP tool `loom_<op>` (dots become underscores).
376 operations in 43 groups. Every op is served at `POST /v1/<op>` and as the MCP tool `loom_<op>` (dots become underscores).

Columns: **Role** is the minimum role (`viewer` < `writer` < `admin` < `org_admin` < `platform_admin`); **Kind** is `read` (idempotent, viewer, unaudited, writes nothing — MCP `readOnlyHint`), `idempotent` (safe to retry) or `write`; **Listed under `all`** says whether an MCP connection on the `all` toolset lists the op in `tools/list` (`code.*` ops only with `LOOM_MCP_CODE_TOOLS=true`); an op marked `no` is still callable by name or through `loom_op`, unless it is hidden from MCP; **Audit** marks ops that leave an `audit_log` row.

Expand Down Expand Up @@ -476,6 +476,7 @@ Fields for every `memory` op: [operations/memory.md](operations/memory.md).
| --- | --- | --- | --- | --- | --- |
| [`memory.agents_list`](operations/memory.md#memoryagents_list) | Distinct agent names that have ever written into (org, namespace). agent + user_id are forced sentinels — we aggregate ACROSS them to discover which agents exist. | viewer | read | yes | |
| [`memory.aggregate`](operations/memory.md#memoryaggregate) | Templated SQL aggregation over typed records in loom.memory. | viewer | read | no | |
| [`memory.append_batch`](operations/memory.md#memoryappend_batch) | Append distinct episodic occurrences atomically on the Postgres authority. | writer | idempotent | no | |
| [`memory.apply`](operations/memory.md#memoryapply) | One scope grammar for every write that can touch more than one row. | writer | write | yes | yes |
| [`memory.ask`](operations/memory.md#memoryask) | Ask a question and get a grounded answer synthesized from the caller's recalled memories (or, with allow_synthesis=False, just the memories). | viewer | read | yes | |
| [`memory.assign_category`](operations/memory.md#memoryassign_category) | Link a memory to one of the namespace's registered categories. | writer | idempotent | no | yes |
Expand Down
32 changes: 32 additions & 0 deletions docs/reference/operations/memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,38 @@ Templated SQL aggregation over typed records in loom.memory.
| `rows` | `list[_AggregateRow]` | One row per group (a single row when `group_by` is empty). |
| `sql_executed` | `str` | the exact SQL run, for debugging without trace logs |

## `memory.append_batch`

Append distinct episodic occurrences atomically on the Postgres authority.

`POST /v1/memory.append_batch` · MCP `loom_memory_append_batch` · role `writer` · idempotent · not listed under the MCP `all` toolset

| Input field | Type | Default | Meaning |
| --- | --- | --- | --- |
| `org` | `str` | required | The tenant (first element of the identity tuple). |
| `namespace` | `str` | required | The namespace the op reads or writes. |
| `agent` | `str` | required | The writing or reading agent; `'-'` or `'*'` means any agent where the op allows it. |
| `user_id` | `str` | `'-'` | The end user the row belongs to; `'-'` for none. |
| `session_id` | `str` | `'-'` | The session or conversation the row belongs to; `'-'` for none. |
| `run_id` | `str` | `'-'` | The agent run (task scope) the call belongs to; `'-'` for none. |
| `entries` | `list[_AppendBatchEntry]` | required | |

| Output field | Type | Meaning |
| --- | --- | --- |
| `written` | `list[_BulkSetResult]` | One result per entry, in order: key, version (0 when not inserted) and verdict. |
| `skipped_pinned` | `list[str]` | Keys whose live row is pinned: skipped (result version=0) rather than overwritten, since overwriting would also unpin (entries default pinned=False). |
| `skipped_locked` | `list[str]` | Keys whose live row is locked read_only: skipped (result version=0), where memory.set would answer 423 LOCKED. |
| `skipped_blacklisted` | `list[str]` | Keys whose entry the blacklist refused or could not check in the request's budget: not written (result version=0), where memory.set would answer 422 BLACKLISTED. |
| `merged` | `list[str]` | Conflict verdicts, the same four memory.set produces: canonical restatement of an existing row → not inserted, existing row reinforced; close to a pinned peer → not inserted; same fact in other words → inserted as status=duplicate… |
| `value_updates_linked` | `int` | Older rows marked superseded by a value update in this batch. |
| `rejected_pinned` | `list[str]` | Keys not inserted because they are too close to a pinned row. |
| `duplicates` | `list[str]` | Keys inserted as `status=duplicate`: the same fact as a live row in other words. |
| `conflicted` | `list[str]` | Keys inserted and cross-linked as `status=conflicted` with the rows they contradict. |
| `unchecked` | `int` | Entries past bulk_conflict_check_max_entries, written without a check. |
| `retyped` | `list[str]` | Entries declared procedural whose value is not a procedure body, stored as semantic tagged retyped_from:procedural. |
| `identity_duplicates` | `dict[str, str]` | Entries not written because a live row in the same identity already carries the same content: entry key -> the keeper's key. |
| `stage_ms` | `dict[str, float]` | Wall time per phase of the write (existing_lookup, entity_resolution, prepare, embed, verdicts, insert, value_updates, projections, total) — Diagnostic; readers tolerate unknown keys. |

## `memory.apply`

One scope grammar for every write that can touch more than one row.
Expand Down
Loading