Skip to content

Append native session items atomically - #1034

Merged
zlareb1 merged 15 commits into
mainfrom
fix/native-session-atomic-append
Nov 21, 2026
Merged

zlareb1 merged 15 commits into
mainfrom
fix/native-session-atomic-append

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Related: #1059
Related: #1077

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in a release)

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

Native session additions in both SDKs append all items in one Postgres transaction. A refused item no longer leaves an earlier item from the same addition stored.

Why

The Python native session contract requires an atomic addition. Separate point writes in either adapter expose a prefix when a later item fails. Ordinary bulk writes are unsuitable because they can omit a refused entry or fold an equal value into another memory.

memory.append_batch accepts distinct fresh keys and fixed episodic rows. It reuses the bulk row builders, blacklist and sensitivity checks, provenance, policy admission, revision audit and idempotency transaction. Every selected entry must survive preparation, and every key must still be absent under the commit's generation and head locks. Equal values remain separate occurrences. The operation refuses the ClickHouse authority before append preparation and never falls back to point or ordinary bulk writes.

Scope and risk

Both adapters preencode the complete addition, check positive operation availability, and send one request with stable prepared keys. The existing application-owned callback protection and transport response observation remain in place. Existing high-sensitivity fences remain in place, including their embedder model-name lookup; strict episodic preparation makes no embedding or model calls.

The operation writes the existing episodic format under the full identity tuple. It adds no table, migration, dependency or separate content store. Normal tenant, user, session-content, retention and erasure rules apply. Ordinary bulk deduplication and per-entry verdicts are unchanged. This fixes additions only; it does not make concurrent history paging, pops or clears linearizable, and it does not change the supported native SDK dependency range.

A publication failure after commit reports PUBLICATION_PENDING with committed=true. The same idempotency key can replay the committed batch. An uncertain commit remains uncertain until the original request is resolved; no compensation or replacement-key retry is introduced.

Session-content admission can reuse one quiescent private HTTP transport across proofs. Each proof still creates a fresh named session, bootstrap and nonce. Failed, cancelled or in-flight work prevents reuse. The ordinary query pool and the security deadlines remain unchanged; the cached transport carries no authority verdict.

Compatibility, rollout and rollback

  • Wire: adds experimental writer operation memory.append_batch, taking 1–1,000 entries with fresh keys, values and optional provenance downgrades. It is session-content-bearing, idempotent, hidden from MCP and available through generated Python and TypeScript clients.
  • Native additions in both SDKs require a server advertising the operation and using the Postgres memory authority. Older servers refuse; history reads keep working. Deploy the server before upgrading clients that need additions. During a mixed deployment, a missing operation on the selected replica propagates without mutation fallback.
  • The owning native-session architecture describes whole additions, marked discovery and post-commit response observation.
  • The native examples install both SDKs from the guide's checkout and require a server from that same checkout. The commands do not assume an available registry artifact containing the new operation.
  • Operator steps: no settings, migration, re-index or backfill. The Postgres authority is required for additions through this operation.
  • Rollback: episodic rows remain readable by older builds. Reverting removes the operation and the SDK's atomic-add behavior; it requires coordinating clients that depend on the operation.

Tests

Scoped source checks and all six generator checks pass. Sender census changes renew the reviewed-source receipt; sender classes, exception sites and reasons are unchanged.

New server regressions exercise the real handler and preparation with controlled storage seams: repeated equal values, second-entry blacklist refusal, duplicate/live/deleted keys, late locked-head collisions, unavailable head reads, authority loss, provenance and committed replay. Hosted integration cases exercise real Postgres writes and reads, request replay, concurrent collisions, blacklist refusal, authorization and user erasure. Native adapter regressions cover whole-input serialization, one batch, discovery and old-replica refusal, pending publication and retained callback protection. The TypeScript adapter uses the same operation to prevent partial writes; it does not claim the optional transaction-aware session protocol.

The 21 directly related append and replicated-fixture unit tests pass in 1.23 seconds with controlled storage seams. The replicated erasure fixture provisions its own write-lease table. The stale-owner fixture waits for and asserts termination of its old backend before taking the successor lock; the original lock and mode assertions remain.

Full hosted qualification on this head passes: all six unit shards, coverage and benchmark checks, native SDK and wheel checks, every integration shard and isolation gate, both Postgres authorities, mixed-version and Cell acceptance. The rollup contains 54 successful checks and two expected documentation deployment/indexing skips; all five applicable workflows succeed. These hosted checks provide the database qualification; the controlled storage cases alone do not.

Transport regressions cover fresh sessions, reuse, bootstrap failure, cancellation, failed channels, disconnects and concurrent children. The current head passes their hosted checks. No latency improvement is claimed.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated (an agent wrote most of the diff): Codex wrote the implementation and regressions and checked source contracts, generated output and the listed static checks. Independent final-head review and hosted behavioral qualification are required.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Required hosted format, lint, type, unit, native adapter and integration checks pass.
  • Operation SDKs, reference pages and reach manifest are regenerated.
  • The existing row format and erasure coverage are retained; no migration is added.
  • Architecture and feature documentation describe the operation and its compatibility limits.
  • Fresh independent review covers the final pushed source and public body.
  • Commits are signed off.

Signed-off-by: Mohammad Lareb Zafar <lareb.zafar@clickhouse.com>
Signed-off-by: Mohammad Lareb Zafar <lareb.zafar@clickhouse.com>
Signed-off-by: Mohammad Lareb Zafar <lareb.zafar@clickhouse.com>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 20, 2026 10:53pm UTC

Request Review

Signed-off-by: Mohammad Lareb Zafar <lareb.zafar@clickhouse.com>
Use one idle-driven clock for the mock exchange timers and owned close. Keep the six cases and their bounds, with a private-loop task census at teardown.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit e1f47e6)
Specify fresh proof sessions, private transport ownership and failure disposal before implementing bounded reuse.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep one idle channel per client while rebuilding each checked session and discarding failed, replaced or active-child transports.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Preserve the write-fence and atomic-append storage guarantees while composing current entity resolution and commit handling.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Replicated erasure tests require their own write-lease table, and successor-lock tests must confirm the previous backend has exited before proceeding. Preserve production fencing and the original assertions.

Includes the reviewed fixture repair from 96be03c.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the merged replicated-erasure fixture and its private Keeper path, retaining the regression cases and the confirmed backend-termination barrier. Preserve the atomic session implementation and current main changes.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 marked this pull request as ready for review November 21, 2026 01:17
@zlareb1
zlareb1 merged commit 89d7b5e into main Nov 21, 2026
56 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant