Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion docs/architecture/clickhouse.md
Original file line number Diff line number Diff line change
Expand Up @@ -668,7 +668,23 @@ with `purge=true`, runner state, and the supersession rows `knowledge.restore` r
(`server/memory/workers/usage_fold.py`). On ClickHouse Cloud each statement is a patch part and
a Keeper entry, so cost is per statement. A pass is skipped while the table has too many
unfinished mutations. Only "lightweight update unsupported" errors fall back to
`ALTER TABLE … UPDATE`; a transient error is retried next pass.
`ALTER TABLE … UPDATE`; a transient error is retried next pass. Every batch retains
its boundary second because `bumped_at` has second resolution. A cursor ordered by
`bumped_at`, SHA-256 digests of `org` and `namespace`, and `memory_id` pages through
ties. Fixed-width digests fit supported identity sizes within the cursor limit.
`worker_status` stores the cursor when a pass stops at its time budget. A completed scan stores an epoch cursor to
reset the scan: an empty cursor tells the supervisor to keep its previous value.

A full-version insert can carry counters read before a fold and replace its result.
Every pass therefore compares the absolute totals and last-recall time with the live
rows, even when the usage high-water mark is unchanged, and rewinds to the earliest
discrepancy. Comparisons use the same UInt32 saturation as the counter writes, so
an unattainable total cannot keep rewinding the scan. This adds an aggregate join
per nonempty pass. Repair is eventual once
the competing writes are visible; a read followed by an insert is not atomic. Absolute
targets and `greatest` make counter repair idempotent across replicas. The usage events
remain authoritative; the counters on `memory` are a derived projection.

- **Erasure purges** remove rows. On `erasure/store/mutations.py::LIGHTWEIGHT_PURGE_TABLES` (vector,
chunk, summary and evidence tables) a purge uses lightweight `DELETE FROM`, which writes
only the row mask: rows vanish from every read at once and bytes leave at the next merge.
Expand Down
134 changes: 87 additions & 47 deletions server/memory/workers/usage_fold.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,17 +55,20 @@
statements instead of 2,000; on ClickHouse Cloud each lightweight UPDATE
is a patch part and the cost is per statement, not per row.

The watermark is process-local. On a cold start it is derived from the
destination tables (the earliest `bumped_at` whose totals exceed what
`loom.memory` already carries) instead of the epoch, so a redeploy does
not re-walk the whole history at the batch cap before reaching live
rows, and a pass keeps taking batches until it is caught up or its time
budget is spent.
The watermark is process-local and retains the boundary second. Every pass
compares totals with the destination counters: a concurrent full-version
insert can carry counts read before a fold and must be repaired even when
no usage timestamp changes. This is eventual reconciliation, not an atomic
update of future versions. A cold start begins at the earliest discrepancy
instead of the epoch. A cursor on bumped_at, fixed-width identity digests,
and memory_id pages through ties and is saved in worker_status when the time
budget is spent. Digests keep supported identity sizes within its cursor limit.
"""

from __future__ import annotations

import hashlib
import json
import logging
import time
from dataclasses import dataclass, field
Expand Down Expand Up @@ -94,6 +97,10 @@

_UINT32_MAX = 2**32 - 1

_Cursor = tuple[datetime, str, str, str]
# Empty cursors preserve the supervisor's last value; an epoch cursor resets it.
_RESET_CURSOR = json.dumps((UNIX_EPOCH.isoformat(), "", "", ""))

_watermark: datetime = UNIX_EPOCH
_watermark_derived: bool = False

Expand Down Expand Up @@ -284,6 +291,16 @@ def _fold_key(org: Any, namespace: Any, memory_id: Any) -> tuple[str, str, str]:
return str(org), str(namespace), str(memory_id)


def _cursor_identity(org: Any, namespace: Any, memory_id: Any) -> tuple[str, str, str]:
"""Fixed-width sort components fit the supervisor's bounded cursor."""
org, namespace, memory_id = _fold_key(org, namespace, memory_id)
return (
hashlib.sha256(org.encode()).hexdigest().upper(),
hashlib.sha256(namespace.encode()).hexdigest().upper(),
memory_id,
)


def _scope_params(keys: list[tuple[str, str, str]]) -> dict[str, Any]:
"""`_FOLD_SCOPE`'s parameters. `keys` stays in the caller's order: the UPDATE's
transform() pairs it with the targets by position."""
Expand Down Expand Up @@ -454,36 +471,37 @@ async def _fold_reinforcement_rows(ch: _CHClient, targets: list[_Target], chunk:
return statements


async def _derive_cold_start_watermark(ch: _CHClient, hwm: datetime) -> datetime:
"""Where to start after a restart: just before the earliest totals row
whose recall/evidence exceed what `loom.memory` already carries. Targets
async def _derive_watermark(ch: _CHClient, hwm: datetime) -> datetime:
"""Reconcile from the earliest totals row ahead of a live memory. Targets
are absolute, so the join is exact for the two counters that live on
`loom.memory`; `used`-only activity has no destination counter there
and re-folds when its row is next bumped. No unfolded rows → the high
water mark (nothing to do)."""
and re-folds when its row is next bumped. With no discrepancy, retain the
high water mark's second for usage that becomes visible after this read."""
row = await ch.query_one(
"""
f"""
SELECT min(t.bumped) AS first_unfolded
FROM (
SELECT org, namespace, memory_id,
sum(recall_count) AS recall_total,
sum(evidence_count) AS evidence_total,
max(last_recalled_at) AS last_recalled,
max(bumped_at) AS bumped
FROM loom.memory_usage_totals
GROUP BY org, namespace, memory_id
) AS t
INNER JOIN (
SELECT org, namespace, memory_id, recall_count, evidence_count
SELECT org, namespace, memory_id, recall_count, evidence_count, last_recalled_at
FROM loom.memory FINAL
WHERE tombstoned_at IS NULL
) AS m ON m.org = t.org AND m.namespace = t.namespace AND m.memory_id = t.memory_id
WHERE t.recall_total > m.recall_count
OR (1 + t.evidence_total) > m.evidence_count
WHERE least(t.recall_total, {_UINT32_MAX}) > m.recall_count
OR (1 + least(t.evidence_total, {_UINT32_MAX - 1})) > m.evidence_count
OR t.last_recalled > coalesce(m.last_recalled_at, toDateTime(0))
"""
)
first = _utc((row or {}).get("first_unfolded"))
if first <= UNIX_EPOCH:
return hwm
return max(UNIX_EPOCH, hwm - timedelta(seconds=1))
return first - timedelta(seconds=1)


Expand All @@ -509,9 +527,13 @@ async def _derive_cold_start_watermark(ch: _CHClient, hwm: datetime) -> datetime
"""


async def _fold_batch(ch: _CHClient, cfg: UsageFoldConfig, result: UsageFoldResult) -> bool:
"""One batch past the watermark. Returns True when a full batch was
taken (there may be more), False when the backlog is drained."""
async def _fold_batch(
ch: _CHClient,
cfg: UsageFoldConfig,
result: UsageFoldResult,
after: _Cursor,
) -> _Cursor | None:
"""Fold one batch; return its tie-break cursor if more rows may remain."""
global _watermark
# Backpressure: while the destination already carries a
# queue of unfinished mutations, folding more only lengthens it. Skip the
Expand All @@ -526,7 +548,7 @@ async def _fold_batch(ch: _CHClient, cfg: UsageFoldConfig, result: UsageFoldResu
settings.memory_usage_fold_max_pending_mutations,
_watermark.isoformat(),
)
return False
return None
# Absolute totals per memory. The HAVING filter must come AFTER
# the GROUP BY (a row-level bumped_at predicate would drop the
# older unmerged SummingMergeTree part-rows of a freshly-bumped
Expand All @@ -543,14 +565,16 @@ async def _fold_batch(ch: _CHClient, cfg: UsageFoldConfig, result: UsageFoldResu
FROM loom.memory_usage_totals
GROUP BY org, namespace, memory_id
HAVING bumped > {wm:DateTime}
ORDER BY bumped ASC
AND (bumped, hex(SHA256(org)), hex(SHA256(namespace)), memory_id)
> {after:Tuple(DateTime, String, String, String)}
ORDER BY bumped ASC, hex(SHA256(org)) ASC, hex(SHA256(namespace)) ASC, memory_id ASC
LIMIT {lim:UInt32}
""",
params={"wm": _watermark, "lim": int(cfg.max_memories_per_pass)},
params={"wm": _watermark, "after": after, "lim": int(cfg.max_memories_per_pass)},
)
result.candidates += len(rows)
if not rows:
return False
return None
result.batches += 1

# Resolve each (org, namespace, memory_id) to the keys holding it, with their
Expand All @@ -574,17 +598,28 @@ async def _fold_batch(ch: _CHClient, cfg: UsageFoldConfig, result: UsageFoldResu
# Advance the watermark only after every fold in the batch landed — a
# partial batch re-folds from the old watermark, which is safe because
# targets are absolute.
max_bumped = max(_utc(r["bumped"]) for r in rows)
full = len(rows) >= int(cfg.max_memories_per_pass)
if full:
# Full batch: bumped_at is second-resolution, so boundary ties may
# exist past the LIMIT. Step back one second to re-fold the boundary
# next batch (idempotent) — unless that would stall the watermark.
new_wm = max_bumped - timedelta(seconds=1)
_watermark = new_wm if new_wm > _watermark else max_bumped
else:
_watermark = max_bumped
return full
cursor = max(
(_utc(r["bumped"]), *_cursor_identity(r["org"], r["namespace"], r["memory_id"]))
for r in rows
)
# Every batch retains the boundary second for the next pass. The cursor
# makes progress within this pass without dropping ties past the LIMIT.
_watermark = max(UNIX_EPOCH, cursor[0] - timedelta(seconds=1))
return cursor if len(rows) >= int(cfg.max_memories_per_pass) else None


def _resume_after(watermark: datetime) -> _Cursor:
"""Resume a partial scan unless reconciliation found an earlier second."""
try:
saved = json.loads(pass_context.resume_cursor() or "[]")
if isinstance(saved, list) and len(saved) == 4 and all(isinstance(v, str) for v in saved):
at, org_hash, namespace_hash, mid = saved
bumped = _utc(datetime.fromisoformat(at))
if bumped > UNIX_EPOCH and bumped - timedelta(seconds=1) <= watermark:
return bumped, org_hash, namespace_hash, mid
except ValueError:
pass # A truncated cursor restarts from destination reconciliation.
return UNIX_EPOCH, "", "", ""


# ---------------------------------------------------------------- acted
Expand Down Expand Up @@ -824,8 +859,7 @@ async def run_pass(
cfg: UsageFoldConfig | None = None,
ch: _CHClient | None = None,
) -> UsageFoldResult:
"""One fold pass: probe for new usage, derive the watermark on a cold
start, then fold batches until caught up or the time budget is spent."""
"""Reconcile destination counters, then fold batches within the time budget."""
global _watermark, _watermark_derived
cfg = cfg or UsageFoldConfig()
result = UsageFoldResult(watermark=_watermark)
Expand Down Expand Up @@ -857,25 +891,31 @@ def _done(outcome: dict[str, Any]) -> UsageFoldResult:
log.warning("usage fold: acted pass failed (non-fatal): %s", e)
pass_context.note_failure(e, "acted")

# Cheap no-op probe: skip the GROUP BY entirely when nothing new
# has landed since the last pass.
# Only an empty totals table can skip destination reconciliation: a
# stale version may have replaced folded counts without another event.
probe = await ch.query_one("SELECT max(bumped_at) AS hwm FROM loom.memory_usage_totals")
hwm = _utc((probe or {}).get("hwm"))
if hwm <= UNIX_EPOCH:
pass_context.set_cursor(_RESET_CURSOR)
return _done({"candidates": 0, "folded": 0, "skipped_no_memory": 0})

if not _watermark_derived and _watermark <= UNIX_EPOCH:
_watermark = await _derive_cold_start_watermark(ch, hwm)
_watermark_derived = True
log.info("usage fold: cold-start watermark derived as %s", _watermark.isoformat())

if hwm <= _watermark:
return _done({"candidates": 0, "folded": 0, "skipped_no_memory": 0})
resume = await _derive_watermark(ch, hwm)
_watermark = min(_watermark, resume) if _watermark_derived else resume
_watermark_derived = True

after = _resume_after(_watermark)
if after[0] > UNIX_EPOCH:
_watermark = min(_watermark, after[0] - timedelta(seconds=1))
pass_context.set_cursor(json.dumps((after[0].isoformat(), *after[1:])))
while True:
more = await _fold_batch(ch, cfg, result)
if not more:
pass_context.check_stop()
cursor = await _fold_batch(ch, cfg, result, after)
if cursor is None:
if not result.skipped_pending_mutations:
pass_context.set_cursor(_RESET_CURSOR)
break
after = cursor
pass_context.set_cursor(json.dumps((after[0].isoformat(), *after[1:])))
if time.monotonic() - started >= float(cfg.pass_time_budget_seconds):
log.info("usage fold: time budget spent, resuming next pass")
break
Expand Down
2 changes: 2 additions & 0 deletions tests/analyzer/cases/workers.py
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,8 @@ async def _worker_passes(ch: RecordingCH, c: dict[str, Any]) -> None:
last_activity=now,
)
await uf._fold_memory_rows(ch, [target], 10)
await uf._derive_watermark(ch, now)
await uf._fold_batch(ch, uf.UsageFoldConfig(), uf.UsageFoldResult(), (now, ORG, NS, "m1"))


async def _admin_readers(ch: RecordingCH, c: dict[str, Any]) -> None:
Expand Down
3 changes: 3 additions & 0 deletions tests/analyzer/shapes.txt
Original file line number Diff line number Diff line change
Expand Up @@ -678,6 +678,7 @@
516db52eddcc SELECT count() AS n FROM loom.session_brief_revisions WHERE org = {org:String} AND namespace = {ns:String} AND ?
51af520b7286 SELECT DISTINCT tier, qualified_name, symbol_id FROM loom.code_symbol_names WHERE org = {org:String} AND namespace = {ns:String} AND name IN {names:Array(String)} AND (qualified_name = {full:String} O
51f662fe04fe SELECT count() AS n FROM loom.learnings_queue WHERE org = {org:String} AND candidate_id IN {cids:Array(UUID)} AND status = ?
52083af6b301 SELECT count() AS n FROM system.mutations WHERE database = {db:String} AND table = {t:String} AND NOT is_done
52448e9cfd6b ALTER TABLE loom.memory_record_conflicts UPDATE tombstoned_at = {now:DateTime} WHERE (org = {org:String} AND tombstoned_at IS NULL AND namespace = {ns:String}) AND ({erasure_mutation_tag:String} = {er
524493debfe9 SELECT count() AS n FROM loom.datasets WHERE org = {org:String} AND namespace IN {ns:Array(String)}
525d9e3b3774 SELECT count() AS n FROM loom.working_memory FINAL WHERE org = {org:String} AND user_id = {uid:String} AND tombstoned_at IS NULL
Expand Down Expand Up @@ -1439,6 +1440,7 @@ a613795f2fe3 SELECT count() AS n FROM loom.operator_edges FINAL WHERE org = {org
a62720ec9473 SELECT org, namespace, agent, user_id, session_id, queue_id, payload, message_count, content_chars, status, attempts, lease_until, last_error, written, enqueued_at, updated_at, version, enqueued_by FR
a627d0337788 ALTER TABLE loom.memory_ask_precedents DELETE WHERE (org = {org:String} AND namespace IN {ns:Array(String)}) AND ({erasure_mutation_tag:String} = {erasure_mutation_tag:String}) SETTINGS mutations_sync
a63dc159ea03 SELECT count() AS n, argMax(role, ts) AS role, argMax(content, ts) AS content, argMax(run_id, ts) AS run_id, max(ts) AS last_ts, max(saved_at) AS last_saved_at, groupUniqArray(user_id) AS owners, coun
a64937812d3d SELECT org, namespace, toString(memory_id) AS memory_id, sum(recall_count) AS recall_total, sum(evidence_count) AS evidence_total, sum(used_count) AS used_total, sum(harm_count) AS harm_total, max(las
a661d7b26199 ALTER TABLE loom.memory_extraction_queue DELETE WHERE (org = {org:String} AND user_id = {uid:String}) AND ({erasure_mutation_tag:String} = {erasure_mutation_tag:String}) SETTINGS mutations_sync = ?
a662cd95ca59 SELECT memory_key, (status != ? OR tombstoned_at IS NOT NULL OR redacted_at IS NOT NULL) AS taken_out FROM loom.memory FINAL WHERE org = {org:String} AND namespace = {ns:String} AND agent = {agent:Str
a67dc6a1b98a SELECT path, start_line, end_line, content_hash FROM loom.code_chunks FINAL WHERE org = {org:String} AND namespace = {ns:String} AND path IN {paths:Array(String)} AND tombstoned_at IS NULL AND start_l
Expand Down Expand Up @@ -1747,6 +1749,7 @@ c853d7bbd170 SELECT org, agent, count() AS calls, countIf(if(consumer = ?, args[
c85b7e038508 SELECT namespace, kind FROM loom.namespaces FINAL WHERE org = {org:String}
c873c5cd3338 SELECT retention_days, dedup_threshold, hot_quota, conflict_policy FROM loom.namespace_types FINAL WHERE org = {org:String} AND namespace = {ns:String} AND memory_type = {mt:String} LIMIT ?
c87677674058 SELECT DISTINCT namespace FROM loom.search_misses WHERE org = {org:String} AND (has({spellings:Array(String)}, asker) OR has({spellings:Array(String)}, lowerUTF8(user_id)))
c8870c302e09 SELECT min(t.bumped) AS first_unfolded FROM ( SELECT org, namespace, memory_id, sum(recall_count) AS recall_total, sum(evidence_count) AS evidence_total, max(last_recalled_at) AS last_recalled, max(bu
c892414a0cf3 SELECT role, granted_by, status, pending_role, pending_since FROM loom.namespace_grants FINAL WHERE org = {org:String} AND namespace = {ns:String} AND subject = {sub:String} AND revoked_at IS NULL AND
c8ce735d214f SELECT count() AS n FROM loom.memory_embeddings FINAL WHERE org = {org:String} AND tombstoned_at IS NULL AND namespace IN {ns:Array(String)}
c8e4a254b959 SELECT count() AS n FROM loom.conversation_threads FINAL WHERE org = {org:String} AND tombstoned_at IS NULL AND user_id = {uid:String}
Expand Down
Loading
Loading