Skip to content

Session digests recheck authoritative source state - #1071

Open
zlareb1 wants to merge 8 commits into
mainfrom
fix/digest-source-authority-fence
Open

zlareb1 wants to merge 8 commits into
mainfrom
fix/digest-source-authority-fence

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Related: #1087
Related: #1041
Related: #1056
Related: #1077

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in a release)

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

Session digests withhold facts whose authoritative sources changed before their search projections caught up.

Why

A source can be deleted, rewritten, rejected or restricted after its projection was admitted for search. Checking that projection alone can still serve a digest derived from the obsolete source. The digest leg reads each exact source key from Postgres after reading its projection, and requires matching version, content hash and serving metadata. Missing or unreadable authority never falls back to the projection.

Scope and risk

Native FixedString content hashes pass through the existing hash-to-text conversion before projection and authority state are compared. The authority hash must still be valid; changed, empty, padded-empty, nonhex and invalid UTF-8 hashes cannot validate a digest.

The read selects source metadata without copying source values. Every lookup includes org, namespace, agent, user and session. Sources without valid content hashes withhold their digests under Postgres authority. Digests without fact lineage remain transcript evidence. Redacted projected sources cannot validate digests. The analyzer corpus drives known-source checks for current reads and both replay bases. The journal-read inventory classifies this serving filter as sending no rows to ClickHouse. The person-query prerequisite scopes observations to the authorization decision task. The replicated erasure lease fixture is already present on main; its unit controls are now on main with the existing Keeper path. The default-comparison prerequisite computes the canonical mapping once. The cleanup fixture uses one private idle clock for the real owner and connection-close budgets. Deadline values, refusal controls and behavior assertions remain unchanged. No erasure fixture DDL change is included.

Compatibility, rollout and rollback

  • Wire: no operations, fields or defaults change.
  • Operator steps: none. No migration or backfill.
  • Rolling upgrades: readers use the existing schema; both builds can run together. A reader without this check can still serve obsolete digests.
  • Rollback: reverting removes the read fence and leaves stored rows unchanged.

Tests

The native-byte regression initially fails both matching bytes and bytearray cases while all five refusal controls pass. After hash normalization, the identical seven-case command passes. Existing text-hash, identity and lifecycle controls remain unchanged.

  • The initial regression command failed 16 cases and passed three controls before the source fix; the identical command passed all 19 afterward.
  • pytest tests/unit/test_summary_source_authority.py tests/unit/test_summary_sources.py tests/unit/test_summary_leg_links.py tests/unit/test_summary_leg_as_of.py tests/unit/test_session_summary_retrieval.py -q: 43 passed, including 22 authority cases.
  • The journal-read caller inventory failed before its required classification and passed the identical command afterward. The final focused run includes that inventory check alongside the 43 retrieval cases.
  • This exact head passes 54 direct retrieval, pool-cleanup and caller-inventory cases. The 69 prerequisite results are retained prior-source receipts; this composition preserves their source tree exactly.
  • Scoped Ruff checks and formatting cover the changed Python files. Documentation formatting and whitespace checks pass.
  • Analyzer shapes regenerated from builders; the summary case drives eight combinations without errors.
  • A Postgres integration regression reads the metadata query against an authoritative write and checks exact identity scope. It closes its in-process pool in a finally block, including helper and assertion failures. Three unit cases invoke that actual integration coroutine through fake I/O and verify cleanup on every path. The identical cleanup command fails all three before the fixture repair and passes all three afterward. It is left to hosted CI, together with full suites. The replicated erasure fixture is already on main; its hosted integration result remains required for this head.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated (an agent wrote most of the diff): Codex; checked source authority, tenant scope, sparse defaults, failure behavior and regression evidence.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Full required checks and integration suites pass in hosted CI.
  • Generated analyzer shapes come from the builders.
  • No migration, plugin or frontend change.
  • Architecture docs describe the source fence and authority failure behavior.
  • Repository text contains no operational incident details.
  • Commits are signed off.

The prerequisite cases comprise 47 default-comparison, 10 person-query observation, 3 erasure-fixture, 9 cleanup. Fresh independent whole review and complete hosted CI are required for this head.

@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 12:04am UTC

Request Review

@zlareb1
zlareb1 force-pushed the fix/digest-source-authority-fence branch 3 times, most recently from 0d742b6 to e563599 Compare November 20, 2026 23:46
A parked source publication can leave an obsolete digest searchable. Require exact-key authority metadata and content proof before a projected source can validate its digest.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Shared client schema reads run with an independent context. Record only decision-tagged statements while retaining unbounded decision reads as failures.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 57327d5)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Exercise each default-field refusal and JSON numeric distinction before consolidating repeated default comparisons.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 73ef63b)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit cb594d0)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep every default-field and JSON type check while avoiding a separate encoder setup for each field.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 0bfc712)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit a94d2f9)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the blocked close, deadline observers and loop timers on one private
clock. A controlled independent clock advance reproduces the elapsed-time
dependency while the original cleanup and refusal assertions stay intact.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 4b03db4)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 force-pushed the fix/digest-source-authority-fence branch from e563599 to f9ba955 Compare November 21, 2026 00:02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant