Skip to content

Postgres stores memory rows without default fields - #459

Merged
zlareb1 merged 16 commits into
mainfrom
feat/pg-sparse-rows
Nov 21, 2026
Merged

zlareb1 merged 16 commits into
mainfrom
feat/pg-sparse-rows

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Related: #455, #1076, #1077

Changelog category

  • Improvement

Changelog entry

The Postgres authority stores memory heads and revision payloads without fields at their default values, reducing repeated JSON fields while preserving the rows returned to callers.

Why

Most memory fields hold their defaults. Repeating their names and values in every stored JSON row costs space. Sparse-row readers shipped in v1.2.0 and v1.3.0, so the writer can omit those fields in a later release.

Scope and risk

commit_versions builds the stored form before taking transaction locks and uses it for head and revision JSON. Full encoded rows still supply identity, admission, audit metadata, delivery targets and returned rows. Only fields in FIRST_VERSION_DEFAULTS may be omitted. Identity, content, provenance, version, hashes, required timestamps, memory type and confidence stay stored.

The Cell verifier decodes both stored rows before comparing them. Its value, vector, audit, delivery and failure checks remain. Existing usage and redaction updates may store explicit defaults; readers accept either form. The shared replicated-erasure fixture repairs are already on main.

Compatibility, rollout and rollback

  • Wire and operator steps: none.
  • Deploy beside builds that read sparse rows; released v1.2.0 and v1.3.0 contain those readers.
  • A sparse-capable released build reads existing rows after rollback. Reverting the writer requires no rewrite.
  • No table, migration, setting, worker or generated-file change is needed.

Tests

  • All five hosted workflows pass on the published head: CI, Docker smoke, CodeQL, plugin scanner and docs. The rollup has 54 successful checks and two expected publication skips. This includes Postgres 16/18 authority, all integration shards, isolation, mixed-version, unit coverage and Cell acceptance.
  • The authority integration checks actual sparse head/revision storage, full/sparse point-read equality, list/search/bulk reads, revisions, cross-session defaults, usage and erasure export.
  • The verifier regression runs both representations through all original fault cases and assertions. Earlier focused validation produced two sparse failures before the repair and 32 passing cases after it; six related authority cases also passed.
  • Scoped Ruff lint/format and the public prose check pass. Local composition test attempts were refused before execution by the resource guard and remain unrun; final-head behavior is qualified by hosted CI.

AI assistance in the code

  • AI-generated: Codex wrote the sparse writer and verifier regression. Independent source reviews checked the full change, published description, released-reader compatibility and current-main composition.
  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Required hosted format, lint, type, unit, coverage and integration gates pass on the published head.
  • No generated files or migrations change.
  • No plugin or frontend behavior changes.
  • Architecture documentation describes sparse storage and in-place updates.
  • Public text contains no deployment details.
  • Commits are signed off.

zlareb1 and others added 10 commits November 9, 2026 22:07
…loom.memory default

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…, and unit tests pin falsy values and the fields SQL reads as stored

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…the pin covers the export's confidence, and the module docstring reads as the row FINAL serves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…-stored pin names the session it covers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… the pin's docstring wraps evenly

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s docstring names its Python readers as readers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…its on them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…a comment on where

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the sparse writer on the current authority and test paths while preserving the released readers and current storage documentation.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 20, 2026 11:23pm UTC

Request Review

@zlareb1 zlareb1 changed the title Under Postgres, a commit stores each row without the fields at their loom.memory default Postgres stores memory rows without default fields Nov 20, 2026
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 marked this pull request as ready for review November 20, 2026 20:33
Compare full logical heads and revisions through the existing decoder so omitted defaults retain the same verification and fault checks.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Fenced memory inserts require a lease table in the selected schema. Give each fixture database its own Keeper root while preserving the replicated content checks.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 96be03c)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

# Conflicts:
#	tests/integration/test_erasure_replicated_ch.py
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit f6cab47)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 merged commit 4c9bf98 into main Nov 21, 2026
56 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant