Skip to content

Usage folds revisit boundary seconds and repair stale counters - #1040

Merged
zlareb1 merged 2 commits into
mainfrom
fix/issue-1030
Nov 20, 2026
Merged

zlareb1 merged 2 commits into
mainfrom
fix/issue-1030

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Closes: #1030

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in a release)

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

Usage folding includes late events within the watermark's second and repairs counters overwritten by a later memory version, even when no further usage arrives.

Why

A short batch advanced its exclusive watermark past events that became visible later in the same second. A version inserted after the fold could also carry older counters, while an unchanged usage timestamp prevented another pass from repairing them.

Every batch retains its boundary second. A deterministic cursor pages through timestamp ties and is saved when the pass exhausts its time budget. Fixed-width identity digests keep the cursor within its persistence limit for supported org and namespace sizes. Each nonempty pass compares absolute usage totals and last-recall time with live memory rows and rewinds to any earlier discrepancy, using the same UInt32 saturation as the counter writes. This provides eventual repair across replicas without treating a read followed by an insert as atomic.

Scope and risk

The change touches usage-fold scheduling, its statement corpus, regression tests, and architecture documentation. The counter writes remain absolute and preserve larger seeded counts through greatest. Reconciliation adds an aggregate join per nonempty pass and re-reads the boundary second. A timestamp-only overlap cannot repair a stale version inserted behind the cursor, so destination comparison remains necessary.

Compatibility, rollout and rollback

  • Wire: no operations, fields, or defaults change.
  • Operator steps: none; no migration or backfill is required.
  • Rollback: reverting restores the earlier worker behavior. The existing worker-status table holds the cursor; no schema changes remain.

Tests

  • Merge status: all hosted checks except Cell acceptance pass. The acceptance check failed its work deadline, including its single diagnostic retry. Final acceptance remains failed.

  • Four regression cases failed before the fix: late recall, late evidence, stale-version repair, and a timestamp tie group larger than the batch cap.

  • Focused worker, acted-event, and real SQL tests: 69 passed.

  • make format-check, make lint, and make typecheck pass.

  • The regenerated analyzer corpus passes all 12 tests against a migrated database on the pinned server release.

  • Benchmark harness unit tests: 251 passed.

  • Recall, promotion, and authentication integration tests: 40 passed, 5 skipped.

  • A supervisor-persistence regression reproduced a completed scan retaining its old cursor; an explicit reset fixes the same-second repair after restart.

  • The full-row SQL assertion reads the isolated table before selecting its key: an automatic PREWHERE point read returned stale patch columns on the pinned engine. Every field remains compared, and the production reconciliation join observes the discrepancy.

  • Long-identity and UInt32-saturation regressions cover progress across time-limited supervised passes.

  • All six Linux CI unit shards, coverage and benchmark harness checks, integration shards, both PostgreSQL authority versions, isolation, mixed-version checks, and quickstarts pass. Cell acceptance failed the global work deadline during measurement. The same guard also failed during provenance verification in the separate harness repair PR Bound JSON encoder setup in Cell proof checks #1041. The same-commit diagnostic retry completed all API writes and delivery rows, then reached the same deadline during final-arm verification. It produced no final latency statistics or provenance proof. This check has not passed at merge; no deadline or acceptance limit changes here.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated (Codex): checked with regression tests, real SQL, and an independent reviewer.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • All required checks pass on the reviewed commit, including Cell acceptance.
  • The statement corpus is regenerated and passes its analyzer tests.
  • No migration, wire operation, setting, plugin or frontend behavior changes.
  • Architecture documentation describes progress, reconciliation and their limits.
  • Repository text contains no private deployment details.
  • Commits are signed off.

Retain the boundary second and reconcile destination counters on each pass.
Persist scan progress and an explicit reset so late usage and stale memory
versions converge without another event.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
loom-docs Ignored Ignored Preview Nov 20, 2026 1:02pm UTC

Request Review

Persist fixed-width identity digests so supported names do not truncate
scan progress. Compare the UInt32 targets written by the fold so saturated
totals cannot keep rewinding time-limited scans.

Read the isolated SQL fixture before selecting its key to observe patches
despite automatic PREWHERE in the pinned engine.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 merged commit 569d12e into main Nov 20, 2026
59 of 61 checks passed
@zlareb1
zlareb1 deleted the fix/issue-1030 branch November 20, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

usage_fold skips usage that shares its watermark's second

1 participant