Repository navigation
Usage folds revisit boundary seconds and repair stale counters - #1040
Merged
Merged
Conversation
Retain the boundary second and reconcile destination counters on each pass. Persist scan progress and an explicit reset so late usage and stale memory versions converge without another event. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Persist fixed-width identity digests so supported names do not truncate scan progress. Compare the UInt32 targets written by the fold so saturated totals cannot keep rewinding time-limited scans. Read the isolated SQL fixture before selecting its key to observe patches despite automatic PREWHERE in the pinned engine. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes: #1030
Changelog category (leave one):
Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):
Usage folding includes late events within the watermark's second and repairs counters overwritten by a later memory version, even when no further usage arrives.
Why
A short batch advanced its exclusive watermark past events that became visible later in the same second. A version inserted after the fold could also carry older counters, while an unchanged usage timestamp prevented another pass from repairing them.
Every batch retains its boundary second. A deterministic cursor pages through timestamp ties and is saved when the pass exhausts its time budget. Fixed-width identity digests keep the cursor within its persistence limit for supported org and namespace sizes. Each nonempty pass compares absolute usage totals and last-recall time with live memory rows and rewinds to any earlier discrepancy, using the same UInt32 saturation as the counter writes. This provides eventual repair across replicas without treating a read followed by an insert as atomic.
Scope and risk
The change touches usage-fold scheduling, its statement corpus, regression tests, and architecture documentation. The counter writes remain absolute and preserve larger seeded counts through
greatest. Reconciliation adds an aggregate join per nonempty pass and re-reads the boundary second. A timestamp-only overlap cannot repair a stale version inserted behind the cursor, so destination comparison remains necessary.Compatibility, rollout and rollback
Tests
Merge status: all hosted checks except Cell acceptance pass. The acceptance check failed its work deadline, including its single diagnostic retry. Final acceptance remains failed.
Four regression cases failed before the fix: late recall, late evidence, stale-version repair, and a timestamp tie group larger than the batch cap.
Focused worker, acted-event, and real SQL tests: 69 passed.
make format-check,make lint, andmake typecheckpass.The regenerated analyzer corpus passes all 12 tests against a migrated database on the pinned server release.
Benchmark harness unit tests: 251 passed.
Recall, promotion, and authentication integration tests: 40 passed, 5 skipped.
A supervisor-persistence regression reproduced a completed scan retaining its old cursor; an explicit reset fixes the same-second repair after restart.
The full-row SQL assertion reads the isolated table before selecting its key: an automatic PREWHERE point read returned stale patch columns on the pinned engine. Every field remains compared, and the production reconciliation join observes the discrepancy.
Long-identity and UInt32-saturation regressions cover progress across time-limited supervised passes.
All six Linux CI unit shards, coverage and benchmark harness checks, integration shards, both PostgreSQL authority versions, isolation, mixed-version checks, and quickstarts pass. Cell acceptance failed the global work deadline during measurement. The same guard also failed during provenance verification in the separate harness repair PR Bound JSON encoder setup in Cell proof checks #1041. The same-commit diagnostic retry completed all API writes and delivery rows, then reached the same deadline during final-arm verification. It produced no final latency statistics or provenance proof. This check has not passed at merge; no deadline or acceptance limit changes here.
AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated (Codex): checked with regression tests, real SQL, and an independent reviewer.
I can explain every line of this diff and how it interacts with the rest of Loom.
Checklist