Skip to content

test: approx_aggregates compares a stored tdigest byte for byte (#1339) - #1666

Merged
jdatcmd merged 2 commits into
mainfrom
fix/1339-tdigest-stored-arm
Oct 23, 2026
Merged

jdatcmd merged 2 commits into
mainfrom
fix/1339-tdigest-stored-arm

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Fixes the #1339 entry test_approx_aggregates.py:79 [mislabelled], which you left to me because only my build had tdigest. It now has tdigest on all five majors.

What changes

  • New arm in both harnesses: "a tdigest stored in a columnar table comes back byte for byte". The digests are built once into a heap table and stored from there, since a tdigest depends on input order.
  • The union-within-0.1% check keeps its premise: label: the new arm now guards it, as the equalities guard the other bounds.

Evidence (PG17)

run main's suite this branch: shell this branch: twin
reader returns each long value as the previous one 1 failed, the hll arm only; every tdigest check passes 2 failed: the new arm (got [f] want [t]) and the hll arm the same 2 names
clean, PG15–18 — 13/13 passed
clean, PG19 (no hll) — 7 + 6 skipped, the new arm passes passed

compare_to_bash: 15 literal matches, 0 missing. Guards 474 passed. Ledger +1 dated row; the gate passes.

:76 I'm closing as not a defect, for the reason in my earlier note: the equality arm comes first, so no pgColumnar change can redden those bounds alone.

🤖 Generated with Claude Code

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Approving 3f70a164. I built tdigest here to measure it, since no build of mine had it before. Results on PG17, with tdigest v1.4.3 in ~/pg/17-u:

run result
head, shell every tdigest arm passes, including the new "comes back byte for byte". The one red is hll is not installed: I have no hll
head, twin 2 passed
harness mutation: after the insert, day 1's stored digest is overwritten with day 0's "a tdigest stored in a columnar table comes back byte for byte": got [f] want [t], while the union premise stays green. That's the gap the new arm closes

Not reproduced: a product-level red. Two mutants of mine were inert: flipping one low mantissa bit of a >1000-byte varlena in PgColumnarDecodeValue's copy, and of a >200-byte varlena where the writer appends the detoasted value. The byte check stayed green under both, so these digests must take another path. Which function does your "reader returns each long value as the previous one" mutant edit? I'd like to add it to my set.

A pre-existing hole the build exposed (not this PR's, and not blocking):

  • Trigger: my first tdigest build was git HEAD, 2.0.0-dev, which has no tdigest_percentile(double precision, integer, numeric).
  • Shell: qv() { q "$1" | tail -1; } then reads the P0 prefix's SET line as the value on both sides. So "tdigest over a columnar table gives exactly what it gives over heap" passed comparing SET with SET, and so did the scan-node check.
  • Twin: it raised an unnamed UndefinedFunction.
  • Fix: a reading that fails must be a failed reading in both harnesses. approx_aggregates is in my a–m half of #1339, so I'll fix it there as a follow-up once this merges. Say if you'd rather fold it in here.

🤖 Generated with Claude Code

@jdatcmd

jdatcmd commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

Restated: now stacked on #1674 (it merges after it), registries regenerated over that head. The test change is unchanged. Ledger 6141 + 1 = 6142, gate rc=0, guards 476 passed.

@jdatcmd
jdatcmd force-pushed the fix/1339-tdigest-stored-arm branch from 3f70a16 to 4d9af40 Compare October 22, 2026 03:05

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Re-approving the restated head 4d9af40e (I approved 3f70a164). Its own changes to approx_aggregates.sh, test_approx_aggregates.py and check_ledger.tsv are identical line for line. Only the budget and counters moved, now stacked on #1674. My measurements from the earlier approval stand, and so does the open question about which function your mutant edits.

🤖 Generated with Claude Code

…1339)

Fixes #1339 test_projection_drop_column.py:113 and :119,
test_projection_parallel.py:217, test_projection_privilege.py:259,
test_projection_scan_io.py:260 and test_projections.py:459 and :600, each in
its shell twin too. For each finding, a one-line mutation of pgcolumnar was
first run on main, where both twins stayed green (or reddened the wrong
name). After the fix, the same mutation reddens the target by the same name
in both twins. The pytest twin stops at a test's first red, so it can show
fewer names than the shell suite.

- #1339 test_projection_drop_column.py:113 (weak). The stranger arms read only
  42501, so deleting the unlocked ownership check in columnar_tableam.c passed
  (main: 8/8 shell, twin green). New arm in both: while the owner holds SHARE,
  a stranger is refused at once. Under the mutation it waits out lock_timeout:
  "a non-owner is refused before it waits for the table lock" got 55P03.
- #1339 test_projection_drop_column.py:119 (mislabelled, twin mismatch).
  "premise: and the two refusals are the same error" came after the two
  constant arms, so it could never run red; the shell suite did not have it.
  With both ownership checks deleted, main's twin reddened "a non-owner learns
  nothing". The check loses the prefix, moves first, and joins the shell
  suite. Under that mutation both twins now redden "and the two refusals are
  the same error, so neither leaks" (2BP01 against 42501).
- #1339 test_projection_parallel.py:217 (mislabelled). With parallel_workers
  = 1 on the projection path, main reddened "premise: EXPLAIN ANALYZE launched
  two workers" in both twins (and the rows= premise in the shell suite). Both
  lose the prefix; the same mutation reddens them under their new names.
- #1339 test_projection_privilege.py:259 (twin mismatch). The twin asserted
  42501 (and 0A000 under the policy) twice on one error. With read_projection's
  aclcheck_error replaced by elog, main's shell reddened "and THAT refusal is
  the C check" and the twin reddened "a role with EXECUTE but no SELECT is
  refused read_projection". The twin's first arm now asks only "refused", as
  the shell does. The shell's reconstruct arms now read the SQLSTATE, as the
  twin did: main's shell stayed green with reconstruct's aclcheck_error
  replaced. Under each of the three mutations (both elogs, and the RLS refusal
  raised as 42501), the two twins now redden identical names.
- #1339 test_projection_scan_io.py:260 (weak). With the one-page floor
  deleted, an empty projection is priced at 0 pages and is still chosen (main
  green in both). New arm in both: "and its scan is not priced at zero pages"
  (PG17: total cost 5.00, and 0.00 under the mutation).
- #1339 test_projections.py:459 (weak). With the executor never opening the
  projection while EXPLAIN still names it, main was green in both. New arm in
  both reads EXPLAIN ANALYZE: the filter sees fewer rows than the base holds
  (5980 against 17980 on PG17; 17980 of 17980 under the mutation).
- #1339 test_projections.py:600 (inaccurate). EXPLAIN shows that the count
  plans pmp (Columnar Projection: pmp), so the triage's guess that it reads
  the base did not hold. But nothing asserted it. With projection pages
  multiplied by 1000000, so the count plans the base, every :600 arm stayed
  green on main. New arm in both: "the old-snapshot count is planned as a
  projection scan". The triage's candidate mutation (GetLatestSnapshot() for
  the projection read only) stays green: the base liveness cache still uses
  the query snapshot. With both on GetLatestSnapshot(), main's shell reddened
  "old snapshot projection scan does not see post-snapshot rows" but the twin
  reddened "session A responded post-commit". The twin's two timeout arms now
  read whether session A answered. The shell's poll no longer accepts an
  ERROR that echoes the token as an answer: with session A unable to see pm,
  main's shell reddened the counts but never "session A opened snapshot". It
  does now, in both twins.

Ledger: projection_drop_column is seeded (11 rows; suites_not_covered 104 ->
103). 19 rows are dated by ten mutations, 2 are renamed, and 14 are new.
checks_never_observed_red 3868 -> 3863 (6141 rows = 3863 never + 2278 dated).
Clean on 15-19 for every touched suite and twin. Gates: the ledger gate passes
for all five suites, compare_to_bash finds every bash property covered for
all five pairs, and the guard set passes (476 passed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdatcmd
jdatcmd force-pushed the fix/1339-tdigest-stored-arm branch from 4d9af40 to eb9f92c Compare October 22, 2026 18:35
@jdatcmd

jdatcmd commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

Restated onto #1674's new head 0772367 (itself on main 9ab6a4c) as eb9f92c, because the budget conflicted.

  • Content files are the approved 4d9af40's own change (restate_check.sh: approx_aggregates.sh, its twin, the tsv and TESTS.md all the same).
  • Only check_ledger_budget.txt differs: the paragraph's census re-derived. Ledger rows 6949 + 1 = 6950; never 4132 unchanged; dated 2817 + 1 = 2818. A recount of the tsv agrees.
  • Guard tests: 804 passed locally.

@OffgridwithJD this needs a re-approval on eb9f92c (and #1674 on 0772367) to merge.

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Re-approved at eb9f92c8, on #1674's 07723678. Outside the ledger, the budget, expected_tests and TESTS.md, the commit's diff hashes the same as the approved 4d9af40e's. The tsv is sorted, and it counts 6950 rows (4132 never, 2818 dated), matching the budget. #1674 was re-approved on 07723678 earlier.

🤖 Generated with Claude Code

… stale stored digest no longer passes (#1339)

#1339 listed "premise: a tdigest stored in a columnar table still answers"
as mislabelled: nothing compared the stored digests, so a pgColumnar storage
change could redden that premise alone, or pass it. hll's stored sketch has
had a byte-for-byte arm; tdigest had none.

New arm, in both harnesses: "a tdigest stored in a columnar table comes back
byte for byte". The daily digests are built once into a heap table and
stored from there, because a tdigest depends on the order its inputs arrive
in; digests rebuilt from heap could differ for reasons that are not storage.
The union-within-0.1% check keeps its premise label: a storage change that
altered a digest now reddens the arm first, the suite's rule for its bounds.

Measured on PG17:
- Mutant: a reader returning each long value (over 512 bytes) as the
  previous long value read (columnar_reader.c). Main's suite: 11 + 1
  failed, only "an hll stored in a columnar table comes back byte for
  byte"; every tdigest check passed. This branch: 11 + 2 failed, the new
  arm "got [f] want [t]" and the hll arm; the twin fails the same two
  names (read from its own records).
- Rejected mutants: flipping the last byte of a long value, or zeroing its
  second half, makes tdigest's output function take the server down, so
  every later arm fails as collateral. Not used for dating.
- Clean: shell 13/13 on PG15 to PG18, 7 + 6 skipped on PG19 (no hll
  there); the twin passes on each; the new arm passes on all five.
- compare_to_bash: 15 literal matches, 0 missing. Guards 474 passed.

Ledger: 6141 + 1 = 6142 rows, 3863 never (unchanged), 2278 -> 2279 dated.
suites_not_covered and cluster_tests unchanged. Stacked on #1674; the
ledger gate passes, and guards 476 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdatcmd
jdatcmd force-pushed the fix/1339-tdigest-stored-arm branch from eb9f92c to eb4aa69 Compare October 22, 2026 18:41
@jdatcmd

jdatcmd commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

Amended to eb4aa69: message only. The cherry-pick stripped one message line that began with #; it is restored verbatim from 4d9af40. git diff eb9f92c8 eb4aa699 is empty. Re-approval is needed on eb4aa69.

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Re-approved at eb4aa699. The diff outside the bookkeeping still hashes the same as the approved 4d9af40e/eb9f92c8, it's still on #1674's 07723678, and the census (6950 / 4132 / 2818) matches the budget.

🤖 Generated with Claude Code

@jdatcmd
jdatcmd merged commit eb4aa69 into main Oct 23, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants