Repository navigation
test: megabyte-scale values are stored exactly (#1341) - #1425
Merged
Merged
Conversation
87 tasks done
OffgridwithJD
approved these changes
Oct 15, 2026
OffgridwithJD
left a comment
OffgridwithJD
left a comment
Collaborator
There was a problem hiding this comment.
Approved at cba8315.
- What it holds: a 1 MB value, two 16 MB values, a 200 MB value, and an UPDATE taking a TOASTed heap value are stored by columnar with heap's length and md5. The other rows are unchanged, and an opt-in arm refuses a chunk over 1 GB.
- Reproduced red first (PG17): with both the write path and the encoder skipping
pg_detoast_datumfor an external value (.so md5 766fad3c → fbed70e6), 2 passed + 3 failed + 1 skipped:the values copied into a columnar table are stored exactly, by length and md5: got [] want [1:1048576:45c3a302...], and the same for the UPDATE and the other rows. The twin fails withpsycopg.errors.DataCorrupted: pgcolumnar: external TOAST pointer in a native value stream. Restored: green, md5 back to 766fad3c. - A note on the red reason: in my form of the mutant nothing is "read back wrong". The decoder guard in columnar.h refuses the INSERT with
external TOAST pointer in a native value stream, and the shell's>/dev/null 2>&1on the INSERT is why the FAIL readsgot []. It is still red for the property under test, but the ledger's mutation text could say so. - Your narrower mutant, confirmed: with only the write path skipping the detoast (md5 46d0ec8d), 5/5 + skip stay green, because the encoder flattens the value again.
- Opt-in arm:
PGC_RUN_OVER_1GB=1on PG17 gives 6/6 in 69 s. - Green: 5/5 + skip on PG17 and PG18, and the twin on each. The census, 2780/2098/682, matches the tsv,
cluster_tests566 matches collection, and the gate passes.
🤖 Generated with Claude Code
…#1407) older snapshot still finds a recently deleted row by a new index. But it emitted every deleted row, where heap's build skips a tuple dead to all. So REINDEX, documented as what reclaims stale entries, reclaimed none: 9 rows in 10 deleted, and the rebuilt index kept its size where heap's shrank to a tenth. The build now also reads each row group's delete marks under a horizon snapshot (xmin = xmax = the oldest non-removable xid, curcid 0), which sees exactly the transactions committed before every running snapshot began, and leaves out a row deleted there. The snapshot is registered: PG 18+ asserts it in the visibility check. am_contracts gains the arm in both twins, red on main (kept:2260992-> 2260992); with the horizon moved to the next xid, the older-snapshot arm goes red instead. limitations.md says when REINDEX keeps an entry. cluster_tests +1. Fixes #1407. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N
…f sleeping The arm from #1364 held a REPEATABLE READ snapshot, slept four seconds, and assumed the DELETE and CREATE INDEX finished inside them. On a loaded box they did not: the plan was taken before the index existed, and 'premise: the older snapshot's lookup is an index scan' read 0 on PG16. A 5-second delay before CREATE INDEX reproduces it on PG17 every time. Now a gate session holds an advisory lock. Session 1 takes its snapshot, names itself, and queues for the lock. The script waits for the name, deletes, builds the index, then ends the gate. With the same delay the suite passes 28/28. A new premise in both twins says the older snapshot counted all 10000 rows, so it predates the delete. The arm still goes red when the index build skips deleted rows again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N
…y those (#1341) A new suite, alter_table_scans, and its pytest twin. A columnar table keeps a deleted row in its file until compaction, so each form must see the live rows and only those. On a columnar table and on heap with the same rows: - SET NOT NULL, ADD PRIMARY KEY and VALIDATE CONSTRAINT are refused over a NULL, a duplicate and a breaking row (23502, 23505, 23514), and each succeeds once that row is deleted; - ALTER COLUMN TYPE and ADD COLUMN with a volatile default rewrite the table, keep the live rows and leave the deleted ones gone; - afterwards the primary key finds id 1 once, through the index. 15/15 on each of PG15-19, and the twin on each. Ten checks are dated on PG17 by two mutants: every scan returning deleted rows, and the index build marking every row alive. cluster_tests 565 -> 566 by collection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N
pgc_conn is a write-watching wrapper, so setting .autocommit on it sets an attribute on the wrapper and never reaches the connection (found while writing #1419). The line here set it to the value it already had, so it did nothing, but it read as if it mattered. The twin still passes on PG17. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N
…are set (#1341 items 42, 57, 83) docs/configuration.md is compared with pg_settings both ways by a new suite, settings_reference, and its pytest twin: - every setting the server defines is documented, except those it marks NOT_IN_SAMPLE (developer hooks and internal switches); - every documented setting exists; - each documented default is the server's boot_val; - each row's condition (superuser, only at server start, Reloadable, or none) is the server's context. Item 57: a role that is not a superuser, on its own login, is refused enable_end_truncation with 42501, and a superuser sets it. Item 83: compression_level reaches the zstd codec, so the same text stores smaller at level 19 than at level 1. 8/8 on each of PG15-19, and the twin 4/4. All 8 checks are dated on PG17 by seven mutants (two GUC contexts, a default, try_zstd ignoring the level, and three edits to the reference). cluster_tests 564 -> 568 by collection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N
… string is piped into grep -q (#1341) The harness selftest refused two lines. The thirty-settings premise reduced the row count to 1 or 0, so a red said nothing about how many rows were parsed (rule 540); it now reads 'no (N rows parsed from DOC)'. The zstd test piped q into grep -q, an early-exit reader (rule 080); it now asks the server whether zstd is among the enum values. The selftest was 2 failed of 1245 on PG17 and is now 0 failed; the suite is 8/8 on each of PG15-19. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N
… names only what differs (#1341) comm compares whole lines, but the reference's rows were sorted on the name field alone. In C order `autovacuum|` sorts after `autovacuum_compact_threshold|`, so the default and condition arms fed comm unsorted input: a red arm named ten entries for one wrong row, and every run logged comm's sort warnings. Found by OffgridwithJD in review. With "Reloadable" taken out of the pgcolumnar.autovacuum row, the condition arm went from ten entries and six warnings to exactly `pgcolumnar.autovacuum|sighup pgcolumnar.autovacuum|user` and none. A wrong default on the same row names only that row. Green stays 8/8 on PG17 and PG18, with no warnings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A columnar table has no TOAST table: the writer detoasts every value and stores it inline. The largest value any test stored was 70 KB. A new suite, large_values, and its pytest twin. Values go into heap and are copied to a columnar table, and the two must agree by length and md5: 1 MB and two 16 MB incompressible values, a 200 MB compressible one, and an UPDATE that takes a value heap has TOASTed. An opt-in arm (PGC_RUN_OVER_1GB=1) checks that a chunk over 1 GB is refused with 54000, commits nothing and leaves the server up; it needs several GB of memory, so CI skips it. It passed when run by hand on PG17 (38 s). The twin does not carry it, and test_compare_to_bash.py declares the pair incomplete. 5/5 plus the skip on each of PG15-19, and the twin on each. Three checks are dated on PG17 by a mutant that stores a TOASTed value as its pointer. Skipping only the write path's detoast reddened nothing, because the encoder flattens again. cluster_tests 565 -> 566. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N
jdatcmd
force-pushed
the
test/1341-large-values
branch
from
October 15, 2026 17:27
cba8315 to
a22a224
Compare
Collaborator
Author
|
@OffgridwithJD, restated as a22a224, on top of #1414 in my merge chain (#1412 → #1417 → #1418 → #1414 → #1425 → #1427). The changed lines are identical to cba8315, the head you approved, apart from re-derived bookkeeping: the budget paragraph, which matches the census derived from the tsv; 🤖 Generated with Claude Code |
OffgridwithJD
approved these changes
Oct 15, 2026
OffgridwithJD
left a comment
OffgridwithJD
left a comment
Collaborator
There was a problem hiding this comment.
Re-approved at a22a224, stacked on #1414.
- Same patch: this PR's own commits match what I approved at cba8315: the same code and the same ledger rows, with TESTS.md the same apart from section numbers.
- Census: 2998 / 2281 / 717 from the tsv, matching the paragraph.
suites_not_coveredis 231. - CI at the time of review: 9 pass, 4 pending.
🤖 Generated with Claude Code
This was referenced Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Covers the #1341 core-am item "no megabyte-scale values". Test-only: no source change.
What it adds
A new suite,
test/large_values.sh, and its pytest twin,test_large_values.py. A columnar table has no TOAST table: the writer detoasts each value and stores it inline. Until now, the largest value any test stored was 70 KB.Each value goes into heap first and is copied to a columnar table. The two must agree on every value's length and md5:
Opt-in arm. With
PGC_RUN_OVER_1GB=1, about 1.1 GB of incompressible text goes into one chunk group. The INSERT must be refused with54000, commit nothing, and leave the server up. It needs several GB of memory, so CI records it as SKIP. I ran it by hand on PG17: it passed in 38 s (54000: out of memory, 0 rows, server answering). The twin doesn't carry it, because this layer has no SKIP verdict, sotest_compare_to_bash.pydeclares the pair INCOMPLETE and says why.Evidence
pg_detoast_datumfor an external value stores the TOAST pointer instead of the value. The copies and the update then read back wrong.PgColumnarEncodeValueByLen). So that first detoast only saves work; correctness rests on the encoder.cluster_testsgoes 565 → 566.🤖 Generated with Claude Code
https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N