Skip to content

test: ALTER TABLE forms that read every row see the live rows and only those (#1341) - #1418

Merged
jdatcmd merged 5 commits into
mainfrom
test/1341-alter-table-scans
Oct 15, 2026
Merged

jdatcmd merged 5 commits into
mainfrom
test/1341-alter-table-scans

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

Covers the #1341 core-am item "ALTER TABLE forms that scan existing rows through the AM". Test-only: no source change.

What it adds

A new suite, test/alter_table_scans.sh, and its pytest twin, test_alter_table_scans.py. A columnar table keeps a deleted row in its file until compaction, so every form that reads the existing rows must see the live rows and only those. Each form runs on a columnar table and on heap with the same rows (3000 ordinary rows, plus a NULL in v, a second id = 1, and a negative v), and each check states both outcomes, columnar|heap:

form with the violating row after that row is deleted
ALTER COLUMN v SET NOT NULL 23502|23502 ok|ok
ADD PRIMARY KEY (id) 23505|23505 ok|ok
ADD CONSTRAINT … CHECK (v > 0) NOT VALID, then VALIDATE CONSTRAINT ok|ok, then 23514|23514 ok|ok, recorded as validated

Two rewrites follow. ALTER COLUMN v TYPE bigint keeps 3000 rows with sum 4501500. ADD COLUMN z float8 DEFAULT random() gives every live row a value and brings back no deleted row. Finally, the primary key finds id = 1 once, through an index scan.

Evidence

  • Green: 15/15 on each of PG15–19, and the twin on each.
  • Red on PG17:
mutant what went red
every scan returns deleted rows (the reader's delete filter disabled) 8 checks. SET NOT NULL got 23502|ok after the delete; both rewrites got 23505|ok; row counts got 3002/4501499
the index build marks every row alive ADD PRIMARY KEY after the delete got 23505|ok; the index-plan arm, which needs that key

Two checks I first named "premise:" reddened under these mutants, so they are arms under #1236 and carry no prefix.

🤖 Generated with Claude Code

https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at 038eeea.

  • What it holds: SET NOT NULL, ADD PRIMARY KEY and VALIDATE CONSTRAINT are refused over a violating row and accepted once that row is deleted. ALTER COLUMN TYPE and ADD COLUMN with a volatile default keep the live rows and only those, and the primary key lookup works afterwards. Each check reads as columnar|heap.
  • Red (PG17), both mutants as stated:
    • The reader with if (false && deleted && !rs->emitDeleted) (.so 81544b33) gives 7 + 8 failed, for example the deleted rows are gone from both tables, which hold the same rows: got [3002/4501499|3000/4501500], SET NOT NULL succeeds once the NULL row is deleted, as on heap: got [23502|ok] and ALTER COLUMN TYPE rewrites the table, as on heap: got [23505|ok].
    • The index build with alive = true (.so 4c47e347) gives 13 + 2: ADD PRIMARY KEY succeeds once the duplicate is deleted, as on heap: got [23505|ok] and the lookup of id 1 is an index scan on the primary key: got [0|1]. The second follows from the first, since there's no primary key to plan on, and the ledger paragraph says so.
    • The twin fails at the first of each, with the same text.
  • Green: shell 15/15 and twin 1 passed on PG17 and PG18 with the code restored (.so b9726239 again). The ledger, 2789 / 2100 / 689, matches the tsv, the gate agrees against origin/main, all 15 names have rows, and cluster_tests 566 matches collection.

🤖 Generated with Claude Code

jdatcmd and others added 5 commits October 15, 2026 16:36
…1341 item 50)

Two finite float8s, or two finite reals, whose sum is infinite raise
"value out of range: overflow" (22003) in core's float8pl and float4pl.
The grouped node's accumulators carry the same check, and nothing tested
it: only avg's overflow had an arm, in the shell alone. native_groupagg
gains the sum arm in both twins, judged by SQLSTATE against core with the
node off, plus a check that the node fires. Removing either accumulator's
check reddens its own position.

native_groupagg enters the ledger seeded; suites_not_covered 234 -> 233.
cluster_tests +1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N
…#1407)

older snapshot still finds a recently deleted row by a new index. But it
emitted every deleted row, where heap's build skips a tuple dead to all.
So REINDEX, documented as what reclaims stale entries, reclaimed none:
9 rows in 10 deleted, and the rebuilt index kept its size where heap's
shrank to a tenth.

The build now also reads each row group's delete marks under a horizon
snapshot (xmin = xmax = the oldest non-removable xid, curcid 0), which
sees exactly the transactions committed before every running snapshot
began, and leaves out a row deleted there. The snapshot is registered:
PG 18+ asserts it in the visibility check.

am_contracts gains the arm in both twins, red on main (kept:2260992->
2260992); with the horizon moved to the next xid, the older-snapshot arm
goes red instead. limitations.md says when REINDEX keeps an entry.
cluster_tests +1.

Fixes #1407.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N
…f sleeping

The arm from #1364 held a REPEATABLE READ snapshot, slept four seconds,
and assumed the DELETE and CREATE INDEX finished inside them. On a loaded
box they did not: the plan was taken before the index existed, and
'premise: the older snapshot's lookup is an index scan' read 0 on PG16.
A 5-second delay before CREATE INDEX reproduces it on PG17 every time.

Now a gate session holds an advisory lock. Session 1 takes its snapshot,
names itself, and queues for the lock. The script waits for the name,
deletes, builds the index, then ends the gate. With the same delay the
suite passes 28/28. A new premise in both twins says the older snapshot
counted all 10000 rows, so it predates the delete. The arm still goes red
when the index build skips deleted rows again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N
…y those (#1341)

A new suite, alter_table_scans, and its pytest twin. A columnar table
keeps a deleted row in its file until compaction, so each form must see
the live rows and only those. On a columnar table and on heap with the
same rows:

- SET NOT NULL, ADD PRIMARY KEY and VALIDATE CONSTRAINT are refused over
  a NULL, a duplicate and a breaking row (23502, 23505, 23514), and each
  succeeds once that row is deleted;
- ALTER COLUMN TYPE and ADD COLUMN with a volatile default rewrite the
  table, keep the live rows and leave the deleted ones gone;
- afterwards the primary key finds id 1 once, through the index.

15/15 on each of PG15-19, and the twin on each. Ten checks are dated on
PG17 by two mutants: every scan returning deleted rows, and the index
build marking every row alive. cluster_tests 565 -> 566 by collection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N
pgc_conn is a write-watching wrapper, so setting .autocommit on it sets
an attribute on the wrapper and never reaches the connection (found
while writing #1419). The line here set it to the value it already had,
so it did nothing, but it read as if it mattered. The twin still passes
on PG17.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N
@jdatcmd
jdatcmd force-pushed the test/1341-alter-table-scans branch from 038eeea to de6e61d Compare October 15, 2026 16:40

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approved at de6e61d, stacked on #1417.

  • Same patch: fc6a07a matches what I approved at 038eeea, outside the count files and ledger rows included. de6e61d removes pgc_conn.autocommit = True, which is now required: since #1432 the proxy raises on that assignment instead of silently keeping it. Net, the twin is the code I approved.
  • Census: 2984 / 2278 / 706 from the tsv, which matches the paragraph's 2969 + 15, 2273 + 5 and 696 + 10.
  • CI: 13 checks green and 2 still running at the time of review. The pytest job is the one that would catch the proxy change.

🤖 Generated with Claude Code

@jdatcmd
jdatcmd merged commit 1f7b5da into main Oct 15, 2026
15 checks passed
This was referenced Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants