[jdatcmd]

@OffgridwithJD

What this does

Ports test/parallel_copy_dedup.sh to test/pytest/test_parallel_copy_dedup.py (#1341), all 17 checks by the shell's exact name, one test per check. It also fixes two holes in the shell, where a real defect left main's shell green, and drops two premise: prefixes under #1339.

What was fixed in the shell (and the twin)

1. The fingerprint could cover only part of the file. The changed file differed from the first in its first bytes, and the whole file (237,788 bytes) fitted in the coordinator's first 256 KiB read. A fingerprint of the first read alone still loaded the changed file, so main's shell passed 14 of 14. Now:
   • the changed file differs only in its last row;
   • 25,000 rows put that row past the first read, and a new premise checks it ("premise: the changed row lies past the first 256 KiB of the file");
   • a new check, "the records hold the SHA-256 of each loaded file's bytes", compares the table's load_fingerprint rows with the server's sha256(pg_read_binary_file(...)) of each file.
2. The NOTICE was never read. The header promises that a refused load "says what it skipped rather than returning a silent 0". With the NOTICE removed, main's shell passed 14 of 14. A new check reads it: "and says in a NOTICE that it skipped the load, rather than returning a silent 0".
3. Two premises that are not premises (pytest audit: 260 confirmed checks that cannot fail, prove something else, or carry the wrong label #1339). "without dedup a repeated load doubles the table, which is the defect" and "and each of those loads reported its rows" lose the premise: prefix. Dedup is pgcolumnar's own default, and forcing it on reddens both. Both are dated in the ledger.

Red names under each edit and mutant (PG17)

Edit or mutant	main's shell	this shell	twin
columnar_parallel_copy.c: fingerprint only the first read (break after the first pg_cryptohash_update)	0 red, 14/14	4 red: "a changed file at the same path is loaded, not skipped (#403 item 7)", "and reports the rows it loaded", "the skipped load left exactly one record for that table and file", "the records hold the SHA-256 of each loaded file's bytes"	same 4
columnar_parallel_copy.c: the skip's NOTICE never raised (if (false && skipped))	0 red, 14/14	1 red: "and says in a NOTICE that it skipped the load, rather than returning a silent 0"	same 1
columnar_parallel_copy.c: dedup always on (dedup = true)	4 red, both still premise:	4 red: "without dedup a repeated load doubles the table, which is the defect", "and each of those loads reported its rows", "asking for no dedup loads again even though the file is on record", "and reports those rows too"	same 4
columnar_parallel_copy.c: the refused load commits anyway	2 red	2 red: "the second load of the same file stores nothing (#403 item 7)", "a changed file at the same path is loaded, not skipped (#403 item 7)"	same 2
columnar_metadata.c: a fingerprint matches in any table	3 red	3 red: "the same file loads into a DIFFERENT table (the record is per table)", "and reports its rows", "asking for no dedup loads again even though the file is on record"	same 3
columnar_parallel_copy.c: the refused load reports the rows it discarded	1 red	1 red: "and reports 0 rather than claiming it loaded them"	same 1
Harness edit: ROWS=20000	not applicable	1 red: "premise: the changed row lies past the first 256 KiB of the file"	same 1

No mutant crashed the backend, so no red came from collateral damage.

Gates

• Clean runs: 17 of 17 on PG15, 16, 17, 18 and 19, in both the shell and the twin.
• Ledger: 17 rows, 13 dated by the 6 mutations and 4 never. The 4 never are the size premise, "the first load stores its rows normally (Opportunities from the ClickHouse VLDB 2024 paper, ranked against what we already have #403 item 7)", "and reports them", and "no prepared transaction leaked through any of it". Census: rows 6096 -> 6113, never 3853 -> 3857, dated 2243 -> 2256.
• suites_not_covered 105 -> 104; cluster_tests 2677 -> 2694 (by collection); TESTS.md section 265.
• The ledger gate passes on each of the five clean logs against origin/main; orphan-scan finds 0 orphans.
• Guards: 474 passed. test_harness_deps.py and test_capability_decline.py with --pg-config: 47 passed. harness_selftest.sh: 1248 of 1248.
• compare_to_bash.py: "every bash property is covered", 17 literal matches, 0 missing.
• The capability-import scan prints []. The twin catches only psycopg.Error.

In the twin, the prepared-transaction check counts only its own database's prepared transactions. Other suites in a pytest session prepare transactions of their own.

Re-checked by me before pushing (PG17, main 5a94757): m1-notice: main's shell 14/14, this shell 1 failed ("and says in a NOTICE that it skipped the load …: got [0] want [1]"), the twin the same test. m2-firstread: main's shell 14/14, this shell 4 failed, including "the records hold the SHA-256 of each loaded file's bytes", the twin the same 4 tests.

🤖 Generated with Claude Code

[OffgridwithJD]

@jdatcmd Approving dcdec1f5. Measured on PG17 with ~/pg/17-p, the shell then the twin as shipped. Main's shell comes from main at 293fb132.

run	main's shell	this shell	twin
head, clean	—	17/17 (and 17/17 on PG18)	17 passed (and on PG18)
firstread: the hash loop breaks after the first 256 KiB buffer	PASSED	4 red: changed file loaded, reports its rows, one record per table and file, the records hold the SHA-256	SAME 4
nonotice: if (false && skipped) before the NOTICE	PASSED	1 red: "and says in a NOTICE that it skipped the load …"	SAME 1
saltrun (mine, over-apply): the fingerprint XORed with a random byte, so no load is ever recognised	—	6 red: the second load stores nothing, reports 0, the NOTICE, one record, the SHA-256 records, the changed-file arm	SAME 6
harness: server stopped just before the SHA-256 check	—	that check and the prepared-xact check are both red, with the connection error as the reading, not a pass	—

• The two relabelled premises: I checked them in the diff and trust your dedup = true mutant's reds. I didn't re-run that one.
• Non-blocking: "premise: the changed row lies past the first 256 KiB of the file" can't be reddened by a product mutant: it reads pg_stat_file of a file the shell wrote, so the label is right. But it hard-codes 262144, which mirrors PCOPY_FINGERPRINT_BUFSZ. If that buffer grew to 512 KiB, the premise would stay green while the changed row fell inside the first read again, and firstread would no longer redden anything. Reading the #define from src/columnar_parallel_copy.c instead of the literal would close that.

🤖 Generated with Claude Code

[OffgridwithJD]

@jdatcmd Re-approving the restated head 4b8ba9a0 (I approved dcdec1f5). This PR's own changes to parallel_copy_dedup.sh, test_parallel_copy_dedup.py, check_ledger.tsv, test_compare_to_bash.py and unported_suites.txt are identical line for line; only the budget and counters moved. The earlier measurements stand.

🤖 Generated with Claude Code

[jdatcmd]

Restated as a stack: now on #1666 (it merges after it), registries regenerated over that head with the same tooling. The suite and twin are unchanged; the ledger gate passed (rc=0) and the guards passed (476). Head 4b8ba9a.

[jdatcmd]

Restated again, onto #1666's new head (main 9ab6a4c underneath), because the registries conflicted with main. The registries are regenerated over that head by the same tooling as before; the suite and twin are unchanged; the ledger gate and the guards pass (see the commit message). Re-approval needed. Head 7e8c10f.

[OffgridwithJD]

@jdatcmd Re-approved at 7e8c10f7, on #1666's eb4aa699. The diff outside the ledger, the budget, expected_tests and TESTS.md hashes the same as the approved 4b8ba9a0's (unported_suites.txt and the compare list included). The tsv is sorted, and its census (6967 / 4136 / 2831) matches the budget.

🤖 Generated with Claude Code