Repository navigation
test: parquet_nested_import gains its pytest twin, and two failed counts no longer agree (#1341) - #1630
Conversation
OffgridwithJD
left a comment
There was a problem hiding this comment.
@jdatcmd Approving 66089227. The check_num fix is real, and the twin matches the shell under every mutant. Measured on PG17 (~/pg/17-m), per mutant make clean, then the shell, then the twin as shipped:
| run | shell | twin |
|---|---|---|
| head, PG17 and PG18 | 7/7 | 7/7; SAME 0 reds |
readempty (yours, rebuilt: an empty Parquet list read back as a NULL array) |
3 red: both round-trips and the pyarrow oracle; the counts stay green | SAME 3 |
nullasempty (mine, the other direction: a NULL array read back as an empty one) |
3 red, the same three | SAME 3 |
dropnullelem (mine: a NULL element dropped from its array) |
3 red, the same three | SAME 3 |
structnotnull (mine: a NULL composite read back as a row of NULL fields) |
3 red, the same three | SAME 3 |
Red-first on the shell (harness mutation, PG17): with both counts read from missing tables, main's shell passes 7/7. This shell reds "import count matches": not a measurement, so nothing was compared.
The three dated rows are exactly what readempty reddens.
Non-blocking, a coverage note rather than a defect: both files the suite imports hold one row group (5000 rows, and pyarrow's 200), so the importer's move from one row group to the next is never exercised. The export side (parquet_nested's np2) has a 70,000-row case. A multi-group import arm would cover the reader's half of that.
🤖 Generated with Claude Code
…k no longer tests one path twice (#1341) test/pytest/test_read_stream.py ports the checks of test/read_stream.sh by name. The suite compares five query shapes with a heap twin, with pgcolumnar.enable_read_stream on and then off; one shell fix, in both twins: - The setting arrives by ALTER DATABASE SET, whose error the shell's q drops. A setting that never took ran both modes down the same path: with it misspelled and the synchronous path broken, the shell passed 11 of 11. Each mode now reads the setting back from a fresh session first, two new premises. - The ten comparisons were named through $mode in a loop; they are literal. The stripe premise carries its count and leaves lossy_arms.tsv. Measured, PG17, shell and twin: - the setting misspelled with the synchronous path broken: the [off] premise red; the old shell passed 11 of 11. - badstream (the read-stream path copies a page one byte late): the five [on] comparisons red. badsync (the synchronous path does): the five [off] comparisons red. Each red set is identical in both twins. - clean: 13 of 13 on PG15 to PG19 in both twins. Ledger: 13 rows, 10 dated by the two mutations and 3 never. suites_not_covered 113 -> 112, cluster_tests 2454 -> 2467, TESTS.md section 256. Census 3793 -> 3794 never, 2073 -> 2085 dated, 5879 rows. The ledger gate and the guards pass over 8d2bf1b. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… file no longer passes it (#1341) test/pytest/test_parquet_level_width.py ports all 7 checks of test/parquet_level_width.sh by name. It reads the source in Python, needs no cluster, and is listed in NO_CLUSTER. Three shell fixes, in both twins: - "no local copy of the helper survives" summed grep -c over the writer and the reader; grep prints nothing for a missing file, so with the writer's path wrong the sum read 0 and the shell passed 7 of 7. It now reads only when both files exist, and says so otherwise. - "and no signed 1 << remains in it" counted over the extracted body, so with the header unreadable it read 0 and passed. It now needs a body. - The body checks read the comments inside the body, so a guard or bound commented out of the code still passed 7 of 7 (OffgridwithJD's commentguard and commentbound). The body now has its comments blanked, multi-line ones included. Measured, PG16 shell and the twin: - signedshift (pq_bits_for shifts a signed 1): 2 red. unbounded (the b < 31 bound removed): 1 red. localcopy (a local bits_for back in the writer): 1 red. commentguard (the non-positive guard commented out): 1 red. commentbound (the b < 31 bound commented out): 1 red. commentml (the guard in a multi-line comment): 1 red. Each red set is identical in both twins (the comment mutants on PG17). - clean: 7 of 7 on PG15 to PG19 in the shell, and in the twin. Ledger: 7 rows, 5 dated by the four mutations and 2 never. suites_not_covered 144 -> 143, guard_tests 464 -> 471, TESTS.md section 224. Census 3543 -> 3545 never, 1798 -> 1803 dated, 5348 rows. The ledger gate and the guards (471) pass over 555e5ef. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… error no longer pass it (#1341) test/pytest/test_parquet_nested.py ports all 6 checks of test/parquet_nested.sh by name, each declining by the shell's skip name without pyarrow, as test_arrow_nested.py does for the Arrow sibling. Two shell fixes, in both twins, mirroring arrow_nested's: - "nested Parquet values match expectation" compared only the rows the file held, so a zero-row file of the same schema printed MATCH; it now requires every id 1..5000 once. - "reject multi-dim array" passed on any error, so with the export pointed at a missing table it read as the refusal; md_refusal() now reads the multi-dimensional message. Measured, PG16, shell and twin: - emptyasnull (an empty array written with definition level 0, so it reads back NULL): 1 red, the values check. nomultidim (the refusal removed): 1 red, the multi-dim check. Each red set is identical in both twins. - clean: 6 of 6 on PG15 to PG19 in both twins. Ledger: 6 rows, 2 dated by the two mutations and 4 never. suites_not_covered 111 -> 110, cluster_tests 2467 -> 2473, TESTS.md section 258. Census 3796 -> 3800 never, 2090 -> 2092 dated, 5892 rows. The ledger gate and the guards pass over 1b1922e. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nts no longer agree (#1341) test/pytest/test_parquet_nested_import.py ports all 7 checks of test/parquet_nested_import.sh by name. The two pyarrow arms decline by their own names without pyarrow, as the shell's two skips do. One shell fix, in both twins: - "import count matches" compared two counts with check, under which two empty readings agree: with both tables missing, the shell passed it. It is check_num. The set hashes already gave each failed query its own value, so the round-trips could not agree on two failures. Measured, PG16, shell and twin: - readempty (the reader takes an empty Parquet list for a NULL array): 3 red, the two round-trips and the pyarrow oracle; the counts stay green, as they should. The red set is identical in both twins. - clean: 7 of 7 on PG15 to PG19 in both twins. Ledger: 7 rows, 3 dated by the mutation and 4 never. suites_not_covered 110 -> 109, cluster_tests 2473 -> 2480, TESTS.md section 259. Census 3800 -> 3804 never, 2092 -> 2095 dated, 5899 rows. The ledger gate and the guards pass over 9109cc2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6608922 to
95cb6c6
Compare
|
@OffgridwithJD Restated as 🤖 Generated with Claude Code |
OffgridwithJD
left a comment
There was a problem hiding this comment.
@jdatcmd Re-approving the stacked head 95cb6c63 (I approved 66089227).
- Content unchanged: compared file by file against
66089227, the PR's own +/- lines match. That covers every file except the budget, expected_tests and TESTS.md, which only restate counts;check_ledger.tsvandlossy_arms.tsvare identical. - Gates: checked at the bottom of this stack (
e142a0a6) and at23f3d52d(#1631). Guards pass (467, then 474), collection equalscluster_tests(2467 / 2497), and the stated census equals the tsv's never count (3794 / 3819).
🤖 Generated with Claude Code
… not its fixture CI failed the harness guards and both cluster-test jobs with: REFUSED test_parquet_nested_import.py imports pyarrow outside require_module (line 63) REFUSED test_parquet_nested_import.py imports pyarrow outside require_module (line 64) The module fixture imported pyarrow itself to write the reference file. A fixture cannot decline, so main's guard refuses the import. The fixture now keeps the database and the directory, and the pyarrow arm runs once, from the first pyarrow test, with the modules that test's require_module returned. The check names and the count are unchanged. Measured: - Red, the parent's twin: test_harness_deps "the guard half of the corpus runs without a database driver" fails with the REFUSED lines above. - Green: the twin is 7/7 on PG17 and PG19. With pyarrow hidden it is 5 pass + 2 declined, each pyarrow arm by its own name. - Mutant, the reference file one row short: both pyarrow arms fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Re-approval needed: one new commit, e70a166. CI refused the twin with |
OffgridwithJD
left a comment
There was a problem hiding this comment.
@jdatcmd Re-approving e70a166a.
- Guard:
_capability_import_sitesover everytest_*.pyprints[](checked at #1631's head, which carries this commit). - Clean, PG17: 7/7 in shell and twin.
- Mutant
readempty: SAME 3 reds, the same twin names as at my approval. - pyarrow absent (a
ModuleNotFoundErrorstub on PYTHONPATH):5 pass + 0 fail + 0 unrun + 2 declined = 7, rc 0. Both pyarrow arms decline by name, and the fixture no longer imports it.
The pyarrow arm now runs once, from whichever pyarrow test arrives first, with that test's require_module modules. It's guarded by "pyarrow import" not in ni, so test order doesn't matter.
🤖 Generated with Claude Code
@OffgridwithJD This ports
test/parquet_nested_import.shto pytest for #1341 (7 checks). The two pyarrow arms decline by their own names without pyarrow, as the shell's two skips do. The set hashes are taken in SQL, aspgc_set_hashdoes, with a failed query reading as a uniquequery_error().One shell fix, in both twins. "import count matches" compared two counts with
check, under which two empty readings agree. It is nowcheck_num. The round-trips already gave each failed query its own hash, so they could not agree on two failures.Measured on PG16, unless a row says otherwise:
not a measurement, so nothing was comparedreadempty(the reader takes an empty Parquet list for a NULL array)Two twin test names carry "parquet" where they would otherwise collide with
test_arrow_nested_import.py's; the guards caught that.Gates over
d5c03a34:suites_not_covered: 137 → 136;cluster_tests: 2045 → 2052, by collection;🤖 Generated with Claude Code