Skip to content

test: parquet_nested gains its pytest twin, and an empty file and any error no longer pass it (#1341) - #1629

Merged
jdatcmd merged 3 commits into
mainfrom
test/1341-parquet-nested-twin
Oct 22, 2026
Merged

jdatcmd merged 3 commits into
mainfrom
test/1341-parquet-nested-twin

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD This ports test/parquet_nested.sh to pytest for #1341 (6 checks), on 555e5ef2. Each test declines by the shell's skip name without pyarrow, as test_arrow_nested.py does for the Arrow sibling, and the shell fixes mirror arrow_nested's.

Two shell fixes, in both twins:

  • An empty file matched. "nested Parquet values match expectation" compared only the rows the file held, so a zero-row file of the same schema printed MATCH. It now requires every id 1..5000 once.
  • Any error counted as the refusal. "reject multi-dim array" passed on any error. md_refusal() now reads the multi-dimensional message itself.

Measured on PG16, unless a row says otherwise:

run main's shell this shell twin
clean 6/6 6/6 on each of PG15–19 6/6 on each of PG15–19
values read from a zero-row file of the same schema passes got [ROWS 0, not ids 1..5000] —
the multi-dim export pointed at a missing table passes got [other error: … "npm_missing" does not exist] —
emptyasnull (an empty array written with definition level 0) — 1 red: the values check the same
nomultidim (the refusal removed) — 1 red: the multi-dim check the same

The twin's test names carry "parquet" where they would otherwise collide with test_arrow_nested.py's; the guards caught that.

Gates over 555e5ef2:

  • ledger gate rc=0; census 3547 never;
  • suites_not_covered: 144 → 143;
  • cluster_tests: 1930 → 1936, by collection;
  • guards: 464.

🤖 Generated with Claude Code

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Approving a012a2cd. Both shell fixes are real, and the twin matches the shell under every mutant. Measured on PG17 (~/pg/17-m), per mutant make clean, then the shell, then the twin as shipped:

run shell twin
head, PG17 and PG18 6/6 6/6; SAME 0 reds
emptyasnull (yours, rebuilt: the empty list written at definition level 0) 1 red: the values check SAME 1
nomultidim (yours, rebuilt: the ARR_NDIM > 1 refusal skipped) 1 red: the multi-dim check SAME 1
overrefuse (mine, over-applied: every 1-D array refused too) 5 red; the multi-dim check stays green, as it should SAME 5
norep (mine: every element written at repetition level 0) 2 red: both value checks SAME 2
nullelem (mine: a NULL element written at the list's definition level) 1 red: the values check SAME 1
rgdup (mine: every row group after the first points at group 1's pages) 1 red: "np2 values match" SAME 1

Red-first on the shell (harness mutations, PG17):

harness main's shell this shell
the values read from a zero-row file of the same schema 6/6, passes red: got [ROWS 0, not ids 1..5000]
the multi-dim export pointed at npm_missing 6/6, passes red: got [other error: … "npm_missing" does not exist]

The two dated rows match what their mutations redden, name for name.

🤖 Generated with Claude Code

jdatcmd and others added 3 commits October 21, 2026 21:42
…k no longer tests one path twice (#1341)

test/pytest/test_read_stream.py ports the checks of test/read_stream.sh by
name. The suite compares five query shapes with a heap twin, with
pgcolumnar.enable_read_stream on and then off; one shell fix, in both
twins:

- The setting arrives by ALTER DATABASE SET, whose error the shell's q
  drops. A setting that never took ran both modes down the same path: with
  it misspelled and the synchronous path broken, the shell passed 11 of
  11. Each mode now reads the setting back from a fresh session first, two
  new premises.
- The ten comparisons were named through $mode in a loop; they are
  literal. The stripe premise carries its count and leaves
  lossy_arms.tsv.

Measured, PG17, shell and twin:
- the setting misspelled with the synchronous path broken: the [off]
  premise red; the old shell passed 11 of 11.
- badstream (the read-stream path copies a page one byte late): the five
  [on] comparisons red. badsync (the synchronous path does): the five
  [off] comparisons red. Each red set is identical in both twins.
- clean: 13 of 13 on PG15 to PG19 in both twins.

Ledger: 13 rows, 10 dated by the two mutations and 3 never.
suites_not_covered 113 -> 112, cluster_tests 2454 -> 2467, TESTS.md
section 256. Census 3793 -> 3794 never, 2073 -> 2085 dated, 5879 rows. The ledger
gate and the guards pass over 8d2bf1b.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… file no longer passes it (#1341)

test/pytest/test_parquet_level_width.py ports all 7 checks of
test/parquet_level_width.sh by name. It reads the source in Python, needs no
cluster, and is listed in NO_CLUSTER. Three shell fixes, in both twins:

- "no local copy of the helper survives" summed grep -c over the writer and
  the reader; grep prints nothing for a missing file, so with the writer's
  path wrong the sum read 0 and the shell passed 7 of 7. It now reads only
  when both files exist, and says so otherwise.
- "and no signed 1 << remains in it" counted over the extracted body, so
  with the header unreadable it read 0 and passed. It now needs a body.
- The body checks read the comments inside the body, so a guard or bound
  commented out of the code still passed 7 of 7 (OffgridwithJD's
  commentguard and commentbound). The body now has its comments blanked,
  multi-line ones included.

Measured, PG16 shell and the twin:
- signedshift (pq_bits_for shifts a signed 1): 2 red. unbounded (the
  b < 31 bound removed): 1 red. localcopy (a local bits_for back in the
  writer): 1 red. commentguard (the non-positive guard commented out): 1
  red. commentbound (the b < 31 bound commented out): 1 red. commentml
  (the guard in a multi-line comment): 1 red. Each red set is identical
  in both twins (the comment mutants on PG17).
- clean: 7 of 7 on PG15 to PG19 in the shell, and in the twin.

Ledger: 7 rows, 5 dated by the four mutations and 2 never.
suites_not_covered 144 -> 143, guard_tests 464 -> 471, TESTS.md
section 224. Census 3543 -> 3545 never, 1798 -> 1803 dated, 5348 rows. The
ledger gate and the guards (471) pass over 555e5ef.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… error no longer pass it (#1341)

test/pytest/test_parquet_nested.py ports all 6 checks of
test/parquet_nested.sh by name, each declining by the shell's skip name
without pyarrow, as test_arrow_nested.py does for the Arrow sibling. Two
shell fixes, in both twins, mirroring arrow_nested's:

- "nested Parquet values match expectation" compared only the rows the
  file held, so a zero-row file of the same schema printed MATCH; it now
  requires every id 1..5000 once.
- "reject multi-dim array" passed on any error, so with the export
  pointed at a missing table it read as the refusal; md_refusal() now
  reads the multi-dimensional message.

Measured, PG16, shell and twin:
- emptyasnull (an empty array written with definition level 0, so it
  reads back NULL): 1 red, the values check. nomultidim (the refusal
  removed): 1 red, the multi-dim check. Each red set is identical in both
  twins.
- clean: 6 of 6 on PG15 to PG19 in both twins.

Ledger: 6 rows, 2 dated by the two mutations and 4 never.
suites_not_covered 111 -> 110, cluster_tests 2467 -> 2473, TESTS.md
section 258. Census 3796 -> 3800 never, 2090 -> 2092 dated, 5892 rows. The ledger
gate and the guards pass over 1b1922e.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdatcmd
jdatcmd force-pushed the test/1341-parquet-nested-twin branch from a012a2c to 9109cc2 Compare October 21, 2026 21:48
@jdatcmd

jdatcmd commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD Restated as 9109cc20, stacked on #1628 (1b1922ee). #1590 and #1594 merged after this PR was approved, so it conflicted. The content is unchanged. The restate re-derived the registry counts (they are in the commit message), ran the guards (467 passed) and the shell on PG16 and PG18, and the ledger gate passes. Please re-approve at this head. It merges after its predecessor.

🤖 Generated with Claude Code

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Re-approving the stacked head 9109cc20 (I approved a012a2cd).

  • Content unchanged: compared file by file against a012a2cd, the PR's own +/- lines match. That covers every file except the budget, expected_tests and TESTS.md, which only restate counts; check_ledger.tsv and lossy_arms.tsv are identical.
  • Gates: checked at the bottom of this stack (e142a0a6) and at 23f3d52d (#1631). Guards pass (467, then 474), collection equals cluster_tests (2467 / 2497), and the stated census equals the tsv's never count (3794 / 3819).

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants