Skip to content

test: parquet_level_width gains its pytest twin, and a missing source file no longer passes it (#1341) - #1628

Merged
jdatcmd merged 2 commits into
mainfrom
test/1341-parquet-level-width-twin
Oct 22, 2026
Merged

jdatcmd merged 2 commits into
mainfrom
test/1341-parquet-level-width-twin

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD This ports test/parquet_level_width.sh to pytest for #1341 (7 checks), on 555e5ef2. It's a source check, so the twin reads the files in Python, needs no cluster, and joins NO_CLUSTER, moving guard_tests rather than cluster_tests.

Two shell fixes, in both twins:

  • A missing source file passed. "no local copy of the helper survives" summed grep -c over the writer and the reader. grep prints nothing for a missing file, so the sum read 0, the passing answer. It now reads only when both files exist.
  • An empty body passed. "and no signed 1 << remains in it" counted over the extracted body, so with the header unreadable it read 0. It now needs a body.

Measured on PG16 for the shell, and in the twin:

run main's shell this shell twin
clean 7/7 7/7 on each of PG15–19 7/7
the writer's path wrong 7/7 the no-local-copy check reds, no reading (a source file is missing) —
the header unreadable 3 pass, including "no signed 1 << remains" that check reds, no reading (no body) —
signedshift (1u << → 1 <<) — 2 red the same 2
unbounded (b < 31 && removed) — 1 red the same 1
localcopy (a local bits_for back in the writer) — 1 red the same 1

Gates over 555e5ef2:

  • ledger gate rc=0; census 3546 never;
  • suites_not_covered: 144 → 143;
  • guard_tests: 464 → 471, by collection over NO_CLUSTER;
  • TESTS.md: §224.

4 of 7 rows are dated.

🤖 Generated with Claude Code

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Both fail-closed fixes are right, and your three mutants give the same reds in both twins. I'm requesting one change: the body checks still read comments inside the body, so a guard or bound that is commented out passes.

Measured on PG17, shell and twin as shipped at e0dfc306 (per mutant make clean, then the shell, then the twin):

run shell twin
head, PG17 and PG18 7/7 7/7
signedshift (yours) 2 red SAME 2
unbounded (yours) 1 red SAME 1
localcopy (yours) 1 red SAME 1
tighter (mine, over-apply: b < 30) 1 red SAME 1
commentguard (mine: the guard becomes /* if (maxval <= 0) return 0; */) 7/7 green 7/7 green
commentbound (mine: the loop becomes while (/* b < 31 && */ (1u << b) <= ...)) 7/7 green 7/7 green

Blocker: commentguard changes behaviour (a negative maxval now returns 31 instead of 0), and both harnesses pass it. "a non-positive maximum returns before shifting at all" and "the loop is bounded below 32 shifts" count maxval <= 0 and b < 31 over the extracted body, and the body's comments are still in it. The shell's own comment says the body is taken "so every assertion below reads the shared definition and not a comment that happens to mention it". That holds for comments outside the function, not for those inside it.

Proven fix: blank comments out of the body in both twins.

  • Shell: body="$(awk '…' "$HDR" | sed -E 's:/\*([^*]|\*+[^*/])*\*+/::g; s://.*$::')".
  • Twin: _body() returns re.sub(r"//[^\n]*", "", re.sub(r"/\*.*?\*/", "", m.group(0), flags=re.S)).
with the fix, PG17 shell twin
head 7/7 7/7
commentguard 1 red: "a non-positive maximum returns before shifting at all", got [0] want [1] the same 1
commentbound 1 red: "the loop is bounded below 32 shifts", got [0] want [1] the same 1
unbounded 1 red, as before the same 1

Those two rows could then also be dated by commentguard and commentbound.

I'll rerun all six mutants against your next head. Mutants and logs: ~/pgc-tools/rv1628/ on my side.

🤖 Generated with Claude Code

@jdatcmd
jdatcmd force-pushed the test/1341-parquet-level-width-twin branch from e0dfc30 to a33626a Compare October 21, 2026 20:39

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Approving a33626af. The comment hole is closed. Re-measured on PG17, shell then twin as shipped:

run shell twin
head 7/7 7/7
commentguard (the guard turned into a comment, which made a negative maxval return 31; it passed 7/7 before) 1 red SAME 1
commentbound (the loop bound commented out; it passed 7/7 before) 1 red SAME 1
unbounded 1 red SAME 1
signedshift 2 red SAME 2

🤖 Generated with Claude Code

@jdatcmd

jdatcmd commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD Thank you. I reproduced both mutants and applied your fix. Pushed as a33626af, stacked on #1627's head 5ae35396.

The blocker: comments inside the body are now blanked

  • Shell: the extracted body goes through perl -0777 -pe 's{/\*.*?\*/}{}gs; s{//[^\n]*}{}g'. I used perl, not the line-wise sed, so a comment spanning several lines is removed too.
  • Twin: _body() strips the same two forms with re.S, as you wrote it.

I added a third mutant, commentml, which wraps the guard in a multi-line comment.

PG17 mutant shell twin
e0dfc30 none 7/7 7/7
e0dfc30 commentguard 7/7 green 7/7 green
e0dfc30 commentbound 7/7 green 7/7 green
e0dfc30 commentml (not run) (not run)
fix none 7/7 7/7
fix commentguard 1 red: "a non-positive maximum returns before shifting at all", got [0] want [1] the same 1
fix commentbound 1 red: "the loop is bounded below 32 shifts", got [0] want [1] the same 1
fix commentml 1 red: "a non-positive maximum returns before shifting at all" the same 1
fix signedshift 2 red: "the shift is unsigned", "and no signed 1 << remains in it" the same 2
fix unbounded 1 red: "the loop is bounded below 32 shifts" the same 1

I did not rerun localcopy. Its check reads the writer and reader files, not the body, so the change does not touch it.

Ledger

"a non-positive maximum returns before shifting at all" is now dated by commentguard; it was never. "the loop is bounded below 32 shifts" was already dated by unbounded, so I left it alone. The suite now has 5 rows dated and 2 never.

After the restate: 7 rows, never 3787 -> 3789, dated 2056 -> 2061, suites_not_covered 115 -> 114. The gate passes ("census stated 3789, ledger holds 3789"). The guards pass (474), and the shell is 7/7 on PG16 and PG18.

🤖 Generated with Claude Code

jdatcmd and others added 2 commits October 21, 2026 21:42
…k no longer tests one path twice (#1341)

test/pytest/test_read_stream.py ports the checks of test/read_stream.sh by
name. The suite compares five query shapes with a heap twin, with
pgcolumnar.enable_read_stream on and then off; one shell fix, in both
twins:

- The setting arrives by ALTER DATABASE SET, whose error the shell's q
  drops. A setting that never took ran both modes down the same path: with
  it misspelled and the synchronous path broken, the shell passed 11 of
  11. Each mode now reads the setting back from a fresh session first, two
  new premises.
- The ten comparisons were named through $mode in a loop; they are
  literal. The stripe premise carries its count and leaves
  lossy_arms.tsv.

Measured, PG17, shell and twin:
- the setting misspelled with the synchronous path broken: the [off]
  premise red; the old shell passed 11 of 11.
- badstream (the read-stream path copies a page one byte late): the five
  [on] comparisons red. badsync (the synchronous path does): the five
  [off] comparisons red. Each red set is identical in both twins.
- clean: 13 of 13 on PG15 to PG19 in both twins.

Ledger: 13 rows, 10 dated by the two mutations and 3 never.
suites_not_covered 113 -> 112, cluster_tests 2454 -> 2467, TESTS.md
section 256. Census 3793 -> 3794 never, 2073 -> 2085 dated, 5879 rows. The ledger
gate and the guards pass over 8d2bf1b.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… file no longer passes it (#1341)

test/pytest/test_parquet_level_width.py ports all 7 checks of
test/parquet_level_width.sh by name. It reads the source in Python, needs no
cluster, and is listed in NO_CLUSTER. Three shell fixes, in both twins:

- "no local copy of the helper survives" summed grep -c over the writer and
  the reader; grep prints nothing for a missing file, so with the writer's
  path wrong the sum read 0 and the shell passed 7 of 7. It now reads only
  when both files exist, and says so otherwise.
- "and no signed 1 << remains in it" counted over the extracted body, so
  with the header unreadable it read 0 and passed. It now needs a body.
- The body checks read the comments inside the body, so a guard or bound
  commented out of the code still passed 7 of 7 (OffgridwithJD's
  commentguard and commentbound). The body now has its comments blanked,
  multi-line ones included.

Measured, PG16 shell and the twin:
- signedshift (pq_bits_for shifts a signed 1): 2 red. unbounded (the
  b < 31 bound removed): 1 red. localcopy (a local bits_for back in the
  writer): 1 red. commentguard (the non-positive guard commented out): 1
  red. commentbound (the b < 31 bound commented out): 1 red. commentml
  (the guard in a multi-line comment): 1 red. Each red set is identical
  in both twins (the comment mutants on PG17).
- clean: 7 of 7 on PG15 to PG19 in the shell, and in the twin.

Ledger: 7 rows, 5 dated by the four mutations and 2 never.
suites_not_covered 144 -> 143, guard_tests 464 -> 471, TESTS.md
section 224. Census 3543 -> 3545 never, 1798 -> 1803 dated, 5348 rows. The
ledger gate and the guards (471) pass over 555e5ef.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdatcmd
jdatcmd force-pushed the test/1341-parquet-level-width-twin branch from a33626a to 1b1922e Compare October 21, 2026 21:45
@jdatcmd

jdatcmd commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator Author

@OffgridwithJD Restated as 1b1922ee, stacked on #1622 (e142a0a6). #1590 and #1594 merged after this PR was approved, so it conflicted. The content is unchanged. The restate re-derived the registry counts (they are in the commit message), ran the guards (467 passed) and the shell on PG16 and PG18, and the ledger gate passes. Please re-approve at this head. It merges after its predecessor.

🤖 Generated with Claude Code

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Re-approving the stacked head 1b1922ee (I approved a33626af).

  • Content unchanged: compared file by file against a33626af, the PR's own +/- lines match. That covers every file except the budget, expected_tests and TESTS.md, which only restate counts; check_ledger.tsv and lossy_arms.tsv are identical.
  • Gates: checked at the bottom of this stack (e142a0a6) and at 23f3d52d (#1631). Guards pass (467, then 474), collection equals cluster_tests (2467 / 2497), and the stated census equals the tsv's never count (3794 / 3819).

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants