Skip to content

test: nineteen #1339 entries in the harness self-tests, each measured red-first (#1339) - #1664

Merged
jdatcmd merged 1 commit into
mainfrom
test/1339-harness-selftests
Oct 22, 2026
Merged

jdatcmd merged 1 commit into
mainfrom
test/1339-harness-selftests

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

@jdatcmd Nineteen of my a–m #1339 entries, in the harness self-tests. No cluster, no ledger rows (the names are pytest-only), and guard_tests is unchanged at 474.

Method: every entry was measured first, then fixed and re-measured.

  • Before: a planted bug the old test passed, a one-line change to the code under test that reddened a premise:, or a docstring the code contradicts.
  • After: the same plant fails the check by name.

Mutants are in ~/pgc-tools/i1339h/mut.py on my side; run.sh TAG NODE… applies one, runs, and restores.

entry tag before after
test_layer.py :1013 mislabelled nocollapse (the order-collapse scan never applied) reddens premise: the collapse is refused… prefix dropped; same red, as a check
test_layer.py :1037 mislabelled nobroad reddens premise: the broad except is refused prefix dropped
test_layer.py :1067 mislabelled noraises reddens premise: an unpinned compound raises block is refused prefix dropped
test_layer.py :1098 mislabelled numpass (num never fails) reddens premise: a false claim fails…, unnamed: assert {'passed': 1…} == {…} prefix dropped, and outcomes() now names its check on failure
test_layer.py :1166 mislabelled numfail (num fails on equal) reddens premise: an honest test passes, unnamed prefix dropped; named
test_layer.py :837 weak a planted expect.num(got=int(a != b), want=1, name=…) in the corpus: 1 passed the scan reads keyword values: no test hand-rolls an inequality: got "['test_zz_plant_kwineq.py:3']"; a planted-offence arm for the keyword form
test_compare_to_bash.py :319 mislabelled loopnone / firstword / nofstr redden the three loop-reader premises prefixes dropped
test_compare_to_bash.py :412 mislabelled noexpand reddens premise: with both members present… prefix dropped
test_compare_to_bash.py :468 weak constok (constants admitted to the expansion gate): all green. The fixture also interpolates col, so the two-names rule refused it too a constant-only fixture: got "raised KeyError 'FLOAT_RTOL'" want '[]'
test_compare_to_bash.py :1174 inaccurate "ONE suite each": after your parquet_export_stats port, all three remaining helpers live in one suite, parallel_copy.sh docstring corrected (merged with your #1644 edit)
test_compare_to_bash.py :1455 inaccurate "NOTHING ENFORCES IT … #1046 TRACKS": dropping advisory_lock_class from COMPLETE reddens two arms docstring states the two-way enforcement
test_harness_deps.py :771 mislabelled allbound / allfree redden the two partition premises prefixes dropped
test_harness_deps.py :782 weak everyfix (every conftest fixture a root): at_least(2) passed the derived set asserted exactly: got 'pgc_cluster, pgc_conn, pgc_own_db, server_dir'
test_harness_deps.py :964 weak pip show psycopg … || true and a comment-only mention: both passed a code line that runs it and can fail
test_harness_deps.py :1007 weak the guard job's flag moved into a comment: passed, since the file-wide count stayed 2 checked per job, in code lines
test_harness_deps.py :1122 re-opened, mislabelled not vacuous any more (planting a driver install reddens the main check), but inst_empty reddens its premise:, which is a control on the test's own detector relabelled control:
test_failed_query_sentinel.py :186 vacuous an exclusion IS the table's reason, so "excluded with a stated reason" cannot fail; a planted dead kind frame: passed every exclusion kind is the first parameter of a public method, so none is a dead exemption: got 'frame'
test_failed_query_sentinel.py :324 mislabelled mintother (query_error mints another prefix) reddens premise: minted under the real prefix prefix dropped
test_guards_pinned.py :373 mislabelled noemptyplan (the empty-plan refusal disabled) reddens premise: all three were found…, so it is the check that catches the regression prefix dropped
test_guards_pinned.py :1 inaccurate "Every refusal in the layer": the file pins 17 guards; the layer raises VacuityError at more sites, several pinned in their subject's file docstring scoped to what it pins and where the rest live

Also:

  • outcomes() failed with pytest's bare dict comparison. It now prefixes the check name, so the corpus no longer has one assertion that fails without saying what.
  • A comment in partition() cited a test that doesn't exist in that file. It now names test_harness_deps_classifier.py::…. Under noclose, that test is what reddens.

Left:

  • The same premise: the collapse is refused when nobody rebinds exists in test_ordered.py, which is in your half.
  • test_mutation_ledger.py's 13 entries are their own PR.

Gates at 9c860657: guards 474 passed; gate rc=0; self-test PASSED.

🤖 Generated with Claude Code

… red-first (#1339)

Fixes my a-m #1339 entries in test_layer.py, test_compare_to_bash.py,
test_harness_deps.py, test_failed_query_sentinel.py and test_guards_pinned.py.
Every entry was shown first: a planted bug the old test passed, a one-line change
to the code under test that reddened a "premise:", or a docstring contradicted by
the code. Then fixed, and the same plant now fails by name.

Mislabelled premises (12), each reddened by a one-line change to what it reads,
so each loses the prefix:
- test_layer :1013 :1037 :1067 :1098 :1166 -- the layer's own refusals switched
  off or inverted (nocollapse, nobroad, noraises, numpass, numfail).
- test_compare_to_bash :319 (three: loopnone, firstword, nofstr), :412 (noexpand).
- test_harness_deps :771 (allbound, allfree); :1122 re-opened: its "premise:"
  was a control on the test's own detector (inst_empty), now "control:".
- test_failed_query_sentinel :324 (mintother), test_guards_pinned :373
  (noemptyplan: the check is what catches the empty-plan refusal being removed).

Weak or vacuous arms (6):
- test_layer :837: the inequality scan read positional arguments only; a planted
  `expect.num(got=int(a != b), ...)` passed the population arm. Keywords are read.
- test_compare_to_bash :468: the constant refusal was overdetermined (the fixture
  also interpolates `col`); admitting constants (constok) stayed green. A
  constant-only fixture now reddens by name.
- test_harness_deps :782: at_least(2) passed an invented root (everyfix); the
  derived set is asserted exactly. :964: "pip show psycopg" was counted as text,
  passing `|| true` and a comment; now a code line that can fail. :1007: a count
  over the file passed the guard job's flag moved into a comment; now per job.
- test_failed_query_sentinel :186: an excluded method's reason exists by
  construction; replaced by "no dead exemption" (a planted unused kind reddens).

Inaccurate text (3): test_compare_to_bash :1174 (one suite holds all three helpers, not one each) and :1455
(#1046 landed; dropping a stem from COMPLETE reddens two arms), test_guards_pinned
:1 (it pins 17 guards, not every refusal). Also: outcomes() now names its check
when it fails (it raised pytest's bare dict comparison), and a stale citation in
test_harness_deps names the test file the closure arm actually lives in.

Guard set 467 passed (guard_tests unchanged); no ledger rows (pytest-only names).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at 9c86065. I reran everyfix (every conftest fixture taken as a root, in cluster_fixtures) against both versions of test_harness_deps.py:

  • main: the root-fixture test passes;
  • this head: it fails with "conftest's cluster-reaching fixtures are derived exactly: got 'pgc_cluster, pgc_conn, pgc_own_db, server_dir' want 'pgc_cluster, pgc_conn, pgc_own_db'".

Clean at this head, the five touched files with --pg-config: 152 passed. CI 15/15.

Note: pinning the exact set means a new connecting fixture in conftest now reddens this check by name. The old docstring said that needed no edit here. Your comment states the trade, and I agree with it.

The test_ordered.py collapse premise you left for me is relabelled in my #1667.

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging at 9c86065 with CI green; approved above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants