From 4be6bd3a57e41140b7036ba7e43711b70d97d00c Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 17:59:14 +0000 Subject: [PATCH 1/2] ci: the nightly's coverage job installs pg_partman, hll and tdigest (#1382) The coverage job ran partman_columnar, approx_aggregates and the other extension suites without their PGDG packages, so the nightly went red. It now installs postgresql-18-partman, -hll and -tdigest, and their versions join its cache key, as in the suites job. Selftest part 640 derives the list instead of trusting it: every job that runs the suites must install every extension package any such job installs. Its detector names a workflow it cannot read or finds no suite job in, so an empty answer can only mean nothing is missing, and the suite-running jobs are named as a set. The part has a database-free pytest twin with its own Python reader: test_every_suite_job_installs_the_extensions.py (NO_CLUSTER, guard_tests 429 -> 433, TESTS.md section 106). Fixes #1382. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N --- .github/workflows/nightly.yml | 4 +- test/check_ledger.tsv | 6 + test/check_ledger_budget.txt | 14 +- test/pytest/TESTS.md | 18 +++ test/pytest/expected_tests.txt | 4 +- ...every_suite_job_installs_the_extensions.py | 127 ++++++++++++++++++ test/pytest/test_harness_deps.py | 3 + ...every-suite-job-installs-the-extensions.sh | 99 ++++++++++++++ test/selftest/parts.manifest | 1 + 9 files changed, 272 insertions(+), 4 deletions(-) create mode 100644 test/pytest/test_every_suite_job_installs_the_extensions.py create mode 100644 test/selftest/640-every-suite-job-installs-the-extensions.sh diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index a7de62150..02c8b54b4 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -419,7 +419,7 @@ jobs: run: | set -euo pipefail v=$(apt-cache policy \ - postgresql-18 postgresql-server-dev-18 \ + postgresql-18 postgresql-server-dev-18 postgresql-18-partman postgresql-18-hll postgresql-18-tdigest \ liblz4-dev libzstd-dev zlib1g-dev python3-pip lcov \ | awk '/Candidate:/ {print $2}' | sort) # $ImageOS, from the shell, where it actually has a value. Reported @@ -449,7 +449,7 @@ jobs: mkdir -p "$HOME/apt-archives/partial" sudo timeout 600 apt-get -o Acquire::Retries=5 \ -o Dir::Cache::archives="$HOME/apt-archives" install -y --no-install-recommends \ - postgresql-18 postgresql-server-dev-18 \ + postgresql-18 postgresql-server-dev-18 postgresql-18-partman postgresql-18-hll postgresql-18-tdigest \ liblz4-dev libzstd-dev zlib1g-dev python3-pip lcov 2>&1 | tee /tmp/apt-install.log # Installed with sudo because the suites run under sudo; a plain # pip3 install lands in ~/.local where root cannot see it, which is how diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 2e9a01bcf..6a8c8d367 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -1790,6 +1790,12 @@ harness_selftest 630-a-build-with-no-pg-config-must and the refusal is make's ow harness_selftest 630-a-build-with-no-pg-config-must and with stdin closed it refuses, rather than exiting 0 having done nothing 15;16;17;18;19 2026-10-13 sed given 1895. harness_selftest part 640 enters with six checks from five-major +# runs (1245/1245 on each of 15-19). Five are dated: the job check by main's +# nightly.yml (coverage lacked postgresql-18-partman, -hll and -tdigest), and the +# detector, unreadable-file, no-suite-job and job-set checks by four mutations of +# the part itself. The premise that both workflow files exist arrives never. +# +# ledger rows 2466 + 6 = 2472 +# never 1894 + 1 = 1895 +# dated 572 + 5 = 577 +# 1895 + 577 == 2472 +checks_never_observed_red 1895 diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index c7826760e..fca414a22 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -151,6 +151,7 @@ behaviour, the source of that number is named. - [103. test_collation_change.py: a collation change without a rewrite leaves columnar answering as heap does](#103-test_collation_changepy-a-collation-change-without-a-rewrite-leaves-columnar-answering-as-heap-does) - [104. test_build_needs_pg_config.py: a build with no pg_config refuses, it does not wait on stdin](#104-test_build_needs_pg_configpy-a-build-with-no-pg_config-refuses-it-does-not-wait-on-stdin) - [105. test_import_export_privilege.py: import and export check the table's privilege, serial and parallel](#105-test_import_export_privilegepy-import-and-export-check-the-tables-privilege-serial-and-parallel) +- [106. test_every_suite_job_installs_the_extensions.py: every job that runs the suites installs the extension packages](#106-test_every_suite_job_installs_the_extensionspy-every-job-that-runs-the-suites-installs-the-extension-packages) ## 1. How to read a test in here @@ -6829,3 +6830,20 @@ The pytest twin of `test/import_export_privilege.sh` (#559, #566), with the para | `test_a_table_the_caller_holds_no_grant_on_is_refused` | the seven deny arms, serial and parallel, in the shell's order; then `victim` keeps exactly its rows and no file of it was written | | `test_the_owner_is_not_refused` | the owner's own exports, storage id and import are not refused | | `test_the_owner_runs_the_parallel_twins` | the owner's `parallel_copy` loads its 50 rows, and its `parallel_export_parquet` writes some | + +## 106. test_every_suite_job_installs_the_extensions.py: every job that runs the suites installs the extension packages + +The pytest half of selftest part 640 (#1382). Suites such as `partman_columnar` and `approx_aggregates` need a PGDG extension package (`postgresql-N-partman`, `-hll`, `-tdigest`) and record a FAIL without it. The nightly's coverage job ran those suites without the packages, and the nightly went red. So the list is derived, not trusted: every job that runs the suites must install every extension package any such job installs. + +READ, WITH ITS OWN READER. Python reads `ci.yml` and `nightly.yml` line by line, where the part uses awk. A job runs the suites when a line that is not a comment calls `run_all_versions.sh` or `run_coverage.sh`, or runs `pytest -q` in a job whose key names a cluster. The detector names a workflow it cannot read or finds no suite job in, so an empty answer can only mean nothing is missing. The check names match the part's. + +Needs no cluster and no driver. + +### Every test + +| test | what it holds | +| --- | --- | +| `test_both_workflows_are_present` | both workflow files exist (premise) | +| `test_the_detector_names_what_it_finds_and_what_it_cannot_read` | on workflows written to fail, the detector names the job that lacks a package, a workflow it cannot read, and a workflow with no suite job | +| `test_the_suite_jobs_are_the_five_known` | the suite-running jobs are exactly `ci.yml`'s `pytest-cluster` and `suites`, and `nightly.yml`'s `coverage`, `pytest-cluster` and `suites`, so a rename is red | +| `test_every_suite_job_installs_every_extension_package` | no suite job lacks a package another installs | diff --git a/test/pytest/expected_tests.txt b/test/pytest/expected_tests.txt index 74d8e1890..85e77a583 100644 --- a/test/pytest/expected_tests.txt +++ b/test/pytest/expected_tests.txt @@ -281,7 +281,9 @@ # of selftest part 630, collected over the NO_CLUSTER files on this tree. # guard_tests 427 -> 429 for #1236's two gate arms in test_mutation_ledger.py (a dated # premise is refused, and its control), by collection over the NO_CLUSTER files on this tree. -guard_tests 429 +# guard_tests 429 -> 433 for #1382's test_every_suite_job_installs_the_extensions.py, four +# tests, the twin of selftest part 640, by collection over the NO_CLUSTER files. +guard_tests 433 # The complement: tests that need the driver and a throwaway cluster. Until #1016 these ran # in no CI job at all -- a quarter of the corpus, green when somebody ran them by hand and diff --git a/test/pytest/test_every_suite_job_installs_the_extensions.py b/test/pytest/test_every_suite_job_installs_the_extensions.py new file mode 100644 index 000000000..e97b10cdb --- /dev/null +++ b/test/pytest/test_every_suite_job_installs_the_extensions.py @@ -0,0 +1,127 @@ +"""Every job that runs the suites installs the same extension packages (#1382). + +The pytest half of selftest part 640, with its own reader: Python, line by line, +where the part uses awk. The check names match. + +Suites such as partman_columnar and approx_aggregates need a PGDG extension +package (postgresql-N-partman, -hll, -tdigest), and record a FAIL without it +(#1215). The nightly's coverage job ran those suites without the packages, and +the nightly went red (#1382). So the list is derived, not trusted: every job that +runs the suites must install every extension package any such job installs. + +A job runs the suites when a line that is not a comment calls run_all_versions.sh +or run_coverage.sh, or runs `pytest -q` in a job whose key names a cluster. The +suite jobs are also named as a set, so a renamed cluster job cannot drop out +quietly. Needs no cluster and no driver. +""" + +import pathlib +import re + +HERE = pathlib.Path(__file__).resolve().parent +WORKFLOWS = HERE.parent.parent / ".github" / "workflows" +CI = WORKFLOWS / "ci.yml" +NIGHTLY = WORKFLOWS / "nightly.yml" + +JOB_KEY = re.compile(r"^ ([A-Za-z_-]+):\s*$") +RUNS = re.compile(r"run_all_versions\.sh|run_coverage\.sh") +PACKAGE = re.compile(r"postgresql-(?:\$\{\{ matrix\.pg \}\}|\d+)-([a-z0-9]+)") + + +def suite_jobs(path): + """{job: set of package suffixes} for each job in `path` that runs the suites.""" + jobs, job, in_jobs = {}, None, False + runs, packages = set(), {} + for line in path.read_text().splitlines(): + if line.startswith("jobs:"): + in_jobs = True + continue + if not in_jobs: + continue + key = JOB_KEY.match(line) + if key: + job = key.group(1) + packages[job] = set() + continue + if job is None or line.lstrip().startswith("#"): + continue + if RUNS.search(line) or ("/pytest -q" in line and "cluster" in job): + runs.add(job) + packages[job].update(PACKAGE.findall(line)) + for name in packages: + if name in runs: + jobs[name] = packages[name] + return jobs + + +def missing(*paths): + """Space-joined "file:job:pkg" for each package a suite job lacks, sorted. + + A workflow it could not compare is named as "file:unreadable" or + "file:no-suite-job", so an empty answer can only mean nothing is missing. + """ + found, out = {}, [] + for path in paths: + if not path.is_file(): + out.append(f"{path.name}:unreadable") + continue + found[path] = suite_jobs(path) + if not found[path]: + out.append(f"{path.name}:no-suite-job") + every = set().union(*(p for jobs in found.values() for p in jobs.values())) + for path, jobs in found.items(): + for job, packages in jobs.items(): + out.extend(f"{path.name}:{job}:{p}" for p in every - packages) + return " ".join(sorted(out)) + + +FIXTURE = """\ +jobs: + a: + steps: + - run: apt-get install postgresql-${{ matrix.pg }}-partman postgresql-${{ matrix.pg }}-hll + - run: bash test/run_all_versions.sh x + b: + steps: + - run: apt-get install postgresql-18 postgresql-server-dev-18 postgresql-18-partman + - run: bash test/run_coverage.sh x + c: + steps: + # c runs no suite, though this comment names run_all_versions.sh + - run: apt-get install postgresql-18 +""" + + +def test_both_workflows_are_present(expect): + expect.text("yes" if CI.is_file() and NIGHTLY.is_file() else "no", "yes", + "premise: both workflows are present") + + +def test_the_detector_names_what_it_finds_and_what_it_cannot_read(tmp_path, expect): + # Proven on workflows made to fail, so the real check cannot pass by finding + # nothing to compare. + w = tmp_path / "w.yml" + w.write_text(FIXTURE) + expect.text(missing(w), "w.yml:b:hll", + "the detector names a suite job that lacks a package the others install") + + expect.text(missing(tmp_path / "absent.yml"), "absent.yml:unreadable", + "the detector names a workflow it cannot read") + + nosuite = tmp_path / "nosuite.yml" + nosuite.write_text("jobs:\n c:\n" + FIXTURE.split(" c:\n", 1)[1]) + expect.text(missing(nosuite), "nosuite.yml:no-suite-job", + "the detector names a workflow in which it finds no suite job") + + +def test_the_suite_jobs_are_the_five_known(expect): + # Not a premise: renaming, adding or deleting a job reddens it (#1236). + got = " ".join(sorted(f"{p.name}:{job}" for p in (CI, NIGHTLY) for job in suite_jobs(p))) + expect.text(got, "ci.yml:pytest-cluster ci.yml:suites nightly.yml:coverage " + "nightly.yml:pytest-cluster nightly.yml:suites", + "the suite-running jobs are exactly the five this part knows") + + +def test_every_suite_job_installs_every_extension_package(expect): + expect.text(missing(CI, NIGHTLY) or "none", "none", + "every job that runs the suites installs every extension package (#1382)") diff --git a/test/pytest/test_harness_deps.py b/test/pytest/test_harness_deps.py index fd0d715fb..7b10d7fce 100644 --- a/test/pytest/test_harness_deps.py +++ b/test/pytest/test_harness_deps.py @@ -83,6 +83,9 @@ # #1344: drives `make -n` on the Makefile with no pg_config. Needs no cluster # and no driver; it never builds and never runs the shell harness. "test_build_needs_pg_config.py", + # #1382: reads the two workflow files as text, with its own Python reader. Needs + # no cluster and no driver, and never runs the shell harness. + "test_every_suite_job_installs_the_extensions.py", # The classifier's own controls, split out of this file so the job whose file # list IS this list actually runs them. This file cannot be in the list: it # hands cluster-bound file names to pytest, so it needs what they need. diff --git a/test/selftest/640-every-suite-job-installs-the-extensions.sh b/test/selftest/640-every-suite-job-installs-the-extensions.sh new file mode 100644 index 000000000..600ca46f6 --- /dev/null +++ b/test/selftest/640-every-suite-job-installs-the-extensions.sh @@ -0,0 +1,99 @@ +# ---- every job that runs the suites installs the same extension packages ----- +# +# Suites such as partman_columnar and approx_aggregates need a PGDG extension +# package (postgresql-N-partman, -hll, -tdigest). Without it they record a FAIL, +# not a pass (#1215). #1291 and #1296 added those packages to the per-PR jobs and +# to the nightly's suites and pytest jobs, but not to the nightly's coverage job, +# which runs the same suites under run_coverage.sh. The nightly went red (#1382). +# +# So the list is derived, not trusted: every job that runs the suites must +# install every extension package any such job installs. A job runs the suites +# when it calls run_all_versions.sh or run_coverage.sh, or runs pytest against a +# cluster (a job whose key names a cluster). Comment lines are skipped: two jobs +# that run no suite mention run_all_versions.sh in their comments. + +# _ej_scan : one line per suite-running job, "job pkg pkg ...", where +# each pkg is the suffix of postgresql-- on the job's lines. +_ej_scan() { + awk ' + /^jobs:/ { inj = 1; next } + inj && /^ [a-zA-Z_-]+:[[:space:]]*$/ { flush(); job = $1; sub(/:$/, "", job); runs = 0; delete pk; next } + inj && job != "" && $0 !~ /^[[:space:]]*#/ { + if ($0 ~ /run_all_versions\.sh|run_coverage\.sh/ || ($0 ~ /\/pytest -q/ && job ~ /cluster/)) runs = 1 + line = $0 + while (match(line, /postgresql-(\$\{\{ matrix\.pg \}\}|[0-9]+)-[a-z0-9]+/)) { + p = substr(line, RSTART, RLENGTH); sub(/.*-/, "", p); pk[p] = 1 + line = substr(line, RSTART + RLENGTH) + } + } + function flush( s, p) { + if (job != "" && runs) { s = job; for (p in pk) s = s " " p; print s } + } + END { flush() }' "$1" +} + +# _ej_missing ...: "file:job:pkg" for each package a suite job lacks, and +# "file:unreadable" or "file:no-suite-job" for a workflow it could not compare, so +# an empty answer can only mean nothing is missing. +_ej_missing() { + local f all + all="$(for f in "$@"; do [ -r "$f" ] && _ej_scan "$f" | awk '{ $1 = ""; print }' | tr ' ' '\n'; done | grep . | sort -u)" + for f in "$@"; do + if [ ! -r "$f" ]; then echo "${f##*/}:unreadable"; continue; fi + [ -n "$(_ej_scan "$f")" ] || echo "${f##*/}:no-suite-job" + _ej_scan "$f" | while read -r job pkgs; do + for p in $all; do + case " $pkgs " in *" $p "*) ;; *) echo "${f##*/}:$job:$p" ;; esac + done + done + done | sort | tr '\n' ' ' | sed 's/ $//' +} + +_ej_ci="$TESTDIR/../.github/workflows/ci.yml" +_ej_nl="$TESTDIR/../.github/workflows/nightly.yml" + +check "premise: both workflows are present" \ + "$([ -r "$_ej_ci" ] && [ -r "$_ej_nl" ] && echo yes || echo no)" "yes" + +_ej_jobs="$(for f in "$_ej_ci" "$_ej_nl"; do _ej_scan "$f" | sed "s|^|${f##*/}:|"; done)" +echo "-- suite jobs and their extension packages:" +printf '%s\n' "$_ej_jobs" | sed 's/^/ /' + +# The detector proven on a workflow made to lack one package, so the real check +# below cannot pass by finding nothing to compare. +_ej_tmp="$(mktemp -d)" +cat >"$_ej_tmp/w.yml" <<'EOF' +jobs: + a: + steps: + - run: apt-get install postgresql-${{ matrix.pg }}-partman postgresql-${{ matrix.pg }}-hll + - run: bash test/run_all_versions.sh x + b: + steps: + - run: apt-get install postgresql-18 postgresql-server-dev-18 postgresql-18-partman + - run: bash test/run_coverage.sh x + c: + steps: + # c runs no suite, though this comment names run_all_versions.sh + - run: apt-get install postgresql-18 +EOF +check "the detector names a suite job that lacks a package the others install" \ + "$(_ej_missing "$_ej_tmp/w.yml")" "w.yml:b:hll" + +# An empty answer must mean "nothing missing", never "nothing read" (#1385 review): +# a workflow the detector cannot read, or in which it finds no suite job, is named. +sed -n '/^ c:/,$p' "$_ej_tmp/w.yml" | sed '1i jobs:' >"$_ej_tmp/nosuite.yml" +check "the detector names a workflow it cannot read" \ + "$(_ej_missing "$_ej_tmp/absent.yml")" "absent.yml:unreadable" +check "the detector names a workflow in which it finds no suite job" \ + "$(_ej_missing "$_ej_tmp/nosuite.yml")" "nosuite.yml:no-suite-job" +rm -rf "${_ej_tmp:?}" + +# Not a premise: renaming, adding or deleting a job reddens it (#1236). Named as a +# set, so a cluster job renamed without "cluster" in its key cannot drop out quietly. +check "the suite-running jobs are exactly the five this part knows" \ + "$(printf '%s\n' "$_ej_jobs" | awk '{ print $1 }' | sort | tr '\n' ' ' | sed 's/ $//')" \ + "ci.yml:pytest-cluster ci.yml:suites nightly.yml:coverage nightly.yml:pytest-cluster nightly.yml:suites" + +check "every job that runs the suites installs every extension package (#1382)" \ + "$(_ej_m="$(_ej_missing "$_ej_ci" "$_ej_nl")"; echo "${_ej_m:-none}")" "none" diff --git a/test/selftest/parts.manifest b/test/selftest/parts.manifest index c2ba2b882..97f2e2557 100644 --- a/test/selftest/parts.manifest +++ b/test/selftest/parts.manifest @@ -60,3 +60,4 @@ 610-a-declined-capability-is-not-silent.sh 620-a-suite-s-own-trap-must-stop.sh 630-a-build-with-no-pg-config-must.sh +640-every-suite-job-installs-the-extensions.sh From 577c404c4c8e2a7017fa19d2b0f327b077b65482 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Wed, 14 Oct 2026 18:30:35 +0000 Subject: [PATCH 2/2] test: the extension-package twin is declared where it names the runners as data (#1385 review) Its RUNS pattern spells run_all_versions.sh and run_coverage.sh to recognise a workflow step that runs the suites, so the harness- independence inventory counted it as reaching into the shell harness. It reads only the two workflow files and never the runners. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_019VvWuX1zUwk6Y513waRd4N --- test/pytest/test_harness_deps.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/test/pytest/test_harness_deps.py b/test/pytest/test_harness_deps.py index 7b10d7fce..ac8761765 100644 --- a/test/pytest/test_harness_deps.py +++ b/test/pytest/test_harness_deps.py @@ -1285,6 +1285,10 @@ def installs(text): "with its own Python reader, to find an EXIT trap that replaces the one setup " "arms (#1321). It never sources or executes either: the rule is about what " "the shell text says, so the text is the only place to read it", + "test_every_suite_job_installs_the_extensions.py": + "names the matrix and coverage runners' file names as data, to recognise a " + "workflow step that runs the suites (#1382). It reads only the two workflow " + "files and never reads or executes the runners themselves", } _SHELL_NAMES = re.compile(