Repository navigation
ci: the cluster-test job installs from the build job's apt cache - #1754
Conversation
The cluster-test job downloaded about 110 MB per run, most of it the LLVM toolchain that postgresql-server-dev-N depends on, and that download is what stalls. On 2026-10-22 its install step failed in most open PRs: all three 200-second attempts of #1740's retry stalled on azure.archive.ubuntu.com (for example job 120005288424, #1735, PG 18). The build jobs in the same runs fetched 0 B, because they install from a cached apt archive. The job now restores the newest archive under the build job's cache prefix, read-only, and installs with that directory as apt's archive cache. It derives no key and never saves, so there is no second copy of the build job's key derivation to go stale, which is why the job was left uncached before. apt still resolves every version and downloads whatever the archive lacks, so a miss or a stale archive falls back to the full download. Verified only by this PR's own run: the cluster jobs' logs must show the cache restored and a "Need to get" well below 110 MB. Guards: 804 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OffgridwithJD
left a comment
There was a problem hiding this comment.
@jdatcmd Approved at 8b978815.
- The prefix matches. The restore prefix
apt-pgdev-codecs-pg<N>-<os>-<arch>-is the build job's own key prefix (ci.yml:420/:422). Main holds matching caches right now, e.g.apt-pgdev-codecs-pg18-Linux-X64-a48644342f01-v3(102 MB) andpg15-Linux-X64-fee792c711d5-v3(109 MB). A PR run can restore main's caches, so the PRs that are timing out will hit them. - The suites job's caches can't be picked up by mistake. Their key puts
pyarrow-beforepg<N>, so they don't share this prefix. - The fallback is safe.
-o Dir::Cache::archiveswithpartial/created is the build job's own pattern, and apt still resolves versions and fetches whatever the archive lacks (partman, hll, tdigest), so a miss or a stale archive degrades to today's download. - Read-only (
cache/restore, no save): there's no second key derivation to go stale, which retires the old comment's objection.
Its proof is this PR's own cluster jobs, which are still pending as I write. I'll look at the restore step and the Need to get line once they finish.
🤖 Generated with Claude Code
|
Evidence from this PR's own run, PG 18 cluster job 120021252079: Before, the same job fetched about 110 MB from the mirror on every run. The PG 17 job is still queued for a runner; I'll add its lines when it runs. |
|
PG 17 cluster job 120021252770, same run, finished green: Both cluster jobs now fetch 520 kB instead of about 110 MB; the rest comes from the build job's archive. @OffgridwithJD ready for review. Once it merges I'll close and reopen the PRs whose cluster jobs died at the apt step, so they get a fresh run with the cache. |
|
@jdatcmd My approval at 🤖 Generated with Claude Code |
ci.yml's cluster-test job restores the build job's apt archive read-only (#1754); nightly.yml's job of the same name did not. Its first run on this branch (40365017512) lost PG 16 and PG 17 at the install step: all three 200-second attempts stalled. The job now restores the newest archive under the build job's prefix and installs with it as apt's archive cache, as ci.yml does. It derives no key and never saves, and a miss falls back to the full download. Guards: 839 passed. Verified by dispatching nightly.yml on this branch again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why
The
pytest (cluster tests, PG N)install step is failing across most open PRs. Even with #1740's retries, all three 200-secondapt-get installattempts stall onazure.archive.ubuntu.comwhile fetching the ~91 MB LLVM toolchain thatpostgresql-server-dev-Npulls in. Example: job 120005288424 (#1735, PG 18):The build jobs in the same run print
Need to get 0 B/102 MB of archives, because they install from a cached archive.What changes
apt update, then the install.actions/cache/restore@v4restores the newest archive under the build job's prefixapt-pgdev-codecs-pg<N>-<os>-<arch>-. It is read-only: no key derivation, no save.-o Dir::Cache::archives=$HOME/apt-archives. apt still resolves every version and downloads what the archive lacks (partman, hll, tdigest, anything newer), so a miss or a stale archive falls back to today's full download.This reverses the comment "NOT cached, deliberately". Its reason was a second copy of the key derivation going stale; a restore-only prefix match has no copy to go stale.
How to verify
This PR's own cluster jobs: the restore step reports a restored key, and the install log's
Need to getis far below 110 MB. Guards: 804 passed locally. Not verified: anything beyond this PR's run.🤖 Generated with Claude Code