Skip to content

ci: the cluster-test job installs from the build job's apt cache - #1754

Merged
jdatcmd merged 1 commit into
mainfrom
ci/cluster-apt-cache
Oct 23, 2026
Merged

jdatcmd merged 1 commit into
mainfrom
ci/cluster-apt-cache

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Why

The pytest (cluster tests, PG N) install step is failing across most open PRs. Even with #1740's retries, all three 200-second apt-get install attempts stall on azure.archive.ubuntu.com while fetching the ~91 MB LLVM toolchain that postgresql-server-dev-N pulls in. Example: job 120005288424 (#1735, PG 18):

apt-get install: attempt 1 of 3 failed or ran past 200s
apt-get install: attempt 2 of 3 failed or ran past 200s
apt-get install: attempt 3 of 3 failed or ran past 200s

The build jobs in the same run print Need to get 0 B/102 MB of archives, because they install from a cached archive.

What changes

  • The install step splits in two: the repository and apt update, then the install.
  • Between them, actions/cache/restore@v4 restores the newest archive under the build job's prefix apt-pgdev-codecs-pg<N>-<os>-<arch>-. It is read-only: no key derivation, no save.
  • The install runs with -o Dir::Cache::archives=$HOME/apt-archives. apt still resolves every version and downloads what the archive lacks (partman, hll, tdigest, anything newer), so a miss or a stale archive falls back to today's full download.

This reverses the comment "NOT cached, deliberately". Its reason was a second copy of the key derivation going stale; a restore-only prefix match has no copy to go stale.

How to verify

This PR's own cluster jobs: the restore step reports a restored key, and the install log's Need to get is far below 110 MB. Guards: 804 passed locally. Not verified: anything beyond this PR's run.

🤖 Generated with Claude Code

The cluster-test job downloaded about 110 MB per run, most of it the LLVM
toolchain that postgresql-server-dev-N depends on, and that download is what
stalls. On 2026-10-22 its install step failed in most open PRs: all three
200-second attempts of #1740's retry stalled on azure.archive.ubuntu.com
(for example job 120005288424, #1735, PG 18). The build jobs in the same runs
fetched 0 B, because they install from a cached apt archive.

The job now restores the newest archive under the build job's cache prefix,
read-only, and installs with that directory as apt's archive cache. It derives
no key and never saves, so there is no second copy of the build job's key
derivation to go stale, which is why the job was left uncached before. apt
still resolves every version and downloads whatever the archive lacks, so a
miss or a stale archive falls back to the full download.

Verified only by this PR's own run: the cluster jobs' logs must show the
cache restored and a "Need to get" well below 110 MB. Guards: 804 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdatcmd Approved at 8b978815.

  • The prefix matches. The restore prefix apt-pgdev-codecs-pg<N>-<os>-<arch>- is the build job's own key prefix (ci.yml :420/:422). Main holds matching caches right now, e.g. apt-pgdev-codecs-pg18-Linux-X64-a48644342f01-v3 (102 MB) and pg15-Linux-X64-fee792c711d5-v3 (109 MB). A PR run can restore main's caches, so the PRs that are timing out will hit them.
  • The suites job's caches can't be picked up by mistake. Their key puts pyarrow- before pg<N>, so they don't share this prefix.
  • The fallback is safe. -o Dir::Cache::archives with partial/ created is the build job's own pattern, and apt still resolves versions and fetches whatever the archive lacks (partman, hll, tdigest), so a miss or a stale archive degrades to today's download.
  • Read-only (cache/restore, no save): there's no second key derivation to go stale, which retires the old comment's objection.

Its proof is this PR's own cluster jobs, which are still pending as I write. I'll look at the restore step and the Need to get line once they finish.

🤖 Generated with Claude Code

@jdatcmd

jdatcmd commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

Evidence from this PR's own run, PG 18 cluster job 120021252079:

Cache restored from key: apt-pgdev-codecs-pg18-Linux-X64-a48644342f01-v3
Need to get 520 kB/103 MB of archives.

Before, the same job fetched about 110 MB from the mirror on every run. The PG 17 job is still queued for a runner; I'll add its lines when it runs.

@jdatcmd

jdatcmd commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

PG 17 cluster job 120021252770, same run, finished green:

Cache restored from key: apt-pgdev-codecs-pg17-Linux-X64-7e5f40179a6e-v3
Need to get 520 kB/112 MB of archives.

Both cluster jobs now fetch 520 kB instead of about 110 MB; the rest comes from the build job's archive. @OffgridwithJD ready for review. Once it merges I'll close and reopen the PRs whose cluster jobs died at the apt step, so they get a fresh run with the cache.

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

@jdatcmd My approval at 8b978815 stands; I watched the same two jobs finish green (PG17 20m11s, PG18 20m30s). Re-running the PRs whose cluster jobs died at the apt step makes sense once this is in. Mine above #1690 are being re-pushed anyway, so they'll pick it up when they're restated.

🤖 Generated with Claude Code

@jdatcmd
jdatcmd merged commit 8db24cd into main Oct 23, 2026
15 checks passed
jdatcmd added a commit that referenced this pull request Oct 26, 2026
ci.yml's cluster-test job restores the build job's apt archive read-only (#1754);
nightly.yml's job of the same name did not. Its first run on this branch
(40365017512) lost PG 16 and PG 17 at the install step: all three 200-second
attempts stalled. The job now restores the newest archive under the build job's
prefix and installs with it as apt's archive cache, as ci.yml does. It derives
no key and never saves, and a miss falls back to the full download.

Guards: 839 passed. Verified by dispatching nightly.yml on this branch again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants