Repository navigation
ci: the nightly cluster-test job gets 40 minutes, not 20 (#1757) - #1758
Conversation
The nightly deep gate has been red since 2026-10-23 (#1757), mostly on `pytest (cluster tests, PG N)`. Those jobs are cancelled, not failed: on 2026-10-26 (run 40322948037) PG 16 finished in 18m52s, and PG 15, 17 and 18 were cancelled at the 20-minute limit. PG 15's accounting line had already printed "7305 pass + 0 fail + 41 unrun". #1740 raised ci.yml's job of the same name to 40 minutes; nightly.yml kept 20. Guards: 839 passed. Verified by dispatching nightly.yml on this branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OffgridwithJD
left a comment
OffgridwithJD
left a comment
There was a problem hiding this comment.
@jdatcmd Approved at ea1c13de.
- The diff:
nightly.yml'spytest-clusterjob was the only job still at 20 minutes (:731), and the others run 45–120. This brings it in line with ci.yml's same-named job (#1740), with the measured times in the comment. - Your numbers: 18m52s finished, three cancelled at 20m01s–21m11s, and PG 15 cancelled after its accounting line printed. That's a budget problem, not a test failure, so raising the limit is the right fix.
Not blocking: the nightly pytest-cluster job doesn't restore the build job's apt cache, which #1754 added to ci.yml. Porting that would shave the install time there too, as a follow-up.
🤖 Generated with Claude Code
ci.yml's cluster-test job restores the build job's apt archive read-only (#1754); nightly.yml's job of the same name did not. Its first run on this branch (40365017512) lost PG 16 and PG 17 at the install step: all three 200-second attempts stalled. The job now restores the newest archive under the build job's prefix and installs with it as apt's archive cache, as ci.yml does. It derives no key and never saves, and a miss falls back to the full download. Guards: 839 passed. Verified by dispatching nightly.yml on this branch again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OffgridwithJD
left a comment
OffgridwithJD
left a comment
There was a problem hiding this comment.
@jdatcmd Re-approved at ab925a32. The new restore step is ci.yml's #1754 block verbatim: the same build-job key prefix, restore-only, partial/ created, and a miss falls back to the full download. The scheduled nightly runs on main, so it reads main's caches directly. That fits run 40365017512, where PG 16 and 17 stalled on all three install attempts.
🤖 Generated with Claude Code
|
@OffgridwithJD The nightly dispatched on this branch is green on all four cluster jobs (run 40367661645). The first dispatch (40365017512) lost PG 16 and 17 at the apt install, so this PR now also gives the nightly job the read-only apt cache that #1754 gave ci.yml.
Every job ran past the old 20-minute limit and finished under 40. Ready for review. |
|
@jdatcmd Thanks, that's the evidence. My approval at 🤖 Generated with Claude Code |
Fixes the red nightly in #1757.
Why
The nightly's
pytest (cluster tests, PG N)jobs are cancelled at the 20-minute limit, not failing. On 2026-10-26 (run 40322948037):accounting: 7305 pass + 0 fail + 41 unrun = 7346#1740 raised
ci.yml's job of the same name to 40 minutes.nightly.ymlkept 20, and the corpus grew past it when the big stack merged on 2026-10-23.What changes
timeout-minutes: 20becomes40on the nightlypytest-clusterjob, with a comment giving the measured times.How it is verified
🤖 Generated with Claude Code