Mm/ci diagnose hangs#2173
Closed
mmontalbo wants to merge 3 commits into
Closed
Conversation
When a test hangs, prove produces no further output, so the CI job simply
idles until the platform's hard timeout (six hours on GitHub) and is then
cancelled, with nothing in the log to say which test stalled. Diagnosing
such a hang currently means correlating a healthy run's test list against
the truncated one to find the test that never reported, which is slow and
only possible after the fact.
Start a watchdog at the beginning of the build-and-test run that, if it
sees no progress for GIT_TEST_STALL_TIMEOUT seconds (two hours by default,
well beyond a healthy run), dumps state that points at the culprit:
- the tests still in flight: test-lib removes a test's trash directory
only once it passes, so a surviving one has not finished;
- the tail of each such test's --verbose-log trace, i.e. the command it
is stuck on (our CI already runs with --verbose-log -x);
- the process table sorted by resident memory, plus the machine's
memory and kernel log, to spot a memory or OOM cascade.
The watchdog only reports; it neither fails nor kills anything, so it
cannot cause a spurious failure even if it fires on an unusually slow run.
It counts down in short sleeps rather than one long "sleep" so that when
it is stopped on the happy path, no long-lived sleep lingers holding the
job's output open. The logic lives in the shared ci/ scripts, so both the
GitHub and GitLab pipelines benefit.
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
The previous commit makes a stalled test run dump diagnostics, but the job itself still idles until GitHub's six-hour hard limit before it is cancelled. Give the test jobs an explicit timeout-minutes so that a hang fails the job within a few hours instead, shortly after the ci/ stall watchdog has printed its diagnostics, and GitHub tears down the job's process tree cleanly. Three hours leaves ample margin over a healthy run (well under two hours) while still saving half of the wasted budget. Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
|
There is a merge commit in this Pull Request: Please rebase the branch and force-push. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thanks for taking the time to contribute to Git! Please be advised that the
Git community does not use github.com for their contributions. Instead, we use
a mailing list (git@vger.kernel.org) for code submissions, code reviews, and
bug reports. Nevertheless, you can use GitGitGadget (https://gitgitgadget.github.io/)
to conveniently send your Pull Requests commits to our mailing list.
For a single-commit pull request, please leave the pull request description
empty: your commit message itself should describe your changes.
Please read the "guidelines for contributing" linked above!