zuul: capture glance-api log in post-run#2915
Merged
Merged
Conversation
The nightly testbed lanes intermittently fail when the Octavia Amphora
image import gets stuck in Glance's "queued" state. The image-import
step ("osism manage image octavia") only reports that the image is
stuck; the reason (web-download, qcow2->raw conversion, or Ceph write)
is logged by glance-api itself.
kolla-ansible configures glance-api with
"log_file = /var/log/kolla/glance/glance-api.log", so the import error
goes to that on-host file and not to the container's stdout. As a
result "docker logs glance_api" does not contain it, and nothing in the
published job output reveals the failing stage.
Add a best-effort post-run play that captures the tail of
glance-api.log from every testbed host into the job output before the
cleanup play destroys the environment. The Zuul orchestrator can only
reach the manager directly (the nodes are on a private network), so the
play hops through the manager and fans out to every host returned by
"osism get hosts". It never fails the job: a missing manager address,
an unreachable host, or an unreadable log are all tolerated. The final
debug surfaces both stdout and stderr (with an empty-result fallback)
so a failed SSH to the manager still shows why.
Wire the play into abstract-testbed-deploy's post-run so every deploy,
update, and upgrade lane inherits it.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Roger Luethi <luethi@osism.tech>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The nightly testbed lanes intermittently fail when the Octavia Amphora image import gets stuck in Glance's
queuedstate (tracked asoctavia-stuck-queued). The image-import step (osism manage image octavia) only reports that the image is stuck; the actual reason — web-download, qcow2→raw conversion, or Ceph write — is logged by glance-api itself.kolla-ansible configures glance-api with
log_file = /var/log/kolla/glance/glance-api.log, so the import error goes to that on-host file, not the container's stdout. As a resultdocker logs glance_apidoes not contain it, and nothing in the published job output reveals the failing stage — so the failure can't be diagnosed from CI logs today.Latest occurrence (for reference)
testbed-upgrade-stable-ubuntu-24.04, periodic-midnight 2026-06-27 — the import stalled at 05:00 UTC and failed theBootstrap servicestask (the.CHECKSUMfetch returned 200, so this is the Glance import-state face, not the checksum face):In
job-output.txtyou can see... seems stuck in queued statefrom the import step — but there is no glance-api log anywhere in that build's artifacts (onlyjob-output.*and the orchestrator debug files). That missing log is exactly what this PR adds. The prior night (2026-06-26, build4e26fc42,testbed-deploy-stable-in-a-nutshell-with-tempest) shows the same failure on a different lane.Change
Adds a best-effort post-run play (
playbooks/collect-glance-logs.yml) that captures the tail ofglance-api.logfrom every testbed host into the job output, so the failing import stage is visible in the published build logs.osism get hosts— the same shape as the existingtboperatorcontainer-log collector.sudo -nso it can't hang). The final debug surfaces both stdout and stderr (with an empty-result fallback) so a failed SSH still shows why.abstract-testbed-deploy's post-run, so every deploy / update / upgrade lane inherits it.Validation
yamllintclean;ansible-lintpasses at theproductionprofile.bash -non both the local wrapper and the remote (manager-side) heredoc body; host-extractionawkchecked against a sampleosism get hoststable.🤖 Generated with Claude Code