Skip to content

Unblock GitHub Actions queue retries - #464

Merged
trask merged 1 commit into
mainfrom
trask-collector-retry-investigation
Oct 14, 2026
Merged

trask merged 1 commit into
mainfrom
trask-collector-retry-investigation

Conversation

@trask

@trask trask commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Keep saved queue runs moving despite jobs-API 502s and stale GitHub job statuses, without increasing the collection budget or dropping attempts.

A finalized raw record can retain GitHub's stale status when completion evidence is present:

{"job_status":"in_progress","job_conclusion":"success","job_completed_at":"2026-08-13T06:53:33Z"}
  • Fetch jobs in pages of 50, retaining filter=all and the unique-job completeness check. Repository and workflow-run pages stay at 100.
  • Reserve five of each direction's 25 pending retry slots for unresolved failures and 20 for other unfinished runs. Share unused capacity and retain oldest-attempt ordering.
  • Finalize jobs with stale in_progress status only when they also have a completion timestamp and a terminal conclusion. Preserve GitHub's reported status in raw records.
  • Clear resolved lookup errors even when jobs remain unfinished, while retaining cumulative failure counts.

Preserves the checkpointing and anti-starvation fixes from #377, live-first collection and budgets from #385, bounded transport retries from #392, and runnerless/negative-queue filtering from #394 and #412. No state migration or snapshot-writer changes are required.

Smaller pages recovered complete job lists for representative failing runs using operator credentials. Recovery under the collector App's credentials remains unverified; GitHub's underlying 502 cause is not established.

Use smaller jobs pages, share bounded pending retries between failed and unfinished runs, and recognize terminal job evidence when GitHub retains a stale status.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: dce070cb-ab76-4725-80eb-0db99ed545b4
@trask
trask requested a balanced review from Copilot October 13, 2026 20:09
@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Pull request dashboard status

Merged · refreshed 2026-10-14 16:48 UTC

Status above doesn't look right?
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Recovery and budget headroom remain unverified under the production collector credentials.

0 open findings

What changed in this PR

Improves queue collection resilience against jobs-API failures and stale job statuses.

Changes:

  • Uses 50-job pagination with completeness checks.
  • Fairly allocates pending-run retries and clears resolved errors.
  • Recognizes terminal jobs with stale in_progress statuses and adds coverage.
File Description
.github/​scripts/​github-actions-queue/​collect.py Implements pagination, retry allocation, and terminal-state handling.
.github/​scripts/​github-actions-queue/​test_collect.py Tests the new collection behavior and edge cases.
github-actions-queue/​README.md Documents updated collector semantics.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@trask
trask marked this pull request as ready for review October 13, 2026 21:07
@trask
trask requested a review from a team as a code owner October 13, 2026 21:07
@trask
trask requested a review from adrielp October 13, 2026 21:07
@trask
trask merged commit 1a1502e into main Oct 14, 2026
8 checks passed
@trask
trask deleted the trask-collector-retry-investigation branch October 14, 2026 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants