Repository navigation
[release/17.0] Retry transient GraphQL failures in milestone changelog fetch-data - #22011
Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
Mitch Denny (mitchdenny) wants to merge 1 commit into
Conversation
…21845) ## Failure The Milestone Changelog Generator's `fetch-data` job fails whenever GitHub's GraphQL gateway returns a transient **502/504** (or a truncated body) to one of its two `gh pr list --limit 5000` calls. Nothing retries, so one hiccup fails the whole scheduled run: **12 of 127 runs (~9%)** since 2026-10-06 (#21838). Failing runs: [37508147763](https://github.com/microsoft/aspire/actions/runs/37508147763) (504) · [37618244761](https://github.com/microsoft/aspire/actions/runs/37618244761) · [37774115249](https://github.com/microsoft/aspire/actions/runs/37774115249) · [37821147747](https://github.com/microsoft/aspire/actions/runs/37821147747) (docs list) · [37891536120](https://github.com/microsoft/aspire/actions/runs/37891536120) (`unexpected end of JSON input`) · [38611753607](https://github.com/microsoft/aspire/actions/runs/38611753607) · [39345410030](https://github.com/microsoft/aspire/actions/runs/39345410030) · [39471234099](https://github.com/microsoft/aspire/actions/runs/39471234099) · [39649124067](https://github.com/microsoft/aspire/actions/runs/39649124067) · [40374212743](https://github.com/microsoft/aspire/actions/runs/40374212743) (504) · [40556051543](https://github.com/microsoft/aspire/actions/runs/40556051543) · [40869927084](https://github.com/microsoft/aspire/actions/runs/40869927084) Fixes #21838 ## Change `milestone-changelog.md` gets a `gh_pr_list_with_retry OUT ARGS...` helper in the `fetch-data` step, used by both the product-PR and docs-PR `gh pr list` calls: - up to **4 attempts**, backoff **15s / 30s / 60s** (worst case adds ~105s to a job that already runs `--limit 5000` queries); - retries **only** transient failures: `HTTP 5xx` in stderr, `unexpected end of JSON input`, or gh exiting 0 with an incomplete JSON body; - any other error (401/403/404, bad query) fails immediately, as today; - output must parse as a JSON array before it is accepted; `sort_by(.mergedAt)` now runs on the validated file instead of in a pipe. `milestone-changelog.lock.yml` is recompiled with the pinned gh-aw **v0.89.17** (built from the tag with release ldflags; a baseline compile of `main` produces no drift for this file). Lock diff = the same step script + `frontmatter_hash`. ## Confidence - **Root cause — high.** All 12 failure logs show `Extracted N processed … PR numbers` → a single `HTTP 50x … (https://api.github.com/graphql)` (or truncated JSON) → `exit code 1` with no retry, and the next 6-hourly run usually succeeds. - **Fix correctness — high.** The helper was exercised with a stub `gh` under `set -euo pipefail`: 502×2 then success → succeeds on attempt 3; truncated body then success → succeeds on attempt 2; 504 forever → fails after 4 attempts with `::error::`; HTTP 401 → fails after 1 attempt (no retry). - **Regression risk — low.** Same `gh pr list` arguments and output files; only failure handling changed. Non-transient errors behave as before. Not yet exercised in a real Actions run. - **Validation — compile and actionlint.** `gh aw compile milestone-changelog` (v0.89.17): `✓ … 1 succeeded`. `actionlint` on the lock file reports the identical 5 pre-existing findings as `main` (`concurrency.queue`, `copilot-requests` permission — newer GitHub features actionlint v1.7.12 doesn't know), nothing new. `auto-sec/publication-guard.js` passes. - **CI:** the repo's `Compile, lint, and test agentic workflows`, `Lint handwritten workflows` and `zizmor` checks pass on this PR. **How we stop this recurring:** every bulk GraphQL `gh` call in agentic-workflow pre-steps should go through a bounded, transient-only retry wrapper like this one, so a single gateway 5xx can't fail a scheduled run. <details><summary>Source diff (<code>milestone-changelog.md</code>)</summary> ```diff diff --git a/.github/workflows/milestone-changelog.md b/.github/workflows/milestone-changelog.md index bb9da23..e1acfb5 100644 --- a/.github/workflows/milestone-changelog.md +++ b/.github/workflows/milestone-changelog.md @@ -74,6 +74,43 @@ jobs: DATA_DIR="/tmp/gh-aw/pr-data" mkdir -p "$DATA_DIR" + # Run `gh pr list` with bounded retry. The large (--limit 5000) + # GraphQL queries below occasionally hit a transient 502/504 from + # GitHub's GraphQL gateway or get a truncated response body. Retry + # only those transient failures (4 attempts, 15s/30s/60s backoff); + # any other error fails immediately. Usage: gh_pr_list_with_retry OUT ARGS... + gh_pr_list_with_retry() { + local out="$1"; shift + local delays=(15 30 60) + local attempt=1 + local err + err=$(mktemp) + while true; do + if gh pr list "$@" > "$out" 2> "$err" \ + && jq -e 'type == "array"' "$out" > /dev/null 2>&1; then + rm -f "$err" + return 0 + fi + cat "$err" >&2 + local transient=false + if grep -qE 'HTTP 5[0-9][0-9]|unexpected end of JSON input' "$err"; then + transient=true + elif [ ! -s "$err" ] && ! jq -e . "$out" > /dev/null 2>&1; then + # gh exited 0 but returned an incomplete JSON body. + transient=true + fi + if [ "$transient" != true ] || [ "$attempt" -gt "${#delays[@]}" ]; then + rm -f "$err" + echo "::error::gh pr list failed after $attempt attempt(s)" + return 1 + fi + local delay="${delays[$((attempt - 1))]}" + echo "::warning::gh pr list attempt $attempt failed with a transient error; retrying in ${delay}s" + sleep "$delay" + attempt=$((attempt + 1)) + done + } + # 0. Clone the memory branch to read existing state (prs/, changes/, # feedback-issue.json). The full content is copied to # $DATA_DIR/memory/$MILESTONE/ so the agent can read it directly. @@ -144,11 +181,13 @@ jobs: rm -f "$CLONE_ERR" # 1. Fetch ALL merged PRs in milestone, sorted by merge date ascending - gh pr list --repo "$REPO" --state merged --limit 5000 \ + gh_pr_list_with_retry "$DATA_DIR/all-milestone-prs.raw.json" \ + --repo "$REPO" --state merged --limit 5000 \ --search "milestone:$MILESTONE" \ - --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles \ - | jq 'sort_by(.mergedAt)' \ + --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles + jq 'sort_by(.mergedAt)' "$DATA_DIR/all-milestone-prs.raw.json" \ > "$DATA_DIR/all-milestone-prs.json" + rm -f "$DATA_DIR/all-milestone-prs.raw.json" TOTAL=$(jq length "$DATA_DIR/all-milestone-prs.json") echo "Total merged PRs in milestone: $TOTAL" @@ -226,11 +265,13 @@ jobs: DOCS_MERGED_RANGE="merged:${MILESTONE_START}..${BATCH_CUTOFF}" echo "Capping docs PRs to merged on or before $BATCH_CUTOFF (batch is full)" fi - gh pr list --repo "$DOCS_REPO" --state merged --limit 5000 \ + gh_pr_list_with_retry "$DATA_DIR/all-docs-prs.raw.json" \ + --repo "$DOCS_REPO" --state merged --limit 5000 \ --search "$DOCS_MERGED_RANGE" \ - --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles,files \ - | jq 'sort_by(.mergedAt)' \ + --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles,files + jq 'sort_by(.mergedAt)' "$DATA_DIR/all-docs-prs.raw.json" \ > "$DATA_DIR/all-docs-prs.json" + rm -f "$DATA_DIR/all-docs-prs.raw.json" DOCS_TOTAL=$(jq length "$DATA_DIR/all-docs-prs.json") echo "Total docs PRs merged since $MILESTONE_START: $DOCS_TOTAL" ``` </details> ``` .github/workflows/milestone-changelog.lock.yml | 55 ++++++++++++++++++++++---- .github/workflows/milestone-changelog.md | 53 ++++++++++++++++++++++--- 2 files changed, 95 insertions(+), 13 deletions(-) ``` **🤖 Posted by AspireAgent (AI Agent)** --------- Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Ankit Jain <radical@gmail.com> (cherry picked from commit d47e6e8)
Closed
14 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #21845 to
release/17.0(cherry-picked from d47e6e8 with no modifications). Layer 14 of 55 in the release/17.0 backport stack; stacked on #22010.Backport recommendation: No — Workflows run from main.
Backport of #21845
Checklist
<remarks />and<code />elements on your triple slash comments?