From 787e90b37006b62315e5c61525a9440027a958e5 Mon Sep 17 00:00:00 2001 From: David Fowler Date: Fri, 6 Nov 2026 12:25:36 -0800 Subject: [PATCH] Retry transient GraphQL failures in milestone changelog fetch-data (#21845) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Failure The Milestone Changelog Generator's `fetch-data` job fails whenever GitHub's GraphQL gateway returns a transient **502/504** (or a truncated body) to one of its two `gh pr list --limit 5000` calls. Nothing retries, so one hiccup fails the whole scheduled run: **12 of 127 runs (~9%)** since 2026-10-06 (#21838). Failing runs: [37508147763](https://github.com/microsoft/aspire/actions/runs/37508147763) (504) · [37618244761](https://github.com/microsoft/aspire/actions/runs/37618244761) · [37774115249](https://github.com/microsoft/aspire/actions/runs/37774115249) · [37821147747](https://github.com/microsoft/aspire/actions/runs/37821147747) (docs list) · [37891536120](https://github.com/microsoft/aspire/actions/runs/37891536120) (`unexpected end of JSON input`) · [38611753607](https://github.com/microsoft/aspire/actions/runs/38611753607) · [39345410030](https://github.com/microsoft/aspire/actions/runs/39345410030) · [39471234099](https://github.com/microsoft/aspire/actions/runs/39471234099) · [39649124067](https://github.com/microsoft/aspire/actions/runs/39649124067) · [40374212743](https://github.com/microsoft/aspire/actions/runs/40374212743) (504) · [40556051543](https://github.com/microsoft/aspire/actions/runs/40556051543) · [40869927084](https://github.com/microsoft/aspire/actions/runs/40869927084) Fixes #21838 ## Change `milestone-changelog.md` gets a `gh_pr_list_with_retry OUT ARGS...` helper in the `fetch-data` step, used by both the product-PR and docs-PR `gh pr list` calls: - up to **4 attempts**, backoff **15s / 30s / 60s** (worst case adds ~105s to a job that already runs `--limit 5000` queries); - retries **only** transient failures: `HTTP 5xx` in stderr, `unexpected end of JSON input`, or gh exiting 0 with an incomplete JSON body; - any other error (401/403/404, bad query) fails immediately, as today; - output must parse as a JSON array before it is accepted; `sort_by(.mergedAt)` now runs on the validated file instead of in a pipe. `milestone-changelog.lock.yml` is recompiled with the pinned gh-aw **v0.89.17** (built from the tag with release ldflags; a baseline compile of `main` produces no drift for this file). Lock diff = the same step script + `frontmatter_hash`. ## Confidence - **Root cause — high.** All 12 failure logs show `Extracted N processed … PR numbers` → a single `HTTP 50x … (https://api.github.com/graphql)` (or truncated JSON) → `exit code 1` with no retry, and the next 6-hourly run usually succeeds. - **Fix correctness — high.** The helper was exercised with a stub `gh` under `set -euo pipefail`: 502×2 then success → succeeds on attempt 3; truncated body then success → succeeds on attempt 2; 504 forever → fails after 4 attempts with `::error::`; HTTP 401 → fails after 1 attempt (no retry). - **Regression risk — low.** Same `gh pr list` arguments and output files; only failure handling changed. Non-transient errors behave as before. Not yet exercised in a real Actions run. - **Validation — compile and actionlint.** `gh aw compile milestone-changelog` (v0.89.17): `✓ … 1 succeeded`. `actionlint` on the lock file reports the identical 5 pre-existing findings as `main` (`concurrency.queue`, `copilot-requests` permission — newer GitHub features actionlint v1.7.12 doesn't know), nothing new. `auto-sec/publication-guard.js` passes. - **CI:** the repo's `Compile, lint, and test agentic workflows`, `Lint handwritten workflows` and `zizmor` checks pass on this PR. **How we stop this recurring:** every bulk GraphQL `gh` call in agentic-workflow pre-steps should go through a bounded, transient-only retry wrapper like this one, so a single gateway 5xx can't fail a scheduled run.
Source diff (milestone-changelog.md) ```diff diff --git a/.github/workflows/milestone-changelog.md b/.github/workflows/milestone-changelog.md index bb9da2398..e1acfb57d 100644 --- a/.github/workflows/milestone-changelog.md +++ b/.github/workflows/milestone-changelog.md @@ -74,6 +74,43 @@ jobs: DATA_DIR="/tmp/gh-aw/pr-data" mkdir -p "$DATA_DIR" + # Run `gh pr list` with bounded retry. The large (--limit 5000) + # GraphQL queries below occasionally hit a transient 502/504 from + # GitHub's GraphQL gateway or get a truncated response body. Retry + # only those transient failures (4 attempts, 15s/30s/60s backoff); + # any other error fails immediately. Usage: gh_pr_list_with_retry OUT ARGS... + gh_pr_list_with_retry() { + local out="$1"; shift + local delays=(15 30 60) + local attempt=1 + local err + err=$(mktemp) + while true; do + if gh pr list "$@" > "$out" 2> "$err" \ + && jq -e 'type == "array"' "$out" > /dev/null 2>&1; then + rm -f "$err" + return 0 + fi + cat "$err" >&2 + local transient=false + if grep -qE 'HTTP 5[0-9][0-9]|unexpected end of JSON input' "$err"; then + transient=true + elif [ ! -s "$err" ] && ! jq -e . "$out" > /dev/null 2>&1; then + # gh exited 0 but returned an incomplete JSON body. + transient=true + fi + if [ "$transient" != true ] || [ "$attempt" -gt "${#delays[@]}" ]; then + rm -f "$err" + echo "::error::gh pr list failed after $attempt attempt(s)" + return 1 + fi + local delay="${delays[$((attempt - 1))]}" + echo "::warning::gh pr list attempt $attempt failed with a transient error; retrying in ${delay}s" + sleep "$delay" + attempt=$((attempt + 1)) + done + } + # 0. Clone the memory branch to read existing state (prs/, changes/, # feedback-issue.json). The full content is copied to # $DATA_DIR/memory/$MILESTONE/ so the agent can read it directly. @@ -144,11 +181,13 @@ jobs: rm -f "$CLONE_ERR" # 1. Fetch ALL merged PRs in milestone, sorted by merge date ascending - gh pr list --repo "$REPO" --state merged --limit 5000 \ + gh_pr_list_with_retry "$DATA_DIR/all-milestone-prs.raw.json" \ + --repo "$REPO" --state merged --limit 5000 \ --search "milestone:$MILESTONE" \ - --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles \ - | jq 'sort_by(.mergedAt)' \ + --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles + jq 'sort_by(.mergedAt)' "$DATA_DIR/all-milestone-prs.raw.json" \ > "$DATA_DIR/all-milestone-prs.json" + rm -f "$DATA_DIR/all-milestone-prs.raw.json" TOTAL=$(jq length "$DATA_DIR/all-milestone-prs.json") echo "Total merged PRs in milestone: $TOTAL" @@ -226,11 +265,13 @@ jobs: DOCS_MERGED_RANGE="merged:${MILESTONE_START}..${BATCH_CUTOFF}" echo "Capping docs PRs to merged on or before $BATCH_CUTOFF (batch is full)" fi - gh pr list --repo "$DOCS_REPO" --state merged --limit 5000 \ + gh_pr_list_with_retry "$DATA_DIR/all-docs-prs.raw.json" \ + --repo "$DOCS_REPO" --state merged --limit 5000 \ --search "$DOCS_MERGED_RANGE" \ - --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles,files \ - | jq 'sort_by(.mergedAt)' \ + --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles,files + jq 'sort_by(.mergedAt)' "$DATA_DIR/all-docs-prs.raw.json" \ > "$DATA_DIR/all-docs-prs.json" + rm -f "$DATA_DIR/all-docs-prs.raw.json" DOCS_TOTAL=$(jq length "$DATA_DIR/all-docs-prs.json") echo "Total docs PRs merged since $MILESTONE_START: $DOCS_TOTAL" ```
``` .github/workflows/milestone-changelog.lock.yml | 55 ++++++++++++++++++++++---- .github/workflows/milestone-changelog.md | 53 ++++++++++++++++++++++--- 2 files changed, 95 insertions(+), 13 deletions(-) ``` **🤖 Posted by AspireAgent (AI Agent)** --------- Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Ankit Jain (cherry picked from commit d47e6e8a51a92bebab4a33b1eebcfd3462d9d673) --- .github/workflows/README.md | 17 ++++ .../workflows/milestone-changelog.lock.yml | 80 +++++++++++++++++-- .github/workflows/milestone-changelog.md | 78 ++++++++++++++++-- 3 files changed, 162 insertions(+), 13 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 116654f1c6a..f6427ae49ff 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -90,6 +90,23 @@ Node 20 action. There is no supported Node 24 upgrade for that dependency; replacing it requires a separate migration of its authentication and locking behavior, rather than merely changing an action pin. +## Milestone changelog fetch diagnostics + +`milestone-changelog.md` retries PR-list requests up to four times, waiting +15, 30, and 60 seconds between attempts. HTTP 5xx errors and the +`unexpected end of JSON input` diagnostic are retryable. When `gh` succeeds +but returns invalid or empty JSON, validation also retries, even if stderr +contains warnings. Other command failures and valid JSON of the wrong type +fail immediately. + +Each failed attempt logs the output filename, command and validation exit codes, +stderr from both tools, and the first 4096 bytes of the response with its total +byte count. Response previews are bounded because PR bodies can be large. +Diagnostic lines are prefixed and legacy `##[` command markers are escaped so +response text is not interpreted as GitHub Actions workflow commands. +Successful requests also replay any `gh` warnings; +their complete JSON is kept in the normal data files rather than printed. + ## Auto-sec dependency reconciliation `auto-sec.md` runs every 12 hours (and on demand). It reconciles open Dependabot diff --git a/.github/workflows/milestone-changelog.lock.yml b/.github/workflows/milestone-changelog.lock.yml index c3b4aa897a4..446d293c157 100644 --- a/.github/workflows/milestone-changelog.lock.yml +++ b/.github/workflows/milestone-changelog.lock.yml @@ -1,4 +1,4 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"2517472aa8e191e8099a2ea79ad07e27e7945b24b40a5349b058bf8200cfecc2","body_hash":"75a78861feb3dbf57cda665b7587eec5ef46a3999e2a20337ba00a8f8875284a","compiler_version":"v0.89.17","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.85"}} +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"836287fdcebb238122975bb2924c76d9f84d0ae924a13c2695563ce00a7d4378","body_hash":"75a78861feb3dbf57cda665b7587eec5ef46a3999e2a20337ba00a8f8875284a","compiler_version":"v0.89.17","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.85"}} # gh-aw-manifest: {"version":1,"secrets":["GH_AW_DEFAULT_OTLP_ENDPOINT","GH_AW_DEFAULT_OTLP_HEADERS","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"f3b81cdb3070066a47faa9bdf440f4f1dc78b1f0","version":"v0.89.17"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.28.20","digest":"sha256:1bacef0f405d77999d01d8c00cb731222531b3f43ef9dc8312c513d6d9038bb1","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.28.20@sha256:1bacef0f405d77999d01d8c00cb731222531b3f43ef9dc8312c513d6d9038bb1"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.20","digest":"sha256:30ab6d3261dd95364281fa0b52ab78450e1c01aa074eccc3a8d4f04b12a6560b","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.20@sha256:30ab6d3261dd95364281fa0b52ab78450e1c01aa074eccc3a8d4f04b12a6560b"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.28.20","digest":"sha256:cba5f56857e4869c4a00c1cce29c3056516cf1a746e18ce380753ecfbe40f112","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.28.20@sha256:cba5f56857e4869c4a00c1cce29c3056516cf1a746e18ce380753ecfbe40f112"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.25","digest":"sha256:9be0a86220e807a0ecc89e53d7453468f7a53fbc6b3d1efd2299025ffe01d086","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.25@sha256:9be0a86220e807a0ecc89e53d7453468f7a53fbc6b3d1efd2299025ffe01d086"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23"},{"image":"ghcr.io/github/github-mcp-server:v1.12.1","digest":"sha256:0ba840c46a237879c8300e7fddb0b6347f20e029ccb9cbe2ce4a943daa1ff560","pinned_image":"ghcr.io/github/github-mcp-server:v1.12.1@sha256:0ba840c46a237879c8300e7fddb0b6347f20e029ccb9cbe2ce4a943daa1ff560"}],"mcp_servers":[{"name":"github","tools":["get_commit","get_file_contents","get_latest_release","get_pull_request","get_pull_request_comments","get_pull_request_diff","get_pull_request_files","get_pull_request_review_comments","get_pull_request_reviews","get_pull_request_status","get_release_by_tag","get_tag","issue_read","list_branches","list_commits","list_issue_types","list_issues","list_pull_requests","list_releases","list_starred_repositories","list_tags","pull_request_read","search_code","search_issues","search_pull_requests","search_repositories"]},{"name":"safeoutputs","tools":["missing_data","missing_tool","noop","publish"]}]} # This file was automatically generated by gh-aw (v0.89.17). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # @@ -1674,6 +1674,68 @@ jobs: DATA_DIR="/tmp/gh-aw/pr-data" mkdir -p "$DATA_DIR" + # Run `gh pr list` with bounded retry. The large (--limit 5000) + # GraphQL queries below occasionally hit a transient 502/504 from + # GitHub's GraphQL gateway or get a truncated response body. Retry + # only those transient failures (4 attempts, 15s/30s/60s backoff); + # Invalid JSON after a successful command is retryable even if gh + # printed warnings. Other command errors and valid non-array JSON + # fail immediately. Usage: gh_pr_list_with_retry OUT ARGS... + gh_pr_list_with_retry() { + local out="$1"; shift + local delays=(15 30 60) + local attempt=1 + local diagnostics err validation_err + diagnostics=$(mktemp -d) + err="$diagnostics/gh.stderr" + validation_err="$diagnostics/jq.stderr" + while true; do + local status=0 validation_status=0 + local transient=false + : > "$validation_err" + if gh pr list "$@" > "$out" 2> "$err"; then + if jq -s -e 'if length == 1 and (.[0] | type == "array") then true else error("Expected exactly one JSON array") end' \ + "$out" > /dev/null 2> "$validation_err"; then + tr '\r' '\n' < "$err" | sed 's/##\[/##\\[/g; s/^/gh stderr: /' >&2 + rm -f "$err" "$validation_err" + rmdir "$diagnostics" + return 0 + else + validation_status=$? + fi + if ! jq -s -e 'length > 0' "$out" > /dev/null 2>&1; then + transient=true + fi + else + status=$? + # Examples: "HTTP 502: Bad Gateway", "unexpected end of JSON input". + if grep -qE 'HTTP 5[0-9][0-9]|unexpected end of JSON input' "$err"; then + transient=true + fi + fi + + echo "gh pr list attempt $attempt failed for $out (gh exit=$status, jq exit=$validation_status)" >&2 + tr '\r' '\n' < "$err" | sed 's/##\[/##\\[/g; s/^/gh stderr: /' >&2 + tr '\r' '\n' < "$validation_err" | sed 's/##\[/##\\[/g; s/^/jq stderr: /' >&2 + # PR bodies can be large and contain workflow-command syntax. + # Bound the preview and prefix every line, including CR-delimited lines. + # Legacy "##[command]" markers are recognized even after a prefix. + echo "Response preview (first 4096 bytes of $(wc -c < "$out") bytes):" >&2 + head -c 4096 "$out" | tr '\r' '\n' | sed 's/##\[/##\\[/g; s/^/stdout: /' >&2 + printf '\n' >&2 + if [ "$transient" != true ] || [ "$attempt" -gt "${#delays[@]}" ]; then + rm -f "$err" "$validation_err" + rmdir "$diagnostics" + echo "::error::gh pr list failed after $attempt attempt(s)" + return 1 + fi + local delay="${delays[$((attempt - 1))]}" + echo "::warning::gh pr list attempt $attempt failed with a transient error; retrying in ${delay}s" + sleep "$delay" + attempt=$((attempt + 1)) + done + } + # 0. Clone the memory branch to read existing state (prs/, changes/, # feedback-issue.json). The full content is copied to # $DATA_DIR/memory/$MILESTONE/ so the agent can read it directly. @@ -1744,11 +1806,13 @@ jobs: rm -f "$CLONE_ERR" # 1. Fetch ALL merged PRs in milestone, sorted by merge date ascending - gh pr list --repo "$REPO" --state merged --limit 5000 \ + gh_pr_list_with_retry "$DATA_DIR/all-milestone-prs.raw.json" \ + --repo "$REPO" --state merged --limit 5000 \ --search "milestone:$MILESTONE" \ - --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles \ - | jq 'sort_by(.mergedAt)' \ + --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles + jq 'sort_by(.mergedAt)' "$DATA_DIR/all-milestone-prs.raw.json" \ > "$DATA_DIR/all-milestone-prs.json" + rm -f "$DATA_DIR/all-milestone-prs.raw.json" TOTAL=$(jq length "$DATA_DIR/all-milestone-prs.json") echo "Total merged PRs in milestone: $TOTAL" @@ -1826,11 +1890,13 @@ jobs: DOCS_MERGED_RANGE="merged:${MILESTONE_START}..${BATCH_CUTOFF}" echo "Capping docs PRs to merged on or before $BATCH_CUTOFF (batch is full)" fi - gh pr list --repo "$DOCS_REPO" --state merged --limit 5000 \ + gh_pr_list_with_retry "$DATA_DIR/all-docs-prs.raw.json" \ + --repo "$DOCS_REPO" --state merged --limit 5000 \ --search "$DOCS_MERGED_RANGE" \ - --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles,files \ - | jq 'sort_by(.mergedAt)' \ + --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles,files + jq 'sort_by(.mergedAt)' "$DATA_DIR/all-docs-prs.raw.json" \ > "$DATA_DIR/all-docs-prs.json" + rm -f "$DATA_DIR/all-docs-prs.raw.json" DOCS_TOTAL=$(jq length "$DATA_DIR/all-docs-prs.json") echo "Total docs PRs merged since $MILESTONE_START: $DOCS_TOTAL" diff --git a/.github/workflows/milestone-changelog.md b/.github/workflows/milestone-changelog.md index bb9da239833..f991b8484e9 100644 --- a/.github/workflows/milestone-changelog.md +++ b/.github/workflows/milestone-changelog.md @@ -74,6 +74,68 @@ jobs: DATA_DIR="/tmp/gh-aw/pr-data" mkdir -p "$DATA_DIR" + # Run `gh pr list` with bounded retry. The large (--limit 5000) + # GraphQL queries below occasionally hit a transient 502/504 from + # GitHub's GraphQL gateway or get a truncated response body. Retry + # only those transient failures (4 attempts, 15s/30s/60s backoff); + # Invalid JSON after a successful command is retryable even if gh + # printed warnings. Other command errors and valid non-array JSON + # fail immediately. Usage: gh_pr_list_with_retry OUT ARGS... + gh_pr_list_with_retry() { + local out="$1"; shift + local delays=(15 30 60) + local attempt=1 + local diagnostics err validation_err + diagnostics=$(mktemp -d) + err="$diagnostics/gh.stderr" + validation_err="$diagnostics/jq.stderr" + while true; do + local status=0 validation_status=0 + local transient=false + : > "$validation_err" + if gh pr list "$@" > "$out" 2> "$err"; then + if jq -s -e 'if length == 1 and (.[0] | type == "array") then true else error("Expected exactly one JSON array") end' \ + "$out" > /dev/null 2> "$validation_err"; then + tr '\r' '\n' < "$err" | sed 's/##\[/##\\[/g; s/^/gh stderr: /' >&2 + rm -f "$err" "$validation_err" + rmdir "$diagnostics" + return 0 + else + validation_status=$? + fi + if ! jq -s -e 'length > 0' "$out" > /dev/null 2>&1; then + transient=true + fi + else + status=$? + # Examples: "HTTP 502: Bad Gateway", "unexpected end of JSON input". + if grep -qE 'HTTP 5[0-9][0-9]|unexpected end of JSON input' "$err"; then + transient=true + fi + fi + + echo "gh pr list attempt $attempt failed for $out (gh exit=$status, jq exit=$validation_status)" >&2 + tr '\r' '\n' < "$err" | sed 's/##\[/##\\[/g; s/^/gh stderr: /' >&2 + tr '\r' '\n' < "$validation_err" | sed 's/##\[/##\\[/g; s/^/jq stderr: /' >&2 + # PR bodies can be large and contain workflow-command syntax. + # Bound the preview and prefix every line, including CR-delimited lines. + # Legacy "##[command]" markers are recognized even after a prefix. + echo "Response preview (first 4096 bytes of $(wc -c < "$out") bytes):" >&2 + head -c 4096 "$out" | tr '\r' '\n' | sed 's/##\[/##\\[/g; s/^/stdout: /' >&2 + printf '\n' >&2 + if [ "$transient" != true ] || [ "$attempt" -gt "${#delays[@]}" ]; then + rm -f "$err" "$validation_err" + rmdir "$diagnostics" + echo "::error::gh pr list failed after $attempt attempt(s)" + return 1 + fi + local delay="${delays[$((attempt - 1))]}" + echo "::warning::gh pr list attempt $attempt failed with a transient error; retrying in ${delay}s" + sleep "$delay" + attempt=$((attempt + 1)) + done + } + # 0. Clone the memory branch to read existing state (prs/, changes/, # feedback-issue.json). The full content is copied to # $DATA_DIR/memory/$MILESTONE/ so the agent can read it directly. @@ -144,11 +206,13 @@ jobs: rm -f "$CLONE_ERR" # 1. Fetch ALL merged PRs in milestone, sorted by merge date ascending - gh pr list --repo "$REPO" --state merged --limit 5000 \ + gh_pr_list_with_retry "$DATA_DIR/all-milestone-prs.raw.json" \ + --repo "$REPO" --state merged --limit 5000 \ --search "milestone:$MILESTONE" \ - --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles \ - | jq 'sort_by(.mergedAt)' \ + --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles + jq 'sort_by(.mergedAt)' "$DATA_DIR/all-milestone-prs.raw.json" \ > "$DATA_DIR/all-milestone-prs.json" + rm -f "$DATA_DIR/all-milestone-prs.raw.json" TOTAL=$(jq length "$DATA_DIR/all-milestone-prs.json") echo "Total merged PRs in milestone: $TOTAL" @@ -226,11 +290,13 @@ jobs: DOCS_MERGED_RANGE="merged:${MILESTONE_START}..${BATCH_CUTOFF}" echo "Capping docs PRs to merged on or before $BATCH_CUTOFF (batch is full)" fi - gh pr list --repo "$DOCS_REPO" --state merged --limit 5000 \ + gh_pr_list_with_retry "$DATA_DIR/all-docs-prs.raw.json" \ + --repo "$DOCS_REPO" --state merged --limit 5000 \ --search "$DOCS_MERGED_RANGE" \ - --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles,files \ - | jq 'sort_by(.mergedAt)' \ + --json number,title,body,author,mergedBy,mergedAt,labels,additions,deletions,changedFiles,files + jq 'sort_by(.mergedAt)' "$DATA_DIR/all-docs-prs.raw.json" \ > "$DATA_DIR/all-docs-prs.json" + rm -f "$DATA_DIR/all-docs-prs.raw.json" DOCS_TOTAL=$(jq length "$DATA_DIR/all-docs-prs.json") echo "Total docs PRs merged since $MILESTONE_START: $DOCS_TOTAL"