🐛🧑‍💻 - Throttle workflow queue polling and increase workflow timeouts

[efraespada]

Is there an existing issue for this?

• I have searched the existing issues.

Which actions are affected?

Pull Request

Which platforms are affected?

No response

Description

During dogfooding, multiple Copilot workflows failed before executing their business logic because the workflow queue exhausted GitHub's API rate limit.

The current queue polls GitHub every two seconds and performs multiple workflow-run queries per polling cycle. When several runs are queued simultaneously, this creates unnecessary API pressure and can prevent all runs from starting.
The sequential execution guarantee must be preserved: runs must never be cancelled, skipped or overwritten.

Expected behavior

• Workflow runs remain strictly sequential.
• Queued runs wait until previous Copilot runs finish.
• Polling must not exhaust GitHub's API rate limit.
• Rate-limit responses must be handled gracefully.
• Long queues must be allowed to drain without the workflow timing out prematurely.
• The queue must remain reliable during bursts of issue, comment, pull request and release activity.

Reproducing the issue

• Copilot - Issue Comment: run 33379584423
• Copilot - Issue Comment: run 33378311554
• Several Issue, Pull Request, Pull Request Comment, Commit and CI runs also failed during the same period.

copilot Version

master

Relevant Log Output

API rate limit exceeded

Additional context and comments

Acceptance criteria

• No unnecessary API request burst occurs when multiple runs are queued.
• HTTP 429 responses cause a controlled wait instead of immediate workflow failure.
• Sequential execution remains guaranteed.
• All queued runs are eventually processed without cancellation.
• Workflow timeouts are long enough to accommodate the configured queue wait period.
• CI, release, hotfix and setup workflow contract tests pass.
• The behavior is covered by deterministic automated tests.

[vypbot]

🪄 Automatic Actions

1. The issue was assigned to @​efraespada (creator).

2. The issue's title was updated from Throttle workflow queue polling and increase workflow timeouts to 🐛 - Throttle workflow queue polling and increase workflow timeouts.

3. The issue was linked to vypdev and moved to the column Todo.

4. Priority set to P0 in vypdev.

Recommended implementation steps

1. Confirm the queue’s current request pattern

   Instrument the queue and reproduce a burst with several active runs. Verify that ActivePreviousWorkflowRunsRepository currently performs one paginated query for each value in WORKFLOW_ACTIVE_STATUSES, resulting in multiple API requests every two seconds.

2. Reduce polling to one repository query

   Update src/data/repository/workflow/active_previous_workflow_runs_repository.ts to query the repository workflow-runs endpoint once per poll, using per_page: 100 and no single status filter:

   listWorkflowRunsForRepo({
     owner,
     repo,
     per_page: 100,
   });

   Filter locally for:

   • the configured COPILOT_WORKFLOW_NAMES;
   • active statuses;
   • run.id < currentRunId.

   Preserve pagination correctness so an older active run on a later page is never missed.

3. Add adaptive, staggered polling

   Replace the fixed two-second polling interval in WaitForPreviousWorkflowRunsUseCase with a centralized policy containing:

   • an initial delay;
   • exponential backoff while earlier runs remain active;
   • a maximum polling delay;
   • bounded jitter to prevent simultaneous queued runs from polling together;
   • an explicit maximum queue-wait duration.

   Keep provider failures fail-closed: an unsuccessful query must never be interpreted as zero active runs.

4. Handle rate limits using server-provided wait times

   Extend src/data/repository/workflow/workflow_runs_retry.ts to recognize:

   • HTTP 429;
   • HTTP 403 responses with a zero x-ratelimit-remaining value or a rate-limit message;
   • transient 5xx and network errors.

   For rate-limit responses, wait according to Retry-After or x-ratelimit-reset, falling back to exponential backoff when headers are unavailable. Treat rate-limit waits as controlled queue waits rather than immediate failures, bounded by the queue deadline.

5. Preserve sequential execution guarantees

   Retain the shared workflow scope in COPILOT_WORKFLOW_NAMES, the lower run-ID comparison, and all active statuses. Do not introduce GitHub Actions concurrency settings for mutation workflows, since they can cancel or replace queued runs.

   Add diagnostic logging for active-run counts, selected polling delays, and rate-limit reset times without logging credentials or response bodies.

6. Align workflow timeouts with the queue policy

   Update the queue-bearing jobs in both .github/workflows/ and setup/workflows/:

   • all Copilot issue, comment, pull-request, pull-request-comment, and commit jobs;
   • the tag jobs in release and hotfix workflows.

   For example, if the queue allows a 90-minute wait, configure those jobs for at least 120 minutes to leave execution headroom. Add the missing timeout to setup/workflows/copilot_commit.yml, and keep setup templates synchronized with the active .github workflows.

7. Add deterministic regression tests

   Update and extend the existing queue tests:

   • verify mixed active statuses are counted through a single provider query;
   • verify unrelated, current, newer, completed, cancelled, and skipped runs are excluded;
   • verify older matching runs on later pages are still detected;
   • verify 429 and rate-limited 403 responses honor Retry-After and reset headers;
   • verify fallback backoff, jitter, maximum delay, and queue deadlines using injected clocks/schedulers;
   • verify polling errors never allow execution to continue;
   • simulate multiple queued runs and assert that no run is skipped, cancelled, or treated as complete prematurely.

8. Strengthen workflow contract validation

   Extend scripts/validate-workflow-contract.cjs and its tests to assert that every queue-bearing mutation job:

   • has a timeout at least as long as the configured queue budget;
   • has no concurrency block;
   • retains the expected workflow names and runner contract.

9. Update documentation and run all quality gates

   Update docs/features.mdx, relevant development documentation, and the README concurrency section to describe adaptive polling, rate-limit handling, queue limits, and required workflow timeouts.

   Validate with:

   corepack pnpm@​10.12.4 run typecheck
   corepack pnpm@​10.12.4 run lint
   corepack pnpm@​10.12.4 run test:coverage
   corepack pnpm@​10.12.4 run build
   corepack pnpm@​10.12.4 run validate:workflows
   corepack pnpm@​10.12.4 run validate:agent-docs
   corepack pnpm@​10.12.4 run validate:docs-page
   git diff --check

10. Lifecycle state synchronized to planned.

Debug log

[DEBUG] Setup done. Issue number: 344, isSingleAction: false, isIssue: true, isPullRequest: false, isPush: false
[INFO] Running IssueUseCase for issue #344.
[INFO] 📋 Executing IssueUseCase.
[INFO] 🔐 Executing CheckPermissionsUseCase.
[DEBUG] Skipping permission enforcement because a mandatory branch is not required.
[INFO] 👤 Executing AssignMemberToIssueUseCase.
[DEBUG] #344 needs 1 assignees.
[DEBUG] Assigned issue creator @​efraespada to #344.
[INFO] ✏️ Executing UpdateTitleUseCase.
[DEBUG] Issue title updated to: 🐛 - Throttle workflow queue polling and increase workflow timeouts
[INFO] 🏷️ Executing UpdateIssueTypeUseCase.
[DEBUG] Setting issue type for issue 344 to Bug
[DEBUG] Fetched issue ID: I_kwDONSeBW88AAAABO-ujCg
[DEBUG] Successfully updated issue type to Bug
[INFO] 🔗 Executing LinkIssueProjectUseCase.
[DEBUG] Fetched issue ID: I_kwDONSeBW88AAAABO-ujCg
[DEBUG] Linked I_kwDONSeBW88AAAABO-ujCg with id PVTI_lADOA8wRO84Ajh7Bzg4u7qw to project PVT_kwDOA8wRO84Ajh7B
[DEBUG] Target field ID: PVTSSF_lADOA8wRO84Ajh7Bzgb4L2E
[DEBUG] Target option ID: f75ad846
[INFO] 📏 Executing CheckPriorityIssueSizeUseCase.
[DEBUG] Priority: priority: high
[DEBUG] Github Priority Label: P0
[DEBUG] Target field ID: PVTSSF_lADOA8wRO84Ajh7Bzgb4MPQ
[DEBUG] Target option ID: 4af6496f
[INFO] 🧹 Executing RemoveIssueBranchesUseCase.
[INFO] 🧹 Executing RemoveNotNeededBranchesUseCase.
[INFO] 🏷️ Executing DeployAddedUseCase.
[INFO] 🏷️ Executing DeployedAddedUseCase.
[INFO] 💡 Executing RecommendStepsUseCase.
[DEBUG] RecommendSteps: prompt length=4010, issue description length=2078.
[INFO] 🤖 Recommending steps using the configured agent...
[DEBUG] RecommendSteps: agent response received. Steps length=4575.
[INFO] Main run finished. Results: 8, total steps: 6.
[INFO] Publishing result: 9 result(s), 7 step(s), 0 error(s).
[INFO] 📄 Executing PublishResultUseCase.

🚀 Happy coding!

^{Made with ❤️ by vypdev/copilot}