ci(merge-queue): CircleCI-scheduled reconcile heartbeat, surviving Actions outages - #10576
Open
davidfirst wants to merge 2 commits into
Open
ci(merge-queue): CircleCI-scheduled reconcile heartbeat, surviving Actions outages#10576davidfirst wants to merge 2 commits into
davidfirst wants to merge 2 commits into
Conversation
…iving Actions outages
PR Summary by QodoAdd CircleCI merge-queue heartbeat to survive GitHub Actions outages
AI Description
Diagram
High-Level Assessment
Files changed (3)
|
Code Review by Qodo
1. Heartbeat runs can overlap
|
![qodo-free-for-open-source-projects[bot]](https://avatars.githubusercontent.com/in/1068000?s=60&v=4){kind=small}
Comment on lines
+1620
to
+1623
| - schedule: | ||
| # CircleCI's config-level cron rejects step syntax (*/10) — list the minutes explicitly | ||
| cron: '4,14,24,34,44,54 * * * *' | ||
| filters: |
There was a problem hiding this comment.
1. Heartbeat runs can overlap 🐞 Bug ☼ Reliability
The new CircleCI scheduled workflow can start a reconcile every 10 minutes with no serialization, so a slow/stalled run can overlap another run and race the merge-queue script’s GitHub status updates and update-branch mutations. GitHub Actions explicitly prevents overlap via a workflow concurrency group, but the CircleCI heartbeat adds an uncoordinated runner, making last-writer-wins updates and brief inconsistent gate states possible.
Agent Prompt
### Issue description
A new CircleCI schedule runs the merge-queue reconcile every 10 minutes, but there’s no guard against overlapping runs. The reconcile script performs multiple external mutations (posting commit statuses and updating PR branches), so concurrent executions can race and cause transient inconsistent state.
### Issue Context
GitHub Actions’ merge-queue workflow intentionally prevents overlap via `concurrency`, but CircleCI pipelines are not covered by that mechanism. Additionally, the script’s network calls use `fetch` without explicit timeouts, increasing the chance that a run exceeds the 10-minute schedule interval.
### Fix Focus Areas
- Add run-time limiting/serialization in CircleCI:
- .circleci/config.yml[736-745]
- .circleci/config.yml[1618-1629]
- Add explicit request timeouts (reduce hung runs):
- .github/scripts/merge-queue.js[91-112]
- .github/scripts/merge-queue.js[122-130]
### Suggested implementation directions
- In CircleCI, wrap the node invocation with a hard timeout (e.g., `timeout 8m node ...`) and/or add logic to skip if another heartbeat pipeline/workflow is currently running.
- In `merge-queue.js`, use `AbortController` (or equivalent) to enforce reasonable timeouts for GitHub/CircleCI API calls so the job can’t hang indefinitely and drift into overlap.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+742
to
+745
| exit 1 | ||
| fi | ||
| GITHUB_TOKEN=$GH_RELEASE_GITHUB_API_TOKEN node .github/scripts/merge-queue.js | ||
|
|
There was a problem hiding this comment.
2. Unvalidated github token 🐞 Bug ⚙ Maintainability
The new CircleCI merge_queue_reconcile job validates CIRCLE_TOKEN but does not validate GH_RELEASE_GITHUB_API_TOKEN before using it to set GITHUB_TOKEN, so a missing/empty value fails with a generic “GITHUB_TOKEN is required” error from the script. This makes the new scheduled heartbeat harder to diagnose and can leave the queue unreconciled until the misconfiguration is found.
Agent Prompt
### Issue description
`merge_queue_reconcile` exits early if `CIRCLE_TOKEN` is missing, but it does not check `GH_RELEASE_GITHUB_API_TOKEN` even though it is required to populate `GITHUB_TOKEN` for the script. When unset, the job fails with the script’s generic `GITHUB_TOKEN is required` error, obscuring which CircleCI project env var needs to be configured.
### Issue Context
The CircleCI job invokes:
`GITHUB_TOKEN=$GH_RELEASE_GITHUB_API_TOKEN node .github/scripts/merge-queue.js`
The script validates `process.env.GITHUB_TOKEN` and throws if missing.
### Fix Focus Areas
- .circleci/config.yml[739-745]
### Suggested implementation directions
- Add an explicit check:
- if `GH_RELEASE_GITHUB_API_TOKEN` is empty, print a clear message naming that variable and exit 1.
- Optionally, prefer `GITHUB_TOKEN` if already set, falling back to `GH_RELEASE_GITHUB_API_TOKEN`.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
…eue's own check runs
|
Code review by qodo was updated up to the latest commit c62ad96 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Today's GitHub Actions outage froze the merge queue completely: every reconcile failed before reaching the script (runners couldn't even download
actions/checkout), leaving green queued PRs unmergeable for everyone without admin bypass. CircleCI kept working throughout — it's independent infrastructure.merge_queue_heartbeatscheduled workflow (every 10 minutes on master, same config-level cron pattern asnightly) that runs the identical reconcile script from CircleCI. The script is stateless and idempotent, so overlapping runs with the Actions-side reconciles are harmless. This also replaces GitHub's throttled Actions cron as the dependable time-based fallback (20-40m silent gaps observed).CIRCLE_TOKENis now optional everywhere (used when present, e.g. to avoid shared-IP rate limits on hosted runners). A failed CircleCI read still fails the run — the queue never treats "couldn't check bit_merge" as settled.reconcile,ping) when judging a PR green: the outage failed those runs on PR head commits, which then blocked the PRs as "checks failing" — infrastructure artifacts, not PR verdicts.GITHUB_TOKEN=<pat> node .github/scripts/merge-queue.jsfrom any machine (used today to unstick the queue mid-outage).