Repository navigation
feat(app): order a pull request's events; Redis-shared dedupe and locks for several replicas - #623
Merged
Merged
Conversation
…ugh Redis for several replicas
The Node server dispatched every delivery straight to run(): two events for
one pull request arriving together raced the merge gate, which the action's
concurrency: group and the Worker's Durable Object both prevent. And its
delivery memory was per process, so a second replica handled redeliveries
again.
- src/adapters/app/lock.ts: PullRequestLocks; memoryLocks() chains the calls
for one key; pullRequestKey moves here from the Worker, which imports it.
- src/adapters/app/redis.ts: redisDeliveryStore (SET NX EX, a day) and
redisLocks (SET NX PX, released by token through a compare-and-delete
script; waits out a dead holder's ttl once). node-redis 6; Redis Ltd.
deprecates ioredis for new code.
- src/server/node.ts: Stores { deliveries, locks, ready, close } chosen by
REDIS_URL (memory otherwise, with a startup warning when PROW_REPLICAS > 1);
dispatch runs under the pull request's lock, or the delivery id for an
event about no issue; /readyz answers 503 until the stores are ready;
background tracks in-flight deliveries and SIGTERM/SIGINT drain them
(PROW_SHUTDOWN_GRACE_MS, default 25 s), close the stores and exit.
Tests: the memory lock (order, independence of keys, release on throw),
the Redis stores against a fake client and, under REDIS_TEST_URL, a real
server that build-test now provides as a service container (cross-client
dedupe, mutual exclusion, ttl recovery); the server's readiness states,
serialized handling of two deliveries for one pull request, the no-issue
lock key, and shutdown (drain, grace period, exit 1 on a store close
failure). docs/github-app.md gains "One replica or several".
Signed-off-by: Jeffrey Sica <me@jeefy.dev>
Member
Author
|
/kind feature |
Contributor
|
@jeefy is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply. |
mrbobbytables
pushed a commit
that referenced
this pull request
Oct 31, 2026
…behaviour from #623 (#631) main()'s close() cleared the cron interval and stopped the listener, and memoryLocks only dropped a key's tail when it was still its own, without any test noticing either: Stryker reported both calls, the pending > 0 guard, the `tails.get(key) === settled` comparison, redisLocks' ttlMs default and createClient's url as survivors. Each is now asserted by a test that fails with the statement removed or inverted. Signed-off-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
PR A of the Kubernetes install plan. Makes the App server correct on one replica and honest on several.
Bug fixed: the Node server had no per-pull-request ordering —
dispatchwasrundirectly. Two events for one PR arriving together (e.g.labeled+pull_request_review) raced the merge gate. The action'sconcurrency:group and the Worker's Durable Object both prevent this; the server now does too.Changes
src/adapters/app/lock.ts—PullRequestLocks;memoryLocks()serialises calls per key;pullRequestKeymoved here from the Worker.src/adapters/app/redis.ts—redisDeliveryStore(SET NX EX, 24 h) andredisLocks(SET NX PX, token-released via compare-and-delete Lua; waits out a dead holder's TTL). node-redis 6 — ioredis is deprecated by Redis Ltd. for new code.src/server/node.ts—Storeschosen byREDIS_URL(memory otherwise; warns whenPROW_REPLICAS > 1without Redis); locked dispatch;GET /readyz(503 until stores ready;/healthzstays liveness); in-flight tracking;SIGTERM/SIGINTdrain withPROW_SHUTDOWN_GRACE_MS(default 25 s).test.ymlbuild-testgets aredis:7-alpineservice; the real-Redis suite runs there (skipIf(!REDIS_TEST_URL)locally).Topologies (docs/github-app.md → One replica or several)
REDIS_URLKeys used:
prow:delivery:<id>(1 day),prow:lock:<owner/repo#n>(≤ 1 min). Any Redis 6+ / Valkey.Tests
lock.test.ts— same key serialises in order, different keys concurrent, release on throw, key forgotten after last holder.redis.test.ts— fake client (NX/TTL dedupe, hold/release by token, wait-then-take, never release another holder's lock, give up afterwaitMs, release on throw) plus a real server: cross-client dedupe, mutual exclusion between two clients, TTL recovery from a vanished holder. Verified locally against Redis 7 (8/8) and runs in CI via the service container.server/stores.test.ts—storesFromEnvbranches incl. the replicas warning and redis error logging;/readyz503→200 and stuck-503; two deliveries for one PR: second PATCH starts only after the first finishes; no-issue lock key;main()SIGTERM drain → exit 0, grace-period give-up, exit 1 on store close failure.npm run test:coverage: 225 files / 2671 tests, 100 / 99.78 / 100 / 100, per-path 100 % gates onsrc/adapters/app/**+src/server/**(test(server,app): execute src/server's entry point and bundle, pin #614's cron arms, and gate src/adapters/app/** + src/server/** at 100% (main.ts, node.ts:24/67/82, webhook.ts:65, cron.ts:85, vitest.config.mjs) #611) met.test:coverage:e2e99.69 / 99.29 / 99.27 / 99.79.typecheckgreen.dist/stable.Dependencies
redis@^6(runtime, App server only — not in the action bundle;selfContained.test.tsenforces). Lockfile resolves to registry.npmjs.org.Next: PR B (ghcr image + chart publishing from
release.yml), PR C (Helm chart).