Skip to content

feat(app): order a pull request's events; Redis-shared dedupe and locks for several replicas - #623

Merged
jeefy merged 1 commit into
cncf:mainfrom
jeefy:feat/app-server-ha
Oct 29, 2026
Merged

jeefy merged 1 commit into
cncf:mainfrom
jeefy:feat/app-server-ha

Conversation

@jeefy

@jeefy jeefy commented Oct 29, 2026

Copy link
Copy Markdown
Member

What

PR A of the Kubernetes install plan. Makes the App server correct on one replica and honest on several.

Bug fixed: the Node server had no per-pull-request ordering — dispatch was run directly. Two events for one PR arriving together (e.g. labeled + pull_request_review) raced the merge gate. The action's concurrency: group and the Worker's Durable Object both prevent this; the server now does too.

Changes

  • src/adapters/app/lock.ts — PullRequestLocks; memoryLocks() serialises calls per key; pullRequestKey moved here from the Worker.
  • src/adapters/app/redis.ts — redisDeliveryStore (SET NX EX, 24 h) and redisLocks (SET NX PX, token-released via compare-and-delete Lua; waits out a dead holder's TTL). node-redis 6 — ioredis is deprecated by Redis Ltd. for new code.
  • src/server/node.ts — Stores chosen by REDIS_URL (memory otherwise; warns when PROW_REPLICAS > 1 without Redis); locked dispatch; GET /readyz (503 until stores ready; /healthz stays liveness); in-flight tracking; SIGTERM/SIGINT drain with PROW_SHUTDOWN_GRACE_MS (default 25 s).
  • test.yml build-test gets a redis:7-alpine service; the real-Redis suite runs there (skipIf(!REDIS_TEST_URL) locally).

Topologies (docs/github-app.md → One replica or several)

one replica several
dedupe + per-PR lock in process Redis every replica shares
config nothing REDIS_URL

Keys used: prow:delivery:<id> (1 day), prow:lock:<owner/repo#n> (≤ 1 min). Any Redis 6+ / Valkey.

Tests

  • lock.test.ts — same key serialises in order, different keys concurrent, release on throw, key forgotten after last holder.
  • redis.test.ts — fake client (NX/TTL dedupe, hold/release by token, wait-then-take, never release another holder's lock, give up after waitMs, release on throw) plus a real server: cross-client dedupe, mutual exclusion between two clients, TTL recovery from a vanished holder. Verified locally against Redis 7 (8/8) and runs in CI via the service container.
  • server/stores.test.ts — storesFromEnv branches incl. the replicas warning and redis error logging; /readyz 503→200 and stuck-503; two deliveries for one PR: second PATCH starts only after the first finishes; no-issue lock key; main() SIGTERM drain → exit 0, grace-period give-up, exit 1 on store close failure.
  • npm run test:coverage: 225 files / 2671 tests, 100 / 99.78 / 100 / 100, per-path 100 % gates on src/adapters/app/** + src/server/** (test(server,app): execute src/server's entry point and bundle, pin #614's cron arms, and gate src/adapters/app/** + src/server/** at 100% (main.ts, node.ts:24/67/82, webhook.ts:65, cron.ts:85, vitest.config.mjs) #611) met. test:coverage:e2e 99.69 / 99.29 / 99.27 / 99.79. typecheck green. dist/ stable.

Dependencies

redis@^6 (runtime, App server only — not in the action bundle; selfContained.test.ts enforces). Lockfile resolves to registry.npmjs.org.

Next: PR B (ghcr image + chart publishing from release.yml), PR C (Helm chart).

…ugh Redis for several replicas

The Node server dispatched every delivery straight to run(): two events for
one pull request arriving together raced the merge gate, which the action's
concurrency: group and the Worker's Durable Object both prevent. And its
delivery memory was per process, so a second replica handled redeliveries
again.

- src/adapters/app/lock.ts: PullRequestLocks; memoryLocks() chains the calls
  for one key; pullRequestKey moves here from the Worker, which imports it.
- src/adapters/app/redis.ts: redisDeliveryStore (SET NX EX, a day) and
  redisLocks (SET NX PX, released by token through a compare-and-delete
  script; waits out a dead holder's ttl once). node-redis 6; Redis Ltd.
  deprecates ioredis for new code.
- src/server/node.ts: Stores { deliveries, locks, ready, close } chosen by
  REDIS_URL (memory otherwise, with a startup warning when PROW_REPLICAS > 1);
  dispatch runs under the pull request's lock, or the delivery id for an
  event about no issue; /readyz answers 503 until the stores are ready;
  background tracks in-flight deliveries and SIGTERM/SIGINT drain them
  (PROW_SHUTDOWN_GRACE_MS, default 25 s), close the stores and exit.

Tests: the memory lock (order, independence of keys, release on throw),
the Redis stores against a fake client and, under REDIS_TEST_URL, a real
server that build-test now provides as a service container (cross-client
dedupe, mutual exclusion, ttl recovery); the server's readiness states,
serialized handling of two deliveries for one pull request, the no-issue
lock key, and shutdown (drain, grace period, exit 1 on a store close
failure). docs/github-app.md gains "One replica or several".

Signed-off-by: Jeffrey Sica <me@jeefy.dev>
@jeefy
jeefy requested a review from jpmcb as a code owner October 29, 2026 22:25
@jeefy

jeefy commented Oct 29, 2026

Copy link
Copy Markdown
Member Author

/kind feature

@github-actions github-actions Bot added the kind/feature Categorizes issue or PR as related to a new feature. label Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@jeefy is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply.

@jeefy
jeefy merged commit f4760f9 into cncf:main Oct 29, 2026
6 checks passed
mrbobbytables pushed a commit that referenced this pull request Oct 31, 2026
…behaviour from #623 (#631)

main()'s close() cleared the cron interval and stopped the listener, and
memoryLocks only dropped a key's tail when it was still its own, without
any test noticing either: Stryker reported both calls, the pending > 0
guard, the `tails.get(key) === settled` comparison, redisLocks' ttlMs
default and createClient's url as survivors. Each is now asserted by a
test that fails with the statement removed or inverted.

Signed-off-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/feature Categorizes issue or PR as related to a new feature.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant