Skip to content

Harden webhook ingestion against outsiders - #2097

Merged
ppXD merged 1 commit into
mainfrom
fix/harden-webhook-ingestion-against-outsiders
Oct 11, 2026
Merged

ppXD merged 1 commit into
mainfrom
fix/harden-webhook-ingestion-against-outsiders

Conversation

@ppXD

@ppXD ppXD commented Oct 9, 2026

Copy link
Copy Markdown
Owner

Summary

  • Who may start a run.
    • PR trigger payloads now carry authorAssociation, isFork and headRepositoryFullName. For GitLab, the author's project access level is looked up once per delivery; Developer and above counts as a member.
    • trigger.pr.opened and trigger.pr.updated gain an authors filter (any | members). Under members, new commits on a fork head must also be pushed by the author or by a member. Otherwise a member's PR opened from an outsider's fork would run the outsider's later pushes. GitLab's pusher is looked up; GitHub's cannot be, so another user's push to a fork head is refused.
    • An activation with no authors value (every existing one) admits only members when the delivery says the repository is public or internal, anyone when it says private, and only members for a fork PR when it doesn't say.
  • How often. Opened, reopened and synchronize events start one run per (activation, PR, head commit) within 60 s. A close/reopen loop collapses into one run, and a member's follow-up push always gets its own run. A delivery that already started its run is skipped before admission. Posting it again (Redeliver, or a captured body with its own id) cannot restart that window (RunSourceDispatcher.FireIfMatchesAsync).
  • Replay and audit.
    • GitHub deliveries without X-GitHub-Delivery are refused with 400.
    • A signed body is claimed per hook for 7 days, so the same body under a different delivery id starts nothing.
    • Refusals anyone can cause collapse to one row per hook and reason per day, with headers capped.
  • Size and rate.
    • Webhook bodies are capped at 25 MiB and callback bodies at 1 MiB, and the hook is resolved before its body is read.
    • Each remote address gets 10,000 ingress requests a minute, checked first as the app's GlobalLimiter. After that, 2,000 per (hook or token, address). The per-target key is chosen by the sender, so a sender rotating it used to get a new limiter on every request and was never limited.
    • The callback route accepts only the engine's 32-character token ({token:length(32)}).
  • Claims. Claims live in webhook_claim (migration 0244) and are taken in the delivery's transaction. Expired rows are deleted by WebhookClaimPurgeRecurringJob every 5 minutes, using SKIP LOCKED, in batches that each commit on their own. Before this, each delivery purged expired rows inside its own transaction, and two deliveries could deadlock (40P01).

Test plan

  • Unit: full suite, 12,923 passed. Includes the authors default and the pusher rule, the GitHub and GitLab pusher and head-SHA readers, the debounce key, the purge job and handler, and the recurring-job and authorization inventories.
  • Integration (Postgres):
    • admission flows: internal default, pusher (GitHub, and GitLab via a stub host), per-head debounce, a delivery posted again after the window;
    • claim store: concurrent claimants among 150 expired keys no longer deadlock, and the sweep steps around a row a delivery holds (a mutation without SKIP LOCKED goes red);
    • ingress contract pins and the sweep-command census.
  • E2E (real app host):
    • one source changing the target on every request gets 429 once its address budget is spent;
    • 8000-character callback tokens never reach the limiter and leave nothing behind;
    • a token the engine actually minted resumes its run over HTTP;
    • a member's second push within a minute runs;
    • an outsider's push to a member's fork PR on a public repository starts nothing;
    • the internal-repository default;
    • every registered recurring job, including the new sweep, completes a tick on a real Hangfire worker.
  • Frontend: pnpm lint, tsc --noEmit -p tsconfig.app.json, vitest (2,366 passed), pnpm build.
  • Staging: watch the author_not_member and pull_request_debounced refusal rates on public and internal repositories, and 429s on /api/webhooks/* behind the production proxy.

Anyone who could reach the anonymous webhook endpoints could start runs,
replay deliveries and bury the refusal log:

- A fork PR from someone outside a public repository started a run like
  a member's, with nothing in the payload to tell them apart, and every
  close/reopen started another run.
- GitHub signs the body only, so a captured delivery posted again under
  a fresh (or no) X-GitHub-Delivery started new runs for as long as the
  secret was valid.
- Every forged-signature post wrote its own audit row with unbounded
  headers, so fifty of them pushed every genuine refusal off the panel.
- The whole body was read before the hook was looked up, with no size
  or rate limit on the webhook and callback endpoints.

PR trigger payloads now carry authorAssociation, isFork and
headRepositoryFullName. GitLab sends no standing, so the author's
project access level is looked up once per delivery (Developer and
above is a member). trigger.pr.opened and trigger.pr.updated gain an
'authors' filter (any | members). Members means whoever made the code
what it is: the author, and for new commits on a fork head whoever
pushed them. A member can open a PR from an outsider's fork branch, and
the outsider's later pushes must not run on the member's standing.
GitLab's pusher is looked up like the author; GitHub has no lookup
here, so another user's push to a fork head is not admitted.

Existing activations set no 'authors'. They admit only members when the
delivery says the repository is public or internal, anyone when it says
private, and, when it does not say, only members for a fork PR.
Internal counts with public because every user of a GitLab instance or
GitHub enterprise can fork it and open a PR. So existing activations on
public and internal repositories stop running outsiders' PRs; private
ones behave as before.

A run holds off the next one from the same activation for the same PR
at the same head commit for 60 seconds. A close/reopen loop re-sends a
head that already ran; a push moves the head, so a member's follow-up
push always gets its own run. A delivery that already started its run
is skipped as a duplicate before admission, so posting it again (an
operator's Redeliver, or a captured body with its own id) cannot
restart that window.

GitHub deliveries without X-GitHub-Delivery are refused with 400, and a
signed body is claimed per hook for 7 days, so the same body under
another id starts nothing while a redelivery, which keeps its id, still
reaches dispatch. Refusals anyone can cause collapse to one row per
hook and reason per day, with headers capped.

Webhook bodies are capped at 25 MiB and callbacks at 1 MiB, and the
hook is resolved before its body is read. Each remote address gets
10,000 ingress requests a minute across every hook and callback, then
2,000 per (hook or token, address). The address budget is checked
first because the per-target key is a value the sender chooses: a
sender changing it on every request was never limited and left a
limiter behind each time. The callback route admits only the engine's
32-character token shape, so a key can no longer be kilobytes long.

Claims live in a new webhook_claim table (migration 0244), taken inside
the delivery's transaction so a delivery that rolls back releases them.
Expired claims are taken over in place and deleted by a recurring sweep
every five minutes that skips rows a delivery holds; purging them
inside a delivery held their locks until it committed, and two
deliveries could deadlock. Activations are visited in id order so
concurrent deliveries take their debounce claims in the same order.
@ppXD
ppXD merged commit a0eb909 into main Oct 11, 2026
8 checks passed
@ppXD
ppXD deleted the fix/harden-webhook-ingestion-against-outsiders branch October 11, 2026 07:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant