Skip to content

Per-request admission guard for the scan worker - #10

Merged
CompleteDotTech merged 7 commits into
mainfrom
feat/request-admission-guard
Oct 23, 2026
Merged

CompleteDotTech merged 7 commits into
mainfrom
feat/request-admission-guard

Conversation

@CompleteDotTech

@CompleteDotTech CompleteDotTech commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Refs CompleteTech-LLC/complete.tech#17 (step 1, part 2: per-request admission guard). Scanning stays disabled by default; nothing here enables it or provisions a worker.

What changes

Before every connection to the customer's site the sandboxed scanner (uid 65534, no key) asks the root supervisor over a private socket pair; the supervisor calls a new signed POST /ct-scan/v1/admit; WordPress decides under the queue lock that also serializes customer Stop, operator revocation, expiry, claims, starts and results:

  • job running, lease and scope hash match, lease unexpired, recipient confirmed, saved authorization still reads back unchanged/unexpired/unrevoked, frozen profile known;
  • previous grant at least 1 s old (central 1 req/s pacing); budget (r2-header-v1: 20 requests = robots.txt <=6 hops + homepage <=6 hops + one http->https change) not exhausted;
  • the reservation + time are written in one compare-and-set update. Grant = reserved and consumed at once, valid 1 s, one connection.

Supervisor/scanner: no grant => no connection (PinnedOpener asks after DNS/public-IP validation and before any socket, re-checks freshness immediately before connect, reports done); one grant outstanding; sequence-numbered request/reply and a poisoned channel on any error; lifetime and 10 s bound measured from just before the granting call; supervisor kills the scanner on refusal, protocol violation, request_deadline_exceeded, or when a 2 s liveness probe (own thread, also during backoff) reports cancelled/expired/revoked/integration-disabled. SubprocessRunner.run refuses to start without a gate; the scanner refuses to run without the socket. Schema 3 (profile_id, http_admitted, last_admit_ms); Stop and operator revoke now wait up to 3 s for the lock.

Review with git diff -w: runner.py is mostly re-indentation (try/finally to close the socket).

Fail-closed behavior

Unreachable/slow/garbled guard, 4xx/5xx, wrong schema, disabled integration, unknown profile, clock moving backwards, unconfirmed recipient, lost compare-and-set, thrown exception in the locked region, expired grant, out-of-sequence/oversized/unknown message, missing gate, missing socket: all are refusals that end the run as a failed receipt (admission_<reason>). Never a clean result, no automatic rescan, no budget reset by recovery/duplicate delivery/schema upgrade (a job running at the 2->3 upgrade gets no fresh budget).

Tests (all local; no third-party target)

  • test_admission.py (new, 52): gate, pump state machine, child client, desync, opener guard, real scanner entry point, real child process + local server for: budget stop, cancel before start / during an in-flight request / during backoff, deadline kill, protocol violation.
  • integrations/wordpress/tests/admission.php (new): cancellation/admission race in both orders and under real lock contention (second process holds the lock; Stop waits then wins), operator revocation, expiry, lease lapse, budget, pacing, clock anomaly, wrong lease/scope, unconfirmed, unknown profile, changed authorization, duplicate delivery, compare-and-set loss, injected DB fault, schema upgrade, disabled kill switch, store unavailable. Helpers extracted to tests/support.php.
  • deployment/test-http-worker.py: real signed HTTP round trip of admit/probe/pacing/foreign lease/finished job.
  • Root-only: real uid-65534 child speaks the protocol over the inherited fd (CI container step now also runs test_admission); packaged smoke test gets a gate.
  • Locally: Python 118 OK (4 root-only skipped, ran in CI); WordPress suites against a disposable MariaDB 11.4 + WordPress 6.8.3; HTTP test against that WordPress. CI on the head: green (ubuntu+windows, py3.10/3.14, WordPress/Docker job).
  • Independent adversarial and acceptance/mutation reviews found a request/reply desync (fixed with sequence ids + poisoning), refusals reported as completed (fixed), unbounded blocking probe (own thread, 4 s budget), lifetime measured from receipt (fixed), maintenance under the lock (removed), upgrade budget (fixed), and several test gaps (added).

Reviewer: please attack

  1. Any path that opens a socket to the target without a fresh grant (DNS-before-admit ordering, redirects/robots, TLS, exception paths skipping done).
  2. Lock semantics: GET_LOCK lost on a silent $wpdb reconnect (CAS is the backstop); Stop/revoke wait of 3 s vs admission 0 s.
  3. Child can ignore the protocol and open its own sockets: not preventable here; needs host/container egress filtering (deployment/compose.yaml does not enforce it). Supervisor still kills by deadline/probe.
  4. Admission answers are TLS-protected but not signed.
  5. In-flight bound: probe interval 2 s + latency; DNS lookups of the authorized host occur before admission.

Not done / needs owner

  • The Stop page still says "a check that has already started may finish its current requests" (accurate; now at most one request). Updating customer-facing copy is left to the owner.
  • Vendoring into complete.tech (mu-plugins/ct-scan.php, docs/php2shell-source.json) is a separate excluded PR held for the owner.

Second independent review (after the first round of fixes)

Re-verified the earlier fixes and found: (high) a slow-but-successful WordPress round trip made the delivered grant stale and the pump killed the whole scan -> now a grant older than 0.6 s is discarded and asked for again (at most twice; the slot stays spent), then refused; (medium) the in-flight probe was never joined and the admission call could outlive a shutdown or the scan deadline -> the runner waits for the probe, the gate honours the scan deadline, and a shutdown interrupts a slow call. Accepted/documented: an abandoned guard call can still land at WordPress after the supervisor gave up (the slot is simply spent); probe/admit slugs are prefixed by the pump.

Fail-closed review round 2 (coordinator findings)

Fixed in this PR, each with tests that fail without the fix (mutation-checked):

  • Activation requires egress_enforced: true (loader refuses otherwise); docs/README/example config state it; unverified nftables egress sketch added (not a claim of enforcement). In-flight bound is exactly 10 s from the grant (no grace). Authorization read from the DB, not the object cache. A completed result needs >= 1 admitted request. Failed schema upgrade: one error_log, 5-minute back-off, plugin 0.2.0. Special-purpose IP ranges listed explicitly (CGNAT, documentation, benchmarking, 3fff::/20, 6to4, NAT64, ...), tested independent of the interpreter's ipaddress.

Review round 3

  • A probe that hit the busy queue lock (queue_busy, often the supervisor's own concurrent admission racing for the no-wait lock) or an internal admission_error killed the scan. Such a probe is now retried briefly within one call budget and then treated as inconclusive, like an unreachable guard; the next admission still fails closed, and explicit refusals stay fatal.
  • The stale-grant re-ask path now honours the four-second admission deadline, so one admission stays within about eight seconds as documented.

CompleteDotTech and others added 4 commits October 22, 2026 06:51
The sandboxed scanner holds no key, so before every connection to the
customer's site it now asks the root supervisor over a private socket pair.
The supervisor calls a new signed `admit` route; WordPress decides under the
same queue lock that serializes customer Stop, operator revocation, expiry,
claims, starts and results. It checks job state, lease, recipient
confirmation, saved authorization, the frozen profile, one-per-second pacing
and the request budget, and records the reservation in the same
compare-and-set update as the decision. A grant is consumed when issued, is
valid for one second and covers one connection.

The supervisor kills the scanner on any refusal, protocol violation, or
request running past the disclosed ten-second bound, and probes liveness
every two seconds (including during backoff) so a Stop aborts the one request
already admitted. SubprocessRunner.run refuses to start without a gate, and a
scanner without the supervisor socket refuses to run. Unreachable guards,
malformed answers and unknown profiles are refusals.

Schema 3 adds profile_id, http_admitted and last_admit_ms (existing rows
default to the r2-header-v1 profile with empty counters). Customer Stop and
operator revocation now wait up to three seconds for the queue lock so a
concurrent admission cannot make them fail. Scanning remains disabled by
default; the profile limits are code constants, not configuration.

Refs CompleteTech-LLC/complete.tech#17 (step 1, part 2).

Co-Authored-By: Claude Code <noreply@anthropic.com>
…or test elsewhere

Co-Authored-By: Claude Code <noreply@anthropic.com>
- Sequence-numbered requests and replies; any channel error poisons the
  scanner's channel, so a late grant can never answer a later request.
- Refused admissions now end the run as failures (AdmissionRefused is no
  longer an OSError the checker swallows into a "completed" result).
- Measure the one-second lifetime and ten-second bound from just before the
  call WordPress granted; bound every supervisor call to WordPress to four
  seconds; run liveness probes on their own thread so a slow store cannot
  delay the in-flight deadline or the kill.
- A disabled integration (operator kill switch) reported by a probe now
  stops the scanner; only an unreachable guard is non-fatal for the request
  already admitted.
- Admission no longer runs table-wide maintenance under the queue lock;
  expiry and lease are checked on the row.
- A job already running at the schema 2 to 3 upgrade gets no fresh budget.
- Tests: real scanner entry point, compare-and-set loss, injected database
  fault, operator-revocation lock wait, upgrade budget, desynchronisation.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
CompleteDotTech and others added 2 commits October 22, 2026 07:40
…can deadline

Independent re-review of the admission guard found that a slow but successful
WordPress round trip (about one second) made the delivered grant stale and the
pump then killed the whole scan. The gate now discards a grant older than 0.6 s
and asks again (bounded, slots stay spent), refuses after the scan deadline,
lets shutdown interrupt a slow guard call, and the runner waits for an in-flight
probe before releasing the API client.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…eview fixes

- Config loader refuses to enable unless egress_enforced is literally true (the guard is
  advisory to a scanner that ignores it); docs, README, example config and an unverified
  nftables egress sketch state that activation requires enforced egress.
- The in-flight bound is the disclosed ten seconds from the grant, with no extra grace.
- Authorization is read from the database, not the object cache.
- A completed result requires at least one admitted request.
- A failed schema upgrade is logged once and not retried for five minutes; plugin version 0.2.0.
- The destination check lists special-purpose ranges itself (CGNAT, documentation, benchmarking,
  3fff::/20, 6to4, NAT64, ...) instead of relying on the interpreter's ipaddress properties.

Co-Authored-By: Claude Code <noreply@anthropic.com>
… the admit deadline

A liveness probe that met the no-wait queue lock (queue_busy), often held by
the supervisor's own concurrent admission, or an internal admission_error,
killed the scanner and spent the customer's daily scan. The probe now retries
a busy lock briefly within one call budget and the pump treats queue_busy and
admission_error like an unreachable guard: the admitted request keeps its
ten-second bound and the next admission still fails closed. Explicit refusals
remain fatal.

The stale-grant re-ask path now checks the four-second admission deadline, so
one admission stays within about eight seconds as documented.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@CompleteDotTech
CompleteDotTech marked this pull request as ready for review October 23, 2026 19:38
@CompleteDotTech
CompleteDotTech merged commit c1bb1a9 into main Oct 23, 2026
10 checks passed
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-23T19:38:58.156859Z 0bd4510 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-23T19:40:35.546189Z 0bd4510 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

CompleteDotTech added a commit that referenced this pull request Oct 23, 2026
Brings in the #10 and #11 review fixes (probe busy/inconclusive handling,
bound-evidence pair comparisons, duplicate evidence refusal, accepted-copy
live check).

Resolutions:
- admission.py probe(): keep main's bounded queue_busy retry and use this
  branch's per-kind route ("probe" for http, "control_probe" for control).
  Test added for the control route.
- ct_scan_send_accepted_copy(): main's live-job check allowed only
  queued/leased/running, but confirmation now leaves control-required jobs
  in awaiting_control and sends the copy then, so the default path would
  never send it. awaiting_control is accepted as live; every other check
  (authorization live, recipient, terms hash, compare-and-set) is kept.
  evidence-strict.php covers confirmation into awaiting_control; docs updated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CompleteDotTech added a commit that referenced this pull request Oct 23, 2026
Brings in the merged #10/#11/#12 review fixes (probe busy-retry, list-of-pairs
evidence comparison, control_admitted and 409 already_admitted, fail-closed
ct_scan_control_required(), terms_policy docs) under the retention branch.

Conflict: the schema's required jobs columns now list both control_admitted
and the retention/hold/identity columns. The schema version moves to 7 so an
install that recorded 6 before control_admitted existed re-runs the upgrade.

Retention coherence with the control gate: awaiting_control (pending or
checking) is not terminal and is never closed; a restored row recognised by a
tombstone is no longer deleted while it holds a lease (leased/running scan or a
control check in progress), so the one-at-a-time gate is never freed while its
traffic may be in flight. Tests cover both.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant