feat(reindex): add an inert, triple-gated reaper for abandoned index generations - #2552
feat(reindex): add an inert, triple-gated reaper for abandoned index generations#2552BigSimmo wants to merge 8 commits into
Conversation
…generations `cleanup_abandoned_document_index_generations` has exactly one caller (`npm run reindex:cleanup-staged`) and no schedule, so a crash mid-reindex leaks staged rows and storage objects that nothing revisits. This ships the DETECTION half plus an apply path that cannot fire without two independent switches. - `--alert-on-abandoned`: a read-only dry run that finds abandoned rows now exits non-zero so a scheduled probe can alert. The apply path and the interactive `confirm()` brake are untouched. The decision is a pure exported function, `abandonedReindexGenerationAlertExitCode`, unit-tested with no network. - `.github/workflows/reindex-reaper.yml`: `repository_dispatch` + a weekly cron, with no `workflow_dispatch`/`push`/`pull_request`/`pull_request_target`. The job is gated on `vars.REINDEX_REAPER_ENABLED`, which is deliberately unset, so merging this changes nothing and scheduled runs skip. Deletion additionally requires BOTH `github.event.client_payload.apply` AND `vars.REINDEX_REAPER_APPLY`; the scheduled branch is evaluated first, so cron can never reach `--apply`. Dry run is the default on every path. - `check-github-action-pins.mjs`: a static rule refusing any workflow that can invoke the reaper's apply path without both gates, so a later PR cannot quietly arm it. Comments are stripped before matching — an early draft passed a workflow whose payload gate had been replaced by a hardcoded `true` purely because the header prose still named it. Self-test covers single-gated, comment-only-gated, and correctly double-gated fixtures. The apply path is destructive well beyond what "cleanup" suggests: the RPC deletes generation-bearing artifact rows across seven tables for every tenant, with no owner scoping and no keep-newest fallback, and `p_limit` caps documents rather than rows. A document missing `metadata.index_generation_id` makes all of its artifact rows eligible. That is why nothing here is armed. This does NOT close #Q4Y7TR. The recorded fix there is for the commit RPC to write `storage_cleanup_jobs` rows for superseded generations at commit time; that is a migration, it deploys to the live clinical database on merge, and it belongs to a separate batch. No migration, schema, or SQL change is included here. Verified offline: check:github-actions (incl. self-test, plus a negative proof that removing one gate fails the rule), check:ci-scope, lint, typecheck, full unit suite (949 files, 12068 passed), and the docs index/link/script/inventory checks. No provider-backed command was run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sjekpEw82gMp57C8xzSxZ
…names The static rule added in f61f83b asserted only that the strings `github.event.client_payload.apply` and `vars.REINDEX_REAPER_APPLY` appeared somewhere in a workflow's non-comment lines. It never checked that they gated anything. A security review copied the two regexes into a harness and got past them six ways; three of those armed the destructive path outright, and one was certified as correct by the rule's own self-test. The rule now answers a structural question instead of a textual one. REACHABILITY. Inline `#` comments are stripped with quote awareness rather than whole-line, so a trailing comment on the apply line no longer supplies the gate names. Backslash continuations are joined, `env:` mappings and shell assignments are expanded, and npm aliases are resolved transitively out of package.json, so `--apply` behind a continuation, behind `$FLAGS`, or behind a renamed script is still found. `reindex:cleanup-staged` stays a hardcoded floor so deleting the alias cannot disable the rule. GATING. An apply invocation must sit lexically inside a shell conditional whose condition tests both gates; declaring the env vars and then deleting unconditionally is now a failure. A gate expression that can render "true" on its own (`|| 'true'`) is a defeated gate, not a satisfied one. The job-level `vars.REINDEX_REAPER_ENABLED == 'true'` gate is required, and any file that can reach the apply path is refused a `push`, `pull_request`, `pull_request_target` or `workflow_dispatch` trigger. Every unparsable shape fails closed, and the guard covers the class of workflows rather than the one path the contract test pins. Self-test fixtures are rewritten as A-I: trailing comment, line continuation, env indirection, npm alias, single gate, declared-but-not-gated, `|| 'true'` defaults, correctly gated (the only must-pass), and a manual trigger, plus the comment-only and missing-enable-gate cases. `--self-test --explain` prints each fixture's verdict. The workflow changes with it: - The scheduled alert could not tell "rows are leaking" from "the job broke": `failure()` fired on a failed `npm ci` or an unreachable Supabase just as it did on detection. The probe's exit code carries that distinction already (2 is detection, 1 is a real error), so the step now captures it via `$GITHUB_OUTPUT` under `continue-on-error`, the issue body branches on it, and a final step re-raises so a broken probe cannot report green. - Gate 1 is a GitHub expression and compares case-insensitively, while gates 2 and 3 were POSIX `=` compares, so arming with `TRUE` gave a silent dry run. Both are normalised through `is_true`, and a value that is neither true nor false now raises a `::warning::` instead of being swallowed. - The header documented gate 2 as `client_payload.apply == 'true'`, which is stricter than the shell string compare actually implemented; the prose now matches the code. Four limits are recorded in the header rather than coded around: two of the three gates are standing repository state and only `client_payload.apply` is per-run, so the safe way to arm the apply path is a GitHub Environment with required reviewers on a separate apply job (deliberately not referenced yet, since naming a missing environment auto-creates it unprotected); the probe will very likely be permanently red because documents missing `metadata.index_generation_id` register as abandoned forever; the RPC still has no owner scoping and no row ceiling; and the concurrency group serialises reaper against reaper only, not against a live reindex. A `docs/outstanding-issues-inbox` request records that only the detection half landed. Verified offline: check:github-actions (self-test passed, and the real workflow was re-checked after removing each gate in turn — all three negative proofs fire), check:ci-scope, the two reaper test files (16 passed), lint, typecheck, full unit suite (949 files, 12071 passed, 1 skipped), knip, and the docs index/inventory/scripts/links checks. No migration, no SQL, no provider-backed command, and the reaper was never run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sjekpEw82gMp57C8xzSxZ
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_6874f36a-9927-481b-8537-7a832496254e) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3e3674aa7d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex resolve actionable Codex review findings for this pull request and current head using the repository instructions. This is the pull request's single automatic repair pass: do not perform a fresh review, create new standalone findings, or request another review. Work only the existing unresolved Codex threads on the current head. The only repository destination is BigSimmo/Database, and the only branch destination is the pull request head branch claude/reindex-reaper at starting commit db0b026; never publish fixes to a detached or synthetic work branch and never create a stacked pull request. Use the authenticated GitHub connector to commit each approved fix to BigSimmo/Database:claude/reindex-reaper, then verify that the pull request head contains the published commit before reporting success. Always fix P0 and P1 findings. For P2 and lower findings, fix only clear, scoped, low-risk issues; otherwise disposition them with a concise reason. For a fixed thread, reply with as the first line and as the second line. For a no-code disposition, use followed by . These result markers authorize the workflow to close that exact thread only after it verifies a fixed commit is the pull request head; a local-only commit is not a fix. If publication or verification fails, do not use either result marker, do not claim success, and leave the thread open with the blocker. If human input or new authorization is required, do the same. Finish only after every actionable thread is fixed or dispositioned and closed, or explicitly left open for a human decision. Do not update the branch from main, address unrelated reviews, broaden scope, or create more than one scoped fix commit. Do not use external APIs, paid services, credentials, dependency changes, or broad refactors unless explicitly authorized. Add targeted tests where behavior changes and run the narrowest relevant validation. |
Summary
Testing
|
Codex review found the structural guard could still be walked past twice, both
verified against the real workflow rather than only fixtures.
A gate can be defeated by the SHELL around it, not only inside its `${{ }}`.
`is_true "$APPLY_REQUESTED" && is_true "$APPLY_ALLOWED" || true` names both
gates, satisfies both expression checks, and always enters the destructive
branch. `evaluateGate` reads GitHub expressions and cannot see that. The guard
SHAPE is now allowlisted instead: no top-level `||` (a disjunction is only as
strong as its weakest branch, which is the opposite of gating) and no always-true
term. Reported in place of the per-gate results, because "gate satisfied" printed
beside "the branch always runs" is the confusion this rule exists to remove.
A forbidden trigger can be written in YAML shapes the denylist did not parse.
Quoted flow-sequence items kept their quotes, so `on: ["repository_dispatch",
"workflow_dispatch"]` produced the name `"workflow_dispatch"` and
`has("workflow_dispatch")` missed it; flow mappings were not read at all. Trigger
names are now normalized across bare scalars, sequence items, quoted scalars and
mapping keys, and flow collections are split outside quotes and brackets so a
nested `[{ cron: "45 19 * * 0" }]` is not cut at its own commas.
Four new must-fail fixtures (J, M, K, L) cover the tautology, a constant conjunct
standing in for a gate test, and both trigger shapes. Each attack was also
applied to .github/workflows/reindex-reaper.yml itself and confirmed caught, then
reverted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015sjekpEw82gMp57C8xzSxZ
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_9ee58130-11b9-4422-93f3-f4a514320cf0) |
Both sides added a test to test:ci-workflows; kept both (browser-test-plan from main, reindex-reaper-workflow from this branch). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sjekpEw82gMp57C8xzSxZ
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_d2259042-5387-440f-9639-21bef795127a) |
Summary
cleanup_abandoned_document_index_generationshas exactly one caller,scripts/cleanup-abandoned-reindex-generations.tsvianpm run reindex:cleanup-staged. No cron, no Railway cron, no CI job references it. Any crash mid-reindex leaves staged rows under a never-committedindex_generation_id, plus storage objects, that nothing revisits — unbounded growth that depends on an operator remembering to run a janitor..github/workflows/reindex-reaper.yml, modelled oningestion-autopilot.yml, this repository's already-solved version of the same problem. Weekly schedule,repository_dispatch(neverworkflow_dispatch— that trigger lets a same-repository writer point a secret-bearing workflow at an arbitrary branch, andingestion-autopilot.ymlrecords that reasoning),concurrencywithcancel-in-progress: false,permissions: contents: readplusissues: write, project ref pinned inenv:,check:supabase-projectbefore any RPC call, digest-pinned actions throughout, and an open-or-update alert issue when the probe finds leaked rows.--alert-on-abandonedflag on the cleanup script: after a dry run, exit non-zero when abandoned rows are found, so a read-only probe can raise an alert. The decision is a pure function insrc/lib/reindex-pipeline.tsso it is unit-testable without touching Supabase. The apply path and the interactiveconfirm()brake are unchanged.--applyand then only asserted that the two gate strings appeared somewhere in the file. Five ways past it were demonstrated, including a trailing#comment on the same line — the exact trick the rule's own comment claimed to have fixed — and${{ … || 'true' }}defaults that fully arm both gates while leaving their names in place. Worse, its own self-test fixture declared "declare two env vars, delete unconditionally" to be correctly gated.env:and shell assignments expanded, and npm aliases resolved transitively out ofpackage.jsonso renaming the script cannot evade it. Gating: the apply invocation must sit lexically inside a shell conditional testing both gates; a gate expression that can rendertrueon its own counts as defeated, not satisfied; the job-level enable gate is required; andpush/pull_request/pull_request_target/workflow_dispatchare refused on any file that can reach the apply path. Unparsable shapes fail closed. Nine fixtures, eight of which must fail.No migration file is added and no SQL is applied to any database.
Why the destructive path stays behind three switches
The RPC is more destructive than "cleanup" suggests. Its delete predicate is "artifact row carries a generation id that differs from the document's current
metadata.index_generation_id", andis distinct from NULLis true for any non-null left side — so a document whose metadata key is missing has every generation-bearing artifact row eligible, across seven tables. There is no owner scoping and no keep-newest fallback, andp_limitcaps documents (up to 1000), not rows. Its real brakes are narrow:p_dry_rundefaults true, execute is granted toservice_roleonly, and there is a re-check for open ingestion jobs immediately before deleting.So deletion requires all three of:
vars.REINDEX_REAPER_ENABLED == 'true'— gates the whole job. Unset. Setting only this arms a read-only weekly probe.github.event.client_payload.apply == 'true'— per-run intent, reachable only viarepository_dispatch.vars.REINDEX_REAPER_APPLY == 'true'— standing repository consent. Unset.The shell evaluates the scheduled branch first and does not fall through, so cron reaches only
--alert-on-abandoned;--applyappears exactly once in the file, behind the branch requiring both 2 and 3.Merging this changes nothing. With
REINDEX_REAPER_ENABLEDunset the job-levelif:is evaluated before any step runs, so the weekly run is a skipped job with zero steps — no checkout, no install, nocheck:supabase-project, no RPC — and a dispatch is a no-op. Arming either switch is a repository-settings change only the owner can make, and nothing in this diff can make it.Verification
npm run verify:pr-localVerification not run: verify:pr-local was not invoked as a wrapper.Its constituent gates were run individually and are quoted below.Decisive output:
The rewritten rule was re-verified independently against the real workflow, not only against fixtures — replacing the dispatch gate with a hardcoded
trueis caught, and the message names the line and quotes the conditional it did find:Deleting the job-level enable gate and adding a
workflow_dispatchtrigger are each caught the same way; the file was restored after each probe.What was NOT verified. The workflow has never executed — verifying that would mean arming it against the live clinical database, which is exactly what this PR declines to do. The evidence is the contract test, the static rule, and a reading of the YAML.
npm run verify:ui— not applicable, no UI, routing, styling or browser behaviour changed.npm run verify:release— not claimed.npm run check:production-readiness— not run: provider-backed, and this batch does not contact Supabase. No runtime, deployment, or startup behaviour changes; the workflow is inert and the script flag only affects an exit code on the existing dry-run path.Risk and rollout
REINDEX_REAPER_ENABLEDwould begin a weekly read-only probe against the live Supabase project; armingREINDEX_REAPER_APPLYas well would permit deletion on an explicit dispatch. Both are explicit owner actions in repository settings and are not authorized by this PR.Clinical Governance Preflight
Clinical KB Database(sjrfecxgysukkwxsowpy)The project ref is pinned in the workflow's
env:andcheck:supabase-projectruns before any RPC call; the stale ref does not appear. The service-role key is referenced only as a repository secret inside the workflow and never leaves the runner, and the destructive RPC remainsservice_role-only. No clinical content, retrieval behaviour, source metadata, or decision-support behaviour changes, so the SaMD classification is unaffected.Notes
This does not close
#Q4Y7TR. The recorded fix there is for the commit RPC to enqueuestorage_cleanup_jobsrows for superseded generations at commit time, where it already knows both generation ids. That is a migration, and a migration merged here reaches the live clinical database within seconds, so it belongs to a separately approved batch. What lands here is the Postgres-row detection half plus a gated apply path; the storage-object half of the leak is untouched. That is stated in the workflow header, in the library doc comment, and in the commit message so it cannot be mistaken for a closed issue.Read this before arming it — the review findings recorded rather than coded around. They are the ones that matter for your decision.
"Three gates" is arithmetically true but two of them are standing state. Once
REINDEX_REAPER_APPLYis set it stays set, and the only per-run control left is arepository_dispatch, which needs justcontents: write— held by every collaborator token and by at least two agent-driven workflows in this repository. The genuinely safe way to arm the apply path is a GitHub Environment with required reviewers on a separate apply job. Noenvironment:key was added on purpose: naming an environment that does not exist yet makes GitHub auto-create it unprotected, which would read as safer than it is. Create the protected environment first; that ordering is stated in the workflow header.The weekly probe may well be permanently red. The RPC treats every generation-bearing row as abandoned on any document whose
metadata.index_generation_idis absent — so documents predating generation stamping, first ingests that crashed before commit, and any metadata edit dropping the key would register as abandoned forever. That is a weekly alarm with no way to clear it short of fixing the data, and the remediation it prompts is the destructive path. Worth confirming against the live counts before arming even the read-only switch. The safer shapes, if it turns out that way, are reporting counts to a summary and alerting only on a delta, or excluding metadata-key-absent documents until a migration backfills them.The apply path has no owner scoping and no row ceiling. Seven deletes in one transaction under a 180-second statement timeout. That is the RPC's pre-existing defect, not this diff's, but it is why arming should not be a bare repository variable: a migration adding a row ceiling and a keep-most-recent-generation guard should land before
REINDEX_REAPER_APPLYis ever set.Concurrency serialises reaper against reaper only. Nothing prevents a reaper run overlapping a reindex; the RPC's open-job re-check narrows that window rather than closing it.
Why
repository_dispatchrather than the manual button. Aworkflow_dispatchon a secret-bearing workflow lets a writer choose the branch, and therefore the workflow definition, that runs with the service-role key.ingestion-autopilot.ymlalready carries that reasoning in a comment; this file follows it. Worth noting for whoever next touches the autopilot: its apply branch is evaluated first, so it would apply on a scheduled run if both its gates were open. The reaper's schedule-first ordering avoids that. Not changed here.Refs
#Q4Y7TR— the detection half only.🤖 Generated with Claude Code
https://claude.ai/code/session_015sjekpEw82gMp57C8xzSxZ
Generated by Claude Code
Note
Low Risk
No runtime or database change on merge while repo variables stay unset; future risk is operational if the reaper is armed, because the workflow can call a cross-tenant destructive cleanup RPC with the service-role key.
Overview
Adds abandoned reindex generation detection that ships inert until
REINDEX_REAPER_ENABLEDis set: a newreindex-reaper.ymlworkflow (weeklyschedule+repository_dispatchonly) runsnpm run reindex:cleanup-stagedas a read-only probe and can open/update areindex-reaperissue when abandoned rows are found.The cleanup CLI gains
--alert-on-abandoned, backed byabandonedReindexGenerationAlertExitCodeinreindex-pipeline.ts(exit 2 on dry-run detection). Scheduled runs use that flag;--applystays behind three switches (enable var, dispatchclient_payload.apply, andREINDEX_REAPER_APPLY) with cron blocked from reaching apply.check-github-action-pins.mjsis expanded with a structural guard so workflows cannot reach the reaper apply path without both apply gates, the job enable gate, and allowed triggers—plus executable fixtures and workflow contract tests. An outstanding-issues inbox row records that storage object cleanup (storage_cleanup_jobs) is still out of scope.Reviewed by Cursor Bugbot for commit 64ede4a. Configure here.