ADR-047: operator skip rules

docs/adr/047-operator-skip-rules.md

@@ -0,0 +1,141 @@
+# ADR-047: Operator skip rules
+
+## Status
+
+Proposed. Needs one migration, numbered when it lands, because the next numbers are contested by open PRs. Builds on [ADR-038](038-queue-runtime-overrides.md) runtime overrides and on the poison-job bounds from #546.
+
+## Context
+
+A poison job is one whose handler panics, hangs or crashes its worker. #546 bounded what one costs: a panic is a failed attempt, and heartbeat and deadline rescues now count against `max_attempts`. A bounded poison job can still do harm while it spends those attempts. Each crash takes down every job running in that process, and a kind with many bad jobs keeps a queue busy failing them.
+
+When an operator spots the pattern, for example `ResizeImage` jobs whose `args.width` exceeds 20000, the controls they have are:
+
+| Control | Scope | Gap |
+| --- | --- | --- |
+| Queue pause (`awa queue pause`, `queue_meta.paused`) | Every job on the queue | Stops healthy kinds sharing the queue |
+| Cron pause | One schedule | Only cron-produced jobs |
+| Batch cancel or move ([ADR-030](030-batch-operations.md)) | Matching jobs that exist now | Misses jobs enqueued afterwards, and cannot see running ones |
+| DLQ purge or retry ([ADR-020](020-dead-letter-queue.md)) | Jobs already failed | After the damage |
+| A feature flag in the handler | Whatever the code checks | Needs a deploy, and runs after argument decoding, which may be what crashes |
+
+The job-lifecycle docs answer "stop a kind" with "put risky kinds on their own queue and pause it". That works only if the queue layout anticipated the problem. What is missing is a standing rule, effective within seconds and without a deploy: *don't run jobs of this kind, optionally only those matching this predicate, until I say so.*
+
+## Decision
+
+Add **skip rules**: operator-managed rows that the worker checks after it claims a job and before it runs the handler.
+
+### Rule shape
+
+A small control-plane table, `awa.job_skip_rules`, holds one row per rule:
+
+| Column | Meaning |
+| --- | --- |
+| `id` | Identity, shown in logs, metrics and the job's error history |
+| `kind` | Required. Rules are always scoped to one kind |
+| `queue` | Optional; `NULL` means every queue |
+| `expression` | Optional CEL predicate. `NULL` matches every job of the kind |
+| `action` | `hold`, `dead_letter` or `cancel` |
+| `reason` | Required free text, shown wherever the rule's effect is |
+| `created_by`, `created_at` | Audit |
+| `expires_at` | Optional. An expired rule stops matching |
+| `disabled_at` | Set when the rule is removed. Rows are kept for audit |
+
+Operators change rules rarely, and nothing on the job path writes to the table, so it is not a hot mutable table.
+
+### Matching
+
+Workers load the enabled, unexpired rules on the same slow refresh as queue overrides (ADR-038, 10 s). They compile each expression once per refresh and index the rules by kind. For each claimed job:
+
+1. **No rule for its kind.** One hash lookup, then the job runs as today.
+2. **A rule matches.** Its `queue` is `NULL` or equal to the job's queue, and its expression is absent or evaluates to `true`. The worker applies the rule's action instead of running the handler. If several rules match, the most restrictive action wins: `cancel` over `dead_letter` over `hold`.
+
+The CEL environment is the job's `args`, `metadata`, `kind`, `queue`, `attempt`, `priority` and `tags`. It is evaluated on the worker with the CEL support that callback filters already use. A rule whose expression fails to compile is rejected when it is created.
+
+An expression that errors when evaluated (for example `args.width` on a job without a `width`) **does not match**. The job runs as it would have without the rule. The error is counted in `awa.job.skip_rule_errors{rule}`. Each worker also keeps a per-rule error count and the latest message in the runtime snapshot it already publishes on its heartbeat, so the CLI and admin UI can show them. The worker never writes to the rules table, however often an expression fails. The other choice, treating an error as a match, would let one malformed expression quarantine healthy jobs. CEL's `has()` lets a rule author handle absent fields explicitly.
+
+### Actions
+
+Each action reuses an existing outcome path. None is a new kind of state transition.
+
+- **`hold`, the default.** The job is snoozed: rescheduled without spending an attempt.
+  - **Duration.** It is held until the rule's `expires_at` or for 24 hours, whichever is sooner, not for a short interval. A short interval would re-claim every held job each cycle, costing one claim and one snooze transaction per job per interval. A held job that reaches its horizon while the rule is still active is claimed and held again, at most once a day.
+  - **Ownership.** The snooze's transaction also writes a row to a second small table, `awa.job_skip_holds (rule_id, job_id, held_at)`, with `ON CONFLICT DO NOTHING` for a re-hold. Ownership stays out of the job's metadata, which may be any JSON value, not only an object.
+  - **Release.** When a rule is removed, `awa job skip remove` releases its jobs with one batch operation (ADR-030). The operation sets `run_at = now()` for the scheduled jobs listed under that rule and deletes their rows.
+  - **Cleanup.** Maintenance deletes holds whose job is no longer scheduled, for example because an admin cancelled it.
+- **`dead_letter`.** The job moves to the dead-letter queue with reason `skipped`, keeping its args so a DLQ retry can re-run it after a fix. The attempt is spent: the job did not run, but it was not deferred either.
+- **`cancel`.** The job is cancelled with reason `skipped by rule <id>: <reason>`.
+
+Hooks and registered follow-ups behave as if the handler had returned the corresponding result. A hold is a snooze, so nothing fires. `dead_letter` fires `Exhausted` and `on_exhausted_enqueue`. `cancel` fires `Cancelled` and `on_cancelled_enqueue`. A compensation registered for "this job will not run" therefore runs whichever way that came about. The handler never runs, so nothing is staged (ADR-046).
+
+A rule never touches an attempt that is already running. To stop those as well, cancel them with the admin cancel, which signals the running handler.
+
+### Operator surface
+
+- **CLI:** `awa job skip add --kind K [--queue Q] [--where EXPR] [--action hold|dead_letter|cancel] --reason R [--expires-in D] [--dry-run]`, plus `list` and `remove`.
+- **Preview.** `--dry-run`, and the confirmation step without it, report how many available, scheduled and retryable jobs the rule would match. ADR-030 rejected CEL as a filter for bulk mutation because counts must come from the database. Here the preview narrows by `kind`, `queue` and `state` in SQL, then evaluates the expression on at most 10,000 of those rows. Past that cap it reports "at least N". Enforcement is per claimed job and never depends on the preview, so an approximate count cannot widen what the rule does.
+- **Admin UI:** the rules list, each rule's matches and held count, and a remove button. It uses the same API as the CLI.
+- **Metrics:**
+  - `awa.job.skipped{kind, queue, action, rule}`;
+  - `awa.job.skip_rule_errors{rule}`;
+  - a held-jobs gauge per rule, counted from `awa.job_skip_holds` on the rules refresh, so a hold nobody is watching shows up on a dashboard.
+
+### Rollout
+
+Phase 1 ships rules with `kind`, `queue` and an action, without expressions. That covers "stop this kind on a shared queue" with no CEL dependency. Phase 2 adds `expression`, gated by the `cel` feature that callback filters already use. A worker built without that feature does not match rules that have an expression, and it logs a warning once per rule.
+
+In a mixed fleet, workers older than the migration ignore rules. `awa job skip add` warns when live runtimes report an older version, using the runtime snapshot that the admin overview already shows.
+
+## Fit with the rest of Awa
+
+- **Control plane in the database ([ADR-038](038-queue-runtime-overrides.md)).** Rules follow the queue-override pattern: a small table, read on a slow refresh, applied by every worker without a deploy.
+- **The claim path is untouched ([ADR-033](033-per-key-execution-control.md)).** ADR-033 rules out skipping a row at claim time "as if it were claimed", because that breaks the cursor-monotonicity and prune proofs. A skipped job here is claimed normally and then finalized through the snooze, DLQ or cancel path, each of which the storage models already cover. The cost is one claim per skipped job, paid only by jobs that match a rule.
+- **CEL ([ADR-030](030-batch-operations.md)).** Enforcement evaluates CEL against one job at a time, as callback filters do, which is the use ADR-030 called CEL good for. Only the preview reads in bulk, and it is bounded in SQL by kind and capped, as ADR-030 requires.
+- **Not a workflow engine.** A rule decides whether a job runs at all, the same decision a queue pause makes at a finer grain. It does not decide which work follows, and it keeps no relation between jobs, so it passes the test in ADR-046 ([#556](https://github.com/hardbyte/awa/pull/556)). Rules are an operator control with an audit trail and an expiry. They are not a routing API: they are not offered on `ClientBuilder` or `InsertOpts`.
+
+## Consequences
+
+### Positive
+
+- An operator can quarantine a poison pattern within seconds of spotting it, scoped to one kind and optionally to the arguments that trigger it, without pausing healthy kinds or deploying.
+- `hold` loses nothing: the jobs keep their attempts and run once the rule is removed.
+- Matching happens before the handler runs, so arguments that crash deserialization can be held too.
+- Every effect names its rule in metrics, logs and the job's error history.
+
+### Negative
+
+- Each skipped job costs a claim and a finalize, and a hold adds one row in `awa.job_skip_holds`, written in the snooze's own transaction. Jobs that match no rule write nothing.
+- A rule that matches a large backlog churns through it once, and `hold` keeps that to once per held job per horizon.
+- Up to 10 seconds between adding a rule and every worker applying it. Jobs claimed in that window run. An urgent stop is still a queue pause.
+- Another place where a job can go "missing". The held-jobs gauge and `awa.job_skip_holds` exist so that a hold is visible.
+- A forgotten hold delays jobs indefinitely. Expiry is optional because some fixes take longer than any default, but the CLI asks for confirmation when `--expires-in` is omitted.
+
+## Alternatives considered
+
+### Filter in the claim query
+
+Push the rules into claim SQL so that matching jobs are never claimed. Rejected for three reasons:
+- it changes the claim path that ADR-019, ADR-023 and ADR-033 constrain;
+- CEL cannot run in SQL, which ADR-030 already settled;
+- every claim on every queue would pay for the rules, not only jobs of the affected kinds.
+
+### Reject at enqueue
+
+Refuse new jobs that match a rule. It misses jobs already queued. Rejecting an enqueue inside a caller's transaction also turns an operational problem into an application failure, which is the tension [ADR-035](035-backpressure-flow-control.md) describes. A hold at claim time protects the workers without failing producers.
+
+### Pause by kind
+
+Add per-kind pause to `queue_meta`. This is Phase 1 with a single action and no audit or expiry. A rules table carries that subset at the same cost and leaves room for expressions.
+
+### A skip check in user code
+
+A handler can check a feature flag and return `Snooze`. That works today and needs no engine change. But every kind needs its own check and a deploy to add one, and it runs after argument decoding.
+
+### Make skipped jobs a new state
+
+A `held` state would make holds visible without a side table. It was rejected because it touches every state machine, both engines, the admin UI and the TLA+ models, whereas a hold that is a snooze plus a row in `awa.job_skip_holds` needs no new state.
+
+## Open questions
+
+- **Should `hold` be the default?** It loses nothing, but a pile of held jobs is easy to ignore. The gauge and the expiry prompt are the mitigation; `dead_letter` makes the cost visible but spends an attempt.
+- **Should a rule match on `errors`,** such as the last error message? It would let an operator quarantine "jobs that failed with OOM" without knowing the argument pattern, at the cost of a larger CEL environment.
+- **A faster path than the 10-second refresh,** for example a `NOTIFY` on rule changes, if operators need an immediate stop and a queue pause is too coarse.

docs/adr/README.md

@@ -54,6 +54,7 @@ Each record preserves its status, context, decision, consequences, and alternati
 | 044 | [Gate A — storage evolution for 0.7](044-storage-evolution-gate-a.md) | The #295 segment-engine RFC graduates to 0.8: the allocator ideas landed inside the engine as staged migrations and measured better; the remaining WAL headroom has no in-place delivery path (#295, #383). |  |
 | 045 | [Hot-path triggers never wait on another transaction](045-hot-path-trigger-lock-contract.md) | Triggers on the canonical hot tables may write only append-only, unconstrained mark tables or the job's own unique claim; the maintenance path never truncates; the contract is an executable test and a TLA+ witness (v046, 0.6 patch). |  |
 | 046 | [Handler-staged follow-up jobs](046-handler-staged-follow-ups.md) | A handler stages follow-up requests on its `JobContext`; they are inserted in the attempt's lease-guarded completion transaction, deduplicated by their own unique keys, and discarded on every other outcome. | <span class="awa-status awa-status--accepted">Accepted</span> |
+| 047 | [Operator skip rules](047-operator-skip-rules.md) | Operator rules hold, dead-letter or cancel claimed jobs of a kind, optionally matching a CEL predicate, before the handler runs; enforced after claim so the claim path and its proofs are untouched (#546 follow-up). | <span class="awa-status awa-status--proposed">Proposed</span> |
 | 048 | [Retained job output](048-retained-job-output.md) | A handler may record a size-bounded JSON output for its attempt, sealed with the staging buffer and published only by the attempt's own lease-guarded transition: as the job's output on completion, or in the error entry of a failure, where only the latest entry to record one keeps it. It rides the compact completion batch, and neither the engine nor another job reads it. A kind may declare its output type, which types the API and is published in its descriptor. | <span class="awa-status awa-status--accepted">Accepted</span> |
 | 049 | [One job filter, and tags as the indexed handle](049-job-filter-and-indexed-tags.md) | One `JobFilter` describes "which jobs" for listing, counting, batch preview, batch execution and the DLQ, compiled once per engine with only the predicates a filter sets; SQL narrows candidates and a batched loader settles each job's current state. Tags are the indexed handle for "jobs about X": a partial GIN index over non-empty payload tags on each queue-storage partition, maintained by Postgres for every binary, emptied by pruning, and built online by the migrator. | <span class="awa-status awa-status--accepted">Accepted</span> |
 | 050 | [Admin views use invalidation and authoritative refetch](050-admin-view-observation.md) | Polling-first owning API reads, disposable invalidation and independent bounded reconciliation; push transport is deferred. | <span class="awa-status awa-status--accepted">Accepted</span> |