Repository navigation
ADR-047: operator skip rules - #561
Conversation
Operators can hold, dead-letter or cancel claimed jobs of one kind, optionally only those matching a CEL predicate over their args, without pausing the queue or deploying. Rules are checked after the claim and applied through the existing snooze, DLQ and cancel paths, so the claim query and its storage proofs are untouched. A hold lasts until the rule expires, so held jobs are not re-claimed every few seconds. Phase 1 ships rules by kind and queue; phase 2 adds expressions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oDvumkeByijpFvqb85Qbj
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 45 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bc8ea0fe88
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Evaluation errors are reported through the runtime snapshot each worker already publishes, so a failing expression never writes the rules table. Holds are recorded in awa.job_skip_holds in the snooze's own transaction instead of in job metadata, which may be any JSON value; release joins on that table and maintenance clears rows whose job is no longer scheduled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016oDvumkeByijpFvqb85Qbj
Proposes ADR-047, operator skip rules. An operator can stop jobs of one kind from running, optionally only those whose args match a CEL predicate, without pausing the queue or deploying. This follows on from the poison-job work in #546, which bounded a poison job's damage, and this PR is docs only.
The gap
Today an operator's controls are:
The docs answer "stop a kind" with "put it on its own queue", which only works if the queue layout anticipated the problem.
The decision
Rules table:
awa.job_skip_ruleshas a required kind, an optional queue, an optional CEL expression, an action, a reason, an expiry and audit columns. Operators write it rarely and nothing on the job path writes it, so it is not a hot table.When rules apply: the worker checks rules after it claims a job and before the handler runs. Rules reach workers on the 10-second refresh that ADR-038 queue overrides already use. Expressions are compiled once per refresh. A kind with no rules costs one hash lookup.
Actions reuse existing paths, so no new state is introduced:
hold(the default) snoozes the job without spending an attempt, until the rule expires or for 24 hours, whichever is sooner. Removing the rule releases its held jobs in one batch operation. A short hold interval would re-claim every held job each cycle; holding to the horizon avoids that.dead_lettersends the job to the DLQ with reasonskipped.cancelcancels the job.Hooks and registered follow-ups fire as if the handler had returned that result.
CEL: expressions can read
args,metadata,kind,queue,attempt,priorityandtags. An expression that errors at evaluation does not match, so a bad rule cannot quarantine healthy jobs. Each error is counted per rule.Operator surface:
awa job skip add / list / remove;Phasing: phase 1 matches by kind and queue only. Phase 2 adds expressions, gated by the existing
celfeature.Why it doesn't go against the grain
ClientBuilderorInsertOpts.The rejected alternatives are a claim-time SQL filter, rejection at enqueue, per-kind pause only, a check in user code, and a new
heldstate. The ADR lists each with its reasons.Open questions
holdshould be the default.Notes
mainhasv051, Redesign admin overview around queue pressure and operator actions #543 also usesv051and perf(queue-storage): enqueue a ready batch in one round trip (v056) #538 usesv052.