From 1bffb63604731680b7ac8a2a960f73d12039faea Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 22:46:55 +0000 Subject: [PATCH 1/8] ADR-049: one job filter, and tags as the indexed handle One JobFilter describes which jobs for listing, counting, batch preview, batch execution and the DLQ, compiled once per engine with only the predicates a filter sets. Tags become the indexed handle for jobs about something: a partial GIN index over non-empty payload tags on each queue-storage partition, maintained by Postgres for every binary and built online by the migrator. Amends ADR-030. --- docs/adr/030-batch-operations.md | 2 +- docs/adr/049-job-filter-and-indexed-tags.md | 285 ++++++++++++++++++++ docs/adr/README.md | 1 + 3 files changed, 287 insertions(+), 1 deletion(-) create mode 100644 docs/adr/049-job-filter-and-indexed-tags.md diff --git a/docs/adr/030-batch-operations.md b/docs/adr/030-batch-operations.md index 86511084d..85f526dd5 100644 --- a/docs/adr/030-batch-operations.md +++ b/docs/adr/030-batch-operations.md @@ -2,7 +2,7 @@ ## Status -Accepted. The initial implementation scope is the framework plus `set_priority` and `move_queue` operation kinds. Existing synchronous bulk retry, bulk cancel, and DLQ bulk endpoints stay as-is for `0.6`; migrating them to the framework is deferred. +Accepted. The initial implementation scope is the framework plus `set_priority` and `move_queue` operation kinds. Amended by [ADR-049](049-job-filter-and-indexed-tags.md): the batch-operation filter becomes `JobFilter`, shared with listing, counting and the DLQ. Existing synchronous bulk retry, bulk cancel, and DLQ bulk endpoints stay as-is for `0.6`; migrating them to the framework is deferred. ## Context diff --git a/docs/adr/049-job-filter-and-indexed-tags.md b/docs/adr/049-job-filter-and-indexed-tags.md new file mode 100644 index 000000000..525f8a6e0 --- /dev/null +++ b/docs/adr/049-job-filter-and-indexed-tags.md @@ -0,0 +1,285 @@ +# ADR-049: One job filter, and tags as the indexed handle + +## Status + +Proposed. Amends [ADR-030](030-batch-operations.md): its batch-operation filter becomes the filter for job listing, counting, batch preview, batch execution and the DLQ. No migration: new queue-storage schemas install the tag indexes, and existing ones get them through an online build that the migrator runs. + +## Context + +Operators and applications ask Awa one question in several places: *which jobs?* The admin UI's jobs page, `awa job list`, Python's `list_jobs`, a batch operation's preview, the batch operation itself and the DLQ pages each answer it, and each answers it differently. + +- **Several filter types.** + - `ListJobsFilter`: kind, queue, state, tag and paging. + - ADR-030's `BatchOperationFilter`: kind, queue, state, tag, ids and created-at bounds. + - `ListDlqFilter`, which mixes paging into the filter that `purge_dlq` and `bulk_retry_from_dlq` destroy with. + - The UI's batch-operations form, which accepts only queue, kind and ids. + + So the population an operator found on the jobs page can't be carried into a batch operation. +- **Two compilers per engine.** + - **Listing:** builds a candidate query over every place a job's row can live and fetches 300 to 5,000 candidate ids. It then loads each job with its own seven-query round trip, filters again in Rust, and loops until the page is full. + - **Batch operations:** keep four hand-copied SQL predicates per engine. + + ADR-030 requires preview and execution to share one compiler. The list the operator was reading when they asked for the preview shares none of it. +- **Tags are the only handle on "jobs about X", and queue storage doesn't index them.** + - **Queue storage** keeps tags inside each row's `payload`. The listing path filters tags in Rust after loading each candidate, and the DLQ and batch paths use `payload -> 'tags' ? $tag`, which no index can serve. A rare tag therefore walks every retained job. + - **The canonical engine** has had GIN indexes on tags since v001. + +Every other way to find a job is narrower: +- **A job id** is exact, but the caller has to store it. +- **A unique key** finds only unique kinds, and only while the job's state is inside its unique-states mask. +- **Searching args** is unindexed. Indexing args would tax every job on every table its row passes through. + +### Measurements + +The setup: 50,000 jobs on queue storage, inserted through `insert_many` (the `insert_job_compat` path), each with two tags. One tag is `tenant:N`, shared by one job in 50. The other is `order:N`, unique to the job. All jobs completed. The run used Postgres 16, a debug client build, and `awa/tests/tag_filter_bench.rs`. A tagged job never takes the compact completion path, so its lifecycle (insert, claim, completion) writes 1,725 B of WAL. + +| Lookup | Time | Rows | +| --- | ---: | ---: | +| `list_jobs` by kind, no tag, limit 100 | 7.7 s to >120 s | 100 | +| `list_jobs`, common tag, limit 100 | >120 s | — | +| `list_jobs`, rare tag, limit 20 | >120 s | — | +| GIN over payload tags, common tag, 100 rows loaded with `get_job` | 4.8 s | 100 | +| GIN over payload tags, rare tag | 49 ms | 1 | +| `(tag, job_id)` side table, common tag, 100 rows loaded with `get_job` | 4.6 s | 100 | +| `(tag, job_id)` side table, rare tag | 45 ms | 1 | + +Three things explain the table: + +- **The listing query degrades as a prepared statement.** One execution with literal values took 2.3 s and touched 2.5 million buffers. It binds every filter field and writes `($n IS NULL OR column = $n)`. Once a prepared statement switches to a generic plan, Postgres can't drop those terms, and the same query ran for minutes. +- **Loading dominates once lookup is indexed.** `get_job` on queue storage runs seven queries, one per place a job's row can live, so a page of 100 jobs costs about 4.5 s. +- **The two indexes cost about the same:** + + | Index | Extra WAL per tagged job (two tags) | Other costs | + | --- | --- | --- | + | Side table | +386 B | Its cleanup costs more WAL later | + | GIN over payload tags | +634 B | About 200 B of index per job | + + This run indexed only ready rows that carry tags. Completed rows store a NULL payload ([Lookup paths](#lookup-paths)), so they add nothing. + +## Decision + +### One filter + +`awa_model::filter::JobFilter` describes "which jobs" for every path in [Surfaces](#surfaces): + +```rust +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct JobFilter { + pub kind: Option, + pub queue: Option, + /// Any of these states. Empty means any state the consumer allows. + pub states: Vec, + /// All of these tags. + pub tags: Vec, + /// `Some(true)`: only dead-lettered jobs. `Some(false)`: none. + pub dead_lettered: Option, + pub ids: Option>, + pub created_at_gte: Option>, + pub created_at_lt: Option>, +} +``` + +- **Every field narrows.** A field that's present narrows the population, and an absent or empty field doesn't. `ids` intersects with the other fields, as ADR-030 specifies. +- **`created_at` is the original insert time.** It doesn't change across retries, DLQ moves, or priority and queue moves. +- **No `#[non_exhaustive]`.** That would also forbid `..Default::default()`, which is how every existing caller builds a filter. Instead, the documented way to construct one is `JobFilter { kind: ..., ..Default::default() }` or the builder methods. A field added later then breaks no caller that follows it. +- **The JSON shape stays ADR-030's.** A filter with one tag serializes as `"tag": "..."`, and one state as `"state": "..."`. Only a filter that needs more uses `"tags": [...]`, `"states": [...]` or `"dead_lettered"`. +- **Deserialization is strict.** An unknown field is an error, not an ignored key, so a typo can't widen a stored batch operation. +- **New fields wait for the fleet.** A runner that predates this ADR would ignore a field it doesn't know. So submitting a batch operation whose filter uses a field ADR-030 didn't define is refused while any live runtime reports an older `binary_version`. `awa migrate` refuses the same way while a live runtime is too old. + +`BatchOperationFilter` becomes an alias of `JobFilter`, so batch operations accept the new fields once the fleet is upgraded. + +A filter describes a population, and paging is separate from it. So a filter means the same thing wherever it's stored or sent: +- `find_jobs(pool, &JobFilter, JobPage { before_id, limit })` takes its window separately. +- A batch scan takes its id bounds separately. +- `list_dlq` takes `DlqPage { before_dlq_at, before_id }`. + +`list_jobs(&ListJobsFilter)` and `list_dlq(&ListDlqFilter)` keep working, marked `#[deprecated]`, through 0.8. They're thin conversions. + +Each consumer applies the filter, then its own eligibility: + +| Consumer | Eligible states | Order and window | +| --- | --- | --- | +| `find_jobs`: the admin UI jobs page, `awa job list`, Python `find_jobs` | all | id descending, `before_id` keyset, `limit` clamped to 1–1000 | +| `count_jobs`: the UI's match count | all | exact up to 10,000; above that, "10,000+" | +| Batch preview and scan (`set_priority`, `move_queue`) | `available`, `scheduled` (ADR-030) | id ascending, up to the scan's upper bound | +| `list_dlq`, `purge_dlq`, `bulk_retry_from_dlq` | dead-lettered | `(dlq_at, id)` descending | + +Preview and execution select from the same population, which is ADR-030's rule. The jobs page shows a different population when it isn't filtered to `available` or `scheduled`. So when it offers a batch operation on "everything matching", it shows the eligible count beside the listed count. It disables the action for filters that can match no eligible state, and in read-only mode. + +### One compiler per engine + +Each engine has one function that turns `(filter, eligible states, window)` into SQL. It replaces the listing loop, the batch-operation predicates and the DLQ predicates. + +- **Only present predicates are written.** The compiler emits a term only for a field the filter sets. The plan therefore matches the filter, and a generic plan keeps nothing it can't use. +- **Only reachable storage is read.** On queue storage, a job's row can be in any of: + - the ready lanes; + - `deferred_jobs`; + - leases and receipt claims; + - `done_entries` and compact completions; + - `dlq_entries`. + + The compiler includes only the branches where the filter's states, intersected with the consumer's eligible states, can occur. +- **SQL narrows; the loader decides.** The branches aren't provably disjoint while a claim is in flight. The available branch compares lane sequences against a cursor that sits outside MVCC (ADR-044). So a claimed job can appear in neither branch, or in two, until its claim and cursor both settle. The compiler therefore returns candidate ids, de-duplicated and ordered, with a bounded over-fetch. +- **Rows load in a batch.** The batched loader reads the page's candidates with `job_id = ANY($1)`, seven queries per page. It picks each job's current row by the rule `get_job` uses, which is `get_job` with one id, and re-checks the filter against that row. The compiler fetches further only while the page is short. +- **Within a page, each job appears once, in its current state.** A job in the middle of a claim may be missed or shown in its previous state. That's the same advisory sense in which ADR-030 calls the preview advisory. Keyset paging on ids never repeats a job across pages, but a job can change state between pages. +- **The count is capped and timed out.** `count_jobs` counts distinct candidate ids in SQL, stops at 10,001, and runs under a server-side `statement_timeout`. List reads get the same timeout. The UI fetches the count when the filter changes, debounced, and doesn't poll it. +- **The filter has bounds.** + - `ids` is capped at 1,000 when a filter is submitted. Rows already stored still load. + - A filter tag is capped at 256 bytes. Producers aren't affected. + - `limit` is clamped in the compiler on both engines. +- **Listing without a tag has no new index.** Ready and done partitions are already indexed by `job_id`, so an id-descending page with a kind, queue or state filter walks recent rows and stops at the page. A rare kind or queue walks further, bounded by the timeout. An index on `(kind, job_id)` or `(queue, job_id)` would tax every job, so it's left until measurements call for one. +- **Canonical** compiles to one statement over `awa.jobs`, with the same rule about present predicates. The canonical engine is deprecated ([ADR-037](037-canonical-engine-deprecation.md)) and isn't optimised further. + +### Tags are the indexed handle for "jobs about X" + +A tag is the supported way to find the jobs about something an application cares about: `order:123`, `tenant:acme`, `import:2026-10-29`. +- **Attach it at insert,** with `InsertOpts::tags` or Python's `tags=`. Insert already allows at most 20 tags per job, and the canonical engine enforces the same limit. +- **Every tag must match.** A filter's `tags` finds every retained job carrying all of them, in any state. + +**The index.** Each queue-storage table where a job's row outlives its claim gets a partial GIN index on every partition: `ready_entries` and `done_entries` (each partitioned by ring slot), `deferred_jobs` and `dlq_entries`. + +```sql +CREATE INDEX {partition}_tags_idx ON {schema}.{partition} + USING gin ((payload -> 'tags') jsonb_path_ops) + WITH (fastupdate = off) + WHERE (payload -> 'tags') <> '[]'::jsonb; +``` + +- **The predicate excludes untagged jobs.** Untagged jobs are written two ways: without a `tags` key (the Rust storage paths), or with `"tags": []` (`insert_job_compat`). Either way they fail the predicate and add no index entry. +- **`fastupdate` is off.** A partition's GIN pending list would otherwise be scanned linearly by every lookup, then flushed synchronously by whichever insert overflows it. That's a latency spike on enqueue. +- **One index per partition, no parent index.** The ring partitions are created up front, so a parent index would buy nothing. Creating one would take a `SHARE` lock on the parent, blocking writes to every partition. +- **Old and new binaries both maintain it.** Postgres maintains the index on every write. Producers on any version and any insert path add entries, and pruning's truncation clears them. + +#### Lookup paths + +A tag predicate is written `(payload -> 'tags') <> '[]'::jsonb AND payload -> 'tags' @> $tags`. A job is then found where its tags are physically stored: + +- **Ready, scheduled, retrying, running and dead-lettered jobs** are found by index hits on `ready_entries`, `deferred_jobs` and `dlq_entries`. Leases and claims carry no payload; they're found through their ready row. +- **Completed, cancelled and failed jobs** usually have a `done_entries` row with a NULL payload, because the row would be identical to its ready row's. Their tags exist only on the ready row, in the same ring slot, which is truncated with it. So the terminal branch has two parts: + - done rows that carry their own payload, through the done index; + - ready-index hits joined to their done row, for done rows with a NULL payload. + + The second part can't double count, because it only takes done rows with a NULL payload. +- **The prune carry is covered.** It moves a retained failed row with `COALESCE(done.payload, ready.payload)`, so the carried row carries its own tags. +- **The DLQ, purge and bulk-retry predicates change.** They move from `?` to the compiler's `@>` form; `jsonb_path_ops` can't serve `?`. + +#### Building the index + +There's no migration. A plain `CREATE INDEX` blocks writes for the whole build, and Awa's migrations run in one transaction, which rules out `CONCURRENTLY`. + +- **New schemas.** Queue-storage schemas installed by a binary from this ADR on create each partition's index when the partition is created. +- **Existing schemas: `awa storage build-tag-index`.** It runs as the migrator, because since v049 the runtime role need not own Awa's tables, and only an owner may create an index. It builds one index at a time: + - **A ring partition** is indexed right after pruning empties it, while it isn't the current slot. The build takes a `SHARE` lock with `NOWAIT`, confirms the partition is empty, creates the index (instant on an empty table) and commits. A populated partition is retried on a later pass, so the whole ring is covered within one ring cycle. + - **`--now`** builds populated partitions with `CREATE INDEX CONCURRENTLY` instead, and so does every build on `deferred_jobs` and `dlq_entries`. Its costs: + - **It holds back vacuum.** A partial expression index isn't exempt from the vacuum horizon, so the build holds `backend_xmin` while it runs. That delays vacuum on the HOT-updated lane tables and terminal-delta folding. + - **It blocks pruning.** `prune_oldest`'s `NOWAIT` lock on that partition fails, so prune reports `Blocked` until the build finishes. + - **It can lose a deadlock.** A build that loses one leaves an invalid index, which the next run repairs. +- **It's resumable.** Each index has a pinned name (`{partition}_tags_idx`). An existing invalid index is rebuilt with `REINDEX INDEX CONCURRENTLY`, never skipped by `IF NOT EXISTS`. Lock attempts use `lock_timeout` and retry. `awa storage status` reports how many partitions are indexed. +- **Callers can trigger it.** It runs from the CLI, from Rust as `storage::build_tag_indexes`, and from Python as `build_tag_indexes()`. `awa migrate` prints a reminder to run it but doesn't run it. External migration runners run the CLI; a static SQL file can't enumerate each database's partitions or repair invalid indexes. +- **Lookups stay correct during the build.** Until the build finishes, the planner uses the indexes that exist and scans the partitions that have none. + +### Surfaces + +| Surface | Change | +| --- | --- | +| Rust | `JobFilter`, `JobPage` and `DlqPage`. `admin::find_jobs` and `admin::count_jobs`. Batch-operation functions take `JobFilter`. `list_jobs` and `list_dlq` stay, deprecated. | +| HTTP | `GET /api/jobs` accepts every `JobFilter` field as a query parameter, plus `before_id` and `limit`. Repeated `tag=` and `state=` mean all-of and any-of; `ids` is comma-separated, up to 1,000. `GET /api/jobs/count` gives the capped count. Batch-operation bodies are unchanged. An older server ignores parameters it doesn't know and returns a wider list, which `docs/ui-design.md` documents. | +| Admin UI | The search bar accepts several `tag:` terms, plus `id:`, `created>=` and `created<`, and shows the capped match count. The batch-operations form takes the same filter, and the jobs page links to it pre-filled, as in [One filter](#one-filter). | +| CLI | `awa job list` and `awa dlq list` gain `--tag` (repeatable), `--state` (repeatable), `--id`, `--created-at-gte` and `--created-at-lt`. | +| Python | `find_jobs(...)`, `count_jobs(...)` and `list_dlq(...)` take the same fields as keyword arguments or as a `filter=` dict, which is the shape batch operations take. `list_jobs` gains `tag` and `before_id`. | + +These paths stay outside `JobFilter`, because each acts on an explicit selection rather than a population: +- bulk retry and cancel by selected ids; +- `bulk_move_failed_to_dlq(kind, queue)`. + +## What a tag can't tell you + +- **A tag finds only jobs whose producers attached it.** It can't distinguish "never tagged", "pruned", and "never inserted": each is an empty result. Retained jobs inserted before a producer started tagging aren't backfilled. +- **A tag lives as long as the job's rows.** Retention prunes both together. Status history beyond retention belongs in change data capture (v047 keeps every Awa table replicable) or in traces. +- **Tags must not carry secrets or personal data.** Tags are shown in the admin UI, written to logs, and replicated by change data capture, and now they're indexed too. Use an opaque identifier (`order:123`), not a value such as an email address. + +## Deferred: deriving tags from args + +A kind could declare tags derived from its args, for example `#[awa(tag = "order:{order_id}")]` in Rust or `task(tags=lambda a: [...])` in Python. + +For: +- **One declaration replaces every insert site.** A forgotten `tags:` at one call site makes those jobs silently unfindable. +- **It's discoverable.** The descriptor can advertise the tag's prefix to the UI. +- **Rust can check the field exists** at compile time. + +Against: +- **Coverage would be partial.** Derivation runs only where the typed kind is known. Raw inserts by kind name, the bridge, `insert_job_compat` and other services wouldn't apply it, and a lookup that silently misses some jobs is worse than one that needs a tag passed. +- **Producers in two languages must render identical strings.** Otherwise the population splits. +- **Template changes aren't retroactive.** +- **It would copy arg values into tags without the caller seeing it,** which makes the privacy rule above easy to break. +- **It's a small template language.** Every rule in it becomes a compatibility promise. + +It's deferred because the index serves explicit tags today, and derivation can be added later without changing storage. If it's added, it should be plain code rather than a template: a `JobArgs::tags(&self)` method and the equivalent Python hook, with a golden test that the two languages agree. + +## Non-goals + +- **Searching args or metadata.** No index covers them, and a filter field without an index path can't be bounded. ADR-030's rule stands: add a field when an operator needs it and storage can answer it predictably. +- **CEL or free-form predicates.** ADR-030's reasons hold. +- **An exact census of jobs mid-transition.** Listing and counting are advisory for jobs whose claim is in flight. + +## Alternatives considered + +- **A `(tag, job_id)` side table.** It writes less WAL before its cleanup is counted. But: + - pruning truncates partitions without reading job ids, so its rows would need a cleanup sweep, or a check that each job still exists on every read; + - every insert path in every binary version would have to write it, and producers on an older version would leave gaps during an upgrade; + - retained jobs would need a backfill. +- **A partial predicate of `payload ? 'tags'`.** It indexes every job inserted through `insert_job_compat`, which writes `"tags": []`, and so taxes untagged traffic. +- **A parent index created by a migration.** `CREATE INDEX ON ONLY` takes a `SHARE` lock on the parent, held until the migration commits, which blocks every insert. On the unpartitioned tables it does a full blocking build. The ring partitions already exist, so a parent index adds nothing. +- **A GIN index on `args`.** Every job would pay on every table, to answer a question tags answer for the jobs that opt in. +- **Keep the filters separate and just index tags.** That fixes neither the generic-plan collapse nor the per-row loads. It also leaves previews selecting a different population from the list the operator was reading. +- **External search** (Elasticsearch, a warehouse). Change data capture already supports it for history. Operators still need live state, which only Awa's tables have. + +## Consequences + +### Positive + +- **Every listing, count, batch and DLQ path agrees** on which jobs a filter selects, and the UI can carry a search into a batch operation. +- **Finding jobs by tag becomes an indexed lookup** in every state, including dead-lettered jobs, and listing stops degrading under generic plans. +- **The tag index needs no code on any insert path** and no cleanup, and covers producers on older versions. + +### Negative + +- **Tagged jobs write more WAL.** With two tags per job, the lifecycle writes about a third more. Untagged jobs are unaffected. +- **Upgrading gains an online step** for existing schemas. Until it finishes, tag lookups on unindexed partitions still scan. `--now` holds back vacuum and blocks pruning of the partition it's building. +- **Some names are deprecated.** `list_jobs`, `ListJobsFilter`, `list_dlq` and `ListDlqFilter` are deprecated, and `BatchOperationFilter` gains fields. Code that builds it with an exhaustive struct literal adds `..Default::default()`. + +## Rollout + +- **No schema migration.** Binaries before and after this ADR read and write the same tables, and the indexes are invisible to them. +- **A binary from this ADR on uses the new compiler** whether or not the indexes exist. Without them, a tag predicate scans in SQL, which is still one statement and one batched load per page instead of a loop of per-row loads. +- **Filters with new fields wait for the fleet.** Batch operations whose filter uses a field ADR-030 didn't define are refused until no live runtime is older than this ADR. + +## Validation + +- **Every job is in exactly one branch.** A property test checks this across every transition, with claims racing, on both engines. The only exception is the documented in-flight window. +- **Same population, both engines.** For filters drawn from every field combination, `find_jobs`, `count_jobs`, the batch preview and the batch scan all agree, after eligibility. +- **Tag lookups across a job's life.** Every state and transition is covered: + - a completed job with a NULL done payload; + - a retry; + - a DLQ move and a DLQ retry; + - priority and queue moves; + - a rescue; + - pruning, with the failed-row carry; + - jobs inserted through `insert_job_compat`, Rust COPY and the bridge. +- **Untagged jobs add no index entries,** on every insert path. +- **The build is safe to interrupt and to run alongside pruning.** Specifically: + - an interrupted `--now` build is repaired by the next run; + - a ring build never queues an exclusive lock that blocks a lane; + - pruning during a build reports `Blocked` and then succeeds. +- **A mixed fleet.** A 0.6.9 producer's tagged jobs are found by tag once the build has run. A batch operation using `tags` is refused while a 0.6.9 runtime is live. +- **Benchmark.** `awa/tests/tag_filter_bench.rs`, on a realistic mix of tagged and untagged jobs, checks: + - a 100-row page loads in one batched read; + - a rare-tag lookup stays under 100 ms at 50,000 retained jobs; + - with `fastupdate = off`, p99 enqueue latency for tagged jobs stays within 10% of today. + +## Relationship to other ADRs + +- **ADR-030.** Its filter becomes `JobFilter`, and its rule that preview and execution share a compiler extends to listing, counting and the DLQ. Its planned tag-editing operation stays compatible, because Postgres re-indexes any row it rewrites. +- **ADR-026 / ADR-048.** The compact completion path is unchanged. Tagged jobs already leave it. +- **ADR-044.** The lane cursor sits outside MVCC, which is why listing re-checks each job's current row instead of trusting the branches. +- **ADR-045.** No trigger is added. diff --git a/docs/adr/README.md b/docs/adr/README.md index e0bfc0918..8934179c4 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -55,6 +55,7 @@ Each record preserves its status, context, decision, consequences, and alternati | 045 | [Hot-path triggers never wait on another transaction](045-hot-path-trigger-lock-contract.md) | Triggers on the canonical hot tables may write only append-only, unconstrained mark tables or the job's own unique claim; the maintenance path never truncates; the contract is an executable test and a TLA+ witness (v046, 0.6 patch). | | | 046 | [Handler-staged follow-up jobs](046-handler-staged-follow-ups.md) | A handler stages follow-up requests on its `JobContext`; they are inserted in the attempt's lease-guarded completion transaction, deduplicated by their own unique keys, and discarded on every other outcome. | Accepted | | 048 | [Retained job output](048-retained-job-output.md) | A handler may record a size-bounded JSON output for its attempt, sealed with the staging buffer and published only by the attempt's own lease-guarded transition: as the job's output on completion, or in the error entry of a failure, where only the latest entry to record one keeps it. It rides the compact completion batch, and neither the engine nor another job reads it. A kind may declare its output type, which types the API and is published in its descriptor. | Accepted | +| 049 | [One job filter, and tags as the indexed handle](049-job-filter-and-indexed-tags.md) | One `JobFilter` describes "which jobs" for listing, counting, batch preview, batch execution and the DLQ, compiled once per engine with only the predicates a filter sets; SQL narrows candidates and a batched loader settles each job's current state. Tags are the indexed handle for "jobs about X": a partial GIN index over non-empty payload tags on each queue-storage partition, maintained by Postgres for every binary, emptied by pruning, and built online by the migrator. | Proposed | ## Correctness evidence From a49ebc7af12ffcaa9eb49855cf319e9a99765ae7 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 23:22:04 +0000 Subject: [PATCH 2/8] ADR-049: match the implementation's index pass, version floor and type change Migration and schema preparation index empty partitions on the spot, extended batch filters wait for 0.7.0-alpha.2 runtimes, and BatchOperationFilter's state and tag become the lists states and tags. --- docs/adr/049-job-filter-and-indexed-tags.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/adr/049-job-filter-and-indexed-tags.md b/docs/adr/049-job-filter-and-indexed-tags.md index 525f8a6e0..3f2c8e28d 100644 --- a/docs/adr/049-job-filter-and-indexed-tags.md +++ b/docs/adr/049-job-filter-and-indexed-tags.md @@ -84,7 +84,7 @@ pub struct JobFilter { - **No `#[non_exhaustive]`.** That would also forbid `..Default::default()`, which is how every existing caller builds a filter. Instead, the documented way to construct one is `JobFilter { kind: ..., ..Default::default() }` or the builder methods. A field added later then breaks no caller that follows it. - **The JSON shape stays ADR-030's.** A filter with one tag serializes as `"tag": "..."`, and one state as `"state": "..."`. Only a filter that needs more uses `"tags": [...]`, `"states": [...]` or `"dead_lettered"`. - **Deserialization is strict.** An unknown field is an error, not an ignored key, so a typo can't widen a stored batch operation. -- **New fields wait for the fleet.** A runner that predates this ADR would ignore a field it doesn't know. So submitting a batch operation whose filter uses a field ADR-030 didn't define is refused while any live runtime reports an older `binary_version`. `awa migrate` refuses the same way while a live runtime is too old. +- **New fields wait for the fleet.** A runner that predates this ADR would ignore a field it doesn't know. So submitting a batch operation whose filter uses a field ADR-030 didn't define is refused while any live runtime reports a version older than 0.7.0-alpha.2, the first release with this ADR. `awa migrate` refuses the same way while a live runtime is too old. `BatchOperationFilter` becomes an alias of `JobFilter`, so batch operations accept the new fields once the fleet is upgraded. @@ -167,7 +167,7 @@ A tag predicate is written `(payload -> 'tags') <> '[]'::jsonb AND payload -> 't There's no migration. A plain `CREATE INDEX` blocks writes for the whole build, and Awa's migrations run in one transaction, which rules out `CONCURRENTLY`. -- **New schemas.** Queue-storage schemas installed by a binary from this ADR on create each partition's index when the partition is created. +- **New schemas and empty partitions.** `awa migrate` and schema preparation index every partition that is empty, owned by the caller, and free to lock without waiting. That costs nothing on an empty table, so new and freshly installed schemas come out fully indexed. - **Existing schemas: `awa storage build-tag-index`.** It runs as the migrator, because since v049 the runtime role need not own Awa's tables, and only an owner may create an index. It builds one index at a time: - **A ring partition** is indexed right after pruning empties it, while it isn't the current slot. The build takes a `SHARE` lock with `NOWAIT`, confirms the partition is empty, creates the index (instant on an empty table) and commits. A populated partition is retried on a later pass, so the whole ring is covered within one ring cycle. - **`--now`** builds populated partitions with `CREATE INDEX CONCURRENTLY` instead, and so does every build on `deferred_jobs` and `dlq_entries`. Its costs: @@ -175,7 +175,7 @@ There's no migration. A plain `CREATE INDEX` blocks writes for the whole build, - **It blocks pruning.** `prune_oldest`'s `NOWAIT` lock on that partition fails, so prune reports `Blocked` until the build finishes. - **It can lose a deadlock.** A build that loses one leaves an invalid index, which the next run repairs. - **It's resumable.** Each index has a pinned name (`{partition}_tags_idx`). An existing invalid index is rebuilt with `REINDEX INDEX CONCURRENTLY`, never skipped by `IF NOT EXISTS`. Lock attempts use `lock_timeout` and retry. `awa storage status` reports how many partitions are indexed. -- **Callers can trigger it.** It runs from the CLI, from Rust as `storage::build_tag_indexes`, and from Python as `build_tag_indexes()`. `awa migrate` prints a reminder to run it but doesn't run it. External migration runners run the CLI; a static SQL file can't enumerate each database's partitions or repair invalid indexes. +- **Callers can trigger it.** It runs from the CLI or from Rust as `tag_index::build_tag_indexes`. External migration runners run the CLI; a static SQL file can't enumerate each database's partitions or repair invalid indexes. - **Lookups stay correct during the build.** Until the build finishes, the planner uses the indexes that exist and scans the partitions that have none. ### Surfaces @@ -246,7 +246,7 @@ It's deferred because the index serves explicit tags today, and derivation can b - **Tagged jobs write more WAL.** With two tags per job, the lifecycle writes about a third more. Untagged jobs are unaffected. - **Upgrading gains an online step** for existing schemas. Until it finishes, tag lookups on unindexed partitions still scan. `--now` holds back vacuum and blocks pruning of the partition it's building. -- **Some names are deprecated.** `list_jobs`, `ListJobsFilter`, `list_dlq` and `ListDlqFilter` are deprecated, and `BatchOperationFilter` gains fields. Code that builds it with an exhaustive struct literal adds `..Default::default()`. +- **Some names are deprecated, and one type changes.** `list_jobs`, `ListJobsFilter`, `list_dlq`, `purge_dlq`, `bulk_retry_from_dlq` and `ListDlqFilter` are deprecated. `BatchOperationFilter` becomes `JobFilter`, whose `state` and `tag` fields are now the lists `states` and `tags`; code that set them in a struct literal moves to the builder. ## Rollout From f4c7fbc162c5318a32dd663551cb8c1007d3781d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 30 Oct 2026 00:05:31 +0000 Subject: [PATCH 3/8] ADR-049: fail closed on older readers, bound filter tags, probe-based plans Several tags or states travel as a list under ADR-030's tag and state keys, which a 0.6 reader refuses rather than ignoring, and the DLQ bulk bodies nest the filter so a 0.6 server sees none and refuses. A filter may require at most 20 tags. The status line names what migrate and build-tag-index each index. DLQ bulk actions need a field that narrows the DLQ. Branch joins are lateral probes, measured against the plans that timed out at 20,000 jobs. --- docs/adr/030-batch-operations.md | 2 +- docs/adr/049-job-filter-and-indexed-tags.md | 24 +++++++++++++++------ 2 files changed, 18 insertions(+), 8 deletions(-) diff --git a/docs/adr/030-batch-operations.md b/docs/adr/030-batch-operations.md index 85f526dd5..5d54a68ca 100644 --- a/docs/adr/030-batch-operations.md +++ b/docs/adr/030-batch-operations.md @@ -2,7 +2,7 @@ ## Status -Accepted. The initial implementation scope is the framework plus `set_priority` and `move_queue` operation kinds. Amended by [ADR-049](049-job-filter-and-indexed-tags.md): the batch-operation filter becomes `JobFilter`, shared with listing, counting and the DLQ. Existing synchronous bulk retry, bulk cancel, and DLQ bulk endpoints stay as-is for `0.6`; migrating them to the framework is deferred. +Accepted. The initial implementation scope is the framework plus `set_priority` and `move_queue` operation kinds. Amended by [ADR-049](049-job-filter-and-indexed-tags.md): the batch-operation filter becomes `JobFilter`, shared with listing, counting and the DLQ. Existing synchronous bulk retry, bulk cancel, and DLQ bulk endpoints stay synchronous, and moving them onto the framework is deferred. In 0.7, ADR-049 changes the filter the DLQ bulk endpoints take; bulk retry and bulk cancel by selected ids are unchanged. ## Context diff --git a/docs/adr/049-job-filter-and-indexed-tags.md b/docs/adr/049-job-filter-and-indexed-tags.md index 3f2c8e28d..aa8810136 100644 --- a/docs/adr/049-job-filter-and-indexed-tags.md +++ b/docs/adr/049-job-filter-and-indexed-tags.md @@ -2,7 +2,7 @@ ## Status -Proposed. Amends [ADR-030](030-batch-operations.md): its batch-operation filter becomes the filter for job listing, counting, batch preview, batch execution and the DLQ. No migration: new queue-storage schemas install the tag indexes, and existing ones get them through an online build that the migrator runs. +Proposed. Amends [ADR-030](030-batch-operations.md): its batch-operation filter becomes the filter for job listing, counting, batch preview, batch execution and the DLQ. No migration. `awa migrate` and schema preparation index the partitions that are empty, so new schemas come out fully indexed; populated partitions are indexed by `awa storage build-tag-index`, which the migrator runs after upgrading. ## Context @@ -82,9 +82,10 @@ pub struct JobFilter { - **Every field narrows.** A field that's present narrows the population, and an absent or empty field doesn't. `ids` intersects with the other fields, as ADR-030 specifies. - **`created_at` is the original insert time.** It doesn't change across retries, DLQ moves, or priority and queue moves. - **No `#[non_exhaustive]`.** That would also forbid `..Default::default()`, which is how every existing caller builds a filter. Instead, the documented way to construct one is `JobFilter { kind: ..., ..Default::default() }` or the builder methods. A field added later then breaks no caller that follows it. -- **The JSON shape stays ADR-030's.** A filter with one tag serializes as `"tag": "..."`, and one state as `"state": "..."`. Only a filter that needs more uses `"tags": [...]`, `"states": [...]` or `"dead_lettered"`. +- **The JSON shape stays ADR-030's, and fails closed on older readers.** `tag` and `state` take one value or a list: `"tag": "a"`, `"tag": ["a", "b"]`. There are no `tags` or `states` keys. A 0.6 server or runner ignores a key it doesn't know, so a new key would be dropped and the filter widened, but it refuses a list where it expects one value. A filter with several tags or states therefore fails on an older reader instead of acting on more jobs. `dead_lettered` is the one new key an older reader drops; batch operations refuse `dead_lettered: true` and `false` changes nothing for them, so dropping it widens nothing. - **Deserialization is strict.** An unknown field is an error, not an ignored key, so a typo can't widen a stored batch operation. -- **New fields wait for the fleet.** A runner that predates this ADR would ignore a field it doesn't know. So submitting a batch operation whose filter uses a field ADR-030 didn't define is refused while any live runtime reports a version older than 0.7.0-alpha.2, the first release with this ADR. `awa migrate` refuses the same way while a live runtime is too old. +- **The DLQ bulk endpoints nest the filter.** Their 0.6 bodies are flat `kind`, `queue` and `tag`. The full filter goes under `filter`, which a 0.6 server ignores, then refuses the request as having no filter. The flat fields still work, and a body may use one form or the other, not both. +- **New fields wait for the fleet.** Submitting a batch operation whose filter has several tags or states, or `dead_lettered`, is refused while any live runtime reports a version older than 0.7.0-alpha.2, the first release with this ADR. An older runner would fail to load the filter; the check turns that failure into a clear error at submission. `BatchOperationFilter` becomes an alias of `JobFilter`, so batch operations accept the new fields once the fleet is upgraded. @@ -104,6 +105,8 @@ Each consumer applies the filter, then its own eligibility: | Batch preview and scan (`set_priority`, `move_queue`) | `available`, `scheduled` (ADR-030) | id ascending, up to the scan's upper bound | | `list_dlq`, `purge_dlq`, `bulk_retry_from_dlq` | dead-lettered | `(dlq_at, id)` descending | +A DLQ purge or bulk retry needs a field that narrows the DLQ (kind, queue, tag, ids or a created bound) unless the caller passes `allow_all`. `state: failed` and `dead_lettered: true` don't count: every DLQ row already matches them. + Preview and execution select from the same population, which is ADR-030's rule. The jobs page shows a different population when it isn't filtered to `available` or `scheduled`. So when it offers a batch operation on "everything matching", it shows the eligible count beside the listed count. It disables the action for filters that can match no eligible state, and in read-only mode. ### One compiler per engine @@ -122,11 +125,17 @@ Each engine has one function that turns `(filter, eligible states, window)` into - **SQL narrows; the loader decides.** The branches aren't provably disjoint while a claim is in flight. The available branch compares lane sequences against a cursor that sits outside MVCC (ADR-044). So a claimed job can appear in neither branch, or in two, until its claim and cursor both settle. The compiler therefore returns candidate ids, de-duplicated and ordered, with a bounded over-fetch. - **Rows load in a batch.** The batched loader reads the page's candidates with `job_id = ANY($1)`, seven queries per page. It picks each job's current row by the rule `get_job` uses, which is `get_job` with one id, and re-checks the filter against that row. The compiler fetches further only while the page is short. - **Within a page, each job appears once, in its current state.** A job in the middle of a claim may be missed or shown in its previous state. That's the same advisory sense in which ADR-030 calls the preview advisory. Keyset paging on ids never repeats a job across pages, but a job can change state between pages. -- **The count is capped and timed out.** `count_jobs` counts distinct candidate ids in SQL, stops at 10,001, and runs under a server-side `statement_timeout`. List reads get the same timeout. The UI fetches the count when the filter changes, debounced, and doesn't poll it. +- **The count is capped and timed out.** `count_jobs` reads at most 10,001 candidates from each branch, counts the distinct ids, and reports "10,000+" when any branch reached its limit. It runs under a server-side `statement_timeout`. List reads get the same timeout. The UI fetches the count when the filter changes, debounced, and doesn't poll it. - **The filter has bounds.** - `ids` is capped at 1,000 when a filter is submitted. Rows already stored still load. - A filter tag is capped at 256 bytes. Producers aren't affected. + - `tags` holds at most 20 entries, the per-job insert limit. A longer list is rejected, not truncated, because dropping an all-of term widens the match. - `limit` is clamped in the compiler on both engines. +- **Every join is a primary-key or `job_id` probe.** Each branch reaches the rows it joins through a lateral subquery fenced with `OFFSET 0`, so the planner can't swap the probe for a hash join over every retained row. Planned freely, it misjudges partial-index hits and the cursor's reach, and a 20,000-job page took over 20 seconds. + - Available jobs are read per lane and ring slot, from the claim cursor up the ready primary key, so retained ready rows below the cursor are never read. + - A claim, lease or completion finds its ready row by `job_id` in that slot's partition. The lane's priority isn't compared, because aging can leave a lease's priority different from its ready row's. + - A claim batch whose receipts closure batches have all closed is skipped before it's unnested. + - JIT is off for these reads. Their estimated cost can cross the JIT threshold, and compiling them takes about a second. - **Listing without a tag has no new index.** Ready and done partitions are already indexed by `job_id`, so an id-descending page with a kind, queue or state filter walks recent rows and stops at the page. A rare kind or queue walks further, bounded by the timeout. An index on `(kind, job_id)` or `(queue, job_id)` would tax every job, so it's left until measurements call for one. - **Canonical** compiles to one statement over `awa.jobs`, with the same rule about present predicates. The canonical engine is deprecated ([ADR-037](037-canonical-engine-deprecation.md)) and isn't optimised further. @@ -183,7 +192,7 @@ There's no migration. A plain `CREATE INDEX` blocks writes for the whole build, | Surface | Change | | --- | --- | | Rust | `JobFilter`, `JobPage` and `DlqPage`. `admin::find_jobs` and `admin::count_jobs`. Batch-operation functions take `JobFilter`. `list_jobs` and `list_dlq` stay, deprecated. | -| HTTP | `GET /api/jobs` accepts every `JobFilter` field as a query parameter, plus `before_id` and `limit`. Repeated `tag=` and `state=` mean all-of and any-of; `ids` is comma-separated, up to 1,000. `GET /api/jobs/count` gives the capped count. Batch-operation bodies are unchanged. An older server ignores parameters it doesn't know and returns a wider list, which `docs/ui-design.md` documents. | +| HTTP | `GET /api/jobs` accepts every `JobFilter` field as a query parameter, plus `before_id` and `limit`. Repeated `tag=` and `state=` mean all-of and any-of; `ids` is comma-separated, up to 1,000. `GET /api/jobs/count` gives the capped count. Batch-operation bodies keep ADR-030's shape, with lists allowed under `tag` and `state`; DLQ bulk bodies take the filter under `filter`. An older server ignores parameters it doesn't know and returns a wider list, which `docs/ui-design.md` documents. | | Admin UI | The search bar accepts several `tag:` terms, plus `id:`, `created>=` and `created<`, and shows the capped match count. The batch-operations form takes the same filter, and the jobs page links to it pre-filled, as in [One filter](#one-filter). | | CLI | `awa job list` and `awa dlq list` gain `--tag` (repeatable), `--state` (repeatable), `--id`, `--created-at-gte` and `--created-at-lt`. | | Python | `find_jobs(...)`, `count_jobs(...)` and `list_dlq(...)` take the same fields as keyword arguments or as a `filter=` dict, which is the shape batch operations take. `list_jobs` gains `tag` and `before_id`. | @@ -252,7 +261,8 @@ It's deferred because the index serves explicit tags today, and derivation can b - **No schema migration.** Binaries before and after this ADR read and write the same tables, and the indexes are invisible to them. - **A binary from this ADR on uses the new compiler** whether or not the indexes exist. Without them, a tag predicate scans in SQL, which is still one statement and one batched load per page instead of a loop of per-row loads. -- **Filters with new fields wait for the fleet.** Batch operations whose filter uses a field ADR-030 didn't define are refused until no live runtime is older than this ADR. +- **Older readers fail closed.** A filter with several tags or states reaches a 0.6 server or runner as a list where it expects one value, which it refuses. A DLQ bulk body reaches a 0.6 server as an empty filter, which it refuses. So a client newer than the server it calls, such as a browser holding the new admin UI during a rolling deploy, gets an error rather than a wider operation. +- **Filters with new fields wait for the fleet.** Batch operations whose filter has several tags or states, or `dead_lettered`, are refused until no live runtime is older than this ADR. ## Validation @@ -271,7 +281,7 @@ It's deferred because the index serves explicit tags today, and derivation can b - an interrupted `--now` build is repaired by the next run; - a ring build never queues an exclusive lock that blocks a lane; - pruning during a build reports `Blocked` and then succeeds. -- **A mixed fleet.** A 0.6.9 producer's tagged jobs are found by tag once the build has run. A batch operation using `tags` is refused while a 0.6.9 runtime is live. +- **A mixed fleet.** A 0.6.9 producer's tagged jobs are found by tag once the build has run. A batch operation with several tags is refused while a 0.6.9 runtime is live, and the 0.6 filter type fails to read one. - **Benchmark.** `awa/tests/tag_filter_bench.rs`, on a realistic mix of tagged and untagged jobs, checks: - a 100-row page loads in one batched read; - a rare-tag lookup stays under 100 ms at 50,000 retained jobs; From e391d9db0f1a0e7f1a00e435de38f249aa358210 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 30 Oct 2026 00:36:46 +0000 Subject: [PATCH 4/8] ADR-049: record measured lookup costs and the loader's plan settings --- docs/adr/049-job-filter-and-indexed-tags.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/adr/049-job-filter-and-indexed-tags.md b/docs/adr/049-job-filter-and-indexed-tags.md index aa8810136..f89b7261d 100644 --- a/docs/adr/049-job-filter-and-indexed-tags.md +++ b/docs/adr/049-job-filter-and-indexed-tags.md @@ -136,6 +136,7 @@ Each engine has one function that turns `(filter, eligible states, window)` into - A claim, lease or completion finds its ready row by `job_id` in that slot's partition. The lane's priority isn't compared, because aging can leave a lease's priority different from its ready row's. - A claim batch whose receipts closure batches have all closed is skipped before it's unnested. - JIT is off for these reads. Their estimated cost can cross the JIT threshold, and compiling them takes about a second. + - The batched loader's statements keep a connection's first plan. Postgres otherwise re-plans a statement over partitioned tables on every call, because a generic plan prices partitions that runtime pruning skips, and planning costs more than running these probes. - **Listing without a tag has no new index.** Ready and done partitions are already indexed by `job_id`, so an id-descending page with a kind, queue or state filter walks recent rows and stops at the page. A rare kind or queue walks further, bounded by the timeout. An index on `(kind, job_id)` or `(queue, job_id)` would tax every job, so it's left until measurements call for one. - **Canonical** compiles to one statement over `awa.jobs`, with the same rule about present predicates. The canonical engine is deprecated ([ADR-037](037-canonical-engine-deprecation.md)) and isn't optimised further. @@ -284,7 +285,7 @@ It's deferred because the index serves explicit tags today, and derivation can b - **A mixed fleet.** A 0.6.9 producer's tagged jobs are found by tag once the build has run. A batch operation with several tags is refused while a 0.6.9 runtime is live, and the 0.6 filter type fails to read one. - **Benchmark.** `awa/tests/tag_filter_bench.rs`, on a realistic mix of tagged and untagged jobs, checks: - a 100-row page loads in one batched read; - - a rare-tag lookup stays under 100 ms at 50,000 retained jobs; + - a rare-tag lookup aims for under 100 ms at 50,000 retained jobs. Measured when this was implemented: a page of 100 in about 200 ms and a rare tag in 120–160 ms, against a 120-second listing before. About half of a rare-tag lookup is loading the one job: `terminal_jobs` expands every compact completion batch before it can filter by id, which also slows `get_job`. An id-keyed read for it needs a migration and is a follow-up; - with `fastupdate = off`, p99 enqueue latency for tagged jobs stays within 10% of today. ## Relationship to other ADRs From 8389870a62059f28944fb00ed31e47c166facf67 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 30 Oct 2026 01:11:26 +0000 Subject: [PATCH 5/8] ADR-049: the capped count reports 10,000+ on the distinct total too --- docs/adr/049-job-filter-and-indexed-tags.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/049-job-filter-and-indexed-tags.md b/docs/adr/049-job-filter-and-indexed-tags.md index f89b7261d..60840c162 100644 --- a/docs/adr/049-job-filter-and-indexed-tags.md +++ b/docs/adr/049-job-filter-and-indexed-tags.md @@ -125,7 +125,7 @@ Each engine has one function that turns `(filter, eligible states, window)` into - **SQL narrows; the loader decides.** The branches aren't provably disjoint while a claim is in flight. The available branch compares lane sequences against a cursor that sits outside MVCC (ADR-044). So a claimed job can appear in neither branch, or in two, until its claim and cursor both settle. The compiler therefore returns candidate ids, de-duplicated and ordered, with a bounded over-fetch. - **Rows load in a batch.** The batched loader reads the page's candidates with `job_id = ANY($1)`, seven queries per page. It picks each job's current row by the rule `get_job` uses, which is `get_job` with one id, and re-checks the filter against that row. The compiler fetches further only while the page is short. - **Within a page, each job appears once, in its current state.** A job in the middle of a claim may be missed or shown in its previous state. That's the same advisory sense in which ADR-030 calls the preview advisory. Keyset paging on ids never repeats a job across pages, but a job can change state between pages. -- **The count is capped and timed out.** `count_jobs` reads at most 10,001 candidates from each branch, counts the distinct ids, and reports "10,000+" when any branch reached its limit. It runs under a server-side `statement_timeout`. List reads get the same timeout. The UI fetches the count when the filter changes, debounced, and doesn't poll it. +- **The count is capped and timed out.** `count_jobs` reads at most 10,001 candidates from each branch and counts the distinct ids across all of them. It reports "10,000+" when that distinct count passes 10,000, or when any branch reached its limit, since that branch alone has more. It runs under a server-side `statement_timeout`. List reads get the same timeout. The UI fetches the count when the filter changes, debounced, and doesn't poll it. - **The filter has bounds.** - `ids` is capped at 1,000 when a filter is submitted. Rows already stored still load. - A filter tag is capped at 256 bytes. Producers aren't affected. From 79605315f48baa9e9031486fda255813c405dc9f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 30 Oct 2026 01:47:17 +0000 Subject: [PATCH 6/8] ADR-049: say what the tag-filter bench asserts and what it only reports --- docs/adr/049-job-filter-and-indexed-tags.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/adr/049-job-filter-and-indexed-tags.md b/docs/adr/049-job-filter-and-indexed-tags.md index 60840c162..8e91b1335 100644 --- a/docs/adr/049-job-filter-and-indexed-tags.md +++ b/docs/adr/049-job-filter-and-indexed-tags.md @@ -283,10 +283,10 @@ It's deferred because the index serves explicit tags today, and derivation can b - a ring build never queues an exclusive lock that blocks a lane; - pruning during a build reports `Blocked` and then succeeds. - **A mixed fleet.** A 0.6.9 producer's tagged jobs are found by tag once the build has run. A batch operation with several tags is refused while a 0.6.9 runtime is live, and the 0.6 filter type fails to read one. -- **Benchmark.** `awa/tests/tag_filter_bench.rs`, on a realistic mix of tagged and untagged jobs, checks: - - a 100-row page loads in one batched read; - - a rare-tag lookup aims for under 100 ms at 50,000 retained jobs. Measured when this was implemented: a page of 100 in about 200 ms and a rare tag in 120–160 ms, against a 120-second listing before. About half of a rare-tag lookup is loading the one job: `terminal_jobs` expands every compact completion batch before it can filter by id, which also slows `get_job`. An id-keyed read for it needs a migration and is a follow-up; - - with `fastupdate = off`, p99 enqueue latency for tagged jobs stays within 10% of today. +- **Benchmark.** `awa/tests/tag_filter_bench.rs` loads a mix of tagged and untagged completed jobs (50,000 by default). + - **It asserts** that a rare-tag lookup finds its one job, with a warm median under 250 ms. The aim is 100 ms; about half of today's time is loading the one job, because `terminal_jobs` expands every compact completion batch before it can filter by id, which also slows `get_job`. An id-keyed read for it needs a migration and is a follow-up. + - **It reports** cold and warm times for unfiltered, available-only, kind, common-tag and rare-tag pages, a capped count, lifecycle WAL per job, and the tag indexes' size. Measured when this was implemented: a page of 100 in about 200 ms and a rare tag in 120–160 ms, against a 120-second listing before. + - **Not measured by it:** how many reads a page takes, and p99 enqueue latency for tagged jobs with `fastupdate = off`. ## Relationship to other ADRs From d2cc29a8c70d6d3f41a13cb863e7b626f9a85f17 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 30 Oct 2026 03:29:42 +0000 Subject: [PATCH 7/8] ADR-049: finding jobs is queue-storage only; canonical returns the DLQ APIs' error --- docs/adr/049-job-filter-and-indexed-tags.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/049-job-filter-and-indexed-tags.md b/docs/adr/049-job-filter-and-indexed-tags.md index 8e91b1335..58860072b 100644 --- a/docs/adr/049-job-filter-and-indexed-tags.md +++ b/docs/adr/049-job-filter-and-indexed-tags.md @@ -138,7 +138,7 @@ Each engine has one function that turns `(filter, eligible states, window)` into - JIT is off for these reads. Their estimated cost can cross the JIT threshold, and compiling them takes about a second. - The batched loader's statements keep a connection's first plan. Postgres otherwise re-plans a statement over partitioned tables on every call, because a generic plan prices partitions that runtime pruning skips, and planning costs more than running these probes. - **Listing without a tag has no new index.** Ready and done partitions are already indexed by `job_id`, so an id-descending page with a kind, queue or state filter walks recent rows and stops at the page. A rare kind or queue walks further, bounded by the timeout. An index on `(kind, job_id)` or `(queue, job_id)` would tax every job, so it's left until measurements call for one. -- **Canonical** compiles to one statement over `awa.jobs`, with the same rule about present predicates. The canonical engine is deprecated ([ADR-037](037-canonical-engine-deprecation.md)) and isn't optimised further. +- **Queue storage only.** The canonical engine is deprecated ([ADR-037](037-canonical-engine-deprecation.md)), and in 0.7 a migrated cluster is finalized onto queue storage; only a fresh install that hasn't started a worker is still canonical. Finding, counting and batch operations therefore return the error the DLQ APIs already return when no queue-storage backend is active, rather than keep a second compiler alive until 0.8 removes canonical. ### Tags are the indexed handle for "jobs about X" From a98508635736fb29c1e6575ba5523cd7bb511e87 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 30 Oct 2026 04:55:32 +0000 Subject: [PATCH 8/8] ADR-049: accepted --- docs/adr/049-job-filter-and-indexed-tags.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/049-job-filter-and-indexed-tags.md b/docs/adr/049-job-filter-and-indexed-tags.md index 58860072b..2139a79ec 100644 --- a/docs/adr/049-job-filter-and-indexed-tags.md +++ b/docs/adr/049-job-filter-and-indexed-tags.md @@ -2,7 +2,7 @@ ## Status -Proposed. Amends [ADR-030](030-batch-operations.md): its batch-operation filter becomes the filter for job listing, counting, batch preview, batch execution and the DLQ. No migration. `awa migrate` and schema preparation index the partitions that are empty, so new schemas come out fully indexed; populated partitions are indexed by `awa storage build-tag-index`, which the migrator runs after upgrading. +Accepted. Amends [ADR-030](030-batch-operations.md): its batch-operation filter becomes the filter for job listing, counting, batch preview, batch execution and the DLQ. No migration. `awa migrate` and schema preparation index the partitions that are empty, so new schemas come out fully indexed; populated partitions are indexed by `awa storage build-tag-index`, which the migrator runs after upgrading. ## Context