From 0f9aba3330c028e8334e43e3b047704670573858 Mon Sep 17 00:00:00 2001 From: Brian Thorne Date: Sat, 31 Oct 2026 09:38:01 +1300 Subject: [PATCH 01/11] docs(adr): reserve ADR-050 for real-time subscriptions --- docs/adr/050-realtime-subscriptions.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 docs/adr/050-realtime-subscriptions.md diff --git a/docs/adr/050-realtime-subscriptions.md b/docs/adr/050-realtime-subscriptions.md new file mode 100644 index 000000000..41530a4dc --- /dev/null +++ b/docs/adr/050-realtime-subscriptions.md @@ -0,0 +1,14 @@ +# ADR-050: Real-time resource subscriptions + +## Status + +Proposed. Number reserved before authoring, per the ADR index conventions. + +## Context + +Design proposal for admin-first resource subscriptions: invalidation and refetch, +recovery after disconnect, bounded fanout, and the boundary with durable jobs. + +## Decision + +Pending analysis. This PR is documentation only. From e272bfe07e41312b188c93663d0dc7181001a5a2 Mon Sep 17 00:00:00 2001 From: Brian Thorne Date: Sat, 31 Oct 2026 09:43:31 +1300 Subject: [PATCH 02/11] docs(adr): propose resource invalidation and recovery over SSE --- docs/adr/050-realtime-subscriptions.md | 397 ++++++++++++++++++++++++- docs/adr/README.md | 2 + 2 files changed, 395 insertions(+), 4 deletions(-) diff --git a/docs/adr/050-realtime-subscriptions.md b/docs/adr/050-realtime-subscriptions.md index 41530a4dc..154f328a7 100644 --- a/docs/adr/050-realtime-subscriptions.md +++ b/docs/adr/050-realtime-subscriptions.md @@ -2,13 +2,402 @@ ## Status -Proposed. Number reserved before authoring, per the ADR index conventions. +Proposed. Documentation only; no subscription API is implemented by this record. +Number reserved in [#576](https://github.com/hardbyte/awa/pull/576). ## Context -Design proposal for admin-first resource subscriptions: invalidation and refetch, -recovery after disconnect, bounded fanout, and the boundary with durable jobs. +The first consumer is Awa's admin UI. Its question is broader than whether a job +changed state: is a queue falling behind, is a capable worker present, did rescue +recover progress, has a schedule reconciled, or has an operator's batch finished? +Applications may later want to refresh a view when their jobs complete. + +The priority order is strict: job correctness, reliability and recovery first; +performance second; UX third. A lost observation must never lose a job or become +the only trigger for business work. Few production users are moving from 0.6 to +0.7, so a simpler public interface is preferable to preserving an accidental one. +That freedom does not remove the database rolling-upgrade contract. + +Today the UI polls ordinary HTTP reads. Aggregate endpoints have TTL caches with +request coalescing; the server advertises a polling interval at least as large as +the cache TTL. Job detail and batch progress have their own intervals. This +proposal builds on those reads rather than creating another model of job state. + +[ADR-049](049-job-filter-and-indexed-tags.md) is accepted, although its index row +still says Proposed. Its implementation is separately under review in +[#572](https://github.com/hardbyte/awa/pull/572). Neither this proposal nor its +recommendation depends on that PR's independent adversarial review. ## Decision -Pending analysis. This PR is documentation only. +Use **resource invalidation followed by authoritative refetch**, transported over +**one multiplexed SSE connection per admin tab**. An invalidation means only +"this resource may have changed". Reconnect always resynchronizes current state; +there is no durable event cursor, transition replay or delivery acknowledgement. + +Start with a process-local hub, periodic resource invalidation, and immediate +hints after successful local admin mutations. Add no subscription writes, +triggers, outbox or mandatory listener to enqueue, claim, heartbeat, finalization +or rescue. Periodic client reconciliation remains mandatory even while SSE is +healthy. A failed or overloaded subscription system falls back to bounded reads. + +This deliberately leaves worker-originated updates at a sampling cadence in the +first delivery. SSE alone does not make them instantaneous or cheaper than +polling. Faster hints are a later, measured optimization; they cannot replace +reconciliation. Durable business reactions remain ordinary Awa jobs, using the +transactional mechanisms in [ADR-029](029-transactional-followup-jobs.md) and +[ADR-046](046-handler-staged-follow-ups.md) where applicable. + +### Resources, not storage events + +The vocabulary describes observable resources, not tables or event names such as +`job.completed`. These are candidate use cases and an extensible vocabulary, +not a commitment to build every resource in the first release: + +| Resource | What a consumer would refetch | +| --- | --- | +| `jobs(filter)` / `job(id)` | A filtered population or one job, including retained output and progress. | +| `queues(names?)` | Health, backlog, oldest waiting age, throughput and concurrency pressure. | +| `workers(ids?)` | Presence, liveness evidence, capabilities and queue coverage. | +| `recovery(queues?)` | Rescue and DLQ summaries or retained notices. | +| `schedules(names?)` | Definitions, pause/trigger lifecycle and reconciliation status. | +| `operations(ids?)` | Bulk operator action progress, cancellation and results. | +| `maintenance(scope?)` | Storage maintenance, pruning and transition progress. | + +"Authoritative refetch" means the normal read API, with its documented accuracy +and retention limits. It does not turn sampled queue health, worker heartbeats +or a capped count into an exact instantaneous census. Oldest waiting age changes +with time without a write; worker liveness expires without a disconnect event; +throughput needs a measured time window. Their reads must define those meanings +and expose observation time before subscriptions can improve their presentation. +An invalidation cannot invent a metric the read API does not yet supply. + +Application completion notifications use `job(id)` or `jobs(filter)` to refresh +current application-visible state. A job that completes and is pruned between +reads may never be observed as completed. A consumer needing that fact for a +business action must arrange a durable job or application record at the source. +Similarly, an ephemeral rescue/DLQ toast is not an audit trail; historical notices +need a separately specified retained read model. + +### Share JobFilter without a second evaluator + +`jobs(filter)` and DLQ populations use ADR-049's `JobFilter`, validation and wire +semantics: fields intersect, states are any-of, tags are all-of, and creation +bounds refer to original insertion. Reuse its strict deserializer and bounds; +reject unknown resource kinds, fields and unsupported protocol versions rather +than silently widening a subscription. Authorization is an additional scope, +never something inferred from a tag. + +Paging stays outside the population. A subscription invalidates the population; +the client refetches its bounded page using the normal page parameters. Filter +changes create a new client subscription identity and discard old responses. +There is no promise of a stable multi-page snapshot or an endless live list. + +Do not evaluate every transition against every subscriber's filter. Initially, +coarse resource-category invalidation is sufficient, even if it over-invalidates. +If later hints carry queue or job identity, narrow routing only when it is +provably safe; unknown routing information invalidates the broader category. +The read API, not hub routing, decides membership. + +This handles **leaving** a population as well as entering it. For example, a job +moving from queue A to B or from `running` to `completed` must invalidate its old +view too. Routing solely on the new row misses the leave. No before-images or +per-client membership sets are needed for coarse invalidation. The replacement +page removes departed jobs and fills vacancies from current matches. A short +transition entirely between reads can be invisible, by design. + +Counts retain ADR-049's cap, timeout and default policy of fetching on filter +change rather than polling. A population invalidation may mark the count stale; +it does not automatically run a count per notification. Explicit refresh or a +debounced refresh after an operator action can renew it. Live subscribed counts +would need their own measured read budget and an explicit change to that policy. + +### Establishing a view and recovering it + +The protocol should have a small versioned envelope: `ready` (connection epoch, +accepted resource identifiers and refresh guidance), `invalidate` (a bounded set +of resource identifiers), and `resync` (discard freshness assumptions for all +accepted resources). These names describe the proposal, not shipped endpoints. +An epoch is local connection identity, never a database commit position. + +1. Validate and authorize the complete requested resource set, register it with + the hub, then send `ready`. The client can display a preliminary read while + connecting, but must fetch again after `ready`. +2. Refetch accepted resources. Track a client-local dirty generation per resource + when invalidations arrive. A response does not clear invalidations received + since that fetch started: keep one pending refetch, paced by the read budget. + Render successful snapshots even under continuous traffic; never wait for + the stream to become quiet. +3. Ignore responses from superseded filters/connections and serialize or fence + requests per query key so an older response cannot overwrite a newer one. +4. Reconcile visible active resources periodically regardless of hints. Reconnect, + tab resume, server restart, overflow or a subscription change triggers a full + resync. Refetch failures retain stale status and retry with bounded backoff; + they do not count as successful reconciliation. + +This closes the ordinary fetch-before-subscribe gap without claiming an atomic +snapshot plus database stream. A missing hint or a job invisible during a claim +is repaired by a later read if that state is still retained and readable. The +guarantee is eventual convergence of observable retained state after changes +settle, given successful reads and fair scheduling. It is **not** observation of +every job or every transition. ADR-049 explicitly permits jobs mid-claim to be +absent or shown in their previous state because the lane cursor is outside MVCC. +Neither a repeatable-read transaction nor a local epoch removes that limitation. + +The first implementation should advertise configurable refresh and reconciliation +intervals. A reasonable starting experiment is the existing server poll interval +for periodic invalidations, a 30-second client reconciliation watchdog, and +immediate refetch on reconnect/resume. These are proposed defaults, not measured +SLOs. Effective freshness also includes scheduler delays, upstream telemetry +refresh, query/cache time and replica lag; transport heartbeats prove none of +those are current. Display the last successful observation time and stale/error +state. Keep existing polling when the stream is unsupported or unhealthy. + +### Producing invalidations safely + +Each admin process maintains a bounded hub for its active resource categories. +Its timer emits coarse invalidations at the advertised cadence without scanning +jobs or comparing every filter. Local mutation handlers invalidate affected +server caches and signal the hub only after their database operation commits. +Other processes' mutations and worker activity are covered by periodic reads; +there is no dependency on a worker or maintenance leader emitting UI hints. + +Refetch must not repeatedly return a server cache entry from before an immediate +hint. Existing dashboard cache invalidation is a starting point, but races with +an already in-flight cache fill need generation fencing: a fill started before +invalidation must not republish itself as fresh afterwards. Timer-driven refetch +must respect cache expiry. Retaining a bounded TTL and later reconciliation is +the recovery backstop; bypassing caches on every event would defeat coalescing. + +No transaction or database connection is held for the life of an SSE stream. +No new writer permission is needed, so read-only admin instances work too. +Against a replica, refresh reads that replica's state and reports its lag/unknown +freshness; a primary hint would not prove replay has caught up. The initial +timer design works through transaction-mode poolers and on replicas without +requiring `LISTEN` support. + +### Bounded work and multiple admin instances + +Merge repeated invalidations into a dirty set, with at most one pending marker +per admitted resource. Use a short debounce with a maximum delay, one in-flight +read per query key, bounded read concurrency, statement timeouts and jitter. +Resource counts, descriptor bytes, connections per principal/instance, event +size, pending bytes and send time must all have hard limits before exposure. +Exact limits and admission behavior require load evidence; they are not supplied +by this ADR's design analysis. + +Overflow collapses to one all-resources `resync`, rather than a queue of changes. +If even that cannot be delivered promptly, close the connection; reconnect +resynchronizes. Slow clients never block the shared hub, a database reader, +workers or producers. Cap fanout and connection admission; answer excess +admission with a retryable failure and continue paced polling. Under sustained +database overload, serve stale/error state and reduce refresh frequency rather +than building an unbounded retry backlog. A subscription is optional visibility, +not an obligation to keep every browser fresh at the expense of jobs. + +Share aggregate reads through the existing typed caches. If filtered-page reads +are later coalesced, key them by backend, authorization scope, full normalized +filter and page; bound cache cardinality. Do not introduce an unbounded cache +entry or database task per distinct filter. One hub fanout is still proportional +to admitted connections, and uncached reads grow with active client views. + +Several admin processes may serve the same database. Each owns its own caches, +timer and clients; no leader, broker, sticky routing or shared browser session +is required. A load balancer can reconnect a client to any instance, which sends +`ready` and requires refetch again. An immediate local hint need not reach peer +instances; their cadence determines freshness. Independent read cost grows with +instance count, so measure the combined budget. Links to UIs serving different +databases remain navigation only, as they are today. + +### Read scope and transport + +Subscriptions are read-only and inherit the read API's trust boundary. Awa's +admin currently has no built-in authentication; ingress/private-network controls +must protect streams as well as reads. `--read-only` disables mutations, not +access to sensitive job data. Start with this trusted admin boundary; do not +expose it as an application tenant API merely because it accepts filters. + +When scoped consumers are introduced, authorize both subscription admission and +every refetch, key shared state by authorization scope, and close/resync streams +when access changes. Reject unauthorized requested resources before `ready`. +Events must not reveal unauthorized job ids, queue names or activity timing. +The stream must have an explicit same-origin/CORS and credential policy rather +than inherit the current router's permissive CORS accidentally. Never put tokens, +args, output, raw filters or secrets into URLs, logs or notification payloads. + +Choose SSE because the contract is one-way hints with ordinary HTTP reads and +mutations. Resource-set changes replace the connection, with overlap bounded and +old responses fenced; no bidirectional command protocol is needed. Use an HTTP +streaming client where authenticated headers or a structured request body are +needed; native `EventSource` is optional, not the API boundary. Reuse the exact +JobFilter wire shape inside a versioned bounded request. Keep native URL-based +subscription descriptors to non-sensitive resource identifiers only. + +[SSE](https://html.spec.whatwg.org/multipage/server-sent-events.html) supplies +event framing and browser reconnect behavior, not replay storage. Do not emit an +SSE `id` implying resumable history; `Last-Event-ID` cannot suppress the required +resync. Proxy buffering, idle timeouts, authentication expiry, tab suspension and +HTTP connection limits need deployment tests. Send small keepalives, disable +proxy response buffering where required, and fall back to polling if streaming +is unavailable. Non-browser clients can consume the same envelope later. + +## Alternatives considered + +### Durable event log or transactional outbox + +A semantic append-only log could support replay after downtime and retained +notices. An outbox written in the job transaction could couple a fact atomically +to a transition, unlike a post-commit in-process hook. This is the right family +of design if observing every retained event becomes an actual requirement. + +It also requires complete coverage of Rust/Python/raw/bridge inserts, callbacks, +rescue, compact completion batches, bulk actions, schedule reconciliation and +pruning. It adds WAL, retention, schema/version contracts, publication lag and +write-path failure modes for a UI that needs current state. A sequence allocated +before commit is not a safe replay watermark: a later id may commit first and +cause a consumer to skip the earlier transaction. Commit-safe ordering, an +initial snapshot boundary, idempotence, gap detection, retention floors and +expired-cursor resync all need a proven protocol. + +Not selected. It would duplicate durable reaction machinery already expressed by +queue jobs. The proposed stream has no event table, consumer offset or replay +retention to prune. Ordinary job/output/operation retention still applies, and +pruning can make resources disappear; reads must not present absence as proof +that a job never existed. + +### PostgreSQL LISTEN / NOTIFY as delivery or hints + +[PostgreSQL NOTIFY](https://www.postgresql.org/docs/17/sql-notify.html) delivers +after commit to connected listeners, can fold duplicates, has a small payload +limit, and exposes notifications to database users. A full notification queue +can make notifying transactions fail at commit. It is not replayable history. +Adding it to every transition would couple optional UI delivery to job commits. + +Not selected as delivery or as a new mandatory write-path mechanism. Existing +queue wakeups are incomplete hints for this vocabulary: they do not mean that +completion, worker expiry, schedules or prune changes were all observed. A later +optimization may use one dedicated session listener per admin process, retaining +timers and broad resync on listener reconnect. It must not create one PostgreSQL +connection per browser, rely on transaction-mode poolers for session state, or +send untrusted payloads straight to clients. + +The [LISTEN startup rule](https://www.postgresql.org/docs/17/sql-listen.html) is +to commit registration before inspecting state. Even when followed, it does not +repair an incomplete signal source or a missed notification. Any new emission +path needs measured commit/WAL cost and review against +[ADR-045](045-hot-path-trigger-lock-contract.md); no trigger change is implied. + +### Logical decoding / CDC + +[Logical decoding](https://www.postgresql.org/docs/17/logicaldecoding-explanation.html) +can provide a database change stream with an exported snapshot boundary. It has +replication-slot, privilege and failover requirements; consumers must tolerate +duplicates after crashes. A stalled slot can retain WAL and catalog resources. + +Awa's physical representation includes ready rows, cursors outside MVCC, leases, +receipts, compact completions and partition truncation. Decoding those into +public resource changes would expose storage evolution or require a semantic +adapter with its own recovery proof. CDC remains useful for external history +and analytics as documented in +[managed Postgres](../deploying-on-managed-postgres.md#logical-replication-and-cdc), +but is not a required dependency for the admin UI. + +### Polling only, pushed snapshots, and WebSocket + +**Keep polling only.** This has the fewest moving parts and remains a valid +deployment/fallback. The initial timer-driven SSE design does not claim fewer +database reads. SSE buys a common resource vocabulary and immediate local +invalidation, with room for safe hints later. If that benefit does not justify +connection/proxy complexity in an implementation benchmark, retain polling +behind the same vocabulary instead of shipping an idle streaming layer. + +**Push snapshots or row deltas.** Shared sampled aggregate snapshots can save +refetch round trips, but filtered job snapshots add per-subscriber query/state +work. Row deltas need correct enter/leave, paging and deletion semantics plus +the same recovery machinery. Start with refetch; consider pushed aggregate +snapshots only when measurements show the read cost matters. + +**WebSocket.** It supports bidirectional subscription changes without reopening +and would suit a later interactive command protocol. This design needs neither; +it would add framing, reconnect and connection-state machinery without stronger +delivery guarantees. Revisit if resource-set churn or actual bidirectional +requirements justify it. The envelope and recovery semantics should survive a +transport change. + +## Consequences + +- Job execution and durable reactions remain independent of browser health. + There is no new schema migration, per-job write tax or notification retention + job. Admin read load still competes for database resources and must be bounded. +- Consumers get convergent views, not a replayable event API. Transient states, + completions before pruning and historical recovery notices may be missed. +- Coarse invalidation simplifies correctness at the cost of extra reads and + cadence-limited remote updates. Finer routing must not weaken leave/resync + behavior. Low-latency coverage is explicitly incomplete in the first stage. +- The initial public surface can change for 0.7, but must negotiate capabilities + and reject unsupported subscription shapes. An old server gets ordinary reads + and polling only for filter semantics it supports; otherwise show an unsupported + filter error. Never silently drop predicates to make fallback work. + +## Validation required before implementation ships + +This is a design recommendation grounded in the sources below, not a subscription +prototype or benchmark. Implementation must demonstrate: + +- A write before registration, during the first read, during refetch and while + disconnected; dropped/duplicate/coalesced hints; server and listener restart; + a filter change with a late response; and convergence after an in-flight claim. +- Enter and leave for state/queue changes, retries, DLQ move/retry/purge, + progress/output, pruning, schedule changes and bulk-operation progress. + An event wholly between samples is allowed to be invisible; tests must not + accidentally assert transition delivery. +- Cache-fill/invalidation races, read errors/timeouts, an empty install, + old-server fallback, a read-only replica with lag, and two admin processes + handling mutations and reconnects independently. +- Hard admission and memory bounds under slow/disconnected clients, maximum + distinct filters and event storms, with prompt resync/close and fair refresh + under continuous traffic. Verify database pool/connection counts and idle + cleanup, not just heap bounds. +- Unauthorized admission/refetch/revocation, origin/credential handling, and a + streaming deployment through the supported ingress/pooler topology. +- Job throughput and p95/p99 enqueue/claim/finalization latency, WAL per job, + admin query concurrency/QPS and browser refresh latency with zero clients, + polling, SSE, many clients and multiple admin instances. Set numeric budgets + and overload defaults from those measurements before enabling by default. + +Unresolved implementation choices are exact endpoint/request encoding, numeric +admission limits, cadence and stale-state SLOs, proxy/auth integration, and which +candidate resource reads already have adequate semantics. Application-scoped +authorization and historical notice retention require further design before +those consumers are exposed. None changes the invalidation/reconciliation +contract; a requirement for replay or guaranteed completion observation would. + +## Evidence and relationship to existing decisions + +Inspected at main `f644f321ad110c6436382b64bdc0d935dcdeba5a` and #572 head +`512f477785d327ac092b94c6a399db4a8b5c4555` on 2026-10-30: + +- [Dashboard cache](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/cache.rs) + and [AppState](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/state.rs): + typed TTL caches, request coalescing, poll guidance and local invalidation. +- [Frontend polling](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/frontend/src/hooks/use-poll-interval.ts) + and [router](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/lib.rs): + ordinary reads, existing resource families and current permissive CORS. +- [Existing worker notification hub](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-worker/src/notify_hub.rs): + a shared listener, poll fallback and broad wakeup after reconnect; precedent + for hints with recovery, not evidence of full resource change coverage. +- [#572 JobFilter](https://github.com/hardbyte/awa/blob/512f477785d327ac092b94c6a399db4a8b5c4555/awa-model/src/filter.rs) + and [job reads](https://github.com/hardbyte/awa/blob/512f477785d327ac092b94c6a399db4a8b5c4555/awa-ui/src/handlers/jobs.rs): + strict population semantics and shared bounded reads under implementation. +- [ADR-049](049-job-filter-and-indexed-tags.md): shared filter, advisory + in-flight reads, capped counts, tag retention and fleet compatibility. + [ADR-030](030-batch-operations.md) owns durable operator actions; + subscriptions only observe their progress. +- [ADR-029](029-transactional-followup-jobs.md) and + [ADR-046](046-handler-staged-follow-ups.md) own durable follow-up work. + This proposal neither replaces nor strengthens the guarantees of those paths. +- [ADR-041](041-rolling-upgrade-policy.md) and + [ADR-045](045-hot-path-trigger-lock-contract.md) remain binding for any future + schema or hot-path hint changes. The + [operations skill](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/skills/awa-operations/SKILL.md) + documents the trusted admin boundary and read-only operation. diff --git a/docs/adr/README.md b/docs/adr/README.md index 8934179c4..cf0dd5fe7 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -57,6 +57,8 @@ Each record preserves its status, context, decision, consequences, and alternati | 048 | [Retained job output](048-retained-job-output.md) | A handler may record a size-bounded JSON output for its attempt, sealed with the staging buffer and published only by the attempt's own lease-guarded transition: as the job's output on completion, or in the error entry of a failure, where only the latest entry to record one keeps it. It rides the compact completion batch, and neither the engine nor another job reads it. A kind may declare its output type, which types the API and is published in its descriptor. | Accepted | | 049 | [One job filter, and tags as the indexed handle](049-job-filter-and-indexed-tags.md) | One `JobFilter` describes "which jobs" for listing, counting, batch preview, batch execution and the DLQ, compiled once per engine with only the predicates a filter sets; SQL narrows candidates and a batched loader settles each job's current state. Tags are the indexed handle for "jobs about X": a partial GIN index over non-empty payload tags on each queue-storage partition, maintained by Postgres for every binary, emptied by pruning, and built online by the migrator. | Proposed | +| 050 | [Real-time resource subscriptions](050-realtime-subscriptions.md) | Admin-first resource invalidation over SSE with bounded refetch, reconnect resync and periodic reconciliation; durable business triggers remain queue jobs. | Proposed | + ## Correctness evidence Executable TLA+ models live under [`correctness/`](https://github.com/hardbyte/awa/tree/main/correctness). The storage models cover segmented storage, storage races, lock ordering, and trace refinement; the runtime models cover claim, rescue, callbacks, batching, cron, and view-trigger concurrency. Benchmark evidence belongs with the benchmark artifacts, not in this decision index. From 3dd7ee3696e706d80c4a2777d7d0642f1e3480cc Mon Sep 17 00:00:00 2001 From: Brian Thorne Date: Sat, 31 Oct 2026 09:45:15 +1300 Subject: [PATCH 03/11] docs(adr): keep subscription entry inside ADR index table --- docs/adr/README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/adr/README.md b/docs/adr/README.md index cf0dd5fe7..d57e19ef1 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -56,7 +56,6 @@ Each record preserves its status, context, decision, consequences, and alternati | 046 | [Handler-staged follow-up jobs](046-handler-staged-follow-ups.md) | A handler stages follow-up requests on its `JobContext`; they are inserted in the attempt's lease-guarded completion transaction, deduplicated by their own unique keys, and discarded on every other outcome. | Accepted | | 048 | [Retained job output](048-retained-job-output.md) | A handler may record a size-bounded JSON output for its attempt, sealed with the staging buffer and published only by the attempt's own lease-guarded transition: as the job's output on completion, or in the error entry of a failure, where only the latest entry to record one keeps it. It rides the compact completion batch, and neither the engine nor another job reads it. A kind may declare its output type, which types the API and is published in its descriptor. | Accepted | | 049 | [One job filter, and tags as the indexed handle](049-job-filter-and-indexed-tags.md) | One `JobFilter` describes "which jobs" for listing, counting, batch preview, batch execution and the DLQ, compiled once per engine with only the predicates a filter sets; SQL narrows candidates and a batched loader settles each job's current state. Tags are the indexed handle for "jobs about X": a partial GIN index over non-empty payload tags on each queue-storage partition, maintained by Postgres for every binary, emptied by pruning, and built online by the migrator. | Proposed | - | 050 | [Real-time resource subscriptions](050-realtime-subscriptions.md) | Admin-first resource invalidation over SSE with bounded refetch, reconnect resync and periodic reconciliation; durable business triggers remain queue jobs. | Proposed | ## Correctness evidence From 14554cf522d247bc567d362eca60b0b7e4ff08aa Mon Sep 17 00:00:00 2001 From: Brian Thorne Date: Sat, 31 Oct 2026 10:07:20 +1300 Subject: [PATCH 04/11] docs(adr): adopt bounded opt-in subscription v1 contract --- docs/adr/050-realtime-subscriptions.md | 430 ++++++++++------------- docs/adr/README.md | 2 +- docs/design/realtime-subscriptions-v1.md | 428 ++++++++++++++++++++++ 3 files changed, 612 insertions(+), 248 deletions(-) create mode 100644 docs/design/realtime-subscriptions-v1.md diff --git a/docs/adr/050-realtime-subscriptions.md b/docs/adr/050-realtime-subscriptions.md index 154f328a7..24a7b696b 100644 --- a/docs/adr/050-realtime-subscriptions.md +++ b/docs/adr/050-realtime-subscriptions.md @@ -25,221 +25,153 @@ proposal builds on those reads rather than creating another model of job state. [ADR-049](049-job-filter-and-indexed-tags.md) is accepted, although its index row still says Proposed. Its implementation is separately under review in -[#572](https://github.com/hardbyte/awa/pull/572). Neither this proposal nor its -recommendation depends on that PR's independent adversarial review. +[#572](https://github.com/hardbyte/awa/pull/572). The initial recommendation was +formed independently of the separate filtering review. Subsequent document-specific review amendments are recorded below. ## Decision -Use **resource invalidation followed by authoritative refetch**, transported over -**one multiplexed SSE connection per admin tab**. An invalidation means only -"this resource may have changed". Reconnect always resynchronizes current state; -there is no durable event cursor, transition replay or delivery acknowledgement. - -Start with a process-local hub, periodic resource invalidation, and immediate -hints after successful local admin mutations. Add no subscription writes, -triggers, outbox or mandatory listener to enqueue, claim, heartbeat, finalization -or rescue. Periodic client reconciliation remains mandatory even while SSE is -healthy. A failed or overloaded subscription system falls back to bounded reads. - -This deliberately leaves worker-originated updates at a sampling cadence in the -first delivery. SSE alone does not make them instantaneous or cheaper than -polling. Faster hints are a later, measured optimization; they cannot replace -reconciliation. Durable business reactions remain ordinary Awa jobs, using the -transactional mechanisms in [ADR-029](029-transactional-followup-jobs.md) and -[ADR-046](046-handler-staged-follow-ups.md) where applicable. - -### Resources, not storage events - -The vocabulary describes observable resources, not tables or event names such as -`job.completed`. These are candidate use cases and an extensible vocabulary, -not a commitment to build every resource in the first release: - -| Resource | What a consumer would refetch | -| --- | --- | -| `jobs(filter)` / `job(id)` | A filtered population or one job, including retained output and progress. | -| `queues(names?)` | Health, backlog, oldest waiting age, throughput and concurrency pressure. | -| `workers(ids?)` | Presence, liveness evidence, capabilities and queue coverage. | -| `recovery(queues?)` | Rescue and DLQ summaries or retained notices. | -| `schedules(names?)` | Definitions, pause/trigger lifecycle and reconciliation status. | -| `operations(ids?)` | Bulk operator action progress, cancellation and results. | -| `maintenance(scope?)` | Storage maintenance, pruning and transition progress. | - -"Authoritative refetch" means the normal read API, with its documented accuracy -and retention limits. It does not turn sampled queue health, worker heartbeats -or a capped count into an exact instantaneous census. Oldest waiting age changes -with time without a write; worker liveness expires without a disconnect event; -throughput needs a measured time window. Their reads must define those meanings -and expose observation time before subscriptions can improve their presentation. -An invalidation cannot invent a metric the read API does not yet supply. - -Application completion notifications use `job(id)` or `jobs(filter)` to refresh -current application-visible state. A job that completes and is pruned between -reads may never be observed as completed. A consumer needing that fact for a -business action must arrange a durable job or application record at the source. -Similarly, an ephemeral rescue/DLQ toast is not an audit trail; historical notices -need a separately specified retained read model. - -### Share JobFilter without a second evaluator - -`jobs(filter)` and DLQ populations use ADR-049's `JobFilter`, validation and wire -semantics: fields intersect, states are any-of, tags are all-of, and creation -bounds refer to original insertion. Reuse its strict deserializer and bounds; -reject unknown resource kinds, fields and unsupported protocol versions rather -than silently widening a subscription. Authorization is an additional scope, -never something inferred from a tag. - -Paging stays outside the population. A subscription invalidates the population; -the client refetches its bounded page using the normal page parameters. Filter -changes create a new client subscription identity and discard old responses. -There is no promise of a stable multi-page snapshot or an endless live list. - -Do not evaluate every transition against every subscriber's filter. Initially, -coarse resource-category invalidation is sufficient, even if it over-invalidates. -If later hints carry queue or job identity, narrow routing only when it is -provably safe; unknown routing information invalidates the broader category. -The read API, not hub routing, decides membership. - -This handles **leaving** a population as well as entering it. For example, a job -moving from queue A to B or from `running` to `completed` must invalidate its old -view too. Routing solely on the new row misses the leave. No before-images or -per-client membership sets are needed for coarse invalidation. The replacement -page removes departed jobs and fills vacancies from current matches. A short -transition entirely between reads can be invisible, by design. - -Counts retain ADR-049's cap, timeout and default policy of fetching on filter -change rather than polling. A population invalidation may mark the count stale; -it does not automatically run a count per notification. Explicit refresh or a -debounced refresh after an operator action can renew it. Live subscribed counts -would need their own measured read budget and an explicit change to that policy. - -### Establishing a view and recovering it - -The protocol should have a small versioned envelope: `ready` (connection epoch, -accepted resource identifiers and refresh guidance), `invalidate` (a bounded set -of resource identifiers), and `resync` (discard freshness assumptions for all -accepted resources). These names describe the proposal, not shipped endpoints. -An epoch is local connection identity, never a database commit position. - -1. Validate and authorize the complete requested resource set, register it with - the hub, then send `ready`. The client can display a preliminary read while - connecting, but must fetch again after `ready`. -2. Refetch accepted resources. Track a client-local dirty generation per resource - when invalidations arrive. A response does not clear invalidations received - since that fetch started: keep one pending refetch, paced by the read budget. - Render successful snapshots even under continuous traffic; never wait for - the stream to become quiet. -3. Ignore responses from superseded filters/connections and serialize or fence - requests per query key so an older response cannot overwrite a newer one. -4. Reconcile visible active resources periodically regardless of hints. Reconnect, - tab resume, server restart, overflow or a subscription change triggers a full - resync. Refetch failures retain stale status and retry with bounded backoff; - they do not count as successful reconciliation. - -This closes the ordinary fetch-before-subscribe gap without claiming an atomic -snapshot plus database stream. A missing hint or a job invisible during a claim -is repaired by a later read if that state is still retained and readable. The -guarantee is eventual convergence of observable retained state after changes -settle, given successful reads and fair scheduling. It is **not** observation of -every job or every transition. ADR-049 explicitly permits jobs mid-claim to be -absent or shown in their previous state because the lane cursor is outside MVCC. -Neither a repeatable-read transaction nor a local epoch removes that limitation. - -The first implementation should advertise configurable refresh and reconciliation -intervals. A reasonable starting experiment is the existing server poll interval -for periodic invalidations, a 30-second client reconciliation watchdog, and -immediate refetch on reconnect/resume. These are proposed defaults, not measured -SLOs. Effective freshness also includes scheduler delays, upstream telemetry -refresh, query/cache time and replica lag; transport heartbeats prove none of -those are current. Display the last successful observation time and stale/error -state. Keep existing polling when the stream is unsupported or unhealthy. - -### Producing invalidations safely - -Each admin process maintains a bounded hub for its active resource categories. -Its timer emits coarse invalidations at the advertised cadence without scanning -jobs or comparing every filter. Local mutation handlers invalidate affected -server caches and signal the hub only after their database operation commits. -Other processes' mutations and worker activity are covered by periodic reads; -there is no dependency on a worker or maintenance leader emitting UI hints. - -Refetch must not repeatedly return a server cache entry from before an immediate -hint. Existing dashboard cache invalidation is a starting point, but races with -an already in-flight cache fill need generation fencing: a fill started before -invalidation must not republish itself as fresh afterwards. Timer-driven refetch -must respect cache expiry. Retaining a bounded TTL and later reconciliation is -the recovery backstop; bypassing caches on every event would defeat coalescing. - -No transaction or database connection is held for the life of an SSE stream. -No new writer permission is needed, so read-only admin instances work too. -Against a replica, refresh reads that replica's state and reports its lag/unknown -freshness; a primary hint would not prove replay has caught up. The initial -timer design works through transaction-mode poolers and on replicas without -requiring `LISTEN` support. - -### Bounded work and multiple admin instances - -Merge repeated invalidations into a dirty set, with at most one pending marker -per admitted resource. Use a short debounce with a maximum delay, one in-flight -read per query key, bounded read concurrency, statement timeouts and jitter. -Resource counts, descriptor bytes, connections per principal/instance, event -size, pending bytes and send time must all have hard limits before exposure. -Exact limits and admission behavior require load evidence; they are not supplied -by this ADR's design analysis. - -Overflow collapses to one all-resources `resync`, rather than a queue of changes. -If even that cannot be delivered promptly, close the connection; reconnect -resynchronizes. Slow clients never block the shared hub, a database reader, -workers or producers. Cap fanout and connection admission; answer excess -admission with a retryable failure and continue paced polling. Under sustained -database overload, serve stale/error state and reduce refresh frequency rather -than building an unbounded retry backlog. A subscription is optional visibility, -not an obligation to keep every browser fresh at the expense of jobs. - -Share aggregate reads through the existing typed caches. If filtered-page reads -are later coalesced, key them by backend, authorization scope, full normalized -filter and page; bound cache cardinality. Do not introduce an unbounded cache -entry or database task per distinct filter. One hub fanout is still proportional -to admitted connections, and uncached reads grow with active client views. - -Several admin processes may serve the same database. Each owns its own caches, -timer and clients; no leader, broker, sticky routing or shared browser session -is required. A load balancer can reconnect a client to any instance, which sends -`ready` and requires refetch again. An immediate local hint need not reach peer -instances; their cadence determines freshness. Independent read cost grows with -instance count, so measure the combined budget. Links to UIs serving different -databases remain navigation only, as they are today. - -### Read scope and transport - -Subscriptions are read-only and inherit the read API's trust boundary. Awa's -admin currently has no built-in authentication; ingress/private-network controls -must protect streams as well as reads. `--read-only` disables mutations, not -access to sensitive job data. Start with this trusted admin boundary; do not -expose it as an application tenant API merely because it accepts filters. - -When scoped consumers are introduced, authorize both subscription admission and -every refetch, key shared state by authorization scope, and close/resync streams -when access changes. Reject unauthorized requested resources before `ready`. -Events must not reveal unauthorized job ids, queue names or activity timing. -The stream must have an explicit same-origin/CORS and credential policy rather -than inherit the current router's permissive CORS accidentally. Never put tokens, -args, output, raw filters or secrets into URLs, logs or notification payloads. - -Choose SSE because the contract is one-way hints with ordinary HTTP reads and -mutations. Resource-set changes replace the connection, with overlap bounded and -old responses fenced; no bidirectional command protocol is needed. Use an HTTP -streaming client where authenticated headers or a structured request body are -needed; native `EventSource` is optional, not the API boundary. Reuse the exact -JobFilter wire shape inside a versioned bounded request. Keep native URL-based -subscription descriptors to non-sensitive resource identifiers only. - -[SSE](https://html.spec.whatwg.org/multipage/server-sent-events.html) supplies -event framing and browser reconnect behavior, not replay storage. Do not emit an -SSE `id` implying resumable history; `Last-Event-ID` cannot suppress the required -resync. Proxy buffering, idle timeouts, authentication expiry, tab suspension and -HTTP connection limits need deployment tests. Send small keepalives, disable -proxy response buffering where required, and fall back to polling if streaming -is unavailable. Non-browser clients can consume the same envelope later. +Use **resource invalidation followed by authoritative refetch** over one +multiplexed **fetch-based SSE connection per visible admin tab**. An invalidation +means only "this resource may have changed". Reconnect resynchronizes current +state; there is no durable cursor, transition replay or acknowledgement. + +The v1 implementation contract is +[Real-time subscriptions v1](../design/realtime-subscriptions-v1.md). It fixes +`POST /api/subscriptions/v1`, strict versioned JSON reads, finite experimental +limits, one refresh scheduler, cache-generation fences and the trusted-admin +origin policy. These are requirements of this proposal, not optional follow-up +choices. **Streaming is disabled by default** and requires explicit server +opt-in until load and deployment validation supports default enablement. + +Start with a process-local hub, periodic invalidation and hints after committed +local admin mutations. Opening a stream performs zero database connections and +zero state queries. Add no subscription writes, triggers, outbox, transactional +directory, mandatory listener or worker changes on enqueue, claim, heartbeat, +finalization or rescue. Neither delivery nor reconciliation depends on a worker +or maintenance leader producing notifications. + +Worker-originated updates remain cadence-limited in v1. SSE alone makes them +neither instantaneous nor cheaper than polling. Durable business reactions remain +ordinary Awa jobs, using [ADR-029](029-transactional-followup-jobs.md) and +[ADR-046](046-handler-staged-follow-ups.md) where applicable. Existing rescue +follow-ups remain best effort after rescue; this proposal does not strengthen +them. + +### Resources and shared filters + +V1 admits `jobs`, `job`, `dlq`, `queues`, `workers`, `schedules`, `operations`, +`maintenance` and `stats`, with exact descriptors and existing read mappings in +[the resource contract](../design/realtime-subscriptions-v1.md#initial-resources). +These cover current admin observations, not new metrics. Queue backlog, oldest +waiting age, throughput and concurrency pressure may only use existing fields +with documented accuracy. Presence is heartbeat evidence, and age can change +without a write; stream traffic proves neither liveness nor source freshness. + +Standalone recovery resources, retained notices/history, named queue/worker/ +schedule selectors and application consumers are excluded from v1. A DLQ view +can observe current retained state, but an ephemeral notice is not an audit +trail. `/storage?history=true` does not provide server-retained history. Catalog +and kinds views keep their existing read behavior. + +`jobs` and `dlq` share ADR-049's `JobFilter` deserialization **and validation**. +Fields intersect, states are any-of, tags are all-of, and creation bounds refer +to original insertion. Paging stays separate. The hub routes coarse categories +and retains keys/category mappings rather than evaluating a second filter. +Thus a queue/state move invalidates both old and new populations; a replacement +page removes departed jobs and refills vacancies. Routing solely on a new row +would miss the leave. Filter changes abort/fence old streams and reads. + +Capabilities are advisory across mixed-server routing. New filtered reads use +strict v1 JSON POST adapters; every response must agree on MIME, envelope and +version. An old server may return SPA HTML with HTTP 200, which is an error. +Legacy fallback permits unfiltered lists/details only; a nonempty filter remains +unsupported, never rewritten by dropping predicates. Counts retain ADR-049's +cap and are excluded from automatic refresh; invalidation only marks them stale. + +### Recovery uses one bounded scheduler + +Register and validate the complete resource set before `ready`, then refetch. +Client dirty generations preserve a hint received during a read as one pending +refresh, and superseded responses cannot overwrite the new view. Render a +successful response even if newer demand remains; do not wait for quiet traffic. + +One tab-level scheduler handles hints, a jittered 30-second reconciliation +watchdog, fallback polling, focus/resume, reconnect and mutations. Managed queries +have no independent automatic intervals or focus/reconnect fetches. Hints and +keepalives do not postpone the watchdog. Every demand, including resync, obeys +the same concurrency, cooldown, admission and retry budgets. A transport failure +must not add a second polling loop. Hidden tabs pause reads and streams; resume +requests one paced resync. Selected jobs remain paused and stale until resumed, +so refreshing cannot replace a selection underneath an operator. + +The guarantee is convergence to retained state returned by ordinary APIs after +changes settle and reads succeed. It is not observation of every job or +transition, nor stronger cross-table consistency. Claims can be absent or stale +while the lane cursor is outside MVCC; ordinary deferred-to-ready handoffs can +also be missed by a composite read. Repeatable Read alone does not make ring +`TRUNCATE` MVCC-safe. A local epoch is not a database position. A completion +pruned before a read may never be observed; a reliable business reaction needs +a durable job or application record at the source. + +The experimental defaults bound a stream to 16 descriptors and 64 KiB, a process +to 64 streams and four expensive reads, and a tab to two concurrent reads. A +whole read has a 15-second deadline covering admission, pool acquisition, every +query/pass and serialization, with SQL and lock-wait limits inside it. Releasing +a Rust future is not proof that PostgreSQL work stopped: cancellation and pool +reuse must be tested. Full limits, overflow/slow-consumer behavior and the +60-second maximum retry backoff are in the companion contract. + +### Server caches, observations and multiple instances + +A post-commit helper advances a bounded cache generation, retires its old cache +entries, then signals the hub. A fill may publish/return as current only if its +captured generation still matches. New requests cannot join an obsolete fill; +retries stay within the remaining read budget. Moka `invalidate_all()` alone is +not an established proof. Client dirty generations and server cache generations +solve different races and both are required. + +Read responses preserve source observation time on cache hits and distinguish +`observed_at`, `served_at` and `cache_age_ms`. Report replica freshness as unknown +unless measured. A primary hint does not prove a replica has replayed the change. +Cache hits may bypass expensive-read admission, but each fill acquires it once. +Slow clients cannot hold a database connection or block shared fanout; dirty +state collapses to resync, then close if it cannot be delivered within budget. + +Every admin process owns its hub, caches and timer. Peer mutations and worker +activity converge through periodic reads; no broker, leader or sticky routing is +needed. Reconnection to a different instance always resynchronizes. Combined +read cost still grows with active views and process count and must be measured. +Links to UIs for different databases remain navigation only. + +### Trusted-admin scope and transport + +V1 is the existing trusted admin surface, protected by ingress/private networking. +It adds no tenant authorization framework. `--read-only` disables mutations, +not access to sensitive job data. New stream/read adapters use same-origin JSON +POSTs, explicit configured-origin checks and credential policy; an outer +permissive CORS layer must not undo that boundary. An origin check neither +authorizes a filter nor makes a public unauthenticated admin safe. + +Never place credentials, args, outputs or secrets in any URL. Do not place raw +filter descriptors in subscription URLs or automatic subscription logs. Existing +shareable search URLs may contain documented nonsecret opaque filter values. +Descriptors travel in JSON request bodies and are not echoed in events/logs. + +Choose SSE because the contract is one-way hints with ordinary HTTP mutations +and reads. V1 uses `fetch` plus AbortController; it has no native EventSource GET +variant, registration service or server-side subscription session. Resource-set +changes abort the old stream before opening its replacement, retaining displayed +data as stale. The client implements bounded parsing and reconnect/backoff; +The [SSE standard](https://html.spec.whatwg.org/multipage/server-sent-events.html) +defines framing; EventSource's automatic reconnect does not apply to fetch. +Epochs are connection identity, +SSE `id` is absent, and `Last-Event-ID` never bypasses resync. + +Five-minute stream lifetimes permit ingress reauthentication but do not promise +immediate per-user revocation in a server without identity. Deployment checks +must cover the actual authentication proxy, buffering, compression and timeouts. +Protocol/admission errors follow the contract's status/MIME and retry rules, +including respecting `Retry-After` for both streams and refetches. ## Alternatives considered @@ -336,41 +268,36 @@ transport change. behavior. Low-latency coverage is explicitly incomplete in the first stage. - The initial public surface can change for 0.7, but must negotiate capabilities and reject unsupported subscription shapes. An old server gets ordinary reads - and polling only for filter semantics it supports; otherwise show an unsupported - filter error. Never silently drop predicates to make fallback work. - -## Validation required before implementation ships - -This is a design recommendation grounded in the sources below, not a subscription -prototype or benchmark. Implementation must demonstrate: - -- A write before registration, during the first read, during refetch and while - disconnected; dropped/duplicate/coalesced hints; server and listener restart; - a filter change with a late response; and convergence after an in-flight claim. -- Enter and leave for state/queue changes, retries, DLQ move/retry/purge, - progress/output, pruning, schedule changes and bulk-operation progress. - An event wholly between samples is allowed to be invisible; tests must not - accidentally assert transition delivery. -- Cache-fill/invalidation races, read errors/timeouts, an empty install, - old-server fallback, a read-only replica with lag, and two admin processes - handling mutations and reconnects independently. -- Hard admission and memory bounds under slow/disconnected clients, maximum - distinct filters and event storms, with prompt resync/close and fair refresh - under continuous traffic. Verify database pool/connection counts and idle - cleanup, not just heap bounds. -- Unauthorized admission/refetch/revocation, origin/credential handling, and a - streaming deployment through the supported ingress/pooler topology. -- Job throughput and p95/p99 enqueue/claim/finalization latency, WAL per job, - admin query concurrency/QPS and browser refresh latency with zero clients, - polling, SSE, many clients and multiple admin instances. Set numeric budgets - and overload defaults from those measurements before enabling by default. - -Unresolved implementation choices are exact endpoint/request encoding, numeric -admission limits, cadence and stale-state SLOs, proxy/auth integration, and which -candidate resource reads already have adequate semantics. Application-scoped -authorization and historical notice retention require further design before -those consumers are exposed. None changes the invalidation/reconciliation -contract; a requirement for replay or guaranteed completion observation would. + and polling only for unfiltered legacy lists/details; nonempty filters are + unsupported. Never silently drop predicates to make fallback work. + +## Delivery and validation gates + +Implementation belongs in a separate draft PR, stacked on #572 or after its +merge, reusing its actual filter/compiler rather than copying one into the UI +crate. The [acceptance sequence](../design/realtime-subscriptions-v1.md#implementation-and-acceptance) +requires strict protocol adapters first, bounded hub/read work, deterministic +cache/cancellation tests, one frontend scheduler and complete local-mutation +coverage. It includes empty-install and read-only behavior without a worker, +mixed-version routing, slow transports, peer-process convergence and paused +selection/count exceptions. + +Opt-in implementation may begin once these amendments are adopted in the draft; +this documentation PR implements none of it. Keep the implementation PR draft +until deterministic correctness and read-budget/cancellation tests pass. Default +enablement additionally needs PostgreSQL 17/18 load evidence and actual +ingress/pooler validation. Timer-driven SSE is not inherently cheaper than +polling; measure local-mutation freshness and total read/connection costs. + +The earlier uniqueness-prune and DLQ downgrade findings remain separate release +gates. Subscriptions neither cause nor repair them. This record does not assert +those issues have been resolved. + +The contract now fixes endpoint/encoding, initial resource scope, experimental +limits, scheduler, cache fencing, origin and fallback semantics. Production SLOs, +measured default enablement, application authorization and historical notice +retention remain unproven or outside v1. A requirement for replay or guaranteed +completion observation changes the contract and needs a further decision. ## Evidence and relationship to existing decisions @@ -401,3 +328,12 @@ Inspected at main `f644f321ad110c6436382b64bdc0d935dcdeba5a` and #572 head schema or hot-path hint changes. The [operations skill](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/skills/awa-operations/SKILL.md) documents the trusted admin boundary and read-only operation. + +The actual ADR draft at `3dd7ee3696e706d80c4a2777d7d0642f1e3480cc` +(blob `154f328a7249229ad3e0a25db83827595ea95f21`) subsequently received a +document-specific adversarial review. This amendment adopts its concrete v1 +contract in the companion design document. That review supports the direction +subject to implementation tests; neither agreement nor docs CI establishes +architectural correctness, load safety or a tested implementation. No executable +review-model artifacts were supplied to this documentation session, so none are +claimed as verified evidence here. diff --git a/docs/adr/README.md b/docs/adr/README.md index d57e19ef1..4cb0c0c40 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -56,7 +56,7 @@ Each record preserves its status, context, decision, consequences, and alternati | 046 | [Handler-staged follow-up jobs](046-handler-staged-follow-ups.md) | A handler stages follow-up requests on its `JobContext`; they are inserted in the attempt's lease-guarded completion transaction, deduplicated by their own unique keys, and discarded on every other outcome. | Accepted | | 048 | [Retained job output](048-retained-job-output.md) | A handler may record a size-bounded JSON output for its attempt, sealed with the staging buffer and published only by the attempt's own lease-guarded transition: as the job's output on completion, or in the error entry of a failure, where only the latest entry to record one keeps it. It rides the compact completion batch, and neither the engine nor another job reads it. A kind may declare its output type, which types the API and is published in its descriptor. | Accepted | | 049 | [One job filter, and tags as the indexed handle](049-job-filter-and-indexed-tags.md) | One `JobFilter` describes "which jobs" for listing, counting, batch preview, batch execution and the DLQ, compiled once per engine with only the predicates a filter sets; SQL narrows candidates and a batched loader settles each job's current state. Tags are the indexed handle for "jobs about X": a partial GIN index over non-empty payload tags on each queue-storage partition, maintained by Postgres for every binary, emptied by pruning, and built online by the migrator. | Proposed | -| 050 | [Real-time resource subscriptions](050-realtime-subscriptions.md) | Admin-first resource invalidation over SSE with bounded refetch, reconnect resync and periodic reconciliation; durable business triggers remain queue jobs. | Proposed | +| 050 | [Real-time resource subscriptions](050-realtime-subscriptions.md) | Opt-in admin resource invalidation over fetch SSE, strict versioned reads and one bounded reconciliation scheduler; durable business triggers remain queue jobs. | Proposed | ## Correctness evidence diff --git a/docs/design/realtime-subscriptions-v1.md b/docs/design/realtime-subscriptions-v1.md new file mode 100644 index 000000000..a0e81b239 --- /dev/null +++ b/docs/design/realtime-subscriptions-v1.md @@ -0,0 +1,428 @@ +# Real-time subscriptions v1: implementation contract + +Status: proposed companion contract to +[ADR-050](../adr/050-realtime-subscriptions.md). Streaming is **disabled by +default**, enabled only by explicit server opt-in. The settings below are finite +experimental engineering limits, not measured production SLOs. This document +specifies a future implementation; it does not describe a shipped API. + +V1 serves the existing trusted admin UI. It uses process-local coarse +invalidation, ordinary read semantics and mandatory reconciliation. No +transactional directory, outbox, CDC integration, tenant authorization framework +or worker change is required. Durable business triggers remain queue jobs. + +## Streaming request and validation + +Use **`POST /api/subscriptions/v1`**, accepting strict JSON and returning +`text/event-stream`. The tab uses `fetch` and an AbortController. This is one +streaming request/response, not a registration/session service. There is no native +EventSource GET variant in v1. Resource changes abort the old stream before +opening its replacement. Retain displayed data, marked stale, until successful +reconciliation. + +```json +{ + "version": 1, + "resources": [ + { + "key": "r1", + "type": "jobs", + "filter": { + "queue": "emails", + "tag": ["tenant:opaque", "order:opaque"] + } + }, + {"key": "r2", "type": "job", "job_id": "123"}, + {"key": "r3", "type": "workers"} + ] +} +``` + +Validate the complete request before registration or HTTP 200: + +- Exact protocol version; unknown fields rejected at envelope, resource and + filter levels; unique opaque keys and type-specific fields. +- Both `JobFilter` deserialization **and `validate()`**, using the existing + implementation and bounds. Strict parsing alone does not enforce validation. +- Valid IDs/UUIDs and every byte/count bound. Subscription job IDs are decimal + strings to avoid another JavaScript integer-precision dependency. + +Body/descriptor limits apply while receiving and parsing, before allocation can +grow without bound. Do not echo raw descriptors or filters in events or logs. +After validation the hub may retain only keys and category mappings; it never +evaluates filters. Opening a stream performs **zero database connections and zero +state queries**. + +## Initial resources + +These types invalidate existing read results; they promise no new metrics or +retained history. The UI owns explicit active-query mappings. + +| Type | Additional fields | Existing reads invalidated | +| --- | --- | --- | +| `jobs` | Required `filter: JobFilter` | Jobs pages and dashboard job sample; count becomes stale only. | +| `job` | Required decimal-string `job_id` | One job's current detail, output, progress and source. | +| `dlq` | Required `filter: JobFilter` | DLQ page/detail and depth. | +| `queues` | None | Queue list/detail and `/queues/runtime`. | +| `workers` | None | `/runtime`, derived runtime details, capabilities and coverage. | +| `schedules` | None | `/cron` and `/cron/reconciliation`. | +| `operations` | Optional `operation_id: UUID` | Batch-operation list or one operation. | +| `maintenance` | None | `/storage` and existing sampling behavior. | +| `stats` | None | `/stats` and current timeseries views. | + +Exclude standalone `recovery`, retained notices/history, application consumers +and named queue/worker/schedule selectors from v1. They need further read +contracts or add little with coarse routing. `/storage?history=true` supplies no +server-retained history; subscriptions must not imply otherwise. Catalog/kinds +views keep existing read behavior. Queue health, throughput, age and recovery +displays use only existing fields with documented accuracy. + +`JobFilter` describes a population; pages remain separate. Invalidation refetches +the active bounded page, removing jobs that left and filling vacancies from +current matches. It does not promise stable multi-page snapshots, an infinite +list or visibility of every transition. Unknown routing information invalidates +the broader category rather than risking a missed leave. + +## Events and stream parsing + +The only data event types are `ready`, `invalidate` and `resync`: + +```text +event: ready +data: {"version":1,"epoch":"opaque-connection-id","keys":["r1","r2","r3"],"refresh_ms":5000,"reconcile_ms":30000,"keepalive_ms":15000} + +event: invalidate +data: {"version":1,"epoch":"opaque-connection-id","keys":["r1","r2"]} + +event: resync +data: {"version":1,"epoch":"opaque-connection-id","reason":"overflow"} + +``` + +`ready.keys` equals the requested set: never partially accept. Registration +precedes `ready`; a client displaying a preliminary read must refetch after +`ready`. Epoch is connection identity, not a database position. No SSE `id`, +offset, acknowledgement or transition payload is emitted. Keepalives are comments +and do not increment dirty generations. + +Check version, envelope, accepted keys and epoch. Unknown versions or malformed +events close the stream and enter safe fallback with a visible protocol error; +invalid protocol does not cause an automatic stream retry loop. Enforce frame +limits even when a peer never finishes a frame. Parse chunked SSE, comments, +split UTF-8 and split line boundaries with finite buffers. A fetch client must +implement reconnect itself; native EventSource reconnect behavior is not automatic. + +## Capabilities and strict versioned reads + +Add optional structured `subscriptions` and separate `job_filter_protocols` +fields to `/api/capabilities`. Advertise enabled protocols, accepted types, +limits, refresh policy and disabled status. Absence means unsupported; do not +infer support from semver. Capabilities are advisory: the next request can reach +an older server. + +Add thin strict JSON adapters: + +- `POST /api/v1/jobs/query` +- `POST /api/v1/dlq/query` +- `POST /api/v1/jobs/count` + +Query bodies use the existing filter and page semantics: + +```json +{"version":1,"filter":{},"page":{}} +``` + +Count bodies omit `page`. Reject unknown fields, including page fields through +strict wrappers where existing page deserializers are permissive. Reuse the +model/compiler and existing response conversion, not a second evaluator or +storage model. Responses wrap the existing payload without changing its meaning: + +```json +{ + "version": 1, + "data": "", + "observation": { + "observed_at": "", + "served_at": "", + "cache_age_ms": 0 + } +} +``` + +The strings above mark payload/timestamp placeholders, not literal wire values. +The new UI's filtered subscription/refetch path uses these adapters. Existing +permissive GET endpoints may remain for compatibility. Unknown `/api/*` routes +in the new router return API errors, not SPA HTML. + +Unsupported routes, wrong MIME, malformed envelopes and mismatched versions are +errors. Check JSON/version agreement for every read and `text/event-stream` plus +valid `ready` for streams. HTTP 200 alone proves nothing: an older server's SPA +fallback may return login/HTML content. Never retry by dropping predicates. + +Without v1 reads, permit legacy polling of **unfiltered** lists/details only. +Nonempty filters remain unsupported rather than rewritten; only a separately +tested legacy subset may later relax this. Do not display an unsupported job-match count +as zero. New-server read budgets apply to SSE-triggered reads and their legacy +fallback paths alike; fallback does not establish new server-side guarantees on +an old binary. + +Never place credentials, args, outputs or secrets in any URL. Do not place raw +filter descriptors in subscription URLs or automatic subscription logs. Existing +shareable search URLs may contain documented nonsecret opaque filter values. +JSON adapters avoid filters in new API URLs without removing intentional search +and bookmark URLs. + +### Errors and retry policy + +Before streaming headers, return structured JSON errors: + +| Condition | Status | +| --- | --- | +| Invalid/unsupported version, type or fields | 400 | +| Size limit | 413 | +| Authorization/origin failure | 401/403 | +| Admission limit | 429 with `Retry-After` | +| Temporary overload or disabled transport | 503 | + +404/405 or wrong MIME means unsupported transport. After streaming headers, use +resync/close; do not pretend to change HTTP status. + +- Network/transport failures retain stale data, use the **same paced read + scheduler**, and reconnect with exponential backoff/jitter capped at 60 seconds. +- 429/503 honor `Retry-After` across stream attempts **and refetches**. Overload + must not create immediate additional polling. +- 401/403 stop retries until credentials or user action change. +- Invalid protocol/request stops stream retries and shows the error. Safe read + fallback still requires supported filter semantics and the same scheduler. +- Reprobe absent transport capability no more than every 60 seconds, unless the + user explicitly requests it. + +## Finite experimental budgets + +All limits are configurable with sanity checks. Zero/unlimited must not defeat +the contract. These values are the initial opt-in implementation contract, not +production capacity or freshness promises. + +| Bound | Initial setting | +| --- | --- | +| Feature default | Disabled; explicit server opt-in. | +| Resources | 16 descriptors per stream. | +| Resource keys | Unique; 1–64 ASCII alphanumeric, `_` or `-` characters. | +| Request size | 64 KiB total; 32 KiB per descriptor. | +| Streams | 64 per process. | +| Principal quotas | No invented identities in no-auth mode; ingress may enforce quotas. | +| Pending stream state | One dirty bit per admitted key plus one resync flag. | +| Event/frame budget | 4 KiB application pending/frame budget. | +| Debounce | 100 ms target; 500 ms maximum from first dirty event. | +| Keepalive | Comment every 15 seconds. | +| Client liveness | Reconnect after 45 seconds without bytes. | +| Connect/ready deadline | 10 seconds. | +| Slow transport | Abort after a frame is pending for 5 seconds without body-consumer progress. | +| Stream lifetime | 5 minutes, then reconnect/resync. | +| Periodic hints | Existing `poll_interval_ms`, currently at least 5 seconds and cache TTL. | +| Reconciliation | Jittered 30-second watchdog per active resource. | +| Client read concurrency | 2 per tab; one in flight and one pending marker per exact query key. | +| Refresh floor | Cached reads respect advertised TTL/poll interval; uncached detail/pages at least 2 seconds apart. | +| Server expensive reads | 4 simultaneous operations per process. | +| Read admission | At most 16 waiters; 250 ms admission wait; excess gets 429. | +| Whole-read deadline | 15 seconds, including admission, pool acquisition, all queries/passes and serialization. | +| SQL statement limit | At most 5 seconds, also bounded by remaining whole-read deadline. | +| SQL lock-wait limit | At most 1 second, also bounded by remaining deadline. | +| Repeated failure | Exponential backoff to 60 seconds; retain one pending demand. | + +Bound request/descriptor parsing before large allocation. Coalesce repeated +invalidation into dirty bits. Overflow collapses to one all-resources resync; +close if that cannot make transport progress within budget. Slow consumers never +block the shared hub, readers, workers or producers. Release stream permits on +body drop, shutdown and failure. Idle time without a pending frame is not +backpressure. Enqueueing into an application channel is not evidence of transport +delivery; local body-consumer progress is not a remote receipt acknowledgement. + +The process-wide expensive-read budget covers SSE-triggered and fallback reads. +Cache hits may bypass admission; each cache fill acquires it once. Do not create +a database task or pool connection per subscription. Bound obsolete fills, +filtered cache cardinality and total fanout as well as per-stream state. Share +aggregate reads; any filtered coalescing key includes backend, authorization +scope, full normalized filter and page. + +A Rust future timeout alone does not prove SQL cancellation. Apply server-side +deadlines to **all** constituent queries through appropriately configured admin +read connections or deadline-aware transaction helpers. Do not alter mutation +session timeouts accidentally. Verify cancellation rolls back or closes +outstanding work before a connection becomes reusable. + +Audit candidate queries, separate loader transactions, all listing passes, +aggregates and DLQ reads. The document-specific review identified a gap in +#572's candidate-query-only timing: its loader opens a separate transaction with +planning settings, and listing may make 32 candidate passes. This contract +requires the entire operation, not each pass independently, to fit the deadline. +The implementation must verify coverage against its actual base revision. + +## One frontend refresh scheduler + +Use one tab-level scheduler for SSE hints, watchdogs, legacy polling, +focus/resume, reconnect and mutation refreshes. Managed queries disable +independent automatic intervals, focus fetching and reconnect fetching. Migrate +the existing two-second detail/operation polling explicitly. Every cause marks +the same bounded dirty state; no component opens its own stream. + +With healthy SSE, periodic hints and reconciliation drive reads. With unhealthy +SSE, polling takes over **through the same scheduler**. Do not run both plus +independent React Query timers. Hints and keepalives alone never postpone the +reconciliation watchdog. Resync obeys admission, cooldown and backoff. A failed +read does not count as successful reconciliation or clear pending demand. + +A request captures its query's dirty generation at start. Its response clears +only that captured demand. A hint during the request leaves one pending refresh; +the client may render the successful response while still showing stale/dirty +state. Fence superseded filter/stream epochs and late responses. Continuous hints +must neither starve visible results nor build an unbounded queue. + +Respect pauses and explicit actions: + +- Jobs currently pause automatic refresh while IDs are selected. Hints mark the + view stale without replacing the selection. Resume/manual refresh re-enters + the scheduler; mutations still use explicit selected IDs and server guards. +- Filtered job-match counts are excluded from automatic SSE/watchdog/focus refresh. Mark them stale; + refresh only on filter change, explicit refresh or the specified debounced + operator action. Prefix invalidation must not accidentally poll counts. +- Hidden tabs pause streams and reads. Visible resume performs one paced resync. + +Convergence means retained state returned by ordinary APIs after changes settle +and reads succeed. It does not strengthen cross-table consistency or guarantee +observation of every transient transition. Composite loading may miss ordinary +deferred-to-ready handoffs as well as claims; Repeatable Read alone does not make +ring `TRUNCATE` MVCC-safe. Preserve reconciliation rather than asserting a perfect +snapshot/stream boundary. Absence after retention is not proof of never existing. + +## Cache generations and truthful observations + +Use a bounded monotonic generation per aggregate cache group. After a successful +local mutation commits, one post-commit helper: + +1. Advances the cache generation. +2. Invalidates or retires the old generation. +3. Signals the hub. + +A fill captures its generation before starting. It may publish/return as current +only if that generation matches at completion. Requests after invalidation must +not join an old-generation fill. Use generation-keyed caches with bounded obsolete +work or an explicit synchronized check; never hold a cache lock over a database +query. Discard/retry obsolete completions only within the remaining read budget. +Do not relabel them fresh or loop indefinitely under continuous invalidations. +Existing Moka `invalidate_all()` alone is not an established proof. Require a +deterministic old-fill-after-invalidation test. + +Client and server generation fences are both required. The client cannot detect +an obsolete cache fill served as fresh. Read metadata distinguishes: + +- `observed_at`: source-fetch start, or a conservative start/end observation + window for a composite read. +- `served_at`: response time. +- `cache_age_ms`: age of the cached observation. + +Preserve original observation time on cache hits. A fresh HTTP response does not +make an old sample newly observed. Use a common wrapper or headers for existing +payloads; the new v1 adapters use the envelope above. Worker liveness remains +heartbeat evidence under its documented window. Replica freshness/lag is +**unknown** unless actually measured; primary hints do not prove replica replay. + +V1 may conservatively invalidate all admitted categories and aggregate caches +after any successful local mutation. Include every family, notably DLQ +retry/purge/move and batch cancellation, whose current handlers do not uniformly +invalidate caches. Rolled-back mutations emit no hints. Worker and peer-process +activity is timer-covered, with no added hooks or triggers. Each admin process +owns its hub/cache/timer and reads its backend; no broker or sticky routing is +needed. Measure aggregate read costs across processes. + +## Trusted-admin origin and credentials + +V1 is the existing trusted admin surface, not an application tenant API. +`--read-only` disables mutations, not access to sensitive data. Ingress/private +network controls protect reads and streams; an origin check does not authorize a +JobFilter or make a public unauthenticated admin safe. + +Default new adapters to same-origin browser access with existing ingress auth: + +- Require JSON POST bodies. Reject a present Origin unless it matches the + configured public admin origin. Never derive an allowlist from arbitrary + forwarded-host headers. +- No wildcard credentialed CORS. Isolate new routes or deliberately replace + existing permissive CORS so an outer layer cannot undo this policy. +- Browser fetch uses `credentials: "same-origin"` and `cache: "no-store"`. + Reject cross-origin redirects/login HTML as protocol data. +- Non-browser requests without Origin retain the existing ingress/private-network + trust requirement. + +SSE response headers: + +```text +Content-Type: text/event-stream +Cache-Control: private, no-store, no-transform +X-Accel-Buffering: no +``` + +Use the actual ingress's buffering controls, testing authentication proxy, +compression, idle timeouts and streaming behavior. The five-minute stream +lifetime allows ingress reauthentication; a server without identity cannot +promise immediate per-user revocation. Application-scoped consumers remain +outside v1. Do not put secrets, args or output in notifications or transport logs. + +## Implementation and acceptance + +Use a separate draft implementation PR, stacked on #572 or after its merge, +reusing its actual `JobFilter` compiler. Do not copy a second compiler into the +UI crate. Adopting this contract permits opt-in implementation, not default +enablement, release or merge. + +1. Adopt ADR-050 and this protocol/resource/fallback/privacy/limit/scheduler/cache/ + origin contract. Keep experimental defaults and disabled-by-default status. +2. Add protocol types/capabilities, strict read adapters and status/MIME checks. + Verify read-only and empty-install behavior without starting a worker. +3. Add the bounded process-local hub/timer and post-commit helper. Prove stream + opening does zero state queries/connections. Test descriptor/frame bounds, + permits, disconnect cleanup, shutdown and blocked response bodies. +4. Add cache generations, observations and whole-read budgets. Use deterministic + barriers for stale-fill races. Prove timed-out SQL stops/rolls back and pool + connections return to baseline before reuse. +5. Add one tab-level fetch-SSE manager/scheduler with explicit active-query + mappings, dirty generations, abort/fencing, count/selection exceptions and + fallback. Test finite parsing across chunks, comments, UTF-8 and line splits. +6. Cover every local mutation family. Prove rollback emits no hint and peer + changes converge through periodic reads. Test filter entry/exit, page refill, + retained-not-found presentation, errors, resume and heartbeat expiry without + writes. Preserve "where applicable" for follow-ups: rescue follow-ups remain + best effort after rescue. +7. Test old-server 200-HTML/404, mixed-version routing of capabilities/stream/read, + 401/403, 429 `Retry-After`, 503, healthy SSE with failed database reads, + restarts, continuous hints, overflow/resync, slow consumers and fallback + without duplicate loops. Counts/paused selections remain unchanged until + explicitly resumed/refreshed under their rules. +8. Benchmark PostgreSQL 17 and 18 with zero clients, current polling, opted-in SSE, + maximum admitted views and multiple admin processes. Measure p95/p99 enqueue, + claim and finalization latency, rescue progress, WAL, query QPS, permits, + database connections, cache fills and browser freshness. Validate through + supported ingress/pooler topology. + +Keep the implementation PR draft until deterministic correctness and +read-budget/cancellation tests pass. Default enablement requires load/deployment +evidence; missing evidence does not prevent writing an opt-in implementation. +Timer-driven SSE is not inherently cheaper than polling: measure local-mutation +freshness and total read/connection costs before default enablement. + +The earlier uniqueness-prune and DLQ downgrade findings remain separate release +gates. This stream neither causes nor repairs them. No transactional-directory +proposal is included. New metrics, retained notices/history and application +authorization need separate work. + +## Evidence boundary + +This contract adopts the actual-document review of ADR-050 at +`3dd7ee3696e706d80c4a2777d7d0642f1e3480cc`, verified blob +`154f328a7249229ad3e0a25db83827595ea95f21`. The original repository and +PostgreSQL/SSE source evidence remains linked in +[ADR-050](../adr/050-realtime-subscriptions.md#evidence-and-relationship-to-existing-decisions). +Review agreement and documentation CI do not establish a tested subscription +implementation, SQL cancellation, load safety or deployment correctness. The +transferred review evidence paragraph was incomplete and executable model +artifacts were not supplied here; this document claims no verified model result. From f3a65a49bd313e712c9b1b0b8f24c73fec6964b4 Mon Sep 17 00:00:00 2001 From: Brian Thorne Date: Sat, 31 Oct 2026 10:31:42 +1300 Subject: [PATCH 05/11] docs(adr): clarify watchdogs browser lifecycle and mutation authority --- docs/adr/050-realtime-subscriptions.md | 12 +++++-- docs/design/realtime-subscriptions-v1.md | 46 +++++++++++++++++++++++- 2 files changed, 54 insertions(+), 4 deletions(-) diff --git a/docs/adr/050-realtime-subscriptions.md b/docs/adr/050-realtime-subscriptions.md index 24a7b696b..77844ff3a 100644 --- a/docs/adr/050-realtime-subscriptions.md +++ b/docs/adr/050-realtime-subscriptions.md @@ -104,6 +104,12 @@ must not add a second polling loop. Hidden tabs pause reads and streams; resume requests one paced resync. Selected jobs remain paused and stale until resumed, so refreshing cannot replace a selection underneath an operator. +Transport deadlines run independently of the read iterator so a silent hung +connection is detectable. Empty scope, cap rejection, cancellation and BFCache +restore have explicit behavior in the contract. Preserve operator drafts too; +optimistic UI and notifications never replace atomic server-side mutation guards +or confer ownership through viewing/presence. + The guarantee is convergence to retained state returned by ordinary APIs after changes settle and reads succeed. It is not observation of every job or transition, nor stronger cross-table consistency. Claims can be absent or stale @@ -162,10 +168,10 @@ and reads. V1 uses `fetch` plus AbortController; it has no native EventSource GE variant, registration service or server-side subscription session. Resource-set changes abort the old stream before opening its replacement, retaining displayed data as stale. The client implements bounded parsing and reconnect/backoff; -The [SSE standard](https://html.spec.whatwg.org/multipage/server-sent-events.html) +the [SSE standard](https://html.spec.whatwg.org/multipage/server-sent-events.html) defines framing; EventSource's automatic reconnect does not apply to fetch. -Epochs are connection identity, -SSE `id` is absent, and `Last-Event-ID` never bypasses resync. +Epochs are connection identity, SSE `id` is absent, and `Last-Event-ID` never +bypasses resync. Five-minute stream lifetimes permit ingress reauthentication but do not promise immediate per-user revocation in a server without identity. Deployment checks diff --git a/docs/design/realtime-subscriptions-v1.md b/docs/design/realtime-subscriptions-v1.md index a0e81b239..f24fe1eb1 100644 --- a/docs/design/realtime-subscriptions-v1.md +++ b/docs/design/realtime-subscriptions-v1.md @@ -53,6 +53,12 @@ After validation the hub may retain only keys and category mappings; it never evaluates filters. Opening a stream performs **zero database connections and zero state queries**. +An empty active resource set opens no stream and schedules no resource reads. +The server rejects an empty `resources` array with 400. More than 16 descriptors +rejects the complete request under the size-limit policy; never silently truncate, +partially subscribe or open extra streams to bypass the cap. Display the error +while retaining safe, paced reads of supported views. + ## Initial resources These types invalidate existing read results; they promise no new metrics or @@ -112,6 +118,11 @@ limits even when a peer never finishes a frame. Parse chunked SSE, comments, split UTF-8 and split line boundaries with finite buffers. A fetch client must implement reconnect itself; native EventSource reconnect behavior is not automatic. +Run connect/ready and byte-liveness deadlines on independent timers, not inside +or after the stream reader's await/iterator. A pending read that never returns +must still be aborted at its deadline. Bytes/keepalives establish transport +activity only; they neither establish source freshness nor reset reconciliation. + ## Capabilities and strict versioned reads Add optional structured `subscriptions` and separate `job_filter_protocols` @@ -271,12 +282,16 @@ SSE, polling takes over **through the same scheduler**. Do not run both plus independent React Query timers. Hints and keepalives alone never postpone the reconciliation watchdog. Resync obeys admission, cooldown and backoff. A failed read does not count as successful reconciliation or clear pending demand. +This includes a missed final invalidation: a retained completed job's detail +must still refresh through reconciliation even when SSE continues sending bytes. A request captures its query's dirty generation at start. Its response clears only that captured demand. A hint during the request leaves one pending refresh; the client may render the successful response while still showing stale/dirty state. Fence superseded filter/stream epochs and late responses. Continuous hints must neither starve visible results nor build an unbounded queue. +Keep demand by resource/query key, not one global latest-resource slot; hints +for a second resource cannot overwrite the first resource's pending demand. Respect pauses and explicit actions: @@ -288,6 +303,24 @@ Respect pauses and explicit actions: operator action. Prefix invalidation must not accidentally poll counts. - Hidden tabs pause streams and reads. Visible resume performs one paced resync. +Resource-set changes cancel/fence old work before replacement. On `pagehide`, +abort streams and managed reads and stop timers, including when entering the +browser back-forward cache (BFCache). On persisted `pageshow`, discard the old +transport epoch and resume exactly one scheduler, with one paced resync only if +visible and nonempty. Cancelled reads do not clear demand or count as successful +reconciliation. Test repeated scope churn and restore without leaked permits, +read tasks or duplicate loops. + +Preserve unsaved operator form/dialog drafts as well as selected IDs during +refetch; do not silently replace the values the operator is reviewing. Optimistic +UI coordination is separate from atomic domain mutation guards. Destructive +operations validate current eligibility and applicable version/lease/fencing +conditions atomically on the server; notifications and disabled buttons cannot +enforce exclusivity or idempotency. Retain the existing server guards rather +than inventing a subscription ownership protocol. Viewing a job or observing +worker presence grants no ownership; active attempt ownership remains a lease +and domain-state question. + Convergence means retained state returned by ordinary APIs after changes settle and reads succeed. It does not strengthen cross-table consistency or guarantee observation of every transient transition. Composite loading may miss ordinary @@ -397,13 +430,24 @@ enablement, release or merge. 401/403, 429 `Retry-After`, 503, healthy SSE with failed database reads, restarts, continuous hints, overflow/resync, slow consumers and fallback without duplicate loops. Counts/paused selections remain unchanged until - explicitly resumed/refreshed under their rules. + explicitly resumed/refreshed under their rules. Include a silently hung reader, + a missed final event with continuing keepalives, interleaved resource hints, + empty/over-cap scope, scope churn, cancellation, BFCache restore and preserved + operator drafts. Independently verify the existing atomic mutation guards. 8. Benchmark PostgreSQL 17 and 18 with zero clients, current polling, opted-in SSE, maximum admitted views and multiple admin processes. Measure p95/p99 enqueue, claim and finalization latency, rescue progress, WAL, query QPS, permits, database connections, cache fills and browser freshness. Validate through supported ingress/pooler topology. +Distinguish source/pipeline observation lag where measurable, transport activity, +API refetch latency and browser read/render freshness in diagnostic evidence. +Missing source/pipeline lag remains unknown. A healthy stream can coexist with +an unhealthy queue or stale source, and a consumer-stage metric cannot establish +universal subsecond browser freshness. Report histogram resolution and censored +top-bucket observations rather than claiming an exact delay from the last bucket. +These are measurement requirements, not additional v1 resource/metric APIs. + Keep the implementation PR draft until deterministic correctness and read-budget/cancellation tests pass. Default enablement requires load/deployment evidence; missing evidence does not prevent writing an opt-in implementation. From fa12096106811597a1e02e108fda7159bfefb274 Mon Sep 17 00:00:00 2001 From: Brian Thorne Date: Sat, 31 Oct 2026 12:22:50 +1300 Subject: [PATCH 06/11] docs(adr): focus admin view decision on observation semantics --- docs/adr/050-realtime-subscriptions.md | 544 +++++++++-------------- docs/adr/README.md | 2 +- docs/design/realtime-subscriptions-v1.md | 2 +- 3 files changed, 213 insertions(+), 335 deletions(-) diff --git a/docs/adr/050-realtime-subscriptions.md b/docs/adr/050-realtime-subscriptions.md index 77844ff3a..29b550388 100644 --- a/docs/adr/050-realtime-subscriptions.md +++ b/docs/adr/050-realtime-subscriptions.md @@ -1,345 +1,223 @@ -# ADR-050: Real-time resource subscriptions +# ADR-050: Admin views use invalidation and authoritative refetch ## Status -Proposed. Documentation only; no subscription API is implemented by this record. -Number reserved in [#576](https://github.com/hardbyte/awa/pull/576). - -## Context - -The first consumer is Awa's admin UI. Its question is broader than whether a job -changed state: is a queue falling behind, is a capable worker present, did rescue -recover progress, has a schedule reconciled, or has an operator's batch finished? -Applications may later want to refresh a view when their jobs complete. - -The priority order is strict: job correctness, reliability and recovery first; -performance second; UX third. A lost observation must never lose a job or become -the only trigger for business work. Few production users are moving from 0.6 to -0.7, so a simpler public interface is preferable to preserving an accidental one. -That freedom does not remove the database rolling-upgrade contract. - -Today the UI polls ordinary HTTP reads. Aggregate endpoints have TTL caches with -request coalescing; the server advertises a polling interval at least as large as -the cache TTL. Job detail and batch progress have their own intervals. This -proposal builds on those reads rather than creating another model of job state. - -[ADR-049](049-job-filter-and-indexed-tags.md) is accepted, although its index row -still says Proposed. Its implementation is separately under review in -[#572](https://github.com/hardbyte/awa/pull/572). The initial recommendation was -formed independently of the separate filtering review. Subsequent document-specific review amendments are recorded below. - -## Decision - -Use **resource invalidation followed by authoritative refetch** over one -multiplexed **fetch-based SSE connection per visible admin tab**. An invalidation -means only "this resource may have changed". Reconnect resynchronizes current -state; there is no durable cursor, transition replay or acknowledgement. - -The v1 implementation contract is -[Real-time subscriptions v1](../design/realtime-subscriptions-v1.md). It fixes -`POST /api/subscriptions/v1`, strict versioned JSON reads, finite experimental -limits, one refresh scheduler, cache-generation fences and the trusted-admin -origin policy. These are requirements of this proposal, not optional follow-up -choices. **Streaming is disabled by default** and requires explicit server -opt-in until load and deployment validation supports default enablement. - -Start with a process-local hub, periodic invalidation and hints after committed -local admin mutations. Opening a stream performs zero database connections and -zero state queries. Add no subscription writes, triggers, outbox, transactional -directory, mandatory listener or worker changes on enqueue, claim, heartbeat, -finalization or rescue. Neither delivery nor reconciliation depends on a worker -or maintenance leader producing notifications. - -Worker-originated updates remain cadence-limited in v1. SSE alone makes them -neither instantaneous nor cheaper than polling. Durable business reactions remain -ordinary Awa jobs, using [ADR-029](029-transactional-followup-jobs.md) and -[ADR-046](046-handler-staged-follow-ups.md) where applicable. Existing rescue -follow-ups remain best effort after rescue; this proposal does not strengthen -them. - -### Resources and shared filters - -V1 admits `jobs`, `job`, `dlq`, `queues`, `workers`, `schedules`, `operations`, -`maintenance` and `stats`, with exact descriptors and existing read mappings in -[the resource contract](../design/realtime-subscriptions-v1.md#initial-resources). -These cover current admin observations, not new metrics. Queue backlog, oldest -waiting age, throughput and concurrency pressure may only use existing fields -with documented accuracy. Presence is heartbeat evidence, and age can change -without a write; stream traffic proves neither liveness nor source freshness. - -Standalone recovery resources, retained notices/history, named queue/worker/ -schedule selectors and application consumers are excluded from v1. A DLQ view -can observe current retained state, but an ephemeral notice is not an audit -trail. `/storage?history=true` does not provide server-retained history. Catalog -and kinds views keep their existing read behavior. - -`jobs` and `dlq` share ADR-049's `JobFilter` deserialization **and validation**. -Fields intersect, states are any-of, tags are all-of, and creation bounds refer -to original insertion. Paging stays separate. The hub routes coarse categories -and retains keys/category mappings rather than evaluating a second filter. -Thus a queue/state move invalidates both old and new populations; a replacement -page removes departed jobs and refills vacancies. Routing solely on a new row -would miss the leave. Filter changes abort/fence old streams and reads. - -Capabilities are advisory across mixed-server routing. New filtered reads use -strict v1 JSON POST adapters; every response must agree on MIME, envelope and -version. An old server may return SPA HTML with HTTP 200, which is an error. -Legacy fallback permits unfiltered lists/details only; a nonempty filter remains -unsupported, never rewritten by dropping predicates. Counts retain ADR-049's -cap and are excluded from automatic refresh; invalidation only marks them stale. - -### Recovery uses one bounded scheduler - -Register and validate the complete resource set before `ready`, then refetch. -Client dirty generations preserve a hint received during a read as one pending -refresh, and superseded responses cannot overwrite the new view. Render a -successful response even if newer demand remains; do not wait for quiet traffic. - -One tab-level scheduler handles hints, a jittered 30-second reconciliation -watchdog, fallback polling, focus/resume, reconnect and mutations. Managed queries -have no independent automatic intervals or focus/reconnect fetches. Hints and -keepalives do not postpone the watchdog. Every demand, including resync, obeys -the same concurrency, cooldown, admission and retry budgets. A transport failure -must not add a second polling loop. Hidden tabs pause reads and streams; resume -requests one paced resync. Selected jobs remain paused and stale until resumed, -so refreshing cannot replace a selection underneath an operator. - -Transport deadlines run independently of the read iterator so a silent hung -connection is detectable. Empty scope, cap rejection, cancellation and BFCache -restore have explicit behavior in the contract. Preserve operator drafts too; -optimistic UI and notifications never replace atomic server-side mutation guards -or confer ownership through viewing/presence. - -The guarantee is convergence to retained state returned by ordinary APIs after -changes settle and reads succeed. It is not observation of every job or -transition, nor stronger cross-table consistency. Claims can be absent or stale -while the lane cursor is outside MVCC; ordinary deferred-to-ready handoffs can -also be missed by a composite read. Repeatable Read alone does not make ring -`TRUNCATE` MVCC-safe. A local epoch is not a database position. A completion -pruned before a read may never be observed; a reliable business reaction needs -a durable job or application record at the source. - -The experimental defaults bound a stream to 16 descriptors and 64 KiB, a process -to 64 streams and four expensive reads, and a tab to two concurrent reads. A -whole read has a 15-second deadline covering admission, pool acquisition, every -query/pass and serialization, with SQL and lock-wait limits inside it. Releasing -a Rust future is not proof that PostgreSQL work stopped: cancellation and pool -reuse must be tested. Full limits, overflow/slow-consumer behavior and the -60-second maximum retry backoff are in the companion contract. - -### Server caches, observations and multiple instances - -A post-commit helper advances a bounded cache generation, retires its old cache -entries, then signals the hub. A fill may publish/return as current only if its -captured generation still matches. New requests cannot join an obsolete fill; -retries stay within the remaining read budget. Moka `invalidate_all()` alone is -not an established proof. Client dirty generations and server cache generations -solve different races and both are required. - -Read responses preserve source observation time on cache hits and distinguish -`observed_at`, `served_at` and `cache_age_ms`. Report replica freshness as unknown -unless measured. A primary hint does not prove a replica has replayed the change. -Cache hits may bypass expensive-read admission, but each fill acquires it once. -Slow clients cannot hold a database connection or block shared fanout; dirty -state collapses to resync, then close if it cannot be delivered within budget. - -Every admin process owns its hub, caches and timer. Peer mutations and worker -activity converge through periodic reads; no broker, leader or sticky routing is -needed. Reconnection to a different instance always resynchronizes. Combined -read cost still grows with active views and process count and must be measured. -Links to UIs for different databases remain navigation only. - -### Trusted-admin scope and transport - -V1 is the existing trusted admin surface, protected by ingress/private networking. -It adds no tenant authorization framework. `--read-only` disables mutations, -not access to sensitive job data. New stream/read adapters use same-origin JSON -POSTs, explicit configured-origin checks and credential policy; an outer -permissive CORS layer must not undo that boundary. An origin check neither -authorizes a filter nor makes a public unauthenticated admin safe. - -Never place credentials, args, outputs or secrets in any URL. Do not place raw -filter descriptors in subscription URLs or automatic subscription logs. Existing -shareable search URLs may contain documented nonsecret opaque filter values. -Descriptors travel in JSON request bodies and are not echoed in events/logs. - -Choose SSE because the contract is one-way hints with ordinary HTTP mutations -and reads. V1 uses `fetch` plus AbortController; it has no native EventSource GET -variant, registration service or server-side subscription session. Resource-set -changes abort the old stream before opening its replacement, retaining displayed -data as stale. The client implements bounded parsing and reconnect/backoff; -the [SSE standard](https://html.spec.whatwg.org/multipage/server-sent-events.html) -defines framing; EventSource's automatic reconnect does not apply to fetch. -Epochs are connection identity, SSE `id` is absent, and `Last-Event-ID` never -bypasses resync. - -Five-minute stream lifetimes permit ingress reauthentication but do not promise -immediate per-user revocation in a server without identity. Deployment checks -must cover the actual authentication proxy, buffering, compression and timeouts. -Protocol/admission errors follow the contract's status/MIME and retry rules, -including respecting `Retry-After` for both streams and refetches. +Proposed. The SSE transport experiment is opt-in and disabled by default. +[Draft PR #576](https://github.com/hardbyte/awa/pull/576) contains this decision +and its supporting design contract; it implements no subscription feature. + +## Problem and constraints + +The admin UI needs current views of jobs, queues, workers, schedules, the DLQ +and operator progress. Those views answer operational questions, not just whether +a job changed status. Today ordinary HTTP reads and TTL caches serve them, with +polling spread across pages. Adding push without a common refresh policy risks +multiplying reads while leaving missed notifications and stale cache fills +unresolved. + +The priority order is job correctness, reliability and recovery first, +performance second, UX third. Observation must not become a dependency of +enqueue, claim, finalization or rescue. The initial consumer is the trusted admin +UI; a future application API must not acquire stronger delivery or authorization +promises by accident. The 0.6-to-0.7 transition permits interface simplification, +while the rolling-upgrade and database safety contracts still apply. + +## Decision and scope + +**Admin views use disposable invalidation hints, authoritative API reads and +independent bounded reconciliation.** This observation model is the enduring +choice. Fetch-based SSE is the proposed experimental transport for v1, not the +source of correctness and not a commitment to permanent transport infrastructure. + +A hint means that a resource may have changed. The owning read API decides its +current contents, eligibility and accuracy. Filtered job views reuse +[ADR-049's JobFilter](049-job-filter-and-indexed-tags.md), including its bounds, +strict parsing and separation of population from paging. Unsupported filters +fail closed, including when successive requests reach servers of different +versions. Routing hints does not create a second filter evaluator. + +The model has these invariants: + +- **Recovery does not depend on hints.** Establish observation before acknowledging + readiness, then refetch. Reconnect and scope changes resynchronize. Independent + reconciliation repairs a missed final hint even while the transport looks + healthy. A hint received during a read remains pending rather than being + cleared by that response. Both entering and leaving a filtered population must + invalidate the view. +- **Demand is bounded and shared.** Hints, polling, reconciliation, resume and + retries feed one paced refresh policy. Repeated demand coalesces without one + resource overwriting another. Read work, waiting work, fanout and slow clients + have finite budgets. Overload reduces freshness rather than creating more + polling or delaying job execution through an unbounded read backlog. +- **Invalidation cannot be undone by a late cache fill.** Once local state is + invalidated, an older in-flight fill cannot reappear as current. Observation + timestamps retain the source sample's age. Transport activity is distinct from + source freshness; it cannot prove a worker alive or a replica caught up. +- **Observation stays off the job write path.** V1 adds no worker hooks, database + triggers, mandatory listener, event log or subscription writes to job + transitions. Each admin process can refresh independently; peer-process and + worker changes converge through reads without a shared broker or leader. +- **Domain guards and access boundaries remain authoritative.** Hints and + optimistic UI cannot enforce ownership, exclusivity or idempotency. Mutations + retain atomic server guards, and refresh preserves selections and operator + drafts. Read-only access remains sensitive admin access. Scopes cannot silently + widen, and a connection does not extend authorization indefinitely. + +Authoritative reads retain their existing limitations. Composite loading can +miss an in-flight claim or deferred-to-ready handoff; neither a stream nor +Repeatable Read makes ring pruning an atomic snapshot. Convergence means retained +state returned by ordinary APIs after changes settle and reads succeed, not +observation of every job or transition. A completion pruned between reads may +never be observed. Durable business reactions remain queue jobs through +[ADR-029](029-transactional-followup-jobs.md) and +[ADR-046](046-handler-staged-follow-ups.md) where applicable; existing best-effort +rescue follow-ups are not strengthened. + +V1 remains within the existing trusted-admin boundary and existing read models. +It adds no tenant authorization framework, presence ownership, new queue metrics +or retained notification history. Protocol shapes, finite experimental limits, +cache mechanisms, lifecycle handling and acceptance tests live in +[the supporting v1 contract](../design/realtime-subscriptions-v1.md). + +### Why experiment with SSE + +A one-way HTTP stream fits invalidation from an admin process to its connected +views. Its immediate incremental opportunity is fanout after a committed local +admin mutation, so other views connected to the same process can refetch sooner +than their next poll. This does not accelerate worker-originated or peer-process +changes beyond the sampling cadence, and timer-driven SSE does not inherently +reduce database reads. + +Streaming therefore stays disabled by default. The experiment must compare it +with improved polling, measuring local-mutation freshness alongside total read, +connection and operational costs. PostgreSQL load tests must show job execution +and recovery remain protected; actual ingress/pooler tests must establish that +authentication, buffering and connection lifecycle behave as intended. A useful +observation model does not by itself justify enabling a transport. ## Alternatives considered +### Improved polling with the same observation model + +This is the strongest baseline and the default path while streaming is disabled. +One scheduler, +coalesced reads, bounded caches, clear stale state and reconciliation solve much +of the problem without long-lived HTTP connections. It handles worker and +peer-process changes at the same sampling cadence as timer-driven SSE. + +Polling lacks immediate server fanout of a local mutation to other connected +views. SSE is worth retaining only if that incremental freshness, or another +measured benefit, justifies its costs. If it does not, keep improved polling; +none of the enduring invariants requires streaming. + +### Fetch-based SSE + +Selected for the opt-in v1 experiment because one-way hints fit the existing +HTTP admin surface and ordinary requests still perform reads and mutations. +It adds reconnect, parsing, proxy, authorization-lifetime and browser lifecycle +work; using fetch does not provide EventSource's automatic recovery behavior. +The supporting contract bounds that work and defines safe fallback, but those +requirements are costs to validate, not evidence of an advantage already earned. + +### WebSocket + +WebSocket can change subscriptions bidirectionally without replacing a stream +and would suit an actual interactive command protocol. V1 needs one-way hints, +so that bidirectionality is unused and SSE fits the current HTTP request model. +Both transports still need bounded state, parsing, reconnect, authorization and +lifecycle handling. Choose between them on demonstrated requirements and deployed +behavior, not on a claim that fetch-SSE avoids that complexity. + +### Pushed snapshots or row deltas + +Shared aggregate snapshots might save refetch round trips. Filtered snapshots +and row deltas require server-side query or membership state, including page +refill, population entry/exit and deletion semantics. Refetch keeps the existing +read API authoritative and avoids maintaining that second view model. Consider +pushed aggregates if measurements identify a material benefit without weakening +recovery. + ### Durable event log or transactional outbox -A semantic append-only log could support replay after downtime and retained -notices. An outbox written in the job transaction could couple a fact atomically -to a transition, unlike a post-commit in-process hook. This is the right family -of design if observing every retained event becomes an actual requirement. - -It also requires complete coverage of Rust/Python/raw/bridge inserts, callbacks, -rescue, compact completion batches, bulk actions, schedule reconciliation and -pruning. It adds WAL, retention, schema/version contracts, publication lag and -write-path failure modes for a UI that needs current state. A sequence allocated -before commit is not a safe replay watermark: a later id may commit first and -cause a consumer to skip the earlier transaction. Commit-safe ordering, an -initial snapshot boundary, idempotence, gap detection, retention floors and -expired-cursor resync all need a proven protocol. - -Not selected. It would duplicate durable reaction machinery already expressed by -queue jobs. The proposed stream has no event table, consumer offset or replay -retention to prune. Ordinary job/output/operation retention still applies, and -pruning can make resources disappear; reads must not present absence as proof -that a job never existed. - -### PostgreSQL LISTEN / NOTIFY as delivery or hints - -[PostgreSQL NOTIFY](https://www.postgresql.org/docs/17/sql-notify.html) delivers -after commit to connected listeners, can fold duplicates, has a small payload -limit, and exposes notifications to database users. A full notification queue -can make notifying transactions fail at commit. It is not replayable history. -Adding it to every transition would couple optional UI delivery to job commits. - -Not selected as delivery or as a new mandatory write-path mechanism. Existing -queue wakeups are incomplete hints for this vocabulary: they do not mean that -completion, worker expiry, schedules or prune changes were all observed. A later -optimization may use one dedicated session listener per admin process, retaining -timers and broad resync on listener reconnect. It must not create one PostgreSQL -connection per browser, rely on transaction-mode poolers for session state, or -send untrusted payloads straight to clients. - -The [LISTEN startup rule](https://www.postgresql.org/docs/17/sql-listen.html) is -to commit registration before inspecting state. Even when followed, it does not -repair an incomplete signal source or a missed notification. Any new emission -path needs measured commit/WAL cost and review against -[ADR-045](045-hot-path-trigger-lock-contract.md); no trigger change is implied. +A log can retain transitions and support replay; a transactional outbox can couple +publication to a committed state change. Those are appropriate capabilities when +every transition is required. They also need complete writer coverage, commit-safe +ordering, snapshot/replay boundaries, idempotence, retention and gap recovery. A +sequence allocated before commit is not itself a safe replay watermark. + +Current admin views need retained state, while durable reactions already use +jobs. Paying WAL, publication and retention costs on job writes to obtain UI +freshness would introduce an unnecessary dependency. No log, outbox or +transactional directory is selected here. + +### PostgreSQL LISTEN / NOTIFY + +Existing wakeups can be useful disposable hints but do not cover all admin +resources or provide replay. New transactional emissions would add job-path cost; +a full notification queue can make notifying transactions fail at commit. +A mandatory listener would add session and deployment requirements without +removing the need for reconciliation. A later optional hint optimization must +preserve the observation model and earn its write/connection cost separately. ### Logical decoding / CDC -[Logical decoding](https://www.postgresql.org/docs/17/logicaldecoding-explanation.html) -can provide a database change stream with an exported snapshot boundary. It has -replication-slot, privilege and failover requirements; consumers must tolerate -duplicates after crashes. A stalled slot can retain WAL and catalog resources. - -Awa's physical representation includes ready rows, cursors outside MVCC, leases, -receipts, compact completions and partition truncation. Decoding those into -public resource changes would expose storage evolution or require a semantic -adapter with its own recovery proof. CDC remains useful for external history -and analytics as documented in -[managed Postgres](../deploying-on-managed-postgres.md#logical-replication-and-cdc), -but is not a required dependency for the admin UI. - -### Polling only, pushed snapshots, and WebSocket - -**Keep polling only.** This has the fewest moving parts and remains a valid -deployment/fallback. The initial timer-driven SSE design does not claim fewer -database reads. SSE buys a common resource vocabulary and immediate local -invalidation, with room for safe hints later. If that benefit does not justify -connection/proxy complexity in an implementation benchmark, retain polling -behind the same vocabulary instead of shipping an idle streaming layer. - -**Push snapshots or row deltas.** Shared sampled aggregate snapshots can save -refetch round trips, but filtered job snapshots add per-subscriber query/state -work. Row deltas need correct enter/leave, paging and deletion semantics plus -the same recovery machinery. Start with refetch; consider pushed aggregate -snapshots only when measurements show the read cost matters. - -**WebSocket.** It supports bidirectional subscription changes without reopening -and would suit a later interactive command protocol. This design needs neither; -it would add framing, reconnect and connection-state machinery without stronger -delivery guarantees. Revisit if resource-set churn or actual bidirectional -requirements justify it. The envelope and recovery semantics should survive a -transport change. - -## Consequences - -- Job execution and durable reactions remain independent of browser health. - There is no new schema migration, per-job write tax or notification retention - job. Admin read load still competes for database resources and must be bounded. -- Consumers get convergent views, not a replayable event API. Transient states, - completions before pruning and historical recovery notices may be missed. -- Coarse invalidation simplifies correctness at the cost of extra reads and - cadence-limited remote updates. Finer routing must not weaken leave/resync - behavior. Low-latency coverage is explicitly incomplete in the first stage. -- The initial public surface can change for 0.7, but must negotiate capabilities - and reject unsupported subscription shapes. An old server gets ordinary reads - and polling only for unfiltered legacy lists/details; nonempty filters are - unsupported. Never silently drop predicates to make fallback work. - -## Delivery and validation gates - -Implementation belongs in a separate draft PR, stacked on #572 or after its -merge, reusing its actual filter/compiler rather than copying one into the UI -crate. The [acceptance sequence](../design/realtime-subscriptions-v1.md#implementation-and-acceptance) -requires strict protocol adapters first, bounded hub/read work, deterministic -cache/cancellation tests, one frontend scheduler and complete local-mutation -coverage. It includes empty-install and read-only behavior without a worker, -mixed-version routing, slow transports, peer-process convergence and paused -selection/count exceptions. - -Opt-in implementation may begin once these amendments are adopted in the draft; -this documentation PR implements none of it. Keep the implementation PR draft -until deterministic correctness and read-budget/cancellation tests pass. Default -enablement additionally needs PostgreSQL 17/18 load evidence and actual -ingress/pooler validation. Timer-driven SSE is not inherently cheaper than -polling; measure local-mutation freshness and total read/connection costs. - -The earlier uniqueness-prune and DLQ downgrade findings remain separate release -gates. Subscriptions neither cause nor repair them. This record does not assert -those issues have been resolved. - -The contract now fixes endpoint/encoding, initial resource scope, experimental -limits, scheduler, cache fencing, origin and fallback semantics. Production SLOs, -measured default enablement, application authorization and historical notice -retention remain unproven or outside v1. A requirement for replay or guaranteed -completion observation changes the contract and needs a further decision. - -## Evidence and relationship to existing decisions - -Inspected at main `f644f321ad110c6436382b64bdc0d935dcdeba5a` and #572 head -`512f477785d327ac092b94c6a399db4a8b5c4555` on 2026-10-30: - -- [Dashboard cache](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/cache.rs) - and [AppState](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/state.rs): - typed TTL caches, request coalescing, poll guidance and local invalidation. -- [Frontend polling](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/frontend/src/hooks/use-poll-interval.ts) - and [router](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/lib.rs): - ordinary reads, existing resource families and current permissive CORS. -- [Existing worker notification hub](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-worker/src/notify_hub.rs): - a shared listener, poll fallback and broad wakeup after reconnect; precedent - for hints with recovery, not evidence of full resource change coverage. -- [#572 JobFilter](https://github.com/hardbyte/awa/blob/512f477785d327ac092b94c6a399db4a8b5c4555/awa-model/src/filter.rs) - and [job reads](https://github.com/hardbyte/awa/blob/512f477785d327ac092b94c6a399db4a8b5c4555/awa-ui/src/handlers/jobs.rs): - strict population semantics and shared bounded reads under implementation. -- [ADR-049](049-job-filter-and-indexed-tags.md): shared filter, advisory - in-flight reads, capped counts, tag retention and fleet compatibility. - [ADR-030](030-batch-operations.md) owns durable operator actions; - subscriptions only observe their progress. -- [ADR-029](029-transactional-followup-jobs.md) and - [ADR-046](046-handler-staged-follow-ups.md) own durable follow-up work. - This proposal neither replaces nor strengthens the guarantees of those paths. +CDC can provide a change stream with a snapshot boundary, but requires slot, +privilege, failover and retention operations. Mapping Awa's leases, cursors, +compact completions and partition pruning into public changes would also require +a semantic adapter. It remains useful for external history and analytics, not a +required dependency of current admin views. + +## Accepted consequences + +Consumers can converge after lost hints, reconnects and process changes +without introducing a second event-processing engine. Job execution and durable +reactions remain independent of browser availability. + +The tradeoff is sampled freshness, potentially redundant reads and no guarantee +of observing transient or already-pruned state. Coarse invalidation deliberately +trades routing precision for simpler entry/exit correctness. Multiple admin +processes have independent caches and read costs; their combined load still +needs a budget. A successful stream does not imply a healthy queue or current +source data. + +The transport adds operational and client lifecycle costs while it remains an +experiment. Production SLOs and default enablement need evidence; documentation +checks establish neither implementation correctness nor capacity. + +## Revisit conditions + +Revisit the rationale when: + +- A consumer requires replay, audit history or guaranteed completion observation; + current-state refetch cannot provide those guarantees. +- Required worker or peer-process update latency cannot be met by sampled reads; + investigate a safe source of hints rather than treating SSE alone as a fix. +- Measured read/connection cost or ingress behavior makes streaming less useful + than improved polling, or makes shared pushed aggregates worthwhile. +- Application consumers require scoped authorization beyond trusted admin access, + or real bidirectional interaction changes the transport requirements. + +Revising a transport or sampling policy must preserve the enduring invariants +unless a later decision explicitly replaces them. + +## Links and evidence + +- [V1 implementation contract](../design/realtime-subscriptions-v1.md): protocol, + budgets, cache fencing, browser lifecycle, security policy and validation gates. +- [ADR-049](049-job-filter-and-indexed-tags.md) and + [implementation PR #572](https://github.com/hardbyte/awa/pull/572): shared job + selection and advisory read behavior. The proposed adapter reuses that compiler. +- [Existing dashboard cache](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/cache.rs), + [admin state](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/state.rs) + and [frontend polling](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/frontend/src/hooks/use-poll-interval.ts): + the current read/cache baseline, not a measured comparison with SSE. +- [Existing notification hub](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-worker/src/notify_hub.rs): + hints with polling recovery, not complete admin change coverage. +- [PostgreSQL NOTIFY](https://www.postgresql.org/docs/17/sql-notify.html), + [LISTEN startup](https://www.postgresql.org/docs/17/sql-listen.html), + [logical decoding](https://www.postgresql.org/docs/17/logicaldecoding-explanation.html) + and [SSE framing](https://html.spec.whatwg.org/multipage/server-sent-events.html): + substrate behavior, not proof of end-to-end freshness or delivery. - [ADR-041](041-rolling-upgrade-policy.md) and - [ADR-045](045-hot-path-trigger-lock-contract.md) remain binding for any future - schema or hot-path hint changes. The - [operations skill](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/skills/awa-operations/SKILL.md) - documents the trusted admin boundary and read-only operation. - -The actual ADR draft at `3dd7ee3696e706d80c4a2777d7d0642f1e3480cc` -(blob `154f328a7249229ad3e0a25db83827595ea95f21`) subsequently received a -document-specific adversarial review. This amendment adopts its concrete v1 -contract in the companion design document. That review supports the direction -subject to implementation tests; neither agreement nor docs CI establishes -architectural correctness, load safety or a tested implementation. No executable -review-model artifacts were supplied to this documentation session, so none are -claimed as verified evidence here. + [ADR-045](045-hot-path-trigger-lock-contract.md): existing rolling-upgrade and + hot-path constraints remain in force. diff --git a/docs/adr/README.md b/docs/adr/README.md index 4cb0c0c40..db24516b7 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -56,7 +56,7 @@ Each record preserves its status, context, decision, consequences, and alternati | 046 | [Handler-staged follow-up jobs](046-handler-staged-follow-ups.md) | A handler stages follow-up requests on its `JobContext`; they are inserted in the attempt's lease-guarded completion transaction, deduplicated by their own unique keys, and discarded on every other outcome. | Accepted | | 048 | [Retained job output](048-retained-job-output.md) | A handler may record a size-bounded JSON output for its attempt, sealed with the staging buffer and published only by the attempt's own lease-guarded transition: as the job's output on completion, or in the error entry of a failure, where only the latest entry to record one keeps it. It rides the compact completion batch, and neither the engine nor another job reads it. A kind may declare its output type, which types the API and is published in its descriptor. | Accepted | | 049 | [One job filter, and tags as the indexed handle](049-job-filter-and-indexed-tags.md) | One `JobFilter` describes "which jobs" for listing, counting, batch preview, batch execution and the DLQ, compiled once per engine with only the predicates a filter sets; SQL narrows candidates and a batched loader settles each job's current state. Tags are the indexed handle for "jobs about X": a partial GIN index over non-empty payload tags on each queue-storage partition, maintained by Postgres for every binary, emptied by pruning, and built online by the migrator. | Proposed | -| 050 | [Real-time resource subscriptions](050-realtime-subscriptions.md) | Opt-in admin resource invalidation over fetch SSE, strict versioned reads and one bounded reconciliation scheduler; durable business triggers remain queue jobs. | Proposed | +| 050 | [Admin views use invalidation and authoritative refetch](050-realtime-subscriptions.md) | Disposable hints, owning API reads and independent bounded reconciliation; SSE remains an opt-in transport experiment. | Proposed | ## Correctness evidence diff --git a/docs/design/realtime-subscriptions-v1.md b/docs/design/realtime-subscriptions-v1.md index f24fe1eb1..2a1c89534 100644 --- a/docs/design/realtime-subscriptions-v1.md +++ b/docs/design/realtime-subscriptions-v1.md @@ -465,7 +465,7 @@ This contract adopts the actual-document review of ADR-050 at `3dd7ee3696e706d80c4a2777d7d0642f1e3480cc`, verified blob `154f328a7249229ad3e0a25db83827595ea95f21`. The original repository and PostgreSQL/SSE source evidence remains linked in -[ADR-050](../adr/050-realtime-subscriptions.md#evidence-and-relationship-to-existing-decisions). +[ADR-050](../adr/050-realtime-subscriptions.md#links-and-evidence). Review agreement and documentation CI do not establish a tested subscription implementation, SQL cancellation, load safety or deployment correctness. The transferred review evidence paragraph was incomplete and executable model From b9306dda5778fdfb6820d1c82a67d6b358c92907 Mon Sep 17 00:00:00 2001 From: Brian Thorne Date: Sat, 31 Oct 2026 21:56:58 +1300 Subject: [PATCH 07/11] docs(adr): choose polling-first admin observation and defer push --- docs/adr/050-admin-view-observation.md | 177 +++++++++ docs/adr/050-realtime-subscriptions.md | 223 ----------- docs/adr/README.md | 2 +- docs/design/admin-observation-v1.md | 315 +++++++++++++++ docs/design/realtime-subscriptions-v1.md | 472 ----------------------- 5 files changed, 493 insertions(+), 696 deletions(-) create mode 100644 docs/adr/050-admin-view-observation.md delete mode 100644 docs/adr/050-realtime-subscriptions.md create mode 100644 docs/design/admin-observation-v1.md delete mode 100644 docs/design/realtime-subscriptions-v1.md diff --git a/docs/adr/050-admin-view-observation.md b/docs/adr/050-admin-view-observation.md new file mode 100644 index 000000000..0fc84dbab --- /dev/null +++ b/docs/adr/050-admin-view-observation.md @@ -0,0 +1,177 @@ +# ADR-050: Admin views use invalidation and authoritative refetch + +## Status + +Proposed. V1 uses improved polling. SSE, WebSocket and inter-process notification +listeners are deferred to a separate decision, not included as opt-in features. +[Draft PR #576](https://github.com/hardbyte/awa/pull/576) is documentation only. + +## Problem and constraints + +Operators need current views of jobs, queues, workers, schedules, the DLQ and +administrative progress. The UI already reads those resources through HTTP and +TTL caches, but independent page timers, ambiguous filter compatibility and +in-flight cache fills make freshness and load difficult to reason about. + +The priority order is correctness, reliability and recovery first, performance +second, UX third. A fresh view must not become a dependency of enqueue, claim, +finalization or rescue. The initial consumer is the trusted admin UI. Durable +business triggers and an application-facing notification API are different +requirements. + +A common refresh scheduler, whole-read deadlines, fenced cache fills, truthful +source timestamps and strict read contracts improve the current UI without a +stream. There is no measured operator need or transport comparison that justifies +adopting SSE alongside those changes. + +## Decision and scope + +**Use disposable invalidation hints, authoritative owning-API reads and independent +bounded reconciliation, implemented with polling in v1.** A local mutation can +mark the initiating tab's views stale and invalidate that server's caches. Other +tabs and admin processes discover changes through their own paced reads. + +The five invariants are: + +1. **Read authority.** A hint only requests a read. The owning API and shared + `JobFilter` define contents; unsupported semantics fail closed. An older cache + fill cannot undo invalidation or be relabelled as newly observed state. +2. **Independent recovery.** Register active read demand and arm reconciliation + before the first read. Lost hints, errors and resumes are repaired by scheduled + reads; a dirty mark received during a read remains pending afterwards. +3. **Bounded shared demand.** Polling, reconciliation, focus, retries and mutations + use one scheduler and finite client/server budgets. Coalescing retains demand + for every resource; overload reduces freshness rather than multiplying reads. +4. **Job-path independence.** Observation adds no job-transition writes, worker + hooks, mandatory listener or event-processing engine. Reliable business work + remains durable queue jobs, not a reaction to a UI observation. +5. **Trusted administration.** Read-only still exposes sensitive admin data; + scopes cannot silently widen. UI refresh preserves selections/drafts and does + not replace atomic mutation guards or confer ownership through presence. + +V1 reuses existing read models and ADR-049's filter/compiler. It adds no new +metrics, retained notice history, tenant authorization framework or application +completion guarantee. The [polling-first v1 contract](../design/admin-observation-v1.md) +specifies startup, read protocols, deadlines, cache fencing and validation. + +Convergence means retained state returned by ordinary APIs after changes settle +and reads succeed. It is not observation of every transition or a stronger +cross-table snapshot. Composite reads can miss claims or deferred-to-ready +handoffs; Repeatable Read does not make ring truncation MVCC-safe. A completed +job pruned before a read may never be observed. Follow-up jobs in +[ADR-029](029-transactional-followup-jobs.md) and +[ADR-046](046-handler-staged-follow-ups.md) remain the durable mechanism where +applicable; best-effort rescue follow-ups are not strengthened. + +## Alternatives considered + +### Improved polling — chosen + +One scheduler, strict read boundaries, bounded work and truthful stale state +address current correctness and operational problems without persistent browser +connections. Each increment is useful without a transport project. Reads remain +sampled and worker/peer changes wait for the configured cadence. We accept that +tradeoff rather than fund unproven low-latency delivery. + +### SSE or WebSocket — deferred + +A stream could invalidate other views immediately after a committed local admin +mutation. A process-local hub reaches only viewers connected to that process. +With independent uniform routing over N processes, a particular viewer shares +the mutation's process with probability 1/N; other routing/topologies differ. +Worker changes gain no speed from a timer-driven stream, and adding such a stream +does not inherently reduce database reads. + +SSE fits one-way HTTP hints; WebSocket supplies bidirectionality and subscription +changes on an existing connection. Both require bounded parsing/state, reconnect, +expiry, ingress/authentication handling and browser lifecycle work. Those costs +are not justified by the observation invariants, which polling already satisfies. +There is no v1 stream endpoint, descriptor protocol, parser or stream budget. +A future hint descriptor should carry only an opaque key, type and necessary job +identity; full filters belong in authoritative reads unless measured filtered +routing earns the extra complexity and privacy exposure. + +### Admin-to-admin PostgreSQL hints — optional future optimization + +A dedicated listener per admin process could invalidate peer caches after admin +mutations, without touching worker/job-execution writes. It would not push new +data to browsers: they still wait for polling. One replica already has local +invalidation; two replicas need evidence that stale-cache windows and mutation +frequency justify dedicated sessions and reconnect handling. A listener needs a +direct/session-pooled connection; transaction pooling cannot preserve its session +registration. We do not select this optimization for v1. + +`NOTIFY` is delivered after its transaction commits, but queue exhaustion can +fail that commit. To isolate hint failure from the authoritative mutation, a +future design would use a separate bounded best-effort post-commit attempt. +Crash/loss is acceptable because reconciliation remains independent. Commit +`LISTEN` before the initial cache resync and resync on reconnect. Payloads are +visible to database users, so use coarse nonsecret categories, never filters, +args or outputs. No listener is required for correctness. + +### Pushed snapshots or row deltas + +Shared aggregate snapshots might save refetches. Filtered snapshots/deltas need +membership, population entry/exit, deletion and page-refill semantics in a second +view model. Current-state refetch keeps those responsibilities in existing APIs. +Consider pushed aggregates only if measured read cost makes that complexity useful. + +### Durable event log/outbox or logical decoding + +A log/outbox can support retained events and replay, but needs complete writer +coverage, commit-safe ordering, idempotence, retention and gap recovery. A sequence +allocated before commit is not a replay watermark. CDC additionally requires +slot/privilege/failover operations and a semantic adapter over leases, cursors, +compact completions and pruning. Those can serve historical/event consumers; +they impose unnecessary dependencies for sampled admin views. + +## Consequences + +Operators get bounded, recoverable views with explicit stale state. Missed hints +cannot leave a retained job's detail stale indefinitely after successful polling +resumes. Late fills and overlapping refreshes do not silently reverse invalidation. + +Freshness remains sampled. The initiating tab can refresh after its own mutation; +other clients are not promised immediate invalidation. Separate processes have +independent caches and aggregate read costs. Current-state reads cannot guarantee +an audit trail, every completion, or an exact census during transitions. + +Shared read hardening and strict adapters are independently useful follow-ups to +#572. They should not wait for a streaming decision. Existing GET parsing and +per-statement timeouts must not be represented as strict mixed-version reads or +an end-to-end deadline; the supporting contract records the verified gaps. + +## Evidence required for a later transport decision + +Reopen transport selection only with an unmet operator freshness requirement or +a concrete push consumer, a measured improved-polling baseline and the intended +deployment topology. Compare coverage of local mutations, worker changes and +peer changes; database/read amplification, connections and browser freshness; +and supported ingress, authentication, expiry and reconnect costs. Name an owner +and explicit, measurable retain/reject criteria for benefit, coverage and cost +before starting any experiment. +This ADR invents no numeric success target or release deadline. + +A durable application event/completion requirement needs a different contract, +not a faster version of these hints. Scoped application authorization or actual +bidirectional interaction also changes the decision boundary. + +## Links and evidence + +- [Polling-first v1 contract](../design/admin-observation-v1.md): normative read, + scheduler, cache and access requirements with incremental delivery gates. +- [ADR-049](049-job-filter-and-indexed-tags.md) and + [PR #572](https://github.com/hardbyte/awa/pull/572): shared population semantics; + [current source evidence](../design/admin-observation-v1.md#dependencies-and-source-evidence) + distinguishes remaining GET/deadline gaps from resolved DLQ/UI issues. +- [Existing cache](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/cache.rs), + [admin state](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/state.rs) + and [polling policy](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/frontend/src/hooks/use-poll-interval.ts): + the baseline, not an SSE performance comparison. +- [PostgreSQL NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) + and [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html): + transaction failure, payload visibility and registration behavior for the + deferred admin-cache optimization. +- [ADR-041](041-rolling-upgrade-policy.md) and + [ADR-045](045-hot-path-trigger-lock-contract.md): rolling-upgrade and hot-path + constraints remain in force. diff --git a/docs/adr/050-realtime-subscriptions.md b/docs/adr/050-realtime-subscriptions.md deleted file mode 100644 index 29b550388..000000000 --- a/docs/adr/050-realtime-subscriptions.md +++ /dev/null @@ -1,223 +0,0 @@ -# ADR-050: Admin views use invalidation and authoritative refetch - -## Status - -Proposed. The SSE transport experiment is opt-in and disabled by default. -[Draft PR #576](https://github.com/hardbyte/awa/pull/576) contains this decision -and its supporting design contract; it implements no subscription feature. - -## Problem and constraints - -The admin UI needs current views of jobs, queues, workers, schedules, the DLQ -and operator progress. Those views answer operational questions, not just whether -a job changed status. Today ordinary HTTP reads and TTL caches serve them, with -polling spread across pages. Adding push without a common refresh policy risks -multiplying reads while leaving missed notifications and stale cache fills -unresolved. - -The priority order is job correctness, reliability and recovery first, -performance second, UX third. Observation must not become a dependency of -enqueue, claim, finalization or rescue. The initial consumer is the trusted admin -UI; a future application API must not acquire stronger delivery or authorization -promises by accident. The 0.6-to-0.7 transition permits interface simplification, -while the rolling-upgrade and database safety contracts still apply. - -## Decision and scope - -**Admin views use disposable invalidation hints, authoritative API reads and -independent bounded reconciliation.** This observation model is the enduring -choice. Fetch-based SSE is the proposed experimental transport for v1, not the -source of correctness and not a commitment to permanent transport infrastructure. - -A hint means that a resource may have changed. The owning read API decides its -current contents, eligibility and accuracy. Filtered job views reuse -[ADR-049's JobFilter](049-job-filter-and-indexed-tags.md), including its bounds, -strict parsing and separation of population from paging. Unsupported filters -fail closed, including when successive requests reach servers of different -versions. Routing hints does not create a second filter evaluator. - -The model has these invariants: - -- **Recovery does not depend on hints.** Establish observation before acknowledging - readiness, then refetch. Reconnect and scope changes resynchronize. Independent - reconciliation repairs a missed final hint even while the transport looks - healthy. A hint received during a read remains pending rather than being - cleared by that response. Both entering and leaving a filtered population must - invalidate the view. -- **Demand is bounded and shared.** Hints, polling, reconciliation, resume and - retries feed one paced refresh policy. Repeated demand coalesces without one - resource overwriting another. Read work, waiting work, fanout and slow clients - have finite budgets. Overload reduces freshness rather than creating more - polling or delaying job execution through an unbounded read backlog. -- **Invalidation cannot be undone by a late cache fill.** Once local state is - invalidated, an older in-flight fill cannot reappear as current. Observation - timestamps retain the source sample's age. Transport activity is distinct from - source freshness; it cannot prove a worker alive or a replica caught up. -- **Observation stays off the job write path.** V1 adds no worker hooks, database - triggers, mandatory listener, event log or subscription writes to job - transitions. Each admin process can refresh independently; peer-process and - worker changes converge through reads without a shared broker or leader. -- **Domain guards and access boundaries remain authoritative.** Hints and - optimistic UI cannot enforce ownership, exclusivity or idempotency. Mutations - retain atomic server guards, and refresh preserves selections and operator - drafts. Read-only access remains sensitive admin access. Scopes cannot silently - widen, and a connection does not extend authorization indefinitely. - -Authoritative reads retain their existing limitations. Composite loading can -miss an in-flight claim or deferred-to-ready handoff; neither a stream nor -Repeatable Read makes ring pruning an atomic snapshot. Convergence means retained -state returned by ordinary APIs after changes settle and reads succeed, not -observation of every job or transition. A completion pruned between reads may -never be observed. Durable business reactions remain queue jobs through -[ADR-029](029-transactional-followup-jobs.md) and -[ADR-046](046-handler-staged-follow-ups.md) where applicable; existing best-effort -rescue follow-ups are not strengthened. - -V1 remains within the existing trusted-admin boundary and existing read models. -It adds no tenant authorization framework, presence ownership, new queue metrics -or retained notification history. Protocol shapes, finite experimental limits, -cache mechanisms, lifecycle handling and acceptance tests live in -[the supporting v1 contract](../design/realtime-subscriptions-v1.md). - -### Why experiment with SSE - -A one-way HTTP stream fits invalidation from an admin process to its connected -views. Its immediate incremental opportunity is fanout after a committed local -admin mutation, so other views connected to the same process can refetch sooner -than their next poll. This does not accelerate worker-originated or peer-process -changes beyond the sampling cadence, and timer-driven SSE does not inherently -reduce database reads. - -Streaming therefore stays disabled by default. The experiment must compare it -with improved polling, measuring local-mutation freshness alongside total read, -connection and operational costs. PostgreSQL load tests must show job execution -and recovery remain protected; actual ingress/pooler tests must establish that -authentication, buffering and connection lifecycle behave as intended. A useful -observation model does not by itself justify enabling a transport. - -## Alternatives considered - -### Improved polling with the same observation model - -This is the strongest baseline and the default path while streaming is disabled. -One scheduler, -coalesced reads, bounded caches, clear stale state and reconciliation solve much -of the problem without long-lived HTTP connections. It handles worker and -peer-process changes at the same sampling cadence as timer-driven SSE. - -Polling lacks immediate server fanout of a local mutation to other connected -views. SSE is worth retaining only if that incremental freshness, or another -measured benefit, justifies its costs. If it does not, keep improved polling; -none of the enduring invariants requires streaming. - -### Fetch-based SSE - -Selected for the opt-in v1 experiment because one-way hints fit the existing -HTTP admin surface and ordinary requests still perform reads and mutations. -It adds reconnect, parsing, proxy, authorization-lifetime and browser lifecycle -work; using fetch does not provide EventSource's automatic recovery behavior. -The supporting contract bounds that work and defines safe fallback, but those -requirements are costs to validate, not evidence of an advantage already earned. - -### WebSocket - -WebSocket can change subscriptions bidirectionally without replacing a stream -and would suit an actual interactive command protocol. V1 needs one-way hints, -so that bidirectionality is unused and SSE fits the current HTTP request model. -Both transports still need bounded state, parsing, reconnect, authorization and -lifecycle handling. Choose between them on demonstrated requirements and deployed -behavior, not on a claim that fetch-SSE avoids that complexity. - -### Pushed snapshots or row deltas - -Shared aggregate snapshots might save refetch round trips. Filtered snapshots -and row deltas require server-side query or membership state, including page -refill, population entry/exit and deletion semantics. Refetch keeps the existing -read API authoritative and avoids maintaining that second view model. Consider -pushed aggregates if measurements identify a material benefit without weakening -recovery. - -### Durable event log or transactional outbox - -A log can retain transitions and support replay; a transactional outbox can couple -publication to a committed state change. Those are appropriate capabilities when -every transition is required. They also need complete writer coverage, commit-safe -ordering, snapshot/replay boundaries, idempotence, retention and gap recovery. A -sequence allocated before commit is not itself a safe replay watermark. - -Current admin views need retained state, while durable reactions already use -jobs. Paying WAL, publication and retention costs on job writes to obtain UI -freshness would introduce an unnecessary dependency. No log, outbox or -transactional directory is selected here. - -### PostgreSQL LISTEN / NOTIFY - -Existing wakeups can be useful disposable hints but do not cover all admin -resources or provide replay. New transactional emissions would add job-path cost; -a full notification queue can make notifying transactions fail at commit. -A mandatory listener would add session and deployment requirements without -removing the need for reconciliation. A later optional hint optimization must -preserve the observation model and earn its write/connection cost separately. - -### Logical decoding / CDC - -CDC can provide a change stream with a snapshot boundary, but requires slot, -privilege, failover and retention operations. Mapping Awa's leases, cursors, -compact completions and partition pruning into public changes would also require -a semantic adapter. It remains useful for external history and analytics, not a -required dependency of current admin views. - -## Accepted consequences - -Consumers can converge after lost hints, reconnects and process changes -without introducing a second event-processing engine. Job execution and durable -reactions remain independent of browser availability. - -The tradeoff is sampled freshness, potentially redundant reads and no guarantee -of observing transient or already-pruned state. Coarse invalidation deliberately -trades routing precision for simpler entry/exit correctness. Multiple admin -processes have independent caches and read costs; their combined load still -needs a budget. A successful stream does not imply a healthy queue or current -source data. - -The transport adds operational and client lifecycle costs while it remains an -experiment. Production SLOs and default enablement need evidence; documentation -checks establish neither implementation correctness nor capacity. - -## Revisit conditions - -Revisit the rationale when: - -- A consumer requires replay, audit history or guaranteed completion observation; - current-state refetch cannot provide those guarantees. -- Required worker or peer-process update latency cannot be met by sampled reads; - investigate a safe source of hints rather than treating SSE alone as a fix. -- Measured read/connection cost or ingress behavior makes streaming less useful - than improved polling, or makes shared pushed aggregates worthwhile. -- Application consumers require scoped authorization beyond trusted admin access, - or real bidirectional interaction changes the transport requirements. - -Revising a transport or sampling policy must preserve the enduring invariants -unless a later decision explicitly replaces them. - -## Links and evidence - -- [V1 implementation contract](../design/realtime-subscriptions-v1.md): protocol, - budgets, cache fencing, browser lifecycle, security policy and validation gates. -- [ADR-049](049-job-filter-and-indexed-tags.md) and - [implementation PR #572](https://github.com/hardbyte/awa/pull/572): shared job - selection and advisory read behavior. The proposed adapter reuses that compiler. -- [Existing dashboard cache](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/cache.rs), - [admin state](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/state.rs) - and [frontend polling](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/frontend/src/hooks/use-poll-interval.ts): - the current read/cache baseline, not a measured comparison with SSE. -- [Existing notification hub](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-worker/src/notify_hub.rs): - hints with polling recovery, not complete admin change coverage. -- [PostgreSQL NOTIFY](https://www.postgresql.org/docs/17/sql-notify.html), - [LISTEN startup](https://www.postgresql.org/docs/17/sql-listen.html), - [logical decoding](https://www.postgresql.org/docs/17/logicaldecoding-explanation.html) - and [SSE framing](https://html.spec.whatwg.org/multipage/server-sent-events.html): - substrate behavior, not proof of end-to-end freshness or delivery. -- [ADR-041](041-rolling-upgrade-policy.md) and - [ADR-045](045-hot-path-trigger-lock-contract.md): existing rolling-upgrade and - hot-path constraints remain in force. diff --git a/docs/adr/README.md b/docs/adr/README.md index db24516b7..6618fcb56 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -56,7 +56,7 @@ Each record preserves its status, context, decision, consequences, and alternati | 046 | [Handler-staged follow-up jobs](046-handler-staged-follow-ups.md) | A handler stages follow-up requests on its `JobContext`; they are inserted in the attempt's lease-guarded completion transaction, deduplicated by their own unique keys, and discarded on every other outcome. | Accepted | | 048 | [Retained job output](048-retained-job-output.md) | A handler may record a size-bounded JSON output for its attempt, sealed with the staging buffer and published only by the attempt's own lease-guarded transition: as the job's output on completion, or in the error entry of a failure, where only the latest entry to record one keeps it. It rides the compact completion batch, and neither the engine nor another job reads it. A kind may declare its output type, which types the API and is published in its descriptor. | Accepted | | 049 | [One job filter, and tags as the indexed handle](049-job-filter-and-indexed-tags.md) | One `JobFilter` describes "which jobs" for listing, counting, batch preview, batch execution and the DLQ, compiled once per engine with only the predicates a filter sets; SQL narrows candidates and a batched loader settles each job's current state. Tags are the indexed handle for "jobs about X": a partial GIN index over non-empty payload tags on each queue-storage partition, maintained by Postgres for every binary, emptied by pruning, and built online by the migrator. | Proposed | -| 050 | [Admin views use invalidation and authoritative refetch](050-realtime-subscriptions.md) | Disposable hints, owning API reads and independent bounded reconciliation; SSE remains an opt-in transport experiment. | Proposed | +| 050 | [Admin views use invalidation and authoritative refetch](050-admin-view-observation.md) | Polling-first owning API reads, disposable invalidation and independent bounded reconciliation; push transport is deferred. | Proposed | ## Correctness evidence diff --git a/docs/design/admin-observation-v1.md b/docs/design/admin-observation-v1.md new file mode 100644 index 000000000..66bd6bdc8 --- /dev/null +++ b/docs/design/admin-observation-v1.md @@ -0,0 +1,315 @@ +# Admin observation v1: polling, strict reads and bounded reconciliation + +Status: proposed supporting contract for +[ADR-050](../adr/050-admin-view-observation.md). V1 uses polling. It includes no +SSE/WebSocket endpoint, browser subscription service, worker hooks, database +notification listener, outbox or CDC integration. Future transport alternatives +and their evidence gate belong in the ADR, not this delivery sequence. + +## Owning reads and startup + +The owning APIs remain authoritative for jobs, queues, workers, DLQ, schedules, +operator progress, storage and existing statistics. Reuse their current meanings, +accuracy and retention. Do not invent metrics or historical notices. Catalog/kinds +keep their existing read behavior; `/storage?history=true` does not supply +server-retained history. Application consumers and tenant authorization are +outside v1. + +For each visible tab, the automatic-read startup sequence below excludes +paused views and manually refreshed counts. Their explicit reads still share +the budgets but do not acquire an automatic watchdog. + +1. Establish the backend identity and supported read protocols. Capabilities are + advisory; verify the actual response to every request. +2. Register active exact query keys with the one tab-level scheduler, mark each + initially dirty and arm its independent reconciliation timer **before** + dispatching the first bounded read. A key includes backend, read scope, full + normalized filter and page. There is no stream registration, `ready` event or + snapshot cursor to wait for. +3. Read through the shared client concurrency/cooldown policy. Capture the key's + dirty generation at request start. A response clears only that captured + demand; a mutation/focus mark arriving during the read leaves one pending + refresh. Successful data may render while still marked dirty/stale. +4. Fence old filter/backend identities and late responses. A filter or active-view + change cancels/retires its old key before registering the replacement. Retain + already displayed data as stale where appropriate rather than claiming a new + successful observation. + +No active keys means no resource reads or resource timers. Enforce the active-key +cap before registration; reject excess demand visibly rather than silently +truncating or creating another scheduler. One pending marker per key prevents +interleaved demand for different resources from overwriting each other. + +Polling and reconciliation use one request-dispatch path. The periodic refresh +cadence may renew normal views sooner than the independent watchdog; neither a +local dirty hint nor a failed request postpones the watchdog. Successful reads +renew the observation. The watchdog marks demand through the same scheduler, +never launches a second request loop. Overload/cooldown/backoff still apply, so +these are scheduling bounds rather than a promised maximum data age. + +## Filter and protocol boundary + +Use the actual ADR-049 `JobFilter` compiler, deserializer **and `validate()`**. +Fields intersect, states are any-of, tags are all-of, and creation bounds refer +to original insertion. Page/window semantics remain separate from the population. +Refetching a page removes jobs that left and fills vacancies from current matches; +there is no stable multi-page snapshot or guarantee of every transition. + +Add thin strict JSON read adapters: + +- `POST /api/v1/jobs/query` +- `POST /api/v1/dlq/query` +- `POST /api/v1/jobs/count` + +These paths are an additive, explicitly versioned **read contract**, alongside +Awa's existing unversioned `/api/*` endpoints. They do not rename or version all +existing APIs. In particular, retain `/api/dlq/retry-matching`, +`/api/dlq/purge-matching` and legacy mutation routes; cosmetic consistency does +not justify another mutation protocol change. + +A version in the URI or JSON is not proof of strict behavior. Enforce exact +version agreement and reject unknown envelope/filter/page fields, using strict +page wrappers where needed. Reuse existing response conversion and model logic; +do not create a UI-side compiler. Query request: + +```json +{"version":1,"filter":{},"page":{}} +``` + +Count requests omit `page`. Apply shared filter bounds and a 64 KiB request-body +limit while receiving/parsing, before allocation can grow without bound. Existing +`JobFilter` and payload ID representations remain unchanged; the current browser +rejects IDs outside JavaScript's safe-integer range rather than rounding them. +Do not claim a new arbitrary-precision or decimal-string ID protocol here. + +Response envelope: + +```json +{ + "version": 1, + "data": "", + "observation": { + "observed_at": "", + "served_at": "", + "cache_age_ms": 0 + } +} +``` + +The example strings stand for real payloads/timestamps. Add optional structured +`job_filter_protocols` capability metadata; absent support is unsupported, never +inferred from semver. Do not advertise a `subscriptions` transport capability. +Capabilities can come from a different server version than the next read, so +check status, JSON MIME, complete envelope and exact response version every time. +Unknown `/api/*` paths on the new router return API errors, not SPA HTML. + +An older server may return 200 HTML or 404/405 for the route. That is unsupported, +not an empty result. Permit unfiltered legacy lists/details through the same +scheduler; nonempty filters remain unsupported. Never retry by dropping +predicates, switching a filtered request to a permissive GET, or displaying an +unsupported count as zero. New-server budgets cover permitted legacy fallbacks +too; they do not establish new server-side guarantees on old binaries. + +Fixing unknown/empty GET parsing on a current server is a separate improvement +from this strict mixed-version read boundary. Neither strict JSON parsing in one +path nor a `/v1` name makes an older GET implementation strict. + +## Errors, deadlines and shared read budgets + +Use structured JSON API errors: invalid versions/fields/values return 400, +body-size excess 413, authorization/origin failure 401/403, admission excess +429 with `Retry-After`, and temporary overload 503. Unsupported routes, wrong +MIME and malformed envelopes are protocol errors, even with HTTP 200. + +Network/read failures retain stale data and one pending demand, with exponential +backoff/jitter capped at 60 seconds. Honor 429/503 `Retry-After` across reads to +the affected backend; do not turn overload into an immediate extra poll. Stop +401/403 retries until credentials/user action change. Invalid filters/protocols +show an error instead of an automatic retry loop. Reprobe absent read capability +no more often than every 60 seconds unless explicitly requested. + +The following are finite experimental read/scheduler defaults, not production +capacity commitments or freshness SLOs. Configure them with sanity checks; zero +or unlimited settings must not defeat the bounds. + +| Bound | Initial setting | +| --- | --- | +| Active managed query keys | 16 per tab; no silent eviction. | +| Client read concurrency | 2 per tab; one in flight and one pending marker per exact key. | +| Debounced dirty demand | 100 ms target, 500 ms maximum before becoming eligible; cooldown/backoff still applies. | +| Cached refresh floor | At least advertised cache TTL/poll interval, currently at least 5 seconds. | +| Uncached detail/page floor | At least 2 seconds between reads. | +| Independent reconciliation | Nominal 30-second watchdog per active key, finite bounded jitter; expose the configured maximum. | +| Server expensive reads | 4 simultaneous operations per process. | +| Read admission | At most 16 waiters; at most 250 ms waiting; excess receives 429. | +| Whole-read deadline | 15 seconds including admission, pool acquisition, all queries/passes and serialization. | +| SQL statement limit | At most 5 seconds, also bounded by remaining whole-read deadline. | +| SQL lock wait | At most 1 second, also bounded by remaining deadline. | +| Repeated failure | Exponential backoff to 60 seconds; one pending demand retained. | + +One frontend scheduler handles periodic reads, reconciliation, focus/resume, +mutation refreshes, retry and manual refresh. Managed React Query queries disable +independent automatic intervals, focus fetches and reconnect fetches. Migrate +existing two-second job-detail/operation intervals into this scheduler explicitly. +Repeated demand coalesces; continuous changes must neither starve results nor +create an unbounded queue. The client deadline runs independently of pending +network/body reads so a silent hung request is still aborted. + +A Rust future timeout does not prove SQL cancellation. Apply server-side limits +to all candidate queries, loader transactions, repeated passes, aggregates and +DLQ reads, bounded by the remaining whole-read budget. Cancellation must roll +back or close outstanding work before pool reuse. Do not leak read timeout +settings into mutation sessions. Cache hits may bypass expensive-read admission; +each fill acquires it once. Bound obsolete fills, filter cache cardinality and +pool/connection use. Do not create a database task per registered tab query. + +## Cache fencing and source observations + +Use one local post-commit invalidation helper across admin mutation families, +including DLQ retry/purge/move and batch cancellation. After the authoritative +mutation commits, advance the cache generation, retire its old cache entries, +and mark affected server read state stale. Rolled-back mutations invalidate +nothing. The initiating browser marks its own affected keys dirty after a +successful action; other browsers/processes receive no v1 push and refresh on +schedule. Broader local invalidation is acceptable until measured routing is +worthwhile. + +A fill captures a bounded monotonic generation before starting. It may +publish/return as current only if the generation still matches at completion. +Requests after invalidation cannot join an obsolete fill. Use generation-keyed +caches with bounded obsolete work or an explicit synchronized check, without +holding a cache lock across a query. Discard/retry obsolete completions only +within remaining read budget; never loop indefinitely or relabel old data fresh. +Moka `invalidate_all()` alone is not proof of this invariant. + +Client dirty generations and server cache generations solve different races; +both are required. `observed_at` is source-fetch start (a conservative start for +composite reads), `served_at` is response time and `cache_age_ms` is cached +observation age. Preserve source time on cache hits. Existing reads can carry +common metadata/headers without changing payload meaning; strict adapters use +the envelope above. A response served now is not necessarily a new observation. + +Each admin process owns its caches. Worker and peer mutations are covered by TTL +and periodic authoritative reads; a local generation does not create a global +snapshot. Heartbeat evidence determines worker liveness, never browser activity. +Replica freshness/lag is unknown unless actually measured. + +## UI lifecycle, domain guards and access + +Hidden tabs pause managed reads and timers. `pagehide`, including BFCache entry, +cancels them; persisted `pageshow` retires old request identities and starts +exactly one scheduler with one paced refresh if visible and nonempty. Cancelled +reads do not clear pending demand. Filter/view churn releases registrations and +obsolete work without losing unrelated keys' demand. + +Preserve selected IDs and unsaved operator form/dialog values during refresh. +Jobs with an active selection remain paused and stale until explicitly resumed; +manual refresh re-enters the same scheduler. Filtered job-match counts are +excluded from periodic/watchdog/focus refresh and prefix invalidation: mark them +stale and fetch only on filter change, explicit refresh or the specified debounced +operator action. Counts remain capped under ADR-049. Selection/draft state never +replaces server-side eligibility, version/lease/fencing and idempotency guards. +Viewing/presence does not confer ownership; no viewer membership protocol is added. + +This is the existing trusted-admin surface, protected by ingress/private network +controls. `--read-only` disables mutations, not sensitive reads. Authenticate and +authorize each read under that boundary; do not create longer-lived access via +client cache state. There is no new tenant authorization model. + +The inspected router currently applies permissive CORS and a SPA fallback. Those +are current observations, not verification of configured origins in a deployment. +New strict JSON POST adapters must explicitly reject a present Origin unless it +matches a configured public admin origin; do not derive trust from arbitrary +forwarded-host headers. Isolate their route policy so outer permissive CORS +cannot undo it. No wildcard credentialed CORS. Browser fetch uses +`credentials: "same-origin"` and `cache: "no-store"`; reject cross-origin redirects +and login HTML as protocol data. Origin-less non-browser requests still require +ingress/private-network trust. Any later credentialed streaming proposal must +resolve/tighten this boundary before exposure, not inherit permissive CORS. + +Never put credentials, args, outputs or secrets in URLs or observation logs. +Strict reads carry filters in bodies. Existing shareable search URLs may contain +documented nonsecret opaque values; no raw filter descriptors belong in automatic +logs. An origin check does not authorize a filter or make public unauthenticated +admin access safe. + +## Incremental delivery and acceptance + +Each increment is independently useful with polling: + +1. Harden owning reads: strict adapters/capability and response checks, API error + fallback, bounded admission, whole-read deadlines and cancellation coverage. + Reuse #572's compiler; GET parsing/deadline repairs are shared follow-up work, + not a transport dependency. Test empty install/read-only operation without a + worker. Do not rename mutation endpoints for URI consistency. +2. Consolidate the tab scheduler and lifecycle. Register dirty demand before + initial reads, disable competing automatic fetch loops, enforce per-key + generations/concurrency/cooldowns/backoff and preserve paused selections/counts. +3. Add generation-fenced caches, conservative source metadata and complete local + post-commit invalidation. Deterministic barriers must reproduce an old fill + completing after invalidation and prove it cannot become current. +4. Complete independent reconciliation and failure recovery through that same + scheduler. Prove convergence after lost local marks and peer/worker changes, + request failure, scope changes, hidden/resume and BFCache restoration. No + listener or transport handshake is a prerequisite for any read. +5. Measure the improved-polling baseline on PostgreSQL 17 and 18, with zero, + ordinary and maximum admitted views and multiple admin processes. Check job + enqueue/claim/finalization p95/p99, rescue progress, WAL, query QPS, pool counts, + cache fills and browser read/render freshness under supported ingress/poolers. + +Deterministic acceptance covers: initial/empty/over-cap registration; a dirty +mark during the first or later read; interleaved resource demand; old responses +following filter/backend changes; cache races; SQL timeout rollback and pool +recovery; slow/hung/aborted reads; continuous demand; errors/backoff/Retry-After; +old-server HTML/404/405 and capability/read requests routed to different versions; +401/403; counts, paused selections and drafts; local rollback; peer mutations; +heartbeat expiry without writes; page refill and retained-not-found presentation. +Tests must not demand every transient transition or a stronger snapshot than +the owning API provides. + +Keep correctness/read-budget work separate from claims of production freshness. +Distinguish source/pipeline lag where measured, transport/network activity, API +latency and browser render age. Missing lag remains unknown, consumer-stage +metrics cannot establish universal subsecond freshness, and the last histogram +bucket does not identify an exact delay. No numeric release/SLO commitment is +introduced. A later transport requires the ADR's separate evidence/owner decision. + +## Dependencies and source evidence + +Source inspection on 2026-10-31 used #572 head +`c09b23ebed0d65614025271611c720ea584a55c9`, which includes change +`ac1e39440d89e6c0c5a5bb86a7d3af5d8db47d86`. These observations +are evidence of current gaps/fixes, not a claim that this proposed contract ships. + +- [Candidate listing](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-model/src/job_query.rs) + loops up to 32 passes with a 30-second statement timeout per candidate query. + [The loader](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-model/src/queue_storage.rs) + starts a separate transaction using planning settings without installing that + timeout. This does not establish a whole-read deadline. Fixes belong in the + shared read layer/#572 follow-ups; verify the actual implementation base. +- [GET parsing](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/src/handlers/filter_params.rs) + deliberately ignores unknown and empty values, including a test of that + behavior. [Finding-jobs documentation](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/docs/finding-jobs.md) + gives a general unknown-field rejection promise; it must not be interpreted as + current GET strictness. Parser/documentation repair and strict mixed-version + adapters are distinct work, both useful without this ADR or any streaming. +- [DLQ handlers](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/src/handlers/dlq.rs), + [router](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/src/lib.rs) + and [API test](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/tests/admin_api_test.rs) + use separate retry/purge-matching paths for the full filter, legacy bodies on + legacy paths and refusal of `all` with a nonempty filter. The obsolete richer- + body-on-legacy-route downgrade concern is not retained as an unresolved v1 gate. + The PR reports a real 0.6.9 rejection check; source inspection here is not a + fresh execution of that rehearsal. +- [Search validation](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/frontend/src/components/SearchBar.tsx) + accepts positive whole IDs only when `Number.isSafeInteger` succeeds; + [the batch form](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/frontend/src/routes/batch-ops.tsx) + uses that validator. This rejects unsupported IDs; it is not full-precision + browser ID support. +- [Router](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/src/lib.rs) + currently applies `CorsLayer::permissive()` and the SPA fallback. Access-policy + claims above are implementation requirements, not assertions about a deployed + origin configuration. + +Other storage/uniqueness release concerns were not revalidated here; this +contract neither clears nor replaces their existing release gates. diff --git a/docs/design/realtime-subscriptions-v1.md b/docs/design/realtime-subscriptions-v1.md deleted file mode 100644 index 2a1c89534..000000000 --- a/docs/design/realtime-subscriptions-v1.md +++ /dev/null @@ -1,472 +0,0 @@ -# Real-time subscriptions v1: implementation contract - -Status: proposed companion contract to -[ADR-050](../adr/050-realtime-subscriptions.md). Streaming is **disabled by -default**, enabled only by explicit server opt-in. The settings below are finite -experimental engineering limits, not measured production SLOs. This document -specifies a future implementation; it does not describe a shipped API. - -V1 serves the existing trusted admin UI. It uses process-local coarse -invalidation, ordinary read semantics and mandatory reconciliation. No -transactional directory, outbox, CDC integration, tenant authorization framework -or worker change is required. Durable business triggers remain queue jobs. - -## Streaming request and validation - -Use **`POST /api/subscriptions/v1`**, accepting strict JSON and returning -`text/event-stream`. The tab uses `fetch` and an AbortController. This is one -streaming request/response, not a registration/session service. There is no native -EventSource GET variant in v1. Resource changes abort the old stream before -opening its replacement. Retain displayed data, marked stale, until successful -reconciliation. - -```json -{ - "version": 1, - "resources": [ - { - "key": "r1", - "type": "jobs", - "filter": { - "queue": "emails", - "tag": ["tenant:opaque", "order:opaque"] - } - }, - {"key": "r2", "type": "job", "job_id": "123"}, - {"key": "r3", "type": "workers"} - ] -} -``` - -Validate the complete request before registration or HTTP 200: - -- Exact protocol version; unknown fields rejected at envelope, resource and - filter levels; unique opaque keys and type-specific fields. -- Both `JobFilter` deserialization **and `validate()`**, using the existing - implementation and bounds. Strict parsing alone does not enforce validation. -- Valid IDs/UUIDs and every byte/count bound. Subscription job IDs are decimal - strings to avoid another JavaScript integer-precision dependency. - -Body/descriptor limits apply while receiving and parsing, before allocation can -grow without bound. Do not echo raw descriptors or filters in events or logs. -After validation the hub may retain only keys and category mappings; it never -evaluates filters. Opening a stream performs **zero database connections and zero -state queries**. - -An empty active resource set opens no stream and schedules no resource reads. -The server rejects an empty `resources` array with 400. More than 16 descriptors -rejects the complete request under the size-limit policy; never silently truncate, -partially subscribe or open extra streams to bypass the cap. Display the error -while retaining safe, paced reads of supported views. - -## Initial resources - -These types invalidate existing read results; they promise no new metrics or -retained history. The UI owns explicit active-query mappings. - -| Type | Additional fields | Existing reads invalidated | -| --- | --- | --- | -| `jobs` | Required `filter: JobFilter` | Jobs pages and dashboard job sample; count becomes stale only. | -| `job` | Required decimal-string `job_id` | One job's current detail, output, progress and source. | -| `dlq` | Required `filter: JobFilter` | DLQ page/detail and depth. | -| `queues` | None | Queue list/detail and `/queues/runtime`. | -| `workers` | None | `/runtime`, derived runtime details, capabilities and coverage. | -| `schedules` | None | `/cron` and `/cron/reconciliation`. | -| `operations` | Optional `operation_id: UUID` | Batch-operation list or one operation. | -| `maintenance` | None | `/storage` and existing sampling behavior. | -| `stats` | None | `/stats` and current timeseries views. | - -Exclude standalone `recovery`, retained notices/history, application consumers -and named queue/worker/schedule selectors from v1. They need further read -contracts or add little with coarse routing. `/storage?history=true` supplies no -server-retained history; subscriptions must not imply otherwise. Catalog/kinds -views keep existing read behavior. Queue health, throughput, age and recovery -displays use only existing fields with documented accuracy. - -`JobFilter` describes a population; pages remain separate. Invalidation refetches -the active bounded page, removing jobs that left and filling vacancies from -current matches. It does not promise stable multi-page snapshots, an infinite -list or visibility of every transition. Unknown routing information invalidates -the broader category rather than risking a missed leave. - -## Events and stream parsing - -The only data event types are `ready`, `invalidate` and `resync`: - -```text -event: ready -data: {"version":1,"epoch":"opaque-connection-id","keys":["r1","r2","r3"],"refresh_ms":5000,"reconcile_ms":30000,"keepalive_ms":15000} - -event: invalidate -data: {"version":1,"epoch":"opaque-connection-id","keys":["r1","r2"]} - -event: resync -data: {"version":1,"epoch":"opaque-connection-id","reason":"overflow"} - -``` - -`ready.keys` equals the requested set: never partially accept. Registration -precedes `ready`; a client displaying a preliminary read must refetch after -`ready`. Epoch is connection identity, not a database position. No SSE `id`, -offset, acknowledgement or transition payload is emitted. Keepalives are comments -and do not increment dirty generations. - -Check version, envelope, accepted keys and epoch. Unknown versions or malformed -events close the stream and enter safe fallback with a visible protocol error; -invalid protocol does not cause an automatic stream retry loop. Enforce frame -limits even when a peer never finishes a frame. Parse chunked SSE, comments, -split UTF-8 and split line boundaries with finite buffers. A fetch client must -implement reconnect itself; native EventSource reconnect behavior is not automatic. - -Run connect/ready and byte-liveness deadlines on independent timers, not inside -or after the stream reader's await/iterator. A pending read that never returns -must still be aborted at its deadline. Bytes/keepalives establish transport -activity only; they neither establish source freshness nor reset reconciliation. - -## Capabilities and strict versioned reads - -Add optional structured `subscriptions` and separate `job_filter_protocols` -fields to `/api/capabilities`. Advertise enabled protocols, accepted types, -limits, refresh policy and disabled status. Absence means unsupported; do not -infer support from semver. Capabilities are advisory: the next request can reach -an older server. - -Add thin strict JSON adapters: - -- `POST /api/v1/jobs/query` -- `POST /api/v1/dlq/query` -- `POST /api/v1/jobs/count` - -Query bodies use the existing filter and page semantics: - -```json -{"version":1,"filter":{},"page":{}} -``` - -Count bodies omit `page`. Reject unknown fields, including page fields through -strict wrappers where existing page deserializers are permissive. Reuse the -model/compiler and existing response conversion, not a second evaluator or -storage model. Responses wrap the existing payload without changing its meaning: - -```json -{ - "version": 1, - "data": "", - "observation": { - "observed_at": "", - "served_at": "", - "cache_age_ms": 0 - } -} -``` - -The strings above mark payload/timestamp placeholders, not literal wire values. -The new UI's filtered subscription/refetch path uses these adapters. Existing -permissive GET endpoints may remain for compatibility. Unknown `/api/*` routes -in the new router return API errors, not SPA HTML. - -Unsupported routes, wrong MIME, malformed envelopes and mismatched versions are -errors. Check JSON/version agreement for every read and `text/event-stream` plus -valid `ready` for streams. HTTP 200 alone proves nothing: an older server's SPA -fallback may return login/HTML content. Never retry by dropping predicates. - -Without v1 reads, permit legacy polling of **unfiltered** lists/details only. -Nonempty filters remain unsupported rather than rewritten; only a separately -tested legacy subset may later relax this. Do not display an unsupported job-match count -as zero. New-server read budgets apply to SSE-triggered reads and their legacy -fallback paths alike; fallback does not establish new server-side guarantees on -an old binary. - -Never place credentials, args, outputs or secrets in any URL. Do not place raw -filter descriptors in subscription URLs or automatic subscription logs. Existing -shareable search URLs may contain documented nonsecret opaque filter values. -JSON adapters avoid filters in new API URLs without removing intentional search -and bookmark URLs. - -### Errors and retry policy - -Before streaming headers, return structured JSON errors: - -| Condition | Status | -| --- | --- | -| Invalid/unsupported version, type or fields | 400 | -| Size limit | 413 | -| Authorization/origin failure | 401/403 | -| Admission limit | 429 with `Retry-After` | -| Temporary overload or disabled transport | 503 | - -404/405 or wrong MIME means unsupported transport. After streaming headers, use -resync/close; do not pretend to change HTTP status. - -- Network/transport failures retain stale data, use the **same paced read - scheduler**, and reconnect with exponential backoff/jitter capped at 60 seconds. -- 429/503 honor `Retry-After` across stream attempts **and refetches**. Overload - must not create immediate additional polling. -- 401/403 stop retries until credentials or user action change. -- Invalid protocol/request stops stream retries and shows the error. Safe read - fallback still requires supported filter semantics and the same scheduler. -- Reprobe absent transport capability no more than every 60 seconds, unless the - user explicitly requests it. - -## Finite experimental budgets - -All limits are configurable with sanity checks. Zero/unlimited must not defeat -the contract. These values are the initial opt-in implementation contract, not -production capacity or freshness promises. - -| Bound | Initial setting | -| --- | --- | -| Feature default | Disabled; explicit server opt-in. | -| Resources | 16 descriptors per stream. | -| Resource keys | Unique; 1–64 ASCII alphanumeric, `_` or `-` characters. | -| Request size | 64 KiB total; 32 KiB per descriptor. | -| Streams | 64 per process. | -| Principal quotas | No invented identities in no-auth mode; ingress may enforce quotas. | -| Pending stream state | One dirty bit per admitted key plus one resync flag. | -| Event/frame budget | 4 KiB application pending/frame budget. | -| Debounce | 100 ms target; 500 ms maximum from first dirty event. | -| Keepalive | Comment every 15 seconds. | -| Client liveness | Reconnect after 45 seconds without bytes. | -| Connect/ready deadline | 10 seconds. | -| Slow transport | Abort after a frame is pending for 5 seconds without body-consumer progress. | -| Stream lifetime | 5 minutes, then reconnect/resync. | -| Periodic hints | Existing `poll_interval_ms`, currently at least 5 seconds and cache TTL. | -| Reconciliation | Jittered 30-second watchdog per active resource. | -| Client read concurrency | 2 per tab; one in flight and one pending marker per exact query key. | -| Refresh floor | Cached reads respect advertised TTL/poll interval; uncached detail/pages at least 2 seconds apart. | -| Server expensive reads | 4 simultaneous operations per process. | -| Read admission | At most 16 waiters; 250 ms admission wait; excess gets 429. | -| Whole-read deadline | 15 seconds, including admission, pool acquisition, all queries/passes and serialization. | -| SQL statement limit | At most 5 seconds, also bounded by remaining whole-read deadline. | -| SQL lock-wait limit | At most 1 second, also bounded by remaining deadline. | -| Repeated failure | Exponential backoff to 60 seconds; retain one pending demand. | - -Bound request/descriptor parsing before large allocation. Coalesce repeated -invalidation into dirty bits. Overflow collapses to one all-resources resync; -close if that cannot make transport progress within budget. Slow consumers never -block the shared hub, readers, workers or producers. Release stream permits on -body drop, shutdown and failure. Idle time without a pending frame is not -backpressure. Enqueueing into an application channel is not evidence of transport -delivery; local body-consumer progress is not a remote receipt acknowledgement. - -The process-wide expensive-read budget covers SSE-triggered and fallback reads. -Cache hits may bypass admission; each cache fill acquires it once. Do not create -a database task or pool connection per subscription. Bound obsolete fills, -filtered cache cardinality and total fanout as well as per-stream state. Share -aggregate reads; any filtered coalescing key includes backend, authorization -scope, full normalized filter and page. - -A Rust future timeout alone does not prove SQL cancellation. Apply server-side -deadlines to **all** constituent queries through appropriately configured admin -read connections or deadline-aware transaction helpers. Do not alter mutation -session timeouts accidentally. Verify cancellation rolls back or closes -outstanding work before a connection becomes reusable. - -Audit candidate queries, separate loader transactions, all listing passes, -aggregates and DLQ reads. The document-specific review identified a gap in -#572's candidate-query-only timing: its loader opens a separate transaction with -planning settings, and listing may make 32 candidate passes. This contract -requires the entire operation, not each pass independently, to fit the deadline. -The implementation must verify coverage against its actual base revision. - -## One frontend refresh scheduler - -Use one tab-level scheduler for SSE hints, watchdogs, legacy polling, -focus/resume, reconnect and mutation refreshes. Managed queries disable -independent automatic intervals, focus fetching and reconnect fetching. Migrate -the existing two-second detail/operation polling explicitly. Every cause marks -the same bounded dirty state; no component opens its own stream. - -With healthy SSE, periodic hints and reconciliation drive reads. With unhealthy -SSE, polling takes over **through the same scheduler**. Do not run both plus -independent React Query timers. Hints and keepalives alone never postpone the -reconciliation watchdog. Resync obeys admission, cooldown and backoff. A failed -read does not count as successful reconciliation or clear pending demand. -This includes a missed final invalidation: a retained completed job's detail -must still refresh through reconciliation even when SSE continues sending bytes. - -A request captures its query's dirty generation at start. Its response clears -only that captured demand. A hint during the request leaves one pending refresh; -the client may render the successful response while still showing stale/dirty -state. Fence superseded filter/stream epochs and late responses. Continuous hints -must neither starve visible results nor build an unbounded queue. -Keep demand by resource/query key, not one global latest-resource slot; hints -for a second resource cannot overwrite the first resource's pending demand. - -Respect pauses and explicit actions: - -- Jobs currently pause automatic refresh while IDs are selected. Hints mark the - view stale without replacing the selection. Resume/manual refresh re-enters - the scheduler; mutations still use explicit selected IDs and server guards. -- Filtered job-match counts are excluded from automatic SSE/watchdog/focus refresh. Mark them stale; - refresh only on filter change, explicit refresh or the specified debounced - operator action. Prefix invalidation must not accidentally poll counts. -- Hidden tabs pause streams and reads. Visible resume performs one paced resync. - -Resource-set changes cancel/fence old work before replacement. On `pagehide`, -abort streams and managed reads and stop timers, including when entering the -browser back-forward cache (BFCache). On persisted `pageshow`, discard the old -transport epoch and resume exactly one scheduler, with one paced resync only if -visible and nonempty. Cancelled reads do not clear demand or count as successful -reconciliation. Test repeated scope churn and restore without leaked permits, -read tasks or duplicate loops. - -Preserve unsaved operator form/dialog drafts as well as selected IDs during -refetch; do not silently replace the values the operator is reviewing. Optimistic -UI coordination is separate from atomic domain mutation guards. Destructive -operations validate current eligibility and applicable version/lease/fencing -conditions atomically on the server; notifications and disabled buttons cannot -enforce exclusivity or idempotency. Retain the existing server guards rather -than inventing a subscription ownership protocol. Viewing a job or observing -worker presence grants no ownership; active attempt ownership remains a lease -and domain-state question. - -Convergence means retained state returned by ordinary APIs after changes settle -and reads succeed. It does not strengthen cross-table consistency or guarantee -observation of every transient transition. Composite loading may miss ordinary -deferred-to-ready handoffs as well as claims; Repeatable Read alone does not make -ring `TRUNCATE` MVCC-safe. Preserve reconciliation rather than asserting a perfect -snapshot/stream boundary. Absence after retention is not proof of never existing. - -## Cache generations and truthful observations - -Use a bounded monotonic generation per aggregate cache group. After a successful -local mutation commits, one post-commit helper: - -1. Advances the cache generation. -2. Invalidates or retires the old generation. -3. Signals the hub. - -A fill captures its generation before starting. It may publish/return as current -only if that generation matches at completion. Requests after invalidation must -not join an old-generation fill. Use generation-keyed caches with bounded obsolete -work or an explicit synchronized check; never hold a cache lock over a database -query. Discard/retry obsolete completions only within the remaining read budget. -Do not relabel them fresh or loop indefinitely under continuous invalidations. -Existing Moka `invalidate_all()` alone is not an established proof. Require a -deterministic old-fill-after-invalidation test. - -Client and server generation fences are both required. The client cannot detect -an obsolete cache fill served as fresh. Read metadata distinguishes: - -- `observed_at`: source-fetch start, or a conservative start/end observation - window for a composite read. -- `served_at`: response time. -- `cache_age_ms`: age of the cached observation. - -Preserve original observation time on cache hits. A fresh HTTP response does not -make an old sample newly observed. Use a common wrapper or headers for existing -payloads; the new v1 adapters use the envelope above. Worker liveness remains -heartbeat evidence under its documented window. Replica freshness/lag is -**unknown** unless actually measured; primary hints do not prove replica replay. - -V1 may conservatively invalidate all admitted categories and aggregate caches -after any successful local mutation. Include every family, notably DLQ -retry/purge/move and batch cancellation, whose current handlers do not uniformly -invalidate caches. Rolled-back mutations emit no hints. Worker and peer-process -activity is timer-covered, with no added hooks or triggers. Each admin process -owns its hub/cache/timer and reads its backend; no broker or sticky routing is -needed. Measure aggregate read costs across processes. - -## Trusted-admin origin and credentials - -V1 is the existing trusted admin surface, not an application tenant API. -`--read-only` disables mutations, not access to sensitive data. Ingress/private -network controls protect reads and streams; an origin check does not authorize a -JobFilter or make a public unauthenticated admin safe. - -Default new adapters to same-origin browser access with existing ingress auth: - -- Require JSON POST bodies. Reject a present Origin unless it matches the - configured public admin origin. Never derive an allowlist from arbitrary - forwarded-host headers. -- No wildcard credentialed CORS. Isolate new routes or deliberately replace - existing permissive CORS so an outer layer cannot undo this policy. -- Browser fetch uses `credentials: "same-origin"` and `cache: "no-store"`. - Reject cross-origin redirects/login HTML as protocol data. -- Non-browser requests without Origin retain the existing ingress/private-network - trust requirement. - -SSE response headers: - -```text -Content-Type: text/event-stream -Cache-Control: private, no-store, no-transform -X-Accel-Buffering: no -``` - -Use the actual ingress's buffering controls, testing authentication proxy, -compression, idle timeouts and streaming behavior. The five-minute stream -lifetime allows ingress reauthentication; a server without identity cannot -promise immediate per-user revocation. Application-scoped consumers remain -outside v1. Do not put secrets, args or output in notifications or transport logs. - -## Implementation and acceptance - -Use a separate draft implementation PR, stacked on #572 or after its merge, -reusing its actual `JobFilter` compiler. Do not copy a second compiler into the -UI crate. Adopting this contract permits opt-in implementation, not default -enablement, release or merge. - -1. Adopt ADR-050 and this protocol/resource/fallback/privacy/limit/scheduler/cache/ - origin contract. Keep experimental defaults and disabled-by-default status. -2. Add protocol types/capabilities, strict read adapters and status/MIME checks. - Verify read-only and empty-install behavior without starting a worker. -3. Add the bounded process-local hub/timer and post-commit helper. Prove stream - opening does zero state queries/connections. Test descriptor/frame bounds, - permits, disconnect cleanup, shutdown and blocked response bodies. -4. Add cache generations, observations and whole-read budgets. Use deterministic - barriers for stale-fill races. Prove timed-out SQL stops/rolls back and pool - connections return to baseline before reuse. -5. Add one tab-level fetch-SSE manager/scheduler with explicit active-query - mappings, dirty generations, abort/fencing, count/selection exceptions and - fallback. Test finite parsing across chunks, comments, UTF-8 and line splits. -6. Cover every local mutation family. Prove rollback emits no hint and peer - changes converge through periodic reads. Test filter entry/exit, page refill, - retained-not-found presentation, errors, resume and heartbeat expiry without - writes. Preserve "where applicable" for follow-ups: rescue follow-ups remain - best effort after rescue. -7. Test old-server 200-HTML/404, mixed-version routing of capabilities/stream/read, - 401/403, 429 `Retry-After`, 503, healthy SSE with failed database reads, - restarts, continuous hints, overflow/resync, slow consumers and fallback - without duplicate loops. Counts/paused selections remain unchanged until - explicitly resumed/refreshed under their rules. Include a silently hung reader, - a missed final event with continuing keepalives, interleaved resource hints, - empty/over-cap scope, scope churn, cancellation, BFCache restore and preserved - operator drafts. Independently verify the existing atomic mutation guards. -8. Benchmark PostgreSQL 17 and 18 with zero clients, current polling, opted-in SSE, - maximum admitted views and multiple admin processes. Measure p95/p99 enqueue, - claim and finalization latency, rescue progress, WAL, query QPS, permits, - database connections, cache fills and browser freshness. Validate through - supported ingress/pooler topology. - -Distinguish source/pipeline observation lag where measurable, transport activity, -API refetch latency and browser read/render freshness in diagnostic evidence. -Missing source/pipeline lag remains unknown. A healthy stream can coexist with -an unhealthy queue or stale source, and a consumer-stage metric cannot establish -universal subsecond browser freshness. Report histogram resolution and censored -top-bucket observations rather than claiming an exact delay from the last bucket. -These are measurement requirements, not additional v1 resource/metric APIs. - -Keep the implementation PR draft until deterministic correctness and -read-budget/cancellation tests pass. Default enablement requires load/deployment -evidence; missing evidence does not prevent writing an opt-in implementation. -Timer-driven SSE is not inherently cheaper than polling: measure local-mutation -freshness and total read/connection costs before default enablement. - -The earlier uniqueness-prune and DLQ downgrade findings remain separate release -gates. This stream neither causes nor repairs them. No transactional-directory -proposal is included. New metrics, retained notices/history and application -authorization need separate work. - -## Evidence boundary - -This contract adopts the actual-document review of ADR-050 at -`3dd7ee3696e706d80c4a2777d7d0642f1e3480cc`, verified blob -`154f328a7249229ad3e0a25db83827595ea95f21`. The original repository and -PostgreSQL/SSE source evidence remains linked in -[ADR-050](../adr/050-realtime-subscriptions.md#links-and-evidence). -Review agreement and documentation CI do not establish a tested subscription -implementation, SQL cancellation, load safety or deployment correctness. The -transferred review evidence paragraph was incomplete and executable model -artifacts were not supplied here; this document claims no verified model result. From aab31aeee309a3ef6c1eff02620c1efec8e6de21 Mon Sep 17 00:00:00 2001 From: Brian Thorne Date: Sat, 31 Oct 2026 22:02:12 +1300 Subject: [PATCH 08/11] docs(adr): bound client read deadline and late response handling --- docs/design/admin-observation-v1.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/design/admin-observation-v1.md b/docs/design/admin-observation-v1.md index 66bd6bdc8..ffdaf9b9b 100644 --- a/docs/design/admin-observation-v1.md +++ b/docs/design/admin-observation-v1.md @@ -136,6 +136,7 @@ or unlimited settings must not defeat the bounds. | --- | --- | | Active managed query keys | 16 per tab; no silent eviction. | | Client read concurrency | 2 per tab; one in flight and one pending marker per exact key. | +| Client request deadline | 20 seconds from dispatch through response-body decoding and protocol validation; finite and configured greater than the server whole-read deadline. | | Debounced dirty demand | 100 ms target, 500 ms maximum before becoming eligible; cooldown/backoff still applies. | | Cached refresh floor | At least advertised cache TTL/poll interval, currently at least 5 seconds. | | Uncached detail/page floor | At least 2 seconds between reads. | @@ -155,6 +156,22 @@ Repeated demand coalesces; continuous changes must neither starve results nor create an unbounded queue. The client deadline runs independently of pending network/body reads so a silent hung request is still aborted. +The client deadline is one elapsed-time budget, not reset by headers, body bytes +or retries within that attempt. On expiry, abort the request and fence its attempt +identity before accepting any late completion. It cannot publish into the client +cache/UI, clear dirty demand or renew successful-observation time after expiry; +retain one pending demand under the normal backoff. Recheck expiry before +publishing decoded data even if cancellation was ignored. The 20-second client +default leaves transport margin over the 15-second server budget, not a network +latency guarantee. Configuration must preserve a finite client budget greater +than the server budget. + +The server independently enforces its 15-second whole-read deadline from entry +to read admission (including waiting) through serialization. Client waiting/cancellation cannot extend it; +candidate passes, loader queries and retries cannot reset it. Server cache-fill +generation fences still apply: neither timeout nor late completion permits an +obsolete fill to become current. + A Rust future timeout does not prove SQL cancellation. Apply server-side limits to all candidate queries, loader transactions, repeated passes, aggregates and DLQ reads, bounded by the remaining whole-read budget. Cancellation must roll @@ -261,6 +278,8 @@ Deterministic acceptance covers: initial/empty/over-cap registration; a dirty mark during the first or later read; interleaved resource demand; old responses following filter/backend changes; cache races; SQL timeout rollback and pool recovery; slow/hung/aborted reads; continuous demand; errors/backoff/Retry-After; +client expiry with an ignored abort/late response and an invalidated late cache +fill, proving neither is published as current and the server budget is not reset; old-server HTML/404/405 and capability/read requests routed to different versions; 401/403; counts, paused selections and drafts; local rollback; peer mutations; heartbeat expiry without writes; page refill and retained-not-found presentation. From 4649c56932955416d2fee55accec277b32be5c78 Mon Sep 17 00:00:00 2001 From: Brian Thorne Date: Sun, 1 Nov 2026 07:10:22 +1300 Subject: [PATCH 09/11] docs(adr): clarify interactive read scope and rollout tradeoffs --- docs/adr/050-admin-view-observation.md | 7 ++- docs/design/admin-observation-v1.md | 72 +++++++++++++++++++------- 2 files changed, 60 insertions(+), 19 deletions(-) diff --git a/docs/adr/050-admin-view-observation.md b/docs/adr/050-admin-view-observation.md index 0fc84dbab..93e705134 100644 --- a/docs/adr/050-admin-view-observation.md +++ b/docs/adr/050-admin-view-observation.md @@ -140,6 +140,10 @@ Shared read hardening and strict adapters are independently useful follow-ups to #572. They should not wait for a streaming decision. Existing GET parsing and per-statement timeouts must not be represented as strict mixed-version reads or an end-to-end deadline; the supporting contract records the verified gaps. +Its interactive-read budgets do not replace batch-scan or DLQ bulk-mutation +policies. When strict adapters ship, upgrade notes must explain that a new UI +routed to an old replica loses filtered-view availability rather than widening +the query through a permissive GET fallback. ## Evidence required for a later transport decision @@ -150,7 +154,8 @@ peer changes; database/read amplification, connections and browser freshness; and supported ingress, authentication, expiry and reconnect costs. Name an owner and explicit, measurable retain/reject criteria for benefit, coverage and cost before starting any experiment. -This ADR invents no numeric success target or release deadline. +The transport proposer supplies the numeric targets and evaluation timeframe +from that baseline. A durable application event/completion requirement needs a different contract, not a faster version of these hints. Scoped application authorization or actual diff --git a/docs/design/admin-observation-v1.md b/docs/design/admin-observation-v1.md index ffdaf9b9b..e49776e59 100644 --- a/docs/design/admin-observation-v1.md +++ b/docs/design/admin-observation-v1.md @@ -110,12 +110,29 @@ predicates, switching a filtered request to a permissive GET, or displaying an unsupported count as zero. New-server budgets cover permitted legacy fallbacks too; they do not establish new server-side guarantees on old binaries. +When the strict adapters ship, their implementation must include upgrade notes +and release notes for this availability tradeoff: a new UI routed to an older +replica shows "filter unsupported" instead of filtered results. During a rolling +deploy, filtered views can therefore alternate between success and unsupported +as routing changes. Document that behavior and the deployment/routing mitigation; +never hide it by falling back to a wider filtered GET. These adapters are proposed, +not claimed to have shipped here. + Fixing unknown/empty GET parsing on a current server is a separate improvement from this strict mixed-version read boundary. Neither strict JSON parsing in one path nor a `/v1` name makes an older GET implementation strict. ## Errors, deadlines and shared read budgets +The admission, client/whole-read, statement and lock-wait budgets below apply to +**interactive admin reads**: lists, details, previews, counts and status/aggregate +reads. Batch-operation execution scans and DLQ bulk mutations keep their own +policy and safeguards; this contract neither subjects them to the interactive +budgets nor exempts them from all limits. Do not apply them globally to compiler +paths or pooled sessions also used for execution scans or mutations. This preserves +ADR-049's scan policy rather than making an interactive timeout fail a whole +operation. + Use structured JSON API errors: invalid versions/fields/values return 400, body-size excess 413, authorization/origin failure 401/403, admission excess 429 with `Retry-After`, and temporary overload 503. Unsupported routes, wrong @@ -132,7 +149,7 @@ The following are finite experimental read/scheduler defaults, not production capacity commitments or freshness SLOs. Configure them with sanity checks; zero or unlimited settings must not defeat the bounds. -| Bound | Initial setting | +| Interactive-read bound | Initial setting | | --- | --- | | Active managed query keys | 16 per tab; no silent eviction. | | Client read concurrency | 2 per tab; one in flight and one pending marker per exact key. | @@ -141,13 +158,19 @@ or unlimited settings must not defeat the bounds. | Cached refresh floor | At least advertised cache TTL/poll interval, currently at least 5 seconds. | | Uncached detail/page floor | At least 2 seconds between reads. | | Independent reconciliation | Nominal 30-second watchdog per active key, finite bounded jitter; expose the configured maximum. | -| Server expensive reads | 4 simultaneous operations per process. | -| Read admission | At most 16 waiters; at most 250 ms waiting; excess receives 429. | -| Whole-read deadline | 15 seconds including admission, pool acquisition, all queries/passes and serialization. | -| SQL statement limit | At most 5 seconds, also bounded by remaining whole-read deadline. | -| SQL lock wait | At most 1 second, also bounded by remaining deadline. | +| Server expensive interactive reads | 4 simultaneous operations per process. | +| Interactive-read admission | At most 16 waiters; at most 250 ms waiting; excess receives 429. | +| Interactive whole-read deadline | 15 seconds including admission, pool acquisition, all queries/passes and serialization. | +| Interactive SQL statement limit | At most 5 seconds, also bounded by remaining whole-read deadline. | +| Interactive SQL lock wait | At most 1 second, also bounded by remaining deadline. | | Repeated failure | Exponential backoff to 60 seconds; one pending demand retained. | +During incidents, uncached filtered job pages are expected to degrade first as +operators multiply demand: reads may receive 429 and the UI retains stale data. +Cached aggregate views bypass expensive-read admission only on a cache hit. +Their misses/refills compete for the same permits and waiters and can also be +rejected; aggregate views have no blanket immunity from overload. + One frontend scheduler handles periodic reads, reconciliation, focus/resume, mutation refreshes, retry and manual refresh. Managed React Query queries disable independent automatic intervals, focus fetches and reconnect fetches. Migrate @@ -166,17 +189,18 @@ default leaves transport margin over the 15-second server budget, not a network latency guarantee. Configuration must preserve a finite client budget greater than the server budget. -The server independently enforces its 15-second whole-read deadline from entry -to read admission (including waiting) through serialization. Client waiting/cancellation cannot extend it; -candidate passes, loader queries and retries cannot reset it. Server cache-fill +The server independently enforces its 15-second interactive whole-read deadline +from entry to read admission (including waiting) through serialization. Client +waiting/cancellation cannot extend it; candidate passes, loader queries and +retries cannot reset it. Server cache-fill generation fences still apply: neither timeout nor late completion permits an obsolete fill to become current. -A Rust future timeout does not prove SQL cancellation. Apply server-side limits -to all candidate queries, loader transactions, repeated passes, aggregates and -DLQ reads, bounded by the remaining whole-read budget. Cancellation must roll -back or close outstanding work before pool reuse. Do not leak read timeout -settings into mutation sessions. Cache hits may bypass expensive-read admission; +A Rust future timeout does not prove SQL cancellation. For interactive reads, +propagate server-side limits through all candidate queries, loader transactions, +repeated passes, aggregates and DLQ inspection reads, bounded by the remaining +interactive whole-read budget. Cancellation must roll back or close outstanding +work before pool reuse. Do not leak read timeout settings into mutation sessions. Cache hits may bypass expensive-read admission; each fill acquires it once. Bound obsolete fills, filter cache cardinality and pool/connection use. Do not create a database task per registered tab query. @@ -254,8 +278,10 @@ admin access safe. Each increment is independently useful with polling: -1. Harden owning reads: strict adapters/capability and response checks, API error - fallback, bounded admission, whole-read deadlines and cancellation coverage. +1. Harden interactive owning reads: strict adapters/capability and response checks, + API error fallback, bounded admission, whole-read deadlines and cancellation + coverage. Verify that batch execution scans and DLQ bulk mutations retain + their own policies. Ship the adapters' rolling-upgrade availability notes. Reuse #572's compiler; GET parsing/deadline repairs are shared follow-up work, not a transport dependency. Test empty install/read-only operation without a worker. Do not rename mutation endpoints for URI consistency. @@ -286,12 +312,20 @@ heartbeat expiry without writes; page refill and retained-not-found presentation Tests must not demand every transient transition or a stronger snapshot than the owning API provides. +Acceptance must distinguish interactive reads from execution/mutation paths: +assert the interactive admission/whole-read/statement limits on the former +without imposing them on batch scans or DLQ bulk mutations. Test uncached +filtered-page overload, aggregate cache hits versus fills, and fail-closed filtered +views when a new UI reaches an older replica; include the required upgrade notes +with the adapters' release. + Keep correctness/read-budget work separate from claims of production freshness. Distinguish source/pipeline lag where measured, transport/network activity, API latency and browser render age. Missing lag remains unknown, consumer-stage metrics cannot establish universal subsecond freshness, and the last histogram -bucket does not identify an exact delay. No numeric release/SLO commitment is -introduced. A later transport requires the ADR's separate evidence/owner decision. +bucket does not identify an exact delay. Read-budget defaults are not freshness +SLOs. Any later transport proposer supplies its numeric criteria and evaluation +timeframe through the ADR's separate evidence/owner decision. ## Dependencies and source evidence @@ -306,6 +340,8 @@ are evidence of current gaps/fixes, not a claim that this proposed contract ship starts a separate transaction using planning settings without installing that timeout. This does not establish a whole-read deadline. Fixes belong in the shared read layer/#572 follow-ups; verify the actual implementation base. + The batch-scan helpers deliberately pass no interactive timeout because a + timed-out scan fails the operation; those paths retain their separate policy. - [GET parsing](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/src/handlers/filter_params.rs) deliberately ignores unknown and empty values, including a test of that behavior. [Finding-jobs documentation](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/docs/finding-jobs.md) From 05e4c70d6cfed2e2e2149bcfd6d7faff434f50f1 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 1 Nov 2026 07:32:03 +0000 Subject: [PATCH 10/11] docs(adr): accept ADR-050 and refresh #572 source evidence Mark ADR-050 and its supporting contract accepted. The contract's source evidence now cites #572 at 15b8e3ff, which gives each interactive listing one 30-second budget and refuses unknown or empty GET parameters; the contract notes that its adapters tighten that budget to 15 s/5 s. The ADR index also shows ADR-049 as accepted, matching its status. --- docs/adr/050-admin-view-observation.md | 14 ++--- docs/adr/README.md | 4 +- docs/design/admin-observation-v1.md | 72 ++++++++++++++------------ 3 files changed, 49 insertions(+), 41 deletions(-) diff --git a/docs/adr/050-admin-view-observation.md b/docs/adr/050-admin-view-observation.md index 93e705134..8eb448b3c 100644 --- a/docs/adr/050-admin-view-observation.md +++ b/docs/adr/050-admin-view-observation.md @@ -2,9 +2,10 @@ ## Status -Proposed. V1 uses improved polling. SSE, WebSocket and inter-process notification +Accepted. V1 uses improved polling. SSE, WebSocket and inter-process notification listeners are deferred to a separate decision, not included as opt-in features. -[Draft PR #576](https://github.com/hardbyte/awa/pull/576) is documentation only. +The [supporting contract](../design/admin-observation-v1.md) describes work not +yet implemented. ## Problem and constraints @@ -137,9 +138,10 @@ independent caches and aggregate read costs. Current-state reads cannot guarante an audit trail, every completion, or an exact census during transitions. Shared read hardening and strict adapters are independently useful follow-ups to -#572. They should not wait for a streaming decision. Existing GET parsing and -per-statement timeouts must not be represented as strict mixed-version reads or -an end-to-end deadline; the supporting contract records the verified gaps. +#572. They should not wait for a streaming decision. #572 makes the current +server's GET parameters strict and gives each interactive listing one deadline, +but neither makes an older server strict; mixed-version strictness needs the +versioned adapters. Its interactive-read budgets do not replace batch-scan or DLQ bulk-mutation policies. When strict adapters ship, upgrade notes must explain that a new UI routed to an old replica loses filtered-view availability rather than widening @@ -168,7 +170,7 @@ bidirectional interaction also changes the decision boundary. - [ADR-049](049-job-filter-and-indexed-tags.md) and [PR #572](https://github.com/hardbyte/awa/pull/572): shared population semantics; [current source evidence](../design/admin-observation-v1.md#dependencies-and-source-evidence) - distinguishes remaining GET/deadline gaps from resolved DLQ/UI issues. + records what #572 already provides and what the adapters add. - [Existing cache](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/cache.rs), [admin state](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/state.rs) and [polling policy](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/frontend/src/hooks/use-poll-interval.ts): diff --git a/docs/adr/README.md b/docs/adr/README.md index 6618fcb56..1d52697c9 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -55,8 +55,8 @@ Each record preserves its status, context, decision, consequences, and alternati | 045 | [Hot-path triggers never wait on another transaction](045-hot-path-trigger-lock-contract.md) | Triggers on the canonical hot tables may write only append-only, unconstrained mark tables or the job's own unique claim; the maintenance path never truncates; the contract is an executable test and a TLA+ witness (v046, 0.6 patch). | | | 046 | [Handler-staged follow-up jobs](046-handler-staged-follow-ups.md) | A handler stages follow-up requests on its `JobContext`; they are inserted in the attempt's lease-guarded completion transaction, deduplicated by their own unique keys, and discarded on every other outcome. | Accepted | | 048 | [Retained job output](048-retained-job-output.md) | A handler may record a size-bounded JSON output for its attempt, sealed with the staging buffer and published only by the attempt's own lease-guarded transition: as the job's output on completion, or in the error entry of a failure, where only the latest entry to record one keeps it. It rides the compact completion batch, and neither the engine nor another job reads it. A kind may declare its output type, which types the API and is published in its descriptor. | Accepted | -| 049 | [One job filter, and tags as the indexed handle](049-job-filter-and-indexed-tags.md) | One `JobFilter` describes "which jobs" for listing, counting, batch preview, batch execution and the DLQ, compiled once per engine with only the predicates a filter sets; SQL narrows candidates and a batched loader settles each job's current state. Tags are the indexed handle for "jobs about X": a partial GIN index over non-empty payload tags on each queue-storage partition, maintained by Postgres for every binary, emptied by pruning, and built online by the migrator. | Proposed | -| 050 | [Admin views use invalidation and authoritative refetch](050-admin-view-observation.md) | Polling-first owning API reads, disposable invalidation and independent bounded reconciliation; push transport is deferred. | Proposed | +| 049 | [One job filter, and tags as the indexed handle](049-job-filter-and-indexed-tags.md) | One `JobFilter` describes "which jobs" for listing, counting, batch preview, batch execution and the DLQ, compiled once per engine with only the predicates a filter sets; SQL narrows candidates and a batched loader settles each job's current state. Tags are the indexed handle for "jobs about X": a partial GIN index over non-empty payload tags on each queue-storage partition, maintained by Postgres for every binary, emptied by pruning, and built online by the migrator. | Accepted | +| 050 | [Admin views use invalidation and authoritative refetch](050-admin-view-observation.md) | Polling-first owning API reads, disposable invalidation and independent bounded reconciliation; push transport is deferred. | Accepted | ## Correctness evidence diff --git a/docs/design/admin-observation-v1.md b/docs/design/admin-observation-v1.md index e49776e59..e80e2cbe9 100644 --- a/docs/design/admin-observation-v1.md +++ b/docs/design/admin-observation-v1.md @@ -1,7 +1,8 @@ # Admin observation v1: polling, strict reads and bounded reconciliation -Status: proposed supporting contract for -[ADR-050](../adr/050-admin-view-observation.md). V1 uses polling. It includes no +Status: supporting contract for the accepted +[ADR-050](../adr/050-admin-view-observation.md). It specifies work that is not +yet implemented. V1 uses polling. It includes no SSE/WebSocket endpoint, browser subscription service, worker hooks, database notification listener, outbox or CDC integration. Future transport alternatives and their evidence gate belong in the ADR, not this delivery sequence. @@ -118,9 +119,10 @@ as routing changes. Document that behavior and the deployment/routing mitigation never hide it by falling back to a wider filtered GET. These adapters are proposed, not claimed to have shipped here. -Fixing unknown/empty GET parsing on a current server is a separate improvement -from this strict mixed-version read boundary. Neither strict JSON parsing in one -path nor a `/v1` name makes an older GET implementation strict. +#572 makes the current server's GET parsing refuse unknown and empty +parameters. That is a separate improvement from this strict mixed-version read +boundary: neither strict GET parsing on a new server nor a `/v1` name makes an +older GET implementation strict. ## Errors, deadlines and shared read budgets @@ -165,6 +167,11 @@ or unlimited settings must not defeat the bounds. | Interactive SQL lock wait | At most 1 second, also bounded by remaining deadline. | | Repeated failure | Exponential backoff to 60 seconds; one pending demand retained. | +#572 ships a 30-second whole-read budget for interactive listing and counting, +passing the remaining time to each candidate query and row load as its +statement timeout. The adapters tighten that to the 15-second whole-read and +5-second statement defaults above. + During incidents, uncached filtered job pages are expected to degrade first as operators multiply demand: reads may receive 429 and the UI retains stale data. Cached aggregate views bypass expensive-read admission only on a cache hit. @@ -282,8 +289,8 @@ Each increment is independently useful with polling: API error fallback, bounded admission, whole-read deadlines and cancellation coverage. Verify that batch execution scans and DLQ bulk mutations retain their own policies. Ship the adapters' rolling-upgrade availability notes. - Reuse #572's compiler; GET parsing/deadline repairs are shared follow-up work, - not a transport dependency. Test empty install/read-only operation without a + Reuse #572's compiler, read budget and strict GET parsing; tightening the + budget to these defaults is not a transport dependency. Test empty install/read-only operation without a worker. Do not rename mutation endpoints for URI consistency. 2. Consolidate the tab scheduler and lifecycle. Register dirty demand before initial reads, disable competing automatic fetch loops, enforce per-key @@ -329,39 +336,38 @@ timeframe through the ADR's separate evidence/owner decision. ## Dependencies and source evidence -Source inspection on 2026-10-31 used #572 head -`c09b23ebed0d65614025271611c720ea584a55c9`, which includes change -`ac1e39440d89e6c0c5a5bb86a7d3af5d8db47d86`. These observations -are evidence of current gaps/fixes, not a claim that this proposed contract ships. - -- [Candidate listing](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-model/src/job_query.rs) - loops up to 32 passes with a 30-second statement timeout per candidate query. - [The loader](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-model/src/queue_storage.rs) - starts a separate transaction using planning settings without installing that - timeout. This does not establish a whole-read deadline. Fixes belong in the - shared read layer/#572 follow-ups; verify the actual implementation base. - The batch-scan helpers deliberately pass no interactive timeout because a - timed-out scan fails the operation; those paths retain their separate policy. -- [GET parsing](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/src/handlers/filter_params.rs) - deliberately ignores unknown and empty values, including a test of that - behavior. [Finding-jobs documentation](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/docs/finding-jobs.md) - gives a general unknown-field rejection promise; it must not be interpreted as - current GET strictness. Parser/documentation repair and strict mixed-version - adapters are distinct work, both useful without this ADR or any streaming. -- [DLQ handlers](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/src/handlers/dlq.rs), - [router](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/src/lib.rs) - and [API test](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/tests/admin_api_test.rs) +Source inspection on 2026-11-01 used #572 head +`15b8e3ff66282cac00b744b6d0483ce2eb4f8cea`. These observations describe what +#572 provides, not a claim that this contract ships. + +- [Candidate listing](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-model/src/job_query.rs) + gives each interactive listing or count one 30-second `ReadBudget`. Every + candidate pass and [loader](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-model/src/queue_storage.rs) + transaction runs with the remaining time as its statement timeout, and a + backstop just past the deadline bounds pool waits. Running out is a 400 asking + for a narrower filter. Batch-scan helpers pass no budget, because a timed-out + scan fails the operation. The adapters' 15-second/5-second defaults and + admission limits are not yet implemented. +- [GET parsing](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-ui/src/handlers/filter_params.rs) + refuses unknown and empty parameters on `GET /api/jobs`, `/api/jobs/count` + and `/api/dlq`, matching [finding-jobs](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/docs/finding-jobs.md). + This is current-server strictness only; a 0.6 server still ignores unknown + parameters. +- [DLQ handlers](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-ui/src/handlers/dlq.rs), + [router](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-ui/src/lib.rs) + and [API test](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-ui/tests/admin_api_test.rs) use separate retry/purge-matching paths for the full filter, legacy bodies on legacy paths and refusal of `all` with a nonempty filter. The obsolete richer- body-on-legacy-route downgrade concern is not retained as an unresolved v1 gate. The PR reports a real 0.6.9 rejection check; source inspection here is not a fresh execution of that rehearsal. -- [Search validation](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/frontend/src/components/SearchBar.tsx) +- [Search validation](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-ui/frontend/src/components/SearchBar.tsx) accepts positive whole IDs only when `Number.isSafeInteger` succeeds; - [the batch form](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/frontend/src/routes/batch-ops.tsx) + [the batch form](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-ui/frontend/src/routes/batch-ops.tsx) uses that validator. This rejects unsupported IDs; it is not full-precision - browser ID support. -- [Router](https://github.com/hardbyte/awa/blob/c09b23ebed0d65614025271611c720ea584a55c9/awa-ui/src/lib.rs) + browser ID support. The jobs and DLQ pages also hide cached rows while the + search holds a malformed term. +- [Router](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-ui/src/lib.rs) currently applies `CorsLayer::permissive()` and the SPA fallback. Access-policy claims above are implementation requirements, not assertions about a deployed origin configuration. From 0efb415ed185a739ec7fffaf89e7394c3d4fa989 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 1 Nov 2026 07:40:53 +0000 Subject: [PATCH 11/11] docs(adr): make read-deadline expiry a retryable 503 in the v1 contract Deadline expiry gets 503 with Retry-After and a read_deadline_exceeded code, so clients retry under backoff rather than treating it as an invalid filter. #572's current 400 mapping is not inherited by the adapters. --- docs/design/admin-observation-v1.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/design/admin-observation-v1.md b/docs/design/admin-observation-v1.md index e80e2cbe9..9d6a50fe5 100644 --- a/docs/design/admin-observation-v1.md +++ b/docs/design/admin-observation-v1.md @@ -140,6 +140,15 @@ body-size excess 413, authorization/origin failure 401/403, admission excess 429 with `Retry-After`, and temporary overload 503. Unsupported routes, wrong MIME and malformed envelopes are protocol errors, even with HTTP 200. +Whole-read deadline expiry is not an invalid request. Return 503 with +`Retry-After` (at least the key's refresh floor) and a structured +`read_deadline_exceeded` error code, so clients keep the request's demand +pending and retry under the normal backoff instead of treating it like a bad +filter. #572 currently reports expiry as a 400 asking for a narrower filter; the +adapters must not inherit that mapping. A filter that exceeds the deadline +repeatedly stays visible as stale with the error, rather than silently retrying +forever without telling the operator. + Network/read failures retain stale data and one pending demand, with exponential backoff/jitter capped at 60 seconds. Honor 429/503 `Retry-After` across reads to the affected backend; do not turn overload into an immediate extra poll. Stop