diff --git a/docs/adr/050-admin-view-observation.md b/docs/adr/050-admin-view-observation.md new file mode 100644 index 000000000..8eb448b3c --- /dev/null +++ b/docs/adr/050-admin-view-observation.md @@ -0,0 +1,184 @@ +# ADR-050: Admin views use invalidation and authoritative refetch + +## Status + +Accepted. V1 uses improved polling. SSE, WebSocket and inter-process notification +listeners are deferred to a separate decision, not included as opt-in features. +The [supporting contract](../design/admin-observation-v1.md) describes work not +yet implemented. + +## Problem and constraints + +Operators need current views of jobs, queues, workers, schedules, the DLQ and +administrative progress. The UI already reads those resources through HTTP and +TTL caches, but independent page timers, ambiguous filter compatibility and +in-flight cache fills make freshness and load difficult to reason about. + +The priority order is correctness, reliability and recovery first, performance +second, UX third. A fresh view must not become a dependency of enqueue, claim, +finalization or rescue. The initial consumer is the trusted admin UI. Durable +business triggers and an application-facing notification API are different +requirements. + +A common refresh scheduler, whole-read deadlines, fenced cache fills, truthful +source timestamps and strict read contracts improve the current UI without a +stream. There is no measured operator need or transport comparison that justifies +adopting SSE alongside those changes. + +## Decision and scope + +**Use disposable invalidation hints, authoritative owning-API reads and independent +bounded reconciliation, implemented with polling in v1.** A local mutation can +mark the initiating tab's views stale and invalidate that server's caches. Other +tabs and admin processes discover changes through their own paced reads. + +The five invariants are: + +1. **Read authority.** A hint only requests a read. The owning API and shared + `JobFilter` define contents; unsupported semantics fail closed. An older cache + fill cannot undo invalidation or be relabelled as newly observed state. +2. **Independent recovery.** Register active read demand and arm reconciliation + before the first read. Lost hints, errors and resumes are repaired by scheduled + reads; a dirty mark received during a read remains pending afterwards. +3. **Bounded shared demand.** Polling, reconciliation, focus, retries and mutations + use one scheduler and finite client/server budgets. Coalescing retains demand + for every resource; overload reduces freshness rather than multiplying reads. +4. **Job-path independence.** Observation adds no job-transition writes, worker + hooks, mandatory listener or event-processing engine. Reliable business work + remains durable queue jobs, not a reaction to a UI observation. +5. **Trusted administration.** Read-only still exposes sensitive admin data; + scopes cannot silently widen. UI refresh preserves selections/drafts and does + not replace atomic mutation guards or confer ownership through presence. + +V1 reuses existing read models and ADR-049's filter/compiler. It adds no new +metrics, retained notice history, tenant authorization framework or application +completion guarantee. The [polling-first v1 contract](../design/admin-observation-v1.md) +specifies startup, read protocols, deadlines, cache fencing and validation. + +Convergence means retained state returned by ordinary APIs after changes settle +and reads succeed. It is not observation of every transition or a stronger +cross-table snapshot. Composite reads can miss claims or deferred-to-ready +handoffs; Repeatable Read does not make ring truncation MVCC-safe. A completed +job pruned before a read may never be observed. Follow-up jobs in +[ADR-029](029-transactional-followup-jobs.md) and +[ADR-046](046-handler-staged-follow-ups.md) remain the durable mechanism where +applicable; best-effort rescue follow-ups are not strengthened. + +## Alternatives considered + +### Improved polling — chosen + +One scheduler, strict read boundaries, bounded work and truthful stale state +address current correctness and operational problems without persistent browser +connections. Each increment is useful without a transport project. Reads remain +sampled and worker/peer changes wait for the configured cadence. We accept that +tradeoff rather than fund unproven low-latency delivery. + +### SSE or WebSocket — deferred + +A stream could invalidate other views immediately after a committed local admin +mutation. A process-local hub reaches only viewers connected to that process. +With independent uniform routing over N processes, a particular viewer shares +the mutation's process with probability 1/N; other routing/topologies differ. +Worker changes gain no speed from a timer-driven stream, and adding such a stream +does not inherently reduce database reads. + +SSE fits one-way HTTP hints; WebSocket supplies bidirectionality and subscription +changes on an existing connection. Both require bounded parsing/state, reconnect, +expiry, ingress/authentication handling and browser lifecycle work. Those costs +are not justified by the observation invariants, which polling already satisfies. +There is no v1 stream endpoint, descriptor protocol, parser or stream budget. +A future hint descriptor should carry only an opaque key, type and necessary job +identity; full filters belong in authoritative reads unless measured filtered +routing earns the extra complexity and privacy exposure. + +### Admin-to-admin PostgreSQL hints — optional future optimization + +A dedicated listener per admin process could invalidate peer caches after admin +mutations, without touching worker/job-execution writes. It would not push new +data to browsers: they still wait for polling. One replica already has local +invalidation; two replicas need evidence that stale-cache windows and mutation +frequency justify dedicated sessions and reconnect handling. A listener needs a +direct/session-pooled connection; transaction pooling cannot preserve its session +registration. We do not select this optimization for v1. + +`NOTIFY` is delivered after its transaction commits, but queue exhaustion can +fail that commit. To isolate hint failure from the authoritative mutation, a +future design would use a separate bounded best-effort post-commit attempt. +Crash/loss is acceptable because reconciliation remains independent. Commit +`LISTEN` before the initial cache resync and resync on reconnect. Payloads are +visible to database users, so use coarse nonsecret categories, never filters, +args or outputs. No listener is required for correctness. + +### Pushed snapshots or row deltas + +Shared aggregate snapshots might save refetches. Filtered snapshots/deltas need +membership, population entry/exit, deletion and page-refill semantics in a second +view model. Current-state refetch keeps those responsibilities in existing APIs. +Consider pushed aggregates only if measured read cost makes that complexity useful. + +### Durable event log/outbox or logical decoding + +A log/outbox can support retained events and replay, but needs complete writer +coverage, commit-safe ordering, idempotence, retention and gap recovery. A sequence +allocated before commit is not a replay watermark. CDC additionally requires +slot/privilege/failover operations and a semantic adapter over leases, cursors, +compact completions and pruning. Those can serve historical/event consumers; +they impose unnecessary dependencies for sampled admin views. + +## Consequences + +Operators get bounded, recoverable views with explicit stale state. Missed hints +cannot leave a retained job's detail stale indefinitely after successful polling +resumes. Late fills and overlapping refreshes do not silently reverse invalidation. + +Freshness remains sampled. The initiating tab can refresh after its own mutation; +other clients are not promised immediate invalidation. Separate processes have +independent caches and aggregate read costs. Current-state reads cannot guarantee +an audit trail, every completion, or an exact census during transitions. + +Shared read hardening and strict adapters are independently useful follow-ups to +#572. They should not wait for a streaming decision. #572 makes the current +server's GET parameters strict and gives each interactive listing one deadline, +but neither makes an older server strict; mixed-version strictness needs the +versioned adapters. +Its interactive-read budgets do not replace batch-scan or DLQ bulk-mutation +policies. When strict adapters ship, upgrade notes must explain that a new UI +routed to an old replica loses filtered-view availability rather than widening +the query through a permissive GET fallback. + +## Evidence required for a later transport decision + +Reopen transport selection only with an unmet operator freshness requirement or +a concrete push consumer, a measured improved-polling baseline and the intended +deployment topology. Compare coverage of local mutations, worker changes and +peer changes; database/read amplification, connections and browser freshness; +and supported ingress, authentication, expiry and reconnect costs. Name an owner +and explicit, measurable retain/reject criteria for benefit, coverage and cost +before starting any experiment. +The transport proposer supplies the numeric targets and evaluation timeframe +from that baseline. + +A durable application event/completion requirement needs a different contract, +not a faster version of these hints. Scoped application authorization or actual +bidirectional interaction also changes the decision boundary. + +## Links and evidence + +- [Polling-first v1 contract](../design/admin-observation-v1.md): normative read, + scheduler, cache and access requirements with incremental delivery gates. +- [ADR-049](049-job-filter-and-indexed-tags.md) and + [PR #572](https://github.com/hardbyte/awa/pull/572): shared population semantics; + [current source evidence](../design/admin-observation-v1.md#dependencies-and-source-evidence) + records what #572 already provides and what the adapters add. +- [Existing cache](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/cache.rs), + [admin state](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/src/state.rs) + and [polling policy](https://github.com/hardbyte/awa/blob/f644f321ad110c6436382b64bdc0d935dcdeba5a/awa-ui/frontend/src/hooks/use-poll-interval.ts): + the baseline, not an SSE performance comparison. +- [PostgreSQL NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) + and [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html): + transaction failure, payload visibility and registration behavior for the + deferred admin-cache optimization. +- [ADR-041](041-rolling-upgrade-policy.md) and + [ADR-045](045-hot-path-trigger-lock-contract.md): rolling-upgrade and hot-path + constraints remain in force. diff --git a/docs/adr/README.md b/docs/adr/README.md index 8934179c4..1d52697c9 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -55,7 +55,8 @@ Each record preserves its status, context, decision, consequences, and alternati | 045 | [Hot-path triggers never wait on another transaction](045-hot-path-trigger-lock-contract.md) | Triggers on the canonical hot tables may write only append-only, unconstrained mark tables or the job's own unique claim; the maintenance path never truncates; the contract is an executable test and a TLA+ witness (v046, 0.6 patch). | | | 046 | [Handler-staged follow-up jobs](046-handler-staged-follow-ups.md) | A handler stages follow-up requests on its `JobContext`; they are inserted in the attempt's lease-guarded completion transaction, deduplicated by their own unique keys, and discarded on every other outcome. | Accepted | | 048 | [Retained job output](048-retained-job-output.md) | A handler may record a size-bounded JSON output for its attempt, sealed with the staging buffer and published only by the attempt's own lease-guarded transition: as the job's output on completion, or in the error entry of a failure, where only the latest entry to record one keeps it. It rides the compact completion batch, and neither the engine nor another job reads it. A kind may declare its output type, which types the API and is published in its descriptor. | Accepted | -| 049 | [One job filter, and tags as the indexed handle](049-job-filter-and-indexed-tags.md) | One `JobFilter` describes "which jobs" for listing, counting, batch preview, batch execution and the DLQ, compiled once per engine with only the predicates a filter sets; SQL narrows candidates and a batched loader settles each job's current state. Tags are the indexed handle for "jobs about X": a partial GIN index over non-empty payload tags on each queue-storage partition, maintained by Postgres for every binary, emptied by pruning, and built online by the migrator. | Proposed | +| 049 | [One job filter, and tags as the indexed handle](049-job-filter-and-indexed-tags.md) | One `JobFilter` describes "which jobs" for listing, counting, batch preview, batch execution and the DLQ, compiled once per engine with only the predicates a filter sets; SQL narrows candidates and a batched loader settles each job's current state. Tags are the indexed handle for "jobs about X": a partial GIN index over non-empty payload tags on each queue-storage partition, maintained by Postgres for every binary, emptied by pruning, and built online by the migrator. | Accepted | +| 050 | [Admin views use invalidation and authoritative refetch](050-admin-view-observation.md) | Polling-first owning API reads, disposable invalidation and independent bounded reconciliation; push transport is deferred. | Accepted | ## Correctness evidence diff --git a/docs/design/admin-observation-v1.md b/docs/design/admin-observation-v1.md new file mode 100644 index 000000000..9d6a50fe5 --- /dev/null +++ b/docs/design/admin-observation-v1.md @@ -0,0 +1,385 @@ +# Admin observation v1: polling, strict reads and bounded reconciliation + +Status: supporting contract for the accepted +[ADR-050](../adr/050-admin-view-observation.md). It specifies work that is not +yet implemented. V1 uses polling. It includes no +SSE/WebSocket endpoint, browser subscription service, worker hooks, database +notification listener, outbox or CDC integration. Future transport alternatives +and their evidence gate belong in the ADR, not this delivery sequence. + +## Owning reads and startup + +The owning APIs remain authoritative for jobs, queues, workers, DLQ, schedules, +operator progress, storage and existing statistics. Reuse their current meanings, +accuracy and retention. Do not invent metrics or historical notices. Catalog/kinds +keep their existing read behavior; `/storage?history=true` does not supply +server-retained history. Application consumers and tenant authorization are +outside v1. + +For each visible tab, the automatic-read startup sequence below excludes +paused views and manually refreshed counts. Their explicit reads still share +the budgets but do not acquire an automatic watchdog. + +1. Establish the backend identity and supported read protocols. Capabilities are + advisory; verify the actual response to every request. +2. Register active exact query keys with the one tab-level scheduler, mark each + initially dirty and arm its independent reconciliation timer **before** + dispatching the first bounded read. A key includes backend, read scope, full + normalized filter and page. There is no stream registration, `ready` event or + snapshot cursor to wait for. +3. Read through the shared client concurrency/cooldown policy. Capture the key's + dirty generation at request start. A response clears only that captured + demand; a mutation/focus mark arriving during the read leaves one pending + refresh. Successful data may render while still marked dirty/stale. +4. Fence old filter/backend identities and late responses. A filter or active-view + change cancels/retires its old key before registering the replacement. Retain + already displayed data as stale where appropriate rather than claiming a new + successful observation. + +No active keys means no resource reads or resource timers. Enforce the active-key +cap before registration; reject excess demand visibly rather than silently +truncating or creating another scheduler. One pending marker per key prevents +interleaved demand for different resources from overwriting each other. + +Polling and reconciliation use one request-dispatch path. The periodic refresh +cadence may renew normal views sooner than the independent watchdog; neither a +local dirty hint nor a failed request postpones the watchdog. Successful reads +renew the observation. The watchdog marks demand through the same scheduler, +never launches a second request loop. Overload/cooldown/backoff still apply, so +these are scheduling bounds rather than a promised maximum data age. + +## Filter and protocol boundary + +Use the actual ADR-049 `JobFilter` compiler, deserializer **and `validate()`**. +Fields intersect, states are any-of, tags are all-of, and creation bounds refer +to original insertion. Page/window semantics remain separate from the population. +Refetching a page removes jobs that left and fills vacancies from current matches; +there is no stable multi-page snapshot or guarantee of every transition. + +Add thin strict JSON read adapters: + +- `POST /api/v1/jobs/query` +- `POST /api/v1/dlq/query` +- `POST /api/v1/jobs/count` + +These paths are an additive, explicitly versioned **read contract**, alongside +Awa's existing unversioned `/api/*` endpoints. They do not rename or version all +existing APIs. In particular, retain `/api/dlq/retry-matching`, +`/api/dlq/purge-matching` and legacy mutation routes; cosmetic consistency does +not justify another mutation protocol change. + +A version in the URI or JSON is not proof of strict behavior. Enforce exact +version agreement and reject unknown envelope/filter/page fields, using strict +page wrappers where needed. Reuse existing response conversion and model logic; +do not create a UI-side compiler. Query request: + +```json +{"version":1,"filter":{},"page":{}} +``` + +Count requests omit `page`. Apply shared filter bounds and a 64 KiB request-body +limit while receiving/parsing, before allocation can grow without bound. Existing +`JobFilter` and payload ID representations remain unchanged; the current browser +rejects IDs outside JavaScript's safe-integer range rather than rounding them. +Do not claim a new arbitrary-precision or decimal-string ID protocol here. + +Response envelope: + +```json +{ + "version": 1, + "data": "", + "observation": { + "observed_at": "", + "served_at": "", + "cache_age_ms": 0 + } +} +``` + +The example strings stand for real payloads/timestamps. Add optional structured +`job_filter_protocols` capability metadata; absent support is unsupported, never +inferred from semver. Do not advertise a `subscriptions` transport capability. +Capabilities can come from a different server version than the next read, so +check status, JSON MIME, complete envelope and exact response version every time. +Unknown `/api/*` paths on the new router return API errors, not SPA HTML. + +An older server may return 200 HTML or 404/405 for the route. That is unsupported, +not an empty result. Permit unfiltered legacy lists/details through the same +scheduler; nonempty filters remain unsupported. Never retry by dropping +predicates, switching a filtered request to a permissive GET, or displaying an +unsupported count as zero. New-server budgets cover permitted legacy fallbacks +too; they do not establish new server-side guarantees on old binaries. + +When the strict adapters ship, their implementation must include upgrade notes +and release notes for this availability tradeoff: a new UI routed to an older +replica shows "filter unsupported" instead of filtered results. During a rolling +deploy, filtered views can therefore alternate between success and unsupported +as routing changes. Document that behavior and the deployment/routing mitigation; +never hide it by falling back to a wider filtered GET. These adapters are proposed, +not claimed to have shipped here. + +#572 makes the current server's GET parsing refuse unknown and empty +parameters. That is a separate improvement from this strict mixed-version read +boundary: neither strict GET parsing on a new server nor a `/v1` name makes an +older GET implementation strict. + +## Errors, deadlines and shared read budgets + +The admission, client/whole-read, statement and lock-wait budgets below apply to +**interactive admin reads**: lists, details, previews, counts and status/aggregate +reads. Batch-operation execution scans and DLQ bulk mutations keep their own +policy and safeguards; this contract neither subjects them to the interactive +budgets nor exempts them from all limits. Do not apply them globally to compiler +paths or pooled sessions also used for execution scans or mutations. This preserves +ADR-049's scan policy rather than making an interactive timeout fail a whole +operation. + +Use structured JSON API errors: invalid versions/fields/values return 400, +body-size excess 413, authorization/origin failure 401/403, admission excess +429 with `Retry-After`, and temporary overload 503. Unsupported routes, wrong +MIME and malformed envelopes are protocol errors, even with HTTP 200. + +Whole-read deadline expiry is not an invalid request. Return 503 with +`Retry-After` (at least the key's refresh floor) and a structured +`read_deadline_exceeded` error code, so clients keep the request's demand +pending and retry under the normal backoff instead of treating it like a bad +filter. #572 currently reports expiry as a 400 asking for a narrower filter; the +adapters must not inherit that mapping. A filter that exceeds the deadline +repeatedly stays visible as stale with the error, rather than silently retrying +forever without telling the operator. + +Network/read failures retain stale data and one pending demand, with exponential +backoff/jitter capped at 60 seconds. Honor 429/503 `Retry-After` across reads to +the affected backend; do not turn overload into an immediate extra poll. Stop +401/403 retries until credentials/user action change. Invalid filters/protocols +show an error instead of an automatic retry loop. Reprobe absent read capability +no more often than every 60 seconds unless explicitly requested. + +The following are finite experimental read/scheduler defaults, not production +capacity commitments or freshness SLOs. Configure them with sanity checks; zero +or unlimited settings must not defeat the bounds. + +| Interactive-read bound | Initial setting | +| --- | --- | +| Active managed query keys | 16 per tab; no silent eviction. | +| Client read concurrency | 2 per tab; one in flight and one pending marker per exact key. | +| Client request deadline | 20 seconds from dispatch through response-body decoding and protocol validation; finite and configured greater than the server whole-read deadline. | +| Debounced dirty demand | 100 ms target, 500 ms maximum before becoming eligible; cooldown/backoff still applies. | +| Cached refresh floor | At least advertised cache TTL/poll interval, currently at least 5 seconds. | +| Uncached detail/page floor | At least 2 seconds between reads. | +| Independent reconciliation | Nominal 30-second watchdog per active key, finite bounded jitter; expose the configured maximum. | +| Server expensive interactive reads | 4 simultaneous operations per process. | +| Interactive-read admission | At most 16 waiters; at most 250 ms waiting; excess receives 429. | +| Interactive whole-read deadline | 15 seconds including admission, pool acquisition, all queries/passes and serialization. | +| Interactive SQL statement limit | At most 5 seconds, also bounded by remaining whole-read deadline. | +| Interactive SQL lock wait | At most 1 second, also bounded by remaining deadline. | +| Repeated failure | Exponential backoff to 60 seconds; one pending demand retained. | + +#572 ships a 30-second whole-read budget for interactive listing and counting, +passing the remaining time to each candidate query and row load as its +statement timeout. The adapters tighten that to the 15-second whole-read and +5-second statement defaults above. + +During incidents, uncached filtered job pages are expected to degrade first as +operators multiply demand: reads may receive 429 and the UI retains stale data. +Cached aggregate views bypass expensive-read admission only on a cache hit. +Their misses/refills compete for the same permits and waiters and can also be +rejected; aggregate views have no blanket immunity from overload. + +One frontend scheduler handles periodic reads, reconciliation, focus/resume, +mutation refreshes, retry and manual refresh. Managed React Query queries disable +independent automatic intervals, focus fetches and reconnect fetches. Migrate +existing two-second job-detail/operation intervals into this scheduler explicitly. +Repeated demand coalesces; continuous changes must neither starve results nor +create an unbounded queue. The client deadline runs independently of pending +network/body reads so a silent hung request is still aborted. + +The client deadline is one elapsed-time budget, not reset by headers, body bytes +or retries within that attempt. On expiry, abort the request and fence its attempt +identity before accepting any late completion. It cannot publish into the client +cache/UI, clear dirty demand or renew successful-observation time after expiry; +retain one pending demand under the normal backoff. Recheck expiry before +publishing decoded data even if cancellation was ignored. The 20-second client +default leaves transport margin over the 15-second server budget, not a network +latency guarantee. Configuration must preserve a finite client budget greater +than the server budget. + +The server independently enforces its 15-second interactive whole-read deadline +from entry to read admission (including waiting) through serialization. Client +waiting/cancellation cannot extend it; candidate passes, loader queries and +retries cannot reset it. Server cache-fill +generation fences still apply: neither timeout nor late completion permits an +obsolete fill to become current. + +A Rust future timeout does not prove SQL cancellation. For interactive reads, +propagate server-side limits through all candidate queries, loader transactions, +repeated passes, aggregates and DLQ inspection reads, bounded by the remaining +interactive whole-read budget. Cancellation must roll back or close outstanding +work before pool reuse. Do not leak read timeout settings into mutation sessions. Cache hits may bypass expensive-read admission; +each fill acquires it once. Bound obsolete fills, filter cache cardinality and +pool/connection use. Do not create a database task per registered tab query. + +## Cache fencing and source observations + +Use one local post-commit invalidation helper across admin mutation families, +including DLQ retry/purge/move and batch cancellation. After the authoritative +mutation commits, advance the cache generation, retire its old cache entries, +and mark affected server read state stale. Rolled-back mutations invalidate +nothing. The initiating browser marks its own affected keys dirty after a +successful action; other browsers/processes receive no v1 push and refresh on +schedule. Broader local invalidation is acceptable until measured routing is +worthwhile. + +A fill captures a bounded monotonic generation before starting. It may +publish/return as current only if the generation still matches at completion. +Requests after invalidation cannot join an obsolete fill. Use generation-keyed +caches with bounded obsolete work or an explicit synchronized check, without +holding a cache lock across a query. Discard/retry obsolete completions only +within remaining read budget; never loop indefinitely or relabel old data fresh. +Moka `invalidate_all()` alone is not proof of this invariant. + +Client dirty generations and server cache generations solve different races; +both are required. `observed_at` is source-fetch start (a conservative start for +composite reads), `served_at` is response time and `cache_age_ms` is cached +observation age. Preserve source time on cache hits. Existing reads can carry +common metadata/headers without changing payload meaning; strict adapters use +the envelope above. A response served now is not necessarily a new observation. + +Each admin process owns its caches. Worker and peer mutations are covered by TTL +and periodic authoritative reads; a local generation does not create a global +snapshot. Heartbeat evidence determines worker liveness, never browser activity. +Replica freshness/lag is unknown unless actually measured. + +## UI lifecycle, domain guards and access + +Hidden tabs pause managed reads and timers. `pagehide`, including BFCache entry, +cancels them; persisted `pageshow` retires old request identities and starts +exactly one scheduler with one paced refresh if visible and nonempty. Cancelled +reads do not clear pending demand. Filter/view churn releases registrations and +obsolete work without losing unrelated keys' demand. + +Preserve selected IDs and unsaved operator form/dialog values during refresh. +Jobs with an active selection remain paused and stale until explicitly resumed; +manual refresh re-enters the same scheduler. Filtered job-match counts are +excluded from periodic/watchdog/focus refresh and prefix invalidation: mark them +stale and fetch only on filter change, explicit refresh or the specified debounced +operator action. Counts remain capped under ADR-049. Selection/draft state never +replaces server-side eligibility, version/lease/fencing and idempotency guards. +Viewing/presence does not confer ownership; no viewer membership protocol is added. + +This is the existing trusted-admin surface, protected by ingress/private network +controls. `--read-only` disables mutations, not sensitive reads. Authenticate and +authorize each read under that boundary; do not create longer-lived access via +client cache state. There is no new tenant authorization model. + +The inspected router currently applies permissive CORS and a SPA fallback. Those +are current observations, not verification of configured origins in a deployment. +New strict JSON POST adapters must explicitly reject a present Origin unless it +matches a configured public admin origin; do not derive trust from arbitrary +forwarded-host headers. Isolate their route policy so outer permissive CORS +cannot undo it. No wildcard credentialed CORS. Browser fetch uses +`credentials: "same-origin"` and `cache: "no-store"`; reject cross-origin redirects +and login HTML as protocol data. Origin-less non-browser requests still require +ingress/private-network trust. Any later credentialed streaming proposal must +resolve/tighten this boundary before exposure, not inherit permissive CORS. + +Never put credentials, args, outputs or secrets in URLs or observation logs. +Strict reads carry filters in bodies. Existing shareable search URLs may contain +documented nonsecret opaque values; no raw filter descriptors belong in automatic +logs. An origin check does not authorize a filter or make public unauthenticated +admin access safe. + +## Incremental delivery and acceptance + +Each increment is independently useful with polling: + +1. Harden interactive owning reads: strict adapters/capability and response checks, + API error fallback, bounded admission, whole-read deadlines and cancellation + coverage. Verify that batch execution scans and DLQ bulk mutations retain + their own policies. Ship the adapters' rolling-upgrade availability notes. + Reuse #572's compiler, read budget and strict GET parsing; tightening the + budget to these defaults is not a transport dependency. Test empty install/read-only operation without a + worker. Do not rename mutation endpoints for URI consistency. +2. Consolidate the tab scheduler and lifecycle. Register dirty demand before + initial reads, disable competing automatic fetch loops, enforce per-key + generations/concurrency/cooldowns/backoff and preserve paused selections/counts. +3. Add generation-fenced caches, conservative source metadata and complete local + post-commit invalidation. Deterministic barriers must reproduce an old fill + completing after invalidation and prove it cannot become current. +4. Complete independent reconciliation and failure recovery through that same + scheduler. Prove convergence after lost local marks and peer/worker changes, + request failure, scope changes, hidden/resume and BFCache restoration. No + listener or transport handshake is a prerequisite for any read. +5. Measure the improved-polling baseline on PostgreSQL 17 and 18, with zero, + ordinary and maximum admitted views and multiple admin processes. Check job + enqueue/claim/finalization p95/p99, rescue progress, WAL, query QPS, pool counts, + cache fills and browser read/render freshness under supported ingress/poolers. + +Deterministic acceptance covers: initial/empty/over-cap registration; a dirty +mark during the first or later read; interleaved resource demand; old responses +following filter/backend changes; cache races; SQL timeout rollback and pool +recovery; slow/hung/aborted reads; continuous demand; errors/backoff/Retry-After; +client expiry with an ignored abort/late response and an invalidated late cache +fill, proving neither is published as current and the server budget is not reset; +old-server HTML/404/405 and capability/read requests routed to different versions; +401/403; counts, paused selections and drafts; local rollback; peer mutations; +heartbeat expiry without writes; page refill and retained-not-found presentation. +Tests must not demand every transient transition or a stronger snapshot than +the owning API provides. + +Acceptance must distinguish interactive reads from execution/mutation paths: +assert the interactive admission/whole-read/statement limits on the former +without imposing them on batch scans or DLQ bulk mutations. Test uncached +filtered-page overload, aggregate cache hits versus fills, and fail-closed filtered +views when a new UI reaches an older replica; include the required upgrade notes +with the adapters' release. + +Keep correctness/read-budget work separate from claims of production freshness. +Distinguish source/pipeline lag where measured, transport/network activity, API +latency and browser render age. Missing lag remains unknown, consumer-stage +metrics cannot establish universal subsecond freshness, and the last histogram +bucket does not identify an exact delay. Read-budget defaults are not freshness +SLOs. Any later transport proposer supplies its numeric criteria and evaluation +timeframe through the ADR's separate evidence/owner decision. + +## Dependencies and source evidence + +Source inspection on 2026-11-01 used #572 head +`15b8e3ff66282cac00b744b6d0483ce2eb4f8cea`. These observations describe what +#572 provides, not a claim that this contract ships. + +- [Candidate listing](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-model/src/job_query.rs) + gives each interactive listing or count one 30-second `ReadBudget`. Every + candidate pass and [loader](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-model/src/queue_storage.rs) + transaction runs with the remaining time as its statement timeout, and a + backstop just past the deadline bounds pool waits. Running out is a 400 asking + for a narrower filter. Batch-scan helpers pass no budget, because a timed-out + scan fails the operation. The adapters' 15-second/5-second defaults and + admission limits are not yet implemented. +- [GET parsing](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-ui/src/handlers/filter_params.rs) + refuses unknown and empty parameters on `GET /api/jobs`, `/api/jobs/count` + and `/api/dlq`, matching [finding-jobs](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/docs/finding-jobs.md). + This is current-server strictness only; a 0.6 server still ignores unknown + parameters. +- [DLQ handlers](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-ui/src/handlers/dlq.rs), + [router](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-ui/src/lib.rs) + and [API test](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-ui/tests/admin_api_test.rs) + use separate retry/purge-matching paths for the full filter, legacy bodies on + legacy paths and refusal of `all` with a nonempty filter. The obsolete richer- + body-on-legacy-route downgrade concern is not retained as an unresolved v1 gate. + The PR reports a real 0.6.9 rejection check; source inspection here is not a + fresh execution of that rehearsal. +- [Search validation](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-ui/frontend/src/components/SearchBar.tsx) + accepts positive whole IDs only when `Number.isSafeInteger` succeeds; + [the batch form](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-ui/frontend/src/routes/batch-ops.tsx) + uses that validator. This rejects unsupported IDs; it is not full-precision + browser ID support. The jobs and DLQ pages also hide cached rows while the + search holds a malformed term. +- [Router](https://github.com/hardbyte/awa/blob/15b8e3ff66282cac00b744b6d0483ce2eb4f8cea/awa-ui/src/lib.rs) + currently applies `CorsLayer::permissive()` and the SPA fallback. Access-policy + claims above are implementation requirements, not assertions about a deployed + origin configuration. + +Other storage/uniqueness release concerns were not revalidated here; this +contract neither clears nor replaces their existing release gates.