You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The audit implementation and review fixes are committed and pushed in draft PR #136, currently at e0e5b47. The PR is unmerged and unreleased; v0.3.0 remains the published version. The 22 original implementation/planning steps below are complete. Review added eligible retention indexing/cursors, active-queue cleanup filtering, custom retry-policy precedence, strict active-attempt heartbeats and related regressions. PostgreSQL upgrades use additive V7–V11 migrations; Redis upgrades use the offline/resumable format-2 migration. Stop all old workers/producers, back up and recompile; mixed versions are unsupported. See docs/compatibility.md.
Current validation
./gradlew productionCheck -PdependencyScanRequired=true --max-workers=2 --no-build-cache passed in 10m 19s: 1,669 JUnit cases, zero failures/errors/skips, including 173 PostgreSQL, 413 Redis and 32 fixed-soak cases. The gate also passed 41 UI tests, five browser scenarios, all three short correctness simulations, both process-separated nudge simulations, required OSV/npm scans, the example, Javadoc and artifact inspection. All 14 binary JARs were also inspected.
The final operations-guide-only clarification passed ./gradlew check in 4s. Runtime JAR checksums are identical to the full production gate; the source-manifest record identifies that documentation-only delta.
Permanent regressions cover lost claim acknowledgement, exact active-attempt heartbeat matching, active-queue counter availability during idle cleanup, and retention deletion after the one-minute metadata grace. All 33 review threads received an implementation or reasoned response and were resolved.
Endurance and remaining performance qualification
A new frozen campaign started on e0e5b47, using dedicated PostgreSQL 18 and standalone Redis 7.4 containers/volumes. PostgreSQL preflight passed with 3001 completed jobs through its planned pause and restart, plus a passing independent datastore audit; Redis preflight completed; see the audit correction below. The backends run sequentially, each with 30-minute mixed and retention baselines, 12-hour mixed and retention runs, and 30-minute retry/check-in/nudge phases. The rate is 25 jobs/second (long-running deliberately 1/second with one-quarter planned timeouts). Long phases include node churn every ten minutes and 20-second datastore pauses at hours 2, 6 and 10; PostgreSQL also restarts at hour 8. Planned duration is approximately 53 hours plus startup, drain and reporting. Endurance/performance/stability results remain pending.
Artifacts and the supervisor are under .local-reference/qualification/20260909T2225Z; 43 runtime JAR checksums, exact candidate, environment/settings, source manifests and gate reports are archived. The combined five-minute PR/soak monitor is active. Local PR branch/worktree cleanup occurs only after confirmed merge, preserving unrelated work and qualification evidence.
The earlier 0fc9881 campaign was superseded and stopped gracefully. Its five completed preflight/baseline phases passed correctness/drain for that old candidate, and its Redis retention baseline is interrupted evidence. Old runs and volumes are preserved, and none qualifies the reviewed candidate.
The indexed pending-retry / large-FAILED-population benchmark is an explicit deferred performance follow-up: the 30-second inter-pass pause and bounded recovery pages do not establish efficient scanning through a very large retained 30-day FAILED backlog. Add that populated-store measurement before claiming capacity for that shape; consider a PostgreSQL retry scalar/index and Redis ordered pending-retry index if the measured budget is exceeded. Hours-scale Redis 8.6/Sentinel/Cluster qualification remains separate. Keep this issue open for review, merge and final qualification.
22. Full validation and separate PostgreSQL/Redis long-running soak plans
Separate long-running soak plans (planned, not executed)
The complete runbook is prepared locally in docs/soak-plan-1.0.md. Use dedicated disposable datastores: harness startup resets Threadmill data. Freeze the revision plus patch, resolved image digests, configuration and resource limits after the release gate. Run PostgreSQL and Redis separately on fixed hardware.
For each backend, first record a 30-minute baseline without node churn. Then run 12 hours mixed workload and 12 hours retention churn, sequentially, with one producer, three nodes, eight workers/node and an initial offered rate of 50 jobs/s. Choose the sustainable rate from the baseline and keep it fixed. Each phase needs a distinct output path and final datastore snapshot.
Repeat each command with -Pscenario=retention-churn and distinct retention run IDs/output directories. That deterministic sequence recipe uses 8 KiB payloads, fresh concurrency keys, queue families, workflows, first-attempt retries, duplicate producer requests, ten-second retention and five-second dedup TTL. The existing mixed workload has no exposed seed; retain its full trace. Do not run concurrent Gradle builds against the active harness classpath.
PostgreSQL 18: durable fsync/commit settings, normal autovacuum; pause the dedicated server for 20 seconds at hours 2/6/10, and restart it normally with volumes retained at hour 8. Record waits, dead/live tuples, vacuum, WAL, table/index sizes and exact state/queue counter reconciliation after drain.
Redis: 7.4 first, then the newer supported line (bounded tests also cover 8.6); AOF and explicit fsync policy, noeviction, memory headroom, the same pause schedule. Record independent-client latency, SLOWLOG, memory/RSS, AOF/rewrite behavior, commands, metadata cardinality and index reconciliation. Add separate eight-hour Sentinel and Cluster retention phases with real failure domains, controlled handovers and Cluster slot migration; see Add real Redis Sentinel, Cluster, authentication, and TLS tests #98. Quantify asynchronous acknowledged-write loss in a distinct abrupt-failure experiment.
Add separate 30-minute retry-storm, long-running/check-in and nudge-pump phases for both backends.
Retain progress, raw traces, latency samples, metrics, invariants, system/datastore samples, fault timeline and final reports. Monitoring includes operation p50/p95/p99 over the recent 4,096 calls, actual retention/group deletions and heap. Full-run percentile reporting still uses memory proportional to attempts; report generation is part of qualification.
Exit criteria: zero definite invariant violations or unexplained stuck/failed jobs; complete drain and reconciled counters; at least 95% of offered throughput outside faults/recovery; stable-window p95 within 20% and p99 within 2x of the same-host baseline; no growing backlog/maintenance lag; retained jobs and metadata plateau; no unexplained heap/RSS/disk/index growth; current monitoring and zero Redis evictions. Record every fault's recovery time. Investigate any breach without changing thresholds afterward. Redis data loss remains a failed zero-loss qualification even when caused by deployment replication policy, and requires an explicit deployment-owner decision.
Remaining before 1.0: review and merge the uncommitted changes, run and assess these endurance phases on the frozen candidate, then complete the release/tag workflow. The successful bounded gate alone is not endurance or production-capacity certification.
Threadmill 1.0 readiness audit
Audited 9 September 2026. Repository: hemju/threadmill. Revision: 0af7e0ac22f36f255e00610c93aac566ba0dfe87, also tagged v0.3.0. The findings below describe the original v0.3.0 audit before implementation; the current status and fresh validation are recorded above.
Assessment
I would not release this revision as 1.0. The architecture is suitable for the product, but executable probes found violations of exclusive execution, claim liveness, workflow preservation, retry policy, and finalization. These deserve priority over additional features or general refactoring.
The strongest parts are the explicit state machine, separation of core and adapters, persisted version discipline, snapshot serialization, shared store contracts, store-backed maintenance leadership, additive PostgreSQL migrations, scoped dashboard permissions/redaction, and substantial fault-oriented tests and simulation tooling. Preserve these foundations.
The largest recurring weakness is the interaction between individually reasonable mechanisms: bounded candidate windows and global ordering; FAILED persistence and later retry hooks; node-wide heartbeats and abandoned finalizers; accepted payload size and subsequent lifecycle growth. The tests cover many individual regressions but need stronger coverage of these combinations.
Recommended release scope: correctness and durability hardening, explicit supported-version and security contracts, sustainable maintenance capacity, and fresh production validation. Native image, additional framework adapters, more scheduling features, and a dashboard redesign can wait.
Scope and evidence
The repository inventory contains 489 tracked files, including 323 Java files, 15 Lua resources, six SQL migrations, and 52 Markdown files. The review covered the Java model, engine, scheduling, stores, Spring integration, dashboard API/UI, metrics/tracing, SQL schema and hot queries, all 15 Lua resources, relevant inline Lua, build/release gates, tests/simulations, and public documentation. This is a broad source audit with targeted execution, not a formal proof of every interleaving or a performance certification.
Evidence labels used below:
Reproduced: an executable probe demonstrated the behavior on the audited revision.
Source-supported: the implementation establishes the concern; the exact operational scenario was not executed here.
Validation gap/design decision: a qualification or contract needed before 1.0, not a claimed reproduced defect.
Executable reproductions and audit logs are retained locally. Reproduced scenarios below will become permanent tests as each item is implemented.
Historical audit-time validation (before fixes and Docker recovery)
Check
Result and limits
./gradlew check
Passed initially, with almost all Java tasks up to date. Vitest actually ran: 33 tests passed. This alone is insufficient release evidence.
./gradlew check --rerun-tasks
Fresh Java checks ran; the aggregate failed because the Docker daemon was unavailable. Recorded JUnit results: 979 cases, 910 passed, nine failures attributable to container initialization, 60 skipped. These counts do not represent every test that would expand after successful container startup.
Real datastore integration
PostgreSQL contract/version tests and Redis standalone/Cluster/TLS/Sentinel tests could not initialize. This is an environment limitation, not evidence of nine product bugs. No PostgreSQL runtime correctness claim is made from this run.
Independent Redis probes
Ran against a disposable local Redis 8.6.1 server on port 16389. Reproduced Redis defects below. This supplements, and does not replace, Testcontainers or minimum-version/topology testing.
:threadmill-dashboard-spring:browserTest --rerun
Five Playwright scenarios passed freshly against the Spring server: authentication, queue controls, job controls, recurring controls, and redaction. Dependencies remained up to date.
:threadmill-simulation:simulateMemory
400 jobs succeeded, drained completely, zero invariant violations. The run exercised 25 failure events and five timeouts. It was a short in-memory simulation.
dependencySecurityScan
Failed on npm audit: one advisory affects two installed Vitest packages.
Independent OSV scan
Scanned exactly 18 tracked Gradle/npm lockfiles, using the repository's configuration and lockfile extractor. Same Vitest advisory; no additional reported Maven findings. This is known-advisory coverage, not proof that dependencies are vulnerability-free.
javadoc artifactInspection
Passed. Artifact inspection ran; most Javadoc tasks were up to date. This was not a clean rebuild of every document/JAR.
Production gate/endurance
productionCheck, real datastore simulations, and hours-scale endurance were not completed. Fresh build-logic tests also remain a release-gate requirement: the nested build retained up-to-date test results.
Validation logs and module JUnit totals are retained with the local audit. OSV scanned only unchanged, public dependency metadata.
Must change before 1.0
“Must” is my release recommendation. Source-supported findings are deliberately distinguished from reproduced failures. Effort is relative implementation scope, including appropriate regression coverage, not a calendar commitment.
M01 — Redis can run two EXCLUSIVE jobs for the same key
Reproduced · correctness · small/medium fix. Claim exclusive job A. Delete pending exclusive job B with the normal saveAtomic transition. A subsequent claim admits exclusive C while A is still PROCESSING: the probe observes two simultaneous exclusive jobs.
The terminal transition in save_atomic.lua decrements a workflow hold without first establishing that this root owns a hold. Decrementing a missing field creates a negative count, which then causes release of a shared/exclusive counter belonging to other work. Clamping the result to zero hides the bookkeeping violation. Other deletion/quarantine scripts already check that the hold exists. This is reachable through dashboard deletion, not just a custom SPI call.
Change: release only holds actually acquired by the affected root, consistently across every terminal mutation. Audit hold creation/release as one invariant across all scripts and stores.
Acceptance: a permanent shared contract/regression case for pending deletion while another root holds the key; include SHARED/EXCLUSIVE, workflow descendants, quarantine, replacement, and repeated/stale writes. No admitted overlap and exact counters after each operation.
M02 — Redis candidate selection can permanently starve runnable jobs
Reproduced · liveness/consistency · medium. Two independent inputs produce zero claims over 20 polls:
Twenty earlier SHARED jobs in queue other, followed by a SHARED job in queue target, all on one key. Poll only target with capacity one. Earlier SHARED work in another queue should not prohibit this claim.
One earliest SHARED job and 20 later EXCLUSIVE jobs with identical timestamps. No job is running, yet no claim succeeds.
RedisJobStore.admissibleIdsForKeys truncates the global pending set before queue filtering. For equal timestamps, Redis truncates by the MODE:uuid member order; sorting that truncated window by UUID cannot restore an omitted earliest job. Polling repeats the same unusable window.
Change: gather queue-relevant candidates with a representation whose stored total order agrees with admission order. Preserve the distinction between SHARED compatibility and an earlier EXCLUSIVE barrier. Do not fix this by merely increasing the window or reading the entire backlog.
Acceptance: shared contract cases for cross-queue SHARED progress, equal-time mixed modes, large same-key populations, and active workflow holds, with bounded work and a stated eventual-progress guarantee.
M03 — Workflow reconciliation deletes children whose parents will retry
Reproduced · workflow loss · medium/large. The recovery code treats a persisted FAILED parent as settled before it actually knows its retry disposition.
A parent fails less than five minutes before maintenance. Retry recovery intentionally skips the young failure; workflow reconciliation immediately deletes its AWAITING child.
With 501 old retryable parents and a page size of 500, recovery reschedules the first 500. Its increasing OFFSET then skips the remaining parent because the result set shrank. Reconciliation deletes that parent's child. The probe reports 500 recovered instead of 501 and a DELETED child.
Change: make final failure versus pending retry a durable decision. Reconcile only conclusively final parents. Replace OFFSET-over-mutating-state recovery with stable keyset/cursor semantics or an explicitly bounded pending-recovery queue. An age check alone does not fix the second reproduction.
Acceptance: permanent tests for both inputs, process death between failure and retry writes, partial recovery-store failure, concurrent reconciliation, and more than two pages. Children of retryable parents must never be abandoned.
Reproduced · public behavior · medium/large; implement with M03. Register policyFor(IllegalArgumentException.class, RetryPolicy.noRetry()). The normal failure hook leaves the first attempt FAILED. The later recovery sweep reschedules it using the global policy because the original exception/policy decision was not persisted.
RetryInterceptor explicitly documents this tradeoff. It is still inappropriate for a 1.0 API that exposes exception-specific retry policies: idempotency does not make an explicit no-retry instruction disposable. Shutdown recovery also needs to retain its distinct attempt-budget semantics after a crash.
Change: persist the effective retry disposition and timing alongside failure, with an atomic, recoverable handoff. Preserve the central failure/interceptor design while removing inference from absent in-memory exceptions.
Acceptance: no-retry remains final after restart and reconciliation; custom backoff/attempt limits and shutdown-neutral attempts survive the same crash boundaries.
M05 — Quarantine abandons finalization after a transient store failure
Reproduced · stranded jobs · small fix plus failure-path audit. Force handler resolution to fail, then fail the first QUARANTINED save. run() returns with the durable job still PROCESSING. Node owner-heartbeat refresh continues to shield it, although no worker owns finalization.
JobRunner.quarantine performs one direct save and catches/logs the error, unlike the retained terminal-save retry path.
Change: apply the same finalization-ownership rule to quarantine and every terminal fallback: retry transient persistence while responsible; on shutdown relinquish ownership so orphan recovery can work. Also review the failed-success-save reload branch, which can return without a finalizer if reload itself fails.
Acceptance: outages at quarantine, success fallback reload, and final failure persistence recover after the store returns without restarting a healthy node. A heartbeat-shielded PROCESSING attempt must always have an active responsible execution/finalizer.
M06 — Job-size handling can strand claims and failed executions
Reproduced · correctness/availability · medium. Two inputs expose gaps in the size invariant:
Insert a normal job followed by a job whose serialized form is accepted just below the configured size limit. Claim both. Adding PROCESSING state/owner/attempt metadata pushes the second over the limit. In memory and Redis, claimReady throws after the first job has already become PROCESSING; the caller receives neither job. PostgreSQL uses a transaction, so its corresponding batch should roll back, but the accepted poison candidate can still abort claims; that PostgreSQL behavior is a source inference, not a runtime result here.
A handler sets a 300 KB progress message and throws. FAILED serialization fails and the runner returns with the job still PROCESSING.
JsonJobSerializer bounds several mutable sections but carries progress through unchanged; JobProgress imposes no message bound. The current “terminal save can never be blocked” statement is too strong.
Change: define an enqueue payload budget that leaves room for bounded lifecycle data, bound every handler-growable field, and ensure a poison candidate cannot discard the successful portion of a claim operation. A minimal durable terminal representation may be needed for deterministic serialization failures.
Acceptance: test the byte boundary with valid payloads, multi-byte Unicode, success/failure/quarantine, many attempts, and mixed claim batches on all stores. Every accepted job is processable or durably rejected/quarantined; no unreturned claim remains protected indefinitely.
M07 — Late execution updates can move liveness backward
Reproduced through the memory SPI; analogous writes are source-supported in PostgreSQL/Redis · concurrency · medium. A fresh check-in is saved, then an older snapshot from the same attempt is saved. The job becomes eligible for orphan detection despite the newer check-in.
Change: make persisted heartbeat/check-in timestamps monotonic and serialize/coalesce context flushes or give them an attempt-local revision. Keep body/scalars consistent; timestamp maxima alone do not prevent older progress/log snapshots replacing newer ones.
Acceptance: delay/reorder same-attempt writes and race check-ins with node refresh across all backends. No false orphan and no regression of acknowledged execution progress.
M08 — PostgreSQL transaction cleanup can commit on Error
Source-supported; not exercised against PostgreSQL here · atomicity · small.OwningPostgresTransactionBoundary and MigrationRunner roll back RuntimeException and SQLException, but not Error. Their finally restores auto-commit. If an Error escapes after a write and the connection remains usable, switching auto-commit back on commits the open transaction according to the Java 25 JDBC contract.
Change: attempt rollback for Error as well, rethrow the original failure, and preserve rollback/reset failures as suppressed exceptions. Avoid committing during cleanup after any failed work unit. Apply consistently to both transaction helpers.
Acceptance: real PostgreSQL test writes a row then throws AssertionError; neither row nor partial migration survives. Include rollback/reset failure handling without masking the primary exception.
M09 — Default retention cannot keep up with modest sustained throughput
Source-supported and arithmetic · capacity · medium.MaintenanceCycle deletes at most 100 × 50 = 5,000 jobs per terminal state per retention interval. The default interval is one hour: about 1.39 jobs/second. At a steady 50 successful jobs/second, 180,000 jobs age into eligibility per hour; only 5,000 can leave, so eligible backlog grows by about 175,000/hour once retention age is reached. Expired dedup cleanup has the same per-interval cap.
Bounded batches are sensible; deferring unfinished work for another hour is the problem. An eight-hour soak with seven-day success retention does not exercise this steady-state failure.
Change: retain bounded/time-budgeted sweeps but resume unfinished work promptly with fair scheduling. Publish deletion throughput and oldest eligible age, and ensure retention cannot monopolize promotion or recovery.
Acceptance: sustained input above 50/s with shortened retention reaches stable storage size; measure cleanup rate, lag, datastore load, and promotion latency. Include high-cardinality dedup keys and node churn.
M10 — Dashboard installation can remove host-wide default security
Source-supported and explicitly encoded in existing tests/docs · integration/security design · medium. The auto-configured dashboard SecurityFilterChain matches only dashboard paths. Its existence makes Spring Boot's default catch-all chain back off. A host previously relying on Boot defaults can therefore expose unrelated endpoints after adding the dashboard.
This is an acknowledged design decision in AGENTS, not a claim that dashboard endpoints themselves lack protection. Nevertheless, a library should not weaken host security as an incidental default.
Change: choose a fail-safe integration contract before 1.0: require explicit dashboard security opt-in/host configuration, or preserve a safe catch-all only when the host supplies none. Document chain precedence and test application endpoints alongside dashboard endpoints.
Acceptance: installing/enabling the dashboard cannot turn a previously authenticated host endpoint into an anonymous endpoint. Cover starter-only applications, custom chains, actuator, custom API path, CSRF, and disabled dashboard security.
M11 — Declare and enforce the actual minimum Redis version
Source-supported compatibility requirement · small. Claim discovery always calls HSCAN NOVALUES. Redis documents NOVALUES as introduced in 7.4. The repository uses broad redis:7-alpine references and does not clearly establish/enforce this minimum.
Change: either declare Redis 7.4+ and validate at startup, including the managed-service validation policy, or remove the newer command dependency if older Redis 7 is intended. Pin test images to the supported minimum and at least one newer version. A moving 7 image does not prove compatibility with 7.0/7.2.
Acceptance: unsupported versions fail early with an actionable message; the full store contract passes on the declared minimum. Include Sentinel data nodes and Cluster in the compatibility policy.
M12 — Correct the transactional-outbox example and guarantees
Source-supported documentation defect · small.docs/transactions.md calls a remote email send inside a database transaction, then records an outbox marker. That does not atomically commit the remote effect and database record. A crash after send, a rollback, or concurrent runners can send twice. Later text correctly describes that duplicate window, then incorrectly says the outbox makes that scenario safe.
Change: show a real transactional outbox: atomically write business state plus an outgoing intent in the application database, then deliver that intent separately with downstream idempotency where supported. A local uniqueness constraint handles local deduplication; it does not make an external send transactional. Remove blanket exactly-once claims about arbitrary providers.
Acceptance: the example and failure table agree for crashes before/after commit and delivery, rollback, and simultaneous duplicate attempts. Keep Threadmill's at-least-once guarantee explicit, qualified by successful durable enqueue and the configured datastore durability.
M13 — Resolve the dependency gate failure
Verified by npm audit and OSV · release tooling · small. The UI lockfile resolves vitest and @vitest/mocker 4.1.7. Both scanners report GHSA-82fw-gwwq-j7x9; the fixed version is 4.1.11. This concerns development/test tooling, not evidence of the same vulnerability in the static dashboard runtime. It still fails the repository's required low-threshold dependency gate.
Change: update the Vitest family consistently to a patched compatible version, update the lockfile, run component/browser tests and the dependency gate. Do not suppress the advisory to get a green release.
Acceptance: npm audit and OSV pass under the pinned CI toolchain, with no unjustified ignores. Update dependency-security documentation if needed.
Should change or qualify before 1.0
S01 — Measure and remove PostgreSQL monitoring costs proportional to backlog
Source-supported performance risk.queueDepths/listEnqueuedQueues still aggregate/distinct over ENQUEUED jobs. oldestEnqueuedAt lacks an index directly matching (queue, current_state_at) for its filter/order; the queue/key/time index cannot provide global age order within a queue with many keys. Metrics refresh requests all queue depths and then one age query per selected queue. Bounded metric tags do not bound the cost of the initial query.
Run PostgreSQL 18 EXPLAIN (ANALYZE, BUFFERS) and concurrent claim benchmarks at 10k/100k/1m pending jobs, with metrics/dashboard polling enabled. Consider sharded queue counters and a targeted additive partial index after measurements. Keep monitoring refresh off latency-critical scrape paths if it can block on the store. Do not edit the released V1 baseline.
S02 — Bound Redis script and admission work, not just returned candidates
Source-supported complexity risk; no latency benchmark claimed.claim_commit.lua reads all earlier pending members and all members at the same score to determine whether an admission barrier exists. Java also reads all active workflow-root fields with HKEYS. Bulk insertion has no explicit job/byte batch ceiling. The Redis scripting model blocks other server activity during script execution; a Lua call being atomic does not make its running time bounded. SCAN COUNT is a work hint rather than a strict result cap.
Use indexes that answer “earliest blocking EXCLUSIVE” and “next eligible member” directly; impose explicit input/work budgets and resume cursors. Benchmark long same-key SHARED runs, timestamp collisions, many active workflows, and large bulk calls while measuring unrelated Redis command latency. Document that the common {threadmill} hash tag places all Threadmill keys in one Cluster slot: Cluster provides topology/failover integration here, not horizontal sharding of one namespace's job load.
S03 — Give maintenance bounded work and independent progress
Recovery and workflow reconciliation page across entire state populations; reconciliation retains a HashSet of every handled parent during the pass. Recurring materialization visits every definition, and the cron DROP path can walk all missed fires. Promotion is limited to 100 jobs per maintenance tick. The current caps and all-population scans need one coherent capacity model.
Introduce resumable cursors and wall-time budgets per activity, separate correctness recovery from the hour-scale retention cadence, and expose oldest scheduled/recovery/retention lag. Test many dormant cron definitions, long downtime, large AWAITING/FAILED populations, and promotion bursts. Correctness work must not disappear behind expensive housekeeping.
S04 — Clean up concurrency metadata for keys that never return
The PostgreSQL group rows and Redis counter hashes can remain after their counts return to zero; no general reclamation for lifetime-distinct concurrency keys was found. Job retention alone does not bound this metadata. This matters when applications use one fresh key per business operation rather than a fixed tenant set.
Add bounded, race-safe reclamation only when no active hold or pending/outstanding job still needs the key; lock/verify against concurrent enqueue and claim. Include metadata cardinality in retention endurance assertions.
S05 — Use the application classloader consistently for payloads
Source-supported integration risk.SpringJobHandlerResolver uses the application context classloader, but JobRunner.deserializePayload uses the library classloader. Parent-library/child-application deployments can resolve a handler and still quarantine its payload as unknown.
Provide a coherent payload type-resolution strategy/classloader, retaining the existing load-without-initialization and assignability checks. Test an actual child-classloader application and Spring restart/deployment shape; do not simply turn on unrestricted polymorphic deserialization.
S06 — Close observability state on every execution exit
Source-supported lifecycle risk.ThreadmillTracing keys active scopes by job ID, not attempt, and assumes start/finish hooks always pair on one thread. Stale terminal-save paths return before success/failure hooks; orphan recovery can invoke failure hooks on a different thread. Metrics also maintain in-flight timing state that depends on finish hooks.
Tie lifecycle state to an execution/attempt token and guarantee cleanup in an engine finally path, including stale completion, deterministic save failure, shutdown, and overlapping orphan recovery. Avoid closing a thread-bound scope from another execution. Add regression tests that assert no leaked span/timer state and correct attempt attribution.
S07 — Make Redis search limitations explicit at the SPI boundary
Reproduced, lower impact. Redis searchJobs pages by state/time before applying queue/handler filters. The probe gets an empty first page although a matching job exists outside that initial unfiltered window. PostgreSQL/memory filter before pagination.
The dashboard already checks supportsRichSearch=false, which limits exposure. Decide whether unsupported SPI filters should throw clearly or be implemented consistently using indexes. Do not silently return a misleading filtered page. Document the exact supported search subset and add a cross-store contract or explicit unsupported-operation test.
S08 — Make after-commit enqueue failure visible to applications
The default Spring mode writes after the business transaction commits; deferred enqueue errors are logged rather than delivered to the original caller, and a returned ID is not proof of durable enqueue. The detailed transaction guide largely explains this, but the main adoption path should make the boundary unavoidable.
Keep explicit transaction modes, prominently show join_transaction for PostgreSQL business-write/job-write atomicity, and expose an observable deferred-enqueue failure hook/counter. Document an application-owned durable outbox for cross-datastore requirements. Consider bounded batching of deferred enqueue callbacks. This is a durability contract decision, not a request to claim exactly-once execution.
S09 — Freeze the 1.0 compatibility contract and prove upgrades
Before promising SemVer stability, review public JobStore operations/defaults/capabilities, public engine mutators and configuration records, retry/finalization hooks, serialized fields, and dashboard DTOs together. Add old-version golden wire fixtures, versioned upgrade tests from v0.3.0, and an explicit rolling-upgrade/downgrade policy. Class names are durable type tags today; drain/migrate-before-rename must remain explicit.
Keep PostgreSQL migrations additive/checksum-validated; add a Redis representation/index migration strategy for changes required by M01/M02/M03. Do not silently reinterpret existing keys. Verify both fresh initialization and nonempty upgrade, including workflows, dedup, recurring state/nudges, paused queues, and in-flight jobs.
Quick wins
These are small, coherent contributions. Some are also mandatory findings above; “quick” does not mean optional.
Order
Change
Why it pays off
1
Patch Vitest and rerun the dependency/UI gates (M13).
Removes a verified release-gate failure.
2
Correct the outbox example and its failure table (M12).
Prevents applications from adopting an unsafe side-effect pattern.
3
Guard Redis hold release (M01), with permanent regressions.
Small surface area, severe correctness impact.
4
Reuse retained finalization for quarantine (M05).
Prevents a one-write outage from stranding work.
5
Roll back PostgreSQL Error paths (M08).
Closes an atomicity hole in two small helpers.
6
Declare/enforce the minimum Redis version (M11).
Converts obscure runtime command errors into clear compatibility behavior.
A paused empty/drained queue otherwise disappears from the queue controls even though its pause remains durable. Union queue-depth keys with paused-queue keys.
9
Show safe API error details and failure history in the UI.
api.ts discards structured error detail. Its history type uses detail where the Java entry has message, and the UI shows only state/time. Preserve server-side redaction while making failures actionable.
10
Protect UI reloads/mutations against overlapping responses.
Request generations/abort and disabled pending action buttons prevent older polls from overwriting current selections and repeated operator submissions.
11
Reconcile documentation and standing instructions.
Replace hand-maintained test counts with generated evidence; remove the impossible “subclass final CronExpression” guidance; align metadata/tag name rules with validation; remove obsolete Boot-4-milestone rationale now that the project pins Boot 4.0.8. Keep history separate from current guarantees.
12
Consider AutoCloseable for RedisJobStore before the API freeze.
It already has explicit close ownership semantics; try-with-resources support makes correct application cleanup easier. Confirm consistency with the store SPI and injected-client ownership.
Release qualification still required
Real topology and durability evidence
The open issue #98 describes the need for real Sentinel/Cluster/authentication/TLS coverage. The current repository already has real Cluster contract and authenticated TLS/Sentinel smoke tests, so the issue's claim that coverage is only unit-level is stale. The remaining qualification is substantial: actual replica promotion, network disruption, topology refresh/resharding, and recovery under load. Startup/round-trip tests do not establish failover behavior.
Run a supported-version matrix with PostgreSQL 18 and the declared Redis minimum plus a newer version. Include real multi-node Sentinel/Cluster failover, verified TLS/ACLs, certificate rejection, wrong credentials, timeouts, noeviction/OOM, and managed-service CONFIG restrictions. Exercise process kills at claim, terminal write, retry handoff, workflow propagation, enqueue/nudge, and lease takeover.
Keep durability claims tied to Redis persistence and replication configuration. The current AOF/noeviction guidance is useful. Add an explicit acknowledged-write-loss/failover boundary and recovery-point expectations; server-side Lua atomicity alone does not imply durable acknowledged writes. See Redis persistence documentation.
Better tests for the failures that matter
Turn every reproduced audit case into a named permanent test. Cross-backend behavior belongs in the shared contract; Redis candidate/index details and PostgreSQL transaction cleanup also need backend regressions. Add generated/model-based sequences combining enqueue, claim, pause, delete, retry, replace, workflow completion, retention, and crash. Compare durable states, counters, hold ownership, and eventual progress against a small reference model.
Use fault injection at durable boundaries, including “write committed but acknowledgement lost.” For safety, assert no overlapping exclusive holders; for liveness, assert eventually runnable jobs are claimed and no abandoned PROCESSING row remains shielded. A zero-violation throughput run is not sufficient if it never generates the triggering combination.
Extend endurance to a steady retained population using shortened retention windows, accumulating concurrency/dedup keys, metrics/dashboard polling, realistic payload sizes, queue families, and node churn. Record p50/p95/p99 enqueue/claim/terminal latency, throughput, oldest queued/scheduled/retention age, database/Redis memory, JVM heap, and per-tick maintenance duration. Compare against a recorded baseline rather than a single jobs/second headline.
Targeted regressions and affected gates green; public examples accurately state behavior.
B — Core correctness
M02–M04, M06–M07; failure/retry/finalizer model and stable candidate/recovery ordering.
All audit probes converted to passing permanent tests; identical store guarantees; process-crash tests pass.
C — Operational bounds
M09–M10; measured S01–S06 improvements and topology/upgrade qualification.
Stable retention/storage under sustained load, fail-safe host security, no starving maintenance, successful failover and nonempty upgrades.
D — 1.0 candidate
Freeze API and wire/security/durability contracts; run release workflow against the exact candidate.
Fresh check, required dependency scan, complete productionCheck, browser tests, real datastore simulations, and Postgres/Redis endurance pass; retain artifacts with revision, versions, settings, and seed.
I would not substitute a date or a feature-count target for these exit criteria. A 1.0 candidate should have no known violations of job ownership, concurrency exclusion, retry policy, workflow preservation, or sustainable retention, and every advertised topology should have fresh qualification evidence.
Overall recommendation
Keep the current architecture and concentrate the next release cycle on the interaction defects above. The fastest route to a trustworthy 1.0 is a series of small verified fixes around a few shared invariants, followed by measured operational and upgrade qualification. Broad rewrites or additional features would make that qualification harder without addressing the reproduced failures.
Redis qualification audit correction
The Redis preflight finished all 2,251 jobs and passed the harness invariants, but monitoring found a private-auditor bug: its HVALS-based active-group check treated 11 idle_since timestamps as live work. The original RDB was restored into an isolated read-only Redis instance; all 20 shared/exclusive counter pairs were zero, pending/workflow holds were zero, and job records, state indexes and counters agreed. The original run recorded zero evictions. The original failed audit remains intact alongside audit-review.json and snapshot-audit-corrected.json, which explicitly accept preflight correctness/drain. Three auditor regressions now pass, including the preserved snapshot. No Threadmill source or frozen JAR changed.
The original PostgreSQL campaign continues unchanged. Its supervisor already marked Redis blocked, so only the seven unrun Redis phases are armed under .local-reference/qualification/20260909T2258Z-redis-remaining. That runner waits for the original PostgreSQL supervisor to exit, verifies the same 43-JAR manifest, and then uses new isolated volumes. No completed phase is repeated and no original datastore is reset. The five-minute monitor now tracks both roots. Endurance, performance and stability qualification remain pending.
1.0 hardening implementation tracker
Current status
The audit implementation and review fixes are committed and pushed in draft PR #136, currently at
e0e5b47. The PR is unmerged and unreleased; v0.3.0 remains the published version. The 22 original implementation/planning steps below are complete. Review added eligible retention indexing/cursors, active-queue cleanup filtering, custom retry-policy precedence, strict active-attempt heartbeats and related regressions. PostgreSQL upgrades use additive V7–V11 migrations; Redis upgrades use the offline/resumable format-2 migration. Stop all old workers/producers, back up and recompile; mixed versions are unsupported. Seedocs/compatibility.md.Current validation
./gradlew productionCheck -PdependencyScanRequired=true --max-workers=2 --no-build-cachepassed in 10m 19s: 1,669 JUnit cases, zero failures/errors/skips, including 173 PostgreSQL, 413 Redis and 32 fixed-soak cases. The gate also passed 41 UI tests, five browser scenarios, all three short correctness simulations, both process-separated nudge simulations, required OSV/npm scans, the example, Javadoc and artifact inspection. All 14 binary JARs were also inspected../gradlew checkin 4s. Runtime JAR checksums are identical to the full production gate; the source-manifest record identifies that documentation-only delta.Endurance and remaining performance qualification
A new frozen campaign started on e0e5b47, using dedicated PostgreSQL 18 and standalone Redis 7.4 containers/volumes. PostgreSQL preflight passed with 3001 completed jobs through its planned pause and restart, plus a passing independent datastore audit; Redis preflight completed; see the audit correction below. The backends run sequentially, each with 30-minute mixed and retention baselines, 12-hour mixed and retention runs, and 30-minute retry/check-in/nudge phases. The rate is 25 jobs/second (long-running deliberately 1/second with one-quarter planned timeouts). Long phases include node churn every ten minutes and 20-second datastore pauses at hours 2, 6 and 10; PostgreSQL also restarts at hour 8. Planned duration is approximately 53 hours plus startup, drain and reporting. Endurance/performance/stability results remain pending.
Artifacts and the supervisor are under
.local-reference/qualification/20260909T2225Z; 43 runtime JAR checksums, exact candidate, environment/settings, source manifests and gate reports are archived. The combined five-minute PR/soak monitor is active. Local PR branch/worktree cleanup occurs only after confirmed merge, preserving unrelated work and qualification evidence.The earlier 0fc9881 campaign was superseded and stopped gracefully. Its five completed preflight/baseline phases passed correctness/drain for that old candidate, and its Redis retention baseline is interrupted evidence. Old runs and volumes are preserved, and none qualifies the reviewed candidate.
The indexed pending-retry / large-FAILED-population benchmark is an explicit deferred performance follow-up: the 30-second inter-pass pause and bounded recovery pages do not establish efficient scanning through a very large retained 30-day FAILED backlog. Add that populated-store measurement before claiming capacity for that shape; consider a PostgreSQL retry scalar/index and Redis ordered pending-retry index if the measured budget is exceeded. Hours-scale Redis 8.6/Sentinel/Cluster qualification remains separate. Keep this issue open for review, merge and final qualification.
Sequential work
Separate long-running soak plans (planned, not executed)
The complete runbook is prepared locally in
docs/soak-plan-1.0.md. Use dedicated disposable datastores: harness startup resets Threadmill data. Freeze the revision plus patch, resolved image digests, configuration and resource limits after the release gate. Run PostgreSQL and Redis separately on fixed hardware.For each backend, first record a 30-minute baseline without node churn. Then run 12 hours mixed workload and 12 hours retention churn, sequentially, with one producer, three nodes, eight workers/node and an initial offered rate of 50 jobs/s. Choose the sustainable rate from the baseline and keep it fixed. Each phase needs a distinct output path and final datastore snapshot.
./gradlew :threadmill-soak:soakPostgres \ -Pscenario=mixed-workload -Pduration=12h -PjobsPerSecond=50 \ -Pproducers=1 -Pnodes=3 -PworkerCount=8 -PnodeChurn=10m \ -PprogressInterval=30s -PfailFast=true \ '-PpostgresUrl=jdbc:postgresql://localhost:54320/threadmill?user=threadmill&password=threadmill' \ -PrunId=pg18-mixed-candidate -PoutputDir=.local-reference/qualification/pg18-mixed-candidate ./gradlew :threadmill-soak:soakRedis \ -Pscenario=mixed-workload -Pduration=12h -PjobsPerSecond=50 \ -Pproducers=1 -Pnodes=3 -PworkerCount=8 -PnodeChurn=10m \ -PprogressInterval=30s -PfailFast=true \ -PredisTopology=standalone -PredisUrl=redis://localhost:63790 \ -PrunId=redis74-mixed-candidate -PoutputDir=.local-reference/qualification/redis74-mixed-candidateRepeat each command with
-Pscenario=retention-churnand distinct retention run IDs/output directories. That deterministic sequence recipe uses 8 KiB payloads, fresh concurrency keys, queue families, workflows, first-attempt retries, duplicate producer requests, ten-second retention and five-second dedup TTL. The existing mixed workload has no exposed seed; retain its full trace. Do not run concurrent Gradle builds against the active harness classpath.Exit criteria: zero definite invariant violations or unexplained stuck/failed jobs; complete drain and reconciled counters; at least 95% of offered throughput outside faults/recovery; stable-window p95 within 20% and p99 within 2x of the same-host baseline; no growing backlog/maintenance lag; retained jobs and metadata plateau; no unexplained heap/RSS/disk/index growth; current monitoring and zero Redis evictions. Record every fault's recovery time. Investigate any breach without changing thresholds afterward. Redis data loss remains a failed zero-loss qualification even when caused by deployment replication policy, and requires an explicit deployment-owner decision.
Remaining before 1.0: review and merge the uncommitted changes, run and assess these endurance phases on the frozen candidate, then complete the release/tag workflow. The successful bounded gate alone is not endurance or production-capacity certification.
Threadmill 1.0 readiness audit
Audited 9 September 2026. Repository:
hemju/threadmill. Revision:0af7e0ac22f36f255e00610c93aac566ba0dfe87, also taggedv0.3.0. The findings below describe the original v0.3.0 audit before implementation; the current status and fresh validation are recorded above.Assessment
I would not release this revision as 1.0. The architecture is suitable for the product, but executable probes found violations of exclusive execution, claim liveness, workflow preservation, retry policy, and finalization. These deserve priority over additional features or general refactoring.
The strongest parts are the explicit state machine, separation of core and adapters, persisted version discipline, snapshot serialization, shared store contracts, store-backed maintenance leadership, additive PostgreSQL migrations, scoped dashboard permissions/redaction, and substantial fault-oriented tests and simulation tooling. Preserve these foundations.
The largest recurring weakness is the interaction between individually reasonable mechanisms: bounded candidate windows and global ordering; FAILED persistence and later retry hooks; node-wide heartbeats and abandoned finalizers; accepted payload size and subsequent lifecycle growth. The tests cover many individual regressions but need stronger coverage of these combinations.
Recommended release scope: correctness and durability hardening, explicit supported-version and security contracts, sustainable maintenance capacity, and fresh production validation. Native image, additional framework adapters, more scheduling features, and a dashboard redesign can wait.
Scope and evidence
The repository inventory contains 489 tracked files, including 323 Java files, 15 Lua resources, six SQL migrations, and 52 Markdown files. The review covered the Java model, engine, scheduling, stores, Spring integration, dashboard API/UI, metrics/tracing, SQL schema and hot queries, all 15 Lua resources, relevant inline Lua, build/release gates, tests/simulations, and public documentation. This is a broad source audit with targeted execution, not a formal proof of every interleaving or a performance certification.
Evidence labels used below:
Executable reproductions and audit logs are retained locally. Reproduced scenarios below will become permanent tests as each item is implemented.
Historical audit-time validation (before fixes and Docker recovery)
./gradlew check./gradlew check --rerun-tasks:threadmill-dashboard-spring:browserTest --rerun:threadmill-simulation:simulateMemorydependencySecurityScanjavadoc artifactInspectionproductionCheck, real datastore simulations, and hours-scale endurance were not completed. Fresh build-logic tests also remain a release-gate requirement: the nested build retained up-to-date test results.Validation logs and module JUnit totals are retained with the local audit. OSV scanned only unchanged, public dependency metadata.
Must change before 1.0
“Must” is my release recommendation. Source-supported findings are deliberately distinguished from reproduced failures. Effort is relative implementation scope, including appropriate regression coverage, not a calendar commitment.
M01 — Redis can run two EXCLUSIVE jobs for the same key
Reproduced · correctness · small/medium fix. Claim exclusive job A. Delete pending exclusive job B with the normal
saveAtomictransition. A subsequent claim admits exclusive C while A is still PROCESSING: the probe observes two simultaneous exclusive jobs.The terminal transition in save_atomic.lua decrements a workflow hold without first establishing that this root owns a hold. Decrementing a missing field creates a negative count, which then causes release of a shared/exclusive counter belonging to other work. Clamping the result to zero hides the bookkeeping violation. Other deletion/quarantine scripts already check that the hold exists. This is reachable through dashboard deletion, not just a custom SPI call.
Change: release only holds actually acquired by the affected root, consistently across every terminal mutation. Audit hold creation/release as one invariant across all scripts and stores.
Acceptance: a permanent shared contract/regression case for pending deletion while another root holds the key; include SHARED/EXCLUSIVE, workflow descendants, quarantine, replacement, and repeated/stale writes. No admitted overlap and exact counters after each operation.
M02 — Redis candidate selection can permanently starve runnable jobs
Reproduced · liveness/consistency · medium. Two independent inputs produce zero claims over 20 polls:
other, followed by a SHARED job in queuetarget, all on one key. Poll onlytargetwith capacity one. Earlier SHARED work in another queue should not prohibit this claim.RedisJobStore.admissibleIdsForKeys truncates the global pending set before queue filtering. For equal timestamps, Redis truncates by the
MODE:uuidmember order; sorting that truncated window by UUID cannot restore an omitted earliest job. Polling repeats the same unusable window.Change: gather queue-relevant candidates with a representation whose stored total order agrees with admission order. Preserve the distinction between SHARED compatibility and an earlier EXCLUSIVE barrier. Do not fix this by merely increasing the window or reading the entire backlog.
Acceptance: shared contract cases for cross-queue SHARED progress, equal-time mixed modes, large same-key populations, and active workflow holds, with bounded work and a stated eventual-progress guarantee.
M03 — Workflow reconciliation deletes children whose parents will retry
Reproduced · workflow loss · medium/large. The recovery code treats a persisted FAILED parent as settled before it actually knows its retry disposition.
See RetryInterceptor recovery, WorkflowInterceptor reconciliation, and maintenance ordering.
Change: make final failure versus pending retry a durable decision. Reconcile only conclusively final parents. Replace OFFSET-over-mutating-state recovery with stable keyset/cursor semantics or an explicitly bounded pending-recovery queue. An age check alone does not fix the second reproduction.
Acceptance: permanent tests for both inputs, process death between failure and retry writes, partial recovery-store failure, concurrent reconciliation, and more than two pages. Children of retryable parents must never be abandoned.
M04 — Failure recovery overrides exception-specific
noRetryReproduced · public behavior · medium/large; implement with M03. Register
policyFor(IllegalArgumentException.class, RetryPolicy.noRetry()). The normal failure hook leaves the first attempt FAILED. The later recovery sweep reschedules it using the global policy because the original exception/policy decision was not persisted.RetryInterceptor explicitly documents this tradeoff. It is still inappropriate for a 1.0 API that exposes exception-specific retry policies: idempotency does not make an explicit no-retry instruction disposable. Shutdown recovery also needs to retain its distinct attempt-budget semantics after a crash.
Change: persist the effective retry disposition and timing alongside failure, with an atomic, recoverable handoff. Preserve the central failure/interceptor design while removing inference from absent in-memory exceptions.
Acceptance: no-retry remains final after restart and reconciliation; custom backoff/attempt limits and shutdown-neutral attempts survive the same crash boundaries.
M05 — Quarantine abandons finalization after a transient store failure
Reproduced · stranded jobs · small fix plus failure-path audit. Force handler resolution to fail, then fail the first QUARANTINED save.
run()returns with the durable job still PROCESSING. Node owner-heartbeat refresh continues to shield it, although no worker owns finalization.JobRunner.quarantine performs one direct save and catches/logs the error, unlike the retained terminal-save retry path.
Change: apply the same finalization-ownership rule to quarantine and every terminal fallback: retry transient persistence while responsible; on shutdown relinquish ownership so orphan recovery can work. Also review the failed-success-save reload branch, which can return without a finalizer if reload itself fails.
Acceptance: outages at quarantine, success fallback reload, and final failure persistence recover after the store returns without restarting a healthy node. A heartbeat-shielded PROCESSING attempt must always have an active responsible execution/finalizer.
M06 — Job-size handling can strand claims and failed executions
Reproduced · correctness/availability · medium. Two inputs expose gaps in the size invariant:
claimReadythrows after the first job has already become PROCESSING; the caller receives neither job. PostgreSQL uses a transaction, so its corresponding batch should roll back, but the accepted poison candidate can still abort claims; that PostgreSQL behavior is a source inference, not a runtime result here.JsonJobSerializer bounds several mutable sections but carries progress through unchanged; JobProgress imposes no message bound. The current “terminal save can never be blocked” statement is too strong.
Change: define an enqueue payload budget that leaves room for bounded lifecycle data, bound every handler-growable field, and ensure a poison candidate cannot discard the successful portion of a claim operation. A minimal durable terminal representation may be needed for deterministic serialization failures.
Acceptance: test the byte boundary with valid payloads, multi-byte Unicode, success/failure/quarantine, many attempts, and mixed claim batches on all stores. Every accepted job is processable or durably rejected/quarantined; no unreturned claim remains protected indefinitely.
M07 — Late execution updates can move liveness backward
Reproduced through the memory SPI; analogous writes are source-supported in PostgreSQL/Redis · concurrency · medium. A fresh check-in is saved, then an older snapshot from the same attempt is saved. The job becomes eligible for orphan detection despite the newer check-in.
The previous-attempt guards do not reject out-of-order writes within one attempt. ExecutionContext.flushIfDue can initiate overlapping flushes; Postgres saveExecutionUpdate and Redis saveExecutionUpdate overwrite timestamps and body. An old handler snapshot can also overwrite a newer node-level owner heartbeat.
Change: make persisted heartbeat/check-in timestamps monotonic and serialize/coalesce context flushes or give them an attempt-local revision. Keep body/scalars consistent; timestamp maxima alone do not prevent older progress/log snapshots replacing newer ones.
Acceptance: delay/reorder same-attempt writes and race check-ins with node refresh across all backends. No false orphan and no regression of acknowledged execution progress.
M08 — PostgreSQL transaction cleanup can commit on
ErrorSource-supported; not exercised against PostgreSQL here · atomicity · small. OwningPostgresTransactionBoundary and MigrationRunner roll back
RuntimeExceptionandSQLException, but notError. Theirfinallyrestores auto-commit. If anErrorescapes after a write and the connection remains usable, switching auto-commit back on commits the open transaction according to the Java 25 JDBC contract.Change: attempt rollback for
Erroras well, rethrow the original failure, and preserve rollback/reset failures as suppressed exceptions. Avoid committing during cleanup after any failed work unit. Apply consistently to both transaction helpers.Acceptance: real PostgreSQL test writes a row then throws
AssertionError; neither row nor partial migration survives. Include rollback/reset failure handling without masking the primary exception.M09 — Default retention cannot keep up with modest sustained throughput
Source-supported and arithmetic · capacity · medium. MaintenanceCycle deletes at most 100 × 50 = 5,000 jobs per terminal state per retention interval. The default interval is one hour: about 1.39 jobs/second. At a steady 50 successful jobs/second, 180,000 jobs age into eligibility per hour; only 5,000 can leave, so eligible backlog grows by about 175,000/hour once retention age is reached. Expired dedup cleanup has the same per-interval cap.
Bounded batches are sensible; deferring unfinished work for another hour is the problem. An eight-hour soak with seven-day success retention does not exercise this steady-state failure.
Change: retain bounded/time-budgeted sweeps but resume unfinished work promptly with fair scheduling. Publish deletion throughput and oldest eligible age, and ensure retention cannot monopolize promotion or recovery.
Acceptance: sustained input above 50/s with shortened retention reaches stable storage size; measure cleanup rate, lag, datastore load, and promotion latency. Include high-cardinality dedup keys and node churn.
M10 — Dashboard installation can remove host-wide default security
Source-supported and explicitly encoded in existing tests/docs · integration/security design · medium. The auto-configured dashboard SecurityFilterChain matches only dashboard paths. Its existence makes Spring Boot's default catch-all chain back off. A host previously relying on Boot defaults can therefore expose unrelated endpoints after adding the dashboard.
This is an acknowledged design decision in AGENTS, not a claim that dashboard endpoints themselves lack protection. Nevertheless, a library should not weaken host security as an incidental default.
Change: choose a fail-safe integration contract before 1.0: require explicit dashboard security opt-in/host configuration, or preserve a safe catch-all only when the host supplies none. Document chain precedence and test application endpoints alongside dashboard endpoints.
Acceptance: installing/enabling the dashboard cannot turn a previously authenticated host endpoint into an anonymous endpoint. Cover starter-only applications, custom chains, actuator, custom API path, CSRF, and disabled dashboard security.
M11 — Declare and enforce the actual minimum Redis version
Source-supported compatibility requirement · small. Claim discovery always calls HSCAN NOVALUES. Redis documents NOVALUES as introduced in 7.4. The repository uses broad
redis:7-alpinereferences and does not clearly establish/enforce this minimum.Change: either declare Redis 7.4+ and validate at startup, including the managed-service validation policy, or remove the newer command dependency if older Redis 7 is intended. Pin test images to the supported minimum and at least one newer version. A moving
7image does not prove compatibility with 7.0/7.2.Acceptance: unsupported versions fail early with an actionable message; the full store contract passes on the declared minimum. Include Sentinel data nodes and Cluster in the compatibility policy.
M12 — Correct the transactional-outbox example and guarantees
Source-supported documentation defect · small. docs/transactions.md calls a remote email send inside a database transaction, then records an outbox marker. That does not atomically commit the remote effect and database record. A crash after send, a rollback, or concurrent runners can send twice. Later text correctly describes that duplicate window, then incorrectly says the outbox makes that scenario safe.
Change: show a real transactional outbox: atomically write business state plus an outgoing intent in the application database, then deliver that intent separately with downstream idempotency where supported. A local uniqueness constraint handles local deduplication; it does not make an external send transactional. Remove blanket exactly-once claims about arbitrary providers.
Acceptance: the example and failure table agree for crashes before/after commit and delivery, rollback, and simultaneous duplicate attempts. Keep Threadmill's at-least-once guarantee explicit, qualified by successful durable enqueue and the configured datastore durability.
M13 — Resolve the dependency gate failure
Verified by npm audit and OSV · release tooling · small. The UI lockfile resolves
vitestand@vitest/mocker4.1.7. Both scanners report GHSA-82fw-gwwq-j7x9; the fixed version is 4.1.11. This concerns development/test tooling, not evidence of the same vulnerability in the static dashboard runtime. It still fails the repository's required low-threshold dependency gate.Change: update the Vitest family consistently to a patched compatible version, update the lockfile, run component/browser tests and the dependency gate. Do not suppress the advisory to get a green release.
Acceptance: npm audit and OSV pass under the pinned CI toolchain, with no unjustified ignores. Update dependency-security documentation if needed.
Should change or qualify before 1.0
S01 — Measure and remove PostgreSQL monitoring costs proportional to backlog
Source-supported performance risk. queueDepths/listEnqueuedQueues still aggregate/distinct over ENQUEUED jobs.
oldestEnqueuedAtlacks an index directly matching(queue, current_state_at)for its filter/order; the queue/key/time index cannot provide global age order within a queue with many keys. Metrics refresh requests all queue depths and then one age query per selected queue. Bounded metric tags do not bound the cost of the initial query.Run PostgreSQL 18
EXPLAIN (ANALYZE, BUFFERS)and concurrent claim benchmarks at 10k/100k/1m pending jobs, with metrics/dashboard polling enabled. Consider sharded queue counters and a targeted additive partial index after measurements. Keep monitoring refresh off latency-critical scrape paths if it can block on the store. Do not edit the released V1 baseline.S02 — Bound Redis script and admission work, not just returned candidates
Source-supported complexity risk; no latency benchmark claimed. claim_commit.lua reads all earlier pending members and all members at the same score to determine whether an admission barrier exists. Java also reads all active workflow-root fields with HKEYS. Bulk insertion has no explicit job/byte batch ceiling. The Redis scripting model blocks other server activity during script execution; a Lua call being atomic does not make its running time bounded. SCAN COUNT is a work hint rather than a strict result cap.
Use indexes that answer “earliest blocking EXCLUSIVE” and “next eligible member” directly; impose explicit input/work budgets and resume cursors. Benchmark long same-key SHARED runs, timestamp collisions, many active workflows, and large bulk calls while measuring unrelated Redis command latency. Document that the common
{threadmill}hash tag places all Threadmill keys in one Cluster slot: Cluster provides topology/failover integration here, not horizontal sharding of one namespace's job load.S03 — Give maintenance bounded work and independent progress
Recovery and workflow reconciliation page across entire state populations; reconciliation retains a
HashSetof every handled parent during the pass. Recurring materialization visits every definition, and the cron DROP path can walk all missed fires. Promotion is limited to 100 jobs per maintenance tick. The current caps and all-population scans need one coherent capacity model.Introduce resumable cursors and wall-time budgets per activity, separate correctness recovery from the hour-scale retention cadence, and expose oldest scheduled/recovery/retention lag. Test many dormant cron definitions, long downtime, large AWAITING/FAILED populations, and promotion bursts. Correctness work must not disappear behind expensive housekeeping.
S04 — Clean up concurrency metadata for keys that never return
The PostgreSQL group rows and Redis counter hashes can remain after their counts return to zero; no general reclamation for lifetime-distinct concurrency keys was found. Job retention alone does not bound this metadata. This matters when applications use one fresh key per business operation rather than a fixed tenant set.
Add bounded, race-safe reclamation only when no active hold or pending/outstanding job still needs the key; lock/verify against concurrent enqueue and claim. Include metadata cardinality in retention endurance assertions.
S05 — Use the application classloader consistently for payloads
Source-supported integration risk. SpringJobHandlerResolver uses the application context classloader, but JobRunner.deserializePayload uses the library classloader. Parent-library/child-application deployments can resolve a handler and still quarantine its payload as unknown.
Provide a coherent payload type-resolution strategy/classloader, retaining the existing load-without-initialization and assignability checks. Test an actual child-classloader application and Spring restart/deployment shape; do not simply turn on unrestricted polymorphic deserialization.
S06 — Close observability state on every execution exit
Source-supported lifecycle risk. ThreadmillTracing keys active scopes by job ID, not attempt, and assumes start/finish hooks always pair on one thread. Stale terminal-save paths return before success/failure hooks; orphan recovery can invoke failure hooks on a different thread. Metrics also maintain in-flight timing state that depends on finish hooks.
Tie lifecycle state to an execution/attempt token and guarantee cleanup in an engine finally path, including stale completion, deterministic save failure, shutdown, and overlapping orphan recovery. Avoid closing a thread-bound scope from another execution. Add regression tests that assert no leaked span/timer state and correct attempt attribution.
S07 — Make Redis search limitations explicit at the SPI boundary
Reproduced, lower impact. Redis
searchJobspages by state/time before applying queue/handler filters. The probe gets an empty first page although a matching job exists outside that initial unfiltered window. PostgreSQL/memory filter before pagination.The dashboard already checks
supportsRichSearch=false, which limits exposure. Decide whether unsupported SPI filters should throw clearly or be implemented consistently using indexes. Do not silently return a misleading filtered page. Document the exact supported search subset and add a cross-store contract or explicit unsupported-operation test.S08 — Make after-commit enqueue failure visible to applications
The default Spring mode writes after the business transaction commits; deferred enqueue errors are logged rather than delivered to the original caller, and a returned ID is not proof of durable enqueue. The detailed transaction guide largely explains this, but the main adoption path should make the boundary unavoidable.
Keep explicit transaction modes, prominently show
join_transactionfor PostgreSQL business-write/job-write atomicity, and expose an observable deferred-enqueue failure hook/counter. Document an application-owned durable outbox for cross-datastore requirements. Consider bounded batching of deferred enqueue callbacks. This is a durability contract decision, not a request to claim exactly-once execution.S09 — Freeze the 1.0 compatibility contract and prove upgrades
Before promising SemVer stability, review public
JobStoreoperations/defaults/capabilities, public engine mutators and configuration records, retry/finalization hooks, serialized fields, and dashboard DTOs together. Add old-version golden wire fixtures, versioned upgrade tests from v0.3.0, and an explicit rolling-upgrade/downgrade policy. Class names are durable type tags today; drain/migrate-before-rename must remain explicit.Keep PostgreSQL migrations additive/checksum-validated; add a Redis representation/index migration strategy for changes required by M01/M02/M03. Do not silently reinterpret existing keys. Verify both fresh initialization and nonempty upgrade, including workflows, dedup, recurring state/nudges, paused queues, and in-flight jobs.
Quick wins
These are small, coherent contributions. Some are also mandatory findings above; “quick” does not mean optional.
Errorpaths (M08).detailwhere the Java entry hasmessage, and the UI shows only state/time. Preserve server-side redaction while making failures actionable.AutoCloseablefor RedisJobStore before the API freeze.Release qualification still required
Real topology and durability evidence
The open issue #98 describes the need for real Sentinel/Cluster/authentication/TLS coverage. The current repository already has real Cluster contract and authenticated TLS/Sentinel smoke tests, so the issue's claim that coverage is only unit-level is stale. The remaining qualification is substantial: actual replica promotion, network disruption, topology refresh/resharding, and recovery under load. Startup/round-trip tests do not establish failover behavior.
Run a supported-version matrix with PostgreSQL 18 and the declared Redis minimum plus a newer version. Include real multi-node Sentinel/Cluster failover, verified TLS/ACLs, certificate rejection, wrong credentials, timeouts, noeviction/OOM, and managed-service CONFIG restrictions. Exercise process kills at claim, terminal write, retry handoff, workflow propagation, enqueue/nudge, and lease takeover.
Keep durability claims tied to Redis persistence and replication configuration. The current AOF/noeviction guidance is useful. Add an explicit acknowledged-write-loss/failover boundary and recovery-point expectations; server-side Lua atomicity alone does not imply durable acknowledged writes. See Redis persistence documentation.
Better tests for the failures that matter
Turn every reproduced audit case into a named permanent test. Cross-backend behavior belongs in the shared contract; Redis candidate/index details and PostgreSQL transaction cleanup also need backend regressions. Add generated/model-based sequences combining enqueue, claim, pause, delete, retry, replace, workflow completion, retention, and crash. Compare durable states, counters, hold ownership, and eventual progress against a small reference model.
Use fault injection at durable boundaries, including “write committed but acknowledgement lost.” For safety, assert no overlapping exclusive holders; for liveness, assert eventually runnable jobs are claimed and no abandoned PROCESSING row remains shielded. A zero-violation throughput run is not sufficient if it never generates the triggering combination.
Extend endurance to a steady retained population using shortened retention windows, accumulating concurrency/dedup keys, metrics/dashboard polling, realistic payload sizes, queue families, and node churn. Record p50/p95/p99 enqueue/claim/terminal latency, throughput, oldest queued/scheduled/retention age, database/Redis memory, JVM heap, and per-tick maintenance duration. Compare against a recorded baseline rather than a single jobs/second headline.
Suggested implementation sequence
check, required dependency scan, completeproductionCheck, browser tests, real datastore simulations, and Postgres/Redis endurance pass; retain artifacts with revision, versions, settings, and seed.I would not substitute a date or a feature-count target for these exit criteria. A 1.0 candidate should have no known violations of job ownership, concurrency exclusion, retry policy, workflow preservation, or sustainable retention, and every advertised topology should have fresh qualification evidence.
Overall recommendation
Keep the current architecture and concentrate the next release cycle on the interaction defects above. The fastest route to a trustworthy 1.0 is a series of small verified fixes around a few shared invariants, followed by measured operational and upgrade qualification. Broad rewrites or additional features would make that qualification harder without addressing the reproduced failures.
Redis qualification audit correction
The Redis preflight finished all 2,251 jobs and passed the harness invariants, but monitoring found a private-auditor bug: its HVALS-based active-group check treated 11
idle_sincetimestamps as live work. The original RDB was restored into an isolated read-only Redis instance; all 20 shared/exclusive counter pairs were zero, pending/workflow holds were zero, and job records, state indexes and counters agreed. The original run recorded zero evictions. The original failed audit remains intact alongsideaudit-review.jsonandsnapshot-audit-corrected.json, which explicitly accept preflight correctness/drain. Three auditor regressions now pass, including the preserved snapshot. No Threadmill source or frozen JAR changed.The original PostgreSQL campaign continues unchanged. Its supervisor already marked Redis blocked, so only the seven unrun Redis phases are armed under
.local-reference/qualification/20260909T2258Z-redis-remaining. That runner waits for the original PostgreSQL supervisor to exit, verifies the same 43-JAR manifest, and then uses new isolated volumes. No completed phase is repeated and no original datastore is reset. The five-minute monitor now tracks both roots. Endurance, performance and stability qualification remain pending.