Messaging and jobs redesign: one IMessageBus, durable jobs runtime, Redis Streams + AWS SQS/SNS transports

[ejsmith]

Adds a unified IMessageBus, typed background jobs through IJobClient, and shared monitoring for jobs and broker-delivered work.

Developer experience

Inject the ordinary application APIs:

await bus.SendAsync(new SendReceipt(1001, "dev@example.com"));
await bus.PublishAsync(new OrderPlaced(1001, "Espresso Machine"));

var job = await jobs.EnqueueAsync<ResizeImageJob, ResizeArgs>(
    new ResizeArgs("product.png", 640, 480));
var completed = await job.WaitForCompletionAsync(TimeSpan.FromSeconds(30));

• Messaging: competing consumers, durable service subscriptions, and separate best-effort SubscribeNodeAsync broadcasts. Implement IMessageHandler<T>; automatic batching keeps single-message calls efficient.
• Jobs: implement IJob or IJob<TArgs> for delays, progress, cancellation, persisted retries, and CRON. IScheduledJobManager supports schedule edits and manual runs.
• One job store: IJobRuntimeStore, JobState, and IJobMonitor cover ordinary jobs and tracked broker work, including paginated history, counts, metadata, heartbeats, and cancellation. Choose Jobs.UseInMemory() or Jobs.UseRedis().
• Delivery controls: scoped handlers, bounded concurrency, prefetch, lease renewal, graceful drain, health checks, and returned MessageOutcome values for success, retry, dead letter, or unsettled work. MessageProcessingContext supplies progress and cooperative cancellation.
• Operations: native Redis resource locks, queue statistics, dead-letter inspection/replay/deletion, and managed node-subscription cleanup. Fresh attempt tokens prevent stale updates from overwriting a newer attempt.

AddFoundatio() configures producers; AddFoundatioWorker(...) starts configured processing. The Quickstart runs without Docker; the messaging sample shows LocalStack, Redis, multiple replicas, and monitoring. Mediator #335 builds queued handlers and live events on these APIs.

Performance

Short deliveries avoid starting an async renewal loop; a timer still supervises expiry from admission. Median jobs/sec, previous native version → this update:

Workload	Before	After
In memory, concurrency 1	93,045	100,537
In memory, concurrency 8	84,916	91,161
In memory, concurrency 64	95,696	100,346
Redis tracking, longer check	7,656	7,341

Untracked memory throughput improves 5–8%, with about 5% fewer allocations and 17% less CPU at concurrency 64. Longer Redis checks were 4% slower, with 5% less CPU; no Redis/SQS speedup is claimed. System .NET 10.0.12, Release, repeated local runs. Workloads, latency and raw results. LocalStack does not measure production AWS capacity.

Additional details

• Broker records share the job store but cannot enter job-worker claims or recovery. Queue delivery is at least once; node broadcasts are best effort. Enqueue/replay are not transactions with business data.
• External references now preserve Release configuration; the integration's CI detects unoptimized assemblies.
• Full build and 2,235 tests pass (24 existing skips), including six lease lifecycle/race regressions, Redis/LocalStack conformance and Quickstart verification. 7.52M comparison jobs had no missing or duplicate deliveries. A ten-minute recovery run verified 69,354 accepted jobs, 40 cancellations, and recovery of all 32 interrupted deliveries, leaving no pending or failed jobs.

[ejsmith]

Pushed two commits addressing the messaging/jobs design-review feedback. All changes are green against the messaging, queue, and jobs suites (plus the conformance harness); full core suite passes.

fix: address messaging/jobs design-review feedback

Durable jobs + CRON

• Hosted runtime driver (JobRuntimeService + AddJobRuntimeService()): pumps occurrence materialization, due-dispatch, recovery, and queued-job execution. The runtime previously never ran end-to-end.
• Lease ownership enforced in TryTransitionAsync (expectedNodeId) so a stale worker can't overwrite the reclaiming node's terminal state.
• Lease renewal heartbeat during execution; the run is cancelled if the lease is lost (RenewClaimAsync was dead code).
• Strong node identity (machine:pid:token), capped exponential retry backoff.
• Replaced the hand-rolled cron parser with the vendored Cronos (moved into core Foundatio.Cronos); materialize every missed occurrence in the misfire window, not just the latest.

Messaging

• Resilient consumer/subscription loops — a poison message or transient receive error no longer silently kills the consumer.
• Fixed an in-memory push-path double-settle (tolerate already-settled receipts in the safety-net abandon).
• In-memory transport now honors visibility timeouts (reaper redelivers unsettled messages) so its advertised at-least-once guarantee is real.
• Log handler exceptions instead of swallowing them; drop the misleading write-only content-type header; add ReceiveDeadLetteredAsync so poison payloads are inspectable.

Conformance harness

• Silent capability skips → Assert.Skip; ordering assertions gated on the declared OrderingGuarantee; added visibility-timeout, competing-consumer, and DLQ-read scenarios.

refactor: hoist shared MessageQueue/PubSub behavior into MessageClientCore (M1, M2)

• Removed the large duplication between MessageQueue and PubSub via an internal MessageClientCore; unified the two handle classes and the two HandleMessageAsync overloads. Pub/sub subscriptions now also honor RedeliveryBackoff (prior drift).
• Enforce ITransportInfo.MaxBatchSize by chunking oversized sends.

Still intentionally not addressed (no external provider on this branch): proving the harness against a real transport before cutting the public API — that's the phased-rollout gate from the plan.