diff --git a/.agents/skills/foundatio/SKILL.md b/.agents/skills/foundatio/SKILL.md index 913fa8b15..8bf7a3a17 100644 --- a/.agents/skills/foundatio/SKILL.md +++ b/.agents/skills/foundatio/SKILL.md @@ -2,95 +2,101 @@ name: foundatio description: > Use when working with Foundatio infrastructure abstractions for .NET -- caching, - queuing, messaging, file storage, distributed locking, or background jobs. Apply - when using ICacheClient, IQueue, IMessageBus, IFileStorage, ILockProvider, IJob, + messaging, background jobs, file storage, distributed locking, or queuing. Apply + when using ICacheClient, IMessageBus, IJobClient, IJob, IFileStorage, ILockProvider, or resilience patterns like retry and circuit breakers. Covers in-memory and - production implementations (Redis, Azure, AWS, Kafka, RabbitMQ). Use context7 - MCP to fetch current API docs and examples. + production implementations (Redis, AWS, Azure). Use context7 MCP to fetch current + API docs and examples. --- # Foundatio -Pluggable infrastructure abstractions for distributed .NET apps. Interface-first, testable, swappable between in-memory (dev/test) and production providers (Redis, Azure, AWS) with zero application code changes. +Pluggable infrastructure abstractions for distributed .NET apps. Interface-first, testable, swappable between in-memory (dev/test) and production providers (Redis, AWS, Azure) with zero application code changes. ## Documentation via context7 Use context7 MCP for complete, up-to-date API docs and examples. The main library ID covers all abstractions and implementations: ```text -query-docs(libraryId="/foundatiofx/foundatio", query="How to configure queue retry policies and dead letter handling") +query-docs(libraryId="/foundatiofx/foundatio", query="How to configure messaging retry policies and dead letter handling") ``` Query with specific questions, not single keywords. All provider docs (Redis, Azure, AWS, Kafka, etc.) are included in the main library. +## Messaging and Jobs (current API) + +- One messaging client: `IMessageBus` in `Foundatio.Messaging`. `SendAsync` targets competing queue consumers; `PublishAsync` fans out to existing event subscriptions. Delivery is at least once where supported, so handlers must tolerate duplicates. Both return application IDs, independently of broker IDs. Supply `MessageSendOptions.MessageId` / `MessagePublishOptions.MessageId` for retry correlation; this does not create exactly-once delivery. Batches return IDs in input order; `MessageSendException.Outcomes` distinguishes accepted, unknown, and unattempted inputs on failure. +- Publish has real pub/sub DROP semantics: a publish to a topic with no existing subscriptions is dropped (subscriptions are created when handlers subscribe or via topology provisioning -- subscribers must exist before the publish). A sent command waits durably on its queue instead. The in-memory transport warns once per topic on zero-subscription drops, and the core logs every produce at debug. +- Implement `IMessageHandler` and explicitly register `.Messaging.AddConsumer()` for queued work or `.Messaging.AddSubscriber("billing")` for events. Each message uses its own DI scope. Dynamic equivalents are `ConsumeAsync` and `SubscribeAsync`, returning an `IMessageSubscription` with its structural `Source` address, Status, RecoveryVersion and WaitUntilReadyAsync. Transient temporary-lease renewal errors retry; definite loss recreates the listener. Derived local state must resynchronize after a recovery gap; HybridCacheClient clears its local cache automatically. +- `MessageConsumerOptions` sets an optional queue destination. MessageTypeName plus Destination/Topic on a handler binds its wire name and producer route; AddMessageType(name, queue: ..., topic: ...) does the same for producers. GetRouteMaps() and startup logs expose mappings; startup validates duplicate wire names even when topology mode is None. `MessageSubscriptionOptions` sets an optional topic and optional durable subscription name (declarative default: UseServiceName, then hosting ApplicationName): replicas using the same name compete. In dynamic SubscribeAsync, null creates a temporary listener with a renewable expiration lease on in-memory/Redis; AWS requires a durable name. Shared `MessageHandlerOptions` controls endpoint concurrency (default 1), retries and acknowledgement. Duplicate concrete handlers and multiple interface/raw fallback handlers on one endpoint are rejected. Manual acknowledgement holds its concurrency slot until settlement. +- Routing is central: `.Messaging.ConfigureRouting(r => r.UseDefaultQueue(...).UseDefaultTopic(...).MapQueue(...).MapTopic(...).UseConvention(...))`. Precedence: operation override > exact map > interface/base-type map > `MessageRouteAttribute` > configured default > convention > kebab-cased type name. Producer routing declares queues/topics, never phantom subscriber groups. +- Routing config doubles as topology declarations (`DestinationDeclaration` with a canonical `DestinationAddress` -- `ForQueue`/`ForTopic`/`ForSubscription`). `IMessageTopology` exposes `GetDeclarations()` / `EnsureAsync()` / `ValidateAsync()`. `.Messaging.ConfigureTopology(TopologyMode.Ensure | Validate | None)` picks whether the client creates missing destinations (default), only verifies they exist (throws at startup when missing), or never touches topology; AddMessageConsumers includes startup topology; producers can opt in with AddMessagingTopology. Registering a transport starts no hosted services. +- The CORE owns retry/dead-lettering identically on every transport: default `RetryPolicy` is `MaxAttempts` 5 with immediate-then-10s/20s/30s backoff (+/-20% jitter); configure via `.Messaging.ConfigureRetry(p => p with { ... })`. Dead-lettered messages go to the transport's native sink or a derived `"{source}.deadletter"` destination, stamped with `message.dead_letter.*` forensics headers (`KnownHeaders.DeadLetter*`). Never configure broker-native redrive policies. +- Settlement succeeds only after the broker operation succeeds. A failed DLQ write leaves the original unsettled. `IMessageContext` exposes application `Id`, diagnostic `BrokerMessageId`, `CompleteAsync`, `RejectAsync`, and cancellation. Expiring delivery leases are supervised and renewed while a handler runs; lease loss cancels the handler and prevents settlement. Direct loops use `await using var message = await bus.ReceiveAsync(options, token)`; disposal returns unfinished work for redelivery. Raw receive requires an explicit destination. +- Transports advertise per-destination capabilities: `ITransportInfo.GetCapabilities(destination)` takes the `DestinationAddress` in question (most transports answer by its role) and returns `TransportCapabilities` (e.g. the AWS transport's queue role has a native 15-minute `MaxDeliveryDelay`; its topic role has none). Delays beyond a ceiling and store-parked retries fall back to the durable runtime store (`IScheduledDispatchStore`, satisfied by any `IJobRuntimeStore`) and are drained by an explicitly hosted ScheduledMessageDispatcher -- never silently truncated. +- Durable jobs: implement `IJob` (`Task RunAsync(JobExecutionContext context)`). `JobResult` is an immutable record -- return the shared `JobResult.Success`/`JobResult.Cancelled` statics or the `SuccessWithMessage`/`FailedWithMessage`/`CancelledWithMessage`/`FromException` factories (there is no `None`). `IJobClient.EnqueueAsync()` / `EnqueueAsync(args)` (typed payloads) returns a `JobHandle`; `IJobMonitor` queries state; `IJobWorker` executes with per-run DI scopes, bounded concurrency, and supervised lease renewal. `JobExecutionContext` gives `JobId`/`Attempt`/`CancellationToken`, `GetArguments()`, `ReportProgressAsync`, `RenewLeaseAsync`, `IsCancellationRequestedAsync`; its public constructor makes a detached context for tests. `GetArguments` enforces the stored payload-type discriminator: requesting a different type than the job was enqueued with throws before deserialization. Hand-wiring outside DI: `JobWorker`/`JobScheduleProcessor` take `JobWorkerOptions`/`JobScheduleProcessorOptions` records for their optional dependencies. +- CRON: `.Jobs.AddCronJob(cron)` or `.Jobs.AddCronJob(cron,args)`; typed jobs implement `IJob`. Schedules persist wire names, serialized payloads, time-zone IDs, retry budgets, and revisions. `ConfigurationVersion` must increase for a changed declaration; same-version restarts preserve runtime edits. `ScheduleAsync` uses revision checks. Global and per-node occurrences share the same job worker/state machine. Global is the default. PerNode requires Jobs.ConfigureWorker(o => o with { NodeId = ... }) or FOUNDATIO_NODE_ID; unclaimed occurrences expire after configurable UnclaimedLifetime (one day). Cache confirmed materializations only: OverlapBlocked must be retried within the misfire window. +- `AddFoundatioWorker` validates missing transports/stores during registration. Receiving options, durable names, concrete job types, and schedule options also fail at registration. With individually hosted roles, startup validation fails fast at boot with actionable messages: CRON jobs registered without a runtime store, or handlers registered without a transport, throw when the corresponding consumer/scheduler host starts (add `.Jobs.UseInMemory()` / `.Messaging.UseInMemory()` or the production `Use*`). +- Jobs exceptions on the trigger/resolve paths: `ScheduledJobNotFoundException` (unknown schedule name), `ScheduledJobDisabledException` (triggering a disabled schedule), and `JobException` (unresolvable job type); all derive from `JobException` : `InvalidOperationException`. +- Schedule management: `IScheduledJobManager` supports inspect, revision-checked updates, enable/disable, reschedule, remove, and manual trigger. The DI-configured manager rejects unregistered job types before saving a schedule. Manual triggers respect disabled/overlap policy. Removing a definition does not cancel already queued jobs. +- Stable wire names: `.Messaging.AddMessageType("order-created.v1")` and `.Jobs.AddJobType("name")` preserve persisted discriminators across refactors. Sends retain the runtime concrete type in the envelope while the declared type selects the route. Interface, abstract and object receivers resolve only explicitly registered names; it never scans loaded assemblies. Concrete handlers can use the default CLR full name. Producers and consumers must use the same serializer/content type. SystemTextJson defaults to application/json; other serializers default to byte-safe application/octet-stream unless ContentType is explicitly configured. Metadata and application IDs survive scheduling and dead-lettering. Dispatch IDs are independently generated; repeated application IDs do not deduplicate sends. Batch MessageBatchItem supplies per-input IDs. Indexed outcomes distinguish Accepted, Rejected, Unknown and NotAttempted; retain error/retryability details. AWS uses native batches of ten; Redis pipelines bounded batches of 64 by default. +- Legacy implementations were removed. For migration, `Messaging.AddLegacyAdapter()` registers the old `IMessageBus`/`IMessagePublisher`/`IMessageSubscriber` interfaces as a thin adapter over the new bus (old handler code compiles unchanged; delete the call when migrated). Old jobs migrate mechanically: `RunAsync(CancellationToken)` becomes `RunAsync(JobExecutionContext)` (use `context.CancellationToken`), `QueueJobBase`/`IQueue` become `IMessageHandler` + `SendAsync`, and `WorkItemJob` becomes `EnqueueAsync(args)` with `ReportProgressAsync`. + ## Core Interfaces | Interface | Purpose | In-Memory | Production | | --------- | ------- | --------- | ---------- | | `ICacheClient` | Key-value caching with TTL | `InMemoryCacheClient` | Redis, Hybrid | -| `IQueue` | FIFO message queuing | `InMemoryQueue` | Redis, SQS, Azure | -| `IMessageBus` | Pub/sub messaging | `InMemoryMessageBus` | Redis, Kafka, RabbitMQ, Azure | +| `IMessageBus` | Commands (`SendAsync`) + events (`PublishAsync`) over one client | `InMemoryMessageTransport` | Redis Streams, AWS SQS/SNS | +| `IJobClient` / `IJobMonitor` | Submit and observe durable background jobs | `InMemoryJobRuntimeStore` | `RedisJobRuntimeStore` | | `IFileStorage` | File storage abstraction | `InMemoryFileStorage` | S3, Azure Blob, Minio | | `ILockProvider` | Distributed locking | `CacheLockProvider` | Redis-backed | -| `IJob` | Background job processing | N/A | Hosted services | | `ISerializer` / `ITextSerializer` | Binary and text serialization | `SystemTextJsonSerializer` | MessagePack, JsonNet | | `IResiliencePolicy` | Retry, circuit breaker, timeout | `ResiliencePolicyBuilder` | N/A | ## DI Registration -All services are **singletons** (maintain internal state/connections). Jobs are scoped. +Use `AddFoundatioWorker(configure)` from Foundatio.Extensions.Hosting (namespace Foundatio) for combined workers; put transport, store, handler, and job registrations in its callback. It hosts consumers, registered jobs and their scheduler, and delayed dispatch when a dispatch store is configured. Use the inert `AddFoundatio()` builder for producer-only apps and manual tests. Infrastructure services register as **singletons**. Handlers and jobs resolve in their own DI scope per message/run, so they can inject scoped dependencies. ```csharp var builder = WebApplication.CreateBuilder(args); -// Quick start -- all in-memory defaults -builder.Services.AddFoundatio(); - -// Or register individually with options -builder.Services.AddSingleton(sp => - new InMemoryCacheClient(o => o.MaxItems(1000) - .LoggerFactory(sp.GetRequiredService()))); -builder.Services.AddSingleton(); -builder.Services.AddSingleton(); -builder.Services.AddSingleton>(sp => - new InMemoryQueue()); - -// Lock provider (message bus optional but enables faster lock release via pub/sub) -builder.Services.AddSingleton(sp => - new CacheLockProvider( - sp.GetRequiredService(), - sp.GetService(), - sp.GetService(), - sp.GetService())); +builder.Services.AddFoundatioWorker(foundatio => foundatio + .Caching.UseInMemory() + .Storage.UseFolder("data") + .Locking.UseCache() + .Messaging + .ConfigureRouting(r => r + .MapQueue("orders") + .MapTopic("order-events", typeof(IOrderEvent))) + .ConfigureRetry(p => p with { MaxAttempts = 5 }) + .UseInMemory() + .AddConsumer() + .Builder.Jobs.UseInMemory() + .AddJobType("search.rebuild")); ``` -Swap to production by changing only DI registration: +Swap to production by changing only the provider lines: ```csharp -// DI owns and disposes the shared multiplexer during host shutdown. -builder.Services.AddSingleton(sp => - ConnectionMultiplexer.Connect("localhost:6379")); +builder.Services.AddFoundatio() + .Messaging.UseRedis(connectionString: "localhost:6379") // Redis Streams transport + .Builder.Jobs.UseRedis(); // Redis job runtime store -builder.Services.AddSingleton(sp => - new RedisCacheClient(o => - { - o.ConnectionMultiplexer = sp.GetRequiredService(); - o.LoggerFactory = sp.GetRequiredService(); - })); -builder.Services.AddSingleton(sp => - new RedisMessageBus(o => - { - o.Subscriber = sp.GetRequiredService().GetSubscriber(); - o.LoggerFactory = sp.GetRequiredService(); - })); -builder.Services.AddSingleton>(sp => - new RedisQueue(o => - { - o.ConnectionMultiplexer = sp.GetRequiredService(); - o.LoggerFactory = sp.GetRequiredService(); - })); +// or AWS (SQS queues, SNS+SQS pub/sub; point ServiceUrl at LocalStack for local dev) +builder.Services.AddFoundatio() + .Messaging.UseAws(o => o.ResourcePrefix = "myapp"); ``` +Custom providers plug in via `.Messaging.UseTransport(...)` (any `IMessageTransport`) and `.Jobs.UseRuntimeStore(...)` (any `IJobRuntimeStore`). The zero-dependency starting point is `samples/Foundatio.QuickstartSample` in the Foundatio repo -- a generic-host console app running messaging and jobs fully in-memory with plain `dotnet run`. + +- Prefer ConfigureMessaging(m => ...) and ConfigureJobs(j => ...) blocks. Every messaging/job method returns its feature builder; .Builder returns to the root. AddSubscriber defaults to one durable subscription per service, while dynamic unnamed SubscribeAsync remains temporary. +- Messaging.UseInMemory/UseRedis supply a matching scheduled dispatch store without registering jobs. The automatic Redis store inherits transport connection, clock and KeyPrefix; configure its budgets with RedisStreamsMessageTransportOptions.Scheduling. AWS needs UseSchedulingStore for non-native delays. HybridCacheClient requires temporary subscriptions and fails immediately on AWS; CacheLockProvider falls back to polling. +- JobRequestOptions supports mutually exclusive Delay/RunAt, MaxAttempts and a persisted JobRetryPolicy (10s initial, multiplier 2, 5min cap, 20% jitter). A failed JobResult with Retryable=false is terminal. JobState.ResultMessage holds success text; Error is reserved for failures. JobHandle.WaitForCompletionAsync defaults to a five-minute wait; cancelling the wait does not cancel work. Context helpers inherit the execution cancellation token by default. +- Hosted job slots replenish independently; RunQueuedAsync remains a bounded drain. Jobs are scoped and disposed, including fallback activation. Shutdown returns owned unsettled messages with a bounded independent token; a lost lease cannot settle replacement work. In-memory transport uses finite visibility and shared pull concurrency. +- AWS automatically coalesces concurrent sends/publishes/deletes with bounded per-destination buffers; completion still requires each broker result. Caller cancellation never cancels a shared batch's other inputs and may leave an Unknown send outcome after dispatch. AWS collects partial operation batches for 2 ms, flushes acknowledgements once the observed receive capacity is filled, and uses up to four overlapping receives with a shared consumer capacity budget. Provider authors can advertise MaxReceiveBatchSize, MaxConcurrentReceives and ReceiveBatchDelay in TransportCapabilities; other providers default to one receive and no coalescing delay. Settled-handler cleanup is separately bounded and drained on shutdown. The versioned fnd.envelope AWS attribute retains readable bodies and all headers. NativeMessageHeaders optionally duplicates up to nine selected headers for SNS filters (empty by default); reserved/invalid names fail at construction and the native-name list is snapshotted; new readers accept legacy envelopes, but old experimental readers cannot read new sends. +- AddFoundatioWorker registers the foundatio health check and Foundatio.Runtime capacity gauges; subscriptions and infrastructure recovery affect health. Malformed AWS envelopes retain raw evidence and are quarantined per entry. Unmatched types back off five seconds with jitter instead of hot-looping. + ## Usage Patterns ### Caching @@ -106,29 +112,39 @@ await _cache.IncrementAsync("requests:today", 1); await _cache.RemoveByPrefixAsync("user:"); ``` -### Queues +### Messaging + +The verb carries the delivery semantic; handlers never choose queue vs. topic: ```csharp -await _queue.EnqueueAsync(new OrderWorkItem { OrderId = orderId }); +// Command: exactly one handler instance across the fleet processes it. +await _bus.SendAsync(new ResizeImage(imageId)); -var entry = await _queue.DequeueAsync(TimeSpan.FromSeconds(5)); -if (entry is not null) +// Event: every subscribing service receives one copy. +await _bus.PublishAsync(new OrderSubmitted(orderId)); +``` + +```csharp +public class SendConfirmationHandler : IMessageHandler { - await ProcessAsync(entry.Value); - await entry.CompleteAsync(); // success - // or: await entry.AbandonAsync(); // retry later + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + => _email.SendConfirmationAsync(context.Message.OrderId, cancellationToken); } + +services.AddFoundatio() + .Messaging.AddConsumer(o => + { + o.MaxConcurrency = 4; // default 1; retries may still reorder work + o.DeadLetterOn(); // retries cannot fix validation failures + }); ``` -### Messaging (Pub/Sub) +Throwing from `HandleAsync` triggers the core retry/dead-letter policy. With `AckMode.Manual`, settle explicitly: ```csharp -await _messageBus.SubscribeAsync(async (msg, ct) => -{ - await HandleOrderCreatedAsync(msg, ct); -}); - -await _messageBus.PublishAsync(new OrderCreated { OrderId = orderId }); +await context.CompleteAsync(); +await context.RejectAsync(new RejectOptions { RedeliveryDelay = TimeSpan.FromSeconds(30) }); +await context.RejectAsync(new RejectOptions { Terminal = true, Reason = "malformed" }); ``` ### File Storage @@ -144,7 +160,8 @@ await _storage.DeleteFilesAsync("reports/old-*"); ### Distributed Locks ```csharp -await using var lck = await _locker.AcquireAsync( +// TryAcquireAsync returns null when the lock is unavailable; AcquireAsync throws instead. +await using var lck = await _locker.TryAcquireAsync( "resource:order-123", timeUntilExpires: TimeSpan.FromMinutes(1)); @@ -172,147 +189,130 @@ await policy.ExecuteAsync(async ct => ## Jobs -### Standard Job +### Durable Job + +Implement `IJob`; enqueue through `IJobClient`. Arguments are typed and persisted with the job: ```csharp -public class CleanupJob : JobBase +public class RebuildSearchIndexJob : IJob { - public CleanupJob( - TimeProvider timeProvider, - IResiliencePolicyProvider resiliencePolicyProvider, - ILoggerFactory loggerFactory = null) - : base(timeProvider, resiliencePolicyProvider, loggerFactory) { } - - protected override async Task RunInternalAsync(JobContext context) + public async Task RunAsync(RebuildSearchIndexArgs args, JobExecutionContext context) { - await CleanupOldRecordsAsync(context.CancellationToken); + + await context.ReportProgressAsync(10, "starting"); + foreach (var batch in GetBatches(args.Index)) + { + if (await context.IsCancellationRequestedAsync()) + return JobResult.Cancelled; + + await IndexBatchAsync(batch, context.CancellationToken); + } + return JobResult.Success; } } + +JobHandle handle = await _jobs.EnqueueAsync( + new RebuildSearchIndexArgs { Index = "orders" }); +JobState? state = await handle.GetStateAsync(); +await handle.RequestCancellationAsync(); ``` -### Job with Lock (Singleton / Leader Election) +Workers claim only registered job types, with a fresh ownership token and a DI scope per run. Leases are supervised; stale tokens cannot mutate a replacement execution. Host interruption returns work to the queue; explicit cancellation is terminal. Persisted MaxAttempts defaults to three; failures use bounded exponential backoff and end in Failed when exhausted. Execution is at least once: protect external side effects with application idempotency. -`JobWithLockBase` acquires a distributed lock before each run. If the lock isn't available the run is cancelled. Implements `IJobWithOptions`. +`IJobMonitor.QueryAsync` returns a bounded `JobPage` ordered by ID; pass ContinuationToken as JobQuery.AfterJobId until null, including after empty filtered pages. Hosted workers clean terminal history older than seven days; manual hosts call CleanupAsync. JobRuntimeStoreOptions separates 100,000 active jobs, 100,000 history records, 1,000,000 deduplication reservations, and 100,000 scheduled messages. History/deduplication default to seven days; history eviction preserves the separate ID reservation. Payload limit defaults to 1 MiB (scheduled messages include UTF-8 header keys/values). GetStatsAsync reports usage. Configure RedisJobRuntimeStoreOptions.Runtime or Jobs.UseInMemory(options). -```csharp -[Job(Description = "Singleton maintenance", Interval = "5s")] -public class MaintenanceJob : JobWithLockBase -{ - private readonly ILockProvider _lockProvider; +### CRON Job - public MaintenanceJob( - ICacheClient cache, IMessageBus messageBus, - TimeProvider timeProvider, IResiliencePolicyProvider resiliencePolicyProvider, - ILoggerFactory loggerFactory) : base(timeProvider, resiliencePolicyProvider, loggerFactory) +```csharp +services.AddFoundatio() + .Jobs.UseInMemory() + .AddCronJob("0 2 * * *", new ExportArgs { Format = "csv" }, o => { - _lockProvider = new CacheLockProvider(cache, messageBus, loggerFactory); - } + o.Scope = ScheduledJobScope.Global; // one instance per tick (default); PerNode = every instance + o.MaxAttempts = 3; // TOTAL run attempts per failed occurrence + o.ConfigurationVersion = 1; + }); +``` - // new CancellationToken(true) = try once, skip if lock is held - protected override Task GetLockAsync(CancellationToken cancellationToken) => - _lockProvider.AcquireAsync(nameof(MaintenanceJob), TimeSpan.FromMinutes(15), - cancellationToken: new CancellationToken(true)); +Start `services.AddJobScheduler()` to reconcile definitions and materialize due occurrences; start `services.AddJobWorker()` to execute them. Registering the store starts neither. - protected override async Task RunInternalAsync(JobContext context) - { - await DoMaintenanceAsync(context.CancellationToken); - return JobResult.Success; - } -} -``` +### Migrating old jobs -### Queue Processor Job +`JobBase`/`QueueJobBase`/`JobWithLockBase`/`JobRunner`/`WorkItemJob` and the hosted `AddJob`/`AddDistributedCronJob` infrastructure were removed. The mappings are mechanical: an old job's `RunAsync(CancellationToken)` becomes `RunAsync(JobExecutionContext)` (use `context.CancellationToken`; `JobResult` is unchanged); a `QueueJobBase` becomes an `IMessageHandler` fed by `SendAsync`; a `WorkItemJob` handler becomes a job enqueued with `EnqueueAsync(args)` reporting progress via `context.ReportProgressAsync`; distributed CRON is `.Jobs.AddCronJob(cron)` on the durable runtime. -```csharp -public class OrderProcessorJob : QueueJobBase -{ - public OrderProcessorJob( - IQueue queue, - TimeProvider timeProvider, - IResiliencePolicyProvider resiliencePolicyProvider, - ILoggerFactory loggerFactory = null) - : base(queue, timeProvider, resiliencePolicyProvider, loggerFactory) { } - - protected override async Task ProcessQueueEntryAsync( - QueueEntryContext context) - { - var item = context.QueueEntry.Value; - await ProcessOrderAsync(item.OrderId, context.CancellationToken); - return JobResult.Success; - } -} -``` +## Testing -### Hosting Integration +### Messaging: Foundatio.Testing harness -Requires `Foundatio.Extensions.Hosting` package: +`Foundatio.Testing` runs the real `IMessageBus` over a recording in-memory transport -- deterministic tests without sleeps, including the retry/dead-letter path: ```csharp -builder.Services.AddJob(o => o.WaitForStartupActions()); -builder.Services.AddCronJob("0 */6 * * *"); -builder.Services.AddDistributedCronJob("0 */6 * * *"); +services.AddFoundatio() + .Messaging.UseTestHarness() + .AddSubscriber("confirmation"); +services.AddMessageConsumers(); + +// resolve MessagingTestHarness from the container; start hosted services, then: +await bus.PublishAsync(new OrderPlaced(42)); +await harness.WaitForIdleAsync(); // blocks until queues and in-flight handlers drain + +Assert.Single(harness.Published()); +Assert.Single(harness.Handled()); +Assert.Empty(harness.DeadLetteredMessages); ``` -## Testing +Recordings: `SentMessages` / `PublishedMessages` / `HandledMessages` / `AbandonedMessages` (retries) / `DeadLetteredMessages`, with typed accessors `Sent()`, `Published()`, `Handled()`, `Abandoned()`, `DeadLettered()`. To await one outcome without draining the whole bus: `WaitForHandledAsync(count)` (returns the handled messages) and `WaitForDeadLetteredAsync(count)` (returns raw `RecordedMessage`s -- assert `Reason`/`Attempts`). `DestinationsWithNoConsumer` lists destinations that received messages nothing consumed -- the usual reason a test is "idle immediately and Handled is empty". -Use `Foundatio.Xunit.v3` for test logging and DI integration. Two base classes: +The harness polls in REAL time (25ms cadence) regardless of any injected `TimeProvider`, while delayed redeliveries execute on the injected `TimeProvider` -- a faked clock must be advanced manually or waits time out. For sleep-free retry tests prefer `RedeliveryBackoff = _ => TimeSpan.Zero` on the subscription instead of faking the clock. -- **`TestWithLoggingBase`** -- lightweight, no DI container. `_logger` (`ILogger`) for logging; `Log` (`ILoggerFactory`) for passing to Foundatio services. -- **`TestLoggerBase`** -- full DI via `TestLoggerFixture`. Override `ConfigureServices` to register services. `Log` (`ILogger`) for logging; `TestLogger` (`ILoggerFactory`) for passing to Foundatio services. +### Jobs: JobsTestHarness + +`.Jobs.UseTestHarness()` registers `JobsTestHarness`: the real in-memory job runtime without hosted workers, so the test decides exactly when work runs. ```csharp -using Foundatio.Caching; -using Foundatio.Xunit; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Logging; -using Xunit; +services.AddFoundatio().Jobs.UseTestHarness(); +var harness = provider.GetRequiredService(); -public class OrderServiceTests : TestLoggerBase -{ - public OrderServiceTests(ITestOutputHelper output, TestLoggerFixture fixture) - : base(output, fixture) { } +var handle = await harness.Client.EnqueueAsync(); +await harness.RunAllQueuedAsync(); // drains currently eligible jobs across batches; future retries remain queued +await harness.RunDueAsync(fixedNow); // materializes due CRON occurrences, then drains eligible jobs +var state = await harness.RunToCompletionAsync(handle); // drives one job to its terminal state +``` - protected override void ConfigureServices(IServiceCollection services) - { - services.AddSingleton(sp => - new InMemoryCacheClient(o => o.LoggerFactory(TestLogger))); - services.AddSingleton(); - } +`Client` (`IJobClient`), `Schedules` (`IScheduledJobManager`), and `Monitor` (`IJobMonitor`) expose the enqueue/manage/assert surface. For running an `IJob` directly without any runtime, `new JobExecutionContext(cancellationToken, arguments: myArgs)` builds a detached context -- progress/lease helpers no-op and `GetArguments()` returns the supplied object. - [Fact] - public async Task GetStatusAsync_WithCachedOrder_ReturnsCachedStatus() - { - // Arrange - var cache = Services.GetRequiredService(); - await cache.SetAsync("order:123", "shipped"); - Log.LogInformation("Seeded cache with order status"); +### Test logging via Foundatio.Xunit.v3 - // Act - var status = await Services.GetRequiredService() - .GetStatusAsync("123"); +Two base classes: - // Assert - Assert.Equal("shipped", status); - } -} -``` +- **`TestWithLoggingBase`** -- lightweight, no DI container. `_logger` (`ILogger`) for logging; `Log` (`ILoggerFactory`) for passing to Foundatio services. +- **`TestLoggerBase`** -- full DI via `TestLoggerFixture`. Override `ConfigureServices` to register services. `Log` (`ILogger`) for logging; `TestLogger` (`ILoggerFactory`) for passing to Foundatio services. + +### Custom providers + +The transport contract is documented on the interfaces themselves (`IMessageTransport` + `ISupports*`): settle semantics (stale receipts SHOULD throw `ReceiptExpiredException`, but the signal is best-effort), the per-delivery `Receipt` token (never settle by entry identity alone), and the growth rule that contract changes only ever add optional init members. + +Validate a custom transport or job store against the shared conformance suites in `Foundatio.TestHarness`: inherit `MessageTransportConformanceTests` (override `CreateTransport`) and `JobRuntimeStoreConformanceTests` (override `CreateStore`). Tests skip automatically for unimplemented optional interfaces or unavailable backends. The suites pin per-message ids (distinct, positionally aligned in batch results), content-type round-trip, and non-destructive dead-letter inspection with explicit deletion/replay. ## Gotchas -- **Lock returns null**: `TryAcquireAsync` returns `null` when the lock cannot be acquired -- always guard with `is not null` before doing work. `AcquireAsync` throws `LockAcquisitionTimeoutException` instead of returning null. +- **Redis lifetime**: UseRedis registers a shared container-owned multiplexer by default. Custom connections should use a singleton factory so DI disposes them after hosted work stops. An already-created singleton instance stays caller-owned; dispose the host before disposing that connection. +- **Shared Redis connection**: messaging and jobs share one multiplexer. Configure ConnectionStrings:Redis, provide one explicit UseRedis connection string, or register the multiplexer. Conflicting explicit strings fail at registration; omit connectionString when using an existing multiplexer. + +- **Explicit receiving intent**: `AddConsumer` registers queued work; `AddSubscriber(..., "stable-group")` registers a durable event subscription. Replicas in the same group compete. DI AddSubscriber defaults to UseServiceName or IHostEnvironment.ApplicationName; set an explicit nonblank name to override; use AddTemporarySubscriber explicitly for temporary listeners. Dynamic unnamed subscriptions require expiring-subscription support (in-memory/Redis); AWS requires a durable name. +- **Do not configure broker redrive policies**: the core owns retry/dead-lettering (SQS `maxReceiveCount`, DLX, etc. would split authority and make behavior transport-specific). +- **Hosting is explicit**: AddFoundatioWorker(configure, jobConcurrency: 1) hosts the roles selected by its callback. Plain AddFoundatio client/storage registrations start no services. For split deployments, add `AddMessageConsumers`, `AddJobWorker(concurrency)`, `AddJobScheduler`, and/or `AddScheduledMessageDispatcher` only where each role should run. Workers, schedulers, and dispatchers are independent. `AddMessagingTopology` is available for producer-only startup checks. +- **Delayed sends beyond transport ceilings need a runtime store**: e.g. > 15 min on SQS, or any delayed publish on SNS topics. Without a store the operation fails loudly rather than truncating the delay. +- **`WaitForIdleAsync` ignores store-parked work**: delayed sends/retries parked in the runtime store are not transport activity -- drain them via ScheduledMessageDispatcher before asserting. +- **Lock returns null**: `TryAcquireAsync` returns `null` when the lock cannot be acquired -- always guard with `is not null`. `AcquireAsync` throws `LockAcquisitionTimeoutException` instead of returning null. - **Dispose streams and locks**: `ILock` is `IAsyncDisposable` -- use `await using`. Streams from `GetFileStreamAsync` are `IDisposable` -- use `using var`. -- **Cache TTL floor**: Expiration values below 5ms are treated as already-expired and the key is silently removed. If you compute TTL dynamically (e.g., `expiresAt - now`), guard against near-zero values. -- **Cache `GetAsync` returns `CacheValue`**: Check `result.HasValue` before accessing `result.Value`. A missing key returns `HasValue = false`, not an exception. -- **Cache stampede (thundering herd)**: The cache-aside pattern (`Get` -> miss -> load -> `Set`) is vulnerable to stampedes when a popular key expires and many callers regenerate simultaneously. Use `CacheLockProvider` to serialize regeneration: acquire a lock keyed on the cache key, double-check the cache after acquiring, and only then call the backing store. See the [Cache Stampede Protection](https://foundatio.readthedocs.io/guide/caching.html#cache-stampede-protection) docs for the full pattern. -- **Queue auto-complete**: `QueueJobBase` auto-completes entries based on `JobResult` by default. Set `AutoComplete = false` only when you need manual `CompleteAsync()`/`AbandonAsync()` control. Manual `DequeueAsync` does NOT auto-complete. -- **GetQueueEntryLockAsync error handling**: If `GetQueueEntryLockAsync` returns `null`, the queue entry is abandoned. If it throws, the entry is also abandoned and a `JobResult.FromException` is returned. Use `TryAcquireAsync` (not `AcquireAsync`) in your override since the return type is `Task`. -- **Failure semantics depend on job type**: `JobResult` only has `IsSuccess` -- there is no separate "failed but don't retry" status. For **queue-processed jobs** (`QueueJobBase.ProcessQueueEntryAsync`, or setting `context.Result` in a `WorkItemJob` handler), a non-success result triggers `AbandonAsync`, which re-queues the entry and eventually dead-letters it after `Retries` is exhausted -- reserve `FailedWithMessage`/`FromException` for transient errors you want retried, and log + return `JobResult.Success`/`SuccessWithMessage(...)` for permanent errors to avoid a pointless retry loop. For **standalone/manual jobs** (`JobBase`, a one-off `RunAsync()`/`RunInConsoleAsync()` run, or scheduled/cron jobs via `Foundatio.Extensions.Hosting`), there is no built-in retry or dead letter queue -- a failed result just produces an error-level log, a non-zero exit code from `RunInConsoleAsync`, or a failed entry in the job run history. Returning `FailedWithMessage`/`FromException` there is correct even for permanent errors, since nothing inside Foundatio will retry it. -- **JobWithLockBase vs manual locking**: Use `JobWithLockBase` when the entire run must be single-instance (leader election). Use manual `ILockProvider.AcquireAsync` inside `JobBase` for finer-grained locking within a job. -- **JobContext.RenewLockAsync**: Call in long-running jobs (both `JobBase` and `QueueJobBase`) to prevent lock expiration mid-processing. -- **Register as singletons**: All infrastructure services (`ICacheClient`, `IMessageBus`, `IQueue`, `IFileStorage`, `ILockProvider`) maintain internal state and connections -- always register as singletons. -- **CacheLockProvider + IMessageBus**: `IMessageBus` is optional but recommended. Without it, lock release falls back to polling. With it, locks are released instantly via pub/sub notification. -- **In-memory for tests**: All in-memory implementations are functionally equivalent to production providers. Swap via DI for fast, isolated unit tests with no external dependencies. +- **Cache `GetAsync` returns `CacheValue`**: check `result.HasValue` before `result.Value`. A missing key returns `HasValue = false`, not an exception. +- **Cache stampede**: serialize regeneration of hot keys with `CacheLockProvider` (lock on the cache key, double-check after acquiring). See the [Cache Stampede Protection](https://foundatio.readthedocs.io/guide/caching.html#cache-stampede-protection) docs. +- **Register as singletons**: infrastructure services (`ICacheClient`, `IMessageBus`, `IFileStorage`, `ILockProvider`) maintain internal state and connections; the `AddFoundatio()` builder does this for you. +- **In-memory for tests**: in-memory implementations run the same applicable conformance suites for fast, isolated tests. Their state is process-local, and optional provider capabilities differ. +- **In-memory visibility timing**: one shared timer reclaims expired deliveries at 50 ms intervals while messages are in flight, and pauses when idle. With a fake TimeProvider, advance past the lease expiry to wake blocked receivers; lock renewal uses the current lease, and completion does not retain one timer per delivery. +- **Legacy name collision during migration**: with `AddLegacyAdapter()`, `Foundatio.Messaging.Legacy.IMessageBus` and `Foundatio.Messaging.IMessageBus` coexist. Disambiguate with a `using` alias in files that reference both namespaces. ## NuGet Packages @@ -320,13 +320,15 @@ public class OrderServiceTests : TestLoggerBase | Package | Provides | | ------- | -------- | -| `Foundatio` | Core interfaces, in-memory implementations, resilience, `SystemTextJsonSerializer` | -| `Foundatio.Extensions.Hosting` | `AddJob`, `AddCronJob`, `AddDistributedCronJob`, startup actions, hosted services | +| `Foundatio` | Core interfaces, in-memory implementations, messaging + durable job runtime, resilience, `SystemTextJsonSerializer` | +| `Foundatio.Extensions.Hosting` | Explicit message consumers, workers, schedulers, dispatchers, startup actions | ### Serializers `ITextSerializer` extends `ISerializer` for human-readable formats (JSON). `ISerializer` covers binary formats. Default is `SystemTextJsonSerializer` (included in core). +`IBufferSerializer` is optional: byte-array/memory extensions use it automatically, while stream-only serializers retain the existing fallback. The default JSON serializer supports it with identical options and primitive normalization. Implementations return owned output and never retain or modify input memory; callers need no configuration changes. + | Package | Provides | | ------- | -------- | | `Foundatio.JsonNet` | `JsonNetSerializer` : `ITextSerializer` (Newtonsoft.Json) | @@ -337,10 +339,10 @@ public class OrderServiceTests : TestLoggerBase | Package | Provides | | ------- | -------- | -| `Foundatio.Redis` | Redis cache, queue, messaging, locks, storage | +| `Foundatio.Redis` | This revision: Redis Streams messaging and durable jobs. Earlier external packages also supply legacy Redis abstractions; check API compatibility before mixing versions. | +| `Foundatio.Aws` | `AwsMessageTransport` (SQS queues, SNS+SQS pub/sub), S3 storage | | `Foundatio.AzureStorage` | Azure Blob storage, Azure Storage queues | | `Foundatio.AzureServiceBus` | Azure Service Bus queues + messaging | -| `Foundatio.AWS` | SQS queues, SQS messaging, S3 storage | | `Foundatio.Kafka` | Kafka messaging | | `Foundatio.RabbitMQ` | RabbitMQ messaging | | `Foundatio.Minio` | MinIO S3-compatible storage | @@ -351,7 +353,16 @@ public class OrderServiceTests : TestLoggerBase | Package | Provides | | ------- | -------- | -| `Foundatio.TestHarness` | Shared test base classes for validating custom implementations | +| `Foundatio.Testing` | `MessagingTestHarness`, `JobsTestHarness`, and `UseTestHarness()` for explicit test-driven execution | +| `Foundatio.TestHarness` | Conformance suites (`MessageTransportConformanceTests`, `JobRuntimeStoreConformanceTests`) for custom providers | | `Foundatio.Xunit` | xUnit v2 test logging, retry attributes | | `Foundatio.Xunit.v3` | xUnit v3 test logging, retry attributes | | `Foundatio.DataProtection` | ASP.NET Core Data Protection key storage via `IFileStorage` | + +## Broker execution integration + +- Optional broker history uses the normal job store: `.Jobs.UseInMemory()` / `.Jobs.UseRedis()`, `IJobRuntimeStore`, `JobState`, and `IJobMonitor`. Create records with `ExecutionOwner = Broker`; runtime claims and recovery exclude them. `BeginBrokerAttemptAsync` returns a fresh claim token for `ReportJobProgressAsync` / `CompleteJobAsync`. Broker delivery leases stay in the message bus. `MessageExecutionPipeline` and `MessageProcessingContext` connect processing to the shared job store without scheduling the same work twice. +- `ConsumeWithOutcomeAsync` handles returned Success/Retry/DeadLetter/Unsettled outcomes. Endpoint receive capacity, visibility, automatic renewal, and graceful drain are configured with MessageHandlerOptions. +- `SubscribeNodeAsync` supplies independent best-effort node broadcasts. AWS uses managed tagged resources and startup stale cleanup, not native TTL. Acknowledge-before-callback intentionally allows lost notifications. +- Native `MessageAdministration` owns bounded dead-letter inspection/replay. Send-before-delete is at least once, not atomic. Optional replay preparation can create fresh tracked execution IDs. +- `.Locking.UseRedis()` uses native RedisLockProvider with ownership-checked renewal/release. It coordinates live resource ownership, not persistent duplicate detection. diff --git a/.github/workflows/build-workflow.yml b/.github/workflows/build-workflow.yml index 89a1eb450..d964ce7e5 100644 --- a/.github/workflows/build-workflow.yml +++ b/.github/workflows/build-workflow.yml @@ -122,18 +122,23 @@ jobs: - name: Publish CI Packages if: github.event_name != 'pull_request' && github.actor != 'dependabot[bot]' run: | + # CI-feed publishing is best-effort: a feed rejecting one package (e.g. GitHub Packages returns 403 when the + # package name is linked to a different repo) must not fail a build whose compile and tests passed. Each + # failure surfaces as a warning annotation instead. Release publishing to NuGet (below) stays strict. for package in $(find . -name "*.nupkg" | grep -v "minver" | grep -v "/EmptyFiles/"); do # GitHub if [ -n "${{ secrets.GITHUB_TOKEN }}" ]; then echo "${0##*/}": Pushing $package to GitHub... - dotnet nuget push $package --source https://nuget.pkg.github.com/${{ inputs.org }}/index.json --api-key ${{ secrets.GITHUB_TOKEN }} --skip-duplicate + dotnet nuget push $package --source https://nuget.pkg.github.com/${{ inputs.org }}/index.json --api-key ${{ secrets.GITHUB_TOKEN }} --skip-duplicate \ + || echo "::warning::Failed to push ${package##*/} to GitHub Packages; continuing" fi # Feedz (remove once GitHub supports anonymous access) if [ -n "${{ secrets.FEEDZ_KEY }}" ]; then echo "${0##*/}": Pushing $package to Feedz... - dotnet nuget push $package --source https://f.feedz.io/foundatio/foundatio/nuget --api-key ${{ secrets.FEEDZ_KEY }} --skip-duplicate + dotnet nuget push $package --source https://f.feedz.io/foundatio/foundatio/nuget --api-key ${{ secrets.FEEDZ_KEY }} --skip-duplicate \ + || echo "::warning::Failed to push ${package##*/} to Feedz; continuing" fi done diff --git a/.github/workflows/provider-conformance.yml b/.github/workflows/provider-conformance.yml new file mode 100644 index 000000000..fc1c612e5 --- /dev/null +++ b/.github/workflows/provider-conformance.yml @@ -0,0 +1,49 @@ +name: Provider conformance +on: [push, pull_request] + +permissions: + contents: read + +jobs: + providers: + runs-on: ubuntu-latest + timeout-minutes: 15 + services: + redis: + image: redis:8.6-alpine + ports: + - 6379:6379 + options: >- + --health-cmd "redis-cli ping" + --health-interval 5s + --health-timeout 5s + --health-retries 12 + localstack: + image: localstack/localstack:3.8.1 + env: + SERVICES: sqs,sns + ports: + - 4566:4566 + options: >- + --health-cmd "curl --fail http://localhost:4566/_localstack/health" + --health-interval 5s + --health-timeout 5s + --health-retries 24 + env: + FOUNDATIO_REDIS_CONNECTION_STRING: localhost:6379 + FOUNDATIO_AWS_CONNECTION_STRING: serviceurl=http://localhost:4566;accesskey=test;secretkey=test;region=us-east-1 + steps: + - uses: actions/checkout@v6 + - uses: actions/setup-dotnet@v5 + with: + dotnet-version: | + 8.0.x + 10.0.x + - name: Build providers and shared conformance tests + run: | + dotnet build tests/Foundatio.Redis.Tests/Foundatio.Redis.Tests.csproj --configuration Release + dotnet build tests/Foundatio.Aws.Tests/Foundatio.Aws.Tests.csproj --configuration Release + - name: Redis transport and job store conformance + run: dotnet tests/Foundatio.Redis.Tests/bin/Release/net10.0/Foundatio.Redis.Tests.dll -noLogo -noColor + - name: SQS and SNS conformance against LocalStack + run: dotnet tests/Foundatio.Aws.Tests/bin/Release/net10.0/Foundatio.Aws.Tests.dll -noLogo -noColor diff --git a/.gitignore b/.gitignore index 33eacba3d..d60cd7399 100644 --- a/.gitignore +++ b/.gitignore @@ -38,3 +38,6 @@ _NCrunch_* .idea .cursor/rules/ + +# Sustained messaging benchmark run artifacts +/benchmarks/Messaging/results/ diff --git a/Foundatio.All.slnx b/Foundatio.All.slnx index 356595f40..a6974581f 100644 --- a/Foundatio.All.slnx +++ b/Foundatio.All.slnx @@ -1,7 +1,6 @@ - @@ -17,6 +16,9 @@ + + + @@ -69,6 +71,7 @@ + diff --git a/Foundatio.slnx b/Foundatio.slnx index ebe457b12..0e2ea0dd9 100644 --- a/Foundatio.slnx +++ b/Foundatio.slnx @@ -1,7 +1,8 @@ - + + @@ -13,17 +14,26 @@ + + + + + + + + + diff --git a/README.md b/README.md index d0d57a45d..df1aa3de9 100644 --- a/README.md +++ b/README.md @@ -28,9 +28,11 @@ Pluggable foundation blocks for building loosely coupled distributed apps. | [**File Storage**](https://foundatio.dev/guide/storage) | Unified file API for disk, S3, Azure Blob, and more | | [**Resilience**](https://foundatio.dev/guide/resilience) | Retry policies, circuit breakers, and timeouts | +The messaging and job APIs on this branch are unreleased. Use the [getting started guide](docs/guide/getting-started.md) and [quickstart sample](samples/Foundatio.QuickstartSample) from the same revision. Published provider packages may still implement the earlier APIs. + ## 🚀 Quick Start -```bash +```powershell dotnet add package Foundatio ``` @@ -40,10 +42,15 @@ ICacheClient cache = new InMemoryCacheClient(); await cache.SetAsync("user:123", user, TimeSpan.FromMinutes(5)); var cached = await cache.GetAsync("user:123"); -// Queuing -IQueue queue = new InMemoryQueue(); -await queue.EnqueueAsync(new WorkItem { Data = "Hello" }); -var entry = await queue.DequeueAsync(); +// Queued work +using var messageBus = new MessageBus(new InMemoryMessageTransport()); +await messageBus.SendAsync(new WorkItem { Data = "Hello" }); +await using var delivery = await messageBus.ReceiveAsync(); +if (delivery is not null) +{ + Console.WriteLine(delivery.Message.Data); + await delivery.CompleteAsync(); +} // File Storage IFileStorage storage = new InMemoryFileStorage(); @@ -54,8 +61,12 @@ ILockProvider locker = new CacheLockProvider(cache, messageBus); await using var handle = await locker.AcquireAsync("resource-key"); ``` +For a hosted worker, configure consumers, named event subscribers, and optional jobs in one `AddFoundatioWorker(...)` callback. Producer-only applications use `AddFoundatio()`; see [dependency injection](docs/guide/dependency-injection.md). + ## 📦 Provider Implementations +This table describes the broader provider ecosystem. This unreleased transport contract is currently implemented by in-memory, Redis Streams, and AWS SQS/SNS; see the [current capability matrix](docs/guide/messaging.md#provider-guarantees). + | Provider | Caching | Queues | Messaging | Storage | Locks | |----------|---------|--------|-----------|---------|-------| | [In-Memory](https://foundatio.dev/guide/implementations/in-memory) | ✅ | ✅ | ✅ | ✅ | ✅ | @@ -94,7 +105,7 @@ await using var handle = await locker.AcquireAsync("resource-key"); Want the latest CI build before it hits NuGet? Add the Feedz source and install the pre-release version: -```bash +```powershell dotnet nuget add source https://f.feedz.io/foundatio/foundatio/nuget -n foundatio-feedz dotnet add package Foundatio --prerelease ``` diff --git a/benchmarks/Foundatio.Benchmarks.csproj b/benchmarks/Foundatio.Benchmarks.csproj index efedf8481..ba8beed76 100644 --- a/benchmarks/Foundatio.Benchmarks.csproj +++ b/benchmarks/Foundatio.Benchmarks.csproj @@ -15,6 +15,9 @@ true ..\build\Foundatio.snk + + + diff --git a/benchmarks/MESSAGING_JOBS_BENCHMARK_RESULTS.md b/benchmarks/MESSAGING_JOBS_BENCHMARK_RESULTS.md new file mode 100644 index 000000000..c9e31d96f --- /dev/null +++ b/benchmarks/MESSAGING_JOBS_BENCHMARK_RESULTS.md @@ -0,0 +1,54 @@ +# Messaging and job runtime measurements + +For sustained queue and pub/sub load tests, including Redis, SQS/SNS and MassTransit comparisons, see the [distributed messaging results](Messaging/RESULTS.md) and [reproduction instructions](Messaging/README.md). + +Measured locally on September 6, 2026 with .NET 10.0.11, SDK 10.0.111, BenchmarkDotNet 0.15.8 and an AMD Ryzen AI 9 HX 470 Linux host. These are development measurements, not production sizing promises. ShortRun timing intervals are wide on this shared machine; allocation differences and removal of history-dependent work are the stronger evidence. + +## Small-header construction and serialization + +A temporary benchmark copied six or sixteen key/value pairs, optionally froze the dictionary, then performed six lookups. The dictionary stayed privately owned; the public MessageHeaders wrapper remains immutable and case-insensitive. + +| Header count | Copy then freeze | Private dictionary | Allocated before / after | +| --- | ---: | ---: | ---: | +| 6 | 687 ns | 161 ns | 1,848 / 464 bytes | +| 16 | 1,698 ns | 245 ns | 3,288 / 992 bytes | + +An eight-header serialization probe allocated 1,464 bytes when copying into another dictionary first, versus 512 bytes when serializing the existing headers directly. The implementation now serializes its private backing dictionary without the extra copy. These probes isolate the backing-store decisions; they are not a claimed end-to-end messaging speedup. + +## Idle job polling + +The same empty-claim benchmark ran before and after separating active state from retained history. Each store contained zero runnable jobs and either zero or 10,000 completed jobs. + +| Retained jobs | Before | After | Allocated before / after | +| --- | ---: | ---: | ---: | +| 0 | 328 ns | 44 ns | 432 / 48 bytes | +| 10,000 | 175 microseconds | 52 ns | 400,512 / 48 bytes | + +Previously each poll copied the ConcurrentDictionary values, including completed history. An idle worker now checks active state independently. Ready-job ordering and eligibility remain covered by shared store conformance tests. + +## Local transport workloads + +Five rounds of 300 messages with a 256-byte body and two headers; provisioning and one warm-up call were excluded. Each response was checked for acceptance. Redis 8.6 and LocalStack 3.8.1 ran in isolated local containers. These compare individual calls with batching on the revised implementation, not two complete PR revisions. + +| Transport | Inputs per call | Median time for 300 sends | Messages/sec | Call p95 | +| --- | ---: | ---: | ---: | ---: | +| Redis | 1 | 50.5 ms | 5,938 | 0.280 ms | +| Redis | 10 | 10.0 ms | 29,906 | 0.492 ms | +| Redis | 64 | 5.20 ms | 57,717 | 1.89 ms | +| LocalStack SQS | 1 | 755 ms | 397 | 5.75 ms | +| LocalStack SQS | 10 | 219 ms | 1,372 | 11.1 ms | + +Redis still executes one atomic script per message, now with bounded concurrent requests. AWS sends up to ten entries in one native batch request; partial acceptance remains visible per input. LocalStack latency does not predict live AWS latency. Network-call p95 measures an entire batch, so batch-size rows perform different amounts of work per call. + +A separate in-memory receive workload processed 300 messages whose handlers awaited a two-millisecond delay. Configured concurrency 1, 8 and 32 produced observed peaks of 1, 8 and 32, taking approximately 789, 89 and 25 milliseconds. This confirms overlapping execution; shared concurrency and staggered-job-arrival regression tests protect the behavioral contract. + +## Reproduce ongoing hot-path checks + +The checked-in benchmarks exercise actual public APIs and are intended to catch future allocation regressions: + +```powershell +dotnet run --project benchmarks -c Release -- --filter '*MessageHeadersBenchmarks*' '*JobPollingBenchmarks*' --job Dry +dotnet run --project benchmarks -c Release -- --filter '*JobPollingBenchmarks*' +``` + +Use identical runtime, hardware, configuration and data when comparing revisions. Live AWS, Redis Cluster/failover and sustained production load still require deployment-specific validation. diff --git a/benchmarks/Messaging.Tests/AwsResourcesTests.cs b/benchmarks/Messaging.Tests/AwsResourcesTests.cs new file mode 100644 index 000000000..2f5e39a57 --- /dev/null +++ b/benchmarks/Messaging.Tests/AwsResourcesTests.cs @@ -0,0 +1,92 @@ +using Amazon.Runtime; +using Foundatio.Messaging.Benchmarks; +using Xunit; + +namespace Foundatio.Messaging.Benchmarks.Tests; + +[CollectionDefinition("AWS benchmark environment", DisableParallelization = true)] +public class AwsEnvironmentCollection; + +[Collection("AWS benchmark environment")] +public class AwsResourcesTests : IDisposable +{ + private readonly Dictionary _original = new[] { "PERF_AWS_MODE", "PERF_AWS_URL", "PERF_AWS_REGION" } + .ToDictionary(name => name, Environment.GetEnvironmentVariable); + + public AwsResourcesTests() + { + foreach (string name in _original.Keys) + Environment.SetEnvironmentVariable(name, null); + } + + [Theory] + [InlineData(null)] + [InlineData("localstack")] + [InlineData("LOCALSTACK")] + public void Configuration_DefaultOrLocalStack_UsesOnlyEmulatorCredentials(string? mode) + { + Environment.SetEnvironmentVariable("PERF_AWS_MODE", mode); + + Assert.Equal("http://localhost:24566", AwsResources.ServiceUrl); + Assert.Equal("us-east-1", AwsResources.Region.SystemName); + AssertLocalClient(AwsResources.SqsConfig, "http://localhost:24566", "us-east-1"); + AssertLocalClient(AwsResources.SnsConfig, "http://localhost:24566", "us-east-1"); + var credentials = Assert.IsType(AwsResources.LocalCredentials).GetCredentials(); + Assert.Equal("test", credentials.AccessKey); + Assert.Equal("test", credentials.SecretKey); + } + + [Fact] + public void Configuration_LocalOverrides_AppliesEndpointAndSigningRegionToBothServices() + { + Environment.SetEnvironmentVariable("PERF_AWS_URL", "http://localhost:34566"); + Environment.SetEnvironmentVariable("PERF_AWS_REGION", "eu-west-1"); + + AssertLocalClient(AwsResources.SqsConfig, "http://localhost:34566", "eu-west-1"); + AssertLocalClient(AwsResources.SnsConfig, "http://localhost:34566", "eu-west-1"); + Assert.NotNull(AwsResources.LocalCredentials); + } + + [Theory] + [InlineData("live")] + [InlineData("LIVE")] + [InlineData(" live ")] + public void Configuration_Live_IgnoresEmulatorEndpointAndLeavesCredentialsToSdk(string mode) + { + Environment.SetEnvironmentVariable("PERF_AWS_MODE", mode); + Environment.SetEnvironmentVariable("PERF_AWS_URL", "http://localhost:34566"); + Environment.SetEnvironmentVariable("PERF_AWS_REGION", "eu-west-1"); + + Assert.Null(AwsResources.ServiceUrl); + Assert.Null(AwsResources.LocalCredentials); + Assert.Null(AwsResources.SqsConfig.ServiceURL); + Assert.Null(AwsResources.SnsConfig.ServiceURL); + Assert.Equal("eu-west-1", AwsResources.SqsConfig.RegionEndpoint.SystemName); + Assert.Equal("eu-west-1", AwsResources.SnsConfig.RegionEndpoint.SystemName); + } + + [Theory] + [InlineData("aws")] + [InlineData("liev")] + public void Configuration_UnknownMode_FailsBeforeConnecting(string mode) + { + Environment.SetEnvironmentVariable("PERF_AWS_MODE", mode); + + var error = Assert.Throws(() => AwsResources.ServiceUrl); + Assert.Contains("PERF_AWS_MODE", error.Message); + Assert.Contains("localstack", error.Message); + Assert.Contains("live", error.Message); + } + + private static void AssertLocalClient(ClientConfig config, string endpoint, string region) + { + Assert.Equal(new Uri(endpoint), new Uri(config.ServiceURL)); + Assert.Equal(region, config.AuthenticationRegion); + } + + public void Dispose() + { + foreach (var (name, value) in _original) + Environment.SetEnvironmentVariable(name, value); + } +} diff --git a/benchmarks/Messaging.Tests/Foundatio.Messaging.Benchmarks.Tests.csproj b/benchmarks/Messaging.Tests/Foundatio.Messaging.Benchmarks.Tests.csproj new file mode 100644 index 000000000..6b96778cb --- /dev/null +++ b/benchmarks/Messaging.Tests/Foundatio.Messaging.Benchmarks.Tests.csproj @@ -0,0 +1,16 @@ + + + Exe + net10.0 + enable + enable + false + + + + + + + + + diff --git a/benchmarks/Messaging.Tests/MeasurementTests.cs b/benchmarks/Messaging.Tests/MeasurementTests.cs new file mode 100644 index 000000000..e972d19ac --- /dev/null +++ b/benchmarks/Messaging.Tests/MeasurementTests.cs @@ -0,0 +1,93 @@ +using Foundatio.Messaging.Benchmarks; +using Xunit; + +namespace Foundatio.Messaging.Benchmarks.Tests; + +public class MeasurementTests +{ + [Fact] + public async Task RateSchedule_SubMillisecondRemainder_NeverPublishesEarly() + { + await RateSchedule.WaitUntilAsync(System.Diagnostics.Stopwatch.GetTimestamp(), CancellationToken.None); + for (int i = 0; i < 10; i++) + { + long scheduled = System.Diagnostics.Stopwatch.GetTimestamp() + System.Diagnostics.Stopwatch.Frequency / 2000; + await RateSchedule.WaitUntilAsync(scheduled, CancellationToken.None); + Assert.True(System.Diagnostics.Stopwatch.GetTimestamp() >= scheduled); + } + } + + [Fact] + public async Task RateSchedule_CanceledWait_StopsPromptly() + { + using var cancellation = new CancellationTokenSource(); + cancellation.Cancel(); + long scheduled = System.Diagnostics.Stopwatch.GetTimestamp() + System.Diagnostics.Stopwatch.Frequency; + await Assert.ThrowsAnyAsync(() => RateSchedule.WaitUntilAsync(scheduled, cancellation.Token)); + } + + [Fact] + public void Histogram_KnownDistribution_RetainsTailAndMaximum() + { + var histogram = new LatencyHistogram(); + for (int i = 1; i <= 100; i++) histogram.RecordMicroseconds(i * 1000); + var result = histogram.Snapshot(); + Assert.Equal(100, result.Count); + Assert.InRange(result.P50Milliseconds, 50, 51); + Assert.InRange(result.P99Milliseconds, 99, 101); + Assert.Equal(100, result.MaxMilliseconds); + } + + [Fact] + public async Task Tracker_DuplicateSubscriber_CannotHideMissingFanout() + { + using var tracker = new DeliveryTracker("run", 10, 2, 4, "body"); + await tracker.ReserveAsync(1, CancellationToken.None); + tracker.Expect(0); + var message = new LoadMessage("run", 0, System.Diagnostics.Stopwatch.GetTimestamp(), "body"); + tracker.Record(0, message); + tracker.Record(0, message); + Assert.Equal(1, tracker.UniqueDeliveries); + Assert.Equal(1, tracker.Duplicates); + Assert.Equal(1, tracker.OutstandingInputs); + tracker.Record(1, message); + Assert.Equal(2, tracker.UniqueDeliveries); + Assert.Equal(0, tracker.OutstandingInputs); + } + + [Fact] + public async Task Tracker_InvalidPayloadOrRun_IsNotSuccessfulDelivery() + { + using var tracker = new DeliveryTracker("run", 10, 1, 4, "body"); + await tracker.ReserveAsync(1, CancellationToken.None); + tracker.Expect(0); + tracker.Record(0, new("other", 0, 1, "body")); + tracker.Record(0, new("run", 0, 1, "wrong")); + tracker.Record(0, new("run", 9, 1, "body")); + Assert.Equal(3, tracker.InvalidDeliveries); + Assert.Equal(0, tracker.UniqueDeliveries); + Assert.Contains("run=other, expectedRun=run", tracker.FirstInvalid); + } + + [Fact] + public async Task Tracker_ConcurrentFanout_AccountsForEveryDelivery() + { + using var tracker = new DeliveryTracker("run", 100, 4, 100, "body"); + await tracker.ReserveAsync(100, CancellationToken.None); + for (int i = 0; i < 100; i++) tracker.Expect(i); + await Task.WhenAll(Enumerable.Range(0, 4).Select(group => Task.Run(() => + { + for (int i = 0; i < 100; i++) tracker.Record(group, new("run", i, 1, "body")); + }))); + Assert.Equal(400, tracker.UniqueDeliveries); + Assert.Equal(0, tracker.OutstandingInputs); + Assert.Equal(0, tracker.Duplicates); + } + + [Fact] + public void Options_BatchExceedsOutstandingWindow_RejectsDeadlockRisk() + { + Assert.Throws(() => new BenchmarkOptions { ProducerConcurrency = 8, BatchSize = 10, MaxOutstanding = 32 }.Validate()); + Assert.Throws(() => new BenchmarkOptions { Transport = "redis", Engine = "masstransit" }.Validate()); + } +} diff --git a/benchmarks/Messaging.Tests/SummaryTests.ps1 b/benchmarks/Messaging.Tests/SummaryTests.ps1 new file mode 100644 index 000000000..123fbfb2c --- /dev/null +++ b/benchmarks/Messaging.Tests/SummaryTests.ps1 @@ -0,0 +1,49 @@ +$ErrorActionPreference = 'Stop' +$directory = Join-Path ([System.IO.Path]::GetTempPath()) ('foundatio-summary-tests-' + [guid]::NewGuid().ToString('N')) +$summarize = Join-Path $PSScriptRoot '../Messaging/summarize.ps1' +New-Item -ItemType Directory $directory | Out-Null + +function Write-Trial([string]$Name, [string]$Mode, [string]$Region, [string]$Transport = 'sqs', [string]$RuntimeHash = 'runtime-one') { + @{ + Success = $true + Environment = @{ Runtime = 'test'; CoreClrSha256 = $RuntimeHash; Broker = $(if ($Mode -eq 'live') { 'AWS (live)' } else { 'SQS/SNS custom endpoint' }); AwsMode = $Mode; AwsRegion = $Region } + Options = @{ Engine = 'foundatio'; Transport = $Transport; Scenario = 'queue'; ProducerConcurrency = 1; ConsumerConcurrency = 1; DeliveryCopies = 1; PayloadBytes = 1024; BatchSize = 1; RatePerSecond = 0; MaxOutstanding = 32; Prefetch = 1; DurationSeconds = 1; WarmupSeconds = 1; MaxMessages = 1000 } + Measurement = @{ Inputs = 10; InputsPerSecond = 10; DeliveriesPerSecond = 10; DeliveryLatency = @{ P50Milliseconds = 1; P95Milliseconds = 2; P99Milliseconds = 3 }; AllocatedBytesPerInput = 1; CpuMilliseconds = 1; PeakWorkingSetBytes = 1024; Duplicates = 0; Missing = 0 } + } | ConvertTo-Json -Depth 5 | Set-Content (Join-Path $directory "round-$Name.json") +} + +function Assert-Rejected([string]$Message = 'mix AWS modes or regions') { + try { & $summarize -Directory $directory *> $null } + catch { + if ($_.Exception.Message -match $Message) { return } + throw + } + throw 'Summarizer accepted incompatible measurements.' +} + +try { + Write-Trial 'one' 'localstack' 'us-east-1' + Write-Trial 'two' 'live' 'us-east-1' + Assert-Rejected + + Write-Trial 'one' 'live' 'us-east-1' + Write-Trial 'two' 'live' 'eu-west-1' + Assert-Rejected + + Write-Trial 'two' 'live' 'us-east-1' + Write-Trial 'memory' '' '' 'memory' + & $summarize -Directory $directory *> $null + $rows = @(Import-Csv (Join-Path $directory 'summary.csv')) + $sqs = @($rows | Where-Object Case -Like 'foundatio/sqs *') + if ($rows.Count -ne 2 -or $sqs.Count -ne 1 -or $sqs[0].Trials -ne 2) { + throw 'Summarizer did not preserve compatible AWS trials alongside in-memory trials.' + } + $report = Get-Content (Join-Path $directory 'summary.md') -Raw + if ($report -notmatch 'live' -or $report -notmatch 'us-east-1') { + throw 'Summary does not identify the AWS mode and region.' + } + Write-Trial 'two' 'live' 'us-east-1' 'sqs' 'runtime-two' + Assert-Rejected 'mix runtime' + Write-Host 'PASS: mixed AWS targets and runtime binaries rejected; compatible trials grouped and labeled.' +} +finally { Remove-Item -Recurse -Force $directory } diff --git a/benchmarks/Messaging/ALLOCATION_RESULTS.md b/benchmarks/Messaging/ALLOCATION_RESULTS.md new file mode 100644 index 000000000..3d5c6326f --- /dev/null +++ b/benchmarks/Messaging/ALLOCATION_RESULTS.md @@ -0,0 +1,113 @@ +# Messaging allocation results + +The default JSON serializer now writes directly to an owned byte array and reads directly from input memory. Existing serializer extensions select the optional `IBufferSerializer` capability automatically. AWS sends also avoid intermediate dictionaries and single-message batching lists, and receive requests omit unused system attributes. The public messaging calls and wire format are unchanged by this allocation pass. + +The repeated optimization matrix measures `0b3dfdc86687ab55d2ee6037e608fc97ac0a748d` against the previous pipeline implementation, `77c20ea354919fd25ae300e49c5de7f3ed8da598`. Final code is `e677cf9a4c53fdea468344f175a4d40f2a698c9a`, which additionally preserves UTF-8 byte-order-mark handling. Its full test suite, 20 additional load trials, and two allocation traces passed. Results from those revisions/profiles are kept separate below. + +All **76 untraced trials passed**: **118,134,011 inputs and 167,597,066 acknowledged deliveries**, with zero missing, duplicate or invalid deliveries, zero tracking-limit failures and zero benchmark worker crashes. Five diagnostic captures also passed delivery validation; their performance totals are excluded from comparison medians. + +## Repeated AWS comparison + +These are median **managed allocated bytes per input**, including SDK and harness work and excluding broker processes. Four-subscriber fanout requires four acknowledged deliveries per input. Each cell uses three fresh-process, untraced trials against LocalStack. Allocation churn is not retained memory. + +| Payload / workload | Previous Foundatio | Optimized Foundatio | Reduction | MassTransit | +| --- | ---: | ---: | ---: | ---: | +| 1 KiB / queue | 33,911 | 28,548 | 15.8% | 67,496 | +| 1 KiB / fanout | 137,549 | 121,377 | 11.8% | 116,064 | +| 16 KiB / queue | 274,385 | 223,596 | 18.5% | 240,331 | +| 16 KiB / fanout | 624,167 | 602,618 | 3.5% | 491,948 | + +The 16 KiB queue allocation gap against MassTransit is reversed in this matrix: Foundatio allocates about 7% less. Short-run AWS fanout still allocates more: about 5% at 1 KiB and 22% at 16 KiB. The two-minute fanout comparison below has the opposite allocation ordering. Do not generalize one payload or duration to all workloads. + +| Payload / workload | Previous inputs/s | Optimized inputs/s (range) | MassTransit inputs/s (range) | Previous / optimized / MT p99 ms | +| --- | ---: | ---: | ---: | ---: | +| 1 KiB / queue | 2,991 | 3,077 (2,892–3,090) | 2,587 (2,571–2,732) | 688.13 / 704.51 / 729.09 | +| 1 KiB / fanout | 433 | 458 (445–510) | 417 (400–461) | 2,981.89 / 2,686.97 / 3,604.48 | +| 16 KiB / queue | 2,023 | 2,104 (2,036–2,141) | 1,904 (1,872–1,926) | 819.20 / 778.24 / 843.77 | +| 16 KiB / fanout | 313 | 326 (321–343) | 310 (289–334) | 4,161.53 / 4,063.23 / 4,063.23 | + +The 1 KiB queue/fanout median throughput changes versus the previous implementation are approximately +3%/+6%; the 16 KiB changes are +4%/+4%. Several ranges overlap. Saturation p99 includes a bounded backlog and final settlement; it is not unloaded request latency. + +## Final-revision follow-up + +The initial three-run in-memory queue comparison showed 11% fewer allocated bytes but a 6% lower median rate. Five longer repetitions on final code did not reproduce a consistent slowdown. The initial single Redis fanout check allocated 5% more, so that case was repeated three times. Both original and repeated observations remain in the data. + +| Profile | Previous / final bytes per input | Previous / final inputs/s (ranges) | Previous / final p99 ms | +| --- | ---: | ---: | ---: | +| memory-repeat | 12,031 / 10,743 | 246,086 (241,039–272,562) / 264,442 (248,358–267,083) | 5.31 / 5.31 | +| redis-repeat | 287,821 / 248,530 | 3,283 (3,264–3,321) / 3,455 (3,303–3,484) | 356.35 / 454.65 | + +The in-memory queue allocation reduction is about 11% across both studies. Redis fanout allocates about 14% less in the repeated study, but its median p99 is higher; there is no uniform tail-latency improvement. The original in-memory fanout study reduced allocation from 26,588 to 24,852 bytes/input (7%) with a median rate of 168,914 versus 181,925 inputs/s. The single Redis queue check reduced allocation from 147,438 to 59,837 bytes/input; that large change has only one trial per implementation. + +Final code also passed one confirmation per AWS workload and payload: + +| Payload / workload | Final bytes/input | Final inputs/s | Final p99 ms | +| --- | ---: | ---: | ---: | +| 1 KiB / queue | 34,473 | 2,899 | 663.55 | +| 1 KiB / fanout | 126,739 | 449 | 3,211.26 | +| 16 KiB / queue | 223,639 | 2,045 | 876.54 | +| 16 KiB / fanout | 607,459 | 358 | 3,964.93 | + +Small-payload AWS allocation varied materially: the final queue confirmation was 34,473 bytes/input, versus the earlier optimized median of 28,548. The earlier repeated result is not a guaranteed reduction for every run. Exact allocation ranges, CPU, GC pauses, collections and working sets are retained in the summary and raw JSON. + +## Sustained load and process memory + +These are single two-minute 16 KiB trials at the optimization revision, with the same 20-million-input tracker capacity. Peak working set includes SDK, harness, fixed tracking arrays and touched pages; it cannot establish leak freedom. + +| Implementation / workload | Inputs/s | Bytes/input | Peak working set MiB | p99 ms | +| --- | ---: | ---: | ---: | ---: | +| after / queue | 2,138 | 201,235 | 219.7 | 827.39 | +| after / fanout | 345 | 653,892 | 228.8 | 3,604.48 | +| masstransit / queue | 1,959 | 240,295 | 239.4 | 835.58 | +| masstransit / fanout | 316 | 728,937 | 268.6 | 4,259.84 | + +## Allocation attribution + +GC-verbose EventPipe captures cover 1 KiB and 16 KiB fanout before and after, plus MassTransit at 16 KiB. The offline reader weights GCAllocationTick stacks by AllocationAmount64 over seconds 12–30 of each trace. All five windows have allocation stacks and zero reported lost events. These are sampled attribution estimates, separate from untraced allocation counters. + +The default serializer’s intermediate output-stream growth accounted for **6.46%** of weighted allocations in the previous 16 KiB trace and had **no samples** in the final trace. The allocation regression test independently failed before the fix at **10,012,800 allocated bytes for 5,000,200 output bytes** and passes with the buffer path under a 1.25× output-size budget. + +Largest remaining 16 KiB sampled sites include application payload strings (21.3%), SDK response strings (20.5%), SDK receive checksum buffers (11.1%), and Foundatio’s owned receive-body byte arrays (9.3%). The last buffer keeps raw-message, retry and dead-letter payloads independently owned. Checksum validation and the delivery guarantees were retained. Removing these remaining copies would need a separate ownership or SDK change; none is claimed here. + +## Validation and reproducibility + +- Final Release solution build passed for net8.0 and net10.0; only the pre-existing ASPIRE010 warning remains. The sibling-repository aggregate solution is unavailable in this isolated checkout. +- Final suites: **2,193 passed, 24 expected skips, zero failures** across core, AWS, Redis and benchmark validation. Serializer tests include stream-only implementations, custom options, nulls, runtime types, Unicode, primitives, sliced/non-array memory, and BOM-prefixed JSON. +- AWS tests cover automatic and explicit batches, byte limits, native headers, malformed envelopes, missing/duplicate response IDs, partial failure, cancellation, disposal and acknowledged settlement. +- Documentation build and changed-file whitespace checks passed. No dependencies were added to the library. +- Benchmark workers use the official Microsoft .NET 10.0.11 runtime, MassTransit 8.5.10 and matching SDK binaries. CoreCLR SHA-256: `3EBE90CD92B1EDF6742A41FA921A0C6326216FD1CCA45FDB5E055BEA33351BEA`. +- LocalStack 3.8.1: four CPUs, 3 GiB limit. Redis 8.6-alpine: four CPUs, 2 GiB limit, AOF every second. Main trials publish for 15 seconds after up to three seconds of warmup; the memory repeat uses 30 seconds and up to five seconds of warmup, and Redis repeat uses 20 seconds and up to five seconds. Warmup is capped at one million inputs. +- All measured workers run sequentially, without concurrent builds, tests or profiling. Other host applications are left running, so small changes and overlapping ranges require caution. No live AWS account was used; the existing explicit live mode is preserved. +- Cleanup verified zero fperf queues, topics and Redis keys. The two task-owned containers, network and LocalStack anonymous volume were removed; conformance resources were removed with those containers. +- The earlier Ubuntu-runtime native crashes remain unresolved; this pass uses the preserved official runtime and does not alter the system installation. + +Raw trial data, scripts, configuration, hashes and summaries are in [baselines/2026-09-07-allocations](baselines/2026-09-07-allocations/). The accompanying local artifact archive holds complete nettrace captures, allocation-stack JSON, the standalone TraceAnalysis reader, binary snapshots and validation logs. See its methodology for the exact capture command and [Microsoft’s trace documentation](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-trace) for the gc-verbose profile. + +## Scoped source scan + +The five changed production serialization/AWS files were checked using the performance skill recipes. Counts below are code signals, not counts of defects; the remaining lists and dictionaries include bounded native requests, explicit batches and cold error/provisioning paths. The two AWS partial declarations are one sealed primary type; the existing public JSON serializer remains extensible (one of two primary class types sealed). + +| Recipe | Hits | +| --- | ---: | +| IndexOf literal without comparison | 0 | +| Substring | 0 | +| StartsWith or EndsWith literal without comparison | 0 | +| Contains literal without comparison | 0 | +| ToLower or ToUpper without culture | 0 | +| Three Replace calls on one line | 0 | +| params | 0 | +| LINQ character predicate | 0 | +| new HttpClient | 0 | +| new JsonSerializerOptions | 0 | +| async void | 0 | +| Static dictionary | 0 | +| Static frozen dictionary | 0 | +| new List | 5 | +| new Dictionary | 5 | +| CurrentCulture comparer | 0 | +| LINQ chains | 0 | +| Unsealed public or internal class declarations | 1 | +| Sealed class declarations | 2 | +| Synchronous task waits | 0 | + +Three measured allocation opportunities were addressed: intermediate serializer output buffers, single-send batching/attribute scaffolding, and unused receive metadata. No critical pattern was found in this scoped scan. It is not a whole-repository performance audit. diff --git a/benchmarks/Messaging/AWS_BATCHING_RESULTS.md b/benchmarks/Messaging/AWS_BATCHING_RESULTS.md new file mode 100644 index 000000000..d5d4d1651 --- /dev/null +++ b/benchmarks/Messaging/AWS_BATCHING_RESULTS.md @@ -0,0 +1,94 @@ +# AWS automatic batching performance follow-up + +Automatic batching improves concurrent SQS queues by 3.2 times, one-subscriber SNS/SQS pub/sub by 4.0 times, and four-subscriber fanout by 3.3 times versus the preserved implementation. In this LocalStack comparison, Foundatio is faster than MassTransit on serial queues, approximately tied on one-subscriber pub/sub, 10% behind on concurrent queues and 28% behind on four-subscriber fanout. These are measured results, not a claim that Foundatio is universally faster. + +All 54 final trials succeeded: 1,113,065 measured inputs and 1,522,595 acknowledged deliveries, with zero missing, duplicate or invalid deliveries. The earlier 36-case batching comparison and six-case refinement check are retained separately; together these 96 trials validated 1,982,868 deliveries. No native worker crashes occurred in this follow-up. The unresolved native CLR failures in the [earlier baseline](RESULTS.md#harness-control-and-failures) remain a release blocker. + +## Repeated comparison + +Three fresh-process repetitions per cell, ten seconds of publishing after up to three seconds of warmup. Values are medians; throughput includes final acknowledgement and drain. Payload is 1 KiB and the outstanding window is 1,024 inputs. Concurrent queues use 32 producer workers and 32 consumer slots; fanout uses 32 producers and eight slots in each of four subscriptions. Serial means one producer worker and one consumer slot, with the same outstanding window; it is not a one-at-a-time round-trip test. MassTransit prefetch equals its per-endpoint consumer limit. + +| Workload | Before inputs/s | After inputs/s | MassTransit inputs/s | After / before | +| --- | ---: | ---: | ---: | ---: | +| Serial queue | 398 | 385 | 294 | 0.97x | +| Concurrent queue | 736 | 2,328 | 2,584 | 3.16x | +| One-subscriber pub/sub | 324 | 1,295 | 1,293 | 4.00x | +| Four-subscriber fanout | 97 | 317 | 439 | 3.26x | + +Serial throughput is 3% below its previous median, with overlapping observed ranges (before 383–414 inputs/s; after 384–397). The refinement recovered the large serial regression in the first batching candidate: always waiting for partial acknowledgement batches reduced its median to 277 inputs/s. Idle streams of singleton batches now skip repeated collection waits. + +| Workload | Before p99 ms | After p99 ms | MassTransit p99 ms | +| --- | ---: | ---: | ---: | +| Serial queue | 2,750.25 | 2,697.86 | 3,203.50 | +| Concurrent queue | 1,668.59 | 785.79 | 729.09 | +| One-subscriber pub/sub | 4,194.30 | 1,097.73 | 1,040.38 | +| Four-subscriber fanout | 10,542.04 | 4,063.23 | 3,145.73 | + +Saturation latency includes the bounded backlog. It is not unloaded service latency. Three samples and a shared host do not establish statistical significance for small differences; the observed throughput ranges are retained in the per-revision summaries. + +## What changed + +- Ordinary concurrent send, publish and complete calls now coalesce into native AWS requests. Application code keeps using the single-message API. Per-entry outcomes and broker message IDs remain attached to the correct caller. +- Automatic batching defaults to ten entries, at most 100 buffered operations and four active requests per destination and operation. Additional callers await capacity. Encoded bytes are bounded separately: SQS 1 MiB, SNS 256 KiB. Partial batches collect for up to one millisecond; idle singleton streams skip repeated waits. Shared requests and disposal drains have a 30-second timeout. Explicit batch calls retain their existing chunked behavior. +- Acknowledgement waits for the broker response. Missing, failed or invalid delete results do not report success. Canceling one caller cannot cancel other messages sharing its request; an uncertain send remains unknown. Disposal drains admitted work before owned SDK clients are disposed. +- AWS advertises its ten-message receive limit. The core briefly collects freed slots before another small pull, while maintaining a strict per-delivery concurrency budget and allowing a completed delivery to free its slot independently of slower handlers. Other providers retain zero receive delay. +- The batch worker does not inherit the first caller's async context. A regression reproduced retention of that request-scoped state on subsequent calls and now passes. Wire encoding, native headers and normal lease supervision remain enabled. + +## Broker request evidence + +These six additional ten-second trials have no warmup. Counts come from LocalStack operation logs and therefore also include startup, drain and cleanup. The table selects only send/publish, receive and delete operations. Receive counts can include empty polls. Entries per request are calculated from the fully validated measured input/delivery counts. + +| Workload / implementation | Send or publish entries/request | Deliveries/receive request | Receipts/delete request | +| --- | ---: | ---: | ---: | +| Concurrent queue / before | 1.00 | 9.99 | 1.00 | +| Concurrent queue / after | 6.97 | 9.98 | 9.91 | +| Concurrent queue / masstransit | 8.14 | 8.15 | 8.09 | +| Four-subscriber fanout / before | 1.00 | 3.58 | 1.00 | +| Four-subscriber fanout / after | 5.14 | 6.68 | 6.48 | +| Four-subscriber fanout / masstransit | 7.63 | 7.99 | 7.81 | + +The original code used the batch-send endpoint with one entry and deleted each receipt separately. The optimized queue case averages 6.97 entries per send and 9.91 receipts per delete. This directly verifies that ordinary API calls now amortize broker requests. + +For fanout, Foundatio averages 5.14 entries per publish and 6.48 receipts per delete, versus MassTransit's 7.63 and 7.81. That implies about 48% more publish requests and 21% more delete requests for equal work. This is evidence that batch utilization remains an optimization target; it does not isolate every source of the throughput gap. The next focused experiment should improve fanout batch collection without making fast handlers wait indefinitely for a slow handler, then confirm the result against live AWS. + +## Controlled arrival rates + +One thirty-second trial per cell, timestamped at the intended arrival schedule. All cases admitted their target rate to rounding; the acknowledgement denominator includes final drain. These are latency checks, not repeated confidence estimates. + +| Target inputs/s | Workload | Foundatio p99 ms | MassTransit p99 ms | +| ---: | --- | ---: | ---: | +| 10 | Concurrent queue | 9.98 | 11.65 | +| 10 | Four-subscriber fanout | 27.65 | 29.18 | +| 100 | Concurrent queue | 8.00 | 8.96 | +| 100 | Four-subscriber fanout | 438.27 | 479.23 | + +## Two-minute soaks + +| Implementation / workload | Inputs | Inputs/s | p99 ms | Peak working set MiB | +| --- | ---: | ---: | ---: | ---: | +| after/fanout | 36,791 | 304 | 4,194.30 | 128.9 | +| after/queue | 293,275 | 2,439 | 737.28 | 125.3 | +| masstransit/fanout | 52,464 | 434 | 2,916.35 | 165.8 | +| masstransit/queue | 318,875 | 2,652 | 712.70 | 151.6 | + +All four soaks ran the full 120-second publishing window, used five seconds of warmup, and drained every delivery. The same 20-million-input tracking capacity was retained across this follow-up; these memory figures should not be directly compared with the older baseline's larger soak tracker. + +## Client cost + +| Workload | Before allocated bytes/input | After allocated bytes/input | MassTransit allocated bytes/input | +| --- | ---: | ---: | ---: | +| Serial queue | 131,191 | 138,928 | 185,334 | +| Concurrent queue | 109,022 | 59,292 | 67,454 | +| One-subscriber pub/sub | 148,868 | 40,829 | 80,948 | +| Four-subscriber fanout | 377,168 | 190,645 | 84,765 | + +Allocation and CPU counts include the client and harness, and exclude LocalStack. Fanout allocations remain higher than MassTransit even though Foundatio's median client CPU time per input is lower (1.42 versus 1.74 ms). RSS includes fixed delivery-tracking arrays; samples and GC statistics are retained and are not a live-object census. + +## Reproduction and validation + +- Optimized shipping implementation: `abce1c0e`. Preserved before checkout: `5dae40ef`; its unchanged shipping assemblies carry `95983490` informational metadata. The intermediate batching candidate was `1a9f0e62`. Binary SHA-256 manifests distinguish all measured executables. +- MassTransit 8.5.10, identical AWS SDK dependencies, .NET 10.0.11, server GC, Linux x64, AMD Ryzen AI 9 HX 470 / 24 logical processors. LocalStack 3.8.1 used the task-owned loopback endpoint with a four-CPU / 3-GiB container limit. The host was shared with other development services, without CPU affinity. Measured workers ran sequentially; no local builds, tests or profiling ran alongside them. +- Final standard cases were shuffled and interleaved across before, after and MassTransit with seed 534. Rate, soak and request-accounting profiles ran afterward against the same broker. All benchmark queues/topics were absent at the end, and the temporary broker containers were removed. +- The full Release solution build passed with only the existing ASPIRE010 warning. Core: 2,032 passed / 12 skipped; AWS: 39 passed / 8 skipped; Redis: 56 passed / 4 skipped; benchmark measurement: 16 passed. Total: **2,143 passed, 24 expected skips, zero failures**. Summary regressions and the documentation build passed. Tests cover mixed outcomes, byte limits, cancellation within a confirmed shared batch, bounded admission, timeout recovery, disposal, caller-context isolation and consumer slot ownership. +- The [benchmark README](README.md) documents both LocalStack and explicit live AWS mode. No actual AWS account was contacted; these emulator figures do not predict AWS throughput or latency. +- [Raw trials, per-revision summaries, manifests and scripts](baselines/2026-09-06-aws-batching/) are retained. The archive contains all 96 comparison/refinement trials, including the intermediate candidate, with a profile manifest. Earlier short diagnostic experiments are retained separately in the local handoff and are excluded from the final performance conclusions. diff --git a/benchmarks/Messaging/AwsResources.cs b/benchmarks/Messaging/AwsResources.cs new file mode 100644 index 000000000..c2bc95256 --- /dev/null +++ b/benchmarks/Messaging/AwsResources.cs @@ -0,0 +1,60 @@ +using Amazon; +using Amazon.Runtime; +using Amazon.SimpleNotificationService; +using Amazon.SQS; +using Amazon.SQS.Model; + +namespace Foundatio.Messaging.Benchmarks; + +public static class AwsResources +{ + public static string Mode => Environment.GetEnvironmentVariable("PERF_AWS_MODE")?.Trim().ToLowerInvariant() switch + { + null or "" or "localstack" => "localstack", + "live" => "live", + _ => throw new ArgumentException("PERF_AWS_MODE must be 'localstack' or 'live'.") + }; + public static string? ServiceUrl => Mode == "live" ? null : Environment.GetEnvironmentVariable("PERF_AWS_URL") ?? "http://localhost:24566"; + public static RegionEndpoint Region => RegionEndpoint.GetBySystemName(Environment.GetEnvironmentVariable("PERF_AWS_REGION") ?? "us-east-1"); + public static AWSCredentials? LocalCredentials => ServiceUrl is null ? null : new BasicAWSCredentials("test", "test"); + public static AmazonSQSConfig SqsConfig + { + get + { + var config = new AmazonSQSConfig { RegionEndpoint = Region }; + if (ServiceUrl is { } url) { config.ServiceURL = url; config.AuthenticationRegion = Region.SystemName; } + return config; + } + } + public static AmazonSimpleNotificationServiceConfig SnsConfig + { + get + { + var config = new AmazonSimpleNotificationServiceConfig { RegionEndpoint = Region }; + if (ServiceUrl is { } url) { config.ServiceURL = url; config.AuthenticationRegion = Region.SystemName; } + return config; + } + } + + public static async Task CleanupAsync(string prefix) + { + using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(60)); + using var sqs = LocalCredentials is { } credentials ? new AmazonSQSClient(credentials, SqsConfig) : new AmazonSQSClient(SqsConfig); + using var sns = LocalCredentials is { } snsCredentials ? new AmazonSimpleNotificationServiceClient(snsCredentials, SnsConfig) : new AmazonSimpleNotificationServiceClient(SnsConfig); + string? next = null; + do + { + var queues = await sqs.ListQueuesAsync(new ListQueuesRequest { QueueNamePrefix = prefix, NextToken = next, MaxResults = 1000 }, timeout.Token); + foreach (string queue in queues.QueueUrls ?? []) await sqs.DeleteQueueAsync(queue, timeout.Token); + next = queues.NextToken; + } while (next is not null); + do + { + var topics = await sns.ListTopicsAsync(next, timeout.Token); + foreach (var topic in topics.Topics ?? []) + if (topic.TopicArn[(topic.TopicArn.LastIndexOf(':') + 1)..].StartsWith(prefix, StringComparison.Ordinal)) + await sns.DeleteTopicAsync(topic.TopicArn, timeout.Token); + next = topics.NextToken; + } while (next is not null); + } +} diff --git a/benchmarks/Messaging/BenchmarkOptions.cs b/benchmarks/Messaging/BenchmarkOptions.cs new file mode 100644 index 000000000..98894c105 --- /dev/null +++ b/benchmarks/Messaging/BenchmarkOptions.cs @@ -0,0 +1,70 @@ +using System.Globalization; + +namespace Foundatio.Messaging.Benchmarks; + +public sealed record BenchmarkOptions +{ + public string Engine { get; init; } = "foundatio"; + public string Transport { get; init; } = "memory"; + public string Scenario { get; init; } = "queue"; + public int DurationSeconds { get; init; } = 15; + public int WarmupSeconds { get; init; } = 3; + public int DrainSeconds { get; init; } = 120; + public int ProducerConcurrency { get; init; } = 32; + public int ConsumerConcurrency { get; init; } = 32; + public int Prefetch { get; init; } = 32; + public int Subscribers { get; init; } = 4; + public int PayloadBytes { get; init; } = 1024; + public int BatchSize { get; init; } = 1; + public int MaxOutstanding { get; init; } = 4096; + public int MaxMessages { get; init; } = 10_000_000; + public int RatePerSecond { get; init; } + public string Output { get; init; } = "result.json"; + public int DeliveryCopies => Scenario == "queue" ? 1 : Subscribers; + + public void Validate() + { + if (Engine is not ("foundatio" or "masstransit" or "loopback")) throw new ArgumentException("Engine must be foundatio, masstransit or loopback."); + if (Transport is not ("memory" or "redis" or "sqs")) throw new ArgumentException("Transport must be memory, redis or sqs."); + if (Engine == "masstransit" && Transport == "redis") throw new ArgumentException("MassTransit has no Redis Streams transport."); + if (Engine == "loopback" && Transport != "memory") throw new ArgumentException("Loopback measures only harness overhead."); + if (Scenario is not ("queue" or "pubsub")) throw new ArgumentException("Scenario must be queue or pubsub."); + if (DurationSeconds is < 1 or > 3600 || WarmupSeconds is < 0 or > 60 || DrainSeconds is < 1 or > 600) throw new ArgumentException("Invalid measurement/warmup/drain duration."); + if (ProducerConcurrency is < 1 or > 1024 || ConsumerConcurrency is < 1 or > 1024 || Prefetch is < 1 or > 4096) throw new ArgumentException("Invalid concurrency or prefetch."); + if (Subscribers is < 1 or > 32 || PayloadBytes is < 0 or > 131072 || BatchSize is < 1 or > 64) throw new ArgumentException("Invalid fanout, payload size or batch size."); + if (MaxOutstanding < ProducerConcurrency * BatchSize || MaxOutstanding > 1_000_000) throw new ArgumentException("Outstanding window must hold one entire batch for every producer, and cannot exceed one million inputs."); + if (MaxMessages < MaxOutstanding || MaxMessages > 100_000_000 || RatePerSecond < 0) throw new ArgumentException("Invalid tracking capacity or offered rate."); + } + + public static BenchmarkOptions Parse(string[] args) + { + if (args.Length % 2 != 0) throw new ArgumentException("Options use --name value pairs; use --help for examples."); + var values = new Dictionary(StringComparer.Ordinal); + for (int i = 0; i < args.Length; i += 2) + if (!values.TryAdd(args[i], args[i + 1])) throw new ArgumentException($"Duplicate option {args[i]}."); + string Text(string name, string fallback) => values.Remove("--" + name, out var value) ? value : fallback; + int Number(string name, int fallback) => Int32.Parse(Text(name, fallback.ToString(CultureInfo.InvariantCulture)), CultureInfo.InvariantCulture); + var options = new BenchmarkOptions + { + Engine = Text("engine", "foundatio"), + Transport = Text("transport", "memory"), + Scenario = Text("scenario", "queue"), + DurationSeconds = Number("seconds", 15), + WarmupSeconds = Number("warmup", 3), + DrainSeconds = Number("drain", 120), + ProducerConcurrency = Number("producers", 32), + ConsumerConcurrency = Number("consumers", 32), + Prefetch = Number("prefetch", 32), + Subscribers = Number("subscribers", 4), + PayloadBytes = Number("payload", 1024), + BatchSize = Number("batch", 1), + MaxOutstanding = Number("outstanding", 4096), + MaxMessages = Number("max-messages", 10_000_000), + RatePerSecond = Number("rate", 0), + Output = Text("output", "result.json") + }; + if (values.Count > 0) throw new ArgumentException($"Unknown option {values.Keys.First()}."); + options.Validate(); + return options; + } +} diff --git a/benchmarks/Messaging/BenchmarkResult.cs b/benchmarks/Messaging/BenchmarkResult.cs new file mode 100644 index 000000000..5012c475b --- /dev/null +++ b/benchmarks/Messaging/BenchmarkResult.cs @@ -0,0 +1,43 @@ +namespace Foundatio.Messaging.Benchmarks; + +public sealed record BenchmarkResult +{ + public required BenchmarkOptions Options { get; init; } + public required string ResourcePrefix { get; init; } + public required IReadOnlyDictionary Environment { get; init; } + public DateTimeOffset StartedUtc { get; init; } = DateTimeOffset.UtcNow; + public bool Success { get; init; } + public string? Error { get; init; } + public PhaseResult? Measurement { get; init; } +} + +public sealed record PhaseResult +{ + public string? Error { get; init; } + public string? FirstInvalid { get; init; } + public long Inputs { get; init; } + public long Deliveries { get; init; } + public long Duplicates { get; init; } + public long Invalid { get; init; } + public long Missing { get; init; } + public bool HitTrackingLimit { get; init; } + public double PublishSeconds { get; init; } + public double TotalSeconds { get; init; } + public double InputsPerSecond => Inputs / TotalSeconds; + public double DeliveriesPerSecond => Deliveries / TotalSeconds; + public long AllocatedBytes { get; init; } + public double AllocatedBytesPerInput => Inputs > 0 ? AllocatedBytes / (double)Inputs : 0; + public double CpuMilliseconds { get; init; } + public long PeakWorkingSetBytes { get; init; } + public long GcHeapSizeBytes { get; init; } + public long GcCommittedBytes { get; init; } + public long GcFragmentedBytes { get; init; } + public int[] Collections { get; init; } = []; + public double GcPauseMilliseconds { get; init; } + public required LatencySummary DeliveryLatency { get; init; } + public required LatencySummary SendCallLatency { get; init; } + public IReadOnlyList Samples { get; init; } = []; +} + +public sealed record ProgressSample(double Seconds, long Inputs, long Deliveries, long Outstanding, long WorkingSetBytes, long AllocatedBytes, + long GcHeapSizeBytes, long GcCommittedBytes, long GcFragmentedBytes); diff --git a/benchmarks/Messaging/BenchmarkRunner.cs b/benchmarks/Messaging/BenchmarkRunner.cs new file mode 100644 index 000000000..5d2dc1a2b --- /dev/null +++ b/benchmarks/Messaging/BenchmarkRunner.cs @@ -0,0 +1,224 @@ +using System.Diagnostics; +using System.Reflection; +using System.Runtime; +using System.Runtime.InteropServices; +using System.Security.Cryptography; +using System.Text.Json; + +namespace Foundatio.Messaging.Benchmarks; + +public static class BenchmarkRunner +{ + public static async Task RunAsync(BenchmarkOptions options, CancellationToken token) + { + var startedUtc = DateTimeOffset.UtcNow; + string prefix = "fperf-" + Guid.NewGuid().ToString("N")[..12]; + var environment = new Dictionary + { + ["Runtime"] = RuntimeInformation.FrameworkDescription, + ["CoreClrSha256"] = RuntimeFingerprint(), + ["OS"] = RuntimeInformation.OSDescription, + ["Architecture"] = RuntimeInformation.ProcessArchitecture.ToString(), + ["LogicalProcessors"] = Environment.ProcessorCount.ToString(), + ["ServerGC"] = GCSettings.IsServerGC.ToString(), + ["Foundatio"] = VersionOf(typeof(MessageBus).Assembly), + ["MassTransit"] = VersionOf(typeof(MassTransit.IBus).Assembly), + ["SqsSdk"] = VersionOf(typeof(Amazon.SQS.AmazonSQSClient).Assembly), + ["SnsSdk"] = VersionOf(typeof(Amazon.SimpleNotificationService.AmazonSimpleNotificationServiceClient).Assembly), + ["Broker"] = options.Transport == "sqs" ? (AwsResources.ServiceUrl is null ? "AWS (live)" : "SQS/SNS custom endpoint") : options.Transport + }; + if (options.Transport == "sqs") + { + environment["AwsMode"] = AwsResources.Mode; + environment["AwsRegion"] = AwsResources.Region.SystemName; + Console.WriteLine($"AWS mode={AwsResources.Mode} region={AwsResources.Region.SystemName}"); + } + IMessagingDriver driver = options.Engine switch + { + "masstransit" => new MassTransitDriver(options, prefix), + "loopback" => new LoopbackDriver(options), + _ => new FoundatioDriver(options, prefix) + }; + Console.WriteLine($"RUN {prefix} {options.Engine}/{options.Transport}/{options.Scenario}"); + DeliveryTracker? tracker = null; + var trackers = new List(); + PhaseResult? measurement = null; + string? error = null; + try + { + using var startup = CancellationTokenSource.CreateLinkedTokenSource(token); + startup.CancelAfter(TimeSpan.FromSeconds(options.DrainSeconds)); + await driver.StartAsync((group, message) => Volatile.Read(ref tracker)?.Record(group, message), startup.Token); + if (options.WarmupSeconds > 0) + { + var warmup = await PhaseAsync(options.WarmupSeconds, true); + if (!Valid(warmup)) throw new InvalidOperationException("Warmup failed: " + (warmup.Error ?? $"missing={warmup.Missing}, invalid={warmup.Invalid}, duplicates={warmup.Duplicates}, firstInvalid={tracker?.FirstInvalid}")); + } + measurement = await PhaseAsync(options.DurationSeconds, false); + if (!Valid(measurement)) error = measurement.Error ?? "Delivery validation failed or the tracking limit was reached."; + } + catch (Exception ex) { error = ex.ToString(); } + finally + { + try { await driver.DisposeAsync(); } + catch (Exception ex) { error = (error is null ? "" : error + Environment.NewLine) + "Cleanup: " + ex; } + foreach (var item in trackers) item.Dispose(); + } + if (measurement is not null && tracker is not null) + { + measurement = measurement with { Duplicates = tracker.Duplicates, Invalid = tracker.InvalidDeliveries, FirstInvalid = tracker.FirstInvalid, Missing = tracker.ExpectedInputs * options.DeliveryCopies - tracker.UniqueDeliveries }; + if (!Valid(measurement) && error is null) error = "Delivery validation failed during shutdown."; + } + var result = new BenchmarkResult { Options = options, StartedUtc = startedUtc, ResourcePrefix = prefix, Environment = environment, Success = error is null, Error = error, Measurement = measurement }; + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(options.Output))!); + await File.WriteAllTextAsync(options.Output, JsonSerializer.Serialize(result, new JsonSerializerOptions { WriteIndented = true }), CancellationToken.None); + Console.WriteLine($"{(result.Success ? "PASS" : "FAIL")} {options.Engine}/{options.Transport}/{options.Scenario} inputs/s={measurement?.InputsPerSecond:F0} deliveries/s={measurement?.DeliveriesPerSecond:F0} p99={measurement?.DeliveryLatency.P99Milliseconds:F2}ms output={options.Output}"); + if (error is not null) Console.Error.WriteLine(error); + return result.Success ? 0 : 1; + + async Task PhaseAsync(int seconds, bool warmup) + { + Console.WriteLine($"PHASE {(warmup ? "warmup" : "measurement")} {seconds}s"); + string runId = Guid.NewGuid().ToString("N"); + string payload = new('x', options.PayloadBytes); + int capacity = warmup ? Math.Min(options.MaxMessages, 1_000_000) : options.MaxMessages; + var phaseTracker = new DeliveryTracker(runId, capacity, options.DeliveryCopies, options.MaxOutstanding, payload); + trackers.Add(phaseTracker); + Volatile.Write(ref tracker, phaseTracker); + var sendLatency = new LatencyHistogram(); + using var process = Process.GetCurrentProcess(); + using var deadline = CancellationTokenSource.CreateLinkedTokenSource(token); + using var publishing = CancellationTokenSource.CreateLinkedTokenSource(token); + GC.Collect(); GC.WaitForPendingFinalizers(); GC.Collect(); + long allocatedStart = GC.GetTotalAllocatedBytes(true); + TimeSpan cpuStart = process.TotalProcessorTime, pausesStart = GC.GetTotalPauseDuration(); + int[] collections = [GC.CollectionCount(0), GC.CollectionCount(1), GC.CollectionCount(2)]; + long start = Stopwatch.GetTimestamp(); + deadline.CancelAfter(TimeSpan.FromSeconds(seconds + options.DrainSeconds)); + publishing.CancelAfter(TimeSpan.FromSeconds(seconds)); + int next = 0, hitLimit = 0; + double publishSeconds = 0; + string? phaseError = null; + var samples = new List(); + using var sampling = new CancellationTokenSource(); + var sampleTask = SampleAsync(); + try + { + await Task.WhenAll(Enumerable.Range(0, options.ProducerConcurrency).Select(_ => Task.Run(ProduceAsync, CancellationToken.None))); + publishSeconds = Stopwatch.GetElapsedTime(start).TotalSeconds; + while (phaseTracker.OutstandingInputs > 0) + { + driver.ThrowIfFaulted(); + await Task.Delay(5, deadline.Token); + } + driver.ThrowIfFaulted(); + } + catch (Exception ex) { phaseError = ex.ToString(); } + double totalSeconds = Stopwatch.GetElapsedTime(start).TotalSeconds; + long allocated = GC.GetTotalAllocatedBytes(true) - allocatedStart; + TimeSpan cpu = process.TotalProcessorTime - cpuStart, pauses = GC.GetTotalPauseDuration() - pausesStart; + for (int i = 0; i < 3; i++) collections[i] = GC.CollectionCount(i) - collections[i]; + await sampling.CancelAsync(); await sampleTask; + process.Refresh(); + var gcMemory = GC.GetGCMemoryInfo(); + long peak = Math.Max(process.WorkingSet64, samples.Count == 0 ? 0 : samples.Max(s => s.WorkingSetBytes)); + return new PhaseResult + { + Error = phaseError, + FirstInvalid = phaseTracker.FirstInvalid, + Inputs = phaseTracker.ExpectedInputs, + Deliveries = phaseTracker.UniqueDeliveries, + Duplicates = phaseTracker.Duplicates, + Invalid = phaseTracker.InvalidDeliveries, + Missing = phaseTracker.ExpectedInputs * options.DeliveryCopies - phaseTracker.UniqueDeliveries, + HitTrackingLimit = !warmup && hitLimit != 0, + PublishSeconds = publishSeconds, + TotalSeconds = totalSeconds, + AllocatedBytes = allocated, + CpuMilliseconds = cpu.TotalMilliseconds, + GcPauseMilliseconds = pauses.TotalMilliseconds, + PeakWorkingSetBytes = peak, + GcHeapSizeBytes = gcMemory.HeapSizeBytes, + GcCommittedBytes = gcMemory.TotalCommittedBytes, + GcFragmentedBytes = gcMemory.FragmentedBytes, + Collections = collections, + DeliveryLatency = phaseTracker.Latency.Snapshot(), + SendCallLatency = sendLatency.Snapshot(), + Samples = samples + }; + + async Task ProduceAsync() + { + while (!publishing.IsCancellationRequested) + { + try { await phaseTracker.ReserveAsync(options.BatchSize, publishing.Token); } + catch (OperationCanceledException) when (publishing.IsCancellationRequested) { return; } + int sequence = Interlocked.Add(ref next, options.BatchSize) - options.BatchSize; + int count = Math.Min(options.BatchSize, capacity - sequence); + if (count <= 0) { phaseTracker.ReleaseUnused(options.BatchSize); Interlocked.Exchange(ref hitLimit, 1); return; } + if (count < options.BatchSize) phaseTracker.ReleaseUnused(options.BatchSize - count); + long timestamp = options.RatePerSecond == 0 ? Stopwatch.GetTimestamp() + : start + (long)(sequence * (double)Stopwatch.Frequency / options.RatePerSecond); + if (options.RatePerSecond > 0) + { + try { await RateSchedule.WaitUntilAsync(timestamp, publishing.Token); } + catch (OperationCanceledException) when (publishing.IsCancellationRequested) { phaseTracker.ReleaseUnused(count); return; } + } + if (publishing.IsCancellationRequested) { phaseTracker.ReleaseUnused(count); return; } + var batch = new LoadMessage[count]; + for (int i = 0; i < count; i++) + { + phaseTracker.Expect(sequence + i); + batch[i] = new LoadMessage(runId, sequence + i, timestamp, payload); + } + driver.ThrowIfFaulted(); + long sendStart = Stopwatch.GetTimestamp(); + await driver.SendAsync(batch, deadline.Token); + sendLatency.RecordMicroseconds((long)Stopwatch.GetElapsedTime(sendStart).TotalMicroseconds); + } + } + + async Task SampleAsync() + { + try + { + while (true) + { + await Task.Delay(TimeSpan.FromSeconds(1), sampling.Token); + process.Refresh(); + var sampleMemory = GC.GetGCMemoryInfo(); + samples.Add(new(Stopwatch.GetElapsedTime(start).TotalSeconds, phaseTracker.ExpectedInputs, phaseTracker.UniqueDeliveries, + phaseTracker.OutstandingInputs, process.WorkingSet64, GC.GetTotalAllocatedBytes(false) - allocatedStart, + sampleMemory.HeapSizeBytes, sampleMemory.TotalCommittedBytes, sampleMemory.FragmentedBytes)); + } + } + catch (OperationCanceledException) when (sampling.IsCancellationRequested) { } + } + } + } + + private static string RuntimeFingerprint() + { + string? directory = Path.GetDirectoryName(typeof(object).Assembly.Location); + string name = OperatingSystem.IsWindows() ? "coreclr.dll" : OperatingSystem.IsMacOS() ? "libcoreclr.dylib" : "libcoreclr.so"; + if (directory is null || !File.Exists(Path.Combine(directory, name))) return "unavailable"; + using var file = File.OpenRead(Path.Combine(directory, name)); + return Convert.ToHexString(SHA256.HashData(file)); + } + + private static bool Valid(PhaseResult result) => result.Error is null && result.Inputs > 0 && result.Missing == 0 && result.Invalid == 0 && result.Duplicates == 0 && !result.HitTrackingLimit; + private static string VersionOf(Assembly assembly) => assembly.GetCustomAttribute()?.InformationalVersion ?? assembly.GetName().Version!.ToString(); +} + +internal sealed class LoopbackDriver(BenchmarkOptions options) : IMessagingDriver +{ + private Action _received = null!; + public Task StartAsync(Action received, CancellationToken token) { _received = received; return Task.CompletedTask; } + public Task SendAsync(LoadMessage[] messages, CancellationToken token) + { + foreach (var message in messages) for (int group = 0; group < options.DeliveryCopies; group++) _received(group, message); + return Task.CompletedTask; + } + public void ThrowIfFaulted() { } + public ValueTask DisposeAsync() => ValueTask.CompletedTask; +} diff --git a/benchmarks/Messaging/DeliveryTracker.cs b/benchmarks/Messaging/DeliveryTracker.cs new file mode 100644 index 000000000..160d80f4a --- /dev/null +++ b/benchmarks/Messaging/DeliveryTracker.cs @@ -0,0 +1,73 @@ +using System.Diagnostics; + +namespace Foundatio.Messaging.Benchmarks; + +public sealed record LoadMessage(string RunId, int Sequence, long StartedTimestamp, string Payload); + +public sealed class DeliveryTracker : IDisposable +{ + private readonly string _runId; + private readonly string _payload; + private readonly int _subscribers; + private readonly int[] _remaining; + private readonly int[] _seen; + private readonly SemaphoreSlim _window; + private long _unique, _duplicates, _invalid, _expected, _completed; + private long _lastDelivery; + private string? _firstInvalid; + public string? FirstInvalid => Volatile.Read(ref _firstInvalid); + public LatencyHistogram Latency { get; } = new(); + public long UniqueDeliveries => Volatile.Read(ref _unique); + public long Duplicates => Volatile.Read(ref _duplicates); + public long InvalidDeliveries => Volatile.Read(ref _invalid); + public long ExpectedInputs => Volatile.Read(ref _expected); + public long OutstandingInputs => ExpectedInputs - Volatile.Read(ref _completed); + public long LastDeliveryTimestamp => Volatile.Read(ref _lastDelivery); + + public DeliveryTracker(string runId, int maxMessages, int subscribers, int window, string payload) + { + _runId = runId; _payload = payload; _subscribers = subscribers; + _remaining = new int[maxMessages]; + _seen = new int[checked((int)(((long)maxMessages * subscribers + 31) / 32))]; + _window = new SemaphoreSlim(window, window); + } + + public async Task ReserveAsync(int count, CancellationToken token) + { + int reserved = 0; + try { for (; reserved < count; reserved++) await _window.WaitAsync(token).ConfigureAwait(false); } + catch { if (reserved > 0) _window.Release(reserved); throw; } + } + + public void ReleaseUnused(int count) => _window.Release(count); + + public void Expect(int sequence) + { + Volatile.Write(ref _remaining[sequence], _subscribers); + Interlocked.Increment(ref _expected); + } + + public void Record(int subscriber, LoadMessage message) + { + if (message.RunId != _runId || (uint)subscriber >= _subscribers || (uint)message.Sequence >= _remaining.Length + || !String.Equals(message.Payload, _payload, StringComparison.Ordinal)) + { + Interlocked.CompareExchange(ref _firstInvalid, $"run={message.RunId}, expectedRun={_runId}, subscriber={subscriber}/{_subscribers}, sequence={message.Sequence}, payloadLength={message.Payload?.Length}/{_payload.Length}", null); + Interlocked.Increment(ref _invalid); return; + } + long bit = ((long)message.Sequence * _subscribers) + subscriber; + int mask = 1 << (int)(bit % 32); + if ((Interlocked.Or(ref _seen[bit / 32], mask) & mask) != 0) + { Interlocked.Increment(ref _duplicates); return; } + if (Volatile.Read(ref _remaining[message.Sequence]) <= 0) + { Interlocked.CompareExchange(ref _firstInvalid, $"Unregistered sequence {message.Sequence}, subscriber={subscriber}", null); Interlocked.Increment(ref _invalid); return; } + long now = Stopwatch.GetTimestamp(); + Latency.RecordMicroseconds((long)(Stopwatch.GetElapsedTime(message.StartedTimestamp, now).TotalMicroseconds)); + Interlocked.Exchange(ref _lastDelivery, now); + Interlocked.Increment(ref _unique); + if (Interlocked.Decrement(ref _remaining[message.Sequence]) == 0) + { Interlocked.Increment(ref _completed); _window.Release(); } + } + + public void Dispose() => _window.Dispose(); +} diff --git a/benchmarks/Messaging/Foundatio.Messaging.Benchmarks.csproj b/benchmarks/Messaging/Foundatio.Messaging.Benchmarks.csproj new file mode 100644 index 000000000..6d2ae0ffa --- /dev/null +++ b/benchmarks/Messaging/Foundatio.Messaging.Benchmarks.csproj @@ -0,0 +1,18 @@ + + + Exe + net10.0 + enable + enable + false + true + 8.5.10 + + + + + + + + + diff --git a/benchmarks/Messaging/FoundatioDriver.cs b/benchmarks/Messaging/FoundatioDriver.cs new file mode 100644 index 000000000..c7b2c3801 --- /dev/null +++ b/benchmarks/Messaging/FoundatioDriver.cs @@ -0,0 +1,86 @@ +using Foundatio.Messaging; +using Microsoft.Extensions.Logging; +using StackExchange.Redis; + +namespace Foundatio.Messaging.Benchmarks; + +public sealed class FoundatioDriver(BenchmarkOptions options, string prefix) : IMessagingDriver +{ + private readonly List _subscriptions = []; + private readonly ILoggerFactory _logs = LoggerFactory.Create(b => b.AddSimpleConsole().SetMinimumLevel(LogLevel.Warning)); + private IMessageTransport? _transport; + private MessageBus? _bus; + private IConnectionMultiplexer? _redis; + private Exception? _fault; + + public async Task StartAsync(Action received, CancellationToken token) + { + if (options.Transport == "redis") + { + _redis = await ConnectionMultiplexer.ConnectAsync(Environment.GetEnvironmentVariable("PERF_REDIS") ?? "localhost:16379"); + _transport = new RedisStreamsMessageTransport(new() { ConnectionMultiplexer = _redis, KeyPrefix = prefix + ":" }); + } + else if (options.Transport == "sqs") + _transport = new AwsMessageTransport(new AwsMessageTransportOptions { ResourcePrefix = prefix, ServiceUrl = AwsResources.ServiceUrl, Region = AwsResources.Region, Credentials = AwsResources.LocalCredentials }); + else _transport = new InMemoryMessageTransport(); + _bus = new MessageBus(_transport, new() + { + OwnsTransport = false, + LoggerFactory = _logs, + MessageTypes = new MessageTypeRegistry([new("load.v1", typeof(LoadMessage))]) + }); + for (int group = 0; group < options.DeliveryCopies; group++) + { + int subscriber = group; + async Task HandleAsync(IMessageContext context, CancellationToken ct) + { + try { await context.CompleteAsync(ct); received(subscriber, context.Message); } + catch (Exception ex) { Interlocked.CompareExchange(ref _fault, ex, null); throw; } + } + var subscription = options.Scenario == "queue" + ? await _bus.ConsumeAsync(HandleAsync, new() { Destination = "input", AckMode = AckMode.Manual, MaxConcurrency = options.ConsumerConcurrency }, token) + : await _bus.SubscribeAsync(HandleAsync, new() { Topic = "events", Subscription = "group" + group, AckMode = AckMode.Manual, MaxConcurrency = options.ConsumerConcurrency }, token); + _subscriptions.Add(subscription); + await subscription.WaitUntilReadyAsync(token); + } + } + + public async Task SendAsync(LoadMessage[] messages, CancellationToken token) + { + if (options.Scenario == "queue") + { + if (messages.Length == 1) await _bus!.SendAsync(messages[0], new() { Destination = "input" }, token); + else await _bus!.SendBatchAsync(messages, new() { Destination = "input" }, token); + } + else + { + if (messages.Length == 1) await _bus!.PublishAsync(messages[0], new() { Topic = "events" }, token); + else await _bus!.PublishBatchAsync(messages, new() { Topic = "events" }, token); + } + } + + public void ThrowIfFaulted() + { + if (Volatile.Read(ref _fault) is { } fault) throw new InvalidOperationException("Foundatio receive or acknowledgement failed.", fault); + } + + public async ValueTask DisposeAsync() + { + foreach (var subscription in _subscriptions) await subscription.DisposeAsync(); + if (_bus is not null) await _bus.DisposeAsync(); + if (_transport is not null) await _transport.DisposeAsync(); + if (_redis is not null) + { + foreach (var endpoint in _redis.GetEndPoints()) + { + var server = _redis.GetServer(endpoint); + if (server.IsReplica) continue; + var keys = server.Keys(pattern: prefix + ":*").ToArray(); + if (keys.Length > 0) await _redis.GetDatabase().KeyDeleteAsync(keys); + } + await _redis.DisposeAsync(); + } + if (options.Transport == "sqs") await AwsResources.CleanupAsync(prefix); + _logs.Dispose(); + } +} diff --git a/benchmarks/Messaging/IMessagingDriver.cs b/benchmarks/Messaging/IMessagingDriver.cs new file mode 100644 index 000000000..91cf2f3d8 --- /dev/null +++ b/benchmarks/Messaging/IMessagingDriver.cs @@ -0,0 +1,8 @@ +namespace Foundatio.Messaging.Benchmarks; + +public interface IMessagingDriver : IAsyncDisposable +{ + Task StartAsync(Action received, CancellationToken token); + Task SendAsync(LoadMessage[] messages, CancellationToken token); + void ThrowIfFaulted(); +} diff --git a/benchmarks/Messaging/JOB_TRACKING_RESULTS.md b/benchmarks/Messaging/JOB_TRACKING_RESULTS.md new file mode 100644 index 000000000..ee04d751a --- /dev/null +++ b/benchmarks/Messaging/JOB_TRACKING_RESULTS.md @@ -0,0 +1,29 @@ +# Messaging and job tracking — September 12, 2026 + +This pass removes lease-monitoring work from short deliveries while preserving renewal, expiry, cancellation and settlement races. The asynchronous monitor starts at the first scheduled lease check; a timer supervises the delivery from admission. Six new regressions cover its lifecycle. + +External project references now preserve Release/Debug configuration through the entire graph. Previously a Release Mediator solution build could copy Debug native dependencies. The integration's CI smoke check now rejects unoptimized benchmark assemblies. + +## Performance + +Fresh-process medians on system .NET 10.0.12, Release, comparing the previous native implementation with this change: + +| Workload | PR #149 jobs/s | Before jobs/s | After jobs/s | Allocated bytes/job, before → after | +| --- | ---: | ---: | ---: | ---: | +| In memory, concurrency 64 | 181,195 | 95,696 | 100,346 | 8,358 → 7,928 | +| In memory, tracked, concurrency 64 | 36,658 | 33,238 | 34,111 | 15,535 → 15,146 | +| In memory, concurrency 1 | 110,805 | 93,045 | 100,537 | 8,686 → 8,271 | +| In memory, concurrency 8 | 177,069 | 84,916 | 91,161 | 8,340 → 7,934 | +| SQS / LocalStack, concurrency 64 | 2,788 | 3,065 | 2,968 | 47,026 → 46,623 | +| In memory + Redis tracking, concurrency 64 | 10,619 | 7,757 | 7,600 | 45,852 → 45,450 | +| SQS / LocalStack + Redis tracking, concurrency 64 | 2,702 | 2,653 | 2,582 | 84,285 → 83,930 | + +Untracked in-memory throughput improves **5–8%**, allocations fall **about 5%**, and concurrency-64 process CPU falls **17%**. Longer tracked-memory runs are level. Redis tracking is **4% slower** in five longer alternating pairs, with **5% less CPU** and **1% fewer allocated bytes**; no Redis/SQS speedup is claimed. Default 1 ms receive collection improves about **7%**. The bus-only diagnosis drops from **4,870 to 4,250 bytes/delivery**; layer timings are not independently subtractable costs. + +The main matrix verifies **4.32M jobs** in 63 runs. Thirty longer/default checks verify another **3.2M**. Each matrix cell uses three rotating trials, 1,000 warmup messages and a 256-character payload. Timing includes broker drain and tracked completion. LocalStack is not production AWS capacity. [Method, latency, CPU, raw data, source fingerprints and excluded mixed-build trials](https://github.com/FoundatioFx/Foundatio.Mediator/tree/codex/core-distributed-alternative/benchmarks/Foundatio.Mediator.Distributed.Benchmarks/comparison/production-pass-2026-09-12). [Raw native-side data](baselines/job-tracking-production-2026-09-12) and [previous measurements](baselines/job-tracking-pass2-2026-09-08). + +## Correctness + +Full build and **2,235 Foundatio tests pass**, with 24 existing skips and the existing AppHost ASPIRE010 warning. **756 Mediator tests**, 23 browser scenarios, Quickstart, console, frontend and docs checks pass. Mediator core still matches main exactly. + +A ten-minute LocalStack/Redis run accepted **69,354 jobs**; **69,314 completed** and **40 were intentionally cancelled**. All **32** deliveries interrupted by a worker crash retried after replacement; another worker restarted gracefully during arrivals. Nothing remained pending or failed, and no duplicate effects were observed. The harness implements idempotent effects and does not claim exactly-once delivery. Real AWS deployment validation and a longer staging soak remain release work. diff --git a/benchmarks/Messaging/LatencyHistogram.cs b/benchmarks/Messaging/LatencyHistogram.cs new file mode 100644 index 000000000..0057fb9d7 --- /dev/null +++ b/benchmarks/Messaging/LatencyHistogram.cs @@ -0,0 +1,49 @@ +using System.Numerics; + +namespace Foundatio.Messaging.Benchmarks; + +public sealed class LatencyHistogram +{ + private readonly long[] _buckets = new long[2048]; + private long _count; + private long _maximum; + + public void RecordMicroseconds(long microseconds) + { + microseconds = Math.Max(0, microseconds); + int exponent = microseconds < 64 ? 0 : BitOperations.Log2((ulong)microseconds) - 6; + int index = checked((int)(microseconds < 64 ? microseconds : (exponent * 64) + (microseconds >> exponent))); + Interlocked.Increment(ref _buckets[Math.Min(index, _buckets.Length - 1)]); + Interlocked.Increment(ref _count); + long previous = Volatile.Read(ref _maximum); + while (microseconds > previous) + { + long observed = Interlocked.CompareExchange(ref _maximum, microseconds, previous); + if (observed == previous) break; + previous = observed; + } + } + + public LatencySummary Snapshot() + { + long count = Volatile.Read(ref _count); + double Percentile(double p) + { + if (count == 0) return 0; + long target = (long)Math.Ceiling(count * p), accumulated = 0; + for (int index = 0; index < _buckets.Length; index++) + { + accumulated += Volatile.Read(ref _buckets[index]); + if (accumulated >= target) + { + long upper = index < 64 ? index : ((65L + (index % 64)) << ((index / 64) - 1)) - 1; + return Math.Min(upper, Volatile.Read(ref _maximum)) / 1000d; + } + } + return Volatile.Read(ref _maximum) / 1000d; + } + return new(count, Percentile(.5), Percentile(.95), Percentile(.99), Volatile.Read(ref _maximum) / 1000d); + } +} + +public sealed record LatencySummary(long Count, double P50Milliseconds, double P95Milliseconds, double P99Milliseconds, double MaxMilliseconds); diff --git a/benchmarks/Messaging/MassTransitDriver.cs b/benchmarks/Messaging/MassTransitDriver.cs new file mode 100644 index 000000000..674e891ac --- /dev/null +++ b/benchmarks/Messaging/MassTransitDriver.cs @@ -0,0 +1,92 @@ +using MassTransit; + +namespace Foundatio.Messaging.Benchmarks; + +public sealed class MassTransitDriver(BenchmarkOptions options, string prefix) : IMessagingDriver, IReceiveObserver +{ + private IBusControl? _bus; + private ISendEndpoint? _send; + private ConnectHandle? _observer; + private Action _received = null!; + private Exception? _fault; + + public async Task StartAsync(Action received, CancellationToken token) + { + _received = received; + if (options.Transport == "sqs") + { + _bus = Bus.Factory.CreateUsingAmazonSqs(cfg => + { + cfg.Host(AwsResources.Region.SystemName, h => + { + h.Scope(prefix, true); + if (AwsResources.LocalCredentials is { } credentials) h.Credentials(credentials); + h.Config(AwsResources.SqsConfig); h.Config(AwsResources.SnsConfig); + }); + cfg.Message(m => m.SetEntityName(prefix + "events")); + for (int group = 0; group < options.DeliveryCopies; group++) + { + int subscriber = group; + cfg.ReceiveEndpoint(prefix + "input" + group, e => Configure(e, subscriber)); + } + }); + } + else + { + _bus = Bus.Factory.CreateUsingInMemory(cfg => + { + for (int group = 0; group < options.DeliveryCopies; group++) + { + int subscriber = group; + cfg.ReceiveEndpoint(prefix + "input" + group, e => Configure(e, subscriber)); + } + }); + } + _observer = _bus.ConnectReceiveObserver(this); + await _bus.StartAsync(token); + _send = await _bus.GetSendEndpoint(new Uri("queue:" + prefix + "input0")); + } + + private void Configure(IReceiveEndpointConfigurator endpoint, int subscriber) + { + endpoint.PrefetchCount = options.Prefetch; + endpoint.ConcurrentMessageLimit = options.ConsumerConcurrency; + endpoint.ConfigureConsumeTopology = options.Scenario == "pubsub"; + endpoint.Handler(context => + { + context.ReceiveContext.GetOrAddPayload(() => new Receipt(subscriber, context.Message)); + return Task.CompletedTask; + }); + } + + public Task SendAsync(LoadMessage[] messages, CancellationToken token) + { + if (options.Scenario == "queue") return messages.Length == 1 ? _send!.Send(messages[0], token) : _send!.SendBatch(messages, token); + return messages.Length == 1 ? _bus!.Publish(messages[0], token) : _bus!.PublishBatch(messages, token); + } + + public Task PreReceive(ReceiveContext context) => Task.CompletedTask; + public Task PostReceive(ReceiveContext context) + { + if (context.TryGetPayload(out var receipt)) _received(receipt.Subscriber, receipt.Message); + return Task.CompletedTask; + } + public Task PostConsume(ConsumeContext context, TimeSpan duration, string consumerType) where T : class => Task.CompletedTask; + public Task ConsumeFault(ConsumeContext context, TimeSpan duration, string consumerType, Exception exception) where T : class => ReceiveFault(context.ReceiveContext, exception); + public Task ReceiveFault(ReceiveContext context, Exception exception) { Interlocked.CompareExchange(ref _fault, exception, null); return Task.CompletedTask; } + public void ThrowIfFaulted() + { + if (Volatile.Read(ref _fault) is { } fault) throw new InvalidOperationException("MassTransit receive or acknowledgement failed.", fault); + } + public async ValueTask DisposeAsync() + { + if (_bus is not null) + { + using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(30)); + await _bus.StopAsync(timeout.Token); + _observer?.Disconnect(); + } + if (options.Transport == "sqs") await AwsResources.CleanupAsync(prefix); + } + private sealed record Receipt(int Subscriber, LoadMessage Message); +} diff --git a/benchmarks/Messaging/PIPELINE_RESULTS.md b/benchmarks/Messaging/PIPELINE_RESULTS.md new file mode 100644 index 000000000..7c11ca7a9 --- /dev/null +++ b/benchmarks/Messaging/PIPELINE_RESULTS.md @@ -0,0 +1,129 @@ +# Messaging performance and reliability follow-up + +This follow-up improves ordinary queue and pub/sub operations without requiring applications to call batch APIs or tune transport internals. Consumer concurrency remains a strict bound. Sends and publishes still wait for broker acceptance, while settlement waits for the individual broker acknowledgement. Lease supervision, cancellation, retry and shutdown behavior remain enabled. + +## Repeated standard comparison + +| Transport / workload | Foundatio inputs/s | MassTransit inputs/s | Throughput ratio | Foundatio p99 ms | MassTransit p99 ms | +| --- | --- | --- | --- | --- | --- | +| Memory / serial | 234,753 | 127,380 | 1.84x | 6.01 | 12.03 | +| Memory / queue | 262,812 | 102,870 | 2.55x | 5.63 | 14.59 | +| Memory / fanout | 170,397 | 75,645 | 2.25x | 8.13 | 20.73 | +| LocalStack / serial | 402 | 284 | 1.42x | 2,490.37 | 3,260.86 | +| LocalStack / queue | 3,019 | 2,425 | 1.25x | 638.98 | 794.62 | +| LocalStack / pubsub-one | 1,536 | 1,131 | 1.36x | 966.65 | 1,196.03 | +| LocalStack / fanout | 425 | 423 | 1.00x | 2,850.82 | 3,506.18 | + +The four-subscriber LocalStack fanout medians are effectively tied: the small difference falls within overlapping observed ranges (Foundatio 389–430; MassTransit 402–424 inputs/s). In-memory and other standard AWS medians favor Foundatio. Across the final 93 trials, 82,408,651 inputs produced 215,044,147 independently validated acknowledged deliveries, with zero missing, duplicate or invalid deliveries and no worker failures. These results establish performance for this matrix and environment, not universal dominance across workloads, brokers or latency/throughput objectives. + +## Preserved implementation comparison + +| Transport / workload | Before inputs/s | After inputs/s | Ratio | +| --- | --- | --- | --- | +| Memory / serial | 121,746 | 234,753 | 1.93x | +| Memory / queue | 259,229 | 262,812 | 1.01x | +| Memory / fanout | 158,447 | 170,397 | 1.08x | +| Redis / queue | 20,970 | 21,179 | 1.01x | +| Redis / fanout | 7,837 | 7,979 | 1.02x | + +Memory concurrent queues and both Redis cases are close to their previous throughput; the larger gains are in memory serial queues and AWS. The earlier same-runtime AWS comparison is retained separately as `official-aws`; it measured the intermediate `d07031ff` implementation against the preserved build and MassTransit. Do not merge that intermediate comparison into the final repetitions. + +## Payload and batch checks + +One fifteen-second trial per cell, with three seconds of warmup. These are checks of additional workloads, not repeated confidence estimates. + +| LocalStack case | Foundatio inputs/s | MassTransit inputs/s | Foundatio p99 ms | MassTransit p99 ms | +| --- | --- | --- | --- | --- | +| 16 KiB / queue | 2,148 | 1,849 | 770.05 | 868.35 | +| 16 KiB / fanout | 314 | 296 | 4,325.38 | 4,456.45 | +| Batch of ten / queue | 2,966 | 2,552 | 737.28 | 696.32 | +| Batch of ten / fanout | 467 | 446 | 3,309.57 | 2,719.74 | + +Explicit batches use eight producer workers and ten inputs per API call. The batch API comparisons favor Foundatio throughput, but the sampled batch p99 values favor MassTransit. Throughput gains do not imply winning every latency statistic. + +## Offered rates and round trips + +One twenty-second trial per offered-rate cell. Latency starts at the intended schedule, including capacity waits. Both implementations achieved the configured offered rates to rounding. + +| Target inputs/s / workload | Foundatio completed/s | MassTransit completed/s | Foundatio p99 ms | MassTransit p99 ms | +| --- | --- | --- | --- | --- | +| 10 / queue | 10.0 | 10.0 | 10.11 | 11.13 | +| 10 / fanout | 10.0 | 10.0 | 27.90 | 32.00 | +| 100 / queue | 100.0 | 100.0 | 7.36 | 9.47 | +| 100 / fanout | 99.8 | 99.8 | 425.98 | 458.75 | + +With one input outstanding, one producer and one consumer, the repeated queue round-trip medians were 390 completed inputs/s and 4.42 ms p99 for Foundatio, versus 213 inputs/s and 6.46 ms for MassTransit. This window-one test is separate from the standard serial saturation case. + +## Two-minute soaks + +One 120-second publishing window per cell after five seconds of warmup; every admitted input is drained and validated. All soak variants reserve the same 100-million-input tracking capacity, compared with 20 million for the shorter profiles. RSS therefore must not be compared directly across these profile types. + +| Transport / implementation / workload | Inputs | Inputs/s | p99 ms | Peak working set MiB | +| --- | --- | --- | --- | --- | +| aws / after / fanout | 52,517 | 436 | 2,949.12 | 131.4 | +| aws / after / queue | 368,559 | 3,066 | 663.55 | 137.6 | +| aws / masstransit / fanout | 53,328 | 441 | 2,818.05 | 156.2 | +| aws / masstransit / queue | 317,017 | 2,637 | 729.09 | 154.6 | +| memory / after / fanout | 21,320,538 | 177,662 | 6.78 | 218.3 | +| memory / masstransit / fanout | 9,081,729 | 75,674 | 21.76 | 223.9 | +| redis / after / fanout | 990,632 | 8,251 | 165.89 | 226.9 | +| redis / after / queue | 2,585,387 | 21,537 | 88.06 | 124.1 | + +## Client cost + +| Transport / workload | Foundatio bytes/input | MassTransit bytes/input | Foundatio CPU ms/input | MassTransit CPU ms/input | +| --- | --- | --- | --- | --- | +| Memory / serial | 11,905 | 19,673 | 0.028 | 0.028 | +| Memory / queue | 12,031 | 22,653 | 0.035 | 0.036 | +| Memory / fanout | 26,493 | 65,948 | 0.085 | 0.116 | +| LocalStack / serial | 129,701 | 168,693 | 1.162 | 1.934 | +| LocalStack / queue | 24,672 | 67,514 | 0.359 | 0.564 | +| LocalStack / pubsub-one | 38,781 | 62,487 | 0.439 | 0.789 | +| LocalStack / fanout | 126,979 | 86,962 | 1.073 | 1.831 | + +AWS fanout allocation remains higher for Foundatio. The 16 KiB checks also allocate more for Foundatio: 274,385 versus 240,259 bytes/input for queues, and 619,397 versus 407,487 for fanout. Client allocation, end-to-end throughput and tail latency are separate measurements; no across-the-board allocation claim is made. + +## Changes + +- Receive-slot collection dispatches as soon as its batch fills. AWS can overlap up to four receives under one shared consumer budget; collection is serialized so simultaneous receivers do not split a useful batch into tiny requests. +- A settled handler releases its delivery capacity independently of slow cancellation callbacks. Deferred cancellation and lease cleanup have a separate bounded budget and are drained on shutdown. +- Ordinary AWS sends, publishes and acknowledgements coalesce automatically. Partial operation batches wait up to two milliseconds, subject to timer scheduling. SQS sends can dispatch immediately when idle; singleton streams skip repeated idle waits. Acknowledgements learn the maximum requested receive batch size, so an eight-slot consumer does not wait for two additional receipts. Already queued receipts can still fill the native ten-entry batch. Slow handlers retain the bounded partial-batch timeout. +- Batch collection no longer creates a cancellation exception for each timer expiry; normal lease-timer cancellation also avoids exception handling. Batch responses still validate individual outcomes, caller cancellation cannot cancel another caller's shared request, and admitted work drains before owned SDK clients are disposed. +- A versioned `fnd.envelope` AWS attribute carries the ID, content type, encoding and all headers. JSON/text bodies stay readable and binary bodies use base64. An empty-by-default `NativeMessageHeaders` collection can duplicate up to nine selected headers for SNS attribute filters. Names are validated and snapshotted at construction. +- Benchmark reports fingerprint the actual CoreCLR binary. `run.ps1 -DotnetPath` selects the worker host, and the summarizer rejects mixed runtime builds even when their displayed version matches. + +The AWS wire/default-header change intentionally affects the unreleased provider: new readers accept the previous envelope format, but previous experimental readers cannot read new sends. Upgrade endpoints together or use a new resource prefix. Existing native SNS attribute filters must select their header names explicitly; application handlers continue receiving all headers. + +## Measurement method + +The final implementation is `77c20ea354919fd25ae300e49c5de7f3ed8da598`. Preserved before binaries are the previous automatic-batching implementation, `abce1c0e`, from checkout `466e3987`. Binary manifests identify all executables. MassTransit is 8.5.10, pinned to `62ab339afa3bac2e9b3fe1769d0d35d7e44778e9`, using the same AWS SDK assemblies. + +All confirmed trials invoke Microsoft's official .NET 10.0.11 runtime with CoreCLR SHA-256 `3ebe90cd92b1edf6742a41fa921a0c6326216fd1cca45fdb5e055bea33351bea`, server GC, on the same shared Linux x64 host (Ubuntu 26.04.1, AMD Ryzen AI 9 HX 470, 24 logical processors). LocalStack 3.8.1 is limited to four CPUs and 3 GiB; Redis 8.6 to four CPUs and 2 GiB, with AOF everysec. Workers run sequentially in seeded shuffled order, with no concurrent builds, tests or profiling. The host is shared and has no CPU affinity; small differences and overlapping ranges should be treated cautiously. + +Standard cases have three fresh-process repetitions, ten seconds of publishing after up to three seconds of warmup, 1 KiB payloads and a 1,024-input outstanding window. Concurrent queues and one-subscriber pub/sub use 32 producers and 32 consumer slots. Four-subscriber fanout uses 32 producers and eight consumer slots per subscription. MassTransit prefetch equals its per-endpoint consumer limit. Serial queues use one producer and one consumer with the same outstanding window; the separate window-one profile measures one-at-a-time round trips. + +Throughput includes final acknowledgement and drain. Fanout rates count inputs; each input requires four independently validated acknowledged deliveries. Saturation p99 includes the bounded backlog and is not unloaded latency. Offered-rate latency starts at the intended arrival schedule and includes capacity waits. Counts and CPU/allocations include client and harness, excluding broker processes. RSS includes fixed tracker arrays and can vary with the portion touched during a run; comparisons must use the same tracker capacity and cannot by themselves establish a live-object leak. + +Baseline harnesses predate the CoreCLR fingerprint field; their exact official-runtime invocation and DLL hashes are preserved in each profile manifest. Earlier Ubuntu-runtime experiments are retained separately and are excluded from final same-runtime comparisons. + +## Runtime investigation + +Earlier benchmark workers exited with native CLR error `0x80131506`. The system SDK build also terminated with that error during this follow-up. This is separate from a managed assertion or delivery-validation failure. The installed Ubuntu .NET 10.0.11 runtime links external libunwind 1.8.3, whereas Microsoft's same-version runtime does not. The Ubuntu package predates a concurrent-unwinding fix discussed in the [upstream runtime issue](https://github.com/dotnet/runtime/issues/130577) and [libunwind change](https://github.com/libunwind/libunwind/pull/993). + +A standalone .NET console program with no Foundatio dependencies repeatedly threw/caught exceptions with 24 workers. Five five-second trials per runtime produced no crash in either build. The official runtime processed approximately 6.8 times as many exceptions. This demonstrates a material runtime difference, but does not reproduce or prove the cause of the historical crashes. Both contenders now use the same official runtime for confirmation; no system runtime was replaced. Original dumps and failed trials remain retained, and the Ubuntu-host crash remains unresolved. + +## Scope + +LocalStack is an emulator. These results do not establish live AWS throughput or latency, and no AWS account was contacted. Live mode remains explicitly selectable with the SDK credential chain and the same configuration for both contenders. Redis Streams is measured against Foundatio's preserved implementation; there is no MassTransit Redis transport in this comparison. + +## Validation and retained evidence + +- Core: 2,036 passed / 12 skipped. AWS: 63 passed / 8 skipped. Redis: 56 passed / 4 skipped. Measurement: 16 passed. **2,171 passed, 24 expected skips, zero regression-test failures.** All suites used the official runtime. The new acknowledgement-capacity regression failed before the change and passed afterward; the full AWS suite was rerun after the final AWS change. +- `Foundatio.slnx` Release build passed, with only the pre-existing ASPIRE010 warning. The aggregate `Foundatio.All.slnx` could not build in the isolated clone because its external sibling repositories are absent. The repository solution includes the temporary AWS and Redis providers and their tests. +- Targeted whitespace formatting, `git diff --check`, documentation build and benchmark summary regressions passed. The summary tests reject mixed runtime binaries even when the displayed runtime versions match. +- Regression coverage includes full receive-batch dispatch, overlapping receives under one capacity bound, bounded cleanup with blocked cancellation callbacks, sibling cancellation before reprovisioning, shared-batch cancellation, timeout/disposal behavior, per-entry acknowledgement validation, legacy/malformed envelope decoding and native-header validation. A LocalStack SNS tenant-filter test verifies selected native headers. +- The final resource inventory found no benchmark SQS queues, SNS topics or Redis keys. Conformance tests left their own 84 queues and six topics in the disposable broker; these were removed with the task-owned containers. Other development services were not modified. +- The final record audit verified all 93 unique resource prefixes, full publishing windows, bounded sampled outstanding counts, acknowledgement counts and histogram totals. Every admitted delivery drained. No crash dump was generated; temporary crash settings applied only to the completed Redis workers. +- [Raw results, manifests, scripts and summaries](baselines/2026-09-07-pipelines/) are retained. Earlier candidate and Ubuntu-runtime experiments are preserved separately in the local handoff; they are not merged into final medians. The diagnostic trace uses sampled thread time and includes waits; it is not a CPU-only hotspot ranking. + +Changes are committed locally and are not published to PR #533. No hosted CI result is claimed for the unpublished commits. diff --git a/benchmarks/Messaging/Program.cs b/benchmarks/Messaging/Program.cs new file mode 100644 index 000000000..cd06f4445 --- /dev/null +++ b/benchmarks/Messaging/Program.cs @@ -0,0 +1,12 @@ +using Foundatio.Messaging.Benchmarks; + +if (args.Contains("--help")) +{ + Console.WriteLine("Messaging load benchmark: --engine foundatio|masstransit|loopback --transport memory|redis|sqs --scenario queue|pubsub --seconds 15 --warmup 3 --producers 32 --consumers 32 --prefetch 32 --subscribers 4 --payload 1024 --batch 1 --rate 0 --outstanding 4096 --output result.json"); + Console.WriteLine("Connections: PERF_REDIS (localhost:16379), PERF_AWS_MODE=localstack|live (defaults to localstack), PERF_AWS_URL (http://localhost:24566; ignored in live mode), PERF_AWS_REGION (us-east-1). Live AWS uses the SDK credential chain. Each run creates and removes uniquely named queues/topics."); + return 0; +} +using var cancellation = new CancellationTokenSource(); +Console.CancelKeyPress += (_, e) => { e.Cancel = true; cancellation.Cancel(); }; +try { return await BenchmarkRunner.RunAsync(BenchmarkOptions.Parse(args), cancellation.Token); } +catch (Exception ex) { Console.Error.WriteLine(ex.Message); return 1; } diff --git a/benchmarks/Messaging/README.md b/benchmarks/Messaging/README.md new file mode 100644 index 000000000..ccb9296e6 --- /dev/null +++ b/benchmarks/Messaging/README.md @@ -0,0 +1,117 @@ +# Distributed messaging benchmarks + +A sustained-load harness for the unreleased messaging API. It complements the existing BenchmarkDotNet microbenchmarks with acknowledged queue throughput, pub/sub fanout, end-to-end latency, allocations, CPU/GC, backlog and delivery validation. + +See the [allocation profiling follow-up](ALLOCATION_RESULTS.md) for buffer serialization, AWS allocation changes, repeated comparisons and sustained-load validation. The [messaging pipeline follow-up](PIPELINE_RESULTS.md) covers the preceding same-runtime comparisons and low-load latency. The [AWS automatic batching report](AWS_BATCHING_RESULTS.md) retains earlier measurements. See [measured results and findings](RESULTS.md) for the checked-in baseline, the timer-retention fix it exposed, and unresolved native crash evidence. + +## Run locally + +Requires .NET 10, PowerShell 7 and Docker. Start disposable, isolated brokers; the compose file limits each broker to four CPUs and enables Redis AOF with `appendfsync everysec`. + +```powershell +$env:PERF_AWS_MODE = 'localstack' +docker compose -f benchmarks/Messaging/docker-compose.yml up -d +dotnet build benchmarks/Messaging.Tests -c Release +dotnet benchmarks/Messaging.Tests/bin/Release/net10.0/Foundatio.Messaging.Benchmarks.Tests.dll +./benchmarks/Messaging.Tests/SummaryTests.ps1 +./benchmarks/Messaging/run.ps1 -Profile smoke -NoBuild +./benchmarks/Messaging/run.ps1 -Profile standard -Repetitions 3 -Seconds 15 -NoBuild +./benchmarks/Messaging/run.ps1 -Profile extended -Repetitions 3 -Seconds 15 -NoBuild +./benchmarks/Messaging/run.ps1 -Profile soak -NoBuild + +docker compose -f benchmarks/Messaging/docker-compose.yml down -v +``` + +Use `-DotnetPath /path/to/dotnet` to select the worker runtime host; builds still use the SDK on `PATH`. Results record the CoreCLR binary SHA-256, and the summarizer rejects mixed runtime builds even if they report the same .NET version. + +Profiles: + +- `smoke`: one second each of concurrent queues and four-way fanout; correctness only. +- `standard`: serial queues, concurrent queues, one-subscriber events and four-subscriber fanout. Payload is 1 KiB, with up to three seconds of warmup followed by fifteen seconds of publishing by default. Warmup is capped at one million inputs; the measured publishing window must complete in full. +- `extended`: 16 KiB queue/fanout payloads and ten-input queue/fanout batch API calls. +- `soak`: two-minute concurrent queue and four-subscriber fanout runs per implementation. + +The standard profile runs 20 configurations × 3 repetitions = 60 fresh processes. Allow roughly 20–30 minutes, including warmup, broker setup, draining and cleanup. Trials run sequentially in seeded shuffled order, so two contenders never load the same broker simultaneously. Time windows exclude topology creation, startup, warmup, cleanup and JSON report generation. Do not build, run tests, profile, or run other workloads concurrently with measurements. + +Results go to a timestamped `results/` directory: individual JSON/log files, throughput ranges and medians in `summary.md`, `summary.csv`, runtime information and repository state. The measurement executable exits nonzero for send/receive failures, missing or invalid deliveries, duplicates, timeout, cleanup failure or exhausted tracking capacity. Invalid trials are excluded from successful summaries and listed explicitly. Inspect failures before comparing throughput. + +Each matrix invocation requires an empty output directory. This preserves earlier trials and prevents an old successful JSON file from being mistaken for the result of a new worker that crashed. + +## What is compared + +| Engine | Transport | Meaning | +| --- | --- | --- | +| Foundatio | In-memory | Full serialization, routing, receive and acknowledgement path | +| MassTransit 8.5.10 | In-memory | Full MassTransit pipeline using its own in-memory transport | +| Foundatio | Redis Streams | Real Redis broker, durable consumer groups and acknowledgements | +| Foundatio | SQS/SNS | Shared-broker comparison against MassTransit | +| MassTransit 8.5.10 | SQS/SNS | Same SQS queues / SNS fanout semantics and broker instance, with separate run namespaces | + +MassTransit has no Redis Streams transport. Its Redis saga repository is not a message transport. The two in-memory implementations have different internals and are not wire-compatible. The SQS/SNS comparison is the shared-transport comparison. LocalStack is an emulator: use those results to investigate client/API behavior, not to predict AWS service throughput or latency. + +MassTransit 8.5.10 is pinned as the latest Apache-licensed v8 release available when this suite was created. Its package version is an MSBuild property so another supported version can be tested deliberately. The benchmark project is isolated from shipping packages. Both contenders run in the same executable dependency graph and use the same AWS SDK/runtime versions; actual versions are embedded in every JSON result. No application contracts or messages pass between the contenders. + +All cases use the same ASCII payload, producer parallelism and per-endpoint consumer limit. MassTransit prefetch is explicitly matched to that limit. Fanout uses separate durable subscription queues, and per-endpoint concurrency is reported rather than pretending that four subscriptions have the same total concurrency as one. The baseline calls the ordinary send/publish API once per input. Batch cases call each library's public batch API; native batching, pipelining and acknowledgement buffering remain part of the implementation being measured. Collection size does not imply a single atomic broker request. Default serializers/envelopes remain enabled, so equal application payload sizes do not imply equal wire bytes. + +## Measurement contract + +- Delivery latency starts immediately before submission and ends after broker acknowledgement. Foundatio explicitly completes its manual delivery before recording it. MassTransit's `IReceiveObserver.PostReceive` runs after `ReceiveLock.Complete`; its consumer attaches the message to the receive context for that observer. This instrumentation and payload validation are included in client CPU/allocation totals. +- Queue throughput counts unique acknowledged inputs. Pub/sub reports both inputs/second and deliveries/second: one input with four subscribers produces four expected deliveries. The denominator includes draining the last submitted work, preventing an undrained backlog from looking like throughput. +- The outstanding window bounds admitted inputs and releases an input only when all its subscriber copies have acknowledged. It is not a fire-and-forget producer benchmark. Every input/subscriber pair is tracked separately; duplicate copies cannot hide missing fanout. +- The tracking arrays are allocated before measurement. Their size is bounded by `--max-messages`; hitting that limit invalidates the trial rather than silently shortening it. Normal profiles reserve capacity for 20 million inputs; the soak profile reserves 100 million. Long/faster runs may require splitting trials. Working-set results include those fixed tracking arrays and warmup state, so compare memory usage only between trials with equal tracking capacity. +- Per-process allocated bytes, CPU time, GC collections and pauses cover producer, consumers and harness. They exclude Redis/LocalStack processes. One-second samples preserve backlog, throughput, working set and GC heap/commitment/fragmentation trends. GC memory information describes the last completed collection, not a live-object census; growing RSS alone does not prove a leak. `SendCallLatency` is per API call, so a batch of ten represents ten inputs. +- Latency histograms retain all samples, including slow tails, in fixed storage with one-microsecond resolution below 64 microseconds and at most approximately 1.6 percent bucket width above it. Percentiles use bucket upper bounds; maximum is exact to the recorded microsecond. Reported p50/p95/p99 are medians of each trial's percentiles, not a percentile formed by averaging durations. +- Saturation tests (`--rate 0`) are bounded closed-loop tests. Offered-rate tests use each input's intended schedule as its latency origin, including time waiting for producer capacity. A monotonic-clock recheck prevents early submission from timer rounding; scheduling has millisecond granularity. They expose scheduling/backpressure delay instead of hiding coordinated omission. If the configured offered rate is not achieved, report that deficit; these tests do not create an unlimited external arrival queue. +- Automatic retries and at-least-once delivery can produce duplicates. Those are reported and invalidate the performance comparison for investigation; this does not claim either library guarantees exactly-once side effects. + +## Individual and offered-rate cases + +```powershell +$runner = 'benchmarks/Messaging/bin/Release/net10.0/Foundatio.Messaging.Benchmarks.dll' +dotnet $runner --engine foundatio --transport redis --scenario queue --seconds 30 --warmup 5 --producers 32 --consumers 32 --prefetch 32 --outstanding 1024 --payload 1024 --output redis-queue.json + +dotnet $runner --engine masstransit --transport sqs --scenario pubsub --subscribers 4 --seconds 30 --producers 8 --consumers 8 --prefetch 8 --rate 200 --outstanding 1024 --output sqs-rate200.json + +# Sanity-check the shared tracking/generation overhead without serialization or a broker. +dotnet $runner --engine loopback --transport memory --scenario queue --seconds 5 --output harness-overhead.json +``` + +## LocalStack and real AWS + +LocalStack is the default, including when AWS credentials are already available on the machine. Foundatio, MassTransit and resource cleanup all use the same mode, endpoint, region and credential selection. + +| Setting | Default | Behavior | +| --- | --- | --- | +| `PERF_AWS_MODE` | `localstack` | `localstack` or `live`, case insensitive; unknown values fail before connecting | +| `PERF_AWS_URL` | `http://localhost:24566` | LocalStack endpoint; ignored in live mode | +| `PERF_AWS_REGION` | `us-east-1` | Region for both SQS and SNS; also the LocalStack signing region | +| `PERF_REDIS` | `localhost:16379` | Redis connection string | + +LocalStack uses explicit `test` credentials. Live mode uses regional AWS endpoints and the [AWS SDK credential chain](https://docs.aws.amazon.com/sdk-for-net/v4/developer-guide/creds-assign.html), including temporary environment credentials, shared credentials profiles and instance/task roles. The harness does not copy credentials into results. Environment credentials take precedence over `AWS_PROFILE`; use one credential source deliberately. + +To compare both implementations in an AWS account, use a benchmark account/region with permissions to provision SQS queues and SNS topics, configure subscriptions and queue policies, send/receive/acknowledge messages, and delete the run's resources. Adapt the [MassTransit IAM example](https://masstransit.massient.com/configuration/transports/amazon-sqs#example-iam-policy) to your account and `fperf-*` queue/topic names; cleanup additionally requires `sqs:ListQueues`, `sns:ListTopics` and `sns:DeleteTopic`. [SQS listing uses the account/region wildcard queue ARN](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-api-permissions-reference.html); SNS listing uses `Resource: "*"`. Retain the example's `sqs:DeleteQueue` permission. + +```powershell +dotnet build benchmarks/Messaging -c Release +$env:PERF_AWS_MODE = 'live' +$env:PERF_AWS_REGION = 'us-east-1' +# Optional: select a shared-credentials profile when not using an instance/task role. +$env:AWS_PROFILE = 'foundatio-benchmarks' +try { + ./benchmarks/Messaging/run.ps1 -Profile smoke -Engines @('foundatio-sqs', 'masstransit-sqs') -NoBuild + ./benchmarks/Messaging/run.ps1 -Profile standard -Engines @('foundatio-sqs', 'masstransit-sqs') -Repetitions 3 -Seconds 15 -NoBuild +} +finally { + $env:PERF_AWS_MODE = 'localstack' +} +``` + +These commands use real, billable AWS services. No Docker broker is needed in live mode. Use the same client host and AWS region for both contenders, preferably a host in that region, and retain the broker/client configuration with the results. Mode and region appear in worker logs, JSON results and summaries. The summarizer rejects a mixture of LocalStack/live or different AWS regions; keep separate output directories for each environment. The checked-in baseline was measured against LocalStack, not an AWS account. + +Each invocation creates and removes its own `fperf-` resources. Cleanup lists resources and deletes only names starting with that invocation's random prefix; it never purges arbitrary application queues. If interrupted before cleanup, use the prefix in its log/result to identify only that run's resources. + +## References + +- [MassTransit SQS/SNS configuration](https://masstransit.massient.com/configuration/transports/amazon-sqs) +- [Pinned MassTransit acknowledgement/observer ordering](https://github.com/MassTransit/MassTransit/blob/62ab339afa3bac2e9b3fe1769d0d35d7e44778e9/src/MassTransit/Transports/ReceivePipeDispatcher.cs) +- [MassTransit 8.5.10 package](https://www.nuget.org/packages/MassTransit/8.5.10) diff --git a/benchmarks/Messaging/RESULTS.md b/benchmarks/Messaging/RESULTS.md new file mode 100644 index 000000000..6995666d9 --- /dev/null +++ b/benchmarks/Messaging/RESULTS.md @@ -0,0 +1,133 @@ +# Distributed messaging performance results + +The [AWS batching follow-up](AWS_BATCHING_RESULTS.md) contains the newer optimized SQS/SNS measurements. The tables below preserve the original baseline. + +Measured September 6, 2026. The sustained workload exposed and helped fix excessive timer retention in the in-memory transport. Foundatio leads the concurrent in-memory cases; MassTransit leads the concurrent SQS/SNS emulator cases. Serial queues provide counterexamples to any claim of a universal winner. + +141 benchmark trials are retained: 139 succeeded and 2 failed. Successful trials accounted for 170,917,727 inputs and 311,455,223 unique acknowledged deliveries. These totals include the retained before measurements and the loopback control, and exclude warmup. + +Native CLR crashes remain unexplained. They are recorded below and prevent treating this work as a complete reliability qualification. + +## Method and environment + +- AMD Ryzen AI 9 HX 470, 24 logical processors, Linux x64, .NET 10.0.11 / SDK 10.0.111, server GC, Release builds. The client host was shared with other development services; there was no CPU affinity or dedicated host isolation. +- Redis 8.6 is a real Redis broker, with AOF enabled and `appendfsync everysec`. LocalStack 3.8.1 emulates SQS/SNS. Each broker container had a four-CPU limit; Redis had 2 GiB and LocalStack 3 GiB memory limits. Redis acknowledgement does not wait for an fsync on every input. This is not a durability-loss experiment or a live AWS benchmark. +- MassTransit 8.5.10 was pinned. Both drivers use the same executable dependency graph, runtime and AWS SDK versions. Exact versions, source revisions, options and host metadata accompany the raw trials. +- Standard cases use 1 KiB application payloads, a 1,024-input outstanding window, and three fresh-process repetitions of ten seconds of publishing. Warmup runs for up to three seconds or one million inputs. The two-minute soaks use up to five seconds of warmup and a 100-million-input tracking capacity; other profiles use 20 million. +- Concurrent queues have 32 producer workers and 32 consumer slots. Four-way fanout has 32 producers and eight consumer slots in each of four subscriptions. MassTransit prefetch matches the per-endpoint consumer limit. Native batching and default envelopes remain enabled. +- Delivery latency ends after broker acknowledgement. Throughput includes the final drain. Every input/subscriber pair is validated; duplicates cannot hide missing copies. CPU, allocations and memory include the client and harness, and exclude broker processes. All measured workers ran sequentially, without local builds or test suites competing with them. +- Tables show medians of successful trials. The [raw summaries](baselines/2026-09-06/) retain ranges and all failures; three samples do not establish statistical confidence. The in-memory tables use the repeated measurements after the timer fix; distributed baseline code was unchanged by that fix. Profile revisions remain separate. + +## Concurrent queues + +| Implementation | Inputs/s | p99 ms | Allocated bytes/input | Peak working set MiB | +| --- | ---: | ---: | ---: | ---: | +| Foundatio, memory | 258,087 | 5.44 | 12,474 | 121.9 | +| MassTransit, memory | 100,661 | 14.46 | 22,655 | 129.8 | +| Foundatio, Redis | 21,233 | 88.06 | 22,680 | 198.7 | +| Foundatio, SQS/SNS emulator | 715 | 1,802.24 | 21,007 | 131.6 | +| MassTransit, SQS/SNS emulator | 2,637 | 712.70 | 67,436 | 149.0 | + +## Four-subscriber fanout + +| Implementation | Published inputs/s | Acknowledged deliveries/s | p99 ms | +| --- | ---: | ---: | ---: | +| Foundatio, memory | 95,855 | 383,419 | 12.03 | +| MassTransit, memory | 73,940 | 295,761 | 21.50 | +| Foundatio, Redis | 7,975 | 31,899 | 159.74 | +| Foundatio, SQS/SNS emulator | 101 | 403 | 9,961.47 | +| MassTransit, SQS/SNS emulator | 420 | 1,680 | 3,375.10 | + +The saturation latency includes up to 1,024 admitted inputs waiting for their acknowledgements. It is not unloaded network latency; controlled-rate results appear below. + +## Timer retention fix + +The old in-memory transport created a visibility-reclaim timer on every receive and renewal. Completed messages left those timers alive until their deadlines. A regression with 100 completed-and-renewed deliveries observed 200 retained timers. The fix uses one shared timer, disables its polling when idle, restarts it when deliveries arrive, and removes expired receipts only if their lease has not changed. Tests cover bounded timer resources, idle clock advancement, disposal, renewed leases and waking blocked receivers. + +| Workload | Before inputs/s | After inputs/s | Before p99 ms | After p99 ms | Before peak MiB | After peak MiB | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| Concurrent queue | 213,749 | 258,087 | 16.89 | 5.44 | 1,326.4 | 121.9 | +| Four-subscriber fanout | 87,561 | 95,855 | 23.17 | 12.03 | 1,944.9 | 123.4 | + +The before fanout median has two successful trials and one native process crash. All three repeated after trials succeeded. This establishes the timer-resource improvement; it does not establish the cause of the native crashes. + +## Serial queues, larger payloads and batches + +Serial means one producer worker and one consumer slot, with the same outstanding window. It is not a one-message-at-a-time round-trip test. + +- Serial memory: Foundatio 112,277 inputs/s; MassTransit 122,978 inputs/s. +- Serial SQS/SNS emulator: Foundatio 406 inputs/s; MassTransit 282 inputs/s. + +Extended cases have one trial each and should be treated as exploratory. Batch cases use eight producer workers and ten inputs per API call; 16 KiB cases use 32 producers and single-input calls. The producer-count change means these are not isolated A/B estimates of batching alone. + +| Implementation | 16 KiB queue inputs/s | 16 KiB fanout inputs/s | Batch-10 queue inputs/s | Batch-10 fanout inputs/s | +| --- | ---: | ---: | ---: | ---: | +| Foundatio, memory | 79,281 | 59,583 | 282,546 | 98,545 | +| MassTransit, memory | 48,866 | 39,049 | 98,847 | 76,375 | +| Foundatio, Redis | 7,729 | 3,170 | 20,221 | 7,286 | +| Foundatio, SQS/SNS emulator | 646 | 91 | 1,117 | 109 | +| MassTransit, SQS/SNS emulator | 1,261 | 303 | 2,757 | 500 | + +## Two-minute soaks + +| Trial | Inputs | Inputs/s | p99 ms | Peak working set MiB | Missing / duplicates | +| --- | ---: | ---: | ---: | ---: | ---: | +| foundatio/memory pubsub | 11,701,265 | 97,507 | 11.65 | 172.6 | 0 / 0 | +| foundatio/memory queue | 31,348,296 | 261,223 | 5.50 | 238.7 | 0 / 0 | +| round1-foundatio-redis-pubsub-four.json | FAILED | | | | | +| foundatio/redis queue | 2,557,096 | 21,302 | 88.06 | 201.1 | 0 / 0 | +| foundatio/sqs pubsub | 12,723 | 98 | 11,403.26 | 133.4 | 0 / 0 | +| foundatio/sqs queue | 90,903 | 753 | 1,703.93 | 128.5 | 0 / 0 | +| masstransit/memory pubsub | 9,025,543 | 75,202 | 22.02 | 216.4 | 0 / 0 | +| masstransit/memory queue | 12,042,135 | 100,338 | 14.59 | 173.1 | 0 / 0 | +| masstransit/sqs pubsub | 54,104 | 446 | 2,818.05 | 153.8 | 0 / 0 | +| masstransit/sqs queue | 322,970 | 2,686 | 704.51 | 153.2 | 0 / 0 | +| foundatio/redis pubsub (repeat) | 954,234 | 7,947 | 169.98 | 212.3 | 0 / 0 | + +The soak tracking arrays are larger than the short-run arrays. Their pages become resident as sequence numbers advance, so RSS growth can reflect the tracker being touched. GC heap/commitment/fragmentation samples describe the last completed collection and do not count only live application objects. Inspect the raw time series before calling a trend a leak. + +## Controlled arrival rates + +These thirty-second trials timestamp each input at its intended schedule, including capacity delay. The timer rechecks a monotonic clock to prevent early sends from sub-millisecond rounding. Arrival scheduling has millisecond granularity. Admission below target is shown explicitly; the generator bounds outstanding work instead of maintaining an unlimited external arrival queue. + +| Trial | Target inputs/s | Admitted inputs/s | Acknowledged inputs/s including drain | p99 ms | +| --- | ---: | ---: | ---: | ---: | +| foundatio/memory pubsub | 50,000 | 50,000.4 | 49,998.3 | 2.85 | +| foundatio/memory queue | 50,000 | 50,000.1 | 49,998.9 | 1.63 | +| foundatio/redis pubsub | 10 | 10.0 | 10.0 | 103.42 | +| foundatio/redis pubsub | 3,000 | 2,999.9 | 2,997.9 | 27.39 | +| foundatio/redis queue | 10 | 10.0 | 10.0 | 103.42 | +| foundatio/redis queue | 3,000 | 3,000.0 | 2,997.9 | 27.14 | +| foundatio/sqs pubsub | 10 | 10.0 | 10.0 | 28.41 | +| foundatio/sqs pubsub | 100 | 100.0 | 88.2 | 4,325.38 | +| foundatio/sqs queue | 10 | 10.0 | 10.0 | 10.62 | +| foundatio/sqs queue | 100 | 100.0 | 100.0 | 8.96 | +| masstransit/memory pubsub | 50,000 | 49,999.7 | 49,990.1 | 6.46 | +| masstransit/memory queue | 50,000 | 49,999.9 | 49,988.9 | 2.24 | +| masstransit/sqs pubsub | 10 | 10.0 | 10.0 | 30.98 | +| masstransit/sqs pubsub | 100 | 100.0 | 99.9 | 430.08 | +| masstransit/sqs queue | 10 | 10.0 | 10.0 | 11.78 | +| masstransit/sqs queue | 100 | 100.0 | 100.0 | 9.73 | + +## Harness control and failures + +- Loopback pubsub: 1,242,226 inputs/s, 80 allocated bytes/input. This includes generation and validation without serialization or a broker; its costs remain included in every library result. +- Loopback queue: 1,466,421 inputs/s, 80 allocated bytes/input. This includes generation and validation without serialization or a broker; its costs remain included in every library result. +- Retained failed benchmark trial: `standard/round2-foundatio-memory-pubsub-four.json`. Worker exited without a result. RUN fperf-6b4027ac3eba foundatio/memory/pubsub +- Retained failed benchmark trial: `soak/round1-foundatio-redis-pubsub-four.json`. Worker exited without a result. RUN fperf-fd29ab56ca9c foundatio/redis/pubsub +- Two additional exploratory workers, one Foundatio in-memory queue and one Foundatio Redis queue, terminated with native CLR error `0x80131506` before the final profile sequence. Dump collection was enabled after the first occurrence. Native dumps are retained locally and are not committed to the repository. Their root cause is unconfirmed. +- Initial MassTransit fanout experiments received traffic from previous runs because explicitly named queues/topics bypassed the configured namespace. Both names now include the run prefix. Repeated queue/fanout verification finished with zero SQS queues and zero SNS topics; contaminated experiments are excluded from comparisons. +- A timer regression reproduced early scheduled submission, then passed after a monotonic-clock recheck was added. No controlled-rate results here use the earlier implementation. + +## Validation + +The full Release solution build succeeded with the existing AppHost ASPIRE010 warning. All 2,138 regression tests completed: 2,114 passed and 24 expected skips, with zero failures. This includes the Redis and AWS suites against the isolated brokers and seven measurement tests. The documentation build, PowerShell parsing, nonempty-output-directory guard and archived measurement invariants passed. These regression results do not erase the separate native benchmark failures. + +## Recommended next work + +1. Investigate the retained native CLR crashes before declaring release readiness. Passing later trials does not identify their cause. +2. Measure bounded automatic SQS/SNS batching as the next transport optimization. Foundatio currently deletes each completed SQS message individually. The pinned MassTransit implementation coalesces sends, publishes and deletes. Preserve per-input results, cancellation, actual acknowledgement completion and low-rate latency while testing any change. This is a source-based optimization hypothesis, not an isolated causal experiment. +3. Set an explicit Redis idle-latency budget. At ten inputs/second, these queue and fanout trials had p99 near 103 ms, versus about 27 ms at 3,000 inputs/second. The transport starts at a 25 ms poll interval and backs off up to one second when idle. Compare a tighter cap or a wake-up mechanism against idle CPU and broker request cost before changing defaults. +4. Repeat on a dedicated client host against live AWS in the same region, including several offered rates and longer runs. LocalStack numbers describe this emulator and client pipeline; they cannot size AWS. Keep the serial, fanout, payload and low-rate cases so an improvement in saturation throughput does not hide a usability regression. + +[Reproduction instructions and measurement contract](README.md) · [Raw trials, metadata and summaries](baselines/2026-09-06/) · [Pinned MassTransit queue batching](https://github.com/MassTransit/MassTransit/blob/62ab339afa3bac2e9b3fe1769d0d35d7e44778e9/src/Transports/MassTransit.AmazonSqsTransport/AmazonSqsTransport/QueueInfo.cs) · [Pinned MassTransit topic batching](https://github.com/MassTransit/MassTransit/blob/62ab339afa3bac2e9b3fe1769d0d35d7e44778e9/src/Transports/MassTransit.AmazonSqsTransport/AmazonSqsTransport/TopicInfo.cs) diff --git a/benchmarks/Messaging/RateSchedule.cs b/benchmarks/Messaging/RateSchedule.cs new file mode 100644 index 000000000..f95478cd4 --- /dev/null +++ b/benchmarks/Messaging/RateSchedule.cs @@ -0,0 +1,15 @@ +using System.Diagnostics; + +namespace Foundatio.Messaging.Benchmarks; + +public static class RateSchedule +{ + public static async Task WaitUntilAsync(long timestamp, CancellationToken token) + { + while (Stopwatch.GetTimestamp() < timestamp) + { + var remaining = Stopwatch.GetElapsedTime(Stopwatch.GetTimestamp(), timestamp); + await Task.Delay(TimeSpan.FromMilliseconds(Math.Max(1, Math.Ceiling(remaining.TotalMilliseconds))), token); + } + } +} diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/README.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/README.md new file mode 100644 index 000000000..036930404 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/README.md @@ -0,0 +1,21 @@ +# AWS batching measurements + +See [the follow-up report](../../AWS_BATCHING_RESULTS.md) for conclusions and methodology. These results supplement the historical 141-trial baseline; they do not overwrite it. + +| Profile | Trials | Implementation | +| --- | ---: | --- | +| confirmed-standard | 36 | Three interleaved repetitions of before / final / MassTransit across four workloads | +| confirmed-rate10 | 4 | Final / MassTransit, 30 seconds at 10 inputs/s | +| confirmed-rate100 | 4 | Final / MassTransit, 30 seconds at 100 inputs/s | +| confirmed-soak | 4 | Final / MassTransit, 120 seconds each, 20-million-input tracking capacity | +| confirmed-requests | 6 | Before / final / MassTransit, 10 seconds without warmup for broker request accounting | +| final-standard | 36 | Earlier fixed-delay candidate, retained under its original capture-directory name | +| adaptive-check | 6 | Five-second refinement check; exploratory | + +All 96 workers returned success. The final conclusions use the 54 `confirmed-*` trials. The `after` variant in `final-standard` is commit `1a9f0e62`; `after` in the other profiles is `abce1c0e`. The before binary was preserved at checkout `5dae40ef`, with unchanged shipping assemblies built at `95983490`. Each profile's binary manifest retains the actual source revision and SHA-256 of every dependency. + +`raw-trials.tar.gz` contains the individual JSON results, worker logs, native request counts, run options, binary manifests, per-revision summaries, and capture/validation scripts. The scripts record the original local paths; adapt those paths to prepared before/after executable directories when replaying elsewhere. The supported cross-platform entrypoint is the [PowerShell benchmark runner](../../README.md). Do not mix the earlier and final implementation summaries. + +The validation script checks success, exact fanout counts, histogram count, absence of lost/duplicate/invalid deliveries, tracking bounds, throughput arithmetic, and publishing duration with one-millisecond timer tolerance. Two retained durations fall less than 0.3 ms below their nominal boundaries. Both counts and actual elapsed times are retained; throughput uses actual elapsed time. LocalStack request logs include setup and cleanup; `confirmed-requests` omits warmup so measured input counts can be used to calculate messages per selected data-operation request. + +`analysis.json` contains medians, ranges, CPU, allocations, memory and totals per profile/variant. All counts include only measured inputs, excluding warmup. Hardware was recorded during the run in `host.json`; this was a shared development host. Broker limits are in the checked-in compose file. No AWS account was contacted. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/adaptive-check/after/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/adaptive-check/after/summary.csv new file mode 100644 index 000000000..5f4b706b2 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/adaptive-check/after/summary.csv @@ -0,0 +1,4 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","1","324.5","324.5","324.5","1298","2490.367","3702.783","3768.319","192605.6","2.0424","139.4","1996","0","0" +"foundatio/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1","1","348.5","348.5","348.5","348.5","2064.383","2228.223","2241.054","89479.1","2.2046","103.5","2439","0","0" +"foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","1","2088.4","2088.4","2088.4","2088.4","458.751","835.583","843.775","39907.7","0.5378","128.6","10994","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/adaptive-check/after/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/adaptive-check/after/summary.md new file mode 100644 index 000000000..76090c216 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/adaptive-check/after/summary.md @@ -0,0 +1,13 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 1 | 324.5 (324.5-324.5) | 1298 | 2490.367 / 3702.783 / 3768.319 | 192605.6 | 2.0424 | +| foundatio/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 1 | 348.5 (348.5-348.5) | 348.5 | 2064.383 / 2228.223 / 2241.054 | 89479.1 | 2.2046 | +| foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 1 | 2088.4 (2088.4-2088.4) | 2088.4 | 458.751 / 835.583 / 843.775 | 39907.7 | 0.5378 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/adaptive-check/masstransit/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/adaptive-check/masstransit/summary.csv new file mode 100644 index 000000000..91beeca81 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/adaptive-check/masstransit/summary.csv @@ -0,0 +1,4 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"masstransit/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","1","404.1","404.1","404.1","1616.3","2080.767","3112.959","3211.263","205907.6","2.1802","162.1","2408","0","0" +"masstransit/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1","1","269.2","269.2","269.2","269.2","1081.343","1671.167","1700.976","186254.6","3.1554","126.9","1804","0","0" +"masstransit/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","1","2456.5","2456.5","2456.5","2456.5","385.023","712.703","720.895","67620.8","0.8177","147.6","12895","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/adaptive-check/masstransit/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/adaptive-check/masstransit/summary.md new file mode 100644 index 000000000..bfb473540 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/adaptive-check/masstransit/summary.md @@ -0,0 +1,13 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| masstransit/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 1 | 404.1 (404.1-404.1) | 1616.3 | 2080.767 / 3112.959 / 3211.263 | 205907.6 | 2.1802 | +| masstransit/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 1 | 269.2 (269.2-269.2) | 269.2 | 1081.343 / 1671.167 / 1700.976 | 186254.6 | 3.1554 | +| masstransit/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 1 | 2456.5 (2456.5-2456.5) | 2456.5 | 385.023 / 712.703 / 720.895 | 67620.8 | 0.8177 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/analysis.json b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/analysis.json new file mode 100644 index 000000000..8b0e6b81e --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/analysis.json @@ -0,0 +1,784 @@ +{ + "final-standard": { + "after/fanout": { + "trials": 3, + "inputs": 10226, + "deliveries": 40904, + "throughput": 302.49201330086254, + "minimum": 289.88173983464077, + "maximum": 327.20706424160846, + "p50": 3276.799, + "p95": 4030.463, + "p99": 4259.839, + "sendP99": 75.775, + "bytes": 144762.1239157373, + "cpu": 1.4238458802608178, + "peakMiB": 132.82421875, + "admitted": 336.92951499455677 + }, + "after/pubsub-one": { + "trials": 3, + "inputs": 36705, + "deliveries": 36705, + "throughput": 1215.734741922291, + "minimum": 1135.6372166995243, + "maximum": 1225.6288124164557, + "p50": 786.431, + "p95": 1146.879, + "p99": 1163.263, + "sendP99": 31.487, + "bytes": 28470.097514340345, + "cpu": 0.48326601338432124, + "peakMiB": 139.18359375, + "admitted": 1247.2390313306164 + }, + "after/queue": { + "trials": 3, + "inputs": 71686, + "deliveries": 71686, + "throughput": 2327.834811161204, + "minimum": 2284.0151148678533, + "maximum": 2382.2761932273243, + "p50": 421.887, + "p95": 471.039, + "p99": 753.663, + "sendP99": 11.135, + "bytes": 59274.57550418101, + "cpu": 0.31834119668147154, + "peakMiB": 122.3203125, + "admitted": 2386.0621815842705 + }, + "after/serial": { + "trials": 3, + "inputs": 11098, + "deliveries": 11098, + "throughput": 277.17596366989835, + "minimum": 277.1103821787046, + "maximum": 284.151667360686, + "p50": 3473.407, + "p95": 3735.551, + "p99": 3745.342, + "sendP99": 3.071, + "bytes": 115279.74454148472, + "cpu": 1.2093194256299107, + "peakMiB": 102.78125, + "admitted": 369.06954438119766 + }, + "before/fanout": { + "trials": 3, + "inputs": 5526, + "deliveries": 22104, + "throughput": 99.27906674067115, + "minimum": 88.89995017928395, + "maximum": 100.62090828349011, + "p50": 8912.895, + "p95": 9830.399, + "p99": 10092.543, + "sendP99": 421.887, + "bytes": 377842.5997910136, + "cpu": 2.3403996865203758, + "peakMiB": 133.32421875, + "admitted": 186.96376081424137 + }, + "before/pubsub-one": { + "trials": 3, + "inputs": 9815, + "deliveries": 9815, + "throughput": 314.77864178323006, + "minimum": 256.1641157488365, + "maximum": 341.8981421544937, + "p50": 3276.799, + "p95": 4128.767, + "p99": 4194.303, + "sendP99": 376.831, + "bytes": 81818.04524236984, + "cpu": 1.059427289048474, + "peakMiB": 145.25, + "admitted": 337.4895007016332 + }, + "before/queue": { + "trials": 3, + "inputs": 23011, + "deliveries": 23011, + "throughput": 712.770329022115, + "minimum": 703.3627745451312, + "maximum": 732.2265118094954, + "p50": 1327.103, + "p95": 1703.935, + "p99": 1749.397, + "sendP99": 36.351, + "bytes": 109019.14814342222, + "cpu": 0.7151187689070103, + "peakMiB": 133.16015625, + "admitted": 759.9944214430262 + }, + "before/serial": { + "trials": 3, + "inputs": 13581, + "deliveries": 13581, + "throughput": 377.1952836166368, + "minimum": 376.784640192652, + "maximum": 402.112699276335, + "p50": 2260.991, + "p95": 2818.047, + "p99": 2829.273, + "sendP99": 3.135, + "bytes": 127042.12715712989, + "cpu": 0.890263907734057, + "peakMiB": 102.1484375, + "admitted": 442.11665216873314 + }, + "masstransit/fanout": { + "trials": 3, + "inputs": 13824, + "deliveries": 55296, + "throughput": 420.3975345735317, + "minimum": 416.1037810926104, + "maximum": 425.0524220688165, + "p50": 2195.455, + "p95": 2883.583, + "p99": 3014.655, + "sendP99": 49.151, + "bytes": 106786.82926829268, + "cpu": 1.6302394921190893, + "peakMiB": 162.4296875, + "admitted": 458.6451723485582 + }, + "masstransit/pubsub-one": { + "trials": 3, + "inputs": 38848, + "deliveries": 38848, + "throughput": 1270.9640185034496, + "minimum": 1218.198255329849, + "maximum": 1300.087330468451, + "p50": 811.007, + "p95": 1064.959, + "p99": 1130.495, + "sendP99": 22.527, + "bytes": 81061.21055862492, + "cpu": 0.7044520493115594, + "peakMiB": 145.8046875, + "admitted": 1302.5203709208608 + }, + "masstransit/queue": { + "trials": 3, + "inputs": 80286, + "deliveries": 80286, + "throughput": 2613.6807080699004, + "minimum": 2608.172576083676, + "maximum": 2616.8785021922713, + "p50": 380.927, + "p95": 421.887, + "p99": 696.319, + "sendP99": 13.439, + "bytes": 67433.12959496892, + "cpu": 0.43146222954256197, + "peakMiB": 146.55859375, + "admitted": 2675.0452170287463 + }, + "masstransit/serial": { + "trials": 3, + "inputs": 11191, + "deliveries": 11191, + "throughput": 292.7597577452258, + "minimum": 273.89373691778377, + "maximum": 298.0060684872128, + "p50": 1949.695, + "p95": 3047.423, + "p99": 3056.547, + "sendP99": 4.479, + "bytes": 165490.37293821637, + "cpu": 1.8646374056471904, + "peakMiB": 112.96484375, + "admitted": 377.25664943841304 + } + }, + "adaptive-check": { + "after/fanout": { + "trials": 1, + "inputs": 1996, + "deliveries": 7984, + "throughput": 324.5004505776547, + "minimum": 324.5004505776547, + "maximum": 324.5004505776547, + "p50": 2490.367, + "p95": 3702.783, + "p99": 3768.319, + "sendP99": 83.811, + "bytes": 192605.63126252504, + "cpu": 2.042448897795591, + "peakMiB": 139.359375, + "admitted": 397.60683711649466 + }, + "after/queue": { + "trials": 1, + "inputs": 10994, + "deliveries": 10994, + "throughput": 2088.3527057141387, + "minimum": 2088.3527057141387, + "maximum": 2088.3527057141387, + "p50": 458.751, + "p95": 835.583, + "p99": 843.775, + "sendP99": 11.903, + "bytes": 39907.661633618336, + "cpu": 0.5377970711297071, + "peakMiB": 128.59375, + "admitted": 2198.268238913007 + }, + "after/serial": { + "trials": 1, + "inputs": 2439, + "deliveries": 2439, + "throughput": 348.5074087831154, + "minimum": 348.5074087831154, + "maximum": 348.5074087831154, + "p50": 2064.383, + "p95": 2228.223, + "p99": 2241.054, + "sendP99": 3.551, + "bytes": 89479.05863058631, + "cpu": 2.20460106601066, + "peakMiB": 103.52734375, + "admitted": 487.5658026423588 + }, + "masstransit/fanout": { + "trials": 1, + "inputs": 2408, + "deliveries": 9632, + "throughput": 404.0698628067032, + "minimum": 404.0698628067032, + "maximum": 404.0698628067032, + "p50": 2080.767, + "p95": 3112.959, + "p99": 3211.263, + "sendP99": 352.255, + "bytes": 205907.57475083056, + "cpu": 2.1802437707641196, + "peakMiB": 162.0546875, + "admitted": 480.31702519400426 + }, + "masstransit/queue": { + "trials": 1, + "inputs": 12895, + "deliveries": 12895, + "throughput": 2456.5286817011684, + "minimum": 2456.5286817011684, + "maximum": 2456.5286817011684, + "p50": 385.023, + "p95": 712.703, + "p99": 720.895, + "sendP99": 14.207, + "bytes": 67620.80186118651, + "cpu": 0.817681039162466, + "peakMiB": 147.63671875, + "admitted": 2575.6956400633626 + }, + "masstransit/serial": { + "trials": 1, + "inputs": 1804, + "deliveries": 1804, + "throughput": 269.1722002890647, + "minimum": 269.1722002890647, + "maximum": 269.1722002890647, + "p50": 1081.343, + "p95": 1671.167, + "p99": 1700.976, + "sendP99": 4.479, + "bytes": 186254.59423503326, + "cpu": 3.155426829268293, + "peakMiB": 126.9296875, + "admitted": 360.5670376369828 + } + }, + "confirmed-standard": { + "after/fanout": { + "trials": 3, + "inputs": 10529, + "deliveries": 42116, + "throughput": 317.1334284150343, + "minimum": 307.497360752489, + "maximum": 324.4854233620776, + "p50": 2981.887, + "p95": 3833.855, + "p99": 4063.231, + "sendP99": 59.903, + "bytes": 190644.7007963595, + "cpu": 1.4162690058479532, + "peakMiB": 133.6875, + "admitted": 351.5764971111681 + }, + "after/pubsub-one": { + "trials": 3, + "inputs": 38870, + "deliveries": 38870, + "throughput": 1295.2390780632202, + "minimum": 1199.9888522436393, + "maximum": 1298.8350241448447, + "p50": 761.855, + "p95": 1081.343, + "p99": 1097.727, + "sendP99": 27.903, + "bytes": 40829.3309255079, + "cpu": 0.43791128057988526, + "peakMiB": 140.86328125, + "admitted": 1324.301114435785 + }, + "after/queue": { + "trials": 3, + "inputs": 71613, + "deliveries": 71613, + "throughput": 2327.6584184845187, + "minimum": 2296.442530076625, + "maximum": 2365.1091416767717, + "p50": 421.887, + "p95": 491.519, + "p99": 785.794, + "sendP99": 10.751, + "bytes": 59291.87267771021, + "cpu": 0.3205981983513215, + "peakMiB": 120.859375, + "admitted": 2384.206575696729 + }, + "after/serial": { + "trials": 3, + "inputs": 13770, + "deliveries": 13770, + "throughput": 385.2893193408955, + "minimum": 383.8968530789814, + "maximum": 397.20877006448904, + "p50": 2293.759, + "p95": 2686.975, + "p99": 2697.86, + "sendP99": 3.167, + "bytes": 138927.586146427, + "cpu": 1.1455537772087068, + "peakMiB": 101.92578125, + "admitted": 456.1087691239998 + }, + "before/fanout": { + "trials": 3, + "inputs": 5660, + "deliveries": 22640, + "throughput": 97.2777088889498, + "minimum": 97.0309624545128, + "maximum": 104.52920080571522, + "p50": 9043.967, + "p95": 10354.687, + "p99": 10542.037, + "sendP99": 409.599, + "bytes": 377167.693635383, + "cpu": 2.3485782092772385, + "peakMiB": 134.05859375, + "admitted": 188.04075009824706 + }, + "before/pubsub-one": { + "trials": 3, + "inputs": 10180, + "deliveries": 10180, + "throughput": 323.9002080062184, + "minimum": 294.48293068072786, + "maximum": 330.38044449025125, + "p50": 3014.655, + "p95": 4095.999, + "p99": 4194.303, + "sendP99": 385.023, + "bytes": 148867.80886185926, + "cpu": 1.1148612954186414, + "peakMiB": 146.7109375, + "admitted": 344.8533700548409 + }, + "before/queue": { + "trials": 3, + "inputs": 23489, + "deliveries": 23489, + "throughput": 736.4451677531464, + "minimum": 719.9503902187324, + "maximum": 744.3961821647176, + "p50": 1310.719, + "p95": 1654.783, + "p99": 1668.594, + "sendP99": 36.351, + "bytes": 109021.73607932875, + "cpu": 0.7560424540186446, + "peakMiB": 134.2578125, + "admitted": 786.1169232894695 + }, + "before/serial": { + "trials": 3, + "inputs": 13989, + "deliveries": 13989, + "throughput": 397.9420058263809, + "minimum": 382.99385839926236, + "maximum": 414.0212182534093, + "p50": 2326.527, + "p95": 2719.743, + "p99": 2750.253, + "sendP99": 2.975, + "bytes": 131190.7718770157, + "cpu": 0.8662875222024867, + "peakMiB": 101.265625, + "admitted": 465.0931398761868 + }, + "masstransit/fanout": { + "trials": 3, + "inputs": 14488, + "deliveries": 57952, + "throughput": 439.1367537958909, + "minimum": 421.01503528820547, + "maximum": 467.52948031580513, + "p50": 2195.455, + "p95": 2981.887, + "p99": 3145.727, + "sendP99": 40.959, + "bytes": 84765.39465408806, + "cpu": 1.7355969684385382, + "peakMiB": 160.609375, + "admitted": 481.58752206730327 + }, + "masstransit/pubsub-one": { + "trials": 3, + "inputs": 39882, + "deliveries": 39882, + "throughput": 1293.2796730017965, + "minimum": 1244.794947015874, + "maximum": 1353.2310927449512, + "p50": 770.047, + "p95": 983.039, + "p99": 1040.383, + "sendP99": 23.807, + "bytes": 80948.4256425065, + "cpu": 0.6499200784136319, + "peakMiB": 145.2421875, + "admitted": 1325.6555723131871 + }, + "masstransit/queue": { + "trials": 3, + "inputs": 78552, + "deliveries": 78552, + "throughput": 2584.339159184672, + "minimum": 2514.872619977275, + "maximum": 2587.2292015475646, + "p50": 380.927, + "p95": 442.367, + "p99": 729.087, + "sendP99": 14.207, + "bytes": 67453.64715023893, + "cpu": 0.4702256561754346, + "peakMiB": 143.21875, + "admitted": 2639.2309762612317 + }, + "masstransit/serial": { + "trials": 3, + "inputs": 11047, + "deliveries": 11047, + "throughput": 293.8643961604908, + "minimum": 247.40434834842358, + "maximum": 299.18815119718096, + "p50": 1835.007, + "p95": 3178.495, + "p99": 3203.502, + "sendP99": 4.223, + "bytes": 185334.3946965834, + "cpu": 1.8633116279069766, + "peakMiB": 111.734375, + "admitted": 386.9872139424514 + } + }, + "confirmed-rate10": { + "after/fanout": { + "trials": 1, + "inputs": 301, + "deliveries": 1204, + "throughput": 10.02461512605051, + "minimum": 10.02461512605051, + "maximum": 10.02461512605051, + "p50": 22.015, + "p95": 26.367, + "p99": 27.647, + "sendP99": 14.207, + "bytes": 377463.0166112957, + "cpu": 9.897362126245849, + "peakMiB": 114.9921875, + "admitted": 10.029833723750443 + }, + "after/queue": { + "trials": 1, + "inputs": 300, + "deliveries": 300, + "throughput": 10.00000573333662, + "minimum": 10.00000573333662, + "maximum": 10.00000573333662, + "p50": 7.743, + "p95": 9.471, + "p99": 9.983, + "sendP99": 5.631, + "bytes": 181390.10666666666, + "cpu": 5.7204, + "peakMiB": 108.09765625, + "admitted": 10.000006400004096 + }, + "masstransit/fanout": { + "trials": 1, + "inputs": 300, + "deliveries": 1200, + "throughput": 9.999882301385313, + "minimum": 9.999882301385313, + "maximum": 9.999882301385313, + "p50": 23.295, + "p95": 27.647, + "p99": 29.183, + "sendP99": 14.079, + "bytes": 573996.5866666667, + "cpu": 12.293086666666666, + "peakMiB": 133.19921875, + "admitted": 9.99988340135954 + }, + "masstransit/queue": { + "trials": 1, + "inputs": 301, + "deliveries": 301, + "throughput": 10.03016884849553, + "minimum": 10.03016884849553, + "maximum": 10.03016884849553, + "p50": 8.959, + "p95": 10.879, + "p99": 11.647, + "sendP99": 7.295, + "bytes": 193991.84053156147, + "cpu": 7.497691029900332, + "peakMiB": 121.63671875, + "admitted": 10.031989314284935 + } + }, + "confirmed-rate100": { + "after/fanout": { + "trials": 1, + "inputs": 3000, + "deliveries": 12000, + "throughput": 99.78341311699066, + "minimum": 99.78341311699066, + "maximum": 99.78341311699066, + "p50": 55.295, + "p95": 274.431, + "p99": 438.271, + "sendP99": 315.391, + "bytes": 396204.15466666664, + "cpu": 4.028724666666667, + "peakMiB": 118.98828125, + "admitted": 99.95454966672105 + }, + "after/queue": { + "trials": 1, + "inputs": 3000, + "deliveries": 3000, + "throughput": 99.99951533568233, + "minimum": 99.99951533568233, + "maximum": 99.99951533568233, + "p50": 5.055, + "p95": 6.975, + "p99": 7.999, + "sendP99": 3.839, + "bytes": 55241.656, + "cpu": 2.371309, + "peakMiB": 109.4921875, + "admitted": 99.99952166895469 + }, + "masstransit/fanout": { + "trials": 1, + "inputs": 3000, + "deliveries": 12000, + "throughput": 99.62427832504862, + "minimum": 99.62427832504862, + "maximum": 99.62427832504862, + "p50": 88.063, + "p95": 335.871, + "p99": 479.231, + "sendP99": 315.391, + "bytes": 289505.29333333333, + "cpu": 4.179472, + "peakMiB": 148.17578125, + "admitted": 99.92718272885487 + }, + "masstransit/queue": { + "trials": 1, + "inputs": 3000, + "deliveries": 3000, + "throughput": 99.99835369377035, + "minimum": 99.99835369377035, + "maximum": 99.99835369377035, + "p50": 6.335, + "p95": 7.999, + "p99": 8.959, + "sendP99": 4.991, + "bytes": 175012.488, + "cpu": 2.7409369999999997, + "peakMiB": 115.078125, + "admitted": 99.99836336011967 + } + }, + "confirmed-soak": { + "after/fanout": { + "trials": 1, + "inputs": 36791, + "deliveries": 147164, + "throughput": 303.78145254683153, + "minimum": 303.78145254683153, + "maximum": 303.78145254683153, + "p50": 3309.567, + "p95": 3932.159, + "p99": 4194.303, + "sendP99": 45.055, + "bytes": 187601.58625750864, + "cpu": 1.124182571824631, + "peakMiB": 128.90625, + "admitted": 306.56100417956117 + }, + "after/queue": { + "trials": 1, + "inputs": 293275, + "deliveries": 293275, + "throughput": 2438.8671149030133, + "minimum": 2438.8671149030133, + "maximum": 2438.8671149030133, + "p50": 405.503, + "p95": 466.943, + "p99": 737.279, + "sendP99": 9.087, + "bytes": 32788.29920040917, + "cpu": 0.26270785781263317, + "peakMiB": 125.28515625, + "admitted": 2443.853746575704 + }, + "masstransit/fanout": { + "trials": 1, + "inputs": 52464, + "deliveries": 209856, + "throughput": 433.9990339765861, + "minimum": 433.9990339765861, + "maximum": 433.9990339765861, + "p50": 2326.527, + "p95": 2719.743, + "p99": 2916.351, + "sendP99": 26.879, + "bytes": 193997.27767612078, + "cpu": 1.3269034385483378, + "peakMiB": 165.79296875, + "admitted": 437.1846016296896 + }, + "masstransit/queue": { + "trials": 1, + "inputs": 318875, + "deliveries": 318875, + "throughput": 2652.418848320992, + "minimum": 2652.418848320992, + "maximum": 2652.418848320992, + "p50": 372.735, + "p95": 475.135, + "p99": 712.703, + "sendP99": 13.183, + "bytes": 67426.23844453155, + "cpu": 0.36133407134457074, + "peakMiB": 151.59765625, + "admitted": 2657.2715865510445 + } + }, + "confirmed-requests": { + "after/fanout": { + "trials": 1, + "inputs": 3485, + "deliveries": 13940, + "throughput": 314.55478853930236, + "minimum": 314.55478853930236, + "maximum": 314.55478853930236, + "p50": 3211.263, + "p95": 3801.087, + "p99": 3899.391, + "sendP99": 380.927, + "bytes": 196403.6614060258, + "cpu": 2.192842754662841, + "peakMiB": 133.234375, + "admitted": 347.91478645429663 + }, + "after/queue": { + "trials": 1, + "inputs": 22532, + "deliveries": 22532, + "throughput": 2201.2087695876025, + "minimum": 2201.2087695876025, + "maximum": 2201.2087695876025, + "p50": 454.655, + "p95": 516.095, + "p99": 802.815, + "sendP99": 10.751, + "bytes": 59637.292739215336, + "cpu": 0.45896027871471684, + "peakMiB": 119.921875, + "admitted": 2252.350480969093 + }, + "before/fanout": { + "trials": 1, + "inputs": 1676, + "deliveries": 6704, + "throughput": 87.24890924807828, + "minimum": 87.24890924807828, + "maximum": 87.24890924807828, + "p50": 9437.183, + "p95": 11272.191, + "p99": 11403.263, + "sendP99": 434.175, + "bytes": 378277.169451074, + "cpu": 4.2552112171837715, + "peakMiB": 131.515625, + "admitted": 167.1994019369173 + }, + "before/queue": { + "trials": 1, + "inputs": 6771, + "deliveries": 6771, + "throughput": 627.6645733081353, + "minimum": 627.6645733081353, + "maximum": 627.6645733081353, + "p50": 1458.175, + "p95": 2097.151, + "p99": 2129.919, + "sendP99": 38.399, + "bytes": 109404.38930734013, + "cpu": 1.2812928666371288, + "peakMiB": 126.34765625, + "admitted": 670.2042682836297 + }, + "masstransit/fanout": { + "trials": 1, + "inputs": 4816, + "deliveries": 19264, + "throughput": 440.00694034202996, + "minimum": 440.00694034202996, + "maximum": 440.00694034202996, + "p50": 2228.223, + "p95": 2818.047, + "p99": 3047.423, + "sendP99": 35.839, + "bytes": 146111.10631229237, + "cpu": 2.509022425249169, + "peakMiB": 147.98828125, + "admitted": 481.4331930272799 + }, + "masstransit/queue": { + "trials": 1, + "inputs": 27109, + "deliveries": 27109, + "throughput": 2649.2230409990448, + "minimum": 2649.2230409990448, + "maximum": 2649.2230409990448, + "p50": 372.735, + "p95": 413.695, + "p99": 696.319, + "sendP99": 13.055, + "bytes": 45288.33523921945, + "cpu": 0.640758161496182, + "peakMiB": 145.078125, + "admitted": 2709.4494149722123 + } + } +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate10/after/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate10/after/summary.csv new file mode 100644 index 000000000..75ee115d5 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate10/after/summary.csv @@ -0,0 +1,3 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/sqs pubsub p32 c8 s4 1024B b1 r10 w1024 pf8","1","10","10","10","40.1","22.015","26.367","27.647","377463","9.8974","115","301","0","0" +"foundatio/sqs queue p32 c32 s1 1024B b1 r10 w1024 pf32","1","10","10","10","10","7.743","9.471","9.983","181390.1","5.7204","108.1","300","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate10/after/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate10/after/summary.md new file mode 100644 index 000000000..8cab2f611 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate10/after/summary.md @@ -0,0 +1,12 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/sqs pubsub p32 c8 s4 1024B b1 r10 w1024 pf8 | 1 | 10 (10-10) | 40.1 | 22.015 / 26.367 / 27.647 | 377463 | 9.8974 | +| foundatio/sqs queue p32 c32 s1 1024B b1 r10 w1024 pf32 | 1 | 10 (10-10) | 10 | 7.743 / 9.471 / 9.983 | 181390.1 | 5.7204 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate10/masstransit/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate10/masstransit/summary.csv new file mode 100644 index 000000000..db54b8fa8 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate10/masstransit/summary.csv @@ -0,0 +1,3 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"masstransit/sqs pubsub p32 c8 s4 1024B b1 r10 w1024 pf8","1","10","10","10","40","23.295","27.647","29.183","573996.6","12.2931","133.2","300","0","0" +"masstransit/sqs queue p32 c32 s1 1024B b1 r10 w1024 pf32","1","10","10","10","10","8.959","10.879","11.647","193991.8","7.4977","121.6","301","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate10/masstransit/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate10/masstransit/summary.md new file mode 100644 index 000000000..3b0cd3170 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate10/masstransit/summary.md @@ -0,0 +1,12 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| masstransit/sqs pubsub p32 c8 s4 1024B b1 r10 w1024 pf8 | 1 | 10 (10-10) | 40 | 23.295 / 27.647 / 29.183 | 573996.6 | 12.2931 | +| masstransit/sqs queue p32 c32 s1 1024B b1 r10 w1024 pf32 | 1 | 10 (10-10) | 10 | 8.959 / 10.879 / 11.647 | 193991.8 | 7.4977 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate100/after/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate100/after/summary.csv new file mode 100644 index 000000000..00dcfcdb1 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate100/after/summary.csv @@ -0,0 +1,3 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/sqs pubsub p32 c8 s4 1024B b1 r100 w1024 pf8","1","99.8","99.8","99.8","399.1","55.295","274.431","438.271","396204.2","4.0287","119","3000","0","0" +"foundatio/sqs queue p32 c32 s1 1024B b1 r100 w1024 pf32","1","100","100","100","100","5.055","6.975","7.999","55241.7","2.3713","109.5","3000","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate100/after/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate100/after/summary.md new file mode 100644 index 000000000..db22d8e42 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate100/after/summary.md @@ -0,0 +1,12 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/sqs pubsub p32 c8 s4 1024B b1 r100 w1024 pf8 | 1 | 99.8 (99.8-99.8) | 399.1 | 55.295 / 274.431 / 438.271 | 396204.2 | 4.0287 | +| foundatio/sqs queue p32 c32 s1 1024B b1 r100 w1024 pf32 | 1 | 100 (100-100) | 100 | 5.055 / 6.975 / 7.999 | 55241.7 | 2.3713 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate100/masstransit/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate100/masstransit/summary.csv new file mode 100644 index 000000000..298b8016b --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate100/masstransit/summary.csv @@ -0,0 +1,3 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"masstransit/sqs pubsub p32 c8 s4 1024B b1 r100 w1024 pf8","1","99.6","99.6","99.6","398.5","88.063","335.871","479.231","289505.3","4.1795","148.2","3000","0","0" +"masstransit/sqs queue p32 c32 s1 1024B b1 r100 w1024 pf32","1","100","100","100","100","6.335","7.999","8.959","175012.5","2.7409","115.1","3000","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate100/masstransit/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate100/masstransit/summary.md new file mode 100644 index 000000000..192c99b6b --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-rate100/masstransit/summary.md @@ -0,0 +1,12 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| masstransit/sqs pubsub p32 c8 s4 1024B b1 r100 w1024 pf8 | 1 | 99.6 (99.6-99.6) | 398.5 | 88.063 / 335.871 / 479.231 | 289505.3 | 4.1795 | +| masstransit/sqs queue p32 c32 s1 1024B b1 r100 w1024 pf32 | 1 | 100 (100-100) | 100 | 6.335 / 7.999 / 8.959 | 175012.5 | 2.7409 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/after/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/after/summary.csv new file mode 100644 index 000000000..2bc33ca2d --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/after/summary.csv @@ -0,0 +1,3 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","1","314.6","314.6","314.6","1258.2","3211.263","3801.087","3899.391","196403.7","2.1928","133.2","3485","0","0" +"foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","1","2201.2","2201.2","2201.2","2201.2","454.655","516.095","802.815","59637.3","0.459","119.9","22532","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/after/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/after/summary.md new file mode 100644 index 000000000..0277e6346 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/after/summary.md @@ -0,0 +1,12 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 1 | 314.6 (314.6-314.6) | 1258.2 | 3211.263 / 3801.087 / 3899.391 | 196403.7 | 2.1928 | +| foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 1 | 2201.2 (2201.2-2201.2) | 2201.2 | 454.655 / 516.095 / 802.815 | 59637.3 | 0.459 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/before/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/before/summary.csv new file mode 100644 index 000000000..20df45b65 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/before/summary.csv @@ -0,0 +1,3 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","1","87.2","87.2","87.2","349","9437.183","11272.191","11403.263","378277.2","4.2552","131.5","1676","0","0" +"foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","1","627.7","627.7","627.7","627.7","1458.175","2097.151","2129.919","109404.4","1.2813","126.3","6771","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/before/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/before/summary.md new file mode 100644 index 000000000..82443a559 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/before/summary.md @@ -0,0 +1,12 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 1 | 87.2 (87.2-87.2) | 349 | 9437.183 / 11272.191 / 11403.263 | 378277.2 | 4.2552 | +| foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 1 | 627.7 (627.7-627.7) | 627.7 | 1458.175 / 2097.151 / 2129.919 | 109404.4 | 1.2813 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/masstransit/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/masstransit/summary.csv new file mode 100644 index 000000000..4137c3123 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/masstransit/summary.csv @@ -0,0 +1,3 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"masstransit/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","1","440","440","440","1760","2228.223","2818.047","3047.423","146111.1","2.509","148","4816","0","0" +"masstransit/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","1","2649.2","2649.2","2649.2","2649.2","372.735","413.695","696.319","45288.3","0.6408","145.1","27109","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/masstransit/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/masstransit/summary.md new file mode 100644 index 000000000..ffe623ad2 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-requests/masstransit/summary.md @@ -0,0 +1,12 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| masstransit/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 1 | 440 (440-440) | 1760 | 2228.223 / 2818.047 / 3047.423 | 146111.1 | 2.509 | +| masstransit/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 1 | 2649.2 (2649.2-2649.2) | 2649.2 | 372.735 / 413.695 / 696.319 | 45288.3 | 0.6408 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-soak/after/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-soak/after/summary.csv new file mode 100644 index 000000000..7ddef0e0a --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-soak/after/summary.csv @@ -0,0 +1,3 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","1","303.8","303.8","303.8","1215.1","3309.567","3932.159","4194.303","187601.6","1.1242","128.9","36791","0","0" +"foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","1","2438.9","2438.9","2438.9","2438.9","405.503","466.943","737.279","32788.3","0.2627","125.3","293275","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-soak/after/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-soak/after/summary.md new file mode 100644 index 000000000..52f9fd89b --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-soak/after/summary.md @@ -0,0 +1,12 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 1 | 303.8 (303.8-303.8) | 1215.1 | 3309.567 / 3932.159 / 4194.303 | 187601.6 | 1.1242 | +| foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 1 | 2438.9 (2438.9-2438.9) | 2438.9 | 405.503 / 466.943 / 737.279 | 32788.3 | 0.2627 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-soak/masstransit/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-soak/masstransit/summary.csv new file mode 100644 index 000000000..758f34af7 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-soak/masstransit/summary.csv @@ -0,0 +1,3 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"masstransit/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","1","434","434","434","1736","2326.527","2719.743","2916.351","193997.3","1.3269","165.8","52464","0","0" +"masstransit/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","1","2652.4","2652.4","2652.4","2652.4","372.735","475.135","712.703","67426.2","0.3613","151.6","318875","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-soak/masstransit/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-soak/masstransit/summary.md new file mode 100644 index 000000000..844749eea --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-soak/masstransit/summary.md @@ -0,0 +1,12 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| masstransit/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 1 | 434 (434-434) | 1736 | 2326.527 / 2719.743 / 2916.351 | 193997.3 | 1.3269 | +| masstransit/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 1 | 2652.4 (2652.4-2652.4) | 2652.4 | 372.735 / 475.135 / 712.703 | 67426.2 | 0.3613 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/after/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/after/summary.csv new file mode 100644 index 000000000..c6f3da72c --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/after/summary.csv @@ -0,0 +1,5 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32","3","1295.2","1200","1298.8","1295.2","761.855","1081.343","1097.727","40829.3","0.4379","140.9","38870","0","0" +"foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","3","317.1","307.5","324.5","1268.5","2981.887","3833.855","4063.231","190644.7","1.4163","133.7","10529","0","0" +"foundatio/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1","3","385.3","383.9","397.2","385.3","2293.759","2686.975","2697.86","138927.6","1.1456","101.9","13770","0","0" +"foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","3","2327.7","2296.4","2365.1","2327.7","421.887","491.519","785.794","59291.9","0.3206","120.9","71613","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/after/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/after/summary.md new file mode 100644 index 000000000..9228b5b0f --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/after/summary.md @@ -0,0 +1,14 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 1295.2 (1200-1298.8) | 1295.2 | 761.855 / 1081.343 / 1097.727 | 40829.3 | 0.4379 | +| foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 3 | 317.1 (307.5-324.5) | 1268.5 | 2981.887 / 3833.855 / 4063.231 | 190644.7 | 1.4163 | +| foundatio/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 3 | 385.3 (383.9-397.2) | 385.3 | 2293.759 / 2686.975 / 2697.86 | 138927.6 | 1.1456 | +| foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 2327.7 (2296.4-2365.1) | 2327.7 | 421.887 / 491.519 / 785.794 | 59291.9 | 0.3206 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/before/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/before/summary.csv new file mode 100644 index 000000000..fb8640ce7 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/before/summary.csv @@ -0,0 +1,5 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32","3","323.9","294.5","330.4","323.9","3014.655","4095.999","4194.303","148867.8","1.1149","146.7","10180","0","0" +"foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","3","97.3","97","104.5","389.1","9043.967","10354.687","10542.037","377167.7","2.3486","134.1","5660","0","0" +"foundatio/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1","3","397.9","383","414","397.9","2326.527","2719.743","2750.253","131190.8","0.8663","101.3","13989","0","0" +"foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","3","736.4","720","744.4","736.4","1310.719","1654.783","1668.594","109021.7","0.756","134.3","23489","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/before/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/before/summary.md new file mode 100644 index 000000000..58eda089a --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/before/summary.md @@ -0,0 +1,14 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 323.9 (294.5-330.4) | 323.9 | 3014.655 / 4095.999 / 4194.303 | 148867.8 | 1.1149 | +| foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 3 | 97.3 (97-104.5) | 389.1 | 9043.967 / 10354.687 / 10542.037 | 377167.7 | 2.3486 | +| foundatio/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 3 | 397.9 (383-414) | 397.9 | 2326.527 / 2719.743 / 2750.253 | 131190.8 | 0.8663 | +| foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 736.4 (720-744.4) | 736.4 | 1310.719 / 1654.783 / 1668.594 | 109021.7 | 0.756 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/masstransit/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/masstransit/summary.csv new file mode 100644 index 000000000..d7c269871 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/masstransit/summary.csv @@ -0,0 +1,5 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"masstransit/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32","3","1293.3","1244.8","1353.2","1293.3","770.047","983.039","1040.383","80948.4","0.6499","145.2","39882","0","0" +"masstransit/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","3","439.1","421","467.5","1756.5","2195.455","2981.887","3145.727","84765.4","1.7356","160.6","14488","0","0" +"masstransit/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1","3","293.9","247.4","299.2","293.9","1835.007","3178.495","3203.502","185334.4","1.8633","111.7","11047","0","0" +"masstransit/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","3","2584.3","2514.9","2587.2","2584.3","380.927","442.367","729.087","67453.6","0.4702","143.2","78552","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/masstransit/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/masstransit/summary.md new file mode 100644 index 000000000..9d60f570f --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/confirmed-standard/masstransit/summary.md @@ -0,0 +1,14 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| masstransit/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 1293.3 (1244.8-1353.2) | 1293.3 | 770.047 / 983.039 / 1040.383 | 80948.4 | 0.6499 | +| masstransit/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 3 | 439.1 (421-467.5) | 1756.5 | 2195.455 / 2981.887 / 3145.727 | 84765.4 | 1.7356 | +| masstransit/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 3 | 293.9 (247.4-299.2) | 293.9 | 1835.007 / 3178.495 / 3203.502 | 185334.4 | 1.8633 | +| masstransit/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 2584.3 (2514.9-2587.2) | 2584.3 | 380.927 / 442.367 / 729.087 | 67453.6 | 0.4702 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/after/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/after/summary.csv new file mode 100644 index 000000000..a1398a2bc --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/after/summary.csv @@ -0,0 +1,5 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32","3","1215.7","1135.6","1225.6","1215.7","786.431","1146.879","1163.263","28470.1","0.4833","139.2","36705","0","0" +"foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","3","302.5","289.9","327.2","1210","3276.799","4030.463","4259.839","144762.1","1.4238","132.8","10226","0","0" +"foundatio/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1","3","277.2","277.1","284.2","277.2","3473.407","3735.551","3745.342","115279.7","1.2093","102.8","11098","0","0" +"foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","3","2327.8","2284","2382.3","2327.8","421.887","471.039","753.663","59274.6","0.3183","122.3","71686","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/after/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/after/summary.md new file mode 100644 index 000000000..54f3fbf44 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/after/summary.md @@ -0,0 +1,14 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 1215.7 (1135.6-1225.6) | 1215.7 | 786.431 / 1146.879 / 1163.263 | 28470.1 | 0.4833 | +| foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 3 | 302.5 (289.9-327.2) | 1210 | 3276.799 / 4030.463 / 4259.839 | 144762.1 | 1.4238 | +| foundatio/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 3 | 277.2 (277.1-284.2) | 277.2 | 3473.407 / 3735.551 / 3745.342 | 115279.7 | 1.2093 | +| foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 2327.8 (2284-2382.3) | 2327.8 | 421.887 / 471.039 / 753.663 | 59274.6 | 0.3183 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/before/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/before/summary.csv new file mode 100644 index 000000000..62f0aadc4 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/before/summary.csv @@ -0,0 +1,5 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32","3","314.8","256.2","341.9","314.8","3276.799","4128.767","4194.303","81818","1.0594","145.2","9815","0","0" +"foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","3","99.3","88.9","100.6","397.1","8912.895","9830.399","10092.543","377842.6","2.3404","133.3","5526","0","0" +"foundatio/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1","3","377.2","376.8","402.1","377.2","2260.991","2818.047","2829.273","127042.1","0.8903","102.1","13581","0","0" +"foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","3","712.8","703.4","732.2","712.8","1327.103","1703.935","1749.397","109019.1","0.7151","133.2","23011","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/before/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/before/summary.md new file mode 100644 index 000000000..7a22cfd9e --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/before/summary.md @@ -0,0 +1,14 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 314.8 (256.2-341.9) | 314.8 | 3276.799 / 4128.767 / 4194.303 | 81818 | 1.0594 | +| foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 3 | 99.3 (88.9-100.6) | 397.1 | 8912.895 / 9830.399 / 10092.543 | 377842.6 | 2.3404 | +| foundatio/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 3 | 377.2 (376.8-402.1) | 377.2 | 2260.991 / 2818.047 / 2829.273 | 127042.1 | 0.8903 | +| foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 712.8 (703.4-732.2) | 712.8 | 1327.103 / 1703.935 / 1749.397 | 109019.1 | 0.7151 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/masstransit/summary.csv b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/masstransit/summary.csv new file mode 100644 index 000000000..9bc268d78 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/masstransit/summary.csv @@ -0,0 +1,5 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"masstransit/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32","3","1271","1218.2","1300.1","1271","811.007","1064.959","1130.495","81061.2","0.7045","145.8","38848","0","0" +"masstransit/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","3","420.4","416.1","425.1","1681.6","2195.455","2883.583","3014.655","106786.8","1.6302","162.4","13824","0","0" +"masstransit/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1","3","292.8","273.9","298","292.8","1949.695","3047.423","3056.547","165490.4","1.8646","113","11191","0","0" +"masstransit/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","3","2613.7","2608.2","2616.9","2613.7","380.927","421.887","696.319","67433.1","0.4315","146.6","80286","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/masstransit/summary.md b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/masstransit/summary.md new file mode 100644 index 000000000..49511805b --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/final-standard/masstransit/summary.md @@ -0,0 +1,14 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance. + +AWS target: SQS/SNS custom endpoint; mode: localstack; region: us-east-1. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| masstransit/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 1271 (1218.2-1300.1) | 1271 | 811.007 / 1064.959 / 1130.495 | 81061.2 | 0.7045 | +| masstransit/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 3 | 420.4 (416.1-425.1) | 1681.6 | 2195.455 / 2883.583 / 3014.655 | 106786.8 | 1.6302 | +| masstransit/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 3 | 292.8 (273.9-298) | 292.8 | 1949.695 / 3047.423 / 3056.547 | 165490.4 | 1.8646 | +| masstransit/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 2613.7 (2608.2-2616.9) | 2613.7 | 380.927 / 421.887 / 696.319 | 67433.1 | 0.4315 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/host.json b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/host.json new file mode 100644 index 000000000..647c6a2ac --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/host.json @@ -0,0 +1,12 @@ +{ + "captured_utc": "2026-09-07T00:35:24.336133+00:00", + "loadavg": "1.48 1.53 1.46 2/4823 3571347", + "meminfo": "MemTotal: 62357452 kB\nMemFree: 4714944 kB\nMemAvailable: 20154752 kB\nBuffers: 1912572 kB\nCached: 16921432 kB\nSwapCached: 183020 kB\nActive: 16443016 kB\nInactive: 37984784 kB\nActive(anon): 15024796 kB\nInactive(anon): 24956368 kB\nActive(file): 1418220 kB\nInactive(file): 13028416 kB\nUnevictable: 800 kB\nMlocked: 800 kB\nSwapTotal: 8388604 kB\nSwapFree: 2560216 kB\nZswap: 0 kB\nZswapped: 0 kB\nDirty: 7252 kB\nWriteback: 0 kB\nAnonPages: 35494784 kB\nMapped: 1821452 kB\nShmem: 4387796 kB\nKReclaimable: 1701764 kB\nSlab: 2465076 kB\nSReclaimable: 1701764 kB\nSUnreclaim: 763312 kB\nKernelStack: 77552 kB\nPageTables: 230312 kB\nSecPageTables: 5144 kB\nNFS_Unstable: 0 kB\nBounce: 0 kB\nWritebackTmp: 0 kB\nCommitLimit: 39567328 kB\nCommitted_AS: 53280380 kB\nVmallocTotal: 34359738367 kB\nVmallocUsed: 163512 kB\nVmallocChunk: 0 kB\nPercpu: 43392 kB\nHardwareCorrupted: 0 kB\nAnonHugePages: 372736 kB\nShmemHugePages: 0 kB\nShmemPmdMapped: 0 kB\nFileHugePages: 45056 kB\nFilePmdMapped: 0 kB\nCmaTotal: 0 kB\nCmaFree: 0 kB\nUnaccepted: 0 kB\nBalloon: 0 kB\nHugePages_Total: 0\nHugePages_Free: 0\nHugePages_Rsvd: 0\nHugePages_Surp: 0\nHugepagesize: 2048 kB\nHugetlb: 0 kB\nDirectMap4k: 408824 kB\nDirectMap2M: 9709568 kB\nDirectMap1G: 54525952 kB\n", + "cpu": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 24\nOn-line CPU(s) list: 0-23\nVendor ID: AuthenticAMD\nModel name: AMD Ryzen AI 9 HX 470 w/ Radeon 890M\nCPU family: 26\nModel: 36\nThread(s) per core: 2\nCore(s) per socket: 12\nSocket(s): 1\nStepping: 0\nFrequency boost: enabled\nCPU(s) scaling MHz: 92%\nCPU max MHz: 5297.2979\nCPU min MHz: 621.6220\nBogoMIPS: 3992.54\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpuid_fault cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx_vnni avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid bus_lock_detect movdiri movdir64b overflow_recov succor smca fsrm avx512_vp2intersect flush_l1d amd_lbr_pmc_freeze\nVirtualization: AMD-V\nL1d cache: 576 KiB (12 instances)\nL1i cache: 384 KiB (12 instances)\nL2 cache: 12 MiB (12 instances)\nL3 cache: 24 MiB (2 instances)\nNUMA node(s): 1\nNUMA node0 CPU(s): 0-23\nVulnerability Gather data sampling: Not affected\nVulnerability Ghostwrite: Not affected\nVulnerability Indirect target selection: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Old microcode: Not affected\nVulnerability Reg file data sampling: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; IBPB on VMEXIT only\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsa: Not affected\nVulnerability Tsx async abort: Not affected\nVulnerability Vmscape: Mitigation; IBPB on VMEXIT\n", + "os": "PRETTY_NAME=\"Ubuntu 26.04.1 LTS\"\nNAME=\"Ubuntu\"\nVERSION_ID=\"26.04\"\nVERSION=\"26.04.1 LTS (Resolute Raccoon)\"\nVERSION_CODENAME=resolute\nID=ubuntu\nID_LIKE=debian\nHOME_URL=\"https://www.ubuntu.com/\"\nSUPPORT_URL=\"https://help.ubuntu.com/\"\nBUG_REPORT_URL=\"https://bugs.launchpad.net/ubuntu/\"\nPRIVACY_POLICY_URL=\"https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\"\nUBUNTU_CODENAME=resolute\nLOGO=ubuntu-logo\n", + "containers": { + "Image": "sha256:b279c01f4cfb8f985a482e4014cabc1e2697b9d7a6c8c8db2e40f4d9f93687c7", + "Name": "/foundatio-messaging-perf-localstack-1", + "Created": "2026-09-07T00:27:07.214220068Z" + } +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/raw-trials.tar.gz b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/raw-trials.tar.gz new file mode 100644 index 000000000..e98b54def Binary files /dev/null and b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/raw-trials.tar.gz differ diff --git a/benchmarks/Messaging/baselines/2026-09-06-aws-batching/scan.txt b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/scan.txt new file mode 100644 index 000000000..b882bd216 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06-aws-batching/scan.txt @@ -0,0 +1,27 @@ +AWS hot-path scan execution checklist (two partial transport files): +- IndexOf string without comparison: 0 +- Substring: 0 +- StartsWith or EndsWith string without comparison: 0 +- Contains string without comparison: 0 +- async void: 0 +- new HttpClient: 0 +- new JsonSerializerOptions: 0 +- static Dictionary: 0 +- static FrozenDictionary: 0 +- new List: 3 +- new Dictionary: 5 +- CurrentCulture comparer: 0 +- LINQ select/filter: 3 +- LINQ Any/All: 1 +- ToLower/ToUpper culture sensitive: 0 +- three Replace calls on one line: 0 +- params: 0 +- LINQ char predicate: 0 +- sync waits: 0 +- Regex constructions: 0 +- string.Format: 0 +- Task.Run: 0 +- class declarations: 2 +- sealed class declarations: 2 +- JSON calls: 2 +- Replace: 1 diff --git a/benchmarks/Messaging/baselines/2026-09-06/README.md b/benchmarks/Messaging/baselines/2026-09-06/README.md new file mode 100644 index 000000000..216a0ee8b --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/README.md @@ -0,0 +1,19 @@ +# Raw messaging benchmark evidence + +`raw-trials.tar.gz` contains all 141 JSON trial records, worker logs, per-profile metadata and summaries. CSV/Markdown summaries and the two failed trial records are also kept here for direct review. Native dumps remain local and are not included. + +- `standard` and `extended`: initial complete matrix before the in-memory timer fix. +- `memory-fixed` and `memory-extended-fixed`: repeated in-memory comparisons after commit `6a1a9887`. +- `soak`: two-minute trials after that fix; the failed Redis warmup and its labeled second attempt are both retained. +- `rate` and `loopback`: the same built executable as `soak`, invoked individually. Options and runtime/library versions are in each JSON; host/runtime metadata is shared with `soak`. + +Standard configurations have three trials. Extended, offered-rate and loopback cases have one each. Failed workers do not contribute successful measurements. The regular output directory is ignored by Git; this is an intentional baseline snapshot. + +Extract and regenerate a summary in PowerShell: + +```powershell +$results = 'benchmarks/Messaging/results/baseline-20260906' +New-Item -ItemType Directory -Force $results | Out-Null +tar -xzf benchmarks/Messaging/baselines/2026-09-06/raw-trials.tar.gz -C $results +./benchmarks/Messaging/summarize.ps1 -Directory (Join-Path $results 'standard') +``` diff --git a/benchmarks/Messaging/baselines/2026-09-06/brokers.txt b/benchmarks/Messaging/baselines/2026-09-06/brokers.txt new file mode 100644 index 000000000..a34aa7da2 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/brokers.txt @@ -0,0 +1,2 @@ +/foundatio-perf-redis redis:8.6-alpine sha256:2cc044fc5a07c9b701f8f1255a309ae9ad7856e694ac03513bf3648c01e40763 CPUs=4000000000 Memory=2147483648 +/foundatio-perf-aws localstack/localstack:3.8.1 sha256:b279c01f4cfb8f985a482e4014cabc1e2697b9d7a6c8c8db2e40f4d9f93687c7 CPUs=4000000000 Memory=3221225472 diff --git a/benchmarks/Messaging/baselines/2026-09-06/extended/summary.csv b/benchmarks/Messaging/baselines/2026-09-06/extended/summary.csv new file mode 100644 index 000000000..4b4abbb43 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/extended/summary.csv @@ -0,0 +1,21 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/memory pubsub p32 c8 s4 16384B b1 r0 w1024 pf8","1","58595.7","58595.7","58595.7","234382.7","7.167","19.455","34.815","231527.8","0.2561","1598.5","587215","0","0" +"foundatio/memory pubsub p8 c8 s4 1024B b10 r0 w1024 pf8","1","91830.1","91830.1","91830.1","367320.2","0.263","10.111","21.503","29596.7","0.2181","1966.9","919560","0","0" +"foundatio/memory queue p32 c32 s1 16384B b1 r0 w1024 pf32","1","90612.7","90612.7","90612.7","90612.7","11.135","14.847","24.831","122615.7","0.0911","744.2","907554","0","0" +"foundatio/memory queue p8 c32 s1 1024B b10 r0 w1024 pf32","1","218399.9","218399.9","218399.9","218399.9","3.935","6.079","17.919","12017.3","0.0399","1314.9","2186710","0","0" +"foundatio/redis pubsub p32 c8 s4 16384B b1 r0 w1024 pf8","1","3169.8","3169.8","3169.8","12679.3","274.431","339.967","348.159","203552.4","0.5459","177.5","32384","0","0" +"foundatio/redis pubsub p8 c8 s4 1024B b10 r0 w1024 pf8","1","7286.3","7286.3","7286.3","29145.3","123.903","145.407","167.935","61176.8","0.3234","114.7","73600","0","0" +"foundatio/redis queue p32 c32 s1 16384B b1 r0 w1024 pf32","1","7728.7","7728.7","7728.7","7728.7","124.927","169.983","178.175","118620.4","0.205","230","77996","0","0" +"foundatio/redis queue p8 c32 s1 1024B b10 r0 w1024 pf32","1","20220.8","20220.8","20220.8","20220.8","47.615","54.783","91.135","21248.3","0.1152","137.2","203150","0","0" +"foundatio/sqs pubsub p32 c8 s4 16384B b1 r0 w1024 pf8","1","91","91","91","364.2","9175.039","10354.687","10747.903","953889.3","2.8806","162.4","1848","0","0" +"foundatio/sqs pubsub p8 c8 s4 1024B b10 r0 w1024 pf8","1","108.8","108.8","108.8","435.2","7077.887","9437.183","9830.399","327364.5","1.95","143.2","1820","0","0" +"foundatio/sqs queue p32 c32 s1 16384B b1 r0 w1024 pf32","1","645.9","645.9","645.9","645.9","1409.023","1851.391","1867.775","347501.6","0.9147","149.9","7011","0","0" +"foundatio/sqs queue p8 c32 s1 1024B b10 r0 w1024 pf32","1","1117.4","1117.4","1117.4","1117.4","827.391","1261.567","1277.951","58017.9","0.4245","116","11950","0","0" +"masstransit/memory pubsub p32 c8 s4 16384B b1 r0 w1024 pf8","1","37892.9","37892.9","37892.9","151571.7","15.743","33.279","40.959","327066.5","0.2917","337.1","380328","0","0" +"masstransit/memory pubsub p8 c8 s4 1024B b10 r0 w1024 pf8","1","77734.8","77734.8","77734.8","310939.2","6.527","14.975","19.455","65825.4","0.1143","178.3","778860","0","0" +"masstransit/memory queue p32 c32 s1 16384B b1 r0 w1024 pf32","1","48503.7","48503.7","48503.7","48503.7","19.711","28.927","33.791","99460.4","0.1667","245.9","486120","0","0" +"masstransit/memory queue p8 c32 s1 1024B b10 r0 w1024 pf32","1","99020.7","99020.7","99020.7","99020.7","9.983","11.647","14.335","22539","0.0362","125.1","991320","0","0" +"masstransit/sqs pubsub p32 c8 s4 16384B b1 r0 w1024 pf8","1","303.1","303.1","303.1","1212.6","3112.959","4587.519","4718.591","760327.9","2.6512","250.5","3417","0","0" +"masstransit/sqs pubsub p8 c8 s4 1024B b10 r0 w1024 pf8","1","499.6","499.6","499.6","1998.3","1818.623","2523.135","2654.207","77792.4","1.474","162.5","5410","0","0" +"masstransit/sqs queue p32 c32 s1 16384B b1 r0 w1024 pf32","1","1261","1261","1261","1261","778.239","843.775","860.159","240453.6","1.031","216.1","13216","0","0" +"masstransit/sqs queue p8 c32 s1 1024B b10 r0 w1024 pf32","1","2757.5","2757.5","2757.5","2757.5","348.159","421.887","737.279","66182.5","0.4325","156.3","28190","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06/extended/summary.md b/benchmarks/Messaging/baselines/2026-09-06/extended/summary.md new file mode 100644 index 000000000..7b5c908c3 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/extended/summary.md @@ -0,0 +1,28 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. These are local measurements, not cloud sizing claims. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/memory pubsub p32 c8 s4 16384B b1 r0 w1024 pf8 | 1 | 58595.7 (58595.7-58595.7) | 234382.7 | 7.167 / 19.455 / 34.815 | 231527.8 | 0.2561 | +| foundatio/memory pubsub p8 c8 s4 1024B b10 r0 w1024 pf8 | 1 | 91830.1 (91830.1-91830.1) | 367320.2 | 0.263 / 10.111 / 21.503 | 29596.7 | 0.2181 | +| foundatio/memory queue p32 c32 s1 16384B b1 r0 w1024 pf32 | 1 | 90612.7 (90612.7-90612.7) | 90612.7 | 11.135 / 14.847 / 24.831 | 122615.7 | 0.0911 | +| foundatio/memory queue p8 c32 s1 1024B b10 r0 w1024 pf32 | 1 | 218399.9 (218399.9-218399.9) | 218399.9 | 3.935 / 6.079 / 17.919 | 12017.3 | 0.0399 | +| foundatio/redis pubsub p32 c8 s4 16384B b1 r0 w1024 pf8 | 1 | 3169.8 (3169.8-3169.8) | 12679.3 | 274.431 / 339.967 / 348.159 | 203552.4 | 0.5459 | +| foundatio/redis pubsub p8 c8 s4 1024B b10 r0 w1024 pf8 | 1 | 7286.3 (7286.3-7286.3) | 29145.3 | 123.903 / 145.407 / 167.935 | 61176.8 | 0.3234 | +| foundatio/redis queue p32 c32 s1 16384B b1 r0 w1024 pf32 | 1 | 7728.7 (7728.7-7728.7) | 7728.7 | 124.927 / 169.983 / 178.175 | 118620.4 | 0.205 | +| foundatio/redis queue p8 c32 s1 1024B b10 r0 w1024 pf32 | 1 | 20220.8 (20220.8-20220.8) | 20220.8 | 47.615 / 54.783 / 91.135 | 21248.3 | 0.1152 | +| foundatio/sqs pubsub p32 c8 s4 16384B b1 r0 w1024 pf8 | 1 | 91 (91-91) | 364.2 | 9175.039 / 10354.687 / 10747.903 | 953889.3 | 2.8806 | +| foundatio/sqs pubsub p8 c8 s4 1024B b10 r0 w1024 pf8 | 1 | 108.8 (108.8-108.8) | 435.2 | 7077.887 / 9437.183 / 9830.399 | 327364.5 | 1.95 | +| foundatio/sqs queue p32 c32 s1 16384B b1 r0 w1024 pf32 | 1 | 645.9 (645.9-645.9) | 645.9 | 1409.023 / 1851.391 / 1867.775 | 347501.6 | 0.9147 | +| foundatio/sqs queue p8 c32 s1 1024B b10 r0 w1024 pf32 | 1 | 1117.4 (1117.4-1117.4) | 1117.4 | 827.391 / 1261.567 / 1277.951 | 58017.9 | 0.4245 | +| masstransit/memory pubsub p32 c8 s4 16384B b1 r0 w1024 pf8 | 1 | 37892.9 (37892.9-37892.9) | 151571.7 | 15.743 / 33.279 / 40.959 | 327066.5 | 0.2917 | +| masstransit/memory pubsub p8 c8 s4 1024B b10 r0 w1024 pf8 | 1 | 77734.8 (77734.8-77734.8) | 310939.2 | 6.527 / 14.975 / 19.455 | 65825.4 | 0.1143 | +| masstransit/memory queue p32 c32 s1 16384B b1 r0 w1024 pf32 | 1 | 48503.7 (48503.7-48503.7) | 48503.7 | 19.711 / 28.927 / 33.791 | 99460.4 | 0.1667 | +| masstransit/memory queue p8 c32 s1 1024B b10 r0 w1024 pf32 | 1 | 99020.7 (99020.7-99020.7) | 99020.7 | 9.983 / 11.647 / 14.335 | 22539 | 0.0362 | +| masstransit/sqs pubsub p32 c8 s4 16384B b1 r0 w1024 pf8 | 1 | 303.1 (303.1-303.1) | 1212.6 | 3112.959 / 4587.519 / 4718.591 | 760327.9 | 2.6512 | +| masstransit/sqs pubsub p8 c8 s4 1024B b10 r0 w1024 pf8 | 1 | 499.6 (499.6-499.6) | 1998.3 | 1818.623 / 2523.135 / 2654.207 | 77792.4 | 1.474 | +| masstransit/sqs queue p32 c32 s1 16384B b1 r0 w1024 pf32 | 1 | 1261 (1261-1261) | 1261 | 778.239 / 843.775 / 860.159 | 240453.6 | 1.031 | +| masstransit/sqs queue p8 c32 s1 1024B b10 r0 w1024 pf32 | 1 | 2757.5 (2757.5-2757.5) | 2757.5 | 348.159 / 421.887 / 737.279 | 66182.5 | 0.4325 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06/loopback/summary.csv b/benchmarks/Messaging/baselines/2026-09-06/loopback/summary.csv new file mode 100644 index 000000000..aa28a7237 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/loopback/summary.csv @@ -0,0 +1,3 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"loopback/memory pubsub p32 c32 s4 1024B b1 r0 w1024 pf32","1","1242225.8","1242225.8","1242225.8","4968903.4","0","0.003","0.004","80","0.0064","89.6","7172217","0","0" +"loopback/memory queue p32 c32 s1 1024B b1 r0 w1024 pf32","1","1466421","1466421","1466421","1466421","0","0","0","80","0.0041","94.9","8672565","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06/loopback/summary.md b/benchmarks/Messaging/baselines/2026-09-06/loopback/summary.md new file mode 100644 index 000000000..8c6b852f2 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/loopback/summary.md @@ -0,0 +1,10 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. These are local measurements, not cloud sizing claims. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| loopback/memory pubsub p32 c32 s4 1024B b1 r0 w1024 pf32 | 1 | 1242225.8 (1242225.8-1242225.8) | 4968903.4 | 0 / 0.003 / 0.004 | 80 | 0.0064 | +| loopback/memory queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 1 | 1466421 (1466421-1466421) | 1466421 | 0 / 0 / 0 | 80 | 0.0041 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06/memory-extended-fixed/summary.csv b/benchmarks/Messaging/baselines/2026-09-06/memory-extended-fixed/summary.csv new file mode 100644 index 000000000..20475b607 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/memory-extended-fixed/summary.csv @@ -0,0 +1,9 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/memory pubsub p32 c8 s4 16384B b1 r0 w1024 pf8","1","59583.1","59583.1","59583.1","238332.3","4.351","20.479","27.135","230057.7","0.2621","259.7","596600","0","0" +"foundatio/memory pubsub p8 c8 s4 1024B b10 r0 w1024 pf8","1","98545.2","98545.2","98545.2","394181","0.245","10.495","11.519","28552.5","0.2099","119","985910","0","0" +"foundatio/memory queue p32 c32 s1 16384B b1 r0 w1024 pf32","1","79280.6","79280.6","79280.6","79280.6","11.903","18.943","24.575","122308.7","0.1403","239.6","793668","0","0" +"foundatio/memory queue p8 c32 s1 1024B b10 r0 w1024 pf32","1","282546","282546","282546","282546","3.487","4.287","4.927","11707.5","0.0376","119.6","2826750","0","0" +"masstransit/memory pubsub p32 c8 s4 16384B b1 r0 w1024 pf8","1","39049","39049","39049","156195.9","13.951","30.975","39.423","327064.6","0.2658","345.8","391679","0","0" +"masstransit/memory pubsub p8 c8 s4 1024B b10 r0 w1024 pf8","1","76375.4","76375.4","76375.4","305501.7","6.335","15.231","19.711","65826.1","0.116","174.5","764540","0","0" +"masstransit/memory queue p32 c32 s1 16384B b1 r0 w1024 pf32","1","48865.7","48865.7","48865.7","48865.7","19.711","27.647","35.839","99457.2","0.1529","238.1","489492","0","0" +"masstransit/memory queue p8 c32 s1 1024B b10 r0 w1024 pf32","1","98847.4","98847.4","98847.4","98847.4","9.983","11.263","14.207","22539.3","0.0365","122.7","989530","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06/memory-extended-fixed/summary.md b/benchmarks/Messaging/baselines/2026-09-06/memory-extended-fixed/summary.md new file mode 100644 index 000000000..54cd895a6 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/memory-extended-fixed/summary.md @@ -0,0 +1,16 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. These are local measurements, not cloud sizing claims. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/memory pubsub p32 c8 s4 16384B b1 r0 w1024 pf8 | 1 | 59583.1 (59583.1-59583.1) | 238332.3 | 4.351 / 20.479 / 27.135 | 230057.7 | 0.2621 | +| foundatio/memory pubsub p8 c8 s4 1024B b10 r0 w1024 pf8 | 1 | 98545.2 (98545.2-98545.2) | 394181 | 0.245 / 10.495 / 11.519 | 28552.5 | 0.2099 | +| foundatio/memory queue p32 c32 s1 16384B b1 r0 w1024 pf32 | 1 | 79280.6 (79280.6-79280.6) | 79280.6 | 11.903 / 18.943 / 24.575 | 122308.7 | 0.1403 | +| foundatio/memory queue p8 c32 s1 1024B b10 r0 w1024 pf32 | 1 | 282546 (282546-282546) | 282546 | 3.487 / 4.287 / 4.927 | 11707.5 | 0.0376 | +| masstransit/memory pubsub p32 c8 s4 16384B b1 r0 w1024 pf8 | 1 | 39049 (39049-39049) | 156195.9 | 13.951 / 30.975 / 39.423 | 327064.6 | 0.2658 | +| masstransit/memory pubsub p8 c8 s4 1024B b10 r0 w1024 pf8 | 1 | 76375.4 (76375.4-76375.4) | 305501.7 | 6.335 / 15.231 / 19.711 | 65826.1 | 0.116 | +| masstransit/memory queue p32 c32 s1 16384B b1 r0 w1024 pf32 | 1 | 48865.7 (48865.7-48865.7) | 48865.7 | 19.711 / 27.647 / 35.839 | 99457.2 | 0.1529 | +| masstransit/memory queue p8 c32 s1 1024B b10 r0 w1024 pf32 | 1 | 98847.4 (98847.4-98847.4) | 98847.4 | 9.983 / 11.263 / 14.207 | 22539.3 | 0.0365 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06/memory-fixed/summary.csv b/benchmarks/Messaging/baselines/2026-09-06/memory-fixed/summary.csv new file mode 100644 index 000000000..600ef2c58 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/memory-fixed/summary.csv @@ -0,0 +1,9 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/memory pubsub p32 c32 s1 1024B b1 r0 w1024 pf32","3","246508.4","245755.4","247629.1","246508.4","4.159","4.991","5.759","12868.3","0.038","122.4","7403425","0","0" +"foundatio/memory pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","3","95854.8","95518.4","96101.8","383419","0.082","10.879","12.031","29604.8","0.2213","123.4","2876391","0","0" +"foundatio/memory queue p1 c1 s1 1024B b1 r0 w1024 pf1","3","112276.9","108152.1","124646.2","112276.9","9.343","11.135","11.775","13111.9","0.0434","105.8","3453852","0","0" +"foundatio/memory queue p32 c32 s1 1024B b1 r0 w1024 pf32","3","258086.7","252717","264568.7","258086.7","3.967","4.799","5.439","12474.2","0.0372","121.9","7758341","0","0" +"masstransit/memory pubsub p32 c32 s1 1024B b1 r0 w1024 pf32","3","102242.6","99263.1","102515.9","102242.6","9.983","11.135","15.359","23135.7","0.0359","127.9","3044413","0","0" +"masstransit/memory pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","3","73940.2","72783.2","76190.7","295760.8","5.567","16.383","21.503","65939.4","0.1129","174.4","2232407","0","0" +"masstransit/memory queue p1 c1 s1 1024B b1 r0 w1024 pf1","3","122978.3","122442.1","123201.7","122978.3","8.447","10.239","12.543","19675.5","0.0293","121.6","3689928","0","0" +"masstransit/memory queue p32 c32 s1 1024B b1 r0 w1024 pf32","3","100661","100145.9","101434.6","100661","10.239","11.263","14.463","22655","0.036","129.8","3026349","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06/memory-fixed/summary.md b/benchmarks/Messaging/baselines/2026-09-06/memory-fixed/summary.md new file mode 100644 index 000000000..919df6026 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/memory-fixed/summary.md @@ -0,0 +1,16 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. These are local measurements, not cloud sizing claims. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/memory pubsub p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 246508.4 (245755.4-247629.1) | 246508.4 | 4.159 / 4.991 / 5.759 | 12868.3 | 0.038 | +| foundatio/memory pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 3 | 95854.8 (95518.4-96101.8) | 383419 | 0.082 / 10.879 / 12.031 | 29604.8 | 0.2213 | +| foundatio/memory queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 3 | 112276.9 (108152.1-124646.2) | 112276.9 | 9.343 / 11.135 / 11.775 | 13111.9 | 0.0434 | +| foundatio/memory queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 258086.7 (252717-264568.7) | 258086.7 | 3.967 / 4.799 / 5.439 | 12474.2 | 0.0372 | +| masstransit/memory pubsub p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 102242.6 (99263.1-102515.9) | 102242.6 | 9.983 / 11.135 / 15.359 | 23135.7 | 0.0359 | +| masstransit/memory pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 3 | 73940.2 (72783.2-76190.7) | 295760.8 | 5.567 / 16.383 / 21.503 | 65939.4 | 0.1129 | +| masstransit/memory queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 3 | 122978.3 (122442.1-123201.7) | 122978.3 | 8.447 / 10.239 / 12.543 | 19675.5 | 0.0293 | +| masstransit/memory queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 100661 (100145.9-101434.6) | 100661 | 10.239 / 11.263 / 14.463 | 22655 | 0.036 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06/rate/summary.csv b/benchmarks/Messaging/baselines/2026-09-06/rate/summary.csv new file mode 100644 index 000000000..f54871818 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/rate/summary.csv @@ -0,0 +1,17 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/memory pubsub p32 c8 s4 1024B b1 r50000 w1024 pf8","1","49998.3","49998.3","49998.3","199993","0.903","2.303","2.847","27472.1","0.2324","106.1","1500013","0","0" +"foundatio/memory queue p32 c32 s1 1024B b1 r50000 w1024 pf32","1","49998.9","49998.9","49998.9","49998.9","0.687","1.327","1.631","8816.1","0.0487","90.8","1500004","0","0" +"foundatio/redis pubsub p32 c8 s4 1024B b1 r10 w1024 pf8","1","10","10","10","40","27.391","102.399","103.423","91185.1","6.3834","87.8","300","0","0" +"foundatio/redis pubsub p32 c8 s4 1024B b1 r3000 w1024 pf8","1","2997.9","2997.9","2997.9","11991.6","14.207","25.599","27.391","37761.6","0.4277","164.1","89997","0","0" +"foundatio/redis queue p32 c32 s1 1024B b1 r10 w1024 pf32","1","10","10","10","10","26.623","101.375","103.423","36318.5","4.9785","81.7","300","0","0" +"foundatio/redis queue p32 c32 s1 1024B b1 r3000 w1024 pf32","1","2997.9","2997.9","2997.9","2997.9","14.207","25.855","27.135","17162.2","0.1953","119.5","90001","0","0" +"foundatio/sqs pubsub p32 c8 s4 1024B b1 r10 w1024 pf8","1","10","10","10","40","23.039","27.135","28.415","339866.6","8.3583","115","300","0","0" +"foundatio/sqs pubsub p32 c8 s4 1024B b1 r100 w1024 pf8","1","88.2","88.2","88.2","353","2949.119","4161.535","4325.375","382118.1","3.0003","128.5","3001","0","0" +"foundatio/sqs queue p32 c32 s1 1024B b1 r10 w1024 pf32","1","10","10","10","10","7.423","9.599","10.623","173318.9","4.5851","107.5","301","0","0" +"foundatio/sqs queue p32 c32 s1 1024B b1 r100 w1024 pf32","1","100","100","100","100","4.863","7.359","8.959","105989","1.9435","107.6","3000","0","0" +"masstransit/memory pubsub p32 c8 s4 1024B b1 r50000 w1024 pf8","1","49990.1","49990.1","49990.1","199960.4","0.911","2.079","6.463","60968.3","0.0786","163.9","1499990","0","0" +"masstransit/memory queue p32 c32 s1 1024B b1 r50000 w1024 pf32","1","49988.9","49988.9","49988.9","49988.9","0.887","1.887","2.239","16154.9","0.0274","107.3","1499996","0","0" +"masstransit/sqs pubsub p32 c8 s4 1024B b1 r10 w1024 pf8","1","10","10","10","40","23.295","27.135","30.975","236999.9","11.4343","131.4","300","0","0" +"masstransit/sqs pubsub p32 c8 s4 1024B b1 r100 w1024 pf8","1","99.9","99.9","99.9","399.7","59.391","303.103","430.079","259311.4","4.7302","145.5","3000","0","0" +"masstransit/sqs queue p32 c32 s1 1024B b1 r10 w1024 pf32","1","10","10","10","10","9.727","11.135","11.775","193389.8","7.6765","119.5","300","0","0" +"masstransit/sqs queue p32 c32 s1 1024B b1 r100 w1024 pf32","1","100","100","100","100","6.399","8.575","9.727","174643.3","2.4587","116.4","3001","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06/rate/summary.md b/benchmarks/Messaging/baselines/2026-09-06/rate/summary.md new file mode 100644 index 000000000..f377c3acb --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/rate/summary.md @@ -0,0 +1,24 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. These are local measurements, not cloud sizing claims. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/memory pubsub p32 c8 s4 1024B b1 r50000 w1024 pf8 | 1 | 49998.3 (49998.3-49998.3) | 199993 | 0.903 / 2.303 / 2.847 | 27472.1 | 0.2324 | +| foundatio/memory queue p32 c32 s1 1024B b1 r50000 w1024 pf32 | 1 | 49998.9 (49998.9-49998.9) | 49998.9 | 0.687 / 1.327 / 1.631 | 8816.1 | 0.0487 | +| foundatio/redis pubsub p32 c8 s4 1024B b1 r10 w1024 pf8 | 1 | 10 (10-10) | 40 | 27.391 / 102.399 / 103.423 | 91185.1 | 6.3834 | +| foundatio/redis pubsub p32 c8 s4 1024B b1 r3000 w1024 pf8 | 1 | 2997.9 (2997.9-2997.9) | 11991.6 | 14.207 / 25.599 / 27.391 | 37761.6 | 0.4277 | +| foundatio/redis queue p32 c32 s1 1024B b1 r10 w1024 pf32 | 1 | 10 (10-10) | 10 | 26.623 / 101.375 / 103.423 | 36318.5 | 4.9785 | +| foundatio/redis queue p32 c32 s1 1024B b1 r3000 w1024 pf32 | 1 | 2997.9 (2997.9-2997.9) | 2997.9 | 14.207 / 25.855 / 27.135 | 17162.2 | 0.1953 | +| foundatio/sqs pubsub p32 c8 s4 1024B b1 r10 w1024 pf8 | 1 | 10 (10-10) | 40 | 23.039 / 27.135 / 28.415 | 339866.6 | 8.3583 | +| foundatio/sqs pubsub p32 c8 s4 1024B b1 r100 w1024 pf8 | 1 | 88.2 (88.2-88.2) | 353 | 2949.119 / 4161.535 / 4325.375 | 382118.1 | 3.0003 | +| foundatio/sqs queue p32 c32 s1 1024B b1 r10 w1024 pf32 | 1 | 10 (10-10) | 10 | 7.423 / 9.599 / 10.623 | 173318.9 | 4.5851 | +| foundatio/sqs queue p32 c32 s1 1024B b1 r100 w1024 pf32 | 1 | 100 (100-100) | 100 | 4.863 / 7.359 / 8.959 | 105989 | 1.9435 | +| masstransit/memory pubsub p32 c8 s4 1024B b1 r50000 w1024 pf8 | 1 | 49990.1 (49990.1-49990.1) | 199960.4 | 0.911 / 2.079 / 6.463 | 60968.3 | 0.0786 | +| masstransit/memory queue p32 c32 s1 1024B b1 r50000 w1024 pf32 | 1 | 49988.9 (49988.9-49988.9) | 49988.9 | 0.887 / 1.887 / 2.239 | 16154.9 | 0.0274 | +| masstransit/sqs pubsub p32 c8 s4 1024B b1 r10 w1024 pf8 | 1 | 10 (10-10) | 40 | 23.295 / 27.135 / 30.975 | 236999.9 | 11.4343 | +| masstransit/sqs pubsub p32 c8 s4 1024B b1 r100 w1024 pf8 | 1 | 99.9 (99.9-99.9) | 399.7 | 59.391 / 303.103 / 430.079 | 259311.4 | 4.7302 | +| masstransit/sqs queue p32 c32 s1 1024B b1 r10 w1024 pf32 | 1 | 10 (10-10) | 10 | 9.727 / 11.135 / 11.775 | 193389.8 | 7.6765 | +| masstransit/sqs queue p32 c32 s1 1024B b1 r100 w1024 pf32 | 1 | 100 (100-100) | 100 | 6.399 / 8.575 / 9.727 | 174643.3 | 2.4587 | + +Failed trials: 0. diff --git a/benchmarks/Messaging/baselines/2026-09-06/raw-trials.tar.gz b/benchmarks/Messaging/baselines/2026-09-06/raw-trials.tar.gz new file mode 100644 index 000000000..40d73d241 Binary files /dev/null and b/benchmarks/Messaging/baselines/2026-09-06/raw-trials.tar.gz differ diff --git a/benchmarks/Messaging/baselines/2026-09-06/run-status.txt b/benchmarks/Messaging/baselines/2026-09-06/run-status.txt new file mode 100644 index 000000000..ffb16f37a --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/run-status.txt @@ -0,0 +1,22 @@ +2026-09-06T16:18:16.1852714-05:00 standard exit=1 +2026-09-06T16:39:56.5768090-05:00 extended exit=0 +2026-09-06T16:59:12.1418827-05:00 soak exit=1 +2026-09-06T16:59:45.3282487-05:00 round1-foundatio-memory-queue-rate50000 exit=0 +2026-09-06T17:00:18.6773192-05:00 round1-foundatio-memory-pubsub-rate50000 exit=0 +2026-09-06T17:00:51.8870573-05:00 round1-masstransit-memory-queue-rate50000 exit=0 +2026-09-06T17:01:25.0986136-05:00 round1-masstransit-memory-pubsub-rate50000 exit=0 +2026-09-06T17:01:58.4322114-05:00 round1-foundatio-redis-queue-rate3000 exit=0 +2026-09-06T17:02:31.8894808-05:00 round1-foundatio-redis-pubsub-rate3000 exit=0 +2026-09-06T17:03:05.2205930-05:00 round1-foundatio-sqs-queue-rate100 exit=0 +2026-09-06T17:03:43.4008685-05:00 round1-foundatio-sqs-pubsub-rate100 exit=0 +2026-09-06T17:04:17.7609607-05:00 round1-masstransit-sqs-queue-rate100 exit=0 +2026-09-06T17:04:52.4724461-05:00 round1-masstransit-sqs-pubsub-rate100 exit=0 +2026-09-06T17:04:59.3361676-05:00 loopback-queue exit=0 +2026-09-06T17:05:05.9453286-05:00 loopback-pubsub exit=0 +2026-09-06T17:07:12.9979880-05:00 round2-foundatio-redis-pubsub-four exit=0 +2026-09-06T17:07:46.3033080-05:00 round1-foundatio-redis-queue-rate10 exit=0 +2026-09-06T17:08:19.7342634-05:00 round1-foundatio-redis-pubsub-rate10 exit=0 +2026-09-06T17:08:53.0678817-05:00 round1-foundatio-sqs-queue-rate10 exit=0 +2026-09-06T17:09:26.6423171-05:00 round1-foundatio-sqs-pubsub-rate10 exit=0 +2026-09-06T17:10:01.0210439-05:00 round1-masstransit-sqs-queue-rate10 exit=0 +2026-09-06T17:10:35.4796509-05:00 round1-masstransit-sqs-pubsub-rate10 exit=0 diff --git a/benchmarks/Messaging/baselines/2026-09-06/soak/round1-foundatio-redis-pubsub-four.json b/benchmarks/Messaging/baselines/2026-09-06/soak/round1-foundatio-redis-pubsub-four.json new file mode 100644 index 000000000..7535be309 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/soak/round1-foundatio-redis-pubsub-four.json @@ -0,0 +1,17 @@ +{ + "Success": false, + "Options": { + "Scenario": "pubsub", + "ProducerConcurrency": 32, + "Transport": "redis", + "Engine": "foundatio", + "DeliveryCopies": 4, + "BatchSize": 1, + "ConsumerConcurrency": 8, + "MaxOutstanding": 1024, + "PayloadBytes": 1024, + "RatePerSecond": 0, + "Prefetch": 8 + }, + "Error": "Worker exited without a result. RUN fperf-fd29ab56ca9c foundatio/redis/pubsub\nPHASE warmup 5s\nFatal error.\nInternal CLR error. (0x80131506)\n[createdump] Gathering state for process 3077623 dotnet\n[createdump] Crashing thread 2ef630 signal 6 (0006)\n[createdump] Writing minidump to file /tmp/foundatio-perf-dumps/dotnet_3077623_1788731298.dmp\n[createdump] Written 335319040 bytes (81865 pages) to core file\n[createdump] Target process is alive\n[createdump] Dump successfully written in 283ms" +} diff --git a/benchmarks/Messaging/baselines/2026-09-06/soak/summary.csv b/benchmarks/Messaging/baselines/2026-09-06/soak/summary.csv new file mode 100644 index 000000000..4708fa218 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/soak/summary.csv @@ -0,0 +1,11 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/memory pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","1","97506.9","97506.9","97506.9","390027.7","0.051","10.623","11.647","29610.6","0.2186","172.6","11701265","0","0" +"foundatio/memory queue p32 c32 s1 1024B b1 r0 w1024 pf32","1","261223.3","261223.3","261223.3","261223.3","3.935","4.799","5.503","12474.1","0.0375","238.7","31348296","0","0" +"foundatio/redis pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","1","7947.3","7947.3","7947.3","31789.3","23.807","137.215","169.983","59900.8","0.3115","212.3","954234","0","0" +"foundatio/redis queue p32 c32 s1 1024B b1 r0 w1024 pf32","1","21301.8","21301.8","21301.8","21301.8","47.103","51.711","88.063","21729.3","0.1359","201.1","2557096","0","0" +"foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","1","98.1","98.1","98.1","392.5","10223.615","11272.191","11403.263","377580.8","2.169","133.4","12723","0","0" +"foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","1","753.2","753.2","753.2","753.2","1294.335","1638.399","1703.935","119171.3","0.6601","128.5","90903","0","0" +"masstransit/memory pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","1","75201.9","75201.9","75201.9","300807.5","6.143","16.255","22.015","65944.2","0.1202","216.4","9025543","0","0" +"masstransit/memory queue p32 c32 s1 1024B b1 r0 w1024 pf32","1","100338","100338","100338","100338","10.239","11.135","14.591","22656.6","0.0364","173.1","12042135","0","0" +"masstransit/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","1","445.7","445.7","445.7","1782.9","2260.991","2654.207","2818.047","189564.7","1.3981","153.8","54104","0","0" +"masstransit/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","1","2686.4","2686.4","2686.4","2686.4","368.639","413.695","704.511","67429.9","0.3549","153.2","322970","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06/soak/summary.md b/benchmarks/Messaging/baselines/2026-09-06/soak/summary.md new file mode 100644 index 000000000..f5002b701 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/soak/summary.md @@ -0,0 +1,28 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. These are local measurements, not cloud sizing claims. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/memory pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 1 | 97506.9 (97506.9-97506.9) | 390027.7 | 0.051 / 10.623 / 11.647 | 29610.6 | 0.2186 | +| foundatio/memory queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 1 | 261223.3 (261223.3-261223.3) | 261223.3 | 3.935 / 4.799 / 5.503 | 12474.1 | 0.0375 | +| foundatio/redis pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 1 | 7947.3 (7947.3-7947.3) | 31789.3 | 23.807 / 137.215 / 169.983 | 59900.8 | 0.3115 | +| foundatio/redis queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 1 | 21301.8 (21301.8-21301.8) | 21301.8 | 47.103 / 51.711 / 88.063 | 21729.3 | 0.1359 | +| foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 1 | 98.1 (98.1-98.1) | 392.5 | 10223.615 / 11272.191 / 11403.263 | 377580.8 | 2.169 | +| foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 1 | 753.2 (753.2-753.2) | 753.2 | 1294.335 / 1638.399 / 1703.935 | 119171.3 | 0.6601 | +| masstransit/memory pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 1 | 75201.9 (75201.9-75201.9) | 300807.5 | 6.143 / 16.255 / 22.015 | 65944.2 | 0.1202 | +| masstransit/memory queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 1 | 100338 (100338-100338) | 100338 | 10.239 / 11.135 / 14.591 | 22656.6 | 0.0364 | +| masstransit/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 1 | 445.7 (445.7-445.7) | 1782.9 | 2260.991 / 2654.207 / 2818.047 | 189564.7 | 1.3981 | +| masstransit/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 1 | 2686.4 (2686.4-2686.4) | 2686.4 | 368.639 / 413.695 / 704.511 | 67429.9 | 0.3549 | + +Failed trials: 1. +- round1-foundatio-redis-pubsub-four.json: Worker exited without a result. RUN fperf-fd29ab56ca9c foundatio/redis/pubsub +PHASE warmup 5s +Fatal error. +Internal CLR error. (0x80131506) +[createdump] Gathering state for process 3077623 dotnet +[createdump] Crashing thread 2ef630 signal 6 (0006) +[createdump] Writing minidump to file /tmp/foundatio-perf-dumps/dotnet_3077623_1788731298.dmp +[createdump] Written 335319040 bytes (81865 pages) to core file +[createdump] Target process is alive +[createdump] Dump successfully written in 283ms diff --git a/benchmarks/Messaging/baselines/2026-09-06/standard/round2-foundatio-memory-pubsub-four.json b/benchmarks/Messaging/baselines/2026-09-06/standard/round2-foundatio-memory-pubsub-four.json new file mode 100644 index 000000000..89b4621ce --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/standard/round2-foundatio-memory-pubsub-four.json @@ -0,0 +1,17 @@ +{ + "Error": "Worker exited without a result. RUN fperf-6b4027ac3eba foundatio/memory/pubsub\nFatal error.\nInternal CLR error. (0x80131506)\n[createdump] Gathering state for process 2967013 dotnet\n[createdump] Crashing thread 2d461f signal 6 (0006)\n[createdump] Writing minidump to file /tmp/foundatio-perf-dumps/dotnet_2967013_1788728719.dmp\n[createdump] Written 233418752 bytes (56987 pages) to core file\n[createdump] Target process is alive\n[createdump] Dump successfully written in 234ms", + "Success": false, + "Options": { + "ProducerConcurrency": 32, + "BatchSize": 1, + "MaxOutstanding": 1024, + "Transport": "memory", + "DeliveryCopies": 4, + "Engine": "foundatio", + "Prefetch": 8, + "PayloadBytes": 1024, + "ConsumerConcurrency": 8, + "Scenario": "pubsub", + "RatePerSecond": 0 + } +} diff --git a/benchmarks/Messaging/baselines/2026-09-06/standard/summary.csv b/benchmarks/Messaging/baselines/2026-09-06/standard/summary.csv new file mode 100644 index 000000000..de8ed916f --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/standard/summary.csv @@ -0,0 +1,21 @@ +"Case","Trials","InputsPerSecond","MinInputsPerSecond","MaxInputsPerSecond","DeliveriesPerSecond","P50Milliseconds","P95Milliseconds","P99Milliseconds","BytesPerInput","CpuMillisecondsPerInput","PeakWorkingSetMiB","TotalInputs","Duplicates","Missing" +"foundatio/memory pubsub p32 c32 s1 1024B b1 r0 w1024 pf32","3","210216.2","206762","213550.2","210216.2","4.287","8.703","16.127","13189.8","0.0427","1258.1","6311743","0","0" +"foundatio/memory pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","2","87561.1","85996.3","89125.9","350244.4","0.082","10.687","23.167","30732.4","0.2351","1944.9","1756161","0","0" +"foundatio/memory queue p1 c1 s1 1024B b1 r0 w1024 pf1","3","110286.8","108516.2","117264.2","110286.8","8.447","13.823","23.551","13486.6","0.0489","883.6","3368135","0","0" +"foundatio/memory queue p32 c32 s1 1024B b1 r0 w1024 pf32","3","213748.9","213651.9","215298.2","213748.9","4.159","8.447","16.895","12793.2","0.0415","1326.4","6435872","0","0" +"foundatio/redis pubsub p32 c32 s1 1024B b1 r0 w1024 pf32","3","21109.3","20905.6","21208.7","21109.3","47.103","53.247","89.087","22867.9","0.134","200.6","635471","0","0" +"foundatio/redis pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","3","7974.8","7907.6","8089.4","31899.3","106.495","139.263","159.743","62596.3","0.3257","220.7","242341","0","0" +"foundatio/redis queue p1 c1 s1 1024B b1 r0 w1024 pf1","3","5539.9","5530.2","5556.7","5539.9","184.319","200.703","229.375","17779.6","0.3379","108.6","169161","0","0" +"foundatio/redis queue p32 c32 s1 1024B b1 r0 w1024 pf32","3","21233","20886.3","21266.9","21233","47.103","53.247","88.063","22679.7","0.1344","198.7","636748","0","0" +"foundatio/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32","3","325.7","317.2","328.9","325.7","3047.423","3997.695","4095.999","74374.4","1.0478","146.2","10501","0","0" +"foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","3","100.8","87.5","101","403.2","8912.895","9699.327","9961.471","378535.4","2.1723","134","5616","0","0" +"foundatio/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1","3","406.2","398.6","423.8","406.2","2195.455","2686.975","2752.511","127522.5","0.8973","102.5","14265","0","0" +"foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","3","715.1","713.2","720.9","715.1","1294.335","1769.471","1802.239","21007.1","0.7896","131.6","23056","0","0" +"masstransit/memory pubsub p32 c32 s1 1024B b1 r0 w1024 pf32","3","100055","97742.3","102936.6","100055","10.111","13.183","15.231","23135","0.0361","130.6","3011199","0","0" +"masstransit/memory pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","3","73712.5","70149.2","73771.4","294850.2","5.759","16.255","21.503","65940.2","0.1196","173.3","2179134","0","0" +"masstransit/memory queue p1 c1 s1 1024B b1 r0 w1024 pf1","3","124599.5","124385.9","126402.6","124599.5","8.319","9.087","11.391","19676.4","0.029","121.6","3757891","0","0" +"masstransit/memory queue p32 c32 s1 1024B b1 r0 w1024 pf32","3","101358.1","100398.3","102476.1","101358.1","10.239","10.879","14.079","22655.3","0.036","129.1","3045742","0","0" +"masstransit/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32","3","1309.5","1253.5","1338.1","1309.5","745.471","1081.343","1130.495","60884.7","0.7053","145.6","39951","0","0" +"masstransit/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8","3","420.1","404.6","420.5","1680.4","2260.991","3178.495","3375.103","84800.1","1.746","159.3","13624","0","0" +"masstransit/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1","3","282.4","278.3","288.9","282.4","1900.543","3080.191","3145.727","162523.4","1.666","114.8","11072","0","0" +"masstransit/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32","3","2637.1","2630","2678.4","2637.1","372.735","438.271","712.703","67436.5","0.4545","149","81255","0","0" diff --git a/benchmarks/Messaging/baselines/2026-09-06/standard/summary.md b/benchmarks/Messaging/baselines/2026-09-06/standard/summary.md new file mode 100644 index 000000000..b4c6c7d03 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-06/standard/summary.md @@ -0,0 +1,37 @@ +# Messaging benchmark results + +Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. These are local measurements, not cloud sizing claims. + +| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| foundatio/memory pubsub p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 210216.2 (206762-213550.2) | 210216.2 | 4.287 / 8.703 / 16.127 | 13189.8 | 0.0427 | +| foundatio/memory pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 2 | 87561.1 (85996.3-89125.9) | 350244.4 | 0.082 / 10.687 / 23.167 | 30732.4 | 0.2351 | +| foundatio/memory queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 3 | 110286.8 (108516.2-117264.2) | 110286.8 | 8.447 / 13.823 / 23.551 | 13486.6 | 0.0489 | +| foundatio/memory queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 213748.9 (213651.9-215298.2) | 213748.9 | 4.159 / 8.447 / 16.895 | 12793.2 | 0.0415 | +| foundatio/redis pubsub p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 21109.3 (20905.6-21208.7) | 21109.3 | 47.103 / 53.247 / 89.087 | 22867.9 | 0.134 | +| foundatio/redis pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 3 | 7974.8 (7907.6-8089.4) | 31899.3 | 106.495 / 139.263 / 159.743 | 62596.3 | 0.3257 | +| foundatio/redis queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 3 | 5539.9 (5530.2-5556.7) | 5539.9 | 184.319 / 200.703 / 229.375 | 17779.6 | 0.3379 | +| foundatio/redis queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 21233 (20886.3-21266.9) | 21233 | 47.103 / 53.247 / 88.063 | 22679.7 | 0.1344 | +| foundatio/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 325.7 (317.2-328.9) | 325.7 | 3047.423 / 3997.695 / 4095.999 | 74374.4 | 1.0478 | +| foundatio/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 3 | 100.8 (87.5-101) | 403.2 | 8912.895 / 9699.327 / 9961.471 | 378535.4 | 2.1723 | +| foundatio/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 3 | 406.2 (398.6-423.8) | 406.2 | 2195.455 / 2686.975 / 2752.511 | 127522.5 | 0.8973 | +| foundatio/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 715.1 (713.2-720.9) | 715.1 | 1294.335 / 1769.471 / 1802.239 | 21007.1 | 0.7896 | +| masstransit/memory pubsub p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 100055 (97742.3-102936.6) | 100055 | 10.111 / 13.183 / 15.231 | 23135 | 0.0361 | +| masstransit/memory pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 3 | 73712.5 (70149.2-73771.4) | 294850.2 | 5.759 / 16.255 / 21.503 | 65940.2 | 0.1196 | +| masstransit/memory queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 3 | 124599.5 (124385.9-126402.6) | 124599.5 | 8.319 / 9.087 / 11.391 | 19676.4 | 0.029 | +| masstransit/memory queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 101358.1 (100398.3-102476.1) | 101358.1 | 10.239 / 10.879 / 14.079 | 22655.3 | 0.036 | +| masstransit/sqs pubsub p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 1309.5 (1253.5-1338.1) | 1309.5 | 745.471 / 1081.343 / 1130.495 | 60884.7 | 0.7053 | +| masstransit/sqs pubsub p32 c8 s4 1024B b1 r0 w1024 pf8 | 3 | 420.1 (404.6-420.5) | 1680.4 | 2260.991 / 3178.495 / 3375.103 | 84800.1 | 1.746 | +| masstransit/sqs queue p1 c1 s1 1024B b1 r0 w1024 pf1 | 3 | 282.4 (278.3-288.9) | 282.4 | 1900.543 / 3080.191 / 3145.727 | 162523.4 | 1.666 | +| masstransit/sqs queue p32 c32 s1 1024B b1 r0 w1024 pf32 | 3 | 2637.1 (2630-2678.4) | 2637.1 | 372.735 / 438.271 / 712.703 | 67436.5 | 0.4545 | + +Failed trials: 1. +- round2-foundatio-memory-pubsub-four.json: Worker exited without a result. RUN fperf-6b4027ac3eba foundatio/memory/pubsub +Fatal error. +Internal CLR error. (0x80131506) +[createdump] Gathering state for process 2967013 dotnet +[createdump] Crashing thread 2d461f signal 6 (0006) +[createdump] Writing minidump to file /tmp/foundatio-perf-dumps/dotnet_2967013_1788728719.dmp +[createdump] Written 233418752 bytes (56987 pages) to core file +[createdump] Target process is alive +[createdump] Dump successfully written in 234ms diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/README.md b/benchmarks/Messaging/baselines/2026-09-07-allocations/README.md new file mode 100644 index 000000000..88391d852 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/README.md @@ -0,0 +1,7 @@ +# Allocation follow-up data + +See [ALLOCATION_RESULTS.md](../../ALLOCATION_RESULTS.md) for findings and revision boundaries. `raw-results.tar.gz` contains all 76 untraced trials, request counts, logs, source patches, options and binary hashes. `summary.json` and `summary.csv` preserve each profile separately, including original anomalous checks and their repeats. `final-validation.json` records delivery and fingerprint checks. + +Main optimization: `0b3dfdc86687ab55d2ee6037e608fc97ac0a748d`. Final BOM-compatible code: `e677cf9a4c53fdea468344f175a4d40f2a698c9a`. Previous pipeline binaries: `77c20ea354919fd25ae300e49c5de7f3ed8da598`. The scripts preserve the exact Linux paths used; adapt the snapshot paths when replaying elsewhere. Main cross-platform benchmark commands remain in the benchmark README. + +The accompanying local artifact archive includes complete nettraces, allocation-stack JSON, the standalone TraceAnalysis reader, both optimized binary snapshots, test/build logs and a Git bundle. The official runtime and previous binary snapshots remain in the preceding pipeline artifact archive. diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/broker-audit.json b/benchmarks/Messaging/baselines/2026-09-07-allocations/broker-audit.json new file mode 100644 index 000000000..cbe1fe2d3 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/broker-audit.json @@ -0,0 +1,26 @@ +{ + "Utc": "2026-09-07T17:33:12.183021+00:00", + "Containers": [ + { + "Name": "foundatio-messaging-perf-localstack-1", + "Id": "e6cf284255f1a698d4e2af9fd846762d9067353496f01cde00b3e4c1aedc3267", + "Image": "localstack/localstack:3.8.1", + "ImageId": "sha256:b279c01f4cfb8f985a482e4014cabc1e2697b9d7a6c8c8db2e40f4d9f93687c7", + "NanoCpus": 4000000000, + "MemoryLimit": 3221225472 + }, + { + "Name": "foundatio-messaging-perf-redis-1", + "Id": "5a7b36f38e024e98d45becded6e5572c6c55b4834b7a8767a9033aabdb2df225", + "Image": "redis:8.6-alpine", + "ImageId": "sha256:2cc044fc5a07c9b701f8f1255a309ae9ad7856e694ac03513bf3648c01e40763", + "NanoCpus": 4000000000, + "MemoryLimit": 2147483648 + } + ], + "BenchmarkQueues": [], + "BenchmarkTopics": [], + "BenchmarkRedisKeys": [], + "RemainingConformanceQueues": 28, + "RemainingConformanceTopics": 2 +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/broker-shutdown.log b/benchmarks/Messaging/baselines/2026-09-07-allocations/broker-shutdown.log new file mode 100644 index 000000000..63f6e1cd3 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/broker-shutdown.log @@ -0,0 +1,10 @@ + Container foundatio-messaging-perf-localstack-1 Stopping + Container foundatio-messaging-perf-redis-1 Stopping + Container foundatio-messaging-perf-redis-1 Stopped + Container foundatio-messaging-perf-redis-1 Removing + Container foundatio-messaging-perf-redis-1 Removed + Container foundatio-messaging-perf-localstack-1 Stopped + Container foundatio-messaging-perf-localstack-1 Removing + Container foundatio-messaging-perf-localstack-1 Removed + Network foundatio-messaging-perf_default Removing + Network foundatio-messaging-perf_default Removed diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/candidate-binaries.json b/benchmarks/Messaging/baselines/2026-09-07-allocations/candidate-binaries.json new file mode 100644 index 000000000..d6728f1f1 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/candidate-binaries.json @@ -0,0 +1,39 @@ +{ + "MassTransit.dll": "a09c141c529567e022fceee7f7dc667a4d4c9c6b0d8277414d28a127bb603d38", + "AWSSDK.SimpleNotificationService.dll": "6ac92e8dd8a8b50a1853f4cdf299b97f0ff921566ea6fc4da6003ff080ad371c", + "MassTransit.AmazonSqsTransport.dll": "b15248c1f4b43288533c6beba3e8ea1d66a3d18ce0b91aecdbae785d055a8825", + "AWSSDK.SQS.dll": "dc64ed3911962a8ac69deecde14060aafd66621db9fc7f5019d03ec6e8fc12d1", + "AWSSDK.Core.dll": "855bf199a6e3ece420d16c9243d0e7c3704ee7f57b8709934cf2d59fb89458d6", + "MassTransit.Abstractions.dll": "506535fd1cb8db2800a25c377f7343bf72916a451384c07f2897d0355a5f1a0f", + "Microsoft.Extensions.Configuration.Abstractions.dll": "a7ae16937ad2931ec036cefde5bc230f6a29e4bcb4a0aca10a29a699a0a51b33", + "Microsoft.Bcl.TimeProvider.dll": "642edac2b7cbf0ac66db473f5abe08892ec08766a9cd661138190703379fc0e2", + "Microsoft.Extensions.Configuration.dll": "997b6440cff60fc5e4cc38fa7bddff453937b6f1c9bb349c923c08f278f8b121", + "Microsoft.Extensions.Configuration.Binder.dll": "e74c683b76e3f9bfdb9ea136e139244aca2361d7dba9b9c4505db40f2d140cdf", + "Microsoft.Extensions.DependencyInjection.dll": "660201ba6898f047c273a611306e4ea6750f5bbe1f6d6c4702ab301dbb16a869", + "Microsoft.Extensions.DependencyInjection.Abstractions.dll": "621f220163119dfb08c8d1dd5bc0df6c038bd9baab0662050042acd7f906b7b7", + "Microsoft.Extensions.Diagnostics.Abstractions.dll": "9702b8737009bb5f6c943cf7a3ba5c0991c7e4eb58a9e952d54835bb596fe563", + "Microsoft.Extensions.Diagnostics.HealthChecks.dll": "efc6218619ab046c41c24f41d6b0ce9b04ff74f5fb3856f8ddee8e5910f7639d", + "Microsoft.Extensions.Diagnostics.HealthChecks.Abstractions.dll": "6a4d25060c4d4951876db57ef649de2bcd96af7c4d0c0da47885889c34aacf80", + "Microsoft.Extensions.FileProviders.Abstractions.dll": "7ee935aadd8afe2f731d655ef54479ed6206e6d2aa52bfd1889cdf170b0f9399", + "Microsoft.Extensions.Hosting.Abstractions.dll": "89a4428a5d0feedde5aa0565509b9f4c6189efdcaf888cd1c2c3f979e32ccd7c", + "Microsoft.Extensions.Logging.dll": "c012df727dcb65440fbb1b9924744a96932b617a3d058cc9b9b91c855ebd760f", + "Microsoft.Extensions.Logging.Abstractions.dll": "5dcb4934cb0dcc5547aeaebebc5bb687cc2522390b7032d68713b29af64f7fd5", + "Microsoft.Extensions.Logging.Configuration.dll": "8314031fcf6d3c2e5337cb8834fd93e3ad574026d4b5702987c6b39989763a15", + "Microsoft.Extensions.Logging.Console.dll": "4a8a39d30cfdbd6060612fd8a60a42fd847172c8d0534c61976304bd21841d51", + "Microsoft.Extensions.Options.dll": "a5a6cd30705512cc512a12d4086c07b7a76243174299aaef6b0dee61832fb0d3", + "Microsoft.Extensions.Primitives.dll": "d85aa4a5e5acabb4a96798e2d280153ceb073ba924d0c5d6b5d63fa3f7bfceb8", + "Microsoft.Extensions.Options.ConfigurationExtensions.dll": "61a435deaf0535d5e130af286a597b44ca216c731663d5831ef98da71e34cccb", + "Pipelines.Sockets.Unofficial.dll": "fa2cdb1d5ffbb2b06512c92ce8bd18918a1a996171d0a72dfc73035bc586a711", + "StackExchange.Redis.dll": "77333f4bea4139ede071bf9a38fb6651ba06ec62563500c95ee9a9c2bbfdb55a", + "Foundatio.Messaging.Benchmarks.deps.json": "10a2905236aa5f1e9227fa11ea42dc8bfb9ad38668c936a68d108360c2f1e74a", + "Foundatio.Messaging.Benchmarks.runtimeconfig.json": "1d8b4d081d584b36eb80652bec54ad0ad0d4a09da48fd37cdb8b13d5b7f92d75", + "Foundatio.Redis.xml": "06ff65c053912cea1b7ccb312879e5b5bf42f2da0afe6f54dddd9a0f6df94925", + "Foundatio.Aws.dll": "c81bfecbeaa6c2b02de21c6c991be5b4fa7dd8f369dd695079682c273b6ed04f", + "Foundatio.Redis.dll": "faaafcbe418e14d792af0ab3d31183c29288e916a5ebcfb0ea01af0783d0a16d", + "Foundatio.xml": "a64704948f9d020622d17ca98356988be955fc4bd27bfe67c4433b2592b7aae1", + "Foundatio.dll": "f71a1ff8753d4c240b191c1d3ff3ff1ab3d19c142cb0d2187becd64119934ab0", + "Foundatio.Aws.xml": "8638bcd5ee70802edb8020dfcd25f8b67ef975ebab6973e9ef88b2681f308a6d", + "Foundatio.Messaging.Benchmarks": "c1047489e762d38b0b8248e2d106bac2cd10abf953cf6acaad3f34d7f4255c37", + "Foundatio.Messaging.Benchmarks.dll": "5c03b888720e134dd0a626a473298330552c6949bbe82ee3de44b27e6fe11d6a", + "Foundatio.Messaging.Benchmarks.pdb": "b9a7297219af9daae574cfa12f7f20ef6049db172c6020c3a84d5dfb7a824b97" +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/cleanup-audit.json b/benchmarks/Messaging/baselines/2026-09-07-allocations/cleanup-audit.json new file mode 100644 index 000000000..ce6a17104 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/cleanup-audit.json @@ -0,0 +1,24 @@ +{ + "RemainingContainers": [], + "OwnedVolumeEvents": [ + { + "Type": "volume", + "Action": "unmount", + "Actor": { + "ID": "ec0a140d6e9f1b32dc1bc2d185394fd6a8a5957de4af021c75199676a7290011", + "Attributes": { + "container": "e6cf284255f1a698d4e2af9fd846762d9067353496f01cde00b3e4c1aedc3267", + "driver": "local" + } + }, + "scope": "local", + "time": 1788802616, + "timeNano": 1788802616469076396 + } + ], + "RemovedOwnedVolumes": [ + "ec0a140d6e9f1b32dc1bc2d185394fd6a8a5957de4af021c75199676a7290011" + ], + "RemainingNetwork": [], + "RedisImageVolumes": null +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/compare-final.py b/benchmarks/Messaging/baselines/2026-09-07-allocations/compare-final.py new file mode 100644 index 000000000..f10ac678c --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/compare-final.py @@ -0,0 +1,66 @@ +import argparse, collections, datetime, json, os, pathlib, random, subprocess, time, hashlib +p=argparse.ArgumentParser() +p.add_argument('--output', required=True) +p.add_argument('--dotnet', default='dotnet') +p.add_argument('--transport', default='sqs') +p.add_argument('--concurrency', type=int, default=0) +p.add_argument('--producer-count', type=int, default=0) +p.add_argument('--window', type=int, default=1024) +p.add_argument('--max-messages', type=int, default=20000000) +p.add_argument('--seconds', type=int, default=10) +p.add_argument('--warmup', type=int, default=3) +p.add_argument('--repetitions', type=int, default=3) +p.add_argument('--variants', default='before,after,masstransit') +p.add_argument('--workloads', default='queue,fanout') +p.add_argument('--rate', type=int, default=0) +p.add_argument('--payload', type=int, default=1024) +p.add_argument('--batch', type=int, default=1) +a=p.parse_args() +root=pathlib.Path(a.output) +root.mkdir(parents=True, exist_ok=False) +base=pathlib.Path('/tmp/foundatio-fastest') +paths={'before':base/'77c20ea3-binaries/Foundatio.Messaging.Benchmarks.dll','after':pathlib.Path('/tmp/foundatio-allocations/e677cf9a-binaries/Foundatio.Messaging.Benchmarks.dll'),'masstransit':pathlib.Path('/tmp/foundatio-allocations/e677cf9a-binaries/Foundatio.Messaging.Benchmarks.dll')} +paths['delay1']=base/'coherent-binaries/Foundatio.Messaging.Benchmarks.dll' +paths['pipeline']=base/'7bd7c4f8-binaries/Foundatio.Messaging.Benchmarks.dll' +paths['previous']=base/'d07031ff-binaries/Foundatio.Messaging.Benchmarks.dll' +env=dict(os.environ, PERF_AWS_MODE='localstack', PERF_AWS_URL='http://localhost:24566', PERF_AWS_REGION='us-east-1') +crash_capture={} +if a.transport == 'redis' and a.seconds >= 120: + crash_capture={'DOTNET_DbgEnableMiniDump':'1','DOTNET_DbgMiniDumpType':'4','DOTNET_DbgMiniDumpName':'/tmp/foundatio-fastest/confirmed-crashes/%e_%p_%t.dmp','DOTNET_EnableCrashReport':'1'} + env.update(crash_capture) +cases=[(r,v,w) for r in range(1,a.repetitions+1) for v in a.variants.split(',') for w in a.workloads.split(',')] +random.Random(534).shuffle(cases) +metadata={'before_revision':'77c20ea354919fd25ae300e49c5de7f3ed8da598','after_revision':subprocess.check_output(['git','-C','/tmp/foundatio-pr-533-review','rev-parse','HEAD'],text=True).strip(),'started_utc':datetime.datetime.now(datetime.timezone.utc).isoformat(),'binaries':{v:{f.name:hashlib.sha256(f.read_bytes()).hexdigest() for f in paths[v].parent.glob('*.dll')} for v in a.variants.split(',')}} +metadata['crash_capture']=crash_capture +(root/'binary-manifest.json').write_text(json.dumps(metadata,indent=2)) +(root/'run-options.json').write_text(json.dumps(vars(a),indent=2)) +(root/'source.patch').write_text(subprocess.check_output(['git','-C','/tmp/foundatio-pr-533-review','diff','HEAD'],text=True)) +(root/'compare.py').write_text(pathlib.Path(__file__).read_text()) +failed=0 +for index,(r,v,w) in enumerate(cases): + target=root/v + target.mkdir(exist_ok=True) + name=f'round{r}-{w}' + output=target/(name+'.json') + scenario='pubsub' if w in ('fanout','pubsub-one') else 'queue' + consumers=a.concurrency or (1 if w=='serial' else 8 if w=='fanout' else 32) + producers=a.producer_count or (1 if w=='serial' else 8 if a.batch>1 else 32) + args=[a.dotnet,str(paths[v]),'--engine','masstransit' if v=='masstransit' else 'foundatio','--transport',a.transport,'--scenario',scenario,'--seconds',str(a.seconds),'--warmup',str(a.warmup),'--producers',str(producers),'--consumers',str(consumers),'--prefetch',str(consumers),'--subscribers','4' if w=='fanout' else '1','--outstanding',str(a.window),'--max-messages',str(a.max_messages),'--payload',str(a.payload),'--batch',str(a.batch),'--rate',str(a.rate),'--output',str(output)] + started=datetime.datetime.now(datetime.timezone.utc).isoformat() + print(f'[{index+1}/{len(cases)}] {v} {w} round {r}',flush=True) + with (target/(name+'.log')).open('w') as log: + run=subprocess.run(args,env=env,stdout=log,stderr=subprocess.STDOUT,timeout=a.seconds+300) + log=subprocess.run(['docker','logs','--since',started,'foundatio-messaging-perf-localstack-1'],capture_output=True,text=True,check=True) + counts=collections.Counter('.'.join(k) for k in __import__('re').findall(r'AWS (sqs|sns)\.(\w+) =>',log.stdout+log.stderr)) + (target/(name+'-requests.txt')).write_text(json.dumps(dict(counts),indent=2)) + if output.exists(): + result=json.loads(output.read_text()) + m=result.get('Measurement') or {} + print(f" success={result['Success']} inputs/s={m.get('InputsPerSecond',0):.0f} p99={m.get('DeliveryLatency',{}).get('P99Milliseconds',0):.2f} ms",flush=True) + if run.returncode or not output.exists() or not result['Success']: + failed+=1 + failure={'exit_code':run.returncode,'result_exists':output.exists(),'started_utc':started,'command':args} + (target/(name+'-failure.txt')).write_text(json.dumps(failure,indent=2)) + print(' FAILED, log retained',flush=True) +(root/'run-options.json').write_text(json.dumps(vars(a),indent=2)) +raise SystemExit(1 if failed else 0) diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/compare.py b/benchmarks/Messaging/baselines/2026-09-07-allocations/compare.py new file mode 100644 index 000000000..41b979ffe --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/compare.py @@ -0,0 +1,66 @@ +import argparse, collections, datetime, json, os, pathlib, random, subprocess, time, hashlib +p=argparse.ArgumentParser() +p.add_argument('--output', required=True) +p.add_argument('--dotnet', default='dotnet') +p.add_argument('--transport', default='sqs') +p.add_argument('--concurrency', type=int, default=0) +p.add_argument('--producer-count', type=int, default=0) +p.add_argument('--window', type=int, default=1024) +p.add_argument('--max-messages', type=int, default=20000000) +p.add_argument('--seconds', type=int, default=10) +p.add_argument('--warmup', type=int, default=3) +p.add_argument('--repetitions', type=int, default=3) +p.add_argument('--variants', default='before,after,masstransit') +p.add_argument('--workloads', default='queue,fanout') +p.add_argument('--rate', type=int, default=0) +p.add_argument('--payload', type=int, default=1024) +p.add_argument('--batch', type=int, default=1) +a=p.parse_args() +root=pathlib.Path(a.output) +root.mkdir(parents=True, exist_ok=False) +base=pathlib.Path('/tmp/foundatio-fastest') +paths={'before':base/'77c20ea3-binaries/Foundatio.Messaging.Benchmarks.dll','after':pathlib.Path('/tmp/foundatio-allocations/0b3dfdc8-binaries/Foundatio.Messaging.Benchmarks.dll'),'masstransit':pathlib.Path('/tmp/foundatio-allocations/0b3dfdc8-binaries/Foundatio.Messaging.Benchmarks.dll')} +paths['delay1']=base/'coherent-binaries/Foundatio.Messaging.Benchmarks.dll' +paths['pipeline']=base/'7bd7c4f8-binaries/Foundatio.Messaging.Benchmarks.dll' +paths['previous']=base/'d07031ff-binaries/Foundatio.Messaging.Benchmarks.dll' +env=dict(os.environ, PERF_AWS_MODE='localstack', PERF_AWS_URL='http://localhost:24566', PERF_AWS_REGION='us-east-1') +crash_capture={} +if a.transport == 'redis' and a.seconds >= 120: + crash_capture={'DOTNET_DbgEnableMiniDump':'1','DOTNET_DbgMiniDumpType':'4','DOTNET_DbgMiniDumpName':'/tmp/foundatio-fastest/confirmed-crashes/%e_%p_%t.dmp','DOTNET_EnableCrashReport':'1'} + env.update(crash_capture) +cases=[(r,v,w) for r in range(1,a.repetitions+1) for v in a.variants.split(',') for w in a.workloads.split(',')] +random.Random(534).shuffle(cases) +metadata={'before_revision':'77c20ea354919fd25ae300e49c5de7f3ed8da598','after_revision':subprocess.check_output(['git','-C','/tmp/foundatio-pr-533-review','rev-parse','HEAD'],text=True).strip(),'started_utc':datetime.datetime.now(datetime.timezone.utc).isoformat(),'binaries':{v:{f.name:hashlib.sha256(f.read_bytes()).hexdigest() for f in paths[v].parent.glob('*.dll')} for v in a.variants.split(',')}} +metadata['crash_capture']=crash_capture +(root/'binary-manifest.json').write_text(json.dumps(metadata,indent=2)) +(root/'run-options.json').write_text(json.dumps(vars(a),indent=2)) +(root/'source.patch').write_text(subprocess.check_output(['git','-C','/tmp/foundatio-pr-533-review','diff','HEAD'],text=True)) +(root/'compare.py').write_text(pathlib.Path(__file__).read_text()) +failed=0 +for index,(r,v,w) in enumerate(cases): + target=root/v + target.mkdir(exist_ok=True) + name=f'round{r}-{w}' + output=target/(name+'.json') + scenario='pubsub' if w in ('fanout','pubsub-one') else 'queue' + consumers=a.concurrency or (1 if w=='serial' else 8 if w=='fanout' else 32) + producers=a.producer_count or (1 if w=='serial' else 8 if a.batch>1 else 32) + args=[a.dotnet,str(paths[v]),'--engine','masstransit' if v=='masstransit' else 'foundatio','--transport',a.transport,'--scenario',scenario,'--seconds',str(a.seconds),'--warmup',str(a.warmup),'--producers',str(producers),'--consumers',str(consumers),'--prefetch',str(consumers),'--subscribers','4' if w=='fanout' else '1','--outstanding',str(a.window),'--max-messages',str(a.max_messages),'--payload',str(a.payload),'--batch',str(a.batch),'--rate',str(a.rate),'--output',str(output)] + started=datetime.datetime.now(datetime.timezone.utc).isoformat() + print(f'[{index+1}/{len(cases)}] {v} {w} round {r}',flush=True) + with (target/(name+'.log')).open('w') as log: + run=subprocess.run(args,env=env,stdout=log,stderr=subprocess.STDOUT,timeout=a.seconds+300) + log=subprocess.run(['docker','logs','--since',started,'foundatio-messaging-perf-localstack-1'],capture_output=True,text=True,check=True) + counts=collections.Counter('.'.join(k) for k in __import__('re').findall(r'AWS (sqs|sns)\.(\w+) =>',log.stdout+log.stderr)) + (target/(name+'-requests.txt')).write_text(json.dumps(dict(counts),indent=2)) + if output.exists(): + result=json.loads(output.read_text()) + m=result.get('Measurement') or {} + print(f" success={result['Success']} inputs/s={m.get('InputsPerSecond',0):.0f} p99={m.get('DeliveryLatency',{}).get('P99Milliseconds',0):.2f} ms",flush=True) + if run.returncode or not output.exists() or not result['Success']: + failed+=1 + failure={'exit_code':run.returncode,'result_exists':output.exists(),'started_utc':started,'command':args} + (target/(name+'-failure.txt')).write_text(json.dumps(failure,indent=2)) + print(' FAILED, log retained',flush=True) +(root/'run-options.json').write_text(json.dumps(vars(a),indent=2)) +raise SystemExit(1 if failed else 0) diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/confirmation-profiles.json b/benchmarks/Messaging/baselines/2026-09-07-allocations/confirmation-profiles.json new file mode 100644 index 000000000..669f5c3f0 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/confirmation-profiles.json @@ -0,0 +1,58 @@ +[ + [ + "memory-repeat", + [ + "--transport", + "memory", + "--variants", + "before,after", + "--workloads", + "queue", + "--seconds", + "30", + "--warmup", + "5", + "--repetitions", + "5" + ] + ], + [ + "redis-repeat", + [ + "--transport", + "redis", + "--variants", + "before,after", + "--workloads", + "fanout", + "--payload", + "16384", + "--seconds", + "20", + "--warmup", + "5", + "--repetitions", + "3" + ] + ], + [ + "aws-final-1024", + [ + "--variants", + "after", + "--repetitions", + "1" + ] + ], + [ + "aws-final-16384", + [ + "--variants", + "after", + "--payload", + "16384", + "--repetitions", + "1" + ] + ] +] \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/final-binaries.json b/benchmarks/Messaging/baselines/2026-09-07-allocations/final-binaries.json new file mode 100644 index 000000000..614474ba8 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/final-binaries.json @@ -0,0 +1,39 @@ +{ + "MassTransit.dll": "a09c141c529567e022fceee7f7dc667a4d4c9c6b0d8277414d28a127bb603d38", + "AWSSDK.SimpleNotificationService.dll": "6ac92e8dd8a8b50a1853f4cdf299b97f0ff921566ea6fc4da6003ff080ad371c", + "MassTransit.AmazonSqsTransport.dll": "b15248c1f4b43288533c6beba3e8ea1d66a3d18ce0b91aecdbae785d055a8825", + "AWSSDK.SQS.dll": "dc64ed3911962a8ac69deecde14060aafd66621db9fc7f5019d03ec6e8fc12d1", + "AWSSDK.Core.dll": "855bf199a6e3ece420d16c9243d0e7c3704ee7f57b8709934cf2d59fb89458d6", + "MassTransit.Abstractions.dll": "506535fd1cb8db2800a25c377f7343bf72916a451384c07f2897d0355a5f1a0f", + "Microsoft.Extensions.Configuration.Abstractions.dll": "a7ae16937ad2931ec036cefde5bc230f6a29e4bcb4a0aca10a29a699a0a51b33", + "Microsoft.Bcl.TimeProvider.dll": "642edac2b7cbf0ac66db473f5abe08892ec08766a9cd661138190703379fc0e2", + "Microsoft.Extensions.Configuration.dll": "997b6440cff60fc5e4cc38fa7bddff453937b6f1c9bb349c923c08f278f8b121", + "Microsoft.Extensions.Configuration.Binder.dll": "e74c683b76e3f9bfdb9ea136e139244aca2361d7dba9b9c4505db40f2d140cdf", + "Microsoft.Extensions.DependencyInjection.dll": "660201ba6898f047c273a611306e4ea6750f5bbe1f6d6c4702ab301dbb16a869", + "Microsoft.Extensions.DependencyInjection.Abstractions.dll": "621f220163119dfb08c8d1dd5bc0df6c038bd9baab0662050042acd7f906b7b7", + "Microsoft.Extensions.Diagnostics.Abstractions.dll": "9702b8737009bb5f6c943cf7a3ba5c0991c7e4eb58a9e952d54835bb596fe563", + "Microsoft.Extensions.Diagnostics.HealthChecks.dll": "efc6218619ab046c41c24f41d6b0ce9b04ff74f5fb3856f8ddee8e5910f7639d", + "Microsoft.Extensions.Diagnostics.HealthChecks.Abstractions.dll": "6a4d25060c4d4951876db57ef649de2bcd96af7c4d0c0da47885889c34aacf80", + "Microsoft.Extensions.FileProviders.Abstractions.dll": "7ee935aadd8afe2f731d655ef54479ed6206e6d2aa52bfd1889cdf170b0f9399", + "Microsoft.Extensions.Hosting.Abstractions.dll": "89a4428a5d0feedde5aa0565509b9f4c6189efdcaf888cd1c2c3f979e32ccd7c", + "Microsoft.Extensions.Logging.dll": "c012df727dcb65440fbb1b9924744a96932b617a3d058cc9b9b91c855ebd760f", + "Microsoft.Extensions.Logging.Abstractions.dll": "5dcb4934cb0dcc5547aeaebebc5bb687cc2522390b7032d68713b29af64f7fd5", + "Microsoft.Extensions.Logging.Configuration.dll": "8314031fcf6d3c2e5337cb8834fd93e3ad574026d4b5702987c6b39989763a15", + "Microsoft.Extensions.Logging.Console.dll": "4a8a39d30cfdbd6060612fd8a60a42fd847172c8d0534c61976304bd21841d51", + "Microsoft.Extensions.Options.dll": "a5a6cd30705512cc512a12d4086c07b7a76243174299aaef6b0dee61832fb0d3", + "Microsoft.Extensions.Primitives.dll": "d85aa4a5e5acabb4a96798e2d280153ceb073ba924d0c5d6b5d63fa3f7bfceb8", + "Microsoft.Extensions.Options.ConfigurationExtensions.dll": "61a435deaf0535d5e130af286a597b44ca216c731663d5831ef98da71e34cccb", + "Pipelines.Sockets.Unofficial.dll": "fa2cdb1d5ffbb2b06512c92ce8bd18918a1a996171d0a72dfc73035bc586a711", + "StackExchange.Redis.dll": "77333f4bea4139ede071bf9a38fb6651ba06ec62563500c95ee9a9c2bbfdb55a", + "Foundatio.Messaging.Benchmarks.deps.json": "10a2905236aa5f1e9227fa11ea42dc8bfb9ad38668c936a68d108360c2f1e74a", + "Foundatio.Messaging.Benchmarks.runtimeconfig.json": "1d8b4d081d584b36eb80652bec54ad0ad0d4a09da48fd37cdb8b13d5b7f92d75", + "Foundatio.xml": "a64704948f9d020622d17ca98356988be955fc4bd27bfe67c4433b2592b7aae1", + "Foundatio.Redis.dll": "5925d801a3fff92f1d877eef7d5da5d8e6a4d91146ec878a7d6911fba70cbc3d", + "Foundatio.Aws.dll": "f976a0cb9a36bb37372babd0ff1c1498305aaf53873d84ee81e727b022fb7a2a", + "Foundatio.Aws.xml": "8638bcd5ee70802edb8020dfcd25f8b67ef975ebab6973e9ef88b2681f308a6d", + "Foundatio.Redis.xml": "06ff65c053912cea1b7ccb312879e5b5bf42f2da0afe6f54dddd9a0f6df94925", + "Foundatio.dll": "253b51bf35c3299ef088cb7ff0285ec13b33fcf065a160b7586d5d014aa8e6d9", + "Foundatio.Messaging.Benchmarks": "c1047489e762d38b0b8248e2d106bac2cd10abf953cf6acaad3f34d7f4255c37", + "Foundatio.Messaging.Benchmarks.dll": "0a140e7ad5c1ee415be37fb966a2959bae57fc215ffcdae5daffe2372c9fcbd8", + "Foundatio.Messaging.Benchmarks.pdb": "20d670f6730b4f3a18050282f579caf376a73e541239b079b5df8d5f22339b27" +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/final-validation.json b/benchmarks/Messaging/baselines/2026-09-07-allocations/final-validation.json new file mode 100644 index 000000000..a6963a609 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/final-validation.json @@ -0,0 +1,26 @@ +{ + "Trials": 76, + "Profiles": { + "aws-1024": 18, + "aws-16384": 18, + "memory": 12, + "redis-16384": 4, + "aws-soak": 4, + "memory-repeat": 10, + "redis-repeat": 6, + "aws-final-1024": 2, + "aws-final-16384": 2 + }, + "Inputs": 118134011, + "Deliveries": 167597066, + "Missing": 0, + "Duplicates": 0, + "Invalid": 0, + "WorkerFailures": 0, + "UniquePrefixes": 76, + "CandidateBinaryFilesVerified": 37, + "FinalBinaryFilesVerified": 37, + "ValidDiagnosticCaptures": 5, + "Runtime": ".NET 10.0.11", + "CoreClrSha256": "3EBE90CD92B1EDF6742A41FA921A0C6326216FD1CCA45FDB5E055BEA33351BEA" +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/manifest.json b/benchmarks/Messaging/baselines/2026-09-07-allocations/manifest.json new file mode 100644 index 000000000..2996132ba --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/manifest.json @@ -0,0 +1,25 @@ +{ + "README.md": "28bf665dcdc240384496d23d9cb5ec284e68253b70e1f6b410f523fdb7dd42dd", + "broker-shutdown.log": "356251dbfa0be3df099066f92439a02620d04ae81b3e531e6bf616a94d979643", + "cleanup-audit.json": "5e25277257fe32abc15eeab9c35e856da0ba22fb3144e4b5df1a2c1d4c484c3d", + "trace-manifest.json": "7cf8d912d624fa683b96762c9698c4144bbef5237da048cb29e6c0772b9f0fcd", + "render-report.py": "c8c8f3166cd06a26efc031863e873f62b64efad0264d2cfe36dad1957cd321fa", + "validate-final.py": "798566fc0f11f1b7156c88bda1fdfa6200fa8f1a9e5f39825aae73071610e318", + "validate.py": "98d49fd29d15bb93ec4f08b00388ad94b88166feeb1959ef31dadf79ca341d02", + "summarize.py": "ff6f41cadcefffdb86364b92986191c84eeeefdaf52347ce40c5bb2576ca4da4", + "run-confirmation.py": "679a498ed69063fb92ae41fa4add072e52093ede90299647a136dfd43efe3a32", + "run-matrix.py": "6751f743d18ec5b9258217d2e9cfe83d06cd5a9597a242be95aaa97e3764980c", + "compare-final.py": "9fc1fe3a10a0664852dc40e1fa95c4c97db2d973f112503b4c76d5031e0102eb", + "compare.py": "18d3cfa339f7bba68f1d0c7c2fe0c905ad3068de0b30325459f7819a62f4258f", + "methodology.md": "458f396e58f84c2c5a6e4cabb88e6cc353a1dd0a614c34171a2d660bd01bef0f", + "final-binaries.json": "d592bdf1d100dcb3a982b8f5ae63e2ac8553c611bfdc3dea88bb7e8730412505", + "candidate-binaries.json": "ab3306c713a595ae60d9af712ba439ee834d0b17379d3094ac752ece1d749f9c", + "source-scan.json": "a53d0aad226ca72fc60c7e1bcdcd2fcbf70559c780793cd00e9e1ea4321585a7", + "broker-audit.json": "0414c57917b49d53d70270aa27520317a9873e91ed0fd5e07594db885d723783", + "final-validation.json": "84e04642e806c66a3d01990cff832503d329a51e59e864cd738a565cc74aebab", + "confirmation-profiles.json": "f702839c3ede15aed97ef9c587ac1707621756716870db71c8ac9a8d69e136f7", + "profiles.json": "eeb5e553b83461ebf45046c4cf58f873ad425b1934e2882c1a511b4dfdfef951", + "summary.json": "0c3d818fd52645abd3749f536bc63baac08af7b5edd10466fd0a27c2940a072f", + "summary.csv": "1e3f36e1b8e91e4af13dfaf114d9c2c36a3a4555dc97aa2e39edfe2610a0a566", + "raw-results.tar.gz": "3609fff1a0af232f07dd9146db21b1aacae366af50491a2c00d24bbceb9179ee" +} diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/methodology.md b/benchmarks/Messaging/baselines/2026-09-07-allocations/methodology.md new file mode 100644 index 000000000..305d5df60 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/methodology.md @@ -0,0 +1,17 @@ +# Allocation profiling methodology + +Baseline code: `77c20ea354919fd25ae300e49c5de7f3ed8da598` (the previous confirmed pipeline implementation). The main optimized matrix uses `0b3dfdc86687ab55d2ee6037e608fc97ac0a748d`. Final code and the after-traces use `e677cf9a4c53fdea468344f175a4d40f2a698c9a`, which additionally restores BOM-prefixed JSON compatibility. Both run under the preserved official Microsoft .NET 10.0.11 runtime. This pass does not replace the system runtime or contact an AWS account. + +Capture command: `dotnet-trace collect --profile gc-verbose --output .nettrace --show-child-io -- /Foundatio.Messaging.Benchmarks.dll --engine foundatio --transport sqs --scenario pubsub --seconds 30 --warmup 5 --producers 32 --consumers 8 --prefetch 8 --subscribers 4 --outstanding 1024 --max-messages 2000000 --payload <1024-or-16384> --output .json`. The comparison capture uses `--engine masstransit` with the same remaining arguments. All captured workloads use synthetic data and task-owned LocalStack resources. + +`TraceAnalysis` reads GCAllocationTick events and weights each sampled type/stack by AllocationAmount64. The analysis window is 12 to 30 seconds after trace start, excluding startup and final drain. Types and complete stacks are saved in JSON; inclusive method summaries can overlap. Sampling estimates allocation attribution, not precise per-method accounting. No missing events or missing allocation stacks were reported in the baseline analysis windows. The profiler's version is 10.0.731102; the offline reader uses the previously cached TraceEvent 3.1.21 package. See [Microsoft's dotnet-trace documentation](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-trace) for gc-verbose and collection options. + +Traced worker throughput/allocation totals are diagnostic only and are excluded from comparison medians. Untraced fresh-process trials measure total managed allocated bytes with GC.GetTotalAllocatedBytes, including SDK and harness work, excluding broker processes. Allocation per input counts four deliveries in fanout. It is allocation churn, not retained memory or isolated library overhead. Working-set observations include the benchmark's fixed tracking arrays and touched pages and cannot establish leak freedom. + +The untraced comparisons use 15-second publishing phases and 3-second warmup, a 1,024-input window, 32 producers, and 32 consumers for queues or eight consumers per subscriber for four-subscriber fanout. Prefetch equals per-endpoint consumer concurrency. Input tracking capacity is 20 million for every case, including 120-second soaks (5-second warmup). Each AWS payload/implementation/workload combination has three randomized repetitions. Redis large-payload checks and each sustained case have one repetition. In-memory checks compare the preserved baseline to the optimized code. Exact invocations, DLL hashes, logs and JSON results are saved per profile. + +Acknowledgement and final drain remain included in throughput. Every input/delivery is validated for missing, duplicate, and invalid delivery. No builds, test suites, or other profiling jobs run alongside measured workers. Other host workloads were left running, so small rate differences and overlapping ranges are treated cautiously. LocalStack is a broker emulator; these results do not establish live AWS performance. + +The optimization adds optional IBufferSerializer support selected automatically by existing extensions; the default JSON serializer preserves stream options, runtime types, null handling and primitive normalization. Existing serializers retain the stream fallback. AWS sends skip temporary per-message batching lists, build SDK attributes directly, and allocate unknown outcomes only for unconfirmed entries. Receive requests retain all application attributes and request only the system receive count used by the transport. Checksum validation, retry, cancellation, acknowledgements, and bounded batching remain enabled. No wire-format change is introduced in this allocation pass. + +Final-revision confirmation adds five 30-second in-memory queue trials per implementation, three 20-second Redis fanout trials per implementation, and one final AWS trial per workload/payload. Both repeats use up to five seconds of warmup, capped at one million inputs. Main matrix and final-revision profiles are summarized separately, without combining their medians. diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/profiles.json b/benchmarks/Messaging/baselines/2026-09-07-allocations/profiles.json new file mode 100644 index 000000000..ef2955c96 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/profiles.json @@ -0,0 +1,53 @@ +[ + [ + "aws-1024", + [ + "--payload", + "1024" + ] + ], + [ + "aws-16384", + [ + "--payload", + "16384" + ] + ], + [ + "memory", + [ + "--transport", + "memory", + "--variants", + "before,after" + ] + ], + [ + "redis-16384", + [ + "--transport", + "redis", + "--variants", + "before,after", + "--payload", + "16384", + "--repetitions", + "1" + ] + ], + [ + "aws-soak", + [ + "--variants", + "after,masstransit", + "--seconds", + "120", + "--warmup", + "5", + "--repetitions", + "1", + "--payload", + "16384" + ] + ] +] \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/raw-results.tar.gz b/benchmarks/Messaging/baselines/2026-09-07-allocations/raw-results.tar.gz new file mode 100644 index 000000000..8a5c07505 Binary files /dev/null and b/benchmarks/Messaging/baselines/2026-09-07-allocations/raw-results.tar.gz differ diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/render-report.py b/benchmarks/Messaging/baselines/2026-09-07-allocations/render-report.py new file mode 100644 index 000000000..722f0b0ab --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/render-report.py @@ -0,0 +1,77 @@ +import collections, json, pathlib, re +root = pathlib.Path('/tmp/foundatio-allocations') +repo = pathlib.Path('/tmp/foundatio-pr-533-review') +rows = json.loads((root / 'summary.json').read_text()) +lookup = {(r['Profile'], r['Variant'], r['Workload']): r for r in rows} +validation = json.loads((root / 'final-validation.json').read_text()) +lines = ['# Messaging allocation results', '', + 'The default JSON serializer now writes directly to an owned byte array and reads directly from input memory. Existing serializer extensions select the optional `IBufferSerializer` capability automatically. AWS sends also avoid intermediate dictionaries and single-message batching lists, and receive requests omit unused system attributes. The public messaging calls and wire format are unchanged by this allocation pass.', '', + 'The repeated optimization matrix measures `0b3dfdc86687ab55d2ee6037e608fc97ac0a748d` against the previous pipeline implementation, `77c20ea354919fd25ae300e49c5de7f3ed8da598`. Final code is `e677cf9a4c53fdea468344f175a4d40f2a698c9a`, which additionally preserves UTF-8 byte-order-mark handling. Its full test suite, 20 additional load trials, and two allocation traces passed. Results from those revisions/profiles are kept separate below.', '', + f'All **{validation["Trials"]} untraced trials passed**: **{validation["Inputs"]:,} inputs and {validation["Deliveries"]:,} acknowledged deliveries**, with zero missing, duplicate or invalid deliveries, zero tracking-limit failures and zero benchmark worker crashes. Five diagnostic captures also passed delivery validation; their performance totals are excluded from comparison medians.', '', + '## Repeated AWS comparison', '', + 'These are median **managed allocated bytes per input**, including SDK and harness work and excluding broker processes. Four-subscriber fanout requires four acknowledged deliveries per input. Each cell uses three fresh-process, untraced trials against LocalStack. Allocation churn is not retained memory.', '', + '| Payload / workload | Previous Foundatio | Optimized Foundatio | Reduction | MassTransit |', + '| --- | ---: | ---: | ---: | ---: |'] +for payload in (1024, 16384): + for workload in ('queue', 'fanout'): + p = lookup[(f'aws-{payload}', 'before', workload)]['AllocatedBytesPerInput'] + a = lookup[(f'aws-{payload}', 'after', workload)]['AllocatedBytesPerInput'] + m = lookup[(f'aws-{payload}', 'masstransit', workload)]['AllocatedBytesPerInput'] + lines.append(f'| {payload // 1024} KiB / {workload} | {p:,.0f} | {a:,.0f} | {100*(1-a/p):.1f}% | {m:,.0f} |') +lines += ['', 'The 16 KiB queue allocation gap against MassTransit is reversed in this matrix: Foundatio allocates about 7% less. Short-run AWS fanout still allocates more: about 5% at 1 KiB and 22% at 16 KiB. The two-minute fanout comparison below has the opposite allocation ordering. Do not generalize one payload or duration to all workloads.', '', + '| Payload / workload | Previous inputs/s | Optimized inputs/s (range) | MassTransit inputs/s (range) | Previous / optimized / MT p99 ms |', + '| --- | ---: | ---: | ---: | ---: |'] +for payload in (1024, 16384): + for workload in ('queue', 'fanout'): + p, a, m = [lookup[(f'aws-{payload}', v, workload)] for v in ('before', 'after', 'masstransit')] + lines.append(f'| {payload // 1024} KiB / {workload} | {p["InputsPerSecond"]:,.0f} | {a["InputsPerSecond"]:,.0f} ({a["Minimum"]:,.0f}–{a["Maximum"]:,.0f}) | {m["InputsPerSecond"]:,.0f} ({m["Minimum"]:,.0f}–{m["Maximum"]:,.0f}) | {p["P99Milliseconds"]:,.2f} / {a["P99Milliseconds"]:,.2f} / {m["P99Milliseconds"]:,.2f} |') +lines += ['', 'The 1 KiB queue/fanout median throughput changes versus the previous implementation are approximately +3%/+6%; the 16 KiB changes are +4%/+4%. Several ranges overlap. Saturation p99 includes a bounded backlog and final settlement; it is not unloaded request latency.', '', + '## Final-revision follow-up', '', + 'The initial three-run in-memory queue comparison showed 11% fewer allocated bytes but a 6% lower median rate. Five longer repetitions on final code did not reproduce a consistent slowdown. The initial single Redis fanout check allocated 5% more, so that case was repeated three times. Both original and repeated observations remain in the data.', '', + '| Profile | Previous / final bytes per input | Previous / final inputs/s (ranges) | Previous / final p99 ms |', + '| --- | ---: | ---: | ---: |'] +for profile, workload in [('memory-repeat', 'queue'), ('redis-repeat', 'fanout')]: + p, a = [lookup[(profile, v, workload)] for v in ('before', 'after')] + lines.append(f'| {profile} | {p["AllocatedBytesPerInput"]:,.0f} / {a["AllocatedBytesPerInput"]:,.0f} | {p["InputsPerSecond"]:,.0f} ({p["Minimum"]:,.0f}–{p["Maximum"]:,.0f}) / {a["InputsPerSecond"]:,.0f} ({a["Minimum"]:,.0f}–{a["Maximum"]:,.0f}) | {p["P99Milliseconds"]:,.2f} / {a["P99Milliseconds"]:,.2f} |') +lines += ['', 'The in-memory queue allocation reduction is about 11% across both studies. Redis fanout allocates about 14% less in the repeated study, but its median p99 is higher; there is no uniform tail-latency improvement. The original in-memory fanout study reduced allocation from 26,588 to 24,852 bytes/input (7%) with a median rate of 168,914 versus 181,925 inputs/s. The single Redis queue check reduced allocation from 147,438 to 59,837 bytes/input; that large change has only one trial per implementation.', '', + 'Final code also passed one confirmation per AWS workload and payload:', '', + '| Payload / workload | Final bytes/input | Final inputs/s | Final p99 ms |', + '| --- | ---: | ---: | ---: |'] +for payload in (1024, 16384): + for workload in ('queue', 'fanout'): + a = lookup[(f'aws-final-{payload}', 'after', workload)] + lines.append(f'| {payload // 1024} KiB / {workload} | {a["AllocatedBytesPerInput"]:,.0f} | {a["InputsPerSecond"]:,.0f} | {a["P99Milliseconds"]:,.2f} |') +lines += ['', 'Small-payload AWS allocation varied materially: the final queue confirmation was 34,473 bytes/input, versus the earlier optimized median of 28,548. The earlier repeated result is not a guaranteed reduction for every run. Exact allocation ranges, CPU, GC pauses, collections and working sets are retained in the summary and raw JSON.', '', + '## Sustained load and process memory', '', + 'These are single two-minute 16 KiB trials at the optimization revision, with the same 20-million-input tracker capacity. Peak working set includes SDK, harness, fixed tracking arrays and touched pages; it cannot establish leak freedom.', '', + '| Implementation / workload | Inputs/s | Bytes/input | Peak working set MiB | p99 ms |', + '| --- | ---: | ---: | ---: | ---: |'] +for variant in ('after', 'masstransit'): + for workload in ('queue', 'fanout'): + a = lookup[('aws-soak', variant, workload)] + lines.append(f'| {variant} / {workload} | {a["InputsPerSecond"]:,.0f} | {a["AllocatedBytesPerInput"]:,.0f} | {a["PeakWorkingSetMiB"]:.1f} | {a["P99Milliseconds"]:,.2f} |') +lines += ['', '## Allocation attribution', '', + 'GC-verbose EventPipe captures cover 1 KiB and 16 KiB fanout before and after, plus MassTransit at 16 KiB. The offline reader weights GCAllocationTick stacks by AllocationAmount64 over seconds 12–30 of each trace. All five windows have allocation stacks and zero reported lost events. These are sampled attribution estimates, separate from untraced allocation counters.', '', + 'The default serializer’s intermediate output-stream growth accounted for **6.46%** of weighted allocations in the previous 16 KiB trace and had **no samples** in the final trace. The allocation regression test independently failed before the fix at **10,012,800 allocated bytes for 5,000,200 output bytes** and passes with the buffer path under a 1.25× output-size budget.', '', + 'Largest remaining 16 KiB sampled sites include application payload strings (21.3%), SDK response strings (20.5%), SDK receive checksum buffers (11.1%), and Foundatio’s owned receive-body byte arrays (9.3%). The last buffer keeps raw-message, retry and dead-letter payloads independently owned. Checksum validation and the delivery guarantees were retained. Removing these remaining copies would need a separate ownership or SDK change; none is claimed here.', '', + '## Validation and reproducibility', '', + '- Final Release solution build passed for net8.0 and net10.0; only the pre-existing ASPIRE010 warning remains. The sibling-repository aggregate solution is unavailable in this isolated checkout.', + '- Final suites: **2,193 passed, 24 expected skips, zero failures** across core, AWS, Redis and benchmark validation. Serializer tests include stream-only implementations, custom options, nulls, runtime types, Unicode, primitives, sliced/non-array memory, and BOM-prefixed JSON.', + '- AWS tests cover automatic and explicit batches, byte limits, native headers, malformed envelopes, missing/duplicate response IDs, partial failure, cancellation, disposal and acknowledged settlement.', + '- Documentation build and changed-file whitespace checks passed. No dependencies were added to the library.', + '- Benchmark workers use the official Microsoft .NET 10.0.11 runtime, MassTransit 8.5.10 and matching SDK binaries. CoreCLR SHA-256: `3EBE90CD92B1EDF6742A41FA921A0C6326216FD1CCA45FDB5E055BEA33351BEA`.', + '- LocalStack 3.8.1: four CPUs, 3 GiB limit. Redis 8.6-alpine: four CPUs, 2 GiB limit, AOF every second. Main trials publish for 15 seconds after up to three seconds of warmup; the memory repeat uses 30 seconds and up to five seconds of warmup, and Redis repeat uses 20 seconds and up to five seconds. Warmup is capped at one million inputs.', + '- All measured workers run sequentially, without concurrent builds, tests or profiling. Other host applications are left running, so small changes and overlapping ranges require caution. No live AWS account was used; the existing explicit live mode is preserved.', + '- Cleanup verified zero fperf queues, topics and Redis keys. The two task-owned containers, network and LocalStack anonymous volume were removed; conformance resources were removed with those containers.', + '- The earlier Ubuntu-runtime native crashes remain unresolved; this pass uses the preserved official runtime and does not alter the system installation.', '', + 'Raw trial data, scripts, configuration, hashes and summaries are in [baselines/2026-09-07-allocations](baselines/2026-09-07-allocations/). The accompanying local artifact archive holds complete nettrace captures, allocation-stack JSON, the standalone TraceAnalysis reader, binary snapshots and validation logs. See its methodology for the exact capture command and [Microsoft’s trace documentation](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-trace) for the gc-verbose profile.', '', + '## Scoped source scan', '', + 'The five changed production serialization/AWS files were checked using the performance skill recipes. Counts below are code signals, not counts of defects; the remaining lists and dictionaries include bounded native requests, explicit batches and cold error/provisioning paths. The two AWS partial declarations are one sealed primary type; the existing public JSON serializer remains extensible (one of two primary class types sealed).', '', + '| Recipe | Hits |', '| --- | ---: |'] +for r in json.loads((root / 'source-scan.json').read_text())['Recipes']: + lines.append(f'| {r["Recipe"]} | {r["Count"]} |') +lines += ['', 'Three measured allocation opportunities were addressed: intermediate serializer output buffers, single-send batching/attribute scaffolding, and unused receive metadata. No critical pattern was found in this scoped scan. It is not a whole-repository performance audit.', ''] +report = '\n'.join(lines) +(root / 'ALLOCATION_RESULTS.md').write_text(report) +(repo / 'benchmarks/Messaging/ALLOCATION_RESULTS.md').write_text(report) +print('Wrote allocation report with all original and follow-up results.') diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/run-confirmation.py b/benchmarks/Messaging/baselines/2026-09-07-allocations/run-confirmation.py new file mode 100644 index 000000000..c5963d0f6 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/run-confirmation.py @@ -0,0 +1,13 @@ +import json, pathlib, subprocess, sys +root = pathlib.Path('/tmp/foundatio-allocations') +profiles = [ + ('memory-repeat', ['--transport', 'memory', '--variants', 'before,after', '--workloads', 'queue', '--seconds', '30', '--warmup', '5', '--repetitions', '5']), + ('redis-repeat', ['--transport', 'redis', '--variants', 'before,after', '--workloads', 'fanout', '--payload', '16384', '--seconds', '20', '--warmup', '5', '--repetitions', '3']), + ('aws-final-1024', ['--variants', 'after', '--repetitions', '1']), + ('aws-final-16384', ['--variants', 'after', '--payload', '16384', '--repetitions', '1']), +] +(root / 'confirmation-profiles.json').write_text(json.dumps(profiles, indent=2)) +for name, args in profiles: + print('PROFILE', name, flush=True) + result = subprocess.run([sys.executable, str(root / 'compare-final.py'), '--dotnet', '/tmp/foundatio-fastest/official-dotnet/dotnet', '--output', str(root / name), '--seconds', '15', *args]) + if result.returncode: sys.exit(result.returncode) diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/run-matrix.py b/benchmarks/Messaging/baselines/2026-09-07-allocations/run-matrix.py new file mode 100644 index 000000000..c2fad06fc --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/run-matrix.py @@ -0,0 +1,14 @@ +import json, pathlib, subprocess, sys +root = pathlib.Path('/tmp/foundatio-allocations') +profiles = [ + ('aws-1024', ['--payload', '1024']), + ('aws-16384', ['--payload', '16384']), + ('memory', ['--transport', 'memory', '--variants', 'before,after']), + ('redis-16384', ['--transport', 'redis', '--variants', 'before,after', '--payload', '16384', '--repetitions', '1']), + ('aws-soak', ['--variants', 'after,masstransit', '--seconds', '120', '--warmup', '5', '--repetitions', '1', '--payload', '16384']), +] +(root / 'profiles.json').write_text(json.dumps(profiles, indent=2)) +for name, args in profiles: + print('PROFILE', name, flush=True) + result = subprocess.run([sys.executable, str(root / 'compare.py'), '--dotnet', '/tmp/foundatio-fastest/official-dotnet/dotnet', '--output', str(root / name), '--seconds', '15', *args]) + if result.returncode: sys.exit(result.returncode) diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/source-scan.json b/benchmarks/Messaging/baselines/2026-09-07-allocations/source-scan.json new file mode 100644 index 000000000..9bdea0af2 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/source-scan.json @@ -0,0 +1,128 @@ +{ + "Files": [ + "src/Foundatio/Serializer/ISerializer.cs", + "src/Foundatio/Serializer/IBufferSerializer.cs", + "src/Foundatio/Serializer/SystemTextJsonSerializer.cs", + "src/Foundatio.Aws/AwsMessageTransport.Batching.cs", + "src/Foundatio.Aws/AwsMessageTransport.cs" + ], + "Recipes": [ + { + "Recipe": "IndexOf literal without comparison", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "Substring", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "StartsWith or EndsWith literal without comparison", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "Contains literal without comparison", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "ToLower or ToUpper without culture", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "Three Replace calls on one line", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "params", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "LINQ character predicate", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "new HttpClient", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "new JsonSerializerOptions", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "async void", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "Static dictionary", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "Static frozen dictionary", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "new List", + "Count": 5, + "Hits": [ + "src/Foundatio.Aws/AwsMessageTransport.cs:141: var entries = new List(response.Messages.Count);", + "src/Foundatio.Aws/AwsMessageTransport.Batching.cs:35: var prepared = new List(messages.Count);", + "src/Foundatio.Aws/AwsMessageTransport.Batching.cs:51: var batch = new List(10);", + "src/Foundatio.Aws/AwsMessageTransport.Batching.cs:133: var entries = new List(batch.Count);", + "src/Foundatio.Aws/AwsMessageTransport.Batching.cs:147: var entries = new List(batch.Count);" + ] + }, + { + "Recipe": "new Dictionary", + "Count": 5, + "Hits": [ + "src/Foundatio.Aws/AwsMessageTransport.cs:172: headers = MessageHeaders.Create(new Dictionary", + "src/Foundatio.Aws/AwsMessageTransport.cs:391: Attributes = new Dictionary { [\"Policy\"] = BuildQueuePolicy(queueArn, topicArn) }", + "src/Foundatio.Aws/AwsMessageTransport.cs:399: Attributes = new Dictionary { [\"RawMessageDelivery\"] = \"true\" },", + "src/Foundatio.Aws/AwsMessageTransport.cs:529: Condition = new { ArnEquals = new Dictionary { [\"aws:SourceArn\"] = topicArn } }", + "src/Foundatio.Aws/AwsMessageTransport.cs:607: var attributes = new Dictionary(_nativeMessageHeaders.Length + 1, StringComparer.Ordinal)" + ] + }, + { + "Recipe": "CurrentCulture comparer", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "LINQ chains", + "Count": 0, + "Hits": [] + }, + { + "Recipe": "Unsealed public or internal class declarations", + "Count": 1, + "Hits": [ + "src/Foundatio/Serializer/SystemTextJsonSerializer.cs:7:public class SystemTextJsonSerializer : ITextSerializer, IBufferSerializer" + ] + }, + { + "Recipe": "Sealed class declarations", + "Count": 2, + "Hits": [ + "src/Foundatio.Aws/AwsMessageTransport.cs:32:public sealed partial class AwsMessageTransport : IMessageTransport, ISupportsPull, ISupportsVisibilityTimeout,", + "src/Foundatio.Aws/AwsMessageTransport.Batching.cs:16:public sealed partial class AwsMessageTransport" + ] + }, + { + "Recipe": "Synchronous task waits", + "Count": 0, + "Hits": [] + } + ] +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/summarize.py b/benchmarks/Messaging/baselines/2026-09-07-allocations/summarize.py new file mode 100644 index 000000000..872a6075a --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/summarize.py @@ -0,0 +1,26 @@ +import collections,csv,json,pathlib,statistics,sys +base=pathlib.Path('/tmp/foundatio-allocations') +paths=[base/n for n in sys.argv[1:]] if len(sys.argv)>1 else sorted(base.glob('confirmed-*')) +rows=[] +for root in paths: + if not root.is_dir(): continue + groups=collections.defaultdict(list) + for p in root.glob('*/*.json'): + r=json.loads(p.read_text()) + if 'Measurement' not in r: continue + o=r['Options']; m=r.get('Measurement') + name=p.stem.partition('-')[2] + groups[(p.parent.name,name)].append((r,m)) + for (variant,name),runs in sorted(groups.items()): + ms=[m for r,m in runs if r['Success'] and m] + if len(ms)!=len(runs): print('FAILED',root.name,variant,name,file=sys.stderr) + if not ms: continue + median=lambda f:statistics.median(f(m) for m in ms) + rates=[m['InputsPerSecond'] for m in ms] + row={'Profile':root.name,'Variant':variant,'Workload':name,'Trials':len(ms),'Inputs':sum(m['Inputs'] for m in ms),'Deliveries':sum(m['Deliveries'] for m in ms),'InputsPerSecond':statistics.median(rates),'Minimum':min(rates),'Maximum':max(rates),'P50Milliseconds':median(lambda m:m['DeliveryLatency']['P50Milliseconds']),'P99Milliseconds':median(lambda m:m['DeliveryLatency']['P99Milliseconds']),'AllocatedBytesPerInput':median(lambda m:m['AllocatedBytesPerInput']),'MinimumAllocatedBytesPerInput':min(m['AllocatedBytesPerInput'] for m in ms),'MaximumAllocatedBytesPerInput':max(m['AllocatedBytesPerInput'] for m in ms),'CpuMillisecondsPerInput':median(lambda m:m['CpuMilliseconds']/m['Inputs']),'GcPauseMillisecondsPerThousandInputs':median(lambda m:m['GcPauseMilliseconds']*1000/m['Inputs']),'Gen0CollectionsPerMillionInputs':median(lambda m:m['Collections'][0]*1000000/m['Inputs']),'PeakWorkingSetMiB':median(lambda m:m['PeakWorkingSetBytes']/1024**2),'Duplicates':sum(m['Duplicates'] for m in ms),'Missing':sum(m['Missing'] for m in ms),'Invalid':sum(m['Invalid'] for m in ms)} + rows.append(row) + print(f"{root.name:24} {variant:12} {name:11} n={len(ms)} rate={row['InputsPerSecond']:,.0f} ({min(rates):,.0f}-{max(rates):,.0f}) p99={row['P99Milliseconds']:,.2f} alloc={row['AllocatedBytesPerInput']:,.0f}") +if rows: + with (base/'summary.csv').open('w') as stream: + writer=csv.DictWriter(stream,fieldnames=list(rows[0]),lineterminator="\n"); writer.writeheader();writer.writerows(rows) + (base/'summary.json').write_text(json.dumps(rows,indent=2)) diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/summary.csv b/benchmarks/Messaging/baselines/2026-09-07-allocations/summary.csv new file mode 100644 index 000000000..9876555b3 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/summary.csv @@ -0,0 +1,33 @@ +Profile,Variant,Workload,Trials,Inputs,Deliveries,InputsPerSecond,Minimum,Maximum,P50Milliseconds,P99Milliseconds,AllocatedBytesPerInput,MinimumAllocatedBytesPerInput,MaximumAllocatedBytesPerInput,CpuMillisecondsPerInput,GcPauseMillisecondsPerThousandInputs,Gen0CollectionsPerMillionInputs,PeakWorkingSetMiB,Duplicates,Missing,Invalid +aws-1024,after,fanout,3,22270,89080,457.9334255526675,444.90664139000273,510.0235366016004,1982.463,2686.975,121377.13634420857,117160.72472822691,150949.3918322296,1.0744658783302465,11.61148311725317,12370.361114582032,126.3828125,0,0,0 +aws-1024,after,queue,3,137759,137759,3076.767424631212,2892.001418425552,3090.058169569636,315.391,704.511,28547.811802914406,27762.21482712312,29240.82987410684,0.30170781163198157,2.1604632280671767,2927.2253322507586,138.09765625,0,0,0 +aws-1024,before,fanout,3,20251,81004,433.392104268222,411.388614061616,445.78350029064205,2015.231,2981.887,137548.99399485273,134082.21988882025,160884.43356437105,1.0971639901173564,13.884329425221619,13279.802347127858,127.453125,0,0,0 +aws-1024,before,queue,3,138584,138584,2990.8180446902593,2987.2842509534134,3130.1110950228062,315.391,688.127,33911.04933865211,33347.758896151056,35597.392217608154,0.27641032962418643,2.2474490867100565,3102.9246715521226,137.90234375,0,0,0 +aws-1024,masstransit,fanout,3,20346,81384,416.79457637500116,399.97517679830594,461.1227009836365,2260.991,3604.479,116064.34489222118,113862.36352657004,126488.73224043715,1.7052680327868852,13.541213768115941,10153.077163386442,162.60546875,0,0,0 +aws-1024,masstransit,queue,3,120408,120408,2586.5031778519706,2571.2092562225553,2731.798539649258,372.735,729.087,67496.03875320195,67452.82642723389,67512.93986013986,0.44878316592498413,4.124094087730452,2644.628099173554,146.89453125,0,0,0 +aws-16384,after,fanout,3,15929,63716,326.31904383217704,321.4881141679614,342.9770618688676,2818.047,4063.231,602618.4508937151,566345.029299363,665387.671955649,1.6608753412192903,20.471802714586122,16560.509554140128,225.34765625,0,0,0 +aws-16384,after,queue,3,96033,96033,2104.1302415382593,2035.6779960330787,2141.0384053898606,421.887,778.239,223595.7842344304,223529.5561700212,223633.02510294341,0.5237091706588689,6.370596309287784,6470.478442107883,210.0234375,0,0,0 +aws-16384,before,fanout,3,15562,62248,312.82624699366005,308.3515514377824,341.65086069549227,2916.351,4161.535,624166.8552679548,613501.4221335992,636833.9917012448,1.7152036091870215,21.593875318993803,17980.63623789765,229.4765625,0,0,0 +aws-16384,before,queue,3,92166,92166,2022.522297768339,1955.0513287277977,2050.2335994222426,450.559,819.199,274384.58385410026,274371.65436893207,274388.94575613673,0.6097751320981873,7.462394822006472,8025.88996763754,206.3125,0,0,0 +aws-16384,masstransit,fanout,3,15202,60808,310.4255939183654,289.05956380838364,334.2443879188526,3178.495,4063.231,491947.8025276461,481357.40448962303,547260.3692762186,2.608151066350711,26.336887835703003,18212.621770436257,258.828125,0,0,0 +aws-16384,masstransit,queue,3,87347,87347,1904.008406462041,1871.7782751115549,1925.8600711545125,511.999,843.775,240330.51730139944,240326.68675153554,240360.0393513807,0.7677793981402052,7.8255648280073276,6691.143670864359,226.67578125,0,0,0 +memory,after,fanout,3,8067377,32269508,181925.0272858953,171247.04550995142,184424.03344848237,0.663,7.039,24852.416328575888,24847.489039035674,25116.43887586455,0.08415493433105703,0.7248032626675985,847.9741659084349,137.12890625,0,0,0 +memory,after,queue,3,11387509,11387509,247620.8427097453,247596.59397609625,263650.6528485089,4.031,5.631,10741.42624214889,10738.707088648474,10743.558423437724,0.03017443010563423,0.31099724751901264,388.8799181872006,132.9140625,0,0,0 +memory,before,fanout,3,7710009,30840036,168913.61108260372,166035.9351719147,178853.82549526374,0.363,7.487,26588.03404908048,26379.838516352407,26626.785721825483,0.0872089047323088,0.7912150342408196,934.7591118945588,138.390625,0,0,0 +memory,before,queue,3,11837879,11837879,264441.2696173366,257387.93837206915,267051.7461687244,3.871,5.503,12031.404347656839,12031.39731915147,12031.526329596689,0.03449835857439017,0.3395715463297126,425.4531262944505,135.65234375,0,0,0 +redis-16384,after,fanout,1,50714,202856,3329.3807577627013,3329.3807577627013,3329.3807577627013,282.623,360.447,274429.0474425208,274429.0474425208,274429.0474425208,0.5277029222699846,4.298024214220925,9385.968371652798,161.78125,0,0,0 +redis-16384,after,queue,1,122078,122078,8083.735736654274,8083.735736654274,8083.735736654274,121.855,169.983,59836.87498156916,59836.87498156916,59836.87498156916,0.1819287422795262,2.0781385671455954,2973.50874031357,144.59375,0,0,0 +redis-16384,before,fanout,1,50771,203084,3351.9633552123914,3351.9633552123914,3351.9633552123914,212.991,364.543,262058.08244864194,262058.08244864194,262058.08244864194,0.5518220046877155,4.386775915384767,10537.511571566445,147.39453125,0,0,0 +redis-16384,before,queue,1,116201,116201,7701.980053992006,7701.980053992006,7701.980053992006,124.927,174.079,147438.0582611165,147438.0582611165,147438.0582611165,0.21424124577241158,2.1162726654675947,4173.802290858082,133.53125,0,0,0 +aws-soak,after,fanout,1,41704,166816,344.65979946396345,344.65979946396345,344.65979946396345,2654.207,3604.479,653892.4803376175,653892.4803376175,653892.4803376175,1.4193026568194897,22.103659121427203,17720.12276999808,228.78125,0,0,0 +aws-soak,after,queue,1,257140,257140,2138.198766125768,2138.198766125768,2138.198766125768,454.655,827.391,201234.541868243,201234.541868243,201234.541868243,0.43069657385082055,9.442163801820021,6968.966321848021,219.7109375,0,0,0 +aws-soak,masstransit,fanout,1,38321,153284,316.3897526473897,316.3897526473897,316.3897526473897,3047.423,4259.839,728936.541948279,728936.541948279,728936.541948279,2.138235327888103,31.04170037316354,18501.604864173692,268.640625,0,0,0 +aws-soak,masstransit,queue,1,235670,235670,1959.0592966840964,1959.0592966840964,1959.0592966840964,507.903,835.583,240295.1237917427,240295.1237917427,240295.1237917427,0.5924802520473543,11.436928756311792,6882.5051979462805,239.40234375,0,0,0 +memory-repeat,after,queue,5,38917198,38917198,264441.7055904249,248357.98754968753,267082.99610812764,3.839,5.311,10742.68599879125,10742.00058720632,10743.569808152519,0.032986227289221064,0.3032774086418513,388.1037688464586,152.56640625,0,0,0 +memory-repeat,before,queue,5,38024577,38024577,246086.47509034444,241039.33851778696,272561.7946698594,4.063,5.311,12030.60639449439,12026.480290032765,12031.297666693934,0.030982540083013356,0.3234400019956328,413.72606296788797,155.28125,0,0,0 +redis-repeat,after,fanout,3,206895,827580,3455.4586847078895,3302.7109599254304,3483.686525953544,245.759,454.655,248529.98246912175,236179.0834047905,274407.5151769772,0.5131245784027897,4.0043160121191335,9189.390041730978,164.0078125,0,0,0 +redis-repeat,before,fanout,3,199479,797916,3283.0668881615175,3264.056661695995,3320.789585529518,266.239,356.351,287821.2279718612,243265.04897694936,293813.6036626038,0.4982813878621677,4.4717876417439815,10611.848887271844,148.47265625,0,0,0 +aws-final-1024,after,fanout,1,7104,28416,449.4921791872479,449.4921791872479,449.4921791872479,2097.151,3211.263,126738.88513513513,126738.88513513513,126738.88513513513,1.0998486768018019,11.73367117117117,12246.621621621622,141.42578125,0,0,0 +aws-final-1024,after,queue,1,44521,44521,2898.7633418901796,2898.7633418901796,2898.7633418901796,331.775,663.551,34473.18937130792,34473.18937130792,34473.18937130792,0.2566580040879585,1.7819231373958357,1954.1340041778037,142.48046875,0,0,0 +aws-final-16384,after,fanout,1,5751,23004,358.4866321325999,358.4866321325999,358.4866321325999,2490.367,3964.927,607458.6847504781,607458.6847504781,607458.6847504781,1.680133368109894,19.90923317683881,16171.100678142931,237.7890625,0,0,0 +aws-final-16384,after,queue,1,31256,31256,2045.2644987796532,2045.2644987796532,2045.2644987796532,458.751,876.543,223639.01356539543,223639.01356539543,223639.01356539543,0.5686103468134118,6.222997184540568,6398.77143588431,215.49609375,0,0,0 diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/summary.json b/benchmarks/Messaging/baselines/2026-09-07-allocations/summary.json new file mode 100644 index 000000000..c2ca2c958 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/summary.json @@ -0,0 +1,738 @@ +[ + { + "Profile": "aws-1024", + "Variant": "after", + "Workload": "fanout", + "Trials": 3, + "Inputs": 22270, + "Deliveries": 89080, + "InputsPerSecond": 457.9334255526675, + "Minimum": 444.90664139000273, + "Maximum": 510.0235366016004, + "P50Milliseconds": 1982.463, + "P99Milliseconds": 2686.975, + "AllocatedBytesPerInput": 121377.13634420857, + "MinimumAllocatedBytesPerInput": 117160.72472822691, + "MaximumAllocatedBytesPerInput": 150949.3918322296, + "CpuMillisecondsPerInput": 1.0744658783302465, + "GcPauseMillisecondsPerThousandInputs": 11.61148311725317, + "Gen0CollectionsPerMillionInputs": 12370.361114582032, + "PeakWorkingSetMiB": 126.3828125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-1024", + "Variant": "after", + "Workload": "queue", + "Trials": 3, + "Inputs": 137759, + "Deliveries": 137759, + "InputsPerSecond": 3076.767424631212, + "Minimum": 2892.001418425552, + "Maximum": 3090.058169569636, + "P50Milliseconds": 315.391, + "P99Milliseconds": 704.511, + "AllocatedBytesPerInput": 28547.811802914406, + "MinimumAllocatedBytesPerInput": 27762.21482712312, + "MaximumAllocatedBytesPerInput": 29240.82987410684, + "CpuMillisecondsPerInput": 0.30170781163198157, + "GcPauseMillisecondsPerThousandInputs": 2.1604632280671767, + "Gen0CollectionsPerMillionInputs": 2927.2253322507586, + "PeakWorkingSetMiB": 138.09765625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-1024", + "Variant": "before", + "Workload": "fanout", + "Trials": 3, + "Inputs": 20251, + "Deliveries": 81004, + "InputsPerSecond": 433.392104268222, + "Minimum": 411.388614061616, + "Maximum": 445.78350029064205, + "P50Milliseconds": 2015.231, + "P99Milliseconds": 2981.887, + "AllocatedBytesPerInput": 137548.99399485273, + "MinimumAllocatedBytesPerInput": 134082.21988882025, + "MaximumAllocatedBytesPerInput": 160884.43356437105, + "CpuMillisecondsPerInput": 1.0971639901173564, + "GcPauseMillisecondsPerThousandInputs": 13.884329425221619, + "Gen0CollectionsPerMillionInputs": 13279.802347127858, + "PeakWorkingSetMiB": 127.453125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-1024", + "Variant": "before", + "Workload": "queue", + "Trials": 3, + "Inputs": 138584, + "Deliveries": 138584, + "InputsPerSecond": 2990.8180446902593, + "Minimum": 2987.2842509534134, + "Maximum": 3130.1110950228062, + "P50Milliseconds": 315.391, + "P99Milliseconds": 688.127, + "AllocatedBytesPerInput": 33911.04933865211, + "MinimumAllocatedBytesPerInput": 33347.758896151056, + "MaximumAllocatedBytesPerInput": 35597.392217608154, + "CpuMillisecondsPerInput": 0.27641032962418643, + "GcPauseMillisecondsPerThousandInputs": 2.2474490867100565, + "Gen0CollectionsPerMillionInputs": 3102.9246715521226, + "PeakWorkingSetMiB": 137.90234375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-1024", + "Variant": "masstransit", + "Workload": "fanout", + "Trials": 3, + "Inputs": 20346, + "Deliveries": 81384, + "InputsPerSecond": 416.79457637500116, + "Minimum": 399.97517679830594, + "Maximum": 461.1227009836365, + "P50Milliseconds": 2260.991, + "P99Milliseconds": 3604.479, + "AllocatedBytesPerInput": 116064.34489222118, + "MinimumAllocatedBytesPerInput": 113862.36352657004, + "MaximumAllocatedBytesPerInput": 126488.73224043715, + "CpuMillisecondsPerInput": 1.7052680327868852, + "GcPauseMillisecondsPerThousandInputs": 13.541213768115941, + "Gen0CollectionsPerMillionInputs": 10153.077163386442, + "PeakWorkingSetMiB": 162.60546875, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-1024", + "Variant": "masstransit", + "Workload": "queue", + "Trials": 3, + "Inputs": 120408, + "Deliveries": 120408, + "InputsPerSecond": 2586.5031778519706, + "Minimum": 2571.2092562225553, + "Maximum": 2731.798539649258, + "P50Milliseconds": 372.735, + "P99Milliseconds": 729.087, + "AllocatedBytesPerInput": 67496.03875320195, + "MinimumAllocatedBytesPerInput": 67452.82642723389, + "MaximumAllocatedBytesPerInput": 67512.93986013986, + "CpuMillisecondsPerInput": 0.44878316592498413, + "GcPauseMillisecondsPerThousandInputs": 4.124094087730452, + "Gen0CollectionsPerMillionInputs": 2644.628099173554, + "PeakWorkingSetMiB": 146.89453125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-16384", + "Variant": "after", + "Workload": "fanout", + "Trials": 3, + "Inputs": 15929, + "Deliveries": 63716, + "InputsPerSecond": 326.31904383217704, + "Minimum": 321.4881141679614, + "Maximum": 342.9770618688676, + "P50Milliseconds": 2818.047, + "P99Milliseconds": 4063.231, + "AllocatedBytesPerInput": 602618.4508937151, + "MinimumAllocatedBytesPerInput": 566345.029299363, + "MaximumAllocatedBytesPerInput": 665387.671955649, + "CpuMillisecondsPerInput": 1.6608753412192903, + "GcPauseMillisecondsPerThousandInputs": 20.471802714586122, + "Gen0CollectionsPerMillionInputs": 16560.509554140128, + "PeakWorkingSetMiB": 225.34765625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-16384", + "Variant": "after", + "Workload": "queue", + "Trials": 3, + "Inputs": 96033, + "Deliveries": 96033, + "InputsPerSecond": 2104.1302415382593, + "Minimum": 2035.6779960330787, + "Maximum": 2141.0384053898606, + "P50Milliseconds": 421.887, + "P99Milliseconds": 778.239, + "AllocatedBytesPerInput": 223595.7842344304, + "MinimumAllocatedBytesPerInput": 223529.5561700212, + "MaximumAllocatedBytesPerInput": 223633.02510294341, + "CpuMillisecondsPerInput": 0.5237091706588689, + "GcPauseMillisecondsPerThousandInputs": 6.370596309287784, + "Gen0CollectionsPerMillionInputs": 6470.478442107883, + "PeakWorkingSetMiB": 210.0234375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-16384", + "Variant": "before", + "Workload": "fanout", + "Trials": 3, + "Inputs": 15562, + "Deliveries": 62248, + "InputsPerSecond": 312.82624699366005, + "Minimum": 308.3515514377824, + "Maximum": 341.65086069549227, + "P50Milliseconds": 2916.351, + "P99Milliseconds": 4161.535, + "AllocatedBytesPerInput": 624166.8552679548, + "MinimumAllocatedBytesPerInput": 613501.4221335992, + "MaximumAllocatedBytesPerInput": 636833.9917012448, + "CpuMillisecondsPerInput": 1.7152036091870215, + "GcPauseMillisecondsPerThousandInputs": 21.593875318993803, + "Gen0CollectionsPerMillionInputs": 17980.63623789765, + "PeakWorkingSetMiB": 229.4765625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-16384", + "Variant": "before", + "Workload": "queue", + "Trials": 3, + "Inputs": 92166, + "Deliveries": 92166, + "InputsPerSecond": 2022.522297768339, + "Minimum": 1955.0513287277977, + "Maximum": 2050.2335994222426, + "P50Milliseconds": 450.559, + "P99Milliseconds": 819.199, + "AllocatedBytesPerInput": 274384.58385410026, + "MinimumAllocatedBytesPerInput": 274371.65436893207, + "MaximumAllocatedBytesPerInput": 274388.94575613673, + "CpuMillisecondsPerInput": 0.6097751320981873, + "GcPauseMillisecondsPerThousandInputs": 7.462394822006472, + "Gen0CollectionsPerMillionInputs": 8025.88996763754, + "PeakWorkingSetMiB": 206.3125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-16384", + "Variant": "masstransit", + "Workload": "fanout", + "Trials": 3, + "Inputs": 15202, + "Deliveries": 60808, + "InputsPerSecond": 310.4255939183654, + "Minimum": 289.05956380838364, + "Maximum": 334.2443879188526, + "P50Milliseconds": 3178.495, + "P99Milliseconds": 4063.231, + "AllocatedBytesPerInput": 491947.8025276461, + "MinimumAllocatedBytesPerInput": 481357.40448962303, + "MaximumAllocatedBytesPerInput": 547260.3692762186, + "CpuMillisecondsPerInput": 2.608151066350711, + "GcPauseMillisecondsPerThousandInputs": 26.336887835703003, + "Gen0CollectionsPerMillionInputs": 18212.621770436257, + "PeakWorkingSetMiB": 258.828125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-16384", + "Variant": "masstransit", + "Workload": "queue", + "Trials": 3, + "Inputs": 87347, + "Deliveries": 87347, + "InputsPerSecond": 1904.008406462041, + "Minimum": 1871.7782751115549, + "Maximum": 1925.8600711545125, + "P50Milliseconds": 511.999, + "P99Milliseconds": 843.775, + "AllocatedBytesPerInput": 240330.51730139944, + "MinimumAllocatedBytesPerInput": 240326.68675153554, + "MaximumAllocatedBytesPerInput": 240360.0393513807, + "CpuMillisecondsPerInput": 0.7677793981402052, + "GcPauseMillisecondsPerThousandInputs": 7.8255648280073276, + "Gen0CollectionsPerMillionInputs": 6691.143670864359, + "PeakWorkingSetMiB": 226.67578125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "memory", + "Variant": "after", + "Workload": "fanout", + "Trials": 3, + "Inputs": 8067377, + "Deliveries": 32269508, + "InputsPerSecond": 181925.0272858953, + "Minimum": 171247.04550995142, + "Maximum": 184424.03344848237, + "P50Milliseconds": 0.663, + "P99Milliseconds": 7.039, + "AllocatedBytesPerInput": 24852.416328575888, + "MinimumAllocatedBytesPerInput": 24847.489039035674, + "MaximumAllocatedBytesPerInput": 25116.43887586455, + "CpuMillisecondsPerInput": 0.08415493433105703, + "GcPauseMillisecondsPerThousandInputs": 0.7248032626675985, + "Gen0CollectionsPerMillionInputs": 847.9741659084349, + "PeakWorkingSetMiB": 137.12890625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "memory", + "Variant": "after", + "Workload": "queue", + "Trials": 3, + "Inputs": 11387509, + "Deliveries": 11387509, + "InputsPerSecond": 247620.8427097453, + "Minimum": 247596.59397609625, + "Maximum": 263650.6528485089, + "P50Milliseconds": 4.031, + "P99Milliseconds": 5.631, + "AllocatedBytesPerInput": 10741.42624214889, + "MinimumAllocatedBytesPerInput": 10738.707088648474, + "MaximumAllocatedBytesPerInput": 10743.558423437724, + "CpuMillisecondsPerInput": 0.03017443010563423, + "GcPauseMillisecondsPerThousandInputs": 0.31099724751901264, + "Gen0CollectionsPerMillionInputs": 388.8799181872006, + "PeakWorkingSetMiB": 132.9140625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "memory", + "Variant": "before", + "Workload": "fanout", + "Trials": 3, + "Inputs": 7710009, + "Deliveries": 30840036, + "InputsPerSecond": 168913.61108260372, + "Minimum": 166035.9351719147, + "Maximum": 178853.82549526374, + "P50Milliseconds": 0.363, + "P99Milliseconds": 7.487, + "AllocatedBytesPerInput": 26588.03404908048, + "MinimumAllocatedBytesPerInput": 26379.838516352407, + "MaximumAllocatedBytesPerInput": 26626.785721825483, + "CpuMillisecondsPerInput": 0.0872089047323088, + "GcPauseMillisecondsPerThousandInputs": 0.7912150342408196, + "Gen0CollectionsPerMillionInputs": 934.7591118945588, + "PeakWorkingSetMiB": 138.390625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "memory", + "Variant": "before", + "Workload": "queue", + "Trials": 3, + "Inputs": 11837879, + "Deliveries": 11837879, + "InputsPerSecond": 264441.2696173366, + "Minimum": 257387.93837206915, + "Maximum": 267051.7461687244, + "P50Milliseconds": 3.871, + "P99Milliseconds": 5.503, + "AllocatedBytesPerInput": 12031.404347656839, + "MinimumAllocatedBytesPerInput": 12031.39731915147, + "MaximumAllocatedBytesPerInput": 12031.526329596689, + "CpuMillisecondsPerInput": 0.03449835857439017, + "GcPauseMillisecondsPerThousandInputs": 0.3395715463297126, + "Gen0CollectionsPerMillionInputs": 425.4531262944505, + "PeakWorkingSetMiB": 135.65234375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "redis-16384", + "Variant": "after", + "Workload": "fanout", + "Trials": 1, + "Inputs": 50714, + "Deliveries": 202856, + "InputsPerSecond": 3329.3807577627013, + "Minimum": 3329.3807577627013, + "Maximum": 3329.3807577627013, + "P50Milliseconds": 282.623, + "P99Milliseconds": 360.447, + "AllocatedBytesPerInput": 274429.0474425208, + "MinimumAllocatedBytesPerInput": 274429.0474425208, + "MaximumAllocatedBytesPerInput": 274429.0474425208, + "CpuMillisecondsPerInput": 0.5277029222699846, + "GcPauseMillisecondsPerThousandInputs": 4.298024214220925, + "Gen0CollectionsPerMillionInputs": 9385.968371652798, + "PeakWorkingSetMiB": 161.78125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "redis-16384", + "Variant": "after", + "Workload": "queue", + "Trials": 1, + "Inputs": 122078, + "Deliveries": 122078, + "InputsPerSecond": 8083.735736654274, + "Minimum": 8083.735736654274, + "Maximum": 8083.735736654274, + "P50Milliseconds": 121.855, + "P99Milliseconds": 169.983, + "AllocatedBytesPerInput": 59836.87498156916, + "MinimumAllocatedBytesPerInput": 59836.87498156916, + "MaximumAllocatedBytesPerInput": 59836.87498156916, + "CpuMillisecondsPerInput": 0.1819287422795262, + "GcPauseMillisecondsPerThousandInputs": 2.0781385671455954, + "Gen0CollectionsPerMillionInputs": 2973.50874031357, + "PeakWorkingSetMiB": 144.59375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "redis-16384", + "Variant": "before", + "Workload": "fanout", + "Trials": 1, + "Inputs": 50771, + "Deliveries": 203084, + "InputsPerSecond": 3351.9633552123914, + "Minimum": 3351.9633552123914, + "Maximum": 3351.9633552123914, + "P50Milliseconds": 212.991, + "P99Milliseconds": 364.543, + "AllocatedBytesPerInput": 262058.08244864194, + "MinimumAllocatedBytesPerInput": 262058.08244864194, + "MaximumAllocatedBytesPerInput": 262058.08244864194, + "CpuMillisecondsPerInput": 0.5518220046877155, + "GcPauseMillisecondsPerThousandInputs": 4.386775915384767, + "Gen0CollectionsPerMillionInputs": 10537.511571566445, + "PeakWorkingSetMiB": 147.39453125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "redis-16384", + "Variant": "before", + "Workload": "queue", + "Trials": 1, + "Inputs": 116201, + "Deliveries": 116201, + "InputsPerSecond": 7701.980053992006, + "Minimum": 7701.980053992006, + "Maximum": 7701.980053992006, + "P50Milliseconds": 124.927, + "P99Milliseconds": 174.079, + "AllocatedBytesPerInput": 147438.0582611165, + "MinimumAllocatedBytesPerInput": 147438.0582611165, + "MaximumAllocatedBytesPerInput": 147438.0582611165, + "CpuMillisecondsPerInput": 0.21424124577241158, + "GcPauseMillisecondsPerThousandInputs": 2.1162726654675947, + "Gen0CollectionsPerMillionInputs": 4173.802290858082, + "PeakWorkingSetMiB": 133.53125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-soak", + "Variant": "after", + "Workload": "fanout", + "Trials": 1, + "Inputs": 41704, + "Deliveries": 166816, + "InputsPerSecond": 344.65979946396345, + "Minimum": 344.65979946396345, + "Maximum": 344.65979946396345, + "P50Milliseconds": 2654.207, + "P99Milliseconds": 3604.479, + "AllocatedBytesPerInput": 653892.4803376175, + "MinimumAllocatedBytesPerInput": 653892.4803376175, + "MaximumAllocatedBytesPerInput": 653892.4803376175, + "CpuMillisecondsPerInput": 1.4193026568194897, + "GcPauseMillisecondsPerThousandInputs": 22.103659121427203, + "Gen0CollectionsPerMillionInputs": 17720.12276999808, + "PeakWorkingSetMiB": 228.78125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-soak", + "Variant": "after", + "Workload": "queue", + "Trials": 1, + "Inputs": 257140, + "Deliveries": 257140, + "InputsPerSecond": 2138.198766125768, + "Minimum": 2138.198766125768, + "Maximum": 2138.198766125768, + "P50Milliseconds": 454.655, + "P99Milliseconds": 827.391, + "AllocatedBytesPerInput": 201234.541868243, + "MinimumAllocatedBytesPerInput": 201234.541868243, + "MaximumAllocatedBytesPerInput": 201234.541868243, + "CpuMillisecondsPerInput": 0.43069657385082055, + "GcPauseMillisecondsPerThousandInputs": 9.442163801820021, + "Gen0CollectionsPerMillionInputs": 6968.966321848021, + "PeakWorkingSetMiB": 219.7109375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-soak", + "Variant": "masstransit", + "Workload": "fanout", + "Trials": 1, + "Inputs": 38321, + "Deliveries": 153284, + "InputsPerSecond": 316.3897526473897, + "Minimum": 316.3897526473897, + "Maximum": 316.3897526473897, + "P50Milliseconds": 3047.423, + "P99Milliseconds": 4259.839, + "AllocatedBytesPerInput": 728936.541948279, + "MinimumAllocatedBytesPerInput": 728936.541948279, + "MaximumAllocatedBytesPerInput": 728936.541948279, + "CpuMillisecondsPerInput": 2.138235327888103, + "GcPauseMillisecondsPerThousandInputs": 31.04170037316354, + "Gen0CollectionsPerMillionInputs": 18501.604864173692, + "PeakWorkingSetMiB": 268.640625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-soak", + "Variant": "masstransit", + "Workload": "queue", + "Trials": 1, + "Inputs": 235670, + "Deliveries": 235670, + "InputsPerSecond": 1959.0592966840964, + "Minimum": 1959.0592966840964, + "Maximum": 1959.0592966840964, + "P50Milliseconds": 507.903, + "P99Milliseconds": 835.583, + "AllocatedBytesPerInput": 240295.1237917427, + "MinimumAllocatedBytesPerInput": 240295.1237917427, + "MaximumAllocatedBytesPerInput": 240295.1237917427, + "CpuMillisecondsPerInput": 0.5924802520473543, + "GcPauseMillisecondsPerThousandInputs": 11.436928756311792, + "Gen0CollectionsPerMillionInputs": 6882.5051979462805, + "PeakWorkingSetMiB": 239.40234375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "memory-repeat", + "Variant": "after", + "Workload": "queue", + "Trials": 5, + "Inputs": 38917198, + "Deliveries": 38917198, + "InputsPerSecond": 264441.7055904249, + "Minimum": 248357.98754968753, + "Maximum": 267082.99610812764, + "P50Milliseconds": 3.839, + "P99Milliseconds": 5.311, + "AllocatedBytesPerInput": 10742.68599879125, + "MinimumAllocatedBytesPerInput": 10742.00058720632, + "MaximumAllocatedBytesPerInput": 10743.569808152519, + "CpuMillisecondsPerInput": 0.032986227289221064, + "GcPauseMillisecondsPerThousandInputs": 0.3032774086418513, + "Gen0CollectionsPerMillionInputs": 388.1037688464586, + "PeakWorkingSetMiB": 152.56640625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "memory-repeat", + "Variant": "before", + "Workload": "queue", + "Trials": 5, + "Inputs": 38024577, + "Deliveries": 38024577, + "InputsPerSecond": 246086.47509034444, + "Minimum": 241039.33851778696, + "Maximum": 272561.7946698594, + "P50Milliseconds": 4.063, + "P99Milliseconds": 5.311, + "AllocatedBytesPerInput": 12030.60639449439, + "MinimumAllocatedBytesPerInput": 12026.480290032765, + "MaximumAllocatedBytesPerInput": 12031.297666693934, + "CpuMillisecondsPerInput": 0.030982540083013356, + "GcPauseMillisecondsPerThousandInputs": 0.3234400019956328, + "Gen0CollectionsPerMillionInputs": 413.72606296788797, + "PeakWorkingSetMiB": 155.28125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "redis-repeat", + "Variant": "after", + "Workload": "fanout", + "Trials": 3, + "Inputs": 206895, + "Deliveries": 827580, + "InputsPerSecond": 3455.4586847078895, + "Minimum": 3302.7109599254304, + "Maximum": 3483.686525953544, + "P50Milliseconds": 245.759, + "P99Milliseconds": 454.655, + "AllocatedBytesPerInput": 248529.98246912175, + "MinimumAllocatedBytesPerInput": 236179.0834047905, + "MaximumAllocatedBytesPerInput": 274407.5151769772, + "CpuMillisecondsPerInput": 0.5131245784027897, + "GcPauseMillisecondsPerThousandInputs": 4.0043160121191335, + "Gen0CollectionsPerMillionInputs": 9189.390041730978, + "PeakWorkingSetMiB": 164.0078125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "redis-repeat", + "Variant": "before", + "Workload": "fanout", + "Trials": 3, + "Inputs": 199479, + "Deliveries": 797916, + "InputsPerSecond": 3283.0668881615175, + "Minimum": 3264.056661695995, + "Maximum": 3320.789585529518, + "P50Milliseconds": 266.239, + "P99Milliseconds": 356.351, + "AllocatedBytesPerInput": 287821.2279718612, + "MinimumAllocatedBytesPerInput": 243265.04897694936, + "MaximumAllocatedBytesPerInput": 293813.6036626038, + "CpuMillisecondsPerInput": 0.4982813878621677, + "GcPauseMillisecondsPerThousandInputs": 4.4717876417439815, + "Gen0CollectionsPerMillionInputs": 10611.848887271844, + "PeakWorkingSetMiB": 148.47265625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-final-1024", + "Variant": "after", + "Workload": "fanout", + "Trials": 1, + "Inputs": 7104, + "Deliveries": 28416, + "InputsPerSecond": 449.4921791872479, + "Minimum": 449.4921791872479, + "Maximum": 449.4921791872479, + "P50Milliseconds": 2097.151, + "P99Milliseconds": 3211.263, + "AllocatedBytesPerInput": 126738.88513513513, + "MinimumAllocatedBytesPerInput": 126738.88513513513, + "MaximumAllocatedBytesPerInput": 126738.88513513513, + "CpuMillisecondsPerInput": 1.0998486768018019, + "GcPauseMillisecondsPerThousandInputs": 11.73367117117117, + "Gen0CollectionsPerMillionInputs": 12246.621621621622, + "PeakWorkingSetMiB": 141.42578125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-final-1024", + "Variant": "after", + "Workload": "queue", + "Trials": 1, + "Inputs": 44521, + "Deliveries": 44521, + "InputsPerSecond": 2898.7633418901796, + "Minimum": 2898.7633418901796, + "Maximum": 2898.7633418901796, + "P50Milliseconds": 331.775, + "P99Milliseconds": 663.551, + "AllocatedBytesPerInput": 34473.18937130792, + "MinimumAllocatedBytesPerInput": 34473.18937130792, + "MaximumAllocatedBytesPerInput": 34473.18937130792, + "CpuMillisecondsPerInput": 0.2566580040879585, + "GcPauseMillisecondsPerThousandInputs": 1.7819231373958357, + "Gen0CollectionsPerMillionInputs": 1954.1340041778037, + "PeakWorkingSetMiB": 142.48046875, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-final-16384", + "Variant": "after", + "Workload": "fanout", + "Trials": 1, + "Inputs": 5751, + "Deliveries": 23004, + "InputsPerSecond": 358.4866321325999, + "Minimum": 358.4866321325999, + "Maximum": 358.4866321325999, + "P50Milliseconds": 2490.367, + "P99Milliseconds": 3964.927, + "AllocatedBytesPerInput": 607458.6847504781, + "MinimumAllocatedBytesPerInput": 607458.6847504781, + "MaximumAllocatedBytesPerInput": 607458.6847504781, + "CpuMillisecondsPerInput": 1.680133368109894, + "GcPauseMillisecondsPerThousandInputs": 19.90923317683881, + "Gen0CollectionsPerMillionInputs": 16171.100678142931, + "PeakWorkingSetMiB": 237.7890625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "aws-final-16384", + "Variant": "after", + "Workload": "queue", + "Trials": 1, + "Inputs": 31256, + "Deliveries": 31256, + "InputsPerSecond": 2045.2644987796532, + "Minimum": 2045.2644987796532, + "Maximum": 2045.2644987796532, + "P50Milliseconds": 458.751, + "P99Milliseconds": 876.543, + "AllocatedBytesPerInput": 223639.01356539543, + "MinimumAllocatedBytesPerInput": 223639.01356539543, + "MaximumAllocatedBytesPerInput": 223639.01356539543, + "CpuMillisecondsPerInput": 0.5686103468134118, + "GcPauseMillisecondsPerThousandInputs": 6.222997184540568, + "Gen0CollectionsPerMillionInputs": 6398.77143588431, + "PeakWorkingSetMiB": 215.49609375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + } +] \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/trace-manifest.json b/benchmarks/Messaging/baselines/2026-09-07-allocations/trace-manifest.json new file mode 100644 index 000000000..4f166ff7a --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/trace-manifest.json @@ -0,0 +1,62 @@ +[ + { + "Name": "before-1024", + "Samples": 10785, + "WeightedBytes": 1153217056, + "EventsLost": 0, + "SamplesWithoutStacks": 0, + "TraceSha256": "df2706d7c6f94d26cfb7fac7d7185bee9544ab443ffb3ffc428ea468d1784c16", + "WindowMilliseconds": [ + 12000, + 30000 + ] + }, + { + "Name": "before-16384", + "Samples": 38098, + "WeightedBytes": 4476364296, + "EventsLost": 0, + "SamplesWithoutStacks": 0, + "TraceSha256": "1fa5966a79bcc78feacba0ca2fb974bbeea6a90e59928c77df64461d1b0da728", + "WindowMilliseconds": [ + 12000, + 30000 + ] + }, + { + "Name": "masstransit-16384", + "Samples": 44678, + "WeightedBytes": 5233375600, + "EventsLost": 0, + "SamplesWithoutStacks": 0, + "TraceSha256": "ef785c7e8face7199a011bfafdab871979cb8778f7b6d4f0260745174ff18b54", + "WindowMilliseconds": [ + 12000, + 30000 + ] + }, + { + "Name": "after-1024", + "Samples": 11176, + "WeightedBytes": 1195244928, + "EventsLost": 0, + "SamplesWithoutStacks": 0, + "TraceSha256": "fb1f359f2be932f09fad6faf46fe6f69ccef8324a1d36d8d1a39f99f22fe2621", + "WindowMilliseconds": [ + 12000, + 30000 + ] + }, + { + "Name": "after-16384", + "Samples": 33820, + "WeightedBytes": 3962598120, + "EventsLost": 0, + "SamplesWithoutStacks": 0, + "TraceSha256": "8fc37458282bf17f988696a10d4dd4c767a73065750ed9e276c4a23f4b433af9", + "WindowMilliseconds": [ + 12000, + 30000 + ] + } +] \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/validate-final.py b/benchmarks/Messaging/baselines/2026-09-07-allocations/validate-final.py new file mode 100644 index 000000000..36e3ce20c --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/validate-final.py @@ -0,0 +1,47 @@ +import hashlib, json, pathlib +root = pathlib.Path('/tmp/foundatio-allocations') +expected = {'aws-1024': 18, 'aws-16384': 18, 'memory': 12, 'redis-16384': 4, 'aws-soak': 4, 'memory-repeat': 10, 'redis-repeat': 6, 'aws-final-1024': 2, 'aws-final-16384': 2} +runs = [] +for profile, count in expected.items(): + files = sorted((root / profile).glob('*/*.json')) + assert len(files) == count, (profile, len(files), count) + assert not list((root / profile).glob('*/*-failure.txt')), profile + for file in files: + result = json.loads(file.read_text()) + measurement = result['Measurement'] + options = result['Options'] + environment = result['Environment'] + assert result['Success'] and result['Error'] is None, file + assert not any(measurement[k] for k in ('Missing', 'Duplicates', 'Invalid', 'HitTrackingLimit')), file + assert measurement['Error'] is None and measurement['Inputs'] > 0, file + copies = options['Subscribers'] if options['Scenario'] == 'pubsub' else 1 + assert measurement['Deliveries'] == copies * measurement['Inputs'], file + assert options['MaxMessages'] == 20_000_000 and options['MaxOutstanding'] == 1024, file + if options['Transport'] == 'sqs': + assert environment['AwsMode'] == 'localstack' and environment['AwsRegion'] == 'us-east-1', file + runs.append((file, result)) +for key in ('Runtime', 'CoreClrSha256', 'MassTransit', 'SqsSdk', 'SnsSdk'): + assert len({r['Environment'][key] for _, r in runs}) == 1, key +assert len({r['ResourcePrefix'] for _, r in runs}) == len(runs) +manifest = json.loads((root / 'candidate-binaries.json').read_text()) +for name, digest in manifest.items(): + assert hashlib.sha256((root / '0b3dfdc8-binaries' / name).read_bytes()).hexdigest() == digest, name +final_manifest = json.loads((root / 'final-binaries.json').read_text()) +for name, digest in final_manifest.items(): + assert hashlib.sha256((root / 'e677cf9a-binaries' / name).read_bytes()).hexdigest() == digest, name +for name in ('before-1024', 'before-16384', 'masstransit-16384', 'after-1024', 'after-16384'): + trace = json.loads((root / (name + '.nettrace.allocations.json')).read_text()) + assert trace['EventsLost'] == 0 and trace['SamplesWithoutStacks'] == 0 and trace['Samples'] > 0, name + run = json.loads((root / (name + '.json')).read_text()) + assert run['Success'] and all(run['Measurement'][k] == 0 for k in ('Missing','Duplicates','Invalid')), name +result = { + 'Trials': len(runs), 'Profiles': expected, + 'Inputs': sum(r['Measurement']['Inputs'] for _, r in runs), + 'Deliveries': sum(r['Measurement']['Deliveries'] for _, r in runs), + 'Missing': 0, 'Duplicates': 0, 'Invalid': 0, 'WorkerFailures': 0, + 'UniquePrefixes': len(runs), 'CandidateBinaryFilesVerified': len(manifest), 'FinalBinaryFilesVerified': len(final_manifest), 'ValidDiagnosticCaptures': 5, + 'Runtime': runs[0][1]['Environment']['Runtime'], + 'CoreClrSha256': runs[0][1]['Environment']['CoreClrSha256'], +} +(root / 'final-validation.json').write_text(json.dumps(result, indent=2)) +print(json.dumps(result, indent=2)) diff --git a/benchmarks/Messaging/baselines/2026-09-07-allocations/validate.py b/benchmarks/Messaging/baselines/2026-09-07-allocations/validate.py new file mode 100644 index 000000000..d72471831 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-allocations/validate.py @@ -0,0 +1,39 @@ +import hashlib, json, pathlib +root = pathlib.Path('/tmp/foundatio-allocations') +expected = {'aws-1024': 18, 'aws-16384': 18, 'memory': 12, 'redis-16384': 4, 'aws-soak': 4} +runs = [] +for profile, count in expected.items(): + files = sorted((root / profile).glob('*/*.json')) + assert len(files) == count, (profile, len(files), count) + assert not list((root / profile).glob('*/*-failure.txt')), profile + for file in files: + result = json.loads(file.read_text()) + measurement = result['Measurement'] + options = result['Options'] + environment = result['Environment'] + assert result['Success'] and result['Error'] is None, file + assert not any(measurement[k] for k in ('Missing', 'Duplicates', 'Invalid', 'HitTrackingLimit')), file + assert measurement['Error'] is None and measurement['Inputs'] > 0, file + copies = options['Subscribers'] if options['Scenario'] == 'pubsub' else 1 + assert measurement['Deliveries'] == copies * measurement['Inputs'], file + assert options['MaxMessages'] == 20_000_000 and options['MaxOutstanding'] == 1024, file + if options['Transport'] == 'sqs': + assert environment['AwsMode'] == 'localstack' and environment['AwsRegion'] == 'us-east-1', file + runs.append((file, result)) +for key in ('Runtime', 'CoreClrSha256', 'MassTransit', 'SqsSdk', 'SnsSdk'): + assert len({r['Environment'][key] for _, r in runs}) == 1, key +assert len({r['ResourcePrefix'] for _, r in runs}) == len(runs) +manifest = json.loads((root / 'candidate-binaries.json').read_text()) +for name, digest in manifest.items(): + assert hashlib.sha256((root / '0b3dfdc8-binaries' / name).read_bytes()).hexdigest() == digest, name +result = { + 'Trials': len(runs), 'Profiles': expected, + 'Inputs': sum(r['Measurement']['Inputs'] for _, r in runs), + 'Deliveries': sum(r['Measurement']['Deliveries'] for _, r in runs), + 'Missing': 0, 'Duplicates': 0, 'Invalid': 0, 'WorkerFailures': 0, + 'UniquePrefixes': len(runs), 'CandidateBinaryFilesVerified': len(manifest), + 'Runtime': runs[0][1]['Environment']['Runtime'], + 'CoreClrSha256': runs[0][1]['Environment']['CoreClrSha256'], +} +(root / 'validation.json').write_text(json.dumps(result, indent=2)) +print(json.dumps(result, indent=2)) diff --git a/benchmarks/Messaging/baselines/2026-09-07-pipelines/README.md b/benchmarks/Messaging/baselines/2026-09-07-pipelines/README.md new file mode 100644 index 000000000..fac17a3fd --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-pipelines/README.md @@ -0,0 +1,12 @@ +# Confirmed messaging pipeline measurements — 2026-09-07 + +93 successful fresh-process trials: 82,408,651 measured inputs and 215,044,147 acknowledged deliveries. No missing, duplicate or invalid deliveries, tracking exhaustion, worker failure or cleanup failure occurred. The [report](../../PIPELINE_RESULTS.md) explains the comparisons and their limits. + +- `raw-results.tar.gz` contains every trial JSON/log, request counts, per-profile source patches, DLL SHA-256 manifests, exact invocation options and orchestration scripts. `profiles.json` identifies all 11 profiles; `summary.csv` and `summary.json` retain medians, ranges, latency, allocation and CPU data. +- Shipping code is `77c20ea354919fd25ae300e49c5de7f3ed8da598`. Preserved before binaries are `abce1c0e` from checkout `466e3987`. All final profile source patches are empty. Source did not change during measurement. +- All workers used the same Microsoft .NET 10.0.11 runtime; `runtime-provenance.json` records the exact CoreCLR hash. Before binaries predate the report fingerprint field, so their pinned host invocation is recorded in the per-profile options. Runtime build differences must not be attributed to library changes. +- The exact orchestration scripts reference the original isolated checkouts under `/tmp`; adjust those binary paths to re-run preserved revisions elsewhere. The supported portable entry point is [run.ps1](../../run.ps1), including `-DotnetPath`, with standard, extended and soak profiles. The [benchmark README](../../README.md) documents individual configurations and live AWS mode. Extra before/after comparisons used separate executable snapshots. +- Standard profiles use three repetitions and 10 seconds of publishing after up to 3 seconds warmup. Payload/batch checks use one 15-second trial; rates use one 20-second trial; round trips use three 10-second trials. Soaks use 120 seconds after 5 seconds warmup. Standard tracker capacity is 20 million inputs; every soak uses 100 million. A larger touched portion of the fixed tracker can affect RSS as throughput increases. +- Crash capture was enabled only for the two Redis soak workers, using child-process environment variables. The selected settings were verified through the worker environment. Neither worker crashed, no dump was created, and no host-wide settings remain enabled. + +The local handoff also retains the previous experiments, diagnostic trace, runtime-only exception reproducer, validation logs and matching binary snapshots. Original failed trials and dumps remain separate from these successful confirmed results. No AWS account was contacted. diff --git a/benchmarks/Messaging/baselines/2026-09-07-pipelines/broker-cleanup-verification.json b/benchmarks/Messaging/baselines/2026-09-07-pipelines/broker-cleanup-verification.json new file mode 100644 index 000000000..2c6eeb843 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-pipelines/broker-cleanup-verification.json @@ -0,0 +1,7 @@ +{ + "benchmark_sqs_queues": 0, + "benchmark_sns_topics": 0, + "benchmark_redis_keys": 0, + "remaining_conformance_queues": 84, + "remaining_conformance_topics": 6 +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-pipelines/brokers.json b/benchmarks/Messaging/baselines/2026-09-07-pipelines/brokers.json new file mode 100644 index 000000000..2f02fd2ab --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-pipelines/brokers.json @@ -0,0 +1,18 @@ +[ + { + "name": "foundatio-messaging-perf-localstack-1", + "image_id": "sha256:b279c01f4cfb8f985a482e4014cabc1e2697b9d7a6c8c8db2e40f4d9f93687c7", + "image": "localstack/localstack:3.8.1", + "nano_cpus": 4000000000, + "memory_limit_bytes": 3221225472, + "started_at": "2026-09-07T04:59:25.251690302Z" + }, + { + "name": "foundatio-messaging-perf-redis-1", + "image_id": "sha256:2cc044fc5a07c9b701f8f1255a309ae9ad7856e694ac03513bf3648c01e40763", + "image": "redis:8.6-alpine", + "nano_cpus": 4000000000, + "memory_limit_bytes": 2147483648, + "started_at": "2026-09-07T04:59:25.355597207Z" + } +] \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-pipelines/manifest.json b/benchmarks/Messaging/baselines/2026-09-07-pipelines/manifest.json new file mode 100644 index 000000000..34859d9a9 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-pipelines/manifest.json @@ -0,0 +1,14 @@ +{ + "README.md": "d7aea153f4fe983ed104942aa6d01d561c18aceabd8074a3e1b0f6a81576b329", + "broker-cleanup-verification.json": "6443c75b4a21399ef19abc9bbebd84f64caab55486fad46f8b828569d4171710", + "brokers.json": "b57d9f19fc3a4bf677c740d1b53716ade9ee575dcb08f9c6dd914c5f7201b131", + "official-runtime-manifest.json": "f660f3f238f52a99e47ec4c8d3394477db0fb5e216b47d183fd13f123e9921b7", + "pr-state.json": "b936f729be0870caf2c418b9cf5b35caea7f74d75e4cdf3424ef3cc11401b438", + "profiles.json": "89dde60f4546310c920b0153dc4fadff806d6dc47457c9967cf5a86bc9113911", + "raw-results.tar.gz": "6c376bf21fcec7bd09c00f0a41b589bd35bdafdca66d17f858194d365cb3e9e3", + "redis-crash-capture-verification.json": "0243483ad7a9a2347719e4c4468e849f6ffa61c375aceb2ced363a0b0ad9575c", + "runtime-provenance.json": "a8164b3e1f33045dcd089821888e291d380c1da79c29f149f2127576fdf186c5", + "summary.csv": "be9c24da7dd1a4136691cb84dbc470e9de142e5d62a34016fe81cf2e56461c9f", + "summary.json": "88a8f438a8e115befd11888f12d4d9e5e649bd321b48f90dbe27c56768655a7b", + "validation.json": "b309c0ddde9e53b21afafd71afc7b091a1d9d17c2cf5a950c213d9d49a9815af" +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-pipelines/official-runtime-manifest.json b/benchmarks/Messaging/baselines/2026-09-07-pipelines/official-runtime-manifest.json new file mode 100644 index 000000000..fd597c30a --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-pipelines/official-runtime-manifest.json @@ -0,0 +1,7 @@ +{ + "metadata_url": "https://builds.dotnet.microsoft.com/dotnet/release-metadata/10.0/releases.json", + "name": "aspnetcore-runtime-linux-x64.tar.gz", + "rid": "linux-x64", + "url": "https://builds.dotnet.microsoft.com/dotnet/aspnetcore/Runtime/10.0.11/aspnetcore-runtime-10.0.11-linux-x64.tar.gz", + "hash": "4c6be0623330074e699dab8084be15a1baebb7a518c0dd8ce99f93cf79777cd46f3a38ef9d25edc152ed606f084b63736bd9e4082eb32d188fc357bf6ac4d1d6" +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-pipelines/pr-state.json b/benchmarks/Messaging/baselines/2026-09-07-pipelines/pr-state.json new file mode 100644 index 000000000..b077dadfd --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-pipelines/pr-state.json @@ -0,0 +1 @@ +{"baseRefName":"main","headRefName":"feat/messaging-jobs","headRefOid":"ce834f9e241dbeecc0168fb6a73908cd7986a915","isDraft":true,"state":"OPEN","url":"https://github.com/FoundatioFx/Foundatio/pull/533"} diff --git a/benchmarks/Messaging/baselines/2026-09-07-pipelines/profiles.json b/benchmarks/Messaging/baselines/2026-09-07-pipelines/profiles.json new file mode 100644 index 000000000..96d57b01a --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-pipelines/profiles.json @@ -0,0 +1,161 @@ +[ + [ + "confirmed-memory", + [ + "--transport", + "memory", + "--variants", + "before,after,masstransit", + "--workloads", + "serial,queue,fanout" + ] + ], + [ + "confirmed-aws", + [ + "--variants", + "after,masstransit", + "--workloads", + "serial,queue,pubsub-one,fanout" + ] + ], + [ + "confirmed-redis", + [ + "--transport", + "redis", + "--variants", + "before,after", + "--workloads", + "queue,fanout" + ] + ], + [ + "confirmed-16k", + [ + "--variants", + "after,masstransit", + "--workloads", + "queue,fanout", + "--payload", + "16384", + "--repetitions", + "1", + "--seconds", + "15" + ] + ], + [ + "confirmed-batch10", + [ + "--variants", + "after,masstransit", + "--workloads", + "queue,fanout", + "--batch", + "10", + "--repetitions", + "1", + "--seconds", + "15" + ] + ], + [ + "confirmed-rate10", + [ + "--variants", + "after,masstransit", + "--workloads", + "queue,fanout", + "--rate", + "10", + "--repetitions", + "1", + "--seconds", + "20" + ] + ], + [ + "confirmed-rate100", + [ + "--variants", + "after,masstransit", + "--workloads", + "queue,fanout", + "--rate", + "100", + "--repetitions", + "1", + "--seconds", + "20" + ] + ], + [ + "confirmed-roundtrip", + [ + "--variants", + "after,masstransit", + "--workloads", + "serial", + "--window", + "1", + "--repetitions", + "3" + ] + ], + [ + "confirmed-soak-aws", + [ + "--variants", + "after,masstransit", + "--workloads", + "queue,fanout", + "--repetitions", + "1", + "--seconds", + "120", + "--warmup", + "5", + "--max-messages", + "100000000" + ] + ], + [ + "confirmed-soak-memory", + [ + "--transport", + "memory", + "--variants", + "after,masstransit", + "--workloads", + "fanout", + "--repetitions", + "1", + "--seconds", + "120", + "--warmup", + "5", + "--max-messages", + "100000000" + ] + ], + [ + "confirmed-soak-redis", + [ + "--transport", + "redis", + "--variants", + "after", + "--workloads", + "queue,fanout", + "--repetitions", + "1", + "--seconds", + "120", + "--warmup", + "5", + "--max-messages", + "100000000" + ] + ] +] \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-pipelines/raw-results.tar.gz b/benchmarks/Messaging/baselines/2026-09-07-pipelines/raw-results.tar.gz new file mode 100644 index 000000000..fc9a9fbb5 Binary files /dev/null and b/benchmarks/Messaging/baselines/2026-09-07-pipelines/raw-results.tar.gz differ diff --git a/benchmarks/Messaging/baselines/2026-09-07-pipelines/redis-crash-capture-verification.json b/benchmarks/Messaging/baselines/2026-09-07-pipelines/redis-crash-capture-verification.json new file mode 100644 index 000000000..95547d888 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-pipelines/redis-crash-capture-verification.json @@ -0,0 +1,11 @@ +[ + { + "pid": 460472, + "crash_capture": { + "DOTNET_DbgEnableMiniDump": "1", + "DOTNET_DbgMiniDumpType": "4", + "DOTNET_DbgMiniDumpName": "/tmp/foundatio-fastest/confirmed-crashes/%e_%p_%t.dmp", + "DOTNET_EnableCrashReport": "1" + } + } +] \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-pipelines/runtime-provenance.json b/benchmarks/Messaging/baselines/2026-09-07-pipelines/runtime-provenance.json new file mode 100644 index 000000000..ebd455317 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-pipelines/runtime-provenance.json @@ -0,0 +1,15 @@ +{ + "ubuntu": { + "host": "/usr/lib/dotnet/dotnet", + "coreclr_sha256": "31209d59cfda3f45af0a8da5b5312e588f42b1e3ee47393fce1eeebcf5b1eb1b", + "dependencies": "\tlinux-vdso.so.1 (0x00007d4e461b1000)\n\tlibgcc_s.so.1 => /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007d4e4616a000)\n\tlibunwind-x86_64.so.8 => /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8 (0x00007d4e4614e000)\n\tlibunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8 (0x00007d4e46132000)\n\tlibstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007d4e45600000)\n\tlibm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x00007d4e458da000)\n\tlibc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007d4e45200000)\n\t/lib64/ld-linux-x86-64.so.2 (0x00007d4e461b3000)\n\tliblzma.so.5 => /usr/lib/x86_64-linux-gnu/liblzma.so.5 (0x00007d4e458a5000)\n", + "info": ".NET SDK:\n Version: 10.0.111\n Commit: e2f47b0110\n Workload version: 10.0.100-manifests.b0c14421\n MSBuild version: 18.0.11+e2f47b011\n\nRuntime Environment:\n OS Name: ubuntu\n OS Version: 26.04\n OS Platform: Linux\n RID: ubuntu.26.04-x64\n Base Path: /usr/lib/dotnet/sdk/10.0.111/\n\n.NET workloads installed:\nThere are no installed workloads to display.\nConfigured to use workload sets when installing new manifests.\nNo workload sets are installed. Run \"dotnet workload restore\" to install a workload set.\n\nHost:\n Version: 10.0.11\n Architecture: x64\n Commit: e2f47b0110\n\n.NET SDKs installed:\n 10.0.111 [/usr/lib/dotnet/sdk]\n\n.NET runtimes installed:\n Microsoft.AspNetCore.App 10.0.11 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]\n Microsoft.NETCore.App 10.0.11 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]\n\nOther architectures found:\n None\n\nEnvironment variables:\n DOTNET_BUNDLE_EXTRACT_BASE_DIR [/home/ejsmith/.cache/dotnet_bundle_extract]\n\nglobal.json file:\n /tmp/foundatio-pr-533-review/global.json\n\nLearn more:\n https://aka.ms/dotnet/info\n\nDownload .NET:\n https://aka.ms/dotnet/download\n" + }, + "microsoft": { + "host": "/tmp/foundatio-fastest/official-dotnet/dotnet", + "coreclr_sha256": "3ebe90cd92b1edf6742a41fa921a0c6326216fd1cca45fdb5e055bea33351bea", + "dependencies": "\tlinux-vdso.so.1 (0x0000719ced036000)\n\tlibgcc_s.so.1 => /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x0000719cec8cf000)\n\tlibpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x0000719cec8ca000)\n\tlibrt.so.1 => /usr/lib/x86_64-linux-gnu/librt.so.1 (0x0000719cec8c5000)\n\tlibdl.so.2 => /usr/lib/x86_64-linux-gnu/libdl.so.2 (0x0000719cec8c0000)\n\tlibstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x0000719cec600000)\n\tlibm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x0000719cec4da000)\n\tlibc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x0000719cec200000)\n\t/lib64/ld-linux-x86-64.so.2 (0x0000719ced038000)\n", + "info": "\nHost:\n Version: 10.0.11\n Architecture: x64\n Commit: e2f47b0110\n RID: linux-x64\n\n.NET SDKs installed:\n No SDKs were found.\n\n.NET runtimes installed:\n Microsoft.AspNetCore.App 10.0.11 [/tmp/foundatio-fastest/official-dotnet/shared/Microsoft.AspNetCore.App]\n Microsoft.NETCore.App 10.0.11 [/tmp/foundatio-fastest/official-dotnet/shared/Microsoft.NETCore.App]\n\nOther architectures found:\n None\n\nEnvironment variables:\n DOTNET_BUNDLE_EXTRACT_BASE_DIR [/home/ejsmith/.cache/dotnet_bundle_extract]\n\nglobal.json file:\n /tmp/foundatio-pr-533-review/global.json\n\nLearn more:\n https://aka.ms/dotnet/info\n\nDownload .NET:\n https://aka.ms/dotnet/download\n" + }, + "ubuntu_packages": "dotnet-runtime-10.0\t10.0.11-0ubuntu1~26.04.1\nlibunwind8:amd64\t1.8.3-0ubuntu1\n" +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-pipelines/summary.csv b/benchmarks/Messaging/baselines/2026-09-07-pipelines/summary.csv new file mode 100644 index 000000000..7e7b9815f --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-pipelines/summary.csv @@ -0,0 +1,48 @@ +Profile,Variant,Workload,Trials,Inputs,Deliveries,InputsPerSecond,Minimum,Maximum,P50Milliseconds,P99Milliseconds,AllocatedBytesPerInput,CpuMillisecondsPerInput,PeakWorkingSetMiB,Duplicates,Missing,Invalid +confirmed-16k,after,fanout,1,5056,20224,314.4806339889774,314.4806339889774,314.4806339889774,2981.887,4325.375,619396.9193037974,1.8608892405063293,221.265625,0,0,0 +confirmed-16k,after,queue,1,32830,32830,2148.255320256053,2148.255320256053,2148.255320256053,421.887,770.047,274384.99664940604,0.5590262869326835,208.2890625,0,0,0 +confirmed-16k,masstransit,fanout,1,4826,19304,295.8327101700457,295.8327101700457,295.8327101700457,3211.263,4456.447,407486.58765022794,2.604864069622876,241.796875,0,0,0 +confirmed-16k,masstransit,queue,1,28352,28352,1849.2203441266527,1849.2203441266527,1849.2203441266527,524.287,868.351,240258.855248307,0.7566038374717833,220.2890625,0,0,0 +confirmed-aws,after,fanout,3,13402,53608,424.78747397945455,389.060640832779,429.7114123366395,2162.687,2850.815,126978.85578788532,1.0730388014658332,128.0078125,0,0,0 +confirmed-aws,after,pubsub-one,3,47713,47713,1536.4335782617472,1513.544495421213,1620.429119146549,606.207,966.655,38781.14417029511,0.43924802194997453,144.91796875,0,0,0 +confirmed-aws,after,queue,3,92250,92250,3019.152545123461,2906.6572311643727,3118.9390777050735,319.487,638.975,24672.173354526876,0.3585275245726149,138.68359375,0,0,0 +confirmed-aws,after,serial,3,14635,14635,402.2200236870709,399.17385027770405,428.96018665069874,2195.455,2490.367,129701.49524989673,1.1616387856257744,101.01953125,0,0,0 +confirmed-aws,masstransit,fanout,3,13632,54528,422.97240079427917,401.84196422409076,423.9827137861561,2326.527,3506.175,86961.57038391224,1.8310210237659963,161.57421875,0,0,0 +confirmed-aws,masstransit,pubsub-one,3,37225,37225,1130.6517403145126,1119.2338327327157,1382.0814630057612,901.119,1196.031,62487.20164201288,0.7894246575342465,146.34375,0,0,0 +confirmed-aws,masstransit,queue,3,71771,71771,2424.9901029486814,2096.2527603458693,2495.526725237609,401.407,794.623,67514.01005025125,0.5641495367462311,148.0,0,0,0 +confirmed-aws,masstransit,serial,3,11048,11048,283.62988131917876,268.79885848410004,287.47489235696315,1884.159,3260.856,168693.18273841235,1.9343214386859247,113.0390625,0,0,0 +confirmed-batch10,after,fanout,1,7330,29320,467.2541591962841,467.2541591962841,467.2541591962841,1769.471,3309.567,128881.99727148704,0.9208259208731242,125.49609375,0,0,0 +confirmed-batch10,after,queue,1,45090,45090,2966.2466979289816,2966.2466979289816,2966.2466979289816,307.199,737.279,27480.32166777556,0.2211175648702595,137.9140625,0,0,0 +confirmed-batch10,masstransit,fanout,1,7110,28440,446.3810475873839,446.3810475873839,446.3810475873839,2195.455,2719.743,138536.25541490858,1.4440270042194092,165.0390625,0,0,0 +confirmed-batch10,masstransit,queue,1,38860,38860,2552.3980857198267,2552.3980857198267,2552.3980857198267,376.831,696.319,66210.22171899125,0.4035708440555842,155.5,0,0,0 +confirmed-memory,after,fanout,3,5157756,20631024,170397.46509374894,170039.57790365128,175040.46297867334,0.551,8.127,26493.232456093952,0.08497743183449777,131.2109375,0,0,0 +confirmed-memory,after,queue,3,7810961,7810961,262811.6484150234,249727.62941739798,268112.20923279104,3.839,5.631,12030.767348627598,0.03530383474888219,131.69921875,0,0,0 +confirmed-memory,after,serial,3,7032806,7032806,234753.26271917002,225187.55993520454,242960.89246838755,3.615,6.015,11905.153991699322,0.028147473266335248,119.76953125,0,0,0 +confirmed-memory,before,fanout,3,4740727,18962908,158447.26812616544,156308.1023005934,159009.07455512785,0.431,9.087,28249.629495338682,0.10822976017697418,126.078125,0,0,0 +confirmed-memory,before,queue,3,7786110,7786110,259228.84569434202,254179.30058237567,264735.4186429704,3.903,5.631,12474.196437677441,0.034813075799681084,120.515625,0,0,0 +confirmed-memory,before,serial,3,3657969,3657969,121746.24020873827,116956.03050139765,126733.46876783972,8.447,12.159,13116.40984246487,0.04802765485908773,106.0625,0,0,0 +confirmed-memory,masstransit,fanout,3,2278530,9114120,75644.79547201018,74486.21525360418,77418.22221841733,5.503,20.735,65947.70740912996,0.11562921615327339,182.82421875,0,0,0 +confirmed-memory,masstransit,queue,3,3102007,3102007,102870.28180892728,102651.92677883462,104354.30252161813,9.983,14.591,22653.27843572902,0.03558406270894628,134.98828125,0,0,0 +confirmed-memory,masstransit,serial,3,3824382,3824382,127380.33130101823,126267.28574651349,128387.21426791119,8.063,12.031,19673.21043380834,0.028443419160258533,126.0546875,0,0,0 +confirmed-rate10,after,fanout,1,201,804,10.039434248411116,10.039434248411116,10.039434248411116,22.271,27.903,278549.49253731343,11.930542288557215,116.83203125,0,0,0 +confirmed-rate10,after,queue,1,200,200,10.00003690013616,10.00003690013616,10.00003690013616,7.551,10.111,175360.04,7.844925,113.71484375,0,0,0 +confirmed-rate10,masstransit,fanout,1,200,800,9.999693109418471,9.999693109418471,9.999693109418471,25.087,31.999,573438.08,14.788255,132.97265625,0,0,0 +confirmed-rate10,masstransit,queue,1,200,200,9.999771555218821,9.999771555218821,9.999771555218821,9.087,11.135,194979.88,8.36992,122.83203125,0,0,0 +confirmed-rate100,after,fanout,1,2000,8000,99.7706890437157,99.7706890437157,99.7706890437157,65.023,425.983,323416.156,4.415156,116.609375,0,0,0 +confirmed-rate100,after,queue,1,2000,2000,99.99676810445486,99.99676810445486,99.99676810445486,3.647,7.359,167678.632,2.2320435,116.60546875,0,0,0 +confirmed-rate100,masstransit,fanout,1,2000,8000,99.79861089415309,99.79861089415309,99.79861089415309,56.319,458.751,282789.872,4.96201,141.0,0,0,0 +confirmed-rate100,masstransit,queue,1,2000,2000,99.99659261610661,99.99659261610661,99.99659261610661,6.655,9.471,170376.36,3.1843155000000003,115.47265625,0,0,0 +confirmed-redis,after,fanout,3,243648,974592,7979.158236932488,7915.096768377135,8289.86094089882,76.799,161.791,60874.34443283062,0.2865009901238281,177.56640625,0,0,0 +confirmed-redis,after,queue,3,638806,638806,21178.775730304696,21090.846540153503,21239.87365549532,47.103,88.063,22264.57425918896,0.11459486083578202,207.49609375,0,0,0 +confirmed-redis,before,fanout,3,236670,946680,7836.8147810919845,7650.187193701006,7988.842526405276,97.279,165.887,62634.56817843128,0.30791940291093334,152.8671875,0,0,0 +confirmed-redis,before,queue,3,630359,630359,20969.55752586236,20725.439250043764,21052.859379688878,47.103,81.919,22715.44155887309,0.124900928013671,223.33203125,0,0,0 +confirmed-roundtrip,after,serial,3,11864,11864,390.4746024404873,387.97727388638003,407.2624438165218,2.399,4.415,95228.93972179289,1.5126416687995903,104.1953125,0,0,0 +confirmed-roundtrip,masstransit,serial,3,6418,6418,212.65317574134116,208.35022535699989,220.3516218284094,4.479,6.463,163612.02358276644,2.4692263788968827,111.70703125,0,0,0 +confirmed-soak-aws,after,fanout,1,52517,210068,436.10403724219697,436.10403724219697,436.10403724219697,1802.239,2949.119,156528.23291505608,0.8140241826456195,131.4453125,0,0,0 +confirmed-soak-aws,after,queue,1,368559,368559,3066.465475056937,3066.465475056937,3066.465475056937,319.487,663.551,44757.56886685714,0.23091041868466106,137.58203125,0,0,0 +confirmed-soak-aws,masstransit,fanout,1,53328,213312,441.09820715597306,441.09820715597306,441.09820715597306,2260.991,2818.047,194515.33528352834,1.417448788628863,156.1796875,0,0,0 +confirmed-soak-aws,masstransit,queue,1,317017,317017,2636.69325388852,2636.69325388852,2636.69325388852,372.735,729.087,67423.33881148329,0.3692111968758773,154.6484375,0,0,0 +confirmed-soak-memory,after,fanout,1,21320538,85282152,177662.3709619909,177662.3709619909,177662.3709619909,0.483,6.783,26386.28629052419,0.08675890917949632,218.28125,0,0,0 +confirmed-soak-memory,masstransit,fanout,1,9081729,36326916,75673.88024277435,75673.88024277435,75673.88024277435,5.695,21.759,66131.43658878171,0.11949498449028816,223.89453125,0,0,0 +confirmed-soak-redis,after,fanout,1,990632,3962528,8250.72475957179,8250.72475957179,8250.72475957179,20.991,165.887,56797.854783612886,0.2909733927432185,226.8671875,0,0,0 +confirmed-soak-redis,after,queue,1,2585387,2585387,21536.643957602257,21536.643957602257,21536.643957602257,46.591,88.063,21630.563071602046,0.11350233137244056,124.0546875,0,0,0 diff --git a/benchmarks/Messaging/baselines/2026-09-07-pipelines/summary.json b/benchmarks/Messaging/baselines/2026-09-07-pipelines/summary.json new file mode 100644 index 000000000..38ec0c01f --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-pipelines/summary.json @@ -0,0 +1,895 @@ +[ + { + "Profile": "confirmed-16k", + "Variant": "after", + "Workload": "fanout", + "Trials": 1, + "Inputs": 5056, + "Deliveries": 20224, + "InputsPerSecond": 314.4806339889774, + "Minimum": 314.4806339889774, + "Maximum": 314.4806339889774, + "P50Milliseconds": 2981.887, + "P99Milliseconds": 4325.375, + "AllocatedBytesPerInput": 619396.9193037974, + "CpuMillisecondsPerInput": 1.8608892405063293, + "PeakWorkingSetMiB": 221.265625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-16k", + "Variant": "after", + "Workload": "queue", + "Trials": 1, + "Inputs": 32830, + "Deliveries": 32830, + "InputsPerSecond": 2148.255320256053, + "Minimum": 2148.255320256053, + "Maximum": 2148.255320256053, + "P50Milliseconds": 421.887, + "P99Milliseconds": 770.047, + "AllocatedBytesPerInput": 274384.99664940604, + "CpuMillisecondsPerInput": 0.5590262869326835, + "PeakWorkingSetMiB": 208.2890625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-16k", + "Variant": "masstransit", + "Workload": "fanout", + "Trials": 1, + "Inputs": 4826, + "Deliveries": 19304, + "InputsPerSecond": 295.8327101700457, + "Minimum": 295.8327101700457, + "Maximum": 295.8327101700457, + "P50Milliseconds": 3211.263, + "P99Milliseconds": 4456.447, + "AllocatedBytesPerInput": 407486.58765022794, + "CpuMillisecondsPerInput": 2.604864069622876, + "PeakWorkingSetMiB": 241.796875, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-16k", + "Variant": "masstransit", + "Workload": "queue", + "Trials": 1, + "Inputs": 28352, + "Deliveries": 28352, + "InputsPerSecond": 1849.2203441266527, + "Minimum": 1849.2203441266527, + "Maximum": 1849.2203441266527, + "P50Milliseconds": 524.287, + "P99Milliseconds": 868.351, + "AllocatedBytesPerInput": 240258.855248307, + "CpuMillisecondsPerInput": 0.7566038374717833, + "PeakWorkingSetMiB": 220.2890625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-aws", + "Variant": "after", + "Workload": "fanout", + "Trials": 3, + "Inputs": 13402, + "Deliveries": 53608, + "InputsPerSecond": 424.78747397945455, + "Minimum": 389.060640832779, + "Maximum": 429.7114123366395, + "P50Milliseconds": 2162.687, + "P99Milliseconds": 2850.815, + "AllocatedBytesPerInput": 126978.85578788532, + "CpuMillisecondsPerInput": 1.0730388014658332, + "PeakWorkingSetMiB": 128.0078125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-aws", + "Variant": "after", + "Workload": "pubsub-one", + "Trials": 3, + "Inputs": 47713, + "Deliveries": 47713, + "InputsPerSecond": 1536.4335782617472, + "Minimum": 1513.544495421213, + "Maximum": 1620.429119146549, + "P50Milliseconds": 606.207, + "P99Milliseconds": 966.655, + "AllocatedBytesPerInput": 38781.14417029511, + "CpuMillisecondsPerInput": 0.43924802194997453, + "PeakWorkingSetMiB": 144.91796875, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-aws", + "Variant": "after", + "Workload": "queue", + "Trials": 3, + "Inputs": 92250, + "Deliveries": 92250, + "InputsPerSecond": 3019.152545123461, + "Minimum": 2906.6572311643727, + "Maximum": 3118.9390777050735, + "P50Milliseconds": 319.487, + "P99Milliseconds": 638.975, + "AllocatedBytesPerInput": 24672.173354526876, + "CpuMillisecondsPerInput": 0.3585275245726149, + "PeakWorkingSetMiB": 138.68359375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-aws", + "Variant": "after", + "Workload": "serial", + "Trials": 3, + "Inputs": 14635, + "Deliveries": 14635, + "InputsPerSecond": 402.2200236870709, + "Minimum": 399.17385027770405, + "Maximum": 428.96018665069874, + "P50Milliseconds": 2195.455, + "P99Milliseconds": 2490.367, + "AllocatedBytesPerInput": 129701.49524989673, + "CpuMillisecondsPerInput": 1.1616387856257744, + "PeakWorkingSetMiB": 101.01953125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-aws", + "Variant": "masstransit", + "Workload": "fanout", + "Trials": 3, + "Inputs": 13632, + "Deliveries": 54528, + "InputsPerSecond": 422.97240079427917, + "Minimum": 401.84196422409076, + "Maximum": 423.9827137861561, + "P50Milliseconds": 2326.527, + "P99Milliseconds": 3506.175, + "AllocatedBytesPerInput": 86961.57038391224, + "CpuMillisecondsPerInput": 1.8310210237659963, + "PeakWorkingSetMiB": 161.57421875, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-aws", + "Variant": "masstransit", + "Workload": "pubsub-one", + "Trials": 3, + "Inputs": 37225, + "Deliveries": 37225, + "InputsPerSecond": 1130.6517403145126, + "Minimum": 1119.2338327327157, + "Maximum": 1382.0814630057612, + "P50Milliseconds": 901.119, + "P99Milliseconds": 1196.031, + "AllocatedBytesPerInput": 62487.20164201288, + "CpuMillisecondsPerInput": 0.7894246575342465, + "PeakWorkingSetMiB": 146.34375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-aws", + "Variant": "masstransit", + "Workload": "queue", + "Trials": 3, + "Inputs": 71771, + "Deliveries": 71771, + "InputsPerSecond": 2424.9901029486814, + "Minimum": 2096.2527603458693, + "Maximum": 2495.526725237609, + "P50Milliseconds": 401.407, + "P99Milliseconds": 794.623, + "AllocatedBytesPerInput": 67514.01005025125, + "CpuMillisecondsPerInput": 0.5641495367462311, + "PeakWorkingSetMiB": 148.0, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-aws", + "Variant": "masstransit", + "Workload": "serial", + "Trials": 3, + "Inputs": 11048, + "Deliveries": 11048, + "InputsPerSecond": 283.62988131917876, + "Minimum": 268.79885848410004, + "Maximum": 287.47489235696315, + "P50Milliseconds": 1884.159, + "P99Milliseconds": 3260.856, + "AllocatedBytesPerInput": 168693.18273841235, + "CpuMillisecondsPerInput": 1.9343214386859247, + "PeakWorkingSetMiB": 113.0390625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-batch10", + "Variant": "after", + "Workload": "fanout", + "Trials": 1, + "Inputs": 7330, + "Deliveries": 29320, + "InputsPerSecond": 467.2541591962841, + "Minimum": 467.2541591962841, + "Maximum": 467.2541591962841, + "P50Milliseconds": 1769.471, + "P99Milliseconds": 3309.567, + "AllocatedBytesPerInput": 128881.99727148704, + "CpuMillisecondsPerInput": 0.9208259208731242, + "PeakWorkingSetMiB": 125.49609375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-batch10", + "Variant": "after", + "Workload": "queue", + "Trials": 1, + "Inputs": 45090, + "Deliveries": 45090, + "InputsPerSecond": 2966.2466979289816, + "Minimum": 2966.2466979289816, + "Maximum": 2966.2466979289816, + "P50Milliseconds": 307.199, + "P99Milliseconds": 737.279, + "AllocatedBytesPerInput": 27480.32166777556, + "CpuMillisecondsPerInput": 0.2211175648702595, + "PeakWorkingSetMiB": 137.9140625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-batch10", + "Variant": "masstransit", + "Workload": "fanout", + "Trials": 1, + "Inputs": 7110, + "Deliveries": 28440, + "InputsPerSecond": 446.3810475873839, + "Minimum": 446.3810475873839, + "Maximum": 446.3810475873839, + "P50Milliseconds": 2195.455, + "P99Milliseconds": 2719.743, + "AllocatedBytesPerInput": 138536.25541490858, + "CpuMillisecondsPerInput": 1.4440270042194092, + "PeakWorkingSetMiB": 165.0390625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-batch10", + "Variant": "masstransit", + "Workload": "queue", + "Trials": 1, + "Inputs": 38860, + "Deliveries": 38860, + "InputsPerSecond": 2552.3980857198267, + "Minimum": 2552.3980857198267, + "Maximum": 2552.3980857198267, + "P50Milliseconds": 376.831, + "P99Milliseconds": 696.319, + "AllocatedBytesPerInput": 66210.22171899125, + "CpuMillisecondsPerInput": 0.4035708440555842, + "PeakWorkingSetMiB": 155.5, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-memory", + "Variant": "after", + "Workload": "fanout", + "Trials": 3, + "Inputs": 5157756, + "Deliveries": 20631024, + "InputsPerSecond": 170397.46509374894, + "Minimum": 170039.57790365128, + "Maximum": 175040.46297867334, + "P50Milliseconds": 0.551, + "P99Milliseconds": 8.127, + "AllocatedBytesPerInput": 26493.232456093952, + "CpuMillisecondsPerInput": 0.08497743183449777, + "PeakWorkingSetMiB": 131.2109375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-memory", + "Variant": "after", + "Workload": "queue", + "Trials": 3, + "Inputs": 7810961, + "Deliveries": 7810961, + "InputsPerSecond": 262811.6484150234, + "Minimum": 249727.62941739798, + "Maximum": 268112.20923279104, + "P50Milliseconds": 3.839, + "P99Milliseconds": 5.631, + "AllocatedBytesPerInput": 12030.767348627598, + "CpuMillisecondsPerInput": 0.03530383474888219, + "PeakWorkingSetMiB": 131.69921875, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-memory", + "Variant": "after", + "Workload": "serial", + "Trials": 3, + "Inputs": 7032806, + "Deliveries": 7032806, + "InputsPerSecond": 234753.26271917002, + "Minimum": 225187.55993520454, + "Maximum": 242960.89246838755, + "P50Milliseconds": 3.615, + "P99Milliseconds": 6.015, + "AllocatedBytesPerInput": 11905.153991699322, + "CpuMillisecondsPerInput": 0.028147473266335248, + "PeakWorkingSetMiB": 119.76953125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-memory", + "Variant": "before", + "Workload": "fanout", + "Trials": 3, + "Inputs": 4740727, + "Deliveries": 18962908, + "InputsPerSecond": 158447.26812616544, + "Minimum": 156308.1023005934, + "Maximum": 159009.07455512785, + "P50Milliseconds": 0.431, + "P99Milliseconds": 9.087, + "AllocatedBytesPerInput": 28249.629495338682, + "CpuMillisecondsPerInput": 0.10822976017697418, + "PeakWorkingSetMiB": 126.078125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-memory", + "Variant": "before", + "Workload": "queue", + "Trials": 3, + "Inputs": 7786110, + "Deliveries": 7786110, + "InputsPerSecond": 259228.84569434202, + "Minimum": 254179.30058237567, + "Maximum": 264735.4186429704, + "P50Milliseconds": 3.903, + "P99Milliseconds": 5.631, + "AllocatedBytesPerInput": 12474.196437677441, + "CpuMillisecondsPerInput": 0.034813075799681084, + "PeakWorkingSetMiB": 120.515625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-memory", + "Variant": "before", + "Workload": "serial", + "Trials": 3, + "Inputs": 3657969, + "Deliveries": 3657969, + "InputsPerSecond": 121746.24020873827, + "Minimum": 116956.03050139765, + "Maximum": 126733.46876783972, + "P50Milliseconds": 8.447, + "P99Milliseconds": 12.159, + "AllocatedBytesPerInput": 13116.40984246487, + "CpuMillisecondsPerInput": 0.04802765485908773, + "PeakWorkingSetMiB": 106.0625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-memory", + "Variant": "masstransit", + "Workload": "fanout", + "Trials": 3, + "Inputs": 2278530, + "Deliveries": 9114120, + "InputsPerSecond": 75644.79547201018, + "Minimum": 74486.21525360418, + "Maximum": 77418.22221841733, + "P50Milliseconds": 5.503, + "P99Milliseconds": 20.735, + "AllocatedBytesPerInput": 65947.70740912996, + "CpuMillisecondsPerInput": 0.11562921615327339, + "PeakWorkingSetMiB": 182.82421875, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-memory", + "Variant": "masstransit", + "Workload": "queue", + "Trials": 3, + "Inputs": 3102007, + "Deliveries": 3102007, + "InputsPerSecond": 102870.28180892728, + "Minimum": 102651.92677883462, + "Maximum": 104354.30252161813, + "P50Milliseconds": 9.983, + "P99Milliseconds": 14.591, + "AllocatedBytesPerInput": 22653.27843572902, + "CpuMillisecondsPerInput": 0.03558406270894628, + "PeakWorkingSetMiB": 134.98828125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-memory", + "Variant": "masstransit", + "Workload": "serial", + "Trials": 3, + "Inputs": 3824382, + "Deliveries": 3824382, + "InputsPerSecond": 127380.33130101823, + "Minimum": 126267.28574651349, + "Maximum": 128387.21426791119, + "P50Milliseconds": 8.063, + "P99Milliseconds": 12.031, + "AllocatedBytesPerInput": 19673.21043380834, + "CpuMillisecondsPerInput": 0.028443419160258533, + "PeakWorkingSetMiB": 126.0546875, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-rate10", + "Variant": "after", + "Workload": "fanout", + "Trials": 1, + "Inputs": 201, + "Deliveries": 804, + "InputsPerSecond": 10.039434248411116, + "Minimum": 10.039434248411116, + "Maximum": 10.039434248411116, + "P50Milliseconds": 22.271, + "P99Milliseconds": 27.903, + "AllocatedBytesPerInput": 278549.49253731343, + "CpuMillisecondsPerInput": 11.930542288557215, + "PeakWorkingSetMiB": 116.83203125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-rate10", + "Variant": "after", + "Workload": "queue", + "Trials": 1, + "Inputs": 200, + "Deliveries": 200, + "InputsPerSecond": 10.00003690013616, + "Minimum": 10.00003690013616, + "Maximum": 10.00003690013616, + "P50Milliseconds": 7.551, + "P99Milliseconds": 10.111, + "AllocatedBytesPerInput": 175360.04, + "CpuMillisecondsPerInput": 7.844925, + "PeakWorkingSetMiB": 113.71484375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-rate10", + "Variant": "masstransit", + "Workload": "fanout", + "Trials": 1, + "Inputs": 200, + "Deliveries": 800, + "InputsPerSecond": 9.999693109418471, + "Minimum": 9.999693109418471, + "Maximum": 9.999693109418471, + "P50Milliseconds": 25.087, + "P99Milliseconds": 31.999, + "AllocatedBytesPerInput": 573438.08, + "CpuMillisecondsPerInput": 14.788255, + "PeakWorkingSetMiB": 132.97265625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-rate10", + "Variant": "masstransit", + "Workload": "queue", + "Trials": 1, + "Inputs": 200, + "Deliveries": 200, + "InputsPerSecond": 9.999771555218821, + "Minimum": 9.999771555218821, + "Maximum": 9.999771555218821, + "P50Milliseconds": 9.087, + "P99Milliseconds": 11.135, + "AllocatedBytesPerInput": 194979.88, + "CpuMillisecondsPerInput": 8.36992, + "PeakWorkingSetMiB": 122.83203125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-rate100", + "Variant": "after", + "Workload": "fanout", + "Trials": 1, + "Inputs": 2000, + "Deliveries": 8000, + "InputsPerSecond": 99.7706890437157, + "Minimum": 99.7706890437157, + "Maximum": 99.7706890437157, + "P50Milliseconds": 65.023, + "P99Milliseconds": 425.983, + "AllocatedBytesPerInput": 323416.156, + "CpuMillisecondsPerInput": 4.415156, + "PeakWorkingSetMiB": 116.609375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-rate100", + "Variant": "after", + "Workload": "queue", + "Trials": 1, + "Inputs": 2000, + "Deliveries": 2000, + "InputsPerSecond": 99.99676810445486, + "Minimum": 99.99676810445486, + "Maximum": 99.99676810445486, + "P50Milliseconds": 3.647, + "P99Milliseconds": 7.359, + "AllocatedBytesPerInput": 167678.632, + "CpuMillisecondsPerInput": 2.2320435, + "PeakWorkingSetMiB": 116.60546875, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-rate100", + "Variant": "masstransit", + "Workload": "fanout", + "Trials": 1, + "Inputs": 2000, + "Deliveries": 8000, + "InputsPerSecond": 99.79861089415309, + "Minimum": 99.79861089415309, + "Maximum": 99.79861089415309, + "P50Milliseconds": 56.319, + "P99Milliseconds": 458.751, + "AllocatedBytesPerInput": 282789.872, + "CpuMillisecondsPerInput": 4.96201, + "PeakWorkingSetMiB": 141.0, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-rate100", + "Variant": "masstransit", + "Workload": "queue", + "Trials": 1, + "Inputs": 2000, + "Deliveries": 2000, + "InputsPerSecond": 99.99659261610661, + "Minimum": 99.99659261610661, + "Maximum": 99.99659261610661, + "P50Milliseconds": 6.655, + "P99Milliseconds": 9.471, + "AllocatedBytesPerInput": 170376.36, + "CpuMillisecondsPerInput": 3.1843155000000003, + "PeakWorkingSetMiB": 115.47265625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-redis", + "Variant": "after", + "Workload": "fanout", + "Trials": 3, + "Inputs": 243648, + "Deliveries": 974592, + "InputsPerSecond": 7979.158236932488, + "Minimum": 7915.096768377135, + "Maximum": 8289.86094089882, + "P50Milliseconds": 76.799, + "P99Milliseconds": 161.791, + "AllocatedBytesPerInput": 60874.34443283062, + "CpuMillisecondsPerInput": 0.2865009901238281, + "PeakWorkingSetMiB": 177.56640625, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-redis", + "Variant": "after", + "Workload": "queue", + "Trials": 3, + "Inputs": 638806, + "Deliveries": 638806, + "InputsPerSecond": 21178.775730304696, + "Minimum": 21090.846540153503, + "Maximum": 21239.87365549532, + "P50Milliseconds": 47.103, + "P99Milliseconds": 88.063, + "AllocatedBytesPerInput": 22264.57425918896, + "CpuMillisecondsPerInput": 0.11459486083578202, + "PeakWorkingSetMiB": 207.49609375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-redis", + "Variant": "before", + "Workload": "fanout", + "Trials": 3, + "Inputs": 236670, + "Deliveries": 946680, + "InputsPerSecond": 7836.8147810919845, + "Minimum": 7650.187193701006, + "Maximum": 7988.842526405276, + "P50Milliseconds": 97.279, + "P99Milliseconds": 165.887, + "AllocatedBytesPerInput": 62634.56817843128, + "CpuMillisecondsPerInput": 0.30791940291093334, + "PeakWorkingSetMiB": 152.8671875, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-redis", + "Variant": "before", + "Workload": "queue", + "Trials": 3, + "Inputs": 630359, + "Deliveries": 630359, + "InputsPerSecond": 20969.55752586236, + "Minimum": 20725.439250043764, + "Maximum": 21052.859379688878, + "P50Milliseconds": 47.103, + "P99Milliseconds": 81.919, + "AllocatedBytesPerInput": 22715.44155887309, + "CpuMillisecondsPerInput": 0.124900928013671, + "PeakWorkingSetMiB": 223.33203125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-roundtrip", + "Variant": "after", + "Workload": "serial", + "Trials": 3, + "Inputs": 11864, + "Deliveries": 11864, + "InputsPerSecond": 390.4746024404873, + "Minimum": 387.97727388638003, + "Maximum": 407.2624438165218, + "P50Milliseconds": 2.399, + "P99Milliseconds": 4.415, + "AllocatedBytesPerInput": 95228.93972179289, + "CpuMillisecondsPerInput": 1.5126416687995903, + "PeakWorkingSetMiB": 104.1953125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-roundtrip", + "Variant": "masstransit", + "Workload": "serial", + "Trials": 3, + "Inputs": 6418, + "Deliveries": 6418, + "InputsPerSecond": 212.65317574134116, + "Minimum": 208.35022535699989, + "Maximum": 220.3516218284094, + "P50Milliseconds": 4.479, + "P99Milliseconds": 6.463, + "AllocatedBytesPerInput": 163612.02358276644, + "CpuMillisecondsPerInput": 2.4692263788968827, + "PeakWorkingSetMiB": 111.70703125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-soak-aws", + "Variant": "after", + "Workload": "fanout", + "Trials": 1, + "Inputs": 52517, + "Deliveries": 210068, + "InputsPerSecond": 436.10403724219697, + "Minimum": 436.10403724219697, + "Maximum": 436.10403724219697, + "P50Milliseconds": 1802.239, + "P99Milliseconds": 2949.119, + "AllocatedBytesPerInput": 156528.23291505608, + "CpuMillisecondsPerInput": 0.8140241826456195, + "PeakWorkingSetMiB": 131.4453125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-soak-aws", + "Variant": "after", + "Workload": "queue", + "Trials": 1, + "Inputs": 368559, + "Deliveries": 368559, + "InputsPerSecond": 3066.465475056937, + "Minimum": 3066.465475056937, + "Maximum": 3066.465475056937, + "P50Milliseconds": 319.487, + "P99Milliseconds": 663.551, + "AllocatedBytesPerInput": 44757.56886685714, + "CpuMillisecondsPerInput": 0.23091041868466106, + "PeakWorkingSetMiB": 137.58203125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-soak-aws", + "Variant": "masstransit", + "Workload": "fanout", + "Trials": 1, + "Inputs": 53328, + "Deliveries": 213312, + "InputsPerSecond": 441.09820715597306, + "Minimum": 441.09820715597306, + "Maximum": 441.09820715597306, + "P50Milliseconds": 2260.991, + "P99Milliseconds": 2818.047, + "AllocatedBytesPerInput": 194515.33528352834, + "CpuMillisecondsPerInput": 1.417448788628863, + "PeakWorkingSetMiB": 156.1796875, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-soak-aws", + "Variant": "masstransit", + "Workload": "queue", + "Trials": 1, + "Inputs": 317017, + "Deliveries": 317017, + "InputsPerSecond": 2636.69325388852, + "Minimum": 2636.69325388852, + "Maximum": 2636.69325388852, + "P50Milliseconds": 372.735, + "P99Milliseconds": 729.087, + "AllocatedBytesPerInput": 67423.33881148329, + "CpuMillisecondsPerInput": 0.3692111968758773, + "PeakWorkingSetMiB": 154.6484375, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-soak-memory", + "Variant": "after", + "Workload": "fanout", + "Trials": 1, + "Inputs": 21320538, + "Deliveries": 85282152, + "InputsPerSecond": 177662.3709619909, + "Minimum": 177662.3709619909, + "Maximum": 177662.3709619909, + "P50Milliseconds": 0.483, + "P99Milliseconds": 6.783, + "AllocatedBytesPerInput": 26386.28629052419, + "CpuMillisecondsPerInput": 0.08675890917949632, + "PeakWorkingSetMiB": 218.28125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-soak-memory", + "Variant": "masstransit", + "Workload": "fanout", + "Trials": 1, + "Inputs": 9081729, + "Deliveries": 36326916, + "InputsPerSecond": 75673.88024277435, + "Minimum": 75673.88024277435, + "Maximum": 75673.88024277435, + "P50Milliseconds": 5.695, + "P99Milliseconds": 21.759, + "AllocatedBytesPerInput": 66131.43658878171, + "CpuMillisecondsPerInput": 0.11949498449028816, + "PeakWorkingSetMiB": 223.89453125, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-soak-redis", + "Variant": "after", + "Workload": "fanout", + "Trials": 1, + "Inputs": 990632, + "Deliveries": 3962528, + "InputsPerSecond": 8250.72475957179, + "Minimum": 8250.72475957179, + "Maximum": 8250.72475957179, + "P50Milliseconds": 20.991, + "P99Milliseconds": 165.887, + "AllocatedBytesPerInput": 56797.854783612886, + "CpuMillisecondsPerInput": 0.2909733927432185, + "PeakWorkingSetMiB": 226.8671875, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + }, + { + "Profile": "confirmed-soak-redis", + "Variant": "after", + "Workload": "queue", + "Trials": 1, + "Inputs": 2585387, + "Deliveries": 2585387, + "InputsPerSecond": 21536.643957602257, + "Minimum": 21536.643957602257, + "Maximum": 21536.643957602257, + "P50Milliseconds": 46.591, + "P99Milliseconds": 88.063, + "AllocatedBytesPerInput": 21630.563071602046, + "CpuMillisecondsPerInput": 0.11350233137244056, + "PeakWorkingSetMiB": 124.0546875, + "Duplicates": 0, + "Missing": 0, + "Invalid": 0 + } +] \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/2026-09-07-pipelines/validation.json b/benchmarks/Messaging/baselines/2026-09-07-pipelines/validation.json new file mode 100644 index 000000000..2bef9de56 --- /dev/null +++ b/benchmarks/Messaging/baselines/2026-09-07-pipelines/validation.json @@ -0,0 +1,12 @@ +{ + "trials": 93, + "inputs": 82408651, + "acknowledged_deliveries": 215044147, + "missing": 0, + "duplicates": 0, + "invalid": 0, + "worker_failures": 0, + "unique_resource_prefixes": 93, + "source_revision": "77c20ea354919fd25ae300e49c5de7f3ed8da598", + "coreclr_sha256": "3ebe90cd92b1edf6742a41fa921a0c6326216fd1cca45fdb5e055bea33351bea" +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/job-tracking-2026-09-08/final-redis-results.json b/benchmarks/Messaging/baselines/job-tracking-2026-09-08/final-redis-results.json new file mode 100644 index 000000000..7516437bc --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-2026-09-08/final-redis-results.json @@ -0,0 +1,488 @@ +[ + { + "Case": "pr149", + "Scenario": "redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Implementation": "pr149", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 33, + "Gen1Collections": 10, + "Gen2Collections": 2, + "ElapsedMilliseconds": 1216.6535, + "MessagesPerSecond": 8219.267030424027, + "AllocatedBytesPerMessage": 25866.1408, + "CpuMilliseconds": 5098.713, + "AcceptanceP50Milliseconds": 3.0561, + "AcceptanceP99Milliseconds": 9.4473, + "HandlerCompletionP50Milliseconds": 490.8495, + "HandlerCompletionP99Milliseconds": 676.58, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 64, + "Gen1Collections": 29, + "Gen2Collections": 4, + "ElapsedMilliseconds": 5745.974, + "MessagesPerSecond": 1740.3489817392142, + "AllocatedBytesPerMessage": 49721.2944, + "CpuMilliseconds": 7529.544, + "AcceptanceP50Milliseconds": 27.2063, + "AcceptanceP99Milliseconds": 37.5038, + "HandlerCompletionP50Milliseconds": 1172.465, + "HandlerCompletionP99Milliseconds": 1572.4491, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 62, + "Gen1Collections": 27, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1475.2214, + "MessagesPerSecond": 6778.6435310659135, + "AllocatedBytesPerMessage": 47371.1368, + "CpuMilliseconds": 6058.733, + "AcceptanceP50Milliseconds": 4.5676, + "AcceptanceP99Milliseconds": 11.4563, + "HandlerCompletionP50Milliseconds": 545.2493, + "HandlerCompletionP99Milliseconds": 699.3522, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 74, + "Gen1Collections": 31, + "Gen2Collections": 1, + "ElapsedMilliseconds": 3796.9387, + "MessagesPerSecond": 2633.7006704901505, + "AllocatedBytesPerMessage": 59444.776, + "CpuMilliseconds": 10471.403, + "AcceptanceP50Milliseconds": 10.2452, + "AcceptanceP99Milliseconds": 23.5571, + "HandlerCompletionP50Milliseconds": 1384.5851, + "HandlerCompletionP99Milliseconds": 1789.4974, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 112, + "Gen1Collections": 99, + "Gen2Collections": 1, + "ElapsedMilliseconds": 7246.4207, + "MessagesPerSecond": 1379.9916419426215, + "AllocatedBytesPerMessage": 90257.468, + "CpuMilliseconds": 13324.312, + "AcceptanceP50Milliseconds": 27.7499, + "AcceptanceP99Milliseconds": 51.6036, + "HandlerCompletionP50Milliseconds": 1662.0895, + "HandlerCompletionP99Milliseconds": 2755.7273, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 107, + "Gen1Collections": 96, + "Gen2Collections": 1, + "ElapsedMilliseconds": 4181.4689, + "MessagesPerSecond": 2391.5040956062116, + "AllocatedBytesPerMessage": 85468.0664, + "CpuMilliseconds": 12739.414, + "AcceptanceP50Milliseconds": 16.0432, + "AcceptanceP99Milliseconds": 31.7118, + "HandlerCompletionP50Milliseconds": 1054.1013, + "HandlerCompletionP99Milliseconds": 1290.9531, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 66, + "Gen1Collections": 31, + "Gen2Collections": 5, + "ElapsedMilliseconds": 6145.4364, + "MessagesPerSecond": 1627.2237395541185, + "AllocatedBytesPerMessage": 49756.136, + "CpuMilliseconds": 7992.949, + "AcceptanceP50Milliseconds": 28.7617, + "AcceptanceP99Milliseconds": 48.668, + "HandlerCompletionP50Milliseconds": 1318.1914, + "HandlerCompletionP99Milliseconds": 1740.5488, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 62, + "Gen1Collections": 26, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1625.3442, + "MessagesPerSecond": 6152.542950594711, + "AllocatedBytesPerMessage": 47363.9208, + "CpuMilliseconds": 6125.461, + "AcceptanceP50Milliseconds": 5.8696, + "AcceptanceP99Milliseconds": 11.3553, + "HandlerCompletionP50Milliseconds": 653.853, + "HandlerCompletionP99Milliseconds": 683.2333, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Implementation": "pr149", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 33, + "Gen1Collections": 11, + "Gen2Collections": 2, + "ElapsedMilliseconds": 1150.979, + "MessagesPerSecond": 8688.255823955085, + "AllocatedBytesPerMessage": 25863.6232, + "CpuMilliseconds": 3922.382, + "AcceptanceP50Milliseconds": 3.2872, + "AcceptanceP99Milliseconds": 10.2523, + "HandlerCompletionP50Milliseconds": 491.3682, + "HandlerCompletionP99Milliseconds": 592.3018, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 112, + "Gen1Collections": 99, + "Gen2Collections": 1, + "ElapsedMilliseconds": 7066.679, + "MessagesPerSecond": 1415.0918698868309, + "AllocatedBytesPerMessage": 90278.3272, + "CpuMilliseconds": 13278.909, + "AcceptanceP50Milliseconds": 27.6274, + "AcceptanceP99Milliseconds": 51.1597, + "HandlerCompletionP50Milliseconds": 1572.2414, + "HandlerCompletionP99Milliseconds": 2597.3671, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 107, + "Gen1Collections": 97, + "Gen2Collections": 1, + "ElapsedMilliseconds": 3850.7847, + "MessagesPerSecond": 2596.8733074066695, + "AllocatedBytesPerMessage": 85576.424, + "CpuMilliseconds": 13014.363, + "AcceptanceP50Milliseconds": 15.5397, + "AcceptanceP99Milliseconds": 28.1221, + "HandlerCompletionP50Milliseconds": 1127.7601, + "HandlerCompletionP99Milliseconds": 1363.1915, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 74, + "Gen1Collections": 29, + "Gen2Collections": 1, + "ElapsedMilliseconds": 3455.1203, + "MessagesPerSecond": 2894.2552304184605, + "AllocatedBytesPerMessage": 59422.3992, + "CpuMilliseconds": 10487.964, + "AcceptanceP50Milliseconds": 10.3238, + "AcceptanceP99Milliseconds": 19.4395, + "HandlerCompletionP50Milliseconds": 1413.6785, + "HandlerCompletionP99Milliseconds": 1757.9657, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 62, + "Gen1Collections": 26, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1664.1797, + "MessagesPerSecond": 6008.966459571643, + "AllocatedBytesPerMessage": 47365.7496, + "CpuMilliseconds": 6526.116, + "AcceptanceP50Milliseconds": 5.5596, + "AcceptanceP99Milliseconds": 11.9693, + "HandlerCompletionP50Milliseconds": 686.863, + "HandlerCompletionP99Milliseconds": 751.064, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Implementation": "pr149", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 33, + "Gen1Collections": 10, + "Gen2Collections": 2, + "ElapsedMilliseconds": 1138.1917, + "MessagesPerSecond": 8785.866212167952, + "AllocatedBytesPerMessage": 25825.3608, + "CpuMilliseconds": 4032.128, + "AcceptanceP50Milliseconds": 2.8405, + "AcceptanceP99Milliseconds": 9.3112, + "HandlerCompletionP50Milliseconds": 472.9912, + "HandlerCompletionP99Milliseconds": 624.8734, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 65, + "Gen1Collections": 28, + "Gen2Collections": 4, + "ElapsedMilliseconds": 5901.2576, + "MessagesPerSecond": 1694.5540557321206, + "AllocatedBytesPerMessage": 49699.8264, + "CpuMilliseconds": 7344.495, + "AcceptanceP50Milliseconds": 27.6611, + "AcceptanceP99Milliseconds": 46.991, + "HandlerCompletionP50Milliseconds": 1270.5838, + "HandlerCompletionP99Milliseconds": 1655.1731, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 107, + "Gen1Collections": 97, + "Gen2Collections": 1, + "ElapsedMilliseconds": 3737.85, + "MessagesPerSecond": 2675.3347512607515, + "AllocatedBytesPerMessage": 85534.5784, + "CpuMilliseconds": 12636.503, + "AcceptanceP50Milliseconds": 15.2135, + "AcceptanceP99Milliseconds": 26.469, + "HandlerCompletionP50Milliseconds": 1199.1307, + "HandlerCompletionP99Milliseconds": 1300.3625, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 74, + "Gen1Collections": 30, + "Gen2Collections": 1, + "ElapsedMilliseconds": 3818.086, + "MessagesPerSecond": 2619.1133463206434, + "AllocatedBytesPerMessage": 59482.2336, + "CpuMilliseconds": 10799.193, + "AcceptanceP50Milliseconds": 10.8117, + "AcceptanceP99Milliseconds": 22.0715, + "HandlerCompletionP50Milliseconds": 1622.0817, + "HandlerCompletionP99Milliseconds": 2024.366, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 112, + "Gen1Collections": 98, + "Gen2Collections": 1, + "ElapsedMilliseconds": 7456.6969, + "MessagesPerSecond": 1341.076368545971, + "AllocatedBytesPerMessage": 89776.2024, + "CpuMilliseconds": 13008.788, + "AcceptanceP50Milliseconds": 27.9852, + "AcceptanceP99Milliseconds": 50.4919, + "HandlerCompletionP50Milliseconds": 1795.4296, + "HandlerCompletionP99Milliseconds": 3013.6201, + "UniqueProcessed": 10000, + "Duplicates": 0 + } +] diff --git a/benchmarks/Messaging/baselines/job-tracking-2026-09-08/manifest.json b/benchmarks/Messaging/baselines/job-tracking-2026-09-08/manifest.json new file mode 100644 index 000000000..c84b70bcb --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-2026-09-08/manifest.json @@ -0,0 +1,31 @@ +{ + "runtime": "/usr/bin/dotnet (.NET 10.0.11, Ubuntu package)", + "baselineMediator": "1623285b131d08d4c9aa006b6e8042e3006e2c5e", + "baselineFoundatio": "9288e40bfc7fb41a27697132a550c36080189a01", + "pr149": "89bd6d1504b83d03de156b08f24fb578aa8bc97f", + "mediatorMain": "a1480132d701b93c64edcc9a776573c94d1a2153", + "foundatioSourceSha256": "71ca870537add27cedece374a244020a1000e1e7a545d4d598c258b7b16f49e9", + "mediatorSourceSha256": "d714bdd460723472b90204353d4a2a3f9687c0908768e40673245b4a558d2ea7", + "binaryHashes": { + "pr149": { + "Foundatio.Mediator.Abstractions.dll": "2e875f87d0730cae886d5f9ad1883e75cda1883621bbcf02d0d967ca82e40fd3", + "Foundatio.Mediator.Distributed.Aws.dll": "9b23123c4ced306c1c62e9c71362db161b72dc280517a1bd7637e4cd3e383d6a", + "Foundatio.Mediator.Distributed.Redis.dll": "0d69009172b2dfd9a9ceb009de3d3c837b66d669ffce2e00c26d9eacdda984f0", + "Foundatio.Mediator.Distributed.dll": "2083b1a1eec1498212a9011f6935a9801d6e058f18cff9b565e5207f85cbd665" + }, + "before": { + "Foundatio.Aws.dll": "046a5ed5937e8f5e323cbda2b5c9cc9446991d7d137125f4d2a0b4d8ef460d1a", + "Foundatio.Mediator.Abstractions.dll": "b2dfa05db70b7d45fcb94bfcf4c67a47c3f9f35d4aef06680a5bb6c6f12934be", + "Foundatio.Mediator.Distributed.dll": "043b11114f6a16f91bad2b77815b43901ff07eb167352dc6df88e119e0f5ce92", + "Foundatio.Redis.dll": "f8f536e7d9e3a9b9f6d695c800f22827f492b704aef8e4a897a3486f347d7063", + "Foundatio.dll": "24d71042553df3bceacda7a4319cebf56a785007af97065f10b229847e2465be" + }, + "after": { + "Foundatio.Aws.dll": "023b7aec1714a3283574795166d6d6a8f3bdf1b8ec89b9e80f36e5e18d2a8fc1", + "Foundatio.Mediator.Abstractions.dll": "b2dfa05db70b7d45fcb94bfcf4c67a47c3f9f35d4aef06680a5bb6c6f12934be", + "Foundatio.Mediator.Distributed.dll": "e441b70853253de8f685056f17f8f1991ee34b95b60eca19605ddcf4a4092964", + "Foundatio.Redis.dll": "a0b59934fc88ac412b0db075d7f8fc29207f313dd3b718331708e726071384f0", + "Foundatio.dll": "2a3fab2a7aa793b094e98b79fa5ebc025cf41cb736d0c29020bb9a935f17563f" + } + } +} diff --git a/benchmarks/Messaging/baselines/job-tracking-2026-09-08/tracked-validation-results.json b/benchmarks/Messaging/baselines/job-tracking-2026-09-08/tracked-validation-results.json new file mode 100644 index 000000000..143d12f55 --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-2026-09-08/tracked-validation-results.json @@ -0,0 +1,272 @@ +[ + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 120, + "Gen1Collections": 34, + "Gen2Collections": 6, + "ElapsedMilliseconds": 1505.0495, + "MessagesPerSecond": 33221.498694893424, + "AllocatedBytesPerMessage": 19202.46224, + "CpuMilliseconds": 6671.966, + "AcceptanceP50Milliseconds": 0.0253, + "AcceptanceP99Milliseconds": 4.6938, + "HandlerCompletionP50Milliseconds": 711.9154, + "HandlerCompletionP99Milliseconds": 914.0205, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 104, + "Gen1Collections": 32, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1578.1893, + "MessagesPerSecond": 31681.877452850553, + "AllocatedBytesPerMessage": 17052.48832, + "CpuMilliseconds": 5839.85, + "AcceptanceP50Milliseconds": 0.024, + "AcceptanceP99Milliseconds": 4.4849, + "HandlerCompletionP50Milliseconds": 784.5913, + "HandlerCompletionP99Milliseconds": 986.5172, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 103, + "Gen1Collections": 31, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1663.0108, + "MessagesPerSecond": 30065.950263221384, + "AllocatedBytesPerMessage": 17042.21024, + "CpuMilliseconds": 6040.904, + "AcceptanceP50Milliseconds": 0.0207, + "AcceptanceP99Milliseconds": 4.76, + "HandlerCompletionP50Milliseconds": 902.6452, + "HandlerCompletionP99Milliseconds": 1078.8896, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 118, + "Gen1Collections": 36, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1580.7946, + "MessagesPerSecond": 31629.66270254213, + "AllocatedBytesPerMessage": 19210.66928, + "CpuMilliseconds": 7368.639, + "AcceptanceP50Milliseconds": 0.0253, + "AcceptanceP99Milliseconds": 5.9448, + "HandlerCompletionP50Milliseconds": 764.2346, + "HandlerCompletionP99Milliseconds": 972.3887, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 118, + "Gen1Collections": 35, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1545.7268, + "MessagesPerSecond": 32347.242733968254, + "AllocatedBytesPerMessage": 19198.60224, + "CpuMilliseconds": 6769.954, + "AcceptanceP50Milliseconds": 0.026, + "AcceptanceP99Milliseconds": 4.4956, + "HandlerCompletionP50Milliseconds": 804.2085, + "HandlerCompletionP99Milliseconds": 915.7291, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 103, + "Gen1Collections": 32, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1516.1679, + "MessagesPerSecond": 32977.877977762226, + "AllocatedBytesPerMessage": 17044.5192, + "CpuMilliseconds": 6166.816, + "AcceptanceP50Milliseconds": 0.0226, + "AcceptanceP99Milliseconds": 4.931, + "HandlerCompletionP50Milliseconds": 780.7592, + "HandlerCompletionP99Milliseconds": 944.9643, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 3, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 104, + "Gen1Collections": 32, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1551.0846, + "MessagesPerSecond": 32235.50797938423, + "AllocatedBytesPerMessage": 17044.7736, + "CpuMilliseconds": 5620.296, + "AcceptanceP50Milliseconds": 0.0225, + "AcceptanceP99Milliseconds": 4.7065, + "HandlerCompletionP50Milliseconds": 784.9431, + "HandlerCompletionP99Milliseconds": 946.8039, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 3, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 119, + "Gen1Collections": 36, + "Gen2Collections": 5, + "ElapsedMilliseconds": 1580.9998, + "MessagesPerSecond": 31625.557447888354, + "AllocatedBytesPerMessage": 19202.71056, + "CpuMilliseconds": 7098.508, + "AcceptanceP50Milliseconds": 0.0265, + "AcceptanceP99Milliseconds": 4.7925, + "HandlerCompletionP50Milliseconds": 808.3326, + "HandlerCompletionP99Milliseconds": 976.2458, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 4, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 118, + "Gen1Collections": 35, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1576.5568, + "MessagesPerSecond": 31714.683543276078, + "AllocatedBytesPerMessage": 19210.41056, + "CpuMilliseconds": 6636.491, + "AcceptanceP50Milliseconds": 0.026, + "AcceptanceP99Milliseconds": 5.7201, + "HandlerCompletionP50Milliseconds": 819.1381, + "HandlerCompletionP99Milliseconds": 918.7369, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 4, + "Attempt": 0, + "Exit": 0, + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 103, + "Gen1Collections": 31, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1573.3851, + "MessagesPerSecond": 31778.615419708753, + "AllocatedBytesPerMessage": 17041.73792, + "CpuMilliseconds": 6025.803, + "AcceptanceP50Milliseconds": 0.0226, + "AcceptanceP99Milliseconds": 5.2514, + "HandlerCompletionP50Milliseconds": 834.6084, + "HandlerCompletionP99Milliseconds": 978.5715, + "UniqueProcessed": 50000, + "Duplicates": 0 + } +] diff --git a/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/foundatio-source-hashes.json b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/foundatio-source-hashes.json new file mode 100644 index 000000000..c1433b55b --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/foundatio-source-hashes.json @@ -0,0 +1,304 @@ +{ + "src/Directory.Build.props": "393429bb2217dd2585ee030871381156e7871ee816b87b6eb523b20c00532b11", + "src/Foundatio.Aws/AwsFoundatioBuilderExtensions.cs": "134185ab336f0a5e299a93a390b0331768feb0d6936c6feebbfa08f04ef3d983", + "src/Foundatio.Aws/AwsMessageTransport.Administration.cs": "79fef47f9ec6ad044d1d92ea2a8665a4c2a4d59789be94f4420ae8db4a09a68b", + "src/Foundatio.Aws/AwsMessageTransport.AutomaticBatching.cs": "c45f6ca6ce3b027cc2619830582f73521265629d77128871cdb36abf8239219c", + "src/Foundatio.Aws/AwsMessageTransport.Batching.cs": "bc4abae00a3334fe2ed281f171f176231e868bc88acab179bc29d14a5101ddff", + "src/Foundatio.Aws/AwsMessageTransport.NodeSubscriptions.cs": "83067c652f9f0acac346f516d526692c4190d51ee98899e55da2dc149a728b12", + "src/Foundatio.Aws/AwsMessageTransport.cs": "83b412dcefd71d4c6bcabde0df588dc031c0189ce3c3a6ec8e2b60fd210078b5", + "src/Foundatio.Aws/AwsMessageTransportOptions.cs": "ca938a0faa9f24d250775001c1c55ef6ce04cdaad2e4151cc1713175e7b76ed0", + "src/Foundatio.Aws/AwsRequestBatcher.cs": "982c19570ba14a6ac0dc5205048bf1177ae52abba3c4554cd396f190e99692a4", + "src/Foundatio.Aws/Foundatio.Aws.csproj": "5644fbc83967a77037f44c8b5a6fd6c93d28d1e1a463950a696a59b46e96be58", + "src/Foundatio.DataProtection/Extensions/DataProtectionBuilderExtensions.cs": "ac462429eeb9e0437546369ab2287bee1f11c6cad5e8f67fe0c3212150dd91eb", + "src/Foundatio.DataProtection/Foundatio.DataProtection.csproj": "85b8e5885066c0987d101b333c774eae059aa5364941dce0db9718934e5340cd", + "src/Foundatio.DataProtection/FoundatioStorageXmlRepository.cs": "5a4e6d86722061a1600fd84ba20a6a8559fcd0c03c2a90d3e27f1616972a0020", + "src/Foundatio.Extensions.Hosting/Foundatio.Extensions.Hosting.csproj": "ac972a941b4be956bff18c3d72e1259ca91d785778cc6ea0745a0d56561e7cb0", + "src/Foundatio.Extensions.Hosting/FoundatioRuntimeHealth.cs": "6fa670d6723ed33045c0847e5ca1ff55428dbad27acc1e2f567f40f057e48785", + "src/Foundatio.Extensions.Hosting/FoundatioWorkerExtensions.cs": "03f358bfb8fd0f6fcda74b32677889f27ab3c994621077fb58e36462c793f2fc", + "src/Foundatio.Extensions.Hosting/Jobs/JobHostExtensions.cs": "b6d457685d585767c3db93f31d700a95322bee72101943aa78f08a7c92866b4b", + "src/Foundatio.Extensions.Hosting/Jobs/JobSchedulerService.cs": "352d51f889480582a3256e89c6e2a8e21489d7e5084cce81f30f85784edae893", + "src/Foundatio.Extensions.Hosting/Jobs/JobWorkerService.cs": "10227d034734ccb6a74638db3b7600be13bebcb8429e1a549374cd42b90911cb", + "src/Foundatio.Extensions.Hosting/Messaging/MessageHandlerHostedService.cs": "5b7816f1f364b2662febf7c568eda86bc2b3a74b48e8cb40b91d1f7486477877", + "src/Foundatio.Extensions.Hosting/Messaging/MessagingHostExtensions.cs": "b2eb01eee08ba5d25aa8c83d83dfaabe9b30049731db4f479a932f97911217f3", + "src/Foundatio.Extensions.Hosting/Messaging/ScheduledMessageDispatcherService.cs": "fbc7ab3aa1910ceaf457f70fae49c20316d61242301f9a2013fbc82f3ea41c4f", + "src/Foundatio.Extensions.Hosting/Startup/IStartupAction.cs": "849520d3cbde530623d83ea8f2c71421de12f76a072e87962f195d1e545f8290", + "src/Foundatio.Extensions.Hosting/Startup/RunStartupActionsService.cs": "aa510cc2cb09000a8a0af077a5eda876b4b2f14f232c29fde3d557a1661d9dc5", + "src/Foundatio.Extensions.Hosting/Startup/StartupActionRegistration.cs": "92707b49ead7cd6c0c695789645a89c89f7a78d128b80d4fc07f432922950f5c", + "src/Foundatio.Extensions.Hosting/Startup/StartupActionsContext.cs": "9cba3e242c90b9b3f197561cf77cbdab97c4162aab9928e90b989e3ef6b42b94", + "src/Foundatio.Extensions.Hosting/Startup/StartupExtensions.cs": "369a17f9fc81ca3ff7a82568d059ddcbd8defeade518ae30c277dedb22ac9c70", + "src/Foundatio.Extensions.Hosting/Startup/StartupHealthcheck.cs": "80d4090b773beab00fd869c2db129cfc6b5cd84378d80f6746f0254c464cd0f4", + "src/Foundatio.Extensions.Hosting/Startup/StartupPriorityAttribute.cs": "d9edd3fd1f91fc132070bc1fdeb9b141fd448e996cc9e349d8f000de22053286", + "src/Foundatio.Extensions.Hosting/Startup/WaitForStartupActionsBeforeServingRequestsMiddleware.cs": "1f8734059ca10b47bfece5f42f11c836865f4e8c06ba92deaad2ccf2b01e517d", + "src/Foundatio.JsonNet/Foundatio.JsonNet.csproj": "b37f45f18a48514159fd12f459ae209fe5bc07f8322ede352512940419c5d8bf", + "src/Foundatio.JsonNet/JsonNetSerializer.cs": "9f42aa8739a91ca216ef7cc7157e4b54def2cb72b6461a179928712b0ba6cbe1", + "src/Foundatio.MessagePack/Foundatio.MessagePack.csproj": "22acb623d1322c186c6fc07d0725eca0c84bd3c83c97eb6cf0c7d28ab3a4a019", + "src/Foundatio.MessagePack/MessagePackSerializer.cs": "be77d533eeb154f2ffccdd84a1851238a30b86b605aa20d1fa7dc0507d2a265b", + "src/Foundatio.Redis/Foundatio.Redis.csproj": "37a9720d1fb11e15c60c67246549b5c985faf013a93ea5a9af07d2b3349fe83c", + "src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.Scripts.cs": "545ca3a8e14188518520ac84eb72b00c90714d3475d074a6a509725eb00bb23a", + "src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.Subscriptions.cs": "e36ac1d8e28a9bdf927e1017bf96a74528cf3e252118e06ff81b32354d4c275a", + "src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.cs": "2667b11e39f6470490901d74edd3976b17320872a2621c63bd3b4e0de3dbb5eb", + "src/Foundatio.Redis/Messaging/RedisStreamsMessageTransportOptions.cs": "4548f62d18314fe89c5face5f992d1d9440d29e5f0ff2c315c422c4ead0579e7", + "src/Foundatio.Redis/RedisFoundatioBuilderExtensions.cs": "00d3a0df25544cd470cf8f99c29c3d9855c8cae27fe515cf9bf8c37b220bb30c", + "src/Foundatio.Redis/RedisJobRuntimeStore.Broker.cs": "73710853d7de6e17eec586510304623e1e86c913f992bb8e44be995b1bb6e1a6", + "src/Foundatio.Redis/RedisJobRuntimeStore.Claims.cs": "13969aeb92b6ae5d16931ca4cd6fedc091d0c61a63be562ad8bb4fe0032cc33d", + "src/Foundatio.Redis/RedisJobRuntimeStore.Schedules.cs": "ebcb4d61f9618d6b1ceebdfea6cb56706fcc9901dee18087edea41a53daa308f", + "src/Foundatio.Redis/RedisJobRuntimeStore.cs": "4c53ab14c52f66704d1cbef28e63f5ce1e2502690c6fb8655381da1b3ddfa511", + "src/Foundatio.Redis/RedisJobRuntimeStoreOptions.cs": "bb37d246718ae73fb5d80fc71d57d58a14413555c4e5f0533f58bad1d8562f4e", + "src/Foundatio.Redis/RedisLockProvider.cs": "066b0728353d570b569a6faa43f93113d6a6e9b4a08869415882587e6756c0fe", + "src/Foundatio.TestHarness/Caching/CacheClientTestsBase.cs": "d0695bd4a5cf123eb054928582d531826a188516bdc2b343123c3f8ea57f256d", + "src/Foundatio.TestHarness/Caching/HybridCacheClientTestBase.cs": "565f1ca1cbdf4fb5f8daf3144a9548c2495322b00f85195a6e6b58360feb2997", + "src/Foundatio.TestHarness/Extensions/TaskExtensions.cs": "d562f2d889b57aeb078889024ae2bcaebd1ab3466aa62eb25a068bbd30bf5f8e", + "src/Foundatio.TestHarness/Foundatio.TestHarness.csproj": "42667d7e912c915ac7c50ae2ebb83854bc8d8d6cdd4b00d4c586bb534ee9d1c3", + "src/Foundatio.TestHarness/GlobalSuppressions.cs": "37856b955115ec6cbac99fc690ebb0be08ec54e87998872a8e304e4c70ee4e33", + "src/Foundatio.TestHarness/Jobs/JobRuntimeStoreConformanceTests.cs": "99e78b928b6f34591ba3c1ff7c226741a5c7ce46c2d34b4f30fa41f20d88fffa", + "src/Foundatio.TestHarness/Locks/LockTestBase.cs": "cb15bede4f2764aa1483cafb84421d2e0ebef1fdf99e1d0d7bd425138fab5c61", + "src/Foundatio.TestHarness/Messaging/MessageTransportConformanceTests.cs": "adced83525089b8801e1a1236a81609f69037a57328131854ca1e3f35bc538e2", + "src/Foundatio.TestHarness/Messaging/Samples.cs": "6d41aa369e26b65b2a7b7cde159c4c56ebf91e66d44ee3d0ef052171b9919b45", + "src/Foundatio.TestHarness/Serializer/FaultInjectingSerializer.cs": "3ffb6f803f87b426b59b437c9520b084ccfcffb6c63a55291fd31d08be0fb63e", + "src/Foundatio.TestHarness/Serializer/SerializerTestsBase.cs": "3a583207ffb0337ce48fce1596f702f1b88bde94766752438dca1c33778a0154", + "src/Foundatio.TestHarness/Storage/FileStorageTestsBase.cs": "6b2af316dff344c69b1ac9921882c7a1a865a92ed4fe4727b5c3d009d064552d", + "src/Foundatio.TestHarness/Utility/BenchmarkToJson.cs": "af707b43894b111697dcf62f19a7790711bfae726fcaa1e2066f39d54869c9db", + "src/Foundatio.TestHarness/Utility/Configuration.cs": "1cdf9b71eb31b284e3dc1738077019d352420e561b61499241dc7a3f18f91425", + "src/Foundatio.TestHarness/Utility/InMemoryMetrics.cs": "9230181187888b19618a4606392f99648efe81968eb22292495c04e969161f42", + "src/Foundatio.TestHarness/Utility/NonSeekableStream.cs": "bbfe000b68fda254f27475682bc054e4a14bc9a7e2ffa3900b868fdc2e77e57e", + "src/Foundatio.Testing/Foundatio.Testing.csproj": "c7dc9f69b68ebb5ea50dcf7f919d09992622735f59e1b714533836e358a09ae3", + "src/Foundatio.Testing/JobsTestHarness.cs": "4a6d357e923f52658174906b7e5812dd9f46a39137fd69a49c45f7bbf03d793e", + "src/Foundatio.Testing/MessagingTestHarness.cs": "09d3a08ac01e74a15c787d85b4923cf9a278cd8bfdf190bedd8f708748b8ba67", + "src/Foundatio.Testing/RecordingMessageTransport.cs": "11f8394867839ea6f2d121e92523898ea4a228abbf074f3ee54ac0efd14fa30d", + "src/Foundatio.Testing/TestingFoundatioBuilderExtensions.cs": "cc0bdced87d39d676a0526027e5d895a793e994658aa327baad2e24749cbc8b4", + "src/Foundatio.Utf8Json/Foundatio.Utf8Json.csproj": "e5ac77cc4363ed28f735d8b302f4773984201768049695503b9e69e942259f65", + "src/Foundatio.Utf8Json/Utf8JsonSerializer.cs": "768f2a3524211cb75378a499d12f980f753d443e1e274130155c79cf50208a90", + "src/Foundatio.Xunit.v3/Foundatio.Xunit.v3.csproj": "d76c4563545d6e3f4c8ecb3841e662851ff196b738ec044250716e8d48023e7c", + "src/Foundatio.Xunit.v3/Logging/LogEntry.cs": "592d7ba22c99ed9f2c00911421b7c1783d1fc271ebbc1e15856fe787a6c49dd1", + "src/Foundatio.Xunit.v3/Logging/LoggingExtensions.cs": "a55d8aaf4ca1961156e2901fd78d101eb966789c97f66fcd07760a472f7f5c80", + "src/Foundatio.Xunit.v3/Logging/TestLogger.cs": "0f7f36e0ae9c5db00305f294525b320d5364490299dba4a9cc6e7d91c8129fc3", + "src/Foundatio.Xunit.v3/Logging/TestLoggerBase.cs": "8076b2c546720b86799d3d4a6a71cce2923be278059b571cdf6b5817b4b2e174", + "src/Foundatio.Xunit.v3/Logging/TestLoggerFixture.cs": "b055f046ef8be5a420ecc819bed20a07c91ec700d9d93b6b4fb0276c2faf343c", + "src/Foundatio.Xunit.v3/Logging/TestLoggerLogger.cs": "f87d388efaa4cfb37ab6828bc965268c4ca863124e9168a5cb7926cd77801bbc", + "src/Foundatio.Xunit.v3/Logging/TestLoggerOptions.cs": "cac8feb1ab9910d52e29f5ed76623ec146926ff474714e8f276073f23cdd075c", + "src/Foundatio.Xunit.v3/Logging/TestLoggerProvider.cs": "ae462feeb8b51f4d1db200156dee87f5a61519e934a93adf89aee35c9b4e59d1", + "src/Foundatio.Xunit.v3/Logging/TestWithLoggingBase.cs": "8796cb1eaf1279324b549e64347161f694c123df670789f440d669669b3110e2", + "src/Foundatio.Xunit.v3/Retry/DelayedMessageBus.cs": "d101fed7c9590040d31783a60f82091e42494599e8d9aeb88b44b250258d1571", + "src/Foundatio.Xunit.v3/Retry/RetryFactAttribute.cs": "57d1cd13779ed513821dcd474a660205f2241d3c572f2e7e94f060562615fe0c", + "src/Foundatio.Xunit.v3/Retry/RetryFactDiscoverer.cs": "ecfa76e65d5eeeb040e46aa119d3edc6afdb80593abedd787e60a69135788919", + "src/Foundatio.Xunit.v3/Retry/RetryTestCase.cs": "4526d2ee21ab92bc7ce88e42c44634e5585a9835a4f580b121e0f7bf567ee794", + "src/Foundatio.Xunit.v3/Retry/RetryTheoryAttribute.cs": "b8a145721f78e13a133a451e319cb26b50e2402988ffe1514c1e12b7b0519153", + "src/Foundatio.Xunit.v3/Retry/RetryTheoryDiscoverer.cs": "872297bf73fae71ecf823236fdbd2c0a50db275c7950c6f3684d62feb754d96f", + "src/Foundatio.Xunit.v3/Retry/RetryTheoryTestCase.cs": "014cce5fac052176989a1acc82fff575ae0348f5d4dc6b9e69b70c13162e65cf", + "src/Foundatio.Xunit/Foundatio.Xunit.csproj": "e0cba22f94c82b4ce876253a6c05250af35055d23e025bd710ae87ef592ff899", + "src/Foundatio.Xunit/Logging/LogEntry.cs": "592d7ba22c99ed9f2c00911421b7c1783d1fc271ebbc1e15856fe787a6c49dd1", + "src/Foundatio.Xunit/Logging/LoggingExtensions.cs": "4a5c96fda6eb9913c604e6318e934f1b6a7cd3f52f352719075605c34f652fad", + "src/Foundatio.Xunit/Logging/TestLogger.cs": "028fffa8d7b5fa952aed02b44da8bb45345043041e0c73b05b9ac8c622474865", + "src/Foundatio.Xunit/Logging/TestLoggerBase.cs": "3520a1fe2b1f248bd2dd6561b0a6c59f0b60c4086543ba7ef9412044e1e146d2", + "src/Foundatio.Xunit/Logging/TestLoggerFixture.cs": "ef1fc901294280ae2bad64c2cfb2d813d0fab1aced2827ecd28e15ac70de0e76", + "src/Foundatio.Xunit/Logging/TestLoggerLogger.cs": "568fa03e8a1a5e12f396471a6763424fec88ec902d9bc9ea03c7c00b3c9d6ffa", + "src/Foundatio.Xunit/Logging/TestLoggerOptions.cs": "35c945b119474f9e552d221914193e23330e4e9eee45367ec693b34c899b77ec", + "src/Foundatio.Xunit/Logging/TestLoggerProvider.cs": "1d09b2b7611ee712080e26725ac4950f58090d047c92c55852be8e8e40a5b26d", + "src/Foundatio.Xunit/Logging/TestWithLoggingBase.cs": "4bebbbcda6d9e659bcd8ee7f4ec50950ecbe59c12e5b8421eff78a6d4b184ccb", + "src/Foundatio.Xunit/Retry/DelayedMessageBus.cs": "3e2adde0c5dad8a35faa1783be4c4adeb12726855f1d23832bf37dfaf8396206", + "src/Foundatio.Xunit/Retry/RetryAttribute.cs": "81b385e191f6dcc15758e2368390ec97c7d7031fe5e341eac2f9babc8b1c500e", + "src/Foundatio.Xunit/Retry/RetryFactDiscoverer.cs": "0dbea335f249c3aae7767c3fbd08acc459b899d61370beadb256de8d6814501e", + "src/Foundatio.Xunit/Retry/RetryTestCase.cs": "ca098c9d8a1264d0b85b658ff5bc31255db6b66db3e065e88fa71b872d1eebc5", + "src/Foundatio.Xunit/Retry/RetryTheoryAttribute.cs": "91ab8a04c4e138b07547298ecd69083450484ef42e938ed9bc85b447a6a81cc5", + "src/Foundatio.Xunit/Retry/RetryTheoryDiscoverer.cs": "11fc0639f76cb81df766a274aaf789ebd6773a088ef8f05c7503d04df77b9d70", + "src/Foundatio.Xunit/Retry/RetryTheoryTestCase.cs": "c2f707eb4d00db13add4f8a3a377deb85398c5ee34bc7a1680d41681456926a2", + "src/Foundatio/Caching/CacheException.cs": "6fe2dea81205e4b625c7f45f465a20dcb3d57468bc1cfa2e87186e06d959e3d8", + "src/Foundatio/Caching/CacheValue.cs": "2bda219472312fc288f6a4a83ae6e82fd87f11c428c704b2ac96534a6134d94a", + "src/Foundatio/Caching/HybridAwareCacheClient.cs": "45b8c10b72381932be6383dd88402065399ae1e8add53dff4f318d90406e4087", + "src/Foundatio/Caching/HybridCacheClient.cs": "1af3875e57564c655258b61a0aff8cafdb1e0af85a68272b07e8e4d85ffb9c08", + "src/Foundatio/Caching/ICacheClient.cs": "7aacccd6af26d8f65e94fd86e794cd7b641429eaa44dd6188601e8866924744f", + "src/Foundatio/Caching/IMemoryCacheClient.cs": "dfc25946e9ed60ad4f469bfc38fcd32160fa84e9f13df1b6fdf4b9737ac44892", + "src/Foundatio/Caching/InMemoryCacheClient.cs": "ece6bfeacc433955e566c05884ea79c8b500f4b46ed5a2f31cb5e6f060de2b6d", + "src/Foundatio/Caching/InMemoryCacheClientOptions.cs": "58b51d14d30867ff9326fdd53a1337d2f870c5b39a3763e492331d1c9db5982e", + "src/Foundatio/Caching/MaxEntrySizeExceededCacheException.cs": "495eff7eecfc9be934b3b9c6da93a9dd38e6f0daaa4edb54fc0c3a8cb296a8f6", + "src/Foundatio/Caching/NullCacheClient.cs": "255894c156a125dea90f09aa747a90cd5bfa96eb817380e1735c87a2f27df49f", + "src/Foundatio/Caching/ScopedCacheClient.cs": "7d16b8acd591755f45353e9329509f9de40534eee44157fcb7b0874cfee11038", + "src/Foundatio/Cronos/CalendarHelper.cs": "8b9e4f3415301ced6e175b162c676b666ff69fd506b313161c0d684f1547d468", + "src/Foundatio/Cronos/CronExpression.cs": "4ddea92e8e2f9e0ab269ca965b77887cfd155817913979fd835692bb06314ab7", + "src/Foundatio/Cronos/CronExpressionFlag.cs": "1a63bf81ce0bde088339c666b3a0eb61106effe7bcd9c74a6b4b158ea83ff813", + "src/Foundatio/Cronos/CronField.cs": "b988d11adea63bcb3f3d44f6c02a732d27498b0d8b39e0a99a0594269310330a", + "src/Foundatio/Cronos/CronFormat.cs": "cc9da2a50aaf1b17eb37fa07ab010ba590fb8d6a138424e69f2058e3b4717d28", + "src/Foundatio/Cronos/CronFormatException.cs": "d287a4a20f71c9d9435000471a02a2a860c5f9f5437d2883731b6e2d1a3ee1da", + "src/Foundatio/Cronos/TimeZoneHelper.cs": "11d45395152bae71927dd7a23d927dd2c173a8f39e27be1547c120554f3152e6", + "src/Foundatio/Extensions/CacheClientExtensions.cs": "10da6d73ee104823c8bbd455d68135b1f27a1be7fe3822a97c4a922fb40c037a", + "src/Foundatio/Extensions/CollectionExtensions.cs": "a1cdd1ec945bd3930edbc3dfdbc23d9a78eba03be896a48ccc0e8bacfaf34a73", + "src/Foundatio/Extensions/ConcurrentDictionaryExtensions.cs": "0a2abe6dbe053375ced5abcdfd558dfa842f00dfe1b449a45349df4d64f82c0d", + "src/Foundatio/Extensions/ConcurrentQueueExtensions.cs": "47a1f3d67f9cbe55c63a5231cbdbc67be6126e618a861149d2f7919e1cdc1098", + "src/Foundatio/Extensions/DateTimeExtensions.cs": "267cd941051e013fbd25e1b8f06b0d5d7987dc2bc5a34eb47a84a5a5777ae66c", + "src/Foundatio/Extensions/DictionaryExtensions.cs": "2415e242f5ec9de519020c1bf11cca9a83aa3b6b1a24880a886ea051c4e4da9e", + "src/Foundatio/Extensions/EnumExtensions.cs": "b353f30767958951616ab150542ae743e46dce4853b794190ff732d2c1bb3d0f", + "src/Foundatio/Extensions/EnumerableExtensions.cs": "722f2281f6209833a9fcbf2245cff5d60d60c60241ea28bafe434f5cf2ea93b8", + "src/Foundatio/Extensions/ExceptionExtensions.cs": "b08b4a64b6bdf3fc74eb2155be82bd8c9a1be5f4c045143a9dec8eed61638e73", + "src/Foundatio/Extensions/LoggerExtensions.cs": "3989d3d3a8c1c0d905636e3d0f9e4c9c929ae6e50dafcffe6cb290e5dbffcdee", + "src/Foundatio/Extensions/NumberExtensions.cs": "5e3e3323ece20b3e6ea8e24b740bb3e4a6e82c559289fe14e4e94d63d90e1243", + "src/Foundatio/Extensions/ObjectExtensions.cs": "66fcdd7ed75ab2995d95dbf7f057b8b48860da22815137ef6b1b79c468b0d6ac", + "src/Foundatio/Extensions/ServicesExtensions.cs": "c0cecf02953cc6ca6c7713af8f0d60af62892dc586c131605e406951aa1438cd", + "src/Foundatio/Extensions/StringExtensions.cs": "02b1c505e97e915639566affb8b2146319d5696022b2468854842363004ab614", + "src/Foundatio/Extensions/TaskExtensions.cs": "d58d5c036c43432862117216426f1d311a62b2391f9251d4a6b422197124f222", + "src/Foundatio/Extensions/TimespanExtensions.cs": "ff6e728aa46435064e0806042648374dedeb274767558da558bcd2e2eef4b371", + "src/Foundatio/Extensions/TypeExtensions.cs": "c903557ae7fb9097648487d95a5fc0acdd2b2329ee9d5fca2e454fea8a4cc91c", + "src/Foundatio/FastCloner/Code/AhoCorasick.cs": "72c5151506e4485591718d0d01c2b45628412adc967e574c6ac015d7f5349977", + "src/Foundatio/FastCloner/Code/ClonerToExprGenerator.cs": "ad26752fb08ad6c2d5d426a93a0d13848129d546ae709ab62b83acd5cc8f35e3", + "src/Foundatio/FastCloner/Code/Extensions.cs": "f30bbb1f7f30a01c60fcc55cdc09879ac31c90aa952caf324111e181fad54dfd", + "src/Foundatio/FastCloner/Code/FastCloneState.cs": "33714a437d76f594fbc251724079a6bcd2c200f2e33af0cf7ca681c7c20e8840", + "src/Foundatio/FastCloner/Code/FastClonerBehaviorAttribute.cs": "4989b05f71b2d94836d6c0f45e2517810f2fd4fec292502208084f156869e7fa", + "src/Foundatio/FastCloner/Code/FastClonerCache.cs": "daf833dc788c21db0e8316849e9ac3cecbf67014f66a1dbf4b8e7a20fd020e1f", + "src/Foundatio/FastCloner/Code/FastClonerExprGenerator.cs": "413f750d07d26825a88f6ce429d74885eaaf1362de77953369506c3416f34f6b", + "src/Foundatio/FastCloner/Code/FastClonerGenerator.cs": "072bfef95c9fe234af16eaea67124e3c06f7fad6864789138468a90b155df2c5", + "src/Foundatio/FastCloner/Code/FastClonerIgnoreAttribute.cs": "39c375ceb10b989ff84e5640f6e7796267e986dd5460d2e899c09f895de3d93a", + "src/Foundatio/FastCloner/Code/FastClonerReferenceAttribute.cs": "475ba8cba96ae1753e80879790a0c0614ca38bb72e1510318963c4770e65bb33", + "src/Foundatio/FastCloner/Code/FastClonerRuntimeConfig.cs": "8068fe21c395155985f443f1be5903620c49c12bcb4a7d537dfed32c68e1f492", + "src/Foundatio/FastCloner/Code/FastClonerSafeHandleAttribute.cs": "8247b99023106dcdfb67b7af93223cda5a9268e8e6d918f3ef283ec2dba75c8f", + "src/Foundatio/FastCloner/Code/FastClonerSafeTypes.cs": "edb71b5ca97e83ab33f75217343f2fdc57e5aacc0c5a8ae76a48fc958d86b2e8", + "src/Foundatio/FastCloner/Code/FastClonerShallowAttribute.cs": "ac7c16f275e881a1cc2663663e7d30481fd12a1de989385c052c386cb8cb8e54", + "src/Foundatio/FastCloner/Code/FieldAccessorGenerator.cs": "865f78ae75a619c4f5957cf8e5266672f9f8dbf2579b025fdcb8321f39447e32", + "src/Foundatio/FastCloner/Code/Polyfill.cs": "db91dacfe045d040bfc26346e4921e800811d8e55712b5c9d5cbe16f8ed1fffe", + "src/Foundatio/FastCloner/Code/ReflectionHelper.cs": "7147ea1d8a404d91d501c9b2ceb633a1a23170bd5508e722367cd3d4a74ce752", + "src/Foundatio/FastCloner/Code/ShallowClonerGenerator.cs": "0bc3e2b7eff6bab744f0995cb88f514e49604b44d49957393c1ed876105c3b27", + "src/Foundatio/FastCloner/Code/ShallowObjectCloner.cs": "f82c6a9be4be2c1329264a2b8052c7b5f3f76346f217a12b89e9d278b3b351af", + "src/Foundatio/FastCloner/Code/StaticMethodInfos.cs": "40b4983131588250c333bb42c38cb44ca6afd3e42a1951a44ca2aecf0be8f6f6", + "src/Foundatio/FastCloner/FastCloner.cs": "b7f96862805a54f8c693fb88ee0d568ee1105bd79a6ddaf1554ef150987ebe02", + "src/Foundatio/FastCloner/LICENSE": "c0fc02a86ae2b179c4729694c523bbf42da674e5136ed8e12e3cb2f26556ea9f", + "src/Foundatio/Foundatio.csproj": "f63da34036ec49c0c5b88976de98f12a2287709574c9b3e9313cb50f05158a7c", + "src/Foundatio/FoundatioServicesExtensions.cs": "a7de97a09d0247f5c55ff74a938197d49d9039e204f2e2d5d2a9aaa6ca7b0142", + "src/Foundatio/Jobs/IJob.cs": "7b9c420da098a327b76fa722018289d9992837d5798a518d7c02f7b958dcc363", + "src/Foundatio/Jobs/InMemoryJobRuntimeStore.Broker.cs": "667e01409912135f7a7461f0b6b11049dc36f8a726998a2b76c191ba59928fb9", + "src/Foundatio/Jobs/InMemoryJobRuntimeStore.Claims.cs": "9e98a3b72bdbfbf3315cb3a87da189ef19df362279b0b7eb073afef078d0d6f1", + "src/Foundatio/Jobs/InMemoryJobRuntimeStore.Schedules.cs": "5a9a49d36a8f87b71ae2ba4cf2a4b357df1794ca5e4e715d046d3ff14ce0a633", + "src/Foundatio/Jobs/InMemoryScheduledJobStore.cs": "c517940895008636b7ca1eee133a8f45c9c9a0f409d0ab025660a7d7b0e979ac", + "src/Foundatio/Jobs/JobArgumentContract.cs": "43c855c611495628d85d6e92988bcf355c28cf7412348e5ccf608c245ca6d456", + "src/Foundatio/Jobs/JobClaim.cs": "ac73c3ccaac9ebf04db3c57035c6e90ffae892f3ee9b31a15b9f954382f6c0f4", + "src/Foundatio/Jobs/JobClaimValidation.cs": "91e74560455cb936e0fa34fe60dab32c682210b699a46e2f93c5f0513059dc8c", + "src/Foundatio/Jobs/JobCounterStats.cs": "d0fecb43abe9f4427ad1d998bbe700c1cb6ead08eaf65e330bb219c0bc6473a4", + "src/Foundatio/Jobs/JobExceptions.cs": "ee4c71f5c20c2c084e25ac58bd4f1817c8fed6a2087ac2de2370a2c5087314f9", + "src/Foundatio/Jobs/JobMonitorExtensions.cs": "b00d219742c58ce270ccc2a48b3b82972719a61442bcd8070e4e122d47b28166", + "src/Foundatio/Jobs/JobPage.cs": "e8411cf6f3dbb8c3c59a8ae97ef45aa53d14b7f59b91acbbe92531ca15372bd4", + "src/Foundatio/Jobs/JobResult.cs": "ddb0854db17c745e2ad03723dd11e47ce3404103a01183419c0443ba7911452f", + "src/Foundatio/Jobs/JobRetryPolicy.cs": "0fd8592eb1933a68b3ad3d7c2fce19f0b051c5e6118616f8d3b8a4e0bb3cf0d4", + "src/Foundatio/Jobs/JobRuntime.cs": "2aeea15268d7a7fa03dca8478b73a82e7db696d3b521b0daee2249c576c69fd9", + "src/Foundatio/Jobs/JobRuntimeStoreOptions.cs": "c9752f88b1cab3cc6d9910a679dd4433d8347359ecdc60a07e383a63c3835bf0", + "src/Foundatio/Jobs/JobScheduler.cs": "57106b2cbb533f796081c92913beebc74d884a392e939fae50524221057cbf2e", + "src/Foundatio/Jobs/JobWorker.cs": "6e10c9cea22efeb30051b3bbf65230c114707695e11617a839b8655b04ad4f3a", + "src/Foundatio/Jobs/ScheduleQuery.cs": "eb805c8897d276bb2429a666b49b8211816d303d177069f840b16f07fd6f5f8f", + "src/Foundatio/Jobs/ScheduledJobRegistration.cs": "91683e5ec1b9804ffe21e0db0bced8e16acd39c4fca55ab564ddc051dc6eeda6", + "src/Foundatio/Lock/CacheLockProvider.cs": "b70fd5a431a3c56b49cad9892c225f2606c0027a3eb01af7edc216366e9266f4", + "src/Foundatio/Lock/DisposableLock.cs": "72bd9c3cffd80114b3fdb6be75567d324f6200fcb442d799a911957265b1a45d", + "src/Foundatio/Lock/DisposableLockCollection.cs": "ff8fc50a176f7a2d5b699e99ee6dfd7e23bfb115dcb3cbe8828e1324229e1df2", + "src/Foundatio/Lock/ILockProvider.cs": "4f193706e9ba8edff38cbd7453af2e254b20f0f0b47711038b9ced93fe1a81fd", + "src/Foundatio/Lock/LockAcquisitionTimeoutException.cs": "abd752f2494dd245ba9adc5c22e5b88de349c5ebc1512af02c9969e7eaf55631", + "src/Foundatio/Lock/LockException.cs": "05de1e03cbec5acd60dd6303fc5d6d5dca2e80b76853ef6cc69ab9b18808dea3", + "src/Foundatio/Lock/LockOwnershipLostException.cs": "5b375238aabe3e908b7c96cf2f554be32fe8b7d2b7695d379e7153b425005ffb", + "src/Foundatio/Lock/ScopedLockProvider.cs": "c580fabceba94a4941dd0ad926734f00a2695fe57e431711c719e0ba426dec54", + "src/Foundatio/Lock/ThrottlingLockProvider.cs": "57a5ee6b090581a770deb73e1801638443daf774443b961aa75a979d1c953742", + "src/Foundatio/Messaging/DeadLetterQuery.cs": "00ca909db31b6b10095234b0a3313b51647e9c6c6657ebac097b3bebd145ebd2", + "src/Foundatio/Messaging/IMessageBus.cs": "287f266106b8e35eb47b2baf5d02e3deb8c4482836e525a5339e12ab46fc8d6e", + "src/Foundatio/Messaging/IMessageContext.cs": "2229c07c2a865643c7ed93626fadd1850c495978aacd73d7650f53baebc63388", + "src/Foundatio/Messaging/IMessageHandler.cs": "ccd16eb5831b964de4b26ef41067653373cecea16d871b92ec854271b9eb9673", + "src/Foundatio/Messaging/IMessageProcessingObserver.cs": "38864cfbffe740fab917929b27c569813f51a06babb2956b5d496017576931e6", + "src/Foundatio/Messaging/IMessagePublisher.cs": "a9c887676179ea50013dc82e2e310d685549689564eab69cd66cb7605fe2f232", + "src/Foundatio/Messaging/IMessageSubscriber.cs": "400354167c3e7e094aa95377fec7539ce5e8a7a73d1ebdb423d3561c402cc253", + "src/Foundatio/Messaging/InMemoryMessageTransport.Subscriptions.cs": "32256b924c8de140026c72c4256220a15bf8d9e62e78a80bf6822885b365ae30", + "src/Foundatio/Messaging/InMemoryMessageTransport.cs": "7da1af85bc752d9e219192c9040b8954ec4222a132fc2ee07ed7173147e83394", + "src/Foundatio/Messaging/KnownHeaders.cs": "55b6d9176b7a5c498fa4620612c198cc110ef45132174d0a6e4aceeda1e6c06c", + "src/Foundatio/Messaging/LegacyMessageBusAdapter.cs": "69403ad89464e5f60888c82aba85161bddd86a61e32e05c9d4260b1c270d7582", + "src/Foundatio/Messaging/LegacyMessageBusException.cs": "e454e8b5664bf60b684ef9aee9069025a97def925f338d304c4c1b11a55e3ccb", + "src/Foundatio/Messaging/MessageAdministration.cs": "c120f232e33f6214f6c8f4ce4d97b0e3ade21845f36f281cc9d2cb8e162ac2c1", + "src/Foundatio/Messaging/MessageBus.cs": "2d15bf6db27c07d809f1cfa9c2cb5be057e685c7f45ef4c14344571a4380bd8f", + "src/Foundatio/Messaging/MessageBusException.cs": "79593748b500ab25c2c8c6f3159fa26f533d544b67782d0c7d95f65ab527c0bb", + "src/Foundatio/Messaging/MessageClientCore.cs": "ca353b3a5319b402093edd69f9f5d3d5eac106d9ca24d23cf08fac27b6d753c6", + "src/Foundatio/Messaging/MessageDeliveryLease.cs": "846a89f00595051907496e81e4f3db7d16d9a42a566be80c810f682ef53c3184", + "src/Foundatio/Messaging/MessageDestinationNotFoundException.cs": "74711e5d593ec984957294f0771c5cd9256e5b9832d5c0034b5cc08eebce57d8", + "src/Foundatio/Messaging/MessageHandlerRegistration.cs": "42f7466ec3cec53b7ed5fa6a3e251df88d1264a2acac427bfbc0d857312b7490", + "src/Foundatio/Messaging/MessageHeaders.cs": "4736b1cf10a7be5e9ebbc894c1c76d9cc4f9ed0f814f54c4e7dab0b0a02ac5a1", + "src/Foundatio/Messaging/MessageNodeSubscription.cs": "fe6e08537831322b5cf961a2ec05fb1a0e534cb9a9d38136f5da009786b1a941", + "src/Foundatio/Messaging/MessageOutcome.cs": "563440938a521a3500f545ba746e3dd06bf9930fc10c72416d791b16b07d79ba", + "src/Foundatio/Messaging/MessageRouteAttribute.cs": "10a33923655237860004d2be64515a3f66233bdcc63ea913a1f11a76a30349e8", + "src/Foundatio/Messaging/MessageRouting.cs": "7e244c6c12f320a735461dfe07f2ce05ad6e555eea8542002b8830a4de9380ee", + "src/Foundatio/Messaging/MessageSendException.cs": "963292dacccbe28f8680e342580bbdc351d964d6f0d6a453508e5cbf19290a4c", + "src/Foundatio/Messaging/MessageTopology.cs": "e1cd3298886f935640e82e072a30ab7725959d2eed7cbd99c4a215421d9f5a20", + "src/Foundatio/Messaging/MessageTransport.cs": "5e3922ada76e428d6e9f77ed847c1c1736993eaa6fdcd13d5cfdba97c4665d61", + "src/Foundatio/Messaging/MessageTypeRegistry.cs": "f11bb801e65fdb2b4f9f0cecc6cd113a2f39d3332f54a5f0af93be505b70d8e0", + "src/Foundatio/Messaging/ReceivedMessage.cs": "9d8f413864210e1a6abbe619638c3fc48e6012b4a32a4e9916456c5b674d6d88", + "src/Foundatio/Messaging/ScheduledMessageDispatcher.cs": "a4297c39c82282cee5b806b97f234d2f353a8bff501886ddd77b9c1a1bd7a2c5", + "src/Foundatio/Messaging/Tracking/ExecutionHeaders.cs": "cb642cbab97b68baa7d03be6337984a153edd448438a55bcccc8d78b0c28b52b", + "src/Foundatio/Messaging/Tracking/MessageExecutionOptions.cs": "86e6c97daf58c044bfc3b8494a6ab541f2a5e8e48c923fab5c8832f5c374c970", + "src/Foundatio/Messaging/Tracking/MessageExecutionPipeline.cs": "1b8f974919ea57ffac88d3b945b7542937c3dc07b6356499a18432e5cef37196", + "src/Foundatio/Messaging/Tracking/MessageProcessingContext.cs": "a69ac61b323e9cb21722da782d6c6392c48dee2a0289faa589c85865a5a0a440", + "src/Foundatio/Metrics/IHaveSubMetricName.cs": "b6d58fd4aadebbcea87a2cecbf0d145cfb9d101336ac15d68bf10fbd07ce1edb", + "src/Foundatio/Nito.AsyncEx.Coordination/AsyncAutoResetEvent.cs": "1ccd5a6602d5de16cb119ae1da6768d79aaff1724e95d22b38ac321867a40408", + "src/Foundatio/Nito.AsyncEx.Coordination/AsyncConditionVariable.cs": "a50523ce16275512407cea69fc56ea70ba221920a21297f1eec1a9e07c8b5633", + "src/Foundatio/Nito.AsyncEx.Coordination/AsyncCountdownEvent.cs": "78cc9d9e712a5612474a48b9096f3d0812c14c68c563df2923131dcc5a11ad42", + "src/Foundatio/Nito.AsyncEx.Coordination/AsyncLazy.cs": "c1dacae8ebfcd4e2da2ea1060a6dec0848cacf4a41723c68a4a4ddba1fee680d", + "src/Foundatio/Nito.AsyncEx.Coordination/AsyncLock.cs": "79273c3c8ab67a5799ca5eb38b448978fcc7187614461a10a7eb2dd88206b3dd", + "src/Foundatio/Nito.AsyncEx.Coordination/AsyncManualResetEvent.cs": "334061aab2bfde8ea6625572efe26697738766f3ddf13bc709bdbc4255baf0b3", + "src/Foundatio/Nito.AsyncEx.Coordination/AsyncReaderWriterLock.cs": "e5a5f2cdea889971312b7f8af3f50a39663a49644173be5c5292ed262c836a86", + "src/Foundatio/Nito.AsyncEx.Coordination/AsyncSemaphore.cs": "2cad1bed0addd84ea74ee9ee415ee857117e23e5c9eb52c02c92eae31b3a0b50", + "src/Foundatio/Nito.AsyncEx.Coordination/AsyncWaitQueue.cs": "3c8640601f3173c8995e2e37f8c0bb796ef4bc71f3ad0c5d6106f9178612d629", + "src/Foundatio/Nito.AsyncEx.Coordination/IdManager.cs": "73ddd1fbb3af18d003cd66fcd900bba9042d4620eff9907d6a567b19197db655", + "src/Foundatio/Nito.AsyncEx.Coordination/LICENSE": "77e1e288c2f378ad62efb54f2567eb7aa2a4f4c91af342bb869c2ea09037d638", + "src/Foundatio/Nito.AsyncEx.Tasks/AwaitableDisposable.cs": "461c50f351c4350b37494b91b1373bcf9f59542f53c8e83339a659975a8e088f", + "src/Foundatio/Nito.AsyncEx.Tasks/CancellationTokenTaskSource.cs": "0b8a6906ba65bdea16a2845a16c50761b0fd297466db4dcd7841236aa4329a1c", + "src/Foundatio/Nito.AsyncEx.Tasks/ExceptionHelpers.cs": "09ae8834bb72eb1952aa04a82a6571669689726ab5aa0cb42c8c1e909c07d256", + "src/Foundatio/Nito.AsyncEx.Tasks/LICENSE": "77e1e288c2f378ad62efb54f2567eb7aa2a4f4c91af342bb869c2ea09037d638", + "src/Foundatio/Nito.AsyncEx.Tasks/Synchronous/TaskExtensions.cs": "482b600dee78ad368214bccc4c91f792925ddcafcaa86a171964494067374476", + "src/Foundatio/Nito.AsyncEx.Tasks/TaskCompletionSourceExtensions.cs": "967b42242e3ecb543daae6955c1f0b5949ded9900529e9f26aabbdee0714dfd3", + "src/Foundatio/Nito.AsyncEx.Tasks/TaskConstants.cs": "d0bc8748da8774b2ec0ff7781b048eeda0a9d65b9a1be028ca4df9c211493dac", + "src/Foundatio/Nito.AsyncEx.Tasks/TaskExtensions.cs": "ffbcb6a725f00f1bac61f6a6171eec259f52907bdb9dea3d34bcd38321f38e36", + "src/Foundatio/Nito.Collections.Deque/CollectionHelpers.cs": "5740dbcf6617cc5d00d72ef5b825ea097a0bbc27d4e9dd529b623dcd825440fb", + "src/Foundatio/Nito.Collections.Deque/Deque.cs": "88b890710513a1be1eb967be7ce446c200be9aece01d42ffda153e32be9c4102", + "src/Foundatio/Nito.Collections.Deque/LICENSE": "77e1e288c2f378ad62efb54f2567eb7aa2a4f4c91af342bb869c2ea09037d638", + "src/Foundatio/Nito.Disposables/AnonymousDisposable.cs": "12e73acd81a22bfeae5a006c1ecd70eabc4eaac7e062bfe90bb3b3d97866478b", + "src/Foundatio/Nito.Disposables/Internals/BoundAction.cs": "27a1dddc79c22777a854b39028c811d564ed3f6e94dab217b6658985e21efe8e", + "src/Foundatio/Nito.Disposables/LICENSE": "e0a3659f911080603af1a8c4ce974758954f501474c6168add6dbd4106585901", + "src/Foundatio/Nito.Disposables/SingleDisposable.cs": "d3d91e7371162f7ac8422d4d29865e64a49573becf61cd325260947d5391ba6c", + "src/Foundatio/Properties/AssemblyInfo.cs": "04fa53dd2f9d942cd0a9b7f1a8a782f355a5d9da9394d00f0b0a3419248774e1", + "src/Foundatio/Resilience/BrokenCircuitException.cs": "e0624552f576c8ecc01cb8e7cff5ce161971376645889d3e0177a9a601854596", + "src/Foundatio/Resilience/CircuitBreaker.cs": "8bad8d04cb742be9469c8dc91f62d0510ba68a7750178c13e711a01183ca5251", + "src/Foundatio/Resilience/CircuitBreakerBuilder.cs": "0cd9f3779fc753d38521911ff4b5d491697b03293471613719d7ce857a813194", + "src/Foundatio/Resilience/CircuitState.cs": "b57f7e97da1c473efd3c1a8227396b630068120407c5ba795dd8124fb5eb57ee", + "src/Foundatio/Resilience/ICircuitBreaker.cs": "1f841fee53e77d16c770a3c257da13fe4a11d8ba467bd7e0c5ada40b9ec82a58", + "src/Foundatio/Resilience/IResiliencePolicy.cs": "9c55f84054e78e3b381ed14ae948ca1f9b12a035002e88308aefeaee1898f544", + "src/Foundatio/Resilience/IResiliencePolicyProvider.cs": "ff1412935c834a804eb017c87211c64bbeb60ce5637abf05facea34313e91b86", + "src/Foundatio/Resilience/ResiliencePolicy.cs": "cb12b7d1954a580091e40e134f5037bd9107e7c6b4f1ec219d1a0f06ab80bcb0", + "src/Foundatio/Resilience/ResiliencePolicyBuilder.cs": "519bc9fc71e8b5cc5ff648520bf59dcb438dae57a3da16c9e356d5af6d9c4688", + "src/Foundatio/Resilience/ResiliencePolicyExtensions.cs": "314887da32ea86ab2ba78c58507ad10aa2d4bb2b3e1df5d2d7771f3f175d5719", + "src/Foundatio/Resilience/ResiliencePolicyProvider.cs": "fabc20ed03348dac24f113ae3e849bbb4dab95634c52a2141cb81e78d5730204", + "src/Foundatio/Resilience/ResiliencePolicyProviderBuilder.cs": "f2564d8459a238ed64b3384634a1a05eeb2a6ab60197e9ec6cbbc557661dbadf", + "src/Foundatio/Serializer/IBufferSerializer.cs": "7104c8eee0961f5ca78f1fda247aa7f94c800369342cee8aeef6c22ff74c197b", + "src/Foundatio/Serializer/IHaveSerializer.cs": "dd163958d20d231b3a918fbe5127bccde7aa9ef484b0caa3b755503686614cc4", + "src/Foundatio/Serializer/ISerializer.cs": "2fd2f31958a939358fad842c6299afab9306bcf504738f4e33e41d5a818c5199", + "src/Foundatio/Serializer/ReadOnlyMemoryStream.cs": "85e678624594c33c47f40670d125db0e2230be6c0eeb651ed9708042cc5f11a4", + "src/Foundatio/Serializer/SerializerException.cs": "8f2349f0126ddc77f5c80fdc86440fa45cc43af58b2868707bdd1cf1235116d3", + "src/Foundatio/Serializer/SystemTextJsonSerializer.cs": "5bd2c78426eac1af9c8591770f6cf0d16125ab27de7e4431c007e4754fc14444", + "src/Foundatio/Storage/ActionableStream.cs": "fa721fb2a40f8d0ed88a50c8e96cdb32e0742d7d09a50128f3270e7a24d35344", + "src/Foundatio/Storage/FolderFileStorage.cs": "5e5e53471ff86563be60b8177876d53ff6f7fc6b7b392ffe8bdc65dea8673a29", + "src/Foundatio/Storage/FolderFileStorageOptions.cs": "d3b0bf79ed101eb1f4dca8531f7aec0629f0ff5e8e69536f3f1bd6395487f2fd", + "src/Foundatio/Storage/IFileStorage.cs": "07f182f7a54602777cf4aacb27162315cfc3ffe4960f9f7bfbeb639275f7f476", + "src/Foundatio/Storage/InMemoryFileStorage.cs": "0213a1a2966134652c9c420afcf4530058361d9b2fb78251457c8f90b327f150", + "src/Foundatio/Storage/InMemoryFileStorageOptions.cs": "2848318b2e009fea36fcc62767489043b8b30cced0564a78a10e5985a865bca4", + "src/Foundatio/Storage/ScopedFileStorage.cs": "f2f62c37e50129fac3b64ae8996a3a8d765ffbfb0c740c31b9ea9142726ad984", + "src/Foundatio/Storage/StorageException.cs": "fc1e3c545b72a0357e35558aae9163fe72791a903739f75efaff0b4af3cd2048", + "src/Foundatio/Storage/StreamMode.cs": "328023b94d9dc00867082fff2c209182c8af971b9c161398bfa999ffb5193bf7", + "src/Foundatio/Utility/AsyncDisposableAction.cs": "c12f78bf1253de178e01d333f5042692c9aee086834641835a6b864e12f70f76", + "src/Foundatio/Utility/AsyncEvent.cs": "d77d827e11073fa8b23596690d412ed47e90c69d0887b35b7d676cb87f15fae0", + "src/Foundatio/Utility/ConnectionStringParser.cs": "71b939254e0ab0b75a684b740305f4d69eef7a6295960ba779d5b054b80cfcc2", + "src/Foundatio/Utility/DataDictionary.cs": "4b382eae26b219a720ad39a734ac4ae176f67393794144015c78a51228660a8f", + "src/Foundatio/Utility/DisposableAction.cs": "bf7179516286fd5b5b7424bad9336b596a1cfd231ea220d55509f0d4a0496ac4", + "src/Foundatio/Utility/EmptyDisposable.cs": "acf1df1e766f21821af8dda5fbed262dac90eec2ae211c894918678256d90f5b", + "src/Foundatio/Utility/FoundatioDiagnostics.cs": "8505200763ef6a582f0b469254da4af1a68464a900c791c28282aa0822aade6a", + "src/Foundatio/Utility/IAsyncDisposable.cs": "c2c3890172722c706cbd383c37fb895a6ab1fda28c375781e9b1cb2a22650228", + "src/Foundatio/Utility/IAsyncLifetime.cs": "9312c7d492c0c128cb3f11f602b7a7a754e28ca071ed30daced97c1ba2e2c184", + "src/Foundatio/Utility/IHaveLogger.cs": "a3e9ef1daae10ef0feec6896ffa802ba964f7bd8d27e25d21a62d56686b5e1b4", + "src/Foundatio/Utility/IHaveResiliencePolicyProvider.cs": "1a37b6133d6efd7ae60a5ba749b54f6990d5e7195f1cc5a1be2ea2ee8466e9a0", + "src/Foundatio/Utility/IHaveTimeProvider.cs": "16e10aa168a112b8c55bcff6eef4052dc3c3d8d8ba65c1c9232e25fd3bc7a667", + "src/Foundatio/Utility/InstrumentsValues.cs": "a0ee2227c8db63afb897c1766e9387daf0eba52ac0aced8713f0b68bd5fb9736", + "src/Foundatio/Utility/MaintenanceBase.cs": "477e541ff14493f65297d99603bcac26cf99c04b5b826983f93c3d92bf8bdf7a", + "src/Foundatio/Utility/OptionsBuilder.cs": "98cc48261c3760a0c7a6ecf0dfb4f15f1fadf0724c52e597853f1804c9009f6c", + "src/Foundatio/Utility/PathHelper.cs": "8495fefa8b7b7666698bbe9542b6809fb36a5b88fa104a2b0ec44a804a77aa47", + "src/Foundatio/Utility/Run.cs": "f3ce77a63558e4d05dc09d7cbe02321913eee96a82d61b22dc254fad1bfce95c", + "src/Foundatio/Utility/ScheduledTimer.cs": "4f0e37583bf5489ae9b660b1af59eb2f3cf61181ec66189edbda902895c89ea3", + "src/Foundatio/Utility/SharedOptions.cs": "aaec1dcb43fa31f5ef03ce707c65a39e8bcca4525876bfe65ca00a17dd6f9fae", + "src/Foundatio/Utility/SizeCalculator.cs": "afb7d08d046d1cb2e9299278ad66620778ab2d79c22caaf05a6a148a8b78de20", + "src/Foundatio/Utility/TimeUnit.cs": "90bf0bc0d59bcca6b971089b9496b5fd886e2a08c18c0595b06430469d96abd1", + "src/Foundatio/Utility/TypeHelper.cs": "69ef43c30f8bdb31723fae8d6726453f1fef3115579a12ae7b2b851327ee8738" +} diff --git a/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/manifest.json b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/manifest.json new file mode 100644 index 000000000..f7f020bf4 --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/manifest.json @@ -0,0 +1,33 @@ +{ + "runtime": "/usr/bin/dotnet (.NET 10.0.11, Ubuntu package)", + "baselineMediator": "ff9c1558c12259819f39d60db5b791874005f5e3", + "baselineFoundatio": "54006ac7a850c8cd297362cb21238366887a77f7", + "pr149": "89bd6d1504b83d03de156b08f24fb578aa8bc97f", + "mediatorMain": "a1480132d701b93c64edcc9a776573c94d1a2153", + "binaryHashes": { + "pr149": { + "Foundatio.Mediator.Abstractions.dll": "2e875f87d0730cae886d5f9ad1883e75cda1883621bbcf02d0d967ca82e40fd3", + "Foundatio.Mediator.Distributed.Aws.dll": "9b23123c4ced306c1c62e9c71362db161b72dc280517a1bd7637e4cd3e383d6a", + "Foundatio.Mediator.Distributed.Redis.dll": "0d69009172b2dfd9a9ceb009de3d3c837b66d669ffce2e00c26d9eacdda984f0", + "Foundatio.Mediator.Distributed.dll": "2083b1a1eec1498212a9011f6935a9801d6e058f18cff9b565e5207f85cbd665" + }, + "before": { + "Foundatio.Aws.dll": "305496a7894bcd99a7cd56bc6cdd36c19ea3f65fc29cc7ebf2b84c90c9343f6a", + "Foundatio.Mediator.Abstractions.dll": "b2dfa05db70b7d45fcb94bfcf4c67a47c3f9f35d4aef06680a5bb6c6f12934be", + "Foundatio.Mediator.Distributed.Benchmarks.dll": "d942140b18709c9aa22afd53cc8874e329b54ca7cb47f10e2d3562d732f35c0a", + "Foundatio.Mediator.Distributed.dll": "8ac86259542eaea33996b1d9e506db980216e46404ed9e92636f9e73215b1a79", + "Foundatio.Redis.dll": "3b33d5cf508b30e6d213a8689b6ee10b99b3cddb559a3e5b8918e5b11b3f1f1c", + "Foundatio.dll": "578a50491ed8737162fd560e64fb5c6a5af6353ea062f36edb175272d0f455ba" + }, + "after": { + "Foundatio.Aws.dll": "51a478b14341fddbce2412de3b5780395b620fc62d50887b01081a8156bdcc98", + "Foundatio.Mediator.Abstractions.dll": "3acd548e0cd2fa3fffabcd1877c29b6de94a73e6d4da6c2beae32fb9d46af7fa", + "Foundatio.Mediator.Distributed.Benchmarks.dll": "cf8918681897f7be49001a7b02eba4255d2af31b6d5f0abb6f5ada72643ebec7", + "Foundatio.Mediator.Distributed.dll": "ede5081842166c6dfed44a4235f78a876ba34746f414798afccc5dba95cda383", + "Foundatio.Redis.dll": "dc8566972b22cfadb0958a2a017b85544c6c8137582a26986d9854d943409555", + "Foundatio.dll": "1a7a4d1b07ceb2ec3bc59a92734e50d4ba807ac42bbb3d5fe69d9977f42879ac" + } + }, + "foundatioSourceSha256": "753f5b39cda0093978f6c95bf35b13477382585ac0e082f4dbdfa73ba6387877", + "mediatorSourceSha256": "c22c0994c0c6b261cfc55428c3e94ae4e445137ca13b658f38b87b74331184b8" +} diff --git a/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/matrix-results.json b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/matrix-results.json new file mode 100644 index 000000000..32eabbc71 --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/matrix-results.json @@ -0,0 +1,2286 @@ +[ + { + "Case": "pr149", + "Scenario": "memory-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 154, + "Gen1Collections": 34, + "Gen2Collections": 6, + "ElapsedMilliseconds": 1178.3667, + "MessagesPerSecond": 169726.45272477574, + "AllocatedBytesPerMessage": 6157.75652, + "CpuMilliseconds": 11779.863, + "AcceptanceP50Milliseconds": 0.0118, + "AcceptanceP99Milliseconds": 0.2202, + "HandlerCompletionP50Milliseconds": 352.1398, + "HandlerCompletionP99Milliseconds": 481.3709, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 240, + "Gen1Collections": 41, + "Gen2Collections": 4, + "ElapsedMilliseconds": 2650.4453, + "MessagesPerSecond": 75459.01815064812, + "AllocatedBytesPerMessage": 9855.72168, + "CpuMilliseconds": 13021.862, + "AcceptanceP50Milliseconds": 0.0213, + "AcceptanceP99Milliseconds": 2.4833, + "HandlerCompletionP50Milliseconds": 1400.7308, + "HandlerCompletionP99Milliseconds": 1601.4891, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 205, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 2474.9427, + "MessagesPerSecond": 80809.95168090153, + "AllocatedBytesPerMessage": 8408.3664, + "CpuMilliseconds": 12562.695, + "AcceptanceP50Milliseconds": 0.0176, + "AcceptanceP99Milliseconds": 0.4614, + "HandlerCompletionP50Milliseconds": 1446.3133, + "HandlerCompletionP99Milliseconds": 1507.3442, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-tracked-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 72, + "Gen1Collections": 19, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1724.1456, + "MessagesPerSecond": 28999.871008573755, + "AllocatedBytesPerMessage": 11488.36608, + "CpuMilliseconds": 6663.817, + "AcceptanceP50Milliseconds": 0.0191, + "AcceptanceP99Milliseconds": 6.7697, + "HandlerCompletionP50Milliseconds": 0.1359, + "HandlerCompletionP99Milliseconds": 10.3042, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 103, + "Gen1Collections": 32, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1831.9427, + "MessagesPerSecond": 27293.43008381212, + "AllocatedBytesPerMessage": 17060.01184, + "CpuMilliseconds": 7257.147, + "AcceptanceP50Milliseconds": 0.0214, + "AcceptanceP99Milliseconds": 5.8487, + "HandlerCompletionP50Milliseconds": 925.6496, + "HandlerCompletionP99Milliseconds": 1238.9402, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 94, + "Gen1Collections": 29, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1712.0209, + "MessagesPerSecond": 29205.25094056971, + "AllocatedBytesPerMessage": 15546.3856, + "CpuMilliseconds": 7170.44, + "AcceptanceP50Milliseconds": 0.0192, + "AcceptanceP99Milliseconds": 6.7307, + "HandlerCompletionP50Milliseconds": 890.7043, + "HandlerCompletionP99Milliseconds": 1125.8105, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-1", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 80, + "Gen1Collections": 20, + "Gen2Collections": 6, + "ElapsedMilliseconds": 1263.3081, + "MessagesPerSecond": 79157.2538797147, + "AllocatedBytesPerMessage": 6414.67712, + "CpuMilliseconds": 4275.575, + "AcceptanceP50Milliseconds": 0.0038, + "AcceptanceP99Milliseconds": 0.01, + "HandlerCompletionP50Milliseconds": 446.5852, + "HandlerCompletionP99Milliseconds": 705.9882, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-1", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 132, + "Gen1Collections": 34, + "Gen2Collections": 11, + "ElapsedMilliseconds": 1489.308, + "MessagesPerSecond": 67145.27820974572, + "AllocatedBytesPerMessage": 10203.46248, + "CpuMilliseconds": 5137.212, + "AcceptanceP50Milliseconds": 0.0077, + "AcceptanceP99Milliseconds": 0.0189, + "HandlerCompletionP50Milliseconds": 230.6888, + "HandlerCompletionP99Milliseconds": 387.9036, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-1", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 111, + "Gen1Collections": 29, + "Gen2Collections": 9, + "ElapsedMilliseconds": 1380.7662, + "MessagesPerSecond": 72423.55729739039, + "AllocatedBytesPerMessage": 8674.1752, + "CpuMilliseconds": 4533.455, + "AcceptanceP50Milliseconds": 0.0074, + "AcceptanceP99Milliseconds": 0.017, + "HandlerCompletionP50Milliseconds": 230.8049, + "HandlerCompletionP99Milliseconds": 372.8586, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-8", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 77, + "Gen1Collections": 18, + "Gen2Collections": 6, + "ElapsedMilliseconds": 639.1295, + "MessagesPerSecond": 156462.81387418354, + "AllocatedBytesPerMessage": 6124.43104, + "CpuMilliseconds": 5585.004, + "AcceptanceP50Milliseconds": 0.0124, + "AcceptanceP99Milliseconds": 0.0639, + "HandlerCompletionP50Milliseconds": 252.4609, + "HandlerCompletionP99Milliseconds": 289.556, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-8", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 120, + "Gen1Collections": 22, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1493.9409, + "MessagesPerSecond": 66937.05219530438, + "AllocatedBytesPerMessage": 9837.79808, + "CpuMilliseconds": 5985.748, + "AcceptanceP50Milliseconds": 0.0175, + "AcceptanceP99Milliseconds": 0.0535, + "HandlerCompletionP50Milliseconds": 821.9459, + "HandlerCompletionP99Milliseconds": 901.3391, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-8", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 102, + "Gen1Collections": 21, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1299.345, + "MessagesPerSecond": 76961.8538571357, + "AllocatedBytesPerMessage": 8332.54816, + "CpuMilliseconds": 4948.52, + "AcceptanceP50Milliseconds": 0.0134, + "AcceptanceP99Milliseconds": 0.0362, + "HandlerCompletionP50Milliseconds": 602.1462, + "HandlerCompletionP99Milliseconds": 782.2498, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 48, + "Gen1Collections": 11, + "Gen2Collections": 0, + "ElapsedMilliseconds": 4214.07, + "MessagesPerSecond": 2373.0028215003545, + "AllocatedBytesPerMessage": 39811.8672, + "CpuMilliseconds": 6126.34, + "AcceptanceP50Milliseconds": 13.0704, + "AcceptanceP99Milliseconds": 23.739, + "HandlerCompletionP50Milliseconds": 1693.6236, + "HandlerCompletionP99Milliseconds": 2118.3536, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 59, + "Gen1Collections": 54, + "Gen2Collections": 0, + "ElapsedMilliseconds": 4702.7512, + "MessagesPerSecond": 2126.4148526505082, + "AllocatedBytesPerMessage": 48113.2744, + "CpuMilliseconds": 8014.756, + "AcceptanceP50Milliseconds": 20.1457, + "AcceptanceP99Milliseconds": 43.5537, + "HandlerCompletionP50Milliseconds": 898.1532, + "HandlerCompletionP99Milliseconds": 1568.929, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 58, + "Gen1Collections": 54, + "Gen2Collections": 0, + "ElapsedMilliseconds": 4590.6866, + "MessagesPerSecond": 2178.32339066666, + "AllocatedBytesPerMessage": 47132.8736, + "CpuMilliseconds": 7507.05, + "AcceptanceP50Milliseconds": 22.1378, + "AcceptanceP99Milliseconds": 40.5698, + "HandlerCompletionP50Milliseconds": 852.6172, + "HandlerCompletionP99Milliseconds": 1203.5015, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 33, + "Gen1Collections": 11, + "Gen2Collections": 2, + "ElapsedMilliseconds": 1386.1782, + "MessagesPerSecond": 7214.0796904755825, + "AllocatedBytesPerMessage": 25834.2872, + "CpuMilliseconds": 5230.924, + "AcceptanceP50Milliseconds": 3.8807, + "AcceptanceP99Milliseconds": 12.0769, + "HandlerCompletionP50Milliseconds": 534.1799, + "HandlerCompletionP99Milliseconds": 713.6035, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 60, + "Gen1Collections": 29, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1951.4235, + "MessagesPerSecond": 5124.464269288547, + "AllocatedBytesPerMessage": 47344.3792, + "CpuMilliseconds": 7055.597, + "AcceptanceP50Milliseconds": 6.7135, + "AcceptanceP99Milliseconds": 13.4074, + "HandlerCompletionP50Milliseconds": 786.1105, + "HandlerCompletionP99Milliseconds": 886.3895, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 60, + "Gen1Collections": 25, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1916.1216, + "MessagesPerSecond": 5218.875461766101, + "AllocatedBytesPerMessage": 45822.024, + "CpuMilliseconds": 6868.575, + "AcceptanceP50Milliseconds": 6.3469, + "AcceptanceP99Milliseconds": 13.3645, + "HandlerCompletionP50Milliseconds": 749.5329, + "HandlerCompletionP99Milliseconds": 881.4759, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 74, + "Gen1Collections": 30, + "Gen2Collections": 1, + "ElapsedMilliseconds": 4352.972, + "MessagesPerSecond": 2297.281030064057, + "AllocatedBytesPerMessage": 59376.4912, + "CpuMilliseconds": 10622.938, + "AcceptanceP50Milliseconds": 13.5672, + "AcceptanceP99Milliseconds": 26.2495, + "HandlerCompletionP50Milliseconds": 1836.2369, + "HandlerCompletionP99Milliseconds": 2116.149, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 107, + "Gen1Collections": 96, + "Gen2Collections": 1, + "ElapsedMilliseconds": 3961.8261, + "MessagesPerSecond": 2524.0885762249886, + "AllocatedBytesPerMessage": 85382.388, + "CpuMilliseconds": 12609.14, + "AcceptanceP50Milliseconds": 16.1713, + "AcceptanceP99Milliseconds": 33.1027, + "HandlerCompletionP50Milliseconds": 1287.655, + "HandlerCompletionP99Milliseconds": 1430.3803, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 106, + "Gen1Collections": 86, + "Gen2Collections": 1, + "ElapsedMilliseconds": 4233.2111, + "MessagesPerSecond": 2362.2729327153093, + "AllocatedBytesPerMessage": 84388.3528, + "CpuMilliseconds": 12579.16, + "AcceptanceP50Milliseconds": 15.1899, + "AcceptanceP99Milliseconds": 32.3942, + "HandlerCompletionP50Milliseconds": 1246.1519, + "HandlerCompletionP99Milliseconds": 1549.0508, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 240, + "Gen1Collections": 41, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1998.5407, + "MessagesPerSecond": 100073.0182777864, + "AllocatedBytesPerMessage": 9847.4312, + "CpuMilliseconds": 11357.305, + "AcceptanceP50Milliseconds": 0.0208, + "AcceptanceP99Milliseconds": 2.4138, + "HandlerCompletionP50Milliseconds": 1139.5458, + "HandlerCompletionP99Milliseconds": 1177.0049, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 204, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1887.6812, + "MessagesPerSecond": 105950.09369166785, + "AllocatedBytesPerMessage": 8353.43268, + "CpuMilliseconds": 10593.069, + "AcceptanceP50Milliseconds": 0.0173, + "AcceptanceP99Milliseconds": 0.138, + "HandlerCompletionP50Milliseconds": 1080.2072, + "HandlerCompletionP99Milliseconds": 1155.0116, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 154, + "Gen1Collections": 31, + "Gen2Collections": 5, + "ElapsedMilliseconds": 1080.4544, + "MessagesPerSecond": 185107.30300140384, + "AllocatedBytesPerMessage": 6191.86752, + "CpuMilliseconds": 11309.699, + "AcceptanceP50Milliseconds": 0.0109, + "AcceptanceP99Milliseconds": 0.1818, + "HandlerCompletionP50Milliseconds": 390.2636, + "HandlerCompletionP99Milliseconds": 409.5892, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 103, + "Gen1Collections": 31, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1610.3741, + "MessagesPerSecond": 31048.6861406924, + "AllocatedBytesPerMessage": 17049.2208, + "CpuMilliseconds": 5771.844, + "AcceptanceP50Milliseconds": 0.0235, + "AcceptanceP99Milliseconds": 4.9612, + "HandlerCompletionP50Milliseconds": 810.8437, + "HandlerCompletionP99Milliseconds": 982.6008, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 95, + "Gen1Collections": 29, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1634.8283, + "MessagesPerSecond": 30584.251569415577, + "AllocatedBytesPerMessage": 15532.74288, + "CpuMilliseconds": 6208.545, + "AcceptanceP50Milliseconds": 0.021, + "AcceptanceP99Milliseconds": 5.453, + "HandlerCompletionP50Milliseconds": 849.0993, + "HandlerCompletionP99Milliseconds": 1040.692, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-tracked-64", + "Iteration": 1, + "Attempt": 0, + "Exit": -6, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ] + }, + { + "Case": "pr149", + "Scenario": "memory-tracked-64", + "Iteration": 1, + "Attempt": 1, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 72, + "Gen1Collections": 19, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1346.9747, + "MessagesPerSecond": 37120.22207989504, + "AllocatedBytesPerMessage": 11470.21888, + "CpuMilliseconds": 5886.6, + "AcceptanceP50Milliseconds": 0.0181, + "AcceptanceP99Milliseconds": 4.0859, + "HandlerCompletionP50Milliseconds": 0.1573, + "HandlerCompletionP99Milliseconds": 6.8731, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-1", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 133, + "Gen1Collections": 35, + "Gen2Collections": 12, + "ElapsedMilliseconds": 1175.7487, + "MessagesPerSecond": 85052.1884480927, + "AllocatedBytesPerMessage": 10195.1076, + "CpuMilliseconds": 3702.885, + "AcceptanceP50Milliseconds": 0.0062, + "AcceptanceP99Milliseconds": 0.0128, + "HandlerCompletionP50Milliseconds": 168.1407, + "HandlerCompletionP99Milliseconds": 321.3564, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-1", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 113, + "Gen1Collections": 30, + "Gen2Collections": 11, + "ElapsedMilliseconds": 1142.3538, + "MessagesPerSecond": 87538.55416771933, + "AllocatedBytesPerMessage": 8684.63928, + "CpuMilliseconds": 3636.179, + "AcceptanceP50Milliseconds": 0.0053, + "AcceptanceP99Milliseconds": 0.0155, + "HandlerCompletionP50Milliseconds": 250.6152, + "HandlerCompletionP99Milliseconds": 334.3605, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-1", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 80, + "Gen1Collections": 19, + "Gen2Collections": 6, + "ElapsedMilliseconds": 977.7009, + "MessagesPerSecond": 102280.76909819762, + "AllocatedBytesPerMessage": 6414.18424, + "CpuMilliseconds": 3408.482, + "AcceptanceP50Milliseconds": 0.0035, + "AcceptanceP99Milliseconds": 0.008, + "HandlerCompletionP50Milliseconds": 337.2093, + "HandlerCompletionP99Milliseconds": 479.1208, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-8", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 122, + "Gen1Collections": 23, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1247.6731, + "MessagesPerSecond": 80149.19933755083, + "AllocatedBytesPerMessage": 9857.36896, + "CpuMilliseconds": 4846.627, + "AcceptanceP50Milliseconds": 0.0142, + "AcceptanceP99Milliseconds": 0.0434, + "HandlerCompletionP50Milliseconds": 538.2362, + "HandlerCompletionP99Milliseconds": 791.0268, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-8", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 102, + "Gen1Collections": 20, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1169.3384, + "MessagesPerSecond": 85518.44359169253, + "AllocatedBytesPerMessage": 8331.23872, + "CpuMilliseconds": 4647.947, + "AcceptanceP50Milliseconds": 0.0107, + "AcceptanceP99Milliseconds": 0.0318, + "HandlerCompletionP50Milliseconds": 518.8071, + "HandlerCompletionP99Milliseconds": 731.9423, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-8", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 76, + "Gen1Collections": 16, + "Gen2Collections": 3, + "ElapsedMilliseconds": 488.359, + "MessagesPerSecond": 204767.39447824244, + "AllocatedBytesPerMessage": 6127.48624, + "CpuMilliseconds": 4234.555, + "AcceptanceP50Milliseconds": 0.0078, + "AcceptanceP99Milliseconds": 0.0366, + "HandlerCompletionP50Milliseconds": 197.9846, + "HandlerCompletionP99Milliseconds": 210.3302, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 59, + "Gen1Collections": 54, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3560.2201, + "MessagesPerSecond": 2808.815106683994, + "AllocatedBytesPerMessage": 48010.9992, + "CpuMilliseconds": 6659.031, + "AcceptanceP50Milliseconds": 17.1905, + "AcceptanceP99Milliseconds": 31.2417, + "HandlerCompletionP50Milliseconds": 662.1692, + "HandlerCompletionP99Milliseconds": 907.0859, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 58, + "Gen1Collections": 54, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3513.5124, + "MessagesPerSecond": 2846.1547481659663, + "AllocatedBytesPerMessage": 46998.812, + "CpuMilliseconds": 6360.693, + "AcceptanceP50Milliseconds": 14.7487, + "AcceptanceP99Milliseconds": 29.7102, + "HandlerCompletionP50Milliseconds": 632.8914, + "HandlerCompletionP99Milliseconds": 760.3067, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 48, + "Gen1Collections": 14, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3468.8519, + "MessagesPerSecond": 2882.798196140919, + "AllocatedBytesPerMessage": 39822.4528, + "CpuMilliseconds": 5146.519, + "AcceptanceP50Milliseconds": 9.0116, + "AcceptanceP99Milliseconds": 17.2356, + "HandlerCompletionP50Milliseconds": 1407.2102, + "HandlerCompletionP99Milliseconds": 2000.4991, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 62, + "Gen1Collections": 26, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1695.4769, + "MessagesPerSecond": 5898.045558745152, + "AllocatedBytesPerMessage": 47359.7736, + "CpuMilliseconds": 6536.766, + "AcceptanceP50Milliseconds": 5.9702, + "AcceptanceP99Milliseconds": 12.3989, + "HandlerCompletionP50Milliseconds": 684.6769, + "HandlerCompletionP99Milliseconds": 739.5871, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 60, + "Gen1Collections": 24, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1848.7977, + "MessagesPerSecond": 5408.920619059619, + "AllocatedBytesPerMessage": 45854.1296, + "CpuMilliseconds": 6271.986, + "AcceptanceP50Milliseconds": 6.4358, + "AcceptanceP99Milliseconds": 13.1015, + "HandlerCompletionP50Milliseconds": 722.7141, + "HandlerCompletionP99Milliseconds": 813.6706, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 33, + "Gen1Collections": 10, + "Gen2Collections": 2, + "ElapsedMilliseconds": 1186.5015, + "MessagesPerSecond": 8428.139366026928, + "AllocatedBytesPerMessage": 25840.8128, + "CpuMilliseconds": 3959.085, + "AcceptanceP50Milliseconds": 3.3516, + "AcceptanceP99Milliseconds": 9.3911, + "HandlerCompletionP50Milliseconds": 507.6687, + "HandlerCompletionP99Milliseconds": 616.3296, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 107, + "Gen1Collections": 91, + "Gen2Collections": 1, + "ElapsedMilliseconds": 4442.505, + "MessagesPerSecond": 2250.982272389114, + "AllocatedBytesPerMessage": 85478.9872, + "CpuMilliseconds": 12919.947, + "AcceptanceP50Milliseconds": 16.1565, + "AcceptanceP99Milliseconds": 39.7854, + "HandlerCompletionP50Milliseconds": 1196.9691, + "HandlerCompletionP99Milliseconds": 1504.8471, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 106, + "Gen1Collections": 95, + "Gen2Collections": 1, + "ElapsedMilliseconds": 4090.228, + "MessagesPerSecond": 2444.851485051689, + "AllocatedBytesPerMessage": 84327.592, + "CpuMilliseconds": 12694.735, + "AcceptanceP50Milliseconds": 16.1131, + "AcceptanceP99Milliseconds": 32.458, + "HandlerCompletionP50Milliseconds": 1199.5962, + "HandlerCompletionP99Milliseconds": 1483.2424, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 74, + "Gen1Collections": 30, + "Gen2Collections": 1, + "ElapsedMilliseconds": 4143.0905, + "MessagesPerSecond": 2413.6571479672, + "AllocatedBytesPerMessage": 59380.2896, + "CpuMilliseconds": 10546.849, + "AcceptanceP50Milliseconds": 11.5121, + "AcceptanceP99Milliseconds": 23.7627, + "HandlerCompletionP50Milliseconds": 1610.904, + "HandlerCompletionP99Milliseconds": 1945.7299, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 204, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 2056.8528, + "MessagesPerSecond": 97235.93248870313, + "AllocatedBytesPerMessage": 8350.23448, + "CpuMilliseconds": 10911.157, + "AcceptanceP50Milliseconds": 0.0163, + "AcceptanceP99Milliseconds": 0.2, + "HandlerCompletionP50Milliseconds": 1180.3647, + "HandlerCompletionP99Milliseconds": 1273.3835, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 153, + "Gen1Collections": 30, + "Gen2Collections": 5, + "ElapsedMilliseconds": 998.3349, + "MessagesPerSecond": 200333.57543645924, + "AllocatedBytesPerMessage": 6148.91028, + "CpuMilliseconds": 11812.199, + "AcceptanceP50Milliseconds": 0.0106, + "AcceptanceP99Milliseconds": 0.1637, + "HandlerCompletionP50Milliseconds": 371.372, + "HandlerCompletionP99Milliseconds": 410.2119, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 240, + "Gen1Collections": 41, + "Gen2Collections": 5, + "ElapsedMilliseconds": 1958.0901, + "MessagesPerSecond": 102140.3458400612, + "AllocatedBytesPerMessage": 9837.96, + "CpuMilliseconds": 11112.135, + "AcceptanceP50Milliseconds": 0.0211, + "AcceptanceP99Milliseconds": 2.1176, + "HandlerCompletionP50Milliseconds": 1103.4667, + "HandlerCompletionP99Milliseconds": 1128.2168, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 95, + "Gen1Collections": 29, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1575.1161, + "MessagesPerSecond": 31743.691782466067, + "AllocatedBytesPerMessage": 15522.98016, + "CpuMilliseconds": 5679.902, + "AcceptanceP50Milliseconds": 0.02, + "AcceptanceP99Milliseconds": 5.2099, + "HandlerCompletionP50Milliseconds": 783.624, + "HandlerCompletionP99Milliseconds": 988.4902, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-tracked-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 72, + "Gen1Collections": 20, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1324.643, + "MessagesPerSecond": 37746.01911609392, + "AllocatedBytesPerMessage": 11476.55488, + "CpuMilliseconds": 5746.684, + "AcceptanceP50Milliseconds": 0.0175, + "AcceptanceP99Milliseconds": 3.9928, + "HandlerCompletionP50Milliseconds": 0.1906, + "HandlerCompletionP99Milliseconds": 6.7101, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 104, + "Gen1Collections": 32, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1574.166, + "MessagesPerSecond": 31762.85093185852, + "AllocatedBytesPerMessage": 17049.3408, + "CpuMilliseconds": 5831.661, + "AcceptanceP50Milliseconds": 0.0216, + "AcceptanceP99Milliseconds": 4.5501, + "HandlerCompletionP50Milliseconds": 789.7598, + "HandlerCompletionP99Milliseconds": 995.3391, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-1", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 111, + "Gen1Collections": 28, + "Gen2Collections": 9, + "ElapsedMilliseconds": 1109.0732, + "MessagesPerSecond": 90165.37411597358, + "AllocatedBytesPerMessage": 8684.8432, + "CpuMilliseconds": 3521.897, + "AcceptanceP50Milliseconds": 0.0051, + "AcceptanceP99Milliseconds": 0.0148, + "HandlerCompletionP50Milliseconds": 221.1587, + "HandlerCompletionP99Milliseconds": 334.1108, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-1", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 80, + "Gen1Collections": 21, + "Gen2Collections": 7, + "ElapsedMilliseconds": 905.3797, + "MessagesPerSecond": 110450.89701039244, + "AllocatedBytesPerMessage": 6411.63104, + "CpuMilliseconds": 3301.507, + "AcceptanceP50Milliseconds": 0.0035, + "AcceptanceP99Milliseconds": 0.011, + "HandlerCompletionP50Milliseconds": 298.8728, + "HandlerCompletionP99Milliseconds": 361.0386, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-1", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 132, + "Gen1Collections": 33, + "Gen2Collections": 11, + "ElapsedMilliseconds": 1229.3027, + "MessagesPerSecond": 81346.92944219515, + "AllocatedBytesPerMessage": 10195.22536, + "CpuMilliseconds": 4007.15, + "AcceptanceP50Milliseconds": 0.0063, + "AcceptanceP99Milliseconds": 0.0135, + "HandlerCompletionP50Milliseconds": 234.873, + "HandlerCompletionP99Milliseconds": 354.4401, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-8", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 102, + "Gen1Collections": 20, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1180.8514, + "MessagesPerSecond": 84684.66057625879, + "AllocatedBytesPerMessage": 8334.59216, + "CpuMilliseconds": 4838.818, + "AcceptanceP50Milliseconds": 0.0124, + "AcceptanceP99Milliseconds": 0.0335, + "HandlerCompletionP50Milliseconds": 534.2955, + "HandlerCompletionP99Milliseconds": 739.1415, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-8", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 76, + "Gen1Collections": 15, + "Gen2Collections": 3, + "ElapsedMilliseconds": 555.512, + "MessagesPerSecond": 180014.11310646756, + "AllocatedBytesPerMessage": 6129.59312, + "CpuMilliseconds": 5059.714, + "AcceptanceP50Milliseconds": 0.01, + "AcceptanceP99Milliseconds": 0.0398, + "HandlerCompletionP50Milliseconds": 222.5953, + "HandlerCompletionP99Milliseconds": 245.8558, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-8", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 122, + "Gen1Collections": 24, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1226.8028, + "MessagesPerSecond": 81512.69299352757, + "AllocatedBytesPerMessage": 9851.80656, + "CpuMilliseconds": 4959.305, + "AcceptanceP50Milliseconds": 0.0133, + "AcceptanceP99Milliseconds": 0.0395, + "HandlerCompletionP50Milliseconds": 556.7031, + "HandlerCompletionP99Milliseconds": 777.6549, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 58, + "Gen1Collections": 50, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3318.4293, + "MessagesPerSecond": 3013.4738745225036, + "AllocatedBytesPerMessage": 47012.9912, + "CpuMilliseconds": 6174.988, + "AcceptanceP50Milliseconds": 16.178, + "AcceptanceP99Milliseconds": 31.8364, + "HandlerCompletionP50Milliseconds": 644.7194, + "HandlerCompletionP99Milliseconds": 769.7932, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 48, + "Gen1Collections": 36, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3462.7585, + "MessagesPerSecond": 2887.8710426961625, + "AllocatedBytesPerMessage": 39817.7024, + "CpuMilliseconds": 5117.143, + "AcceptanceP50Milliseconds": 9.5728, + "AcceptanceP99Milliseconds": 17.7707, + "HandlerCompletionP50Milliseconds": 1416.6221, + "HandlerCompletionP99Milliseconds": 1895.388, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.11", + "TrackingStore": "memory", + "Gen0Collections": 59, + "Gen1Collections": 52, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3104.1858, + "MessagesPerSecond": 3221.4566537866385, + "AllocatedBytesPerMessage": 47994.2192, + "CpuMilliseconds": 6239.516, + "AcceptanceP50Milliseconds": 14.3461, + "AcceptanceP99Milliseconds": 26.1227, + "HandlerCompletionP50Milliseconds": 587.5785, + "HandlerCompletionP99Milliseconds": 766.6391, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 60, + "Gen1Collections": 25, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1533.3584, + "MessagesPerSecond": 6521.632515920609, + "AllocatedBytesPerMessage": 45830.62, + "CpuMilliseconds": 6008.182, + "AcceptanceP50Milliseconds": 4.608, + "AcceptanceP99Milliseconds": 9.8602, + "HandlerCompletionP50Milliseconds": 574.9345, + "HandlerCompletionP99Milliseconds": 777.3787, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 33, + "Gen1Collections": 11, + "Gen2Collections": 2, + "ElapsedMilliseconds": 1135.011, + "MessagesPerSecond": 8810.487299242033, + "AllocatedBytesPerMessage": 25863.208, + "CpuMilliseconds": 4477.253, + "AcceptanceP50Milliseconds": 3.1196, + "AcceptanceP99Milliseconds": 10.3854, + "HandlerCompletionP50Milliseconds": 478.5586, + "HandlerCompletionP99Milliseconds": 588.8377, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 62, + "Gen1Collections": 26, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1499.653, + "MessagesPerSecond": 6668.209245738848, + "AllocatedBytesPerMessage": 47386.2624, + "CpuMilliseconds": 6230.305, + "AcceptanceP50Milliseconds": 4.8674, + "AcceptanceP99Milliseconds": 9.9498, + "HandlerCompletionP50Milliseconds": 606.967, + "HandlerCompletionP99Milliseconds": 699.2259, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 106, + "Gen1Collections": 94, + "Gen2Collections": 1, + "ElapsedMilliseconds": 3789.639, + "MessagesPerSecond": 2638.7737723830687, + "AllocatedBytesPerMessage": 84583.02, + "CpuMilliseconds": 12945.087, + "AcceptanceP50Milliseconds": 15.4452, + "AcceptanceP99Milliseconds": 29.4407, + "HandlerCompletionP50Milliseconds": 1024.4032, + "HandlerCompletionP99Milliseconds": 1299.1594, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 74, + "Gen1Collections": 29, + "Gen2Collections": 1, + "ElapsedMilliseconds": 3556.9563, + "MessagesPerSecond": 2811.392425597132, + "AllocatedBytesPerMessage": 59402.9472, + "CpuMilliseconds": 10540.743, + "AcceptanceP50Milliseconds": 10.0392, + "AcceptanceP99Milliseconds": 20.3527, + "HandlerCompletionP50Milliseconds": 1513.7194, + "HandlerCompletionP99Milliseconds": 1904.2094, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 107, + "Gen1Collections": 97, + "Gen2Collections": 1, + "ElapsedMilliseconds": 4003.7668, + "MessagesPerSecond": 2497.6479649114426, + "AllocatedBytesPerMessage": 85530.4776, + "CpuMilliseconds": 12705.587, + "AcceptanceP50Milliseconds": 15.9412, + "AcceptanceP99Milliseconds": 30.9604, + "HandlerCompletionP50Milliseconds": 1254.8382, + "HandlerCompletionP99Milliseconds": 1411.4109, + "UniqueProcessed": 10000, + "Duplicates": 0 + } +] diff --git a/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/matrix-summary.json b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/matrix-summary.json new file mode 100644 index 000000000..a4d14d5c6 --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/matrix-summary.json @@ -0,0 +1,163 @@ +{ + "memory-64": { + "pr149": { + "MessagesPerSecond": 185107.303, + "AllocatedBytesPerMessage": 6157.757, + "AcceptanceP99Milliseconds": 0.182, + "HandlerCompletionP99Milliseconds": 410.212, + "CpuMilliseconds": 11779.863 + }, + "before": { + "MessagesPerSecond": 100073.018, + "AllocatedBytesPerMessage": 9847.431, + "AcceptanceP99Milliseconds": 2.414, + "HandlerCompletionP99Milliseconds": 1177.005, + "CpuMilliseconds": 11357.305 + }, + "after": { + "MessagesPerSecond": 97235.932, + "AllocatedBytesPerMessage": 8353.433, + "AcceptanceP99Milliseconds": 0.2, + "HandlerCompletionP99Milliseconds": 1273.383, + "CpuMilliseconds": 10911.157 + } + }, + "memory-tracked-64": { + "pr149": { + "MessagesPerSecond": 37120.222, + "AllocatedBytesPerMessage": 11476.555, + "AcceptanceP99Milliseconds": 4.086, + "HandlerCompletionP99Milliseconds": 6.873, + "CpuMilliseconds": 5886.6 + }, + "before": { + "MessagesPerSecond": 31048.686, + "AllocatedBytesPerMessage": 17049.341, + "AcceptanceP99Milliseconds": 4.961, + "HandlerCompletionP99Milliseconds": 995.339, + "CpuMilliseconds": 5831.661 + }, + "after": { + "MessagesPerSecond": 30584.252, + "AllocatedBytesPerMessage": 15532.743, + "AcceptanceP99Milliseconds": 5.453, + "HandlerCompletionP99Milliseconds": 1040.692, + "CpuMilliseconds": 6208.545 + } + }, + "memory-1": { + "pr149": { + "MessagesPerSecond": 102280.769, + "AllocatedBytesPerMessage": 6414.184, + "AcceptanceP99Milliseconds": 0.01, + "HandlerCompletionP99Milliseconds": 479.121, + "CpuMilliseconds": 3408.482 + }, + "before": { + "MessagesPerSecond": 81346.929, + "AllocatedBytesPerMessage": 10195.225, + "AcceptanceP99Milliseconds": 0.013, + "HandlerCompletionP99Milliseconds": 354.44, + "CpuMilliseconds": 4007.15 + }, + "after": { + "MessagesPerSecond": 87538.554, + "AllocatedBytesPerMessage": 8684.639, + "AcceptanceP99Milliseconds": 0.015, + "HandlerCompletionP99Milliseconds": 334.361, + "CpuMilliseconds": 3636.179 + } + }, + "memory-8": { + "pr149": { + "MessagesPerSecond": 180014.113, + "AllocatedBytesPerMessage": 6127.486, + "AcceptanceP99Milliseconds": 0.04, + "HandlerCompletionP99Milliseconds": 245.856, + "CpuMilliseconds": 5059.714 + }, + "before": { + "MessagesPerSecond": 80149.199, + "AllocatedBytesPerMessage": 9851.807, + "AcceptanceP99Milliseconds": 0.043, + "HandlerCompletionP99Milliseconds": 791.027, + "CpuMilliseconds": 4959.305 + }, + "after": { + "MessagesPerSecond": 84684.661, + "AllocatedBytesPerMessage": 8332.548, + "AcceptanceP99Milliseconds": 0.034, + "HandlerCompletionP99Milliseconds": 739.141, + "CpuMilliseconds": 4838.818 + } + }, + "localstack-64": { + "pr149": { + "MessagesPerSecond": 2882.798, + "AllocatedBytesPerMessage": 39817.702, + "AcceptanceP99Milliseconds": 17.771, + "HandlerCompletionP99Milliseconds": 2000.499, + "CpuMilliseconds": 5146.519 + }, + "before": { + "MessagesPerSecond": 2808.815, + "AllocatedBytesPerMessage": 48010.999, + "AcceptanceP99Milliseconds": 31.242, + "HandlerCompletionP99Milliseconds": 907.086, + "CpuMilliseconds": 6659.031 + }, + "after": { + "MessagesPerSecond": 2846.155, + "AllocatedBytesPerMessage": 47012.991, + "AcceptanceP99Milliseconds": 31.836, + "HandlerCompletionP99Milliseconds": 769.793, + "CpuMilliseconds": 6360.693 + } + }, + "redis-64": { + "pr149": { + "MessagesPerSecond": 8428.139, + "AllocatedBytesPerMessage": 25840.813, + "AcceptanceP99Milliseconds": 10.385, + "HandlerCompletionP99Milliseconds": 616.33, + "CpuMilliseconds": 4477.253 + }, + "before": { + "MessagesPerSecond": 5898.046, + "AllocatedBytesPerMessage": 47359.774, + "AcceptanceP99Milliseconds": 12.399, + "HandlerCompletionP99Milliseconds": 739.587, + "CpuMilliseconds": 6536.766 + }, + "after": { + "MessagesPerSecond": 5408.921, + "AllocatedBytesPerMessage": 45830.62, + "AcceptanceP99Milliseconds": 13.101, + "HandlerCompletionP99Milliseconds": 813.671, + "CpuMilliseconds": 6271.986 + } + }, + "localstack-redis-64": { + "pr149": { + "MessagesPerSecond": 2413.657, + "AllocatedBytesPerMessage": 59380.29, + "AcceptanceP99Milliseconds": 23.763, + "HandlerCompletionP99Milliseconds": 1945.73, + "CpuMilliseconds": 10546.849 + }, + "before": { + "MessagesPerSecond": 2497.648, + "AllocatedBytesPerMessage": 85478.987, + "AcceptanceP99Milliseconds": 33.103, + "HandlerCompletionP99Milliseconds": 1430.38, + "CpuMilliseconds": 12705.587 + }, + "after": { + "MessagesPerSecond": 2444.851, + "AllocatedBytesPerMessage": 84388.353, + "AcceptanceP99Milliseconds": 32.394, + "HandlerCompletionP99Milliseconds": 1483.242, + "CpuMilliseconds": 12694.735 + } + } +} diff --git a/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/mediator-source-hashes.json b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/mediator-source-hashes.json new file mode 100644 index 000000000..010ca12d6 --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/mediator-source-hashes.json @@ -0,0 +1,150 @@ +{ + "src/Foundatio.Mediator.Abstractions/AuthorizationRequirements.cs": "474f67f73067b9dd8260bde3427a3aff27d0a281db5b278bb7dc01ab055dc97e", + "src/Foundatio.Mediator.Abstractions/AuthorizationResult.cs": "fe77221291743dd8cd6aba9e3dd070ef60ea7529ecfb8f436e6cf757a61ecb81", + "src/Foundatio.Mediator.Abstractions/CallContext.cs": "ef99d33ba519139e0e0605d0e47e7701d857f975622582e3129ad45851abe311", + "src/Foundatio.Mediator.Abstractions/DefaultAuthorizationContextProvider.cs": "e5236866be1001d6277a7ad1fdc2a5b3f04527d68b8ab676de68175a51f047eb", + "src/Foundatio.Mediator.Abstractions/DefaultHandlerAuthorizationService.cs": "d65bf9d12b7c30ab7babd2277fe0be57d717ce9899d4e4d1d7788c9880ba35cb", + "src/Foundatio.Mediator.Abstractions/EndpointDiscovery.cs": "f0a8470a2ecbe12adb32a88eb576ac1e04c509eba6f3b6373b08c6b80f0148d2", + "src/Foundatio.Mediator.Abstractions/EndpointStreaming.cs": "690cae10c37ab170346c57a5de33ebeaec60ff34e57d132ff44c4c6c8f00194b", + "src/Foundatio.Mediator.Abstractions/EndpointSummaryStyle.cs": "afb5ede3525d4ead58e9a28b2b0f280da335240698ca252283f6603b68cabb7c", + "src/Foundatio.Mediator.Abstractions/FileResult.cs": "24d3943295b4999304fd99ed1b3fc57ea305e04440acf0445a85558f2c6da7c5", + "src/Foundatio.Mediator.Abstractions/Foundatio.Mediator.Abstractions.csproj": "522c3b2e7ab988e8f1d215b6815bfdb8ce269815a435593bef7009a34d61edc9", + "src/Foundatio.Mediator.Abstractions/FoundatioIgnoreAttribute.cs": "01aec34ef39c0c280bf97de7fba8de8a33007143f2713d8bbf2914d29b48b3a0", + "src/Foundatio.Mediator.Abstractions/FoundatioModuleAttribute.cs": "60ae843ec8042e7b89002f9947efbb67ef19e69f79eb586620a7264c7875dafc", + "src/Foundatio.Mediator.Abstractions/GlobalUsings.cs": "f958e01af3ce3424d7e515da0c9a4ab4dc56707e772140ace4a8bc3995f96af4", + "src/Foundatio.Mediator.Abstractions/HandlerAllowAnonymousAttribute.cs": "dc3c8d842111cddbf68df77b883bb54a3b734bfaa0796e0afb8ced11cdf42e4a", + "src/Foundatio.Mediator.Abstractions/HandlerAttribute.cs": "b3879e6f2c29984307a714b9466a01b1f33d7066fc6994c5a49680c74e482fda", + "src/Foundatio.Mediator.Abstractions/HandlerAttributeMetadata.cs": "bb8324b5bbe0708fd240f8a626eddc02aa6a320cde42aa2ab50764f78659d43b", + "src/Foundatio.Mediator.Abstractions/HandlerAuthorizeAttribute.cs": "cc7d8ca4a0616a17f1bc26f8667273cb05d534ff89cd7995457b01eebcdb7064", + "src/Foundatio.Mediator.Abstractions/HandlerDiscovery.cs": "d17f8b3464d87a7c053ae42cad2ae827bb5680bb14c56b92d8d755ae86655b8f", + "src/Foundatio.Mediator.Abstractions/HandlerEndpointAttribute.cs": "93e4a853b19f38c4ba7d48898d1b54ec606b50bf610243822b4b5ef72c7fcfd0", + "src/Foundatio.Mediator.Abstractions/HandlerEndpointGroupAttribute.cs": "c919efbce710795a9088fb5502e3d7e829bd68aa33e07188198a7511ef6874f3", + "src/Foundatio.Mediator.Abstractions/HandlerExecutionDelegate.cs": "a508b206a59d7f1b1df0b64b50fe78fb8900b44a2cc6c14dea3de3859f597e5c", + "src/Foundatio.Mediator.Abstractions/HandlerExecutionInfo.cs": "3bd9bbe38e72b8b7ab41dc7fd957190ecbe4fcfd6e0b3d65dd80fea50545e35c", + "src/Foundatio.Mediator.Abstractions/HandlerMethod.cs": "c80dcf0fa56d5decdefdff5c09c56128822518e923f3c686eb5e413246f4090a", + "src/Foundatio.Mediator.Abstractions/HandlerRegistration.cs": "db2da1639e7a3a42f9a96f21189c0e1ed86690a2598a9e41c22cd380db5d39eb", + "src/Foundatio.Mediator.Abstractions/HandlerRegistry.cs": "4852987ffaa99f2a187ce3d0c6bc4ff4ceb63acf28878041f91022e220360d32", + "src/Foundatio.Mediator.Abstractions/HandlerResult.cs": "bdb95cffc4e4b9e0d492995de1a5898f2bc8dd61edc0b1ea90bf6547ab7ade13", + "src/Foundatio.Mediator.Abstractions/IAuthorizationContextProvider.cs": "f75a0319adea5d5853e0702a26a1808dd5d152defb10e95ca4350bf404cf8ebd", + "src/Foundatio.Mediator.Abstractions/ICommand.cs": "6f753d77bbf65f4d4a157f144589821cc59fcadbf11de68998c9c4325ff58c62", + "src/Foundatio.Mediator.Abstractions/IEndpointConvention.cs": "d51ca6b09bb60cb9dad8d5ad87dc34de1a784e660948e6e8f1751281fb765a94", + "src/Foundatio.Mediator.Abstractions/IHandler.cs": "de892860c902a60774bcea29ae1cbd2fdc1c5171b7eebe7db9f578122f676e95", + "src/Foundatio.Mediator.Abstractions/IHandlerAuthorizationService.cs": "0ff9cb309c3c49285be26d6964d2c45485aee01df4cb162b229a1821f6018fa0", + "src/Foundatio.Mediator.Abstractions/IMediator.cs": "73c0f765991abbd87530a03abadf2b636cd4367a30a81e50490614feed89fc82", + "src/Foundatio.Mediator.Abstractions/IMediatorBuilder.cs": "a38336a5eae485b64810dc72c0776aca8138f47aab884ff3ff118196c91057aa", + "src/Foundatio.Mediator.Abstractions/IMediatorResultMapper.cs": "ccdf9f61313b48b64a53dffb3fd91bb849230baf09d2fc1abf037fae009e26ec", + "src/Foundatio.Mediator.Abstractions/INotification.cs": "51ebb8d039a95780297481776d4df59680ecf9e30865d3648e6c3a52b5ae1e96", + "src/Foundatio.Mediator.Abstractions/INotificationPublisher.cs": "323f6105cba974ba29aaa28c7fb98287e5edab81ab07e91198b375fdef986b39", + "src/Foundatio.Mediator.Abstractions/IQuery.cs": "123d0ccea446b504a921da503a6095ee2823050da6fcc0a1a0499eb811388214", + "src/Foundatio.Mediator.Abstractions/IRequest.cs": "cd3a6f5031387e37a9c544a0df8a50a6d739e8ae4abb6cd341c3f44849f82ebe", + "src/Foundatio.Mediator.Abstractions/IResult.cs": "bfdd605cc74d0f7af850b10ff00895d55005a946ad66f740af66fdde42053dc3", + "src/Foundatio.Mediator.Abstractions/IsExternalInit.cs": "86c21d23c8e87177dcfc3806390755ecc59620688f841d6cf0184d95ca383750", + "src/Foundatio.Mediator.Abstractions/MaybeNullAttribute.cs": "5857fc2076ac9523d222b5b3a08e53ddaa8433b8ae987d25e66cd28576b4f826", + "src/Foundatio.Mediator.Abstractions/Mediator.cs": "9e592b74a459746baaa878d5cb0e69a70060fdffa41a719c0b9c56365f371538", + "src/Foundatio.Mediator.Abstractions/MediatorActivitySource.cs": "f5089fdcbddf7b0ff08da047a75b62a45262e01fb08eba68a87ff95928f48bcc", + "src/Foundatio.Mediator.Abstractions/MediatorConfigurationAttribute.cs": "aef5d857d2cfb384b6788e913a6741852c15812bcf416e2702471587c0da3399", + "src/Foundatio.Mediator.Abstractions/MediatorEndpointGroupAttribute.cs": "b13c84d53af7ff42b185b3ca217e1a9443c33896753ecc0e4200cbd2b4540769", + "src/Foundatio.Mediator.Abstractions/MediatorEndpointOptions.cs": "5ee303338402c5955ce5ffbf499c51c4db8cd04d36aa47493fd9e5fa20e6eb29", + "src/Foundatio.Mediator.Abstractions/MediatorExtensions.cs": "3e9be699b7ac34f636fadafddb7e33bd0f3d8067f70172a46f25a154c97a4682", + "src/Foundatio.Mediator.Abstractions/MediatorLifetime.cs": "6a1613a694b738faf74745fe8273c87f07d2514beb589b18046be346fb8caea2", + "src/Foundatio.Mediator.Abstractions/MediatorOptions.cs": "519c3d0ecb3e10e217c3a8f47c12a11ed3fe8cc13e10c4d4dba6516832892d56", + "src/Foundatio.Mediator.Abstractions/MediatorResultMapperOptions.cs": "70211b600c031e269ae71b60bd788228d665313f81c003f4f5af8859abda909f", + "src/Foundatio.Mediator.Abstractions/MessageContext.cs": "2eea0b8a9739ba70e0c697c2c85f141cdb992f1d65be034f7bfc1cc3dc9414af", + "src/Foundatio.Mediator.Abstractions/MessageTypeKey.cs": "ed665fda285a81a36a4b5bbe088de242f6f1256220bde50d261efbc6659db370", + "src/Foundatio.Mediator.Abstractions/MiddlewareAttribute.cs": "e608c22a8b7729311cbdb2ae3a64a136631deca215ef70d45a8b85b6688030e8", + "src/Foundatio.Mediator.Abstractions/MiddlewareRegistration.cs": "6b21a1afbae5536a8ea368953aa0ba78c571e461ddfcd63e965a103c019960a7", + "src/Foundatio.Mediator.Abstractions/NotificationPublishStrategy.cs": "7ef9d84c5dfebd48eba18855c7ac7def4200ae1080fe49f96022347bced12457", + "src/Foundatio.Mediator.Abstractions/OpenGenericHandlerDescriptor.cs": "a40f2ae161abc8106e286e23273973e75d7834a95b3ae4bb5ebb1114a9b7c261", + "src/Foundatio.Mediator.Abstractions/Result.Generic.cs": "d86b8bd783e0f5fcc55d6d4a03518d72fff96bbc2b38d6a72a793ab6957eb823", + "src/Foundatio.Mediator.Abstractions/Result.cs": "df138be840e876cb324e7e00698b7e255f2b2560868c9117a9df193389f008a7", + "src/Foundatio.Mediator.Abstractions/ResultStatus.cs": "8447d598518b22cb18b92fe9a51d18828567c0374c28d47deea8bd40efbd2e0b", + "src/Foundatio.Mediator.Abstractions/StringSyntaxAttribute.cs": "606a72eb414a1d646b1312992507fea55063766a570cc573b713a84b04565d11", + "src/Foundatio.Mediator.Abstractions/SubscriberOptions.cs": "4cb1f1e1b521178baca367e695aeba298a873db790b66b9ba6a985c2f88f954b", + "src/Foundatio.Mediator.Abstractions/TopologicalSort.cs": "3713d171662a48ed62d2184bb45d402addcbbc2e9e3f63cf09827baf2faf98be", + "src/Foundatio.Mediator.Abstractions/TypeNameResolver.cs": "f16031702d4381454be54404a5d0240b92f5e4bd05021eb26c0bfdf9013b8db7", + "src/Foundatio.Mediator.Abstractions/UseMiddlewareAttribute.cs": "1f54990c7e022f78db6f9f0f4617650382b1d210038d1b2ef1d594ae54ae4348", + "src/Foundatio.Mediator.Abstractions/ValidationError.cs": "69c37a152bd712cc47256ca44c7cab884f746c72eb025d68e8aa13aade697e48", + "src/Foundatio.Mediator.Abstractions/ValidationSeverity.cs": "2859c51673fe215056293ce265aa154bcea7d05a4ad401206216f052ed629345", + "src/Foundatio.Mediator.CodeFixes/Foundatio.Mediator.CodeFixes.csproj": "83a92e4c8bccce0894885947bd228d945efdd81cee147d693b1ece9a134c9ebf", + "src/Foundatio.Mediator.CodeFixes/GlobalUsings.cs": "5e0770f90ee1b0a3ad5f4720354cc290b2badee9ccf825eae05dfa4bf4581803", + "src/Foundatio.Mediator.CodeFixes/LockEndpointRouteCodeFixProvider.cs": "2b23d58bed8ac5e764f432cfc37620637a698f25b86b7844ad0e150a4af104c0", + "src/Foundatio.Mediator.Distributed/AssemblyInfo.cs": "dc3034dc8af6aeb6329d03ef61a0f8e988255fda5df025b74a3db6583eb0c0a0", + "src/Foundatio.Mediator.Distributed/DistributedConfigurationValidator.cs": "c09c6263ee47775e823982c39c398761718bdcbb7b7385b70f837bfbb610d881", + "src/Foundatio.Mediator.Distributed/DistributedContext.cs": "58f31aec8eeec8036bfa21c0e0b71128c0588007acecdbafac91bf67948b2c7b", + "src/Foundatio.Mediator.Distributed/DistributedInfrastructureInitializer.cs": "87ef3124b37dfe277f993c63e73e4db273e4f9cf3a2917e18cba0d7e691cf5ec", + "src/Foundatio.Mediator.Distributed/DistributedMetrics.cs": "55b54050c3a209c2b0b34d810c866f6c693336b6ec198972f48f67d56db42df0", + "src/Foundatio.Mediator.Distributed/DistributedNotificationAttribute.cs": "01cafe5a2ba0db3689d72d02a6044c32306bbbd04ae6eb1428b85ac7216dfe1a", + "src/Foundatio.Mediator.Distributed/DistributedNotificationOptions.cs": "dfebfbb413c8eae9dee660dc1fd8729e573994e31919ff9b6aaca81edfba2a29", + "src/Foundatio.Mediator.Distributed/DistributedNotificationWorker.cs": "3291a2160228df63d3ffd8526121edc59707658d34d633ba3300279b048308bb", + "src/Foundatio.Mediator.Distributed/DistributedOptions.cs": "d1084d459db1fe164b8813a6969c41afdb5d96d515316e4012a4b8bed1c07320", + "src/Foundatio.Mediator.Distributed/DistributedQueueOptions.cs": "c13dfcd0c2c37e8de3a466e98935d3fb763a774846cbe8c45f58c659af8ef259", + "src/Foundatio.Mediator.Distributed/DistributedServiceExtensions.cs": "10019499d4b4f88055d481e45af8d8bf686786a9598199ab25065314928a5d4b", + "src/Foundatio.Mediator.Distributed/Foundatio.Mediator.Distributed.csproj": "43a88de24deb8e7e3e4efbb673cb4b233082b0509a4ae140f552dad64bdd154b", + "src/Foundatio.Mediator.Distributed/IDistributedNotification.cs": "6ad1f53161822ee5ceabd81921e3ba6ae9d5ef69627a6fd5ec631661b163a5d6", + "src/Foundatio.Mediator.Distributed/IQueueHeaderProvider.cs": "21bc0a2c64a8ec77118b27e4cc6776bc539e2a1d6c5d540495d6b46e81085663", + "src/Foundatio.Mediator.Distributed/IQueueWorkerRegistry.cs": "83b561cfc92bf6d9fedd7ac67ac409bead1493e40f2d7a3ebf3184479a762cf5", + "src/Foundatio.Mediator.Distributed/QueueAdministration.cs": "16a4e13b78f861d0a88cdbac4737a23e599eade3b47dfb122dced4393256f5a1", + "src/Foundatio.Mediator.Distributed/QueueAttribute.cs": "44df9ee0a5c172f59acc298ce8b68b0c33ed1b23d814d57a7dfb30dd60e03dfc", + "src/Foundatio.Mediator.Distributed/QueueDepthMetricsService.cs": "00e48684976d4123c74d8bc5b8c164c7158c4e34d808a36c4dd7b3cb58698d4d", + "src/Foundatio.Mediator.Distributed/QueueLockAttribute.cs": "878dbb3e64e669de9502189f691dfc6504c35044502db27c27730ddde1ec3200", + "src/Foundatio.Mediator.Distributed/QueueLockMiddleware.cs": "d45cb1eba9c1eabc56a52331bb7552732e69c6fff103bea20a7ba4831a1c4934", + "src/Foundatio.Mediator.Distributed/QueueMiddleware.cs": "586a600aacba948beb3d62cd50d187b8d7a134c326d74e6aac80447122765036", + "src/Foundatio.Mediator.Distributed/QueueOperation.cs": "42ad6bbd46957b54bf120082b004a8cc146c96bb210302277a4afe502a778031", + "src/Foundatio.Mediator.Distributed/QueueReceipt.cs": "88eb7149e852f91bafaba2a1f103151efdb7b0f6c8dc1b5fc408043985adc03d", + "src/Foundatio.Mediator.Distributed/QueueRetryDelay.cs": "77cb3aeec19effa755fa081246adc170f4099211788e10ef18787f7a0c5c9bf6", + "src/Foundatio.Mediator.Distributed/QueueRetryPolicy.cs": "59affe0544be908624c6fc691b12b810c5169e2438686b05dda63268f616694d", + "src/Foundatio.Mediator.Distributed/QueueStats.cs": "f512ef671eac27351b2a7485059853aebb0cf0900d262527e4ac933fbf69ddb5", + "src/Foundatio.Mediator.Distributed/QueueTopology.cs": "12835194fe9639605bd06953fc069b2900622a60c9ba2d92e6ee6dd6d18b8a7b", + "src/Foundatio.Mediator.Distributed/QueueWorker.cs": "5b2dba51127b99593a9baa8647d0446a79b8a939958089648389939a88bddd98", + "src/Foundatio.Mediator.Distributed/QueueWorkerInfo.cs": "8a7bbe62bbc305be36cd0b82b37d7a0851800f94d6d63c0a108ae3ba542161d8", + "src/Foundatio.Mediator.Distributed/QueueWorkerOptions.cs": "c179c74daec447650ce9d7cc537e105abf52839f7d6870b330409e0a1e802de9", + "src/Foundatio.Mediator.Distributed/QueueWorkerRegistry.cs": "d4ff8a4bd0227260abcb446f50681d6f8d7667940cc415c61347fa8a9618112e", + "src/Foundatio.Mediator.Distributed/QueueWorkerStats.cs": "84c47c0617d8a38442805d9e437fd4af983f7ff263f2d35cad2d0a316709251a", + "src/Foundatio.Mediator.Distributed/WorkerSelection.cs": "742d4ac382033d01f432e9c07397ad45184e0fd4b3738203ba786937f7058ec0", + "src/Foundatio.Mediator/AGENTS.md": "78e519c7152fa0600fea744e72a219dee7ed5094df47548f29baba585bb624d7", + "src/Foundatio.Mediator/CallSiteAnalyzer.cs": "02f2c143e9986092215d5b3975868f7ba5e4971157dd8005c71070878a81d3eb", + "src/Foundatio.Mediator/CrossAssemblyHandlerScanner.cs": "6246b42d32701021aed4cf90a6b8b903c9f7886c4fbd8c42791d4c958075b934", + "src/Foundatio.Mediator/CrossAssemblyInterceptorGenerator.cs": "41021f9357e5bc429b587a5e52d5a601db52ead370d31176c537f09ea3ac18c6", + "src/Foundatio.Mediator/EndpointGenerator.cs": "fd18ec8ecc172d4e8917e1348e80c326cf146bf1984ec05ef86e634b1d9a1443", + "src/Foundatio.Mediator/Foundatio.Mediator.csproj": "37bc000ec2bb866a896cdf86fd448977f24c3bfe0b910a09f8b8d70b2c3dc9be", + "src/Foundatio.Mediator/Foundatio.Mediator.targets": "54ae7b88ec6ff4d97770e4a7ef1e10ea1664a379a794ae3d3e9ed2803d449966", + "src/Foundatio.Mediator/FoundatioModuleGenerator.cs": "ac1e2c406f636f02cc554a42e1c8c0725fdd819f1f3fd758ec42fa29c6ff3cc2", + "src/Foundatio.Mediator/GlobalUsings.cs": "5e0770f90ee1b0a3ad5f4720354cc290b2badee9ccf825eae05dfa4bf4581803", + "src/Foundatio.Mediator/HandlerAnalyzer.cs": "55012f6a349ff1356e6f9dceca4943c69584b24a02dae842054a3fb06ac9eef1", + "src/Foundatio.Mediator/HandlerGenerator.cs": "05e5c296492a330635d1e02b0fe3183e391998500870727cb5b29e84ba07fe3f", + "src/Foundatio.Mediator/HelpersGenerator.cs": "65f7ff0f3a69c472e9bfb2505539c2e37bb1dfd5fc9bb39969a7fc2c5474744c", + "src/Foundatio.Mediator/InterceptsLocationGenerator.cs": "9f1f58b67658e1f0b2351dc4b0b217f5b5c172b74c3edd23a056155ce21be67d", + "src/Foundatio.Mediator/MediatorGenerator.cs": "1d615d3c9829e0a650af7c96f5d1e4296bb3bd26ba365d536e385c499bfa6450", + "src/Foundatio.Mediator/MediatorInfoAnalyzer.cs": "bfaad2c9cbf7a52cef097e62986a22292cc68cafea87b863b6d97d8b3c4ff638", + "src/Foundatio.Mediator/MetadataMiddlewareScanner.cs": "a55d401ef264778dd11be52c4d957a8e6a658068af251ac4a6f30ac3ad6dfe50", + "src/Foundatio.Mediator/MiddlewareAnalyzer.cs": "233eb624fed569212d814a34f6134b8e2b7c99faace8b0f853d24987b7414ad4", + "src/Foundatio.Mediator/Models/AuthorizationInfo.cs": "ed1fcd8f62a0992508c6bf3e216f8c00c687c8be23697b3a7ca11ad5edca0b6b", + "src/Foundatio.Mediator/Models/CallSiteInfo.cs": "cb37a105ab9540fe37c1908b20c49111300a68ac2316a78f0a35c453358b240c", + "src/Foundatio.Mediator/Models/CompilationInfo.cs": "053cd927567adae8059149064ef2362f45c0a8cb95eadcc0cd737a764a6dfe6d", + "src/Foundatio.Mediator/Models/DiagnosticInfo.cs": "640e68961feabecce4e40785f9fc9af13884ac0533fab6b831790490c584ea5f", + "src/Foundatio.Mediator/Models/EndpointConventionInfo.cs": "2d7a6fda5d638b7bf37a278945406b2d7002f32bc6a94da40f3810251e08c338", + "src/Foundatio.Mediator/Models/EndpointDefaultsInfo.cs": "82fd9a8f9d509ae63a4efb6952bc684628b6fd8102f385cd1390f2656ec16987", + "src/Foundatio.Mediator/Models/EndpointInfo.cs": "478197e5fef3c1bf6bbec45c58e1f8e260f72cc78dca8e21c57f9006c107aa07", + "src/Foundatio.Mediator/Models/GeneratorConfiguration.cs": "f80041b1cfd4aad59ea33db2651b9006f8159ff296cba01627a81957037ea9fc", + "src/Foundatio.Mediator/Models/HandlerAttributeMetadataInfo.cs": "780235d706e9aef0c446689c0df8d40e0ac3a0e9afb419550068255f6aa5f1da", + "src/Foundatio.Mediator/Models/HandlerInfo.cs": "f2bbe14955f951bd8e41803150ff9d1eef7dd18a421044dde477291912429209", + "src/Foundatio.Mediator/Models/HandlerMiddlewareReference.cs": "db3e0cbccb5c4ced8a68aaf9f786ff2aad6220cf06da783e9825d80b18cb66a2", + "src/Foundatio.Mediator/Models/LocationInfo.cs": "01ba3c064c59185608642dcc52ce9b19a768eddb49bfca955cd3a48ad569ba36", + "src/Foundatio.Mediator/Models/MiddlewareInfo.cs": "7e86ce683029be74fe61932a6301807cf00cd88403b7e5108532fccac082988a", + "src/Foundatio.Mediator/Models/TypeSymbolInfo.cs": "9d296be581b091872084de3a34006bc231b611d9602e724153768f10e962764c", + "src/Foundatio.Mediator/Properties/launchSettings.json": "07ccfa4635249f60ffbc54b018d29f6bdee88c01591f15a81556bf4eac45fd74", + "src/Foundatio.Mediator/PublishInterceptorGenerator.cs": "7b50ba9fe3a0e0699589c2bef103637d700c9ddcb31e98a411b5927f7a5dcada", + "src/Foundatio.Mediator/Utility/EquatableArray.cs": "ba3003c40e832cc6f085a1bfa4b29e251cd4ca7e7191e731c1af3c52cf53bc8b", + "src/Foundatio.Mediator/Utility/GeneratorDiagnostics.cs": "fe032733daaf06bf98c9ec52870d466f13010ef0fc008bb8e1f2a8b8401a3a13", + "src/Foundatio.Mediator/Utility/HandlerCodeEmitter.cs": "285ab9178e901267ac7527cee56d436ab8afe0ecf1b2c59251ae04a8d30a9961", + "src/Foundatio.Mediator/Utility/Helpers.cs": "a32fef07606d2269bf08057ad5433ca6c337d8a678549829415e2271e5c0bc09", + "src/Foundatio.Mediator/Utility/IndentedStringBuilder.cs": "e31d0656e0e12f27e6eae9d54ce21e4c82ce70a73158d493609980b3cce802d0", + "src/Foundatio.Mediator/Utility/InterceptorCodeEmitter.cs": "d5a436c04982c9f0991f3b6abb25604008e79f5c5ed574440f1fc9fd8a58b8dc", + "src/Foundatio.Mediator/Utility/IsExternInit.cs": "1fb52a29e5a55a7baa397b9ecc6a000ffcaab8bd2a28c359cb55d79639dde518", + "src/Foundatio.Mediator/Utility/NamespacePatternMatcher.cs": "4e29966507bf9ed98e25cd05ab020a5856888db4642a6577ff6eb95a98c4c321", + "src/Foundatio.Mediator/Utility/RouteConventions.cs": "e2f0aff1ad5fc7dc2dd704e7ae3b7485c95346c40c2d92c7a29beb7a34d1f73e", + "src/Foundatio.Mediator/Utility/SymbolUtilities.cs": "5ec67f7955c3d0ecc884e77da1526f39f3fadfd919575e9683947ad78f3315e7", + "src/Foundatio.Mediator/Utility/TopologicalSort.cs": "5a3dbb106bf92e6556b0285a51b542bbbd2ffb6c8cf3920f9148e780d8637b59", + "src/Foundatio.Mediator/Utility/TrackingNames.cs": "4e6d3da68b04fc61b2b607f60b2e3068e34c300939047bf90d20f178fc9eb319", + "src/Foundatio.Mediator/Utility/TypeExtensions.cs": "c9f8c2ad0b0ed2dae3a473242f482105e8900703a31721351ab9dcff21d98f64" +} diff --git a/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/recovery-results.json b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/recovery-results.json new file mode 100644 index 000000000..b64ed2d5a --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/recovery-results.json @@ -0,0 +1,76 @@ +{ + "Prefix": "recovery-1d1b352a6e374fadabe125181bbbb862", + "Runtime": ".NET 10.0.11", + "ProducerSeconds": 120, + "ElapsedSeconds": 121.2407788, + "Accepted": 46504, + "Completed": 46464, + "QueuedCancelled": 20, + "RunningCancelled": 20, + "RunningCancellationMilliseconds": 4958.3588, + "InterruptedAtKill": 32, + "RetriedJobs": 32, + "HandlerInvocations": 46516, + "DuplicateEffectAttempts": 0, + "Pending": 0, + "Failed": 0, + "DeadLettered": 0, + "Snapshots": [ + { + "Phase": "killed", + "ElapsedSeconds": 0.6581047, + "ProcessId": 1462944, + "Count": 32, + "Workers": [] + }, + { + "Phase": "cancelled", + "ElapsedSeconds": 6.2192256, + "ProcessId": 0, + "Count": 40, + "Workers": [ + { + "Id": 1463026, + "WorkingSet64": 123375616, + "PeakWorkingSet64": 123375616 + }, + { + "Id": 1463027, + "WorkingSet64": 108343296, + "PeakWorkingSet64": 108343296 + } + ] + }, + { + "Phase": "graceful-stop", + "ElapsedSeconds": 26.3104696, + "ProcessId": 1463027, + "Count": 0, + "Workers": [ + { + "Id": 1463026, + "WorkingSet64": 128479232, + "PeakWorkingSet64": 128479232 + } + ] + }, + { + "Phase": "drained", + "ElapsedSeconds": 121.2396349, + "ProcessId": 0, + "Count": 46464, + "Workers": [ + { + "Id": 1463026, + "WorkingSet64": 130609152, + "PeakWorkingSet64": 130609152 + }, + { + "Id": 1464492, + "WorkingSet64": 123256832, + "PeakWorkingSet64": 123256832 + } + ] + } + ] +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/redis-validation-results.json b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/redis-validation-results.json new file mode 100644 index 000000000..7255ef2ca --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/redis-validation-results.json @@ -0,0 +1,372 @@ +[ + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 181, + "Gen1Collections": 81, + "Gen2Collections": 5, + "ElapsedMilliseconds": 4912.7617, + "MessagesPerSecond": 6106.544919530699, + "AllocatedBytesPerMessage": 47331.51866666666, + "CpuMilliseconds": 17304.741, + "AcceptanceP50Milliseconds": 5.5527, + "AcceptanceP99Milliseconds": 13.0505, + "HandlerCompletionP50Milliseconds": 2002.2075, + "HandlerCompletionP99Milliseconds": 2100.5447, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 176, + "Gen1Collections": 77, + "Gen2Collections": 5, + "ElapsedMilliseconds": 4697.0437, + "MessagesPerSecond": 6386.996143978818, + "AllocatedBytesPerMessage": 45841.74773333333, + "CpuMilliseconds": 14290.472, + "AcceptanceP50Milliseconds": 5.6236, + "AcceptanceP99Milliseconds": 10.7371, + "HandlerCompletionP50Milliseconds": 1934.61, + "HandlerCompletionP99Milliseconds": 2025.3295, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 176, + "Gen1Collections": 81, + "Gen2Collections": 5, + "ElapsedMilliseconds": 4324.8948, + "MessagesPerSecond": 6936.584908377425, + "AllocatedBytesPerMessage": 45838.55466666666, + "CpuMilliseconds": 12937.454, + "AcceptanceP50Milliseconds": 5.0448, + "AcceptanceP99Milliseconds": 10.0928, + "HandlerCompletionP50Milliseconds": 1700.3524, + "HandlerCompletionP99Milliseconds": 1861.2469, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 182, + "Gen1Collections": 82, + "Gen2Collections": 5, + "ElapsedMilliseconds": 4559.0087, + "MessagesPerSecond": 6580.37787907709, + "AllocatedBytesPerMessage": 47323.2144, + "CpuMilliseconds": 17348.611, + "AcceptanceP50Milliseconds": 5.0873, + "AcceptanceP99Milliseconds": 10.5292, + "HandlerCompletionP50Milliseconds": 1802.7095, + "HandlerCompletionP99Milliseconds": 2045.9639, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 181, + "Gen1Collections": 85, + "Gen2Collections": 5, + "ElapsedMilliseconds": 4697.468, + "MessagesPerSecond": 6386.419236916569, + "AllocatedBytesPerMessage": 47316.78693333334, + "CpuMilliseconds": 14707.921, + "AcceptanceP50Milliseconds": 5.711, + "AcceptanceP99Milliseconds": 12.5729, + "HandlerCompletionP50Milliseconds": 1887.2762, + "HandlerCompletionP99Milliseconds": 2015.31, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 176, + "Gen1Collections": 80, + "Gen2Collections": 5, + "ElapsedMilliseconds": 4533.9863, + "MessagesPerSecond": 6616.694011625047, + "AllocatedBytesPerMessage": 45823.5752, + "CpuMilliseconds": 14166.398, + "AcceptanceP50Milliseconds": 5.3201, + "AcceptanceP99Milliseconds": 10.7063, + "HandlerCompletionP50Milliseconds": 1785.1358, + "HandlerCompletionP99Milliseconds": 1904.973, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 3, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 176, + "Gen1Collections": 82, + "Gen2Collections": 5, + "ElapsedMilliseconds": 4642.5918, + "MessagesPerSecond": 6461.907764537903, + "AllocatedBytesPerMessage": 45839.893066666664, + "CpuMilliseconds": 15918.828, + "AcceptanceP50Milliseconds": 5.2276, + "AcceptanceP99Milliseconds": 10.4365, + "HandlerCompletionP50Milliseconds": 1870.9831, + "HandlerCompletionP99Milliseconds": 2072.4856, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 3, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 181, + "Gen1Collections": 81, + "Gen2Collections": 5, + "ElapsedMilliseconds": 4819.382, + "MessagesPerSecond": 6224.864515823813, + "AllocatedBytesPerMessage": 47323.825333333334, + "CpuMilliseconds": 16474.052, + "AcceptanceP50Milliseconds": 5.966, + "AcceptanceP99Milliseconds": 11.335, + "HandlerCompletionP50Milliseconds": 1974.3003, + "HandlerCompletionP99Milliseconds": 2059.2992, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 4, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 181, + "Gen1Collections": 84, + "Gen2Collections": 5, + "ElapsedMilliseconds": 4702.4303, + "MessagesPerSecond": 6379.679885951739, + "AllocatedBytesPerMessage": 47309.49413333333, + "CpuMilliseconds": 14767.515, + "AcceptanceP50Milliseconds": 5.6414, + "AcceptanceP99Milliseconds": 10.9754, + "HandlerCompletionP50Milliseconds": 1841.2863, + "HandlerCompletionP99Milliseconds": 1908.7225, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 4, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/performance-pass2/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.11", + "TrackingStore": "redis", + "Gen0Collections": 176, + "Gen1Collections": 78, + "Gen2Collections": 5, + "ElapsedMilliseconds": 4612.9546, + "MessagesPerSecond": 6503.42407445328, + "AllocatedBytesPerMessage": 45815.304533333336, + "CpuMilliseconds": 14890.485, + "AcceptanceP50Milliseconds": 5.3144, + "AcceptanceP99Milliseconds": 11.1213, + "HandlerCompletionP50Milliseconds": 1861.2762, + "HandlerCompletionP99Milliseconds": 1990.6832, + "UniqueProcessed": 30000, + "Duplicates": 0 + } +] diff --git a/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/redis-validation-summary.json b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/redis-validation-summary.json new file mode 100644 index 000000000..f1a4734e9 --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-pass2-2026-09-08/redis-validation-summary.json @@ -0,0 +1,18 @@ +{ + "redis-64": { + "before": { + "MessagesPerSecond": 6379.68, + "AllocatedBytesPerMessage": 47323.214, + "AcceptanceP99Milliseconds": 11.335, + "HandlerCompletionP99Milliseconds": 2045.964, + "CpuMilliseconds": 16474.052 + }, + "after": { + "MessagesPerSecond": 6503.424, + "AllocatedBytesPerMessage": 45838.555, + "AcceptanceP99Milliseconds": 10.706, + "HandlerCompletionP99Milliseconds": 1990.683, + "CpuMilliseconds": 14290.472 + } + } +} diff --git a/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/layers-after-results.json b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/layers-after-results.json new file mode 100644 index 000000000..8f883aa77 --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/layers-after-results.json @@ -0,0 +1,278 @@ +[ + { + "Implementation": "foundatio-native", + "Layer": "transport", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 74, + "Gen1Collections": 29, + "Gen2Collections": 7, + "ElapsedMilliseconds": 668.9707, + "MessagesPerSecond": 298966.7559431228, + "AllocatedBytesPerMessage": 2809.9346, + "CpuMilliseconds": 3791.832, + "AcceptanceP50Milliseconds": 0.0039, + "AcceptanceP99Milliseconds": 0.0128, + "HandlerCompletionP50Milliseconds": 156.5385, + "HandlerCompletionP99Milliseconds": 232.7172, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "bus", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 106, + "Gen1Collections": 31, + "Gen2Collections": 5, + "ElapsedMilliseconds": 1224.7802, + "MessagesPerSecond": 163294.6058402969, + "AllocatedBytesPerMessage": 4249.70892, + "CpuMilliseconds": 5966.502, + "AcceptanceP50Milliseconds": 0.0095, + "AcceptanceP99Milliseconds": 0.0298, + "HandlerCompletionP50Milliseconds": 511.9716, + "HandlerCompletionP99Milliseconds": 616.2701, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "execution", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 151, + "Gen1Collections": 32, + "Gen2Collections": 5, + "ElapsedMilliseconds": 1398.0391, + "MessagesPerSecond": 143057.51534417027, + "AllocatedBytesPerMessage": 6121.77024, + "CpuMilliseconds": 6315.313, + "AcceptanceP50Milliseconds": 0.0094, + "AcceptanceP99Milliseconds": 0.0327, + "HandlerCompletionP50Milliseconds": 692.5949, + "HandlerCompletionP99Milliseconds": 749.793, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 194, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1851.2769, + "MessagesPerSecond": 108033.54160579652, + "AllocatedBytesPerMessage": 7924.9626, + "CpuMilliseconds": 8877.735, + "AcceptanceP50Milliseconds": 0.0176, + "AcceptanceP99Milliseconds": 0.1495, + "HandlerCompletionP50Milliseconds": 1021.412, + "HandlerCompletionP99Milliseconds": 1128.9666, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "bus", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 106, + "Gen1Collections": 31, + "Gen2Collections": 5, + "ElapsedMilliseconds": 1125.9919, + "MessagesPerSecond": 177621.1711647304, + "AllocatedBytesPerMessage": 4250.0346, + "CpuMilliseconds": 5538.469, + "AcceptanceP50Milliseconds": 0.0082, + "AcceptanceP99Milliseconds": 0.0278, + "HandlerCompletionP50Milliseconds": 434.4107, + "HandlerCompletionP99Milliseconds": 501.9349, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "execution", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 151, + "Gen1Collections": 32, + "Gen2Collections": 5, + "ElapsedMilliseconds": 1503.3243, + "MessagesPerSecond": 133038.4934242066, + "AllocatedBytesPerMessage": 6122.08452, + "CpuMilliseconds": 5958.848, + "AcceptanceP50Milliseconds": 0.0078, + "AcceptanceP99Milliseconds": 0.0276, + "HandlerCompletionP50Milliseconds": 675.0227, + "HandlerCompletionP99Milliseconds": 801.3771, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 193, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1738.2986, + "MessagesPerSecond": 115055.03139679224, + "AllocatedBytesPerMessage": 7913.20464, + "CpuMilliseconds": 8280.592, + "AcceptanceP50Milliseconds": 0.0172, + "AcceptanceP99Milliseconds": 0.1347, + "HandlerCompletionP50Milliseconds": 974.2742, + "HandlerCompletionP99Milliseconds": 1080.1789, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "transport", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 71, + "Gen1Collections": 29, + "Gen2Collections": 5, + "ElapsedMilliseconds": 661.2769, + "MessagesPerSecond": 302445.1632893876, + "AllocatedBytesPerMessage": 2809.38164, + "CpuMilliseconds": 3319.241, + "AcceptanceP50Milliseconds": 0.0036, + "AcceptanceP99Milliseconds": 0.0126, + "HandlerCompletionP50Milliseconds": 201.0827, + "HandlerCompletionP99Milliseconds": 275.5085, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "execution", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 151, + "Gen1Collections": 32, + "Gen2Collections": 5, + "ElapsedMilliseconds": 1436.521, + "MessagesPerSecond": 139225.25323333248, + "AllocatedBytesPerMessage": 6121.46564, + "CpuMilliseconds": 6052.249, + "AcceptanceP50Milliseconds": 0.0076, + "AcceptanceP99Milliseconds": 0.0289, + "HandlerCompletionP50Milliseconds": 718.8442, + "HandlerCompletionP99Milliseconds": 795.5796, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 194, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1996.2709, + "MessagesPerSecond": 100186.80330410066, + "AllocatedBytesPerMessage": 7933.07052, + "CpuMilliseconds": 8292.253, + "AcceptanceP50Milliseconds": 0.0163, + "AcceptanceP99Milliseconds": 0.1834, + "HandlerCompletionP50Milliseconds": 1104.5859, + "HandlerCompletionP99Milliseconds": 1210.9511, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "transport", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 72, + "Gen1Collections": 30, + "Gen2Collections": 6, + "ElapsedMilliseconds": 696.5053, + "MessagesPerSecond": 287147.85084908904, + "AllocatedBytesPerMessage": 2810.0204, + "CpuMilliseconds": 3455.183, + "AcceptanceP50Milliseconds": 0.0039, + "AcceptanceP99Milliseconds": 0.0131, + "HandlerCompletionP50Milliseconds": 210.9249, + "HandlerCompletionP99Milliseconds": 294.5844, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "bus", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 106, + "Gen1Collections": 31, + "Gen2Collections": 5, + "ElapsedMilliseconds": 1160.4787, + "MessagesPerSecond": 172342.6720369792, + "AllocatedBytesPerMessage": 4249.35888, + "CpuMilliseconds": 5492.435, + "AcceptanceP50Milliseconds": 0.0078, + "AcceptanceP99Milliseconds": 0.0296, + "HandlerCompletionP50Milliseconds": 443.9492, + "HandlerCompletionP99Milliseconds": 526.9722, + "UniqueProcessed": 200000, + "Duplicates": 0 + } +] \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/layers-before-results.json b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/layers-before-results.json new file mode 100644 index 000000000..4c4c06e94 --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/layers-before-results.json @@ -0,0 +1,278 @@ +[ + { + "Implementation": "foundatio-native", + "Layer": "transport", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 80, + "Gen1Collections": 39, + "Gen2Collections": 12, + "ElapsedMilliseconds": 784.4166, + "MessagesPerSecond": 254966.55731151023, + "AllocatedBytesPerMessage": 2826.33512, + "CpuMilliseconds": 3805.979, + "AcceptanceP50Milliseconds": 0.0036, + "AcceptanceP99Milliseconds": 0.0124, + "HandlerCompletionP50Milliseconds": 234.6403, + "HandlerCompletionP99Milliseconds": 340.676, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "bus", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 131, + "Gen1Collections": 44, + "Gen2Collections": 13, + "ElapsedMilliseconds": 1714.0318, + "MessagesPerSecond": 116683.94950432074, + "AllocatedBytesPerMessage": 4859.57364, + "CpuMilliseconds": 8809.402, + "AcceptanceP50Milliseconds": 0.0094, + "AcceptanceP99Milliseconds": 0.0292, + "HandlerCompletionP50Milliseconds": 847.2854, + "HandlerCompletionP99Milliseconds": 950.2622, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "execution", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 172, + "Gen1Collections": 44, + "Gen2Collections": 13, + "ElapsedMilliseconds": 1969.6718, + "MessagesPerSecond": 101539.75906036732, + "AllocatedBytesPerMessage": 6590.47244, + "CpuMilliseconds": 8713.738, + "AcceptanceP50Milliseconds": 0.0092, + "AcceptanceP99Milliseconds": 0.0276, + "HandlerCompletionP50Milliseconds": 1078.1974, + "HandlerCompletionP99Milliseconds": 1132.1616, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 217, + "Gen1Collections": 54, + "Gen2Collections": 15, + "ElapsedMilliseconds": 2375.4918, + "MessagesPerSecond": 84193.09214201456, + "AllocatedBytesPerMessage": 8381.47828, + "CpuMilliseconds": 12325.73, + "AcceptanceP50Milliseconds": 0.0231, + "AcceptanceP99Milliseconds": 0.0868, + "HandlerCompletionP50Milliseconds": 1359.8243, + "HandlerCompletionP99Milliseconds": 1501.8987, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "bus", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 131, + "Gen1Collections": 43, + "Gen2Collections": 13, + "ElapsedMilliseconds": 1830.7098, + "MessagesPerSecond": 109247.2438832195, + "AllocatedBytesPerMessage": 4870.05248, + "CpuMilliseconds": 8822.498, + "AcceptanceP50Milliseconds": 0.0087, + "AcceptanceP99Milliseconds": 0.0273, + "HandlerCompletionP50Milliseconds": 923.3158, + "HandlerCompletionP99Milliseconds": 1011.9226, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "execution", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 172, + "Gen1Collections": 45, + "Gen2Collections": 13, + "ElapsedMilliseconds": 1960.1342, + "MessagesPerSecond": 102033.83013265113, + "AllocatedBytesPerMessage": 6577.8166, + "CpuMilliseconds": 8931.296, + "AcceptanceP50Milliseconds": 0.0087, + "AcceptanceP99Milliseconds": 0.0285, + "HandlerCompletionP50Milliseconds": 1002.7864, + "HandlerCompletionP99Milliseconds": 1062.622, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 207, + "Gen1Collections": 43, + "Gen2Collections": 6, + "ElapsedMilliseconds": 2245.2583, + "MessagesPerSecond": 89076.61091821818, + "AllocatedBytesPerMessage": 8352.63152, + "CpuMilliseconds": 13726.695, + "AcceptanceP50Milliseconds": 0.0267, + "AcceptanceP99Milliseconds": 2.0784, + "HandlerCompletionP50Milliseconds": 1245.9897, + "HandlerCompletionP99Milliseconds": 1373.9973, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "transport", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 76, + "Gen1Collections": 37, + "Gen2Collections": 8, + "ElapsedMilliseconds": 709.0783, + "MessagesPerSecond": 282056.2975908302, + "AllocatedBytesPerMessage": 2824.3378, + "CpuMilliseconds": 3753.997, + "AcceptanceP50Milliseconds": 0.0038, + "AcceptanceP99Milliseconds": 0.0157, + "HandlerCompletionP50Milliseconds": 224.2195, + "HandlerCompletionP99Milliseconds": 336.0705, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "execution", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 165, + "Gen1Collections": 38, + "Gen2Collections": 7, + "ElapsedMilliseconds": 1588.4751, + "MessagesPerSecond": 125906.91538066918, + "AllocatedBytesPerMessage": 6592.0416, + "CpuMilliseconds": 7708.456, + "AcceptanceP50Milliseconds": 0.0068, + "AcceptanceP99Milliseconds": 0.0321, + "HandlerCompletionP50Milliseconds": 823.9175, + "HandlerCompletionP99Milliseconds": 906.7955, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 206, + "Gen1Collections": 42, + "Gen2Collections": 5, + "ElapsedMilliseconds": 2044.7225, + "MessagesPerSecond": 97812.78388632197, + "AllocatedBytesPerMessage": 8384.76684, + "CpuMilliseconds": 11747.3, + "AcceptanceP50Milliseconds": 0.0188, + "AcceptanceP99Milliseconds": 2.0827, + "HandlerCompletionP50Milliseconds": 1181.1237, + "HandlerCompletionP99Milliseconds": 1239.067, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "transport", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 75, + "Gen1Collections": 35, + "Gen2Collections": 7, + "ElapsedMilliseconds": 723.1769, + "MessagesPerSecond": 276557.5061924683, + "AllocatedBytesPerMessage": 2822.9176, + "CpuMilliseconds": 3561.011, + "AcceptanceP50Milliseconds": 0.0036, + "AcceptanceP99Milliseconds": 0.0163, + "HandlerCompletionP50Milliseconds": 249.9717, + "HandlerCompletionP99Milliseconds": 322.2744, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Implementation": "foundatio-native", + "Layer": "bus", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 126, + "Gen1Collections": 39, + "Gen2Collections": 8, + "ElapsedMilliseconds": 1505.2209, + "MessagesPerSecond": 132870.86300754925, + "AllocatedBytesPerMessage": 4888.11788, + "CpuMilliseconds": 8251.401, + "AcceptanceP50Milliseconds": 0.0076, + "AcceptanceP99Milliseconds": 0.0316, + "HandlerCompletionP50Milliseconds": 700.9642, + "HandlerCompletionP99Milliseconds": 732.4533, + "UniqueProcessed": 200000, + "Duplicates": 0 + } +] \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/manifest.json b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/manifest.json new file mode 100644 index 000000000..ac7bc1906 --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/manifest.json @@ -0,0 +1,54 @@ +{ + "foundatioMain": "99901a483f3da43e16dd47d72956e23853ee8ebb", + "mediatorMain": "3f76a7ee889a3111d8c081cf562aea16ccd13c13", + "previousPublishedFoundatio": "7e6e0a1928eca505f6f8ebd3572a3c323fe31292", + "previousPublishedMediator": "673102f6d7f2b9577dddf43bce088d28934fd606", + "rebasedBeforeFoundatio": "7726d474fa1994dbaddec91cbc90a458acf4def6", + "rebasedBeforeMediator": "45401e6b01cd1a1d71990c7780936d9fc0cd485b", + "pr149": "89bd6d1504b83d03de156b08f24fb578aa8bc97f", + "runtimeExecutable": "/usr/bin/dotnet", + "sdk": "10.0.112", + "binaries": { + "before": { + "Foundatio.Mediator.Distributed.Benchmarks.dll": "b6df55e61e77dbec2f1b80483fb9a69950104fc4a53a1f19155d48b604b54c59", + "Foundatio.Redis.dll": "b32e1f9cf36bb69d314cddef3384d50da27868f9e211b82e1ef37b22a79842de", + "Foundatio.Mediator.Distributed.dll": "203c2d02c3214e156c9c687b1b8837b3854e6a568409b9d0d387f68c8fc770ac", + "Foundatio.Mediator.Abstractions.dll": "f636737d98866455bf8eb150d878f1220b62fa79813f4b9b8b1512157bcb31c5", + "Foundatio.Aws.dll": "550fbef89423d0dcd4d43f1d92e68516e8598de21726c02ec9d17e5f76996d3e", + "Foundatio.dll": "0aebf424a114f31e7554b16dc14a7a82c8ce131bbfcc16ac01890f8252072afe" + }, + "recovery": { + "Foundatio.Redis.dll": "ed87796e54577be9518519843db95fd527d748625d39216ab514c889cfd45f59", + "Foundatio.Mediator.Distributed.dll": "baf2aca52b3a629409a4cfc70b8423e896dc6e752860ec8ed5e48152c4fc86c2", + "Foundatio.Mediator.Abstractions.dll": "f636737d98866455bf8eb150d878f1220b62fa79813f4b9b8b1512157bcb31c5", + "Foundatio.dll": "b9c6bb92bef2fe8e86a7bad65e1addccaeaee7663e75a9c047351f8aaaa45d6c", + "Foundatio.Aws.dll": "c0625097a02a49cd5ecb1a0124e03b4f8fea8a95135340a4dc35b2a432bb5a5a" + }, + "excludedMixedBuild": { + "Foundatio.Mediator.Distributed.Benchmarks.dll": "c942560dc62e389bccf52420b56762ff075efdb81d2e20ab9d17721662a25d16", + "Foundatio.dll": "c0844dce22d36580a7df6b846faa6439790c96d387fff30d2626da059ef980cb", + "Foundatio.Redis.dll": "944380d9550a7fdf926ea4541f866520d2c9c0df96344a282b8f71cad355cd55", + "Foundatio.Aws.dll": "ed4c30ededdd6136b1e7dfe9d78d06f46e8e9813cdbb509805d5f86f5a3c1470", + "Foundatio.Mediator.Distributed.dll": "c209b252ae7a26c00265ffd7ec6432b415a0ffc269e4a22ae87d14870c5829d4", + "Foundatio.Mediator.Abstractions.dll": "f636737d98866455bf8eb150d878f1220b62fa79813f4b9b8b1512157bcb31c5" + }, + "after": { + "Foundatio.Mediator.Distributed.Benchmarks.dll": "2d7ca1371c3e2ddea5cffaa6959bd458f0ebc955cde1db932e93e51205508957", + "Foundatio.Redis.dll": "1104232a274b019b6269b6467ceb5836167aaf7ee673076a66c065d29cc4a6e3", + "Foundatio.Mediator.Distributed.dll": "3ec4f5a3ea39f807151bb85063c19b05b7b9d71ddf6cabcdf0350e472ae211b9", + "Foundatio.dll": "c035a4447aa5a40459bf63bdd80413a7d71ff2ba810de0d4689995c9d5252642", + "Foundatio.Aws.dll": "4b42ec1b46f10d642fcb39d91737b424035f19a97a4e750477d2d0815ed65b75", + "Foundatio.Mediator.Abstractions.dll": "f636737d98866455bf8eb150d878f1220b62fa79813f4b9b8b1512157bcb31c5" + } + }, + "source": { + "/tmp/foundatio-core-alternative/src/Foundatio/Messaging/MessageDeliveryLease.cs": "76833ec9308a59533e8d6198af29132769492fe1f5c6e3f8d78bcbf600d9e6b2", + "/tmp/foundatio-core-alternative/tests/Foundatio.Tests/Messaging/MessageDeliveryLeaseTests.cs": "a1d69d05f94ab9f43471ad1ad8970d55f3587d07f702d14c534985e4363cf8cc", + "/tmp/mediator-core-alternative/benchmarks/Foundatio.Mediator.Distributed.Benchmarks/Program.cs": "8254fba4cc53f8c784d9a98c46c54d9b1c9b7b47cd96b778d4c25cbde653c7a1", + "/tmp/foundatio-core-alternative/build/common.props": "18030a483005bde3763bd8f736470015c76b96f7cba469136bc4a81b5ebd7b0e", + "/tmp/mediator-core-alternative/build/foundatio-core.props": "6596d364da1e333631ab0cc1040852022c590db046187964425d13224d593a00", + "/tmp/mediator-core-alternative/.github/workflows/build.yml": "622360f6ce9d9fcad6d9023091a67f755804bb5eb01b02144bf73c2592230a21" + }, + "runtime": ".NET 10.0.12", + "afterRuntimeCommit": "e583f953" +} diff --git a/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/matrix-results.json b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/matrix-results.json new file mode 100644 index 000000000..8f28bbd92 --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/matrix-results.json @@ -0,0 +1,2312 @@ +[ + { + "Case": "pr149", + "Scenario": "memory-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 155, + "Gen1Collections": 33, + "Gen2Collections": 6, + "ElapsedMilliseconds": 1103.782, + "MessagesPerSecond": 181195.19977676752, + "AllocatedBytesPerMessage": 6174.6714, + "CpuMilliseconds": 12292.901, + "AcceptanceP50Milliseconds": 0.011, + "AcceptanceP99Milliseconds": 0.3183, + "HandlerCompletionP50Milliseconds": 398.0789, + "HandlerCompletionP99Milliseconds": 434.9931, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 203, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1957.8911, + "MessagesPerSecond": 102150.7273821307, + "AllocatedBytesPerMessage": 8343.84312, + "CpuMilliseconds": 10412.641, + "AcceptanceP50Milliseconds": 0.0173, + "AcceptanceP99Milliseconds": 0.1617, + "HandlerCompletionP50Milliseconds": 1131.4993, + "HandlerCompletionP99Milliseconds": 1190.867, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 194, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1940.0444, + "MessagesPerSecond": 103090.42411606663, + "AllocatedBytesPerMessage": 7928.319, + "CpuMilliseconds": 8475.778, + "AcceptanceP50Milliseconds": 0.0175, + "AcceptanceP99Milliseconds": 0.6537, + "HandlerCompletionP50Milliseconds": 1102.316, + "HandlerCompletionP99Milliseconds": 1174.4511, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-tracked-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 72, + "Gen1Collections": 18, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1373.5288, + "MessagesPerSecond": 36402.58580671916, + "AllocatedBytesPerMessage": 11482.16016, + "CpuMilliseconds": 5688.72, + "AcceptanceP50Milliseconds": 0.0173, + "AcceptanceP99Milliseconds": 4.7345, + "HandlerCompletionP50Milliseconds": 0.1214, + "HandlerCompletionP99Milliseconds": 7.966, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 95, + "Gen1Collections": 29, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1602.7859, + "MessagesPerSecond": 31195.68246763339, + "AllocatedBytesPerMessage": 15534.83632, + "CpuMilliseconds": 5791.305, + "AcceptanceP50Milliseconds": 0.0213, + "AcceptanceP99Milliseconds": 4.4872, + "HandlerCompletionP50Milliseconds": 792.1129, + "HandlerCompletionP99Milliseconds": 1034.2686, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 93, + "Gen1Collections": 29, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1405.2739, + "MessagesPerSecond": 35580.252362190746, + "AllocatedBytesPerMessage": 15150.97504, + "CpuMilliseconds": 5089.354, + "AcceptanceP50Milliseconds": 0.0191, + "AcceptanceP99Milliseconds": 4.3143, + "HandlerCompletionP50Milliseconds": 728.882, + "HandlerCompletionP99Milliseconds": 866.2067, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-1", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 80, + "Gen1Collections": 19, + "Gen2Collections": 6, + "ElapsedMilliseconds": 902.4827, + "MessagesPerSecond": 110805.44812659567, + "AllocatedBytesPerMessage": 6413.62296, + "CpuMilliseconds": 3113.494, + "AcceptanceP50Milliseconds": 0.0034, + "AcceptanceP99Milliseconds": 0.0088, + "HandlerCompletionP50Milliseconds": 366.7218, + "HandlerCompletionP99Milliseconds": 426.0144, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-1", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 111, + "Gen1Collections": 27, + "Gen2Collections": 9, + "ElapsedMilliseconds": 1074.7441, + "MessagesPerSecond": 93045.40494802437, + "AllocatedBytesPerMessage": 8684.16208, + "CpuMilliseconds": 3332.278, + "AcceptanceP50Milliseconds": 0.005, + "AcceptanceP99Milliseconds": 0.0122, + "HandlerCompletionP50Milliseconds": 225.8616, + "HandlerCompletionP99Milliseconds": 321.0849, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-1", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 109, + "Gen1Collections": 30, + "Gen2Collections": 11, + "ElapsedMilliseconds": 994.66, + "MessagesPerSecond": 100536.8668690809, + "AllocatedBytesPerMessage": 8271.53088, + "CpuMilliseconds": 1770.779, + "AcceptanceP50Milliseconds": 0.0047, + "AcceptanceP99Milliseconds": 0.0132, + "HandlerCompletionP50Milliseconds": 168.3279, + "HandlerCompletionP99Milliseconds": 281.5513, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-8", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 77, + "Gen1Collections": 18, + "Gen2Collections": 4, + "ElapsedMilliseconds": 572.8751, + "MessagesPerSecond": 174558.11921307107, + "AllocatedBytesPerMessage": 6133.2476, + "CpuMilliseconds": 5933.353, + "AcceptanceP50Milliseconds": 0.0103, + "AcceptanceP99Milliseconds": 0.0323, + "HandlerCompletionP50Milliseconds": 240.2262, + "HandlerCompletionP99Milliseconds": 257.8909, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-8", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 102, + "Gen1Collections": 21, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1118.3072, + "MessagesPerSecond": 89420.86753979586, + "AllocatedBytesPerMessage": 8341.1656, + "CpuMilliseconds": 4164.481, + "AcceptanceP50Milliseconds": 0.0106, + "AcceptanceP99Milliseconds": 0.0314, + "HandlerCompletionP50Milliseconds": 480.2731, + "HandlerCompletionP99Milliseconds": 700.8688, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-8", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 98, + "Gen1Collections": 21, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1096.9576, + "MessagesPerSecond": 91161.2262862302, + "AllocatedBytesPerMessage": 7937.79024, + "CpuMilliseconds": 3092.042, + "AcceptanceP50Milliseconds": 0.0122, + "AcceptanceP99Milliseconds": 0.0324, + "HandlerCompletionP50Milliseconds": 487.9858, + "HandlerCompletionP99Milliseconds": 677.5429, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 48, + "Gen1Collections": 18, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3774.2034, + "MessagesPerSecond": 2649.565733526709, + "AllocatedBytesPerMessage": 39788.4928, + "CpuMilliseconds": 5140.036, + "AcceptanceP50Milliseconds": 10.3057, + "AcceptanceP99Milliseconds": 20.0604, + "HandlerCompletionP50Milliseconds": 1372.9495, + "HandlerCompletionP99Milliseconds": 1778.9107, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 57, + "Gen1Collections": 53, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3158.7795, + "MessagesPerSecond": 3165.779694340805, + "AllocatedBytesPerMessage": 46894.6768, + "CpuMilliseconds": 6059.97, + "AcceptanceP50Milliseconds": 14.7095, + "AcceptanceP99Milliseconds": 47.5444, + "HandlerCompletionP50Milliseconds": 501.0109, + "HandlerCompletionP99Milliseconds": 767.5534, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 57, + "Gen1Collections": 52, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3369.6964, + "MessagesPerSecond": 2967.6264010015857, + "AllocatedBytesPerMessage": 46623.0464, + "CpuMilliseconds": 5916.081, + "AcceptanceP50Milliseconds": 14.3725, + "AcceptanceP99Milliseconds": 28.4577, + "HandlerCompletionP50Milliseconds": 707.6503, + "HandlerCompletionP99Milliseconds": 841.6169, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 33, + "Gen1Collections": 11, + "Gen2Collections": 2, + "ElapsedMilliseconds": 1039.3107, + "MessagesPerSecond": 9621.76180809069, + "AllocatedBytesPerMessage": 25866.052, + "CpuMilliseconds": 4651.474, + "AcceptanceP50Milliseconds": 2.7211, + "AcceptanceP99Milliseconds": 9.0972, + "HandlerCompletionP50Milliseconds": 432.6228, + "HandlerCompletionP99Milliseconds": 547.3125, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 59, + "Gen1Collections": 25, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1505.8942, + "MessagesPerSecond": 6640.57275736901, + "AllocatedBytesPerMessage": 45840.6816, + "CpuMilliseconds": 6418.73, + "AcceptanceP50Milliseconds": 5.4149, + "AcceptanceP99Milliseconds": 10.7907, + "HandlerCompletionP50Milliseconds": 603.6399, + "HandlerCompletionP99Milliseconds": 656.4766, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 59, + "Gen1Collections": 24, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1400.88, + "MessagesPerSecond": 7138.370167323397, + "AllocatedBytesPerMessage": 45449.5456, + "CpuMilliseconds": 6478.119, + "AcceptanceP50Milliseconds": 4.3981, + "AcceptanceP99Milliseconds": 11.3004, + "HandlerCompletionP50Milliseconds": 570.8358, + "HandlerCompletionP99Milliseconds": 663.3455, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 74, + "Gen1Collections": 57, + "Gen2Collections": 1, + "ElapsedMilliseconds": 4602.8089, + "MessagesPerSecond": 2172.5863961026057, + "AllocatedBytesPerMessage": 59277.2312, + "CpuMilliseconds": 12316.922, + "AcceptanceP50Milliseconds": 13.4666, + "AcceptanceP99Milliseconds": 26.2889, + "HandlerCompletionP50Milliseconds": 1744.0931, + "HandlerCompletionP99Milliseconds": 2409.8847, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 106, + "Gen1Collections": 95, + "Gen2Collections": 1, + "ElapsedMilliseconds": 4244.1105, + "MessagesPerSecond": 2356.206324034212, + "AllocatedBytesPerMessage": 84284.6224, + "CpuMilliseconds": 13132.348, + "AcceptanceP50Milliseconds": 16.4589, + "AcceptanceP99Milliseconds": 32.6327, + "HandlerCompletionP50Milliseconds": 1142.8043, + "HandlerCompletionP99Milliseconds": 1281.852, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 105, + "Gen1Collections": 81, + "Gen2Collections": 2, + "ElapsedMilliseconds": 3872.6118, + "MessagesPerSecond": 2582.2366187078187, + "AllocatedBytesPerMessage": 83929.9784, + "CpuMilliseconds": 12802.417, + "AcceptanceP50Milliseconds": 15.8128, + "AcceptanceP99Milliseconds": 56.8855, + "HandlerCompletionP50Milliseconds": 1161.3343, + "HandlerCompletionP99Milliseconds": 1323.4233, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 204, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 2089.9421, + "MessagesPerSecond": 95696.43101595972, + "AllocatedBytesPerMessage": 8357.98008, + "CpuMilliseconds": 10981.601, + "AcceptanceP50Milliseconds": 0.0193, + "AcceptanceP99Milliseconds": 0.2561, + "HandlerCompletionP50Milliseconds": 1195.76, + "HandlerCompletionP99Milliseconds": 1270.1025, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 194, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1993.1, + "MessagesPerSecond": 100346.1943705785, + "AllocatedBytesPerMessage": 7926.36816, + "CpuMilliseconds": 9085.144, + "AcceptanceP50Milliseconds": 0.0169, + "AcceptanceP99Milliseconds": 0.2251, + "HandlerCompletionP50Milliseconds": 1124.1508, + "HandlerCompletionP99Milliseconds": 1220.3718, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 153, + "Gen1Collections": 32, + "Gen2Collections": 5, + "ElapsedMilliseconds": 1083.8027, + "MessagesPerSecond": 184535.4325099947, + "AllocatedBytesPerMessage": 6172.35772, + "CpuMilliseconds": 12105.471, + "AcceptanceP50Milliseconds": 0.011, + "AcceptanceP99Milliseconds": 0.2486, + "HandlerCompletionP50Milliseconds": 375.6087, + "HandlerCompletionP99Milliseconds": 395.1285, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 95, + "Gen1Collections": 29, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1504.2878, + "MessagesPerSecond": 33238.32048627929, + "AllocatedBytesPerMessage": 15537.56224, + "CpuMilliseconds": 5300.711, + "AcceptanceP50Milliseconds": 0.0193, + "AcceptanceP99Milliseconds": 5.3217, + "HandlerCompletionP50Milliseconds": 789.8318, + "HandlerCompletionP99Milliseconds": 903.9109, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 93, + "Gen1Collections": 29, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1465.795, + "MessagesPerSecond": 34111.18198656702, + "AllocatedBytesPerMessage": 15145.77136, + "CpuMilliseconds": 5304.177, + "AcceptanceP50Milliseconds": 0.021, + "AcceptanceP99Milliseconds": 5.2627, + "HandlerCompletionP50Milliseconds": 774.3912, + "HandlerCompletionP99Milliseconds": 875.2828, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-tracked-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 71, + "Gen1Collections": 18, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1274.9199, + "MessagesPerSecond": 39218.150097115904, + "AllocatedBytesPerMessage": 11455.5288, + "CpuMilliseconds": 5633.295, + "AcceptanceP50Milliseconds": 0.0187, + "AcceptanceP99Milliseconds": 5.3382, + "HandlerCompletionP50Milliseconds": 0.2003, + "HandlerCompletionP99Milliseconds": 8.1104, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-1", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 113, + "Gen1Collections": 29, + "Gen2Collections": 10, + "ElapsedMilliseconds": 1026.2934, + "MessagesPerSecond": 97438.02308384718, + "AllocatedBytesPerMessage": 8685.74264, + "CpuMilliseconds": 3184.163, + "AcceptanceP50Milliseconds": 0.0051, + "AcceptanceP99Milliseconds": 0.009, + "HandlerCompletionP50Milliseconds": 220.5677, + "HandlerCompletionP99Milliseconds": 301.8298, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-1", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 109, + "Gen1Collections": 29, + "Gen2Collections": 11, + "ElapsedMilliseconds": 991.1681, + "MessagesPerSecond": 100891.05975061143, + "AllocatedBytesPerMessage": 8271.42648, + "CpuMilliseconds": 1717.055, + "AcceptanceP50Milliseconds": 0.0052, + "AcceptanceP99Milliseconds": 0.0126, + "HandlerCompletionP50Milliseconds": 165.9661, + "HandlerCompletionP99Milliseconds": 267.992, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-1", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 80, + "Gen1Collections": 20, + "Gen2Collections": 6, + "ElapsedMilliseconds": 826.6044, + "MessagesPerSecond": 120976.85422434237, + "AllocatedBytesPerMessage": 6416.3572, + "CpuMilliseconds": 2622.332, + "AcceptanceP50Milliseconds": 0.0024, + "AcceptanceP99Milliseconds": 0.0064, + "HandlerCompletionP50Milliseconds": 245.4687, + "HandlerCompletionP99Milliseconds": 414.2613, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-8", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 103, + "Gen1Collections": 21, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1177.6671, + "MessagesPerSecond": 84913.63985628875, + "AllocatedBytesPerMessage": 8339.86192, + "CpuMilliseconds": 4378.531, + "AcceptanceP50Milliseconds": 0.0117, + "AcceptanceP99Milliseconds": 0.0331, + "HandlerCompletionP50Milliseconds": 567.3213, + "HandlerCompletionP99Milliseconds": 728.7036, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-8", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 97, + "Gen1Collections": 20, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1127.1599, + "MessagesPerSecond": 88718.55714526394, + "AllocatedBytesPerMessage": 7933.9924, + "CpuMilliseconds": 3001.382, + "AcceptanceP50Milliseconds": 0.0118, + "AcceptanceP99Milliseconds": 0.0319, + "HandlerCompletionP50Milliseconds": 515.1992, + "HandlerCompletionP99Milliseconds": 683.588, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-8", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 77, + "Gen1Collections": 16, + "Gen2Collections": 4, + "ElapsedMilliseconds": 564.7504, + "MessagesPerSecond": 177069.37436432095, + "AllocatedBytesPerMessage": 6131.3216, + "CpuMilliseconds": 5392.001, + "AcceptanceP50Milliseconds": 0.0103, + "AcceptanceP99Milliseconds": 0.0317, + "HandlerCompletionP50Milliseconds": 234.4283, + "HandlerCompletionP99Milliseconds": 252.7088, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 58, + "Gen1Collections": 52, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3262.8414, + "MessagesPerSecond": 3064.813386271242, + "AllocatedBytesPerMessage": 47026.0128, + "CpuMilliseconds": 5987.839, + "AcceptanceP50Milliseconds": 13.7954, + "AcceptanceP99Milliseconds": 49.163, + "HandlerCompletionP50Milliseconds": 589.2781, + "HandlerCompletionP99Milliseconds": 800.9062, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 57, + "Gen1Collections": 53, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3240.7072, + "MessagesPerSecond": 3085.7462223060447, + "AllocatedBytesPerMessage": 46627.0296, + "CpuMilliseconds": 6209.664, + "AcceptanceP50Milliseconds": 14.0778, + "AcceptanceP99Milliseconds": 50.4635, + "HandlerCompletionP50Milliseconds": 621.4341, + "HandlerCompletionP99Milliseconds": 956.737, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 48, + "Gen1Collections": 28, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3586.775, + "MessagesPerSecond": 2788.0198785817342, + "AllocatedBytesPerMessage": 39814.6216, + "CpuMilliseconds": 4867.947, + "AcceptanceP50Milliseconds": 9.4236, + "AcceptanceP99Milliseconds": 18.8977, + "HandlerCompletionP50Milliseconds": 1354.6324, + "HandlerCompletionP99Milliseconds": 1739.9177, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 60, + "Gen1Collections": 23, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1289.1503, + "MessagesPerSecond": 7757.047413323334, + "AllocatedBytesPerMessage": 45863.8616, + "CpuMilliseconds": 5891.335, + "AcceptanceP50Milliseconds": 4.1647, + "AcceptanceP99Milliseconds": 9.5431, + "HandlerCompletionP50Milliseconds": 494.1496, + "HandlerCompletionP99Milliseconds": 594.7043, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 59, + "Gen1Collections": 24, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1315.7818, + "MessagesPerSecond": 7600.0443234584955, + "AllocatedBytesPerMessage": 45466.6472, + "CpuMilliseconds": 5870.868, + "AcceptanceP50Milliseconds": 4.5408, + "AcceptanceP99Milliseconds": 11.553, + "HandlerCompletionP50Milliseconds": 512.2408, + "HandlerCompletionP99Milliseconds": 567.0245, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 33, + "Gen1Collections": 10, + "Gen2Collections": 2, + "ElapsedMilliseconds": 930.8194, + "MessagesPerSecond": 10743.22258431657, + "AllocatedBytesPerMessage": 25857.7488, + "CpuMilliseconds": 4446.052, + "AcceptanceP50Milliseconds": 2.3707, + "AcceptanceP99Milliseconds": 7.9752, + "HandlerCompletionP50Milliseconds": 395.2164, + "HandlerCompletionP99Milliseconds": 503.653, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 106, + "Gen1Collections": 95, + "Gen2Collections": 1, + "ElapsedMilliseconds": 3708.7346, + "MessagesPerSecond": 2696.337451593328, + "AllocatedBytesPerMessage": 84333.5328, + "CpuMilliseconds": 12704.218, + "AcceptanceP50Milliseconds": 15.6119, + "AcceptanceP99Milliseconds": 27.564, + "HandlerCompletionP50Milliseconds": 1054.2966, + "HandlerCompletionP99Milliseconds": 1201.587, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 105, + "Gen1Collections": 94, + "Gen2Collections": 1, + "ElapsedMilliseconds": 3962.9561, + "MessagesPerSecond": 2523.3688558901777, + "AllocatedBytesPerMessage": 83917.472, + "CpuMilliseconds": 13723.667, + "AcceptanceP50Milliseconds": 14.9208, + "AcceptanceP99Milliseconds": 28.0182, + "HandlerCompletionP50Milliseconds": 1056.6053, + "HandlerCompletionP99Milliseconds": 1234.1299, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 74, + "Gen1Collections": 30, + "Gen2Collections": 1, + "ElapsedMilliseconds": 3700.4181, + "MessagesPerSecond": 2702.397331804209, + "AllocatedBytesPerMessage": 59327.6504, + "CpuMilliseconds": 10742.672, + "AcceptanceP50Milliseconds": 10.723, + "AcceptanceP99Milliseconds": 21.1192, + "HandlerCompletionP50Milliseconds": 1569.9301, + "HandlerCompletionP99Milliseconds": 1900.0031, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 194, + "Gen1Collections": 38, + "Gen2Collections": 4, + "ElapsedMilliseconds": 2059.8645, + "MessagesPerSecond": 97093.76514814445, + "AllocatedBytesPerMessage": 7933.04308, + "CpuMilliseconds": 10500.649, + "AcceptanceP50Milliseconds": 0.0227, + "AcceptanceP99Milliseconds": 0.2027, + "HandlerCompletionP50Milliseconds": 1108.2991, + "HandlerCompletionP99Milliseconds": 1203.4381, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 153, + "Gen1Collections": 30, + "Gen2Collections": 5, + "ElapsedMilliseconds": 1195.0283, + "MessagesPerSecond": 167360.0533142186, + "AllocatedBytesPerMessage": 6173.37708, + "CpuMilliseconds": 12646.6, + "AcceptanceP50Milliseconds": 0.0121, + "AcceptanceP99Milliseconds": 0.2048, + "HandlerCompletionP50Milliseconds": 458.9281, + "HandlerCompletionP99Milliseconds": 503.1059, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 204, + "Gen1Collections": 38, + "Gen2Collections": 4, + "ElapsedMilliseconds": 2323.5597, + "MessagesPerSecond": 86074.82734357976, + "AllocatedBytesPerMessage": 8369.85464, + "CpuMilliseconds": 13965.884, + "AcceptanceP50Milliseconds": 0.0277, + "AcceptanceP99Milliseconds": 0.2141, + "HandlerCompletionP50Milliseconds": 1271.3037, + "HandlerCompletionP99Milliseconds": 1356.891, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 92, + "Gen1Collections": 28, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1538.8768, + "MessagesPerSecond": 32491.22996720725, + "AllocatedBytesPerMessage": 15146.49648, + "CpuMilliseconds": 5592.862, + "AcceptanceP50Milliseconds": 0.02, + "AcceptanceP99Milliseconds": 5.5031, + "HandlerCompletionP50Milliseconds": 785.3157, + "HandlerCompletionP99Milliseconds": 936.5879, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-tracked-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 72, + "Gen1Collections": 19, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1363.9657, + "MessagesPerSecond": 36657.81331598001, + "AllocatedBytesPerMessage": 11486.63328, + "CpuMilliseconds": 5589.893, + "AcceptanceP50Milliseconds": 0.017, + "AcceptanceP99Milliseconds": 4.2634, + "HandlerCompletionP50Milliseconds": 0.1149, + "HandlerCompletionP99Milliseconds": 7.6546, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "50000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 50000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 94, + "Gen1Collections": 29, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1440.5239, + "MessagesPerSecond": 34709.5941969446, + "AllocatedBytesPerMessage": 15521.9904, + "CpuMilliseconds": 5490.054, + "AcceptanceP50Milliseconds": 0.0172, + "AcceptanceP99Milliseconds": 5.2926, + "HandlerCompletionP50Milliseconds": 696.2633, + "HandlerCompletionP99Milliseconds": 929.1987, + "UniqueProcessed": 50000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-1", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 109, + "Gen1Collections": 29, + "Gen2Collections": 11, + "ElapsedMilliseconds": 994.8361, + "MessagesPerSecond": 100519.07042778202, + "AllocatedBytesPerMessage": 8271.44536, + "CpuMilliseconds": 1775.598, + "AcceptanceP50Milliseconds": 0.0049, + "AcceptanceP99Milliseconds": 0.0099, + "HandlerCompletionP50Milliseconds": 171.7537, + "HandlerCompletionP99Milliseconds": 281.9949, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-1", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 79, + "Gen1Collections": 19, + "Gen2Collections": 5, + "ElapsedMilliseconds": 906.3209, + "MessagesPerSecond": 110336.1954910231, + "AllocatedBytesPerMessage": 6411.8092, + "CpuMilliseconds": 3114.118, + "AcceptanceP50Milliseconds": 0.0035, + "AcceptanceP99Milliseconds": 0.0086, + "HandlerCompletionP50Milliseconds": 321.0898, + "HandlerCompletionP99Milliseconds": 406.5647, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-1", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "1" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 1, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 110, + "Gen1Collections": 28, + "Gen2Collections": 8, + "ElapsedMilliseconds": 1123.5237, + "MessagesPerSecond": 89005.68808650854, + "AllocatedBytesPerMessage": 8686.9144, + "CpuMilliseconds": 3533.672, + "AcceptanceP50Milliseconds": 0.0049, + "AcceptanceP99Milliseconds": 0.0128, + "HandlerCompletionP50Milliseconds": 228.4521, + "HandlerCompletionP99Milliseconds": 365.5762, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-8", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 97, + "Gen1Collections": 21, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1059.8946, + "MessagesPerSecond": 94349.00413682644, + "AllocatedBytesPerMessage": 7932.39912, + "CpuMilliseconds": 3279.651, + "AcceptanceP50Milliseconds": 0.0127, + "AcceptanceP99Milliseconds": 0.0351, + "HandlerCompletionP50Milliseconds": 518.4178, + "HandlerCompletionP99Milliseconds": 622.4809, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "memory-8", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 77, + "Gen1Collections": 17, + "Gen2Collections": 4, + "ElapsedMilliseconds": 560.7781, + "MessagesPerSecond": 178323.65422258823, + "AllocatedBytesPerMessage": 6123.93776, + "CpuMilliseconds": 5258.534, + "AcceptanceP50Milliseconds": 0.0098, + "AcceptanceP99Milliseconds": 0.0472, + "HandlerCompletionP50Milliseconds": 238.192, + "HandlerCompletionP99Milliseconds": 248.7626, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-8", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "100000", + "--concurrency", + "8" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 100000, + "Concurrency": 8, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 102, + "Gen1Collections": 21, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1177.6283, + "MessagesPerSecond": 84916.43755504178, + "AllocatedBytesPerMessage": 8337.3516, + "CpuMilliseconds": 4392.612, + "AcceptanceP50Milliseconds": 0.0108, + "AcceptanceP99Milliseconds": 0.035, + "HandlerCompletionP50Milliseconds": 498.2995, + "HandlerCompletionP99Milliseconds": 758.1277, + "UniqueProcessed": 100000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 57, + "Gen1Collections": 53, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3828.4912, + "MessagesPerSecond": 2611.9950334481637, + "AllocatedBytesPerMessage": 46621.1192, + "CpuMilliseconds": 6001.177, + "AcceptanceP50Milliseconds": 17.2105, + "AcceptanceP99Milliseconds": 33.0872, + "HandlerCompletionP50Milliseconds": 725.3644, + "HandlerCompletionP99Milliseconds": 879.302, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 48, + "Gen1Collections": 35, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3130.4728, + "MessagesPerSecond": 3194.4056501624927, + "AllocatedBytesPerMessage": 39792.0072, + "CpuMilliseconds": 5210.003, + "AcceptanceP50Milliseconds": 8.7862, + "AcceptanceP99Milliseconds": 15.811, + "HandlerCompletionP50Milliseconds": 1310.8215, + "HandlerCompletionP99Milliseconds": 1692.9045, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 58, + "Gen1Collections": 55, + "Gen2Collections": 0, + "ElapsedMilliseconds": 3655.4681, + "MessagesPerSecond": 2735.627757222119, + "AllocatedBytesPerMessage": 47062.0072, + "CpuMilliseconds": 6068.258, + "AcceptanceP50Milliseconds": 15.2003, + "AcceptanceP99Milliseconds": 30.1908, + "HandlerCompletionP50Milliseconds": 723.5064, + "HandlerCompletionP99Milliseconds": 907.1339, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 58, + "Gen1Collections": 23, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1314.0055, + "MessagesPerSecond": 7610.31822165128, + "AllocatedBytesPerMessage": 45442.4688, + "CpuMilliseconds": 5833.678, + "AcceptanceP50Milliseconds": 4.2865, + "AcceptanceP99Milliseconds": 9.0971, + "HandlerCompletionP50Milliseconds": 500.9175, + "HandlerCompletionP99Milliseconds": 608.3963, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "pr149", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 33, + "Gen1Collections": 11, + "Gen2Collections": 2, + "ElapsedMilliseconds": 941.7012, + "MessagesPerSecond": 10619.079597647322, + "AllocatedBytesPerMessage": 25880.0384, + "CpuMilliseconds": 3980.716, + "AcceptanceP50Milliseconds": 2.6147, + "AcceptanceP99Milliseconds": 8.8347, + "HandlerCompletionP50Milliseconds": 382.0151, + "HandlerCompletionP99Milliseconds": 471.5875, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 59, + "Gen1Collections": 24, + "Gen2Collections": 3, + "ElapsedMilliseconds": 1246.1479, + "MessagesPerSecond": 8024.729648864312, + "AllocatedBytesPerMessage": 45851.7224, + "CpuMilliseconds": 6096.837, + "AcceptanceP50Milliseconds": 3.9121, + "AcceptanceP99Milliseconds": 9.1611, + "HandlerCompletionP50Milliseconds": 484.4801, + "HandlerCompletionP99Milliseconds": 590.2952, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "localstack-redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 105, + "Gen1Collections": 93, + "Gen2Collections": 1, + "ElapsedMilliseconds": 3838.0068, + "MessagesPerSecond": 2605.5190939213553, + "AllocatedBytesPerMessage": 83997.0312, + "CpuMilliseconds": 12914.266, + "AcceptanceP50Milliseconds": 15.2478, + "AcceptanceP99Milliseconds": 50.9925, + "HandlerCompletionP50Milliseconds": 1120.5989, + "HandlerCompletionP99Milliseconds": 1281.8331, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "pr149", + "Scenario": "localstack-redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/mediator-core-alternative/.dependencies/PR149Benchmark/bin/Release/net10.0/PR149Benchmark.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "pr149", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 74, + "Gen1Collections": 29, + "Gen2Collections": 1, + "ElapsedMilliseconds": 3494.0495, + "MessagesPerSecond": 2862.0086807585294, + "AllocatedBytesPerMessage": 59337.2936, + "CpuMilliseconds": 10344.158, + "AcceptanceP50Milliseconds": 10.1863, + "AcceptanceP99Milliseconds": 18.9097, + "HandlerCompletionP50Milliseconds": 1445.5999, + "HandlerCompletionP99Milliseconds": 1841.1713, + "UniqueProcessed": 10000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "localstack-redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "10000", + "--concurrency", + "64", + "--aws", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "localstack", + "Messages": 10000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 106, + "Gen1Collections": 87, + "Gen2Collections": 1, + "ElapsedMilliseconds": 3769.942, + "MessagesPerSecond": 2652.560702525397, + "AllocatedBytesPerMessage": 84234.0664, + "CpuMilliseconds": 13715.548, + "AcceptanceP50Milliseconds": 15.399, + "AcceptanceP99Milliseconds": 27.7885, + "HandlerCompletionP50Milliseconds": 1082.5957, + "HandlerCompletionP99Milliseconds": 1306.3156, + "UniqueProcessed": 10000, + "Duplicates": 0 + } +] diff --git a/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/matrix-summary.json b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/matrix-summary.json new file mode 100644 index 000000000..847ff3325 --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/matrix-summary.json @@ -0,0 +1,163 @@ +{ + "memory-64": { + "pr149": { + "MessagesPerSecond": 181195.2, + "AllocatedBytesPerMessage": 6173.377, + "AcceptanceP99Milliseconds": 0.249, + "HandlerCompletionP99Milliseconds": 434.993, + "CpuMilliseconds": 12292.901 + }, + "before": { + "MessagesPerSecond": 95696.431, + "AllocatedBytesPerMessage": 8357.98, + "AcceptanceP99Milliseconds": 0.214, + "HandlerCompletionP99Milliseconds": 1270.102, + "CpuMilliseconds": 10981.601 + }, + "after": { + "MessagesPerSecond": 100346.194, + "AllocatedBytesPerMessage": 7928.319, + "AcceptanceP99Milliseconds": 0.225, + "HandlerCompletionP99Milliseconds": 1203.438, + "CpuMilliseconds": 9085.144 + } + }, + "memory-tracked-64": { + "pr149": { + "MessagesPerSecond": 36657.813, + "AllocatedBytesPerMessage": 11482.16, + "AcceptanceP99Milliseconds": 4.734, + "HandlerCompletionP99Milliseconds": 7.966, + "CpuMilliseconds": 5633.295 + }, + "before": { + "MessagesPerSecond": 33238.32, + "AllocatedBytesPerMessage": 15534.836, + "AcceptanceP99Milliseconds": 5.293, + "HandlerCompletionP99Milliseconds": 929.199, + "CpuMilliseconds": 5490.054 + }, + "after": { + "MessagesPerSecond": 34111.182, + "AllocatedBytesPerMessage": 15146.496, + "AcceptanceP99Milliseconds": 5.263, + "HandlerCompletionP99Milliseconds": 875.283, + "CpuMilliseconds": 5304.177 + } + }, + "memory-1": { + "pr149": { + "MessagesPerSecond": 110805.448, + "AllocatedBytesPerMessage": 6413.623, + "AcceptanceP99Milliseconds": 0.009, + "HandlerCompletionP99Milliseconds": 414.261, + "CpuMilliseconds": 3113.494 + }, + "before": { + "MessagesPerSecond": 93045.405, + "AllocatedBytesPerMessage": 8685.743, + "AcceptanceP99Milliseconds": 0.012, + "HandlerCompletionP99Milliseconds": 321.085, + "CpuMilliseconds": 3332.278 + }, + "after": { + "MessagesPerSecond": 100536.867, + "AllocatedBytesPerMessage": 8271.445, + "AcceptanceP99Milliseconds": 0.013, + "HandlerCompletionP99Milliseconds": 281.551, + "CpuMilliseconds": 1770.779 + } + }, + "memory-8": { + "pr149": { + "MessagesPerSecond": 177069.374, + "AllocatedBytesPerMessage": 6131.322, + "AcceptanceP99Milliseconds": 0.032, + "HandlerCompletionP99Milliseconds": 252.709, + "CpuMilliseconds": 5392.001 + }, + "before": { + "MessagesPerSecond": 84916.438, + "AllocatedBytesPerMessage": 8339.862, + "AcceptanceP99Milliseconds": 0.033, + "HandlerCompletionP99Milliseconds": 728.704, + "CpuMilliseconds": 4378.531 + }, + "after": { + "MessagesPerSecond": 91161.226, + "AllocatedBytesPerMessage": 7933.992, + "AcceptanceP99Milliseconds": 0.032, + "HandlerCompletionP99Milliseconds": 677.543, + "CpuMilliseconds": 3092.042 + } + }, + "localstack-64": { + "pr149": { + "MessagesPerSecond": 2788.02, + "AllocatedBytesPerMessage": 39792.007, + "AcceptanceP99Milliseconds": 18.898, + "HandlerCompletionP99Milliseconds": 1739.918, + "CpuMilliseconds": 5140.036 + }, + "before": { + "MessagesPerSecond": 3064.813, + "AllocatedBytesPerMessage": 47026.013, + "AcceptanceP99Milliseconds": 47.544, + "HandlerCompletionP99Milliseconds": 800.906, + "CpuMilliseconds": 6059.97 + }, + "after": { + "MessagesPerSecond": 2967.626, + "AllocatedBytesPerMessage": 46623.046, + "AcceptanceP99Milliseconds": 33.087, + "HandlerCompletionP99Milliseconds": 879.302, + "CpuMilliseconds": 6001.177 + } + }, + "redis-64": { + "pr149": { + "MessagesPerSecond": 10619.08, + "AllocatedBytesPerMessage": 25866.052, + "AcceptanceP99Milliseconds": 8.835, + "HandlerCompletionP99Milliseconds": 503.653, + "CpuMilliseconds": 4446.052 + }, + "before": { + "MessagesPerSecond": 7757.047, + "AllocatedBytesPerMessage": 45851.722, + "AcceptanceP99Milliseconds": 9.543, + "HandlerCompletionP99Milliseconds": 594.704, + "CpuMilliseconds": 6096.837 + }, + "after": { + "MessagesPerSecond": 7600.044, + "AllocatedBytesPerMessage": 45449.546, + "AcceptanceP99Milliseconds": 11.3, + "HandlerCompletionP99Milliseconds": 608.396, + "CpuMilliseconds": 5870.868 + } + }, + "localstack-redis-64": { + "pr149": { + "MessagesPerSecond": 2702.397, + "AllocatedBytesPerMessage": 59327.65, + "AcceptanceP99Milliseconds": 21.119, + "HandlerCompletionP99Milliseconds": 1900.003, + "CpuMilliseconds": 10742.672 + }, + "before": { + "MessagesPerSecond": 2652.561, + "AllocatedBytesPerMessage": 84284.622, + "AcceptanceP99Milliseconds": 27.788, + "HandlerCompletionP99Milliseconds": 1281.852, + "CpuMilliseconds": 13132.348 + }, + "after": { + "MessagesPerSecond": 2582.237, + "AllocatedBytesPerMessage": 83929.978, + "AcceptanceP99Milliseconds": 50.992, + "HandlerCompletionP99Milliseconds": 1281.833, + "CpuMilliseconds": 12914.266 + } + } +} diff --git a/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/recovery-results.json b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/recovery-results.json new file mode 100644 index 000000000..6ae317eb8 --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/recovery-results.json @@ -0,0 +1,76 @@ +{ + "Prefix": "recovery-6aa4c9bd1dcf4d7d8f984951ad3ed2ac", + "Runtime": ".NET 10.0.12", + "ProducerSeconds": 600, + "ElapsedSeconds": 601.2048925, + "Accepted": 69354, + "Completed": 69314, + "QueuedCancelled": 20, + "RunningCancelled": 20, + "RunningCancellationMilliseconds": 5037.8638, + "InterruptedAtKill": 32, + "RetriedJobs": 32, + "HandlerInvocations": 69366, + "DuplicateEffectAttempts": 0, + "Pending": 0, + "Failed": 0, + "DeadLettered": 0, + "Snapshots": [ + { + "Phase": "killed", + "ElapsedSeconds": 0.691541, + "ProcessId": 3980853, + "Count": 32, + "Workers": [] + }, + { + "Phase": "cancelled", + "ElapsedSeconds": 6.2457078, + "ProcessId": 0, + "Count": 40, + "Workers": [ + { + "Id": 3980986, + "WorkingSet64": 125853696, + "PeakWorkingSet64": 125853696 + }, + { + "Id": 3980987, + "WorkingSet64": 107061248, + "PeakWorkingSet64": 107061248 + } + ] + }, + { + "Phase": "graceful-stop", + "ElapsedSeconds": 26.3027001, + "ProcessId": 3980987, + "Count": 0, + "Workers": [ + { + "Id": 3980986, + "WorkingSet64": 132513792, + "PeakWorkingSet64": 132513792 + } + ] + }, + { + "Phase": "drained", + "ElapsedSeconds": 601.2036094, + "ProcessId": 0, + "Count": 69314, + "Workers": [ + { + "Id": 3980986, + "WorkingSet64": 133586944, + "PeakWorkingSet64": 133586944 + }, + { + "Id": 3983917, + "WorkingSet64": 126136320, + "PeakWorkingSet64": 126136320 + } + ] + } + ] +} \ No newline at end of file diff --git a/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/tracked-validation-results.json b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/tracked-validation-results.json new file mode 100644 index 000000000..b6dacff99 --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/tracked-validation-results.json @@ -0,0 +1,1122 @@ +[ + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "90000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 90000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 168, + "Gen1Collections": 50, + "Gen2Collections": 5, + "ElapsedMilliseconds": 2555.7105, + "MessagesPerSecond": 35215.25618805416, + "AllocatedBytesPerMessage": 15582.915644444445, + "CpuMilliseconds": 9570.487, + "AcceptanceP50Milliseconds": 0.0203, + "AcceptanceP99Milliseconds": 6.3471, + "HandlerCompletionP50Milliseconds": 1382.3252, + "HandlerCompletionP99Milliseconds": 1530.5538, + "UniqueProcessed": 90000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "90000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 90000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 164, + "Gen1Collections": 49, + "Gen2Collections": 5, + "ElapsedMilliseconds": 2588.0001, + "MessagesPerSecond": 34775.88737342012, + "AllocatedBytesPerMessage": 15212.123466666666, + "CpuMilliseconds": 9335.046, + "AcceptanceP50Milliseconds": 0.0208, + "AcceptanceP99Milliseconds": 5.5537, + "HandlerCompletionP50Milliseconds": 1445.1393, + "HandlerCompletionP99Milliseconds": 1543.1829, + "UniqueProcessed": 90000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 176, + "Gen1Collections": 82, + "Gen2Collections": 5, + "ElapsedMilliseconds": 3926.2532, + "MessagesPerSecond": 7640.872473532781, + "AllocatedBytesPerMessage": 45844.5464, + "CpuMilliseconds": 16135.386, + "AcceptanceP50Milliseconds": 4.8468, + "AcceptanceP99Milliseconds": 9.3859, + "HandlerCompletionP50Milliseconds": 1579.6341, + "HandlerCompletionP99Milliseconds": 1651.8953, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 173, + "Gen1Collections": 73, + "Gen2Collections": 5, + "ElapsedMilliseconds": 3906.955, + "MessagesPerSecond": 7678.6141637157325, + "AllocatedBytesPerMessage": 45453.0344, + "CpuMilliseconds": 15146.066, + "AcceptanceP50Milliseconds": 4.7908, + "AcceptanceP99Milliseconds": 9.6002, + "HandlerCompletionP50Milliseconds": 1570.5433, + "HandlerCompletionP99Milliseconds": 1631.9695, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-default-delay-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64", + "--default-delay" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 204, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 2028.9903, + "MessagesPerSecond": 98571.19573218266, + "AllocatedBytesPerMessage": 8356.18652, + "CpuMilliseconds": 11236.692, + "AcceptanceP50Milliseconds": 0.0191, + "AcceptanceP99Milliseconds": 0.1203, + "HandlerCompletionP50Milliseconds": 1153.6213, + "HandlerCompletionP99Milliseconds": 1184.5474, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-default-delay-64", + "Iteration": 0, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64", + "--default-delay" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 194, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 2182.3837, + "MessagesPerSecond": 91642.9132054093, + "AllocatedBytesPerMessage": 7927.5958, + "CpuMilliseconds": 11075.746, + "AcceptanceP50Milliseconds": 0.0261, + "AcceptanceP99Milliseconds": 0.2021, + "HandlerCompletionP50Milliseconds": 1230.315, + "HandlerCompletionP99Milliseconds": 1314.1957, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "90000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 90000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 165, + "Gen1Collections": 49, + "Gen2Collections": 5, + "ElapsedMilliseconds": 2694.5856, + "MessagesPerSecond": 33400.31209251619, + "AllocatedBytesPerMessage": 15212.827377777778, + "CpuMilliseconds": 10026.51, + "AcceptanceP50Milliseconds": 0.0201, + "AcceptanceP99Milliseconds": 6.8068, + "HandlerCompletionP50Milliseconds": 1534.3933, + "HandlerCompletionP99Milliseconds": 1658.593, + "UniqueProcessed": 90000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "90000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 90000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 169, + "Gen1Collections": 51, + "Gen2Collections": 5, + "ElapsedMilliseconds": 2474.9216, + "MessagesPerSecond": 36364.78828258641, + "AllocatedBytesPerMessage": 15591.1688, + "CpuMilliseconds": 10127.019, + "AcceptanceP50Milliseconds": 0.0211, + "AcceptanceP99Milliseconds": 5.7428, + "HandlerCompletionP50Milliseconds": 1417.5633, + "HandlerCompletionP99Milliseconds": 1495.986, + "UniqueProcessed": 90000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 174, + "Gen1Collections": 84, + "Gen2Collections": 5, + "ElapsedMilliseconds": 4086.913, + "MessagesPerSecond": 7340.503700470257, + "AllocatedBytesPerMessage": 45455.5352, + "CpuMilliseconds": 16164.155, + "AcceptanceP50Milliseconds": 5.4579, + "AcceptanceP99Milliseconds": 9.8226, + "HandlerCompletionP50Milliseconds": 1586.8895, + "HandlerCompletionP99Milliseconds": 1774.69, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 177, + "Gen1Collections": 76, + "Gen2Collections": 6, + "ElapsedMilliseconds": 3919.6785, + "MessagesPerSecond": 7653.688944131515, + "AllocatedBytesPerMessage": 45865.14666666667, + "CpuMilliseconds": 15516.075, + "AcceptanceP50Milliseconds": 4.8736, + "AcceptanceP99Milliseconds": 10.405, + "HandlerCompletionP50Milliseconds": 1618.309, + "HandlerCompletionP99Milliseconds": 1669.7024, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-default-delay-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64", + "--default-delay" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 194, + "Gen1Collections": 38, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1736.0142, + "MessagesPerSecond": 115206.43091513883, + "AllocatedBytesPerMessage": 7916.10868, + "CpuMilliseconds": 8012.822, + "AcceptanceP50Milliseconds": 0.0159, + "AcceptanceP99Milliseconds": 0.0978, + "HandlerCompletionP50Milliseconds": 965.7627, + "HandlerCompletionP99Milliseconds": 1074.3805, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-default-delay-64", + "Iteration": 1, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64", + "--default-delay" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 204, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 2127.1523, + "MessagesPerSecond": 94022.41673057448, + "AllocatedBytesPerMessage": 8355.49464, + "CpuMilliseconds": 10038.661, + "AcceptanceP50Milliseconds": 0.0151, + "AcceptanceP99Milliseconds": 0.1325, + "HandlerCompletionP50Milliseconds": 1197.8368, + "HandlerCompletionP99Milliseconds": 1278.8619, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "90000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 90000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 169, + "Gen1Collections": 51, + "Gen2Collections": 5, + "ElapsedMilliseconds": 2657.8349, + "MessagesPerSecond": 33862.148472803936, + "AllocatedBytesPerMessage": 15595.857066666667, + "CpuMilliseconds": 9818.022, + "AcceptanceP50Milliseconds": 0.0216, + "AcceptanceP99Milliseconds": 6.204, + "HandlerCompletionP50Milliseconds": 1466.699, + "HandlerCompletionP99Milliseconds": 1594.1831, + "UniqueProcessed": 90000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "90000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 90000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 164, + "Gen1Collections": 49, + "Gen2Collections": 5, + "ElapsedMilliseconds": 2490.0877, + "MessagesPerSecond": 36143.305314106, + "AllocatedBytesPerMessage": 15211.481155555555, + "CpuMilliseconds": 9136.529, + "AcceptanceP50Milliseconds": 0.0198, + "AcceptanceP99Milliseconds": 5.0248, + "HandlerCompletionP50Milliseconds": 1297.7483, + "HandlerCompletionP99Milliseconds": 1514.6454, + "UniqueProcessed": 90000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 176, + "Gen1Collections": 81, + "Gen2Collections": 5, + "ElapsedMilliseconds": 3918.5331, + "MessagesPerSecond": 7655.926142361794, + "AllocatedBytesPerMessage": 45856.069866666665, + "CpuMilliseconds": 16414.637, + "AcceptanceP50Milliseconds": 4.6943, + "AcceptanceP99Milliseconds": 10.0526, + "HandlerCompletionP50Milliseconds": 1579.9243, + "HandlerCompletionP99Milliseconds": 1620.6491, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 174, + "Gen1Collections": 72, + "Gen2Collections": 5, + "ElapsedMilliseconds": 3896.1103, + "MessagesPerSecond": 7699.987343787469, + "AllocatedBytesPerMessage": 45451.42773333333, + "CpuMilliseconds": 14185.274, + "AcceptanceP50Milliseconds": 4.8707, + "AcceptanceP99Milliseconds": 10.0492, + "HandlerCompletionP50Milliseconds": 1552.3412, + "HandlerCompletionP99Milliseconds": 1581.9087, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-default-delay-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64", + "--default-delay" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 204, + "Gen1Collections": 38, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1969.1078, + "MessagesPerSecond": 101568.84249811005, + "AllocatedBytesPerMessage": 8356.70484, + "CpuMilliseconds": 10723.951, + "AcceptanceP50Milliseconds": 0.0174, + "AcceptanceP99Milliseconds": 0.1487, + "HandlerCompletionP50Milliseconds": 1123.0295, + "HandlerCompletionP99Milliseconds": 1176.7406, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-default-delay-64", + "Iteration": 2, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64", + "--default-delay" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 193, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1801.4799, + "MessagesPerSecond": 111019.8343040075, + "AllocatedBytesPerMessage": 7917.75396, + "CpuMilliseconds": 8364.772, + "AcceptanceP50Milliseconds": 0.0179, + "AcceptanceP99Milliseconds": 0.4879, + "HandlerCompletionP50Milliseconds": 996.8247, + "HandlerCompletionP99Milliseconds": 1095.9982, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 3, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "90000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 90000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 164, + "Gen1Collections": 49, + "Gen2Collections": 5, + "ElapsedMilliseconds": 2569.5997, + "MessagesPerSecond": 35024.910689396485, + "AllocatedBytesPerMessage": 15205.701688888888, + "CpuMilliseconds": 9394.309, + "AcceptanceP50Milliseconds": 0.0202, + "AcceptanceP99Milliseconds": 6.0327, + "HandlerCompletionP50Milliseconds": 1367.0448, + "HandlerCompletionP99Milliseconds": 1526.037, + "UniqueProcessed": 90000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 3, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "90000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 90000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 169, + "Gen1Collections": 51, + "Gen2Collections": 5, + "ElapsedMilliseconds": 2670.3087, + "MessagesPerSecond": 33703.96838388011, + "AllocatedBytesPerMessage": 15599.605422222223, + "CpuMilliseconds": 10080.167, + "AcceptanceP50Milliseconds": 0.0211, + "AcceptanceP99Milliseconds": 5.5349, + "HandlerCompletionP50Milliseconds": 1489.8205, + "HandlerCompletionP99Milliseconds": 1603.059, + "UniqueProcessed": 90000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 3, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 174, + "Gen1Collections": 77, + "Gen2Collections": 5, + "ElapsedMilliseconds": 4294.0012, + "MessagesPerSecond": 6986.490828181417, + "AllocatedBytesPerMessage": 45458.39653333333, + "CpuMilliseconds": 15336.79, + "AcceptanceP50Milliseconds": 5.633, + "AcceptanceP99Milliseconds": 12.4905, + "HandlerCompletionP50Milliseconds": 1744.6957, + "HandlerCompletionP99Milliseconds": 2093.9035, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 3, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 176, + "Gen1Collections": 78, + "Gen2Collections": 5, + "ElapsedMilliseconds": 3748.8152, + "MessagesPerSecond": 8002.528372164091, + "AllocatedBytesPerMessage": 45859.29386666667, + "CpuMilliseconds": 15735.087, + "AcceptanceP50Milliseconds": 4.464, + "AcceptanceP99Milliseconds": 9.3401, + "HandlerCompletionP50Milliseconds": 1522.5429, + "HandlerCompletionP99Milliseconds": 1621.5506, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-default-delay-64", + "Iteration": 3, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64", + "--default-delay" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 194, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1998.5735, + "MessagesPerSecond": 100071.375908867, + "AllocatedBytesPerMessage": 7926.65684, + "CpuMilliseconds": 9560.425, + "AcceptanceP50Milliseconds": 0.0186, + "AcceptanceP99Milliseconds": 0.1087, + "HandlerCompletionP50Milliseconds": 1125.1987, + "HandlerCompletionP99Milliseconds": 1221.5615, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-default-delay-64", + "Iteration": 3, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64", + "--default-delay" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 204, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 2061.6498, + "MessagesPerSecond": 97009.68612613063, + "AllocatedBytesPerMessage": 8385.26068, + "CpuMilliseconds": 12035.4, + "AcceptanceP50Milliseconds": 0.0188, + "AcceptanceP99Milliseconds": 0.1815, + "HandlerCompletionP50Milliseconds": 1163.7953, + "HandlerCompletionP99Milliseconds": 1231.0459, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-tracked-64", + "Iteration": 4, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "90000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 90000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 168, + "Gen1Collections": 51, + "Gen2Collections": 5, + "ElapsedMilliseconds": 2564.661, + "MessagesPerSecond": 35092.35723551767, + "AllocatedBytesPerMessage": 15592.399911111112, + "CpuMilliseconds": 9928.729, + "AcceptanceP50Milliseconds": 0.0218, + "AcceptanceP99Milliseconds": 5.3282, + "HandlerCompletionP50Milliseconds": 1390.8353, + "HandlerCompletionP99Milliseconds": 1523.406, + "UniqueProcessed": 90000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-tracked-64", + "Iteration": 4, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "90000", + "--concurrency", + "64", + "--tracking" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 90000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 164, + "Gen1Collections": 49, + "Gen2Collections": 5, + "ElapsedMilliseconds": 2543.0962, + "MessagesPerSecond": 35389.93137577729, + "AllocatedBytesPerMessage": 15212.082044444445, + "CpuMilliseconds": 9505.442, + "AcceptanceP50Milliseconds": 0.0225, + "AcceptanceP99Milliseconds": 6.285, + "HandlerCompletionP50Milliseconds": 1408.3212, + "HandlerCompletionP99Milliseconds": 1439.1454, + "UniqueProcessed": 90000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "redis-64", + "Iteration": 4, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 176, + "Gen1Collections": 79, + "Gen2Collections": 5, + "ElapsedMilliseconds": 3888.2945, + "MessagesPerSecond": 7715.464967995608, + "AllocatedBytesPerMessage": 45858.12933333333, + "CpuMilliseconds": 16864.83, + "AcceptanceP50Milliseconds": 4.3951, + "AcceptanceP99Milliseconds": 9.2692, + "HandlerCompletionP50Milliseconds": 1550.4683, + "HandlerCompletionP99Milliseconds": 1719.4281, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "redis-64", + "Iteration": 4, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "30000", + "--concurrency", + "64", + "--tracking", + "--redis" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 30000, + "Concurrency": 64, + "Tracking": true, + "Runtime": ".NET 10.0.12", + "TrackingStore": "redis", + "Gen0Collections": 174, + "Gen1Collections": 76, + "Gen2Collections": 5, + "ElapsedMilliseconds": 4153.954, + "MessagesPerSecond": 7222.034716802354, + "AllocatedBytesPerMessage": 45445.9784, + "CpuMilliseconds": 15964.675, + "AcceptanceP50Milliseconds": 5.135, + "AcceptanceP99Milliseconds": 10.124, + "HandlerCompletionP50Milliseconds": 1710.7629, + "HandlerCompletionP99Milliseconds": 1733.8127, + "UniqueProcessed": 30000, + "Duplicates": 0 + }, + { + "Case": "before", + "Scenario": "memory-default-delay-64", + "Iteration": 4, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/before/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64", + "--default-delay" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 203, + "Gen1Collections": 38, + "Gen2Collections": 4, + "ElapsedMilliseconds": 2210.3049, + "MessagesPerSecond": 90485.25386701174, + "AllocatedBytesPerMessage": 8343.67804, + "CpuMilliseconds": 12360.078, + "AcceptanceP50Milliseconds": 0.0208, + "AcceptanceP99Milliseconds": 0.1846, + "HandlerCompletionP50Milliseconds": 1242.4169, + "HandlerCompletionP99Milliseconds": 1317.0695, + "UniqueProcessed": 200000, + "Duplicates": 0 + }, + { + "Case": "after", + "Scenario": "memory-default-delay-64", + "Iteration": 4, + "Attempt": 0, + "Exit": 0, + "Command": [ + "/usr/bin/dotnet", + "/tmp/production-pass-20260912/after/Foundatio.Mediator.Distributed.Benchmarks.dll", + "--count", + "200000", + "--concurrency", + "64", + "--default-delay" + ], + "Implementation": "foundatio-native", + "Layer": "mediator", + "Transport": "memory", + "Messages": 200000, + "Concurrency": 64, + "Tracking": false, + "Runtime": ".NET 10.0.12", + "TrackingStore": "memory", + "Gen0Collections": 194, + "Gen1Collections": 37, + "Gen2Collections": 4, + "ElapsedMilliseconds": 1924.4088, + "MessagesPerSecond": 103928.02194627254, + "AllocatedBytesPerMessage": 7919.37216, + "CpuMilliseconds": 10015.95, + "AcceptanceP50Milliseconds": 0.0248, + "AcceptanceP99Milliseconds": 0.248, + "HandlerCompletionP50Milliseconds": 1050.8031, + "HandlerCompletionP99Milliseconds": 1175.5397, + "UniqueProcessed": 200000, + "Duplicates": 0 + } +] diff --git a/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/tracked-validation-summary.json b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/tracked-validation-summary.json new file mode 100644 index 000000000..136327df4 --- /dev/null +++ b/benchmarks/Messaging/baselines/job-tracking-production-2026-09-12/tracked-validation-summary.json @@ -0,0 +1,50 @@ +{ + "memory-tracked-64": { + "before": { + "MessagesPerSecond": 35092.357, + "AllocatedBytesPerMessage": 15592.4, + "AcceptanceP99Milliseconds": 5.743, + "HandlerCompletionP99Milliseconds": 1530.554, + "CpuMilliseconds": 9928.729 + }, + "after": { + "MessagesPerSecond": 35024.911, + "AllocatedBytesPerMessage": 15212.082, + "AcceptanceP99Milliseconds": 6.033, + "HandlerCompletionP99Milliseconds": 1526.037, + "CpuMilliseconds": 9394.309 + } + }, + "redis-64": { + "before": { + "MessagesPerSecond": 7655.926, + "AllocatedBytesPerMessage": 45858.129, + "AcceptanceP99Milliseconds": 9.386, + "HandlerCompletionP99Milliseconds": 1651.895, + "CpuMilliseconds": 16135.386 + }, + "after": { + "MessagesPerSecond": 7340.504, + "AllocatedBytesPerMessage": 45453.034, + "AcceptanceP99Milliseconds": 10.049, + "HandlerCompletionP99Milliseconds": 1733.813, + "CpuMilliseconds": 15336.79 + } + }, + "memory-default-delay-64": { + "before": { + "MessagesPerSecond": 97009.686, + "AllocatedBytesPerMessage": 8356.187, + "AcceptanceP99Milliseconds": 0.149, + "HandlerCompletionP99Milliseconds": 1231.046, + "CpuMilliseconds": 11236.692 + }, + "after": { + "MessagesPerSecond": 103928.022, + "AllocatedBytesPerMessage": 7919.372, + "AcceptanceP99Milliseconds": 0.202, + "HandlerCompletionP99Milliseconds": 1175.54, + "CpuMilliseconds": 9560.425 + } + } +} diff --git a/benchmarks/Messaging/docker-compose.yml b/benchmarks/Messaging/docker-compose.yml new file mode 100644 index 000000000..b8edf4110 --- /dev/null +++ b/benchmarks/Messaging/docker-compose.yml @@ -0,0 +1,17 @@ +name: foundatio-messaging-perf +services: + redis: + image: redis:8.6-alpine + command: [redis-server, --appendonly, 'yes', --appendfsync, everysec] + cpus: 4 + mem_limit: 2g + ports: + - '127.0.0.1:16379:6379' + localstack: + image: localstack/localstack:3.8.1 + cpus: 4 + mem_limit: 3g + environment: + SERVICES: sqs,sns + ports: + - '127.0.0.1:24566:4566' diff --git a/benchmarks/Messaging/run.ps1 b/benchmarks/Messaging/run.ps1 new file mode 100644 index 000000000..9bbf6e76f --- /dev/null +++ b/benchmarks/Messaging/run.ps1 @@ -0,0 +1,80 @@ +param( + [ValidateSet('smoke', 'standard', 'extended', 'soak')][string]$Profile = 'standard', + [int]$Repetitions = 3, + [int]$Seconds = 15, + [int]$Warmup = 3, + [string[]]$Engines = @('foundatio-memory', 'masstransit-memory', 'foundatio-redis', 'foundatio-sqs', 'masstransit-sqs'), + [string]$OutputDirectory = (Join-Path $PSScriptRoot ('results/' + (Get-Date -Format 'yyyyMMdd-HHmmss'))), + [string]$DotnetPath = 'dotnet', + [switch]$NoBuild +) +$ErrorActionPreference = 'Stop' +$PSNativeCommandUseErrorActionPreference = $true +if ($Repetitions -lt 1 -or $Repetitions -gt 20) { throw 'Repetitions must be 1-20.' } +New-Item -ItemType Directory -Force $OutputDirectory | Out-Null +$OutputDirectory = (Resolve-Path $OutputDirectory).Path +if (@(Get-ChildItem -Force $OutputDirectory).Count -gt 0) { throw 'OutputDirectory must be empty so previous trials cannot be overwritten or mistaken for new results.' } +if (-not $NoBuild) { & dotnet build (Join-Path $PSScriptRoot 'Foundatio.Messaging.Benchmarks.csproj') -c Release --nologo } +$dll = Join-Path $PSScriptRoot 'bin/Release/net10.0/Foundatio.Messaging.Benchmarks.dll' +if (-not (Test-Path $dll)) { throw 'Build the benchmark before using -NoBuild.' } +& $DotnetPath --info | Set-Content (Join-Path $OutputDirectory 'dotnet-info.txt') +& git -C $PSScriptRoot rev-parse HEAD | Set-Content (Join-Path $OutputDirectory 'revision.txt') +& git -C $PSScriptRoot status --short | Set-Content (Join-Path $OutputDirectory 'working-tree.txt') +if (Test-Path '/proc/cpuinfo') { Get-Content '/proc/cpuinfo' | Select-Object -First 30 | Set-Content (Join-Path $OutputDirectory 'cpu.txt') } +if (Test-Path '/proc/loadavg') { Get-Content '/proc/loadavg' | Set-Content (Join-Path $OutputDirectory 'load-before.txt') } +$workloads = @( + @{ Name = 'queue-serial'; Scenario = 'queue'; Producers = 1; Consumers = 1; Subscribers = 1; Payload = 1024; Batch = 1 }, + @{ Name = 'queue-concurrent'; Scenario = 'queue'; Producers = 32; Consumers = 32; Subscribers = 1; Payload = 1024; Batch = 1 }, + @{ Name = 'pubsub-one'; Scenario = 'pubsub'; Producers = 32; Consumers = 32; Subscribers = 1; Payload = 1024; Batch = 1 }, + @{ Name = 'pubsub-four'; Scenario = 'pubsub'; Producers = 32; Consumers = 8; Subscribers = 4; Payload = 1024; Batch = 1 } +) +if ($Profile -eq 'extended') { + $workloads = @( + @{ Name = 'queue-16k'; Scenario = 'queue'; Producers = 32; Consumers = 32; Subscribers = 1; Payload = 16384; Batch = 1 }, + @{ Name = 'pubsub-four-16k'; Scenario = 'pubsub'; Producers = 32; Consumers = 8; Subscribers = 4; Payload = 16384; Batch = 1 }, + @{ Name = 'queue-batch10'; Scenario = 'queue'; Producers = 8; Consumers = 32; Subscribers = 1; Payload = 1024; Batch = 10 }, + @{ Name = 'pubsub-four-batch10'; Scenario = 'pubsub'; Producers = 8; Consumers = 8; Subscribers = 4; Payload = 1024; Batch = 10 } + ) +} +if ($Profile -eq 'smoke') { $Seconds = 1; $Warmup = 1; $Repetitions = 1; $workloads = @($workloads[1], $workloads[3]) } +if ($Profile -eq 'soak') { $Seconds = 120; $Warmup = 5; $Repetitions = 1; $workloads = @($workloads[1], $workloads[3]) } +$maxMessages = if ($Profile -eq 'soak') { 100000000 } else { 20000000 } +$cases = foreach ($engine in $Engines) { + if ($engine -notin @('foundatio-memory', 'masstransit-memory', 'foundatio-redis', 'foundatio-sqs', 'masstransit-sqs')) { throw "Unknown engine $engine" } + foreach ($workload in $workloads) { [pscustomobject]@{ Engine = $engine; Workload = $workload } } +} +$random = [System.Random]::new(533) +$failures = 0 +$index = 0 +foreach ($round in 1..$Repetitions) { + foreach ($case in ($cases | Sort-Object { $random.Next() })) { + $index++ + $w = $case.Workload + $parts = $case.Engine.Split('-') + $name = "round$round-$($case.Engine)-$($w.Name)" + $startedUtc = [DateTimeOffset]::UtcNow + Write-Host "[$index/$($cases.Count * $Repetitions)] $name" + $arguments = @($dll, '--engine', $parts[0], '--transport', $parts[1], '--scenario', $w.Scenario, + '--seconds', $Seconds, '--warmup', $Warmup, '--producers', $w.Producers, '--consumers', $w.Consumers, + '--prefetch', $w.Consumers, '--subscribers', $w.Subscribers, '--payload', $w.Payload, '--batch', $w.Batch, + '--outstanding', 1024, '--max-messages', $maxMessages, '--output', (Join-Path $OutputDirectory "$name.json")) + try { & $DotnetPath @arguments > (Join-Path $OutputDirectory "$name.log") 2>&1 } + catch { + $failures++ + $resultPath = Join-Path $OutputDirectory "$name.json" + if (-not (Test-Path $resultPath)) { + $failure = @{ + Success = $false + StartedUtc = $startedUtc + Error = "Worker exited without a result. " + ((Get-Content (Join-Path $OutputDirectory "$name.log") -Tail 20) -join "`n") + Options = @{ Engine = $parts[0]; Transport = $parts[1]; Scenario = $w.Scenario; DurationSeconds = $Seconds; WarmupSeconds = $Warmup; DrainSeconds = 120; MaxMessages = $maxMessages; ProducerConcurrency = $w.Producers; ConsumerConcurrency = $w.Consumers; Prefetch = $w.Consumers; DeliveryCopies = $w.Subscribers; PayloadBytes = $w.Payload; BatchSize = $w.Batch; RatePerSecond = 0; MaxOutstanding = 1024 } + } + $failure | ConvertTo-Json -Depth 5 | Set-Content $resultPath + } + Write-Warning "$name failed; retained its log and result." + } + } +} +if (Test-Path '/proc/loadavg') { Get-Content '/proc/loadavg' | Set-Content (Join-Path $OutputDirectory 'load-after.txt') } +& (Join-Path $PSScriptRoot 'summarize.ps1') -Directory $OutputDirectory +if ($failures -gt 0) { throw "$failures trials failed. Failed trials are excluded from rankings and listed in the report." } diff --git a/benchmarks/Messaging/summarize.ps1 b/benchmarks/Messaging/summarize.ps1 new file mode 100644 index 000000000..99a43c10f --- /dev/null +++ b/benchmarks/Messaging/summarize.ps1 @@ -0,0 +1,58 @@ +param([Parameter(Mandatory)][string]$Directory) +$ErrorActionPreference = 'Stop' +function Median($Values) { + $sorted = @($Values | Sort-Object) + if ($sorted.Count -eq 0) { return 0 } + if ($sorted.Count % 2) { return $sorted[[int][math]::Floor($sorted.Count / 2)] } + return ($sorted[$sorted.Count / 2 - 1] + $sorted[$sorted.Count / 2]) / 2 +} +$results = @(Get-ChildItem $Directory -Filter 'round*.json' | ForEach-Object { + $result = Get-Content $_.FullName -Raw | ConvertFrom-Json + $o = $result.Options + [pscustomobject]@{ File = $_.Name; Key = "$($o.Engine)/$($o.Transport) $($o.Scenario) p$($o.ProducerConcurrency) c$($o.ConsumerConcurrency) s$($o.DeliveryCopies) $($o.PayloadBytes)B b$($o.BatchSize) r$($o.RatePerSecond) w$($o.MaxOutstanding) pf$($o.Prefetch)"; Result = $result } +}) +$environments = @($results | Where-Object { $_.Result.Success } | ForEach-Object { + $e = $_.Result.Environment + $o = $_.Result.Options + "$($e.Runtime)|$($e.CoreClrSha256)|$($e.OS)|$($e.Architecture)|$($e.LogicalProcessors)|$($e.ServerGC)|$($e.Foundatio)|$($e.MassTransit)|$($e.SqsSdk)|$($e.SnsSdk)|$($o.DurationSeconds)|$($o.WarmupSeconds)|$($o.MaxMessages)" +} | Select-Object -Unique) +if ($environments.Count -gt 1) { throw 'Results mix runtime, library, duration or tracking configurations. Summarize each configuration in a separate directory.' } +$awsEnvironments = @($results | Where-Object { $_.Result.Success -and $_.Result.Options.Transport -eq 'sqs' } | ForEach-Object { + $e = $_.Result.Environment + "$($e.Broker)|$($e.AwsMode)|$($e.AwsRegion)" +} | Select-Object -Unique) +if ($awsEnvironments.Count -gt 1) { throw 'Results mix AWS modes or regions. Summarize LocalStack and each AWS region in separate directories.' } +$rows = @($results | Where-Object { $_.Result.Success } | Group-Object Key | ForEach-Object { + $metrics = @($_.Group.Result.Measurement) + [pscustomobject]@{ + Case = $_.Name; Trials = $_.Count + InputsPerSecond = [math]::Round((Median $metrics.InputsPerSecond), 1) + MinInputsPerSecond = [math]::Round(($metrics.InputsPerSecond | Measure-Object -Minimum).Minimum, 1) + MaxInputsPerSecond = [math]::Round(($metrics.InputsPerSecond | Measure-Object -Maximum).Maximum, 1) + DeliveriesPerSecond = [math]::Round((Median $metrics.DeliveriesPerSecond), 1) + P50Milliseconds = [math]::Round((Median $metrics.DeliveryLatency.P50Milliseconds), 3) + P95Milliseconds = [math]::Round((Median $metrics.DeliveryLatency.P95Milliseconds), 3) + P99Milliseconds = [math]::Round((Median $metrics.DeliveryLatency.P99Milliseconds), 3) + BytesPerInput = [math]::Round((Median $metrics.AllocatedBytesPerInput), 1) + CpuMillisecondsPerInput = [math]::Round((Median @($metrics | ForEach-Object { $_.CpuMilliseconds / $_.Inputs })), 4) + PeakWorkingSetMiB = [math]::Round((Median @($metrics | ForEach-Object { $_.PeakWorkingSetBytes / 1MB })), 1) + TotalInputs = ($metrics.Inputs | Measure-Object -Sum).Sum + Duplicates = ($metrics.Duplicates | Measure-Object -Sum).Sum + Missing = ($metrics.Missing | Measure-Object -Sum).Sum + } +}) +$rows | Export-Csv (Join-Path $Directory 'summary.csv') -NoTypeInformation +$lines = @('# Messaging benchmark results', '', 'Medians across successful fresh-process trials; ranges are observed throughput variation. Latency includes broker acknowledgement. Fanout deliveries/s counts each subscriber copy. Allocation/CPU include the client and measurement harness, and exclude broker processes. Results describe this client and broker configuration; LocalStack results do not predict AWS service performance.', '') +if ($awsEnvironments.Count -eq 1) { + $aws = ($results | Where-Object { $_.Result.Success -and $_.Result.Options.Transport -eq 'sqs' } | Select-Object -First 1).Result.Environment + $lines += @("AWS target: $($aws.Broker); mode: $($aws.AwsMode); region: $($aws.AwsRegion).", '') +} +$lines += @( +'| Case | Trials | Inputs/s (min-max) | Deliveries/s | p50 / p95 / p99 ms | Bytes/input | CPU ms/input |', +'| --- | ---: | ---: | ---: | ---: | ---: | ---: |') +foreach ($row in $rows) { $lines += "| $($row.Case) | $($row.Trials) | $($row.InputsPerSecond) ($($row.MinInputsPerSecond)-$($row.MaxInputsPerSecond)) | $($row.DeliveriesPerSecond) | $($row.P50Milliseconds) / $($row.P95Milliseconds) / $($row.P99Milliseconds) | $($row.BytesPerInput) | $($row.CpuMillisecondsPerInput) |" } +$failures = @($results | Where-Object { -not $_.Result.Success }) +$lines += @('', "Failed trials: $($failures.Count).") +foreach ($failure in $failures) { $lines += "- $($failure.File): $($failure.Result.Error)" } +$lines | Set-Content (Join-Path $Directory 'summary.md') +$rows | Format-Table Case, Trials, InputsPerSecond, P99Milliseconds, BytesPerInput diff --git a/benchmarks/MessagingJobsBenchmarks.cs b/benchmarks/MessagingJobsBenchmarks.cs new file mode 100644 index 000000000..2055ff48a --- /dev/null +++ b/benchmarks/MessagingJobsBenchmarks.cs @@ -0,0 +1,59 @@ +#nullable enable +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using BenchmarkDotNet.Attributes; +using Foundatio.Jobs; +using Foundatio.Messaging; + +namespace Foundatio.Benchmarks; + +[MemoryDiagnoser] +public class MessageHeadersBenchmarks +{ + [Params(6, 16)] + public int Count { get; set; } + + private KeyValuePair[] _values = null!; + private MessageHeaders _headers = null!; + + [GlobalSetup] + public void Setup() + { + _values = Enumerable.Range(0, Count).Select(i => new KeyValuePair($"message.header{i}", $"value-{i}")).ToArray(); + _headers = MessageHeaders.Create(_values); + } + + [Benchmark] + public MessageHeaders Construct() => MessageHeaders.Create(_values); + + [Benchmark] + public string Serialize() => MessageHeaders.SerializeToJson(_headers); +} + +[MemoryDiagnoser] +public class JobPollingBenchmarks +{ + [Params(0, 10000)] + public int History { get; set; } + + private InMemoryJobRuntimeStore _store = null!; + private JobClaimRequest _request = null!; + + [GlobalSetup] + public async Task SetupAsync() + { + _store = new InMemoryJobRuntimeStore(); + for (int i = 0; i < History; i++) + await _store.CreateIfAbsentAsync(new JobState + { + JobId = $"old-{i}", Name = "work", JobType = "work", Status = JobStatus.Completed, + CompletedUtc = DateTimeOffset.UtcNow + }); + _request = new JobClaimRequest { NodeId = "worker", JobTypes = ["work"] }; + } + + [Benchmark] + public Task EmptyClaim() => _store.ClaimNextAsync(_request); +} diff --git a/build/common.props b/build/common.props index 4618d25a7..f8e06669f 100644 --- a/build/common.props +++ b/build/common.props @@ -1,6 +1,8 @@ + + false net8.0;net10.0 Foundatio Pluggable foundation blocks for building distributed apps. diff --git a/docs/design/developer-experience-review.md b/docs/design/developer-experience-review.md new file mode 100644 index 000000000..28b3613a9 --- /dev/null +++ b/docs/design/developer-experience-review.md @@ -0,0 +1,47 @@ +# Developer experience review + +Scope: the unreleased messaging and durable job redesign, including setup, common operations, lifecycle, registration errors, provider configuration, test helpers, and the first-run documentation. + +| Journey | Friction found | Decision | +| --- | --- | --- | +| Run a normal worker | Four unrelated hosting calls in the first example | `AddFoundatioWorker(configure)` configures and hosts the required roles in one explicit call. Keep individual host APIs for split deployments. | +| Run a producer API | Registering a client must not start workers | Keep `AddFoundatio()` inert. Document the producer/worker distinction first. | +| Subscribe durably | Omitting a name silently selected temporary behavior | Require a durable name in `AddSubscriber`; expose `AddTemporarySubscriber` explicitly. | +| Register a handler | Invalid concurrency, attempts, acknowledgement, or destinations failed late | Validate receiving options at registration as well as direct subscription. | +| Register a schedule | Invalid options or an abstract job survived until startup/execution | Validate complete schedule options and concrete job types before adding registrations. | +| Configure Redis | A second explicit connection string was silently ignored | Reject conflicting settings before connecting; allow one explicit setting after a default registration. | +| Test a queue of jobs | `RunAllQueuedAsync` stopped at 100 | Drain ready work across batches with a timeout. Delayed retries remain queued. | +| Test one job | `RunToCompletionAsync(handle)` executed unrelated jobs | Execute only the requested handle. | +| Learn the library | Operational internals dominated the introductory examples | Lead with one complete worker, then sending work, publishing events, and optional typed jobs. Put split hosting and transport administration later. | + +Keep the delivery model small: send queued work, publish events, and add durable jobs only for tracked execution or schedules. Keep `IJob` and its two generic enqueue parameters because they enforce the argument contract at compile time. Keep explicit receipts, claim tokens, and schedule revisions in their advanced contracts; ordinary handlers and jobs do not need to manage them. + +Provider delivery guarantees, idempotency, and schema compatibility remain explicit. Simplifying setup does not change at-least-once execution into exactly-once side effects. + +## Intentional breaking changes + +- `AddSubscriber` now requires a nonblank durable name. Replace unnamed DI registrations with `AddTemporarySubscriber`. Dynamic `SubscribeAsync` retains its options-based lifetime selection. +- Invalid receiving/schedule options and non-concrete job types fail during registration. +- Conflicting Redis connection strings throw instead of silently using the first connection. A pre-registered multiplexer requires omitting `connectionString`. +- `RunToCompletionAsync(handle)` leaves unrelated jobs queued. `RunAllQueuedAsync()` drains more than one batch and has a 30-second safety timeout. + +`AddFoundatioWorker` is additive. The individual role hosts remain useful for separate worker, scheduler, and dispatcher processes. Its callback is the configuration boundary: put worker registrations there or register them before calling it. + +## Design choices retained + +- Queue commands and event subscriptions remain separate receiving APIs, even though they share a bus and handler interface. This makes competing consumption versus fan-out visible at registration. +- Handlers receive `IMessageContext` so metadata, cancellation, and optional settlement are available without a second handler abstraction. Ordinary handlers read `Message` and return a task. +- Provider capabilities stay explicit. Temporary subscriptions, durable storage, ordering, and dead-letter inspection cannot be made identical by a convenience API. +- Stable wire names and durable subscription names remain deliberate choices. Changing CLR names or replica counts should not silently change persisted contracts. +- Job arguments retain compile-time constraints. Removing the second generic enqueue parameter would sacrifice that guarantee or require another binding abstraction. + +The README, introductory pages, primary guides, serializer examples, runnable samples, and repository skill now use the same vocabulary and current APIs. + +## Validation + +- `Foundatio.slnx` builds with zero warnings and zero errors. +- Full solution tests with disposable Redis 8.6 and LocalStack 3.8.1: 2,065 total; 2,042 passed, 23 skipped, zero failed. +- Focused regressions cover combined and single-feature worker startup, missing dependencies, temporary subscriptions, registration validation, conflicting Redis settings, draining 201 jobs, and executing one job without consuming another. +- Documentation site builds successfully. The quickstart processes a command, an event, a typed job with progress, and CRON cleanup; intentional shutdown completes cleanly. +- Whitespace formatting and `git diff --check` pass. No removed `InMemoryQueue` or `JobBase` examples remain in the README or documentation. +- The broader `Foundatio.All.slnx` workspace remains unavailable because sibling provider checkouts are absent. Validation covers all projects in this repository's `Foundatio.slnx`. diff --git a/docs/design/messaging-jobs-implementation.md b/docs/design/messaging-jobs-implementation.md new file mode 100644 index 000000000..126bfef3f --- /dev/null +++ b/docs/design/messaging-jobs-implementation.md @@ -0,0 +1,47 @@ +# Messaging and jobs design decisions + +The unreleased PR is revised around worker queues, explicit pub/sub subscriptions, and optional durable jobs. The current public guides are [Messaging](../guide/messaging.md) and [Durable jobs](../guide/jobs.md). + +## Implemented + +- Explicit queue consumers and event subscribers. Stable durable subscription names; renewable temporary leases for memory and Redis; named subscriptions required on AWS. +- Endpoint concurrency, duplicate-handler validation, and standalone disposable receive/settle. +- Broker-confirmed settlement, original preservation when dead-letter parking fails, lease supervision, and cancellation on lost ownership. +- Stable application IDs, distinct broker IDs and receipts, serialization metadata, allowlisted polymorphism, and partial batch outcomes. +- Consistent topology policy, producer declarations without phantom subscriptions, and fresh-instance AWS resource validation/deletion. +- One worker and persisted retry state machine for ad hoc and CRON jobs, with typed argument contracts and unique claim tokens. +- Atomic occurrence admission, node/type eligibility, fair due claims, stale recovery, and ownership-guarded progress/renewal/completion. +- Serializable schedule definitions with revisions and deployment configuration versions that preserve operator edits across restarts. +- Explicit consumer, worker, scheduler, and delayed-message dispatcher hosting. Registering clients or storage starts no background work. +- Bounded monitoring pages, independent active/history/idempotency/dispatch budgets, configurable retention, payload limits, and atomic capacity rejection. +- Atomic Redis receive/reclaim/settle, orphan pending-entry recovery, safe topic retention, and non-destructive per-subscription dead-letter inspection/replay. +- Dedicated CI services for Redis and SQS/SNS LocalStack conformance; executable local and distributed samples. +- Updated public guides, migration guidance, capability matrix, and repository skill. + +- Consistent messaging/job feature builders, service-based durable subscription defaults, and combined wire-name/route registration with startup diagnostics. +- Supervised subscription renewal and recreation, observable listener health, hybrid-cache resynchronization, and immediate return of owned unsettled messages at shutdown. +- Independently replenished job slots, scoped job disposal, configurable persisted retry policies, delayed enqueue, completion waits, and separate success/failure diagnostics. +- Stable per-node schedule identity, expiry for unclaimed occurrences on retired nodes, cached CRON parsing and confirmed occurrence materialization. +- Indexed send outcomes and individual application IDs, native AWS batches and bounded Redis pipelines, per-entry malformed-envelope quarantine, and amortized safe retention. +- Measured header/routing/polling improvements, reusable hot-path benchmarks and runtime health/capacity reporting. + +## Boundaries + +Delivery and execution are at least once. Business idempotency and transactional outbox/inbox integration remain application responsibilities. Redis durability depends on deployment persistence and availability settings. LocalStack conformance does not certify live AWS behavior. + +The changes intentionally break the unreleased API and Redis state layout. Do not mix old and new runtime binaries or reuse old experimental Redis namespaces; provision an isolated namespace when testing this revision. + +The full external-provider workspace solution references Aliyun, Azure Service Bus, and Minio projects that are absent from this checkout. Its build cannot start. This is an environment limitation, separate from the successful in-repository validation below. + +## Validation of the feedback changes + +- Full in-repository solution rebuilt successfully. The existing sample AppHost emits ASPIRE010 because AspireUseCliBundle is false; there are no compilation errors. +- Core suite: 2,029 tests, 2,017 passed and 12 skipped; zero failures. +- Redis suite: 60 tests, 56 passed and four unsupported-capability skips; zero failures. +- AWS suite: 29 tests, 21 passed and eight unsupported-capability skips; zero failures. +- Redis 8.6 and SQS/SNS through LocalStack 3.8.1 ran in isolated local containers. This does not certify live AWS behavior. +- Documentation site build and git whitespace checks passed. +- Quickstart `--verify` passed: producer-only registration, command processing, durable event subscription, delayed typed job, persisted cancellation, automatic CRON execution and graceful shutdown. +- [Measured costs and repeatable benchmarks](https://github.com/FoundatioFx/Foundatio/blob/feat/messaging-jobs/benchmarks/MESSAGING_JOBS_BENCHMARK_RESULTS.md) cover header allocation, idle polling independent of retained history, batching and overlapping handlers. Timing results are local development evidence, not deployment capacity limits. + +The six reproduced execution/ownership regressions are retained as tests. Additional shared cases cover retained-history pressure without losing idempotency, retry policy persistence, nonretryable failure, per-node expiry, payload/dispatch budgets, and dispatch lease timing. Recovery and configuration tests cover transient/lost subscriptions, cache gaps, cancellation, indexed batch results and startup wire-name collisions. diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 8ec0612b7..5585f1d5d 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -49,91 +49,20 @@ var scopedCache = new ScopedCacheClient( ); ``` -## Queue Configuration +## Messaging and worker queue configuration -### InMemoryQueue +Configure the bus once, then register explicit consumers or subscribers. Queue and topic capabilities differ by provider; unsupported delays require a scheduled-dispatch store and an explicitly hosted dispatcher. ```csharp -var queue = new InMemoryQueue(options => -{ - // Queue name/identifier - options.Name = "work-items"; - - // Work item timeout - options.WorkItemTimeout = TimeSpan.FromMinutes(5); - - // Retry settings - options.Retries = 3; - options.RetryDelay = TimeSpan.FromSeconds(30); - - // Logger - options.LoggerFactory = loggerFactory; - - // Serializer - options.Serializer = serializer; -}); -``` - -### RedisQueue - -```csharp -var queue = new RedisQueue(options => -{ - // Redis connection - options.ConnectionMultiplexer = redis; - - // Queue name - options.Name = "work-items"; - - // Work item timeout - options.WorkItemTimeout = TimeSpan.FromMinutes(5); - - // Dead letter settings - options.DeadLetterTimeToLive = TimeSpan.FromDays(1); - options.DeadLetterMaxItems = 100; - - // Retry settings - options.Retries = 3; - options.RetryDelay = TimeSpan.FromSeconds(30); - - // Logger - options.LoggerFactory = loggerFactory; -}); -``` - -## Messaging Configuration - -### InMemoryMessageBus - -```csharp -var messageBus = new InMemoryMessageBus(options => -{ - // Logger - options.LoggerFactory = loggerFactory; - - // Serializer - options.Serializer = serializer; -}); +builder.Services.AddFoundatioWorker(foundatio => foundatio + .Messaging.ConfigureRetry(policy => policy with { MaxAttempts = 5 }) + .UseInMemory() + .AddConsumer(options => options.MaxConcurrency = 4)); ``` -### RedisMessageBus - -```csharp -var messageBus = new RedisMessageBus(options => -{ - // Redis subscriber - options.Subscriber = redis.GetSubscriber(); - - // Topic prefix - options.Topic = "myapp"; - - // Logger - options.LoggerFactory = loggerFactory; +Use `MessageBusOptions` when constructing a bus manually. Set `Topology` to `Ensure`, `Validate`, or `None`; set `Serializer` and matching `ContentType` when overriding serialization. Consumer concurrency belongs to an endpoint. Named subscriptions are durable; unnamed temporary subscriptions require provider support. - // Serializer - options.Serializer = serializer; -}); -``` +See [Messaging](messaging.md) for full configuration and provider limits. The former `InMemoryQueue`, `RedisQueue`, and publish-only message bus options do not configure the new transport runtime. ## Lock Configuration @@ -298,38 +227,18 @@ var circuitBreaker = new CircuitBreakerBuilder() .Build(); ``` -## Job Configuration +## Job configuration -### JobOptions +Register the store and eligible job types in a worker: ```csharp -var options = new JobOptions -{ - // Job name for logging - Name = "CleanupJob", - - // Interval between runs - Interval = TimeSpan.FromHours(1), - - // Maximum iterations (-1 for unlimited) - IterationLimit = -1, - - // Initial run delay - InitialDelay = TimeSpan.FromMinutes(5) -}; - -await job.RunContinuousAsync(options, stoppingToken); +builder.Services.AddFoundatioWorker(foundatio => foundatio + .Jobs.UseInMemory() + .AddJobType("cleanup.v1") + .AddCronJob("0 2 * * *"), jobConcurrency: 4); ``` -### JobRunner - -```csharp -var runner = new JobRunner( - job: myJob, - instanceCount: 4, // Number of parallel instances - interval: TimeSpan.FromSeconds(5) -); -``` +Set per-request `MaxAttempts` in `JobRequestOptions`; schedule definitions snapshot their own retry budget. Persisted schedule edits use revisions, and changed declarations require a higher `ConfigurationVersion`. See [Durable jobs](jobs.md) for retention, capacity, and deployment behavior. ## Serialization Configuration @@ -345,7 +254,7 @@ var serializer = new SystemTextJsonSerializer(new JsonSerializerOptions // Apply to services var cache = new InMemoryCacheClient(o => o.Serializer = serializer); -var queue = new InMemoryQueue(o => o.Serializer = serializer); +var bus = new MessageBus(new InMemoryMessageTransport(), new MessageBusOptions { Serializer = serializer }); var storage = new InMemoryFileStorage(o => o.Serializer = serializer); ``` @@ -364,7 +273,7 @@ var loggerFactory = LoggerFactory.Create(builder => // Apply to services var cache = new InMemoryCacheClient(o => o.LoggerFactory = loggerFactory); -var queue = new InMemoryQueue(o => o.LoggerFactory = loggerFactory); +var bus = new MessageBus(new InMemoryMessageTransport(), new MessageBusOptions { LoggerFactory = loggerFactory }); ``` ## Environment Variables diff --git a/docs/guide/dependency-injection.md b/docs/guide/dependency-injection.md index 16bdf4a6f..9373ff4be 100644 --- a/docs/guide/dependency-injection.md +++ b/docs/guide/dependency-injection.md @@ -1,484 +1,78 @@ -# Dependency Injection +# Dependency injection -Foundatio is designed to work seamlessly with Microsoft.Extensions.DependencyInjection. All abstractions are interface-based and can be easily registered and resolved. - -## Basic Registration - -### Manual Registration - -```csharp -using Foundatio.Caching; -using Foundatio.Messaging; -using Foundatio.Lock; -using Foundatio.Storage; -using Foundatio.Queues; - -var builder = WebApplication.CreateBuilder(args); - -// Core services -builder.Services.AddSingleton(); -builder.Services.AddSingleton(); -builder.Services.AddSingleton(); - -// Lock provider (depends on cache and message bus) -builder.Services.AddSingleton(sp => - new CacheLockProvider( - sp.GetRequiredService(), - sp.GetRequiredService() - ) -); - -// Queues -builder.Services.AddSingleton>(sp => - new InMemoryQueue() -); -``` - -### Using Extension Methods +Use `AddFoundatioWorker(configure)` for an application that processes messages or jobs. Put its Foundatio registrations in the callback; it starts the roles those registrations require. For producer-only APIs and manually driven tests, use `AddFoundatio()` instead. ```csharp using Foundatio; -builder.Services.AddFoundatio(); // Adds default in-memory implementations -``` - -## Service Lifetimes - -### Recommended Lifetimes - -| Service | Lifetime | Reason | -|---------|----------|--------| -| `ICacheClient` | Singleton | Maintains internal state/connection | -| `IMessageBus` | Singleton | Maintains subscriptions | -| `ILockProvider` | Singleton | Stateless, thread-safe | -| `IFileStorage` | Singleton | Stateless, thread-safe | -| `IQueue` | Singleton | Maintains queue state | -| Jobs | Scoped | Per-execution isolation | - -### Example Registration - -```csharp -// Singletons for infrastructure -builder.Services.AddSingleton(sp => - new InMemoryCacheClient(o => o.MaxItems = 1000)); - -builder.Services.AddSingleton(); - -// Scoped for per-request isolation -builder.Services.AddScoped(sp => - new ScopedLockProvider( - sp.GetRequiredService(), - $"tenant:{GetCurrentTenantId(sp)}" - ) -); -``` - -## Environment-Based Configuration - -### Development vs Production - -```csharp -if (builder.Environment.IsDevelopment()) -{ - // In-memory for development - builder.Services.AddSingleton(); - builder.Services.AddSingleton(); - builder.Services.AddSingleton(); -} -else -{ - // Container-owned Redis connection for production - builder.Services.AddSingleton(sp => - ConnectionMultiplexer.Connect( - builder.Configuration.GetConnectionString("Redis") - )); - - builder.Services.AddSingleton(sp => - new RedisCacheClient(o => o.ConnectionMultiplexer = - sp.GetRequiredService())); - - builder.Services.AddSingleton(sp => - new RedisMessageBus(o => o.Subscriber = - sp.GetRequiredService().GetSubscriber())); - - builder.Services.AddSingleton(sp => - new AzureFileStorage(o => { - o.ConnectionString = builder.Configuration["Azure:StorageConnectionString"]; - o.ContainerName = "files"; - })); -} -``` - -### Using Options Pattern - -```csharp -// appsettings.json -{ - "Foundatio": { - "Cache": { - "Type": "Redis", - "MaxItems": 1000 - }, - "Storage": { - "Type": "Azure", - "ContainerName": "files" - } - } -} - -// Registration -builder.Services.Configure( - builder.Configuration.GetSection("Foundatio")); - -builder.Services.AddSingleton(sp => -{ - var options = sp.GetRequiredService>().Value; - return options.Cache.Type switch - { - "Redis" => new RedisCacheClient(...), - "InMemory" => new InMemoryCacheClient(o => o.MaxItems = options.Cache.MaxItems), - _ => throw new InvalidOperationException() - }; -}); -``` - -## Named/Keyed Services - -### Multiple Implementations - -```csharp -// Multiple caches -builder.Services.AddKeyedSingleton("session", - sp => new InMemoryCacheClient(o => o.MaxItems = 10000)); - -builder.Services.AddKeyedSingleton("data", - sp => new RedisCacheClient(o => o.ConnectionMultiplexer = redis)); - -// Multiple queues -builder.Services.AddKeyedSingleton>("high-priority", - sp => new InMemoryQueue()); - -builder.Services.AddKeyedSingleton>("low-priority", - sp => new InMemoryQueue()); -``` - -### Injecting Keyed Services - -```csharp -public class OrderService -{ - private readonly ICacheClient _sessionCache; - private readonly ICacheClient _dataCache; - - public OrderService( - [FromKeyedServices("session")] ICacheClient sessionCache, - [FromKeyedServices("data")] ICacheClient dataCache) - { - _sessionCache = sessionCache; - _dataCache = dataCache; - } -} -``` - -## Factory Pattern - -### Dynamic Resolution - -```csharp -public interface ICacheClientFactory -{ - ICacheClient GetCache(string name); -} - -public class CacheClientFactory : ICacheClientFactory -{ - private readonly IServiceProvider _services; - private readonly ConcurrentDictionary _caches = new(); - - public CacheClientFactory(IServiceProvider services) - { - _services = services; - } - - public ICacheClient GetCache(string name) - { - return _caches.GetOrAdd(name, n => - { - var baseCache = _services.GetRequiredService(); - return new ScopedCacheClient(baseCache, n); - }); - } -} - -// Registration -builder.Services.AddSingleton(); -builder.Services.AddSingleton(); -``` - -## Multi-Tenant Support - -### Tenant-Scoped Services - -```csharp -public interface ITenantAccessor -{ - string TenantId { get; } -} - -// Scoped cache per tenant -builder.Services.AddScoped(sp => -{ - var baseCache = sp.GetRequiredService(); - var tenant = sp.GetRequiredService(); - return new ScopedCacheClient(baseCache, $"tenant:{tenant.TenantId}"); -}); - -// Scoped storage per tenant -builder.Services.AddScoped(sp => -{ - var baseStorage = sp.GetRequiredService(); - var tenant = sp.GetRequiredService(); - return new ScopedFileStorage(baseStorage, tenant.TenantId); -}); - -// Scoped locks per tenant -builder.Services.AddScoped(sp => -{ - var baseLock = sp.GetRequiredService(); - var tenant = sp.GetRequiredService(); - return new ScopedLockProvider(baseLock, tenant.TenantId); -}); -``` - -## Health Checks - -### Register Health Checks - -```csharp -builder.Services.AddHealthChecks() - .AddCheck("cache") - .AddCheck("storage") - .AddCheck("queue"); - -public class CacheHealthCheck : IHealthCheck -{ - private readonly ICacheClient _cache; - - public CacheHealthCheck(ICacheClient cache) => _cache = cache; - - public async Task CheckHealthAsync( - HealthCheckContext context, - CancellationToken cancellationToken = default) - { - try - { - await _cache.SetAsync("health-check", DateTime.UtcNow); - var result = await _cache.GetAsync("health-check"); - - return result.HasValue - ? HealthCheckResult.Healthy() - : HealthCheckResult.Unhealthy("Cache read failed"); - } - catch (Exception ex) - { - return HealthCheckResult.Unhealthy(ex.Message); - } - } -} -``` - -## Testing - -### Test-Friendly Registration - -```csharp -// In test setup -public class TestStartup -{ - public void ConfigureServices(IServiceCollection services) - { - // Always use in-memory for tests - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(sp => - new CacheLockProvider( - sp.GetRequiredService(), - sp.GetRequiredService() - ) - ); - } -} +builder.Services.AddFoundatioWorker(foundatio => foundatio + .Caching.UseInMemory() + .Storage.UseInMemory() + .Locking.UseCache() + .UseServiceName("billing") + .ConfigureMessaging(messaging => messaging.UseInMemory() + .AddConsumer() + .AddSubscriber()) + .ConfigureJobs(jobs => jobs.UseInMemory() + .AddJobType("generate-report.v1"))); ``` -### Isolated Tests - -```csharp -public class OrderServiceTests -{ - private readonly ServiceProvider _services; - - public OrderServiceTests() - { - var services = new ServiceCollection(); - - // Fresh instances for each test class - services.AddSingleton(); - services.AddSingleton(); - - _services = services.BuildServiceProvider(); - } - - [Fact] - public async Task CreateOrder_CachesOrder() - { - var cache = _services.GetRequiredService(); - var service = new OrderService(cache); - - var order = await service.CreateOrderAsync(new CreateOrderRequest()); - - var cached = await cache.GetAsync($"order:{order.Id}"); - Assert.True(cached.HasValue); - } -} -``` +## Choose host roles explicitly -## Best Practices +`AddFoundatio()` starts no hosted services. For split deployments, register clients, stores, handlers, and job types with it, then choose only the roles that process runs. These individual methods live in `Foundatio.Extensions.Hosting.Messaging` and `Foundatio.Extensions.Hosting.Jobs`. -### 1. Proper Resource Disposal +| Registration | Responsibility | +| --- | --- | +| `AddMessageConsumers()` | Start registered consumers/subscribers and apply startup topology policy | +| `AddMessagingTopology()` | Optional producer-only topology startup checks | +| `AddJobWorker(concurrency)` | Execute registered job types | +| `AddJobScheduler()` | Reconcile declarations and materialize due CRON occurrences | +| `AddScheduledMessageDispatcher()` | Send messages parked in an `IScheduledDispatchStore` | -Foundatio services implement `IDisposable` and/or `IAsyncDisposable`. The DI container handles disposal for registered services, but you must handle disposal correctly for manually created instances. +`AddFoundatioWorker` selects these roles for a combined worker: consumers when a transport is configured, worker and scheduler when job types are registered, and delayed dispatch when both a transport and dispatch store are configured. Set its `jobConcurrency` argument to control simultaneous job executions; set message concurrency on each receiving endpoint. -```csharp -// ✅ Good: DI container handles disposal -builder.Services.AddSingleton(); -// Container disposes when application shuts down +Messaging and jobs methods consistently return their feature builder. Prefer `ConfigureMessaging(m => ...)` and `ConfigureJobs(j => ...)` when configuring several features; `.Builder` explicitly returns to the root. -// ✅ Good: Using statement for short-lived instances -await using var cache = new InMemoryCacheClient(); -await cache.SetAsync("key", "value"); -// Automatically disposed +Scheduler, worker slots, and dispatcher loops run independently. A long-running job does not block delayed-message delivery or schedule materialization. Host registrations are idempotent. -// ✅ Good: Manual disposal when needed -var queue = new InMemoryQueue(); -try -{ - await queue.EnqueueAsync(new WorkItem()); -} -finally -{ - queue.Dispose(); // Or await using for IAsyncDisposable -} +## Service lifetimes and ownership -// ❌ Bad: Not disposing manually created instances -var cache = new InMemoryCacheClient(); -// ... use cache -// Never disposed - resources leak! -``` +Keep infrastructure clients, transport connections, stores, and buses as singletons. Inject interfaces into business services. Handler and job dependencies may be scoped, including database contexts; a fresh scope is created for each invocation and disposed after execution. -### 2. Async Disposal with `await using` +The container owns transports registered through the builder. For manual construction, `MessageBus` owns its supplied transport by default. Set `MessageBusOptions.OwnsTransport = false` only when another owner manages that transport. Dispose directly created buses and use `await using` for subscriptions, received deliveries, and locks. -For services implementing `IAsyncDisposable`, prefer `await using`: +Avoid capturing a scoped dependency inside a singleton factory or long-lived delegate. Class handlers with constructor injection are the usual choice: ```csharp -// Locks implement IAsyncDisposable -await using var lck = await locker.AcquireAsync("resource"); -if (lck is null) - throw new InvalidOperationException("Failed to acquire lock on 'resource'"); - -await DoWork(); -// Lock automatically released - -// Queue entries should be completed/abandoned -var entry = await queue.DequeueAsync(); -if (entry is null) - return; - -try +public sealed class ProcessOrderHandler(OrderService orders) : IMessageHandler { - await ProcessAsync(entry.Value); - await entry.CompleteAsync(); -} -catch -{ - await entry.AbandonAsync(); - throw; + public Task HandleAsync(IMessageContext context, CancellationToken token) + => orders.ProcessAsync(context.Message, token); } ``` -### 3. Use Interfaces for Dependencies - -```csharp -// ✅ Good: Interface dependency -public class OrderService -{ - private readonly ICacheClient _cache; - - public OrderService(ICacheClient cache) - { - _cache = cache; - } -} - -// ❌ Bad: Concrete dependency -public class OrderService -{ - private readonly RedisCacheClient _cache; // Harder to test -} -``` +## Redis connection lifetime -### 4. Avoid Service Locator Pattern +`Messaging.UseRedis()` and `Jobs.UseRedis()` register one shared, container-owned connection by default. The host disposes it after background services stop. To supply a custom connection, register `IConnectionMultiplexer` with a singleton factory and omit `connectionString` on the provider calls: ```csharp -// ✅ Good: Constructor injection -public class MyService -{ - private readonly ICacheClient _cache; - - public MyService(ICacheClient cache) - { - _cache = cache; - } -} - -// ❌ Bad: Service locator -public class MyService -{ - private readonly IServiceProvider _services; - - public void DoWork() - { - var cache = _services.GetService(); - } -} +builder.Services.AddSingleton(_ => + ConnectionMultiplexer.Connect(builder.Configuration.GetConnectionString("Redis")!)); +builder.Services.AddFoundatioWorker(foundatio => foundatio + .ConfigureMessaging(messaging => messaging.UseRedis() + .AddConsumer()) + .ConfigureJobs(jobs => jobs.UseRedis() + .AddJobType("generate-report.v1"))); ``` -### 5. Register as Singletons When Appropriate +The explicit connection example uses `StackExchange.Redis` and `Microsoft.Extensions.DependencyInjection`. A connection passed as an already-created singleton instance remains caller-owned: dispose the host first, then dispose that connection. Never dispose a shared connection while handlers, workers, or lock-release operations are still using it. -```csharp -// Stateless services that maintain connections -builder.Services.AddSingleton(...); -builder.Services.AddSingleton(...); - -// Not scoped unless you need tenant isolation -``` +## Providers and testing -### 6. Validate Configuration at Startup +Swap `.Messaging.UseInMemory()` for a supported production transport, or `.Jobs.UseInMemory()` for a durable store. Check the [provider matrix](messaging.md#provider-guarantees): ordering, temporary subscriptions, native delays, and dead-letter administration are not identical across brokers. -```csharp -builder.Services.AddSingleton(sp => -{ - var connectionString = builder.Configuration["Redis:ConnectionString"]; - if (string.IsNullOrEmpty(connectionString)) - throw new InvalidOperationException("Redis connection string not configured"); - - var redis = ConnectionMultiplexer.Connect(connectionString); - return new RedisCacheClient(o => o.ConnectionMultiplexer = redis); -}); -``` +Use `.Messaging.UseTestHarness()` and `.Jobs.UseTestHarness()` from `Foundatio.Testing` for tests that exercise the real runtime. Messaging tests explicitly start consumers. Job tests drive the harness worker/scheduler themselves; no background worker races their assertions. Dispose each test's service provider to isolate resources. -## Next Steps +For keyed caches or storage services, standard `AddKeyedSingleton` and `[FromKeyedServices]` remain available. A single message bus routes by queue/topic; use explicit destinations and subscriber names instead of registering a separate typed queue service for every message type. -- [Configuration](./configuration) - Configuration options for Foundatio services -- [Caching](./caching) - Deep dive into caching -- [Getting Started](./getting-started) - Initial setup guide +See [Getting started](getting-started.md), [Messaging](messaging.md), and [Durable jobs](jobs.md) for current setup and migration examples. diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index e2a235504..862cc650e 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -1,266 +1,68 @@ -# Getting Started +# Getting started -This guide will walk you through installing Foundatio and using your first abstractions. +Foundatio supplies swappable caching, file storage, locking, messaging, and background job building blocks. Start with in-memory implementations, then choose a production provider for the contracts your application needs. -## Installation +The messaging and durable job APIs shown here are the current unreleased redesign. Existing published provider packages may still use the earlier queue/pub-sub APIs; see the [migration guide](messaging.md#migration). -Foundatio is available on [NuGet](https://www.nuget.org/packages?q=Foundatio). Install the core package: +## Run the example -```bash -dotnet add package Foundatio -``` - -For specific implementations, install the corresponding packages: - -```bash -# Redis implementations -dotnet add package Foundatio.Redis - -# Azure Storage (Queues, Blobs) -dotnet add package Foundatio.AzureStorage - -# Azure Service Bus (Queues, Messaging) -dotnet add package Foundatio.AzureServiceBus - -# AWS (SQS, S3) -dotnet add package Foundatio.AWS - -# RabbitMQ (Messaging) -dotnet add package Foundatio.RabbitMQ +From a checkout of this revision: -# Kafka (Messaging) -dotnet add package Foundatio.Kafka - -# Aliyun OSS (Storage) -dotnet add package Foundatio.Aliyun - -# MinIO (S3-compatible Storage) -dotnet add package Foundatio.Minio - -# SSH/SFTP (Storage) -dotnet add package Foundatio.Storage.SshNet +```powershell +dotnet run --project samples/Foundatio.QuickstartSample ``` -## Basic Setup +The sample starts a host, sends a command, publishes an event, runs a typed job with progress, and schedules a CRON cleanup. It requires no external services. Add `-- --verify` to check message handling, a delayed job, cancellation and an automatic CRON occurrence, then exit. -### 1. Register Services +## A message worker -Configure Foundatio services in your application's dependency injection container: +Reference `Foundatio` and `Foundatio.Extensions.Hosting` from this revision. `AddFoundatioWorker` is in the `Foundatio` namespace. The full message and handler definitions are in the quickstart sample above. ```csharp -using Foundatio.Caching; +using Foundatio; using Foundatio.Messaging; -using Foundatio.Lock; -using Foundatio.Storage; -using Foundatio.Queues; - -var builder = WebApplication.CreateBuilder(args); - -// Register core services -builder.Services.AddSingleton(); -builder.Services.AddSingleton(); -builder.Services.AddSingleton(); - -// Register lock provider (depends on cache and message bus) -builder.Services.AddSingleton(sp => - new CacheLockProvider( - sp.GetRequiredService(), - sp.GetRequiredService() - ) -); - -// Register queues -builder.Services.AddSingleton>(sp => - new InMemoryQueue() -); - -var app = builder.Build(); -``` - -### 2. Use the Services - -Inject and use the services in your application: - -```csharp -public class OrderService -{ - private readonly ICacheClient _cache; - private readonly IQueue _queue; - private readonly ILockProvider _locker; - private readonly IMessageBus _messageBus; - - public OrderService( - ICacheClient cache, - IQueue queue, - ILockProvider locker, - IMessageBus messageBus) - { - _cache = cache; - _queue = queue; - _locker = locker; - _messageBus = messageBus; - } - - public async Task CreateOrderAsync(CreateOrderRequest request) - { - // Use distributed lock to prevent duplicate orders - await using var lck = await _locker.AcquireAsync($"order:{request.CustomerId}"); - if (lck == null) - throw new InvalidOperationException("Could not acquire lock"); - - // Create order - var order = new Order { Id = Guid.NewGuid(), CustomerId = request.CustomerId }; - - // Cache the order - await _cache.SetAsync($"order:{order.Id}", order, TimeSpan.FromHours(1)); - - // Queue for background processing - await _queue.EnqueueAsync(new OrderWorkItem { OrderId = order.Id }); - - // Publish event for other services - await _messageBus.PublishAsync(new OrderCreatedEvent { OrderId = order.Id }); - - return order; - } -} -``` - -## Switching to Production Implementations - -When moving to production, swap in-memory implementations for distributed ones: - -```csharp -using Foundatio.Redis.Cache; -using Foundatio.Redis.Messaging; -using Foundatio.Redis.Queues; -using StackExchange.Redis; - -var builder = WebApplication.CreateBuilder(args); - -// Configure a container-owned Redis connection -builder.Services.AddSingleton(sp => - ConnectionMultiplexer.Connect("localhost:6379")); +using Microsoft.Extensions.Hosting; -// Use Redis implementations -builder.Services.AddSingleton(sp => - new RedisCacheClient(o => o.ConnectionMultiplexer = - sp.GetRequiredService()) -); +var builder = Host.CreateApplicationBuilder(args); +builder.Services.AddFoundatioWorker(foundatio => foundatio + .UseServiceName("billing") + .ConfigureMessaging(messaging => messaging.UseInMemory() + .AddConsumer() + .AddSubscriber())); -builder.Services.AddSingleton(sp => - new RedisMessageBus(o => o.Subscriber = - sp.GetRequiredService().GetSubscriber()) -); - -builder.Services.AddSingleton>(sp => - new RedisQueue(o => o.ConnectionMultiplexer = - sp.GetRequiredService()) -); +await builder.Build().RunAsync(); ``` -Your application code remains unchanged - only the DI registration changes! - -## Working with Extension Methods - -Foundatio provides convenient extension methods through `FoundatioServicesExtensions`: - -```csharp -using Foundatio; - -var builder = WebApplication.CreateBuilder(args); +Handlers implement `IMessageHandler`. Send work with `IMessageBus.SendAsync`; publish events with `PublishAsync`. Queue consumers compete. A named event subscription receives one copy for its group, and replicas in that group compete. Handlers must tolerate duplicate delivery. -// Add Foundatio with default in-memory implementations -builder.Services.AddFoundatio(); +For a producer-only API, use `AddFoundatio().Messaging.UseInMemory()` instead. `AddFoundatio()` registers clients; `AddFoundatioWorker(...)` also starts background processing when the host starts. See [Messaging](messaging.md) for complete handler examples and delivery guarantees. -// Or configure with options -builder.Services.AddFoundatio(options => -{ - options.UseInMemoryCache(); - options.UseInMemoryMessageBus(); - options.UseInMemoryQueues(); - options.UseInMemoryStorage(); -}); -``` +## Choose the operation -## Sample Application +| You need to… | Use | Register on the worker | +| --- | --- | --- | +| Hand work to one available consumer | `bus.SendAsync(message)` | `AddConsumer()` | +| Notify each interested service | `bus.PublishAsync(message)` | `AddSubscriber()` with a stable service name | +| Track execution, progress, cancellation, or schedules | `jobs.EnqueueAsync(args)` | `AddJobType("job-name.v1")` | -Here's a complete example showing all major abstractions working together: +## Add the infrastructure you need ```csharp -using Foundatio.Caching; -using Foundatio.Lock; -using Foundatio.Messaging; -using Foundatio.Queues; -using Foundatio.Storage; - -// Setup services -var cache = new InMemoryCacheClient(); -var messageBus = new InMemoryMessageBus(); -var storage = new InMemoryFileStorage(); -var locker = new CacheLockProvider(cache, messageBus); -var queue = new InMemoryQueue(); - -// Subscribe to messages -await messageBus.SubscribeAsync(msg => -{ - Console.WriteLine($"Work completed: {msg.ItemId}"); -}); - -// Store a file -await storage.SaveFileAsync("config.json", """{"setting": "value"}"""); - -// Queue work -await queue.EnqueueAsync(new WorkItem { Id = "item-1" }); - -// Process queue with locking -while (true) -{ - var entry = await queue.DequeueAsync(TimeSpan.FromSeconds(5)); - if (entry == null) break; - - // Acquire lock for this item - await using var lck = await locker.AcquireAsync($"work:{entry.Value.Id}"); - if (lck != null) - { - // Cache progress - await cache.SetAsync($"progress:{entry.Value.Id}", "processing"); - - // Do work... - - // Complete entry - await entry.CompleteAsync(); - - // Publish completion event - await messageBus.PublishAsync(new WorkCompleted { ItemId = entry.Value.Id }); - } - else - { - // Couldn't get lock, abandon for retry - await entry.AbandonAsync(); - } -} - -public record WorkItem { public string Id { get; init; } } -public record WorkCompleted { public string ItemId { get; init; } } +builder.Services.AddFoundatio() + .Caching.UseInMemory() + .Storage.UseFolder("data") + .Locking.UseCache(); ``` -## Next Steps - -Now that you have the basics working, explore more advanced features: - -- [Caching](./caching) - Deep dive into caching patterns -- [Queues](./queues) - Queue processing and behaviors -- [Locks](./locks) - Distributed locking strategies -- [Messaging](./messaging) - Pub/sub patterns -- [Storage](./storage) - File storage operations -- [Jobs](./jobs) - Background job processing -- [Resilience](./resilience) - Retry policies and circuit breakers +Use `ICacheClient` for cache operations, `IFileStorage` for files, and `ILockProvider` for distributed coordination. Dispose streams and acquired locks. In-memory data is process-local and does not survive restarts. -## LLM-Friendly Documentation +Add [durable jobs](jobs.md) only when you need handles, progress, cancellation, stored retries, or schedules. `AddFoundatioWorker(...)` hosts the required worker, scheduler, and delayed-message dispatcher roles. [Individual hosting methods](dependency-injection.md#choose-host-roles-explicitly) support running those roles in separate processes. -For AI assistants and Large Language Models, we provide optimized documentation formats: +For Redis, the builder shares and owns one connection by default. If you supply your own, keep it alive until the host has stopped; see [connection lifetime](dependency-injection.md#redis-connection-lifetime). -- [📜 LLMs Index](/llms.txt) - Quick reference with links to all sections -- [📖 Complete Documentation](/llms-full.txt) - All docs in one LLM-friendly file +## Next steps -These files follow the [llmstxt.org](https://llmstxt.org/) standard and contain the same information as this documentation in a format optimized for AI consumption. +- [Worker queues](queues.md) for competing consumers and migration from `IQueue`. +- [Messaging](messaging.md) for pub/sub identity, serialization, topology, retries, and provider behavior. +- [Durable jobs](jobs.md) for typed work, CRON definitions, monitoring, and retention. +- [Caching](caching.md), [storage](storage.md), and [locks](locks.md) for other infrastructure contracts. diff --git a/docs/guide/implementations/aws.md b/docs/guide/implementations/aws.md index f18224551..8f2597d1a 100644 --- a/docs/guide/implementations/aws.md +++ b/docs/guide/implementations/aws.md @@ -1,5 +1,10 @@ # Foundatio.AWS +::: info Provider API versions +The queue and publish-only bus examples below describe the earlier external provider packages. This unreleased revision uses `IMessageTransport` with explicit queue consumers and event subscribers; see the [current messaging matrix](../messaging.md#provider-guarantees) and [durable job store guide](../jobs.md). Earlier provider implementations do not implement the new SPI automatically. Cache and file-storage examples retain their existing contracts. +::: + + Foundatio provides AWS implementations for file storage, queuing, and messaging using Amazon S3, Amazon SQS, and Amazon SNS. [View source on GitHub →](https://github.com/FoundatioFx/Foundatio.AWS) ## Overview diff --git a/docs/guide/implementations/azure.md b/docs/guide/implementations/azure.md index 889760fcc..f302063f8 100644 --- a/docs/guide/implementations/azure.md +++ b/docs/guide/implementations/azure.md @@ -1,5 +1,10 @@ # Foundatio.AzureStorage / Foundatio.AzureServiceBus +::: info Provider API versions +The queue and publish-only bus examples below describe the earlier external provider packages. This unreleased revision uses `IMessageTransport` with explicit queue consumers and event subscribers; see the [current messaging matrix](../messaging.md#provider-guarantees) and [durable job store guide](../jobs.md). Earlier provider implementations do not implement the new SPI automatically. Cache and file-storage examples retain their existing contracts. +::: + + Foundatio provides Azure implementations for storage, queuing, and messaging using Azure Blob Storage, Azure Storage Queues, and Azure Service Bus. [View source on GitHub →](https://github.com/FoundatioFx/Foundatio.AzureStorage) | [AzureServiceBus](https://github.com/FoundatioFx/Foundatio.AzureServiceBus) ## Overview diff --git a/docs/guide/implementations/in-memory.md b/docs/guide/implementations/in-memory.md index 5fc790026..5a108c216 100644 --- a/docs/guide/implementations/in-memory.md +++ b/docs/guide/implementations/in-memory.md @@ -7,8 +7,8 @@ Foundatio provides in-memory implementations for all core abstractions. These ar | Implementation | Interface | Package | |----------------|-----------|---------| | `InMemoryCacheClient` | `ICacheClient` | Foundatio | -| `InMemoryQueue` | `IQueue` | Foundatio | -| `InMemoryMessageBus` | `IMessageBus` | Foundatio | +| `InMemoryMessageTransport` / `MessageBus` | `IMessageTransport` / `IMessageBus` | Foundatio | +| `InMemoryJobRuntimeStore` | `IJobRuntimeStore` | Foundatio | | `InMemoryFileStorage` | `IFileStorage` | Foundatio | | `CacheLockProvider` | `ILockProvider` | Foundatio | @@ -149,145 +149,24 @@ services.AddSingleton(sp => })); ``` -## InMemoryQueue - -A thread-safe in-memory queue with retry support and dead letter handling. - -### Basic Usage - -```csharp -using Foundatio.Queues; - -var queue = new InMemoryQueue(); - -// Enqueue items -await queue.EnqueueAsync(new WorkItem { Id = 1, Data = "Hello" }); - -// Dequeue and process -var entry = await queue.DequeueAsync(); -if (entry != null) -{ - // Process the item - Console.WriteLine(entry.Value.Data); - - // Mark as complete - await entry.CompleteAsync(); -} -``` - -### Configuration Options - -```csharp -var queue = new InMemoryQueue(options => -{ - // Queue identifier - options.Name = "work-items"; - - // Work item timeout (for retry) - options.WorkItemTimeout = TimeSpan.FromMinutes(5); - - // Retry settings - options.Retries = 3; - options.RetryDelay = TimeSpan.FromSeconds(30); - - // Processing behaviors - options.Behaviors.Add(new DuplicateDetectionQueueBehavior(cacheClient, loggerFactory)); - - // Logger - options.LoggerFactory = loggerFactory; -}); -``` - -### Processing Patterns - -```csharp -// Continuous processing with handler -await queue.StartWorkingAsync(async (entry, token) => -{ - await ProcessWorkItemAsync(entry.Value); -}); - -// Process until empty -while (await queue.GetQueueStatsAsync() is { Queued: > 0 }) -{ - var entry = await queue.DequeueAsync(); - if (entry is null) - break; - - await entry.CompleteAsync(); -} -``` - -### DI Registration - -```csharp -services.AddSingleton>(sp => - new InMemoryQueue(options => - { - options.Name = "work-items"; - options.WorkItemTimeout = TimeSpan.FromMinutes(5); - options.LoggerFactory = sp.GetRequiredService(); - })); -``` - -## InMemoryMessageBus - -A simple in-memory pub/sub message bus for single-process communication. - -### Basic Usage +## Messaging and durable job contracts ```csharp -using Foundatio.Messaging; - -var messageBus = new InMemoryMessageBus(); - -// Subscribe to messages -await messageBus.SubscribeAsync(message => -{ - Console.WriteLine($"User created: {message.UserId}"); -}); - -// Publish messages -await messageBus.PublishAsync(new UserCreatedEvent { UserId = "123" }); +services.AddFoundatio() + .Messaging.UseInMemory() + .AddConsumer() + .AddSubscriber("billing") + .Builder.Jobs.UseInMemory() + .AddJobType("cleanup.v1"); +services.AddMessageConsumers(); +services.AddJobWorker(); ``` -### Configuration Options +In-memory messaging provides competing queues, named event subscriptions, temporary expiring subscriptions, lease supervision, and non-destructive dead-letter administration. Use `MessageBus` over `InMemoryMessageTransport` for manual construction. It has no native delayed sends; configure a dispatch store and `AddScheduledMessageDispatcher` for delays. -```csharp -var messageBus = new InMemoryMessageBus(options => -{ - options.LoggerFactory = loggerFactory; - options.Serializer = serializer; -}); -``` - -### Subscription Management - -```csharp -// Subscribe with options -await messageBus.SubscribeAsync( - handler: async (message, token) => - { - await ProcessOrderAsync(message); - }, - cancellationToken: stoppingToken); +The job store uses the same claims, retries, cancellation, schedule revisions, pagination, and retention contracts as the Redis provider. All state is lost on process exit. These implementations exercise application behavior without requiring a broker; they do not model a distributed provider's durability or every capability. -// Type hierarchy subscription -await messageBus.SubscribeAsync(message => -{ - // Receives all events that inherit from BaseEvent -}); -``` - -### DI Registration - -```csharp -services.AddSingleton(); -services.AddSingleton(sp => - sp.GetRequiredService()); -services.AddSingleton(sp => - sp.GetRequiredService()); -``` +See [Messaging](../messaging.md), [Durable jobs](../jobs.md), and the `Foundatio.Testing` harnesses for executable usage patterns. ## InMemoryFileStorage @@ -372,7 +251,7 @@ Use `CacheLockProvider` with `InMemoryCacheClient` for in-memory distributed loc using Foundatio.Lock; var cache = new InMemoryCacheClient(); -var messageBus = new InMemoryMessageBus(); +var messageBus = new MessageBus(new InMemoryMessageTransport()); var locker = new CacheLockProvider(cache, messageBus); // Acquire a lock @@ -405,63 +284,22 @@ services.AddSingleton(sp => sp.GetRequiredService())); ``` -## Complete In-Memory Setup - -### All Services +## Complete in-memory setup ```csharp -public static IServiceCollection AddFoundatioInMemory( - this IServiceCollection services) -{ - // Cache - services.AddSingleton(); - - // Message Bus - services.AddSingleton(); - services.AddSingleton(sp => - sp.GetRequiredService()); - services.AddSingleton(sp => - sp.GetRequiredService()); - - // Lock Provider - services.AddSingleton(sp => - new CacheLockProvider( - sp.GetRequiredService(), - sp.GetRequiredService())); - - // File Storage - services.AddSingleton(); - - return services; -} - -// With queues -public static IServiceCollection AddFoundatioQueue( - this IServiceCollection services, - string name) where T : class -{ - services.AddSingleton>(sp => - new InMemoryQueue(options => - { - options.Name = name; - options.LoggerFactory = sp.GetRequiredService(); - })); - - return services; -} +builder.Services.AddFoundatio() + .Caching.UseInMemory() + .Storage.UseInMemory() + .Locking.UseCache() + .Messaging.UseInMemory() + .AddConsumer() + .Builder.Jobs.UseInMemory() + .AddJobType("cleanup.v1"); +builder.Services.AddMessageConsumers(); +builder.Services.AddJobWorker(); ``` -### Usage - -```csharp -var builder = WebApplication.CreateBuilder(args); - -builder.Services.AddFoundatioInMemory(); -builder.Services.AddFoundatioQueue("work-items"); -builder.Services.AddFoundatioQueue("emails"); - -var app = builder.Build(); -``` +Clients and stores are singletons; handlers/jobs receive a scope for each invocation. Add a scheduler or scheduled-message dispatcher only when this process should run that role. ## When to Use In-Memory diff --git a/docs/guide/implementations/redis.md b/docs/guide/implementations/redis.md index 1e195a00c..9df9a6d34 100644 --- a/docs/guide/implementations/redis.md +++ b/docs/guide/implementations/redis.md @@ -1,5 +1,10 @@ # Redis Implementation +::: info Provider API versions +The queue and publish-only bus examples below describe the earlier external provider packages. This unreleased revision uses `IMessageTransport` with explicit queue consumers and event subscribers; see the [current messaging matrix](../messaging.md#provider-guarantees) and [durable job store guide](../jobs.md). Earlier provider implementations do not implement the new SPI automatically. Cache and file-storage examples retain their existing contracts. +::: + + Foundatio provides Redis implementations for caching, queues, messaging, locks, and file storage. Redis enables distributed scenarios across multiple processes and servers. ## Overview @@ -701,3 +706,7 @@ Redis/Valkey replication is asynchronous. When using `PreferReplica`, reads may ## GitHub Repository - [Foundatio.Redis](https://github.com/FoundatioFx/Foundatio.Redis) - View source code and contribute + +## Shared connection configuration + +Messaging and jobs share one `IConnectionMultiplexer`. Set `ConnectionStrings:Redis` in configuration, register a multiplexer yourself, or supply `connectionString` on one `UseRedis` call. Repeating the same explicit string is allowed; conflicting strings fail during registration. When a multiplexer is already registered, omit `connectionString` so that connection is used. diff --git a/docs/guide/jobs.md b/docs/guide/jobs.md index f7380b9ad..7e3006895 100644 --- a/docs/guide/jobs.md +++ b/docs/guide/jobs.md @@ -1,925 +1,128 @@ -# Jobs +# Durable jobs -Jobs allow you to run long-running processes without worrying about them being terminated prematurely. Foundatio provides several base classes that handle the boilerplate of continuous execution, cancellation, locking, queue processing, and hosting integration — so you focus on your business logic. +Use a message consumer for ordinary worker-queue processing. Use durable jobs when callers need a handle, progress, cancellation, persisted retries, or CRON scheduling. Ad hoc jobs and scheduled occurrences use the same execution state machine. -## The IJob Interface - -[View source](https://github.com/FoundatioFx/Foundatio/blob/main/src/Foundatio/Jobs/IJob.cs) - -Every job implements a single method: - -```csharp -public interface IJob -{ - Task RunAsync(CancellationToken cancellationToken = default); -} -``` - -You can implement `IJob` directly, but in practice you'll derive from one of the base classes below. - -## Choosing a Job Type - -| Scenario | Base Class | When to Use | -|----------|-----------|-------------| -| Scheduled or periodic work | `JobBase` | Maintenance tasks, report generation, data sync | -| Singleton / leader-elected work | `JobWithLockBase` | Only one instance should run across all servers | -| Processing queue items | `QueueJobBase` | Each unit of work arrives as a queue message | -| On-demand heterogeneous tasks | `WorkItemJob` + handlers | User-triggered operations, bulk operations with progress | - -### Architectural Tradeoffs - -**`JobBase` vs `QueueJobBase`:** A `JobBase` that polls a database on an interval is simpler to reason about but wastes cycles when there's no work. A `QueueJobBase` reacts instantly to new messages and naturally distributes load across instances, but adds a queue dependency. Use `QueueJobBase` when work arrives unpredictably and latency matters; use `JobBase` when work is periodic or the polling interval is acceptable. - -**`QueueJobBase` vs `WorkItemJob`:** `QueueJobBase` creates one strongly-typed queue per job — ideal when you have a steady stream of homogeneous work (order processing, email sending, image resizing). `WorkItemJob` uses a single shared `IQueue` to multiplex many task types through one queue and job pool. Prefer `WorkItemJob` when tasks are sporadic, one-off, or varied (user-triggered deletes, bulk exports, cache rebuilds) — it avoids creating a dedicated queue and job class for each operation. `WorkItemJob` also supports built-in progress reporting, making it natural for operations that a user is waiting on. - -**Lock timeouts and self-healing:** Locks acquired via `JobWithLockBase` or `ILockProvider.AcquireAsync` have a `timeUntilExpires` parameter (default: 20 minutes). If a server crashes while holding a lock, the lock *automatically releases* after this timeout — no manual intervention needed. Set `timeUntilExpires` to a duration comfortably longer than your expected job duration so the lock doesn't expire mid-run, but short enough that a crash doesn't block the next run for too long. For jobs where you can measure average duration, set the timeout to roughly 2-3x that average. For long or unpredictable jobs, use a shorter timeout and call `context.RenewLockAsync()` periodically to extend the lease. When acquiring a lock in `GetLockAsync`, pass `new CancellationToken(true)` to make the attempt non-blocking — `AcquireAsync` checks `cancellationToken.IsCancellationRequested` to decide whether to wait; an already-cancelled token means "try once and return `null` if the lock is held." This lets interval-based jobs gracefully skip a run rather than pile up waiting for a busy lock. The queue's `WorkItemTimeout` serves the same self-healing purpose for queue entries: entries that aren't completed or renewed within the timeout are redelivered to another consumer. - -## Standard Jobs - -### JobBase - -[View source](https://github.com/FoundatioFx/Foundatio/blob/main/src/Foundatio/Jobs/JobBase.cs) - -`JobBase` provides structured logging (`_logger`), a `TimeProvider`, and a `ResiliencePolicyProvider`. All base classes accept optional `TimeProvider` and `IResiliencePolicyProvider` constructor parameters (defaulting to `TimeProvider.System` and `DefaultResiliencePolicyProvider.Instance`). You override `RunInternalAsync` and receive a `JobContext`: +## Start a job worker ```csharp +using Foundatio; using Foundatio.Jobs; -public class CleanupJob : JobBase -{ - public CleanupJob( - TimeProvider timeProvider, - IResiliencePolicyProvider resiliencePolicyProvider, - ILoggerFactory loggerFactory) - : base(timeProvider, resiliencePolicyProvider, loggerFactory) { } - - protected override async Task RunInternalAsync(JobContext context) - { - var deletedCount = await CleanupOldRecordsAsync(context.CancellationToken); - _logger.LogInformation("Cleaned up {Count} records", deletedCount); - return JobResult.Success; - } -} -``` - -### JobContext - -`JobContext` is passed to `RunInternalAsync` and carries everything your job needs at runtime: - -| Member | Description | -|--------|-------------| -| `CancellationToken` | Signals that the job should stop gracefully | -| `Lock` | The distributed lock held by the job (`null` unless using `JobWithLockBase`) | -| `RenewLockAsync()` | Extends the lock lease — call this in long-running loops to prevent expiration. In `QueueEntryContext`, also renews the queue entry's visibility timeout so the message isn't redelivered to another consumer. | - -```csharp -protected override async Task RunInternalAsync(JobContext context) -{ - foreach (var batch in GetBatches()) - { - context.CancellationToken.ThrowIfCancellationRequested(); - await ProcessBatchAsync(batch); - await context.RenewLockAsync(); // keep the lock alive between batches - } - - return JobResult.Success; -} +builder.Services.AddFoundatioWorker(foundatio => foundatio + .ConfigureJobs(jobs => jobs.UseInMemory() + .AddJobType("resize-image.v1") + .AddCronJob("0 2 * * *"))); ``` -### JobWithLockBase - -[View source](https://github.com/FoundatioFx/Foundatio/blob/main/src/Foundatio/Jobs/JobWithLockBase.cs) +The in-memory store is for tests and local development. Use `.Jobs.UseRedis()` for persistence across processes, with Redis persistence and availability configured for your requirements. -`JobWithLockBase` automatically acquires a distributed lock before each run and releases it afterward. If the lock cannot be acquired, the run is cancelled — your code is never called. This makes it ideal for leader-election scenarios where exactly one instance should execute across a cluster. +`AddFoundatioWorker(..., jobConcurrency: 4)` runs up to four jobs concurrently. Its scheduler runs independently. For a producer-only API, use `AddFoundatio()` to register the same store and job types. For separate scheduler and worker processes, use the [individual hosting methods](dependency-injection.md#choose-host-roles-explicitly). -Override two methods: - -- **`GetLockAsync`** — return the lock to acquire, or `null` to skip the run. -- **`RunInternalAsync`** — your job logic, called only while the lock is held. +## Typed arguments and handles ```csharp -using Foundatio.Jobs; -using Foundatio.Lock; +public sealed record ResizeArgs(string File, int Width); -[Job(Description = "Singleton maintenance job", Interval = "5s")] -public class MaintenanceJob : JobWithLockBase +public sealed class ResizeImageJob(ImageService images) : IJob { - private readonly ILockProvider _lockProvider; - - public MaintenanceJob( - ICacheClient cache, - IMessageBus messageBus, - TimeProvider timeProvider, - IResiliencePolicyProvider resiliencePolicyProvider, - ILoggerFactory loggerFactory) : base(timeProvider, resiliencePolicyProvider, loggerFactory) - { - _lockProvider = new CacheLockProvider(cache, messageBus, loggerFactory); - } - - protected override Task GetLockAsync(CancellationToken cancellationToken) + public async Task RunAsync(ResizeArgs arguments, JobExecutionContext context) { - // Pass an already-cancelled token so AcquireAsync attempts the lock - // exactly once without waiting. If the lock is held by another instance, - // it returns null immediately and this run is skipped. - return _lockProvider.AcquireAsync( - nameof(MaintenanceJob), - timeUntilExpires: TimeSpan.FromMinutes(15), - cancellationToken: new CancellationToken(true)); - } - - protected override async Task RunInternalAsync(JobContext context) - { - _logger.LogInformation("Running maintenance (lock held)..."); - await DoMaintenanceAsync(context.CancellationToken); + await context.ReportProgressAsync(10, "Reading image"); + await images.ResizeAsync(arguments.File, arguments.Width, context.CancellationToken); return JobResult.Success; } } -``` - -> **Why `new CancellationToken(true)`?** `ILockProvider.AcquireAsync` uses the cancellation token to decide whether to wait for a busy lock. A token that is already cancelled tells the provider "try once — if the lock is held, return `null` immediately." This is the standard pattern for jobs that run on an interval and should simply skip the current iteration if another instance is already running. - -**`JobWithLockBase` vs manual locking in `JobBase`:** -- Use **`JobWithLockBase`** when the *entire run* must be single-instance. The lock wraps the full execution and is released automatically — even on exceptions. Set `timeUntilExpires` in `GetLockAsync` to at least 2-3x your expected run duration so the lock self-heals after a crash but doesn't expire during normal operation. -- Use **manual `ILockProvider.AcquireAsync`** inside `JobBase` when you need finer-grained control — for example, locking individual resources while allowing the job itself to run on multiple servers: - -```csharp -public class ResourceSyncJob : JobBase -{ - private readonly ILockProvider _locker; - private readonly IResourceRepository _repository; - - public ResourceSyncJob( - ILockProvider locker, - IResourceRepository repository, - TimeProvider timeProvider, - IResiliencePolicyProvider resiliencePolicyProvider, - ILoggerFactory loggerFactory) - : base(timeProvider, resiliencePolicyProvider, loggerFactory) - { - _locker = locker; - _repository = repository; - } +var handle = await jobs.EnqueueAsync(new ResizeArgs("image.png", 640)); +var state = await handle.WaitForCompletionAsync(TimeSpan.FromMinutes(2)); +Console.WriteLine(state.ResultMessage); - protected override async Task RunInternalAsync(JobContext context) - { - var pendingResources = await _repository.GetPendingSyncAsync(context.CancellationToken); - if (pendingResources.Count == 0) - return JobResult.Success; - - _logger.LogInformation("Found {Count} resources to sync", pendingResources.Count); - - foreach (var resource in pendingResources) - { - context.CancellationToken.ThrowIfCancellationRequested(); - - await using var lck = await _locker.AcquireAsync( - $"resource-sync:{resource.Id}", - cancellationToken: new CancellationToken(true)); - - if (lck is null) - { - _logger.LogDebug("Skipping resource {ResourceId}, another instance is syncing it", resource.Id); - continue; - } - - await _repository.SyncAsync(resource, context.CancellationToken); - } - - return JobResult.Success; - } -} +var delayed = await jobs.EnqueueAsync( + new ResizeArgs("later.png", 640), new JobRequestOptions { Delay = TimeSpan.FromHours(1) }); +await delayed.RequestCancellationAsync(); ``` -### IJobWithOptions +`Delay` and `RunAt` are mutually exclusive. `WaitForCompletionAsync` defaults to a five-minute timeout and polls every 250 ms. Timeout or cancellation stops the wait; cancelling execution requires `RequestCancellationAsync`. -`IJobWithOptions` extends `IJob` with a `JobOptions` property. `JobWithLockBase` implements this interface, and `JobRunner` uses it to pass runtime configuration (name, interval, iteration limit) to job instances. You rarely need to implement it directly. +The argument type is part of `IJob` and is checked before persistence. Argument-free jobs implement `IJob.RunAsync(JobExecutionContext)`. A typed job cannot be submitted without its arguments. Register stable, versioned job names on both submitters and workers; only allowlisted job types execute. Keep the serialized argument contract compatible for as long as old jobs can remain queued or retained. -```csharp -public interface IJobWithOptions : IJob -{ - JobOptions? Options { get; set; } -} -``` - -### Running Jobs +Each execution receives a dependency injection scope, its application job ID, attempt number, and cancellation token. Workers renew leases automatically and poll for cancellation. Progress updates, renewal, and completion require the current unexpired claim token. Restarting with the same node name does not confer ownership of a previous execution. -```csharp -var job = serviceProvider.GetRequiredService(); - -// Run once -await job.RunAsync(); +## Execution and retries -// Run continuously with a 5-minute pause between iterations -await job.RunContinuousAsync( - interval: TimeSpan.FromMinutes(5), - cancellationToken: stoppingToken); +Jobs progress from queued to processing to completed, failed, or cancelled. A failed attempt returns to the queue with a persisted `JobRetryPolicy`: initially 10 seconds, exponential multiplier 2, a five-minute cap, and 20% jitter. Set `JobRequestOptions.RetryPolicy` or `CronJobOptions.RetryPolicy` to change it. The policy is stored with the job so different workers apply the same curve. Return a failed `JobResult` with `Retryable = false` for a terminal business failure. `JobRequestOptions.MaxAttempts` defaults to three total attempts, including crash recovery; CRON definitions snapshot the same budget into each occurrence. Exhausted jobs end in `Failed` with their error retained. -// Run exactly 100 iterations then stop -await job.RunContinuousAsync( - iterationLimit: 100, - cancellationToken: stoppingToken); -``` +An expired processing lease can be claimed with a fresh token. Host shutdown returns unfinished work to the queue; explicit user cancellation is terminal. A worker that loses its lease cannot complete or report progress against the replacement claim. -`RunContinuousAsync` handles the loop, error delays, and cancellation for you. For queue-based jobs, the return value is the number of items processed successfully; for standard jobs, it's the iteration count. +These fences protect job state, not arbitrary external side effects. A process can crash after completing an external operation but before persisting completion. Jobs must tolerate repeated execution. A stable caller-supplied `JobRequestOptions.JobId` makes submission create-if-absent until its idempotency reservation expires; it does not make execution exactly once. -### Job Results +Workers atomically claim the oldest eligible due job from registered types and optional node affinity. Monitoring queries do not drive execution, so old or unknown job types cannot crowd runnable work out of a monitoring page. -`JobResult` communicates the outcome of each run to the framework. When running continuously, a failed result triggers an automatic delay before the next iteration to avoid tight error loops: +## CRON schedules ```csharp -protected override Task RunInternalAsync(JobContext context) -{ - try - { - // Success - return Task.FromResult(JobResult.Success); - - // Success with message - return Task.FromResult(JobResult.SuccessWithMessage("Processed 100 items")); - - // Failed with message - return Task.FromResult(JobResult.FailedWithMessage("Database connection failed")); - - // Cancelled - return Task.FromResult(JobResult.Cancelled); - } - catch (Exception ex) +builder.Services.AddFoundatioWorker(foundatio => foundatio + .ConfigureJobs(jobs => jobs.UseInMemory() + .AddJobType("resize-image.v1") + .AddCronJob("0 2 * * *", new ResizeArgs("banner.png", 640), o => { - // From exception - return Task.FromResult(JobResult.FromException(ex)); - } -} + o.Name = "resize-banner"; + o.TimeZone = TimeZoneInfo.FindSystemTimeZoneById("America/Chicago"); + o.ConfigurationVersion = 1; + }))); ``` -| Factory | `IsSuccess` | Behavior in continuous mode | -|---------|------------|---------------------------| -| `Success` / `SuccessWithMessage` | `true` | Waits `Interval` then runs again | -| `FailedWithMessage` / `FromException` | `false` | Waits at least 100ms (or `Interval`, whichever is longer) | -| `Cancelled` / `CancelledWithMessage` | N/A | Logged as warning; loop continues | +Five-field expressions use minute resolution; six-field expressions include seconds. Definitions persist a wire job name, serialized argument payload, time-zone ID, retry budget, enabled state, scope, overlap policy, and revision. They contain no CLR `Type`, delegates, or live argument objects. -## Queue Processor Jobs +Global schedules create one occurrence per tick across scheduler replicas. `PerNode` creates occurrences with affinity to each scheduler node; the worker on that node must use the same node identity and register the job type. `PerNode` requires an explicit stable `Jobs.ConfigureWorker(o => o with { NodeId = "worker-a" })` or `FOUNDATIO_NODE_ID`; startup fails if neither is configured. Use a unique stable identity per node. Global is the default and needs no node configuration. Unclaimed node-affine occurrences expire after `UnclaimedLifetime` (default one day), releasing backlog and overlap reservations when a node is retired. Once an attempt starts, ordinary retry and lease recovery apply. -### QueueJobBase\ +`SkipIfRunning` prevents a new occurrence while an earlier occurrence remains queued, processing, or waiting for retry. Explicitly allowing overlap permits concurrent occurrences. Unique occurrence IDs prevent duplicate materialization across concurrent scheduler polls. Manual triggers also respect overlap and disabled state. -[View source](https://github.com/FoundatioFx/Foundatio/blob/main/src/Foundatio/Jobs/QueueJobBase.cs) +The misfire window defaults to one minute and is limited to one day. Only ticks inside that window are caught up; this is not a promise to replay every tick after an unlimited outage. Keep high-frequency catch-up windows small. A skipped overlapping tick can be considered again while it remains in the window. -`QueueJobBase` processes items from an `IQueue`. Each call to `RunAsync` dequeues one item and calls your `ProcessQueueEntryAsync` method. It handles dequeue timeouts, cancellation, poison messages (null values), and optional per-entry locking automatically. +### Persisted edits and deployment reconciliation -**Key behaviors:** +`IScheduledJobManager` lists, inspects, reschedules, enables/disables, removes, and manually triggers schedules. `TriggerAsync(name)` returns a durable job handle. `ScheduleAsync` creates a typed runtime schedule. Register its job with `Jobs.AddJobType()` on producers and workers first; the DI-configured manager rejects unknown job types before persisting a schedule, with the same validation as `IJobClient`. -- **AutoComplete (default: `true`)** — entries are completed when `ProcessQueueEntryAsync` returns success, or abandoned on failure/exception. Set `AutoComplete = false` when you need to call `CompleteAsync()` / `AbandonAsync()` yourself. -- **Entry-level locking** — override `GetQueueEntryLockAsync` to acquire a distributed lock per queue entry before processing. The default returns an empty (no-op) lock. If `GetQueueEntryLockAsync` returns `null`, the entry is abandoned. If it throws, the entry is abandoned and a failure `JobResult` is returned. -- **Poison message safety** — entries with `null` values (deserialization failures) are automatically abandoned without calling your code. +Updates to `ScheduledJobDefinition` use its `Revision`; a stale update fails rather than silently replacing another operator's edit. Declarative configuration has a separate `ConfigurationVersion`. Restarting the same deployment preserves runtime edits. Changing a declaration requires increasing that configuration version; older deployments cannot overwrite newer definitions. An intentional higher version applies the new declaration and advances the stored revision. -```csharp -using Foundatio.Jobs; -using Foundatio.Queues; +Disabling or removing a schedule stops future materialization; already queued occurrences remain independent jobs. Cancel those explicitly if needed. -public class OrderProcessorJob : QueueJobBase -{ - private readonly IOrderService _orderService; - - public OrderProcessorJob( - IQueue queue, - IOrderService orderService, - TimeProvider timeProvider, - IResiliencePolicyProvider resiliencePolicyProvider, - ILoggerFactory loggerFactory) - : base(queue, timeProvider, resiliencePolicyProvider, loggerFactory) - { - _orderService = orderService; - } - - protected override async Task ProcessQueueEntryAsync( - QueueEntryContext context) - { - var workItem = context.QueueEntry.Value; - - _logger.LogInformation("Processing order {OrderId}", workItem.OrderId); - - try - { - await _orderService.ProcessAsync(workItem.OrderId, context.CancellationToken); - return JobResult.Success; - } - catch (Exception ex) - { - _logger.LogError(ex, "Failed to process order {OrderId}", workItem.OrderId); - return JobResult.FromException(ex); - } - } -} - -public record OrderWorkItem -{ - public int OrderId { get; init; } -} -``` - -### QueueEntryContext\ - -`QueueEntryContext` extends `JobContext` and is passed to `ProcessQueueEntryAsync`: +## Monitoring, retention, and capacity -| Member | Description | -|--------|-------------| -| `QueueEntry` | The `IQueueEntry` — access `Value`, `Id`, `Attempts`, `CompleteAsync()`, `AbandonAsync()` | -| `CancellationToken` | Inherited from `JobContext` | -| `Lock` | The per-entry lock from `GetQueueEntryLockAsync` | -| `RenewLockAsync()` | Renews the queue entry's visibility timeout (preventing redelivery) *and* the per-entry distributed lock | - -### IQueueJob\ - -`IQueueJob` extends `IJob` and exposes the queue and a direct processing method: - -- **`ProcessAsync(IQueueEntry, CancellationToken)`** — process a single entry obtained externally (e.g., from a test or a different dequeue source). -- **`Queue`** — the underlying `IQueue`. - -### Running Queue Jobs +`IJobMonitor.GetAsync(id)` retrieves a job. `QueryAsync(JobQuery)` returns a `JobPage`, ordered by job ID, with optional name/status filters. Limits range from 1 to 1,000 and default to 100. Continue using the returned token and the same filters: ```csharp -var queue = new InMemoryQueue(); -var job = serviceProvider.GetRequiredService(); - -// Enqueue work -await queue.EnqueueAsync(new OrderWorkItem { OrderId = 123 }); -await queue.EnqueueAsync(new OrderWorkItem { OrderId = 456 }); - -// Process all queued items, then stop (waits up to 30s for an empty queue) -await job.RunUntilEmptyAsync(); - -// Process with an explicit timeout for the empty-queue wait -await job.RunUntilEmptyAsync(TimeSpan.FromSeconds(10)); - -// Run continuously — processes items as they arrive -await job.RunContinuousAsync(cancellationToken: stoppingToken); -``` - -### Queue Processing Behaviors - -Behaviors hook into queue lifecycle events to add cross-cutting concerns without modifying your job. Attach them when creating the queue: - -```csharp -var cache = new InMemoryCacheClient(); -var queue = new InMemoryQueue(o => o - .AddBehavior(new DuplicateDetectionQueueBehavior( - cache, loggerFactory, detectionWindow: TimeSpan.FromMinutes(10)))); -``` - -`DuplicateDetectionQueueBehavior` discards duplicate entries based on `IHaveUniqueIdentifier.UniqueIdentifier`. Implement the interface on your work item type: - -```csharp -public record OrderWorkItem : IHaveUniqueIdentifier +string? cursor = null; +do { - public int OrderId { get; init; } - public string? UniqueIdentifier => $"order:{OrderId}"; -} + var page = await monitor.QueryAsync(new JobQuery { Status = JobStatus.Failed, AfterJobId = cursor }); + foreach (var job in page) + Console.WriteLine($"{job.JobId}: {job.Error}"); + cursor = page.ContinuationToken; +} while (cursor is not null); ``` -You can create custom behaviors by extending `QueueBehaviorBase` and overriding any combination of `OnEnqueuing`, `OnEnqueued`, `OnDequeued`, `OnCompleted`, `OnAbandoned`, `OnLockRenewed`, and `OnQueueDeleted`. - -## Work Item Jobs - -Work item jobs solve a different problem than queue jobs: they process **heterogeneous** tasks from a single shared queue. A `WorkItemJob` dequeues `WorkItemData` messages and dispatches each one to a type-specific handler. This is ideal for user-triggered operations (bulk deletes, imports, exports) where you want progress reporting and don't want to create a separate queue per task type. +Redis reads bounded index pages rather than loading every job. A filtered page can be empty and still have a continuation token. Pages are a live view; concurrent inserts or status changes are not a snapshot. -### Define a Work Item Handler +`JobRuntimeStoreOptions` separates admission, history and idempotency budgets: -Create handlers by extending `WorkItemHandlerBase`: +| Option | Default | +| --- | --- | +| `MaxActiveJobs` | 100,000 queued, scheduled or processing jobs | +| `MaxHistoryJobs` / `HistoryRetention` | 100,000 terminal records / seven days | +| `MaxDeduplicationRecords` / `DeduplicationRetention` | 1,000,000 ID reservations / seven days after completion | +| `MaxScheduledDispatches` | 100,000 delayed messages | +| `MaxPayloadBytes` | 1 MiB per job payload or scheduled message body plus UTF-8 header keys/values | -```csharp -using Foundatio.Jobs; - -public class DeleteEntityWorkItemHandler : WorkItemHandlerBase -{ - private readonly IEntityService _entityService; - - public DeleteEntityWorkItemHandler( - IEntityService entityService, - ILogger logger) : base(logger) - { - _entityService = entityService; - } +History is evicted by age or count independently of active capacity. Eviction preserves the ID reservation until its deduplication deadline; active jobs reserve IDs until they become terminal. Deduplication retention must be at least history retention. Admission fails with an actionable `JobException` when an applicable budget is full. Neither eviction nor a supplied ID makes business side effects exactly once. - public override async Task HandleItemAsync(WorkItemContext ctx) - { - var workItem = ctx.GetData(); - - await ctx.ReportProgressAsync(0, "Starting deletion..."); - - // Delete children with progress reporting - var children = await _entityService.GetChildrenAsync(workItem.EntityId); - var total = children.Count; - var current = 0; - - foreach (var child in children) - { - await _entityService.DeleteAsync(child.Id); - current++; - await ctx.ReportProgressAsync( - (current * 100) / total, - $"Deleted {current} of {total} children"); - } - - await _entityService.DeleteAsync(workItem.EntityId); - await ctx.ReportProgressAsync(100, "Deletion complete"); - } -} +Pass these options to `Jobs.UseInMemory(options)` or `RedisJobRuntimeStoreOptions.Runtime`. The old `MaxJobs` setting now means active capacity. Hosted workers clean up automatically; manual hosts call `CleanupAsync`. `GetStatsAsync` exposes current budget usage. -public record DeleteEntityWorkItem -{ - public int EntityId { get; init; } -} -``` - -### WorkItemContext - -`WorkItemContext` is passed to `HandleItemAsync` and provides everything a handler needs: - -| Member | Description | -|--------|-------------| -| `GetData()` | Deserializes the raw payload to your work item type | -| `Data` | The raw work item payload (use `GetData()` instead) | -| `JobId` | Unique identifier for this job run | -| `WorkItemLock` | Optional distributed lock for the work item | -| `CancellationToken` | Signals that processing should stop | -| `Result` | Set to `JobResult.FailedWithMessage(...)` to indicate failure without throwing. As with `QueueJobBase`, a non-success `Result` abandons and retries the entry -- see [Retry vs Permanent Failure](#retry-vs-permanent-failure) | -| `ReportProgressAsync(progress, message)` | Publishes `WorkItemStatus` updates via `IMessageBus` | -| `RenewLockAsync()` | Extends the work item lock lease | - -### WorkItemHandlers - -`WorkItemHandlers` is a registry mapping work item data types to their handlers. You can register handlers in several ways: - -```csharp -var handlers = new WorkItemHandlers(); - -// Instance registration -handlers.Register( - new DeleteEntityWorkItemHandler(entityService, logger)); - -// Factory registration (lazy — creates a new handler per invocation) -handlers.Register( - () => sp.GetRequiredService()); - -// Inline delegate (for simple tasks that don't need a full handler class) -handlers.Register(async ctx => -{ - var data = ctx.GetData(); - await ProcessAsync(data); -}); -``` - -### Register and Run Work Item Jobs - -```csharp -// DI registration -services.AddSingleton>(sp => new InMemoryQueue()); -services.AddSingleton(sp => new InMemoryMessageBus()); -services.AddSingleton(sp => sp.GetRequiredService()); -services.AddScoped(); -services.AddSingleton(sp => -{ - var handlers = new WorkItemHandlers(); - handlers.Register( - () => sp.GetRequiredService()); - return handlers; -}); - -// Run with multiple instances for parallel processing -var job = serviceProvider.GetRequiredService(); -await new JobRunner(job, serviceProvider, instanceCount: 2).RunAsync(stoppingToken); -``` - -### Trigger Work Items - -Use the `EnqueueAsync` extension method to enqueue strongly-typed work items: - -```csharp -var queue = serviceProvider.GetRequiredService>(); - -// Enqueue a work item (returns a job ID for tracking) -string jobId = await queue.EnqueueAsync(new DeleteEntityWorkItem { EntityId = 123 }); - -// With progress reporting enabled -string jobId = await queue.EnqueueAsync( - new DeleteEntityWorkItem { EntityId = 123 }, - includeProgressReporting: true); - -// Subscribe to progress updates -var messageBus = serviceProvider.GetRequiredService(); -await messageBus.SubscribeAsync(status => -{ - Console.WriteLine($"[{status.WorkItemId}] {status.Progress}% - {status.Message}"); -}); -``` - -## Job Runner - -[View source](https://github.com/FoundatioFx/Foundatio/blob/main/src/Foundatio/Jobs/JobRunner.cs) - -`JobRunner` orchestrates job execution with support for continuous running, multiple parallel instances, initial delays, and console hosting: - -```csharp -using Foundatio.Jobs; - -var job = serviceProvider.GetRequiredService(); -var runner = new JobRunner(job, serviceProvider); - -// Run until cancelled -await runner.RunAsync(stoppingToken); - -// Run in background (fire-and-forget) -runner.RunInBackground(); - -// Multiple parallel instances -var multiRunner = new JobRunner(job, serviceProvider, instanceCount: 4); -await multiRunner.RunAsync(stoppingToken); -``` - -### Console App Hosting - -`RunInConsoleAsync` sets up `Ctrl+C` and Azure WebJobs shutdown file handling, runs the job, and returns a process exit code: - -```csharp -var exitCode = await new JobRunner(job, serviceProvider).RunInConsoleAsync(); -Environment.Exit(exitCode); -// Returns: 0 = success, -1 = failure, 1 = unhandled exception -``` - -## Job Options - -### Job Attribute - -Configure job behavior declaratively with the `[Job]` attribute. These values become the defaults that `JobRunner` and the hosting infrastructure use: - -```csharp -[Job( - Name = "MyJob", - Description = "Processes pending items", - Interval = "5m", - InitialDelay = "10s", - IsContinuous = true, - IterationLimit = -1, - InstanceCount = 1 -)] -public class MyJob : JobBase -{ - public MyJob( - TimeProvider timeProvider, - IResiliencePolicyProvider resiliencePolicyProvider, - ILoggerFactory loggerFactory) - : base(timeProvider, resiliencePolicyProvider, loggerFactory) { } - - protected override Task RunInternalAsync(JobContext context) - { - return Task.FromResult(JobResult.Success); - } -} -``` - -| Property | Type | Default | Description | -|----------|------|---------|-------------| -| `Name` | `string?` | Type name minus "Job" suffix | Display name used in logging and status APIs | -| `Description` | `string?` | `null` | Human-readable description | -| `IsContinuous` | `bool` | `true` | Whether the job runs in a loop | -| `Interval` | `string?` | `null` | Delay between iterations (e.g., `"5m"`, `"30s"`) | -| `InitialDelay` | `string?` | `null` | Delay before first execution | -| `IterationLimit` | `int` | `-1` | Maximum iterations (`-1` = unlimited) | -| `InstanceCount` | `int` | `1` | Number of parallel instances | - -### JobOptions Class - -`JobOptions` holds the same settings programmatically. Values from `[Job]` are applied as defaults, and can be overridden at runtime: - -```csharp -var options = new JobOptions -{ - Name = "CleanupJob", - Interval = TimeSpan.FromHours(1), - IterationLimit = 100, - RunContinuous = true, - InstanceCount = 2, - InitialDelay = TimeSpan.FromSeconds(30) -}; - -await job.RunContinuousAsync(options, stoppingToken); -``` - -## Hosted Service Integration - -`Foundatio.Extensions.Hosting` integrates Foundatio jobs with ASP.NET Core's `IHostedService` pipeline. Jobs are registered as managed background services that start with the host and shut down gracefully. - -### Installation - -```bash -dotnet add package Foundatio.Extensions.Hosting -``` - -### AddJob Extension - -Register jobs as hosted services with a fluent builder: - -```csharp -using Foundatio.Extensions.Hosting.Jobs; - -// Simple registration — runs continuously -services.AddJob(); - -// With configuration -services.AddJob(o => o - .Interval(TimeSpan.FromHours(1)) - .WaitForStartupActions() - .InitialDelay(TimeSpan.FromSeconds(30))); - -// Parallel queue processing -services.AddJob(o => o.InstanceCount(4)); -``` - -The builder exposes: `Name`, `Description`, `JobFactory`, `RunContinuous`, `Interval`, `InitialDelay`, `IterationLimit`, `InstanceCount`, and `WaitForStartupActions`. - -### Cron Job Scheduling - -Schedule jobs using cron expressions: - -```csharp -using Foundatio.Extensions.Hosting.Jobs; - -// Every 6 hours -services.AddCronJob("0 */6 * * *"); - -// Every Monday at midnight -services.AddCronJob("0 0 * * MON"); - -// With configuration -services.AddCronJob("0 2 * * *", o => o - .Name("nightly-maintenance") - .WaitForStartupActions()); - -// Inline action — no job class needed -services.AddCronJob("health-check", "*/5 * * * *", async (sp, ct) => -{ - var healthService = sp.GetRequiredService(); - await healthService.CheckAsync(ct); -}); -``` - -#### Cron Helper Class - -Use the `Cron` helper to generate common cron expressions without memorizing the syntax: - -```csharp -using Foundatio.Extensions.Hosting.Jobs; - -services.AddCronJob(Cron.Hourly()); // every hour at :00 -services.AddCronJob(Cron.Daily(hour: 2)); // daily at 2:00 AM -services.AddCronJob(Cron.Weekly(DayOfWeek.Monday, hour: 9)); // Monday at 9 AM -services.AddCronJob(Cron.Monthly(day: 1)); // 1st of each month -services.AddCronJob(Cron.Minutely(5)); // every 5 minutes -services.AddCronJob(Cron.Yearly(month: 1)); // January 1st -services.AddCronJob(Cron.Never()); // never (disabled) -``` - -#### Scheduled Job Options - -Cron jobs support additional configuration through `ScheduledJobOptionsBuilder`: - -```csharp -services.AddCronJob("0 0 * * *", o => o - .Name("daily-report") - .Description("Generates the daily summary report") - .WaitForStartupActions() - .CronTimeZone("America/New_York") - .Enabled(true)); -``` - -### Distributed Cron Jobs - -Ensure only one instance runs a scheduled job across all servers. This requires an `ICacheClient` registration for distributed lock coordination: - -```csharp -using Foundatio.Extensions.Hosting.Jobs; - -services.AddDistributedCronJob("0 0 * * *"); - -// Requires ICacheClient for distributed locking -services.AddSingleton(sp => new RedisCacheClient(...)); -``` - -### Job Manager - -`IJobManager` provides a runtime API for inspecting, triggering, and managing scheduled jobs. It is automatically registered when you use `AddCronJob` or `AddJobScheduler`: - -```csharp -var jobManager = serviceProvider.GetRequiredService(); - -// View all job statuses -JobStatus[] statuses = jobManager.GetJobStatus(); -foreach (var status in statuses) - Console.WriteLine($"{status.Name}: NextRun={status.NextRun}, LastRun={status.LastRun}"); - -// Trigger a job on-demand (runs immediately regardless of schedule) -await jobManager.RunJobAsync(); - -// Add or update a scheduled job at runtime -jobManager.AddOrUpdate(o => o.CronSchedule(Cron.Hourly())); - -// Disable a job without removing it -jobManager.Update(o => o.Disabled()); - -// Remove a job entirely -jobManager.Remove(); - -// Release a stuck distributed lock (e.g., after a server crash) -await jobManager.ReleaseLockAsync("Cleanup"); -``` - -### Manual BackgroundService - -When the `AddJob` extensions don't fit your needs, you can integrate any Foundatio job with `BackgroundService` directly: - -```csharp -public class CleanupJobHostedService : BackgroundService -{ - private readonly IServiceProvider _services; - private readonly ILogger _logger; - - public CleanupJobHostedService( - IServiceProvider services, ILogger logger) - { - _services = services; - _logger = logger; - } - - protected override async Task ExecuteAsync(CancellationToken stoppingToken) - { - while (!stoppingToken.IsCancellationRequested) - { - using var scope = _services.CreateScope(); - var job = scope.ServiceProvider.GetRequiredService(); - - try { await job.RunAsync(stoppingToken); } - catch (Exception ex) { _logger.LogError(ex, "Cleanup job failed"); } - - await Task.Delay(TimeSpan.FromHours(1), stoppingToken); - } - } -} - -services.AddScoped(); -services.AddHostedService(); -``` - -## Common Patterns - -### Job with Progress Reporting - -Use `IMessageBus` to publish progress from standard jobs: - -```csharp -public class ImportJob : JobBase -{ - private readonly IMessageBus _messageBus; - - public ImportJob( - IMessageBus messageBus, - TimeProvider timeProvider, - IResiliencePolicyProvider resiliencePolicyProvider, - ILoggerFactory loggerFactory) - : base(timeProvider, resiliencePolicyProvider, loggerFactory) - { - _messageBus = messageBus; - } - - protected override async Task RunInternalAsync(JobContext context) - { - var items = await GetItemsToImportAsync(); - var total = items.Count; - - for (int i = 0; i < total; i++) - { - if (context.CancellationToken.IsCancellationRequested) - return JobResult.Cancelled; - - await ImportItemAsync(items[i]); - await _messageBus.PublishAsync(new ImportProgress - { - ProcessedCount = i + 1, - TotalCount = total, - PercentComplete = ((i + 1) * 100) / total - }); - } - - return JobResult.Success; - } -} -``` - -### Retry vs Permanent Failure - -Distinguish between transient errors (retry is useful) and permanent errors (retry would loop forever). Whether returning a failed `JobResult` actually triggers a retry depends on the job type: - -- **Queue-processed jobs** (`QueueJobBase`, `WorkItemJob`) -- a non-success result abandons the queue entry, which re-queues it for retry and eventually moves it to the [dead letter queue](./queues#dead-letter-queue) once `Retries` is exhausted. Return `FailedWithMessage`/`FromException` only for errors you want retried; for permanent errors, log it yourself and return `JobResult.Success` (or `SuccessWithMessage`) so the entry completes instead of retrying forever and dead-lettering. - -```csharp -protected override async Task ProcessQueueEntryAsync(QueueEntryContext context) -{ - try - { - await DoWorkAsync(context.CancellationToken); - return JobResult.Success; - } - catch (TransientException ex) - { - return JobResult.FailedWithMessage(ex.Message); // entry is abandoned, retried, and eventually dead-lettered - } - catch (PermanentException ex) - { - _logger.LogError(ex, "Permanent failure, not retrying"); - return JobResult.Success; // complete the entry instead of retrying/dead-lettering - } -} -``` - -- **Standalone/manual jobs** (`JobBase`, a one-off `RunAsync()`/`RunInConsoleAsync()` run, or scheduled/cron jobs via `Foundatio.Extensions.Hosting`) -- there is no built-in retry or dead letter queue. A failed result just reflects the outcome: an error-level log entry, a non-zero exit code from `RunInConsoleAsync`, or a failed run in the scheduled job history. Returning `FailedWithMessage`/`FromException` for a permanent error is correct here; nothing inside Foundatio retries it, and any retry decision belongs to whatever runs the job (a scheduler, CI pipeline, or Kubernetes restart policy). - -```csharp -protected override async Task RunInternalAsync(JobContext context) -{ - try - { - await DoWorkAsync(context.CancellationToken); - return JobResult.Success; - } - catch (Exception ex) - { - // Nothing in Foundatio retries a standalone job -- return the real outcome either way - return JobResult.FromException(ex); - } -} -``` - -### Idempotent Jobs - -Track progress externally so the job can safely resume after a crash: - -```csharp -protected override async Task RunInternalAsync(JobContext context) -{ - var lastProcessedId = await _state.GetLastProcessedIdAsync(); - var items = await _db.GetItemsAfterAsync(lastProcessedId); - - foreach (var item in items) - { - context.CancellationToken.ThrowIfCancellationRequested(); - await ProcessItemAsync(item); - await _state.SetLastProcessedIdAsync(item.Id); - } - - return JobResult.Success; -} -``` - -## Best Practices - -1. **Always propagate cancellation tokens.** Pass `context.CancellationToken` to every async call and check it in loops. This ensures your job shuts down promptly during host shutdown. - -2. **Renew locks in long-running jobs.** If your job holds a distributed lock (via `JobWithLockBase` or queue entry locking), call `context.RenewLockAsync()` periodically — especially between batches. Lock expiration mid-run causes correctness issues. - -3. **Keep jobs idempotent.** Jobs may be killed at any point (process recycle, deployment, crash). Track progress so they can pick up where they left off rather than re-processing everything. - -4. **Log with structured context.** Use `BeginScope` to correlate all log entries for a unit of work: - -```csharp -using var _ = _logger.BeginScope(s => s.Property("OrderId", workItem.OrderId)); -_logger.LogInformation("Processing order..."); -// every log inside this scope automatically includes OrderId -``` - -5. **Match job type to workload.** Don't force a `QueueJobBase` when a simple `JobBase` with `RunContinuousAsync` suffices. Don't create separate queues for every task type — use `WorkItemJob` for heterogeneous on-demand work. - -6. **Use distributed cron for cluster-wide scheduling.** If you have multiple servers running the same host, use `AddDistributedCronJob` to ensure only one server executes the scheduled run. - -## Dependency Injection - -### Register Standard Jobs - -```csharp -services.AddScoped(); -services.AddScoped(); -services.AddSingleton>(sp => new InMemoryQueue()); -``` - -### Register Queue Jobs with Parallel Processing - -```csharp -services.AddSingleton>(sp => new InMemoryQueue()); -services.AddJob(o => o.InstanceCount(4)); -``` +`AddFoundatioWorker` registers a `foundatio` health check. Map it with ASP.NET Core's `app.MapHealthChecks("/health")` or query `HealthCheckService`. Worker, scheduler, dispatcher and listener recovery affect health. The `Foundatio.Runtime` meter exposes active jobs, retained history, ID reservations and scheduled dispatch counts. Job exceptions are logged with ID, wire type and attempt, with a bounded failure summary in `JobState.Error`; successful messages use `ResultMessage`. -## Next Steps +## Testing and migration -- [Queues](./queues) — Queue implementations for job processing -- [Locks](./locks) — Distributed locking for singleton jobs -- [Resilience](./resilience) — Retry policies for job reliability -- [Serialization](./serialization) — Serializer configuration and performance +`Foundatio.Testing.JobsTestHarness` runs the real worker/scheduler with in-memory state and a controlled clock. `RunAllQueuedAsync()` drains currently eligible work across batches; future delayed jobs remain queued. `RunToCompletionAsync(handle)` runs only that job. `RunDueAsync()` materializes due CRON occurrences and drains eligible jobs; it does not dispatch scheduled messages. Shared `JobRuntimeStoreConformanceTests` cover ownership fencing, eligibility, concurrency, retries, cancellation, schedule revisions, dispatch recovery, pagination, and retention against memory and Redis. +Old `JobBase`, `QueueJobBase`, `JobWithLockBase`, `JobRunner`, and `WorkItemJob` implementations migrate to plain `IJob` or `IMessageHandler`. Replace `RunAsync(CancellationToken)` with `RunAsync(JobExecutionContext)`, queue jobs with explicit message consumers, and work-item payloads with `IJob`. The old automatic runtime pump and generic state-patch store API are removed; host roles and ownership-specific store operations are explicit. diff --git a/docs/guide/locks.md b/docs/guide/locks.md index d8ffdcb1f..b5cef18c2 100644 --- a/docs/guide/locks.md +++ b/docs/guide/locks.md @@ -58,7 +58,7 @@ using Foundatio.Caching; using Foundatio.Messaging; var cache = new InMemoryCacheClient(); -var messageBus = new InMemoryMessageBus(); +var messageBus = new MessageBus(new InMemoryMessageTransport()); var locker = new CacheLockProvider(cache, messageBus); await using var lck = await locker.TryAcquireAsync("my-resource"); @@ -436,7 +436,7 @@ public async Task ProcessRequest(string userId) ```csharp services.AddSingleton(); -services.AddSingleton(); +services.AddFoundatio().Messaging.UseInMemory(); services.AddSingleton(sp => new CacheLockProvider( @@ -676,3 +676,7 @@ if (lck is not null) - [Jobs](./jobs) - Background jobs with distributed locking - [Resilience](./resilience) - Retry policies for lock acquisition - [Serialization](./serialization) - Serializer configuration and performance + +## Native Redis resource locks + +`foundatio.Locking.UseRedis()` registers `RedisLockProvider` using the application's shared `IConnectionMultiplexer`. Acquisition uses Redis `SET NX` with expiration; renewal and release compare the ownership token atomically. `LockOwnershipLostException` signals that a holder may no longer extend its lease. Expired holders cannot release a successor's lock. These are leased resource locks, not deduplication or fencing of external writes. Configure distinct key prefixes for unrelated applications. diff --git a/docs/guide/messaging-jobs-redesign.md b/docs/guide/messaging-jobs-redesign.md new file mode 100644 index 000000000..be78dc8ba --- /dev/null +++ b/docs/guide/messaging-jobs-redesign.md @@ -0,0 +1,9 @@ +# Messaging and jobs redesign + +This unreleased redesign uses explicit queue consumers and event subscribers, with an optional durable job runtime. See [Messaging](messaging.md) and [Durable jobs](jobs.md) for current APIs, runnable examples, guarantees, and migration guidance. + +The common setup is a message bus plus a handler. Add durable jobs only when persisted execution state, progress, cancellation, retries, or CRON schedules are needed. Use `AddFoundatioWorker(configure)` for a combined worker. Plain `AddFoundatio()` registers clients without starting execution; individual hosting methods support separate role deployments. + +The transport handles bytes, metadata, receipts, and broker operations. The messaging core owns routing, serialization, retry policy, lease supervision, and settlement. The job store owns atomic admission, claims, fenced mutations, schedule definitions, and retention. Ad hoc jobs and CRON occurrences run through one worker state machine. + +Delivery is at least once. Stable application IDs, idempotent business operations, and transactional outbox/inbox boundaries remain application responsibilities. The runtime does not promise exactly-once side effects or distributed transactions across a business database and broker. diff --git a/docs/guide/messaging.md b/docs/guide/messaging.md index 39808fecc..e43e132e8 100644 --- a/docs/guide/messaging.md +++ b/docs/guide/messaging.md @@ -1,1061 +1,177 @@ # Messaging -Messaging allows you to publish and subscribe to messages flowing through your application using pub/sub patterns. Foundatio provides multiple message bus implementations through the `IMessageBus` interface. +Use `IMessageBus` for two common patterns: `SendAsync` queues work for competing consumers; `PublishAsync` sends an event to each existing subscription. Queue consumers and event subscribers are registered separately. Publishing without subscriptions drops the event. -## The IMessageBus Interface - -[View source](https://github.com/FoundatioFx/Foundatio/blob/main/src/Foundatio/Messaging/IMessageBus.cs) - -```csharp -public interface IMessageBus : IMessagePublisher, IMessageSubscriber, IDisposable, IAsyncDisposable -{ -} - -public interface IMessagePublisher -{ - Task PublishAsync(Type messageType, object message, - MessageOptions? options = null, - CancellationToken cancellationToken = default); -} - -public interface IMessageSubscriber -{ - Task SubscribeAsync(Func handler, - CancellationToken cancellationToken = default) where T : class; -} -``` - -## Implementations - -### InMemoryMessageBus - -An in-memory message bus for development and testing: - -[View source](https://github.com/FoundatioFx/Foundatio/blob/main/src/Foundatio/Messaging/InMemoryMessageBus.cs) - -```csharp -using Foundatio.Messaging; - -var messageBus = new InMemoryMessageBus(); - -// Subscribe to messages -await messageBus.SubscribeAsync(async msg => -{ - Console.WriteLine($"Order created: {msg.OrderId}"); -}); - -// Publish a message -await messageBus.PublishAsync(new OrderCreated { OrderId = 123 }); -``` - -### AzureServiceBusMessageBus - -Messaging using Azure Service Bus (separate package): - -[View source](https://github.com/FoundatioFx/Foundatio.AzureServiceBus/blob/main/src/Foundatio.AzureServiceBus/Messaging/AzureServiceBusMessageBus.cs) - -```csharp -// dotnet add package Foundatio.AzureServiceBus - -using Foundatio.AzureServiceBus.Messaging; - -var messageBus = new AzureServiceBusMessageBus(o => { - o.ConnectionString = "..."; - o.Topic = "events"; -}); -``` - -### KafkaMessageBus - -Messaging using Apache Kafka (separate package): - -[View source](https://github.com/FoundatioFx/Foundatio.Kafka/blob/main/src/Foundatio.Kafka/Messaging/KafkaMessageBus.cs) - -```csharp -// dotnet add package Foundatio.Kafka - -using Foundatio.Kafka.Messaging; - -var messageBus = new KafkaMessageBus(o => { - o.BootstrapServers = "localhost:9092"; -}); -``` - -### RabbitMQMessageBus - -Messaging using RabbitMQ (separate package): - -[View source](https://github.com/FoundatioFx/Foundatio.RabbitMQ/blob/main/src/Foundatio.RabbitMQ/Messaging/RabbitMQMessageBus.cs) +## Start with a worker ```csharp -// dotnet add package Foundatio.RabbitMQ - -using Foundatio.RabbitMQ.Messaging; - -var messageBus = new RabbitMQMessageBus(o => { - o.ConnectionString = "amqp://guest:guest@localhost:5672"; -}); -``` - -### RedisMessageBus - -Distributed messaging using Redis pub/sub (separate package): - -[View source](https://github.com/FoundatioFx/Foundatio.Redis/blob/main/src/Foundatio.Redis/Messaging/RedisMessageBus.cs) - -```csharp -// dotnet add package Foundatio.Redis - -using Foundatio.Redis.Messaging; -using StackExchange.Redis; - -var redis = await ConnectionMultiplexer.ConnectAsync("localhost:6379"); -var messageBus = new RedisMessageBus(o => o.Subscriber = redis.GetSubscriber()); -``` - -### SQSMessageBus - -Messaging using AWS SNS/SQS (separate package): - -[View source](https://github.com/FoundatioFx/Foundatio.AWS/blob/master/src/Foundatio.AWS/Messaging/SQSMessageBus.cs) - -```csharp -// dotnet add package Foundatio.AWS - +using Foundatio; using Foundatio.Messaging; -var messageBus = new SQSMessageBus(o => { - o.ConnectionString = connectionString; - o.Topic = "events"; - // Optional: Specify queue name for durable subscriptions - // o.SubscriptionQueueName = "my-service-queue"; -}); -``` - -## Basic Usage - -### Publishing Messages - -```csharp -var messageBus = new InMemoryMessageBus(); - -// Simple publish -await messageBus.PublishAsync(new OrderCreated { OrderId = 123 }); - -// With options -await messageBus.PublishAsync(new OrderCreated { OrderId = 123 }, new MessageOptions -{ - CorrelationId = "request-abc", - DeliveryDelay = TimeSpan.FromSeconds(30), - Properties = new Dictionary - { - ["source"] = "order-service" - } -}); - -// Delayed publish (extension method) -await messageBus.PublishAsync( - new OrderReminder { OrderId = 123 }, - TimeSpan.FromHours(1) -); -``` - -### Subscribing to Messages - -```csharp -var messageBus = new InMemoryMessageBus(); - -// Simple subscription -await messageBus.SubscribeAsync(async order => -{ - Console.WriteLine($"Processing order: {order.OrderId}"); -}); - -// With cancellation token -await messageBus.SubscribeAsync( - async (order, ct) => - { - await ProcessOrderAsync(order, ct); - }, - cancellationToken -); - -// Synchronous handler -await messageBus.SubscribeAsync(order => -{ - Console.WriteLine($"Order: {order.OrderId}"); -}); -``` - -### Multiple Subscribers - -Each subscriber receives every message: - -```csharp -var messageBus = new InMemoryMessageBus(); - -// Handler 1: Logging -await messageBus.SubscribeAsync(async order => -{ - _logger.LogInformation("Order {OrderId} created", order.OrderId); -}); - -// Handler 2: Notification -await messageBus.SubscribeAsync(async order => -{ - await _notificationService.SendAsync(order.CustomerId, "Order placed!"); -}); - -// Handler 3: Analytics -await messageBus.SubscribeAsync(async order => -{ - await _analytics.TrackAsync("order_created", order.OrderId); -}); - -// All three handlers receive this message -await messageBus.PublishAsync(new OrderCreated { OrderId = 123 }); -``` - -## Message Types - -### Define Your Messages - -```csharp -// Simple message -public record OrderCreated -{ - public int OrderId { get; init; } - public DateTime CreatedAt { get; init; } - public string CustomerId { get; init; } -} - -// Message with interface for grouping -public interface IOrderEvent { int OrderId { get; } } - -public record OrderShipped : IOrderEvent -{ - public int OrderId { get; init; } - public string TrackingNumber { get; init; } -} - -public record OrderDelivered : IOrderEvent -{ - public int OrderId { get; init; } - public DateTime DeliveredAt { get; init; } -} -``` - -### Subscribe to Interface - -Subscribe to all messages implementing an interface: - -```csharp -// Receives OrderShipped, OrderDelivered, and any other IOrderEvent -await messageBus.SubscribeAsync(async orderEvent => -{ - _logger.LogInformation("Order event: {Type} for {OrderId}", - orderEvent.GetType().Name, orderEvent.OrderId); -}); -``` - -### IMessage Interface - -Use the built-in `IMessage` interface for raw message access: - -```csharp -await messageBus.SubscribeAsync(async (IMessage message, CancellationToken ct) => -{ - Console.WriteLine($"Type: {message.Type}"); - Console.WriteLine($"Correlation ID: {message.CorrelationId}"); - - // Deserialize the data - var order = message.GetBody(); -}); -``` - -#### Breaking change: `IMessage.Data` is now `ReadOnlyMemory` - -`IMessage.Data` exposes the raw payload as `ReadOnlyMemory` instead of `byte[]`. This lets memory-backed transports such as Azure Service Bus and RabbitMQ expose the payload without copying it into a new array. Since it is a struct, follow these patterns: - -- Check for an empty payload with `message.Data.IsEmpty` (not `== null`) -- Read the bytes directly via `message.Data.Span` -- Call `message.Data.ToArray()` only when you need a `byte[]` - -**Buffer validity:** `Data` is only guaranteed valid for the duration of message handling. Some providers (such as RabbitMQ) expose a pooled transport buffer that is reclaimed once your handler returns, so the framework deserializes the body within the handler. If you need to retain the raw payload beyond the current handler invocation, copy it with `message.Data.ToArray()`. - -Most code that uses `GetBody()` / `Body` is unaffected. When constructing a `Message`, you can still pass a `byte[]`; it converts implicitly to `ReadOnlyMemory`. - -## Common Patterns - -### Event-Driven Architecture - -Decouple services with events: - -```csharp -// Order Service -public class OrderService -{ - private readonly IMessageBus _messageBus; - - public async Task CreateOrderAsync(CreateOrderRequest request) - { - var order = await _repository.CreateAsync(request); - - // Publish event for other services - await _messageBus.PublishAsync(new OrderCreated - { - OrderId = order.Id, - CustomerId = request.CustomerId, - CreatedAt = DateTime.UtcNow - }); - } -} - -// Inventory Service (separate process/service) -public class InventoryService -{ - public InventoryService(IMessageBus messageBus) - { - messageBus.SubscribeAsync(async order => - { - await ReserveInventoryAsync(order.OrderId); - }); - } -} - -// Notification Service (separate process/service) -public class NotificationService -{ - public NotificationService(IMessageBus messageBus) - { - messageBus.SubscribeAsync(async order => - { - await SendConfirmationEmailAsync(order.CustomerId); - }); - } -} -``` - -### Cache Invalidation - -Coordinate cache across instances: - -```csharp -public class CacheInvalidationService -{ - private readonly IMessageBus _messageBus; - private readonly ICacheClient _localCache; - - public CacheInvalidationService(IMessageBus messageBus, ICacheClient localCache) - { - _messageBus = messageBus; - _localCache = localCache; - - // Listen for invalidation messages - _messageBus.SubscribeAsync(async msg => - { - await _localCache.RemoveAsync(msg.Key); - }); - } - - public async Task InvalidateAsync(string key) - { - // Remove locally - await _localCache.RemoveAsync(key); - - // Notify other instances - await _messageBus.PublishAsync(new CacheInvalidated { Key = key }); - } -} - -public record CacheInvalidated { public string Key { get; init; } } -``` - -### Real-Time Updates - -Push updates to clients: - -```csharp -// Server-side -public class NotificationHub -{ - private readonly IMessageBus _messageBus; - - public NotificationHub(IMessageBus messageBus) - { - _messageBus = messageBus; - - // Forward bus messages to SignalR/WebSocket - _messageBus.SubscribeAsync(async notification => - { - await _hubContext.Clients - .User(notification.UserId) - .SendAsync("notification", notification); - }); - } -} - -// When something happens -await messageBus.PublishAsync(new UserNotification -{ - UserId = "user-123", - Message = "Your order has shipped!" -}); -``` - -### Saga/Process Manager - -Coordinate multi-step processes: - -```csharp -public class OrderSaga -{ - private readonly IMessageBus _messageBus; - - public OrderSaga(IMessageBus messageBus) - { - _messageBus = messageBus; - - // Step 1: Order created -> Reserve inventory - _messageBus.SubscribeAsync(async order => - { - await ReserveInventoryAsync(order.OrderId); - await _messageBus.PublishAsync(new InventoryReserved { OrderId = order.OrderId }); - }); - - // Step 2: Inventory reserved -> Process payment - _messageBus.SubscribeAsync(async evt => - { - await ProcessPaymentAsync(evt.OrderId); - await _messageBus.PublishAsync(new PaymentProcessed { OrderId = evt.OrderId }); - }); - - // Step 3: Payment processed -> Ship order - _messageBus.SubscribeAsync(async evt => - { - await ShipOrderAsync(evt.OrderId); - }); - } -} -``` - -## Message Options - -Configure message delivery: - -```csharp -await messageBus.PublishAsync(new OrderCreated { OrderId = 123 }, new MessageOptions -{ - // Unique message identifier - UniqueId = Guid.NewGuid().ToString(), - - // For tracing across services - CorrelationId = Activity.Current?.Id, - - // Delayed delivery - DeliveryDelay = TimeSpan.FromMinutes(5), - - // Custom properties - Properties = new Dictionary - { - ["source"] = "order-service", - ["version"] = "1.0" - } -}); -``` - -## Delayed Message Delivery - -The `DeliveryDelay` option schedules messages for future delivery. This is useful for scenarios like: - -- **Eventual consistency** - Wait for data to propagate before processing -- **Scheduled reminders** - Send notifications after a delay -- **Retry with backoff** - Republish failed messages with increasing delays - -### Basic Usage - -```csharp -// Using MessageOptions -await messageBus.PublishAsync(new OrderReminder { OrderId = 123 }, new MessageOptions -{ - DeliveryDelay = TimeSpan.FromMinutes(30) -}); - -// Using extension method -await messageBus.PublishAsync(new OrderReminder { OrderId = 123 }, TimeSpan.FromMinutes(30)); -``` - -### Provider Support - -Different providers handle delayed delivery differently: - -| Provider | Implementation | Persistence | Survives Restart | -|----------|---------------|-------------|------------------| -| **InMemoryMessageBus** | In-memory timer | None | No | -| **AzureServiceBusMessageBus** | Native `ScheduledEnqueueTime` | Azure | Yes | -| **KafkaMessageBus** | In-memory timer | None | No | -| **RabbitMQMessageBus** | Plugin or fallback | Plugin: Yes, Fallback: No | Plugin: Yes, Fallback: No | -| **RedisMessageBus** | In-memory timer | None | No | -| **SQSMessageBus** | In-memory timer | None | No | - -### Native vs Fallback Implementation - -**Native implementations** (Azure Service Bus, RabbitMQ with plugin) persist the delayed message in the broker. The message survives application restarts and is delivered reliably. - -**Fallback implementations** hold the message in memory using a timer. This has important limitations: - -::: warning Fallback Limitations -- **Messages are lost on restart** - If your application restarts before the delay expires, the message is permanently lost -- **Messages are discarded on disposal** - During graceful shutdown, pending delayed messages are discarded -- **Best-effort delivery** - No guarantee the message will be delivered -::: - -### RabbitMQ Plugin - -RabbitMQ requires the `rabbitmq_delayed_message_exchange` plugin for native delayed delivery: - -```bash -# Enable the plugin -rabbitmq-plugins enable rabbitmq_delayed_message_exchange -``` - -The `RabbitMQMessageBus` automatically detects if the plugin is available and uses it when present. Otherwise, it falls back to the in-memory timer. - -### When to Use Delayed Delivery - -**Appropriate use cases (fallback is acceptable):** - -- Cache invalidation -- Non-critical notifications -- Eventual consistency delays (e.g., waiting for Elasticsearch to refresh) - -**NOT appropriate for fallback:** - -- Financial transactions -- Order processing -- Any message where loss is unacceptable - -For guaranteed delayed delivery, use: -- Azure Service Bus (native support) -- RabbitMQ with the delayed message plugin -- `IQueue` with `DeliveryDelay` for work items that must be processed - -## Distributed Tracing - -Foundatio automatically integrates with .NET's distributed tracing infrastructure (`System.Diagnostics.Activity`) to enable end-to-end request tracing across services. - -### Automatic CorrelationId Injection - -When you publish a message, Foundatio automatically captures the current trace context: - -```csharp -// If Activity.Current exists, its ID is automatically used as CorrelationId -await messageBus.PublishAsync(new OrderCreated { OrderId = 123 }); - -// The message will have: -// - CorrelationId = Activity.Current?.Id -// - Properties["TraceState"] = Activity.Current?.TraceStateString (if present) -``` - -### Manual CorrelationId - -You can also set the `CorrelationId` explicitly: - -```csharp -await messageBus.PublishAsync(new OrderCreated { OrderId = 123 }, new MessageOptions -{ - CorrelationId = "my-custom-correlation-id" -}); -``` - -When you provide a `CorrelationId`, the automatic injection is skipped. - -### Trace Propagation - -When a subscriber receives a message, Foundatio: - -1. Creates a new `Activity` with the message's `CorrelationId` as the parent -2. Restores the `TraceState` from message properties -3. Adds the `CorrelationId` to the logging scope - -This enables distributed tracing tools (like Application Insights, Jaeger, or Zipkin) to correlate requests across services. - -### Accessing Trace Information - -In your subscriber, you can access the trace context: - -```csharp -await messageBus.SubscribeAsync(async (message, ct) => -{ - // Access correlation ID - var correlationId = message.CorrelationId; - - // Access custom properties - var traceState = message.Properties.GetValueOrDefault("TraceState"); - - // Activity.Current is automatically set with the message's trace context - _logger.LogInformation("Processing message with trace {TraceId}", Activity.Current?.TraceId); -}); +builder.Services.AddFoundatioWorker(foundatio => foundatio + .UseServiceName("billing") + .ConfigureMessaging(messaging => messaging.UseInMemory() + .AddConsumer() + .AddSubscriber())); ``` -### Integration with OpenTelemetry - -Foundatio's tracing integrates seamlessly with OpenTelemetry: - -```csharp -services.AddOpenTelemetry() - .WithTracing(builder => - { - builder.AddSource(FoundatioDiagnostics.ActivitySource.Name); - // ... other configuration - }); -``` - -## Dependency Injection - -### Basic Registration +Handlers implement `IMessageHandler`: ```csharp -// In-memory (development) -services.AddSingleton(); - -// Redis (production) -services.AddSingleton(sp => -{ - var redis = sp.GetRequiredService(); - return new RedisMessageBus(o => o.Subscriber = redis.GetSubscriber()); -}); -``` - -### Subscribe at Startup - -```csharp -public class MessageSubscriber : IHostedService +public sealed class SendReceiptHandler(ReceiptService receipts) : IMessageHandler { - private readonly IMessageBus _messageBus; - private readonly IServiceProvider _services; - - public MessageSubscriber(IMessageBus messageBus, IServiceProvider services) - { - _messageBus = messageBus; - _services = services; - } - - public async Task StartAsync(CancellationToken cancellationToken) - { - await _messageBus.SubscribeAsync(async (msg, ct) => - { - using var scope = _services.CreateScope(); - var handler = scope.ServiceProvider.GetRequiredService(); - await handler.HandleAsync(msg, ct); - }, cancellationToken); - } - - public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask; + public Task HandleAsync(IMessageContext context, CancellationToken token) + => receipts.SendAsync(context.Message.OrderId, token); } - -// Register -services.AddHostedService(); ``` -## Error Handling - -### MessageBusException - -All message bus implementations throw `MessageBusException` for transport-level errors. This provides a consistent exception type regardless of the underlying provider: +Each invocation gets its own dependency injection scope. A successful handler is acknowledged automatically; an exception follows the retry policy and eventually parks the message in a dead-letter destination. Use the supplied cancellation token for downstream work. ```csharp -try -{ - await messageBus.PublishAsync(new OrderCreated { OrderId = 123 }); -} -catch (MessageBusException ex) -{ - _logger.LogError(ex, "Failed to publish message: {Message}", ex.Message); - // Handle transport error (network, broker unavailable, etc.) -} -catch (OperationCanceledException) -{ - // Handle cancellation -} +await bus.SendAsync(new SendReceipt(1001)); +await bus.PublishAsync(new OrderPlaced(1001)); ``` -**Exception Behavior:** - -| Scenario | Exception Type | Notes | -|----------|---------------|-------| -| Transport error (network, broker) | `MessageBusException` | Wraps underlying exception | -| Null message/type | `ArgumentNullException` | Thrown immediately | -| Cancellation requested | `OperationCanceledException` | Passed through unchanged | -| Serialization error | `MessageBusException` | Wraps serialization exception | +Use `AddFoundatio().Messaging.UseInMemory()` in an API process that only produces messages. Client and storage registration never starts background execution. `AddMessagingTopology()` optionally ensures or validates declared producer destinations at host startup. -### Subscriber Error Handling +The [quickstart sample](https://github.com/FoundatioFx/Foundatio/tree/feat/messaging-jobs/samples/Foundatio.QuickstartSample) is a complete, executable host without external services. The [messaging sample](https://github.com/FoundatioFx/Foundatio/tree/feat/messaging-jobs/samples/Foundatio.MessagingSample) uses SQS/SNS and Redis jobs. -**Important:** Subscriber errors do NOT propagate to the publisher. This behavior is consistent across ALL implementations, including `InMemoryMessageBus`. This design ensures: +## Subscription identity -1. **Consistent behavior** - Code that works with `InMemoryMessageBus` in tests will behave the same with distributed buses in production -2. **Matches distributed reality** - In distributed systems, publishers and subscribers run in separate processes; publisher cannot see subscriber errors -3. **Predictable error handling** - Subscribers are responsible for handling their own errors - -```csharp -// Publisher - will NOT see subscriber errors -await messageBus.PublishAsync(new OrderCreated { OrderId = 123 }); -// Returns successfully even if a subscriber throws - -// Subscriber - handle your own errors -await messageBus.SubscribeAsync(async order => -{ - try - { - await ProcessOrderAsync(order); - } - catch (Exception ex) - { - _logger.LogError(ex, "Failed to process order {OrderId}", order.OrderId); - // Optionally publish failure event, retry, etc. - } -}); -``` - -### Provider Behavior Summary - -| Provider | Publish Errors | Subscriber Errors | Redelivery on Failure | -|----------|---------------|-------------------|----------------------| -| **InMemoryMessageBus** | Throws `MessageBusException` | Logged, swallowed | No | -| **AzureServiceBusMessageBus** | Throws `MessageBusException` | Logged, SDK handles | Yes (`MaxDeliveryCount`) | -| **KafkaMessageBus** | Fire-and-forget with callback | Logged, offset not committed | Yes (redelivered) | -| **RabbitMQMessageBus** | Throws `MessageBusException` | Logged, nack/requeue | Yes (`DeliveryLimit`) | -| **RedisMessageBus** | Throws `MessageBusException` | Logged, swallowed | No (pub/sub has no ack) | -| **SQSMessageBus** | Throws `MessageBusException` | Logged, message not deleted | Yes (redelivered) | - -::: tip Logging -All errors are logged at `Error` level. Subscriber errors are logged exactly once by the base class. -::: - -### In Subscribers - -```csharp -await messageBus.SubscribeAsync(async order => -{ - try - { - await ProcessOrderAsync(order); - } - catch (Exception ex) - { - _logger.LogError(ex, "Failed to process order {OrderId}", order.OrderId); +A named subscription such as `"billing"` is durable. All replicas using that name compete for the subscription's events. A separate `"analytics"` subscription gets its own copy. Names are deployment contracts: keep them stable across restarts and class renames. Register the subscription before publishing events that it must receive; creating one does not replay earlier publications. - // Optionally publish failure event - await _messageBus.PublishAsync(new OrderProcessingFailed - { - OrderId = order.OrderId, - Error = ex.Message - }); - } -}); -``` +`AddSubscriber()` defaults to `UseServiceName(...)`, then `IHostEnvironment.ApplicationName`. This gives one copy per service, with replicas competing. Configure a stable service name for deployed contracts; outside a host, supply it or an explicit subscription name. `AddSubscriber("name")` overrides the default and rejects blank names. Use `.Messaging.AddTemporarySubscriber()` when each running instance needs its own temporary subscription. -### With Retry +For dynamic `SubscribeAsync`, an unnamed subscription is temporary and receives its own copy while its listener is alive. In-memory and Redis transports support renewable two-minute subscription leases. Disposal deletes the subscription, and loss of renewal expires it after a crash. Transient renewal errors retry within the lease; a lost lease stops the old receiver and recreates its subscription. `IMessageSubscription.Status`, `RecoveryVersion`, and `WaitUntilReadyAsync` expose recovery. Consumers maintaining derived state must resynchronize after a possible gap. Redis physically removes expired groups during subsequent stream operations. AWS requires a named durable subscription because SQS/SNS does not provide this expiration contract; unnamed subscriptions fail explicitly. ```csharp -await messageBus.SubscribeAsync(async order => -{ - await _resiliencePolicy.ExecuteAsync(async ct => - { - await ProcessOrderAsync(order, ct); - }); -}); -``` - -## Cancellation Token Behavior - -Understanding how cancellation tokens are handled internally is important for building reliable publishers and subscribers. - -### Resource Creation Uses Disposal Token - -When you call `PublishAsync` or `SubscribeAsync`, the message bus may need to create infrastructure (e.g., Azure Service Bus topics, RabbitMQ exchanges, SQS topics). These setup operations use an internal disposal token — **not** the caller's cancellation token. This means: - -- **Topic and subscription creation only abort when the message bus is disposed**, never because a single caller cancelled their operation. -- A cancelled publish will not leave topic infrastructure in a half-created state. -- Multiple concurrent publishers/subscribers cannot interfere with each other's setup. - -### Linked Cancellation for Publish - -The caller's cancellation token is combined with the disposal token into a linked token for the actual publish operation. This means: - -- Publish cancels when **either** the caller cancels **or** the message bus is disposed. -- Graceful shutdown via `Dispose()` cancels all in-flight publishes promptly. +await using var consumer = await bus.ConsumeAsync( + (context, token) => receipts.SendAsync(context.Message.OrderId, token)); -```csharp -// Topic creation always completes (unless disposed), even if the publish is cancelled -using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5)); -await messageBus.PublishAsync(new OrderCreated { OrderId = 123 }, cancellationToken: cts.Token); +await using var subscriber = await bus.SubscribeAsync( + (context, token) => billing.RecordAsync(context.Message, token), + new MessageSubscriptionOptions { Subscription = "billing" }); ``` -### For Implementation Authors - -If you are writing a custom `IMessageBus` implementation by extending `MessageBusBase`: - -- **`EnsureTopicCreatedAsync`** always receives `DisposedCancellationToken`. Use it for all setup operations (lock acquisition, API calls, etc.). -- **`EnsureTopicSubscriptionAsync`** always receives `DisposedCancellationToken`. Use it for subscription infrastructure setup. -- **`PublishImplAsync`** receives a linked token (caller + disposal). Respect it for the actual message send. +Concurrency belongs to a receiving endpoint and defaults to one. Set `MaxConcurrency` on consumer/subscription options when handlers may run concurrently. One bus rejects duplicate handlers for the same message type on the same endpoint; use separate named subscriptions when multiple event handlers each need a copy. Handlers sharing an endpoint must agree on concurrency; retry overrides belong to each registered handler. An endpoint permits at most one interface/raw fallback, avoiding ambiguous dispatch. -## Best Practices +## Receive and settle directly -### 1. Use Immutable Messages +A hosted handler is optional. Direct receive returns a disposable delivery: ```csharp -// ✅ Good: Immutable record -public record OrderCreated -{ - public int OrderId { get; init; } - public required string CustomerId { get; init; } -} - -// ❌ Bad: Mutable class -public class OrderCreated +await using var delivery = await bus.ReceiveAsync(); +if (delivery is not null) { - public int OrderId { get; set; } - public string CustomerId { get; set; } + await receipts.SendAsync(delivery.Message.OrderId, delivery.CancellationToken); + await delivery.CompleteAsync(); } ``` -### 2. Include Timestamp and Correlation +Disposing an unsettled delivery returns it for redelivery. Raw receive takes `MessageReceiveOptions` with an explicit destination. Lease renewal runs while a delivery is active. Losing the lease cancels its work; disposal, shutdown, or cancellation does not count as successful completion. -```csharp -public record OrderCreated -{ - public int OrderId { get; init; } - public DateTime OccurredAt { get; init; } = DateTime.UtcNow; - public string CorrelationId { get; init; } = Activity.Current?.Id; -} -``` +Manual acknowledgement is available through `AckMode.Manual`. The endpoint retains its concurrency slot until settlement. Use automatic acknowledgement for ordinary handlers. -### 3. Handle Idempotency +## Delivery, identity, and serialization -```csharp -await messageBus.SubscribeAsync(async order => -{ - // Check if already processed - if (await _processedEvents.ContainsAsync(order.EventId)) - { - _logger.LogDebug("Already processed {EventId}", order.EventId); - return; - } +Durable delivery is **at least once**. A worker can finish a business operation and crash before acknowledgement. Lease renewal reduces concurrent execution but cannot guarantee exactly-once side effects. In-memory state is lost when the process stops. An interrupted broker receive can leave a message invisible until its lease expires (the bus requests a one-minute receive lease), including when a client cancels an SQS long poll before receiving its response. Allow for that redelivery delay during worker replacement. - await ProcessOrderAsync(order); - await _processedEvents.AddAsync(order.EventId); -}); -``` +The application `MessageId`, broker entry ID, and per-delivery receipt are distinct. Send/publish return the application ID. Supply `MessageSendOptions.MessageId` or `MessagePublishOptions.MessageId` for retry correlation and consumer deduplication; supplying an ID does not make broker sends idempotent. Scheduling, retries, and dead-lettering preserve that ID. -### 4. Use Specific Message Types +Batch sends are not transactions. On failure, `MessageSendException.Outcomes` describes each input as accepted, rejected, unknown, or not attempted, with its original index, application ID, provider error and retryability when known. An unknown outcome may already have reached the broker. Retrying requires an application deduplication strategy. Preserve individual IDs with the batch-item overload: ```csharp -// ✅ Good: Specific, intentional messages -public record OrderCreated { ... } -public record OrderShipped { ... } -public record OrderCancelled { ... } - -// ❌ Bad: Generic, multi-purpose messages -public record OrderEvent { public string Action { get; set; } } +await bus.SendBatchAsync([ + new MessageBatchItem(new SendReceipt(1001), "receipt-1001"), + new MessageBatchItem(new SendReceipt(1002), "receipt-1002") +]); ``` -### 5. Keep Messages Small +AWS uses native batches of up to ten, respecting encoded payload/attribute limits and retaining mixed per-entry outcomes. Redis pipelines bounded batches (64 by default, configurable up to 256). Durable retry and dead-letter source records are removed only after verified acceptance. -Messages should contain identifiers and essential data only, not full entity payloads. +Concurrent AWS sends, publishes and acknowledgements are automatically combined into native batches. Applications keep using the ordinary single-message methods, and completion still waits for the broker's per-entry response. Automatic batching uses separate send and acknowledgement buffers per destination: 100 buffered messages and four concurrent requests by default; additional callers await capacity. Partial batches collect for up to two milliseconds (subject to timer scheduling); an idle SQS sender can dispatch immediately, and a stream of singleton batches skips repeated collection delays while no request is active. `AwsMessageTransportOptions` exposes `EnableBatching`, `BatchDelay`, `MaxPendingBatchMessages`, `MaxConcurrentBatches` and `BatchTimeout` (30 seconds) for explicit tuning. Acknowledgement collection also learns the requested receive capacity, so a consumer with fewer than ten slots can flush a complete batch immediately; queued receipts can still fill all ten native slots. Disabling automatic batching leaves explicit batch sends available. -```csharp -// ✅ Good: Just identifiers -public record OrderCreated -{ - public int OrderId { get; init; } -} - -// ❌ Bad: Full entity in message -public record OrderCreated -{ - public Order FullOrderWithAllDetails { get; init; } -} -``` +Canceling a caller does not cancel other messages sharing its AWS request. The collector skips canceled buffered operations; cancellation racing dispatch can leave an unknown send outcome. Disposal drains admitted operations and cancels unfinished requests at the batch timeout. Missing or failed delete results never count as acknowledgements. The AWS receiver caps each pull at ten and overlaps up to four receive requests when consumer capacity permits. Receives share one slot budget and collect freed slots together, starting immediately when a batch fills. `MaxConcurrency` remains a strict bound on unacknowledged deliveries; a slow handler does not block unrelated slots. Completed handlers release their slots while bounded cancellation cleanup finishes. Shutdown drains both handlers and cleanup. -## Message Size Limits +AWS stores readable JSON/text payloads directly in the body and binary payloads as base64. A versioned `fnd.envelope` attribute carries the encoding, application ID, content type and headers. Headers are duplicated as native attributes only when selected in `AwsMessageTransportOptions.NativeMessageHeaders`, for SNS filters or external consumers. Select any required application headers (up to nine); the default empty list keeps the wire representation compact. For example, `NativeMessageHeaders = [KnownHeaders.MessageType, "tenant.id"]` exposes the type and tenant for SNS attribute filtering. Invalid or reserved attribute names fail during transport construction. The receiver accepts the earlier separate-attribute encoding, but earlier experimental receivers cannot read this new format. Upgrade producers and consumers together or use a new resource prefix. Existing experimental SNS attribute filters must explicitly select their header names; all consumer headers remain available without this option. These changes are confined to the unreleased provider. -Different message bus implementations have different size limits. Understanding these limits is essential for reliable messaging. - -| Provider | Max Message Size | Notes | -|----------|------------------|-------| -| InMemoryMessageBus | Limited by available memory | No practical limit | -| AzureServiceBusMessageBus | 256 KB (Standard) / 100 MB (Premium) | Use claim check for large payloads | -| KafkaMessageBus | 1 MB (default) | Configurable via `message.max.bytes` | -| RabbitMQMessageBus | 128 MB (default) | Configurable, but keep small | -| RedisMessageBus | 512 MB (Redis limit) | Recommended: < 1 MB for performance | -| SQSMessageBus | 256 KB | Use claim check for large payloads | - -### Claim Check Pattern for Large Payloads - -For large data, store it externally and pass a reference (also known as the Claim Check Pattern): +For long-lived contracts, register versioned wire names on producers and consumers: ```csharp -// Instead of embedding large data -public record DocumentProcessed -{ - public string DocumentId { get; init; } - public string BlobPath { get; init; } // Reference to storage - public long SizeBytes { get; init; } -} - -// Subscriber retrieves from storage -await messageBus.SubscribeAsync(async msg => -{ - var document = await _fileStorage.GetObjectAsync(msg.BlobPath); - await ProcessDocumentAsync(document); -}); +builder.Services.AddFoundatio().Messaging + .AddMessageType("order-placed.v1", topic: "orders"); ``` -## Notification Patterns - -### Real-Time Notifications with SignalR +Bind the stable wire name and producer route together with `AddMessageType(name, queue: ..., topic: ...)`, or set `MessageTypeName` plus `Destination`/`Topic` in a handler registration. Startup topology checks validate wire-name collisions even in `TopologyMode.None`; `MessageRoutingOptions.GetRouteMaps()` and startup logs expose declared mappings. Configure stable queue/topic routes independently of CLR class names. Concrete handlers may use the default CLR full-name discriminator; interface, abstract, and `object` receivers accept only explicitly registered concrete types. Sends and batches preserve the runtime payload type in the wire header, even when a variable is declared as an interface; the declared type still selects the route. The runtime does not scan assemblies or activate a type named by an untrusted header. Producers and consumers must agree on serialization and schema evolution. JSON uses `application/json`; other serializers default to byte-safe `application/octet-stream` unless configured otherwise. -```csharp -public class NotificationService : IHostedService -{ - private readonly IMessageBus _messageBus; - private readonly IHubContext _hubContext; +When updating a business database and publishing must commit together, persist an outbox record in the same database transaction and publish from an outbox dispatcher. Foundatio does not coordinate that transaction. Consumers should commit their deduplication record with their business changes. Scheduled dispatch send/delete and retry park/ack are also at-least-once boundaries. - public NotificationService(IMessageBus messageBus, IHubContext hubContext) - { - _messageBus = messageBus; - _hubContext = hubContext; - } +## Delays and failures - public async Task StartAsync(CancellationToken ct) - { - // Bridge message bus to SignalR - await _messageBus.SubscribeAsync(async (msg, ct) => - { - await _hubContext.Clients - .User(msg.UserId) - .SendAsync("Notification", msg.Title, msg.Body, ct); - }, ct); - - // Broadcast to all users - await _messageBus.SubscribeAsync(async (msg, ct) => - { - await _hubContext.Clients.All - .SendAsync("Announcement", msg.Message, ct); - }, ct); - } - - public Task StopAsync(CancellationToken ct) => Task.CompletedTask; -} -``` - -### Delayed Notifications +Native delays are used when the destination supports them. `Messaging.UseInMemory()` and `Messaging.UseRedis()` automatically supply a matching `IScheduledDispatchStore` without registering job execution. The automatic Redis store shares the transport connection, clock and key prefix; `RedisStreamsMessageTransportOptions.Scheduling` sets its limits. AWS requires an explicit durable store for delays beyond native support; configure `Messaging.UseSchedulingStore(...)` or share a job runtime store. `AddFoundatioWorker` starts its dispatcher when both a transport and dispatch store are registered; split deployments can call `AddScheduledMessageDispatcher()` directly. Messaging depends only on that store contract; a job worker is not required. `IJobRuntimeStore` also implements the dispatch store, so a configured job store can be shared. ```csharp -// Schedule a reminder -await messageBus.PublishAsync(new ReminderNotification -{ - UserId = "user-123", - Message = "Don't forget to complete your order!" -}, new MessageOptions -{ - DeliveryDelay = TimeSpan.FromHours(24) -}); +builder.Services.AddFoundatioWorker(foundatio => foundatio + .ConfigureMessaging(messaging => messaging.UseInMemory())); ``` -### Fan-Out Pattern - -Publish once, process in multiple ways: - -```csharp -// Single publish -await messageBus.PublishAsync(new OrderCreated { OrderId = 123 }); +For production durability use a durable dispatch store, such as Redis. Without a suitable native delay or dispatch store, unsupported delays fail instead of being shortened. The scheduled message dispatcher runs independently of job execution. -// Multiple subscribers handle different concerns -await messageBus.SubscribeAsync(async order => -{ - await _emailService.SendConfirmationAsync(order.OrderId); -}); +Native dead-letter transports expose `ISupportsDeadLetter`: `PeekDeadLetteredAsync` reads a bounded page without removing evidence; `DeleteDeadLetteredAsync` removes an explicit ID; `ReplayDeadLetteredAsync` sends that ID to an explicit queue/topic and resets retry metadata while preserving the application ID. Peeking repeatedly is safe. Replaying can repeat business effects, so apply the same idempotency rules as normal delivery. -await messageBus.SubscribeAsync(async order => -{ - await _inventoryService.ReserveAsync(order.OrderId); -}); +Unmatched message types retry after five seconds with jitter (50 attempts by default), allowing rolling deployments without a tight redelivery loop. Ordinary handler failures use five attempts with immediate-first, then 10/20/30-second jittered delays. Override these through `ConfigureRetry`. Malformed AWS envelope entries retain raw evidence and are quarantined independently, allowing valid entries in the batch to proceed. -await messageBus.SubscribeAsync(async order => -{ - await _analyticsService.TrackAsync("order_created", order.OrderId); -}); -``` +If native dead-lettering is unavailable, the core sends to a fallback queue and only completes the original after that send succeeds. Failure to park the message leaves the original recoverable. AWS fallback queues support ordinary receive/settle operations, not non-destructive peek by ID. -## Resource Management +## Topology -### Disposal Lifecycle +`TopologyMode.Ensure` creates destinations on first use. Successful permanent provisioning is cached briefly; errors invalidate it, and deleted receive destinations are recreated under listener supervision. Expiring declarations are never cached. `Validate` checks existing destinations and fails if missing; it does not create. `None` assumes out-of-band provisioning. This policy applies to sends, publishes, receiving, delayed dispatch, and fallback dead-letter sends. Temporary subscriptions require `Ensure`. -Message buses implement both `IDisposable` and `IAsyncDisposable`. Prefer `await using` (or `DisposeAsync()`) for clean shutdown: +Producer routing declares queues/topics, never phantom subscriber groups. AWS resource existence and deletion work through a fresh transport instance, including SNS bindings. Provider administration should use `ISupportsProvisioning` explicitly. -```csharp -// Preferred: async disposal -await using var messageBus = new InMemoryMessageBus(); -await messageBus.SubscribeAsync(async e => { /* ... */ }); -// DisposeAsync is called when scope ends +## Provider guarantees -// DI container manages lifetime automatically -services.AddSingleton(); -``` +| Behavior | In-memory | Redis Streams | AWS SQS/SNS | +| --- | --- | --- | --- | +| Queued work and named event subscriptions | Yes, process-local | Yes | Yes | +| Temporary expiring subscriptions | Yes, supervised recovery | Yes, supervised recovery | Unsupported; name the subscription | +| Hybrid-cache invalidation | Resynchronizes after listener gaps | Resynchronizes after listener gaps | Fails immediately: temporary subscriptions required | +| Cache-backed lock notifications | Notifications plus polling | Notifications plus polling | Polling fallback | +| Delayed-message persistence | Automatic, process-local | Automatic, Redis | Explicit durable store for non-native delays | +| Execution durability after process loss | No | Depends on Redis persistence/HA | Broker-managed | +| Delivery order | Initial FIFO; priority/retries can reorder | Initial FIFO; retries/concurrency can reorder | Standard queues, no ordering guarantee | +| Native delayed queue send | No | No | Up to 15 minutes | +| Native delayed publish | No | No | No | +| Lease renewal | Yes | Atomic receipt fencing | SQS visibility; stale-receipt detection is best effort | +| Non-destructive DLQ peek/replay by ID | Yes | Yes, per subscription | No; core fallback queue | +| Backlog limit | Process memory | `MaxPendingMessages`, default 100,000 per stream/DLQ | Broker limits | -Disposal follows a **two-phase** sequence to prevent message loss in durable providers: +Redis capacity rejects new messages instead of trimming unread or pending entries. A slow durable subscription therefore applies backpressure to the topic. Acknowledged topic entries are trimmed only when every subscription has progressed past them. Retention checks are amortized to a one-second cadence and forced before capacity rejection. Empty Redis receivers back off from 25 ms to one second; tune `PollInterval` and `MaxIdlePollInterval` when idle latency matters. Delete abandoned durable subscriptions deliberately; temporary leases are not a replacement for durable subscription administration. -1. **Graceful drain** — In-flight handlers finish executing while subscribers and the internal cancellation token are still active. Providers that support processor-level draining (e.g., Azure Service Bus `StopProcessingAsync`) execute it here via `ShutdownAsync`. -2. **Teardown** — The internal cancellation token is cancelled, all subscribers are cleared, and transport infrastructure (connections, channels, clients) is closed and disposed via `CleanupAsync`. +SQS/SNS support varies by destination role. Do not infer topic capabilities from queue capabilities. The shared conformance suite exercises in-memory, Redis, and SQS/SNS via LocalStack in CI; the emulator is not evidence of a live AWS deployment. -> **Note:** The base `MessageBusBase` implementation does not guarantee that all active subscriber callbacks have completed before `DisposeAsync` returns. Provider-specific draining behavior (such as Azure Service Bus `StopProcessingAsync`) is implemented in provider overrides of `ShutdownAsync`. +## Migration -### Message Durability During Shutdown +The former publish-only interfaces live under `Foundatio.Messaging.Legacy`; `Messaging.AddLegacyAdapter()` adapts them to this bus. Legacy `IQueue` workers migrate to an explicit consumer and `SendAsync`. There is no `Deliveries.Both`, handler-name-derived subscriber identity, or `PerInstance` flag. Choose `AddConsumer` or `AddSubscriber`, and use a stable service identity or explicit names for durable subscribers. -What happens to messages that arrive while the bus is disposing depends on the provider: +## Broker-driven execution tracking -| Provider | In-Flight Messages | Arriving During Dispose | After Dispose | -|---|---|---|---| -| **InMemoryMessageBus** | Completed normally | Dropped (no persistence) | Lost | -| **AzureServiceBusMessageBus** | Completed; abandoned if bus disposes mid-handler (PeekLock) | Remain in topic for other subscribers | Persisted in Azure | -| **KafkaMessageBus** | Completed; offset not committed if bus disposes mid-handler | Remain in partition (uncommitted offset) | Persisted in Kafka | -| **RabbitMQMessageBus** | Completed; requeued if bus disposes mid-handler | Remain in queue | Persisted in RabbitMQ | -| **RedisMessageBus** | Completed normally | Dropped (pub/sub has no persistence) | Lost | -| **SQSMessageBus** | Completed; message not deleted if bus disposes mid-handler | Remain in SQS queue | Persisted in SQS | +For queues that need progress, cancellation, and operational history, configure `.Jobs.UseInMemory()` or `.Jobs.UseRedis()`. Tracked deliveries use the existing `IJobRuntimeStore`, `JobState`, and `IJobMonitor`. Set `ExecutionOwner = JobExecutionOwner.Broker` when recording enqueue acceptance: the ordinary job worker cannot claim or recover these records. The message bus owns delivery leases and settlement, while each delivery attempt gets a fresh token for atomic progress and completion updates. Registering the store starts no job worker or scheduler. -::: tip Fire-and-Forget Providers -`InMemoryMessageBus` and `RedisMessageBus` use fire-and-forget delivery. Messages that arrive when no subscriber is listening are permanently lost. This is by design — if you need guaranteed delivery, use a durable provider or `IQueue`. -::: +`MessageExecutionPipeline` runs a raw delivery callback with `MessageProcessingContext`, applies returned `MessageOutcome` values, and persists only confirmed settlement. Pass it to a manual-ack raw `ConsumeAsync` endpoint with `WaitForManualSettlement = false`. The producer creates a unique execution state before sending and puts its ID in `ExecutionHeaders.ExecutionId`. Integrations such as Foundatio.Mediator perform this composition automatically. -### Writing Custom Providers +Processing claims and subsequent progress/status writes are fenced by the broker attempt number. A terminal execution is not started again. Expired history does not block valid broker work; progress/history updates then become no-ops. Tracking is not persistent deduplication or exactly-once delivery. Broker acceptance and persistence are separate operations; uncertain sends and acknowledgments remain nonterminal. Application side effects must tolerate retries. -If you are extending `MessageBusBase` with a custom provider, override the lifecycle hooks: +Endpoint options own `MaxConcurrency`, `PrefetchCount`, `VisibilityTimeout`, `AutoRenewLock`, and `ShutdownTimeout`. Manual and automatic renewal share one conservative delivery deadline. Lost ownership cancels processing. Stopping receiving allows admitted handlers to drain within the configured shutdown timeout. -- **`ShutdownAsync()`** — Called *before* the cancellation token is cancelled and *before* subscribers are cleared. Use this to gracefully drain your transport (e.g., stop a processor, close a consumer group). Subscribers are still active and can finish processing. -- **`CleanupAsync()`** — Called *after* the cancellation token is cancelled and *after* subscribers are cleared. Use this to tear down transport infrastructure (close connections, dispose clients, await background tasks). +`ConsumeWithOutcomeAsync` accepts `MessageOutcome.Success`, `Retry`, `DeadLetter`, or `Unsettled` without requiring expected application failures to throw exceptions. -```csharp -public class MyMessageBus : MessageBusBase -{ - // Phase 1: Stop accepting new messages, drain in-flight work. - // When the body is a single call, return the Task directly (no extra async state machine). - protected override Task ShutdownAsync() => _processor.StopAsync(); +## Per-node broadcasts - protected override async Task CleanupAsync() - { - // Phase 2: Close connections, dispose clients — ConfigureAwait(false) in library overrides - await _connection.CloseAsync().ConfigureAwait(false); - _client.Dispose(); - } -} -``` +`SubscribeNodeAsync` receives an independent best-effort copy on each running node, acknowledging before callbacks. Memory and Redis use native expiring subscriptions. AWS creates a tagged SQS queue and SNS subscription, heartbeats ownership, cleans up on disposal, and reaps stale resources when another node starts. AWS managed resources do not advertise native TTL expiration. Configure IAM for the required resource lifecycle, tagging, and discovery operations, including when durable topology is externally provisioned. -If `ShutdownAsync` needs multiple steps, use `async`/`await` and apply `.ConfigureAwait(false)` to each await (within Foundatio provider projects, the same pattern uses the internal `AnyContext()` helper). +Use this for invalidation and UI refresh signals. A disconnected, paused, or restarting node may miss events and should refresh authoritative state. Use durable service subscriptions for work that needs replay and retries. -## Next Steps +## Operational recovery -- [Queues](./queues) - For guaranteed delivery with acknowledgment -- [Caching](./caching) - Cache invalidation with messaging -- [Jobs](./jobs) - Background processing triggered by messages +`MessageAdministration` supplies statistics, bounded dead-letter inspection, deletion, and replay through native capabilities. Providers without native inspection use bounded receive/hold/release. Scans stop at 1,000 messages or ten seconds; their cleanup has an independent timeout. A successful replacement send precedes deleting the original. That fallback is not transactional and an uncertain send can produce a duplicate. Callers may prepare fresh execution metadata for an operator replay while retaining the failed history. diff --git a/docs/guide/nullable-reference-types.md b/docs/guide/nullable-reference-types.md index 33f3c7577..fcb3aab10 100644 --- a/docs/guide/nullable-reference-types.md +++ b/docs/guide/nullable-reference-types.md @@ -1,5 +1,8 @@ # Nullable Reference Types (NRT) Migration +This migration reference includes earlier queue/job interfaces removed by the unreleased redesign. Current receive and job contracts are documented in [Messaging](messaging.md) and [Durable jobs](jobs.md). + + Foundatio has been fully annotated with C# [nullable reference types](https://learn.microsoft.com/en-us/dotnet/csharp/nullable-references) across the core library and all provider repositories. This document describes the public API changes, design decisions, and remaining areas for improvement. ## Interface Return Type Changes diff --git a/docs/guide/provider-behavioral-gaps.md b/docs/guide/provider-behavioral-gaps.md index f7a3e50c2..bb98c7713 100644 --- a/docs/guide/provider-behavioral-gaps.md +++ b/docs/guide/provider-behavioral-gaps.md @@ -8,7 +8,7 @@ This document catalogs known behavioral differences across Foundatio provider im |-----------|----------|-------|-------|-----|----------|-------|-------|--------|--------| | `IFileStorage` | Full | Full | Full | Partial | — | — | Partial | Full | Full | | `IQueue` | Full | Partial | Partial | Partial | — | — | — | — | — | -| `IMessageBus` | Full | Full | Full | Partial | Full | Full | — | — | — | +| Legacy pub/sub bus | Full | Full | Full | Partial | Full | Full | — | — | — | | `ICacheClient` | Full | Full | — | — | — | — | — | — | — | | `ILockProvider` | Full | Full | — | — | — | — | — | — | — | @@ -106,7 +106,9 @@ This document catalogs known behavioral differences across Foundatio provider im --- -## IMessageBus +## Legacy pub/sub providers + +The tables below describe the former publish-only provider APIs. For current transport contracts, see the [messaging provider matrix](messaging.md#provider-guarantees). These older provider packages do not implement the new transport SPI. ### Delayed Message Delivery diff --git a/docs/guide/queues.md b/docs/guide/queues.md index 5a4bcc5b8..b079c93e9 100644 --- a/docs/guide/queues.md +++ b/docs/guide/queues.md @@ -1,980 +1,30 @@ -# Queues +# Worker queues -Queues offer First In, First Out (FIFO) message delivery with reliable processing semantics. Foundatio provides multiple queue implementations through the `IQueue` interface. - -## The IQueue Interface - -[View source](https://github.com/FoundatioFx/Foundatio/blob/main/src/Foundatio/Queues/IQueue.cs) - -```csharp -public interface IQueue : IQueue where T : class -{ - AsyncEvent> Enqueuing { get; } - AsyncEvent> Enqueued { get; } - AsyncEvent> Dequeued { get; } - AsyncEvent> LockRenewed { get; } - AsyncEvent> Completed { get; } - AsyncEvent> Abandoned { get; } - AsyncEvent> QueueDeleted { get; } - - void AttachBehavior(IQueueBehavior behavior); - Task EnqueueAsync(T data, QueueEntryOptions? options = null); - Task?> DequeueAsync(CancellationToken cancellationToken); - Task?> DequeueAsync(TimeSpan? timeout = null); - Task RenewLockAsync(IQueueEntry queueEntry); - Task CompleteAsync(IQueueEntry queueEntry); - Task AbandonAsync(IQueueEntry queueEntry); - Task> GetDeadletterItemsAsync(CancellationToken cancellationToken = default); - Task StartWorkingAsync(Func, CancellationToken, Task> handler, - bool autoComplete = false, - CancellationToken cancellationToken = default); -} - -public interface IQueue : IHaveSerializer, IDisposable -{ - Task GetQueueStatsAsync(); - Task DeleteQueueAsync(); - string QueueId { get; } -} -``` - -## Implementations - -### InMemoryQueue - -An in-memory queue implementation for development and testing: - -[View source](https://github.com/FoundatioFx/Foundatio/blob/main/src/Foundatio/Queues/InMemoryQueue.cs) - -```csharp -using Foundatio.Queues; - -var queue = new InMemoryQueue(); - -// Enqueue work -await queue.EnqueueAsync(new WorkItem { Id = 1, Data = "Hello" }); - -// Dequeue and process -var entry = await queue.DequeueAsync(); -if (entry != null) -{ - Console.WriteLine($"Processing: {entry.Value.Data}"); - await entry.CompleteAsync(); -} -``` - -### RedisQueue - -Distributed queue using Redis (separate package): - -[View source](https://github.com/FoundatioFx/Foundatio.Redis/blob/main/src/Foundatio.Redis/Queues/RedisQueue.cs) - -```csharp -// dotnet add package Foundatio.Redis - -using Foundatio.Redis.Queues; - -var queue = new RedisQueue(o => { - o.ConnectionMultiplexer = redis; - o.Name = "work-items"; - o.WorkItemTimeout = TimeSpan.FromMinutes(5); -}); -``` - -### AzureServiceBusQueue - -Queue using Azure Service Bus (separate package): - -[View source](https://github.com/FoundatioFx/Foundatio.AzureServiceBus/blob/main/src/Foundatio.AzureServiceBus/Queues/AzureServiceBusQueue.cs) - -```csharp -// dotnet add package Foundatio.AzureServiceBus - -using Foundatio.AzureServiceBus.Queues; - -var queue = new AzureServiceBusQueue(o => { - o.ConnectionString = "..."; - o.Name = "work-items"; -}); -``` - -### AzureStorageQueue - -Queue using Azure Storage Queues (separate package): - -[View source](https://github.com/FoundatioFx/Foundatio.AzureStorage/blob/main/src/Foundatio.AzureStorage/Queues/AzureStorageQueue.cs) - -```csharp -// dotnet add package Foundatio.AzureStorage - -using Foundatio.AzureStorage.Queues; - -var queue = new AzureStorageQueue(o => { - o.ConnectionString = "..."; - o.Name = "work-items"; -}); -``` - -### SQSQueue - -Queue using AWS SQS (separate package): - -[View source](https://github.com/FoundatioFx/Foundatio.AWS/blob/main/src/Foundatio.AWS/Queues/SQSQueue.cs) - -```csharp -// dotnet add package Foundatio.AWS - -using Foundatio.AWS.Queues; - -var queue = new SQSQueue(o => { - o.Region = RegionEndpoint.USEast1; - o.QueueName = "work-items"; -}); -``` - -## Queue Entry Lifecycle - -Each dequeued message goes through a lifecycle: - -```txt - ┌─────────┐ - │ Queued │ - └────┬────┘ - │ - ▼ - ┌──────────────────┐ - │ Dequeued/Working │ - └────┬─────────────┘ - │ - ▼ - ┌──────────────┐ - │ Processing │ - └──┬────────┬──┘ - │ │ - Success│ │Failure - │ │ - ▼ ▼ - ┌────────┐ ┌───────────┐ - │Complete│ │ Abandoned │ - └────────┘ └─────┬─────┘ - │ - ▼ - ┌─────────┐ - │ Retry? │ - └──┬───┬──┘ - Yes │ │ No - │ │ - ▼ ▼ - ┌─────────┐ ┌──────────────┐ - │ Queued │ │ Dead Letter │ - └─────────┘ └──────────────┘ -``` - -### Completing Entries - -Mark an entry as successfully processed: - -```csharp -var entry = await queue.DequeueAsync(); -if (entry is null) - return; - -try -{ - await ProcessAsync(entry.Value); - await entry.CompleteAsync(); -} -catch -{ - await entry.AbandonAsync(); - throw; -} -``` - -### Abandoning Entries - -Return an entry to the queue for retry: - -```csharp -var entry = await queue.DequeueAsync(); -if (entry is null) - return; - -if (!CanProcess(entry.Value)) -{ - // Return to queue for later processing - await entry.AbandonAsync(); - return; -} -``` - -### Lock Renewal - -When processing takes longer than the `WorkItemTimeout`, the queue entry's lock may expire, causing another worker to pick up the same item. Use `RenewLockAsync` to extend the lock duration. - -**Why lock renewal matters:** - -- Prevents duplicate processing when work takes longer than expected -- Avoids entries being re-queued while still being processed -- Essential for variable-duration workloads - -::: tip Recommended Approach -Use `QueueJobBase` for queue processing (see [Jobs - Queue Processor Jobs](/guide/jobs#queue-processor-jobs)). For manual processing, call `RenewLockAsync()` periodically within your processing logic. -::: - -**Best practices for `WorkItemTimeout`:** - -- Set `WorkItemTimeout` to your typical processing time plus padding (e.g., 2x normal duration) -- Call `RenewLockAsync()` before the timeout expires if processing takes longer than expected -- Monitor your processing times to adjust the timeout appropriately - -#### Manual Renewal in Queue Jobs - -For long-running operations in a `QueueJobBase`, renew the lock during processing: - -```csharp -public class VideoProcessorJob : QueueJobBase -{ - private readonly IVideoService _videoService; - - public VideoProcessorJob(IQueue queue, IVideoService videoService) - : base(queue) => _videoService = videoService; - - protected override async Task ProcessQueueEntryAsync( - QueueEntryContext context) - { - var workItem = context.QueueEntry.Value; - var startTime = DateTime.UtcNow; - - try - { - // Start processing - await _videoService.StartProcessingAsync(workItem.VideoId); - - // Renew lock if processing is taking longer than expected - if (DateTime.UtcNow - startTime > TimeSpan.FromMinutes(3)) - { - await context.QueueEntry.RenewLockAsync(); - } - - await _videoService.CompleteProcessingAsync(workItem.VideoId); - return JobResult.Success; - } - catch (Exception ex) - { - return JobResult.FromException(ex); - } - } -} -``` - -::: warning Manual Lock Renewal -Most processing should complete within the `WorkItemTimeout`. If you regularly need lock renewal, increase the `WorkItemTimeout` instead. Manual renewal should only be used for truly variable-duration workloads where you cannot predict processing time accurately. -::: - -#### Ensuring Single Processing with GetQueueEntryLockAsync - -Override `GetQueueEntryLockAsync` to acquire a distributed lock based on a unique value from the work item. This guarantees that even if the same item is enqueued multiple times (e.g., due to retries or system failures), only one instance will process it at a time. - -**When to use this:** - -- Processing must be guaranteed to occur only once per unique identifier -- Work items can be re-queued due to failures, but duplicate processing would cause issues -- You need to lock on a business key (e.g., user ID, order ID) rather than the queue entry ID - -```csharp -public class OrderProcessorJob : QueueJobBase -{ - private readonly ILockProvider _lockProvider; - private readonly IOrderService _orderService; - - public OrderProcessorJob( - IQueue queue, - ILockProvider lockProvider, - IOrderService orderService) : base(queue) - { - _lockProvider = lockProvider; - _orderService = orderService; - } - - // Override to lock on the order ID instead of the queue entry ID - protected override Task GetQueueEntryLockAsync( - IQueueEntry queueEntry, - CancellationToken cancellationToken = default) - { - // Lock on the business key (order ID) to prevent concurrent processing - // of the same order across all queue entries - string lockKey = $"order:{queueEntry.Value.OrderId}"; - return _lockProvider.TryAcquireAsync(lockKey, TimeSpan.FromMinutes(5), cancellationToken); - } - - protected override async Task ProcessQueueEntryAsync( - QueueEntryContext context) - { - // This will only execute if we successfully acquired the lock - // Multiple queue entries for the same order will be serialized - var orderId = context.QueueEntry.Value.OrderId; - - await _orderService.ProcessAsync(orderId, context.CancellationToken); - return JobResult.Success; - } -} -``` - -**How it works:** - -1. When `QueueJobBase` dequeues an entry, it calls `GetQueueEntryLockAsync` before processing -2. If the lock cannot be acquired (returns `null`), the entry is abandoned and returned to the queue -3. If the lock acquisition throws an exception (e.g., network failure), the entry is abandoned and a `JobResult.FromException` is returned -4. If the lock is acquired, processing continues and the lock is automatically released after completion -5. The lock is also used for manual renewal within `ProcessQueueEntryAsync` via `await context.QueueEntry.RenewLockAsync()` - -::: tip Lock Provider Selection -Use a distributed lock provider (e.g., `CacheLockProvider` with Redis) in production to coordinate across multiple instances. For single-instance scenarios, `CacheLockProvider` with `InMemoryCacheClient` is sufficient. -::: - -## Processing Patterns - -::: tip Recommended Approach -For production applications, use `QueueJobBase` with `Foundatio.Extensions.Hosting` for reliable, automatic background processing. See [Jobs - Queue Processor Jobs](/guide/jobs#queue-processor-jobs) for details. The patterns below are for advanced scenarios or custom integrations. -::: - -### Simple Processing Loop - -```csharp -while (!cancellationToken.IsCancellationRequested) -{ - var entry = await queue.DequeueAsync(cancellationToken); - if (entry == null) - continue; - - try - { - await ProcessAsync(entry.Value); - await entry.CompleteAsync(); - } - catch (Exception ex) - { - _logger.LogError(ex, "Failed to process {Id}", entry.Value.Id); - await entry.AbandonAsync(); - } -} -``` - -### Using StartWorkingAsync - -Simplified background processing: - -```csharp -// Start processing in background -await queue.StartWorkingAsync( - async (entry, ct) => - { - await ProcessAsync(entry.Value); - }, - autoComplete: true, // Automatically complete on success - cancellationToken -); -``` - -## Queue Entry Options - -Configure enqueue behavior: - -```csharp -await queue.EnqueueAsync(new WorkItem { Id = 1 }, new QueueEntryOptions -{ - UniqueId = "unique-id", // Dedupe by ID - CorrelationId = "request-123", // For tracing - DeliveryDelay = TimeSpan.FromMinutes(5), // Delayed delivery - Properties = new Dictionary - { - ["priority"] = "high" - } -}); -``` - -## Queue Events - -Subscribe to queue lifecycle events: - -```csharp -var queue = new InMemoryQueue(); - -queue.Enqueuing.AddHandler((sender, args) => -{ - _logger.LogInformation("Enqueuing: {Data}", args.Data); - return Task.CompletedTask; -}); - -queue.Enqueued.AddHandler((sender, args) => -{ - _logger.LogInformation("Enqueued: {Id}", args.Entry.Id); - return Task.CompletedTask; -}); - -queue.Dequeued.AddHandler((sender, args) => -{ - _logger.LogInformation("Dequeued: {Id}", args.Entry.Id); - return Task.CompletedTask; -}); - -queue.Completed.AddHandler((sender, args) => -{ - _logger.LogInformation("Completed: {Id}", args.Entry.Id); - return Task.CompletedTask; -}); - -queue.Abandoned.AddHandler((sender, args) => -{ - _logger.LogWarning("Abandoned: {Id}", args.Entry.Id); - return Task.CompletedTask; -}); - -queue.QueueDeleted.AddHandler((sender, args) => -{ - _logger.LogInformation("Queue deleted"); - return Task.CompletedTask; -}); -``` - -## Queue Behaviors - -Extend queue functionality with behaviors. Behaviors hook into queue events to add cross-cutting concerns like logging, metrics, or deduplication. - -### Creating Custom Behaviors - -```csharp -public class LoggingQueueBehavior : QueueBehaviorBase where T : class -{ - private readonly ILogger _logger; - - public LoggingQueueBehavior(ILogger logger) => _logger = logger; - - protected override Task OnEnqueued(object sender, EnqueuedEventArgs args) - { - _logger.LogInformation("Enqueued {Id}", args.Entry.Id); - return Task.CompletedTask; - } - - protected override Task OnDequeued(object sender, DequeuedEventArgs args) - { - _logger.LogInformation("Dequeued {Id}", args.Entry.Id); - return Task.CompletedTask; - } - - protected override Task OnCompleted(object sender, CompletedEventArgs args) - { - _logger.LogInformation("Completed {Id} in {Duration}ms", - args.Entry.Id, args.Entry.ProcessingTime.TotalMilliseconds); - return Task.CompletedTask; - } - - protected override Task OnAbandoned(object sender, AbandonedEventArgs args) - { - _logger.LogWarning("Abandoned {Id}, attempt {Attempt}", - args.Entry.Id, args.Entry.Attempts); - return Task.CompletedTask; - } - - protected override Task OnQueueDeleted(object sender, QueueDeletedEventArgs args) - { - _logger.LogInformation("Queue deleted"); - return Task.CompletedTask; - } -} - -// Attach to queue -queue.AttachBehavior(new LoggingQueueBehavior(logger)); -``` - -### Built-in: Duplicate Detection Behavior - -Foundatio includes `DuplicateDetectionQueueBehavior` to prevent duplicate messages from being enqueued. This is useful for scenarios where the same work item might be submitted multiple times. - -```csharp -// Your message must implement IHaveUniqueIdentifier -public class OrderWorkItem : IHaveUniqueIdentifier -{ - public int OrderId { get; set; } - public string UniqueIdentifier => $"order:{OrderId}"; -} - -// Attach the behavior -var cache = new InMemoryCacheClient(); -queue.AttachBehavior(new DuplicateDetectionQueueBehavior( - cache, - loggerFactory, - detectionWindow: TimeSpan.FromMinutes(10) // How long to remember seen IDs -)); - -// Duplicates are automatically discarded -await queue.EnqueueAsync(new OrderWorkItem { OrderId = 123 }); // ✅ Enqueued -await queue.EnqueueAsync(new OrderWorkItem { OrderId = 123 }); // ❌ Discarded (duplicate) -await queue.EnqueueAsync(new OrderWorkItem { OrderId = 456 }); // ✅ Enqueued -``` - -**How it works:** - -1. On enqueue, the behavior checks if the `UniqueIdentifier` exists in the cache -2. If found, the message is discarded (not enqueued) -3. If not found, the identifier is cached with the specified TTL -4. On dequeue, the identifier is removed from the cache (allowing re-submission) - -### Behavior Attachment Rules - -Each behavior instance can only be attached to a single queue. Attempting to attach the same behavior instance to multiple queues or attaching it twice to the same queue throws a `QueueException`. This prevents subtle bugs where event handlers could fire against the wrong queue reference. - -```csharp -// ✅ Correct: separate instances for each queue -queue1.AttachBehavior(new LoggingQueueBehavior(logger)); -queue2.AttachBehavior(new LoggingQueueBehavior(logger)); - -// ❌ Throws QueueException: same instance attached twice -var behavior = new LoggingQueueBehavior(logger); -queue1.AttachBehavior(behavior); -queue2.AttachBehavior(behavior); // throws QueueException -``` - -### Attaching Multiple Behaviors - -You can attach multiple different behavior instances to a single queue: - -```csharp -var queue = new InMemoryQueue(o => o - .Behaviors( - new LoggingQueueBehavior(logger), - new DuplicateDetectionQueueBehavior(cache, loggerFactory), - new MetricsQueueBehavior(metrics) - )); -``` - -## Queue Statistics - -Monitor queue health: - -```csharp -var stats = await queue.GetQueueStatsAsync(); - -Console.WriteLine($"Queued: {stats.Queued}"); -Console.WriteLine($"Working: {stats.Working}"); -Console.WriteLine($"Dead Letter: {stats.Deadletter}"); -Console.WriteLine($"Enqueued: {stats.Enqueued}"); -Console.WriteLine($"Dequeued: {stats.Dequeued}"); -Console.WriteLine($"Completed: {stats.Completed}"); -Console.WriteLine($"Abandoned: {stats.Abandoned}"); -Console.WriteLine($"Errors: {stats.Errors}"); -Console.WriteLine($"Timeouts: {stats.Timeouts}"); -``` - -## Dead Letter Queue - -Handle failed messages that have exceeded the retry limit: - -```csharp -// Get dead letter items -var deadLetters = await queue.GetDeadletterItemsAsync(); - -foreach (var item in deadLetters) -{ - _logger.LogWarning("Dead letter: {Id}", item.Id); - - // Optionally re-queue for retry - await queue.EnqueueAsync(item); -} -``` - -### When Messages Go to Dead Letter - -Messages are moved to the dead letter queue when: - -1. The message has been abandoned more times than the configured `Retries` count -2. Processing repeatedly fails and the retry limit is exhausted - -### Monitoring Dead Letters - -```csharp -var stats = await queue.GetQueueStatsAsync(); -if (stats.Deadletter > 0) -{ - _logger.LogWarning("Dead letter queue has {Count} items", stats.Deadletter); - // Alert operations team, trigger investigation -} -``` - -### Poison Message Handling - -When a message cannot be deserialized during dequeue (e.g., corrupted data, schema changes, or serializer misconfiguration), all Foundatio queue implementations handle it gracefully: - -1. **The deserialization exception is caught** and logged as a warning with the message ID and current attempt count -2. **The message is abandoned** through the normal `AbandonAsync` flow, which increments the attempt counter and applies retry delay/backoff -3. **`null` is returned** from `DequeueAsync`, so the consumer never sees the undeserializable message -4. **After exhausting retries**, the message is moved to the dead letter queue through the standard dead-lettering path - -This approach gives operators a window to fix transient issues (such as a missing `JsonConverter` or incorrect serializer configuration) before messages are permanently dead-lettered. If the serializer configuration is corrected and redeployed before retries are exhausted, the message will deserialize successfully on the next attempt. - -```text -Dequeue → Deserialize fails → Abandon (attempt incremented) - → Still has retries? → Re-queued with backoff delay - → Retries exhausted? → Moved to dead letter queue -``` - -## Retry Policies - -All Foundatio queue implementations share common retry behavior configured via `SharedQueueOptions`: - -| Option | Default | Description | -|--------|---------|-------------| -| `Retries` | 2 | Maximum number of retry attempts before dead-lettering | -| `WorkItemTimeout` | 5 minutes | How long a worker can hold a message before it's considered abandoned | - -### WorkItemTimeout Best Practices - -The `WorkItemTimeout` determines how long a dequeued entry stays locked before being considered abandoned and returned to the queue for retry. Setting this value correctly is critical for reliable queue processing. - -**Guidelines for setting `WorkItemTimeout`:** - -```csharp -var queue = new RedisQueue(o => -{ - // For predictable workloads: typical duration + padding - // Example: If processing takes 2 minutes, set to 4-5 minutes - o.WorkItemTimeout = TimeSpan.FromMinutes(5); - - // For variable workloads: maximum expected duration + buffer - // Example: If processing can take up to 10 minutes, set to 15 minutes - o.WorkItemTimeout = TimeSpan.FromMinutes(15); -}); -``` - -**Sizing recommendations:** - -- **Fast operations (< 30 seconds)**: Set to 1-2 minutes to allow for retries without long delays -- **Standard operations (1-5 minutes)**: Set to 2x your average processing time (e.g., 3 minutes avg → 6 minute timeout) -- **Long operations (> 5 minutes)**: Set to 1.5x your maximum expected time, but consider using manual lock renewal if highly variable -- **Always include padding**: Account for network latency, temporary slowdowns, and system load - -**What happens when timeout expires:** - -1. The queue entry lock is released -2. Another worker can pick up the same entry -3. The original worker may still be processing (potentially duplicate work) -4. Entry's `Attempts` counter increments -5. After `Retries` attempts, the entry moves to the dead letter queue - -::: warning Timeout Too Short -If `WorkItemTimeout` is too short, entries will be re-queued before processing completes, leading to duplicate processing attempts and wasted resources. -::: - -::: tip Monitoring and Adjustment -Monitor your queue processing times and adjust `WorkItemTimeout` based on actual metrics. Use Application Insights, logging, or custom telemetry to track processing duration over time. -::: - -### InMemoryQueue Retry Options - -The in-memory queue provides additional retry configuration: - -```csharp -var queue = new InMemoryQueue(o => -{ - o.Retries = 3; // Max retry attempts - o.RetryDelay = TimeSpan.FromMinutes(1); // Base delay between retries - o.RetryMultipliers = new[] { 1, 3, 5, 10 }; // Exponential backoff multipliers -}); -``` - -**Retry delay calculation:** `RetryDelay × RetryMultipliers[attempt - 1]` - -For example, with defaults: - -- 1st retry: 1 minute × 1 = 1 minute -- 2nd retry: 1 minute × 3 = 3 minutes -- 3rd retry: 1 minute × 5 = 5 minutes -- 4th+ retry: 1 minute × 10 = 10 minutes - -### Provider-Specific Retry Behavior - -| Provider | Retry Mechanism | Dead Letter Support | -|----------|-----------------|---------------------| -| InMemoryQueue | Built-in with configurable backoff | In-memory dead letter queue | -| RedisQueue | Built-in with configurable backoff | Redis-backed dead letter queue | -| AzureServiceBusQueue | Native Service Bus retries | Native DLQ with message metadata | -| AzureStorageQueue | Built-in retries | Poison message queue | -| SQSQueue | Native SQS retries | Native DLQ (requires configuration) | - -## Message Size Limits - -Different queue providers have different message size limits. Understanding these limits is crucial for designing your message contracts. - -| Provider | Max Message Size | Notes | -|----------|------------------|-------| -| InMemoryQueue | Limited by available memory | No practical limit | -| RedisQueue | 512 MB (Redis limit) | Recommended: < 1 MB for performance | -| AzureServiceBusQueue | 256 KB (Standard) / 100 MB (Premium) | Use claim check pattern for large payloads | -| AzureStorageQueue | 64 KB | Base64 encoded, effective ~48 KB | -| SQSQueue | 256 KB | Use S3 for larger messages | - -### Best Practice: Keep Messages Small - -```csharp -// ✅ Good: Small message with reference -public record ProcessImageWorkItem -{ - public required string ImageBlobPath { get; init; } // Reference to storage - public required string OutputPath { get; init; } - public required ImageProcessingOptions Options { get; init; } -} - -// ❌ Bad: Large payload in message -public record ProcessImageWorkItem -{ - public required byte[] ImageData { get; init; } // Could be megabytes! - public required ImageProcessingOptions Options { get; init; } -} -``` - -### Claim Check Pattern for Large Payloads - -When you need to process large data, store it externally and pass a reference: - -```csharp -// Store large data in blob storage -var blobPath = $"work-items/{Guid.NewGuid()}.json"; -await fileStorage.SaveObjectAsync(blobPath, largePayload); - -// Enqueue reference only -await queue.EnqueueAsync(new WorkItem -{ - PayloadPath = blobPath, - PayloadSize = largePayload.Length -}); - -// In worker: retrieve the payload -var entry = await queue.DequeueAsync(); -var payload = await fileStorage.GetObjectAsync(entry.Value.PayloadPath); -await ProcessAsync(payload); -await entry.CompleteAsync(); - -// Clean up blob after processing -await fileStorage.DeleteFileAsync(entry.Value.PayloadPath); -``` - -## Dependency Injection - -### Basic Registration - -```csharp -// In-memory (development) -services.AddSingleton>(sp => - new InMemoryQueue()); - -// Redis (production) -services.AddSingleton>(sp => - new RedisQueue(o => { - o.ConnectionMultiplexer = sp.GetRequiredService(); - o.Name = "work-items"; - })); -``` - -::: tip Automatic Queue Processing -For automatic background processing of queue items, use `QueueJobBase` with `Foundatio.Extensions.Hosting`. See [Jobs - Queue Processor Jobs](/guide/jobs#queue-processor-jobs) for details. - -```csharp -// Register queue and processor job -services.AddSingleton>(sp => new InMemoryQueue()); -services.AddJob(); // Automatically processes queue items -``` - -### Multiple Queues - -```csharp -services.AddSingleton>(sp => - new InMemoryQueue(o => o.Name = "orders")); - -services.AddSingleton>(sp => - new InMemoryQueue(o => o.Name = "emails")); -``` - -## Queue Exceptions - -Queue operations throw `QueueException` for queue-specific error conditions. This provides a consistent, predictable exception type across all queue implementations (in-memory, Redis, Azure, AWS, etc.). - -```csharp -using Foundatio.Queues; - -try -{ - // Attempting to reuse a behavior instance throws QueueException - var behavior = new LoggingQueueBehavior(logger); - queue1.AttachBehavior(behavior); - queue2.AttachBehavior(behavior); // throws QueueException -} -catch (QueueException ex) -{ - logger.LogError(ex, "Queue operation failed: {Message}", ex.Message); -} -``` - -## Cancellation Token Behavior - -Understanding how cancellation tokens are handled internally is important for building reliable queue consumers. - -### Resource Creation Uses Disposal Token - -When you call `EnqueueAsync`, `DequeueAsync`, or `GetDeadletterItemsAsync`, the queue may need to create infrastructure (e.g., SQS queues, Azure Service Bus queues, Redis streams). These setup operations use an internal disposal token — **not** the caller's cancellation token. This means: - -- **Queue creation only aborts when the queue is disposed**, never because a single caller cancelled their operation. -- A cancelled `DequeueAsync` call (e.g., from a zero timeout) will not prevent queue creation from completing. -- Multiple concurrent callers cannot interfere with each other's setup. - -### Linked Cancellation for Operations - -The caller's cancellation token is combined with the disposal token into a linked token for the actual operation (dequeue, deadletter retrieval, etc.). This means: - -- Operations cancel when **either** the caller cancels **or** the queue is disposed. -- Graceful shutdown via `Dispose()` cancels all in-flight operations promptly. - -```csharp -// This will never prevent queue creation, even though it times out immediately -var entry = await queue.DequeueAsync(TimeSpan.Zero); - -// The cancellation token only affects the dequeue wait, not infrastructure setup -using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5)); -var entry = await queue.DequeueAsync(cts.Token); -``` - -### For Implementation Authors - -If you are writing a custom `IQueue` implementation by extending `QueueBase`: - -- **`EnsureQueueCreatedAsync`** always receives `DisposedCancellationToken`. Use it for all setup operations (lock acquisition, API calls, etc.). -- **`DequeueImplAsync`** receives a linked token (caller + disposal). Respect it for the wait/poll operation. -- **`EnqueueImplAsync`** does not receive a cancellation token — keep enqueue fast and non-blocking. - -## Best Practices - -### 1. Proper Resource Disposal - -Queues implement `IDisposable` and should be properly disposed: +Queued work uses the same `IMessageBus` client as pub/sub, with explicit consumer registration. `SendAsync` targets competing consumers; `PublishAsync` targets event subscriptions. ```csharp -// ✅ Good: Using statement for short-lived queues -await using var queue = new InMemoryQueue(); -await queue.EnqueueAsync(new WorkItem { Id = 1 }); - -// ✅ Good: DI container manages lifetime -services.AddSingleton>(sp => - new InMemoryQueue()); +builder.Services.AddFoundatioWorker(foundatio => foundatio + .Messaging.UseInMemory() + .AddConsumer()); -// ❌ Bad: Not disposing -var queue = new InMemoryQueue(); -// ... use queue -// Queue is never disposed, resources leak +await bus.SendAsync(new ProcessOrder(1001)); ``` -### 2. Use Typed Messages +Implement `IMessageHandler` and process the message in `HandleAsync`. Successful handlers are acknowledged automatically; failures use the retry/dead-letter policy. Replicas compete for the queue, and each invocation gets its own dependency injection scope. Use idempotent processing because a message may be delivered again after a crash or uncertain acknowledgement. -```csharp -// ✅ Good: Typed, versioned messages -public record OrderWorkItem -{ - public int Version { get; init; } = 1; - public required int OrderId { get; init; } - public required DateTime CreatedAt { get; init; } -} - -// ❌ Bad: Generic, untyped -public class WorkItem -{ - public object Data { get; set; } -} -``` +For manual loops, `ReceiveAsync` returns a disposable delivery with complete, reject, and renewal operations. Disposing an unfinished delivery returns it for redelivery. See [Messaging](messaging.md) for direct receive examples, concurrency, delays, topology, dead-letter administration, and provider guarantees. -### 2. Handle Idempotency - -```csharp -var entry = await queue.DequeueAsync(); -if (entry is null) - return; - -// Check if already processed -if (await _processedIds.ContainsAsync(entry.Value.Id)) -{ - await entry.CompleteAsync(); - return; -} - -// Process -await ProcessAsync(entry.Value); - -// Mark as processed -await _processedIds.AddAsync(entry.Value.Id); -await entry.CompleteAsync(); -``` - -### 3. Set Appropriate Timeouts - -```csharp -var queue = new RedisQueue(o => { - o.WorkItemTimeout = TimeSpan.FromMinutes(5); // How long to process - o.RetryDelay = TimeSpan.FromSeconds(30); // Delay before retry - o.Retries = 3; // Max retries -}); -``` - -### 4. Monitor Queue Depth - -```csharp -var stats = await queue.GetQueueStatsAsync(); -if (stats.Queued > 1000) -{ - _logger.LogWarning("Queue depth is high: {Depth}", stats.Queued); - // Consider scaling workers -} -``` - -### 5. Use Delayed Delivery for Scheduling - -```csharp -// Schedule for later -await queue.EnqueueAsync(reminder, new QueueEntryOptions -{ - DeliveryDelay = TimeSpan.FromHours(24) -}); -``` - -## Queue Name vs Queue ID - -Every queue has two distinct identifiers that serve different purposes: - -| Property | Purpose | Stable across restarts? | Shared across processes? | -|----------|---------|------------------------|--------------------------| -| `Name` | Identifies the queue in the **backing store** (Redis key prefix, SQS queue name, etc.) | ✅ Yes | ✅ Yes — two processes with the same `Name` share the same data | -| `QueueId` | Runtime **instance identifier** used only for logging and diagnostics | ❌ No (random suffix by default) | N/A — never used for data routing | - -**`Name`** is what controls which data is read and written. All distributed queue implementations (Redis, SQS, Azure Service Bus, Azure Storage) route messages based on `Name`, not `QueueId`. Two processes or application restarts using the same `Name` will naturally share queue data and continue from where the other left off. - -**`QueueId`** exists purely so that multiple queue instances within the same process (e.g., priority queues or keyed queues) produce distinguishable log output. It has no effect on the backing store. - -::: tip Sharing queues across processes -If you want multiple processes (or application restarts) to share a queue, just ensure they all configure the same `Name` value. The default is `typeof(T).Name` (the message type name), so it is consistent by default as long as you use the same message type. - -```csharp -// Both processes use the same Name → they share the same queue in Redis -var queue = new RedisQueue(o => { - o.ConnectionMultiplexer = redis; - o.Name = "work-items"; // This is the stable backing-store identifier -}); -``` -::: +Use [durable jobs](jobs.md) when a caller needs a job handle, progress, persisted execution retries, cancellation, or CRON scheduling. -::: info InMemoryQueue -`InMemoryQueue` is an in-process implementation only. It cannot share data across processes regardless of `Name` or `QueueId`. Use a distributed implementation (Redis, SQS, Azure) for cross-process sharing. -::: +## Migrating IQueue -## Next Steps +| Former API | Current pattern | +| --- | --- | +| `IQueue.EnqueueAsync` | `IMessageBus.SendAsync` | +| `QueueJobBase` / queue worker callbacks | `IMessageHandler` and `AddConsumer` | +| `DequeueAsync` | `ReceiveAsync` and `await using` | +| Queue entry completion/abandonment | Delivery `CompleteAsync` / `RejectAsync` | +| `WorkItemJob` handlers | Typed `IJob` and `EnqueueAsync` | +| Queue-specific retry settings | Consumer retry options and the message bus retry policy | -- [Jobs](./jobs) - Queue processor jobs for automatic background processing with `QueueJobBase` -- [Messaging](./messaging) - Pub/sub for event-driven patterns -- [Locks](./locks) - Coordinate queue processing across instances -- [Serialization](./serialization) - Serializer configuration and performance +Earlier external provider packages implement the former queue interfaces; they do not automatically implement the new transport contract. This unreleased revision supplies in-memory, Redis Streams, and SQS/SNS transports. Check the [provider matrix](messaging.md#provider-guarantees) before changing implementations. diff --git a/docs/guide/resilience.md b/docs/guide/resilience.md index 98fc94010..79f5b8c0a 100644 --- a/docs/guide/resilience.md +++ b/docs/guide/resilience.md @@ -488,7 +488,6 @@ var resilientCache = new ResilientCacheClient( ```csharp public class ResilientQueueProcessor { - private readonly IQueue _queue; private readonly IResiliencePolicy _policy; public async Task ProcessAsync(WorkItem item) diff --git a/docs/guide/serialization.md b/docs/guide/serialization.md index 9543cc541..d6d0a696a 100644 --- a/docs/guide/serialization.md +++ b/docs/guide/serialization.md @@ -36,6 +36,8 @@ public interface ITextSerializer : ISerializer { } This abstraction allows you to swap serializers without changing your code. +Serializers can optionally implement `IBufferSerializer` to serialize directly to an owned byte array and deserialize from `ReadOnlyMemory`. `SerializeToBytes` and the byte-array/memory `Deserialize` extensions select this path automatically. The default `SystemTextJsonSerializer` supports it, avoiding intermediate streams and payload copies with no configuration changes. Existing stream-only serializers continue to work. Buffer implementations must preserve their stream serializer's options and null handling, return independently owned output, and never retain or modify the input memory. + ## Extension Methods The `SerializerExtensions` class provides convenient methods for common serialization scenarios: @@ -105,8 +107,7 @@ var jsonOptions = new JsonSerializerOptions var serializer = new SystemTextJsonSerializer(jsonOptions); var cache = new InMemoryCacheClient(o => o.Serializer = serializer); -var queue = new InMemoryQueue(o => o.Serializer = serializer); -var messageBus = new InMemoryMessageBus(o => o.Serializer = serializer); +var messageBus = new MessageBus(new InMemoryMessageTransport(), new MessageBusOptions { Serializer = serializer }); ``` ## Global Default Serializer @@ -119,7 +120,7 @@ DefaultSerializer.Instance = new SystemTextJsonSerializer(myJsonOptions); // Now all new instances use your custom serializer var cache = new InMemoryCacheClient(); // Uses your custom serializer -var queue = new InMemoryQueue(); // Uses your custom serializer +var messageBus = new MessageBus(new InMemoryMessageTransport()); // Uses your custom serializer ``` **How it works:** @@ -254,11 +255,8 @@ var serializer = new MessagePackSerializer(); // Caching var cache = new InMemoryCacheClient(o => o.Serializer = serializer); -// Queues -var queue = new InMemoryQueue(o => o.Serializer = serializer); - -// Messaging -var messageBus = new InMemoryMessageBus(o => o.Serializer = serializer); +// Queued work and pub/sub +var messageBus = new MessageBus(new InMemoryMessageTransport(), new MessageBusOptions { Serializer = serializer }); // Storage (for metadata serialization) var storage = new InMemoryFileStorage(o => o.Serializer = serializer); diff --git a/docs/guide/what-is-foundatio.md b/docs/guide/what-is-foundatio.md index d88884370..7773a4130 100644 --- a/docs/guide/what-is-foundatio.md +++ b/docs/guide/what-is-foundatio.md @@ -16,7 +16,7 @@ Foundatio was built with several key principles in mind: ### Abstract Interfaces -All core functionality is exposed through clean interfaces (`ICacheClient`, `IQueue`, `ILockProvider`, `IMessageBus`, `IFileStorage`). This allows you to: +All core functionality is exposed through clean interfaces (`ICacheClient`, `IJobClient`, `ILockProvider`, `IMessageBus`, `IFileStorage`). This allows you to: - **Swap implementations** without changing application code - **Test easily** using in-memory implementations @@ -28,7 +28,7 @@ Every component is designed to work seamlessly with Microsoft.Extensions.Depende ```csharp services.AddSingleton(sp => new InMemoryCacheClient()); -services.AddSingleton(sp => new InMemoryMessageBus()); +services.AddSingleton(sp => new MessageBus(new InMemoryMessageTransport())); services.AddSingleton(sp => new CacheLockProvider( sp.GetRequiredService(), sp.GetRequiredService() @@ -73,9 +73,9 @@ var cached = await cache.GetAsync("user:123"); Reliable message delivery with at-least-once semantics: ```csharp -IQueue queue = new InMemoryQueue(); -await queue.EnqueueAsync(new WorkItem { Id = 1 }); -var entry = await queue.DequeueAsync(); +await using var bus = new MessageBus(new InMemoryMessageTransport()); +await bus.SendAsync(new WorkItem { Id = 1 }); +await using var entry = await bus.ReceiveAsync(); if (entry != null) { // Process and complete @@ -105,8 +105,8 @@ if (lck != null) Publish/subscribe messaging: ```csharp -IMessageBus bus = new InMemoryMessageBus(); -await bus.SubscribeAsync(msg => ProcessOrder(msg)); +IMessageBus bus = new MessageBus(new InMemoryMessageTransport()); +await using var subscription = await bus.SubscribeAsync((context, token) => ProcessOrder(context.Message)); await bus.PublishAsync(new OrderCreated { OrderId = 123 }); ``` @@ -129,9 +129,9 @@ var file = await storage.GetFileStreamAsync("reports/2024/report.pdf", StreamMod Background job processing: ```csharp -public class MyJob : JobBase +public class MyJob : IJob { - protected override Task RunInternalAsync(JobContext context) + public Task RunAsync(JobExecutionContext context) { // Do work return Task.FromResult(JobResult.Success); diff --git a/docs/guide/why-foundatio.md b/docs/guide/why-foundatio.md index 11ac54fdb..023ee8bcd 100644 --- a/docs/guide/why-foundatio.md +++ b/docs/guide/why-foundatio.md @@ -23,12 +23,12 @@ Write your code against interfaces, not implementations: public class OrderProcessor { private readonly ICacheClient _cache; - private readonly IQueue _queue; + private readonly IMessageBus _bus; - public OrderProcessor(ICacheClient cache, IQueue queue) + public OrderProcessor(ICacheClient cache, IMessageBus bus) { _cache = cache; - _queue = queue; + _bus = bus; } } ``` @@ -54,15 +54,15 @@ public async Task Should_Process_Order_With_Caching() { // Arrange - use in-memory implementations var cache = new InMemoryCacheClient(); - var queue = new InMemoryQueue(); - var processor = new OrderProcessor(cache, queue); + using var bus = new MessageBus(new InMemoryMessageTransport()); + var processor = new OrderProcessor(cache, bus); // Act await processor.ProcessAsync(new Order { Id = 1 }); // Assert var cached = await cache.GetAsync("order:1"); - Assert.NotNull(cached); + Assert.True(cached.HasValue); } ``` @@ -79,8 +79,7 @@ Start coding immediately without external dependencies: ```csharp // Works out of the box - no Redis, no Azure, no AWS var cache = new InMemoryCacheClient(); -var queue = new InMemoryQueue(); -var messageBus = new InMemoryMessageBus(); +var messageBus = new MessageBus(new InMemoryMessageTransport()); var storage = new InMemoryFileStorage(); ``` diff --git a/docs/index.md b/docs/index.md index da0c82b5c..fb816307e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -72,12 +72,13 @@ var value = await cache.GetAsync("test"); ### Queues ```csharp -using Foundatio.Queues; - -IQueue queue = new InMemoryQueue(); +using Foundatio.Messaging; -await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello" }); -var workItem = await queue.DequeueAsync(); +await using var bus = new MessageBus(new InMemoryMessageTransport()); +await bus.SendAsync(new SimpleWorkItem { Data = "Hello" }); +await using var workItem = await bus.ReceiveAsync(); +if (workItem is not null) + await workItem.CompleteAsync(); ``` [Learn more about Queues →](./guide/queues) @@ -89,7 +90,7 @@ using Foundatio.Lock; ILockProvider locker = new CacheLockProvider( new InMemoryCacheClient(), - new InMemoryMessageBus() + new MessageBus(new InMemoryMessageTransport()) ); await using var lck = await locker.AcquireAsync("resource"); @@ -107,10 +108,9 @@ await ProcessAsync(); ```csharp using Foundatio.Messaging; -IMessageBus messageBus = new InMemoryMessageBus(); -await messageBus.SubscribeAsync(msg => { - // Got message -}); +IMessageBus messageBus = new MessageBus(new InMemoryMessageTransport()); +await using var subscription = await messageBus.SubscribeAsync((context, token) => + ProcessAsync(context.Message, token)); await messageBus.PublishAsync(new SimpleMessage { Data = "Hello" }); ``` diff --git a/samples/Foundatio.AppHost/Foundatio.AppHost.csproj b/samples/Foundatio.AppHost/Foundatio.AppHost.csproj index e6c715ecd..e77bedf87 100644 --- a/samples/Foundatio.AppHost/Foundatio.AppHost.csproj +++ b/samples/Foundatio.AppHost/Foundatio.AppHost.csproj @@ -14,7 +14,7 @@ - + diff --git a/samples/Foundatio.AppHost/Program.cs b/samples/Foundatio.AppHost/Program.cs index e5d8cc8ff..93ebe54aa 100644 --- a/samples/Foundatio.AppHost/Program.cs +++ b/samples/Foundatio.AppHost/Program.cs @@ -11,17 +11,26 @@ .WithRedisInsight(b => b.WithEndpointProxySupport(false).WithContainerName("Foundatio-RedisInsight") .WithUrlForEndpoint("http", u => u.DisplayText = "Cache")); -builder.AddProject("Foundatio-HostingSample") +// LocalStack provides AWS SQS/SNS locally so the messaging sample's AWS transport works with no cloud account. +var localstack = builder.AddContainer("localstack", "localstack/localstack", "3") + .WithContainerName("Foundatio-LocalStack") + .WithEnvironment("SERVICES", "sqs,sns") + .WithEndpoint(port: 4566, targetPort: 4566, scheme: "http", name: "gateway") + .WithHttpHealthCheck("/_localstack/health", endpointName: "gateway"); + +// The redesigned messaging + durable-jobs sample, scaled to 3 replicas so you can watch the queue load-balance across +// instances, the pub/sub topic fan out to every instance, and durable/CRON jobs get claimed by a single instance. +// Messaging runs on AWS (SQS/SNS via LocalStack) and durable jobs on Redis. WithReference(cache) supplies the "Redis" +// connection string UseRedis() reads; the Aws__* settings point UseAws() at LocalStack (which accepts any credentials). +builder.AddProject("Foundatio-MessagingSample") .WithExternalHttpEndpoints() .WithReplicas(3) .WithReference(cache) .WaitFor(cache) - .WithArgs("all") - .WithUrls(u => - { - u.Urls.Clear(); - u.Urls.Add(new ResourceUrlAnnotation { Url = "/jobs/status", DisplayText = "Job Status", Endpoint = u.GetEndpoint("http") }); - u.Urls.Add(new ResourceUrlAnnotation { Url = "/jobs/run", DisplayText = "Run Job", Endpoint = u.GetEndpoint("http") }); - }); + .WaitFor(localstack) + .WithEnvironment("Aws__ServiceUrl", localstack.GetEndpoint("gateway")) + .WithEnvironment("Aws__AccessKey", "test") + .WithEnvironment("Aws__SecretKey", "test") + .WithEnvironment("Aws__ResourcePrefix", "fnd-sample-"); await builder.Build().RunAsync(); diff --git a/samples/Foundatio.HostingSample/Foundatio.HostingSample.csproj b/samples/Foundatio.HostingSample/Foundatio.HostingSample.csproj deleted file mode 100644 index 6fbcff86d..000000000 --- a/samples/Foundatio.HostingSample/Foundatio.HostingSample.csproj +++ /dev/null @@ -1,24 +0,0 @@ - - - - net10.0 - False - false - REDIS - - - - - - - - - - - - - - - - - diff --git a/samples/Foundatio.HostingSample/Jobs/EveryMinuteJob.cs b/samples/Foundatio.HostingSample/Jobs/EveryMinuteJob.cs deleted file mode 100644 index e879b104a..000000000 --- a/samples/Foundatio.HostingSample/Jobs/EveryMinuteJob.cs +++ /dev/null @@ -1,33 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Caching; -using Foundatio.Jobs; -using Microsoft.Extensions.Logging; - -namespace Foundatio.HostingSample; - -public class EveryMinuteJob : IJob -{ - private readonly ICacheClient _cacheClient; - private readonly ILogger _logger; - - public EveryMinuteJob(ILoggerFactory loggerFactory, ICacheClient cacheClient) - { - _cacheClient = cacheClient; - _logger = loggerFactory.CreateLogger(); - } - - public async Task RunAsync(CancellationToken cancellationToken = default) - { - var runCount = await _cacheClient.IncrementAsync("EveryMinuteJob"); - - _logger.LogInformation("EveryMinuteJob Run Count={Count} Thread={ManagedThreadId}", runCount, Thread.CurrentThread.ManagedThreadId); - - await Task.Delay(TimeSpan.FromSeconds(30)); - - _logger.LogInformation("EveryMinuteJob Complete"); - - return JobResult.Success; - } -} diff --git a/samples/Foundatio.HostingSample/Jobs/Sample1Job.cs b/samples/Foundatio.HostingSample/Jobs/Sample1Job.cs deleted file mode 100644 index bea203a60..000000000 --- a/samples/Foundatio.HostingSample/Jobs/Sample1Job.cs +++ /dev/null @@ -1,42 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Jobs; -using Foundatio.Resilience; -using Microsoft.Extensions.Logging; - -namespace Foundatio.HostingSample; - -[Job(Description = "Sample 1 job", Interval = "5s", IterationLimit = 5)] -public class Sample1Job : IJob -{ - private readonly IResiliencePolicy _policy; - private readonly ILogger _logger; - private int _iterationCount = 0; - - public Sample1Job(IResiliencePolicyProvider provider, ILoggerFactory loggerFactory) - { - // get policy for Sample1Job and if not found, try to get policy for IJob, then fallback to default policy - _policy = provider.GetPolicy(); - _logger = loggerFactory.CreateLogger(); - } - - public async Task RunAsync(CancellationToken cancellationToken = default) - { - return await _policy.ExecuteAsync(async _ => - { - int count = Interlocked.Increment(ref _iterationCount); - _logger.LogTrace("Sample1Job Run #{IterationCount} Thread={ManagedThreadId}", _iterationCount, Thread.CurrentThread.ManagedThreadId); - - if (count < 3) - { - _logger.LogInformation("Sample1Job Run #{IterationCount} Thread={ManagedThreadId} - Simulating failure", _iterationCount, Thread.CurrentThread.ManagedThreadId); - throw new InvalidOperationException("Simulated failure"); - } - - await Task.Delay(5000, cancellationToken); - - return JobResult.Success; - }, cancellationToken); - } -} diff --git a/samples/Foundatio.HostingSample/Jobs/Sample2Job.cs b/samples/Foundatio.HostingSample/Jobs/Sample2Job.cs deleted file mode 100644 index dc7fbdccd..000000000 --- a/samples/Foundatio.HostingSample/Jobs/Sample2Job.cs +++ /dev/null @@ -1,41 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Jobs; -using Microsoft.Extensions.Diagnostics.HealthChecks; -using Microsoft.Extensions.Logging; - -namespace Foundatio.HostingSample; - -[Job(Description = "Sample 2 job", Interval = "15s", IterationLimit = 24)] -public class Sample2Job : IJob, IHealthCheck -{ - private readonly ILogger _logger; - private int _iterationCount = 0; - private DateTime? _lastRun = null; - - public Sample2Job(ILoggerFactory loggerFactory) - { - _logger = loggerFactory.CreateLogger(); - } - - public Task RunAsync(CancellationToken cancellationToken = default) - { - _lastRun = DateTime.UtcNow; - Interlocked.Increment(ref _iterationCount); - _logger.LogTrace("Sample2Job Run #{IterationCount} Thread={ManagedThreadId}", _iterationCount, Thread.CurrentThread.ManagedThreadId); - - return Task.FromResult(JobResult.Success); - } - - public Task CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) - { - if (!_lastRun.HasValue) - return Task.FromResult(HealthCheckResult.Healthy("Job has not been run yet.")); - - if (DateTime.UtcNow.Subtract(_lastRun.Value) > TimeSpan.FromSeconds(5)) - return Task.FromResult(HealthCheckResult.Unhealthy("Job has not run in the last 5 seconds.")); - - return Task.FromResult(HealthCheckResult.Healthy("Job has run in the last 5 seconds.")); - } -} diff --git a/samples/Foundatio.HostingSample/Jobs/SampleLockJob.cs b/samples/Foundatio.HostingSample/Jobs/SampleLockJob.cs deleted file mode 100644 index 9cae2fc83..000000000 --- a/samples/Foundatio.HostingSample/Jobs/SampleLockJob.cs +++ /dev/null @@ -1,32 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Caching; -using Foundatio.Jobs; -using Foundatio.Lock; -using Microsoft.Extensions.Logging; - -namespace Foundatio.HostingSample; - -[Job(Description = "Sample lock job", Interval = "5s")] -public class SampleLockJob : JobWithLockBase -{ - private readonly ILockProvider _lockProvider; - - public SampleLockJob(ICacheClient cache, ILoggerFactory loggerFactory) : base(loggerFactory) - { - _lockProvider = new ThrottlingLockProvider(cache, 1, TimeSpan.FromMinutes(1), _timeProvider, _resiliencePolicyProvider, loggerFactory); - } - - protected override Task GetLockAsync(CancellationToken cancellationToken = default) - { - return _lockProvider.AcquireAsync(nameof(SampleLockJob), TimeSpan.FromMinutes(15), cancellationToken); - } - - protected override Task RunInternalAsync(JobContext context) - { - _logger.LogTrace("SampleLockJob Run Thread={ManagedThreadId}", Thread.CurrentThread.ManagedThreadId); - - return Task.FromResult(JobResult.Success); - } -} diff --git a/samples/Foundatio.HostingSample/MyCriticalHealthCheck.cs b/samples/Foundatio.HostingSample/MyCriticalHealthCheck.cs deleted file mode 100644 index 7d3d7ac84..000000000 --- a/samples/Foundatio.HostingSample/MyCriticalHealthCheck.cs +++ /dev/null @@ -1,18 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Microsoft.Extensions.Diagnostics.HealthChecks; - -namespace Foundatio.HostingSample; - -public class MyCriticalHealthCheck : IHealthCheck -{ - private static DateTime _startTime = DateTime.Now; - - public Task CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = new CancellationToken()) - { - return DateTime.Now.Subtract(_startTime) > TimeSpan.FromSeconds(3) ? - Task.FromResult(HealthCheckResult.Healthy("Critical resource is available.")) - : Task.FromResult(HealthCheckResult.Unhealthy("Critical resource not available.")); - } -} diff --git a/samples/Foundatio.HostingSample/Program.cs b/samples/Foundatio.HostingSample/Program.cs deleted file mode 100644 index 8560a9786..000000000 --- a/samples/Foundatio.HostingSample/Program.cs +++ /dev/null @@ -1,222 +0,0 @@ -using System; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; -using Foundatio; -using Foundatio.Extensions.Hosting.Jobs; -using Foundatio.Extensions.Hosting.Startup; -using Foundatio.HostingSample; -using Foundatio.Resilience; -using Foundatio.Serializer; -using Microsoft.AspNetCore.Builder; -using Microsoft.AspNetCore.Http; -using Microsoft.AspNetCore.Http.Json; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Hosting; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Options; -#if REDIS -using Microsoft.Extensions.Configuration; -using Foundatio.Redis; -using StackExchange.Redis; -#endif - -bool all = args.Contains("all", StringComparer.OrdinalIgnoreCase); -bool sample1 = all || args.Contains("sample1", StringComparer.OrdinalIgnoreCase); -bool sample2 = all || args.Contains("sample2", StringComparer.OrdinalIgnoreCase); -bool everyMinute = all || args.Contains("everyMinute", StringComparer.OrdinalIgnoreCase); -bool evenMinutes = all || args.Contains("evenMinutes", StringComparer.OrdinalIgnoreCase); - -var builder = WebApplication.CreateBuilder(args); - -// configure Foundatio services -builder.Services.AddFoundatio() - .Storage.UseFolder() - .Caching.UseInMemory() - .Locking.UseCache() - .Messaging.UseInMemory() - .AddSerializer(sp => new SystemTextJsonSerializer(sp.GetRequiredService>().Value.SerializerOptions)) - .AddResilience(b => b.WithPolicy(p => p.WithMaxAttempts(5).WithLinearDelay().WithJitter())); - -ConfigureServices(); - -// shutdown the host if no jobs are running, cron jobs are not considered running jobs -builder.Services.AddJobLifetimeService(); - -// inserts a startup action that does not complete until the critical health checks are healthy -// gets inserted as 1st startup action so that any other startup actions don't run until the critical resources are available -builder.Services.AddStartupActionToWaitForHealthChecks("Critical"); - -builder.Services.AddHealthChecks().AddCheck("My Critical Resource", tags: ["Critical"]); - -// add health check that does not return healthy until the startup actions have completed -// useful for readiness checks -builder.Services.AddHealthChecks().AddCheckForStartupActions("Critical"); - -// this gets added automatically by any AddJob call, but we might not be running any jobs, and we need it for doing dynamic jobs -builder.Services.AddJobScheduler(); - -if (everyMinute) - builder.Services.AddDistributedCronJob("* * * * *"); - -builder.Services.AddCronJob(b => b.Name("Tokyo").CronSchedule("44 4 * * *").CronTimeZone("Asia/Tokyo").JobAction(async sp => -{ - var logger = sp.GetRequiredService>(); - logger.LogInformation("Tokyo 4:44am Run Thread={ManagedThreadId}", Thread.CurrentThread.ManagedThreadId); - await Task.Delay(TimeSpan.FromSeconds(5)); -})); - -if (evenMinutes) - builder.Services.AddCronJob("EvenMinutes", "*/2 * * * *", async sp => - { - var logger = sp.GetRequiredService>(); - logger.LogInformation("EvenMinuteJob Run Thread={ManagedThreadId}", Thread.CurrentThread.ManagedThreadId); - await Task.Delay(TimeSpan.FromSeconds(30)); - logger.LogInformation("EvenMinuteJob Complete"); - }); - -if (sample1) - builder.Services.AddJob("Sample1", sp => new Sample1Job(sp.GetService(), sp.GetService()), o => o.ApplyDefaults().WaitForStartupActions().InitialDelay(TimeSpan.FromSeconds(4))); - -builder.Services.AddJob(o => o.WaitForStartupActions()); - -if (sample2) -{ - builder.Services.AddHealthChecks().AddCheck("Sample2Job"); - builder.Services.AddJob(o => o.WaitForStartupActions()); -} - -// if you don't specify priority, actions will automatically be assigned an incrementing priority starting at 0 -builder.Services.AddStartupAction("Test1", async sp => -{ - var logger = sp.GetRequiredService>(); - logger.LogTrace("Running startup 1 action"); - for (int i = 0; i < 3; i++) - { - await Task.Delay(100); - logger.LogTrace("Running startup 1 action..."); - } - - logger.LogTrace("Done running startup 1 action"); -}); - -// then these startup actions will run concurrently since they both have the same priority -builder.Services.AddStartupAction(priority: 100); -builder.Services.AddStartupAction(priority: 100); - -/*builder.Services.AddStartupAction("Test2", async sp => -{ - var logger = sp.GetRequiredService>(); - logger.LogTrace("Running startup 2 action"); - for (int i = 0; i < 2; i++) - { - await Task.Delay(50); - logger.LogTrace("Running startup 2 action..."); - } - //throw new ApplicationException("Boom goes the startup"); - logger.LogTrace("Done running startup 2 action"); -});*/ - -//s.AddStartupAction("Boom", () => throw new ApplicationException("Boom goes the startup")); - -var app = builder.Build(); - -app.MapGet("/", () => "Foundatio!"); - -app.MapGet("/jobs/status", (IJobManager jobManager, string name = null, bool? running = null, bool history = true) => - { - if (!String.IsNullOrEmpty(name)) - return Results.Ok(jobManager.GetJobStatus(name, includeHistory: history)); - - if (running.HasValue && running.Value) - return Results.Ok(jobManager.GetJobStatus(true, includeHistory: history)); - - return Results.Ok(jobManager.GetJobStatus(includeHistory: history)); - }); - -app.MapGet("/jobs/run", async (IJobManager jobManager, string name) => - { - if (String.IsNullOrWhiteSpace(name)) - return Results.BadRequest("Job name is required."); - - await jobManager.RunJobAsync(name); - - return Results.Accepted($"Job {name} started successfully."); - }); - -app.MapGet("/jobs/enable", (IJobManager jobManager, string name) => - { - if (String.IsNullOrWhiteSpace(name)) - return Results.BadRequest("Job name is required."); - - jobManager.Update(name, c => c.Enabled()); - - return Results.Ok($"Job {name} enabled successfully."); - }); - -app.MapGet("/jobs/disable", (IJobManager jobManager, string name) => - { - if (String.IsNullOrWhiteSpace(name)) - return Results.BadRequest("Job name is required."); - - jobManager.Update(name, c => c.Disabled()); - - return Results.Ok($"Job {name} disabled successfully."); - }); - -app.MapGet("/jobs/schedule", (IJobManager jobManager, string name, string cron) => -{ - if (String.IsNullOrWhiteSpace(name)) - return Results.BadRequest("Job name is required."); - - jobManager.Update(name, c => c.CronSchedule(cron)); - - return Results.Ok($"Job {name} updated successfully."); -}); - -app.MapGet("/jobs/release", async (IJobManager jobManager, string name) => -{ - if (String.IsNullOrWhiteSpace(name)) - return Results.BadRequest("Job name is required."); - - await jobManager.ReleaseLockAsync(name); - - return Results.Ok($"Job {name} lock released successfully."); -}); - -app.UseHealthChecks("/health"); -app.UseReadyHealthChecks("Critical"); - -// this middleware will return Service Unavailable until the startup actions have completed -app.UseWaitForStartupActionsBeforeServingRequests(); - -// add mvc or other request middleware after the UseWaitForStartupActionsBeforeServingRequests call - -app.Run(); - -void ConfigureServices() -{ - builder.Services.AddLogging(opt => - { - opt.AddSimpleConsole(c => c.TimestampFormat = "[HH:mm:ss] "); - }); - - builder.AddServiceDefaults(); - builder.Services.ConfigureHttpJsonOptions(o => { o.SerializerOptions.WriteIndented = true; }); - -#if REDIS - builder.Services.AddSingleton(sp => - { - var connectionString = builder.Configuration.GetConnectionString("Redis")!; - connectionString += ",abortConnect=false"; - return ConnectionMultiplexer.Connect(connectionString); - // enable redis logging - //return ConnectionMultiplexer.Connect(connectionString, o => o.LoggerFactory = sp.GetRequiredService()); - }); - - // distributed cache and messaging using redis (replaces in memory cache) - builder.Services.AddFoundatio() - .Caching.UseRedis() - .Messaging.UseRedis(); -#endif -} diff --git a/samples/Foundatio.HostingSample/Properties/launchSettings.json b/samples/Foundatio.HostingSample/Properties/launchSettings.json deleted file mode 100644 index f520048a1..000000000 --- a/samples/Foundatio.HostingSample/Properties/launchSettings.json +++ /dev/null @@ -1,27 +0,0 @@ -{ - "$schema": "https://json.schemastore.org/launchsettings.json", - "profiles": { - "http": { - "commandName": "Project", - "commandLineArgs": "all", - "dotnetRunMessages": true, - "launchBrowser": false, - "launchUrl": "jobstatus", - "applicationUrl": "http://localhost:5324", - "environmentVariables": { - "ASPNETCORE_ENVIRONMENT": "Development" - } - }, - "https": { - "commandName": "Project", - "commandLineArgs": "all", - "dotnetRunMessages": true, - "launchBrowser": false, - "launchUrl": "jobstatus", - "applicationUrl": "https://localhost:7580;http://localhost:5324", - "environmentVariables": { - "ASPNETCORE_ENVIRONMENT": "Development" - } - } - } -} diff --git a/samples/Foundatio.HostingSample/ServiceDefaults.cs b/samples/Foundatio.HostingSample/ServiceDefaults.cs deleted file mode 100644 index fbd3284d7..000000000 --- a/samples/Foundatio.HostingSample/ServiceDefaults.cs +++ /dev/null @@ -1,57 +0,0 @@ -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Logging; -using OpenTelemetry; -using OpenTelemetry.Metrics; -using OpenTelemetry.Trace; - -namespace Microsoft.Extensions.Hosting; - -public static class Extensions -{ - public static TBuilder AddServiceDefaults(this TBuilder builder) where TBuilder : IHostApplicationBuilder - { - builder.ConfigureOpenTelemetry(); - - return builder; - } - - public static TBuilder ConfigureOpenTelemetry(this TBuilder builder) where TBuilder : IHostApplicationBuilder - { - builder.Logging.AddOpenTelemetry(logging => - { - logging.IncludeFormattedMessage = true; - logging.IncludeScopes = true; - }); - - builder.Services.AddOpenTelemetry() - .WithMetrics(metrics => - { - metrics.AddAspNetCoreInstrumentation() - .AddHttpClientInstrumentation() - .AddRuntimeInstrumentation() - .AddMeter("Foundatio"); - }) - .WithTracing(tracing => - { - tracing.AddAspNetCoreInstrumentation() - .AddHttpClientInstrumentation() - .AddSource("Foundatio"); - }); - - builder.AddOpenTelemetryExporters(); - - return builder; - } - - private static TBuilder AddOpenTelemetryExporters(this TBuilder builder) where TBuilder : IHostApplicationBuilder - { - var useOtlpExporter = !string.IsNullOrWhiteSpace(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]); - - if (useOtlpExporter) - { - builder.Services.AddOpenTelemetry().UseOtlpExporter(); - } - - return builder; - } -} diff --git a/samples/Foundatio.HostingSample/Startup/MyStartupAction.cs b/samples/Foundatio.HostingSample/Startup/MyStartupAction.cs deleted file mode 100644 index 8a87e059d..000000000 --- a/samples/Foundatio.HostingSample/Startup/MyStartupAction.cs +++ /dev/null @@ -1,42 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Caching; -using Foundatio.Extensions.Hosting.Jobs; -using Foundatio.Extensions.Hosting.Startup; -using Microsoft.Extensions.Logging; - -namespace Foundatio.HostingSample; - -public class MyStartupAction : IStartupAction -{ - private readonly IJobManager _jobManager; - private readonly ICacheClient _cacheClient; - private readonly ILogger _logger; - - public MyStartupAction(IJobManager jobManager, ICacheClient cacheClient, ILogger logger) - { - _jobManager = jobManager; - _cacheClient = cacheClient; - _logger = logger; - } - - public async Task RunAsync(CancellationToken cancellationToken = default) - { - // set next run to be far in the past so it runs immediately - await _cacheClient.SetAsync("jobs:every_minute:nextrun", DateTime.UtcNow.AddDays(-1)); - - for (int i = 0; i < 5; i++) - { - _logger.LogTrace("MyStartupAction Run Thread={ManagedThreadId}", Thread.CurrentThread.ManagedThreadId); - await Task.Delay(500); - } - - _jobManager.AddOrUpdate("MyJob", j => j.CronSchedule("* * * * *").JobAction(async () => - { - _logger.LogInformation("Running MyJob"); - await Task.Delay(1000); - _logger.LogInformation("MyJob Complete"); - })); - } -} diff --git a/samples/Foundatio.HostingSample/Startup/OtherStartupAction.cs b/samples/Foundatio.HostingSample/Startup/OtherStartupAction.cs deleted file mode 100644 index 96cce9a33..000000000 --- a/samples/Foundatio.HostingSample/Startup/OtherStartupAction.cs +++ /dev/null @@ -1,25 +0,0 @@ -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Extensions.Hosting.Startup; -using Microsoft.Extensions.Logging; - -namespace Foundatio.HostingSample; - -public class OtherStartupAction : IStartupAction -{ - private readonly ILogger _logger; - - public OtherStartupAction(ILogger logger) - { - _logger = logger; - } - - public async Task RunAsync(CancellationToken cancellationToken = default) - { - for (int i = 0; i < 5; i++) - { - _logger.LogTrace("OtherStartupAction Run Thread={ManagedThreadId}", Thread.CurrentThread.ManagedThreadId); - await Task.Delay(900); - } - } -} diff --git a/samples/Foundatio.HostingSample/appsettings.Development.json b/samples/Foundatio.HostingSample/appsettings.Development.json deleted file mode 100644 index a34cd70c5..000000000 --- a/samples/Foundatio.HostingSample/appsettings.Development.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Logging": { - "LogLevel": { - "Default": "Information", - "Microsoft.AspNetCore": "Warning" - } - } -} diff --git a/samples/Foundatio.HostingSample/appsettings.json b/samples/Foundatio.HostingSample/appsettings.json deleted file mode 100644 index 49bf8624f..000000000 --- a/samples/Foundatio.HostingSample/appsettings.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "Logging": { - "LogLevel": { - "Default": "Information", - "Microsoft.AspNetCore": "Warning", - "Foundatio": "Information" - } - }, - "AllowedHosts": "*" -} diff --git a/samples/Foundatio.MessagingSample/Foundatio.MessagingSample.csproj b/samples/Foundatio.MessagingSample/Foundatio.MessagingSample.csproj new file mode 100644 index 000000000..fb51537ff --- /dev/null +++ b/samples/Foundatio.MessagingSample/Foundatio.MessagingSample.csproj @@ -0,0 +1,17 @@ + + + + net10.0 + enable + enable + false + + + + + + + + + + diff --git a/samples/Foundatio.MessagingSample/Handlers.cs b/samples/Foundatio.MessagingSample/Handlers.cs new file mode 100644 index 000000000..51805ad65 --- /dev/null +++ b/samples/Foundatio.MessagingSample/Handlers.cs @@ -0,0 +1,33 @@ +using Foundatio.Messaging; + +namespace Foundatio.MessagingSample; + +/// A short per-process id so you can see which instance handled each message/job when scaled to replicas. +public sealed record InstanceInfo(string Id); + +/// +/// Handles orders. Registration carries no topology — orders arrive here because the endpoint calls +/// bus.SendAsync, and running instances compete for deliveries. Resolved from DI +/// per message; throwing would trigger retry/dead-letter. +/// +public sealed class ProcessOrderHandler(InstanceInfo instance, ILogger logger) : IMessageHandler +{ + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + logger.LogInformation("[{Instance}] processed order: {Quantity} x {Product}", instance.Id, context.Message.Quantity, context.Message.Product); + return Task.CompletedTask; + } +} + +/// +/// Handles announcements published via bus.PublishAsync. The service name supplies the durable subscription +/// identity; this service's replicas compete for each delivery. +/// +public sealed class AnnouncementHandler(InstanceInfo instance, ILogger logger) : IMessageHandler +{ + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + logger.LogInformation("[{Instance}] announcement: {Text}", instance.Id, context.Message.Text); + return Task.CompletedTask; + } +} diff --git a/samples/Foundatio.MessagingSample/Jobs.cs b/samples/Foundatio.MessagingSample/Jobs.cs new file mode 100644 index 000000000..cf23defaf --- /dev/null +++ b/samples/Foundatio.MessagingSample/Jobs.cs @@ -0,0 +1,64 @@ +using Foundatio.Jobs; + +namespace Foundatio.MessagingSample; + +/// Typed arguments for , serialized into the durable job payload. +public sealed record ReportArgs(string Format, string RequestedBy); + +/// +/// A durable, on-demand job (submitted via POST /reports with typed ). It runs on +/// whichever instance's job worker claims it, reads its arguments back with +/// context.GetArguments<ReportArgs>(), and reports progress through its +/// so GET /reports/{id} can observe it. +/// +public sealed class GenerateReportJob(InstanceInfo instance, ILogger logger) : IJob +{ + public async Task RunAsync(ReportArgs args, JobExecutionContext context) + { + logger.LogInformation("[{Instance}] generating {Format} report {JobId} for {RequestedBy}", instance.Id, args.Format, context.JobId, args.RequestedBy); + + for (int percent = 25; percent <= 100; percent += 25) + { + await Task.Delay(TimeSpan.FromMilliseconds(250), context.CancellationToken); + await context.ReportProgressAsync(percent, $"{percent}% complete", context.CancellationToken); + } + + return JobResult.Success; + } +} + +// The recurring (CRON) jobs below are scheduled in Program.cs. Every instance registers the same schedules, but each +// occurrence is materialized once into the shared runtime store, so scope decides how many instances run it: +// * Global (default) -> exactly ONE instance runs each tick (a leader/singleton task). +// * PerNode -> EVERY instance runs its own occurrence each tick (per-instance maintenance). +// A job uses its JobExecutionContext when it wants progress/heartbeat/identity, or ignores it (as these do). + +/// Global, every minute: a simple liveness heartbeat that runs on a single instance per tick. +public sealed class HeartbeatJob(InstanceInfo instance, ILogger logger) : IJob +{ + public Task RunAsync(JobExecutionContext context) + { + logger.LogInformation("[{Instance}] heartbeat {Time:HH:mm:ss} (one instance per tick)", instance.Id, DateTimeOffset.UtcNow); + return Task.FromResult(JobResult.Success); + } +} + +/// PerNode, every minute: each instance refreshes its own local state — so every instance runs this each tick. +public sealed class RefreshCacheJob(InstanceInfo instance, ILogger logger) : IJob +{ + public Task RunAsync(JobExecutionContext context) + { + logger.LogInformation("[{Instance}] refreshed local cache (every instance per tick)", instance.Id); + return Task.FromResult(JobResult.Success); + } +} + +/// Global, every 2 minutes: a periodic maintenance sweep that runs on a single instance per tick. +public sealed class SweepStaleOrdersJob(InstanceInfo instance, ILogger logger) : IJob +{ + public Task RunAsync(JobExecutionContext context) + { + logger.LogInformation("[{Instance}] swept stale orders (one instance per tick)", instance.Id); + return Task.FromResult(JobResult.Success); + } +} diff --git a/samples/Foundatio.MessagingSample/Messages.cs b/samples/Foundatio.MessagingSample/Messages.cs new file mode 100644 index 000000000..5701b2d27 --- /dev/null +++ b/samples/Foundatio.MessagingSample/Messages.cs @@ -0,0 +1,25 @@ +using Foundatio.Messaging; + +namespace Foundatio.MessagingSample; + +/// +/// A command / unit of work, delivered with bus.SendAsync — running instances compete for deliveries. Handlers must tolerate redelivery. +/// The names the destination ("orders"); without it the kebab-cased type name +/// ("process-order") is used. +/// +[MessageRoute("orders")] +public class ProcessOrder +{ + public string Product { get; set; } = ""; + public int Quantity { get; set; } = 1; +} + +/// +/// An event, delivered with bus.PublishAsync — each subscribing service receives one copy (and this sample's +/// handler uses the durable announcements group, whose replicas compete). +/// +[MessageRoute("announcements")] +public class Announcement +{ + public string Text { get; set; } = ""; +} diff --git a/samples/Foundatio.MessagingSample/Program.cs b/samples/Foundatio.MessagingSample/Program.cs new file mode 100644 index 000000000..fa45981e9 --- /dev/null +++ b/samples/Foundatio.MessagingSample/Program.cs @@ -0,0 +1,54 @@ +using Foundatio; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Foundatio.MessagingSample; + +var builder = WebApplication.CreateBuilder(args); + +// A short id so log lines make it obvious WHICH instance handled each message/job when scaled to multiple replicas. +builder.Services.AddSingleton(new InstanceInfo(Guid.NewGuid().ToString("N")[..6])); + +builder.Services.AddFoundatioWorker(foundatio => foundatio + .UseServiceName("messaging-sample") + .ConfigureMessaging(messaging => messaging.UseAws() + .AddConsumer() + .AddSubscriber()) + .ConfigureJobs(jobs => jobs.UseRedis() + .AddJobType("generate-report") + .AddCronJob("* * * * *") + .AddCronJob("* * * * *") + .AddCronJob("*/2 * * * *"))); + +var app = builder.Build(); +app.MapHealthChecks("/health"); + +app.MapGet("/", (InstanceInfo instance) => Results.Ok(new { service = "Foundatio messaging sample", instance = instance.Id })); + +// SEND — a command / unit of work: replicas compete to process each order (handled by ProcessOrderHandler). +app.MapPost("/orders", async (ProcessOrder order, IMessageBus bus) => + Results.Accepted(value: new { queued = await bus.SendAsync(order) })); + +// PUBLISH — one copy for the durable announcements group; its replicas compete. +app.MapPost("/announcements", async (Announcement announcement, IMessageBus bus) => +{ + await bus.PublishAsync(announcement); + return Results.Accepted(value: new { published = announcement.Text }); +}); + +// DURABLE JOB — submitted here with typed arguments (persisted in the job payload; the job reads them back with +// context.GetArguments()), executed on whichever instance's job worker claims it. +app.MapPost("/reports", async (IJobClient jobs) => +{ + var handle = await jobs.EnqueueAsync(new ReportArgs("pdf", "sample-user")); + return Results.Accepted($"/reports/{handle.JobId}", new { jobId = handle.JobId }); +}); + +app.MapGet("/reports/{id}", async (string id, IJobMonitor monitor) => +{ + var state = await monitor.GetAsync(id); + return state is null + ? Results.NotFound() + : Results.Ok(new { state.JobId, status = state.Status.ToString(), state.Progress, state.ProgressMessage }); +}); + +app.Run(); diff --git a/samples/Foundatio.MessagingSample/Properties/launchSettings.json b/samples/Foundatio.MessagingSample/Properties/launchSettings.json new file mode 100644 index 000000000..6c31385a1 --- /dev/null +++ b/samples/Foundatio.MessagingSample/Properties/launchSettings.json @@ -0,0 +1,12 @@ +{ + "$schema": "https://json.schemastore.org/launchsettings.json", + "profiles": { + "http": { + "commandName": "Project", + "applicationUrl": "http://localhost:5280", + "environmentVariables": { + "ASPNETCORE_ENVIRONMENT": "Development" + } + } + } +} diff --git a/samples/Foundatio.MessagingSample/README.md b/samples/Foundatio.MessagingSample/README.md new file mode 100644 index 000000000..384d7465f --- /dev/null +++ b/samples/Foundatio.MessagingSample/README.md @@ -0,0 +1,30 @@ +# Messaging sample + +An ASP.NET host with three Aspire replicas, SQS/SNS through LocalStack, and durable jobs on Redis. + +- `POST /orders` sends queued work to competing `ProcessOrderHandler` instances. +- `POST /announcements` publishes to the durable `announcements` subscription. Its replicas compete for each event. A different named subscription would receive another copy. +- `POST /reports` enqueues typed report arguments and returns a job ID; `GET /reports/{id}` shows progress. +- Global CRON schedules share one occurrence per tick; `PerNode` schedules create node-affine work for each replica. + +Delivery and execution are at least once. Production business operations must tolerate retries and duplicate delivery. + +[Program.cs](Program.cs) uses `AddFoundatioWorker(...)` to configure and host message consumers, jobs, schedules, and delayed dispatch together. API-only hosts use `AddFoundatio()` to register clients without starting workers. + +```powershell +dotnet run --project samples/Foundatio.AppHost +``` + +After opening the service endpoint from the Aspire dashboard: + +```powershell +$serviceUrl = 'https://localhost:' +1..6 | ForEach-Object { + Invoke-RestMethod "$serviceUrl/orders" -Method Post -ContentType application/json -Body (@{ product = 'widget'; quantity = $_ } | ConvertTo-Json) +} +Invoke-RestMethod "$serviceUrl/announcements" -Method Post -ContentType application/json -Body '{"text":"hello"}' +$job = Invoke-RestMethod "$serviceUrl/reports" -Method Post +Invoke-RestMethod "$serviceUrl/reports/$($job.jobId)" +``` + +For a runnable example without Redis, Docker, or AWS, use `dotnet run --project samples/Foundatio.QuickstartSample`. diff --git a/samples/Foundatio.MessagingSample/appsettings.json b/samples/Foundatio.MessagingSample/appsettings.json new file mode 100644 index 000000000..10f68b8c8 --- /dev/null +++ b/samples/Foundatio.MessagingSample/appsettings.json @@ -0,0 +1,9 @@ +{ + "Logging": { + "LogLevel": { + "Default": "Information", + "Microsoft.AspNetCore": "Warning" + } + }, + "AllowedHosts": "*" +} diff --git a/samples/Foundatio.QuickstartSample/Foundatio.QuickstartSample.csproj b/samples/Foundatio.QuickstartSample/Foundatio.QuickstartSample.csproj new file mode 100644 index 000000000..a4ec3e62e --- /dev/null +++ b/samples/Foundatio.QuickstartSample/Foundatio.QuickstartSample.csproj @@ -0,0 +1,20 @@ + + + + Exe + net10.0 + enable + enable + false + + + + + + + + + + + + diff --git a/samples/Foundatio.QuickstartSample/Handlers.cs b/samples/Foundatio.QuickstartSample/Handlers.cs new file mode 100644 index 000000000..bc703b0ba --- /dev/null +++ b/samples/Foundatio.QuickstartSample/Handlers.cs @@ -0,0 +1,33 @@ +using Foundatio.Messaging; +using Microsoft.Extensions.Logging; + +namespace Foundatio.QuickstartSample; + +/// +/// Handles the event. Registration carries no topology decision — this receives events +/// because Program.cs calls bus.PublishAsync. Resolved from DI in its own scope per message; throwing here +/// would trigger the core retry/dead-letter policy. +/// +public sealed class OrderPlacedHandler(ILogger logger, SampleActivity activity) : IMessageHandler +{ + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + logger.LogInformation("EVENT handled: order {OrderId} placed for {Product}", context.Message.OrderId, context.Message.Product); + activity.EventHandled.TrySetResult(); + return Task.CompletedTask; + } +} + +/// +/// Handles the command — competing consumers process queued work, because +/// Program.cs delivers it with bus.SendAsync. +/// +public sealed class SendReceiptHandler(ILogger logger, SampleActivity activity) : IMessageHandler +{ + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + logger.LogInformation("COMMAND handled: receipt for order {OrderId} sent to {Email}", context.Message.OrderId, context.Message.Email); + activity.CommandHandled.TrySetResult(); + return Task.CompletedTask; + } +} diff --git a/samples/Foundatio.QuickstartSample/Jobs.cs b/samples/Foundatio.QuickstartSample/Jobs.cs new file mode 100644 index 000000000..ac1cd3bcf --- /dev/null +++ b/samples/Foundatio.QuickstartSample/Jobs.cs @@ -0,0 +1,43 @@ +using Foundatio.Jobs; +using Microsoft.Extensions.Logging; + +namespace Foundatio.QuickstartSample; + +/// Typed arguments for , serialized into the durable job payload. +public sealed record ResizeArgs(string FileName, int Width, int Height); + +/// +/// A durable, on-demand job enqueued with jobs.EnqueueAsync<ResizeImageJob, ResizeArgs>(args). It reads +/// its typed arguments back with context.GetArguments<ResizeArgs>() and reports progress through the +/// runtime store as it works. +/// +public sealed class ResizeImageJob(ILogger logger) : IJob +{ + public async Task RunAsync(ResizeArgs args, JobExecutionContext context) + { + logger.LogInformation("JOB {JobId} started: resizing {FileName} to {Width}x{Height}", context.JobId, args.FileName, args.Width, args.Height); + + for (int percent = 25; percent <= 100; percent += 25) + { + await Task.Delay(TimeSpan.FromMilliseconds(200), context.CancellationToken); + await context.ReportProgressAsync(percent, $"{percent}% complete"); + logger.LogInformation("JOB {JobId} progress: {Percent}%", context.JobId, percent); + } + + return JobResult.SuccessWithMessage($"{args.FileName} resized to {args.Width}x{args.Height}"); + } +} + +/// +/// A recurring (CRON) job registered with AddCronJob<CleanupJob>("*/1 * * * *") in Program.cs — the +/// scheduler materializes a durable occurrence every minute and the job worker executes it. +/// +public sealed class CleanupJob(ILogger logger, SampleActivity activity) : IJob +{ + public Task RunAsync(JobExecutionContext context) + { + logger.LogInformation("CRON tick: cleanup ran at {Time:HH:mm:ss}", DateTimeOffset.Now); + activity.CleanupRan.TrySetResult(); + return Task.FromResult(JobResult.Success); + } +} diff --git a/samples/Foundatio.QuickstartSample/Messages.cs b/samples/Foundatio.QuickstartSample/Messages.cs new file mode 100644 index 000000000..4e7f393a4 --- /dev/null +++ b/samples/Foundatio.QuickstartSample/Messages.cs @@ -0,0 +1,12 @@ +namespace Foundatio.QuickstartSample; + +/// +/// An EVENT — published with bus.PublishAsync: every subscribing service receives one copy. +/// +public record OrderPlaced(int OrderId, string Product); + +/// +/// A COMMAND / unit of work — sent with bus.SendAsync: exactly one handler instance across the fleet +/// processes each one (competing consumers). +/// +public record SendReceipt(int OrderId, string Email); diff --git a/samples/Foundatio.QuickstartSample/Program.cs b/samples/Foundatio.QuickstartSample/Program.cs new file mode 100644 index 000000000..b288c8d02 --- /dev/null +++ b/samples/Foundatio.QuickstartSample/Program.cs @@ -0,0 +1,49 @@ +// Foundatio quickstart: messaging + durable jobs with ZERO external dependencies (everything in-memory). +// Just `dotnet run` — publish an event, send a command, run a durable job with typed args, and watch a CRON +// job tick once a minute. Swap UseInMemory() for UseRedis()/UseAws() to go to production without touching +// any handler or job code. +using Foundatio; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Foundatio.QuickstartSample; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; + +bool verify = args.Contains("--verify"); +var builder = Host.CreateApplicationBuilder(args.Where(arg => arg != "--verify").ToArray()); +builder.Services.AddSingleton(); + +builder.Services.AddFoundatioWorker(foundatio => foundatio + .UseServiceName("quickstart") + .ConfigureMessaging(messaging => messaging.UseInMemory() + .AddSubscriber() + .AddConsumer()) + .ConfigureJobs(jobs => jobs.UseInMemory() + .AddJobType("resize-image") + .AddCronJob("*/1 * * * *"))); + +using var host = builder.Build(); +await host.StartAsync(); // handlers attach and the job worker starts here + +var bus = host.Services.GetRequiredService(); +var jobs = host.Services.GetRequiredService(); + +// EVENT — every subscribing service receives a copy (OrderPlacedHandler logs it). +await bus.PublishAsync(new OrderPlaced(1001, "Espresso Machine")); + +// COMMAND — competing consumers process it (SendReceiptHandler logs it). +await bus.SendAsync(new SendReceipt(1001, "dev@example.com")); + +// DURABLE JOB with typed arguments — a worker claims it, the job reads the args back and reports progress. +var handle = await jobs.EnqueueAsync(new ResizeArgs("product-1001.png", 640, 480), new JobRequestOptions { Delay = TimeSpan.FromSeconds(1) }); +var completed = await handle.WaitForCompletionAsync(TimeSpan.FromSeconds(30)); +Console.WriteLine($"Resize completed: {completed.Status}"); +Console.WriteLine($"Enqueued ResizeImageJob {handle.JobId}; CleanupJob (CRON) ticks within a minute. Ctrl+C to exit."); + +if (verify) +{ + try { await SampleVerification.RunAsync(host, completed); } + finally { await host.StopAsync(); } +} +else + await host.WaitForShutdownAsync(); diff --git a/samples/Foundatio.QuickstartSample/SampleActivity.cs b/samples/Foundatio.QuickstartSample/SampleActivity.cs new file mode 100644 index 000000000..8fa88ab77 --- /dev/null +++ b/samples/Foundatio.QuickstartSample/SampleActivity.cs @@ -0,0 +1,8 @@ +namespace Foundatio.QuickstartSample; + +public sealed class SampleActivity +{ + public TaskCompletionSource EventHandled { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public TaskCompletionSource CommandHandled { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public TaskCompletionSource CleanupRan { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); +} diff --git a/samples/Foundatio.QuickstartSample/SampleVerification.cs b/samples/Foundatio.QuickstartSample/SampleVerification.cs new file mode 100644 index 000000000..ba2861944 --- /dev/null +++ b/samples/Foundatio.QuickstartSample/SampleVerification.cs @@ -0,0 +1,40 @@ +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; + +namespace Foundatio.QuickstartSample; + +internal static class SampleVerification +{ + public static async Task RunAsync(IHost host, JobState completed) + { + if (completed.Status != JobStatus.Completed || completed.Error is not null || completed.ResultMessage is null) + throw new InvalidOperationException("The delayed job did not complete successfully."); + + var activity = host.Services.GetRequiredService(); + await Task.WhenAll(activity.EventHandled.Task, activity.CommandHandled.Task, activity.CleanupRan.Task) + .WaitAsync(TimeSpan.FromSeconds(70)); + + var jobs = host.Services.GetRequiredService(); + var cancelled = await jobs.EnqueueAsync(new ResizeArgs("cancelled.png", 32, 32), + new JobRequestOptions { Delay = TimeSpan.FromHours(1) }); + await cancelled.RequestCancellationAsync(); + if ((await cancelled.WaitForCompletionAsync(TimeSpan.FromSeconds(5))).Status != JobStatus.Cancelled) + throw new InvalidOperationException("Cancellation was not persisted."); + + var services = new ServiceCollection(); + services.AddFoundatio().ConfigureMessaging(messaging => messaging.UseInMemory() + .AddMessageType("send-receipt.v1", queue: "receipts")); + await using var producer = services.BuildServiceProvider(); + if (producer.GetServices().Any()) + throw new InvalidOperationException("Producer registration started a hosted service."); + var bus = producer.GetRequiredService(); + await bus.SendAsync(new SendReceipt(2002, "producer@example.com")); + await using var delivery = await bus.ReceiveAsync(); + if (delivery?.Message.OrderId != 2002) + throw new InvalidOperationException("The producer-only message was not available."); + await delivery.CompleteAsync(); + Console.WriteLine("Verified: producer-only registration, command, durable subscriber, delayed job, cancellation, and automatic CRON execution."); + } +} diff --git a/src/Foundatio.Aws/AwsFoundatioBuilderExtensions.cs b/src/Foundatio.Aws/AwsFoundatioBuilderExtensions.cs new file mode 100644 index 000000000..871a4d02c --- /dev/null +++ b/src/Foundatio.Aws/AwsFoundatioBuilderExtensions.cs @@ -0,0 +1,46 @@ +using System; +using Amazon; +using Amazon.Runtime; +using Foundatio.Messaging; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; + +namespace Foundatio; + +public static class AwsFoundatioBuilderExtensions +{ + /// + /// Runs messaging (queues on SQS, pub/sub on SNS+SQS) over AWS. With no arguments it relies on the AWS SDK's default + /// region and credential resolution; common settings (ServiceUrl, Region, ResourcePrefix, AccessKey/SecretKey) are + /// also bound from an "Aws" configuration section when present, and can override + /// anything. Point ServiceUrl at LocalStack to run without a cloud account. + /// + public static FoundatioBuilder.MessagingBuilder UseAws(this FoundatioBuilder.MessagingBuilder builder, Action? configure = null) + { + return builder.UseTransport(sp => + { + var options = new AwsMessageTransportOptions { LoggerFactory = sp.GetService() }; + BindFromConfiguration(options, sp.GetService()?.GetSection("Aws")); + configure?.Invoke(options); + return new AwsMessageTransport(options); + }); + } + + private static void BindFromConfiguration(AwsMessageTransportOptions options, IConfiguration? section) + { + if (section is null) + return; + + if (section["ServiceUrl"] is { Length: > 0 } serviceUrl) + options.ServiceUrl = serviceUrl; + + if (section["Region"] is { Length: > 0 } region) + options.Region = RegionEndpoint.GetBySystemName(region); + + if (section["ResourcePrefix"] is { Length: > 0 } prefix) + options.ResourcePrefix = prefix; + + if (section["AccessKey"] is { Length: > 0 } accessKey && section["SecretKey"] is { Length: > 0 } secretKey) + options.Credentials = new BasicAWSCredentials(accessKey, secretKey); + } +} diff --git a/src/Foundatio.Aws/AwsMessageTransport.Administration.cs b/src/Foundatio.Aws/AwsMessageTransport.Administration.cs new file mode 100644 index 000000000..84d67c473 --- /dev/null +++ b/src/Foundatio.Aws/AwsMessageTransport.Administration.cs @@ -0,0 +1,60 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Amazon.SQS.Model; + +namespace Foundatio.Messaging; + +public sealed partial class AwsMessageTransport +{ + /// Validates existence and declared SQS attributes, reporting all mismatches together. + public async Task ValidateAsync(IReadOnlyList declarations, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(declarations); + var problems = new List(); + foreach (var declaration in declarations) + { + if (!await ExistsAsync(declaration.Address, cancellationToken).ConfigureAwait(false)) + { + problems.Add($"Destination '{declaration.Address.Key}' does not exist."); + continue; + } + if (declaration.Address.Role != DestinationRole.Queue) continue; + ValidateQueueArguments(declaration.ProviderArguments); + string url = await ResolveQueueUrlAsync(declaration.Address, cancellationToken).ConfigureAwait(false); + var response = await _sqs.Value.GetQueueAttributesAsync(new GetQueueAttributesRequest + { + QueueUrl = url, + AttributeNames = ["All"] + }, cancellationToken).ConfigureAwait(false); + foreach (var expected in declaration.ProviderArguments ?? new Dictionary()) + { + response.Attributes.TryGetValue(expected.Key, out var actual); + if (!String.Equals(expected.Value, actual, StringComparison.Ordinal)) + problems.Add($"Queue '{declaration.Address.Name}' attribute {expected.Key} is '{actual ?? "(unset)"}', expected '{expected.Value}'."); + } + if (response.Attributes.TryGetValue("RedrivePolicy", out string? redrive) && !String.IsNullOrEmpty(redrive)) + problems.Add($"Queue '{declaration.Address.Name}' has a broker RedrivePolicy. Foundatio owns delivery attempts; remove the broker policy."); + } + if (problems.Count > 0) throw new InvalidOperationException(String.Join(Environment.NewLine, problems)); + } + + private static void ValidateQueueArguments(IReadOnlyDictionary? arguments) + { + if (arguments is null) return; + if (arguments.TryGetValue("RedrivePolicy", out var policy) && !String.IsNullOrEmpty(policy)) + throw new ArgumentException("Foundatio owns retry and dead-letter policy; do not configure a broker RedrivePolicy.", nameof(arguments)); + foreach (var argument in arguments) + { + if (argument.Key is not ("VisibilityTimeout" or "MessageRetentionPeriod")) continue; + int min = argument.Key == "VisibilityTimeout" ? 1 : 60; + int max = argument.Key == "VisibilityTimeout" ? 43200 : 1209600; + if (!Int32.TryParse(argument.Value, NumberStyles.Integer, CultureInfo.InvariantCulture, out int value) || value < min || value > max) + throw new ArgumentException($"SQS {argument.Key} must be between {min} and {max} seconds.", nameof(arguments)); + } + } + +} diff --git a/src/Foundatio.Aws/AwsMessageTransport.AutomaticBatching.cs b/src/Foundatio.Aws/AwsMessageTransport.AutomaticBatching.cs new file mode 100644 index 000000000..94b29fe83 --- /dev/null +++ b/src/Foundatio.Aws/AwsMessageTransport.AutomaticBatching.cs @@ -0,0 +1,84 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using Amazon.SQS.Model; + +namespace Foundatio.Messaging; + +public sealed partial class AwsMessageTransport +{ + private readonly object _batchersLock = new(); + private readonly Dictionary> _sendBatchers = new(StringComparer.Ordinal); + private readonly Dictionary> _deleteBatchers = new(StringComparer.Ordinal); + + private AwsRequestBatcher GetSendBatcher(bool topic, string address, int maximumBytes) + { + string key = (topic ? "sns:" : "sqs:") + address; + lock (_batchersLock) + { + ThrowIfDisposed(); + if (_sendBatchers.TryGetValue(key, out var batcher)) + return batcher; + batcher = new AwsRequestBatcher(_options, maximumBytes, static message => message.Bytes, + (batch, ct) => SendPreparedBatchAsync(topic, address, batch, ct), delayWhenIdle: topic); + _sendBatchers.Add(key, batcher); + return batcher; + } + } + + private AwsRequestBatcher GetDeleteBatcher(string queueUrl) + { + lock (_batchersLock) + { + ThrowIfDisposed(); + if (_deleteBatchers.TryGetValue(queueUrl, out var batcher)) + return batcher; + batcher = new AwsRequestBatcher(_options, Int32.MaxValue, static _ => 0, + (receipts, ct) => DeleteBatchAsync(queueUrl, receipts, ct)); + _deleteBatchers.Add(queueUrl, batcher); + return batcher; + } + } + + private async Task DeleteBatchAsync(string queueUrl, IReadOnlyList receipts, CancellationToken ct) + { + var entries = new List(receipts.Count); + for (int i = 0; i < receipts.Count; i++) + entries.Add(new DeleteMessageBatchRequestEntry(i.ToString(CultureInfo.InvariantCulture), receipts[i])); + var response = await _sqs.Value.DeleteMessageBatchAsync(new DeleteMessageBatchRequest { QueueUrl = queueUrl, Entries = entries }, ct).ConfigureAwait(false); + var results = new Exception?[receipts.Count]; + var seen = new bool[receipts.Count]; + foreach (var entry in response.Successful ?? []) + MarkSeen(entry.Id); + foreach (var entry in response.Failed ?? []) + results[MarkSeen(entry.Id)] = new MessageBusException($"SQS did not acknowledge deletion ({entry.Code}): {entry.Message}"); + for (int i = 0; i < receipts.Count; i++) + if (!seen[i]) results[i] = new MessageBusException("SQS did not return an acknowledgement for this receipt."); + return results; + + int MarkSeen(string id) + { + if (!Int32.TryParse(id, CultureInfo.InvariantCulture, out int index) || index < 0 || index >= receipts.Count || seen[index]) + throw new MessageBusException("SQS returned an invalid or duplicate acknowledgement ID."); + seen[index] = true; + return index; + } + } + + private Task DisposeBatchersAsync() + { + var tasks = new List(); + lock (_batchersLock) + { + foreach (var batcher in _sendBatchers.Values) + tasks.Add(batcher.DisposeAsync().AsTask()); + foreach (var batcher in _deleteBatchers.Values) + tasks.Add(batcher.DisposeAsync().AsTask()); + _sendBatchers.Clear(); + _deleteBatchers.Clear(); + } + return Task.WhenAll(tasks); + } +} diff --git a/src/Foundatio.Aws/AwsMessageTransport.Batching.cs b/src/Foundatio.Aws/AwsMessageTransport.Batching.cs new file mode 100644 index 000000000..98bdd9719 --- /dev/null +++ b/src/Foundatio.Aws/AwsMessageTransport.Batching.cs @@ -0,0 +1,178 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Net; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Amazon.Runtime; +using Amazon.SQS.Model; +using Amazon.SimpleNotificationService.Model; +using SnsAttribute = Amazon.SimpleNotificationService.Model.MessageAttributeValue; +using SqsAttribute = Amazon.SQS.Model.MessageAttributeValue; + +namespace Foundatio.Messaging; + +public sealed partial class AwsMessageTransport +{ + private sealed record PreparedMessage(int Index, string Body, string Envelope, MessageHeaders Headers, int Bytes, DateTimeOffset? DeliverAt); + + public async Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + ArgumentNullException.ThrowIfNull(messages); + ArgumentNullException.ThrowIfNull(options); + ct.ThrowIfCancellationRequested(); + bool topic = destination.Role == DestinationRole.Topic; + if (topic && options.DeliverAt > DateTimeOffset.UtcNow) + throw new NotSupportedException("SNS cannot delay publication. Configure Messaging.UseSchedulingStore(...)."); + int maximumBytes = topic ? 262144 : 1048576; + if (messages.Count == 1 && _options.EnableBatching) + return await SendSingleAsync(destination, PrepareMessage(0, messages[0], options.DeliverAt), topic, maximumBytes, ct).ConfigureAwait(false); + + var results = new SendItemResult[messages.Count]; + var prepared = new List(messages.Count); + for (int index = 0; index < messages.Count; index++) + { + results[index] = new SendItemResult { Index = index, Status = MessageSendStatus.NotAttempted }; + var entry = PrepareMessage(index, messages[index], options.DeliverAt); + if (entry.Bytes > maximumBytes) + { + results[index] = MessageTooLarge(index, maximumBytes); + continue; + } + prepared.Add(entry); + } + if (prepared.Count == 0) return new SendResult { Items = results }; + string address = topic ? await ResolveTopicArnAsync(destination.Name, ct).ConfigureAwait(false) : await ResolveQueueUrlAsync(destination, ct).ConfigureAwait(false); + for (int offset = 0; offset < prepared.Count;) + { + var batch = new List(_options.MaxBatchSize); + int bytes = 0; + while (offset < prepared.Count && batch.Count < _options.MaxBatchSize && bytes + prepared[offset].Bytes <= maximumBytes) + { + var entry = prepared[offset++]; + batch.Add(entry); + bytes += entry.Bytes; + } + if (ct.IsCancellationRequested) break; + try + { + var response = await SendPreparedBatchAsync(topic, address, batch, ct).ConfigureAwait(false); + for (int i = 0; i < batch.Count; i++) + results[batch[i].Index] = response[i].Index == batch[i].Index ? response[i] : response[i] with { Index = batch[i].Index }; + } + catch (Exception ex) + { + InvalidateAddress(destination, ex); + foreach (var entry in batch) + results[entry.Index] = SendFailure(entry.Index, ex); + break; + } + } + return new SendResult { Items = results }; + } + + private async Task SendSingleAsync(DestinationAddress destination, PreparedMessage message, bool topic, int maximumBytes, CancellationToken ct) + { + if (message.Bytes > maximumBytes) + return new SendResult { Items = [MessageTooLarge(0, maximumBytes)] }; + + string address = topic ? await ResolveTopicArnAsync(destination.Name, ct).ConfigureAwait(false) : await ResolveQueueUrlAsync(destination, ct).ConfigureAwait(false); + if (ct.IsCancellationRequested) + return new SendResult { Items = [new SendItemResult { Index = 0, Status = MessageSendStatus.NotAttempted }] }; + + SendItemResult result; + try + { + result = await GetSendBatcher(topic, address, maximumBytes).ExecuteAsync(message, ct).ConfigureAwait(false); + if (result.Index != 0) result = result with { Index = 0 }; + } + catch (Exception ex) + { + InvalidateAddress(destination, ex); + result = SendFailure(0, ex); + } + return new SendResult { Items = [result] }; + } + + private static SendItemResult MessageTooLarge(int index, int maximumBytes) => new() + { + Index = index, + Status = MessageSendStatus.Rejected, + ErrorCode = "MessageTooLarge", + ErrorMessage = $"Encoded message and attributes exceed {maximumBytes} bytes.", + Retryable = false + }; + + private void InvalidateAddress(DestinationAddress destination, Exception exception) + { + if (exception is QueueDoesNotExistException) _queueUrls.TryRemove(destination.Key, out _); + if (exception is Amazon.SimpleNotificationService.Model.NotFoundException) _topicArns.TryRemove(destination.Name, out _); + } + + private static SendItemResult SendFailure(int index, Exception exception) + { + bool rejected = exception is AmazonServiceException aws && aws.StatusCode is >= HttpStatusCode.BadRequest and < HttpStatusCode.InternalServerError; + return new SendItemResult + { + Index = index, + Status = rejected ? MessageSendStatus.Rejected : MessageSendStatus.Unknown, + ErrorCode = (exception as AmazonServiceException)?.ErrorCode ?? exception.GetType().Name, + ErrorMessage = exception.Message.Length > 1024 ? exception.Message[..1024] : exception.Message, + Retryable = exception is OperationCanceledException ? null : !rejected || (exception as AmazonServiceException)?.ErrorCode?.Contains("Throttl", StringComparison.OrdinalIgnoreCase) == true + }; + } + + private async Task SendPreparedBatchAsync(bool topic, string address, IReadOnlyList batch, CancellationToken ct) + { + var results = new SendItemResult[batch.Count]; + if (topic) + { + var entries = new List(batch.Count); + for (int i = 0; i < batch.Count; i++) + { + var attributes = BuildAttributes(batch[i], static value => new SnsAttribute { DataType = "String", StringValue = value }); + entries.Add(new PublishBatchRequestEntry { Id = i.ToString(CultureInfo.InvariantCulture), Message = batch[i].Body, MessageAttributes = attributes }); + } + var response = await _sns.Value.PublishBatchAsync(new PublishBatchRequest { TopicArn = address, PublishBatchRequestEntries = entries }, ct).ConfigureAwait(false); + foreach (var success in response.Successful ?? []) + SetOutcome(results, success.Id, MessageSendStatus.Accepted, success.MessageId); + foreach (var failure in response.Failed ?? []) + SetOutcome(results, failure.Id, MessageSendStatus.Rejected, null, failure.Code, failure.Message, failure.SenderFault is { } senderFault ? !senderFault : null); + } + else + { + var entries = new List(batch.Count); + for (int i = 0; i < batch.Count; i++) + { + var attributes = BuildAttributes(batch[i], static value => new SqsAttribute { DataType = "String", StringValue = value }); + entries.Add(new SendMessageBatchRequestEntry + { + Id = i.ToString(CultureInfo.InvariantCulture), + MessageBody = batch[i].Body, + DelaySeconds = ToDelaySeconds(batch[i].DeliverAt), + MessageAttributes = attributes + }); + } + var response = await _sqs.Value.SendMessageBatchAsync(new SendMessageBatchRequest { QueueUrl = address, Entries = entries }, ct).ConfigureAwait(false); + foreach (var success in response.Successful ?? []) + SetOutcome(results, success.Id, MessageSendStatus.Accepted, success.MessageId); + foreach (var failure in response.Failed ?? []) + SetOutcome(results, failure.Id, MessageSendStatus.Rejected, null, failure.Code, failure.Message, failure.SenderFault is { } senderFault ? !senderFault : null); + } + for (int i = 0; i < results.Length; i++) + results[i] ??= new SendItemResult { Index = i, Status = MessageSendStatus.Unknown }; + return results; + } + + private static void SetOutcome(SendItemResult[] results, string id, MessageSendStatus status, string? messageId, string? code = null, string? error = null, bool? retryable = null) + { + if (!Int32.TryParse(id, CultureInfo.InvariantCulture, out int index) || index < 0 || index >= results.Length) + throw new MessageBusException("AWS returned an unknown batch entry ID."); + if (results[index] is not null) + throw new MessageBusException("AWS returned a duplicate batch entry ID."); + results[index] = new SendItemResult { Index = index, Status = status, MessageId = messageId, ErrorCode = code, ErrorMessage = error, Retryable = retryable }; + } +} diff --git a/src/Foundatio.Aws/AwsMessageTransport.NodeSubscriptions.cs b/src/Foundatio.Aws/AwsMessageTransport.NodeSubscriptions.cs new file mode 100644 index 000000000..5bbd477f6 --- /dev/null +++ b/src/Foundatio.Aws/AwsMessageTransport.NodeSubscriptions.cs @@ -0,0 +1,141 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using Amazon.SQS.Model; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Foundatio.Messaging; + +public sealed partial class AwsMessageTransport +{ + private const string NodeRoleTag = "fnd:role"; + private const string NodeTopicTag = "fnd:topic"; + private const string NodeHeartbeatTag = "fnd:heartbeat"; + + /// Owns a tagged SQS subscription for one node; active nodes renew it and startup reaps stale peers. + public async Task OpenNodeSubscriptionAsync(MessageNodeSubscriptionOptions options, CancellationToken cancellationToken = default) + { + options.Validate(); + ArgumentOutOfRangeException.ThrowIfLessThan(options.MessageRetention, TimeSpan.FromMinutes(1)); + ArgumentOutOfRangeException.ThrowIfGreaterThan(options.MessageRetention, TimeSpan.FromDays(14)); + // A fresh identity avoids queue-deleted-recently failures and prevents two processes adopting one receipt namespace. + var source = DestinationAddress.ForSubscription(options.Topic, $"node-{options.NodeId}-{Guid.NewGuid():N}"); + await EnsureSubscriptionAsync(source, cancellationToken).ConfigureAwait(false); + string url = await ResolveQueueUrlAsync(source, cancellationToken).ConfigureAwait(false); + try + { + await _sqs.Value.SetQueueAttributesAsync(new SetQueueAttributesRequest + { + QueueUrl = url, + Attributes = new Dictionary + { + ["MessageRetentionPeriod"] = Math.Ceiling(options.MessageRetention.TotalSeconds).ToString(CultureInfo.InvariantCulture) + } + }, cancellationToken).ConfigureAwait(false); + await TagNodeAsync(url, options.Topic, cancellationToken).ConfigureAwait(false); + await SweepNodesAsync(options.Topic, options.StaleAfter, cancellationToken).ConfigureAwait(false); + return new NodeOwner(this, source, url, options); + } + catch + { + using var cleanup = new CancellationTokenSource(TimeSpan.FromSeconds(10)); + try { await DeleteAsync(source, cleanup.Token).ConfigureAwait(false); } catch { } + throw; + } + } + + private Task TagNodeAsync(string url, string topic, CancellationToken cancellationToken) + => _sqs.Value.TagQueueAsync(new TagQueueRequest + { + QueueUrl = url, + Tags = new Dictionary + { + [NodeRoleTag] = "node-subscription", + [NodeTopicTag] = topic, + [NodeHeartbeatTag] = DateTimeOffset.UtcNow.ToString("O", CultureInfo.InvariantCulture) + } + }, cancellationToken); + + private async Task SweepNodesAsync(string topic, TimeSpan staleAfter, CancellationToken cancellationToken) + { + string physicalPrefix = SanitizeResourceName(_options.ResourcePrefix + topic + "/node-"); + physicalPrefix = physicalPrefix[..Math.Min(32, physicalPrefix.Length)]; + string? cursor = null; + do + { + var page = await _sqs.Value.ListQueuesAsync(new ListQueuesRequest + { + QueueNamePrefix = physicalPrefix, + MaxResults = 100, + NextToken = cursor + }, cancellationToken).ConfigureAwait(false); + foreach (string url in page.QueueUrls ?? []) + { + try + { + var tags = (await _sqs.Value.ListQueueTagsAsync(new ListQueueTagsRequest { QueueUrl = url }, cancellationToken).ConfigureAwait(false)).Tags; + if (tags is null || !tags.TryGetValue(NodeRoleTag, out var role) || role != "node-subscription" + || !tags.TryGetValue(NodeTopicTag, out var ownedTopic) || ownedTopic != topic + || !tags.TryGetValue(NodeHeartbeatTag, out var heartbeat) + || !DateTimeOffset.TryParse(heartbeat, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind, out var lastSeen) + || lastSeen >= DateTimeOffset.UtcNow - staleAfter) continue; + // Re-read ownership immediately before deletion; a peer may have resumed during the scan. + var current = (await _sqs.Value.ListQueueTagsAsync(new ListQueueTagsRequest { QueueUrl = url }, cancellationToken).ConfigureAwait(false)).Tags; + if (current is null || !current.TryGetValue(NodeHeartbeatTag, out var value) || value != heartbeat) continue; + var attributes = await _sqs.Value.GetQueueAttributesAsync(new GetQueueAttributesRequest { QueueUrl = url, AttributeNames = ["QueueArn"] }, cancellationToken).ConfigureAwait(false); + string? topicArn = await FindTopicArnAsync(topic, cancellationToken).ConfigureAwait(false); + if (topicArn is not null && await FindSubscriptionArnAsync(topicArn, attributes.QueueARN, cancellationToken).ConfigureAwait(false) is { } arn) + await _sns.Value.UnsubscribeAsync(arn, cancellationToken).ConfigureAwait(false); + await _sqs.Value.DeleteQueueAsync(url, cancellationToken).ConfigureAwait(false); + } + catch (QueueDoesNotExistException) { } + } + cursor = page.NextToken; + } while (!String.IsNullOrEmpty(cursor)); + } + + private sealed class NodeOwner : IManagedNodeSubscription + { + private readonly AwsMessageTransport _owner; + private readonly CancellationTokenSource _stop = new(); + private readonly Task _heartbeat; + private readonly ILogger _logger; + private int _disposed; + public DestinationAddress Source { get; } + + public NodeOwner(AwsMessageTransport owner, DestinationAddress source, string url, MessageNodeSubscriptionOptions options) + { + _owner = owner; + _logger = owner._options.LoggerFactory?.CreateLogger() ?? NullLogger.Instance; + Source = source; + _heartbeat = HeartbeatAsync(url, options); + } + + private async Task HeartbeatAsync(string url, MessageNodeSubscriptionOptions options) + { + while (!_stop.IsCancellationRequested) + { + try + { + await Task.Delay(options.HeartbeatInterval, _stop.Token).ConfigureAwait(false); + await _owner.TagNodeAsync(url, options.Topic, _stop.Token).ConfigureAwait(false); + } + catch (OperationCanceledException) when (_stop.IsCancellationRequested) { return; } + catch (Exception exception) { _logger.LogWarning(exception, "Unable to renew node subscription {Source}; retrying on the next heartbeat", Source); } + } + } + + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _disposed, 1) != 0) return; + await _stop.CancelAsync().ConfigureAwait(false); + await _heartbeat.ConfigureAwait(false); + using var cleanup = new CancellationTokenSource(TimeSpan.FromSeconds(10)); + try { await _owner.DeleteAsync(Source, cleanup.Token).ConfigureAwait(false); } + finally { _stop.Dispose(); } + } + } +} diff --git a/src/Foundatio.Aws/AwsMessageTransport.cs b/src/Foundatio.Aws/AwsMessageTransport.cs new file mode 100644 index 000000000..054b30702 --- /dev/null +++ b/src/Foundatio.Aws/AwsMessageTransport.cs @@ -0,0 +1,693 @@ +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Linq; +using System.Security.Cryptography; +using System.Text.Json; +using System.Text; +using System.Threading.Tasks; +using System.Threading; +using System; +using Amazon.SQS.Model; +using Amazon.SQS; +using Amazon.SimpleNotificationService.Model; +using Amazon.SimpleNotificationService; +using SnsMessageAttributeValue = Amazon.SimpleNotificationService.Model.MessageAttributeValue; +using SqsMessage = Amazon.SQS.Model.Message; +using SqsMessageAttributeValue = Amazon.SQS.Model.MessageAttributeValue; + +namespace Foundatio.Messaging; + +/// +/// An over AWS SQS (queues + competing-consumer subscriptions) and SNS (topics). This +/// is a temporary in-repo provider used to validate the redesigned transport contract against a real broker. Queue and +/// subscription destinations are SQS queues; topic destinations are SNS topics fanned out to SQS subscription queues. +/// +/// +/// Capability mapping: pull receive (SQS long poll), visibility timeout, redelivery delay (ChangeMessageVisibility, +/// 12h cap), delayed delivery on queues only (SQS DelaySeconds, 15-minute cap — SNS topics have no native delay, so +/// delayed publishes route through the runtime-store fallback), provisioning, and stats. SQS has no per-message +/// priority, per-message TTL, or push delivery, and no transport-native dead-letter that the core controls the timing +/// of, so those capabilities are intentionally not implemented (the core owns retry/dead-lettering). +/// +public sealed partial class AwsMessageTransport : IMessageTransport, ISupportsPull, ISupportsVisibilityTimeout, + ISupportsManagedNodeSubscriptions, ISupportsLockRenewal, ISupportsRedeliveryDelay, ISupportsProvisioning, ISupportsStats, ITransportInfo +{ + private const string EnvelopeAttributeName = "fnd.envelope"; + private const string HeadersAttributeName = "fnd.headers"; + private const string EncodingAttributeName = "fnd.encoding"; + private const string MessageIdAttributeName = "fnd.id"; + private const string ContentTypeAttributeName = "fnd.content_type"; + + private static readonly IReadOnlySet _supportedRoles = + new HashSet { DestinationRole.Queue, DestinationRole.Topic, DestinationRole.Subscription, DestinationRole.Binding }; + + private readonly AwsMessageTransportOptions _options; + private readonly string[] _nativeMessageHeaders; + private readonly Lazy _sqs; + private readonly Lazy _sns; + private readonly ConcurrentDictionary _queueUrls = new(StringComparer.Ordinal); + private readonly ConcurrentDictionary _topicArns = new(StringComparer.Ordinal); + private int _isDisposed; + private readonly bool _ownsSqs = true; + private readonly bool _ownsSns = true; + + public AwsMessageTransport(AwsMessageTransportOptions options) + { + _options = options ?? throw new ArgumentNullException(nameof(options)); + options.Validate(); + _nativeMessageHeaders = options.NativeMessageHeaders.ToArray(); + _sqs = new Lazy(CreateSqsClient); + _sns = new Lazy(CreateSnsClient); + } + + /// Uses caller-owned SDK clients, allowing shared connection configuration and deterministic tests. + public AwsMessageTransport(AwsMessageTransportOptions options, IAmazonSQS sqs, IAmazonSimpleNotificationService sns) : this(options) + { + ArgumentNullException.ThrowIfNull(sqs); + ArgumentNullException.ThrowIfNull(sns); + _sqs = new Lazy(() => sqs); + _sns = new Lazy(() => sns); + _ownsSqs = false; + _ownsSns = false; + } + + /// Uses a caller-owned SQS client; SNS is created lazily only when needed. + public AwsMessageTransport(AwsMessageTransportOptions options, IAmazonSQS sqs) : this(options) + { + ArgumentNullException.ThrowIfNull(sqs); + _sqs = new Lazy(() => sqs); + _ownsSqs = false; + } + + public AwsMessageTransport(string connectionString) : this(AwsMessageTransportOptions.FromConnectionString(connectionString)) { } + + // Capabilities differ by role: SQS queues take a native DelaySeconds (15-minute cap), SNS topics have no native + // delay at all — a delayed publish must route through the runtime-store fallback, never silently drop the delay. + // SQS accepts up to 1 MiB; SNS accepts up to 256 KiB, including message attributes. + private static readonly TransportCapabilities _queueCapabilities = new() + { + DelayedDelivery = true, + MaxDeliveryDelay = TimeSpan.FromMinutes(15), // SQS DelaySeconds maximum + MaxMessageBytes = 1048576, + MaxBatchSize = 10, + MaxReceiveBatchSize = 10, + MaxConcurrentReceives = 4, + ReceiveBatchDelay = TimeSpan.FromMilliseconds(1) + }; + + private static readonly TransportCapabilities _topicCapabilities = new() + { + MaxMessageBytes = 262144, + MaxBatchSize = 10 + }; + + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => _supportedRoles; + + public TransportCapabilities GetCapabilities(DestinationAddress destination) => + destination.Role == DestinationRole.Topic ? _topicCapabilities : _queueCapabilities; + + public TimeSpan? MaxRedeliveryDelay => TimeSpan.FromHours(12); // SQS ChangeMessageVisibility maximum + public TimeSpan? MaxVisibilityTimeout => TimeSpan.FromHours(12); // SQS visibility maximum + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + { + return ReceiveAsync(source, request, _options.DefaultVisibilityTimeout, ct); + } + + public async Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, TimeSpan visibility, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(source); + ArgumentNullException.ThrowIfNull(request); + + string queueUrl = await ResolveQueueUrlAsync(source, ct).ConfigureAwait(false); + + var sqsRequest = new ReceiveMessageRequest + { + QueueUrl = queueUrl, + MaxNumberOfMessages = Math.Clamp(request.MaxMessages <= 0 ? 1 : request.MaxMessages, 1, 10), + VisibilityTimeout = (int)Math.Clamp(visibility.TotalSeconds, 0, 43200), + MessageAttributeNames = ["All"], + MessageSystemAttributeNames = ["ApproximateReceiveCount", "SentTimestamp"] + }; + if (request.MaxWaitTime is { } wait) + sqsRequest.WaitTimeSeconds = (int)Math.Clamp(wait.TotalSeconds, 0, 20); + + var receiveStarted = DateTimeOffset.UtcNow; + ReceiveMessageResponse response; + try { response = await _sqs.Value.ReceiveMessageAsync(sqsRequest, ct).ConfigureAwait(false); } + catch (QueueDoesNotExistException ex) + { + _queueUrls.TryRemove(source.Key, out _); + throw new MessageDestinationNotFoundException(source, ex); + } + if (response.Messages is not { Count: > 0 }) + return []; + + if (_options.EnableBatching) + GetDeleteBatcher(queueUrl).ObserveBatchSize(sqsRequest.MaxNumberOfMessages.GetValueOrDefault(1)); + + var entries = new List(response.Messages.Count); + foreach (var message in response.Messages) + { + ReadOnlyMemory body; + MessageHeaders headers; + string? applicationMessageId = GetAttribute(message.MessageAttributes, MessageIdAttributeName); + string? contentType = GetAttribute(message.MessageAttributes, ContentTypeAttributeName); + Exception? envelopeError = null; + try + { + string encodedBody = message.Body ?? throw new FormatException("Missing message body."); + if (GetAttribute(message.MessageAttributes, EnvelopeAttributeName) is { } envelopeJson) + { + var envelope = JsonSerializer.Deserialize(envelopeJson); + if (envelope is null || envelope.Version != 1 || envelope.Encoding is not ("text" or "base64") || envelope.Headers is null) + throw new FormatException("Invalid or unsupported Foundatio AWS envelope."); + body = DecodeBody(encodedBody, envelope.Encoding); + headers = MessageHeaders.Create(envelope.Headers); + applicationMessageId = envelope.MessageId; + contentType = envelope.ContentType; + } + else + { + body = DecodeBody(encodedBody, GetAttribute(message.MessageAttributes, EncodingAttributeName)); + headers = FromSqsAttributes(message.MessageAttributes); + } + } + catch (Exception ex) when (ex is FormatException or JsonException or ArgumentException) + { + envelopeError = ex; + body = Encoding.UTF8.GetBytes(message.Body ?? ""); + headers = MessageHeaders.Create(new Dictionary + { + ["transport.raw.attributes"] = JsonSerializer.Serialize(message.MessageAttributes) + }); + } + entries.Add(new TransportEntry + { + Id = message.MessageId, + ApplicationMessageId = applicationMessageId, + ContentType = contentType, + Destination = source, + LockExpiresUtc = receiveStarted.AddSeconds(sqsRequest.VisibilityTimeout.GetValueOrDefault()), + EnqueuedUtc = message.Attributes is not null && message.Attributes.TryGetValue("SentTimestamp", out var timestamp) && Int64.TryParse(timestamp, out long milliseconds) + ? DateTimeOffset.FromUnixTimeMilliseconds(milliseconds) : null, + Body = body, + Headers = headers, + EnvelopeError = envelopeError, + DeliveryCount = GetReceiveCount(message), + Receipt = new Receipt { TransportState = message.ReceiptHandle } + }); + } + + return entries; + } + + public async Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) + { + ThrowIfDisposed(); + string queueUrl = await ResolveQueueUrlAsync(entry.Destination, ct).ConfigureAwait(false); + string receipt = GetReceiptHandle(entry); + if (!_options.EnableBatching) + { + await _sqs.Value.DeleteMessageAsync(queueUrl, receipt, ct).ConfigureAwait(false); + return; + } + var error = await GetDeleteBatcher(queueUrl).ExecuteAsync(receipt, ct).ConfigureAwait(false); + if (error is not null) + throw error; + } + + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) + { + return AbandonAsync(entry, TimeSpan.Zero, ct); + } + + public async Task AbandonAsync(TransportEntry entry, TimeSpan redeliveryDelay, CancellationToken ct) + { + ThrowIfDisposed(); + string queueUrl = await ResolveQueueUrlAsync(entry.Destination, ct).ConfigureAwait(false); + // Returning a message to the queue is a visibility change to the requested delay (0 = immediately visible). + await _sqs.Value.ChangeMessageVisibilityAsync(queueUrl, GetReceiptHandle(entry), (int)Math.Clamp(redeliveryDelay.TotalSeconds, 0, 43200), ct).ConfigureAwait(false); + } + + public async Task RenewLockAsync(TransportEntry entry, TimeSpan? duration, CancellationToken ct) + { + ThrowIfDisposed(); + string queueUrl = await ResolveQueueUrlAsync(entry.Destination, ct).ConfigureAwait(false); + int seconds = (int)Math.Clamp((duration ?? _options.DefaultVisibilityTimeout).TotalSeconds, 0, 43200); + await _sqs.Value.ChangeMessageVisibilityAsync(queueUrl, GetReceiptHandle(entry), seconds, ct).ConfigureAwait(false); + } + + public async Task EnsureAsync(IReadOnlyList declarations, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(declarations); + + foreach (var declaration in declarations) + { + if (declaration.AutoDeleteAfter is not null) + throw new NotSupportedException("SQS/SNS do not provide expiring subscription resources. Use an explicitly named durable subscription."); + switch (declaration.Address.Role) + { + case DestinationRole.Topic: + await ResolveTopicArnAsync(declaration.Address.Name, allowCreate: true, ct).ConfigureAwait(false); + break; + case DestinationRole.Subscription: + case DestinationRole.Binding: + await EnsureSubscriptionAsync(declaration.Address, ct).ConfigureAwait(false); + break; + default: + ValidateQueueArguments(declaration.ProviderArguments); + string url = await ResolveQueueUrlAsync(declaration.Address, allowCreate: true, ct).ConfigureAwait(false); + if (declaration.ProviderArguments is { Count: > 0 } arguments) + await _sqs.Value.SetQueueAttributesAsync(new SetQueueAttributesRequest + { + QueueUrl = url, + Attributes = new Dictionary(arguments) + }, ct).ConfigureAwait(false); + break; + } + } + } + + public async Task DeleteAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + if (destination.Role == DestinationRole.Topic) + { + string? arn = await FindTopicArnAsync(destination.Name, ct).ConfigureAwait(false); + if (arn is not null) + await _sns.Value.DeleteTopicAsync(arn, ct).ConfigureAwait(false); + _topicArns.TryRemove(destination.Name, out _); + return; + } + + if (destination.Topic is { Length: > 0 } topic) + { + string? topicArn = await FindTopicArnAsync(topic, ct).ConfigureAwait(false); + if (topicArn is not null) + { + string queueArn = topicArn[..topicArn.LastIndexOf(':')].Replace(":sns:", ":sqs:", StringComparison.Ordinal) + ":" + ResourceName(destination.Key); + string? subscriptionArn = await FindSubscriptionArnAsync(topicArn, queueArn, ct).ConfigureAwait(false); + if (subscriptionArn is not null) + await _sns.Value.UnsubscribeAsync(subscriptionArn, ct).ConfigureAwait(false); + } + } + try + { + var response = await _sqs.Value.GetQueueUrlAsync(ResourceName(destination.Key), ct).ConfigureAwait(false); + await _sqs.Value.DeleteQueueAsync(response.QueueUrl, ct).ConfigureAwait(false); + } + catch (QueueDoesNotExistException) + { + } + _queueUrls.TryRemove(destination.Key, out _); + } + + public async Task ExistsAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + if (destination.Role == DestinationRole.Topic) + return await FindTopicArnAsync(destination.Name, ct).ConfigureAwait(false) is not null; + try + { + var queue = await _sqs.Value.GetQueueUrlAsync(ResourceName(destination.Key), ct).ConfigureAwait(false); + if (destination.Topic is not { Length: > 0 } topic) + return true; + string? topicArn = await FindTopicArnAsync(topic, ct).ConfigureAwait(false); + if (topicArn is null) + return false; + string queueArn = await GetQueueArnAsync(queue.QueueUrl, ct).ConfigureAwait(false); + return await FindSubscriptionArnAsync(topicArn, queueArn, ct).ConfigureAwait(false) is not null; + } + catch (QueueDoesNotExistException) + { + return false; + } + } + + private async Task FindTopicArnAsync(string name, CancellationToken ct) + { + string resourceName = ResourceName(name); + string? nextToken = null; + do + { + var page = await _sns.Value.ListTopicsAsync(new ListTopicsRequest { NextToken = nextToken }, ct).ConfigureAwait(false); + foreach (var topic in page.Topics ?? []) + { + if (topic.TopicArn.EndsWith(":" + resourceName, StringComparison.Ordinal)) + return topic.TopicArn; + } + nextToken = page.NextToken; + } while (!String.IsNullOrEmpty(nextToken)); + return null; + } + + private async Task FindSubscriptionArnAsync(string topicArn, string queueArn, CancellationToken ct) + { + string? nextToken = null; + do + { + var page = await _sns.Value.ListSubscriptionsByTopicAsync(new ListSubscriptionsByTopicRequest { TopicArn = topicArn, NextToken = nextToken }, ct).ConfigureAwait(false); + foreach (var subscription in page.Subscriptions ?? []) + { + if (subscription.Protocol == "sqs" && subscription.Endpoint == queueArn) + return subscription.SubscriptionArn; + } + nextToken = page.NextToken; + } while (!String.IsNullOrEmpty(nextToken)); + return null; + } + + public async Task GetStatsAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + string queueUrl = await ResolveQueueUrlAsync(destination, ct).ConfigureAwait(false); + var response = await _sqs.Value.GetQueueAttributesAsync(new GetQueueAttributesRequest + { + QueueUrl = queueUrl, + AttributeNames = ["All"] + }, ct).ConfigureAwait(false); + + return new MessageDestinationStats + { + Queued = response.ApproximateNumberOfMessages, + Working = response.ApproximateNumberOfMessagesNotVisible, + Delayed = response.ApproximateNumberOfMessagesDelayed + }; + } + + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _isDisposed, 1) == 1) + return; + + await DisposeBatchersAsync().ConfigureAwait(false); + if (_ownsSqs && _sqs.IsValueCreated) + _sqs.Value.Dispose(); + if (_ownsSns && _sns.IsValueCreated) + _sns.Value.Dispose(); + } + + private async Task EnsureSubscriptionAsync(DestinationAddress address, CancellationToken ct) + { + string queueUrl = await ResolveQueueUrlAsync(address, allowCreate: true, ct).ConfigureAwait(false); + + if (String.IsNullOrEmpty(address.Topic)) + return; + + string topicArn = await ResolveTopicArnAsync(address.Topic, allowCreate: true, ct).ConfigureAwait(false); + string queueArn = await GetQueueArnAsync(queueUrl, ct).ConfigureAwait(false); + + // Allow the topic to deliver to the queue, then subscribe with raw delivery so the SQS body/attributes match a + // direct SQS send (no SNS envelope). + await _sqs.Value.SetQueueAttributesAsync(new SetQueueAttributesRequest + { + QueueUrl = queueUrl, + Attributes = new Dictionary { ["Policy"] = BuildQueuePolicy(queueArn, topicArn) } + }, ct).ConfigureAwait(false); + + await _sns.Value.SubscribeAsync(new SubscribeRequest + { + TopicArn = topicArn, + Protocol = "sqs", + Endpoint = queueArn, + Attributes = new Dictionary { ["RawMessageDelivery"] = "true" }, + ReturnSubscriptionArn = true + }, ct).ConfigureAwait(false); + } + + // Queue and subscription destinations are both backed by an SQS queue whose logical name is the address key + // (Name for queues, "topic/subscription" for subscriptions), so provisioning and every runtime path resolve the + // same physical queue from the same address. + private Task ResolveQueueUrlAsync(DestinationAddress address, CancellationToken ct) => + ResolveQueueUrlAsync(address, allowCreate: false, ct); + + private async Task ResolveQueueUrlAsync(DestinationAddress address, bool allowCreate, CancellationToken ct) + { + string key = address.Key; + if (_queueUrls.TryGetValue(key, out string? cached)) + return cached; + + string resourceName = ResourceName(key); + try + { + var response = await _sqs.Value.GetQueueUrlAsync(resourceName, ct).ConfigureAwait(false); + _queueUrls[key] = response.QueueUrl; + return response.QueueUrl; + } + catch (QueueDoesNotExistException) when (allowCreate) + { + var response = await _sqs.Value.CreateQueueAsync(new CreateQueueRequest { QueueName = resourceName }, ct).ConfigureAwait(false); + _queueUrls[key] = response.QueueUrl; + return response.QueueUrl; + } + } + + // Sending and receiving resolve existing resources; explicit provisioning via EnsureAsync + // creates missing resources according to the caller's topology policy. + private Task ResolveTopicArnAsync(string name, CancellationToken ct) => + ResolveTopicArnAsync(name, allowCreate: false, ct); + + private async Task ResolveTopicArnAsync(string name, bool allowCreate, CancellationToken ct) + { + if (_topicArns.TryGetValue(name, out string? cached)) + return cached; + + if (allowCreate) + { + // CreateTopic is idempotent and returns the ARN of an existing topic with the same name. + var response = await _sns.Value.CreateTopicAsync(new CreateTopicRequest { Name = ResourceName(name) }, ct).ConfigureAwait(false); + _topicArns[name] = response.TopicArn; + return response.TopicArn; + } + + // Auto-create is disabled (locked-down broker): look the topic up instead of creating it, and fail loudly when + // it has not been provisioned out of band. + var existing = await FindTopicArnAsync(name, ct).ConfigureAwait(false); + if (existing is null) + throw new InvalidOperationException($"SNS topic \"{ResourceName(name)}\" does not exist and implicit creation is disabled. Provision it with EnsureAsync or through the message bus topology policy."); + + _topicArns[name] = existing; + return existing; + } + + // SQS queue / SNS topic names allow only [A-Za-z0-9_-] (max 80 chars). Most logical names already conform, but a + // subscription's key (see DestinationAddress.Key) is the opaque "topic/subscription" form which contains '/'. + // Encode any illegal name deterministically and collision-free — sanitize, then append a short stable hash of the + // original — so EnsureAsync/ReceiveAsync/CompleteAsync all resolve the same queue from the same logical name. + // Legal names are returned unchanged (no behavior change for plain queues/topics). + private string ResourceName(string logicalName) => EncodeResourceName(_options.ResourcePrefix, logicalName); + + private static string EncodeResourceName(string prefix, string logicalName) + { + string candidate = prefix + logicalName; + if (IsResourceNameLegal(candidate)) + return candidate; + + string suffix = "-" + StableHash(candidate); + string sanitized = SanitizeResourceName(candidate); + if (sanitized.Length > 80 - suffix.Length) + sanitized = sanitized[..(80 - suffix.Length)]; + return sanitized + suffix; + } + + private static bool IsResourceNameLegal(string name) + { + if (name.Length is 0 or > 80) + return false; + foreach (char c in name) + { + if (!(Char.IsAsciiLetterOrDigit(c) || c is '-' or '_')) + return false; + } + + return true; + } + + private static string SanitizeResourceName(string name) + { + var builder = new StringBuilder(name.Length); + foreach (char c in name) + builder.Append(Char.IsAsciiLetterOrDigit(c) || c is '-' or '_' ? c : '-'); + return builder.ToString(); + } + + private static string StableHash(string value) + { + byte[] hash = SHA256.HashData(Encoding.UTF8.GetBytes(value)); + return Convert.ToHexString(hash, 0, 4).ToLowerInvariant(); // 8 hex chars + } + + private async Task GetQueueArnAsync(string queueUrl, CancellationToken ct) + { + var response = await _sqs.Value.GetQueueAttributesAsync(new GetQueueAttributesRequest + { + QueueUrl = queueUrl, + AttributeNames = ["QueueArn"] + }, ct).ConfigureAwait(false); + return response.QueueARN; + } + + private static string BuildQueuePolicy(string queueArn, string topicArn) + { + return JsonSerializer.Serialize(new + { + Version = "2012-10-17", + Statement = new[] + { + new + { + Effect = "Allow", + Principal = new { Service = "sns.amazonaws.com" }, + Action = "sqs:SendMessage", + Resource = queueArn, + Condition = new { ArnEquals = new Dictionary { ["aws:SourceArn"] = topicArn } } + } + } + }); + } + + private int? ToDelaySeconds(DateTimeOffset? deliverAt) + { + if (deliverAt is not { } at) + return null; + + double seconds = (at - DateTimeOffset.UtcNow).TotalSeconds; + if (seconds <= 0) + return null; + + return (int)Math.Clamp(seconds, 1, 900); // SQS DelaySeconds maximum is 900 (15 minutes) + } + + private static int GetReceiveCount(SqsMessage message) + { + if (message.Attributes is not null && message.Attributes.TryGetValue("ApproximateReceiveCount", out string? value) && Int32.TryParse(value, out int count) && count > 0) + return count; + return 1; + } + + private static string GetReceiptHandle(TransportEntry entry) + { + return entry.Receipt.TransportState as string + ?? throw new ReceiptExpiredException("The transport entry does not carry an SQS receipt handle."); + } + + // A text body (e.g. JSON, the default) is stored as-is so it is human-readable in the console and avoids base64 + // overhead; anything else is base64-encoded so arbitrary bytes round-trip through SQS/SNS string bodies. The chosen + // encoding is recorded in a native attribute for the receive side. + private static (string Body, string Encoding) EncodeBody(TransportMessage message) + { + return IsTextContent(message.ContentType) + ? (Encoding.UTF8.GetString(message.Body.Span), "text") + : (Convert.ToBase64String(message.Body.Span), "base64"); + } + + private static ReadOnlyMemory DecodeBody(string body, string? encoding) + { + if (String.IsNullOrEmpty(body)) + return ReadOnlyMemory.Empty; + + return String.Equals(encoding, "text", StringComparison.Ordinal) + ? Encoding.UTF8.GetBytes(body) + : Convert.FromBase64String(body); + } + + private static bool IsTextContent(string? contentType) + { + return !String.IsNullOrEmpty(contentType) + && (contentType.Contains("json", StringComparison.OrdinalIgnoreCase) + || contentType.Contains("xml", StringComparison.OrdinalIgnoreCase) + || contentType.StartsWith("text/", StringComparison.OrdinalIgnoreCase)); + } + + private PreparedMessage PrepareMessage(int index, TransportMessage message, DateTimeOffset? deliverAt) + { + var (body, encoding) = EncodeBody(message); + var headers = message.Headers; + string envelope = JsonSerializer.Serialize(new AwsEnvelope(1, encoding, message.MessageId, message.ContentType, headers)); + int bytes = checked(Encoding.UTF8.GetByteCount(body) + AttributeBytes(EnvelopeAttributeName, envelope)); + foreach (string name in NativeHeaders(headers)) + { + string? value = headers.GetValueOrDefault(name); + if (!String.IsNullOrEmpty(value)) + bytes = checked(bytes + AttributeBytes(name, value)); + } + return new PreparedMessage(index, body, envelope, headers, bytes, deliverAt); + + static int AttributeBytes(string name, string value) => checked(Encoding.UTF8.GetByteCount(name) + Encoding.UTF8.GetByteCount(value) + 6); + } + + private IEnumerable NativeHeaders(MessageHeaders headers) + { + if (!_options.ExposeAllNativeHeaders) return _nativeMessageHeaders; + var names = headers.Keys.Where(AwsMessageTransportOptions.IsValidNativeHeader).Union(_nativeMessageHeaders, StringComparer.Ordinal).ToArray(); + return names.Length <= 9 ? names : _nativeMessageHeaders; + } + + private Dictionary BuildAttributes(PreparedMessage message, Func createAttribute) + { + var attributes = new Dictionary(_nativeMessageHeaders.Length + 1, StringComparer.Ordinal) + { + [EnvelopeAttributeName] = createAttribute(message.Envelope) + }; + foreach (string name in NativeHeaders(message.Headers)) + { + string? value = message.Headers.GetValueOrDefault(name); + if (!String.IsNullOrEmpty(value)) + attributes[name] = createAttribute(value); + } + return attributes; + } + + private static string? GetAttribute(Dictionary? attributes, string name) + { + return attributes is not null && attributes.TryGetValue(name, out var value) ? value.StringValue : null; + } + + private sealed record AwsEnvelope(int Version, string Encoding, string? MessageId, string? ContentType, IReadOnlyDictionary? Headers); + + private static MessageHeaders FromSqsAttributes(Dictionary? attributes) + { + if (attributes is null || !attributes.TryGetValue(HeadersAttributeName, out var value) || String.IsNullOrEmpty(value.StringValue)) + return MessageHeaders.Empty; + + return MessageHeaders.DeserializeFromJson(value.StringValue); + } + + private IAmazonSQS CreateSqsClient() + { + var config = new AmazonSQSConfig(); + ApplyEndpoint(config); + return _options.Credentials is { } credentials ? new AmazonSQSClient(credentials, config) : new AmazonSQSClient(config); + } + + private IAmazonSimpleNotificationService CreateSnsClient() + { + var config = new AmazonSimpleNotificationServiceConfig(); + ApplyEndpoint(config); + return _options.Credentials is { } credentials ? new AmazonSimpleNotificationServiceClient(credentials, config) : new AmazonSimpleNotificationServiceClient(config); + } + + private void ApplyEndpoint(Amazon.Runtime.ClientConfig config) + { + if (!String.IsNullOrEmpty(_options.ServiceUrl)) + { + config.ServiceURL = _options.ServiceUrl; + config.AuthenticationRegion = (_options.Region ?? Amazon.RegionEndpoint.USEast1).SystemName; + } + else if (_options.Region is { } region) + { + config.RegionEndpoint = region; + } + } + + private void ThrowIfDisposed() + { + ObjectDisposedException.ThrowIf(Volatile.Read(ref _isDisposed) == 1, this); + } +} diff --git a/src/Foundatio.Aws/AwsMessageTransportOptions.cs b/src/Foundatio.Aws/AwsMessageTransportOptions.cs new file mode 100644 index 000000000..f383c1fe8 --- /dev/null +++ b/src/Foundatio.Aws/AwsMessageTransportOptions.cs @@ -0,0 +1,141 @@ +using System; +using System.Collections.Generic; +using Amazon; +using Amazon.Runtime; + +namespace Foundatio.Messaging; + +public class AwsMessageTransportOptions +{ + /// Optional logging for managed node lifecycle operations. + public Microsoft.Extensions.Logging.ILoggerFactory? LoggerFactory { get; set; } + + /// AWS credentials. When null, the SDK's default credential chain is used. + public AWSCredentials? Credentials { get; set; } + + /// AWS region. When null, the SDK's default region resolution is used (ignored when is set). + public RegionEndpoint? Region { get; set; } + + /// Custom service endpoint, e.g. http://localhost:4566 for LocalStack. + public string? ServiceUrl { get; set; } + + /// + /// Optional prefix applied to the underlying SQS queue and SNS topic names (not the logical destination names used + /// by callers). Useful to isolate runs/environments on a shared broker — e.g. a unique prefix per conformance run + /// so leftover messages from a prior run can't leak in. + /// + public string ResourcePrefix { get; set; } = ""; + + /// + /// Headers to also expose as native AWS message attributes for SNS filters or external consumers. + /// Empty by default; all headers remain available in the Foundatio envelope. At most nine names are allowed. + /// + public IReadOnlyCollection NativeMessageHeaders { get; set; } = []; + + /// Expose all valid headers when they fit AWS's attribute limit; otherwise retain only explicitly selected native headers. + public bool ExposeAllNativeHeaders { get; set; } + + /// Maximum entries in a native batch, between one and ten. + public int MaxBatchSize { get; set; } = 10; + + /// Default receive visibility timeout when none is supplied. Maps to the SQS visibility window. + public TimeSpan DefaultVisibilityTimeout { get; set; } = TimeSpan.FromSeconds(30); + + /// Coalesce concurrent single sends, publishes and acknowledgements into native AWS batches. + public bool EnableBatching { get; set; } = true; + + /// Maximum time to collect a partial batch; idle single-operation streams dispatch immediately. Zero batches only operations already waiting. + public TimeSpan BatchDelay { get; set; } = TimeSpan.FromMilliseconds(2); + + /// Maximum concurrent automatically collected requests per destination and operation (send or acknowledge). + public int MaxConcurrentBatches { get; set; } = 4; + + /// Maximum buffered operations per destination and operation. Further callers await capacity. + public int MaxPendingBatchMessages { get; set; } = 100; + + /// Timeout for a shared AWS batch request and for draining batchers during transport disposal. + public TimeSpan BatchTimeout { get; set; } = TimeSpan.FromSeconds(30); + + internal void Validate() + { + ArgumentNullException.ThrowIfNull(NativeMessageHeaders); + ArgumentOutOfRangeException.ThrowIfLessThan(MaxBatchSize, 1); + ArgumentOutOfRangeException.ThrowIfGreaterThan(MaxBatchSize, 10); + ArgumentOutOfRangeException.ThrowIfGreaterThan(NativeMessageHeaders.Count, 9, nameof(NativeMessageHeaders)); + var names = new HashSet(StringComparer.Ordinal); + foreach (string name in NativeMessageHeaders) + if (!IsValidNativeHeader(name) || !names.Add(name)) + throw new ArgumentException("Native message header names must be unique AWS attribute names, without AWS., Amazon. or fnd. prefixes.", nameof(NativeMessageHeaders)); + + ArgumentOutOfRangeException.ThrowIfLessThan(BatchDelay, TimeSpan.Zero); + ArgumentOutOfRangeException.ThrowIfGreaterThan(BatchDelay, TimeSpan.FromMilliseconds(100)); + ArgumentOutOfRangeException.ThrowIfLessThan(MaxConcurrentBatches, 1); + ArgumentOutOfRangeException.ThrowIfGreaterThan(MaxConcurrentBatches, 64); + ArgumentOutOfRangeException.ThrowIfLessThan(MaxPendingBatchMessages, 1); + ArgumentOutOfRangeException.ThrowIfGreaterThan(MaxPendingBatchMessages, 1_000_000); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(BatchTimeout, TimeSpan.Zero); + ArgumentOutOfRangeException.ThrowIfGreaterThan(BatchTimeout, TimeSpan.FromMinutes(5)); + } + + internal static bool IsValidNativeHeader(string name) + { + if (String.IsNullOrEmpty(name) || name.Length > 256 || name[0] == '.' || name[^1] == '.' || name.Contains("..", StringComparison.Ordinal) + || name.StartsWith("AWS.", StringComparison.OrdinalIgnoreCase) || name.StartsWith("Amazon.", StringComparison.OrdinalIgnoreCase) + || name.StartsWith("fnd.", StringComparison.OrdinalIgnoreCase)) + return false; + foreach (char c in name) + if (!(Char.IsAsciiLetterOrDigit(c) || c is '_' or '-' or '.')) + return false; + return true; + } + + /// + /// Parses a connection string of the form + /// serviceurl=http://localhost:4566;accesskey=...;secretkey=...;region=us-east-1 into options. Any subset of + /// keys may be provided; unknown keys are ignored. + /// + public static AwsMessageTransportOptions FromConnectionString(string connectionString) + { + ArgumentException.ThrowIfNullOrEmpty(connectionString); + + string? accessKey = null, secretKey = null, region = null, serviceUrl = null; + foreach (string pair in connectionString.Split(';', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)) + { + int separator = pair.IndexOf('='); + if (separator < 0) + continue; + + string key = pair[..separator].Trim().ToLowerInvariant().Replace(" ", ""); + string value = pair[(separator + 1)..].Trim(); + + switch (key) + { + case "accesskey": + case "accesskeyid": + case "id": + accessKey = value; + break; + case "secretkey": + case "secret": + secretKey = value; + break; + case "region": + case "endpoint": + region = value; + break; + case "serviceurl": + case "service": + serviceUrl = value; + break; + } + } + + var options = new AwsMessageTransportOptions { ServiceUrl = serviceUrl }; + if (!String.IsNullOrEmpty(accessKey) && !String.IsNullOrEmpty(secretKey)) + options.Credentials = new BasicAWSCredentials(accessKey, secretKey); + if (!String.IsNullOrEmpty(region)) + options.Region = RegionEndpoint.GetBySystemName(region); + + return options; + } +} diff --git a/src/Foundatio.Aws/AwsRequestBatcher.cs b/src/Foundatio.Aws/AwsRequestBatcher.cs new file mode 100644 index 000000000..c88803af2 --- /dev/null +++ b/src/Foundatio.Aws/AwsRequestBatcher.cs @@ -0,0 +1,200 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Channels; +using System.Threading.Tasks; + +namespace Foundatio.Messaging; + +internal sealed class AwsRequestBatcher : IAsyncDisposable +{ + private readonly Channel _channel; + private readonly Func, CancellationToken, Task> _execute; + private readonly Func _size; + private readonly int _maximumBytes; + private readonly int _concurrency; + private readonly int _maxBatchSize; + private readonly TimeSpan _delay; + private readonly TimeSpan _timeout; + private readonly bool _delayWhenIdle; + private readonly CancellationTokenSource _stop = new(); + private readonly Task _worker; + private int _disposed; + private int _activeRequests; + private int _observedBatchSize; + + public AwsRequestBatcher(AwsMessageTransportOptions options, int maximumBytes, Func size, + Func, CancellationToken, Task> execute, bool delayWhenIdle = true) + { + _maximumBytes = maximumBytes; + _size = size; + _execute = execute; + _delayWhenIdle = delayWhenIdle; + _concurrency = options.MaxConcurrentBatches; + _maxBatchSize = options.MaxBatchSize; + _delay = options.BatchDelay; + _timeout = options.BatchTimeout; + _channel = Channel.CreateBounded(new BoundedChannelOptions(options.MaxPendingBatchMessages) + { + SingleReader = true, + AllowSynchronousContinuations = false, + FullMode = BoundedChannelFullMode.Wait + }); + if (ExecutionContext.IsFlowSuppressed()) + _worker = Task.Run(RunAsync); + else + { + using (ExecutionContext.SuppressFlow()) + _worker = Task.Run(RunAsync); + } + } + + public void ObserveBatchSize(int count) + { + count = Math.Clamp(count, 1, _maxBatchSize); + int previous = Volatile.Read(ref _observedBatchSize); + while (previous < count) + { + int observed = Interlocked.CompareExchange(ref _observedBatchSize, count, previous); + if (observed == previous) + break; + previous = observed; + } + } + + public async Task ExecuteAsync(T value, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + var pending = new Pending(value); + using var registration = cancellationToken.UnsafeRegister(static (state, token) => + ((Pending)state!).Completion.TrySetCanceled(token), pending); + try + { + await _channel.Writer.WriteAsync(pending, cancellationToken).ConfigureAwait(false); + return await pending.Completion.Task.ConfigureAwait(false); + } + catch (ChannelClosedException) + { + throw new ObjectDisposedException(nameof(AwsMessageTransport)); + } + } + + private async Task RunAsync() + { + var executing = new List(_concurrency); + int previousBatchSize = 0; + try + { + while (await _channel.Reader.WaitToReadAsync(_stop.Token).ConfigureAwait(false)) + { + executing.RemoveAll(static task => task.IsCompleted); + if (executing.Count >= _concurrency) + { + await Task.WhenAny(executing).ConfigureAwait(false); + executing.RemoveAll(static task => task.IsCompleted); + } + var batch = await ReadBatchAsync(Volatile.Read(ref _activeRequests) > 0 || (_delayWhenIdle && previousBatchSize != 1)).ConfigureAwait(false); + if (batch.Count > 0) + { + previousBatchSize = batch.Count; + executing.Add(ExecuteBatchAsync(batch)); + } + } + } + catch (OperationCanceledException) when (_stop.IsCancellationRequested) + { + } + finally + { + _channel.Writer.TryComplete(); + while (_channel.Reader.TryRead(out var pending)) + pending.Completion.TrySetException(new ObjectDisposedException(nameof(AwsMessageTransport))); + await Task.WhenAll(executing).ConfigureAwait(false); + } + } + + private async Task> ReadBatchAsync(bool waitForMore) + { + var batch = new List(_maxBatchSize); + int bytes = 0; + Task? deadline = null; + try + { + while (batch.Count < _maxBatchSize) + { + if (_channel.Reader.TryPeek(out var pending)) + { + if (pending.Completion.Task.IsCompleted) + { + _channel.Reader.TryRead(out _); + continue; + } + int size = _size(pending.Value); + if (batch.Count > 0 && bytes + size > _maximumBytes) + break; + _channel.Reader.TryRead(out _); + batch.Add(pending); + bytes += size; + } + else + { + int observedBatchSize = Volatile.Read(ref _observedBatchSize); + if (!waitForMore || _delay == TimeSpan.Zero || (observedBatchSize > 0 && batch.Count >= observedBatchSize)) + break; + deadline ??= Task.Delay(_delay, _stop.Token); + var available = _channel.Reader.WaitToReadAsync(_stop.Token).AsTask(); + if (await Task.WhenAny(available, deadline).ConfigureAwait(false) != available || !await available.ConfigureAwait(false)) + break; + } + } + } + catch (OperationCanceledException) when (_stop.IsCancellationRequested) + { + } + return batch; + } + + private async Task ExecuteBatchAsync(List batch) + { + batch.RemoveAll(static entry => entry.Completion.Task.IsCompleted); + if (batch.Count == 0) + return; + try + { + using var timeout = CancellationTokenSource.CreateLinkedTokenSource(_stop.Token); + timeout.CancelAfter(_timeout); + var values = new T[batch.Count]; + for (int i = 0; i < batch.Count; i++) + values[i] = batch[i].Value; + TResult[] results; + Interlocked.Increment(ref _activeRequests); + try { results = await _execute(values, timeout.Token).ConfigureAwait(false); } + finally { Interlocked.Decrement(ref _activeRequests); } + if (results.Length != batch.Count) + throw new MessageBusException("AWS returned an incomplete batch result."); + for (int i = 0; i < batch.Count; i++) + batch[i].Completion.TrySetResult(results[i]); + } + catch (Exception ex) + { + foreach (var pending in batch) + pending.Completion.TrySetException(ex); + } + } + + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _disposed, 1) != 0) + return; + _channel.Writer.TryComplete(); + _stop.CancelAfter(_timeout); + try { await _worker.ConfigureAwait(false); } + finally { _stop.Dispose(); } + } + + private sealed class Pending(T value) + { + public T Value { get; } = value; + public TaskCompletionSource Completion { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + } +} diff --git a/src/Foundatio.Aws/Foundatio.Aws.csproj b/src/Foundatio.Aws/Foundatio.Aws.csproj new file mode 100644 index 000000000..2c8635702 --- /dev/null +++ b/src/Foundatio.Aws/Foundatio.Aws.csproj @@ -0,0 +1,12 @@ + + + AWS (SQS/SNS) IMessageTransport for Foundatio messaging. Temporary in-repo provider for validating the redesigned transport contract against a real broker. + + + + + + + + + diff --git a/src/Foundatio.DataProtection/Foundatio.DataProtection.csproj b/src/Foundatio.DataProtection/Foundatio.DataProtection.csproj index b76cf052e..ca1adfd27 100644 --- a/src/Foundatio.DataProtection/Foundatio.DataProtection.csproj +++ b/src/Foundatio.DataProtection/Foundatio.DataProtection.csproj @@ -4,6 +4,7 @@ + diff --git a/src/Foundatio.Extensions.Hosting/Foundatio.Extensions.Hosting.csproj b/src/Foundatio.Extensions.Hosting/Foundatio.Extensions.Hosting.csproj index 346f4eaaa..0de0bd9ad 100644 --- a/src/Foundatio.Extensions.Hosting/Foundatio.Extensions.Hosting.csproj +++ b/src/Foundatio.Extensions.Hosting/Foundatio.Extensions.Hosting.csproj @@ -1,4 +1,4 @@ - + true net8.0;net10.0 @@ -9,8 +9,4 @@ - - - - diff --git a/src/Foundatio.Extensions.Hosting/FoundatioRuntimeHealth.cs b/src/Foundatio.Extensions.Hosting/FoundatioRuntimeHealth.cs new file mode 100644 index 000000000..d3036404a --- /dev/null +++ b/src/Foundatio.Extensions.Hosting/FoundatioRuntimeHealth.cs @@ -0,0 +1,53 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Diagnostics.Metrics; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Extensions.Hosting.Messaging; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Hosting; + +namespace Foundatio; + +/// Last observed infrastructure health and capacity of the hosted runtime. +public sealed class FoundatioRuntimeHealth : IDisposable +{ + private readonly ConcurrentDictionary _failures = new(StringComparer.Ordinal); + private readonly Meter _meter = new("Foundatio.Runtime"); + private JobRuntimeStoreStats? _capacity; + private long _scheduledDispatches = -1; + + public FoundatioRuntimeHealth() + { + _meter.CreateObservableGauge("foundatio.jobs.active", () => Volatile.Read(ref _capacity)?.ActiveJobs ?? 0); + _meter.CreateObservableGauge("foundatio.jobs.history", () => Volatile.Read(ref _capacity)?.HistoryJobs ?? 0); + _meter.CreateObservableGauge("foundatio.jobs.idempotency_records", () => Volatile.Read(ref _capacity)?.DeduplicationRecords ?? 0); + _meter.CreateObservableGauge("foundatio.messaging.scheduled_dispatches", () => ScheduledDispatches ?? Volatile.Read(ref _capacity)?.ScheduledDispatches ?? 0); + } + + public IReadOnlyDictionary Components => new Dictionary(_failures); + public long? ScheduledDispatches => Interlocked.Read(ref _scheduledDispatches) is >= 0 and var count ? count : null; + internal void UpdateScheduledDispatches(long count) => Interlocked.Exchange(ref _scheduledDispatches, count); + public JobRuntimeStoreStats? Capacity => Volatile.Read(ref _capacity); + internal void Healthy(string component) => _failures[component] = null; + internal void Failed(string component, Exception error) => _failures[component] = error.Message; + internal void UpdateCapacity(JobRuntimeStoreStats capacity) => Volatile.Write(ref _capacity, capacity); + public void Dispose() => _meter.Dispose(); +} + +internal sealed class FoundatioHealthCheck(IServiceProvider services, FoundatioRuntimeHealth runtime) : IHealthCheck +{ + public Task CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) + { + var failures = runtime.Components.Where(p => p.Value is not null).Select(p => $"{p.Key}: {p.Value}").ToList(); + if (services.GetService() is { IsHealthy: false }) failures.Add("Job worker is recovering."); + foreach (var host in services.GetServices().OfType()) + failures.AddRange(host.Subscriptions.Where(s => s.Status != MessageSubscriptionStatus.Healthy).Select(s => $"{s.Source}: {s.Status}")); + return Task.FromResult(failures.Count == 0 ? HealthCheckResult.Healthy() : HealthCheckResult.Unhealthy(String.Join("; ", failures))); + } +} diff --git a/src/Foundatio.Extensions.Hosting/FoundatioWorkerExtensions.cs b/src/Foundatio.Extensions.Hosting/FoundatioWorkerExtensions.cs new file mode 100644 index 000000000..22ac33179 --- /dev/null +++ b/src/Foundatio.Extensions.Hosting/FoundatioWorkerExtensions.cs @@ -0,0 +1,54 @@ +using System; +using System.Linq; +using Foundatio.Extensions.Hosting.Jobs; +using Foundatio.Extensions.Hosting.Messaging; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; + +namespace Foundatio; + +/// Configures a worker application and starts the background roles its configuration requires. +public static class FoundatioWorkerExtensions +{ + /// + /// Configures and hosts message consumers, registered jobs, their scheduler, and delayed-message dispatch. + /// Put the worker's Foundatio registrations in this callback. Producer-only apps use AddFoundatio instead. + /// Individual hosting extensions remain available when roles run in separate processes. + /// + /// The application's services. + /// Transport, store, handler, and job registrations for this worker. + /// Maximum simultaneous job executions. Message concurrency is configured per consumer. + public static IServiceCollection AddFoundatioWorker(this IServiceCollection services, Action configure, int? jobConcurrency = null) + { + ArgumentNullException.ThrowIfNull(services); + ArgumentNullException.ThrowIfNull(configure); + if (jobConcurrency is { } value) ArgumentOutOfRangeException.ThrowIfLessThan(value, 1); + services.TryAddSingleton(); + services.AddHealthChecks().AddCheck("foundatio"); + configure(services.AddFoundatio()); + + bool handlers = services.Any(d => d.ServiceType == typeof(MessageHandlerRegistration)); + bool transport = services.Any(d => d.ServiceType == typeof(IMessageTransport)); + bool jobs = services.Any(d => d.ServiceType == typeof(JobTypeRegistration)); + bool jobStore = services.Any(d => d.ServiceType == typeof(IJobRuntimeStore)); + bool dispatchStore = jobStore || services.Any(d => d.ServiceType == typeof(IScheduledDispatchStore)); + + if (handlers && !transport) + throw new InvalidOperationException("The worker has message handlers but no transport. Configure Messaging.UseInMemory(), Messaging.UseRedis(), or Messaging.UseAws() in AddFoundatioWorker."); + if (jobs && !jobStore) + throw new InvalidOperationException("The worker has jobs but no runtime store. Configure Jobs.UseInMemory(), Jobs.UseRedis(), or Jobs.UseRuntimeStore(...) in AddFoundatioWorker."); + + if (transport) + services.AddMessageConsumers(); + if (jobs) + { + services.AddJobWorker(jobConcurrency); + services.AddJobScheduler(); + } + if (transport && dispatchStore) + services.AddScheduledMessageDispatcher(); + return services; + } +} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/Cron.cs b/src/Foundatio.Extensions.Hosting/Jobs/Cron.cs deleted file mode 100644 index ffc193481..000000000 --- a/src/Foundatio.Extensions.Hosting/Jobs/Cron.cs +++ /dev/null @@ -1,232 +0,0 @@ -// This file is part of Hangfire. Copyright © 2013-2014 Hangfire OÜ. -// -// Hangfire is free software: you can redistribute it and/or modify -// it under the terms of the GNU Lesser General Public License as -// published by the Free Software Foundation, either version 3 -// of the License, or any later version. -// -// Hangfire is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -// GNU Lesser General Public License for more details. -// -// You should have received a copy of the GNU Lesser General Public -// License along with Hangfire. If not, see . - -using System; - -namespace Foundatio.Extensions.Hosting.Jobs; - -/// -/// Helper class that provides common values for the cron expressions. -/// -public static class Cron -{ - /// - /// Returns cron expression that fires every minute. - /// - public static string Minutely() - { - return "* * * * *"; - } - - /// - /// Returns cron expression that fires every Nth minute. - /// - public static string Minutely(int minute) - { - return $"0/{minute} * * * *"; - } - - /// - /// Returns cron expression that fires every hour at the first minute. - /// - public static string Hourly() - { - return Hourly(minute: 0); - } - - /// - /// Returns cron expression that fires every hour at the specified minute. - /// - /// The minute in which the schedule will be activated (0-59). - public static string Hourly(int minute) - { - return $"{minute} * * * *"; - } - - /// - /// Returns cron expression that fires every day at 00:00 UTC. - /// - public static string Daily() - { - return Daily(hour: 0); - } - - /// - /// Returns cron expression that fires every day at the first minute of - /// the specified hour in UTC. - /// - /// The hour in which the schedule will be activated (0-23). - public static string Daily(int hour) - { - return Daily(hour, minute: 0); - } - - /// - /// Returns cron expression that fires every day at the specified hour and minute - /// in UTC. - /// - /// The hour in which the schedule will be activated (0-23). - /// The minute in which the schedule will be activated (0-59). - public static string Daily(int hour, int minute) - { - return $"{minute} {hour} * * *"; - } - - /// - /// Returns cron expression that fires every week at Monday, 00:00 UTC. - /// - public static string Weekly() - { - return Weekly(DayOfWeek.Monday); - } - - /// - /// Returns cron expression that fires every week at 00:00 UTC of the specified - /// day of the week. - /// - /// The day of week in which the schedule will be activated. - public static string Weekly(DayOfWeek dayOfWeek) - { - return Weekly(dayOfWeek, hour: 0); - } - - /// - /// Returns cron expression that fires every week at the first minute - /// of the specified day of week and hour in UTC. - /// - /// The day of week in which the schedule will be activated. - /// The hour in which the schedule will be activated (0-23). - public static string Weekly(DayOfWeek dayOfWeek, int hour) - { - return Weekly(dayOfWeek, hour, minute: 0); - } - - /// - /// Returns cron expression that fires every week at the specified day - /// of week, hour and minute in UTC. - /// - /// The day of week in which the schedule will be activated. - /// The hour in which the schedule will be activated (0-23). - /// The minute in which the schedule will be activated (0-59). - public static string Weekly(DayOfWeek dayOfWeek, int hour, int minute) - { - return $"{minute} {hour} * * {(int)dayOfWeek}"; - } - - /// - /// Returns cron expression that fires every month at 00:00 UTC of the first - /// day of month. - /// - public static string Monthly() - { - return Monthly(day: 1); - } - - /// - /// Returns cron expression that fires every month at 00:00 UTC of the specified - /// day of month. - /// - /// The day of month in which the schedule will be activated (1-31). - public static string Monthly(int day) - { - return Monthly(day, hour: 0); - } - - /// - /// Returns cron expression that fires every month at the first minute of the - /// specified day of month and hour in UTC. - /// - /// The day of month in which the schedule will be activated (1-31). - /// The hour in which the schedule will be activated (0-23). - public static string Monthly(int day, int hour) - { - return Monthly(day, hour, minute: 0); - } - - /// - /// Returns cron expression that fires every month at the specified day of month, - /// hour and minute in UTC. - /// - /// The day of month in which the schedule will be activated (1-31). - /// The hour in which the schedule will be activated (0-23). - /// The minute in which the schedule will be activated (0-59). - public static string Monthly(int day, int hour, int minute) - { - return $"{minute} {hour} {day} * *"; - } - - /// - /// Returns cron expression that fires every year on Jan, 1st at 00:00 UTC. - /// - public static string Yearly() - { - return Yearly(month: 1); - } - - /// - /// Returns cron expression that fires every year in the first day at 00:00 UTC - /// of the specified month. - /// - /// The month in which the schedule will be activated (1-12). - public static string Yearly(int month) - { - return Yearly(month, day: 1); - } - - /// - /// Returns cron expression that fires every year at 00:00 UTC of the specified - /// month and day of month. - /// - /// The month in which the schedule will be activated (1-12). - /// The day of month in which the schedule will be activated (1-31). - public static string Yearly(int month, int day) - { - return Yearly(month, day, hour: 0); - } - - /// - /// Returns cron expression that fires every year at the first minute of the - /// specified month, day and hour in UTC. - /// - /// The month in which the schedule will be activated (1-12). - /// The day of month in which the schedule will be activated (1-31). - /// The hour in which the schedule will be activated (0-23). - public static string Yearly(int month, int day, int hour) - { - return Yearly(month, day, hour, minute: 0); - } - - /// - /// Returns cron expression that fires every year at the specified month, day, - /// hour and minute in UTC. - /// - /// The month in which the schedule will be activated (1-12). - /// The day of month in which the schedule will be activated (1-31). - /// The hour in which the schedule will be activated (0-23). - /// The minute in which the schedule will be activated (0-59). - public static string Yearly(int month, int day, int hour, int minute) - { - return $"{minute} {hour} {day} {month} *"; - } - - /// - /// Returns cron expression that never fires. Specifically 31st of February - /// - /// - public static string Never() - { - return Yearly(2, 31); - } -} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/DynamicJob.cs b/src/Foundatio.Extensions.Hosting/Jobs/DynamicJob.cs deleted file mode 100644 index 1eda91928..000000000 --- a/src/Foundatio.Extensions.Hosting/Jobs/DynamicJob.cs +++ /dev/null @@ -1,26 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Jobs; -using Foundatio.Utility; - -namespace Foundatio.Extensions.Hosting.Jobs; - -internal class DynamicJob : IJob -{ - private readonly IServiceProvider _serviceProvider; - private readonly Func _action; - - public DynamicJob(IServiceProvider serviceProvider, Func action) - { - _serviceProvider = serviceProvider; - _action = action; - } - - public async Task RunAsync(CancellationToken cancellationToken = default) - { - await _action(_serviceProvider, cancellationToken).AnyContext(); - - return JobResult.Success; - } -} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/HostedJobOptions.cs b/src/Foundatio.Extensions.Hosting/Jobs/HostedJobOptions.cs deleted file mode 100644 index ce9488ed7..000000000 --- a/src/Foundatio.Extensions.Hosting/Jobs/HostedJobOptions.cs +++ /dev/null @@ -1,8 +0,0 @@ -using Foundatio.Jobs; - -namespace Foundatio.Extensions.Hosting.Jobs; - -public class HostedJobOptions : JobOptions -{ - public bool WaitForStartupActions { get; set; } -} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/HostedJobService.cs b/src/Foundatio.Extensions.Hosting/Jobs/HostedJobService.cs deleted file mode 100644 index d5d7b2eba..000000000 --- a/src/Foundatio.Extensions.Hosting/Jobs/HostedJobService.cs +++ /dev/null @@ -1,105 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Extensions.Hosting.Startup; -using Foundatio.Jobs; -using Foundatio.Utility; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Hosting; -using Microsoft.Extensions.Logging; - -namespace Foundatio.Extensions.Hosting.Jobs; - -public class HostedJobService : IHostedService, IJobStatus, IDisposable -{ - private readonly CancellationTokenSource _stoppingCts = new(); - private Task? _executingTask; - private readonly IServiceProvider _serviceProvider; - private readonly ILoggerFactory _loggerFactory; - private readonly ILogger _logger; - private readonly HostedJobOptions _jobOptions; - private bool _hasStarted = false; - - public HostedJobService(IServiceProvider serviceProvider, HostedJobOptions jobOptions, ILoggerFactory loggerFactory) - { - _serviceProvider = serviceProvider; - _loggerFactory = loggerFactory; - _logger = loggerFactory.CreateLogger(); - _jobOptions = jobOptions; - - var lifetime = serviceProvider.GetService(); - lifetime?.RegisterHostedJobInstance(this); - } - - private async Task ExecuteAsync(CancellationToken stoppingToken) - { - if (_jobOptions.WaitForStartupActions) - { - var startupContext = _serviceProvider.GetService(); - if (startupContext != null) - { - var result = await startupContext.WaitForStartupAsync(stoppingToken).AnyContext(); - if (!result.Success) - { - _logger.LogError("Unable to start {JobName} job due to startup actions failure", _jobOptions.Name); - return; - } - } - } - - var runner = new JobRunner(_jobOptions, _serviceProvider, _loggerFactory); - - try - { - await runner.RunAsync(stoppingToken).AnyContext(); -#if NET8_0_OR_GREATER - await _stoppingCts.CancelAsync().AnyContext(); -#else - _stoppingCts.Cancel(); -#endif - } - finally - { - _logger.LogInformation("{JobName} job completed", _jobOptions.Name); - } - } - - public Task StartAsync(CancellationToken cancellationToken) - { - _executingTask = ExecuteAsync(_stoppingCts.Token); - _hasStarted = true; - return _executingTask.IsCompleted ? _executingTask : Task.CompletedTask; - } - - public async Task StopAsync(CancellationToken cancellationToken) - { - if (_executingTask == null) - return; - - try - { -#if NET8_0_OR_GREATER - await _stoppingCts.CancelAsync().AnyContext(); -#else - _stoppingCts.Cancel(); -#endif - } - finally - { - await Task.WhenAny(_executingTask, Task.Delay(-1, cancellationToken)).AnyContext(); - } - } - - public void Dispose() - { - _stoppingCts.Cancel(); - _stoppingCts.Dispose(); - } - - public bool IsRunning => _hasStarted == false || (_executingTask != null && !_executingTask.IsCompleted); -} - -public interface IJobStatus -{ - bool IsRunning { get; } -} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/JobHostExtensions.cs b/src/Foundatio.Extensions.Hosting/Jobs/JobHostExtensions.cs index 1a00ac6c0..c0e6ff548 100644 --- a/src/Foundatio.Extensions.Hosting/Jobs/JobHostExtensions.cs +++ b/src/Foundatio.Extensions.Hosting/Jobs/JobHostExtensions.cs @@ -1,203 +1,30 @@ -using System; +using System; using System.Linq; -using System.Threading; -using System.Threading.Tasks; using Foundatio.Jobs; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; using Microsoft.Extensions.Hosting; -using Microsoft.Extensions.Logging; namespace Foundatio.Extensions.Hosting.Jobs; public static class JobHostExtensions { - public static IServiceCollection AddJob(this IServiceCollection services, HostedJobOptions jobOptions) - { - if (jobOptions.JobFactory == null) - throw new ArgumentNullException(nameof(jobOptions), "jobOptions.JobFactory is required"); - - return services.AddTransient(s => new HostedJobService(s, jobOptions, s.GetRequiredService())); - } - - public static IServiceCollection AddJob(this IServiceCollection services, HostedJobOptions? jobOptions = null) where T : class, IJob - { - services.AddTransient(); - return services.AddTransient(s => - { - if (jobOptions == null) - { - jobOptions = new HostedJobOptions(); - jobOptions.ApplyDefaults(); - } - - jobOptions.Name ??= JobOptions.GetDefaultJobName(typeof(T)); - jobOptions.JobFactory ??= sp => sp.GetRequiredService(); - - return new HostedJobService(s, jobOptions, s.GetRequiredService()); - }); - } - - public static IServiceCollection AddJob(this IServiceCollection services, Action configureJobOptions) where T : class, IJob - { - var jobOptionsBuilder = new HostedJobOptionsBuilder(); - jobOptionsBuilder.ApplyDefaults(); - jobOptionsBuilder.Name(JobOptions.GetDefaultJobName(typeof(T))); - configureJobOptions?.Invoke(jobOptionsBuilder); - return services.AddJob(jobOptionsBuilder.Target); - } - - public static IServiceCollection AddJob(this IServiceCollection services, Action configureJobOptions) - { - var jobOptionsBuilder = new HostedJobOptionsBuilder(); - configureJobOptions?.Invoke(jobOptionsBuilder); - return services.AddJob(jobOptionsBuilder.Target); - } - - public static IServiceCollection AddJob(this IServiceCollection services, string name, Func jobFactory, Action configureJobOptions) - { - var jobOptionsBuilder = new HostedJobOptionsBuilder(); - jobOptionsBuilder.Name(name).JobFactory(jobFactory); - configureJobOptions?.Invoke(jobOptionsBuilder); - return services.AddJob(jobOptionsBuilder.Target); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, ScheduledJobOptions jobOptions) - { - if (jobOptions.JobFactory == null) - throw new ArgumentNullException(nameof(jobOptions), "jobOptions.JobFactory is required"); - - services.AddJobScheduler(); - - return services.AddTransient(s => new ScheduledJobRegistration(jobOptions)); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, Action configureJobOptions) - { - var jobOptionsBuilder = new ScheduledJobOptionsBuilder(); - configureJobOptions?.Invoke(jobOptionsBuilder); - return services.AddCronJob(jobOptionsBuilder.Target); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, string cronSchedule, Action? configureJobOptions = null) where T : class, IJob - { - services.AddTransient(); - var jobOptionsBuilder = new ScheduledJobOptionsBuilder(); - jobOptionsBuilder.Name(JobOptions.GetDefaultJobName(typeof(T))).CronSchedule(cronSchedule).JobFactory(sp => sp.GetRequiredService()); - configureJobOptions?.Invoke(jobOptionsBuilder); - return services.AddCronJob(jobOptionsBuilder.Target); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) - { - return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, action))); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) - { - return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (xp, _) => action(xp)))); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) - { - return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, _) => action()))); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) - { - return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (xp, ct) => - { - action(xp, ct); - return Task.CompletedTask; - }))); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) - { - return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, ct) => - { - action(ct); - return Task.CompletedTask; - }))); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) - { - return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, _) => - { - action(); - return Task.CompletedTask; - }))); - } - - public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string cronSchedule, Action? configureJobOptions = null) where T : class, IJob - { - services.AddTransient(); - var jobOptionsBuilder = new ScheduledJobOptionsBuilder(); - jobOptionsBuilder.Name(JobOptions.GetDefaultJobName(typeof(T))).Distributed().CronSchedule(cronSchedule).JobFactory(sp => sp.GetRequiredService()); - configureJobOptions?.Invoke(jobOptionsBuilder); - return services.AddCronJob(jobOptionsBuilder.Target); - } - - public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) - { - return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, action))); - } - - public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) - { - return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (xp, _) => action(xp)))); - } - - public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) - { - return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, _) => action()))); - } - - public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) - { - return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (xp, ct) => - { - action(xp, ct); - return Task.CompletedTask; - }))); - } - - public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) - { - return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, ct) => - { - action(ct); - return Task.CompletedTask; - }))); - } - - public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) - { - return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, _) => - { - action(); - return Task.CompletedTask; - }))); - } - - public static IServiceCollection AddJobScheduler(this IServiceCollection services) - { - if (!services.Any(s => s.ServiceType == typeof(IHostedService) && s.ImplementationType == typeof(ScheduledJobService))) - services.AddTransient(); - - if (!services.Any(s => s.ServiceType == typeof(JobManager) && s.ImplementationType == typeof(JobManager))) - services.AddSingleton(); - - if (!services.Any(s => s.ServiceType == typeof(IJobManager) && s.ImplementationType == typeof(JobManager))) - services.AddSingleton(sp => sp.GetRequiredService()); - + /// Runs registered durable job types on this host. Configure a runtime store first. + public static IServiceCollection AddJobWorker(this IServiceCollection services, int? concurrency = null) + { + if (concurrency is { } value) ArgumentOutOfRangeException.ThrowIfLessThan(value, 1); + var options = services.LastOrDefault(d => d.ServiceType == typeof(JobWorkerOptions))?.ImplementationInstance as JobWorkerOptions ?? new(); + services.Replace(ServiceDescriptor.Singleton(options with { MaxConcurrency = concurrency ?? options.MaxConcurrency })); + services.TryAddSingleton(); + services.TryAddEnumerable(ServiceDescriptor.Singleton()); return services; } - public static IServiceCollection AddJobLifetimeService(this IServiceCollection services) + /// Registers declared schedules and materializes due occurrences. Job execution requires AddJobWorker. + public static IServiceCollection AddJobScheduler(this IServiceCollection services) { - services.AddSingleton(); - services.AddSingleton(x => x.GetRequiredService()); + services.TryAddSingleton(); + services.TryAddEnumerable(ServiceDescriptor.Singleton()); return services; } } diff --git a/src/Foundatio.Extensions.Hosting/Jobs/JobManager.cs b/src/Foundatio.Extensions.Hosting/Jobs/JobManager.cs deleted file mode 100644 index d84fe2d8d..000000000 --- a/src/Foundatio.Extensions.Hosting/Jobs/JobManager.cs +++ /dev/null @@ -1,237 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Linq; -using System.Text.Json.Serialization; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Caching; -using Foundatio.Jobs; -using Foundatio.Utility; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Logging; - -namespace Foundatio.Extensions.Hosting.Jobs; - -public interface IJobManager -{ - void AddOrUpdate(Action? configure = null) where TJob : class, IJob; - void AddOrUpdate(string jobName, Action? configure = null); - void Update(Action? configure = null); - void Update(string jobName, Action? configure = null); - void Remove() where TJob : class, IJob; - void Remove(string jobName); - JobStatus[] GetJobStatus(bool runningOnly = false, bool includeHistory = true); - JobStatus GetJobStatus(string jobName, bool includeHistory = true); - Task RunJobAsync(CancellationToken cancellationToken = default) where TJob : class, IJob; - Task RunJobAsync(string jobName, CancellationToken cancellationToken = default); - Task ReleaseLockAsync(string jobName); -} - -public class JobManager : IJobManager -{ - private readonly IServiceProvider _serviceProvider; - private readonly ILoggerFactory _loggerFactory; - private readonly ICacheClient _cacheClient; - private readonly List _jobs = []; - private ScheduledJobInstance[] _jobsArray; - private readonly object _lock = new(); - - public JobManager(IServiceProvider serviceProvider, ILoggerFactory loggerFactory) - { - _serviceProvider = serviceProvider; - _loggerFactory = loggerFactory; - var cacheClient = serviceProvider.GetService(); - bool hasCacheClient = cacheClient is not null; - _cacheClient = cacheClient ?? new InMemoryCacheClient(o => o.LoggerFactory(loggerFactory)); - _jobs.AddRange(serviceProvider.GetServices().Select(j => new ScheduledJobInstance(j.Options, serviceProvider, _cacheClient, loggerFactory))); - _jobsArray = _jobs.ToArray(); - if (_jobs.Any(j => j.Options.IsDistributed && !hasCacheClient)) - throw new ArgumentException("A distributed cache client is required to run distributed jobs."); - } - - public void AddOrUpdate(Action? configure = null) where TJob : class, IJob - { - string jobName = JobOptions.GetDefaultJobName(typeof(TJob)); - lock (_lock) - { - var job = GetJob(jobName); - if (job == null) - { - var options = new ScheduledJobOptions - { - Name = jobName, - JobFactory = sp => sp.GetRequiredService() - }; - var builder = new ScheduledJobOptionsBuilder(options); - configure?.Invoke(builder); - _jobs.Add(new ScheduledJobInstance(options, _serviceProvider, _cacheClient, _loggerFactory)); - _jobsArray = _jobs.ToArray(); - } - else - { - var builder = new ScheduledJobOptionsBuilder(job.Options); - configure?.Invoke(builder); - } - } - } - - public void AddOrUpdate(string jobName, Action? configure = null) - { - lock (_lock) - { - var job = GetJob(jobName); - if (job == null) - { - var options = new ScheduledJobOptions - { - Name = jobName, - }; - var builder = new ScheduledJobOptionsBuilder(options); - configure?.Invoke(builder); - _jobs.Add(new ScheduledJobInstance(options, _serviceProvider, _cacheClient, _loggerFactory)); - _jobsArray = _jobs.ToArray(); - } - else - { - var builder = new ScheduledJobOptionsBuilder(job.Options); - configure?.Invoke(builder); - } - } - } - - public void Update(Action? configure = null) - { - string jobName = JobOptions.GetDefaultJobName(typeof(TJob)); - lock (_lock) - { - var job = GetJob(jobName); - if (job == null) - throw new ArgumentException("Job not found.", nameof(jobName)); - - var builder = new ScheduledJobOptionsBuilder(job.Options); - configure?.Invoke(builder); - } - } - - public void Update(string jobName, Action? configure = null) - { - lock (_lock) - { - var job = GetJob(jobName); - if (job == null) - throw new ArgumentException("Job not found.", nameof(jobName)); - - var builder = new ScheduledJobOptionsBuilder(job.Options); - configure?.Invoke(builder); - } - } - - public void Remove() where TJob : class, IJob - { - string jobName = JobOptions.GetDefaultJobName(typeof(TJob)); - lock (_lock) - { - var job = GetJob(jobName); - if (job == null) - return; - - _jobs.Remove(job); - _jobsArray = _jobs.ToArray(); - } - } - - public void Remove(string jobName) - { - lock (_lock) - { - var job = GetJob(jobName); - if (job == null) - return; - - _jobs.Remove(job); - _jobsArray = _jobs.ToArray(); - } - } - - public JobStatus[] GetJobStatus(bool runningOnly = false, bool includeHistory = true) - { - if (runningOnly) - return Jobs.Where(j => j.Running).Select(j => new JobStatus - { - Name = j.Options.Name, - Description = j.Options.Description, - Schedule = j.Options.CronSchedule, - Running = j.Running, - Enabled = j.Options.IsEnabled, - Distributed = j.Options.IsDistributed, - LastRun = j.LastRun, - NextRun = j.NextRun, - LastSuccess = j.LastSuccess, - History = includeHistory ? j.History ?? [] : null - }).ToArray(); - - return Jobs.Select(j => new JobStatus - { - Name = j.Options.Name, - Description = j.Options.Description, - Schedule = j.Options.CronSchedule, - Running = j.Running, - Enabled = j.Options.IsEnabled, - Distributed = j.Options.IsDistributed, - LastRun = j.LastRun, - NextRun = j.NextRun, - LastSuccess = j.LastSuccess, - History = includeHistory ? j.History ?? [] : null - }).ToArray(); - } - - public JobStatus GetJobStatus(string jobName, bool includeHistory = true) => - GetJobStatus(includeHistory: includeHistory).FirstOrDefault(j => String.Equals(j.Name, jobName, StringComparison.OrdinalIgnoreCase)) - ?? throw new ArgumentException("Job not found.", nameof(jobName)); - - public async Task RunJobAsync(CancellationToken cancellationToken = default) where TJob : class, IJob - { - string jobName = JobOptions.GetDefaultJobName(typeof(TJob)); - await RunJobAsync(jobName, cancellationToken).AnyContext(); - } - - public async Task RunJobAsync(string jobName, CancellationToken cancellationToken = default) - { - var job = GetJob(jobName); - if (job == null) - throw new ArgumentException("Job not found.", nameof(jobName)); - - await job.StartAsync(true, cancellationToken).AnyContext(); - } - - public async Task ReleaseLockAsync(string jobName) - { - var job = GetJob(jobName); - if (job == null) - throw new ArgumentException("Job not found.", nameof(jobName)); - - await job.ReleaseLockAsync().AnyContext(); - } - - internal ScheduledJobInstance? GetJob(string jobName) - { - return Jobs.FirstOrDefault(j => String.Equals(j.Options.Name, jobName, StringComparison.OrdinalIgnoreCase)); - } - - internal ScheduledJobInstance[] Jobs => _jobsArray; -} - -public class JobStatus -{ - public string? Name { get; set; } - public string? Description { get; set; } - public bool Running { get; set; } - public bool Enabled { get; set; } - public bool Distributed { get; set; } - public string? Schedule { get; set; } - public DateTime? LastRun { get; set; } - public DateTime? LastSuccess { get; set; } - public DateTime? NextRun { get; set; } - [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] - public List? History { get; set; } -} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/JobOptionsBuilder.cs b/src/Foundatio.Extensions.Hosting/Jobs/JobOptionsBuilder.cs deleted file mode 100644 index 9adb13a94..000000000 --- a/src/Foundatio.Extensions.Hosting/Jobs/JobOptionsBuilder.cs +++ /dev/null @@ -1,80 +0,0 @@ -using System; -using Foundatio.Jobs; - -namespace Foundatio.Extensions.Hosting.Jobs; - -public class HostedJobOptionsBuilder -{ - public HostedJobOptionsBuilder(HostedJobOptions? target = null) - { - Target = target ?? new HostedJobOptions(); - } - - public HostedJobOptions Target { get; } - - public HostedJobOptionsBuilder ApplyDefaults() where T : IJob - { - Target.ApplyDefaults(); - return this; - } - - public HostedJobOptionsBuilder ApplyDefaults(Type jobType) - { - JobOptions.ApplyDefaults(Target, jobType); - return this; - } - - public HostedJobOptionsBuilder Name(string value) - { - Target.Name = value; - return this; - } - - public HostedJobOptionsBuilder Description(string value) - { - Target.Description = value; - return this; - } - - public HostedJobOptionsBuilder JobFactory(Func value) - { - Target.JobFactory = value; - return this; - } - - public HostedJobOptionsBuilder RunContinuous(bool value = true) - { - Target.RunContinuous = value; - return this; - } - - public HostedJobOptionsBuilder Interval(TimeSpan? value) - { - Target.Interval = value; - return this; - } - - public HostedJobOptionsBuilder InitialDelay(TimeSpan? value) - { - Target.InitialDelay = value; - return this; - } - - public HostedJobOptionsBuilder IterationLimit(int value) - { - Target.IterationLimit = value; - return this; - } - - public HostedJobOptionsBuilder InstanceCount(int value) - { - Target.InstanceCount = value; - return this; - } - - public HostedJobOptionsBuilder WaitForStartupActions(bool value = true) - { - Target.WaitForStartupActions = value; - return this; - } -} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/JobSchedulerService.cs b/src/Foundatio.Extensions.Hosting/Jobs/JobSchedulerService.cs new file mode 100644 index 000000000..ff9628606 --- /dev/null +++ b/src/Foundatio.Extensions.Hosting/Jobs/JobSchedulerService.cs @@ -0,0 +1,55 @@ +using System.Collections.Generic; +using System.Threading.Tasks; +using System.Threading; +using System; +using Foundatio.Jobs; +using Foundatio.Utility; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; + +namespace Foundatio.Extensions.Hosting.Jobs; + +/// Registers declared schedules at startup and materializes due work without executing jobs. +internal sealed class JobSchedulerService(IServiceProvider services, ILogger logger, FoundatioRuntimeHealth health) : BackgroundService +{ + private JobScheduleProcessor? _processor; + + public override async Task StartAsync(CancellationToken cancellationToken) + { + if (services.GetService() is null) + throw new InvalidOperationException("A job scheduler was registered but no runtime store is configured, so jobs would never run. Call AddFoundatio().Jobs.UseInMemory() or UseRuntimeStore(...)."); + _processor = services.GetRequiredService(); + var store = services.GetRequiredService(); + foreach (var definition in services.GetServices()) + { + if (definition.Scope == ScheduledJobScope.PerNode) + NodeIdentity.RequireStable(services.GetService()?.NodeId); + await store.ReconcileAsync(definition, cancellationToken).AnyContext(); + } + await base.StartAsync(cancellationToken).AnyContext(); + } + + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + while (!stoppingToken.IsCancellationRequested) + { + try + { + await _processor!.EnqueueDueOccurrencesAsync(stoppingToken).AnyContext(); + health.Healthy("scheduler"); + await Task.Delay(TimeSpan.FromSeconds(1), stoppingToken).AnyContext(); + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + break; + } + catch (Exception ex) + { + health.Failed("scheduler", ex); + logger.LogError(ex, "Error creating scheduled job occurrences"); + await Task.Delay(TimeSpan.FromSeconds(1), stoppingToken).AnyContext(); + } + } + } +} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/JobWorkerService.cs b/src/Foundatio.Extensions.Hosting/Jobs/JobWorkerService.cs new file mode 100644 index 000000000..6262f0be6 --- /dev/null +++ b/src/Foundatio.Extensions.Hosting/Jobs/JobWorkerService.cs @@ -0,0 +1,48 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Utility; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; + +namespace Foundatio.Extensions.Hosting.Jobs; + +/// Executes jobs and maintains retention independently of long-running executions. +internal sealed class JobWorkerService(IJobWorker worker, IJobRuntimeStore store, ILogger logger, FoundatioRuntimeHealth health) : BackgroundService +{ + protected override Task ExecuteAsync(CancellationToken stoppingToken) + => Task.WhenAll(worker.RunContinuouslyAsync(stoppingToken), CleanupAsync(stoppingToken)); + + private async Task CleanupAsync(CancellationToken stoppingToken) + { + while (!stoppingToken.IsCancellationRequested) + { + var delay = TimeSpan.FromMinutes(1); + try + { + health.UpdateCapacity(await store.GetStatsAsync(stoppingToken).AnyContext()); + health.Healthy("job-store"); + if (await store.CleanupAsync(cancellationToken: stoppingToken).AnyContext() == 1000) + delay = TimeSpan.FromSeconds(1); + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + break; + } + catch (Exception ex) + { + health.Failed("job-store", ex); + logger.LogError(ex, "Error cleaning up job history"); + } + try + { + await Task.Delay(delay, stoppingToken).AnyContext(); + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + break; + } + } + } +} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobInstance.cs b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobInstance.cs deleted file mode 100644 index 8fe31d801..000000000 --- a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobInstance.cs +++ /dev/null @@ -1,450 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Diagnostics; -using System.Text.Json.Serialization; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Caching; -using Foundatio.Extensions.Hosting.Cronos; -using Foundatio.Jobs; -using Foundatio.Lock; -using Foundatio.Messaging; -using Foundatio.Utility; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Logging.Abstractions; - -namespace Foundatio.Extensions.Hosting.Jobs; - -internal class ScheduledJobInstance -{ - private readonly ScheduledJobOptions _jobOptions; - private readonly IServiceProvider _serviceProvider; - private readonly ICacheClient _cacheClient; - private readonly IMessageBus _messageBus; - private readonly TimeProvider _timeProvider; - private CronExpression? _cronExpression; - private readonly ILockProvider _lockProvider; - private readonly ILogger _logger; - private readonly DateTime _baseDate = new(2010, 1, 1); - - public ScheduledJobInstance(ScheduledJobOptions jobOptions, IServiceProvider serviceProvider, ICacheClient cacheClient, ILoggerFactory? loggerFactory = null) - { - _jobOptions = jobOptions; - _jobOptions.Name ??= Guid.NewGuid().ToString("N").Substring(0, 10); - CacheKey = _jobOptions.Name.ToLower().Replace(' ', '_'); - _serviceProvider = serviceProvider; - _timeProvider = serviceProvider.GetService() ?? TimeProvider.System; - _cacheClient = new ScopedCacheClient(cacheClient, "jobs"); - _logger = loggerFactory?.CreateLogger() ?? NullLogger.Instance; - - Id = Guid.NewGuid().ToString("N").Substring(0, 10); - - UpdateCronExpression(); - - _messageBus = serviceProvider.GetService() ?? new InMemoryMessageBus(); - _lockProvider = new CacheLockProvider(cacheClient, _messageBus, loggerFactory); - - _jobOptions.PropertyChanged += (_, args) => - { - if (args.PropertyName == nameof(ScheduledJobOptions.CronSchedule)) - { - UpdateCronExpression(); - - NextRun = GetNextScheduledRun(); - - _logger.LogDebug("Cron schedule changed for job {JobName} ({JobId}): {CronSchedule}", _jobOptions.Name, Id, _jobOptions.CronSchedule); - - // NOTE: Do we want to cancel this via DisposedCancellationToken? - Task.Run(() => UpdateDistributedStateAsync(true, "Cron schedule changed")); - } - - if (args.PropertyName == nameof(ScheduledJobOptions.IsEnabled)) - { - NextRun = GetNextScheduledRun(); - - // NOTE: Do we want to cancel this via DisposedCancellationToken? - Task.Run(() => UpdateDistributedStateAsync(true, "Enabled changed")); - } - }; - } - - private void UpdateCronExpression() - { - if (String.IsNullOrEmpty(_jobOptions.CronSchedule)) - { - _cronExpression = null; - return; - } - - try - { - _cronExpression = CronExpression.Parse(_jobOptions.CronSchedule); - } - catch (Exception) - { - _logger.LogError("Failed to parse cron expression: {CronSchedule}", _jobOptions.CronSchedule); - _cronExpression = null; - } - } - - public string Id { get; } - - public ScheduledJobOptions Options => _jobOptions; - - public DateTime? LastStateSync { get; internal set; } - public bool Running { get; internal set; } - public DateTime? NextRun { get; internal set; } - public DateTime? LastSuccess { get; internal set; } - public DateTime? LastRun { get; internal set; } - public List History { get; set; } = new(); - - internal bool SkipUpdate { get; set; } - - public Task? RunTask { get; private set; } - - internal string CacheKey { get; } - - public DateTime? GetNextScheduledRun() - { - if (Options.IsEnabled == false || _cronExpression == null) - return null; - - var lastRun = LastRun ?? _timeProvider.GetUtcNowDateTime(false).AddSeconds(-5); - var nextRun = _cronExpression.GetNextOccurrence(lastRun, _jobOptions.CronTimeZone ?? TimeZoneInfo.Local); - if (nextRun == null) - return null; - - if (nextRun < _timeProvider.GetUtcNowDateTime(false)) - { - var futureRun = _cronExpression.GetNextOccurrence(_timeProvider.GetUtcNowDateTime(false), _jobOptions.CronTimeZone ?? TimeZoneInfo.Local); - - // if next run is more than an hour in the past, use the future run - if (_timeProvider.GetUtcNowDateTime(false).Subtract(nextRun.Value) > TimeSpan.FromHours(1)) - nextRun = futureRun; - - // if the next run is within 10 minutes, use it - if (futureRun.HasValue && futureRun.Value.Subtract(_timeProvider.GetUtcNowDateTime(false)) < TimeSpan.FromMinutes(10)) - nextRun = futureRun; - } - - return nextRun; - } - - internal bool ShouldRun() - { - if (!Options.IsEnabled) - return false; - - if (!NextRun.HasValue) - return false; - - // not time yet - if (NextRun > _timeProvider.GetUtcNowDateTime(false)) - return false; - - // check if already run - if (LastRun != null && LastRun.Value == NextRun.Value) - return false; - - return true; - } - - public async Task ReleaseLockAsync() - { - if (!Options.IsDistributed) - return; - - _logger.LogDebug("Releasing lock for {JobName} ({JobId})", Options.Name, Id); - - try - { - await _lockProvider.ReleaseAsync(CacheKey).AnyContext(); - await _lockProvider.ReleaseAsync(GetLockKey(_baseDate)).AnyContext(); - - Running = false; - - await UpdateDistributedStateAsync(true); - } - catch (Exception ex) - { - _logger.LogError(ex, "Error releasing lock for {JobName} ({JobId}): {Message}", Options.Name, Id, ex.Message); - } - } - - public Task StartAsync(CancellationToken cancellationToken = default) - { - return StartAsync(false, cancellationToken); - } - - public async Task StartAsync(bool isManual, CancellationToken cancellationToken = default) - { - using var activity = FoundatioDiagnostics.ActivitySource.StartActivity("Job: " + Options.Name); - - var scheduledTime = isManual ? _baseDate : NextRun!.Value; - - ILock? jobRunningLock = null; - ILock? scheduledTimeLock = null; - if (Options.IsDistributed) - { - // using lock provider in a cluster with a distributed cache implementation keeps cron jobs from running duplicates - try - { - // hold this lock for 1 hour to prevent duplicates - scheduledTimeLock = await _lockProvider.TryAcquireAsync(GetLockKey(scheduledTime), TimeSpan.FromHours(1), TimeSpan.Zero).AnyContext(); - - if (scheduledTimeLock is not null) - { - // hold this lock while the job is running to prevent multiple instances of the job running at the same time - jobRunningLock = await _lockProvider.TryAcquireAsync(CacheKey, TimeSpan.FromMinutes(15), TimeSpan.Zero).AnyContext(); - - if (jobRunningLock is null) - await scheduledTimeLock.ReleaseAsync().AnyContext(); - } - } - catch (Exception ex) - { - _logger.LogError(ex, "Error acquiring locks for job ({JobName})", Options.Name); - if (scheduledTimeLock is not null) - await scheduledTimeLock.ReleaseAsync().AnyContext(); - scheduledTimeLock = null; - if (jobRunningLock is not null) - await jobRunningLock.ReleaseAsync().AnyContext(); - jobRunningLock = null; - } - - if (isManual && (scheduledTimeLock is null || jobRunningLock is null)) - _logger.LogWarning("Job ({JobName}) is already running, skipping manual request", Options.Name); - else if (jobRunningLock is null || scheduledTimeLock is null) - _logger.LogDebug("Job ({JobName}) scheduled on another instance", Options.Name); - - if (scheduledTimeLock is null || jobRunningLock is null) - { - // sync distributed state - await GetDistributedStateAsync(); - - return; - } - } - - // start running the job in a thread - RunTask = Task.Factory.StartNew(async () => - { - await using (jobRunningLock) - { - var utcNow = _timeProvider.GetUtcNowDateTime(false); - var jobRunResult = new JobRunResult { Date = utcNow }; - if (isManual) - jobRunResult.Manual = true; - else - jobRunResult.Scheduled = scheduledTime; - - var sw = new Stopwatch(); - - try - { - string jobRunId = Guid.NewGuid().ToString("N").Substring(0, 10); - using var _ = _logger.BeginScope(s => s.Property("job.name", Options.Name ?? String.Empty).Property("job.id", Id).Property("job.run_id", jobRunId)); - - _logger.LogDebug("{JobType} {JobName} ({JobId}) starting for time: {ScheduledTime}", Options.IsDistributed ? "Distributed job" : "Job", Options.Name, - Id, isManual ? "Manual" : NextRun!.Value.ToString("t")); - - await using var scope = _serviceProvider.CreateAsyncScope(); - - if (Options.JobFactory is null) - throw new InvalidOperationException($"JobFactory is not configured for job '{Options.Name}'."); - - var job = Options.JobFactory(scope.ServiceProvider); - - Running = true; - LastRun = isManual ? utcNow : NextRun; - NextRun = GetNextScheduledRun(); - - await UpdateDistributedStateAsync(true).AnyContext(); - - sw.Start(); - var result = await job.TryRunAsync(cancellationToken).AnyContext(); - sw.Stop(); - jobRunResult.Duration = sw.Elapsed; - - _logger.LogJobResult(result, Options.Name); - if (result.IsSuccess) - { - jobRunResult.Success = true; - LastSuccess = _timeProvider.GetUtcNowDateTime(false); - } - else - { - jobRunResult.Success = false; - - // TODO set next run time to retry, but need max retry count - } - } - catch (TaskCanceledException) - { - } - catch (Exception ex) - { - sw.Stop(); - jobRunResult.Duration = sw.Elapsed; - jobRunResult.Success = false; - jobRunResult.Error = ex.Message; - - if (scheduledTimeLock is not null) - await scheduledTimeLock.ReleaseAsync(); - - if (jobRunningLock is not null) - await jobRunningLock.ReleaseAsync(); - - // TODO set next run time to retry, but need max retry count - } - finally - { - Running = false; - AddJobRunResult(jobRunResult); - - await UpdateDistributedStateAsync(); - - if (isManual && scheduledTimeLock is not null) - await scheduledTimeLock.ReleaseAsync(); - } - } - }, cancellationToken).Unwrap(); - } - - private void AddJobRunResult(JobRunResult result) - { - if (result == null) - return; - - const int maxCount = 10; - - History.Insert(0, result); - if (History.Count > maxCount) - History.RemoveRange(maxCount, History.Count - maxCount); - } - - internal async Task UpdateDistributedStateAsync(bool setNextRun = false, string? reason = null) - { - if (!Options.IsDistributed || SkipUpdate) - return; - - try - { - var jobState = new JobInstanceState - { - Enabled = Options.IsEnabled, - Schedule = Options.CronSchedule, - Running = Running, - LastRun = LastRun, - LastSuccess = LastSuccess, - History = History - }; - - _logger.LogDebug("Updating distributed state for {JobName} ({JobId}): {JobState}", Options.Name, Id, Options.CronSchedule); - - if (setNextRun) - await _cacheClient.SetAsync(CacheKey + ":nextrun", NextRun).AnyContext(); - - await _cacheClient.SetAsync(CacheKey + ":state", jobState).AnyContext(); - - LastStateSync = _timeProvider.GetUtcNowDateTime(false); - - // send out change notification - await _messageBus.PublishAsync(new JobStateChangedMessage - { - Id = Id, - JobName = Options.Name, - Enabled = Options.IsEnabled, - Schedule = Options.CronSchedule, - Running = Running, - LastRun = LastRun, - LastSuccess = LastSuccess, - History = History, - Reason = reason - }).AnyContext(); - } - catch (Exception ex) - { - _logger.LogError(ex, "Error updating job state for {JobName} ({JobId}): {Message}", Options.Name, Id, ex.Message); - } - } - - internal async Task GetDistributedStateAsync() - { - if (!Options.IsDistributed) - return; - - try - { - _logger.LogDebug("Getting job state for {JobName} ({JobId})", Options.Name, Id); - - LastStateSync = _timeProvider.GetUtcNowDateTime(false); - - var cacheState = await _cacheClient.GetAsync(CacheKey + ":state").AnyContext(); - if (!cacheState.HasValue || cacheState.Value == null) - return; - - ApplyState(cacheState.Value); - } - catch (Exception ex) - { - _logger.LogError(ex, "Error getting job state for {JobName} ({JobId}): {Message}", Options.Name, Id, ex.Message); - } - } - - internal void ApplyState(JobInstanceState state, string? cronSchedule = null) - { - if (!Options.IsDistributed || state == null) - return; - - _logger.LogDebug("Applying job state for {JobName} ({JobId})", Options.Name, Id); - - Options.IsEnabled = state.Enabled; - Options.CronSchedule = cronSchedule ?? state.Schedule; - Running = state.Running; - LastRun = state.LastRun; - LastSuccess = state.LastSuccess; - History = state.History; - NextRun = GetNextScheduledRun(); - - LastStateSync = _timeProvider.GetUtcNowDateTime(false); - } - - private string GetLockKey(DateTime date) - { - long minute = (long)date.Subtract(_baseDate).TotalMinutes; - - return CacheKey + ":" + minute; - } -} - -public class JobInstanceState -{ - public string? Schedule { get; set; } - public bool Enabled { get; set; } - public bool Running { get; set; } - public DateTime? LastRun { get; set; } - public DateTime? LastSuccess { get; set; } - public List History { get; set; } = []; -} - -public class JobRunResult -{ - public DateTime? Date { get; set; } - [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] - public DateTime? Scheduled { get; set; } - [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] - public bool? Manual { get; set; } - public bool Success { get; set; } - public TimeSpan? Duration { get; set; } - [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] - public string? Error { get; set; } -} - -public class JobStateChangedMessage : JobInstanceState -{ - public string? Id { get; set; } - public string? JobName { get; set; } - public string? Reason { get; set; } -} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobOptions.cs b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobOptions.cs deleted file mode 100644 index b1279a284..000000000 --- a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobOptions.cs +++ /dev/null @@ -1,79 +0,0 @@ -using System; -using System.ComponentModel; -using System.Runtime.CompilerServices; -using Foundatio.Jobs; - -namespace Foundatio.Extensions.Hosting.Jobs; - -public class ScheduledJobOptions : INotifyPropertyChanged -{ - private string? _name; - private string? _description; - private Func? _jobFactory; - private bool _waitForStartupActions; - private string? _cronSchedule; - private TimeZoneInfo? _cronTimeZone; - private bool _isDistributed; - private bool _isEnabled = true; - - public string? Name - { - get => _name; - set => SetField(ref _name, value); - } - - public string? Description - { - get => _description; - set => SetField(ref _description, value); - } - - public Func? JobFactory - { - get => _jobFactory; - set => SetField(ref _jobFactory, value); - } - - public bool WaitForStartupActions - { - get => _waitForStartupActions; - set => SetField(ref _waitForStartupActions, value); - } - - public string? CronSchedule - { - get => _cronSchedule; - set => SetField(ref _cronSchedule, value); - } - - public TimeZoneInfo? CronTimeZone - { - get => _cronTimeZone; - set => SetField(ref _cronTimeZone, value); - } - - public bool IsDistributed - { - get => _isDistributed; - set => SetField(ref _isDistributed, value); - } - - public bool IsEnabled - { - get => _isEnabled; - set => SetField(ref _isEnabled, value); - } - - public event PropertyChangedEventHandler? PropertyChanged; - - private bool SetField(ref T field, T value, [CallerMemberName] string propertyName = "") - { - if (Equals(field, value)) - return false; - - field = value; - PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(propertyName)); - return true; - } -} - diff --git a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobOptionsBuilder.cs b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobOptionsBuilder.cs deleted file mode 100644 index 5fda06683..000000000 --- a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobOptionsBuilder.cs +++ /dev/null @@ -1,114 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Jobs; - -namespace Foundatio.Extensions.Hosting.Jobs; - -public class ScheduledJobOptionsBuilder -{ - public ScheduledJobOptionsBuilder(ScheduledJobOptions? target = null) - { - Target = target ?? new ScheduledJobOptions(); - } - - public ScheduledJobOptions Target { get; } - - public ScheduledJobOptionsBuilder Name(string value) - { - Target.Name = value; - return this; - } - - public ScheduledJobOptionsBuilder Description(string value) - { - Target.Description = value; - return this; - } - - public ScheduledJobOptionsBuilder CronSchedule(string value) - { - Target.CronSchedule = value; - return this; - } - - public ScheduledJobOptionsBuilder CronTimeZone(string id) - { - Target.CronTimeZone = TimeZoneInfo.FindSystemTimeZoneById(id); - return this; - } - - public ScheduledJobOptionsBuilder CronTimeZone(TimeZoneInfo value) - { - Target.CronTimeZone = value; - return this; - } - - public ScheduledJobOptionsBuilder JobFactory(Func value) - { - Target.JobFactory = value; - return this; - } - - public ScheduledJobOptionsBuilder JobAction(Func action) - { - Target.JobFactory = sp => new DynamicJob(sp, action); - return this; - } - - public ScheduledJobOptionsBuilder JobAction(Func action) - { - Target.JobFactory = sp => new DynamicJob(sp, (xp, _) => action(xp)); - return this; - } - - public ScheduledJobOptionsBuilder JobAction(Func action) - { - Target.JobFactory = sp => new DynamicJob(sp, (_, _) => action()); - return this; - } - - public ScheduledJobOptionsBuilder JobAction(Action action) - { - Target.JobFactory = sp => new DynamicJob(sp, (_, ct) => - { - action(ct); - return Task.CompletedTask; - }); - return this; - } - - public ScheduledJobOptionsBuilder JobAction(Action action) - { - Target.JobFactory = sp => new DynamicJob(sp, (_, _) => - { - action(); - return Task.CompletedTask; - }); - return this; - } - - public ScheduledJobOptionsBuilder WaitForStartupActions(bool value = true) - { - Target.WaitForStartupActions = value; - return this; - } - - public ScheduledJobOptionsBuilder Distributed(bool value = true) - { - Target.IsDistributed = value; - return this; - } - - public ScheduledJobOptionsBuilder Enabled(bool value = true) - { - Target.IsEnabled = value; - return this; - } - - public ScheduledJobOptionsBuilder Disabled() - { - Target.IsEnabled = false; - return this; - } -} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobRegistration.cs b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobRegistration.cs deleted file mode 100644 index 6d7269e75..000000000 --- a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobRegistration.cs +++ /dev/null @@ -1,11 +0,0 @@ -namespace Foundatio.Extensions.Hosting.Jobs; - -public class ScheduledJobRegistration -{ - public ScheduledJobRegistration(ScheduledJobOptions options) - { - Options = options; - } - - public ScheduledJobOptions Options { get; private set; } -} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobService.cs b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobService.cs deleted file mode 100644 index 39bcdfbd9..000000000 --- a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobService.cs +++ /dev/null @@ -1,155 +0,0 @@ -using System; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Caching; -using Foundatio.Extensions.Hosting.Startup; -using Foundatio.Messaging; -using Foundatio.Utility; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Hosting; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Logging.Abstractions; - -namespace Foundatio.Extensions.Hosting.Jobs; - -public class ScheduledJobService : BackgroundService -{ - private readonly IServiceProvider _serviceProvider; - private readonly JobManager _jobManager; - private readonly TimeProvider _timeProvider; - private readonly ICacheClient _cacheClient; - private readonly ILogger _logger; - private readonly IMessageBus _messageBus; - - public ScheduledJobService(IServiceProvider serviceProvider, JobManager jobManager) - { - _serviceProvider = serviceProvider; - _jobManager = jobManager; - _timeProvider = serviceProvider.GetService() ?? TimeProvider.System; - var loggerFactory = serviceProvider.GetService() ?? NullLoggerFactory.Instance; - var cacheClient = serviceProvider.GetService() ?? new InMemoryCacheClient(o => o.LoggerFactory(loggerFactory)); - _cacheClient = new ScopedCacheClient(cacheClient, "jobs"); - _messageBus = serviceProvider.GetService() ?? new NullMessageBus(); - _logger = serviceProvider.GetService>() ?? NullLogger.Instance; - } - - protected override async Task ExecuteAsync(CancellationToken stoppingToken) - { - var startupContext = _serviceProvider.GetService(); - if (startupContext != null) - { - var result = await startupContext.WaitForStartupAsync(stoppingToken).AnyContext(); - if (!result.Success) - { - throw new StartupActionsException("Failed to wait for startup actions to complete"); - } - } - - await _messageBus.SubscribeAsync(s => - { - if (String.IsNullOrEmpty(s.JobName)) - { - _logger.LogWarning("Received job state change message with no job name, ignoring"); - return Task.CompletedTask; - } - - var job = _jobManager.GetJob(s.JobName); - if (job is null) - { - _logger.LogWarning("Received job state change for unknown job {JobName}, ignoring", s.JobName); - return Task.CompletedTask; - } - - if (String.Equals(s.Id, job.Id, StringComparison.Ordinal)) - return Task.CompletedTask; - - if (!String.IsNullOrEmpty(s.Reason)) - _logger.LogInformation("Received job state change for {JobName} from {Id} ({JobId}) Reason: {Reason}", s.JobName, s.Id, job.Id, s.Reason); - else - _logger.LogDebug("Received job state change for {JobName} from {Id} ({JobId})", s.JobName, s.Id, job.Id); - - // skip update to prevent infinite loop - job.SkipUpdate = true; - job.ApplyState(s); - job.SkipUpdate = false; - - return Task.CompletedTask; - }, cancellationToken: stoppingToken); - - try - { - _logger.LogDebug("Applying initial distributed job states..."); - var distributedJobs = _jobManager.Jobs.Where(j => j.Options.IsDistributed).ToDictionary(j => j.CacheKey + ":state", j => j); - var distributedJobStates = await _cacheClient.GetAllAsync(distributedJobs.Keys).AnyContext(); - - foreach (var distributedJob in distributedJobs) - { - var job = distributedJob.Value; - - if (!distributedJobStates.TryGetValue(distributedJob.Key, out var jobState) || !jobState.HasValue) - continue; - - _logger.LogDebug("Applying distributed state for job {JobName} ({JobId})", distributedJob.Value.Options.Name, job.Id); - - if (!String.Equals(job.Options.CronSchedule, jobState.Value.Schedule, StringComparison.Ordinal)) - { - _logger.LogInformation("Cron schedule changed for job {JobName} from {OldCronSchedule} to {NewCronSchedule} ({JobId})", - job.Options.Name, jobState.Value.Schedule, job.Options.CronSchedule, job.Id); - - // if cron schedule is different from distributed state, set it explicitly - job.ApplyState(jobState.Value, job.Options.CronSchedule); - await job.UpdateDistributedStateAsync(true, "Cron schedule changed").AnyContext(); - } - else - { - job.ApplyState(jobState.Value); - } - } - } - catch (Exception ex) - { - _logger.LogError(ex, "Error applying initial distributed job states: {Message}", ex.Message); - } - - // delay until right after next minute starts to sync with cron schedules - await _timeProvider.Delay(TimeSpan.FromMinutes(1) - TimeSpan.FromSeconds(_timeProvider.GetUtcNow().Second) - TimeSpan.FromMilliseconds(_timeProvider.GetUtcNow().Millisecond), stoppingToken).AnyContext(); - - while (!stoppingToken.IsCancellationRequested) - { - using (FoundatioDiagnostics.ActivitySource.StartActivity("Job Scheduler")) - { - try - { - var jobNextRuns = _jobManager.Jobs.ToDictionary(j => j.CacheKey + ":nextrun", j => j); - var jobNextRunTimes = await _cacheClient.GetAllAsync(jobNextRuns.Keys).AnyContext(); - - foreach ((string nextRunKey, ScheduledJobInstance job) in jobNextRuns) - { - if (jobNextRunTimes.TryGetValue(nextRunKey, out var nextRunTime) && nextRunTime.HasValue) - { - if (!nextRunTime.IsNull) - job.NextRun = DateTime.SpecifyKind(nextRunTime.Value, DateTimeKind.Utc); - else - job.NextRun = null; - } - - job.NextRun ??= job.GetNextScheduledRun(); - } - } - catch (Exception ex) - { - _logger.LogError(ex, "Error retrieving job next run times: {Message}", ex.Message); - } - } - - foreach (var jobToRun in _jobManager.Jobs.Where(j => j.ShouldRun())) - { - await jobToRun.StartAsync(stoppingToken).AnyContext(); - } - - // shortest cron schedule is 1 minute so only check every minute - await _timeProvider.Delay(TimeSpan.FromMinutes(1) - TimeSpan.FromSeconds(_timeProvider.GetUtcNow().Second) - TimeSpan.FromMilliseconds(_timeProvider.GetUtcNow().Millisecond), stoppingToken).AnyContext(); - } - } -} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/ShutdownHostIfNoJobsRunningService.cs b/src/Foundatio.Extensions.Hosting/Jobs/ShutdownHostIfNoJobsRunningService.cs deleted file mode 100644 index bb5a6c74d..000000000 --- a/src/Foundatio.Extensions.Hosting/Jobs/ShutdownHostIfNoJobsRunningService.cs +++ /dev/null @@ -1,82 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Extensions.Hosting.Startup; -using Foundatio.Utility; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Hosting; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Logging.Abstractions; - -namespace Foundatio.Extensions.Hosting.Jobs; - -public class ShutdownHostIfNoJobsRunningService : IHostedService, IDisposable -{ - private Timer? _timer; - private readonly List _jobs = new(); - private readonly IHostApplicationLifetime _lifetime; - private readonly IServiceProvider _serviceProvider; - private bool _isStarted = false; - private readonly ILogger _logger; - - public ShutdownHostIfNoJobsRunningService(IHostApplicationLifetime applicationLifetime, IServiceProvider serviceProvider, ILogger logger) - { - ArgumentNullException.ThrowIfNull(applicationLifetime); - - _lifetime = applicationLifetime; - _serviceProvider = serviceProvider; - _logger = logger ?? NullLogger.Instance; - - _lifetime.ApplicationStarted.Register(() => - { - _timer = new Timer(e => CheckForShutdown(), null, TimeSpan.FromSeconds(3), TimeSpan.FromSeconds(2)); - }); - } - - public Task StartAsync(CancellationToken cancellationToken) - { - // if there are startup actions, don't allow shutdown to happen until after the startup actions have completed - _ = Task.Run(async () => - { - var startupContext = _serviceProvider.GetService(); - if (startupContext != null) - await startupContext.WaitForStartupAsync(cancellationToken).AnyContext(); - - _isStarted = true; - }, cancellationToken); - - return Task.CompletedTask; - } - - public Task StopAsync(CancellationToken cancellationToken) - { - _timer?.Change(Timeout.Infinite, 0); - return Task.CompletedTask; - } - - public void RegisterHostedJobInstance(IJobStatus job) - { - _jobs.Add(job); - } - - public void CheckForShutdown() - { - if (!_isStarted) - return; - - int runningJobCount = _jobs.Count(s => s.IsRunning); - if (runningJobCount != 0) - return; - - _timer?.Change(Timeout.Infinite, 0); - _logger.LogInformation("Stopping host due to no running jobs"); - _lifetime.StopApplication(); - } - - public void Dispose() - { - _timer?.Dispose(); - } -} diff --git a/src/Foundatio.Extensions.Hosting/Messaging/MessageHandlerHostedService.cs b/src/Foundatio.Extensions.Hosting/Messaging/MessageHandlerHostedService.cs new file mode 100644 index 000000000..304bb1695 --- /dev/null +++ b/src/Foundatio.Extensions.Hosting/Messaging/MessageHandlerHostedService.cs @@ -0,0 +1,139 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Foundatio.Utility; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Foundatio.Extensions.Hosting.Messaging; + +/// +/// Applies the app's declared topology at startup for EVERY app with a configured transport — including publish-only +/// apps that register no handlers. Ensure creates what the routing config declares; Validate proves it exists and +/// fails boot when it doesn't (a missing destination should stop the app at startup, not surface as runtime send +/// errors); None trusts out-of-band provisioning entirely. +/// +internal sealed class MessagingTopologyStartupService : IHostedService +{ + private readonly IServiceProvider _serviceProvider; + private readonly ILogger _logger; + + public MessagingTopologyStartupService(IServiceProvider serviceProvider, ILoggerFactory? loggerFactory = null) + { + _serviceProvider = serviceProvider; + _logger = loggerFactory?.CreateLogger() ?? NullLogger.Instance; + } + + public async Task StartAsync(CancellationToken cancellationToken) + { + var types = _serviceProvider.GetService(); + var routes = _serviceProvider.GetService(); + if (types is not null && routes is not null) + { + foreach (var mapping in routes.GetRouteMaps()) + _logger.LogInformation("Message contract {MessageType} uses wire name {WireName} and {Role} route {Route}", + mapping.MessageType.FullName, types.GetName(mapping.MessageType), mapping.Role, mapping.Route); + } + var mode = (_serviceProvider.GetService(typeof(MessagingTopologyOptions)) as MessagingTopologyOptions)?.Mode ?? TopologyMode.Ensure; + if (mode == TopologyMode.None) + return; + + if (_serviceProvider.GetService(typeof(IMessageTopology)) is not IMessageTopology topology) + return; + + if (mode == TopologyMode.Validate) + { + await topology.ValidateAsync(cancellationToken).AnyContext(); + _logger.LogInformation("Validated declared message topology"); + return; + } + + try + { + await topology.EnsureAsync(cancellationToken).AnyContext(); + _logger.LogInformation("Ensured declared message topology"); + } + catch (NotSupportedException) + { + // The transport cannot provision; the runtime use-time paths no-op the same way, so startup should not fail. + _logger.LogDebug("Transport does not support topology provisioning; skipping startup ensure"); + } + } + + public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask; +} + +/// +/// Hosts every declaratively-registered message handler for the app's lifetime: on start it launches each handler's +/// consumer/subscription; on stop it disposes them. Auto-registered when the first handler is added, so users register +/// handlers in configuration and never hand-write a hosted service. Programmatic +/// remain available for dynamic use. +/// +internal sealed class MessageHandlerHostedService : IHostedService +{ + private readonly IServiceProvider _serviceProvider; + private readonly IEnumerable _registrations; + private readonly ILogger _logger; + private readonly List _started = new(); + internal IReadOnlyList Subscriptions { get { lock (_started) return _started.OfType().ToArray(); } } + + public MessageHandlerHostedService(IServiceProvider serviceProvider, IEnumerable registrations, ILoggerFactory? loggerFactory = null) + { + _serviceProvider = serviceProvider; + _registrations = registrations; + _logger = loggerFactory?.CreateLogger() ?? NullLogger.Instance; + } + + public async Task StartAsync(CancellationToken cancellationToken) + { + if (_registrations.Any() && _serviceProvider.GetService() is null) + throw new InvalidOperationException("Message consumers were registered but no message transport is configured. Call AddFoundatio().Messaging.UseTransport(...) or UseInMemory()."); + try + { + foreach (var registration in _registrations) + { + var disposable = await registration.StartAsync(_serviceProvider, cancellationToken).AnyContext(); + lock (_started) _started.Add(disposable); + _logger.LogInformation("Started message handler {Handler}", registration.Description); + } + } + catch + { + // A hosted service whose StartAsync throws is not sent StopAsync, so dispose whatever we already started + // rather than leaking those consumers' background receive loops. + await DisposeStartedAsync().AnyContext(); + throw; + } + } + + public Task StopAsync(CancellationToken cancellationToken) => DisposeStartedAsync(); + + private async Task DisposeStartedAsync() + { + try + { + // Dispose every started consumer even if one throws (e.g. a broker connection dropped mid-shutdown), so a + // single failure can't leak the rest. + foreach (var disposable in _started) + { + try + { + await disposable.DisposeAsync().AnyContext(); + } + catch (Exception ex) + { + _logger.LogError(ex, "Error disposing message handler consumer: {Message}", ex.Message); + } + } + } + finally + { + lock (_started) _started.Clear(); + } + } +} diff --git a/src/Foundatio.Extensions.Hosting/Messaging/MessagingHostExtensions.cs b/src/Foundatio.Extensions.Hosting/Messaging/MessagingHostExtensions.cs new file mode 100644 index 000000000..3a4922a9c --- /dev/null +++ b/src/Foundatio.Extensions.Hosting/Messaging/MessagingHostExtensions.cs @@ -0,0 +1,39 @@ +using System; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; + +namespace Foundatio.Extensions.Hosting.Messaging; + +public static class MessagingHostExtensions +{ + /// Starts registered queue consumers and event subscribers for the host lifetime. + public static IServiceCollection AddMessageConsumers(this IServiceCollection services) + { + services.AddMessagingTopology(); + services.TryAddEnumerable(ServiceDescriptor.Singleton()); + return services; + } + + /// Ensures or validates declared messaging topology at startup using the configured topology mode. + public static IServiceCollection AddMessagingTopology(this IServiceCollection services) + { + services.TryAddEnumerable(ServiceDescriptor.Singleton()); + return services; + } + + /// Dispatches persisted delayed messages independently of job execution. + public static IServiceCollection AddScheduledMessageDispatcher(this IServiceCollection services) + { + services.TryAddSingleton(); + services.TryAddSingleton(sp => new ScheduledMessageDispatcher( + sp.GetService() ?? sp.GetRequiredService(), + sp.GetRequiredService(), + new ScheduledMessageDispatcherOptions { TimeProvider = sp.GetService(), LoggerFactory = sp.GetService(), TopologyMode = sp.GetService()?.Mode ?? TopologyMode.Ensure })); + services.TryAddEnumerable(ServiceDescriptor.Singleton()); + return services; + } +} diff --git a/src/Foundatio.Extensions.Hosting/Messaging/ScheduledMessageDispatcherService.cs b/src/Foundatio.Extensions.Hosting/Messaging/ScheduledMessageDispatcherService.cs new file mode 100644 index 000000000..deab29ba0 --- /dev/null +++ b/src/Foundatio.Extensions.Hosting/Messaging/ScheduledMessageDispatcherService.cs @@ -0,0 +1,46 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Foundatio.Jobs; +using Microsoft.Extensions.DependencyInjection; +using Foundatio.Utility; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; + +namespace Foundatio.Extensions.Hosting.Messaging; + +internal sealed class ScheduledMessageDispatcherService(ScheduledMessageDispatcher dispatcher, ILogger logger, FoundatioRuntimeHealth health, IServiceProvider services) : BackgroundService +{ + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + var nextStats = DateTimeOffset.MinValue; + var store = services.GetService() as IJobRuntimeStore ?? services.GetService(); + while (!stoppingToken.IsCancellationRequested) + { + try + { + if (store is not null && DateTimeOffset.UtcNow >= nextStats) + { + health.UpdateScheduledDispatches((await store.GetStatsAsync(stoppingToken).AnyContext()).ScheduledDispatches); + nextStats = DateTimeOffset.UtcNow.AddMinutes(1); + } + int dispatched = await dispatcher.DispatchDueAsync(cancellationToken: stoppingToken).AnyContext(); + if (dispatcher.LastFailure is { } failure) health.Failed("dispatcher", failure); + else health.Healthy("dispatcher"); + if (dispatched == 0) + await Task.Delay(TimeSpan.FromSeconds(1), stoppingToken).AnyContext(); + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + break; + } + catch (Exception ex) + { + health.Failed("dispatcher", ex); + logger.LogError(ex, "Error dispatching scheduled messages"); + await Task.Delay(TimeSpan.FromSeconds(1), stoppingToken).AnyContext(); + } + } + } +} diff --git a/src/Foundatio.Redis/Foundatio.Redis.csproj b/src/Foundatio.Redis/Foundatio.Redis.csproj new file mode 100644 index 000000000..f70dd8073 --- /dev/null +++ b/src/Foundatio.Redis/Foundatio.Redis.csproj @@ -0,0 +1,11 @@ + + + Redis-backed durable job runtime store (IJobRuntimeStore) for Foundatio. Temporary in-repo provider for validating the redesigned job runtime against a real distributed store. + + + + + + + + diff --git a/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.Scripts.cs b/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.Scripts.cs new file mode 100644 index 000000000..a594be36c --- /dev/null +++ b/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.Scripts.cs @@ -0,0 +1,160 @@ +namespace Foundatio.Messaging; + +public sealed partial class RedisStreamsMessageTransport +{ + private const string TopicRetentionFunctions = """ + local function hex(value) + return (string.gsub(value, '.', function(c) return string.format('%02X', string.byte(c)) end)) + end + local function cleanupSubscriptions(stream, now) + local leases = stream .. ':subscriptions' + for _, group in ipairs(redis.call('ZRANGEBYSCORE', leases, '-inf', now, 'LIMIT', 0, 100)) do + if redis.call('EXISTS', stream) == 1 then redis.call('XGROUP', 'DESTROY', stream, group) end + redis.call('DEL', stream .. ':lock:' .. hex(group), stream .. ':meta:' .. hex(group), stream .. ':dead:' .. hex(group)) + redis.call('ZREM', leases, group) + end + end + local function less(a, b) + local am, as = string.match(a, '^(%d+)%-(%d+)$') + local bm, bs = string.match(b, '^(%d+)%-(%d+)$') + return tonumber(am) < tonumber(bm) or (tonumber(am) == tonumber(bm) and tonumber(as) < tonumber(bs)) + end + local function trimTopic(stream, now, force) + local cadence = stream .. ':trim' + if not force and redis.call('EXISTS', cadence) == 1 then return end + redis.call('SET', cadence, '1', 'PX', 1000) + cleanupSubscriptions(stream, now) + if redis.call('EXISTS', stream) == 0 then return end + local groups = redis.call('XINFO', 'GROUPS', stream) + if #groups == 0 then redis.call('XTRIM', stream, 'MAXLEN', 0) return end + local boundary, keep + for _, group in ipairs(groups) do + local name, last + for index = 1, #group, 2 do + if group[index] == 'name' then name = group[index + 1] end + if group[index] == 'last-delivered-id' then last = group[index + 1] end + end + local pending = redis.call('XPENDING', stream, name, '-', '+', 1) + local candidate = #pending > 0 and pending[1][1] or last + local preserve = #pending > 0 + if not boundary or less(candidate, boundary) then + boundary, keep = candidate, preserve + elseif candidate == boundary and preserve then keep = true end + end + redis.call('XTRIM', stream, 'MINID', boundary) + if not keep then redis.call('XDEL', stream, boundary) end + end + """; + + private const string SendScript = TopicRetentionFunctions + """ + + if ARGV[1] == '1' then trimTopic(KEYS[1], ARGV[3], redis.call('XLEN', KEYS[1]) >= tonumber(ARGV[2])) end + if redis.call('XLEN', KEYS[1]) >= tonumber(ARGV[2]) then + return redis.error_reply('The destination has reached its pending-message capacity.') + end + local id = redis.call('XADD', KEYS[1], '*', unpack(ARGV, 4)) + if ARGV[1] == '1' then trimTopic(KEYS[1], ARGV[3]) end + return id + """; + + private const string ReplayScript = TopicRetentionFunctions + """ + + local entries = redis.call('XRANGE', KEYS[1], ARGV[1], ARGV[1], 'COUNT', 1) + if #entries == 0 then return 0 end + if ARGV[2] == '1' then trimTopic(KEYS[2], ARGV[4], redis.call('XLEN', KEYS[2]) >= tonumber(ARGV[3])) end + if redis.call('XLEN', KEYS[2]) >= tonumber(ARGV[3]) then return redis.error_reply('Replay destination is full.') end + local fields = entries[1][2] + for index = 1, #fields, 2 do + if fields[index] == 'h' then + local headers = cjson.decode(fields[index + 1]) + for key, _ in pairs(headers) do + local normalized = string.lower(key) + if normalized == 'message.attempts' or normalized == 'message.expiration' or string.sub(normalized, 1, 20) == 'message.dead_letter.' then headers[key] = nil end + end + fields[index + 1] = cjson.encode(headers) + end + end + redis.call('XADD', KEYS[2], '*', unpack(fields)) + if ARGV[2] == '1' then trimTopic(KEYS[2], ARGV[4]) end + redis.call('XDEL', KEYS[1], ARGV[1]) + return 1 + """; + + private const string ReceiveScript = TopicRetentionFunctions + """ + + cleanupSubscriptions(KEYS[1], ARGV[3]) + local result = {} + local now, visibility, maximum = tonumber(ARGV[3]), tonumber(ARGV[4]), tonumber(ARGV[5]) + local function track(entry, deliveries) + local token = ARGV[6] .. ':' .. entry[1] + redis.call('HSET', KEYS[3], entry[1], token .. '|' .. deliveries) + redis.call('ZADD', KEYS[2], now + visibility, entry[1]) + table.insert(result, {entry[1], entry[2], deliveries, token}) + end + local due = redis.call('ZRANGEBYSCORE', KEYS[2], '-inf', now, 'LIMIT', 0, maximum) + for _, id in ipairs(due) do + local claimed = redis.call('XCLAIM', KEYS[1], ARGV[1], ARGV[2], 0, id) + if #claimed > 0 then + local meta = redis.call('HGET', KEYS[3], id) or '' + local deliveries = tonumber(string.match(meta, '|(%d+)$') or '0') + 1 + track(claimed[1], deliveries) + else + redis.call('ZREM', KEYS[2], id) + redis.call('HDEL', KEYS[3], id) + end + end + if #result < maximum then + local cursor = redis.call('HGET', KEYS[3], '@orphan-cursor') or '-' + local pending = redis.call('XPENDING', KEYS[1], ARGV[1], cursor, '+', 100) + local visited = 0 + for _, item in ipairs(pending) do + if #result >= maximum then break end + visited = visited + 1 + cursor = '(' .. item[1] + if tonumber(item[3]) >= tonumber(ARGV[7]) and + (redis.call('HEXISTS', KEYS[3], item[1]) == 0 or not redis.call('ZSCORE', KEYS[2], item[1])) then + local claimed = redis.call('XCLAIM', KEYS[1], ARGV[1], ARGV[2], 0, item[1]) + if #claimed > 0 then track(claimed[1], tonumber(item[4]) + 1) end + end + end + if visited == #pending and #pending < 100 then cursor = '-' end + redis.call('HSET', KEYS[3], '@orphan-cursor', cursor) + end + if #result < maximum then + local fresh = redis.call('XREADGROUP', 'GROUP', ARGV[1], ARGV[2], 'COUNT', maximum - #result, 'STREAMS', KEYS[1], '>') + if fresh then + for _, entry in ipairs(fresh[1][2]) do track(entry, 1) end + end + end + return result + """; + + private const string SettleScript = TopicRetentionFunctions + """ + + local meta = redis.call('HGET', KEYS[3], ARGV[2]) + local expires = tonumber(redis.call('ZSCORE', KEYS[2], ARGV[2]) or '0') + if not meta or string.match(meta, '^([^|]*)') ~= ARGV[3] or expires <= tonumber(ARGV[4]) then return 0 end + if #redis.call('XPENDING', KEYS[1], ARGV[1], ARGV[2], ARGV[2], 1) == 0 then return 0 end + if ARGV[5] == 'renew' then + redis.call('ZADD', KEYS[2], ARGV[6], ARGV[2]) + return 1 + end + if ARGV[5] == 'abandon' then + redis.call('ZADD', KEYS[2], ARGV[6], ARGV[2]) + redis.call('HSET', KEYS[3], ARGV[2], '|' .. (string.match(meta, '|(%d+)$') or '1')) + return 1 + end + if ARGV[5] == 'deadletter' then + if redis.call('XLEN', KEYS[4]) >= tonumber(ARGV[8]) then + return redis.error_reply('The dead-letter destination has reached its capacity.') + end + redis.call('XADD', KEYS[4], '*', unpack(ARGV, 9)) + end + redis.call('XACK', KEYS[1], ARGV[1], ARGV[2]) + if ARGV[7] == '1' then redis.call('XDEL', KEYS[1], ARGV[2]) end + redis.call('ZREM', KEYS[2], ARGV[2]) + redis.call('HDEL', KEYS[3], ARGV[2]) + if ARGV[7] == '0' then trimTopic(KEYS[1], ARGV[4]) end + return 1 + """; +} diff --git a/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.Subscriptions.cs b/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.Subscriptions.cs new file mode 100644 index 000000000..1a7f763e0 --- /dev/null +++ b/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.Subscriptions.cs @@ -0,0 +1,50 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using StackExchange.Redis; + +namespace Foundatio.Messaging; + +public sealed partial class RedisStreamsMessageTransport +{ + private async Task EnsureTemporarySubscriptionAsync(DestinationAddress source, TimeSpan lease, CancellationToken cancellationToken) + { + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(lease, TimeSpan.Zero); + if (source.Role != DestinationRole.Subscription) + throw new ArgumentException("Only subscriptions can have expiration leases.", nameof(source)); + cancellationToken.ThrowIfCancellationRequested(); + var resolved = Resolve(source); + long now = _timeProvider.GetUtcNow().ToUnixTimeMilliseconds(); + await _db.ScriptEvaluateAsync(TopicRetentionFunctions + """ + + cleanupSubscriptions(KEYS[1], ARGV[2]) + local created = redis.pcall('XGROUP', 'CREATE', KEYS[1], ARGV[1], '$', 'MKSTREAM') + if type(created) == 'table' and created.err then + if not string.find(created.err, 'BUSYGROUP', 1, true) then return redis.error_reply(created.err) end + if not redis.call('ZSCORE', KEYS[2], ARGV[1]) then return redis.error_reply('Cannot change a durable subscription into a temporary subscription.') end + end + redis.call('ZADD', KEYS[2], ARGV[3], ARGV[1]) + return 1 + """, new RedisKey[] { resolved.StreamKey, (RedisKey)$"{resolved.StreamKey}:subscriptions" }, + new RedisValue[] { resolved.Group, now, now + (long)lease.TotalMilliseconds }).ConfigureAwait(false); + _ensuredGroups.TryAdd(GroupKey(resolved), 0); + } + + public async Task RenewSubscriptionAsync(DestinationAddress source, TimeSpan lease, CancellationToken cancellationToken = default) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(source); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(lease, TimeSpan.Zero); + cancellationToken.ThrowIfCancellationRequested(); + var resolved = Resolve(source); + long now = _timeProvider.GetUtcNow().ToUnixTimeMilliseconds(); + var result = await _db.ScriptEvaluateAsync(""" + local expires = tonumber(redis.call('ZSCORE', KEYS[1], ARGV[1]) or '0') + if expires <= tonumber(ARGV[2]) then return 0 end + redis.call('ZADD', KEYS[1], ARGV[3], ARGV[1]) + return 1 + """, new RedisKey[] { (RedisKey)$"{resolved.StreamKey}:subscriptions" }, + new RedisValue[] { resolved.Group, now, now + (long)lease.TotalMilliseconds }).ConfigureAwait(false); + return (long)result == 1; + } +} diff --git a/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.cs b/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.cs new file mode 100644 index 000000000..e07b5f8d8 --- /dev/null +++ b/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.cs @@ -0,0 +1,517 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Globalization; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using StackExchange.Redis; + +namespace Foundatio.Messaging; + +/// +/// An over Redis Streams + consumer groups (at-least-once). Temporary in-repo provider +/// used to validate the redesigned transport contract — and the core's retry/dead-letter machinery — against a real +/// broker. A stream is a queue/topic; a consumer group is a subscription (the default group for a plain queue gives +/// competing consumers; one group per named subscription gives topic fan-out). +/// +/// +/// Streams has no per-message visible-until or per-message delay, so this transport keeps the lease explicitly: a +/// per-group sorted set (member = stream entry id, score = visible-until unix-ms) is the authoritative +/// in-flight lease and a per-group hash holds token|delivery-count per entry. Reclaim (abandon, redelivery +/// delay, lock expiry, crashed consumer) is driven by that sorted set — entries whose lease has lapsed are +/// XCLAIMed and redelivered (same stream id, delivery count incremented). Because the lease lives in Redis, a +/// message held by a crashed instance is recovered by any other instance. A stale receipt (already settled, or the +/// entry was redelivered to someone else) is detected by an owner token and surfaced as . +/// +public sealed partial class RedisStreamsMessageTransport : IMessageTransport, ISupportsPull, ISupportsVisibilityTimeout, + ISupportsLockRenewal, ISupportsRedeliveryDelay, ISupportsDeadLetter, ISupportsEphemeralSubscriptions, ISupportsStats, ITransportInfo +{ + private readonly ConcurrentDictionary _idlePolls = new(StringComparer.Ordinal); + + private static readonly IReadOnlySet _supportedRoles = + new HashSet { DestinationRole.Queue, DestinationRole.Topic, DestinationRole.Subscription, DestinationRole.Binding }; + + private readonly RedisStreamsMessageTransportOptions _options; + private readonly IDatabase _db; + private readonly TimeProvider _timeProvider; + private readonly string _prefix; + private readonly string _consumer; + private readonly ConcurrentDictionary _ensuredGroups = new(StringComparer.Ordinal); + private int _isDisposed; + + public RedisStreamsMessageTransport(RedisStreamsMessageTransportOptions options) + { + _options = options ?? throw new ArgumentNullException(nameof(options)); + ArgumentOutOfRangeException.ThrowIfLessThan(options.MaxPendingMessages, 1); + ArgumentOutOfRangeException.ThrowIfLessThan(options.MaxBatchSize, 1); + ArgumentOutOfRangeException.ThrowIfGreaterThan(options.MaxBatchSize, 256); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(options.PollInterval, TimeSpan.Zero); + ArgumentOutOfRangeException.ThrowIfLessThan(options.MaxIdlePollInterval, options.PollInterval); + ArgumentNullException.ThrowIfNull(options.ConnectionMultiplexer); + _db = options.ConnectionMultiplexer.GetDatabase(); + _timeProvider = options.TimeProvider ?? TimeProvider.System; + _prefix = options.KeyPrefix ?? ""; + _consumer = !String.IsNullOrEmpty(options.ConsumerName) ? options.ConsumerName : $"c-{Guid.NewGuid():N}"[..16]; + } + + // Streams append FIFO; there is no native priority, per-message expiration, or delayed delivery (delays route + // through the runtime-store fallback), and no broker-imposed size or batch limits. + private static readonly TransportCapabilities _capabilities = new() { Ordering = OrderingGuarantee.Fifo }; + + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => _supportedRoles; + public TransportCapabilities GetCapabilities(DestinationAddress destination) => _capabilities with { MaxBatchSize = _options.MaxBatchSize }; + public TimeSpan? MaxRedeliveryDelay => null; // lease is tracked in Redis, so any delay is honored + public TimeSpan? MaxVisibilityTimeout => null; + + public async Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + ArgumentNullException.ThrowIfNull(messages); + + // Streams have no native delayed delivery; the core routes delayed sends through the runtime-store fallback + // (no DelayedDelivery capability is advertised), so a future DeliverAt reaching here is a contract violation — + // refuse loudly rather than deliver immediately and silently drop the delay. + if (options.DeliverAt is { } deliverAt && deliverAt > _timeProvider.GetUtcNow()) + throw new NotSupportedException($"Transport \"{nameof(RedisStreamsMessageTransport)}\" does not support native delayed delivery. Register a job runtime store so delayed sends use the scheduled-dispatch fallback."); + + // The stream IS the queue/topic; subscriptions read it through their own group. The address role picks the + // stream namespace so a queue and a topic sharing a route name never cross-deliver. + RedisKey streamKey = destination.Role == DestinationRole.Topic ? TopicStreamKey(destination.Name) : QueueStreamKey(destination.Name); + var items = new SendItemResult[messages.Count]; + for (int index = 0; index < items.Length; index++) + items[index] = new SendItemResult { Index = index, Status = MessageSendStatus.NotAttempted }; + for (int offset = 0; offset < messages.Count; offset += _options.MaxBatchSize) + { + if (ct.IsCancellationRequested) break; + int count = Math.Min(_options.MaxBatchSize, messages.Count - offset); + var pending = new Task[count]; + for (int index = 0; index < count; index++) + pending[index] = SendOneAsync(offset + index); + await Task.WhenAll(pending).ConfigureAwait(false); + } + return new SendResult { Items = items }; + + async Task SendOneAsync(int index) + { + items[index] = items[index] with { Status = MessageSendStatus.Unknown }; + try + { + var arguments = new List { destination.Role == DestinationRole.Topic ? "1" : "0", _options.MaxPendingMessages, _timeProvider.GetUtcNow().ToUnixTimeMilliseconds() }; + foreach (var field in BuildFields(messages[index])) { arguments.Add(field.Name); arguments.Add(field.Value); } + var id = await _db.ScriptEvaluateAsync(SendScript, [streamKey], arguments.ToArray()).WaitAsync(ct).ConfigureAwait(false); + items[index] = new SendItemResult { Index = index, Status = MessageSendStatus.Accepted, MessageId = (string)id! }; + } + catch (Exception ex) + { + bool rejected = ex is RedisServerException && ex.Message.Contains("The destination has reached its pending-message capacity.", StringComparison.Ordinal); + items[index] = new SendItemResult + { + Index = index, + Status = rejected ? MessageSendStatus.Rejected : MessageSendStatus.Unknown, + ErrorCode = rejected ? "CapacityExceeded" : ex.GetType().Name, + ErrorMessage = ex.Message, + Retryable = ex is not OperationCanceledException + }; + } + } + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + => ReceiveAsync(source, request, _options.DefaultVisibilityTimeout, ct); + + public async Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, TimeSpan visibility, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(source); + ArgumentNullException.ThrowIfNull(request); + + var resolved = Resolve(source); + int max = Math.Max(1, request.MaxMessages); + long visibilityMs = (long)Math.Max(0, visibility.TotalMilliseconds); + var deadline = _timeProvider.GetUtcNow() + (request.MaxWaitTime ?? TimeSpan.Zero); + + while (true) + { + ct.ThrowIfCancellationRequested(); + List entries; + try { entries = await PollOnceAsync(source, resolved, max, visibilityMs, ct).ConfigureAwait(false); } + catch (RedisServerException ex) when (ex.Message.Contains("NOGROUP", StringComparison.Ordinal)) + { + _ensuredGroups.TryRemove(GroupKey(resolved), out _); + throw new MessageDestinationNotFoundException(source, ex); + } + if (entries.Count > 0) + { + _idlePolls.TryRemove(GroupKey(resolved), out _); + return entries; + } + + var remaining = deadline - _timeProvider.GetUtcNow(); + if (remaining <= TimeSpan.Zero) + return []; + + int idle = _idlePolls.AddOrUpdate(GroupKey(resolved), 0, (_, current) => Math.Min(10, current + 1)); + var delay = TimeSpan.FromMilliseconds(Math.Min(_options.MaxIdlePollInterval.TotalMilliseconds, _options.PollInterval.TotalMilliseconds * (1 << idle))); + await Task.Delay(remaining < delay ? remaining : delay, _timeProvider, ct).ConfigureAwait(false); + } + } + + private async Task> PollOnceAsync(DestinationAddress source, ResolvedSource resolved, int max, long visibilityMs, CancellationToken ct) + { + ct.ThrowIfCancellationRequested(); + long nowMs = _timeProvider.GetUtcNow().ToUnixTimeMilliseconds(); + var snapshot = await _db.ScriptEvaluateAsync(ReceiveScript, + new RedisKey[] { resolved.StreamKey, LockKey(resolved), MetaKey(resolved) }, + new RedisValue[] { resolved.Group, _consumer, nowMs, visibilityMs, max, Guid.NewGuid().ToString("N"), (long)_options.DefaultVisibilityTimeout.TotalMilliseconds }).ConfigureAwait(false); + var rows = (RedisResult[]?)snapshot ?? []; + var result = new List(rows.Length); + foreach (var row in rows) + { + var values = (RedisResult[])row!; + var fields = (RedisResult[])values[1]!; + var entries = new NameValueEntry[fields.Length / 2]; + for (int index = 0; index < entries.Length; index++) + entries[index] = new NameValueEntry((string)fields[index * 2]!, (byte[])fields[index * 2 + 1]!); + var entry = new StreamEntry((string)values[0]!, entries); + result.Add(ToEntry(source, resolved, entry, (int)values[2], (string)values[3]!) with + { + LockExpiresUtc = DateTimeOffset.FromUnixTimeMilliseconds(nowMs + visibilityMs) + }); + } + return result; + } + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) + => SettleAsync(entry, "complete", null, null, ct); + + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) + => AbandonAsync(entry, TimeSpan.Zero, ct); + + public Task AbandonAsync(TransportEntry entry, TimeSpan redeliveryDelay, CancellationToken ct) + => SettleAsync(entry, "abandon", redeliveryDelay, null, ct); + + public Task RenewLockAsync(TransportEntry entry, TimeSpan? duration, CancellationToken ct) + => SettleAsync(entry, "renew", duration ?? _options.DefaultVisibilityTimeout, null, ct); + + public Task DeadLetterAsync(TransportEntry entry, string? reason, CancellationToken ct) + => SettleAsync(entry, "deadletter", null, reason, ct); + + private async Task SettleAsync(TransportEntry entry, string operation, TimeSpan? duration, string? reason, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(entry); + ct.ThrowIfCancellationRequested(); + if (entry.Receipt.TransportState is not StreamReceipt receipt) + throw new ReceiptExpiredException("The entry does not carry a Redis Streams receipt."); + long nowMs = _timeProvider.GetUtcNow().ToUnixTimeMilliseconds(); + var arguments = new List + { + receipt.Group, receipt.EntryId, receipt.Token, nowMs, operation, + nowMs + (long)Math.Max(0, duration?.TotalMilliseconds ?? 0), IsTopicStream(receipt.StreamKey) ? "0" : "1", _options.MaxPendingMessages + }; + if (operation == "deadletter") + { + var headers = entry.Headers.ToBuilder(); + if (!String.IsNullOrEmpty(reason)) + headers.Set(KnownHeaders.DeadLetterReason, reason); + foreach (var field in BuildFields(entry.ApplicationMessageId, entry.Body, headers.Build(), entry.ContentType)) + { + arguments.Add(field.Name); + arguments.Add(field.Value); + } + } + var result = await _db.ScriptEvaluateAsync(SettleScript, + new RedisKey[] { receipt.StreamKey, LockKey(receipt), MetaKey(receipt), DeadKey(receipt) }, arguments.ToArray()).ConfigureAwait(false); + if ((long)result != 1) + throw new ReceiptExpiredException(); + } + + public async Task> PeekDeadLetteredAsync(DestinationAddress destination, DeadLetterQuery? query = null, CancellationToken cancellationToken = default) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + cancellationToken.ThrowIfCancellationRequested(); + query ??= new DeadLetterQuery(); + query.Validate(); + var entries = await _db.StreamRangeAsync(DeadKey(Resolve(destination)), minId: query.AfterId is null ? "-" : "(" + query.AfterId, count: query.Limit).ConfigureAwait(false); + var result = new List(entries.Length); + foreach (var entry in entries) + result.Add(ToEntry(destination, null, entry, 1, "")); + return result; + } + + public async Task DeleteDeadLetteredAsync(DestinationAddress destination, string id, CancellationToken cancellationToken = default) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + ArgumentException.ThrowIfNullOrWhiteSpace(id); + cancellationToken.ThrowIfCancellationRequested(); + return await _db.StreamDeleteAsync(DeadKey(Resolve(destination)), new RedisValue[] { id }).ConfigureAwait(false) > 0; + } + + public async Task ReplayDeadLetteredAsync(DestinationAddress source, string id, DestinationAddress target, CancellationToken cancellationToken = default) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(source); + ArgumentNullException.ThrowIfNull(target); + ArgumentException.ThrowIfNullOrWhiteSpace(id); + cancellationToken.ThrowIfCancellationRequested(); + if (target.Role is not (DestinationRole.Queue or DestinationRole.Topic)) + throw new ArgumentException("Replay targets must be a queue or topic.", nameof(target)); + var result = await _db.ScriptEvaluateAsync(ReplayScript, new RedisKey[] { DeadKey(Resolve(source)), Resolve(target).StreamKey }, + new RedisValue[] { id, target.Role == DestinationRole.Topic ? "1" : "0", _options.MaxPendingMessages, _timeProvider.GetUtcNow().ToUnixTimeMilliseconds() }).ConfigureAwait(false); + return (long)result == 1; + } + + public async Task EnsureAsync(IReadOnlyList declarations, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(declarations); + + foreach (var declaration in declarations) + { + if (declaration.AutoDeleteAfter is { } lease) + { + await EnsureTemporarySubscriptionAsync(declaration.Address, lease, ct).ConfigureAwait(false); + continue; + } + switch (declaration.Address.Role) + { + case DestinationRole.Topic: + await _db.ScriptEvaluateAsync(""" + if redis.call('EXISTS', KEYS[1]) == 0 then + local id = redis.call('XADD', KEYS[1], '*', 'init', '1') + redis.call('XDEL', KEYS[1], id) + end + return 1 + """, new RedisKey[] { TopicStreamKey(declaration.Address.Name) }).ConfigureAwait(false); + break; + default: + // Queue, subscription, and binding declarations all materialize as a consumer group on the stream + // the address resolves to. + await EnsureGroupAsync(Resolve(declaration.Address)).ConfigureAwait(false); + break; + } + } + } + + public async Task DeleteAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + + // A subscription deletes only that group's state (never the shared topic stream — other subscriptions still + // read it); a queue or topic deletes its own stream and everything scoped to it. + if (destination.Role is DestinationRole.Subscription or DestinationRole.Binding) + { + var sub = Resolve(destination); + await _db.StreamDeleteConsumerGroupAsync(sub.StreamKey, sub.Group).ConfigureAwait(false); + await _db.KeyDeleteAsync([LockKey(sub), MetaKey(sub), DeadKey(sub)]).ConfigureAwait(false); + await _db.SortedSetRemoveAsync((RedisKey)$"{sub.StreamKey}:subscriptions", sub.Group).ConfigureAwait(false); + _ensuredGroups.TryRemove(GroupKey(sub), out _); + return; + } + + var resolved = Resolve(destination); + + // Drop each consumer group's lease/meta state before the stream itself (topic streams can carry several). + if (await _db.KeyExistsAsync(resolved.StreamKey).ConfigureAwait(false)) + { + foreach (var group in await _db.StreamGroupInfoAsync(resolved.StreamKey).ConfigureAwait(false)) + { + var groupSource = resolved with { Group = group.Name }; + await _db.KeyDeleteAsync([LockKey(groupSource), MetaKey(groupSource), DeadKey(groupSource)]).ConfigureAwait(false); + _ensuredGroups.TryRemove(GroupKey(groupSource), out _); + } + } + + await _db.KeyDeleteAsync([resolved.StreamKey, DeadKey(resolved), (RedisKey)$"{resolved.StreamKey}:subscriptions"]).ConfigureAwait(false); + _ensuredGroups.TryRemove(GroupKey(resolved), out _); + } + + public async Task ExistsAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + + var resolved = Resolve(destination); + await _db.ScriptEvaluateAsync(TopicRetentionFunctions + "\ncleanupSubscriptions(KEYS[1], ARGV[1]); return 1", + new RedisKey[] { resolved.StreamKey }, new RedisValue[] { _timeProvider.GetUtcNow().ToUnixTimeMilliseconds() }).ConfigureAwait(false); + if (!await _db.KeyExistsAsync(resolved.StreamKey).ConfigureAwait(false)) + return false; + + // A subscription exists when its consumer group exists on the topic stream; a queue/topic exists when its + // stream key does. + if (destination.Role is not (DestinationRole.Subscription or DestinationRole.Binding)) + return true; + + foreach (var group in await _db.StreamGroupInfoAsync(resolved.StreamKey).ConfigureAwait(false)) + { + if (String.Equals(group.Name, resolved.Group, StringComparison.Ordinal)) + return true; + } + + return false; + } + + public async Task GetStatsAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + ct.ThrowIfCancellationRequested(); + var resolved = Resolve(destination); + var result = await _db.ScriptEvaluateAsync(TopicRetentionFunctions + """ + + cleanupSubscriptions(KEYS[1], ARGV[4]) + local queued, working, found = 0, 0, false + if redis.call('EXISTS', KEYS[1]) == 1 then + for _, group in ipairs(redis.call('XINFO', 'GROUPS', KEYS[1])) do + local name, pending, lag + for index = 1, #group, 2 do + if group[index] == 'name' then name = group[index + 1] end + if group[index] == 'pending' then pending = group[index + 1] end + if group[index] == 'lag' then lag = group[index + 1] end + end + if ARGV[2] == '1' or name == ARGV[1] then + found = true + queued = queued + (tonumber(lag) or 0) + working = working + (tonumber(pending) or 0) + end + end + if not found and ARGV[3] == '1' then queued = redis.call('XLEN', KEYS[1]) end + end + return {queued, working, redis.call('XLEN', KEYS[2])} + """, new RedisKey[] { resolved.StreamKey, DeadKey(resolved) }, + new RedisValue[] { resolved.Group, destination.Role == DestinationRole.Topic ? "1" : "0", destination.Role == DestinationRole.Queue ? "1" : "0", _timeProvider.GetUtcNow().ToUnixTimeMilliseconds() }).ConfigureAwait(false); + var values = (RedisResult[])result!; + return new MessageDestinationStats { Queued = (long)values[0], Working = (long)values[1], Deadletter = (long)values[2] }; + } + + public ValueTask DisposeAsync() + { + Interlocked.Exchange(ref _isDisposed, 1); + return ValueTask.CompletedTask; // the connection multiplexer is owned by the caller + } + + private async Task EnsureGroupAsync(ResolvedSource resolved) + { + if (_ensuredGroups.ContainsKey(GroupKey(resolved))) + return; + + try + { + await _db.StreamCreateConsumerGroupAsync(resolved.StreamKey, resolved.Group, resolved.Position, createStream: true).ConfigureAwait(false); + } + catch (RedisServerException ex) when (ex.Message.Contains("BUSYGROUP", StringComparison.Ordinal)) + { + // Group already exists — creation is idempotent. + } + _ensuredGroups.TryAdd(GroupKey(resolved), 0); + } + + // The address is structural, so the physical mapping is derived from it directly — topology declarations and the + // runtime resolve the SAME address to the SAME stream/group, with no registration cache to drift. A subscription is + // a consumer group on its owning topic's stream: the group is named by the bare subscription name and scoped by the + // topic stream key. A queue is its own stream read through the shared default group (competing consumers). + private ResolvedSource Resolve(DestinationAddress address) => address.Role switch + { + DestinationRole.Topic => new ResolvedSource(TopicStreamKey(address.Name), _options.DefaultConsumerGroup, "$"), + DestinationRole.Subscription or DestinationRole.Binding => new ResolvedSource(TopicStreamKey(address.Topic ?? address.Name), address.Name, "$"), + _ => new ResolvedSource(QueueStreamKey(address.Name), _options.DefaultConsumerGroup, "0") + }; + + private TransportEntry ToEntry(DestinationAddress destination, ResolvedSource? resolved, StreamEntry entry, int deliveries, string token) + { + var headers = MessageHeaders.DeserializeFromJson(GetField(entry, "h")); + Receipt receipt = resolved is null + ? default + : new Receipt { TransportState = new StreamReceipt(resolved.StreamKey.ToString(), resolved.Group, entry.Id.ToString(), token) }; + + return new TransportEntry + { + Id = entry.Id.ToString(), + ApplicationMessageId = GetField(entry, "id"), + ContentType = GetField(entry, "ct"), + Destination = destination, + Body = GetBody(entry), + Headers = headers, + DeliveryCount = deliveries, + EnqueuedUtc = ParseStreamIdTime(entry.Id), + Receipt = receipt + }; + } + + private static NameValueEntry[] BuildFields(TransportMessage message) + => BuildFields(message.MessageId, message.Body, message.Headers, message.ContentType); + + private static NameValueEntry[] BuildFields(string? messageId, ReadOnlyMemory body, MessageHeaders headers, string? contentType = null) + { + return + [ + new NameValueEntry("id", messageId ?? ""), + new NameValueEntry("ct", contentType ?? ""), + new NameValueEntry("h", MessageHeaders.SerializeToJson(headers)), + new NameValueEntry("b", body.ToArray()) + ]; + } + + private static string? GetField(StreamEntry entry, string name) + { + foreach (var value in entry.Values) + { + if (value.Name == name) + return value.Value.IsNull ? null : value.Value.ToString(); + } + + return null; + } + + private static ReadOnlyMemory GetBody(StreamEntry entry) + { + foreach (var value in entry.Values) + { + if (value.Name == "b") + return value.Value.IsNullOrEmpty ? ReadOnlyMemory.Empty : (byte[])value.Value!; + } + + return ReadOnlyMemory.Empty; + } + + + // Stream ids are "-"; the timestamp half is the broker enqueue time. + private static DateTimeOffset? ParseStreamIdTime(RedisValue id) + { + string s = id.ToString(); + int dash = s.IndexOf('-'); + string ms = dash > 0 ? s[..dash] : s; + return Int64.TryParse(ms, NumberStyles.Integer, CultureInfo.InvariantCulture, out long unixMs) + ? DateTimeOffset.FromUnixTimeMilliseconds(unixMs) + : null; + } + + // Streams are namespaced by role ("q:" queue, "t:" topic) because an XADD lands on whichever stream the key names: + // without the split, a message type both sent and published would share one stream and cross-deliver (a publish + // consumed as queue work and vice versa). Subscriptions are consumer groups on the topic stream. + private static string EncodeKeyPart(string value) => Convert.ToHexString(Encoding.UTF8.GetBytes(value)); + private RedisKey QueueStreamKey(string name) => $"{_prefix}q:{EncodeKeyPart(name)}"; + private RedisKey TopicStreamKey(string name) => $"{_prefix}t:{EncodeKeyPart(name)}"; + private bool IsTopicStream(string streamKey) => streamKey.StartsWith($"{_prefix}t:", StringComparison.Ordinal); + private RedisKey DeadKey(ResolvedSource source) + => IsTopicStream(source.StreamKey.ToString()) ? $"{source.StreamKey}:dead:{EncodeKeyPart(source.Group)}" : $"{source.StreamKey}:dead"; + private RedisKey DeadKey(StreamReceipt receipt) + => IsTopicStream(receipt.StreamKey) ? $"{receipt.StreamKey}:dead:{EncodeKeyPart(receipt.Group)}" : $"{receipt.StreamKey}:dead"; + private static RedisKey LockKey(ResolvedSource r) => $"{r.StreamKey}:lock:{EncodeKeyPart(r.Group)}"; + private static RedisKey MetaKey(ResolvedSource r) => $"{r.StreamKey}:meta:{EncodeKeyPart(r.Group)}"; + private static RedisKey LockKey(StreamReceipt r) => $"{r.StreamKey}:lock:{EncodeKeyPart(r.Group)}"; + private static RedisKey MetaKey(StreamReceipt r) => $"{r.StreamKey}:meta:{EncodeKeyPart(r.Group)}"; + private static string GroupKey(ResolvedSource r) => $"{r.StreamKey}|{r.Group}"; + + private void ThrowIfDisposed() => ObjectDisposedException.ThrowIf(Volatile.Read(ref _isDisposed) == 1, this); + + private sealed record ResolvedSource(RedisKey StreamKey, string Group, RedisValue Position); + + private sealed record StreamReceipt(string StreamKey, string Group, string EntryId, string Token); +} diff --git a/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransportOptions.cs b/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransportOptions.cs new file mode 100644 index 000000000..e297a4515 --- /dev/null +++ b/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransportOptions.cs @@ -0,0 +1,36 @@ +using System; +using Foundatio.Jobs; +using StackExchange.Redis; + +namespace Foundatio.Messaging; + +public class RedisStreamsMessageTransportOptions +{ + /// The Redis connection to use. Required. + public IConnectionMultiplexer ConnectionMultiplexer { get; set; } = null!; + + /// Prefix applied to every stream/key this transport creates. Isolates environments/runs on a shared Redis. + public string KeyPrefix { get; set; } = "fnd:msg:"; + + /// Consumer-group name used for plain queue destinations (its members are competing consumers). + public string DefaultConsumerGroup { get; set; } = "foundatio"; + + /// How long a received message stays invisible to other consumers before it can be reclaimed (the lease). + public TimeSpan DefaultVisibilityTimeout { get; set; } = TimeSpan.FromSeconds(30); + + /// Maximum retained messages per destination. Sends fail at capacity; unread or pending work is never trimmed. + public int MaxPendingMessages { get; set; } = 100_000; + /// Maximum concurrently pipelined sends per call. + public int MaxBatchSize { get; set; } = 64; + /// Budgets for the automatic delayed-message store when no shared job runtime store is configured. + public JobRuntimeStoreOptions Scheduling { get; set; } = new(); + public TimeSpan PollInterval { get; set; } = TimeSpan.FromMilliseconds(25); + /// Idle polling backs off to this ceiling; lower it when arrival latency matters more than idle broker traffic. + public TimeSpan MaxIdlePollInterval { get; set; } = TimeSpan.FromSeconds(1); + + /// This node's consumer name within every group (defaults to a stable per-instance id). Distinct instances are competing consumers. + public string? ConsumerName { get; set; } + + /// Time source (defaults to ). + public TimeProvider? TimeProvider { get; set; } +} diff --git a/src/Foundatio.Redis/RedisFoundatioBuilderExtensions.cs b/src/Foundatio.Redis/RedisFoundatioBuilderExtensions.cs new file mode 100644 index 000000000..fa8023614 --- /dev/null +++ b/src/Foundatio.Redis/RedisFoundatioBuilderExtensions.cs @@ -0,0 +1,105 @@ +using System; +using System.Linq; +using Foundatio.Jobs; +using Foundatio.Lock; +using Foundatio.Messaging; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using StackExchange.Redis; + +namespace Foundatio; + +public static class RedisFoundatioBuilderExtensions +{ + /// + /// Backs the durable job runtime with Redis. Uses an already registered in DI, + /// otherwise connects using or the "Redis" connection string from configuration + /// (falling back to localhost). When both messaging and jobs use Redis a single connection is shared, so the + /// explicit connection settings must agree. Conflicting settings fail during registration. + /// + public static FoundatioBuilder.JobsBuilder UseRedis(this FoundatioBuilder.JobsBuilder builder, Action? configure = null, string? connectionString = null) + { + EnsureConnection(((IFoundatioBuilder)builder).Services, connectionString); + return builder.UseRuntimeStore(sp => + { + var options = new RedisJobRuntimeStoreOptions { ConnectionMultiplexer = sp.GetRequiredService() }; + configure?.Invoke(options); + return new RedisJobRuntimeStore(options); + }); + } + + /// + /// Runs messaging (queues + pub/sub) over Redis Streams. Uses an already + /// registered in DI, otherwise connects using or the "Redis" connection string + /// from configuration (falling back to localhost). When both messaging and jobs use Redis a single connection is + /// shared. Explicit connection settings must agree; conflicting settings fail during registration. + /// + public static FoundatioBuilder.MessagingBuilder UseRedis(this FoundatioBuilder.MessagingBuilder builder, Action? configure = null, string? connectionString = null) + { + EnsureConnection(((IFoundatioBuilder)builder).Services, connectionString); + var services = ((IFoundatioBuilder)builder).Services; + services.AddSingleton(sp => + { + var options = new RedisStreamsMessageTransportOptions + { + ConnectionMultiplexer = sp.GetRequiredService(), + TimeProvider = sp.GetService() + }; + configure?.Invoke(options); + return options; + }); + services.TryAddSingleton(sp => + { + if (sp.GetService() is { } store) return store; + var options = sp.GetRequiredService(); + return new RedisJobRuntimeStore(new RedisJobRuntimeStoreOptions + { + ConnectionMultiplexer = options.ConnectionMultiplexer, + KeyPrefix = options.KeyPrefix + "dispatch:", + TimeProvider = options.TimeProvider, + Runtime = options.Scheduling + }); + }); + return builder.UseTransport(sp => new RedisStreamsMessageTransport(sp.GetRequiredService())); + } + + /// Coordinates resources across workers with ownership-checked Redis locks. + public static FoundatioBuilder UseRedis(this FoundatioBuilder.LockingBuilder builder, string keyPrefix = "fnd:locks:", string? connectionString = null) + { + var services = ((IFoundatioBuilder)builder).Services; + EnsureConnection(services, connectionString); + return builder.Use(sp => new RedisLockProvider(sp.GetRequiredService(), keyPrefix)); + } + + private static void EnsureConnection(IServiceCollection services, string? connectionString) + { + var settings = services.FirstOrDefault(d => d.ServiceType == typeof(RedisConnectionSettings))?.ImplementationInstance as RedisConnectionSettings; + if (settings is not null) + { + if (connectionString is not null && settings.ConnectionString is not null && !String.Equals(connectionString, settings.ConnectionString, StringComparison.Ordinal)) + throw new ArgumentException("Messaging and jobs share one Redis connection. Supply the same connection string, or configure it once and omit it on subsequent UseRedis calls.", nameof(connectionString)); + settings.ConnectionString ??= connectionString; + return; + } + + if (services.Any(d => d.ServiceType == typeof(IConnectionMultiplexer))) + { + if (connectionString is not null) + throw new ArgumentException("A Redis connection is already registered. Omit connectionString from UseRedis to use that connection.", nameof(connectionString)); + return; + } + + settings = new RedisConnectionSettings { ConnectionString = connectionString }; + services.AddSingleton(settings); + services.AddSingleton(sp => ConnectionMultiplexer.Connect( + settings.ConnectionString + ?? sp.GetService()?.GetConnectionString("Redis") + ?? "localhost:6379")); + } + + private sealed class RedisConnectionSettings + { + public string? ConnectionString { get; set; } + } +} diff --git a/src/Foundatio.Redis/RedisJobRuntimeStore.Broker.cs b/src/Foundatio.Redis/RedisJobRuntimeStore.Broker.cs new file mode 100644 index 000000000..6708236f6 --- /dev/null +++ b/src/Foundatio.Redis/RedisJobRuntimeStore.Broker.cs @@ -0,0 +1,246 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using StackExchange.Redis; + +namespace Foundatio.Jobs; + +public sealed partial class RedisJobRuntimeStore +{ + public bool IsShared => true; + + private const string MonitoringFunctions = """ + local function hex(value) + return (string.gsub(value, '.', function(c) return string.format('%02X', string.byte(c)) end)) + end + local function monitoringKeys(job, status) + local id = redis.call('HGET', job, 'jobId') + local prefix = string.sub(job, 1, #job - #id - 4) + local keys = {prefix .. 'created', prefix .. 'created-status:' .. status} + local name = redis.call('HGET', job, 'monitorName') + if not name then + local value = redis.call('HGET', job, 'name') + if value then name = hex(value); redis.call('HSET', job, 'monitorName', name) end + end + if name then + table.insert(keys, prefix .. 'created-name:' .. name) + table.insert(keys, prefix .. 'created-name:' .. name .. ':' .. status) + end + local queue = redis.call('HGET', job, 'monitorQueue') + if not queue then + local value = redis.call('HGET', job, 'queueName') + if value then queue = hex(value); redis.call('HSET', job, 'monitorQueue', queue) end + end + if queue then + table.insert(keys, prefix .. 'created-queue:' .. queue) + table.insert(keys, prefix .. 'created-queue:' .. queue .. ':' .. status) + end + return keys, prefix, id + end + local function removeMonitoring(job) + local status = redis.call('HGET', job, 'monitorStatus') or redis.call('HGET', job, 'status') + if not status then return end + local keys, prefix, id = monitoringKeys(job, status) + for _, key in ipairs(keys) do redis.call('ZREM', key, id) end + redis.call('ZREM', prefix .. 'broker-expiry', id) + end + local function syncMonitoring(job) + local values = redis.call('HMGET', job, 'status', 'monitorStatus', 'createdUtc', 'historyExpiresUtc', 'monitorExpiry', 'jobId') + local status, previous = values[1], values[2] + if not status then return end + local id = values[6] + local prefix = string.sub(job, 1, #job - #id - 4) + if previous ~= status then + local keys = monitoringKeys(job, status) + for i = 1, #keys, 2 do + if not previous then redis.call('ZADD', keys[i], values[3], id) + else + local oldKey = keys[i] .. (i == 1 and '-status:' or ':') .. previous + redis.call('ZREM', oldKey, id) + end + redis.call('ZADD', keys[i+1], values[3], id) + end + redis.call('HSET', job, 'monitorStatus', status) + end + if values[4] and values[4] ~= values[5] then + redis.call('ZADD', prefix .. 'broker-expiry', values[4], id) + redis.call('HSET', job, 'monitorExpiry', values[4]) + end + end + local function refreshBrokerHistory(job, now) + if redis.call('HGET', job, 'executionOwner') ~= 'Broker' then return end + local retention = tonumber(redis.call('HGET', job, 'historyRetention') or '0') + if retention > 0 then redis.call('HSET', job, 'historyExpiresUtc', string.format('%.0f', now + retention)) end + end + local function forgetJob(job, prefix, id) + removeMonitoring(job) + local status, name = redis.call('HGET', job, 'status'), redis.call('HGET', job, 'name') + if status then redis.call('ZREM', prefix .. 'status:' .. status, id) end + if name then redis.call('ZREM', prefix .. 'name:' .. name, id) end + for _, suffix in ipairs({'all', 'terminal', 'unclaimed'}) do redis.call('ZREM', prefix .. suffix, id) end + if redis.call('HGET', job, 'executionOwner') == 'Broker' then redis.call('ZREM', prefix .. 'deduplication', id) end + local ready, schedule = redis.call('HGET', job, 'readyKey'), redis.call('HGET', job, 'activeScheduleKey') + if ready then redis.call('ZREM', ready, id) end + if schedule then redis.call('SREM', schedule, id) end + redis.call('DEL', job) + end + local function expireJob(job, now) + local expires = tonumber(redis.call('HGET', job, 'historyExpiresUtc')) + if not expires or expires > now or redis.call('HGET', job, 'executionOwner') ~= 'Broker' then return false end + local _, prefix, id = monitoringKeys(job, redis.call('HGET', job, 'status')) + forgetJob(job, prefix, id) + return true + end + local function purgeBrokerHistory(prefix, now, limit) + local expiry = prefix .. 'broker-expiry' + local ids = redis.call('ZRANGEBYSCORE', expiry, '-inf', now, 'LIMIT', 0, limit) + for _, id in ipairs(ids) do + local job = prefix .. 'job:' .. id + if redis.call('HGET', job, 'executionOwner') == 'Broker' then forgetJob(job, prefix, id) end + redis.call('ZREM', expiry, id) + end + return #ids + end + """; + + public async Task BeginBrokerAttemptAsync(string jobId, int attempt, string nodeId, CancellationToken cancellationToken = default) + { + ArgumentOutOfRangeException.ThrowIfLessThan(attempt, 1); + ArgumentException.ThrowIfNullOrWhiteSpace(nodeId); + const string script = MonitoringFunctions + "\n" + """ + if expireJob(KEYS[1], tonumber(ARGV[4])) then return {} end + if redis.call('HGET', KEYS[1], 'executionOwner') ~= 'Broker' then return {} end + local status = redis.call('HGET', KEYS[1], 'status') + if status ~= 'Queued' and status ~= 'Processing' and status ~= 'RetryPending' and status ~= 'EnqueueUnknown' then return {} end + if tonumber(redis.call('HGET', KEYS[1], 'attempt') or '0') >= tonumber(ARGV[1]) then return {} end + redis.call('ZREM', ARGV[5] .. 'status:' .. status, ARGV[6]) + redis.call('ZADD', ARGV[5] .. 'status:Processing', 0, ARGV[6]) + redis.call('HSET', KEYS[1], 'status', 'Processing', 'attempt', ARGV[1], 'nodeId', ARGV[2], 'claimToken', ARGV[3], + 'startedUtc', ARGV[4], 'lastHeartbeatUtc', ARGV[4], 'lastUpdatedUtc', ARGV[4], 'progress', 0) + redis.call('HDEL', KEYS[1], 'progressMessage', 'completedUtc') + refreshBrokerHistory(KEYS[1], tonumber(ARGV[4])) + syncMonitoring(KEYS[1]) + return redis.call('HGETALL', KEYS[1]) + """; + var raw = await _db.ScriptEvaluateAsync(script, [JobKey(jobId)], [attempt, nodeId, Guid.NewGuid().ToString("N"), Ticks(_timeProvider.GetUtcNow()), _prefix, jobId]).WaitAsync(cancellationToken).ConfigureAwait(false); + return ((RedisResult[])raw!).Length == 0 ? null : ReadJobSnapshot(raw); + } + + public async Task MarkEnqueueUnknownAsync(string jobId, string error, CancellationToken cancellationToken = default) + { + const string script = MonitoringFunctions + "\n" + """ + if expireJob(KEYS[1], tonumber(ARGV[2])) then return 0 end + if redis.call('HGET', KEYS[1], 'executionOwner') ~= 'Broker' or redis.call('HGET', KEYS[1], 'status') ~= 'Queued' or redis.call('HGET', KEYS[1], 'attempt') ~= '0' then return 0 end + redis.call('HSET', KEYS[1], 'status', 'EnqueueUnknown', 'error', ARGV[1], 'lastUpdatedUtc', ARGV[2]) + redis.call('ZREM', ARGV[3] .. 'status:Queued', ARGV[4]) + redis.call('ZADD', ARGV[3] .. 'status:EnqueueUnknown', 0, ARGV[4]) + refreshBrokerHistory(KEYS[1], tonumber(ARGV[2])) + syncMonitoring(KEYS[1]) + return 1 + """; + return (long)await _db.ScriptEvaluateAsync(script, [JobKey(jobId)], [error, Ticks(_timeProvider.GetUtcNow()), _prefix, jobId]).WaitAsync(cancellationToken).ConfigureAwait(false) == 1; + } + + public Task HeartbeatJobAsync(string jobId, string claimToken, CancellationToken cancellationToken = default) + => ReportJobProgressAsync(jobId, claimToken, cancellationToken: cancellationToken); + + public async Task RemoveAsync(string jobId, CancellationToken cancellationToken = default) + { + const string script = MonitoringFunctions + "\n" + """ + local status = redis.call('HGET', KEYS[1], 'status') + if not status then return 0 end + if redis.call('HGET', KEYS[1], 'executionOwner') ~= 'Broker' and (status == 'Queued' or status == 'Scheduled' or status == 'Processing') then return 0 end + forgetJob(KEYS[1], ARGV[1], ARGV[2]) + return 1 + """; + return (long)await _db.ScriptEvaluateAsync(script, [JobKey(jobId)], [_prefix, jobId]).WaitAsync(cancellationToken).ConfigureAwait(false) == 1; + } + + private async Task PurgeBrokerHistoryAsync(CancellationToken cancellationToken) + { + const string script = MonitoringFunctions + "\n" + """ + return purgeBrokerHistory(ARGV[2], tonumber(ARGV[1]), 128) + """; + while ((long)await _db.ScriptEvaluateAsync(script, [$"{_prefix}broker-expiry"], [Ticks(_timeProvider.GetUtcNow()), _prefix]).WaitAsync(cancellationToken).ConfigureAwait(false) == 128) + cancellationToken.ThrowIfCancellationRequested(); + } + + private RedisKey MonitoringIndex(JobQuery query) + { + string scope = query.QueueName is { } queue ? "created-queue:" + EncodeKey(queue) : query.Name is { } name ? "created-name:" + EncodeKey(name) : "created"; + return $"{_prefix}{scope}{(query.Status is { } status ? (scope == "created" ? "-status:" : ":") + status : "")}"; + } + + public async Task CountAsync(JobQuery query, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(query); + query.Validate(); + await PurgeBrokerHistoryAsync(cancellationToken).ConfigureAwait(false); + if (query.QueueName is null || query.Name is null) + return await _db.SortedSetLengthAsync(MonitoringIndex(query)).WaitAsync(cancellationToken).ConfigureAwait(false); + const string script = """ + local count, offset = 0, 0 + repeat + local ids = redis.call('ZRANGE', KEYS[1], offset, offset + 199) + for _, id in ipairs(ids) do + if redis.call('HGET', ARGV[1] .. 'job:' .. id, 'name') == ARGV[2] then count = count + 1 end + end + offset = offset + #ids + until #ids < 200 + return count + """; + return (long)await _db.ScriptEvaluateAsync(script, [MonitoringIndex(query)], [_prefix, query.Name]).WaitAsync(cancellationToken).ConfigureAwait(false); + } + + private async Task QueryNewestAsync(JobQuery query, CancellationToken cancellationToken) + { + const string script = """ + local result, offset, skipped = {}, 0, 0 + repeat + local ids = redis.call('ZREVRANGE', KEYS[1], offset, offset + 199) + for _, id in ipairs(ids) do + local job = ARGV[1] .. 'job:' .. id + if ARGV[2] == '' or redis.call('HGET', job, 'name') == ARGV[2] then + if skipped < tonumber(ARGV[3]) then skipped = skipped + 1 + else table.insert(result, redis.call('HGETALL', job)) end + if #result >= tonumber(ARGV[4]) then return result end + end + end + offset = offset + #ids + until #ids < 200 + return result + """; + var result = (RedisResult[])(await _db.ScriptEvaluateAsync(script, [MonitoringIndex(query)], [_prefix, query.Name ?? "", query.Skip, query.Limit]).WaitAsync(cancellationToken).ConfigureAwait(false))!; + return new JobPage(result.Select(ReadJobSnapshot).ToArray(), null); + } + + public Task IncrementCounterAsync(string name, string counterName, long value = 1, CancellationToken cancellationToken = default) + => _db.ScriptEvaluateAsync("redis.call('HINCRBY', KEYS[1], ARGV[1], ARGV[2]); redis.call('PEXPIRE', KEYS[1], 172800000); return 1", + [CounterKey(name, _timeProvider.GetUtcNow())], [counterName, value]).WaitAsync(cancellationToken); + + public async Task GetCounterStatsAsync(string name, TimeSpan? window = null, CancellationToken cancellationToken = default) + { + var duration = window ?? TimeSpan.FromHours(24); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(duration, TimeSpan.Zero); + ArgumentOutOfRangeException.ThrowIfGreaterThan(duration, TimeSpan.FromHours(48)); + var now = _timeProvider.GetUtcNow(); + var hours = new List(); + for (var hour = TruncateHour(now - duration); hour <= TruncateHour(now); hour = hour.AddHours(1)) hours.Add(hour); + var snapshots = await Task.WhenAll(hours.Select(hour => _db.HashGetAllAsync(CounterKey(name, hour)))).WaitAsync(cancellationToken).ConfigureAwait(false); + var totals = new Dictionary(); + var buckets = new List(); + for (int i = 0; i < hours.Count; i++) + { + var counters = snapshots[i].ToDictionary(pair => (string)pair.Name!, pair => (long)pair.Value); + foreach (var pair in counters) totals[pair.Key] = totals.GetValueOrDefault(pair.Key) + pair.Value; + buckets.Add(new() { Hour = hours[i], Counters = counters }); + } + return new JobCounterStats { Totals = totals, Buckets = buckets }; + } + + private RedisKey CounterKey(string name, DateTimeOffset timestamp) => $"{_prefix}counters:{EncodeKey(name)}:{timestamp:yyyy-MM-ddTHH}"; + private static DateTimeOffset TruncateHour(DateTimeOffset value) => new(value.Year, value.Month, value.Day, value.Hour, 0, 0, TimeSpan.Zero); +} diff --git a/src/Foundatio.Redis/RedisJobRuntimeStore.Claims.cs b/src/Foundatio.Redis/RedisJobRuntimeStore.Claims.cs new file mode 100644 index 000000000..e236c8e7f --- /dev/null +++ b/src/Foundatio.Redis/RedisJobRuntimeStore.Claims.cs @@ -0,0 +1,269 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using StackExchange.Redis; + +namespace Foundatio.Jobs; + +public sealed partial class RedisJobRuntimeStore +{ + public async Task CreateOccurrenceAsync(JobState initial, bool allowOverlap = false, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(initial); + ArgumentException.ThrowIfNullOrWhiteSpace(initial.ScheduleName); + if (initial.ExecutionOwner != JobExecutionOwner.Runtime) throw new ArgumentException("Scheduled occurrences must be owned by the job runtime.", nameof(initial)); + ArgumentException.ThrowIfNullOrWhiteSpace(initial.JobType); + cancellationToken.ThrowIfCancellationRequested(); + ValidatePayload(initial.Payload?.Length ?? 0); + initial.RetryPolicy.Validate(); + var now = _timeProvider.GetUtcNow(); + var state = initial with { CreatedUtc = initial.CreatedUtc == default ? now : initial.CreatedUtc, LastUpdatedUtc = now }; + const string script = MonitoringFunctions + "\n" + """ + if redis.call('EXISTS', KEYS[1]) == 1 then return 0 end + if ARGV[2] == '0' and redis.call('SCARD', KEYS[6]) > 0 then return 2 end + redis.call('ZREMRANGEBYSCORE', KEYS[7], '-inf', ARGV[5]) + if redis.call('ZSCORE', KEYS[7], ARGV[1]) then return 0 end + if redis.call('ZCARD', KEYS[2]) - redis.call('ZCARD', KEYS[8]) >= tonumber(ARGV[4]) then return -1 end + if redis.call('ZCARD', KEYS[7]) >= tonumber(ARGV[6]) then return -2 end + redis.call('HSET', KEYS[1], unpack(ARGV, 7)) + redis.call('ZADD', KEYS[7], '+inf', ARGV[1]) + local expiry = redis.call('HGET', KEYS[1], 'expiresUtc') + if expiry then redis.call('ZADD', KEYS[9], expiry, ARGV[1]) end + redis.call('ZADD', KEYS[2], 0, ARGV[1]) + redis.call('ZADD', KEYS[3], 0, ARGV[1]) + redis.call('ZADD', KEYS[4], 0, ARGV[1]) + redis.call('ZADD', KEYS[5], ARGV[3], ARGV[1]) + redis.call('SADD', KEYS[6], ARGV[1]) + syncMonitoring(KEYS[1]) + return 1 + """; + var arguments = new List { state.JobId, allowOverlap ? "1" : "0", Ticks(state.AvailableUtc ?? state.CreatedUtc), _options.MaxActiveJobs, Ticks(now), _options.MaxDeduplicationRecords }; + foreach (var field in ToHash(state)) + { + arguments.Add(field.Name); + arguments.Add(field.Value); + } + + var result = await _db.ScriptEvaluateAsync(script, + new RedisKey[] { JobKey(state.JobId), AllKey, StatusKey(state.Status), NameKey(state.Name), ReadyKey(state.JobType, state.RequiredNodeId), ActiveScheduleKey(state.ScheduleName, state.RequiredNodeId), DeduplicationKey, TerminalKey, UnclaimedKey }, + arguments.ToArray()).ConfigureAwait(false); + ThrowIfCapacityExceeded((long)result); + return (JobOccurrenceResult)(int)result; + } + + private const string ClaimJobScript = RetentionFunctions + "\n" + """ + local now = tonumber(ARGV[1]) + for scan = 1, 100 do + local id, key, score + for _, candidateKey in ipairs(KEYS) do + local candidate + if ARGV[6] ~= '' then + local value = redis.call('ZSCORE', candidateKey, ARGV[6]) + if value then candidate = {ARGV[6], value} else candidate = {} end + else + candidate = redis.call('ZRANGE', candidateKey, 0, 0, 'WITHSCORES') + end + if #candidate > 0 then + local due = tonumber(candidate[2]) + if due <= now and (not score or due < score or (due == score and candidate[1] < id)) then + id, key, score = candidate[1], candidateKey, due + end + end + end + if not id then return {} end + local job = ARGV[5] .. 'job:' .. id + local status = redis.call('HGET', job, 'status') + if redis.call('HGET', job, 'executionOwner') == 'Broker' or (status ~= 'Queued' and status ~= 'Scheduled' and status ~= 'Processing') then + redis.call('ZREM', key, id) + else + local due = redis.call('HGET', job, status == 'Processing' and 'leaseExpiresUtc' or 'availableUtc') + if not due or due == '' then due = redis.call('HGET', job, 'createdUtc') end + if tonumber(due) > now then + redis.call('ZADD', key, due, id) + else + local attempt = tonumber(redis.call('HGET', job, 'attempt') or '0') + local maximum = tonumber(redis.call('HGET', job, 'maxAttempts') or '3') + local expiry = tonumber(redis.call('HGET', job, 'expiresUtc')) + local expired = attempt == 0 and redis.call('HEXISTS', job, 'requiredNodeId') == 1 and expiry and expiry <= now + local cancelled = expired or redis.call('HGET', job, 'cancellationRequested') == '1' + if expired then redis.call('HSET', job, 'resultMessage', 'Unclaimed per-node occurrence expired.') end + redis.call('ZREM', ARGV[5] .. 'status:' .. status, id) + if cancelled or attempt >= maximum then + local terminal = cancelled and 'Cancelled' or 'Failed' + redis.call('HSET', job, 'status', terminal, 'completedUtc', ARGV[1], 'lastUpdatedUtc', ARGV[1]) + redis.call('HDEL', job, 'nodeId', 'claimToken', 'leaseExpiresUtc') + if not cancelled then redis.call('HSET', job, 'error', 'Execution attempts exhausted after lease expiration.') end + redis.call('ZADD', ARGV[5] .. 'status:' .. terminal, 0, id) + redis.call('ZADD', ARGV[5] .. 'terminal', ARGV[1], id) + redis.call('ZREM', key, id) + local active = redis.call('HGET', job, 'activeScheduleKey') + if active then redis.call('SREM', active, id) end + syncMonitoring(job) + finishJob(job, id, ARGV[5], now, tonumber(ARGV[7]), tonumber(ARGV[8]), tonumber(ARGV[9])) + else + redis.call('HSET', job, 'status', 'Processing', 'nodeId', ARGV[2], 'claimToken', ARGV[3], + 'leaseExpiresUtc', ARGV[4], 'startedUtc', ARGV[1], 'lastUpdatedUtc', ARGV[1], 'attempt', attempt + 1) + redis.call('HDEL', job, 'completedUtc') + redis.call('ZADD', ARGV[5] .. 'status:Processing', 0, id) + redis.call('ZADD', key, ARGV[4], id) + redis.call('ZREM', ARGV[5] .. 'unclaimed', id) + syncMonitoring(job) + return redis.call('HGETALL', job) + end + end + end + end + return {} + """; + + private const string CompleteJobScript = RetentionFunctions + "\n" + """ + if expireJob(KEYS[1], tonumber(ARGV[2])) then return 0 end + if redis.call('HGET', KEYS[1], 'status') ~= 'Processing' or redis.call('HGET', KEYS[1], 'claimToken') ~= ARGV[1] then return 0 end + if redis.call('HGET', KEYS[1], 'executionOwner') == 'Broker' then + local kind = tonumber(ARGV[3]) + local status = kind == 0 and 'Completed' or kind == 2 and 'Cancelled' or (kind == 3 or (kind == 1 and ARGV[7] == '1')) and 'RetryPending' or 'Failed' + local id = redis.call('HGET', KEYS[1], 'jobId') + redis.call('ZREM', ARGV[5] .. 'status:Processing', id) + redis.call('ZADD', ARGV[5] .. 'status:' .. status, 0, id) + redis.call('HSET', KEYS[1], 'status', status, 'lastUpdatedUtc', ARGV[2], 'error', ARGV[4], 'resultMessage', ARGV[6]) + redis.call('HDEL', KEYS[1], 'claimToken', 'leaseExpiresUtc', 'availableUtc') + refreshBrokerHistory(KEYS[1], tonumber(ARGV[2])) + syncMonitoring(KEYS[1]) + if status ~= 'RetryPending' then + redis.call('HSET', KEYS[1], 'completedUtc', ARGV[2]) + if status == 'Completed' then redis.call('HSET', KEYS[1], 'progress', 100) end + redis.call('ZADD', ARGV[5] .. 'terminal', ARGV[2], id) + finishJob(KEYS[1], id, ARGV[5], tonumber(ARGV[2]), tonumber(ARGV[9]), tonumber(ARGV[10]), tonumber(ARGV[11])) + end + return 1 + end + if tonumber(redis.call('HGET', KEYS[1], 'leaseExpiresUtc') or '0') <= tonumber(ARGV[2]) then return 0 end + local kind = tonumber(ARGV[3]) + if redis.call('HGET', KEYS[1], 'cancellationRequested') == '1' then kind = 2 end + local attempt = tonumber(redis.call('HGET', KEYS[1], 'attempt') or '0') + local maximum = tonumber(redis.call('HGET', KEYS[1], 'maxAttempts') or '3') + local retry = kind == 1 and attempt < maximum and ARGV[7] == '1' + local status = kind == 0 and 'Completed' or kind == 2 and 'Cancelled' or (kind == 3 or retry) and 'Queued' or 'Failed' + local available = ARGV[2] + if retry then + local initial = tonumber(redis.call('HGET', KEYS[1], 'retryInitialSeconds') or '10') + local maximumDelay = tonumber(redis.call('HGET', KEYS[1], 'retryMaxSeconds') or '300') + local multiplier = tonumber(redis.call('HGET', KEYS[1], 'retryMultiplier') or '2') + local jitter = tonumber(redis.call('HGET', KEYS[1], 'retryJitter') or '0.2') + local baseDelay = initial == 0 and 0 or math.min(maximumDelay, initial * multiplier ^ math.min(100, attempt - 1)) + local seconds = math.min(maximumDelay, baseDelay * (1 + jitter * (2 * tonumber(ARGV[8]) - 1))) + available = string.format('%.0f', tonumber(ARGV[2]) + seconds * 10000000) + end + redis.call('HSET', KEYS[1], 'status', status, 'lastUpdatedUtc', ARGV[2], 'availableUtc', available) + redis.call('HDEL', KEYS[1], 'nodeId', 'claimToken', 'leaseExpiresUtc') + if ARGV[4] ~= '' then redis.call('HSET', KEYS[1], 'error', ARGV[4]) else redis.call('HDEL', KEYS[1], 'error') end + if ARGV[6] ~= '' then redis.call('HSET', KEYS[1], 'resultMessage', ARGV[6]) else redis.call('HDEL', KEYS[1], 'resultMessage') end + if status == 'Queued' then redis.call('HDEL', KEYS[1], 'completedUtc') else redis.call('HSET', KEYS[1], 'completedUtc', ARGV[2]) end + if status == 'Completed' then redis.call('HSET', KEYS[1], 'progress', '100') end + local id = redis.call('HGET', KEYS[1], 'jobId') + local ready = redis.call('HGET', KEYS[1], 'readyKey') + if status == 'Queued' then redis.call('ZADD', ready, available, id) else redis.call('ZREM', ready, id) end + if status ~= 'Queued' then + local active = redis.call('HGET', KEYS[1], 'activeScheduleKey') + if active then redis.call('SREM', active, id) end + end + redis.call('ZREM', ARGV[5] .. 'status:Processing', id) + redis.call('ZADD', ARGV[5] .. 'status:' .. status, 0, id) + if status ~= 'Queued' then + redis.call('ZADD', ARGV[5] .. 'terminal', ARGV[2], id) + finishJob(KEYS[1], id, ARGV[5], tonumber(ARGV[2]), tonumber(ARGV[9]), tonumber(ARGV[10]), tonumber(ARGV[11])) + end + syncMonitoring(KEYS[1]) + return 1 + """; + + private const string RenewJobLeaseScript = """ + if redis.call('HGET', KEYS[1], 'status') ~= 'Processing' or redis.call('HGET', KEYS[1], 'claimToken') ~= ARGV[1] then return 0 end + if tonumber(redis.call('HGET', KEYS[1], 'leaseExpiresUtc') or '0') <= tonumber(ARGV[2]) then return 0 end + redis.call('HSET', KEYS[1], 'leaseExpiresUtc', ARGV[3], 'lastUpdatedUtc', ARGV[2]) + local id = redis.call('HGET', KEYS[1], 'jobId') + redis.call('ZADD', redis.call('HGET', KEYS[1], 'readyKey'), ARGV[3], id) + return 1 + """; + + private const string ReportJobProgressScript = MonitoringFunctions + "\n" + """ + if expireJob(KEYS[1], tonumber(ARGV[2])) then return 0 end + if redis.call('HGET', KEYS[1], 'status') ~= 'Processing' or redis.call('HGET', KEYS[1], 'claimToken') ~= ARGV[1] then return 0 end + if redis.call('HGET', KEYS[1], 'executionOwner') ~= 'Broker' and tonumber(redis.call('HGET', KEYS[1], 'leaseExpiresUtc') or '0') <= tonumber(ARGV[2]) then return 0 end + if ARGV[3] ~= '' then redis.call('HSET', KEYS[1], 'progress', ARGV[3]) end + if ARGV[4] == '1' then redis.call('HSET', KEYS[1], 'progressMessage', ARGV[5]) end + redis.call('HSET', KEYS[1], 'lastUpdatedUtc', ARGV[2], 'lastHeartbeatUtc', ARGV[2]) + refreshBrokerHistory(KEYS[1], tonumber(ARGV[2])) + syncMonitoring(KEYS[1]) + return 1 + """; + + public Task ClaimNextAsync(JobClaimRequest request, CancellationToken cancellationToken = default) + => ClaimJobCoreAsync(null, request, cancellationToken); + + public Task ClaimJobAsync(string jobId, JobClaimRequest request, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(jobId); + return ClaimJobCoreAsync(jobId, request, cancellationToken); + } + + private async Task ClaimJobCoreAsync(string? jobId, JobClaimRequest request, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(request); + ArgumentException.ThrowIfNullOrWhiteSpace(request.NodeId); + ArgumentNullException.ThrowIfNull(request.JobTypes); + if (request.JobTypes.Count == 0 || request.JobTypes.Any(String.IsNullOrWhiteSpace)) + throw new ArgumentException("Register the job types this worker can execute.", nameof(request)); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(request.Lease, TimeSpan.Zero); + cancellationToken.ThrowIfCancellationRequested(); + var now = _timeProvider.GetUtcNow(); + var result = await _db.ScriptEvaluateAsync(ClaimJobScript, request.JobTypes.Distinct(StringComparer.Ordinal).SelectMany(t => new[] { ReadyKey(t), ReadyKey(t, request.NodeId) }).ToArray(), + new RedisValue[] { Ticks(now), request.NodeId, Guid.NewGuid().ToString("N"), Ticks(now.Add(request.Lease)), _prefix, jobId ?? "", _options.MaxHistoryJobs, _options.HistoryRetention.Ticks, _options.DeduplicationRetention.Ticks }).ConfigureAwait(false); + var values = (RedisResult[])result!; + if (values.Length == 0) + return null; + return ReadJobSnapshot(result); + } + + public Task CompleteJobAsync(string jobId, string claimToken, JobCompletion completion, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(completion); + if (!Enum.IsDefined(completion.Kind)) + throw new ArgumentOutOfRangeException(nameof(completion)); + return MutateClaimAsync(CompleteJobScript, jobId, claimToken, + new RedisValue[] { (int)completion.Kind, completion.Kind == JobCompletionKind.Failed ? completion.Error ?? "" : "", _prefix, completion.Message ?? "", completion.Retryable ? "1" : "0", Random.Shared.NextDouble(), _options.MaxHistoryJobs, _options.HistoryRetention.Ticks, _options.DeduplicationRetention.Ticks }, cancellationToken); + } + + public Task RenewJobLeaseAsync(string jobId, string claimToken, TimeSpan lease, CancellationToken cancellationToken = default) + { + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(lease, TimeSpan.Zero); + return MutateClaimAsync(RenewJobLeaseScript, jobId, claimToken, + new RedisValue[] { Ticks(_timeProvider.GetUtcNow().Add(lease)), _prefix }, cancellationToken); + } + + public Task ReportJobProgressAsync(string jobId, string claimToken, int? percent = null, string? message = null, CancellationToken cancellationToken = default) + { + if (percent is < 0 or > 100) + throw new ArgumentOutOfRangeException(nameof(percent)); + return MutateClaimAsync(ReportJobProgressScript, jobId, claimToken, + new RedisValue[] { percent?.ToString(CultureInfo.InvariantCulture) ?? "", message is null ? "0" : "1", message ?? "" }, cancellationToken); + } + + private async Task MutateClaimAsync(string script, string jobId, string claimToken, RedisValue[] arguments, CancellationToken cancellationToken) + { + ArgumentException.ThrowIfNullOrWhiteSpace(jobId); + ArgumentException.ThrowIfNullOrWhiteSpace(claimToken); + cancellationToken.ThrowIfCancellationRequested(); + RedisValue[] values = [claimToken, Ticks(_timeProvider.GetUtcNow()), .. arguments]; + var result = await _db.ScriptEvaluateAsync(script, new RedisKey[] { JobKey(jobId) }, values).ConfigureAwait(false); + return (long)result == 1; + } + + private RedisKey ReadyKey(string jobType, string? nodeId = null) => $"{_prefix}ready:{EncodeKey(jobType)}:{EncodeKey(nodeId ?? "")}"; + private RedisKey ActiveScheduleKey(string name, string? nodeId) => $"{_prefix}active-schedule:{EncodeKey(name)}:{EncodeKey(nodeId ?? "")}"; + private static string EncodeKey(string value) => Convert.ToHexString(Encoding.UTF8.GetBytes(value)); +} diff --git a/src/Foundatio.Redis/RedisJobRuntimeStore.Schedules.cs b/src/Foundatio.Redis/RedisJobRuntimeStore.Schedules.cs new file mode 100644 index 000000000..eef87a3c2 --- /dev/null +++ b/src/Foundatio.Redis/RedisJobRuntimeStore.Schedules.cs @@ -0,0 +1,94 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using StackExchange.Redis; + +namespace Foundatio.Jobs; + +public sealed partial class RedisJobRuntimeStore +{ + private const string SaveScheduleScript = """ + local revision = tonumber(redis.call('HGET', KEYS[1], 'revision') or '0') + local version = tonumber(redis.call('HGET', KEYS[1], 'configurationVersion') or '0') + local incoming = cjson.decode(ARGV[1]) + if ARGV[2] == '1' then + if incoming.ConfigurationVersion < version then return 0 end + if incoming.ConfigurationVersion == version then + if redis.call('HGET', KEYS[1], 'configuration') ~= ARGV[1] then return -2 end + return 0 + end + redis.call('HSET', KEYS[1], 'configuration', ARGV[1]) + version = incoming.ConfigurationVersion + elseif incoming.Revision ~= revision then + return -1 + end + incoming.Revision = revision + 1 + incoming.ConfigurationVersion = version + redis.call('HSET', KEYS[1], 'definition', cjson.encode(incoming), 'revision', revision + 1, 'configurationVersion', version) + redis.call('ZADD', KEYS[2], 0, ARGV[3]) + return 1 + """; + + public Task ScheduleAsync(ScheduledJobDefinition definition, CancellationToken cancellationToken = default) + => SaveScheduleAsync(definition, false, cancellationToken); + + public Task ReconcileAsync(ScheduledJobDefinition definition, CancellationToken cancellationToken = default) + => SaveScheduleAsync(definition, true, cancellationToken); + + private async Task SaveScheduleAsync(ScheduledJobDefinition definition, bool reconcile, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(definition); + definition.Validate(); + if (reconcile) + ArgumentOutOfRangeException.ThrowIfLessThan(definition.ConfigurationVersion, 1); + cancellationToken.ThrowIfCancellationRequested(); + var result = await _db.ScriptEvaluateAsync(SaveScheduleScript, [ScheduleKey(definition.Name), SchedulesKey], + new RedisValue[] { JsonSerializer.Serialize(reconcile ? definition with { Revision = 0 } : definition), reconcile ? "1" : "0", definition.Name }).ConfigureAwait(false); + if ((long)result == -1) + throw new JobException($"Schedule {definition.Name} changed. Reload it before saving."); + if ((long)result == -2) + throw new JobException($"Declared schedule {definition.Name} changed. Increase ConfigurationVersion to apply it."); + } + + public async Task GetScheduleAsync(string name, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(name); + cancellationToken.ThrowIfCancellationRequested(); + var value = await _db.HashGetAsync(ScheduleKey(name), "definition").ConfigureAwait(false); + return value.IsNull ? null : JsonSerializer.Deserialize((string)value!); + } + + public Task UnscheduleAsync(string name, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(name); + cancellationToken.ThrowIfCancellationRequested(); + return _db.ScriptEvaluateAsync(""" + redis.call('DEL', KEYS[1]) + redis.call('ZREM', KEYS[2], ARGV[1]) + return 1 + """, [ScheduleKey(name), SchedulesKey], [name]); + } + + public async Task> GetSchedulesAsync(ScheduleQuery? query = null, CancellationToken cancellationToken = default) + { + query ??= new ScheduleQuery(); + query.Validate(); + cancellationToken.ThrowIfCancellationRequested(); + var result = await _db.ScriptEvaluateAsync(""" + local names = redis.call('ZRANGEBYLEX', KEYS[1], ARGV[1], '+', 'LIMIT', 0, ARGV[2]) + local definitions = {} + for _, name in ipairs(names) do + local definition = redis.call('HGET', ARGV[3] .. name, 'definition') + if definition then table.insert(definitions, definition) end + end + return definitions + """, [SchedulesKey], new RedisValue[] { query.AfterName is null ? "-" : "(" + query.AfterName, query.Limit, $"{_prefix}schedule:" }).ConfigureAwait(false); + return ((RedisValue[]?)result ?? []).Select(value => JsonSerializer.Deserialize((string)value!)!).ToArray(); + } + + private RedisKey ScheduleKey(string name) => $"{_prefix}schedule:{name}"; + private RedisKey SchedulesKey => $"{_prefix}schedules"; +} diff --git a/src/Foundatio.Redis/RedisJobRuntimeStore.cs b/src/Foundatio.Redis/RedisJobRuntimeStore.cs new file mode 100644 index 000000000..177a00169 --- /dev/null +++ b/src/Foundatio.Redis/RedisJobRuntimeStore.cs @@ -0,0 +1,513 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using StackExchange.Redis; + +namespace Foundatio.Jobs; + +/// +/// A Redis-backed durable job and scheduled-dispatch store. Admission, claims, and ownership-guarded mutations +/// are atomic. Sorted indexes bound monitoring pages, due claims, and terminal retention cleanup. +/// +public sealed partial class RedisJobRuntimeStore : IJobRuntimeStore +{ + private const string ClaimDueScript = """ + local ids = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', ARGV[1], 'LIMIT', 0, tonumber(ARGV[2])) + local claimed = {} + for _, id in ipairs(ids) do + local dkey = ARGV[5] .. id + if redis.call('EXISTS', dkey) == 1 then + local owner = redis.call('HGET', dkey, 'claimOwner') + local expires = redis.call('HGET', dkey, 'claimExpiresUtc') + if (not owner or owner == '') or (expires and expires ~= '' and tonumber(expires) <= tonumber(ARGV[1])) then + redis.call('HSET', dkey, 'claimOwner', ARGV[3], 'claimExpiresUtc', ARGV[4]) + redis.call('HINCRBY', dkey, 'attempts', 1) + redis.call('ZADD', KEYS[1], ARGV[4], id) + table.insert(claimed, redis.call('HGETALL', dkey)) + end + else + redis.call('ZREM', KEYS[1], id) + end + end + return claimed + """; + + private readonly IDatabase _db; + private readonly string _prefix; + private readonly TimeProvider _timeProvider; + private readonly JobRuntimeStoreOptions _options; + + public RedisJobRuntimeStore(RedisJobRuntimeStoreOptions options) + { + ArgumentNullException.ThrowIfNull(options); + ArgumentNullException.ThrowIfNull(options.ConnectionMultiplexer); + options.Runtime.Validate(); + _options = options.Runtime; + _db = options.ConnectionMultiplexer.GetDatabase(); + _prefix = options.KeyPrefix ?? ""; + _timeProvider = options.TimeProvider ?? TimeProvider.System; + } + + public RedisJobRuntimeStore(IConnectionMultiplexer connectionMultiplexer, string keyPrefix = "fnd:jobs:", TimeProvider? timeProvider = null) + : this(new RedisJobRuntimeStoreOptions { ConnectionMultiplexer = connectionMultiplexer, KeyPrefix = keyPrefix, TimeProvider = timeProvider }) { } + + public async Task CreateIfAbsentAsync(JobState initial, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(initial); + cancellationToken.ThrowIfCancellationRequested(); + ValidatePayload(initial.Payload?.Length ?? 0); + initial.Validate(); + initial.RetryPolicy.Validate(); + var now = _timeProvider.GetUtcNow(); + var state = initial with { CreatedUtc = initial.CreatedUtc == default ? now : initial.CreatedUtc, LastUpdatedUtc = initial.LastUpdatedUtc == default ? now : initial.LastUpdatedUtc }; + const string script = MonitoringFunctions + "\n" + """ + if purgeBrokerHistory(ARGV[8], tonumber(ARGV[6]), 128) == 128 then return -3 end + if redis.call('EXISTS', KEYS[1]) == 1 then return 0 end + redis.call('ZREMRANGEBYSCORE', KEYS[6], '-inf', ARGV[6]) + if redis.call('ZSCORE', KEYS[6], ARGV[1]) then return 0 end + if redis.call('ZCARD', KEYS[2]) - redis.call('ZCARD', KEYS[5]) >= tonumber(ARGV[2]) then return -1 end + if redis.call('ZCARD', KEYS[6]) >= tonumber(ARGV[5]) then return -2 end + redis.call('HSET', KEYS[1], unpack(ARGV, 9)) + redis.call('ZADD', KEYS[6], ARGV[7], ARGV[1]) + local expiry = redis.call('HGET', KEYS[1], 'expiresUtc') + if expiry then redis.call('ZADD', KEYS[7], expiry, ARGV[1]) end + redis.call('ZADD', KEYS[2], 0, ARGV[1]) + redis.call('ZADD', KEYS[3], 0, ARGV[1]) + redis.call('ZADD', KEYS[4], 0, ARGV[1]) + local ready = redis.call('HGET', KEYS[1], 'readyKey') + local active = redis.call('HGET', KEYS[1], 'activeScheduleKey') + if ARGV[4] == '1' then + if ready then redis.call('ZADD', ready, ARGV[3], ARGV[1]) end + if active then redis.call('SADD', active, ARGV[1]) end + else + local completed = redis.call('HGET', KEYS[1], 'completedUtc') + if completed then redis.call('ZADD', KEYS[5], completed, ARGV[1]) end + end + refreshBrokerHistory(KEYS[1], tonumber(ARGV[6])) + syncMonitoring(KEYS[1]) + return 1 + """; + var args = new List { state.JobId, _options.MaxActiveJobs, Ticks(state.Status == JobStatus.Processing ? state.LeaseExpiresUtc ?? now : state.AvailableUtc ?? state.CreatedUtc), state.Status is JobStatus.Queued or JobStatus.Scheduled or JobStatus.Processing or JobStatus.RetryPending or JobStatus.EnqueueUnknown ? "1" : "0", _options.MaxDeduplicationRecords, Ticks(now), state.CompletedUtc is { } completed ? Ticks(completed.Add(_options.DeduplicationRetention)) : "+inf" }; + args.Add(_prefix); + foreach (var field in ToHash(state)) { args.Add(field.Name); args.Add(field.Value); } + RedisKey[] keys = [JobKey(state.JobId), AllKey, StatusKey(state.Status), NameKey(state.Name), TerminalKey, DeduplicationKey, UnclaimedKey]; + var values = args.ToArray(); + long result; + do + { + cancellationToken.ThrowIfCancellationRequested(); + result = (long)await _db.ScriptEvaluateAsync(script, keys, values).WaitAsync(cancellationToken).ConfigureAwait(false); + } while (result == -3); + ThrowIfCapacityExceeded(result); + } + + public async Task GetAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + const string script = MonitoringFunctions + "\n" + """ + if expireJob(KEYS[1], tonumber(ARGV[1])) then return {} end + return redis.call('HGETALL', KEYS[1]) + """; + var snapshot = await _db.ScriptEvaluateAsync(script, [JobKey(jobId)], [Ticks(_timeProvider.GetUtcNow())]).WaitAsync(cancellationToken).ConfigureAwait(false); + return ((RedisResult[])snapshot!).Length == 0 ? null : ReadJobSnapshot(snapshot); + } + + public async Task QueryAsync(JobQuery query, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(query); + query.Validate(); + cancellationToken.ThrowIfCancellationRequested(); + await PurgeBrokerHistoryAsync(cancellationToken).ConfigureAwait(false); + if (query.NewestFirst) return await QueryNewestAsync(query, cancellationToken).ConfigureAwait(false); + var index = query.Name is not null ? NameKey(query.Name) : query.Status is { } status ? StatusKey(status) : AllKey; + const string script = """ + local ids = redis.call('ZRANGEBYLEX', KEYS[1], ARGV[1], '+', 'LIMIT', 0, 1001) + local result, cursor = {}, '' + for i = 1, math.min(#ids, 1000) do + local id = ids[i] + local job = ARGV[2] .. id + if (ARGV[3] == '' or redis.call('HGET', job, 'status') == ARGV[3]) and (ARGV[5] == '' or redis.call('HGET', job, 'queueName') == ARGV[5]) then + table.insert(result, redis.call('HGETALL', job)) + end + if #result >= tonumber(ARGV[4]) or i == 1000 then + if i < #ids then cursor = id end + break + end + end + return {cursor, result} + """; + var raw = (RedisResult[])(await _db.ScriptEvaluateAsync(script, new RedisKey[] { index }, new RedisValue[] { query.AfterJobId is null ? "-" : "(" + query.AfterJobId, $"{_prefix}job:", query.Status?.ToString() ?? "", query.Limit, query.QueueName ?? "" }).ConfigureAwait(false))!; + var states = ((RedisResult[])raw[1]!).Select(ReadJobSnapshot).ToArray(); + string? cursor = (string?)raw[0]; + return new JobPage(states, String.IsNullOrEmpty(cursor) ? null : cursor); + } + + private static JobState ReadJobSnapshot(RedisResult snapshot) + { + var values = (RedisResult[])snapshot!; + var fields = new Dictionary(values.Length / 2); + for (int i = 0; i < values.Length; i += 2) + fields.Add((RedisValue)values[i], (RedisValue)values[i + 1]); + return FromHash(fields); + } + + private const string RetentionFunctions = MonitoringFunctions + "\n" + """ + local function trimHistory(prefix, now, maximum, retention, limit) + if limit <= 0 then return 0 end + local terminal = prefix .. 'terminal' + local count = redis.call('ZCARD', terminal) + local removeCount = math.min(limit, math.max(count - maximum, redis.call('ZCOUNT', terminal, '-inf', now - retention))) + if removeCount <= 0 then return 0 end + local candidates = redis.call('ZRANGE', terminal, 0, removeCount - 1) + for _, id in ipairs(candidates) do + local job = prefix .. 'job:' .. id + forgetJob(job, prefix, id) + end + return #candidates + end + local function finishJob(job, id, prefix, now, maximum, retention, dedupRetention) + if redis.call('HGET', job, 'executionOwner') ~= 'Broker' then + redis.call('ZADD', prefix .. 'deduplication', string.format('%.0f', now + dedupRetention), id) + end + redis.call('ZREM', prefix .. 'unclaimed', id) + return trimHistory(prefix, now, maximum, retention, 1000) + end + """; + + private void ValidatePayload(long bytes) + { + if (bytes > _options.MaxPayloadBytes) + throw new JobException($"Payload exceeds the configured {_options.MaxPayloadBytes} byte limit."); + } + + private void ThrowIfCapacityExceeded(long result) + { + if (result is -1 or -2) + throw new JobException(result == -1 + ? $"Active job capacity ({_options.MaxActiveJobs}) reached. Configure RedisJobRuntimeStoreOptions.Runtime.MaxActiveJobs." + : $"Idempotency capacity ({_options.MaxDeduplicationRecords}) reached. Configure RedisJobRuntimeStoreOptions.Runtime.MaxDeduplicationRecords."); + } + + public async Task GetStatsAsync(CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + await PurgeBrokerHistoryAsync(cancellationToken).ConfigureAwait(false); + var raw = (RedisResult[])(await _db.ScriptEvaluateAsync("return {redis.call('ZCARD', KEYS[1]), redis.call('ZCARD', KEYS[2]), redis.call('ZCARD', KEYS[3]), redis.call('ZCARD', KEYS[4])}", + [AllKey, TerminalKey, DeduplicationKey, DueKey]).ConfigureAwait(false))!; + return new JobRuntimeStoreStats((long)raw[0] - (long)raw[1], (long)raw[1], (long)raw[2], (long)raw[3]); + } + + public async Task CleanupAsync(int limit = 1000, CancellationToken cancellationToken = default) + { + ArgumentOutOfRangeException.ThrowIfLessThan(limit, 1); + ArgumentOutOfRangeException.ThrowIfGreaterThan(limit, 1000); + cancellationToken.ThrowIfCancellationRequested(); + await PurgeBrokerHistoryAsync(cancellationToken).ConfigureAwait(false); + const string script = RetentionFunctions + "\n" + """ + local now, prefix = tonumber(ARGV[1]), ARGV[3] + redis.call('ZREMRANGEBYSCORE', KEYS[3], '-inf', ARGV[1]) + local expired = redis.call('ZRANGEBYSCORE', KEYS[4], '-inf', ARGV[1], 'LIMIT', 0, ARGV[2]) + local retired = 0 + for _, id in ipairs(expired) do + local job = prefix .. 'job:' .. id + local status = redis.call('HGET', job, 'status') + if (status == 'Queued' or status == 'Scheduled') and redis.call('HGET', job, 'attempt') == '0' then + redis.call('ZREM', prefix .. 'status:' .. status, id) + redis.call('ZADD', prefix .. 'status:Cancelled', 0, id) + redis.call('HSET', job, 'status', 'Cancelled', 'completedUtc', ARGV[1], 'lastUpdatedUtc', ARGV[1], 'resultMessage', 'Unclaimed per-node occurrence expired.') + redis.call('ZADD', KEYS[1], ARGV[1], id) + syncMonitoring(job) + local ready, active = redis.call('HGET', job, 'readyKey'), redis.call('HGET', job, 'activeScheduleKey') + if ready then redis.call('ZREM', ready, id) end + if active then redis.call('SREM', active, id) end + redis.call('ZADD', KEYS[3], string.format('%.0f', now + tonumber(ARGV[6])), id) + retired = retired + 1 + end + redis.call('ZREM', KEYS[4], id) + end + return retired + trimHistory(prefix, now, tonumber(ARGV[4]), tonumber(ARGV[5]), tonumber(ARGV[2]) - retired) + """; + return (int)(await _db.ScriptEvaluateAsync(script, [TerminalKey, AllKey, DeduplicationKey, UnclaimedKey], + [Ticks(_timeProvider.GetUtcNow()), limit, _prefix, _options.MaxHistoryJobs, _options.HistoryRetention.Ticks, _options.DeduplicationRetention.Ticks]).ConfigureAwait(false)); + } + + public async Task RequestCancellationAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + await PurgeBrokerHistoryAsync(cancellationToken).ConfigureAwait(false); + const string script = RetentionFunctions + "\n" + """ + if redis.call('EXISTS', KEYS[1]) == 0 then return 0 end + local status = redis.call('HGET', KEYS[1], 'status') + if redis.call('HGET', KEYS[1], 'executionOwner') == 'Broker' and (status == 'Completed' or status == 'Failed' or status == 'Cancelled' or status == 'DeadLettered') then return 0 end + redis.call('HSET', KEYS[1], 'cancellationRequested', '1', 'lastUpdatedUtc', ARGV[2]) + if redis.call('HGET', KEYS[1], 'executionOwner') ~= 'Broker' and (status == 'Queued' or status == 'Scheduled') then + redis.call('HSET', KEYS[1], 'status', 'Cancelled', 'completedUtc', ARGV[2]) + redis.call('ZREM', KEYS[2], ARGV[1]) + redis.call('ZREM', KEYS[3], ARGV[1]) + redis.call('ZADD', KEYS[4], 0, ARGV[1]) + redis.call('ZADD', KEYS[5], ARGV[2], ARGV[1]) + local ready = redis.call('HGET', KEYS[1], 'readyKey') + local active = redis.call('HGET', KEYS[1], 'activeScheduleKey') + if ready then redis.call('ZREM', ready, ARGV[1]) end + if active then redis.call('SREM', active, ARGV[1]) end + finishJob(KEYS[1], ARGV[1], ARGV[3], tonumber(ARGV[2]), tonumber(ARGV[4]), tonumber(ARGV[5]), tonumber(ARGV[6])) + end + syncMonitoring(KEYS[1]) + return 1 + """; + var result = await _db.ScriptEvaluateAsync(script, + new RedisKey[] { JobKey(jobId), StatusKey(JobStatus.Queued), StatusKey(JobStatus.Scheduled), StatusKey(JobStatus.Cancelled), TerminalKey }, + new RedisValue[] { jobId, Ticks(_timeProvider.GetUtcNow()), _prefix, _options.MaxHistoryJobs, _options.HistoryRetention.Ticks, _options.DeduplicationRetention.Ticks }).ConfigureAwait(false); + return (long)result == 1; + } + + public async Task IsCancellationRequestedAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + const string script = MonitoringFunctions + "\n" + """ + if expireJob(KEYS[1], tonumber(ARGV[1])) then return 0 end + return redis.call('HGET', KEYS[1], 'cancellationRequested') == '1' and 1 or 0 + """; + return (long)await _db.ScriptEvaluateAsync(script, [JobKey(jobId)], [Ticks(_timeProvider.GetUtcNow())]).WaitAsync(cancellationToken).ConfigureAwait(false) == 1; + } + + public async Task ScheduleDispatchAsync(ScheduledDispatchState dispatch, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(dispatch); + cancellationToken.ThrowIfCancellationRequested(); + ValidatePayload(dispatch.Body.Length + dispatch.Headers.Sum(h => (long)System.Text.Encoding.UTF8.GetByteCount(h.Key) + System.Text.Encoding.UTF8.GetByteCount(h.Value))); + const string script = """ + if redis.call('EXISTS', KEYS[1]) == 1 then return 0 end + if redis.call('ZCARD', KEYS[2]) >= tonumber(ARGV[3]) then return -1 end + redis.call('HSET', KEYS[1], unpack(ARGV, 4)) + redis.call('ZADD', KEYS[2], ARGV[2], ARGV[1]) + return 1 + """; + var arguments = new List { dispatch.DispatchId, Ticks(dispatch.DueUtc), _options.MaxScheduledDispatches }; + foreach (var field in ToHash(dispatch)) { arguments.Add(field.Name); arguments.Add(field.Value); } + if ((long)await _db.ScriptEvaluateAsync(script, [DispatchKey(dispatch.DispatchId), DueKey], arguments.ToArray()).ConfigureAwait(false) == -1) + throw new JobException($"Scheduled dispatch capacity ({_options.MaxScheduledDispatches}) reached."); + } + + public async Task> ClaimDueDispatchesAsync(DateTimeOffset now, int limit, string nodeId, TimeSpan lease, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + ArgumentException.ThrowIfNullOrEmpty(nodeId); + + var result = await _db.ScriptEvaluateAsync(ClaimDueScript, + [DueKey], + [now.UtcTicks, Math.Max(1, limit), nodeId, Ticks(_timeProvider.GetUtcNow().Add(lease)), $"{_prefix}dispatch:"]).ConfigureAwait(false); + + var snapshots = (RedisResult[]?)result ?? []; + var dispatches = new List(snapshots.Length); + foreach (var snapshot in snapshots) + { + var values = (RedisValue[]?)snapshot ?? []; + var entries = new HashEntry[values.Length / 2]; + for (int index = 0; index < entries.Length; index++) + entries[index] = new HashEntry(values[index * 2], values[index * 2 + 1]); + dispatches.Add(DispatchFromHash(entries)); + } + + return dispatches; + } + + public async Task CompleteDispatchAsync(string dispatchId, string nodeId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + return (int)await _db.ScriptEvaluateAsync(""" + if redis.call('HGET', KEYS[1], 'claimOwner') ~= ARGV[1] then return 0 end + if tonumber(redis.call('HGET', KEYS[1], 'claimExpiresUtc') or '0') <= tonumber(ARGV[2]) then return 0 end + redis.call('DEL', KEYS[1]) + redis.call('ZREM', KEYS[2], ARGV[3]) + return 1 + """, [DispatchKey(dispatchId), DueKey], [nodeId, Ticks(_timeProvider.GetUtcNow()), dispatchId]).ConfigureAwait(false) == 1; + } + + public Task ReleaseDispatchAsync(string dispatchId, string nodeId, DateTimeOffset nextDueUtc, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + return _db.ScriptEvaluateAsync(""" + if redis.call('HGET', KEYS[1], 'claimOwner') ~= ARGV[1] then return 0 end + if tonumber(redis.call('HGET', KEYS[1], 'claimExpiresUtc') or '0') <= tonumber(ARGV[2]) then return 0 end + redis.call('HDEL', KEYS[1], 'claimOwner', 'claimExpiresUtc') + redis.call('HSET', KEYS[1], 'dueUtc', ARGV[4]) + redis.call('ZADD', KEYS[2], ARGV[4], ARGV[3]) + return 1 + """, [DispatchKey(dispatchId), DueKey], [nodeId, Ticks(_timeProvider.GetUtcNow()), dispatchId, Ticks(nextDueUtc)]); + } + + private RedisKey JobKey(string id) => $"{_prefix}job:{id}"; + private RedisKey StatusKey(JobStatus status) => $"{_prefix}status:{status}"; + private RedisKey NameKey(string name) => $"{_prefix}name:{name}"; + private RedisKey DispatchKey(string id) => $"{_prefix}dispatch:{id}"; + private RedisKey TerminalKey => $"{_prefix}terminal"; + private RedisKey AllKey => $"{_prefix}all"; + private RedisKey DeduplicationKey => $"{_prefix}deduplication"; + private RedisKey UnclaimedKey => $"{_prefix}unclaimed"; + private RedisKey DueKey => $"{_prefix}dispatches:due"; + + private static string Ticks(DateTimeOffset value) => value.UtcTicks.ToString(CultureInfo.InvariantCulture); + + private static DateTimeOffset? ParseTime(RedisValue value) + { + return value.IsNullOrEmpty || !Int64.TryParse((string?)value, NumberStyles.Integer, CultureInfo.InvariantCulture, out long ticks) + ? null + : new DateTimeOffset(ticks, TimeSpan.Zero); + } + + private static string? ToStringOrNull(RedisValue value) => value.IsNullOrEmpty ? null : (string)value!; + + private HashEntry[] ToHash(JobState state) + { + var entries = new List + { + new("jobId", state.JobId), + new("name", state.Name), + new("monitorName", EncodeKey(state.Name)), + new("executionOwner", state.ExecutionOwner.ToString()), + new("status", state.Status.ToString()), + new("attempt", state.Attempt), + new("maxAttempts", state.MaxAttempts), + new("retryPolicy", JsonSerializer.Serialize(state.RetryPolicy)), + new("retryInitialSeconds", state.RetryPolicy.InitialDelay.TotalSeconds), + new("retryMaxSeconds", state.RetryPolicy.MaxDelay.TotalSeconds), + new("retryMultiplier", state.RetryPolicy.Multiplier), + new("retryJitter", state.RetryPolicy.JitterFactor), + new("cancellationRequested", state.CancellationRequested ? "1" : "0"), + new("createdUtc", Ticks(state.CreatedUtc)), + new("lastUpdatedUtc", Ticks(state.LastUpdatedUtc)) + }; + + if (state.QueueName is not null) + { + entries.Add(new("queueName", state.QueueName)); + entries.Add(new("monitorQueue", EncodeKey(state.QueueName))); + } + if (state.Metadata is not null) entries.Add(new("metadata", JsonSerializer.Serialize(state.Metadata))); + if (state.LastHeartbeatUtc is { } heartbeat) entries.Add(new("lastHeartbeatUtc", Ticks(heartbeat))); + if (state.HistoryRetention is { } retention) entries.Add(new("historyRetention", retention.Ticks)); + if (state.HistoryExpiresUtc is { } historyExpiry) entries.Add(new("historyExpiresUtc", Ticks(historyExpiry))); + if (state.JobType is not null) + { + entries.Add(new("jobType", state.JobType)); + if (state.ExecutionOwner == JobExecutionOwner.Runtime) entries.Add(new("readyKey", ReadyKey(state.JobType, state.RequiredNodeId).ToString())); + } + if (state.ExpiresUtc is { } expires) entries.Add(new("expiresUtc", Ticks(expires))); + if (state.RequiredNodeId is not null) entries.Add(new("requiredNodeId", state.RequiredNodeId)); + if (state.ScheduleName is not null) + { + entries.Add(new("scheduleName", state.ScheduleName)); + entries.Add(new("activeScheduleKey", ActiveScheduleKey(state.ScheduleName, state.RequiredNodeId).ToString())); + } + if (state.Payload is { } payload) entries.Add(new("payload", Convert.ToBase64String(payload.Span))); + if (state.PayloadType is not null) entries.Add(new("payloadType", state.PayloadType)); + if (state.Progress is { } progress) entries.Add(new("progress", progress)); + if (state.ProgressMessage is not null) entries.Add(new("progressMessage", state.ProgressMessage)); + if (state.NodeId is not null) entries.Add(new("nodeId", state.NodeId)); + if (state.ClaimToken is not null) entries.Add(new("claimToken", state.ClaimToken)); + if (state.AvailableUtc is { } available) entries.Add(new("availableUtc", Ticks(available))); + if (state.StartedUtc is { } started) entries.Add(new("startedUtc", Ticks(started))); + if (state.CompletedUtc is { } completed) entries.Add(new("completedUtc", Ticks(completed))); + if (state.LeaseExpiresUtc is { } leaseExpires) entries.Add(new("leaseExpiresUtc", Ticks(leaseExpires))); + if (state.Error is not null) entries.Add(new("error", state.Error)); + if (state.ScheduledForUtc is { } scheduledFor) entries.Add(new("scheduledForUtc", Ticks(scheduledFor))); + + return entries.ToArray(); + } + + private static JobState FromHash(Dictionary map) + { + RedisValue Get(string field) => map.TryGetValue(field, out var value) ? value : RedisValue.Null; + + return new JobState + { + JobId = (string)Get("jobId")!, + Name = (string)Get("name")!, + JobType = ToStringOrNull(Get("jobType")), + ExecutionOwner = Get("executionOwner") == "Broker" ? JobExecutionOwner.Broker : JobExecutionOwner.Runtime, + QueueName = ToStringOrNull(Get("queueName")), + Metadata = Get("metadata").IsNullOrEmpty ? null : JsonSerializer.Deserialize>((string)Get("metadata")!), + LastHeartbeatUtc = ParseTime(Get("lastHeartbeatUtc")), + HistoryRetention = Get("historyRetention").IsNullOrEmpty ? null : TimeSpan.FromTicks((long)Get("historyRetention")), + HistoryExpiresUtc = ParseTime(Get("historyExpiresUtc")), + Payload = Get("payload").IsNullOrEmpty ? null : Convert.FromBase64String((string)Get("payload")!), + PayloadType = ToStringOrNull(Get("payloadType")), + Status = Enum.Parse((string)Get("status")!), + Progress = Get("progress").IsNullOrEmpty ? null : (int)Get("progress"), + ProgressMessage = ToStringOrNull(Get("progressMessage")), + Attempt = Get("attempt").IsNullOrEmpty ? 0 : (int)Get("attempt"), + NodeId = ToStringOrNull(Get("nodeId")), + ClaimToken = ToStringOrNull(Get("claimToken")), + RequiredNodeId = ToStringOrNull(Get("requiredNodeId")), + ScheduleName = ToStringOrNull(Get("scheduleName")), + MaxAttempts = Get("maxAttempts").IsNullOrEmpty ? 3 : (int)Get("maxAttempts"), + AvailableUtc = ParseTime(Get("availableUtc")), + ExpiresUtc = ParseTime(Get("expiresUtc")), + RetryPolicy = Get("retryPolicy").IsNullOrEmpty ? new() : JsonSerializer.Deserialize((string)Get("retryPolicy")!)!, + CreatedUtc = ParseTime(Get("createdUtc")) ?? default, + LastUpdatedUtc = ParseTime(Get("lastUpdatedUtc")) ?? default, + StartedUtc = ParseTime(Get("startedUtc")), + CompletedUtc = ParseTime(Get("completedUtc")), + LeaseExpiresUtc = ParseTime(Get("leaseExpiresUtc")), + Error = ToStringOrNull(Get("error")), + ResultMessage = ToStringOrNull(Get("resultMessage")), + CancellationRequested = Get("cancellationRequested") == "1", + ScheduledForUtc = ParseTime(Get("scheduledForUtc")) + }; + } + + private static HashEntry[] ToHash(ScheduledDispatchState dispatch) + { + var headers = new Dictionary(StringComparer.Ordinal); + foreach (var header in dispatch.Headers) + headers[header.Key] = header.Value; + + var entries = new List + { + new("dispatchId", dispatch.DispatchId), + new("kind", dispatch.Kind.ToString()), + new("body", Convert.ToBase64String(dispatch.Body.Span)), + new("headers", JsonSerializer.Serialize(headers)), + new("options", JsonSerializer.Serialize(dispatch.Options)), + new("dueUtc", Ticks(dispatch.DueUtc)), + new("attempts", dispatch.Attempts) + }; + + if (dispatch.Destination is not null) entries.Add(new("destination", JsonSerializer.Serialize(dispatch.Destination))); + if (dispatch.ClaimOwner is not null) entries.Add(new("claimOwner", dispatch.ClaimOwner)); + if (dispatch.ClaimExpiresUtc is { } claimExpires) entries.Add(new("claimExpiresUtc", Ticks(claimExpires))); + + return entries.ToArray(); + } + + private static ScheduledDispatchState DispatchFromHash(HashEntry[] entries) + { + var map = entries.ToDictionary(e => (string)e.Name!, e => e.Value); + RedisValue Get(string field) => map.TryGetValue(field, out var value) ? value : RedisValue.Null; + + string headersJson = (string?)Get("headers") ?? "{}"; + var headerMap = JsonSerializer.Deserialize>(headersJson) ?? []; + var options = JsonSerializer.Deserialize((string?)Get("options") ?? "{}") ?? new TransportSendOptions(); + var destination = Get("destination"); + + return new ScheduledDispatchState + { + DispatchId = (string)Get("dispatchId")!, + Kind = Enum.Parse((string)Get("kind")!), + Destination = destination.IsNullOrEmpty ? null : JsonSerializer.Deserialize((string)destination!), + Body = Get("body").IsNullOrEmpty ? ReadOnlyMemory.Empty : Convert.FromBase64String((string)Get("body")!), + Headers = MessageHeaders.Create(headerMap), + Options = options, + DueUtc = ParseTime(Get("dueUtc")) ?? default, + ClaimOwner = ToStringOrNull(Get("claimOwner")), + ClaimExpiresUtc = ParseTime(Get("claimExpiresUtc")), + Attempts = Get("attempts").IsNullOrEmpty ? 0 : (int)Get("attempts") + }; + } +} diff --git a/src/Foundatio.Redis/RedisJobRuntimeStoreOptions.cs b/src/Foundatio.Redis/RedisJobRuntimeStoreOptions.cs new file mode 100644 index 000000000..69c0d1ad0 --- /dev/null +++ b/src/Foundatio.Redis/RedisJobRuntimeStoreOptions.cs @@ -0,0 +1,22 @@ +using System; +using StackExchange.Redis; + +namespace Foundatio.Jobs; + +public class RedisJobRuntimeStoreOptions +{ + /// The Redis connection to use. Required. + public IConnectionMultiplexer ConnectionMultiplexer { get; set; } = null!; + + /// Prefix applied to every key this store creates. Useful to isolate environments/runs on a shared Redis. + public string KeyPrefix { get; set; } = "fnd:jobs:"; + + /// Independent active, history, idempotency, dispatch and payload budgets. + public JobRuntimeStoreOptions Runtime { get; set; } = new(); + + /// Maximum active jobs; retained terminal history has its own budget. + public int MaxJobs { get => Runtime.MaxActiveJobs; set => Runtime = Runtime with { MaxActiveJobs = value }; } + + /// Time source (defaults to ). + public TimeProvider? TimeProvider { get; set; } +} diff --git a/src/Foundatio.Redis/RedisLockProvider.cs b/src/Foundatio.Redis/RedisLockProvider.cs new file mode 100644 index 000000000..f06885f53 --- /dev/null +++ b/src/Foundatio.Redis/RedisLockProvider.cs @@ -0,0 +1,72 @@ +using System; +using System.Diagnostics; +using System.Threading; +using System.Threading.Tasks; +using StackExchange.Redis; + +namespace Foundatio.Lock; + +/// Redis resource locks with ownership-checked renewal and release. +public sealed class RedisLockProvider(IConnectionMultiplexer connection, string keyPrefix = "fnd:locks:") : ILockProvider +{ + private const string ReleaseScript = "if redis.call('GET', KEYS[1]) == ARGV[1] then return redis.call('DEL', KEYS[1]) else return 0 end"; + private const string RenewScript = "if redis.call('GET', KEYS[1]) == ARGV[1] then return redis.call('PEXPIRE', KEYS[1], ARGV[2]) else return 0 end"; + + public async Task AcquireAsync(string resource, TimeSpan? timeUntilExpires = null, bool releaseOnDispose = true, CancellationToken cancellationToken = default) + => await TryAcquireAsync(resource, timeUntilExpires, releaseOnDispose, cancellationToken).ConfigureAwait(false) + ?? throw new LockAcquisitionTimeoutException(resource); + + public async Task TryAcquireAsync(string resource, TimeSpan? timeUntilExpires = null, bool releaseOnDispose = true, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(resource); + var lifetime = timeUntilExpires ?? TimeSpan.FromMinutes(20); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(lifetime, TimeSpan.Zero); + string id = Guid.NewGuid().ToString("N"); + long started = Stopwatch.GetTimestamp(); + var database = connection.GetDatabase(); + do + { + // Observe every acquisition outcome even if cancellation arrives during Redis I/O, so a late successful acquisition can still be released by the caller. + if (await database.StringSetAsync(keyPrefix + resource, id, lifetime, When.NotExists).ConfigureAwait(false)) + return new Handle(this, resource, id, Stopwatch.GetElapsedTime(started), releaseOnDispose); + if (cancellationToken.IsCancellationRequested) return null; + try { await Task.Delay(TimeSpan.FromMilliseconds(50), cancellationToken).ConfigureAwait(false); } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) { return null; } + } while (true); + } + + public Task IsLockedAsync(string resource) => connection.GetDatabase().KeyExistsAsync(keyPrefix + resource); + public async Task ReleaseAsync(string resource, string lockId) + => _ = await connection.GetDatabase().ScriptEvaluateAsync(ReleaseScript, [keyPrefix + resource], [lockId]).ConfigureAwait(false); + public async Task ReleaseAsync(string resource) + => _ = await connection.GetDatabase().KeyDeleteAsync(keyPrefix + resource).ConfigureAwait(false); + public async Task RenewAsync(string resource, string lockId, TimeSpan? timeUntilExpires = null) + { + var lifetime = timeUntilExpires ?? TimeSpan.FromMinutes(20); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(lifetime, TimeSpan.Zero); + if ((int)await connection.GetDatabase().ScriptEvaluateAsync(RenewScript, [keyPrefix + resource], [lockId, (long)Math.Ceiling(lifetime.TotalMilliseconds)]).ConfigureAwait(false) == 0) + throw new LockOwnershipLostException($"The lock on '{resource}' is no longer owned by this holder."); + } + + private sealed class Handle(RedisLockProvider owner, string resource, string id, TimeSpan waited, bool releaseOnDispose) : ILock + { + private int _released; + private int _renewals; + public string LockId => id; + public string Resource => resource; + public DateTime AcquiredTimeUtc { get; } = DateTime.UtcNow; + public TimeSpan TimeWaitedForLock => waited; + public int RenewalCount => Volatile.Read(ref _renewals); + public async Task RenewAsync(TimeSpan? timeUntilExpires = null) + { + if (Volatile.Read(ref _released) != 0) throw new LockOwnershipLostException("This lock was already released."); + await owner.RenewAsync(resource, id, timeUntilExpires).ConfigureAwait(false); + Interlocked.Increment(ref _renewals); + } + public async Task ReleaseAsync() + { + if (Interlocked.Exchange(ref _released, 1) == 0) await owner.ReleaseAsync(resource, id).ConfigureAwait(false); + } + public ValueTask DisposeAsync() => releaseOnDispose ? new ValueTask(ReleaseAsync()) : default; + } +} diff --git a/src/Foundatio.TestHarness/Caching/HybridCacheClientTestBase.cs b/src/Foundatio.TestHarness/Caching/HybridCacheClientTestBase.cs index 5963d16c9..04832678d 100644 --- a/src/Foundatio.TestHarness/Caching/HybridCacheClientTestBase.cs +++ b/src/Foundatio.TestHarness/Caching/HybridCacheClientTestBase.cs @@ -18,7 +18,7 @@ public class HybridCacheClientTestBase : CacheClientTestsBase, IDisposable public HybridCacheClientTestBase(ITestOutputHelper output) : base(output) { _distributedCache = new InMemoryCacheClient(o => o.CloneValues(true).ShouldThrowOnSerializationError(true).LoggerFactory(Log)); - _messageBus = new InMemoryMessageBus(o => o.LoggerFactory(Log)); + _messageBus = new MessageBus(new InMemoryMessageTransport(), new MessageBusOptions { LoggerFactory = Log }); } /// @@ -593,6 +593,6 @@ public virtual async Task SetAsync_WithMultipleInstances_UsesLocalCache() public void Dispose() { _distributedCache.Dispose(); - _messageBus.Dispose(); + _messageBus.DisposeAsync().AsTask().GetAwaiter().GetResult(); } } diff --git a/src/Foundatio.TestHarness/Jobs/HelloWorldJob.cs b/src/Foundatio.TestHarness/Jobs/HelloWorldJob.cs deleted file mode 100644 index b79a2a0ff..000000000 --- a/src/Foundatio.TestHarness/Jobs/HelloWorldJob.cs +++ /dev/null @@ -1,79 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Jobs; -using Microsoft.Extensions.Logging; - -namespace Foundatio.Tests.Jobs; - -public class HelloWorldJob : JobBase -{ - private readonly string _id; - - public HelloWorldJob(TimeProvider? timeProvider, ILoggerFactory loggerFactory) : base(timeProvider, null, loggerFactory) - { - _id = Guid.NewGuid().ToString("N").Substring(0, 10); - } - - public static int GlobalRunCount; - public int RunCount { get; set; } - - protected override Task RunInternalAsync(JobContext context) - { - RunCount++; - Interlocked.Increment(ref GlobalRunCount); - - _logger.LogTrace("HelloWorld Running: instance={Id} runs={RunCount} global={GlobalRunCount}", _id, RunCount, GlobalRunCount); - - return Task.FromResult(JobResult.Success); - } -} - -public class FailingJob : JobBase -{ - private readonly string _id; - - public int RunCount { get; set; } - - public FailingJob(TimeProvider timeProvider, ILoggerFactory loggerFactory) : base(timeProvider, null, loggerFactory) - { - _id = Guid.NewGuid().ToString("N").Substring(0, 10); - } - - protected override Task RunInternalAsync(JobContext context) - { - RunCount++; - - _logger.LogTrace("FailingJob Running: instance={Id} runs={RunCount}", _id, RunCount); - - return Task.FromResult(JobResult.FailedWithMessage("Test failure")); - } -} - -public class LongRunningJob : JobBase -{ - private readonly string _id; - private int _iterationCount; - - public LongRunningJob(TimeProvider? timeProvider, ILoggerFactory loggerFactory) : base(timeProvider, null, loggerFactory) - { - _id = Guid.NewGuid().ToString("N").Substring(0, 10); - } - - public int IterationCount => _iterationCount; - - protected override Task RunInternalAsync(JobContext context) - { - do - { - Interlocked.Increment(ref _iterationCount); - if (context.CancellationToken.IsCancellationRequested) - break; - - if (_iterationCount % 10000 == 0) - _logger.LogTrace("LongRunningJob Running: instance={Id} iterations={IterationCount}", _id, IterationCount); - } while (true); - - return Task.FromResult(JobResult.Success); - } -} diff --git a/src/Foundatio.TestHarness/Jobs/JobQueueTestsBase.cs b/src/Foundatio.TestHarness/Jobs/JobQueueTestsBase.cs deleted file mode 100644 index 3d1844582..000000000 --- a/src/Foundatio.TestHarness/Jobs/JobQueueTestsBase.cs +++ /dev/null @@ -1,218 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Diagnostics; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; -using Exceptionless; -using Foundatio.Caching; -using Foundatio.Jobs; -using Foundatio.Lock; -using Foundatio.Queues; -using Foundatio.Xunit; -using Microsoft.Extensions.Logging; -using Xunit; - -namespace Foundatio.Tests.Jobs; - -public abstract class JobQueueTestsBase : TestWithLoggingBase -{ - private readonly ActivitySource _activitySource = new(nameof(JobQueueTestsBase)); - - public JobQueueTestsBase(ITestOutputHelper output) : base(output) - { - } - - protected abstract IQueue GetSampleWorkItemQueue(int retries, TimeSpan retryDelay); - - public virtual async Task ActivityWillFlowThroughQueueJobAsync() - { - using var queue = GetSampleWorkItemQueue(retries: 0, retryDelay: TimeSpan.Zero); - await queue.DeleteQueueAsync(); - - Activity? parentActivity = null; - using var listener = new ActivityListener - { - ShouldListenTo = s => s.Name == nameof(JobQueueTestsBase) || s.Name == "Foundatio", - Sample = (ref ActivityCreationOptions _) => ActivitySamplingResult.AllDataAndRecorded, - ActivityStarted = a => - { - if (a.OperationName != "ProcessQueueEntry") - return; - - Assert.NotNull(parentActivity); - Assert.Equal(parentActivity.RootId, a.RootId); - Assert.Equal(parentActivity.SpanId, a.ParentSpanId); - }, - ActivityStopped = a => { } - }; - ActivitySource.AddActivityListener(listener); - - parentActivity = _activitySource.StartActivity("Parent"); - Assert.NotNull(parentActivity); - - string? enqueueTask = await queue.EnqueueAsync(new SampleQueueWorkItem - { - Created = DateTime.UtcNow, - Path = "somepath" - }); - - // clear activity and then verify that Activity.Current = null; - var job = new SampleQueueJob(queue, loggerFactory: Log); - await job.RunAsync(); - - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(0, stats.Queued); - Assert.Equal(1, stats.Enqueued); - Assert.Equal(1, stats.Dequeued); - } - - public virtual async Task CanRunQueueJobAsync() - { - const int workItemCount = 100; - using var queue = GetSampleWorkItemQueue(retries: 0, retryDelay: TimeSpan.Zero); - await queue.DeleteQueueAsync(); - - var enqueueTask = Parallel.ForEachAsync(Enumerable.Range(1, workItemCount), async (index, _) => await queue.EnqueueAsync(new SampleQueueWorkItem - { - Created = DateTime.UtcNow, - Path = "somepath" + index - })); - - var job = new SampleQueueJob(queue, loggerFactory: Log); - await Task.Delay(10); - await Task.WhenAll(job.RunUntilEmptyAsync(), enqueueTask); - - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(0, stats.Queued); - Assert.Equal(workItemCount, stats.Enqueued); - Assert.Equal(workItemCount, stats.Dequeued); - } - - public virtual async Task CanRunQueueJobWithLockFailAsync() - { - const int workItemCount = 10; - const int allowedLockCount = 5; - Log.SetLogLevel(LogLevel.Trace); - - using var queue = GetSampleWorkItemQueue(retries: 3, retryDelay: TimeSpan.Zero); - Assert.NotNull(queue); - - await queue.DeleteQueueAsync(); - - var enqueueTask = Parallel.ForEachAsync(Enumerable.Range(1, workItemCount), async (index, _) => - { - _logger.LogInformation("Enqueue #{Index}", index); - await queue.EnqueueAsync(new SampleQueueWorkItem - { - Created = DateTime.UtcNow, - Path = "somepath" + index - }); - }); - - var lockProvider = new ThrottlingLockProvider(new InMemoryCacheClient(o => o.LoggerFactory(Log)), allowedLockCount, TimeSpan.FromDays(1), null, null, Log); - var job = new SampleQueueJobWithLocking(queue, lockProvider, loggerFactory: Log); - await Task.Delay(10); - _logger.LogInformation("Starting RunUntilEmptyAsync"); - await Task.WhenAll(job.RunUntilEmptyAsync(), enqueueTask); - _logger.LogInformation("Done RunUntilEmptyAsync"); - - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(0, stats.Queued); - Assert.Equal(workItemCount, stats.Enqueued); - Assert.Equal(allowedLockCount, stats.Completed); - Assert.Equal(allowedLockCount * 4, stats.Abandoned); - Assert.Equal(allowedLockCount, stats.Deadletter); - } - - public virtual async Task CanRunMultipleQueueJobsAsync() - { - const int jobCount = 5; - const int workItemCount = 100; - - Log.SetLogLevel(LogLevel.Information); - - var queues = new List>(); - try - { - for (int i = 0; i < jobCount; i++) - { - var q = GetSampleWorkItemQueue(retries: 1, retryDelay: TimeSpan.Zero); - await q.DeleteQueueAsync(); - queues.Add(q); - } - _logger.LogInformation("Done setting up queues"); - - var enqueueTask = Parallel.ForEachAsync(Enumerable.Range(1, workItemCount), async (_, _) => - { - var queue = queues[RandomData.GetInt(0, jobCount - 1)]; - await queue.EnqueueAsync(new SampleQueueWorkItem - { - Created = DateTime.UtcNow, - Path = RandomData.GetString() - }); - }); - _logger.LogInformation("Done enqueueing"); - - using var cancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(TestCancellationToken); - await Parallel.ForEachAsync(Enumerable.Range(1, jobCount), TestCancellationToken, async (index, _) => - { - var queue = queues[index - 1]; - var job = new SampleQueueWithRandomErrorsAndAbandonsJob(queue, loggerFactory: Log); - await job.RunUntilEmptyAsync(cancellationTokenSource.Token); - await cancellationTokenSource.CancelAsync(); - }); - _logger.LogInformation("Done running jobs until empty"); - - await enqueueTask; - - var queueStats = new List(); - for (int i = 0; i < queues.Count; i++) - { - var stats = await queues[i].GetQueueStatsAsync(); - _logger.LogInformation("Queue#{Id}: Working: {Working} Completed: {Completed} Abandoned: {Abandoned} Error: {Errors} Deadletter: {Deadletter}", i, stats.Working, stats.Completed, stats.Abandoned, stats.Errors, stats.Deadletter); - queueStats.Add(stats); - } - _logger.LogInformation("Done getting queue stats"); - - Assert.InRange(queueStats.Sum(s => s.Completed), 0, workItemCount); - } - finally - { - foreach (var q in queues) - { - await q.DeleteQueueAsync(); - q.Dispose(); - } - } - } - - public virtual async Task GetQueueEntryLockAsync_WhenLockThrows_AbandonsQueueEntry() - { - // Arrange - using var queue = GetSampleWorkItemQueue(retries: 0, retryDelay: TimeSpan.Zero); - await queue.DeleteQueueAsync(); - - await queue.EnqueueAsync(new SampleQueueWorkItem - { - Created = DateTime.UtcNow, - Path = "somepath" - }); - - var job = new SampleQueueJobWithThrowingLock(queue, loggerFactory: Log); - - // Act - var result = await job.RunAsync(); - - // Assert - Assert.False(result.IsSuccess); - Assert.NotNull(result.Error); - Assert.IsType(result.Error); - - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(0, stats.Queued); - Assert.Equal(1, stats.Enqueued); - Assert.Equal(1, stats.Abandoned); - Assert.Equal(0, stats.Completed); - } -} diff --git a/src/Foundatio.TestHarness/Jobs/JobRuntimeStoreConformanceTests.cs b/src/Foundatio.TestHarness/Jobs/JobRuntimeStoreConformanceTests.cs new file mode 100644 index 000000000..1bf8ce933 --- /dev/null +++ b/src/Foundatio.TestHarness/Jobs/JobRuntimeStoreConformanceTests.cs @@ -0,0 +1,861 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Foundatio.Xunit; +using Microsoft.Extensions.Time.Testing; +using Xunit; + +namespace Foundatio.Tests.Jobs; + +/// +/// Shared conformance suite every implementation must satisfy: state round-trips, +/// optimistic-concurrency transitions, leases/claims, stale recovery (including the renew-during-reclaim race), and +/// scheduled-dispatch claiming. The in-memory reference and any real store (Redis, etc.) run the same assertions so a +/// new backend is validated against the exact behavior the runtime depends on. +/// +/// +/// A drives time so lease-expiry and claim-steal paths are deterministic without real +/// sleeps. returns null when the backing store is unavailable (e.g. Redis not +/// configured), in which case every test skips. +/// +public abstract class JobRuntimeStoreConformanceTests : TestWithLoggingBase +{ + [Fact] + public virtual async Task BrokerAdmission_ReclaimsExpiredHistoryAcrossCleanupBatchesAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time, new JobRuntimeStoreOptions { MaxActiveJobs = 129, MaxDeduplicationRecords = 129 }); + Assert.SkipWhen(store is null, "Job runtime store not configured."); + var token = TestCancellationToken; + for (int i = 0; i < 129; i++) + await store.CreateIfAbsentAsync(new JobState { JobId = $"expired-{i}", Name = "exports", QueueName = "exports", ExecutionOwner = JobExecutionOwner.Broker, HistoryRetention = TimeSpan.FromMinutes(1) }, token); + time.Advance(TimeSpan.FromMinutes(1)); + await store.CreateIfAbsentAsync(new JobState { JobId = "new", Name = "exports", QueueName = "exports", ExecutionOwner = JobExecutionOwner.Broker, HistoryRetention = TimeSpan.FromMinutes(1) }, token); + Assert.Equal(new JobRuntimeStoreStats(1, 0, 1, 0), await store.GetStatsAsync(token)); + Assert.Equal("new", Assert.Single(await store.QueryAsync(new JobQuery { NewestFirst = true }, token)).JobId); + } + + [Fact] + public virtual async Task BrokerMonitoring_PreservesOrderAndCountsAcrossProgressRetryAndRemovalAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + Assert.SkipWhen(store is null, "Job runtime store not configured."); + var token = TestCancellationToken; + const string name = "reports:月次"; + const string queue = "exports:é"; + await store.CreateIfAbsentAsync(new JobState { JobId = "older", Name = name, QueueName = queue, ExecutionOwner = JobExecutionOwner.Broker, HistoryRetention = TimeSpan.FromHours(1) }, token); + time.Advance(TimeSpan.FromSeconds(1)); + await store.CreateIfAbsentAsync(new JobState { JobId = "newer", Name = name, QueueName = queue, ExecutionOwner = JobExecutionOwner.Broker, HistoryRetention = TimeSpan.FromHours(1) }, token); + var claim = Assert.IsType(await store.BeginBrokerAttemptAsync("older", 1, "worker", token)); + time.Advance(TimeSpan.FromMinutes(30)); + Assert.True(await store.ReportJobProgressAsync("older", claim.ClaimToken!, 50, "Halfway", token)); + Assert.True(await store.CompleteJobAsync("older", claim.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Failed }, token)); + claim = Assert.IsType(await store.BeginBrokerAttemptAsync("older", 2, "worker", token)); + Assert.True(await store.CompleteJobAsync("older", claim.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Succeeded }, token)); + + foreach (var query in new[] { new JobQuery(), new JobQuery { Name = name }, new JobQuery { QueueName = queue }, new JobQuery { Name = name, QueueName = queue } }) + { + Assert.Equal(2, await store.CountAsync(query, token)); + Assert.Equal(new[] { "newer", "older" }, (await store.QueryAsync(query with { NewestFirst = true }, token)).Select(job => job.JobId)); + Assert.Equal(1, await store.CountAsync(query with { Status = JobStatus.Completed }, token)); + Assert.Equal(1, await store.CountAsync(query with { Status = JobStatus.Queued }, token)); + Assert.Equal(0, await store.CountAsync(query with { Status = JobStatus.Processing }, token)); + Assert.Equal(0, await store.CountAsync(query with { Status = JobStatus.RetryPending }, token)); + } + + time.Advance(TimeSpan.FromMinutes(31)); + Assert.Equal("older", Assert.Single(await store.QueryAsync(new JobQuery { QueueName = queue, NewestFirst = true }, token)).JobId); + Assert.True(await store.RemoveAsync("older", token)); + Assert.Equal(0, await store.CountAsync(new JobQuery { Name = name }, token)); + Assert.Equal(0, await store.CountAsync(new JobQuery { QueueName = queue, Status = JobStatus.Completed }, token)); + } + + [Fact] + public virtual async Task CleanupAsync_RespectsBatchLimitAndRetentionBoundaryAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time, new JobRuntimeStoreOptions { HistoryRetention = TimeSpan.FromMinutes(1), MaxHistoryJobs = 10 }); + Assert.SkipWhen(store is null, "Job runtime store not configured."); + var token = TestCancellationToken; + for (int i = 0; i < 4; i++) + { + await store.CreateIfAbsentAsync(NewJob(time, $"job-{i}"), token); + var claim = Assert.IsType(await store.ClaimJobAsync($"job-{i}", new JobClaimRequest { NodeId = "node", JobTypes = ["work.v1"] }, token)); + Assert.True(await store.CompleteJobAsync(claim.JobId, claim.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Succeeded }, token)); + } + time.Advance(TimeSpan.FromSeconds(59)); + Assert.Equal(0, await store.CleanupAsync(2, token)); + time.Advance(TimeSpan.FromSeconds(1)); + Assert.Equal(2, await store.CleanupAsync(2, token)); + Assert.Equal(2, await store.CountAsync(new JobQuery { Status = JobStatus.Completed }, token)); + Assert.Equal(2, await store.CleanupAsync(2, token)); + Assert.Equal(0, await store.CountAsync(new JobQuery(), token)); + } + + [Fact] + public virtual async Task BrokerHistoryPressure_ReleasesCapacityAndExpiredIdentityAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time, new JobRuntimeStoreOptions { MaxActiveJobs = 1, MaxHistoryJobs = 1, MaxDeduplicationRecords = 2 }); + Assert.SkipWhen(store is null, "Job runtime store not configured."); + var token = TestCancellationToken; + for (int i = 0; i < 5; i++) + { + var id = $"broker-{i}"; + await store.CreateIfAbsentAsync(new JobState { JobId = id, Name = "exports", QueueName = "exports", ExecutionOwner = JobExecutionOwner.Broker, HistoryRetention = TimeSpan.FromMinutes(1) }, token); + var claim = Assert.IsType(await store.BeginBrokerAttemptAsync(id, 1, "worker", token)); + Assert.True(await store.CompleteJobAsync(id, claim.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Succeeded }, token)); + time.Advance(TimeSpan.FromSeconds(1)); + } + Assert.Equal(new JobRuntimeStoreStats(0, 1, 1, 0), await store.GetStatsAsync(token)); + time.Advance(TimeSpan.FromMinutes(2)); + await store.CreateIfAbsentAsync(new JobState { JobId = "broker-4", Name = "new", QueueName = "exports", ExecutionOwner = JobExecutionOwner.Broker, HistoryRetention = TimeSpan.FromMinutes(1) }, token); + Assert.Equal("new", (await store.GetAsync("broker-4", token))!.Name); + time.Advance(TimeSpan.FromMinutes(2)); + Assert.Equal(new JobRuntimeStoreStats(0, 0, 0, 0), await store.GetStatsAsync(token)); + } + + [Fact] + public virtual async Task BrokerAndRuntimeJobs_ShareQueriesMetadataAndCountersAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + Assert.SkipWhen(store is null, "Job runtime store not configured."); + var token = TestCancellationToken; + await store.CreateIfAbsentAsync(new JobState { JobId = "runtime", Name = "reports", JobType = "work" }, token); + for (int i = 0; i < 28; i++) + { + time.Advance(TimeSpan.FromMilliseconds(1)); + await store.CreateIfAbsentAsync(new JobState + { + JobId = $"broker-{i:D2}", + Name = "reports", + ExecutionOwner = JobExecutionOwner.Broker, + QueueName = "exports", + PayloadType = "Export", + HistoryRetention = TimeSpan.FromDays(1), + Metadata = new Dictionary { ["tenant"] = "acme" } + }, token); + } + Assert.Equal(29, await store.CountAsync(new JobQuery { Name = "reports" }, token)); + Assert.Equal(28, await store.CountAsync(new JobQuery { QueueName = "exports" }, token)); + var query = new JobQuery { QueueName = "exports", NewestFirst = true, Limit = 25 }; + var first = await store.QueryAsync(query, token); + var second = await store.QueryAsync(query with { Skip = 25 }, token); + Assert.Equal(25, first.Count); + Assert.Equal(3, second.Count); + Assert.Equal("broker-27", first[0].JobId); + Assert.Equal("broker-02", second[0].JobId); + Assert.Empty(first.Select(j => j.JobId).Intersect(second.Select(j => j.JobId))); + Assert.All(first, job => Assert.Equal("acme", job.Metadata!["tenant"])); + var claim = Assert.IsType(await store.BeginBrokerAttemptAsync("broker-00", 1, "worker", token)); + Assert.Equal(1, await store.CountAsync(query with { Status = JobStatus.Processing }, token)); + Assert.Equal("broker-00", Assert.Single(await store.QueryAsync(query with { Status = JobStatus.Processing }, token)).JobId); + Assert.Equal(27, await store.CountAsync(query with { Status = JobStatus.Queued }, token)); + await store.IncrementCounterAsync("exports", "processed", 2, token); + time.Advance(TimeSpan.FromHours(1)); + await store.IncrementCounterAsync("exports", "processed", 3, token); + var counters = await store.GetCounterStatsAsync("exports", cancellationToken: token); + Assert.Equal(5, counters.Totals["processed"]); + Assert.Equal(2, counters.Buckets.Count(bucket => bucket.Counters.ContainsKey("processed"))); + Assert.True(await store.RemoveAsync(claim.JobId, token)); + Assert.Equal(0, await store.CountAsync(query with { Status = JobStatus.Processing }, token)); + } + + [Fact] + public virtual async Task BrokerJobs_ShareMonitoringButNeverEnterRuntimeClaimsAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + Assert.SkipWhen(store is null, "Job runtime store not configured."); + var token = TestCancellationToken; + await store.CreateIfAbsentAsync(new JobState + { + JobId = "broker", + Name = "work", + JobType = "work", + QueueName = "exports", + ExecutionOwner = JobExecutionOwner.Broker, + HistoryRetention = TimeSpan.FromHours(1) + }, token); + var request = new JobClaimRequest { NodeId = "runtime", JobTypes = ["work"] }; + Assert.Null(await store.ClaimNextAsync(request, token)); + Assert.Null(await store.ClaimJobAsync("broker", request, token)); + var first = Assert.IsType(await store.BeginBrokerAttemptAsync("broker", 1, "worker", token)); + Assert.Null(await store.BeginBrokerAttemptAsync("broker", 1, "duplicate", token)); + Assert.True(await store.ReportJobProgressAsync("broker", first.ClaimToken!, 25, "Running", token)); + Assert.False(await store.RenewJobLeaseAsync("broker", first.ClaimToken!, TimeSpan.FromMinutes(1), token)); + time.Advance(TimeSpan.FromMinutes(5)); + Assert.Null(await store.ClaimNextAsync(request, token)); + var second = Assert.IsType(await store.BeginBrokerAttemptAsync("broker", 2, "replacement", token)); + Assert.NotEqual(first.ClaimToken, second.ClaimToken); + Assert.False(await store.ReportJobProgressAsync("broker", first.ClaimToken!, 90, cancellationToken: token)); + Assert.False(await store.CompleteJobAsync("broker", first.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Succeeded }, token)); + Assert.True(await store.CompleteJobAsync("broker", second.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Interrupted }, token)); + Assert.Equal(JobStatus.RetryPending, (await store.GetAsync("broker", token))!.Status); + Assert.False(await store.ReportJobProgressAsync("broker", second.ClaimToken!, 90, cancellationToken: token)); + Assert.Null(await store.ClaimNextAsync(request, token)); + Assert.Null(await store.BeginBrokerAttemptAsync("broker", 2, "duplicate", token)); + var third = Assert.IsType(await store.BeginBrokerAttemptAsync("broker", 3, "worker", token)); + Assert.True(await store.CompleteJobAsync("broker", third.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Succeeded }, token)); + Assert.Equal(JobStatus.Completed, Assert.Single(await store.QueryAsync(new JobQuery { QueueName = "exports", NewestFirst = true }, token)).Status); + Assert.Equal(1, await store.CountAsync(new JobQuery { QueueName = "exports", Status = JobStatus.Completed }, token)); + Assert.False(await store.MarkEnqueueUnknownAsync("broker", "Late send timeout", token)); + Assert.Null(await store.BeginBrokerAttemptAsync("broker", 4, "late", token)); + } + + [Fact] + public virtual async Task BrokerCancellationAndRetention_DoNotScheduleOrCancelRuntimeJobsAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + Assert.SkipWhen(store is null, "Job runtime store not configured."); + var token = TestCancellationToken; + await store.CreateIfAbsentAsync(new JobState { JobId = "runtime", Name = "work", JobType = "work" }, token); + await store.CreateIfAbsentAsync(new JobState { JobId = "broker", Name = "work", JobType = "work", QueueName = "exports", ExecutionOwner = JobExecutionOwner.Broker, HistoryRetention = TimeSpan.FromMinutes(1) }, token); + Assert.True(await store.RequestCancellationAsync("broker", token)); + Assert.Equal(JobStatus.Queued, (await store.GetAsync("broker", token))!.Status); + Assert.False(await store.RemoveAsync("runtime", token)); + var claim = Assert.IsType(await store.BeginBrokerAttemptAsync("broker", 1, "worker", token)); + Assert.True(claim.CancellationRequested); + Assert.True(await store.ReportJobProgressAsync("broker", claim.ClaimToken!, 50, cancellationToken: token)); + Assert.True(await store.IsCancellationRequestedAsync("broker", token)); + Assert.True(await store.CompleteJobAsync("broker", claim.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Interrupted }, token)); + Assert.Equal(JobStatus.RetryPending, (await store.GetAsync("broker", token))!.Status); + claim = Assert.IsType(await store.BeginBrokerAttemptAsync("broker", 2, "replacement", token)); + Assert.True(claim.CancellationRequested); + Assert.True(await store.CompleteJobAsync("broker", claim.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Cancelled }, token)); + Assert.False(await store.RequestCancellationAsync("broker", token)); + time.Advance(TimeSpan.FromSeconds(59)); + Assert.True(await store.IsCancellationRequestedAsync("broker", token)); + time.Advance(TimeSpan.FromSeconds(1)); + Assert.False(await store.IsCancellationRequestedAsync("broker", token)); + Assert.False(await store.IsCancellationRequestedAsync("missing", token)); + Assert.False(await store.IsCancellationRequestedAsync("runtime", token)); + await store.CleanupAsync(cancellationToken: token); + Assert.Null(await store.GetAsync("broker", token)); + Assert.Equal(0, await store.CountAsync(new JobQuery { QueueName = "exports" }, token)); + Assert.Equal("runtime", (await store.ClaimNextAsync(new JobClaimRequest { NodeId = "node", JobTypes = ["work"] }, token))!.JobId); + } + + [Fact] + public virtual async Task RetentionPressure_EvictsHistoryAndPreservesIdempotencyAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time, new JobRuntimeStoreOptions { MaxActiveJobs = 1, MaxHistoryJobs = 1, MaxDeduplicationRecords = 10 }); + Assert.SkipWhen(store is null, "Job runtime store not configured."); + var token = TestCancellationToken; + var request = new JobClaimRequest { NodeId = "node", JobTypes = ["work"] }; + for (int i = 0; i < 3; i++) + { + await store.CreateIfAbsentAsync(new JobState { JobId = $"job-{i}", Name = "work", JobType = "work" }, token); + var claim = Assert.IsType(await store.ClaimNextAsync(request, token)); + Assert.True(await store.CompleteJobAsync(claim.JobId, claim.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Succeeded, Message = "Done" }, token)); + time.Advance(TimeSpan.FromSeconds(1)); + } + Assert.Single(await store.QueryAsync(new JobQuery(), token)); + await store.CreateIfAbsentAsync(new JobState { JobId = "job-0", Name = "work", JobType = "work" }, token); + Assert.Null(await store.ClaimNextAsync(request, token)); + Assert.Equal(new JobRuntimeStoreStats(0, 1, 3, 0), await store.GetStatsAsync(token)); + time.Advance(TimeSpan.FromDays(8)); + await store.CleanupAsync(cancellationToken: token); + await store.CreateIfAbsentAsync(new JobState { JobId = "job-0", Name = "work", JobType = "work" }, token); + Assert.NotNull(await store.ClaimNextAsync(request, token)); + } + + [Fact] + public virtual async Task RetryPolicy_PersistsDelayAndHonorsTerminalFailureAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + Assert.SkipWhen(store is null, "Job runtime store not configured."); + var token = TestCancellationToken; + await store.CreateIfAbsentAsync(new JobState + { + JobId = "retry-policy", + Name = "work", + JobType = "work", + MaxAttempts = 5, + RetryPolicy = new JobRetryPolicy { InitialDelay = TimeSpan.FromMinutes(2), MaxDelay = TimeSpan.FromMinutes(3), JitterFactor = 0 } + }, token); + var request = new JobClaimRequest { NodeId = "node", JobTypes = ["work"] }; + var claim = Assert.IsType(await store.ClaimNextAsync(request, token)); + await store.CompleteJobAsync(claim.JobId, claim.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Failed, Error = "Transient" }, token); + Assert.Null(await store.ClaimNextAsync(request, token)); + time.Advance(TimeSpan.FromMinutes(2)); + claim = Assert.IsType(await store.ClaimNextAsync(request, token)); + await store.CompleteJobAsync(claim.JobId, claim.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Failed, Retryable = false, Error = "Permanent" }, token); + Assert.Equal(JobStatus.Failed, (await store.GetAsync(claim.JobId, token))!.Status); + Assert.Null(await store.ClaimNextAsync(request, token)); + } + + [Fact] + public virtual async Task DispatchCapacity_RejectsNewWorkWithoutLosingAcceptedMessagesAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time, new JobRuntimeStoreOptions { MaxScheduledDispatches = 1, MaxPayloadBytes = 32 }); + Assert.SkipWhen(store is null, "Job runtime store not configured."); + var token = TestCancellationToken; + var dispatch = new ScheduledDispatchState { DispatchId = "one", Body = "data"u8.ToArray(), DueUtc = time.GetUtcNow(), Destination = DestinationAddress.ForQueue("work") }; + await store.ScheduleDispatchAsync(dispatch, token); + await Assert.ThrowsAsync(() => store.ScheduleDispatchAsync(dispatch with { DispatchId = "two" }, token)); + var claim = Assert.Single(await store.ClaimDueDispatchesAsync(time.GetUtcNow(), 1, "claim", TimeSpan.FromMinutes(1), token)); + Assert.Equal("one", claim.DispatchId); + Assert.True(await store.CompleteDispatchAsync(claim.DispatchId, "claim", token)); + await store.ScheduleDispatchAsync(dispatch with { DispatchId = "two" }, token); + await Assert.ThrowsAsync(() => store.CreateIfAbsentAsync(new JobState { JobId = "large", Name = "large", Payload = new byte[33] }, token)); + } + + [Fact] + public virtual async Task PerNodeExpiry_RetiresOnlyUnclaimedOccurrencesAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + Assert.SkipWhen(store is null, "Job runtime store not configured."); + var token = TestCancellationToken; + await store.CreateIfAbsentAsync(new JobState { JobId = "retired", Name = "work", JobType = "work", RequiredNodeId = "retired-node", ExpiresUtc = time.GetUtcNow().AddMinutes(1) }, token); + await store.CreateIfAbsentAsync(new JobState { JobId = "normal", Name = "work", JobType = "work" }, token); + time.Advance(TimeSpan.FromDays(2)); + await store.CleanupAsync(cancellationToken: token); + Assert.Equal(JobStatus.Cancelled, (await store.GetAsync("retired", token))!.Status); + Assert.Equal(JobStatus.Queued, (await store.GetAsync("normal", token))!.Status); + } + + [Fact] + public virtual async Task DispatchLease_UsesAcquisitionClockRatherThanDueCutoffAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + Assert.SkipWhen(store is null, "Job runtime store not configured."); + var token = TestCancellationToken; + var cutoff = time.GetUtcNow(); + await store.ScheduleDispatchAsync(new ScheduledDispatchState { DispatchId = "old", Body = "data"u8.ToArray(), DueUtc = cutoff }, token); + time.Advance(TimeSpan.FromMinutes(2)); + var claim = Assert.Single(await store.ClaimDueDispatchesAsync(cutoff, 1, "owner", TimeSpan.FromMinutes(1), token)); + Assert.Equal(time.GetUtcNow().AddMinutes(1), claim.ClaimExpiresUtc); + Assert.True(await store.CompleteDispatchAsync(claim.DispatchId, "owner", token)); + Assert.False(await store.CompleteDispatchAsync(claim.DispatchId, "owner", token)); + } + + [Fact] + public virtual async Task CreateIfAbsentAsync_UnspecifiedTimestamps_UsesStoreClockAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + Assert.SkipWhen(store is null, "Store unavailable"); + var token = TestCancellationToken; + await store.CreateIfAbsentAsync(new JobState { JobId = "clock", Name = "work", JobType = "work.v1" }, token); + var state = await store.GetAsync("clock", token); + Assert.NotNull(state); + Assert.Equal(time.GetUtcNow(), state.CreatedUtc); + Assert.Equal(time.GetUtcNow(), state.LastUpdatedUtc); + Assert.NotNull(await store.ClaimJobAsync("clock", new JobClaimRequest { NodeId = "node", JobTypes = ["work.v1"] }, token)); + } + + [Fact] + public virtual async Task Schedules_PageByNameWithoutLoadingOtherDefinitionsAsync() + { + var store = CreateStore(new FakeTimeProvider()); + Assert.SkipWhen(store is null, "Store unavailable"); + var token = TestContext.Current.CancellationToken; + foreach (var name in new[] { "e", "a", "d", "b", "c" }) + await store.ScheduleAsync(new ScheduledJobDefinition { Name = name, Cron = "0 0 * * *", JobType = "work.v1" }, token); + var first = await store.GetSchedulesAsync(new ScheduleQuery { Limit = 2 }, token); + var second = await store.GetSchedulesAsync(new ScheduleQuery { Limit = 2, AfterName = first[^1].Name }, token); + var third = await store.GetSchedulesAsync(new ScheduleQuery { Limit = 2, AfterName = second[^1].Name }, token); + Assert.Equal(new[] { "a", "b", "c", "d", "e" }, first.Concat(second).Concat(third).Select(d => d.Name)); + } + + [Fact] + public virtual async Task Schedules_ReconciliationPreservesEditsAndRejectsStaleWritersAsync() + { + var store = CreateStore(new FakeTimeProvider()); + Assert.SkipWhen(store is null, "Store unavailable"); + var schedules = Assert.IsAssignableFrom(store); + var token = TestContext.Current.CancellationToken; + var declared = new ScheduledJobDefinition { Name = "nightly", Cron = "0 3 * * *", JobType = "work.v1", ConfigurationVersion = 1 }; + await schedules.ReconcileAsync(declared, token); + var initial = await schedules.GetScheduleAsync("nightly", token); + Assert.NotNull(initial); + Assert.Equal(1, initial.Revision); + await schedules.ScheduleAsync(initial with { Enabled = false, Cron = "0 4 * * *" }, token); + await schedules.ReconcileAsync(declared, token); + var edited = await schedules.GetScheduleAsync("nightly", token); + Assert.NotNull(edited); + Assert.False(edited.Enabled); + Assert.Equal("0 4 * * *", edited.Cron); + await Assert.ThrowsAsync(() => schedules.ScheduleAsync(initial with { Cron = "0 5 * * *" }, token)); + await Assert.ThrowsAsync(() => schedules.ReconcileAsync(declared with { Cron = "0 6 * * *" }, token)); + await schedules.ReconcileAsync(declared with { Cron = "0 6 * * *", ConfigurationVersion = 2 }, token); + await schedules.ReconcileAsync(declared, token); + var latest = await schedules.GetScheduleAsync("nightly", token); + Assert.NotNull(latest); + Assert.Equal("0 6 * * *", latest.Cron); + Assert.True(latest.Enabled); + Assert.Equal(2, latest.ConfigurationVersion); + Assert.Equal(3, latest.Revision); + } + + [Fact] + public virtual async Task ScheduledDispatches_LeasedHeadDoesNotHideEligibleWorkAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + Assert.SkipWhen(store is null, "Store unavailable"); + var token = TestContext.Current.CancellationToken; + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = "first", + Destination = DestinationAddress.ForQueue("work"), + Body = ReadOnlyMemory.Empty, + DueUtc = time.GetUtcNow() + }, token); + Assert.Single(await store.ClaimDueDispatchesAsync(time.GetUtcNow(), 1, "first-claim", TimeSpan.FromMinutes(1), token)); + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = "second", + Destination = DestinationAddress.ForQueue("work"), + Body = ReadOnlyMemory.Empty, + DueUtc = time.GetUtcNow() + }, token); + Assert.Equal("second", Assert.Single(await store.ClaimDueDispatchesAsync(time.GetUtcNow(), 1, "second-claim", TimeSpan.FromMinutes(1), token)).DispatchId); + + time.Advance(TimeSpan.FromMinutes(2)); + await store.CompleteDispatchAsync("first", "first-claim", token); + var reclaimed = await store.ClaimDueDispatchesAsync(time.GetUtcNow(), 10, "fresh-claim", TimeSpan.FromMinutes(1), token); + Assert.Equal(2, reclaimed.Count); + await store.ReleaseDispatchAsync("first", "first-claim", time.GetUtcNow().AddDays(1), token); + await store.CompleteDispatchAsync("first", "first-claim", token); + time.Advance(TimeSpan.FromMinutes(2)); + Assert.Equal(2, (await store.ClaimDueDispatchesAsync(time.GetUtcNow(), 10, "another-claim", TimeSpan.FromMinutes(1), token)).Count); + } + + [Fact] + public virtual async Task CreateOccurrenceAsync_AtomicallyPreventsOverlapAndHonorsNodeAffinityAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + if (store is null) + { + Assert.Skip("Job runtime store not configured."); + return; + } + + var token = TestCancellationToken; + var occurrence = NewJob(time, "occurrence-1") with { JobType = "work.v1", ScheduleName = "periodic", RequiredNodeId = "node-a" }; + var creates = await Task.WhenAll(Enumerable.Range(0, 20).Select(i => store.CreateOccurrenceAsync(occurrence with { JobId = $"occurrence-{i}" }, cancellationToken: token))); + Assert.Single(creates.Where(created => created == JobOccurrenceResult.Created)); + var request = new JobClaimRequest { NodeId = "node-b", JobTypes = new[] { "work.v1" } }; + Assert.Null(await store.ClaimNextAsync(request, token)); + var claimed = await store.ClaimNextAsync(request with { NodeId = "node-a" }, token); + Assert.NotNull(claimed); + Assert.True(await store.CompleteJobAsync(claimed.JobId, claimed.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Succeeded }, token)); + Assert.Equal(JobOccurrenceResult.Created, await store.CreateOccurrenceAsync(occurrence with { JobId = "next-occurrence" }, cancellationToken: token)); + } + + [Fact] + public virtual async Task ClaimNextAsync_FiltersEligibilityAndFencesRepeatedWorkerIdentityAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + if (store is null) + { + Assert.Skip("Job runtime store not configured."); + return; + } + + var token = TestCancellationToken; + await store.CreateIfAbsentAsync(NewJob(time, "other") with { JobType = "other.v1" }, token); + await store.CreateIfAbsentAsync(NewJob(time, "eligible") with { JobType = "work.v1" }, token); + var request = new JobClaimRequest { NodeId = "same-node", JobTypes = new[] { "work.v1" }, Lease = TimeSpan.FromSeconds(10) }; + var first = await store.ClaimNextAsync(request, token); + Assert.NotNull(first); + Assert.Equal("eligible", first.JobId); + Assert.NotEmpty(first.ClaimToken!); + Assert.Equal(1, first.Attempt); + Assert.Null(await store.ClaimNextAsync(request, token)); + + time.Advance(TimeSpan.FromSeconds(11)); + var second = await store.ClaimNextAsync(request, token); + Assert.NotNull(second); + Assert.NotEqual(first.ClaimToken, second.ClaimToken); + Assert.Equal(2, second.Attempt); + Assert.False(await store.CompleteJobAsync(first.JobId, first.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Succeeded }, token)); + Assert.False(await store.RenewJobLeaseAsync(first.JobId, first.ClaimToken!, request.Lease, token)); + Assert.False(await store.ReportJobProgressAsync(first.JobId, first.ClaimToken!, 99, "stale", token)); + Assert.True(await store.CompleteJobAsync(second.JobId, second.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Succeeded }, token)); + Assert.Equal(JobStatus.Completed, (await store.GetAsync("eligible", token))!.Status); + Assert.Equal(JobStatus.Queued, (await store.GetAsync("other", token))!.Status); + } + + [Fact] + public virtual async Task CompleteJobAsync_FailurePersistsRetryAvailabilityAndBudgetAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + if (store is null) + { + Assert.Skip("Job runtime store not configured."); + return; + } + + var token = TestCancellationToken; + await store.CreateIfAbsentAsync(NewJob(time, "retry") with { JobType = "work.v1", MaxAttempts = 2 }, token); + var request = new JobClaimRequest { NodeId = "worker", JobTypes = new[] { "work.v1" } }; + var first = await store.ClaimNextAsync(request, token); + Assert.NotNull(first); + Assert.True(await store.CompleteJobAsync(first.JobId, first.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Failed, Error = "temporary" }, token)); + Assert.Null(await store.ClaimNextAsync(request, token)); + var pending = await store.GetAsync(first.JobId, token); + Assert.NotNull(pending); + Assert.Equal(JobStatus.Queued, pending.Status); + Assert.NotNull(pending.AvailableUtc); + Assert.Null(pending.CompletedUtc); + time.Advance(pending.AvailableUtc.Value - time.GetUtcNow()); + var second = await store.ClaimNextAsync(request, token); + Assert.NotNull(second); + Assert.True(await store.CompleteJobAsync(second.JobId, second.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Failed, Error = "permanent" }, token)); + Assert.Equal(JobStatus.Failed, (await store.GetAsync(second.JobId, token))!.Status); + Assert.Null(await store.ClaimNextAsync(request, token)); + } + + protected JobRuntimeStoreConformanceTests(ITestOutputHelper output) : base(output) { } + + /// Creates a fresh, isolated store bound to , or null when unavailable. + protected abstract IJobRuntimeStore? CreateStore(TimeProvider timeProvider, JobRuntimeStoreOptions? options = null); + + protected static JobState NewJob(TimeProvider time, string id, string name = "conformance-job", JobStatus status = JobStatus.Queued) + { + var now = time.GetUtcNow(); + return new JobState { JobId = id, Name = name, JobType = "work.v1", Status = status, CreatedUtc = now, LastUpdatedUtc = now }; + } + + [Fact] + public virtual async Task JobLifecycle_RoundTripsAndTransitionsAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + if (store is null) + { + Assert.Skip("Job runtime store not configured."); + return; + } + + var ct = TestCancellationToken; + var created = time.GetUtcNow(); + + // Create with a rich, fully-populated state and assert every field survives the round-trip. + var job = NewJob(time, "job-1", "emailer") with + { + JobType = "Acme.EmailJob", + Payload = new byte[] { 1, 2, 3, 4 }, + PayloadType = "Acme.EmailJobArgs", + Progress = 10, + ProgressMessage = "starting", + Attempt = 1, + ScheduledForUtc = created.AddMinutes(1), + AvailableUtc = created.AddMinutes(1) + }; + await store.CreateIfAbsentAsync(job, ct); + + var got = await store.GetAsync("job-1", ct); + Assert.NotNull(got); + Assert.Equal("emailer", got.Name); + Assert.Equal("Acme.EmailJob", got.JobType); + Assert.NotNull(got.Payload); + Assert.Equal(new byte[] { 1, 2, 3, 4 }, got.Payload.Value.ToArray()); + Assert.Equal("Acme.EmailJobArgs", got.PayloadType); + Assert.Equal(JobStatus.Queued, got.Status); + Assert.Equal(10, got.Progress); + Assert.Equal("starting", got.ProgressMessage); + Assert.Equal(1, got.Attempt); + Assert.Equal(created, got.CreatedUtc); + Assert.Equal(created.AddMinutes(1), got.ScheduledForUtc); + + // Create-if-absent is a no-op once the row exists: a second create must not overwrite. + await store.CreateIfAbsentAsync(job with { Name = "overwritten" }, ct); + Assert.Equal("emailer", (await store.GetAsync("job-1", ct))!.Name); + + var request = new JobClaimRequest { NodeId = "node-a", JobTypes = new[] { "Acme.EmailJob" } }; + Assert.Null(await store.ClaimJobAsync("job-1", request, ct)); + time.Advance(TimeSpan.FromMinutes(1)); + var claimed = await store.ClaimJobAsync("job-1", request, ct); + Assert.NotNull(claimed); + Assert.Equal(JobStatus.Processing, claimed.Status); + Assert.Equal("node-a", claimed.NodeId); + Assert.Equal(time.GetUtcNow().AddMinutes(5), claimed.LeaseExpiresUtc); + Assert.Equal(2, claimed.Attempt); + Assert.Equal(time.GetUtcNow(), claimed.StartedUtc); + Assert.True(await store.ReportJobProgressAsync("job-1", claimed.ClaimToken!, 55, "halfway", ct)); + got = await store.GetAsync("job-1", ct); + Assert.Equal(55, got!.Progress); + Assert.Equal("halfway", got.ProgressMessage); + Assert.False(await store.CompleteJobAsync("job-1", "wrong-claim", new JobCompletion { Kind = JobCompletionKind.Succeeded }, ct)); + Assert.True(await store.CompleteJobAsync("job-1", claimed.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Succeeded }, ct)); + got = await store.GetAsync("job-1", ct); + Assert.Equal(JobStatus.Completed, got!.Status); + Assert.Null(got.NodeId); + Assert.Null(got.ClaimToken); + Assert.Null(got.LeaseExpiresUtc); + Assert.Equal(time.GetUtcNow(), got.CompletedUtc); + Assert.False(await store.ReportJobProgressAsync("job-1", claimed.ClaimToken!, 12, "late", ct)); + Assert.False(await store.RenewJobLeaseAsync("job-1", claimed.ClaimToken!, request.Lease, ct)); + + await store.CreateIfAbsentAsync(NewJob(time, "job-2", "worker"), ct); + Assert.False(await store.IsCancellationRequestedAsync("job-2", ct)); + Assert.True(await store.RequestCancellationAsync("job-2", ct)); + Assert.True(await store.IsCancellationRequestedAsync("job-2", ct)); + + // Operating on a missing job is a benign no-op (returns false / does not throw). + Assert.False(await store.RequestCancellationAsync("missing", ct)); + Assert.Null(await store.GetAsync("missing", ct)); + } + + [Fact] + public virtual async Task Query_FiltersByNameStatusAndLimitAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + if (store is null) + { + Assert.Skip("Job runtime store not configured."); + return; + } + + var ct = TestCancellationToken; + var t = time.GetUtcNow(); + + // Monitoring uses stable ID ordering, independent of execution progress updates. + await store.CreateIfAbsentAsync(NewJob(time, "a", "alpha", JobStatus.Queued) with { LastUpdatedUtc = t }, ct); + await store.CreateIfAbsentAsync(NewJob(time, "b", "alpha", JobStatus.Processing) with { LastUpdatedUtc = t.AddSeconds(1) }, ct); + await store.CreateIfAbsentAsync(NewJob(time, "c", "beta", JobStatus.Queued) with { LastUpdatedUtc = t.AddSeconds(2) }, ct); + + var byName = await store.QueryAsync(new JobQuery { Name = "alpha" }, ct); + Assert.Equal(["a", "b"], byName.Select(j => j.JobId)); + + var byStatus = await store.QueryAsync(new JobQuery { Status = JobStatus.Queued }, ct); + Assert.Equal(new HashSet { "a", "c" }, byStatus.Select(j => j.JobId).ToHashSet()); + + var byBoth = await store.QueryAsync(new JobQuery { Name = "alpha", Status = JobStatus.Queued }, ct); + Assert.Equal("a", Assert.Single(byBoth).JobId); + + var all = await store.QueryAsync(new JobQuery(), ct); + Assert.Equal(new HashSet { "a", "b", "c" }, all.Select(j => j.JobId).ToHashSet()); + + // Continue with the returned cursor and the same filters. + var limited = await store.QueryAsync(new JobQuery { Limit = 1 }, ct); + Assert.Equal("a", Assert.Single(limited).JobId); + var next = await store.QueryAsync(new JobQuery { Limit = 1, AfterJobId = limited.ContinuationToken }, ct); + Assert.Equal("b", Assert.Single(next).JobId); + + + } + + [Fact] + public virtual async Task CleanupAsync_OnlyRemovesExpiredTerminalJobsAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + Assert.SkipWhen(store is null, "Store unavailable"); + var token = TestCancellationToken; + await store.CreateIfAbsentAsync(NewJob(time, "completed"), token); + await store.CreateIfAbsentAsync(NewJob(time, "cancelled"), token); + await store.CreateIfAbsentAsync(NewJob(time, "queued"), token); + var claim = await store.ClaimJobAsync("completed", new JobClaimRequest { NodeId = "node", JobTypes = ["work.v1"] }, token); + Assert.NotNull(claim); + await store.CompleteJobAsync("completed", claim.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Succeeded }, token); + await store.RequestCancellationAsync("cancelled", token); + time.Advance(TimeSpan.FromDays(6)); + Assert.Equal(0, await store.CleanupAsync(cancellationToken: token)); + time.Advance(TimeSpan.FromDays(2)); + Assert.Equal(1, await store.CleanupAsync(1, token)); + Assert.Equal(1, await store.CleanupAsync(1, token)); + Assert.Equal("queued", Assert.Single(await store.QueryAsync(new JobQuery(), token)).JobId); + Assert.NotNull(await store.ClaimJobAsync("queued", new JobClaimRequest { NodeId = "node", JobTypes = ["work.v1"] }, token)); + } + + [Fact] + public virtual async Task Leasing_RenewalPreventsRecoveryAndInterruptionReturnsWorkAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + Assert.SkipWhen(store is null, "Store unavailable"); + var token = TestContext.Current.CancellationToken; + await store.CreateIfAbsentAsync(NewJob(time, "job-1"), token); + var request = new JobClaimRequest { NodeId = "node-a", JobTypes = new[] { "work.v1" }, Lease = TimeSpan.FromMinutes(1) }; + var first = await store.ClaimJobAsync("job-1", request, token); + Assert.NotNull(first); + Assert.Null(await store.ClaimJobAsync("job-1", request, token)); + time.Advance(TimeSpan.FromSeconds(30)); + Assert.True(await store.RenewJobLeaseAsync("job-1", first.ClaimToken!, request.Lease, token)); + time.Advance(TimeSpan.FromSeconds(40)); + Assert.Null(await store.ClaimJobAsync("job-1", request with { NodeId = "node-b" }, token)); + Assert.True(await store.CompleteJobAsync("job-1", first.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Interrupted }, token)); + var second = await store.ClaimJobAsync("job-1", request with { NodeId = "node-b" }, token); + Assert.NotNull(second); + Assert.NotEqual(first.ClaimToken, second.ClaimToken); + Assert.Equal("node-b", second.NodeId); + Assert.Equal(2, second.Attempt); + Assert.False(await store.CompleteJobAsync("job-1", first.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Succeeded }, token)); + } + + [Fact] + public virtual async Task ScheduledDispatches_ClaimCompleteAndRescheduleAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + if (store is null) + { + Assert.Skip("Job runtime store not configured."); + return; + } + + var ct = TestCancellationToken; + var t = time.GetUtcNow(); + + var headers = MessageHeaders.Create(new Dictionary { ["message.type"] = "order.created", ["tenant"] = "acme" }); + var options = new TransportSendOptions { Priority = MessagePriority.High }; + byte[] body = [0x01, 0x02, 0xFF, 0x00, 0x10]; + + var due = new ScheduledDispatchState + { + DispatchId = "d1", + Kind = ScheduledDispatchKind.QueueMessage, + Destination = DestinationAddress.ForQueue("jobs"), + Body = body, + Headers = headers, + Options = options, + DueUtc = t.AddMinutes(-1) + }; + var future = new ScheduledDispatchState + { + DispatchId = "d2", + Kind = ScheduledDispatchKind.QueueMessage, + Destination = DestinationAddress.ForQueue("later"), + Body = body, + DueUtc = t.AddHours(1) + }; + await store.ScheduleDispatchAsync(due, ct); + await store.ScheduleDispatchAsync(future, ct); + // Re-scheduling the same id is a no-op (must not overwrite the dispatch). + await store.ScheduleDispatchAsync(due with { Destination = DestinationAddress.ForQueue("overwritten") }, ct); + + // Only the due dispatch is claimed; the full payload round-trips and the attempt counter increments. + var claimed = await store.ClaimDueDispatchesAsync(t, 100, "node-a", TimeSpan.FromMinutes(5), ct); + var d = Assert.Single(claimed); + Assert.Equal("d1", d.DispatchId); + Assert.Equal(ScheduledDispatchKind.QueueMessage, d.Kind); + Assert.Equal(DestinationAddress.ForQueue("jobs"), d.Destination); + Assert.Equal(body, d.Body.ToArray()); + Assert.Equal("acme", d.Headers["tenant"]); + Assert.Equal("order.created", d.Headers["message.type"]); + Assert.Equal(MessagePriority.High, d.Options.Priority); + Assert.Equal("node-a", d.ClaimOwner); + Assert.Equal(1, d.Attempts); + + // A competing claim sees nothing while the lease is live (and d2 is not yet due). + Assert.Empty(await store.ClaimDueDispatchesAsync(t, 100, "node-b", TimeSpan.FromMinutes(5), ct)); + + // A complete from the wrong owner is ignored: after the lease lapses the dispatch is re-claimable, attempt 2. + await store.CompleteDispatchAsync("d1", "node-b", ct); + time.Advance(TimeSpan.FromMinutes(6)); + var reclaimed = await store.ClaimDueDispatchesAsync(t.AddMinutes(6), 100, "node-a", TimeSpan.FromMinutes(5), ct); + Assert.Equal(2, Assert.Single(reclaimed).Attempts); + + // The owning node completes it for good. + await store.CompleteDispatchAsync("d1", "node-a", ct); + Assert.Empty(await store.ClaimDueDispatchesAsync(t.AddMinutes(12), 100, "node-a", TimeSpan.FromMinutes(5), ct)); + + // Release reschedules a claimed dispatch to its next due time and clears ownership (recurring-occurrence path). + var recurring = new ScheduledDispatchState + { + DispatchId = "d3", + Kind = ScheduledDispatchKind.QueueMessage, + Destination = DestinationAddress.ForQueue("cron"), + Body = body, + DueUtc = t.AddMinutes(20) + }; + await store.ScheduleDispatchAsync(recurring, ct); + Assert.Equal("d3", Assert.Single(await store.ClaimDueDispatchesAsync(t.AddMinutes(21), 100, "node-a", TimeSpan.FromMinutes(5), ct)).DispatchId); + + await store.ReleaseDispatchAsync("d3", "node-b", t.AddMinutes(50), ct); // wrong owner: ignored + await store.ReleaseDispatchAsync("d3", "node-a", t.AddMinutes(50), ct); + Assert.Empty(await store.ClaimDueDispatchesAsync(t.AddMinutes(40), 100, "node-c", TimeSpan.FromMinutes(5), ct)); + var rescheduled = Assert.Single(await store.ClaimDueDispatchesAsync(t.AddMinutes(51), 100, "node-c", TimeSpan.FromMinutes(5), ct)); + Assert.Equal("d3", rescheduled.DispatchId); + Assert.Equal("node-c", rescheduled.ClaimOwner); + } + + [Fact] + public virtual async Task ClaimAsync_ExpiredUnclaimedOccurrence_RetiresBeforeExecutionAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + Assert.SkipWhen(store is null, "Store unavailable"); + var token = TestCancellationToken; + await store.CreateOccurrenceAsync(NewJob(time, "expired") with + { + ScheduleName = "daily", + RequiredNodeId = "node", + ExpiresUtc = time.GetUtcNow().AddMinutes(1) + }, cancellationToken: token); + time.Advance(TimeSpan.FromMinutes(2)); + Assert.Null(await store.ClaimJobAsync("expired", new JobClaimRequest { NodeId = "node", JobTypes = ["work.v1"] }, token)); + Assert.Equal(JobStatus.Cancelled, (await store.GetAsync("expired", token))!.Status); + } + + [Fact] + public virtual Task ScheduleDispatchAsync_HeadersExceedPayloadBudget_RejectsAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time, new JobRuntimeStoreOptions { MaxPayloadBytes = 16 }); + Assert.SkipWhen(store is null, "Store unavailable"); + return Assert.ThrowsAsync(() => store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = "large-headers", + Destination = DestinationAddress.ForQueue("q"), + DueUtc = time.GetUtcNow(), + Body = ReadOnlyMemory.Empty, + Headers = MessageHeaders.Empty.ToBuilder().Set("evidence", new string('x', 32)).Build() + }, TestCancellationToken)); + } + + [Fact] + public virtual async Task Concurrency_OptimisticControlElectsSingleWinnerAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + if (store is null) + { + Assert.Skip("Job runtime store not configured."); + return; + } + + var ct = TestCancellationToken; + const int contenders = 25; + + // Many nodes race to claim the same unclaimed job: exactly one may win, and the store must agree on the owner. + await store.CreateIfAbsentAsync(NewJob(time, "claim-race"), ct); + var claims = await Task.WhenAll(Enumerable.Range(0, contenders) + .Select(i => Task.Run(() => store.ClaimJobAsync("claim-race", new JobClaimRequest { NodeId = $"node-{i}", JobTypes = new[] { "work.v1" } }, ct), ct))); + Assert.Equal(1, claims.Count(claimed => claimed is not null)); + var ownedBy = (await store.GetAsync("claim-race", ct))!.NodeId; + Assert.StartsWith("node-", ownedBy); + + // A single due dispatch contested by many claimers must be handed to exactly one. + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = "dispatch-race", + Destination = DestinationAddress.ForQueue("q"), + Body = new byte[] { 1 }, + DueUtc = time.GetUtcNow().AddMinutes(-1) + }, ct); + var dispatchClaims = await Task.WhenAll(Enumerable.Range(0, contenders) + .Select(i => Task.Run(() => store.ClaimDueDispatchesAsync(time.GetUtcNow(), 100, $"node-{i}", TimeSpan.FromMinutes(5), ct), ct))); + Assert.Equal(1, dispatchClaims.Sum(claimed => claimed.Count(d => d.DispatchId == "dispatch-race"))); + } +} diff --git a/src/Foundatio.TestHarness/Jobs/SampleQueueJob.cs b/src/Foundatio.TestHarness/Jobs/SampleQueueJob.cs deleted file mode 100644 index c5dfde3de..000000000 --- a/src/Foundatio.TestHarness/Jobs/SampleQueueJob.cs +++ /dev/null @@ -1,112 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Exceptionless; -using Foundatio.Jobs; -using Foundatio.Lock; -using Foundatio.Queues; -using Foundatio.Resilience; -using Microsoft.Extensions.Logging; - -namespace Foundatio.Tests.Jobs; - -public class SampleQueueWithRandomErrorsAndAbandonsJob : QueueJobBase -{ - public SampleQueueWithRandomErrorsAndAbandonsJob(IQueue queue, TimeProvider? timeProvider = null, IResiliencePolicyProvider? resiliencePolicyProvider = null, ILoggerFactory? loggerFactory = null) : base(queue, timeProvider, resiliencePolicyProvider, loggerFactory) - { - } - - protected override Task ProcessQueueEntryAsync(QueueEntryContext context) - { - if (RandomData.GetBool(10)) - { - throw new Exception("Boom!"); - } - - if (RandomData.GetBool(10)) - { - return Task.FromResult(JobResult.FailedWithMessage("Abandoned")); - } - - return Task.FromResult(JobResult.Success); - } -} - -public class SampleQueueJob : QueueJobBase -{ - public SampleQueueJob(IQueue queue, TimeProvider? timeProvider = null, IResiliencePolicyProvider? resiliencePolicyProvider = null, ILoggerFactory? loggerFactory = null) : base(queue, timeProvider, resiliencePolicyProvider, loggerFactory) - { - } - - protected override Task ProcessQueueEntryAsync(QueueEntryContext context) - { - return Task.FromResult(JobResult.Success); - } -} - -public class SampleQueueJobWithLocking : QueueJobBase -{ - private readonly ILockProvider _lockProvider; - - public SampleQueueJobWithLocking(IQueue queue, ILockProvider lockProvider, TimeProvider? timeProvider = null, IResiliencePolicyProvider? resiliencePolicyProvider = null, ILoggerFactory? loggerFactory = null) : base(queue, timeProvider, resiliencePolicyProvider, loggerFactory) - { - _lockProvider = lockProvider; - } - - protected override Task GetQueueEntryLockAsync(IQueueEntry queueEntry, CancellationToken cancellationToken = default(CancellationToken)) - { - return _lockProvider.TryAcquireAsync("job", TimeSpan.FromMilliseconds(100), TimeSpan.Zero); - } - - protected override Task ProcessQueueEntryAsync(QueueEntryContext context) - { - return Task.FromResult(JobResult.Success); - } -} - -public class SampleQueueJobWithThrowingLock : QueueJobBase -{ - public SampleQueueJobWithThrowingLock(IQueue queue, TimeProvider? timeProvider = null, IResiliencePolicyProvider? resiliencePolicyProvider = null, ILoggerFactory? loggerFactory = null) : base(queue, timeProvider, resiliencePolicyProvider, loggerFactory) - { - } - - protected override Task GetQueueEntryLockAsync(IQueueEntry queueEntry, CancellationToken cancellationToken = default) - { - throw new InvalidOperationException("Lock provider is unavailable"); - } - - protected override Task ProcessQueueEntryAsync(QueueEntryContext context) - { - return Task.FromResult(JobResult.Success); - } -} - -public class SampleQueueWorkItem -{ - public string? Path { get; set; } - public DateTime Created { get; set; } -} - -public class SampleJob : JobBase -{ - public SampleJob(TimeProvider? timeProvider, ILoggerFactory loggerFactory) : base(timeProvider, null, loggerFactory) - { - } - - protected override Task RunInternalAsync(JobContext context) - { - if (RandomData.GetBool(10)) - { - throw new Exception("Boom!"); - } - - if (RandomData.GetBool(10)) - { - return Task.FromResult(JobResult.FailedWithMessage("Failed")); - } - - return Task.FromResult(JobResult.Success); - } -} - -#pragma warning restore 612, 618 diff --git a/src/Foundatio.TestHarness/Jobs/ThrottledJob.cs b/src/Foundatio.TestHarness/Jobs/ThrottledJob.cs deleted file mode 100644 index bdbfb56af..000000000 --- a/src/Foundatio.TestHarness/Jobs/ThrottledJob.cs +++ /dev/null @@ -1,32 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Caching; -using Foundatio.Jobs; -using Foundatio.Lock; -using Microsoft.Extensions.Logging; - -namespace Foundatio.Tests.Jobs; - -public class ThrottledJob : JobWithLockBase -{ - public ThrottledJob(ICacheClient client, ILoggerFactory? loggerFactory = null) : base(loggerFactory) - { - _locker = new ThrottlingLockProvider(client, 1, TimeSpan.FromMilliseconds(100), null, null, loggerFactory); - } - - private readonly ILockProvider _locker; - public int RunCount { get; set; } - - protected override Task GetLockAsync(CancellationToken cancellationToken = default) - { - return _locker.TryAcquireAsync(nameof(ThrottledJob), acquireTimeout: TimeSpan.Zero); - } - - protected override Task RunInternalAsync(JobContext context) - { - RunCount++; - _logger.LogDebug("Incremented Run Count: {RunCount}", RunCount); - return Task.FromResult(JobResult.Success); - } -} diff --git a/src/Foundatio.TestHarness/Jobs/WithDependencyJob.cs b/src/Foundatio.TestHarness/Jobs/WithDependencyJob.cs deleted file mode 100644 index 054b123a0..000000000 --- a/src/Foundatio.TestHarness/Jobs/WithDependencyJob.cs +++ /dev/null @@ -1,29 +0,0 @@ -using System.Threading.Tasks; -using Foundatio.Jobs; -using Microsoft.Extensions.Logging; - -namespace Foundatio.Tests.Jobs; - -public class WithDependencyJob : JobBase -{ - public WithDependencyJob(MyDependency dependency, ILoggerFactory? loggerFactory = null) : base(null, null, loggerFactory) - { - Dependency = dependency; - } - - public MyDependency Dependency { get; private set; } - - public int RunCount { get; set; } - - protected override Task RunInternalAsync(JobContext context) - { - RunCount++; - - return Task.FromResult(JobResult.Success); - } -} - -public class MyDependency -{ - public int MyProperty { get; set; } -} diff --git a/src/Foundatio.TestHarness/Jobs/WithLockingJob.cs b/src/Foundatio.TestHarness/Jobs/WithLockingJob.cs deleted file mode 100644 index 9bbfd120f..000000000 --- a/src/Foundatio.TestHarness/Jobs/WithLockingJob.cs +++ /dev/null @@ -1,38 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Caching; -using Foundatio.Jobs; -using Foundatio.Lock; -using Foundatio.Messaging; -using Microsoft.Extensions.Logging; -using Xunit; - -namespace Foundatio.Tests.Jobs; - -public class WithLockingJob : JobWithLockBase -{ - private readonly ILockProvider _locker; - - public WithLockingJob(ILoggerFactory loggerFactory) : base(loggerFactory) - { - _locker = new CacheLockProvider(new InMemoryCacheClient(o => o.LoggerFactory(loggerFactory)), new InMemoryMessageBus(o => o.LoggerFactory(loggerFactory)), null, null, loggerFactory); - } - - public int RunCount { get; set; } - - protected override Task GetLockAsync(CancellationToken cancellationToken = default(CancellationToken)) - { - return _locker.TryAcquireAsync(nameof(WithLockingJob), TimeSpan.FromSeconds(1), TimeSpan.Zero); - } - - protected override async Task RunInternalAsync(JobContext context) - { - RunCount++; - - await Task.Delay(150, context.CancellationToken); - Assert.True(await _locker.IsLockedAsync("WithLockingJob")); - - return JobResult.Success; - } -} diff --git a/src/Foundatio.TestHarness/Messaging/MessageBusTestBase.cs b/src/Foundatio.TestHarness/Messaging/MessageBusTestBase.cs deleted file mode 100644 index 523fa5bf3..000000000 --- a/src/Foundatio.TestHarness/Messaging/MessageBusTestBase.cs +++ /dev/null @@ -1,1266 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Diagnostics; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; -using Exceptionless; -using Foundatio.AsyncEx; -using Foundatio.Messaging; -using Foundatio.Tests.Extensions; -using Foundatio.Tests.Serializer; -using Foundatio.Tests.Utility; -using Foundatio.Utility; -using Foundatio.Xunit; -using Microsoft.Extensions.Logging; -using Xunit; - -namespace Foundatio.Tests.Messaging; - -public abstract class MessageBusTestBase : TestWithLoggingBase -{ - protected MessageBusTestBase(ITestOutputHelper output) : base(output) - { - Log.SetLogLevel(LogLevel.Debug); - } - - protected virtual IMessageBus? GetMessageBus(Func? config = null) - { - return null; - } - - protected virtual Task CleanupMessageBusAsync(IMessageBus messageBus) - { - messageBus?.Dispose(); - return Task.CompletedTask; - } - - public virtual async Task CanUseMessageOptionsAsync() - { - using var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - using var metrics = new InMemoryMetrics(FoundatioDiagnostics.Meter.Name, _logger); - - try - { - using var listener = new ActivityListener - { - ShouldListenTo = s => s.Name == FoundatioDiagnostics.ActivitySource.Name, - Sample = (ref ActivityCreationOptions _) => ActivitySamplingResult.AllDataAndRecorded, - ActivityStarted = activity => _logger.LogInformation("Start: {ActivityDisplayName}", activity.DisplayName), - ActivityStopped = activity => _logger.LogInformation("Stop: {ActivityDisplayName}", activity.DisplayName), - }; - - ActivitySource.AddActivityListener(listener); - - using var activity = FoundatioDiagnostics.ActivitySource.StartActivity("Parent"); - Assert.NotNull(activity); - Assert.NotNull(Activity.Current); - Assert.Equal(Activity.Current, activity); - - var countdown = new AsyncCountdownEvent(1); - await messageBus.SubscribeAsync>(msg => - { - _logger.LogTrace("Got message"); - - Assert.Equal("Hello", msg.Body.Data); - Assert.True(msg.Body.Items.ContainsKey("Test")); - - Assert.Equal(activity.Id, msg.CorrelationId); - Assert.Equal(Activity.Current.ParentId, activity.Id); - Assert.Single(msg.Properties); - Assert.Contains(msg.Properties, i => i.Key == "hey" && i.Value.ToString() == "now"); - countdown.Signal(); - _logger.LogTrace("Set event"); - }); - - await Task.Delay(1000); - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "Hello", - Items = { { "Test", "Test" } } - }, new MessageOptions - { - Properties = new Dictionary - { - { "hey", "now" } - } - }, TestCancellationToken); - _logger.LogTrace("Published one..."); - - await countdown.WaitAsync(TimeSpan.FromSeconds(5)); - Assert.Equal(0, countdown.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task CanSendMessageAsync() - { - using var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - try - { - var countdown = new AsyncCountdownEvent(1); - await messageBus.SubscribeAsync(msg => - { - _logger.LogTrace("Got message"); - Assert.Equal("Hello", msg.Data); - Assert.True(msg.Items.ContainsKey("Test")); - countdown.Signal(); - _logger.LogTrace("Set event"); - }, TestCancellationToken); - - await Task.Delay(100, TestCancellationToken); - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "Hello", - Items = { { "Test", "Test" } } - }, cancellationToken: TestCancellationToken); - _logger.LogTrace("Published one..."); - - await countdown.WaitAsync(TimeSpan.FromSeconds(5)); - Assert.Equal(0, countdown.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task CanHandleNullMessageAsync() - { - using var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - try - { - // Publishing null should throw ArgumentNullException - await Assert.ThrowsAsync(async () => await messageBus.PublishAsync(null!)); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task CanSendDerivedMessageAsync() - { - using var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - try - { - var countdown = new AsyncCountdownEvent(1); - await messageBus.SubscribeAsync(msg => - { - _logger.LogTrace("Got message"); - Assert.Equal("Hello", msg.Data); - countdown.Signal(); - _logger.LogTrace("Set event"); - }, TestCancellationToken); - - await Task.Delay(100, TestCancellationToken); - await messageBus.PublishAsync(new DerivedSimpleMessageA - { - Data = "Hello" - }); - _logger.LogTrace("Published one..."); - await countdown.WaitAsync(TimeSpan.FromSeconds(5)); - Assert.Equal(0, countdown.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task CanSendMappedMessageAsync() - { - using var messageBus = GetMessageBus(b => - { - b.MessageTypeMappings.Add(nameof(SimpleMessageA), typeof(SimpleMessageA)); - return b; - }); - if (messageBus == null) - return; - - try - { - var countdown = new AsyncCountdownEvent(1); - await messageBus.SubscribeAsync(msg => - { - _logger.LogTrace("Got message"); - Assert.Equal("Hello", msg.Data); - countdown.Signal(); - _logger.LogTrace("Set event"); - }, TestCancellationToken); - - await Task.Delay(100, TestCancellationToken); - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - _logger.LogTrace("Published one..."); - await countdown.WaitAsync(TimeSpan.FromSeconds(5)); - Assert.Equal(0, countdown.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task CanSendDelayedMessageAsync() - { - const int numConcurrentMessages = 1000; - using var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - try - { - // Arrange - var countdown = new AsyncCountdownEvent(numConcurrentMessages); - int messages = 0; - int optionsVerifiedCount = 0; - - await messageBus.SubscribeAsync>(msg => - { - Assert.Equal("Hello", msg.Body.Data); - - // Verify options are preserved through delayed delivery - if (!String.IsNullOrEmpty(msg.CorrelationId) && msg.CorrelationId.StartsWith("correlation-")) - { - Assert.True(msg.Properties.TryGetValue("TestKey", out var value)); - Assert.Equal("TestValue", value); - Interlocked.Increment(ref optionsVerifiedCount); - } - - if (Interlocked.Increment(ref messages) % 50 == 0) - _logger.LogTrace("Total Processed {Messages} messages", messages); - - countdown.Signal(); - }); - - // Act - var sw = Stopwatch.StartNew(); - await Parallel.ForEachAsync(Enumerable.Range(1, numConcurrentMessages), async (i, _) => - { - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "Hello", - Count = i - }, new MessageOptions - { - DeliveryDelay = TimeSpan.FromMilliseconds(RandomData.GetInt(0, 100)), - CorrelationId = $"correlation-{i}", - Properties = new Dictionary { { "TestKey", "TestValue" } } - }, TestCancellationToken); - - if (i % 500 == 0) - _logger.LogTrace("Published 500 messages..."); - }); - - await countdown.WaitAsync(TimeSpan.FromSeconds(30)); - sw.Stop(); - - // Assert - _logger.LogTrace("Processed {Processed} in {Duration:g}", numConcurrentMessages - countdown.CurrentCount, sw.Elapsed); - Assert.Equal(0, countdown.CurrentCount); - Assert.InRange(sw.Elapsed.TotalMilliseconds, 50, 30000); - Assert.Equal(numConcurrentMessages, optionsVerifiedCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task CanSubscribeConcurrentlyAsync() - { - const int iterations = 100; - using var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - try - { - var countdown = new AsyncCountdownEvent(iterations * 10); - await Parallel.ForEachAsync(Enumerable.Range(1, 10), async (_, ct) => - { - await messageBus.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - countdown.Signal(); - }, cancellationToken: ct); - }); - - await Parallel.ForEachAsync(Enumerable.Range(1, iterations), async (_, _) => await messageBus.PublishAsync(new SimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken)); - await countdown.WaitAsync(TimeSpan.FromSeconds(5)); - Assert.Equal(0, countdown.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task CanReceiveMessagesConcurrentlyAsync() - { - const int iterations = 100; - var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - var messageBuses = new List(10); - try - { - var countdown = new AsyncCountdownEvent(iterations * 10); - await Parallel.ForEachAsync(Enumerable.Range(1, 10), async (_, ct) => - { - var bus = GetMessageBus(); - Assert.NotNull(bus); - await bus.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - countdown.Signal(); - }, cancellationToken: ct); - - messageBuses.Add(bus); - }); - - var subscribe = Parallel.ForEachAsync(Enumerable.Range(1, iterations), async (i, ct) => - { - await Task.Delay(RandomData.GetInt(0, 10), ct); - var randomBus = messageBuses.Random(); - Assert.NotNull(randomBus); - await randomBus.SubscribeAsync(msg => Task.CompletedTask, cancellationToken: ct); - }); - - var publish = Parallel.ForEachAsync(Enumerable.Range(1, iterations + 3), async (i, _) => - { - await (i switch - { - 1 => messageBus.PublishAsync(new DerivedSimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken), - 2 => messageBus.PublishAsync(new Derived2SimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken), - 3 => messageBus.PublishAsync(new Derived3SimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken), - 4 => messageBus.PublishAsync(new Derived4SimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken), - 5 => messageBus.PublishAsync(new Derived5SimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken), - 6 => messageBus.PublishAsync(new Derived6SimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken), - 7 => messageBus.PublishAsync(new Derived7SimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken), - 8 => messageBus.PublishAsync(new Derived8SimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken), - 9 => messageBus.PublishAsync(new Derived9SimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken), - 10 => messageBus.PublishAsync(new Derived10SimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken), - iterations + 1 => messageBus.PublishAsync(new { Data = "Hello" }, cancellationToken: TestCancellationToken), - iterations + 2 => messageBus.PublishAsync(new SimpleMessageC { Data = "Hello" }, cancellationToken: TestCancellationToken), - iterations + 3 => messageBus.PublishAsync(new SimpleMessageB { Data = "Hello" }, cancellationToken: TestCancellationToken), - _ => messageBus.PublishAsync(new SimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken) - }); - }); - - await Task.WhenAll(subscribe, publish); - await countdown.WaitAsync(TimeSpan.FromSeconds(4)); - Assert.Equal(0, countdown.CurrentCount); - } - finally - { - foreach (var mb in messageBuses) - await CleanupMessageBusAsync(mb); - - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task CanSendMessageToMultipleSubscribersAsync() - { - using var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - try - { - var countdown = new AsyncCountdownEvent(3); - await messageBus.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - countdown.Signal(); - }, TestCancellationToken); - await messageBus.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - countdown.Signal(); - }, TestCancellationToken); - await messageBus.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - countdown.Signal(); - }, TestCancellationToken); - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - - await countdown.WaitAsync(TimeSpan.FromSeconds(2)); - Assert.Equal(0, countdown.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task CanTolerateSubscriberFailureAsync() - { - using var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - try - { - var countdown = new AsyncCountdownEvent(4); - await messageBus.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - countdown.Signal(); - }, TestCancellationToken); - await messageBus.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - countdown.Signal(); - }, TestCancellationToken); - await messageBus.SubscribeAsync(msg => throw new Exception()); - await messageBus.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - countdown.Signal(); - }, TestCancellationToken); - await messageBus.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - countdown.Signal(); - }, TestCancellationToken); - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - - await countdown.WaitAsync(TimeSpan.FromSeconds(2)); - Assert.Equal(0, countdown.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task WillOnlyReceiveSubscribedMessageTypeAsync() - { - using var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - try - { - var countdown = new AsyncCountdownEvent(1); - await messageBus.SubscribeAsync(msg => - { - Assert.Fail("Received wrong message type"); - }, TestCancellationToken); - await messageBus.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - countdown.Signal(); - }, TestCancellationToken); - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - - await countdown.WaitAsync(TimeSpan.FromSeconds(2)); - Assert.Equal(0, countdown.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task WillReceiveDerivedMessageTypesAsync() - { - using var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - try - { - var countdown = new AsyncCountdownEvent(2); - await messageBus.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - countdown.Signal(); - }); - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - await messageBus.PublishAsync(new SimpleMessageB - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - await messageBus.PublishAsync(new SimpleMessageC - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - - await countdown.WaitAsync(TimeSpan.FromSeconds(5)); - Assert.Equal(0, countdown.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task CanSubscribeToRawMessagesAsync() - { - using var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - try - { - var countdown = new AsyncCountdownEvent(3); - await messageBus.SubscribeAsync(msg => - { - Assert.NotNull(msg.Type); - Assert.True(msg.Type.Contains(nameof(SimpleMessageA)) - || msg.Type.Contains(nameof(SimpleMessageB)) - || msg.Type.Contains(nameof(SimpleMessageC))); - countdown.Signal(); - }); - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - await messageBus.PublishAsync(new SimpleMessageB - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - await messageBus.PublishAsync(new SimpleMessageC - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - - await countdown.WaitAsync(TimeSpan.FromSeconds(5)); - Assert.Equal(0, countdown.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task CanSubscribeToAllMessageTypesAsync() - { - using var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - try - { - var countdown = new AsyncCountdownEvent(3); - await messageBus.SubscribeAsync(msg => - { - countdown.Signal(); - }, TestCancellationToken); - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - await messageBus.PublishAsync(new SimpleMessageB - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - await messageBus.PublishAsync(new SimpleMessageC - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - - await countdown.WaitAsync(TimeSpan.FromSeconds(2)); - Assert.Equal(0, countdown.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task WontKeepMessagesWithNoSubscribersAsync() - { - using var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - try - { - var countdown = new AsyncCountdownEvent(1); - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - - await Task.Delay(100, TestCancellationToken); - await messageBus.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - countdown.Signal(); - }, TestCancellationToken); - - await Assert.ThrowsAsync(async () => await countdown.WaitAsync(TimeSpan.FromMilliseconds(100))); - Assert.Equal(1, countdown.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task CanCancelSubscriptionAsync() - { - using var messageBus = GetMessageBus(); - if (messageBus == null) - return; - - try - { - var countdown = new AsyncCountdownEvent(2); - - long messageCount = 0; - using var cancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(TestCancellationToken); - await messageBus.SubscribeAsync(async msg => - { - _logger.LogTrace("SimpleAMessage received"); - Interlocked.Increment(ref messageCount); - await cancellationTokenSource.CancelAsync(); - countdown.Signal(); - }, cancellationTokenSource.Token); - - // NOTE: This subscriber will not be canceled. - await messageBus.SubscribeAsync(_ => countdown.Signal(), TestCancellationToken); - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - - await countdown.WaitAsync(TimeSpan.FromSeconds(2)); - Assert.Equal(0, countdown.CurrentCount); - Assert.Equal(1, messageCount); - - countdown.AddCount(1); - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - - await countdown.WaitAsync(TimeSpan.FromSeconds(2)); - Assert.Equal(0, countdown.CurrentCount); - Assert.Equal(1, messageCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task CanReceiveFromMultipleSubscribersAsync() - { - using var messageBus1 = GetMessageBus(); - if (messageBus1 == null) - return; - - try - { - var countdown1 = new AsyncCountdownEvent(1); - await messageBus1.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - countdown1.Signal(); - }, TestCancellationToken); - - using var messageBus2 = GetMessageBus(); - Assert.NotNull(messageBus2); - - try - { - var countdown2 = new AsyncCountdownEvent(1); - await messageBus2.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - countdown2.Signal(); - }, TestCancellationToken); - - await messageBus1.PublishAsync(new SimpleMessageA - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - - await countdown1.WaitAsync(TimeSpan.FromSeconds(5)); - Assert.Equal(0, countdown1.CurrentCount); - await countdown2.WaitAsync(TimeSpan.FromSeconds(5)); - Assert.Equal(0, countdown2.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus2); - } - } - finally - { - await CleanupMessageBusAsync(messageBus1); - } - } - - public virtual async Task CanDisposeWithNoSubscribersOrPublishersAsync() - { - using var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - using (messageBus) - { - // Empty using statement to ensure Dispose is called - } - - await CleanupMessageBusAsync(messageBus); - } - - public virtual async Task CanHandlePoisonedMessageAsync() - { - // Arrange - using var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - long handlerInvocations = 0; - - try - { - await messageBus.SubscribeAsync(_ => - { - _logger.LogTrace("SimpleAMessage received"); - Interlocked.Increment(ref handlerInvocations); - throw new Exception("Poisoned message"); - }); - - // Act - await messageBus.PublishAsync(new SimpleMessageA(), cancellationToken: TestCancellationToken); - _logger.LogTrace("Published one..."); - - await Task.Delay(TimeSpan.FromSeconds(3)); - - // Assert - Assert.InRange(handlerInvocations, 1, 5); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task DisposeAsync_CalledMultipleTimes_IsIdempotentAsync() - { - // Arrange - var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - try - { - // Act - await messageBus.DisposeAsync(); - await messageBus.DisposeAsync(); - await messageBus.DisposeAsync(); - - // Assert - no exception thrown - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task DisposeAsync_WhilePublishing_CompletesWithoutDeadlockAsync() - { - // Arrange - var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - try - { - var subscriberStarted = new AsyncAutoResetEvent(false); - - await messageBus.SubscribeAsync(async msg => - { - subscriberStarted.Set(); - await Task.Delay(500, TestCancellationToken); - }); - - // Act - _ = messageBus.PublishAsync(new SimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken); - await subscriberStarted.WaitAsync(TestCancellationToken); - - await messageBus.DisposeAsync(); - - // Assert - no deadlock or exception thrown - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task DisposeAsync_WithNoSubscribersOrPublishers_CompletesWithoutExceptionAsync() - { - // Arrange - var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - try - { - // Act - await messageBus.DisposeAsync(); - - // Assert - no exception thrown - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task PublishAsync_AfterDispose_ThrowsMessageBusExceptionAsync() - { - // Arrange - var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - try - { - await messageBus.DisposeAsync(); - - // Act & Assert - await Assert.ThrowsAsync(async () => - await messageBus.PublishAsync(new SimpleMessageA { Data = "Hello" })); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - /// - /// Verifies that cancellation is surfaced as OperationCanceledException, not swallowed - /// or wrapped in MessageBusException. This ensures callers can distinguish between - /// cancellation and actual publish failures. - /// - public virtual async Task PublishAsync_WithCancellation_ThrowsOperationCanceledExceptionAsync() - { - // Arrange - using var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - try - { - // Act & Assert - await Assert.ThrowsAnyAsync(async () => - await messageBus.PublishAsync(new SimpleMessageA(), cancellationToken: new CancellationToken(true))); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task PublishAsync_WithDelayedMessageAndDisposeBeforeDelivery_DiscardsMessageAsync() - { - // Arrange - var messageReceived = new AsyncAutoResetEvent(false); - var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - try - { - await messageBus.SubscribeAsync(msg => - { - _logger.LogTrace("Got message - this should NOT happen"); - messageReceived.Set(); - }, TestCancellationToken); - - // Act - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "ShouldBeDiscarded" - }, new MessageOptions { DeliveryDelay = TimeSpan.FromSeconds(1) }, TestCancellationToken); - - _logger.LogTrace("Published delayed message, disposing immediately..."); - messageBus.Dispose(); - messageBus = null; - - // Assert - using var cts = CancellationTokenSource.CreateLinkedTokenSource(TestCancellationToken); - cts.CancelAfter(TimeSpan.FromMilliseconds(250)); - await Assert.ThrowsAnyAsync(() => messageReceived.WaitAsync(cts.Token)); - } - finally - { - if (messageBus is not null) - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task PublishAsync_WithSerializationFailure_ThrowsSerializerExceptionAsync() - { - // Arrange - var faultSerializer = new FaultInjectingSerializer { ShouldFailOnSerialize = true }; - using var messageBus = GetMessageBus(o => { o.Serializer = faultSerializer; return o; }); - if (messageBus is null) - return; - - try - { - await messageBus.SubscribeAsync(_ => { }); - - // Act & Assert - await Assert.ThrowsAsync(async () => - await messageBus.PublishAsync(new SimpleMessageA { Data = "test" }, cancellationToken: TestCancellationToken)); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task SubscribeAsync_AfterDispose_ThrowsMessageBusExceptionAsync() - { - // Arrange - var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - try - { - await messageBus.DisposeAsync(); - - // Act & Assert - await Assert.ThrowsAsync(async () => - await messageBus.SubscribeAsync(_ => { })); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - /// - /// Cancelling a subscription token should only remove that subscriber, not tear down - /// the underlying transport (connections, channels, polling loops). Other active - /// subscribers must continue to receive messages. - /// - public virtual async Task SubscribeAsync_CancelledToken_DoesNotTearDownInfrastructureAsync() - { - // Arrange - var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - try - { - using var cts = CancellationTokenSource.CreateLinkedTokenSource(TestCancellationToken); - var cancelledHandlerCount = new AsyncCountdownEvent(1); - - await messageBus.SubscribeAsync(msg => - { - cancelledHandlerCount.Signal(); - }, cts.Token); - - await cts.CancelAsync(); - - var activeHandlerReceived = new AsyncCountdownEvent(1); - await messageBus.SubscribeAsync(msg => - { - Assert.Equal("Hello", msg.Data); - activeHandlerReceived.Signal(); - }, TestCancellationToken); - - // Act - await messageBus.PublishAsync(new SimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken); - await activeHandlerReceived.WaitAsync(TimeSpan.FromSeconds(5)); - - // Assert - Assert.Equal(0, activeHandlerReceived.CurrentCount); - Assert.Equal(1, cancelledHandlerCount.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - /// - /// Verifies that cancellation is surfaced as OperationCanceledException, not swallowed - /// or wrapped in MessageBusException. This ensures callers can distinguish between - /// cancellation and actual subscribe failures. - /// - public virtual async Task SubscribeAsync_WithCancellation_ThrowsOperationCanceledExceptionAsync() - { - // Arrange - using var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - try - { - // Act & Assert - await Assert.ThrowsAnyAsync(async () => - await messageBus.SubscribeAsync(_ => { }, cancellationToken: new CancellationToken(true))); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task SubscribeAsync_WithDeserializationFailure_SkipsMessageAsync() - { - // Arrange - var faultSerializer = new FaultInjectingSerializer { ShouldFailOnDeserialize = true }; - using var messageBus = GetMessageBus(o => { o.Serializer = faultSerializer; return o; }); - if (messageBus is null) - return; - - long handlerInvocations = 0; - - try - { - await messageBus.SubscribeAsync(_ => - { - Interlocked.Increment(ref handlerInvocations); - }); - - // Act - await messageBus.PublishAsync(new SimpleMessageA { Data = "poison" }, cancellationToken: TestCancellationToken); - await Task.Delay(TimeSpan.FromSeconds(2), TestCancellationToken); - - // Assert - Assert.Equal(0, handlerInvocations); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task SubscribeAsync_WithValidThenPoisonedMessage_DeliversOnlyValidMessageAsync() - { - // Arrange - var faultSerializer = new FaultInjectingSerializer(); - using var messageBus = GetMessageBus(o => { o.Serializer = faultSerializer; return o; }); - if (messageBus is null) - return; - - long handlerInvocations = 0; - var messageReceived = new AsyncAutoResetEvent(false); - - try - { - await messageBus.SubscribeAsync(_ => - { - _logger.LogTrace("SimpleAMessage received"); - Interlocked.Increment(ref handlerInvocations); - messageReceived.Set(); - }); - - await messageBus.PublishAsync(new SimpleMessageA { Data = "valid" }, cancellationToken: TestCancellationToken); - await messageReceived.WaitAsync(TestCancellationToken); - Assert.Equal(1, handlerInvocations); - - faultSerializer.ShouldFailOnDeserialize = true; - - // Act - await messageBus.PublishAsync(new SimpleMessageA { Data = "poison" }, cancellationToken: TestCancellationToken); - await Task.Delay(TimeSpan.FromSeconds(2), TestCancellationToken); - - // Assert - Assert.Equal(1, handlerInvocations); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task PublishAsync_WithDeliveryDelayExtension_DelaysDeliveryAsync() - { - // Arrange - using var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - try - { - var countdown = new AsyncCountdownEvent(1); - await messageBus.SubscribeAsync(msg => - { - Assert.Equal("Delayed", msg.Data); - countdown.Signal(); - }, TestCancellationToken).AnyContext(); - - // Allow subscription to propagate in distributed providers - await Task.Delay(250, TestCancellationToken).AnyContext(); - - // Act - await messageBus.PublishAsync(new SimpleMessageA { Data = "Delayed" }, TimeSpan.FromSeconds(1), TestCancellationToken).AnyContext(); - - // Assert - message should NOT be received immediately - await Assert.ThrowsAsync(async () => - await countdown.WaitAsync(TimeSpan.FromMilliseconds(250))).AnyContext(); - Assert.Equal(1, countdown.CurrentCount); - - // Assert - message SHOULD arrive after the delay - await countdown.WaitAsync(TimeSpan.FromSeconds(10)).AnyContext(); - Assert.Equal(0, countdown.CurrentCount); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task PublishAsync_WithUniqueId_PropagatesUniqueIdToSubscriberAsync() - { - // Arrange - using var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - try - { - var countdown = new AsyncCountdownEvent(1); - string? receivedUniqueId = null; - - await messageBus.SubscribeAsync>(msg => - { - receivedUniqueId = msg.UniqueId; - countdown.Signal(); - }, TestCancellationToken).AnyContext(); - - // Act - await messageBus.PublishAsync(new SimpleMessageA { Data = "Hello" }, new MessageOptions - { - UniqueId = "test-unique-123" - }, TestCancellationToken).AnyContext(); - - // Assert - await countdown.WaitAsync(TimeSpan.FromSeconds(5)).AnyContext(); - Assert.Equal(0, countdown.CurrentCount); - Assert.Equal("test-unique-123", receivedUniqueId); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task SubscribeAsync_ToRawIMessage_CanAccessAllPropertiesAsync() - { - // Arrange - using var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - try - { - var countdown = new AsyncCountdownEvent(1); - string? receivedCorrelationId = null; - string? receivedUniqueId = null; - IDictionary? receivedProperties = null; - - await messageBus.SubscribeAsync(msg => - { - receivedCorrelationId = msg.CorrelationId; - receivedUniqueId = msg.UniqueId; - receivedProperties = msg.Properties; - countdown.Signal(); - }, TestCancellationToken).AnyContext(); - - // Act - await messageBus.PublishAsync(new SimpleMessageA { Data = "Hello" }, new MessageOptions - { - CorrelationId = "corr-456", - UniqueId = "unique-789", - Properties = new Dictionary { { "env", "test" }, { "version", "1.0" } } - }, TestCancellationToken).AnyContext(); - - // Assert - await countdown.WaitAsync(TimeSpan.FromSeconds(5)).AnyContext(); - Assert.Equal(0, countdown.CurrentCount); - Assert.Equal("corr-456", receivedCorrelationId); - Assert.Equal("unique-789", receivedUniqueId); - Assert.NotNull(receivedProperties); - Assert.Equal("test", receivedProperties["env"]); - Assert.Equal("1.0", receivedProperties["version"]); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - - public virtual async Task SubscribeAsync_WithCancellationTokenHandler_ReceivesCancellationTokenAsync() - { - // Arrange - using var messageBus = GetMessageBus(); - if (messageBus is null) - return; - - try - { - var countdown = new AsyncCountdownEvent(1); - bool tokenWasCancelled = true; - - await messageBus.SubscribeAsync((msg, ct) => - { - tokenWasCancelled = ct.IsCancellationRequested; - countdown.Signal(); - return Task.CompletedTask; - }, TestCancellationToken).AnyContext(); - - // Act - await messageBus.PublishAsync(new SimpleMessageA { Data = "Hello" }, cancellationToken: TestCancellationToken).AnyContext(); - - // Assert - await countdown.WaitAsync(TimeSpan.FromSeconds(5)).AnyContext(); - Assert.Equal(0, countdown.CurrentCount); - Assert.False(tokenWasCancelled); - } - finally - { - await CleanupMessageBusAsync(messageBus); - } - } - -} diff --git a/src/Foundatio.TestHarness/Messaging/MessageTransportConformanceTests.cs b/src/Foundatio.TestHarness/Messaging/MessageTransportConformanceTests.cs new file mode 100644 index 000000000..86ac7e26d --- /dev/null +++ b/src/Foundatio.TestHarness/Messaging/MessageTransportConformanceTests.cs @@ -0,0 +1,856 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Foundatio.Xunit; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public abstract class MessageTransportConformanceTests : TestWithLoggingBase +{ + protected MessageTransportConformanceTests(ITestOutputHelper output) : base(output) { } + + protected virtual IMessageTransport? CreateTransport() + { + return null; + } + + protected virtual ValueTask CleanupTransportAsync(IMessageTransport transport) + { + return transport.DisposeAsync(); + } + + [Fact] + public virtual async Task TemporarySubscription_ExpiresWithoutListenerDisposalAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsEphemeralSubscriptions temporary) + { + Assert.Skip("Transport does not support expiring subscriptions."); + return; + } + try + { + var token = TestCancellationToken; + var source = DestinationAddress.ForSubscription("temporary-events", "temporary-listener"); + await temporary.EnsureAsync([new DestinationDeclaration { Address = source, AutoDeleteAfter = TimeSpan.FromMilliseconds(100) }], token); + Assert.True(await temporary.ExistsAsync(source, token)); + await Task.Delay(200, token); + Assert.False(await temporary.RenewSubscriptionAsync(source, TimeSpan.FromMinutes(1), token)); + Assert.False(await temporary.ExistsAsync(source, token)); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task CanSendAndReceiveBatchAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull) + { + Assert.Skip("Transport does not support pull receive (ISupportsPull)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("orders"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + + var result = await transport.SendAsync(queue, [ + CreateMessage("one", ("tenant", "acme")), + CreateMessage("two", ("tenant", "acme")) + ], new TransportSendOptions(), TestCancellationToken); + + // Send is throw-on-failure, so reaching here means both messages were accepted; assert the accepted ids. + Assert.Equal(2, result.Items.Count); + + var entries = await pull.ReceiveAsync(queue, new ReceiveRequest + { + MaxMessages = 2, + MaxWaitTime = TimeSpan.FromSeconds(1) + }, TestCancellationToken); + + Assert.Equal(2, entries.Count); + var bodies = entries.Select(ReadBody).ToList(); + Assert.Contains("one", bodies); + Assert.Contains("two", bodies); + + // Only assert positional FIFO order when the transport actually guarantees ordering; a best-effort + // (OrderingGuarantee.None) transport may legitimately deliver out of order. + if (GetCapabilities(transport, queue).Ordering != OrderingGuarantee.None) + { + Assert.Equal("one", ReadBody(entries[0])); + Assert.Equal("two", ReadBody(entries[1])); + } + + Assert.All(entries, e => Assert.Equal("acme", e.Headers["tenant"])); + Assert.Equal(1, entries[0].DeliveryCount); + + await transport.CompleteAsync(entries[0], TestCancellationToken); + await transport.CompleteAsync(entries[1], TestCancellationToken); + + if (transport is ISupportsStats stats) + { + // Assert only the point-in-time gauges every broker can report, and tolerate eventual consistency + // (e.g. SQS ApproximateNumberOf* lag). Lifetime counters such as Completed are not universally + // available across transports, so they are not part of the shared contract. + await AssertQueueDrainedAsync(stats, queue, TestCancellationToken); + } + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task SendAsync_ReturnsOneAcceptedIdPerMessageAsync() + { + var transport = CreateTransport(); + if (transport is null) + { + Assert.Skip("No transport configured."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("send-ids"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + + var result = await transport.SendAsync(queue, [ + CreateMessage("a"), + CreateMessage("b"), + CreateMessage("c") + ], new TransportSendOptions(), TestCancellationToken); + + // One accepted id per message, positionally aligned (see SendResult): every id present and distinct, so + // per-message settlement and tracing can never alias two messages from one batch. + Assert.Equal(3, result.Items.Count); + var ids = result.Items.Select(i => i.MessageId).ToList(); + Assert.All(ids, id => Assert.False(String.IsNullOrEmpty(id))); + Assert.Equal(3, ids.Distinct(StringComparer.Ordinal).Count()); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task TextContentType_RoundTripsBodyAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull) + { + Assert.Skip("Transport does not support pull receive (ISupportsPull)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("text-content"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + + // A text ContentType lets text-native transports (SQS/SNS) store the body directly instead of base64; + // whichever encoding the provider picks, the bytes must round-trip exactly. + byte[] body = Encoding.UTF8.GetBytes("""{"hello":"wörld"}"""); + await transport.SendAsync(queue, [new TransportMessage + { + Body = body, + MessageId = "application-message-id", + ContentType = "application/json" + }], new TransportSendOptions(), TestCancellationToken); + + var entry = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, TestCancellationToken)); + Assert.Equal(body, entry.Body.ToArray()); + Assert.Equal("application-message-id", entry.ApplicationMessageId); + Assert.Equal("application/json", entry.ContentType); + + await transport.CompleteAsync(entry, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task AbandonAsync_RedeliversWithIncrementedDeliveryCountAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull) + { + Assert.Skip("Transport does not support pull receive (ISupportsPull)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("retry"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("retry-me")], new TransportSendOptions(), TestCancellationToken); + + var first = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(1) }, TestCancellationToken)); + Assert.Equal(1, first.DeliveryCount); + + await transport.AbandonAsync(first, TestCancellationToken); + + var second = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(1) }, TestCancellationToken)); + Assert.Equal(first.Id, second.Id); + Assert.Equal(2, second.DeliveryCount); + Assert.Equal("retry-me", ReadBody(second)); + + await transport.CompleteAsync(second, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task CompleteAsync_WithExpiredReceipt_ThrowsReceiptExpiredExceptionAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull) + { + Assert.Skip("Transport does not support pull receive (ISupportsPull)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("receipts"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("done")], new TransportSendOptions(), TestCancellationToken); + + var entry = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(1) }, TestCancellationToken)); + await transport.CompleteAsync(entry, TestCancellationToken); + + await Assert.ThrowsAsync(async () => + await transport.CompleteAsync(entry, TestCancellationToken)); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task SubscribeAsync_DeliversPushMessagesAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPush push) + { + Assert.Skip("Transport does not support push delivery (ISupportsPush)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("push"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + + var received = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var subscription = await push.SubscribeAsync(queue, async (entry, ct) => + { + await transport.CompleteAsync(entry, ct); + received.TrySetResult(entry); + }, new PushOptions(), TestCancellationToken); + + await transport.SendAsync(queue, [CreateMessage("pushed")], new TransportSendOptions(), TestCancellationToken); + + var completed = await Task.WhenAny(received.Task, Task.Delay(TimeSpan.FromSeconds(3), TestCancellationToken)); + Assert.Equal(received.Task, completed); + Assert.Equal("pushed", ReadBody(await received.Task)); + Assert.Equal(queue, subscription.Source); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task SendAsync_ToTopic_FansOutToSubscriptionsAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull || transport is not ISupportsProvisioning) + { + Assert.Skip("Fan-out verification requires pull receive plus provisioning (ISupportsPull + ISupportsProvisioning) to create the topic's subscriptions up front. A transport that supports topics without ISupportsProvisioning (subscriptions created out of band) must cover fan-out in its own tests."); + return; + } + + try + { + var topic = DestinationAddress.ForTopic("orders-topic"); + var subscriptionA = DestinationAddress.ForSubscription("orders-topic", "orders-subscription-a"); + var subscriptionB = DestinationAddress.ForSubscription("orders-topic", "orders-subscription-b"); + await EnsureAsync(transport, + new DestinationDeclaration { Address = topic }, + new DestinationDeclaration { Address = subscriptionA }, + new DestinationDeclaration { Address = subscriptionB }); + + // The address states the destination role; publishing to a topic must use a topic-role address. + await transport.SendAsync(topic, [CreateMessage("fanout")], new TransportSendOptions(), TestCancellationToken); + + var first = Assert.Single(await pull.ReceiveAsync(subscriptionA, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, TestCancellationToken)); + var second = Assert.Single(await pull.ReceiveAsync(subscriptionB, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(1) }, TestCancellationToken)); + + Assert.Equal("fanout", ReadBody(first)); + Assert.Equal("fanout", ReadBody(second)); + + await transport.CompleteAsync(first, TestCancellationToken); + await transport.CompleteAsync(second, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task ProvisioningLifecycle_EnsureExistsDeleteAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsProvisioning provisioning) + { + Assert.Skip("Transport does not support provisioning (ISupportsProvisioning)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("prov-queue"); + var topic = DestinationAddress.ForTopic("prov-topic"); + var subscription = DestinationAddress.ForSubscription("prov-topic", "prov-sub"); + + Assert.False(await provisioning.ExistsAsync(queue, TestCancellationToken)); + Assert.False(await provisioning.ExistsAsync(topic, TestCancellationToken)); + Assert.False(await provisioning.ExistsAsync(subscription, TestCancellationToken)); + + DestinationDeclaration[] declarations = [ + new DestinationDeclaration { Address = queue }, + new DestinationDeclaration { Address = topic }, + new DestinationDeclaration { Address = subscription } + ]; + await provisioning.EnsureAsync(declarations, TestCancellationToken); + + Assert.True(await provisioning.ExistsAsync(queue, TestCancellationToken)); + Assert.True(await provisioning.ExistsAsync(topic, TestCancellationToken)); + Assert.True(await provisioning.ExistsAsync(subscription, TestCancellationToken)); + + // Ensure is idempotent: re-declaring destinations that already exist must not throw. + await provisioning.EnsureAsync(declarations, TestCancellationToken); + + await provisioning.DeleteAsync(subscription, TestCancellationToken); + await provisioning.DeleteAsync(topic, TestCancellationToken); + await provisioning.DeleteAsync(queue, TestCancellationToken); + + Assert.False(await provisioning.ExistsAsync(queue, TestCancellationToken)); + Assert.False(await provisioning.ExistsAsync(topic, TestCancellationToken)); + Assert.False(await provisioning.ExistsAsync(subscription, TestCancellationToken)); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task SendAsync_ToTopic_WithDeliverAt_WithoutNativeDelay_ThrowsAsync() + { + var transport = CreateTransport(); + if (transport is null) + { + Assert.Skip("No transport configured."); + return; + } + + if (transport is ITransportInfo { SupportedRoles: { } roles } && !roles.Contains(DestinationRole.Topic)) + { + Assert.Skip("Transport does not support topic destinations."); + return; + } + + if (GetCapabilities(transport, DestinationAddress.ForTopic("delayed-topic")).DelayedDelivery) + { + Assert.Skip("Transport honors delayed delivery natively for topics; nothing to refuse."); + return; + } + + try + { + // A transport that cannot honor DeliverAt for a role must refuse it, never publish immediately and + // silently drop the delay — the core only routes a delayed send here when the role advertises the + // capability, so acceptance would mean a lost delay (the AWS SNS delayed-publish bug shape). + await Assert.ThrowsAsync(() => transport.SendAsync(DestinationAddress.ForTopic("delayed-topic"), + [CreateMessage("later")], + new TransportSendOptions { DeliverAt = DateTimeOffset.UtcNow.AddMinutes(5) }, + TestCancellationToken)); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task ReceiveAsync_RespectsPriorityAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull || !GetCapabilities(transport, DestinationAddress.ForQueue("priority")).Priority) + { + Assert.Skip("Transport does not support pull receive with queue priority (ISupportsPull + Priority capability)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("priority"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("low")], new TransportSendOptions { Priority = MessagePriority.Low }, TestCancellationToken); + await transport.SendAsync(queue, [CreateMessage("high")], new TransportSendOptions { Priority = MessagePriority.High }, TestCancellationToken); + await transport.SendAsync(queue, [CreateMessage("normal")], new TransportSendOptions { Priority = MessagePriority.Normal }, TestCancellationToken); + + var entries = await pull.ReceiveAsync(queue, new ReceiveRequest + { + MaxMessages = 3, + MaxWaitTime = TimeSpan.FromSeconds(1) + }, TestCancellationToken); + + Assert.Equal(3, entries.Count); + Assert.Equal("high", ReadBody(entries[0])); + Assert.Equal("normal", ReadBody(entries[1])); + Assert.Equal("low", ReadBody(entries[2])); + + foreach (var entry in entries) + await transport.CompleteAsync(entry, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task SendAsync_WithDeliverAt_DelaysVisibilityAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull || !GetCapabilities(transport, DestinationAddress.ForQueue("delayed")).DelayedDelivery) + { + Assert.Skip("Transport does not support pull receive with native queue delayed delivery (ISupportsPull + DelayedDelivery capability)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("delayed"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("later")], new TransportSendOptions + { + DeliverAt = DateTimeOffset.UtcNow.AddMilliseconds(250) + }, TestCancellationToken); + + var immediate = await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromMilliseconds(50) }, TestCancellationToken); + Assert.Empty(immediate); + + var delayed = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, TestCancellationToken)); + Assert.Equal("later", ReadBody(delayed)); + await transport.CompleteAsync(delayed, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task DeadLetterAsync_MovesEntryToDeadletterStatsAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull || transport is not ISupportsDeadLetter || transport is not ISupportsStats stats) + { + Assert.Skip("Transport does not support pull receive with dead-letter and stats (ISupportsPull + ISupportsDeadLetter + ISupportsStats)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("deadletter"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("poison")], new TransportSendOptions(), TestCancellationToken); + + var entry = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(1) }, TestCancellationToken)); + await ((ISupportsDeadLetter)transport).DeadLetterAsync(entry, "bad-payload", TestCancellationToken); + + MessageDestinationStats queueStats = await stats.GetStatsAsync(queue, TestCancellationToken); + Assert.Equal(0, queueStats.Working); + Assert.Equal(1, queueStats.Deadletter); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task ReceiveAsync_WithExpiredMessage_DeadlettersAndSkipsAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull || !GetCapabilities(transport, DestinationAddress.ForQueue("expiration")).Expiration || transport is not ISupportsStats stats) + { + Assert.Skip("Transport does not support pull receive with expiration and stats (ISupportsPull + Expiration capability + ISupportsStats)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("expiration"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + var expired = new TransportMessage + { + Body = Encoding.UTF8.GetBytes("expired"), + Headers = MessageHeaders.Create([ + new KeyValuePair(KnownHeaders.Expiration, DateTimeOffset.UtcNow.AddMinutes(-1).ToString("O")) + ]) + }; + + await transport.SendAsync(queue, [expired], new TransportSendOptions(), TestCancellationToken); + + var entries = await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromMilliseconds(50) }, TestCancellationToken); + Assert.Empty(entries); + + MessageDestinationStats queueStats = await stats.GetStatsAsync(queue, TestCancellationToken); + Assert.Equal(0, queueStats.Queued); + Assert.Equal(1, queueStats.Deadletter); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + [Fact] + public virtual async Task ReceiveAsync_AfterVisibilityTimeout_RedeliversAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsVisibilityTimeout visibility) + { + Assert.Skip("Transport does not support visibility timeout (ISupportsVisibilityTimeout)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("visibility"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("lease")], new TransportSendOptions(), TestCancellationToken); + + // Whole-second visibility window: real brokers (e.g. SQS) only support second-resolution visibility timeouts. + var visibilityWindow = TimeSpan.FromSeconds(2); + var first = Assert.Single(await visibility.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, visibilityWindow, TestCancellationToken)); + Assert.Equal(1, first.DeliveryCount); + + // Still within the visibility window: a competing receive must not see the in-flight message. + var hidden = await visibility.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromMilliseconds(100) }, visibilityWindow, TestCancellationToken); + Assert.Empty(hidden); + + // After the visibility window lapses without settlement the message must be redelivered (at-least-once). A + // long poll observes the lapse — a transport wakes a blocked receive when a visibility window expires — so + // this is robust to coarse/variable redelivery latency without a fixed sleep. + var second = Assert.Single(await visibility.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = visibilityWindow + TimeSpan.FromSeconds(5) }, visibilityWindow, TestCancellationToken)); + Assert.Equal(first.Id, second.Id); + Assert.Equal(2, second.DeliveryCount); + + await transport.CompleteAsync(second, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task AbandonAsync_WithRedeliveryDelay_RedeliversAfterDelayAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull || transport is not ISupportsRedeliveryDelay redelivery) + { + Assert.Skip("Transport does not support pull receive with redelivery delay (ISupportsPull + ISupportsRedeliveryDelay)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("redelivery-delay"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("delay-me")], new TransportSendOptions(), TestCancellationToken); + + var first = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, TestCancellationToken)); + Assert.Equal(1, first.DeliveryCount); + + // Whole-second redelivery delay: SQS serves this via ChangeMessageVisibility, which is second-resolution. + var redeliveryDelay = TimeSpan.FromSeconds(2); + await redelivery.AbandonAsync(first, redeliveryDelay, TestCancellationToken); + + // Within the delay window the message must not be visible again. + var early = await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromMilliseconds(100) }, TestCancellationToken); + Assert.Empty(early); + + // After the delay lapses it is redelivered with an incremented delivery count. Long poll for robustness. + var second = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = redeliveryDelay + TimeSpan.FromSeconds(5) }, TestCancellationToken)); + Assert.Equal(first.Id, second.Id); + Assert.Equal(2, second.DeliveryCount); + Assert.Equal("delay-me", ReadBody(second)); + + await transport.CompleteAsync(second, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task RenewLockAsync_ExtendsVisibilityWindowAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsVisibilityTimeout visibility || transport is not ISupportsLockRenewal lockRenewal) + { + Assert.Skip("Transport does not support visibility timeout with lock renewal (ISupportsVisibilityTimeout + ISupportsLockRenewal)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("lock-renewal"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("hold")], new TransportSendOptions(), TestCancellationToken); + + // Whole-second windows so the test maps onto second-resolution brokers (e.g. SQS). + var originalWindow = TimeSpan.FromSeconds(2); + var renewedWindow = TimeSpan.FromSeconds(8); + var first = Assert.Single(await visibility.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, originalWindow, TestCancellationToken)); + Assert.Equal(1, first.DeliveryCount); + + // Renew before the original window lapses, extending it well past the original expiry. + await Task.Delay(TimeSpan.FromSeconds(1), TestCancellationToken); + await lockRenewal.RenewLockAsync(first, renewedWindow, TestCancellationToken); + + // Past the original window but inside the renewed window: the message must still be held, so a competing + // receive sees nothing rather than a premature redelivery. + await Task.Delay(originalWindow, TestCancellationToken); + var held = await visibility.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromMilliseconds(100) }, originalWindow, TestCancellationToken); + Assert.Empty(held); + + await transport.CompleteAsync(first, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task CompetingConsumers_DoNotReceiveTheSameInFlightMessageAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull) + { + Assert.Skip("Transport does not support pull receive (ISupportsPull)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("competing"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("once")], new TransportSendOptions(), TestCancellationToken); + + var first = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(1) }, TestCancellationToken)); + + // A competing consumer must not receive the same message while it is in flight. + var second = await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromMilliseconds(100) }, TestCancellationToken); + Assert.Empty(second); + + await transport.CompleteAsync(first, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task ReplayDeadLetteredAsync_PreservesApplicationIdAndResetsAttemptsAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsDeadLetter dead || transport is not ISupportsPull pull) + { + Assert.Skip("Transport has no dead-letter administration."); + return; + } + try + { + var token = TestCancellationToken; + var source = DestinationAddress.ForQueue("dead-source"); + var target = DestinationAddress.ForQueue("replay-target"); + await EnsureAsync(transport, new DestinationDeclaration { Address = source }); + await EnsureAsync(transport, new DestinationDeclaration { Address = target }); + await transport.SendAsync(source, [CreateMessage("payload", (KnownHeaders.Attempts, "5")) with { MessageId = "stable-id" }], new TransportSendOptions(), token); + var original = Assert.Single(await pull.ReceiveAsync(source, new ReceiveRequest(), token)); + await dead.DeadLetterAsync(original, "failure", token); + var entry = Assert.Single(await dead.PeekDeadLetteredAsync(source, cancellationToken: token)); + Assert.True(await dead.ReplayDeadLetteredAsync(source, entry.Id, target, token)); + Assert.False(await dead.ReplayDeadLetteredAsync(source, entry.Id, target, token)); + Assert.Empty(await dead.PeekDeadLetteredAsync(source, cancellationToken: token)); + var replayed = Assert.Single(await pull.ReceiveAsync(target, new ReceiveRequest(), token)); + Assert.Equal("stable-id", replayed.ApplicationMessageId); + Assert.Equal("payload", ReadBody(replayed)); + Assert.Equal(1, replayed.DeliveryCount); + Assert.False(replayed.Headers.ContainsKey(KnownHeaders.Attempts)); + await transport.CompleteAsync(replayed, token); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task PeekDeadLetteredAsync_PreservesEvidenceUntilExplicitDeletionAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull || transport is not ISupportsDeadLetter deadLetter) + { + Assert.Skip("Transport does not support pull receive and dead-letter (ISupportsPull + ISupportsDeadLetter)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("dlq-read"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("poison", ("tenant", "acme"))], new TransportSendOptions(), TestCancellationToken); + + var entry = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(1) }, TestCancellationToken)); + await deadLetter.DeadLetterAsync(entry, "bad-payload", TestCancellationToken); + + // The raw (un-deserialized) payload and the dead-letter reason must be inspectable. + var deadLettered = Assert.Single(await deadLetter.PeekDeadLetteredAsync(queue, new DeadLetterQuery { Limit = 10 }, TestCancellationToken)); + Assert.Equal("poison", ReadBody(deadLettered)); + Assert.Equal("acme", deadLettered.Headers["tenant"]); + Assert.Equal("bad-payload", deadLettered.Headers[KnownHeaders.DeadLetterReason]); + + Assert.Equal(deadLettered.Id, Assert.Single(await deadLetter.PeekDeadLetteredAsync(queue, cancellationToken: TestCancellationToken)).Id); + Assert.True(await deadLetter.DeleteDeadLetteredAsync(queue, deadLettered.Id, TestCancellationToken)); + Assert.False(await deadLetter.DeleteDeadLetteredAsync(queue, deadLettered.Id, TestCancellationToken)); + Assert.Empty(await deadLetter.PeekDeadLetteredAsync(queue, cancellationToken: TestCancellationToken)); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task SendAsync_PreservesBinaryBodyAndCaseInsensitiveHeadersAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull) + { + Assert.Skip("Transport does not support pull receive (ISupportsPull)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("binary"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + + // Arbitrary, non-UTF-8 bytes with no content type must round-trip exactly (catches body-encoding bugs — a + // provider must not assume text), and header keys must round-trip case-insensitively across the wire. + byte[] payload = [0x00, 0x01, 0xFF, 0xFE, 0x10, 0x80, 0x7F]; + await transport.SendAsync(queue, [new TransportMessage + { + Body = payload, + Headers = MessageHeaders.Create([ + new KeyValuePair("tenant", "acme"), + new KeyValuePair("Mixed.Case", "x") + ]) + }], new TransportSendOptions(), TestCancellationToken); + + var entry = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, TestCancellationToken)); + Assert.Equal(payload, entry.Body.ToArray()); + Assert.Equal("acme", entry.Headers["tenant"]); + Assert.Equal("x", entry.Headers["MIXED.CASE"]); + + await transport.CompleteAsync(entry, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + private async ValueTask CleanupTransportIfNotNullAsync(IMessageTransport? transport) + { + if (transport is not null) + await CleanupTransportAsync(transport); + } + + // Polls until the destination reports no queued or in-flight messages (the point-in-time gauges every broker can + // report), tolerating transports whose stats are only eventually consistent (e.g. SQS ApproximateNumberOf*). + private async Task AssertQueueDrainedAsync(ISupportsStats stats, DestinationAddress destination, CancellationToken cancellationToken) + { + var current = await stats.GetStatsAsync(destination, cancellationToken); + for (int attempt = 0; attempt < 50 && (current.Queued != 0 || current.Working != 0); attempt++) + { + await Task.Delay(100, cancellationToken); + current = await stats.GetStatsAsync(destination, cancellationToken); + } + + Assert.Equal(0, current.Queued); + Assert.Equal(0, current.Working); + } + + private static TransportCapabilities GetCapabilities(IMessageTransport transport, DestinationAddress destination) + { + return transport is ITransportInfo info ? info.GetCapabilities(destination) : TransportCapabilities.None; + } + + private static async Task EnsureAsync(IMessageTransport transport, params DestinationDeclaration[] declarations) + { + if (transport is ISupportsProvisioning provisioning) + await provisioning.EnsureAsync(declarations, CancellationToken.None); + } + + private static TransportMessage CreateMessage(string body, params (string Key, string Value)[] headers) + { + return new TransportMessage + { + Body = Encoding.UTF8.GetBytes(body), + Headers = MessageHeaders.Create(ToKeyValuePairs(headers)) + }; + } + + private static IEnumerable> ToKeyValuePairs((string Key, string Value)[] headers) + { + foreach (var header in headers) + yield return new KeyValuePair(header.Key, header.Value); + } + + private static string ReadBody(TransportEntry entry) + { + return Encoding.UTF8.GetString(entry.Body.Span); + } +} diff --git a/src/Foundatio.TestHarness/Queue/QueueTestBase.cs b/src/Foundatio.TestHarness/Queue/QueueTestBase.cs deleted file mode 100644 index 9c8ed42df..000000000 --- a/src/Foundatio.TestHarness/Queue/QueueTestBase.cs +++ /dev/null @@ -1,2220 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Diagnostics; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; -using Exceptionless; -using Foundatio.AsyncEx; -using Foundatio.Caching; -using Foundatio.Jobs; -using Foundatio.Lock; -using Foundatio.Messaging; -using Foundatio.Queues; -using Foundatio.Serializer; -using Foundatio.Tests.Extensions; -using Foundatio.Tests.Serializer; -using Foundatio.Tests.Utility; -using Foundatio.Utility; -using Foundatio.Xunit; -using Microsoft.Extensions.Logging; -using Xunit; - -namespace Foundatio.Tests.Queue; - -public abstract class QueueTestBase : TestWithLoggingBase -{ - protected QueueTestBase(ITestOutputHelper output) : base(output) - { - Log.SetLogLevel(LogLevel.Debug); - Log.SetLogLevel(LogLevel.Debug); - } - - protected virtual IQueue? GetQueue(int retries = 1, TimeSpan? workItemTimeout = null, TimeSpan? retryDelay = null, int[]? retryMultipliers = null, int deadLetterMaxItems = 100, bool runQueueMaintenance = true, TimeProvider? timeProvider = null, ISerializer? serializer = null) - { - return null; - } - - protected virtual async Task CleanupQueueAsync(IQueue queue) - { - if (queue == null) - return; - - try - { - await queue.DeleteQueueAsync(); - } - catch (Exception ex) - { - _logger.LogError(ex, "Error cleaning up queue: {Message}", ex.Message); - } - finally - { - queue.Dispose(); - } - } - - protected bool _assertStats = true; - - public virtual async Task CanQueueAndDequeueWorkItemAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - using var metrics = new InMemoryMetrics(FoundatioDiagnostics.Meter.Name, _logger); - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello", - SubMetricName = "myitem" - }); - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Enqueued); - - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - if (_assertStats) - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Dequeued); - - await workItem.CompleteAsync(); - Assert.False(workItem.IsAbandoned); - Assert.True(workItem.IsCompleted); - - metrics.RecordObservableInstruments(); - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(1, stats.Completed); - Assert.Equal(0, stats.Queued); - - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.enqueued")); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.dequeued")); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.completed")); - - Assert.Equal(0, metrics.Value("foundatio.simpleworkitem.count")); - Assert.Equal(0, metrics.Value("foundatio.simpleworkitem.working")); - Assert.Equal(0, metrics.Value("foundatio.simpleworkitem.deadletter")); - - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.myitem.enqueued")); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.myitem.dequeued")); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.myitem.completed")); - } - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanQueueAndDequeueWorkItemWithDelayAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - using var metrics = new InMemoryMetrics(FoundatioDiagnostics.Meter.Name, _logger); - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello" - }, new QueueEntryOptions { DeliveryDelay = TimeSpan.FromSeconds(1) }); - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Enqueued); - - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.Null(workItem); - - workItem = await queue.DequeueAsync(TimeSpan.FromSeconds(2)); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - if (_assertStats) - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Dequeued); - - await workItem.CompleteAsync(); - Assert.False(workItem.IsAbandoned); - Assert.True(workItem.IsCompleted); - - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(1, stats.Completed); - Assert.Equal(0, stats.Queued); - - metrics.RecordObservableInstruments(); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.enqueued")); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.dequeued")); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.completed")); - - Assert.Equal(0, metrics.Value("foundatio.simpleworkitem.count")); - Assert.Equal(0, metrics.Value("foundatio.simpleworkitem.working")); - Assert.Equal(0, metrics.Value("foundatio.simpleworkitem.deadletter")); - } - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanUseQueueOptionsAsync() - { - using var queue = GetQueue(retryDelay: TimeSpan.Zero); - if (queue == null) - return; - - using var metrics = new InMemoryMetrics(FoundatioDiagnostics.Meter.Name, _logger); - - try - { - using var listener = new ActivityListener - { - ShouldListenTo = s => s.Name == "Foundatio", - Sample = (ref ActivityCreationOptions _) => ActivitySamplingResult.AllDataAndRecorded, - ActivityStarted = activity => _logger.LogInformation("Start: {DisplayName}", activity.DisplayName), - ActivityStopped = activity => _logger.LogInformation("Stop: {DisplayName}", activity.DisplayName) - }; - - Activity.Current = new Activity("Parent"); - - ActivitySource.AddActivityListener(listener); - - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello" - }, new QueueEntryOptions - { - CorrelationId = "123+456", - Properties = new Dictionary { - { "hey", "now" } - } - }); - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Enqueued); - - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - Assert.Equal("123+456", workItem.CorrelationId); - Assert.Single(workItem.Properties); - Assert.Contains(workItem.Properties, i => i.Key == "hey" && i.Value.ToString() == "now"); - if (_assertStats) - { - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Dequeued); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.dequeued")); - } - - await workItem.AbandonAsync(); - Assert.True(workItem.IsAbandoned); - Assert.False(workItem.IsCompleted); - await Task.Delay(100, TestCancellationToken); - - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(1, stats.Abandoned); - Assert.Equal(0, stats.Completed); - Assert.Equal(1, stats.Queued); - - metrics.RecordObservableInstruments(); - Assert.Equal(0, metrics.Sum("foundatio.simpleworkitem.completed")); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.abandoned")); - Assert.Equal(1, metrics.Value("foundatio.simpleworkitem.count")); - } - - workItem = await queue.DequeueAsync(TimeSpan.FromSeconds(10)); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - Assert.Equal("123+456", workItem.CorrelationId); - Assert.Equal(2, workItem.Attempts); - Assert.Single(workItem.Properties); - Assert.Contains(workItem.Properties, i => i.Key == "hey" && i.Value.ToString() == "now"); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanDiscardDuplicateQueueEntriesAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - using var metrics = new InMemoryMetrics(FoundatioDiagnostics.Meter.Name, _logger); - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - queue.AttachBehavior(new DuplicateDetectionQueueBehavior(new InMemoryCacheClient(o => o.LoggerFactory(Log)), Log)); - - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello", - UniqueIdentifier = "123" - }); - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(1, stats.Enqueued); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.enqueued")); - } - - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello", - UniqueIdentifier = "123" - }); - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(1, stats.Enqueued); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.enqueued")); - } - - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(1, stats.Dequeued); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.dequeued")); - } - - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello", - UniqueIdentifier = "123" - }); - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(2, stats.Enqueued); - Assert.Equal(2, metrics.Sum("foundatio.simpleworkitem.enqueued")); - } - - await workItem.CompleteAsync(); - Assert.False(workItem.IsAbandoned); - Assert.True(workItem.IsCompleted); - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(2, stats.Enqueued); - Assert.Equal(1, stats.Dequeued); - Assert.Equal(1, stats.Completed); - Assert.Equal(0, stats.Abandoned); - - Assert.Equal(1, stats.Queued); - Assert.Equal(0, stats.Working); - Assert.Equal(0, stats.Deadletter); - Assert.Equal(0, stats.Errors); - Assert.Equal(0, stats.Timeouts); - - metrics.RecordObservableInstruments(); - Assert.Equal(2, metrics.Sum("foundatio.simpleworkitem.enqueued")); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.dequeued")); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.completed")); - Assert.Equal(0, metrics.Sum("foundatio.simpleworkitem.abandoned")); - - Assert.Equal(1, metrics.Value("foundatio.simpleworkitem.count")); - Assert.Equal(0, metrics.Value("foundatio.simpleworkitem.working")); - Assert.Equal(0, metrics.Value("foundatio.simpleworkitem.deadletter")); - } - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task DuplicateDetection_WithDifferentIdentifiers_AcceptsBothItemsAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - queue.AttachBehavior(new DuplicateDetectionQueueBehavior(new InMemoryCacheClient(o => o.LoggerFactory(Log)), Log)); - - // Act - await queue.EnqueueAsync(new SimpleWorkItem { Data = "First", UniqueIdentifier = "aaa" }); - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Second", UniqueIdentifier = "bbb" }); - - // Assert - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(2, stats.Enqueued); - } - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task DuplicateDetection_WithExpiredWindow_AcceptsDuplicateAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - queue.AttachBehavior(new DuplicateDetectionQueueBehavior( - new InMemoryCacheClient(o => o.LoggerFactory(Log)), Log, detectionWindow: TimeSpan.FromMilliseconds(100))); - - // Act - enqueue first item - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello", UniqueIdentifier = "abc" }); - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(1, stats.Enqueued); - } - - // Act - wait for detection window to expire, then enqueue same identifier - await Task.Delay(250); - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello", UniqueIdentifier = "abc" }); - - // Assert - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(2, stats.Enqueued); - } - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task DuplicateDetection_WithNullIdentifier_AcceptsAllItemsAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - queue.AttachBehavior(new DuplicateDetectionQueueBehavior(new InMemoryCacheClient(o => o.LoggerFactory(Log)), Log)); - - // Act - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello", UniqueIdentifier = null }); - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello", UniqueIdentifier = null }); - - // Assert - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(2, stats.Enqueued); - } - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task VerifyRetryAttemptsAsync() - { - const int retryCount = 2; - using var queue = GetQueue(retryCount, TimeSpan.FromSeconds(1), TimeSpan.Zero, [1]); - if (queue == null) - return; - - await VerifyRetryAttemptsImplAsync(queue, retryCount, TimeSpan.FromSeconds(10)); - } - - public virtual async Task VerifyDelayedRetryAttemptsAsync() - { - const int retryCount = 2; - using var queue = GetQueue(retryCount, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(1), [1]); - if (queue == null) - return; - - await VerifyRetryAttemptsImplAsync(queue, retryCount, TimeSpan.FromSeconds(30)); - } - - private async Task VerifyRetryAttemptsImplAsync(IQueue queue, int retryCount, TimeSpan waitTime) - { - using var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(30)); - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - using var metrics = new InMemoryMetrics(FoundatioDiagnostics.Meter.Name, _logger); - - var countdown = new AsyncCountdownEvent(retryCount + 1); - int attempts = 0; - - await queue.StartWorkingAsync(async w => - { - Interlocked.Increment(ref attempts); - _logger.LogInformation("Starting Attempt {Attempt} to work on queue item", attempts); - Assert.NotNull(w.Value); - Assert.Equal("Hello", w.Value.Data); - - var queueEntryMetadata = (IQueueEntryMetadata)w; - Assert.Equal(attempts, queueEntryMetadata.Attempts); - - await w.AbandonAsync(); - countdown.Signal(); - - _logger.LogInformation("Finished Attempt {Attempt} to work on queue item, Metadata Attempts: {QueueEntryAttempts}", attempts, queueEntryMetadata.Attempts); - }, cancellationToken: cancellationTokenSource.Token); - - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello" - }); - - await countdown.WaitAsync(waitTime); - Assert.Equal(0, countdown.CurrentCount); - - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(retryCount + 1, attempts); - Assert.Equal(0, stats.Completed); - Assert.Equal(0, stats.Queued); - Assert.Equal(0, stats.Errors); - Assert.Equal(retryCount + 1, stats.Dequeued); - Assert.Equal(retryCount + 1, stats.Abandoned); - - metrics.RecordObservableInstruments(); - Assert.Equal(retryCount + 1, metrics.Sum("foundatio.simpleworkitem.dequeued")); - Assert.Equal(0, metrics.Sum("foundatio.simpleworkitem.completed")); - Assert.Equal(retryCount + 1, metrics.Sum("foundatio.simpleworkitem.abandoned")); - - Assert.Equal(0, metrics.Value("foundatio.simpleworkitem.count")); - } - } - finally - { - await cancellationTokenSource.CancelAsync(); - await CleanupQueueAsync(queue); - } - } - - /// - /// When a cancelled token is passed into Dequeue, it will only try to dequeue one time and then exit. - /// - /// - public virtual async Task CanDequeueWithCancelledTokenAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - using var metrics = new InMemoryMetrics(FoundatioDiagnostics.Meter.Name, _logger); - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello" - }); - if (_assertStats) - { - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Enqueued); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.enqueued")); - } - - var workItem = await queue.DequeueAsync(new CancellationToken(true)); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Dequeued); - - // TODO: We should verify that only one retry occurred. - await workItem.CompleteAsync(); - - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(1, stats.Completed); - Assert.Equal(0, stats.Queued); - Assert.Equal(1, metrics.Sum("foundatio.simpleworkitem.completed")); - } - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanDequeueEfficientlyAsync() - { - const int iterations = 100; - - using var queue = GetQueue(runQueueMaintenance: false); - if (queue == null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Initialize queue to create more accurate metrics" }); - Assert.NotNull(await queue.DequeueAsync(TimeSpan.FromSeconds(1))); - - using var metrics = new InMemoryMetrics(FoundatioDiagnostics.Meter.Name, _logger); - - _ = Task.Run(async () => - { - _logger.LogTrace("Starting enqueue loop"); - for (int index = 0; index < iterations; index++) - { - await Task.Delay(RandomData.GetInt(10, 30)); - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello" }); - } - _logger.LogTrace("Finished enqueuing"); - }); - - _logger.LogTrace("Starting dequeue loop"); - for (int index = 0; index < iterations; index++) - { - var item = await queue.DequeueAsync(TimeSpan.FromSeconds(3)); - Assert.NotNull(item); - await item.CompleteAsync(); - } - _logger.LogTrace("Finished dequeuing"); - - Assert.InRange(metrics.Avg("foundatio.simpleworkitem.queuetime"), 0, 100); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanResumeDequeueEfficientlyAsync() - { - const int iterations = 10; - - using var queue = GetQueue(runQueueMaintenance: false); - if (queue == null) - return; - - try - { - using var metrics = new InMemoryMetrics(FoundatioDiagnostics.Meter.Name, _logger); - - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - for (int index = 0; index < iterations; index++) - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello" }); - - using var secondQueue = GetQueue(runQueueMaintenance: false); - Assert.NotNull(secondQueue); - - _logger.LogTrace("Starting dequeue loop"); - for (int index = 0; index < iterations; index++) - { - _logger.LogTrace("[{Index}] Calling Dequeue", index); - var item = await secondQueue.DequeueAsync(TimeSpan.FromSeconds(3)); - Assert.NotNull(item); - await item.CompleteAsync(); - } - - metrics.RecordObservableInstruments(); - Assert.InRange(metrics.Avg("foundatio.simpleworkitem.queuetime"), 0, 100); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanQueueAndDequeueMultipleWorkItemsAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - try - { - using var metrics = new InMemoryMetrics(FoundatioDiagnostics.Meter.Name, _logger); - - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - const int workItemCount = 25; - for (int i = 0; i < workItemCount; i++) - { - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello" - }); - } - metrics.RecordObservableInstruments(); - Assert.Equal(workItemCount, metrics.Value("foundatio.simpleworkitem.count")); - Assert.Equal(workItemCount, (await queue.GetQueueStatsAsync()).Queued); - - var sw = Stopwatch.StartNew(); - for (int i = 0; i < workItemCount; i++) - { - var workItem = await queue.DequeueAsync(TimeSpan.FromSeconds(5)); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - await workItem.CompleteAsync(); - } - sw.Stop(); - _logger.LogTrace("Time {Elapsed:g}", sw.Elapsed); - Assert.InRange(sw.Elapsed.TotalSeconds, 0, 5); - - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(workItemCount, stats.Dequeued); - Assert.Equal(workItemCount, stats.Completed); - Assert.Equal(0, stats.Queued); - } - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task WillNotWaitForItemAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - var sw = Stopwatch.StartNew(); - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - sw.Stop(); - _logger.LogTrace("Time {Elapsed:g}", sw.Elapsed); - Assert.Null(workItem); - Assert.InRange(sw.Elapsed.TotalMilliseconds, 0, 100); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task WillWaitForItemAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - var sw = Stopwatch.StartNew(); - var workItem = await queue.DequeueAsync(TimeSpan.FromMilliseconds(100)); - sw.Stop(); - _logger.LogTrace("Time {Elapsed:g}", sw.Elapsed); - Assert.Null(workItem); - Assert.InRange(sw.Elapsed, TimeSpan.FromMilliseconds(50), TimeSpan.FromMilliseconds(5000)); - - _ = Task.Run(async () => - { - await Task.Delay(500); - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello" - }); - }); - - sw.Restart(); - workItem = await queue.DequeueAsync(TimeSpan.FromSeconds(10)); - sw.Stop(); - _logger.LogTrace("Time {Elapsed:g}", sw.Elapsed); - Assert.True(sw.Elapsed > TimeSpan.FromMilliseconds(400)); - Assert.NotNull(workItem); - await workItem.CompleteAsync(); - - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task DequeueAsync_AfterAbandonWithMutatedValue_ReturnsOriginalValueAsync() - { - using var queue = GetQueue(retries: 1, retryDelay: TimeSpan.Zero); - if (queue == null) - return; - - using var metrics = new InMemoryMetrics(FoundatioDiagnostics.Meter.Name, _logger); - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello" }); - - // Act: first dequeue, mutate, abandon - var workItem = await queue.DequeueAsync(TimeSpan.FromSeconds(5)); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - - workItem.Value.Data = "Mutated"; - Assert.True(await metrics.WaitForCounterAsync("foundatio.simpleworkitem.abandoned", () => workItem.AbandonAsync(), cancellationToken: TestCancellationToken)); - - // Assert: original entry retains abandoned state after abandon - Assert.True(workItem.IsAbandoned); - Assert.False(workItem.IsCompleted); - - // Assert: verify stats after abandon - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(1, stats.Queued); - Assert.Equal(1, stats.Dequeued); - Assert.Equal(1, stats.Abandoned); - Assert.Equal(0, stats.Completed); - } - - // Act: second dequeue (retry) should have pristine value - var retryItem = await queue.DequeueAsync(TimeSpan.FromSeconds(5)); - Assert.NotNull(retryItem?.Value); - Assert.Equal("Hello", retryItem.Value.Data); - - // Assert: retry entry has fresh state, original entry is still abandoned - Assert.False(retryItem.IsAbandoned); - Assert.False(retryItem.IsCompleted); - Assert.True(workItem.IsAbandoned); - - Assert.True(await metrics.WaitForCounterAsync("foundatio.simpleworkitem.completed", () => retryItem.CompleteAsync(), cancellationToken: TestCancellationToken)); - - // Assert: final entry states - Assert.True(retryItem.IsCompleted); - Assert.False(retryItem.IsAbandoned); - Assert.True(workItem.IsAbandoned); - Assert.False(workItem.IsCompleted); - - // Assert: verify final stats - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(0, stats.Queued); - Assert.Equal(2, stats.Dequeued); - Assert.Equal(1, stats.Abandoned); - Assert.Equal(1, stats.Completed); - } - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task DequeueWaitWillGetSignaledAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - _ = Task.Run(async () => - { - await Task.Delay(250); - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello" - }); - }); - - var sw = Stopwatch.StartNew(); - var workItem = await queue.DequeueAsync(TimeSpan.FromSeconds(2)); - sw.Stop(); - _logger.LogTrace("Time {Elapsed:g}", sw.Elapsed); - Assert.NotNull(workItem); - Assert.InRange(sw.Elapsed.TotalSeconds, 0, 2); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanUseQueueWorkerAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - using var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(30)); - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - var resetEvent = new AsyncManualResetEvent(false); - await queue.StartWorkingAsync(async w => - { - Assert.NotNull(w.Value); - Assert.Equal("Hello", w.Value.Data); - await w.CompleteAsync(); - resetEvent.Set(); - }, cancellationToken: cancellationTokenSource.Token); - - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello" - }); - - await resetEvent.WaitAsync(); - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(1, stats.Completed); - Assert.Equal(0, stats.Queued); - Assert.Equal(0, stats.Errors); - } - } - finally - { - await cancellationTokenSource.CancelAsync(); - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanHandleErrorInWorkerAsync() - { - using var queue = GetQueue(retries: 0); - if (queue == null) - return; - - using var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(30)); - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.StartWorkingAsync(w => - { - _logger.LogDebug("WorkAction"); - Assert.NotNull(w.Value); - Assert.Equal("Hello", w.Value.Data); - throw new Exception(); - }, cancellationToken: cancellationTokenSource.Token); - - var resetEvent = new AsyncManualResetEvent(false); - using (queue.Abandoned.AddSyncHandler((o, args) => resetEvent.Set())) - { - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello" }); - await resetEvent.WaitAsync(TimeSpan.FromSeconds(200)); - - await Task.Delay(100, TestCancellationToken); // give time for the stats to reflect the changes. - - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - _logger.LogInformation("Completed: {Completed} Errors: {Errors} Deadletter: {Deadletter} Working: {Working} ", stats.Completed, stats.Errors, stats.Deadletter, stats.Working); - Assert.Equal(0, stats.Completed); - Assert.Equal(1, stats.Errors); - Assert.Equal(1, stats.Deadletter); - } - } - } - finally - { - await cancellationTokenSource.CancelAsync(); - await CleanupQueueAsync(queue); - } - } - - public virtual async Task WorkItemsWillTimeoutAsync() - { - using var queue = GetQueue(retryDelay: TimeSpan.Zero, workItemTimeout: TimeSpan.FromSeconds(1)); - if (queue == null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello" - }); - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - - var sw = Stopwatch.StartNew(); - if (_assertStats) - { - // wait for the entry to be auto abandoned - do - { - var stats = await queue.GetQueueStatsAsync(); - if (stats.Abandoned > 0) - break; - await Task.Delay(1250); - } while (sw.Elapsed < TimeSpan.FromSeconds(10)); - } - - // should throw because the item has already been auto abandoned - if (_assertStats) - await Assert.ThrowsAnyAsync(async () => await workItem.CompleteAsync().AnyContext()); - - sw = Stopwatch.StartNew(); - workItem = await queue.DequeueAsync(TimeSpan.FromSeconds(5)); - sw.Stop(); - _logger.LogTrace("Time {Elapsed:g}", sw.Elapsed); - Assert.NotNull(workItem); - await workItem.CompleteAsync(); - if (_assertStats) - Assert.Equal(0, (await queue.GetQueueStatsAsync()).Queued); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task WorkItemsWillGetMovedToDeadletterAsync() - { - using var queue = GetQueue(retryDelay: TimeSpan.Zero); - if (queue == null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello" - }); - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Dequeued); - - await workItem.AbandonAsync(); - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Abandoned); - - // work item should be retried 1 time. - workItem = await queue.DequeueAsync(TimeSpan.FromSeconds(5)); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - Assert.Equal(2, (await queue.GetQueueStatsAsync()).Dequeued); - - await workItem.AbandonAsync(); - - if (_assertStats) - { - // work item should be moved to deadletter _queue after retries. - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(1, stats.Deadletter); - Assert.Equal(2, stats.Abandoned); - } - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task AbandonAsync_WhenRetriesExceeded_MovesToDeadletterAsync() - { - // Arrange - const int retryCount = 1; - using var queue = GetQueue(retryCount, retryDelay: TimeSpan.Zero); - if (queue is null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello" }); - - // Act - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - Assert.Equal(1, workItem.Attempts); - await workItem.AbandonAsync(); - - workItem = await queue.DequeueAsync(TimeSpan.FromSeconds(5)); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - Assert.Equal(2, workItem.Attempts); - await workItem.AbandonAsync(); - - // Assert - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - _logger.LogInformation("Stats after abandon: Queued={Queued} Working={Working} Deadletter={Deadletter} Abandoned={Abandoned} Dequeued={Dequeued}", - stats.Queued, stats.Working, stats.Deadletter, stats.Abandoned, stats.Dequeued); - - Assert.Equal(1, stats.Deadletter); - Assert.Equal(0, stats.Queued); - Assert.Equal(0, stats.Working); - } - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task DequeueAsync_WithPoisonMessage_MovesToDeadletterAsync() - { - // Use retries > 0 to prove poison messages go through the normal abandon/retry - // cycle before being dead-lettered, allowing transient serializer misconfigurations - // to self-heal on redeploy. - const int retries = 2; - var faultSerializer = new FaultInjectingSerializer(); - var queue = GetQueue(retries: retries, retryDelay: TimeSpan.Zero, retryMultipliers: [1], serializer: faultSerializer); - if (queue is null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - // Arrange: enqueue a valid message (serializer works normally) - await queue.EnqueueAsync(new SimpleWorkItem { Data = "poison-test" }); - - // Flip the flag so deserialization throws on every dequeue - faultSerializer.ShouldFailOnDeserialize = true; - - // Act: dequeue enough times to exhaust retries (initial attempt + retries) - for (int attempt = 0; attempt <= retries; attempt++) - { - var entry = await queue.DequeueAsync(TimeSpan.FromSeconds(5)); - Assert.Null(entry); - - var intermediateStats = await queue.GetQueueStatsAsync(); - _logger.LogInformation("Poison message attempt {Attempt}: Queued={Queued} Deadletter={Deadletter} Abandoned={Abandoned}", - attempt + 1, intermediateStats.Queued, intermediateStats.Deadletter, intermediateStats.Abandoned); - } - - // Assert: message should be dead-lettered after exhausting retries - var stats = await queue.GetQueueStatsAsync(); - _logger.LogInformation("Poison message final stats: Queued={Queued} Deadletter={Deadletter} Abandoned={Abandoned}", - stats.Queued, stats.Deadletter, stats.Abandoned); - Assert.Equal(1, stats.Deadletter); - Assert.Equal(0, stats.Queued); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task EnqueueAsync_WithSerializationError_ThrowsAndLeavesQueueEmptyAsync() - { - var faultSerializer = new FaultInjectingSerializer(); - var queue = GetQueue(serializer: faultSerializer); - if (queue is null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - // Arrange: enable serialization failure before enqueue - faultSerializer.ShouldFailOnSerialize = true; - - // Act & Assert: enqueue should throw since the message can't be serialized - await Assert.ThrowsAnyAsync(() => - queue.EnqueueAsync(new SimpleWorkItem { Data = "should-fail" })); - - // Assert: queue should remain empty — no corrupt data was persisted - var stats = await queue.GetQueueStatsAsync(); - _logger.LogInformation("Enqueue serialization error stats: Queued={Queued} Deadletter={Deadletter}", - stats.Queued, stats.Deadletter); - Assert.Equal(0, stats.Queued); - Assert.Equal(0, stats.Deadletter); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanAutoCompleteWorkerAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - using var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(30)); - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - var resetEvent = new AsyncManualResetEvent(false); - await queue.StartWorkingAsync(w => - { - Assert.NotNull(w.Value); - Assert.Equal("Hello", w.Value.Data); - return Task.CompletedTask; - }, true, cancellationTokenSource.Token); - - using (queue.Completed.AddSyncHandler((s, e) => { resetEvent.Set(); })) - { - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello" }); - - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Enqueued); - await resetEvent.WaitAsync(TimeSpan.FromSeconds(2)); - - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(0, stats.Queued); - Assert.Equal(0, stats.Errors); - Assert.Equal(1, stats.Completed); - } - } - } - finally - { - await cancellationTokenSource.CancelAsync(); - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanHaveMultipleQueueInstancesAsync() - { - using var queue = GetQueue(retries: 0, retryDelay: TimeSpan.Zero); - if (queue == null) - return; - - using var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(30)); - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - const int workItemCount = 500; - const int workerCount = 3; - var countdown = new AsyncCountdownEvent(workItemCount); - var info = new WorkInfo(); - var workers = new List> { queue }; - - try - { - for (int i = 0; i < workerCount; i++) - { - var q = GetQueue(retries: 0, retryDelay: TimeSpan.Zero); - Assert.NotNull(q); - - _logger.LogTrace("Queue Id: {QueueId}, I: {Instance}", q.QueueId, i); - await q.StartWorkingAsync(w => DoWorkAsync(w, countdown, info), cancellationToken: cancellationTokenSource.Token); - workers.Add(q); - } - - await Parallel.ForEachAsync(Enumerable.Range(1, workItemCount), cancellationTokenSource.Token, async (i, _) => - { - string? id = await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello", - Id = i - }); - _logger.LogTrace("Enqueued Index: {Instance} Id: {QueueEntryId}", i, id); - }); - - await countdown.WaitAsync(cancellationTokenSource.Token); - await Task.Delay(50, cancellationTokenSource.Token); - - _logger.LogInformation("Work Info Stats: Completed: {Completed} Abandoned: {Abandoned} Error: {Errors}", info.CompletedCount, info.AbandonCount, info.ErrorCount); - Assert.Equal(workItemCount, info.CompletedCount + info.AbandonCount + info.ErrorCount); - - // In memory queue doesn't share state. - if (queue.GetType() == typeof(InMemoryQueue)) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(0, stats.Working); - Assert.Equal(0, stats.Timeouts); - Assert.Equal(workItemCount, stats.Enqueued); - Assert.Equal(workItemCount, stats.Dequeued); - Assert.Equal(info.CompletedCount, stats.Completed); - Assert.Equal(info.ErrorCount, stats.Errors); - Assert.Equal(info.AbandonCount, stats.Abandoned - info.ErrorCount); - Assert.Equal(info.AbandonCount + stats.Errors, stats.Deadletter); - } - else if (_assertStats) - { - var workerStats = new List(); - for (int i = 0; i < workers.Count; i++) - { - var stats = await workers[i].GetQueueStatsAsync(); - _logger.LogInformation("Worker#{Id} Working: {Working} Completed: {Completed} Abandoned: {Abandoned} Error: {Errors} Deadletter: {Deadletter}", i, stats.Working, stats.Completed, stats.Abandoned, stats.Errors, stats.Deadletter); - workerStats.Add(stats); - } - - Assert.Equal(info.CompletedCount, workerStats.Sum(s => s.Completed)); - Assert.Equal(info.ErrorCount, workerStats.Sum(s => s.Errors)); - Assert.Equal(info.AbandonCount, workerStats.Sum(s => s.Abandoned) - info.ErrorCount); - Assert.Equal(info.AbandonCount + workerStats.Sum(s => s.Errors), (workerStats.LastOrDefault()?.Deadletter ?? 0)); - //Expected: 260 - //Actual: 125 - } - } - finally - { - await cancellationTokenSource.CancelAsync(); - foreach (var q in workers) - await CleanupQueueAsync(q); - } - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanDelayRetryAsync() - { - using var queue = GetQueue(workItemTimeout: TimeSpan.FromSeconds(1), retryDelay: TimeSpan.FromSeconds(1)); - if (queue == null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello" - }); - - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - - var startTime = DateTime.UtcNow; - await workItem.AbandonAsync(); - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Abandoned); - - workItem = await queue.DequeueAsync(TimeSpan.FromSeconds(5)); - var elapsed = DateTime.UtcNow.Subtract(startTime); - _logger.LogTrace("Time {Elapsed}", elapsed); - Assert.NotNull(workItem); - Assert.InRange(elapsed, TimeSpan.FromMilliseconds(900), TimeSpan.FromSeconds(10)); - await workItem.CompleteAsync(); - - if (_assertStats) - Assert.Equal(0, (await queue.GetQueueStatsAsync()).Queued); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanRunWorkItemWithMetricsAsync() - { - int completedCount = 0; - - using var queue = new InMemoryQueue(o => o.LoggerFactory(Log).MetricsPollingInterval(TimeSpan.Zero)); - - Task Handler(object sender, CompletedEventArgs e) - { - completedCount++; - return Task.CompletedTask; - } - - using var metrics = new InMemoryMetrics(FoundatioDiagnostics.Meter.Name, _logger); - - using (queue.Completed.AddHandler(Handler)) - { - _logger.LogTrace("Before enqueue"); - await queue.EnqueueAsync(new SimpleWorkItem { Id = 1, Data = "Testing" }); - await queue.EnqueueAsync(new SimpleWorkItem { Id = 2, Data = "Testing" }); - await queue.EnqueueAsync(new SimpleWorkItem { Id = 3, Data = "Testing" }); - - await Task.Delay(100, TestCancellationToken); - - _logger.LogTrace("Before dequeue"); - var item = await queue.DequeueAsync(); - Assert.NotNull(item); - - await Task.Delay(100, TestCancellationToken); - await item.CompleteAsync(); - - item = await queue.DequeueAsync(); - Assert.NotNull(item); - - await Task.Delay(100, TestCancellationToken); - await item.CompleteAsync(); - - item = await queue.DequeueAsync(); - Assert.NotNull(item); - - await Task.Delay(100, TestCancellationToken); - await item.AbandonAsync(); - - _logger.LogTrace("Before asserts"); - Assert.Equal(2, completedCount); - - metrics.RecordObservableInstruments(); - Assert.InRange(metrics.Max("foundatio.workitemdata.count"), 1, 3); - Assert.InRange(metrics.Max("foundatio.workitemdata.working"), 0, 1); - - Assert.Equal(3, metrics.Sum("foundatio.workitemdata.simple.enqueued")); - Assert.Equal(3, metrics.Sum("foundatio.workitemdata.enqueued")); - - Assert.Equal(3, metrics.Sum("foundatio.workitemdata.simple.dequeued")); - Assert.Equal(3, metrics.Sum("foundatio.workitemdata.dequeued")); - - Assert.Equal(2, metrics.Sum("foundatio.workitemdata.simple.completed")); - Assert.Equal(2, metrics.Sum("foundatio.workitemdata.completed")); - - Assert.Equal(1, metrics.Sum("foundatio.workitemdata.simple.abandoned")); - Assert.Equal(1, metrics.Sum("foundatio.workitemdata.abandoned")); - - var measurements = metrics.GetMeasurements("foundatio.workitemdata.simple.queuetime"); - Assert.Equal(3, measurements.Count); - measurements = metrics.GetMeasurements("foundatio.workitemdata.queuetime"); - Assert.Equal(3, measurements.Count); - - measurements = metrics.GetMeasurements("foundatio.workitemdata.simple.processtime"); - Assert.Equal(3, measurements.Count); - measurements = metrics.GetMeasurements("foundatio.workitemdata.processtime"); - Assert.Equal(3, measurements.Count); - } - } - - public virtual async Task CanRenewLockAsync() - { - Log.SetLogLevel>(LogLevel.Trace); - - // Need large value to reproduce this test - var workItemTimeout = TimeSpan.FromSeconds(1); - // Slightly shorter than the timeout to ensure we haven't lost the lock - var renewWait = TimeSpan.FromSeconds(workItemTimeout.TotalSeconds * .25d); - - using var queue = GetQueue(retryDelay: TimeSpan.Zero, workItemTimeout: workItemTimeout); - if (queue == null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello" - }); - var entry = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(entry?.Value); - Assert.Equal("Hello", entry.Value.Data); - - _logger.LogTrace("Waiting for {RenewWait:g} before renewing lock", renewWait); - await Task.Delay(renewWait); - _logger.LogTrace("Renewing lock"); - await entry.RenewLockAsync(); - _logger.LogTrace("Waiting for {RenewWait:g} to see if lock was renewed", renewWait); - await Task.Delay(renewWait); - - // We shouldn't get another item here if RenewLock works. - _logger.LogTrace("Attempting to dequeue item that shouldn't exist"); - var nullWorkItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.Null(nullWorkItem); - await entry.CompleteAsync(); - - if (_assertStats) - Assert.Equal(0, (await queue.GetQueueStatsAsync()).Queued); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanAbandonQueueEntryOnceAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello" }); - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Enqueued); - - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Dequeued); - - await workItem.AbandonAsync(); - Assert.True(workItem.IsAbandoned); - Assert.False(workItem.IsCompleted); - await Assert.ThrowsAnyAsync(() => workItem.AbandonAsync()); - await Assert.ThrowsAnyAsync(() => workItem.CompleteAsync()); - await Assert.ThrowsAnyAsync(() => workItem.CompleteAsync()); - - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(1, stats.Abandoned); - Assert.Equal(0, stats.Completed); - Assert.Equal(0, stats.Deadletter); - Assert.Equal(1, stats.Dequeued); - Assert.Equal(1, stats.Enqueued); - Assert.Equal(0, stats.Errors); - Assert.InRange(stats.Queued, 0, 1); - Assert.Equal(0, stats.Timeouts); - Assert.Equal(0, stats.Working); - } - - if (workItem is QueueEntry queueEntry) - Assert.Equal(1, queueEntry.Attempts); - - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello" }); - workItem = await queue.DequeueAsync(TimeSpan.Zero); - - Assert.NotNull(workItem); - await queue.AbandonAsync(workItem); - Assert.True(workItem.IsAbandoned); - Assert.False(workItem.IsCompleted); - await Assert.ThrowsAnyAsync(() => workItem.CompleteAsync()); - await Assert.ThrowsAnyAsync(() => workItem.AbandonAsync()); - await Assert.ThrowsAnyAsync(() => queue.AbandonAsync(workItem)); - await Assert.ThrowsAnyAsync(() => queue.CompleteAsync(workItem)); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanCompleteQueueEntryOnceAsync() - { - using var queue = GetQueue(); - if (queue == null) - return; - - try - { - await queue.DeleteQueueAsync(); - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello" }); - - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Enqueued); - - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem?.Value); - Assert.Equal("Hello", workItem.Value.Data); - Assert.Equal(1, (await queue.GetQueueStatsAsync()).Dequeued); - - await workItem.CompleteAsync(); - await Assert.ThrowsAnyAsync(() => workItem.CompleteAsync()); - await Assert.ThrowsAnyAsync(() => workItem.AbandonAsync()); - await Assert.ThrowsAnyAsync(() => workItem.AbandonAsync()); - - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(0, stats.Abandoned); - Assert.Equal(1, stats.Completed); - Assert.Equal(0, stats.Deadletter); - Assert.Equal(1, stats.Dequeued); - Assert.Equal(1, stats.Enqueued); - Assert.Equal(0, stats.Errors); - Assert.Equal(0, stats.Queued); - Assert.Equal(0, stats.Timeouts); - Assert.Equal(0, stats.Working); - } - - if (workItem is QueueEntry queueEntry) - Assert.Equal(1, queueEntry.Attempts); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanDequeueWithLockingAsync() - { - using var cache = new InMemoryCacheClient(o => o.LoggerFactory(Log)); - using var messageBus = new InMemoryMessageBus(o => o.LoggerFactory(Log)); - - var distributedLock = new CacheLockProvider(cache, messageBus, null, null, Log); - await CanDequeueWithLockingImpAsync(distributedLock); - } - - protected async Task CanDequeueWithLockingImpAsync(CacheLockProvider distributedLock) - { - using var queue = GetQueue(retryDelay: TimeSpan.Zero, retries: 0); - if (queue == null) - return; - - using var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(30)); - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - var resetEvent = new AsyncAutoResetEvent(); - await queue.StartWorkingAsync(async w => - { - _logger.LogInformation("Acquiring distributed lock in work item"); - var l = await distributedLock.AcquireAsync("test", cancellationToken: cancellationTokenSource.Token); - _logger.LogInformation("Acquired distributed lock"); - await Task.Delay(TimeSpan.FromMilliseconds(250)); - await l.ReleaseAsync(); - _logger.LogInformation("Released distributed lock"); - - await w.CompleteAsync(); - resetEvent.Set(); - }, cancellationToken: cancellationTokenSource.Token); - - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Hello" }); - await resetEvent.WaitAsync(TimeSpan.FromSeconds(5)); - - if (_assertStats) - { - await Task.Delay(1); - var stats = await queue.GetQueueStatsAsync(); - _logger.LogInformation("Completed: {Completed} Errors: {Errors} Deadletter: {Deadletter} Working: {Working} ", stats.Completed, stats.Errors, stats.Deadletter, stats.Working); - Assert.Equal(1, stats.Completed); - } - } - finally - { - await cancellationTokenSource.CancelAsync(); - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanHaveMultipleQueueInstancesWithLockingAsync() - { - using var cache = new InMemoryCacheClient(o => o.LoggerFactory(Log)); - using var messageBus = new InMemoryMessageBus(o => o.LoggerFactory(Log)); - - var distributedLock = new CacheLockProvider(cache, messageBus, null, null, Log); - await CanHaveMultipleQueueInstancesWithLockingImplAsync(distributedLock); - } - - protected async Task CanHaveMultipleQueueInstancesWithLockingImplAsync(CacheLockProvider distributedLock) - { - using var queue = GetQueue(retries: 0, retryDelay: TimeSpan.Zero); - if (queue == null) - return; - - using var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(30)); - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - const int workItemCount = 16; - const int workerCount = 4; - var countdown = new AsyncCountdownEvent(workItemCount); - var info = new WorkInfo(); - var workers = new List> { queue }; - - try - { - for (int i = 0; i < workerCount; i++) - { - var q = GetQueue(retries: 0, retryDelay: TimeSpan.Zero); - Assert.NotNull(q); - int instanceCount = i; - await q.StartWorkingAsync(async w => - { - _logger.LogInformation("[{Instance}] Acquiring distributed lock in work item: {QueueEntryId}", instanceCount, w.Id); - var l = await distributedLock.AcquireAsync("test", cancellationToken: cancellationTokenSource.Token); - _logger.LogInformation("[{Instance}] Acquired distributed lock: {QueueEntryId}", instanceCount, w.Id); - await Task.Delay(TimeSpan.FromMilliseconds(50), cancellationTokenSource.Token); - await l.ReleaseAsync(); - _logger.LogInformation("[{Instance}] Released distributed lock: {QueueEntryId}", instanceCount, w.Id); - - await w.CompleteAsync(); - info.IncrementCompletedCount(); - countdown.Signal(); - _logger.LogInformation("[{Instance}] Signaled countdown: {QueueEntryId}", instanceCount, w.Id); - }, cancellationToken: cancellationTokenSource.Token); - workers.Add(q); - } - - await Parallel.ForEachAsync(Enumerable.Range(1, workItemCount), cancellationTokenSource.Token, async (i, _) => - { - string? id = await queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello", - Id = i - }); - _logger.LogTrace("Enqueued Index: {Instance} Id: {QueueEntryId}", i, id); - }); - - await countdown.WaitAsync(TimeSpan.FromSeconds(5)); - await Task.Delay(50, cancellationTokenSource.Token); - _logger.LogTrace("Completed: {Completed} Abandoned: {Abandoned} Error: {Errors}", info.CompletedCount, info.AbandonCount, info.ErrorCount); - - _logger.LogInformation("Work Info Stats: Completed: {Completed} Abandoned: {Abandoned} Error: {Errors}", info.CompletedCount, info.AbandonCount, info.ErrorCount); - Assert.Equal(workItemCount, info.CompletedCount + info.AbandonCount + info.ErrorCount); - - // In memory queue doesn't share state. - if (queue.GetType() == typeof(InMemoryQueue)) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(info.CompletedCount, stats.Completed); - } - else - { - var workerStats = new List(); - for (int i = 0; i < workers.Count; i++) - { - var stats = await workers[i].GetQueueStatsAsync(); - _logger.LogInformation("Worker#{Id} Working: {Working} Completed: {Completed} Abandoned: {Abandoned} Error: {Errors} Deadletter: {Deadletter}", i, stats.Working, stats.Completed, stats.Abandoned, stats.Errors, stats.Deadletter); - workerStats.Add(stats); - } - - Assert.Equal(info.CompletedCount, workerStats.Sum(s => s.Completed)); - } - } - finally - { - await cancellationTokenSource.CancelAsync(); - foreach (var q in workers) - await CleanupQueueAsync(q); - } - } - finally - { - await CleanupQueueAsync(queue); - } - } - - protected async Task DoWorkAsync(IQueueEntry w, AsyncCountdownEvent countdown, WorkInfo info) - { - Assert.NotNull(w.Value); - _logger.LogTrace("Starting: {Id}", w.Value.Id); - Assert.Equal("Hello", w.Value.Data); - - try - { - // randomly complete, abandon or blowup. - if (RandomData.GetBool()) - { - _logger.LogTrace("Completing: {Id}", w.Value.Id); - await w.CompleteAsync(); - info.IncrementCompletedCount(); - } - else if (RandomData.GetBool()) - { - _logger.LogTrace("Abandoning: {Id}", w.Value.Id); - await w.AbandonAsync(); - info.IncrementAbandonCount(); - } - else - { - _logger.LogTrace("Erroring: {Id}", w.Value.Id); - info.IncrementErrorCount(); - throw new Exception(); - } - } - finally - { - _logger.LogTrace("Signal {CurrentCount}", countdown.CurrentCount); - countdown.Signal(); - } - } - - protected async Task AssertEmptyQueueAsync(IQueue queue) - { - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(0, stats.Abandoned); - Assert.Equal(0, stats.Completed); - Assert.Equal(0, stats.Deadletter); - Assert.Equal(0, stats.Dequeued); - Assert.Equal(0, stats.Enqueued); - Assert.Equal(0, stats.Errors); - Assert.Equal(0, stats.Queued); - Assert.Equal(0, stats.Timeouts); - Assert.Equal(0, stats.Working); - } - } - - public virtual async Task MaintainJobNotAbandon_NotWorkTimeOutEntry() - { - using var queue = GetQueue(retries: 0, workItemTimeout: TimeSpan.FromSeconds(1), retryDelay: TimeSpan.Zero); - if (queue == null) - return; - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - var enqueueTask1 = queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello World", - Id = 1 - }); - var enqueueTask2 = queue.EnqueueAsync(new SimpleWorkItem - { - Data = "Hello World", - Id = 2 - }); - - var dequeuedQueueItem = Assert.IsAssignableFrom>(await queue.DequeueAsync()); - Assert.NotNull(dequeuedQueueItem.Value); - // The first dequeued item works for 900 milliseconds less than work timeout(1s). - await Task.Delay(900); - await dequeuedQueueItem.CompleteAsync(); - Assert.True(dequeuedQueueItem.IsCompleted); - Assert.False(dequeuedQueueItem.IsAbandoned); - - dequeuedQueueItem = Assert.IsAssignableFrom>(await queue.DequeueAsync()); - Assert.NotNull(dequeuedQueueItem.Value); - // The second dequeued item works for 900 milliseconds less than work timeout(1s). - await Task.Delay(900); - await dequeuedQueueItem.CompleteAsync(); - Assert.True(dequeuedQueueItem.IsCompleted); - Assert.False(dequeuedQueueItem.IsAbandoned); - - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(0, stats.Working); - Assert.Equal(0, stats.Abandoned); - Assert.Equal(2, stats.Completed); - } - - await Task.WhenAll(enqueueTask1, enqueueTask2); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task CanHandleAutoAbandonInWorker() - { - // create queue with short work item timeout so it will be auto-abandoned - using var queue = GetQueue(workItemTimeout: TimeSpan.FromSeconds(1)); - if (queue == null) - return; - - using var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(30)); - try - { - await queue.DeleteQueueAsync(); - - var successEvent = new AsyncAutoResetEvent(); - var errorEvent = new AsyncAutoResetEvent(); - - await queue.StartWorkingAsync(async item => - { - Assert.NotNull(item.Value); - _logger.LogDebug("Processing item: {QueueEntryId} Value={Value}", item.Id, item.Value.Data); - if (item.Value is { Data: "Delay" }) - { - // wait for queue item to get auto abandoned - var stats = await queue.GetQueueStatsAsync(); - var sw = Stopwatch.StartNew(); - do - { - if (stats.Abandoned > 0) - { - _logger.LogTrace("Breaking, queue item was abandoned"); - break; - } - - stats = await queue.GetQueueStatsAsync(); - _logger.LogTrace("Getting updated stats... Queued={Queued}, Working={Working}, Abandoned={Abandoned} Deadletter={Deadletter}, Enqueued={Enqueued}, Dequeued={Dequeued}, Completed={Completed}, Errors={Errors}, Timeouts={Timeouts}", - stats.Queued, stats.Working, stats.Abandoned, stats.Deadletter, stats.Enqueued, stats.Dequeued, stats.Completed, stats.Errors, stats.Timeouts); - - await Task.Delay(250, cancellationTokenSource.Token); - } while (sw.Elapsed < TimeSpan.FromSeconds(5)); - - Assert.Equal(1, stats.Abandoned); - } - - try - { - await item.CompleteAsync(); - } - catch (Exception ex) - { - _logger.LogDebug(ex, "Error completing item: {Message}", ex.Message); - errorEvent.Set(); - throw; - } - - successEvent.Set(); - }, cancellationToken: cancellationTokenSource.Token); - - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Delay" }); - await queue.EnqueueAsync(new SimpleWorkItem { Data = "No Delay" }); - - await errorEvent.WaitAsync(TimeSpan.FromSeconds(10)); - await successEvent.WaitAsync(TimeSpan.FromSeconds(10)); - } - finally - { - await cancellationTokenSource.CancelAsync(); - await CleanupQueueAsync(queue); - } - } - - public virtual async Task DequeueAsync_WithDispose_AutoAbandonsEntryAsync() - { - // Arrange - using var queue = GetQueue(retries: 1, retryDelay: TimeSpan.Zero); - if (queue is null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem { Data = "dispose-test" }); - - // Act - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem); - Assert.NotNull(workItem.Value); - Assert.Equal("dispose-test", workItem.Value.Data); - await workItem.DisposeAsync(); - - // Assert - await Task.Delay(100, TestCancellationToken); - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.True(stats.Abandoned > 0 || stats.Queued > 0, - $"Expected item to be abandoned or re-queued after dispose. Stats: Abandoned={stats.Abandoned}, Queued={stats.Queued}"); - } - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task EnqueueAsync_WithUniqueId_UsesProvidedIdAsync() - { - // Arrange - using var queue = GetQueue(); - if (queue is null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - // Act - string? entryId = await queue.EnqueueAsync(new SimpleWorkItem { Data = "unique-id-test" }, - new QueueEntryOptions { UniqueId = "my-custom-id-123" }); - - // Assert - Assert.NotNull(entryId); - Assert.Equal("my-custom-id-123", entryId); - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem); - Assert.Equal("my-custom-id-123", workItem.Id); - await workItem.CompleteAsync(); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task GetDeadletterItemsAsync_WithDeadletteredEntry_ReturnsItemsAsync() - { - // Arrange - using var queue = GetQueue(retries: 0, retryDelay: TimeSpan.Zero); - if (queue is null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem { Data = "deadletter-test" }); - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem); - await workItem.AbandonAsync(); - - if (_assertStats) - { - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(1, stats.Deadletter); - } - - // Act - var deadletterItems = await queue.GetDeadletterItemsAsync(); - - // Assert - var items = new List(deadletterItems); - Assert.Single(items); - Assert.Equal("deadletter-test", items[0].Data); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task GetQueueActivity_AfterEnqueueAndDequeue_ReturnsTimestampsAsync() - { - // Arrange - using var queue = GetQueue(); - if (queue is null) - return; - - if (queue is not IQueueActivity activity) - return; - - try - { - await queue.DeleteQueueAsync(); - Assert.Null(activity.LastEnqueueActivity); - Assert.Null(activity.LastDequeueActivity); - - // Act - await queue.EnqueueAsync(new SimpleWorkItem { Data = "activity-test" }); - - // Assert - Assert.NotNull(activity.LastEnqueueActivity); - var enqueueTime = activity.LastEnqueueActivity.Value; - Assert.True(enqueueTime <= DateTimeOffset.UtcNow); - - // Act - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem); - - // Assert - Assert.NotNull(activity.LastDequeueActivity); - Assert.True(activity.LastDequeueActivity.Value >= enqueueTime); - - await workItem.CompleteAsync(); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task GetQueueEntryMetadata_AfterDequeue_ReturnsValidTimestampsAsync() - { - // Arrange - using var queue = GetQueue(); - if (queue is null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - var beforeEnqueue = DateTime.UtcNow; - await queue.EnqueueAsync(new SimpleWorkItem { Data = "metadata-test" }); - - // Act - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem); - - // Assert - if (workItem is IQueueEntryMetadata metadata) - { - Assert.True(metadata.EnqueuedTimeUtc >= beforeEnqueue.AddSeconds(-1), - $"EnqueuedTimeUtc {metadata.EnqueuedTimeUtc} should be >= {beforeEnqueue.AddSeconds(-1)}"); - Assert.True(metadata.DequeuedTimeUtc >= metadata.EnqueuedTimeUtc, - $"DequeuedTimeUtc {metadata.DequeuedTimeUtc} should be >= EnqueuedTimeUtc {metadata.EnqueuedTimeUtc}"); - Assert.True(metadata.ProcessingTime >= TimeSpan.Zero); - Assert.True(metadata.TotalTime >= TimeSpan.Zero); - } - - await workItem.CompleteAsync(); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task QueueEntry_EntryType_ReturnsCorrectTypeAsync() - { - // Arrange - using var queue = GetQueue(); - if (queue is null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem { Data = "entrytype-test" }); - - // Act - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem); - - // Assert - Assert.Equal(typeof(SimpleWorkItem), ((IQueueEntry)workItem).EntryType); - - await workItem.CompleteAsync(); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task QueueEntry_GetValue_ReturnsUntypedValueAsync() - { - // Arrange - using var queue = GetQueue(); - if (queue is null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem { Data = "getvalue-test" }); - - // Act - var workItem = await queue.DequeueAsync(TimeSpan.Zero); - Assert.NotNull(workItem); - object untypedValue = ((IQueueEntry)workItem).GetValue(); - - // Assert - Assert.NotNull(untypedValue); - Assert.IsType(untypedValue); - Assert.Equal("getvalue-test", ((SimpleWorkItem)untypedValue).Data); - - await workItem.CompleteAsync(); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public virtual async Task Dispose_WithMaintenanceRunning_DoesNotThrowObjectDisposedException() - { - // Arrange - var queue = GetQueue(runQueueMaintenance: true); - if (queue is null) - return; - - try - { - await queue.DeleteQueueAsync(); - await AssertEmptyQueueAsync(queue); - - await queue.EnqueueAsync(new SimpleWorkItem { Data = "trigger-maintenance" }); - var item = await queue.DequeueAsync(TimeSpan.FromSeconds(5)); - Assert.NotNull(item); - await item.CompleteAsync(); - await Task.Delay(100); - - // Act - var exception = Record.Exception(() => queue.Dispose()); - - // Assert - Assert.Null(exception); - } - finally - { - await CleanupQueueAsync(queue); - } - } - - public override async ValueTask DisposeAsync() - { - await base.DisposeAsync(); - - using var queue = GetQueue(); - if (queue is null) - return; - - await queue.DeleteQueueAsync(); - GC.SuppressFinalize(this); - } -} - -public class WorkInfo -{ - private int _abandonCount; - private int _errorCount; - private int _completedCount; - - public int AbandonCount => _abandonCount; - public int ErrorCount => _errorCount; - public int CompletedCount => _completedCount; - - public void IncrementAbandonCount() - { - Interlocked.Increment(ref _abandonCount); - } - - public void IncrementErrorCount() - { - Interlocked.Increment(ref _errorCount); - } - - public void IncrementCompletedCount() - { - Interlocked.Increment(ref _completedCount); - } -} diff --git a/src/Foundatio.TestHarness/Queue/Samples.cs b/src/Foundatio.TestHarness/Queue/Samples.cs deleted file mode 100644 index ab6a97746..000000000 --- a/src/Foundatio.TestHarness/Queue/Samples.cs +++ /dev/null @@ -1,12 +0,0 @@ -using Foundatio.Metrics; -using Foundatio.Queues; - -namespace Foundatio.Tests.Queue; - -public class SimpleWorkItem : IHaveSubMetricName, IHaveUniqueIdentifier -{ - public string? Data { get; set; } - public int Id { get; set; } - public string? UniqueIdentifier { get; set; } - public string? SubMetricName { get; set; } -} diff --git a/src/Foundatio.Testing/Foundatio.Testing.csproj b/src/Foundatio.Testing/Foundatio.Testing.csproj new file mode 100644 index 000000000..b4ab27a31 --- /dev/null +++ b/src/Foundatio.Testing/Foundatio.Testing.csproj @@ -0,0 +1,8 @@ + + + Test harnesses for Foundatio messaging and jobs: run the real message bus over a recording in-memory transport, await quiescence, and assert on the messages that were sent, published, handled, retried, or dead-lettered; drive the in-memory job runtime deterministically without the pump. + + + + + diff --git a/src/Foundatio.Testing/JobsTestHarness.cs b/src/Foundatio.Testing/JobsTestHarness.cs new file mode 100644 index 000000000..8ad98cc4b --- /dev/null +++ b/src/Foundatio.Testing/JobsTestHarness.cs @@ -0,0 +1,105 @@ +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Jobs.Testing; + +/// +/// Deterministic job tests over the real in-memory runtime without hosted workers. Tests decide when queued jobs run (), when CRON +/// occurrences materialize and execute ( with a fixed "now"), and when a single job is +/// driven to its terminal state () — no polling loop ever races the assertions. +/// +/// var services = new ServiceCollection(); +/// services.AddFoundatio().Jobs.UseTestHarness(); +/// var harness = provider.GetRequiredService<JobsTestHarness>(); +/// var handle = await harness.Client.EnqueueAsync<SendWelcomeEmailJob>(); +/// await harness.RunAllQueuedAsync(); +/// Assert.Equal(JobStatus.Completed, (await handle.GetStateAsync())!.Status); +/// +/// +public sealed class JobsTestHarness +{ + private static readonly TimeSpan DefaultRunTimeout = TimeSpan.FromSeconds(30); + + private readonly IJobRuntimeStore _store; + private readonly IJobWorker _worker; + private readonly JobScheduleProcessor _processor; + + public JobsTestHarness(IJobRuntimeStore store, IJobWorker worker, JobScheduleProcessor processor, IJobClient client, IScheduledJobManager schedules) + { + _store = store ?? throw new ArgumentNullException(nameof(store)); + _worker = worker ?? throw new ArgumentNullException(nameof(worker)); + _processor = processor ?? throw new ArgumentNullException(nameof(processor)); + Client = client ?? throw new ArgumentNullException(nameof(client)); + Schedules = schedules ?? throw new ArgumentNullException(nameof(schedules)); + } + + /// The client for enqueueing the jobs under test. + public IJobClient Client { get; } + + /// Runtime management of scheduled (CRON) jobs: add/replace definitions, enable/disable, trigger. + public IScheduledJobManager Schedules { get; } + + /// Read access to job state for assertions. + public IJobMonitor Monitor => _store; + + /// Runs all currently eligible queued jobs, across batches. Future delayed retries remain queued. + public async Task RunAllQueuedAsync(CancellationToken cancellationToken = default) + { + using var timeout = new CancellationTokenSource(DefaultRunTimeout); + using var operation = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeout.Token); + int total = 0; + try + { + while (true) + { + operation.Token.ThrowIfCancellationRequested(); + int executed = await _worker.RunQueuedAsync(cancellationToken: operation.Token).WaitAsync(operation.Token).ConfigureAwait(false); + total += executed; + if (executed < 100) + return total; + } + } + catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested && timeout.IsCancellationRequested) + { + throw new TimeoutException("Queued jobs did not become idle within 30 seconds. Check for a blocked job or work that continually enqueues more jobs."); + } + } + + /// + /// Materializes CRON occurrences due at the supplied time and runs all currently eligible queued jobs. + /// Scheduled messages are drained separately through ScheduledMessageDispatcher. + /// + public async Task RunDueAsync(DateTimeOffset? now = null, CancellationToken cancellationToken = default) + { + var utcNow = now ?? DateTimeOffset.UtcNow; + await _processor.EnqueueDueOccurrencesAsync(utcNow, cancellationToken).ConfigureAwait(false); + return await RunAllQueuedAsync(cancellationToken).ConfigureAwait(false); + } + + /// + /// Runs only the handle's job until it reaches a terminal state (Completed, Failed, Cancelled, or + /// DeadLettered) and returns that state. Throws naming the job's current status + /// when it is still non-terminal after 30s. + /// + public async Task RunToCompletionAsync(JobHandle handle, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(handle); + + long deadline = Environment.TickCount64 + (long)DefaultRunTimeout.TotalMilliseconds; + while (true) + { + cancellationToken.ThrowIfCancellationRequested(); + await _worker.RunAsync(handle.JobId, cancellationToken).ConfigureAwait(false); + + var state = await handle.GetStateAsync(cancellationToken).ConfigureAwait(false); + if (state is { Status: JobStatus.Completed or JobStatus.Failed or JobStatus.Cancelled or JobStatus.DeadLettered }) + return state; + + if (Environment.TickCount64 >= deadline) + throw new TimeoutException($"Job \"{handle.JobId}\" did not reach a terminal state in time; current status: {state?.Status.ToString() ?? "not found"}."); + + await Task.Delay(25, cancellationToken).ConfigureAwait(false); + } + } +} diff --git a/src/Foundatio.Testing/MessagingTestHarness.cs b/src/Foundatio.Testing/MessagingTestHarness.cs new file mode 100644 index 000000000..8cf9956f4 --- /dev/null +++ b/src/Foundatio.Testing/MessagingTestHarness.cs @@ -0,0 +1,242 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Serializer; + +namespace Foundatio.Messaging.Testing; + +/// +/// One recorded message movement: a send/publish accepted by the transport, or a settlement (handled, abandoned for +/// retry, dead-lettered) of a delivered message. +/// +public sealed record RecordedMessage +{ + public required string Destination { get; init; } + + /// + /// The role the message was sent to (Queue for sends, Topic for publishes). Settlement recordings always report + /// Queue: every delivery settles on a queue-shaped channel, including topic deliveries via their subscriptions. + /// + public required DestinationRole Role { get; init; } + + public string? MessageType { get; init; } + public required ReadOnlyMemory Body { get; init; } + public MessageHeaders Headers { get; init; } = MessageHeaders.Empty; + + /// The dead-letter reason, for dead-lettered recordings. + public string? Reason { get; init; } + + /// The delivery count at settlement, for settlement recordings. + public int Attempts { get; init; } +} + +/// +/// Deterministic messaging tests without sleeps: the harness runs the real bus over a recording in-memory transport, +/// so tests act (send/publish), until every queue and in-flight handler drains, then +/// assert on what actually happened — including the core retry/dead-letter path (a message redelivered N times and +/// then dead-lettered is directly assertable). +/// +/// var services = new ServiceCollection(); +/// services.AddFoundatio() +/// .Messaging.UseTestHarness() +/// .Messaging.AddSubscriber<OrderPlaced, SendConfirmationHandler>("confirmation"); +/// services.AddMessageConsumers(); +/// // start hosted services, then: +/// await bus.PublishAsync(new OrderPlaced(42)); +/// await harness.WaitForIdleAsync(); +/// Assert.Single(harness.Published<OrderPlaced>()); +/// Assert.Empty(harness.DeadLetteredMessages); +/// +/// The harness waits in REAL time: , , and +/// poll on a real 25ms cadence regardless of any injected +/// . Delayed redeliveries and backoffs, however, execute on the injected TimeProvider — +/// so a test that injects a fake TimeProvider must advance it itself or the retry never fires and the wait times +/// out. For sleep-free retry tests prefer RedeliveryBackoff = _ => TimeSpan.Zero on the subscription +/// instead of faking the clock. +/// +public sealed class MessagingTestHarness : IAsyncDisposable +{ + private static readonly TimeSpan DefaultIdleTimeout = TimeSpan.FromSeconds(30); + + private readonly RecordingMessageTransport _transport; + private readonly ISerializer _serializer; + private readonly IMessageTypeRegistry _typeRegistry; + + public MessagingTestHarness(ISerializer? serializer = null, IMessageTypeRegistry? typeRegistry = null, TimeProvider? timeProvider = null) + { + _serializer = serializer ?? DefaultSerializer.Instance; + _typeRegistry = typeRegistry ?? new MessageTypeRegistry(); + _transport = new RecordingMessageTransport(timeProvider); + } + + /// The transport to run the bus over (an in-memory transport that records every movement). + public IMessageTransport Transport => _transport; + + /// Every message accepted by a queue-role send (a command on its way to one handler). + public IReadOnlyList SentMessages => _transport.Sent; + + /// Every message accepted by a topic-role send (an event on its way to each subscriber). + public IReadOnlyList PublishedMessages => _transport.Published; + + /// Every delivered message that settled as completed (handled successfully or auto-acked). + public IReadOnlyList HandledMessages => _transport.Handled; + + /// Every delivered message returned for redelivery (a retry). + public IReadOnlyList AbandonedMessages => _transport.Abandoned; + + /// + /// Every delivered message that settled terminally into the dead-letter sink. Records deaths made through the + /// transport API (the core's retry-exhausted/unrecoverable path); a broker-internal death — such as a message + /// whose TimeToLive lapsed before delivery — is visible in destination stats and ReceiveDeadLetteredAsync but + /// not recorded here. + /// + public IReadOnlyList DeadLetteredMessages => _transport.DeadLettered; + + /// The sent (queue-role) messages of type , deserialized. + public IReadOnlyList Sent() where T : class => Deserialize(_transport.Sent); + + /// The published (topic-role) messages of type , deserialized. + public IReadOnlyList Published() where T : class => Deserialize(_transport.Published); + + /// The successfully handled messages of type , deserialized. + public IReadOnlyList Handled() where T : class => Deserialize(_transport.Handled); + + /// The retried (abandoned for redelivery) messages of type , deserialized. + public IReadOnlyList Abandoned() where T : class => Deserialize(_transport.Abandoned); + + /// The dead-lettered messages of type , deserialized. + public IReadOnlyList DeadLettered() where T : class => Deserialize(_transport.DeadLettered); + + /// + /// Destination keys that received sends/publishes but were never received from or subscribed to — the usual + /// reason a test is "idle immediately and Handled is empty": the message went to a destination nothing consumes + /// (no handler registered, hosted services never started, or a topic published before any subscription existed). + /// + public IReadOnlyList DestinationsWithNoConsumer => _transport.DestinationsWithNoConsumer; + + /// + /// Waits (polling in real time) until at least recorded HANDLED messages deserialize to + /// , then returns them — so a test can await one outcome without draining the whole bus. + /// Throws describing everything that WAS recorded when the timeout (default 30s) + /// lapses first. + /// + public Task> WaitForHandledAsync(int count = 1, TimeSpan? timeout = null, CancellationToken cancellationToken = default) where T : class + { + return WaitForRecordedAsync(() => Handled(), count, timeout, "handled", typeof(T), cancellationToken); + } + + /// + /// Waits (polling in real time) until at least recorded dead-lettered messages carry a + /// MessageType header matching (its registered name or full name), then returns the raw + /// s so the caller can assert and + /// . Throws describing everything that WAS + /// recorded when the timeout (default 30s) lapses first. + /// + public Task> WaitForDeadLetteredAsync(int count = 1, TimeSpan? timeout = null, CancellationToken cancellationToken = default) where T : class + { + string registeredName = _typeRegistry.GetName(typeof(T)); + string? fullName = typeof(T).FullName; + return WaitForRecordedAsync(() => _transport.DeadLettered + .Where(r => String.Equals(r.MessageType, registeredName, StringComparison.Ordinal) + || String.Equals(r.MessageType, fullName, StringComparison.Ordinal)) + .ToList(), count, timeout, "dead-lettered", typeof(T), cancellationToken); + } + + /// + /// Waits until the transport is quiescent — every known destination has nothing queued and nothing in flight — + /// so assertions observe the final state. Returns quickly when already idle (fast negative assertions). Throws + /// naming the still-busy destinations when the timeout (default 30s) lapses; + /// waits until idle or cancellation. + /// Store-parked work (delayed sends / delayed retries through a runtime store) is not transport activity; drain + /// it explicitly via the job schedule processor before waiting. + /// + public async Task WaitForIdleAsync(TimeSpan? timeout = null, CancellationToken cancellationToken = default) + { + var effectiveTimeout = timeout ?? DefaultIdleTimeout; + if (effectiveTimeout < TimeSpan.Zero && effectiveTimeout != Timeout.InfiniteTimeSpan) + throw new ArgumentOutOfRangeException(nameof(timeout), timeout, "Timeout must be non-negative or Timeout.InfiniteTimeSpan."); + + long deadline = effectiveTimeout == Timeout.InfiniteTimeSpan + ? Int64.MaxValue + : Environment.TickCount64 + (long)effectiveTimeout.TotalMilliseconds; + int stableChecks = 0; + + while (true) + { + cancellationToken.ThrowIfCancellationRequested(); + var pending = await _transport.GetPendingAsync(cancellationToken).ConfigureAwait(false); + + if (pending.Count == 0) + { + // Require two consecutive idle observations: a settling message can synchronously cascade into a new + // send, which a single snapshot could miss. + if (++stableChecks >= 2) + return; + } + else + { + stableChecks = 0; + + if (Environment.TickCount64 >= deadline) + { + var detail = new StringBuilder("The message bus did not become idle in time. Still busy: "); + detail.AppendJoin(", ", pending.Select(p => $"{p.Name} (queued={p.Queued}, working={p.Working})")); + throw new TimeoutException(detail.ToString()); + } + } + + await Task.Delay(25, cancellationToken).ConfigureAwait(false); + } + } + + public ValueTask DisposeAsync() => _transport.DisposeAsync(); + + private async Task> WaitForRecordedAsync(Func> snapshot, int count, TimeSpan? timeout, string outcome, Type messageType, CancellationToken cancellationToken) + { + ArgumentOutOfRangeException.ThrowIfLessThan(count, 1); + var effectiveTimeout = timeout ?? DefaultIdleTimeout; + if (effectiveTimeout < TimeSpan.Zero && effectiveTimeout != Timeout.InfiniteTimeSpan) + throw new ArgumentOutOfRangeException(nameof(timeout), timeout, "Timeout must be non-negative or Timeout.InfiniteTimeSpan."); + + long deadline = effectiveTimeout == Timeout.InfiniteTimeSpan + ? Int64.MaxValue + : Environment.TickCount64 + (long)effectiveTimeout.TotalMilliseconds; + + while (true) + { + cancellationToken.ThrowIfCancellationRequested(); + var matches = snapshot(); + if (matches.Count >= count) + return matches; + + if (Environment.TickCount64 >= deadline) + { + // Name what WAS recorded: the usual failure is the message settling some other way (or reaching a + // destination nothing consumes), and these counts point straight at which. + var detail = new StringBuilder(); + detail.Append($"Timed out waiting for {count} {outcome} message(s) of type {messageType.Name}; observed {matches.Count}. "); + detail.Append($"Recorded so far: sent={_transport.Sent.Count}, published={_transport.Published.Count}, handled={_transport.Handled.Count}, abandoned={_transport.Abandoned.Count}, deadLettered={_transport.DeadLettered.Count}."); + var unconsumed = _transport.DestinationsWithNoConsumer; + if (unconsumed.Count > 0) + detail.Append($" Destinations with no consumer: {String.Join(", ", unconsumed)}."); + throw new TimeoutException(detail.ToString()); + } + + await Task.Delay(25, cancellationToken).ConfigureAwait(false); + } + } + + private IReadOnlyList Deserialize(IReadOnlyList recordings) where T : class + { + string typeName = _typeRegistry.GetName(typeof(T)); + return recordings + .Where(r => String.Equals(r.MessageType, typeName, StringComparison.Ordinal)) + .Select(r => _serializer.Deserialize(r.Body, typeof(T)) as T) + .Where(m => m is not null) + .Select(m => m!) + .ToList(); + } +} diff --git a/src/Foundatio.Testing/RecordingMessageTransport.cs b/src/Foundatio.Testing/RecordingMessageTransport.cs new file mode 100644 index 000000000..0ebae742b --- /dev/null +++ b/src/Foundatio.Testing/RecordingMessageTransport.cs @@ -0,0 +1,225 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Messaging.Testing; + +/// +/// The harness transport: a fully-capable in-memory transport that records every send and settlement so tests can +/// assert on what actually moved through the bus, and tracks the destinations/sources it has seen so +/// can detect quiescence. +/// +internal sealed class RecordingMessageTransport : IMessageTransport, ISupportsPull, ISupportsVisibilityTimeout, + ISupportsDeadLetter, ISupportsRedeliveryDelay, ISupportsLockRenewal, ISupportsStats, + ISupportsEphemeralSubscriptions, ITransportInfo, IMessageProcessingObserver +{ + // A delayed redelivery lives only in the inner transport's timer until it fires — neither queued nor in flight — + // so idle detection would report quiescent while a retry is pending. Give the timer this long past its due time to + // materialize the redelivered message back into stats before the pending marker is dropped. + private static readonly TimeSpan _redeliveryGrace = TimeSpan.FromMilliseconds(250); + + private readonly ConcurrentDictionary _processing = new(); + public void ProcessingStarted(TransportEntry entry) => _processing[entry.Receipt] = entry.Destination; + public void ProcessingFinished(TransportEntry entry) => _processing.TryRemove(entry.Receipt, out _); + + private readonly InMemoryMessageTransport _inner; + private readonly TimeProvider _timeProvider; + private readonly ConcurrentQueue _sent = new(); + private readonly ConcurrentQueue _published = new(); + private readonly ConcurrentQueue _handled = new(); + private readonly ConcurrentQueue _abandoned = new(); + private readonly ConcurrentQueue _deadLettered = new(); + private readonly ConcurrentDictionary _knownNames = new(); + private readonly ConcurrentDictionary _sendDestinations = new(); + private readonly ConcurrentDictionary _consumeSources = new(); + private readonly ConcurrentDictionary _pendingRedeliveries = new(); + + public RecordingMessageTransport(TimeProvider? timeProvider = null) + { + _timeProvider = timeProvider ?? TimeProvider.System; + _inner = new InMemoryMessageTransport(timeProvider); + } + + public IReadOnlyList Sent => [.. _sent]; + public IReadOnlyList Published => [.. _published]; + public IReadOnlyList Handled => [.. _handled]; + public IReadOnlyList Abandoned => [.. _abandoned]; + public IReadOnlyList DeadLettered => [.. _deadLettered]; + + public DeliveryGuarantee DeliveryGuarantee => _inner.DeliveryGuarantee; + public IReadOnlySet SupportedRoles => _inner.SupportedRoles; + public TransportCapabilities GetCapabilities(DestinationAddress destination) => _inner.GetCapabilities(destination); + public TimeSpan? MaxVisibilityTimeout => _inner.MaxVisibilityTimeout; + public TimeSpan? MaxRedeliveryDelay => _inner.MaxRedeliveryDelay; + + public async Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + var result = await _inner.SendAsync(destination, messages, options, ct).ConfigureAwait(false); + + _knownNames.TryAdd(destination, 0); + _sendDestinations.TryAdd(destination, 0); + var recordings = destination.Role == DestinationRole.Topic ? _published : _sent; + foreach (var message in messages) + { + recordings.Enqueue(new RecordedMessage + { + Destination = destination.Key, + Role = destination.Role, + MessageType = message.Headers.GetValueOrDefault(KnownHeaders.MessageType), + Body = message.Body, + Headers = message.Headers + }); + } + + return result; + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct = default) + { + _knownNames.TryAdd(source, 0); + _consumeSources.TryAdd(source, 0); + return _inner.ReceiveAsync(source, request, ct); + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, TimeSpan visibility, CancellationToken ct = default) + { + _knownNames.TryAdd(source, 0); + _consumeSources.TryAdd(source, 0); + return _inner.ReceiveAsync(source, request, visibility, ct); + } + + + public async Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) + { + await _inner.CompleteAsync(entry, ct).ConfigureAwait(false); + _handled.Enqueue(Record(entry)); + } + + public async Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) + { + await _inner.AbandonAsync(entry, ct).ConfigureAwait(false); + _abandoned.Enqueue(Record(entry)); + } + + public async Task AbandonAsync(TransportEntry entry, TimeSpan redeliveryDelay, CancellationToken ct = default) + { + Guid pendingToken = Guid.NewGuid(); + if (redeliveryDelay > TimeSpan.Zero) + _pendingRedeliveries[pendingToken] = (entry.Destination, _timeProvider.GetUtcNow().Add(redeliveryDelay)); + + try + { + await _inner.AbandonAsync(entry, redeliveryDelay, ct).ConfigureAwait(false); + } + catch + { + _pendingRedeliveries.TryRemove(pendingToken, out _); + throw; + } + + _abandoned.Enqueue(Record(entry)); + } + + public async Task DeadLetterAsync(TransportEntry entry, string? reason, CancellationToken ct = default) + { + await _inner.DeadLetterAsync(entry, reason, ct).ConfigureAwait(false); + _deadLettered.Enqueue(Record(entry) with { Reason = reason }); + } + + public Task> PeekDeadLetteredAsync(DestinationAddress destination, DeadLetterQuery? query = null, CancellationToken cancellationToken = default) + => _inner.PeekDeadLetteredAsync(destination, query, cancellationToken); + public Task DeleteDeadLetteredAsync(DestinationAddress destination, string id, CancellationToken cancellationToken = default) + => _inner.DeleteDeadLetteredAsync(destination, id, cancellationToken); + + public Task ReplayDeadLetteredAsync(DestinationAddress source, string id, DestinationAddress target, CancellationToken cancellationToken = default) + => _inner.ReplayDeadLetteredAsync(source, id, target, cancellationToken); + + public Task RenewSubscriptionAsync(DestinationAddress source, TimeSpan lease, CancellationToken cancellationToken = default) + => _inner.RenewSubscriptionAsync(source, lease, cancellationToken); + + public Task RenewLockAsync(TransportEntry entry, TimeSpan? duration, CancellationToken ct = default) + => _inner.RenewLockAsync(entry, duration, ct); + + public Task GetStatsAsync(DestinationAddress destination, CancellationToken ct = default) + => _inner.GetStatsAsync(destination, ct); + + public Task EnsureAsync(IReadOnlyList declarations, CancellationToken ct = default) + { + foreach (var declaration in declarations) + _knownNames.TryAdd(declaration.Address, 0); + return _inner.EnsureAsync(declarations, ct); + } + + public Task DeleteAsync(DestinationAddress destination, CancellationToken ct = default) => _inner.DeleteAsync(destination, ct); + + public Task ExistsAsync(DestinationAddress destination, CancellationToken ct = default) => _inner.ExistsAsync(destination, ct); + + public ValueTask DisposeAsync() => _inner.DisposeAsync(); + + // Destinations that received sends/publishes but were never received from or subscribed to. A topic publish + // counts as consumed when anything consumes one of the topic's subscriptions; a topic with none is exactly the + // zero-subscription publish the inner transport drops (real pub/sub semantics), so it is included here. + public IReadOnlyList DestinationsWithNoConsumer + { + get + { + var consumers = _consumeSources.Keys.ToArray(); + return _sendDestinations.Keys + .Where(sent => !consumers.Any(consumer => Consumes(consumer, sent))) + .Select(sent => sent.Key) + .OrderBy(key => key, StringComparer.Ordinal) + .ToList(); + } + } + + private static bool Consumes(DestinationAddress consumer, DestinationAddress sent) + { + if (consumer == sent) + return true; + + // A topic is consumed through its subscriptions, which carry the owning topic in their address. + return sent.Role == DestinationRole.Topic + && consumer.Role is DestinationRole.Subscription or DestinationRole.Binding + && String.Equals(consumer.Topic, sent.Name, StringComparison.Ordinal); + } + + // Aggregate pending work across every destination/source this transport has seen; idle means nothing queued, + // nothing in flight, and no delayed redelivery still waiting on its timer. + public async Task> GetPendingAsync(CancellationToken ct = default) + { + var now = _timeProvider.GetUtcNow(); + var scheduled = new Dictionary(); + foreach (var redelivery in _pendingRedeliveries) + { + if (now >= redelivery.Value.DueAt + _redeliveryGrace) + _pendingRedeliveries.TryRemove(redelivery.Key, out _); + else + scheduled[redelivery.Value.Destination] = scheduled.GetValueOrDefault(redelivery.Value.Destination) + 1; + } + + var pending = new List<(string, long, long)>(); + foreach (var address in _knownNames.Keys.OrderBy(a => a.Key, StringComparer.Ordinal)) + { + var stats = await _inner.GetStatsAsync(address, ct).ConfigureAwait(false); + long queued = stats.Queued + scheduled.GetValueOrDefault(address); + long working = Math.Max(stats.Working, _processing.Values.LongCount(source => source == address)); + if (queued > 0 || working > 0) + pending.Add((address.Key, queued, working)); + } + + return pending; + } + + private static RecordedMessage Record(TransportEntry entry) => new() + { + Destination = entry.Destination.Key, + Role = DestinationRole.Queue, + MessageType = entry.Headers.GetValueOrDefault(KnownHeaders.MessageType), + Body = entry.Body, + Headers = entry.Headers, + Attempts = entry.DeliveryCount + }; +} diff --git a/src/Foundatio.Testing/TestingFoundatioBuilderExtensions.cs b/src/Foundatio.Testing/TestingFoundatioBuilderExtensions.cs new file mode 100644 index 000000000..135985ec3 --- /dev/null +++ b/src/Foundatio.Testing/TestingFoundatioBuilderExtensions.cs @@ -0,0 +1,46 @@ +using System; +using Foundatio.Jobs; +using Foundatio.Jobs.Testing; +using Foundatio.Messaging; +using Foundatio.Messaging.Testing; +using Foundatio.Serializer; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; + +namespace Foundatio; + +public static class TestingFoundatioBuilderExtensions +{ + /// + /// Runs messaging over a recording in-memory transport for tests. Resolve from + /// the container to await quiescence () and assert on the + /// messages that were sent, published, handled, retried, or dead-lettered. + /// + public static FoundatioBuilder.MessagingBuilder UseTestHarness(this FoundatioBuilder.MessagingBuilder builder) + { + var services = ((IFoundatioBuilder)builder).Services; + services.TryAddSingleton(sp => new MessagingTestHarness( + sp.GetService(), + sp.GetService(), + sp.GetService())); + return builder.UseTransport(sp => sp.GetRequiredService().Transport); + } + + /// + /// Runs jobs over the in-memory runtime without hosted workers, so tests drive execution explicitly. + /// Resolve from the container to enqueue jobs, tick schedules deterministically, + /// and run work to completion ( / + /// / ). + /// + public static FoundatioBuilder.JobsBuilder UseTestHarness(this FoundatioBuilder.JobsBuilder builder) + { + var services = ((IFoundatioBuilder)builder).Services; + services.TryAddSingleton(sp => new JobsTestHarness( + sp.GetRequiredService(), + sp.GetRequiredService(), + sp.GetRequiredService(), + sp.GetRequiredService(), + sp.GetRequiredService())); + return builder.UseInMemory(); + } +} diff --git a/src/Foundatio.Xunit.v3/Logging/TestLoggerBase.cs b/src/Foundatio.Xunit.v3/Logging/TestLoggerBase.cs index 58597e3bd..b675ce80b 100644 --- a/src/Foundatio.Xunit.v3/Logging/TestLoggerBase.cs +++ b/src/Foundatio.Xunit.v3/Logging/TestLoggerBase.cs @@ -27,7 +27,7 @@ protected TestLoggerBase(ITestOutputHelper output, TestLoggerFixture fixture) /// /// Gets a cancellation token that is cancelled when the current test completes or /// when the test run is aborted/timed out. Pass this token to - /// + /// /// and other async operations to ensure automatic cleanup between tests. /// /// diff --git a/src/Foundatio.Xunit.v3/Logging/TestWithLoggingBase.cs b/src/Foundatio.Xunit.v3/Logging/TestWithLoggingBase.cs index 9dc993a44..e96c6a8f4 100644 --- a/src/Foundatio.Xunit.v3/Logging/TestWithLoggingBase.cs +++ b/src/Foundatio.Xunit.v3/Logging/TestWithLoggingBase.cs @@ -22,7 +22,7 @@ protected TestWithLoggingBase(ITestOutputHelper output) /// /// Gets a cancellation token that is cancelled when the current test completes or /// when the test run is aborted/timed out. Pass this token to - /// + /// /// and other async operations to ensure automatic cleanup between tests. /// /// diff --git a/src/Foundatio.Xunit/Logging/TestLoggerBase.cs b/src/Foundatio.Xunit/Logging/TestLoggerBase.cs index 2e1b29c02..9adf3d4b4 100644 --- a/src/Foundatio.Xunit/Logging/TestLoggerBase.cs +++ b/src/Foundatio.Xunit/Logging/TestLoggerBase.cs @@ -26,7 +26,7 @@ protected TestLoggerBase(ITestOutputHelper output, TestLoggerFixture fixture) /// /// Gets a cancellation token that is cancelled when the current test completes. - /// Pass this token to + /// Pass this token to /// and other async operations to ensure automatic cleanup between tests. /// /// diff --git a/src/Foundatio.Xunit/Logging/TestWithLoggingBase.cs b/src/Foundatio.Xunit/Logging/TestWithLoggingBase.cs index 4189b92c0..c9afc52c9 100644 --- a/src/Foundatio.Xunit/Logging/TestWithLoggingBase.cs +++ b/src/Foundatio.Xunit/Logging/TestWithLoggingBase.cs @@ -21,7 +21,7 @@ protected TestWithLoggingBase(ITestOutputHelper output) /// /// Gets a cancellation token that is cancelled when the current test completes. - /// Pass this token to + /// Pass this token to /// and other async operations to ensure automatic cleanup between tests. /// /// diff --git a/src/Foundatio/Caching/HybridAwareCacheClient.cs b/src/Foundatio/Caching/HybridAwareCacheClient.cs index 81532fc98..c5ac4046e 100644 --- a/src/Foundatio/Caching/HybridAwareCacheClient.cs +++ b/src/Foundatio/Caching/HybridAwareCacheClient.cs @@ -23,14 +23,14 @@ public interface IHybridAwareCacheClient : ICacheClient public class HybridAwareCacheClient : IHybridAwareCacheClient, IHaveTimeProvider, IHaveLogger, IHaveLoggerFactory, IHaveResiliencePolicyProvider { protected readonly ICacheClient _distributedCache; - protected readonly IMessagePublisher _messagePublisher; + protected readonly IMessageBus _messagePublisher; private readonly string _cacheId = Guid.NewGuid().ToString("N"); private readonly ILogger _logger; private readonly ILoggerFactory _loggerFactory; private readonly TimeProvider _timeProvider; private readonly IResiliencePolicyProvider _resiliencePolicyProvider; - public HybridAwareCacheClient(ICacheClient distributedCacheClient, IMessagePublisher messagePublisher, ILoggerFactory? loggerFactory = null) + public HybridAwareCacheClient(ICacheClient distributedCacheClient, IMessageBus messagePublisher, ILoggerFactory? loggerFactory = null) { _loggerFactory = loggerFactory ?? distributedCacheClient.GetLoggerFactory() ?? NullLoggerFactory.Instance; _logger = _loggerFactory.CreateLogger(); diff --git a/src/Foundatio/Caching/HybridCacheClient.cs b/src/Foundatio/Caching/HybridCacheClient.cs index e248b49e1..197035118 100644 --- a/src/Foundatio/Caching/HybridCacheClient.cs +++ b/src/Foundatio/Caching/HybridCacheClient.cs @@ -29,6 +29,9 @@ public class HybridCacheClient : IHybridCacheClient, IHaveTimeProvider, IHaveLog private readonly IResiliencePolicyProvider _resiliencePolicyProvider; private readonly CancellationTokenSource _disposedCancellationTokenSource = new(); private readonly AsyncLazy _lazySubscription; + private IMessageSubscription? _invalidationSubscription; + private long _subscriptionVersion; + private readonly SemaphoreSlim _subscriptionRecovery = new(1, 1); private long _localCacheHits; private long _invalidateCacheCalls; private bool _isDisposed; @@ -41,10 +44,16 @@ public HybridCacheClient(ICacheClient distributedCacheClient, IMessageBus messag _resiliencePolicyProvider = distributedCacheClient.GetResiliencePolicyProvider() ?? localCacheOptions?.ResiliencePolicyProvider ?? DefaultResiliencePolicyProvider.Instance; _distributedCache = distributedCacheClient; _messageBus = messageBus; + if (!messageBus.SupportsTemporarySubscriptions) + throw new NotSupportedException("HybridCacheClient requires per-instance invalidation subscriptions. Use an in-memory or Redis messaging transport with TopologyMode.Ensure."); _lazySubscription = new AsyncLazy(async () => { - await _messageBus.SubscribeAsync( - OnRemoteCacheItemExpiredAsync, _disposedCancellationTokenSource.Token).AnyContext(); + // Invalidations are events every node must see: published-only (no queue channel) and per-instance so + // each hybrid client gets its own copy instead of instances competing for one. + _invalidationSubscription = await _messageBus.SubscribeAsync( + (context, _) => OnRemoteCacheItemExpiredAsync(context.Message), + new MessageSubscriptionOptions(), + _disposedCancellationTokenSource.Token).AnyContext(); return true; }, AsyncLazyFlags.RetryOnFailure | AsyncLazyFlags.ExecuteOnCallingThread); localCacheOptions ??= new InMemoryCacheClientOptions @@ -65,9 +74,23 @@ await _messageBus.SubscribeAsync( TimeProvider IHaveTimeProvider.TimeProvider => _timeProvider; IResiliencePolicyProvider IHaveResiliencePolicyProvider.ResiliencePolicyProvider => _resiliencePolicyProvider; - private Task EnsureSubscribedAsync() + private async Task EnsureSubscribedAsync() { - return _lazySubscription.Task; + await _lazySubscription.Task.AnyContext(); + await _invalidationSubscription!.WaitUntilReadyAsync(_disposedCancellationTokenSource.Token).AnyContext(); + long version = _invalidationSubscription.RecoveryVersion; + if (Interlocked.Read(ref _subscriptionVersion) == version) + return; + await _subscriptionRecovery.WaitAsync(_disposedCancellationTokenSource.Token).AnyContext(); + try + { + if (_subscriptionVersion != version) + { + await _localCache.RemoveAllAsync().AnyContext(); + Interlocked.Exchange(ref _subscriptionVersion, version); + } + } + finally { _subscriptionRecovery.Release(); } } private Task OnRemoteCacheItemExpiredAsync(InvalidateCache message) @@ -806,6 +829,7 @@ public virtual void Dispose() _isDisposed = true; _disposedCancellationTokenSource.Cancel(); _disposedCancellationTokenSource.Dispose(); + _invalidationSubscription?.DisposeAsync().AsTask().GetAwaiter().GetResult(); _localCache.Dispose(); } diff --git a/src/Foundatio.Extensions.Hosting/Cronos/CalendarHelper.cs b/src/Foundatio/Cronos/CalendarHelper.cs similarity index 99% rename from src/Foundatio.Extensions.Hosting/Cronos/CalendarHelper.cs rename to src/Foundatio/Cronos/CalendarHelper.cs index c447b746b..57630deca 100644 --- a/src/Foundatio.Extensions.Hosting/Cronos/CalendarHelper.cs +++ b/src/Foundatio/Cronos/CalendarHelper.cs @@ -5,7 +5,7 @@ using System; using System.Runtime.CompilerServices; -namespace Foundatio.Extensions.Hosting.Cronos; +namespace Foundatio.Cronos; internal static class CalendarHelper { diff --git a/src/Foundatio.Extensions.Hosting/Cronos/CronExpression.cs b/src/Foundatio/Cronos/CronExpression.cs similarity index 99% rename from src/Foundatio.Extensions.Hosting/Cronos/CronExpression.cs rename to src/Foundatio/Cronos/CronExpression.cs index 9068ed36d..0518952e3 100644 --- a/src/Foundatio.Extensions.Hosting/Cronos/CronExpression.cs +++ b/src/Foundatio/Cronos/CronExpression.cs @@ -26,7 +26,7 @@ using System.Runtime.CompilerServices; using System.Text; -namespace Foundatio.Extensions.Hosting.Cronos; +namespace Foundatio.Cronos; /// /// Provides a parser and scheduler for cron expressions. diff --git a/src/Foundatio.Extensions.Hosting/Cronos/CronExpressionFlag.cs b/src/Foundatio/Cronos/CronExpressionFlag.cs similarity index 96% rename from src/Foundatio.Extensions.Hosting/Cronos/CronExpressionFlag.cs rename to src/Foundatio/Cronos/CronExpressionFlag.cs index df7b2fc7e..74b7bba04 100644 --- a/src/Foundatio.Extensions.Hosting/Cronos/CronExpressionFlag.cs +++ b/src/Foundatio/Cronos/CronExpressionFlag.cs @@ -22,7 +22,7 @@ using System; -namespace Foundatio.Extensions.Hosting.Cronos; +namespace Foundatio.Cronos; [Flags] internal enum CronExpressionFlag : byte diff --git a/src/Foundatio.Extensions.Hosting/Cronos/CronField.cs b/src/Foundatio/Cronos/CronField.cs similarity index 98% rename from src/Foundatio.Extensions.Hosting/Cronos/CronField.cs rename to src/Foundatio/Cronos/CronField.cs index 44c9fe8ef..5065f0a01 100644 --- a/src/Foundatio.Extensions.Hosting/Cronos/CronField.cs +++ b/src/Foundatio/Cronos/CronField.cs @@ -20,7 +20,7 @@ // OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE // SOFTWARE. -namespace Foundatio.Extensions.Hosting.Cronos; +namespace Foundatio.Cronos; internal sealed class CronField { diff --git a/src/Foundatio.Extensions.Hosting/Cronos/CronFormat.cs b/src/Foundatio/Cronos/CronFormat.cs similarity index 97% rename from src/Foundatio.Extensions.Hosting/Cronos/CronFormat.cs rename to src/Foundatio/Cronos/CronFormat.cs index 2e762e5e6..776e2684a 100644 --- a/src/Foundatio.Extensions.Hosting/Cronos/CronFormat.cs +++ b/src/Foundatio/Cronos/CronFormat.cs @@ -22,7 +22,7 @@ using System; -namespace Foundatio.Extensions.Hosting.Cronos; +namespace Foundatio.Cronos; /// /// Defines the cron format options that customize string parsing for . diff --git a/src/Foundatio.Extensions.Hosting/Cronos/CronFormatException.cs b/src/Foundatio/Cronos/CronFormatException.cs similarity index 97% rename from src/Foundatio.Extensions.Hosting/Cronos/CronFormatException.cs rename to src/Foundatio/Cronos/CronFormatException.cs index 7df867527..fe13770a8 100644 --- a/src/Foundatio.Extensions.Hosting/Cronos/CronFormatException.cs +++ b/src/Foundatio/Cronos/CronFormatException.cs @@ -22,7 +22,7 @@ using System; -namespace Foundatio.Extensions.Hosting.Cronos; +namespace Foundatio.Cronos; /// /// Represents an exception that's thrown, when invalid Cron expression is given. diff --git a/src/Foundatio.Extensions.Hosting/Cronos/TimeZoneHelper.cs b/src/Foundatio/Cronos/TimeZoneHelper.cs similarity index 99% rename from src/Foundatio.Extensions.Hosting/Cronos/TimeZoneHelper.cs rename to src/Foundatio/Cronos/TimeZoneHelper.cs index 587cfa248..4eaef9ef0 100644 --- a/src/Foundatio.Extensions.Hosting/Cronos/TimeZoneHelper.cs +++ b/src/Foundatio/Cronos/TimeZoneHelper.cs @@ -22,7 +22,7 @@ using System; -namespace Foundatio.Extensions.Hosting.Cronos; +namespace Foundatio.Cronos; internal static class TimeZoneHelper { diff --git a/src/Foundatio/Foundatio.csproj b/src/Foundatio/Foundatio.csproj index 04a1ec8d7..3fb6df955 100644 --- a/src/Foundatio/Foundatio.csproj +++ b/src/Foundatio/Foundatio.csproj @@ -1,9 +1,16 @@ + + + true + + + diff --git a/src/Foundatio/FoundatioServicesExtensions.cs b/src/Foundatio/FoundatioServicesExtensions.cs index 07b184b78..00b68b82c 100644 --- a/src/Foundatio/FoundatioServicesExtensions.cs +++ b/src/Foundatio/FoundatioServicesExtensions.cs @@ -1,14 +1,20 @@ using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; using Foundatio.Caching; using Foundatio.Extensions; +using Foundatio.Jobs; using Foundatio.Lock; using Foundatio.Messaging; -using Foundatio.Queues; using Foundatio.Resilience; using Foundatio.Serializer; using Foundatio.Storage; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Hosting; +using Legacy = Foundatio.Messaging.Legacy; namespace Foundatio; @@ -21,6 +27,7 @@ public static class FoundatioServicesExtensions /// public static FoundatioBuilder AddFoundatio(this IServiceCollection services) { + ArgumentNullException.ThrowIfNull(services); return new FoundatioBuilder(services); } } @@ -35,7 +42,7 @@ internal FoundatioBuilder(IServiceCollection services) Caching = new CachingBuilder(this); Storage = new StorageBuilder(this); Messaging = new MessagingBuilder(this); - Queueing = new QueueingBuilder(this); + Jobs = new JobsBuilder(this); Locking = new LockingBuilder(this); } @@ -58,9 +65,9 @@ internal FoundatioBuilder(IServiceCollection services) public MessagingBuilder Messaging { get; } /// - /// Configure queueing services for Foundatio. + /// Configure background job runtime services for Foundatio. /// - public QueueingBuilder Queueing { get; } + public JobsBuilder Jobs { get; } /// /// Configure locking services for Foundatio. @@ -137,6 +144,30 @@ public FoundatioBuilder AddSerializer(ITextSerializer textSerializer, ISerialize return this; } + /// Configures messaging in one feature block. + public FoundatioBuilder ConfigureMessaging(Action configure) + { + ArgumentNullException.ThrowIfNull(configure); + configure(Messaging); + return this; + } + + /// Configures durable jobs in one feature block. + public FoundatioBuilder ConfigureJobs(Action configure) + { + ArgumentNullException.ThrowIfNull(configure); + configure(Jobs); + return this; + } + + /// Stable service identity used for default durable event subscriptions. + public FoundatioBuilder UseServiceName(string name) + { + ArgumentException.ThrowIfNullOrWhiteSpace(name); + _services.ReplaceSingleton(_ => new FoundatioServiceIdentity(name)); + return this; + } + public class CachingBuilder : IFoundatioBuilder { private readonly FoundatioBuilder _builder; @@ -237,6 +268,9 @@ public class MessagingBuilder : IFoundatioBuilder { private readonly FoundatioBuilder _builder; private readonly IServiceCollection _services; + private bool _routingServicesRegistered; + private bool _topologyServicesRegistered; + private TopologyMode _topologyMode = TopologyMode.Ensure; internal MessagingBuilder(IFoundatioBuilder builder) { @@ -244,78 +278,390 @@ internal MessagingBuilder(IFoundatioBuilder builder) _services = builder.Services; } + /// + /// Selects how the messaging client administers topology: creates missing + /// destinations on use and at handler-host startup (default), only checks + /// they exist and throws when missing, and never touches topology. + /// + public MessagingBuilder ConfigureTopology(TopologyMode mode) + { + if (!Enum.IsDefined(mode)) + throw new ArgumentOutOfRangeException(nameof(mode)); + _topologyMode = mode; + return this; + } + + /// Returns the root builder for configuring another feature. + public FoundatioBuilder Builder => _builder; + IServiceCollection IFoundatioBuilder.Services => _services; FoundatioBuilder IFoundatioBuilder.Builder => _builder; - public FoundatioBuilder Use(IMessageBus messageBus) + /// + /// Registers the legacy // + /// interfaces as a thin adapter over the redesigned + /// , so existing consuming code keeps compiling while it migrates. There is no + /// legacy bus behind it — remove this call once call sites are on the new API. + /// + public MessagingBuilder AddLegacyAdapter() { - _services.ReplaceSingleton(_ => messageBus); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); - return _builder; + _services.ReplaceSingleton(sp => new Legacy.LegacyMessageBusAdapter(sp.GetRequiredService())); + _services.ReplaceSingleton(sp => sp.GetRequiredService()); + _services.ReplaceSingleton(sp => sp.GetRequiredService()); + return this; + } + + public MessagingBuilder ConfigureRouting(Action configure) + { + ArgumentNullException.ThrowIfNull(configure); + + _services.AddSingleton>(configure); + RegisterRoutingServices(); + return this; + } + + // The core owns retry and dead-letter behavior so it is identical across transports. This configures the + // default policy applied to queue and pub/sub consumers; a consumer can still override MaxAttempts/backoff. + public MessagingBuilder ConfigureRetry(RetryPolicy policy) + { + ArgumentNullException.ThrowIfNull(policy); + _services.ReplaceSingleton(_ => policy); + return this; + } + + public MessagingBuilder ConfigureRetry(Func configure) + { + ArgumentNullException.ThrowIfNull(configure); + return ConfigureRetry(configure(new RetryPolicy())); } - public FoundatioBuilder Use(Func factory) + // Registers a stable wire name for a message type so the discriminator survives assembly/namespace moves and + // grouped/interface consumers can resolve and deserialize the concrete payload type. + public MessagingBuilder AddMessageType(string name, string? queue = null, string? topic = null) where T : class { + ArgumentException.ThrowIfNullOrEmpty(name); + _services.AddSingleton(new MessageTypeRegistration(name, typeof(T))); + if (queue is not null) ConfigureRouting(r => r.MapQueue(queue)); + if (topic is not null) ConfigureRouting(r => r.MapTopic(topic)); + return this; + } + + /// Uses the in-memory transport — the all-defaults setup for development and tests. + public MessagingBuilder UseInMemory(JobRuntimeStoreOptions? scheduling = null) + { + _services.TryAddSingleton(sp => sp.GetService() ?? new InMemoryJobRuntimeStore(scheduling ?? new(), sp.GetService())); + RegisterMessagingRuntime(sp => new InMemoryMessageTransport(sp.GetService(), sp.GetService())); + return this; + } + + /// Configures delayed messaging without registering job execution services. + public MessagingBuilder UseSchedulingStore(Func factory) + { + ArgumentNullException.ThrowIfNull(factory); _services.ReplaceSingleton(factory); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); - return _builder; + return this; } - public FoundatioBuilder UseInMemory(InMemoryMessageBusOptions? options = null) + public MessagingBuilder UseTransport(IMessageTransport transport) { - _services.ReplaceSingleton(sp => new InMemoryMessageBus(options.UseServices(sp))); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); - return _builder; + ArgumentNullException.ThrowIfNull(transport); + RegisterMessagingRuntime(_ => transport); + return this; } - public FoundatioBuilder UseInMemory(Builder config) + public MessagingBuilder UseTransport(Func factory) { - _services.ReplaceSingleton(sp => new InMemoryMessageBus(b => b.Configure(config).UseServices(sp))); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); - return _builder; + ArgumentNullException.ThrowIfNull(factory); + RegisterMessagingRuntime(factory); + return this; + } + + /// Runs queued work in a scoped handler. Replicas compete for the same queue. + public MessagingBuilder AddConsumer(Action? configure = null) + where TMessage : class where THandler : class, IMessageHandler + { + _services.TryAddScoped(); + return AddConsumer((sp, message, ct) => DispatchAsync(sp, message, ct), configure); + } + + /// Runs queued work in a delegate handler. + public MessagingBuilder AddConsumer(Func, CancellationToken, Task> handler, Action? configure = null) + where TMessage : class + { + ArgumentNullException.ThrowIfNull(handler); + return AddConsumer((_, message, ct) => handler(message, ct), configure); + } + + private MessagingBuilder AddConsumer(Func, CancellationToken, Task> dispatch, Action? configure) + where TMessage : class + { + var options = new MessageConsumerOptions(); + configure?.Invoke(options); + options.Validate(); + if (options.MessageTypeName is { } wireName) AddMessageType(wireName, queue: options.Destination); + else if (options.Destination is not null) ConfigureRouting(r => r.MapQueue(options.Destination)); + return AddHandlerRegistration($"consumer:{typeof(TMessage).Name}", (sp, ct) => + sp.GetRequiredService().ConsumeAsync((message, token) => dispatch(sp, message, token), options, ct)); + } + + /// + /// Receives published events in a scoped handler. Supply a stable subscription name for durable delivery + /// shared by replicas. Use AddTemporarySubscriber for a temporary subscription on every instance. + /// + public MessagingBuilder AddSubscriber(string? subscription = null, Action? configure = null) + where TMessage : class where THandler : class, IMessageHandler + { + _services.TryAddScoped(); + return AddSubscriber((sp, message, ct) => DispatchAsync(sp, message, ct), subscription, configure); + } + + /// Receives published events in a delegate handler on a named durable subscription. + public MessagingBuilder AddSubscriber(Func, CancellationToken, Task> handler, string? subscription = null, Action? configure = null) + where TMessage : class + { + ArgumentNullException.ThrowIfNull(handler); + return AddSubscriber((_, message, ct) => handler(message, ct), subscription, configure); + } + + /// Receives a copy of each event for this process using an expiring subscription. Requires provider support. + public MessagingBuilder AddTemporarySubscriber(Action? configure = null) + where TMessage : class where THandler : class, IMessageHandler + { + _services.TryAddScoped(); + return AddSubscriber((sp, message, ct) => DispatchAsync(sp, message, ct), null, configure, temporary: true); + } + + /// Receives events in a delegate using an expiring subscription. Requires provider support. + public MessagingBuilder AddTemporarySubscriber(Func, CancellationToken, Task> handler, Action? configure = null) + where TMessage : class + { + ArgumentNullException.ThrowIfNull(handler); + return AddSubscriber((_, message, ct) => handler(message, ct), null, configure, temporary: true); + } + + private MessagingBuilder AddSubscriber(Func, CancellationToken, Task> dispatch, string? subscription, Action? configure, bool temporary = false) + where TMessage : class + { + if (subscription is not null) + ArgumentException.ThrowIfNullOrWhiteSpace(subscription); + var options = new MessageSubscriptionOptions { Subscription = subscription }; + configure?.Invoke(options); + options.Validate(); + if (options.Subscription != subscription) + throw new ArgumentException("Set the durable name with the subscription argument. Use AddTemporarySubscriber for a temporary subscription.", nameof(configure)); + if (options.MessageTypeName is { } wireName) AddMessageType(wireName, topic: options.Topic); + else if (options.Topic is not null) ConfigureRouting(r => r.MapTopic(options.Topic)); + return AddHandlerRegistration($"subscriber:{typeof(TMessage).Name}", (sp, ct) => + { + var subscriptionOptions = options.Copy(); + if (!temporary && subscriptionOptions.Subscription is null) + subscriptionOptions.Subscription = sp.GetService()?.Name ?? sp.GetService()?.ApplicationName + ?? throw new InvalidOperationException("A default durable subscription requires UseServiceName(...), a hosting ApplicationName, or an explicit subscription name."); + return sp.GetRequiredService().SubscribeAsync((message, token) => dispatch(sp, message, token), subscriptionOptions, ct); + }); + } + + private MessagingBuilder AddHandlerRegistration(string description, Func> start) + { + _services.AddSingleton(new MessageHandlerRegistration + { + Description = description, + StartAsync = async (sp, ct) => await start(sp, ct).ConfigureAwait(false) + }); + return this; + } + + private static async Task DispatchAsync(IServiceProvider serviceProvider, IMessageContext message, CancellationToken cancellationToken) + where TMessage : class where THandler : class, IMessageHandler + { + await using var scope = serviceProvider.CreateAsyncScope(); + var handler = scope.ServiceProvider.GetRequiredService(); + await handler.HandleAsync(message, cancellationToken).ConfigureAwait(false); + } + + private void RegisterMessagingRuntime(Func factory) + { + _services.ReplaceSingleton(factory); + // Resolved lazily so ConfigureTopology can be called before or after the Use* transport registration. + _services.ReplaceSingleton(_ => new MessagingTopologyOptions(_topologyMode)); + RegisterMessageTopology(); + RegisterMessageClients(); + + } + + private void RegisterRoutingServices() + { + if (_routingServicesRegistered) + return; + + _routingServicesRegistered = true; + _services.ReplaceSingleton(sp => + { + var options = new MessageRoutingOptions(); + var builder = new MessageRoutingOptionsBuilder(options); + foreach (var configure in sp.GetServices>()) + configure(builder); + + return options; + }); + _services.ReplaceSingleton(sp => new DefaultMessageRouter(sp.GetRequiredService())); + } + + private void RegisterMessageTopology() + { + RegisterRoutingServices(); + + if (_topologyServicesRegistered) + return; + + _topologyServicesRegistered = true; + _services.ReplaceSingleton(sp => new MessageTopology( + sp.GetRequiredService(), + sp.GetRequiredService())); + } + + private void RegisterMessageClients() + { + RegisterRoutingServices(); + _services.ReplaceSingleton(sp => new MessageTypeRegistry(sp.GetServices())); + _services.ReplaceSingleton(sp => new MessageBus(sp.GetRequiredService(), new MessageBusOptions + { + Serializer = sp.GetService() ?? DefaultSerializer.Instance, + Router = sp.GetService() ?? DefaultMessageRouter.Instance, + MessageTypes = sp.GetService() ?? new MessageTypeRegistry(), + RuntimeStore = sp.GetService() ?? sp.GetService(), + RetryPolicy = sp.GetService() ?? new RetryPolicy(), + Topology = sp.GetService()?.Mode ?? TopologyMode.Ensure, + // The transport is a shared DI singleton owned by the container; the bus must not dispose it. + OwnsTransport = false, + TimeProvider = sp.GetService() ?? TimeProvider.System, + LoggerFactory = sp.GetService() + })); } } - public class QueueingBuilder : IFoundatioBuilder + public class JobsBuilder : IFoundatioBuilder { private readonly FoundatioBuilder _builder; private readonly IServiceCollection _services; - internal QueueingBuilder(IFoundatioBuilder builder) + internal JobsBuilder(IFoundatioBuilder builder) { _builder = builder.Builder; _services = builder.Services; } + /// Returns the root builder for configuring another feature. + public FoundatioBuilder Builder => _builder; + IServiceCollection IFoundatioBuilder.Services => _services; FoundatioBuilder IFoundatioBuilder.Builder => _builder; - public FoundatioBuilder Use(IQueue storage) where T : class + public JobsBuilder UseRuntimeStore(IJobRuntimeStore store) { - _services.ReplaceSingleton(_ => storage); - return _builder; + ArgumentNullException.ThrowIfNull(store); + _services.ReplaceSingleton(_ => store); + RegisterJobServices(); + return this; } - public FoundatioBuilder Use(Func> factory) where T : class + public JobsBuilder UseRuntimeStore(Func factory) { + ArgumentNullException.ThrowIfNull(factory); _services.ReplaceSingleton(factory); - return _builder; + RegisterJobServices(); + return this; } - public FoundatioBuilder UseInMemory(InMemoryQueueOptions? options = null) where T : class + /// Uses the in-memory job runtime — the all-defaults setup for development and tests. + public JobsBuilder UseInMemory(JobRuntimeStoreOptions? options = null) { - _services.ReplaceSingleton>(sp => new InMemoryQueue(options.UseServices(sp))); - return _builder; + _services.ReplaceSingleton(sp => new InMemoryJobRuntimeStore(options ?? new(), sp.GetService())); + RegisterJobServices(); + return this; } - public FoundatioBuilder UseInMemory(Builder, InMemoryQueueOptions> config) where T : class + /// Configures execution slots, stable node identity, lease and polling settings. + public JobsBuilder ConfigureWorker(Func configure) { - _services.ReplaceSingleton>(sp => new InMemoryQueue(b => b.Configure(config).UseServices(sp))); - return _builder; + ArgumentNullException.ThrowIfNull(configure); + var existing = _services.LastOrDefault(d => d.ServiceType == typeof(JobWorkerOptions))?.ImplementationInstance as JobWorkerOptions ?? new(); + var options = configure(existing); + ArgumentOutOfRangeException.ThrowIfLessThan(options.MaxConcurrency, 1); + if (options.NodeId is not null) ArgumentException.ThrowIfNullOrWhiteSpace(options.NodeId); + _services.Replace(ServiceDescriptor.Singleton(options)); + return this; + } + + public JobsBuilder AddJobType(string? name = null) where TJob : IJob + { + JobArgumentContract.ValidateType(typeof(TJob)); + if (name is not null) + ArgumentException.ThrowIfNullOrWhiteSpace(name); + _services.TryAddScoped(typeof(TJob)); + _services.AddSingleton(new JobTypeRegistration(name ?? typeof(TJob).FullName ?? typeof(TJob).Name, typeof(TJob))); + return this; + } + + /// + /// Registers a recurring (CRON) job. The schedule is materialized once into the shared runtime store per + /// occurrence, so decides fan-out (Global = one instance per tick, + /// PerNode = every instance per tick). Scheduled when the job scheduler starts — no manual + /// call needed. Requires a runtime store ( + /// / ). + /// + public JobsBuilder AddCronJob(string cronSchedule, Action? configure = null) where TJob : IJob + => AddCronJob(typeof(TJob), cronSchedule, null, configure); + + /// Declares a recurring job with arguments constrained to its typed job contract. + public JobsBuilder AddCronJob(string cronSchedule, TArgs arguments, Action? configure = null) + where TJob : IJob where TArgs : class + => AddCronJob(typeof(TJob), cronSchedule, arguments, configure); + + private JobsBuilder AddCronJob(Type jobType, string cronSchedule, object? arguments, Action? configure) + { + ArgumentException.ThrowIfNullOrWhiteSpace(cronSchedule); + JobScheduleProcessor.ValidateCron(cronSchedule); + JobArgumentContract.Validate(jobType, arguments); + var options = new CronJobOptions(); + configure?.Invoke(options); + var registration = new ScheduledJobRegistration(jobType, cronSchedule, options, arguments); + registration.Validate(); + if (_services.Any(d => d.ImplementationInstance is ScheduledJobRegistration existing && existing.Name == registration.Name)) + throw new InvalidOperationException($"A CRON job named {registration.Name} is already registered. Give one an explicit CronJobOptions.Name."); + if (!_services.Any(d => d.ImplementationInstance is JobTypeRegistration existing && existing.JobType == jobType)) + _services.AddSingleton(new JobTypeRegistration(jobType.FullName ?? jobType.Name, jobType)); + _services.TryAddScoped(jobType); + _services.AddSingleton(registration); + _services.AddSingleton(sp => registration.Create(sp.GetRequiredService(), sp.GetService() ?? DefaultSerializer.Instance)); + return this; + } + + private void RegisterJobServices() + { + _services.ReplaceSingleton(sp => new JobTypeRegistry(sp.GetServices())); + _services.ReplaceSingleton(sp => sp.GetRequiredService()); + _services.ReplaceSingleton(sp => new JobClient(sp.GetRequiredService(), sp.GetService(), sp.GetRequiredService(), sp.GetService())); + _services.ReplaceSingleton(sp => + { + var options = sp.GetService() ?? new(); + return new JobWorker(sp.GetRequiredService(), sp, options with + { + TimeProvider = options.TimeProvider ?? sp.GetService(), + JobTypes = options.JobTypes ?? sp.GetRequiredService(), + Serializer = options.Serializer ?? sp.GetService() + }); + }); + _services.ReplaceSingleton(sp => sp.GetRequiredService()); + _services.ReplaceSingleton(sp => new ScheduledJobManager( + sp.GetRequiredService(), + sp.GetRequiredService(), + sp.GetRequiredService(), + sp.GetService(), + sp.GetService(), sp.GetService()?.NodeId)); + _services.ReplaceSingleton(sp => new JobScheduleProcessor(sp.GetRequiredService(), sp.GetRequiredService(), new JobScheduleProcessorOptions { TimeProvider = sp.GetService(), NodeId = sp.GetService()?.NodeId })); + + } } @@ -373,3 +719,6 @@ public interface IFoundatioBuilder IServiceCollection Services { get; } FoundatioBuilder Builder { get; } } + +/// Stable application identity for durable subscription defaults. +public sealed record FoundatioServiceIdentity(string Name); diff --git a/src/Foundatio/Jobs/IJob.cs b/src/Foundatio/Jobs/IJob.cs index ac5cefd10..1c2120a52 100644 --- a/src/Foundatio/Jobs/IJob.cs +++ b/src/Foundatio/Jobs/IJob.cs @@ -1,46 +1,46 @@ using System; -using System.Linq; -using System.Threading; using System.Threading.Tasks; using Foundatio.Utility; -using Microsoft.Extensions.Logging; namespace Foundatio.Jobs; /// -/// Represents a unit of background work that can be executed once or continuously. -/// Implement this interface to create custom jobs for scheduled tasks, queue processing, or maintenance operations. +/// Represents a unit of background work run by the durable job runtime. Every run is handed a +/// carrying its cancellation token, identity, attempt number, and store-backed +/// progress/heartbeat helpers — a job uses what it needs and ignores the rest. /// public interface IJob { /// /// Executes the job's work. /// - /// Token to signal that the job should stop. + /// The execution context for this run (cancellation, identity, progress, heartbeat). /// A result indicating success, failure, or cancellation. - Task RunAsync(CancellationToken cancellationToken = default); + Task RunAsync(JobExecutionContext context); } -/// -/// A job that exposes configurable options for execution behavior. -/// -public interface IJobWithOptions : IJob +/// A job whose required argument contract is checked when it is submitted. +public interface IJob : IJob where TArgs : class { - /// - /// Gets or sets the options controlling job execution (name, interval, iteration limit). - /// - JobOptions? Options { get; set; } + /// Executes with the deserialized arguments and the current execution context. + Task RunAsync(TArgs arguments, JobExecutionContext context); + + Task IJob.RunAsync(JobExecutionContext context) + => RunAsync(context.GetArguments(), context); } public static class JobExtensions { - public static async Task TryRunAsync(this IJob job, CancellationToken cancellationToken = default) + /// + /// Runs the job, converting cancellation and unhandled exceptions into a instead of throwing. + /// + public static async Task TryRunAsync(this IJob job, JobExecutionContext context) { try { - return await job.RunAsync(cancellationToken).AnyContext(); + return await job.RunAsync(context).AnyContext(); } - catch (OperationCanceledException) + catch (OperationCanceledException) when (context.CancellationToken.IsCancellationRequested) { return JobResult.Cancelled; } @@ -49,97 +49,4 @@ public static async Task TryRunAsync(this IJob job, CancellationToken return JobResult.FromException(ex); } } - - /// - /// Runs the job continuously until the cancellation token is set or the iteration limit is reached. - /// - /// Returns the iteration count for normal jobs. For queue-based jobs this will be the number of items processed successfully. - public static Task RunContinuousAsync(this IJob job, TimeSpan? interval = null, int iterationLimit = -1, - CancellationToken cancellationToken = default, Func>? continuationCallback = null) - { - var options = JobOptions.GetDefaults(job); - options.Interval = interval; - options.IterationLimit = iterationLimit; - return RunContinuousAsync(job, options, cancellationToken, continuationCallback); - } - - /// - /// Runs the job continuously until the cancellation token is set or the iteration limit is reached. - /// - /// Returns the iteration count for normal jobs. For queue based jobs this will be the amount of items processed successfully. - public static async Task RunContinuousAsync(this IJob job, JobOptions options, CancellationToken cancellationToken = default, Func>? continuationCallback = null) - { - int iterations = 0; - var logger = job.GetLogger(); - - int queueItemsProcessed = 0; - bool isQueueJob = job.GetType().GetInterfaces().Any(x => x.IsGenericType && x.GetGenericTypeDefinition() == typeof(IQueueJob<>)); - - string jobId = Guid.NewGuid().ToString("N").Substring(0, 10); - using var jobScope = logger.BeginScope(s => s.Property("job.name", options.Name ?? String.Empty).Property("job.id", jobId)); - logger.LogInformation("Starting continuous job type {JobName} on machine {MachineName}...", options.Name, Environment.MachineName); - - while (!cancellationToken.IsCancellationRequested) - { - using var activity = FoundatioDiagnostics.ActivitySource.StartActivity($"Job: {options.Name}"); - - string jobRunId = Guid.NewGuid().ToString("N").Substring(0, 10); - using var _ = logger.BeginScope(s => s.Property("job.run_id", jobRunId)); - var result = await job.TryRunAsync(cancellationToken).AnyContext(); - logger.LogJobResult(result, options.Name); - - iterations++; - if (isQueueJob && result.IsSuccess) - queueItemsProcessed++; - - if (cancellationToken.IsCancellationRequested || (options.IterationLimit > -1 && options.IterationLimit <= iterations)) - break; - - if (result.Error != null) - { - await job.GetTimeProvider().SafeDelay(TimeSpan.FromMilliseconds(Math.Max((int)(options.Interval?.TotalMilliseconds ?? 0), 100)), cancellationToken).AnyContext(); - } - else if (options.Interval.HasValue && options.Interval.Value > TimeSpan.Zero) - { - await job.GetTimeProvider().SafeDelay(options.Interval.Value, cancellationToken).AnyContext(); - } - - // needed to yield back a task for jobs that aren't async - await Task.Yield(); - - if (cancellationToken.IsCancellationRequested) - break; - - if (continuationCallback is null) - continue; - - try - { - if (!await continuationCallback().AnyContext()) - break; - } - catch (Exception ex) - { - logger.LogError(ex, "Error in continuation callback: {Message}", ex.Message); - } - } - - if (cancellationToken.IsCancellationRequested) - logger.LogTrace("Job cancellation requested"); - - if (options.IterationLimit > 0) - { - logger.LogInformation( - "Stopping continuous job type {JobName} on machine {MachineName}: Job ran {Iterations} times (Limit={IterationLimit})", - options.Name, Environment.MachineName, iterations, options.IterationLimit); - } - else - { - logger.LogInformation( - "Stopping continuous job type {JobName} on machine {MachineName}: Job ran {Iterations} times", - options.Name, Environment.MachineName, iterations); - } - - return isQueueJob ? queueItemsProcessed : iterations; - } } diff --git a/src/Foundatio/Jobs/IQueueJob.cs b/src/Foundatio/Jobs/IQueueJob.cs deleted file mode 100644 index 2f396e3b2..000000000 --- a/src/Foundatio/Jobs/IQueueJob.cs +++ /dev/null @@ -1,69 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Queues; -using Foundatio.Utility; -using Microsoft.Extensions.Logging; - -namespace Foundatio.Jobs; - -/// -/// A job that processes items from a queue. Each invocation of -/// dequeues and processes a single item. -/// -/// The type of message payload in the queue. -public interface IQueueJob : IJob where T : class -{ - /// - /// Processes a single queue entry. Called by after dequeuing an item. - /// Can also be called directly when the queue entry is obtained externally. - /// - /// The queue entry to process. - /// Token to signal that processing should stop. - /// A result indicating success or failure of processing. - Task ProcessAsync(IQueueEntry queueEntry, CancellationToken cancellationToken); - - /// - /// Gets the queue this job processes items from. - /// - IQueue Queue { get; } -} - -public static class QueueJobExtensions -{ - /// - /// Will run until the queue is empty or the wait time is exceeded. - /// - /// The amount of queue items processed. - public static async Task RunUntilEmptyAsync(this IQueueJob job, TimeSpan waitTimeout, - CancellationToken cancellationToken = default) where T : class - { - if (waitTimeout <= TimeSpan.Zero) - throw new ArgumentException("Acquire timeout must be greater than zero", nameof(waitTimeout)); - - using var linkedCancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); - linkedCancellationTokenSource.CancelAfter(waitTimeout); - - // NOTE: This has to be awaited otherwise the linkedCancellationTokenSource cancel timer will not fire. - return await job.RunUntilEmptyAsync(linkedCancellationTokenSource.Token).AnyContext(); - } - - /// - /// Will wait up to thirty seconds if queue is empty, otherwise will run until the queue is empty or cancelled. - /// - /// The amount of queue items processed. - public static Task RunUntilEmptyAsync(this IQueueJob job, CancellationToken cancellationToken = default) where T : class - { - var logger = job.GetLogger(); - - return job.RunContinuousAsync(cancellationToken: cancellationToken, continuationCallback: async () => - { - // Allow abandoned items to be added in a background task. - Thread.Yield(); - - var stats = await job.Queue.GetQueueStatsAsync().AnyContext(); - logger.LogTrace("RunUntilEmpty continuation: Queued={Queued}, Working={Working}, Abandoned={Abandoned}", stats.Queued, stats.Working, stats.Abandoned); - return stats.Queued + stats.Working > 0; - }); - } -} diff --git a/src/Foundatio/Jobs/InMemoryJobRuntimeStore.Broker.cs b/src/Foundatio/Jobs/InMemoryJobRuntimeStore.Broker.cs new file mode 100644 index 000000000..fb65bbf6b --- /dev/null +++ b/src/Foundatio/Jobs/InMemoryJobRuntimeStore.Broker.cs @@ -0,0 +1,127 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Jobs; + +public sealed partial class InMemoryJobRuntimeStore +{ + public bool IsShared => false; + private readonly SortedSet<(DateTimeOffset Expires, string Id)> _brokerExpiry = new(); + private readonly Dictionary<(string Name, DateTimeOffset Hour), Dictionary> _counters = new(); + + public Task BeginBrokerAttemptAsync(string jobId, int attempt, string nodeId, CancellationToken cancellationToken = default) + { + ArgumentOutOfRangeException.ThrowIfLessThan(attempt, 1); + ArgumentException.ThrowIfNullOrWhiteSpace(nodeId); + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + PurgeBrokerHistory(); + if (!_jobs.TryGetValue(jobId, out var state) || state.ExecutionOwner != JobExecutionOwner.Broker || !IsActive(state) || attempt <= state.Attempt) + return Task.FromResult(null); + var now = _timeProvider.GetUtcNow(); + StoreJob(state with + { + Status = JobStatus.Processing, + Attempt = attempt, + ClaimToken = Guid.NewGuid().ToString("N"), + NodeId = nodeId, + StartedUtc = now, + LastHeartbeatUtc = now, + LastUpdatedUtc = now, + CompletedUtc = null, + Progress = 0, + ProgressMessage = null + }); + return Task.FromResult(_jobs[jobId]); + } + } + + public Task MarkEnqueueUnknownAsync(string jobId, string error, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + PurgeBrokerHistory(); + if (!_jobs.TryGetValue(jobId, out var state) || state.ExecutionOwner != JobExecutionOwner.Broker || state.Status != JobStatus.Queued || state.Attempt != 0) + return Task.FromResult(false); + StoreJob(state with { Status = JobStatus.EnqueueUnknown, Error = error, LastUpdatedUtc = _timeProvider.GetUtcNow() }); + return Task.FromResult(true); + } + } + + public Task HeartbeatJobAsync(string jobId, string claimToken, CancellationToken cancellationToken = default) + => ReportJobProgressAsync(jobId, claimToken, cancellationToken: cancellationToken); + + public Task RemoveAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + PurgeBrokerHistory(); + if (!_jobs.TryGetValue(jobId, out var state) || state.ExecutionOwner == JobExecutionOwner.Runtime && IsActive(state)) + return Task.FromResult(false); + ForgetJob(state); + return Task.FromResult(true); + } + } + + private void ForgetJob(JobState state) + { + if (state.HistoryExpiresUtc is { } expires) _brokerExpiry.Remove((expires, state.JobId)); + _jobs.Remove(state.JobId); + if (IsActive(state)) _activeJobs--; + _active.Remove(state.JobId); + if (state.ExecutionOwner == JobExecutionOwner.Broker) _deduplication.Remove(state.JobId); + } + + private void PurgeBrokerHistory() + { + var now = _timeProvider.GetUtcNow(); + while (_brokerExpiry.Count > 0 && _brokerExpiry.Min.Expires <= now) + { + var item = _brokerExpiry.Min; + _brokerExpiry.Remove(item); + if (_jobs.TryGetValue(item.Id, out var state) && state.HistoryExpiresUtc == item.Expires) ForgetJob(state); + } + } + + public Task IncrementCounterAsync(string name, string counterName, long value = 1, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + var hour = TruncateHour(_timeProvider.GetUtcNow()); + foreach (var key in _counters.Keys.Where(k => k.Hour < hour.AddHours(-48)).ToArray()) _counters.Remove(key); + if (!_counters.TryGetValue((name, hour), out var bucket)) _counters[(name, hour)] = bucket = new(); + bucket[counterName] = bucket.GetValueOrDefault(counterName) + value; + } + return Task.CompletedTask; + } + + public Task GetCounterStatsAsync(string name, TimeSpan? window = null, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + var duration = window ?? TimeSpan.FromHours(24); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(duration, TimeSpan.Zero); + ArgumentOutOfRangeException.ThrowIfGreaterThan(duration, TimeSpan.FromHours(48)); + lock (_lock) + { + var now = _timeProvider.GetUtcNow(); + var totals = new Dictionary(); + var buckets = new List(); + for (var hour = TruncateHour(now - duration); hour <= TruncateHour(now); hour = hour.AddHours(1)) + { + var values = _counters.TryGetValue((name, hour), out var bucket) ? new Dictionary(bucket) : new(); + foreach (var pair in values) totals[pair.Key] = totals.GetValueOrDefault(pair.Key) + pair.Value; + buckets.Add(new() { Hour = hour, Counters = values }); + } + return Task.FromResult(new JobCounterStats { Totals = totals, Buckets = buckets }); + } + } + + private static DateTimeOffset TruncateHour(DateTimeOffset value) => new(value.Year, value.Month, value.Day, value.Hour, 0, 0, TimeSpan.Zero); +} diff --git a/src/Foundatio/Jobs/InMemoryJobRuntimeStore.Claims.cs b/src/Foundatio/Jobs/InMemoryJobRuntimeStore.Claims.cs new file mode 100644 index 000000000..0d0810d76 --- /dev/null +++ b/src/Foundatio/Jobs/InMemoryJobRuntimeStore.Claims.cs @@ -0,0 +1,176 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Jobs; + +public sealed partial class InMemoryJobRuntimeStore +{ + public Task CreateOccurrenceAsync(JobState initial, bool allowOverlap = false, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(initial); + ArgumentException.ThrowIfNullOrWhiteSpace(initial.ScheduleName); + if (initial.ExecutionOwner != JobExecutionOwner.Runtime) throw new ArgumentException("Scheduled occurrences must be owned by the job runtime.", nameof(initial)); + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + ValidatePayload(initial.Payload?.Length ?? 0); + initial.RetryPolicy.Validate(); + PurgeDeduplication(); + if (_jobs.ContainsKey(initial.JobId) || _deduplication.ContainsKey(initial.JobId)) + return Task.FromResult(JobOccurrenceResult.AlreadyExists); + if (!allowOverlap && _active.Values.Any(s => s.ScheduleName == initial.ScheduleName + && s.RequiredNodeId == initial.RequiredNodeId && s.Status is JobStatus.Queued or JobStatus.Scheduled or JobStatus.Processing)) + return Task.FromResult(JobOccurrenceResult.OverlapBlocked); + EnsureCapacity(); + var now = _timeProvider.GetUtcNow(); + StoreJob(initial with + { + CreatedUtc = initial.CreatedUtc == default ? now : initial.CreatedUtc, + LastUpdatedUtc = now + }); + return Task.FromResult(JobOccurrenceResult.Created); + } + } + + public Task ClaimNextAsync(JobClaimRequest request, CancellationToken cancellationToken = default) + => ClaimAsync(null, request, cancellationToken); + + public Task ClaimJobAsync(string jobId, JobClaimRequest request, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(jobId); + return ClaimAsync(jobId, request, cancellationToken); + } + + private Task ClaimAsync(string? jobId, JobClaimRequest request, CancellationToken cancellationToken) + { + JobClaimValidation.Validate(request); + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + var now = _timeProvider.GetUtcNow(); + if (_active.Count == 0) + return Task.FromResult(null); + var candidates = _active.Values.Where(s => s.ExecutionOwner == JobExecutionOwner.Runtime && (jobId is null || s.JobId == jobId) + && (s.RequiredNodeId is null || s.RequiredNodeId == request.NodeId) + && s.JobType is not null && request.JobTypes.Contains(s.JobType, StringComparer.Ordinal) + && ((s.Status is JobStatus.Queued or JobStatus.Scheduled && (s.AvailableUtc ?? s.CreatedUtc) <= now) + || (s.Status == JobStatus.Processing && s.LeaseExpiresUtc <= now))) + .OrderBy(s => s.Status == JobStatus.Processing ? s.LeaseExpiresUtc : s.AvailableUtc ?? s.CreatedUtc) + .ThenBy(s => s.CreatedUtc).ThenBy(s => s.JobId, StringComparer.Ordinal); + foreach (var state in candidates) + { + bool expired = state.Attempt == 0 && state.RequiredNodeId is not null && state.ExpiresUtc <= now; + if (expired || state.CancellationRequested || state.Attempt >= state.MaxAttempts) + { + StoreJob(state with + { + Status = expired || state.CancellationRequested ? JobStatus.Cancelled : JobStatus.Failed, + Error = expired || state.CancellationRequested ? null : "Execution attempts exhausted after lease expiration.", + ResultMessage = expired ? "Unclaimed per-node occurrence expired." : null, + CompletedUtc = now, + LastUpdatedUtc = now, + NodeId = null, + ClaimToken = null, + LeaseExpiresUtc = null + }); + continue; + } + + var claimed = state with + { + Status = JobStatus.Processing, + NodeId = request.NodeId, + ClaimToken = Guid.NewGuid().ToString("N"), + LeaseExpiresUtc = now.Add(request.Lease), + StartedUtc = now, + CompletedUtc = null, + LastUpdatedUtc = now, + Attempt = state.Attempt + 1 + }; + StoreJob(claimed); + return Task.FromResult(claimed); + } + + return Task.FromResult(null); + } + } + + public Task CompleteJobAsync(string jobId, string claimToken, JobCompletion completion, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(completion); + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + var now = _timeProvider.GetUtcNow(); + if (!TryGetOwnedJob(jobId, claimToken, now, out var state)) + return Task.FromResult(false); + + bool broker = state.ExecutionOwner == JobExecutionOwner.Broker; + var kind = !broker && state.CancellationRequested ? JobCompletionKind.Cancelled : completion.Kind; + bool retry = kind == JobCompletionKind.Failed && completion.Retryable && (broker || state.Attempt < state.MaxAttempts); + var status = kind switch + { + JobCompletionKind.Succeeded => JobStatus.Completed, + JobCompletionKind.Cancelled => JobStatus.Cancelled, + JobCompletionKind.Interrupted => broker ? JobStatus.RetryPending : JobStatus.Queued, + JobCompletionKind.Failed => retry ? (broker ? JobStatus.RetryPending : JobStatus.Queued) : JobStatus.Failed, + _ => throw new ArgumentOutOfRangeException(nameof(completion)) + }; + StoreJob(state with + { + Status = status, + Error = kind == JobCompletionKind.Failed ? completion.Error : null, + ResultMessage = completion.Message, + NodeId = broker ? state.NodeId : null, + ClaimToken = null, + LeaseExpiresUtc = null, + LastUpdatedUtc = now, + CompletedUtc = status is JobStatus.Queued or JobStatus.RetryPending ? null : now, + AvailableUtc = broker ? null : retry ? now.Add(state.RetryPolicy.GetDelay(state.Attempt)) : now, + Progress = status == JobStatus.Completed ? 100 : state.Progress + }); + return Task.FromResult(true); + } + } + + public Task RenewJobLeaseAsync(string jobId, string claimToken, TimeSpan lease, CancellationToken cancellationToken = default) + { + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(lease, TimeSpan.Zero); + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + var now = _timeProvider.GetUtcNow(); + if (!TryGetOwnedJob(jobId, claimToken, now, out var state)) + return Task.FromResult(false); + if (state.ExecutionOwner == JobExecutionOwner.Broker) return Task.FromResult(false); + StoreJob(state with { LeaseExpiresUtc = now.Add(lease), LastUpdatedUtc = now }); + return Task.FromResult(true); + } + } + + public Task ReportJobProgressAsync(string jobId, string claimToken, int? percent = null, string? message = null, CancellationToken cancellationToken = default) + { + if (percent is < 0 or > 100) + throw new ArgumentOutOfRangeException(nameof(percent)); + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + var now = _timeProvider.GetUtcNow(); + if (!TryGetOwnedJob(jobId, claimToken, now, out var state)) + return Task.FromResult(false); + StoreJob(state with { Progress = percent ?? state.Progress, ProgressMessage = message ?? state.ProgressMessage, LastHeartbeatUtc = now, LastUpdatedUtc = now }); + return Task.FromResult(true); + } + } + + private bool TryGetOwnedJob(string jobId, string claimToken, DateTimeOffset now, out JobState state) + { + ArgumentException.ThrowIfNullOrWhiteSpace(jobId); + ArgumentException.ThrowIfNullOrWhiteSpace(claimToken); + PurgeBrokerHistory(); + return _jobs.TryGetValue(jobId, out state!) && state.Status == JobStatus.Processing + && state.ClaimToken == claimToken && (state.ExecutionOwner == JobExecutionOwner.Broker || state.LeaseExpiresUtc > now); + } +} diff --git a/src/Foundatio/Jobs/InMemoryJobRuntimeStore.Schedules.cs b/src/Foundatio/Jobs/InMemoryJobRuntimeStore.Schedules.cs new file mode 100644 index 000000000..f9cdd00b4 --- /dev/null +++ b/src/Foundatio/Jobs/InMemoryJobRuntimeStore.Schedules.cs @@ -0,0 +1,21 @@ +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Jobs; + +public sealed partial class InMemoryJobRuntimeStore +{ + private readonly InMemoryScheduledJobStore _schedules = new(); + + public Task ScheduleAsync(ScheduledJobDefinition definition, CancellationToken cancellationToken = default) + => _schedules.ScheduleAsync(definition, cancellationToken); + public Task ReconcileAsync(ScheduledJobDefinition definition, CancellationToken cancellationToken = default) + => _schedules.ReconcileAsync(definition, cancellationToken); + public Task GetScheduleAsync(string name, CancellationToken cancellationToken = default) + => _schedules.GetScheduleAsync(name, cancellationToken); + public Task UnscheduleAsync(string name, CancellationToken cancellationToken = default) + => _schedules.UnscheduleAsync(name, cancellationToken); + public Task> GetSchedulesAsync(ScheduleQuery? query = null, CancellationToken cancellationToken = default) + => _schedules.GetSchedulesAsync(query, cancellationToken); +} diff --git a/src/Foundatio/Jobs/InMemoryScheduledJobStore.cs b/src/Foundatio/Jobs/InMemoryScheduledJobStore.cs new file mode 100644 index 000000000..5d21d7644 --- /dev/null +++ b/src/Foundatio/Jobs/InMemoryScheduledJobStore.cs @@ -0,0 +1,90 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Jobs; + +/// Versioned schedule storage for tests and single-process applications. +public sealed class InMemoryScheduledJobStore : IScheduledJobStore +{ + private sealed record Entry(ScheduledJobDefinition Definition, string? Configuration); + private readonly Dictionary _definitions = new(StringComparer.Ordinal); + private readonly object _lock = new(); + + public Task ScheduleAsync(ScheduledJobDefinition definition, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(definition); + definition.Validate(); + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + _definitions.TryGetValue(definition.Name, out var current); + if (definition.Revision != (current?.Definition.Revision ?? 0)) + throw new JobException($"Schedule {definition.Name} changed. Reload it before saving."); + _definitions[definition.Name] = new Entry(Snapshot(definition with + { + Revision = definition.Revision + 1, + ConfigurationVersion = current?.Definition.ConfigurationVersion ?? 0 + }), current?.Configuration); + } + return Task.CompletedTask; + } + + public Task ReconcileAsync(ScheduledJobDefinition definition, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(definition); + definition.Validate(); + ArgumentOutOfRangeException.ThrowIfLessThan(definition.ConfigurationVersion, 1); + cancellationToken.ThrowIfCancellationRequested(); + string configuration = JsonSerializer.Serialize(definition with { Revision = 0 }); + lock (_lock) + { + _definitions.TryGetValue(definition.Name, out var current); + if (current is not null) + { + if (definition.ConfigurationVersion < current.Definition.ConfigurationVersion) + return Task.CompletedTask; + if (definition.ConfigurationVersion == current.Definition.ConfigurationVersion) + { + if (configuration != current.Configuration) + throw new JobException($"Declared schedule {definition.Name} changed. Increase ConfigurationVersion to apply it."); + return Task.CompletedTask; + } + } + _definitions[definition.Name] = new Entry(Snapshot(definition with { Revision = (current?.Definition.Revision ?? 0) + 1 }), configuration); + } + return Task.CompletedTask; + } + + public Task GetScheduleAsync(string name, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(name); + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + return Task.FromResult(_definitions.TryGetValue(name, out var entry) ? Snapshot(entry.Definition) : null); + } + + public Task UnscheduleAsync(string name, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(name); + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + _definitions.Remove(name); + return Task.CompletedTask; + } + + public Task> GetSchedulesAsync(ScheduleQuery? query = null, CancellationToken cancellationToken = default) + { + query ??= new ScheduleQuery(); + query.Validate(); + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + return Task.FromResult>(_definitions.Values.Where(e => query.AfterName is null || StringComparer.Ordinal.Compare(e.Definition.Name, query.AfterName) > 0).OrderBy(e => e.Definition.Name, StringComparer.Ordinal).Take(query.Limit).Select(e => Snapshot(e.Definition)).ToArray()); + } + + private static ScheduledJobDefinition Snapshot(ScheduledJobDefinition definition) + => definition with { Payload = definition.Payload is { } payload ? (ReadOnlyMemory?)payload.ToArray() : null }; +} diff --git a/src/Foundatio/Jobs/JobArgumentContract.cs b/src/Foundatio/Jobs/JobArgumentContract.cs new file mode 100644 index 000000000..58d5e4bd8 --- /dev/null +++ b/src/Foundatio/Jobs/JobArgumentContract.cs @@ -0,0 +1,37 @@ +using System; +using System.Collections.Concurrent; +using System.Linq; + +namespace Foundatio.Jobs; + +internal static class JobArgumentContract +{ + private static readonly ConcurrentDictionary _contracts = new(); + + public static void ValidateType(Type jobType) + { + ArgumentNullException.ThrowIfNull(jobType); + if (!typeof(IJob).IsAssignableFrom(jobType) || jobType.IsAbstract || jobType.IsInterface || jobType.ContainsGenericParameters) + throw new ArgumentException($"Job {jobType.Name} must be a concrete type implementing IJob.", nameof(jobType)); + } + + public static void Validate(Type jobType, object? arguments) + { + ValidateType(jobType); + var types = _contracts.GetOrAdd(jobType, static type => type.GetInterfaces() + .Where(i => i.IsGenericType && i.GetGenericTypeDefinition() == typeof(IJob<>)) + .Select(i => i.GenericTypeArguments[0]).ToArray()); + if (types.Length > 1) + throw new ArgumentException($"Job {jobType.Name} must declare only one argument contract.", nameof(jobType)); + + if (types.Length == 0) + { + if (arguments is not null) + throw new ArgumentException($"Job {jobType.Name} does not declare an IJob argument contract.", nameof(arguments)); + return; + } + + if (arguments is null || arguments.GetType() != types[0]) + throw new ArgumentException($"Job {jobType.Name} requires arguments of type {types[0].Name}. Use EnqueueAsync(args).", nameof(arguments)); + } +} diff --git a/src/Foundatio/Jobs/JobAttribute.cs b/src/Foundatio/Jobs/JobAttribute.cs deleted file mode 100644 index 8b32f5ab4..000000000 --- a/src/Foundatio/Jobs/JobAttribute.cs +++ /dev/null @@ -1,15 +0,0 @@ -using System; - -namespace Foundatio.Jobs; - -[AttributeUsage(AttributeTargets.Class, AllowMultiple = false, Inherited = true)] -public class JobAttribute : Attribute -{ - public string? Name { get; set; } - public string? Description { get; set; } - public bool IsContinuous { get; set; } = true; - public string? Interval { get; set; } - public string? InitialDelay { get; set; } - public int IterationLimit { get; set; } = -1; - public int InstanceCount { get; set; } = 1; -} diff --git a/src/Foundatio/Jobs/JobBase.cs b/src/Foundatio/Jobs/JobBase.cs deleted file mode 100644 index ead6e3b48..000000000 --- a/src/Foundatio/Jobs/JobBase.cs +++ /dev/null @@ -1,43 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Resilience; -using Foundatio.Utility; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Logging.Abstractions; - -namespace Foundatio.Jobs; - -public abstract class JobBase : IJob, IHaveLogger, IHaveLoggerFactory, IHaveTimeProvider, IHaveResiliencePolicyProvider -{ - protected readonly TimeProvider _timeProvider; - protected readonly ILogger _logger; - protected readonly ILoggerFactory _loggerFactory; - protected readonly IResiliencePolicyProvider _resiliencePolicyProvider; - - public JobBase(ILoggerFactory? loggerFactory = null) : this(null, null, loggerFactory) - { - } - - public JobBase(TimeProvider? timeProvider, IResiliencePolicyProvider? resiliencePolicyProvider, ILoggerFactory? loggerFactory = null) - { - _timeProvider = timeProvider ?? TimeProvider.System; - _resiliencePolicyProvider = resiliencePolicyProvider ?? DefaultResiliencePolicyProvider.Instance; - _loggerFactory = loggerFactory ?? NullLoggerFactory.Instance; - _logger = _loggerFactory.CreateLogger(GetType()); - - } - - public string JobId { get; } = Guid.NewGuid().ToString("N").Substring(0, 10); - ILogger IHaveLogger.Logger => _logger; - ILoggerFactory IHaveLoggerFactory.LoggerFactory => _loggerFactory; - TimeProvider IHaveTimeProvider.TimeProvider => _timeProvider; - IResiliencePolicyProvider IHaveResiliencePolicyProvider.ResiliencePolicyProvider => _resiliencePolicyProvider; - - public virtual Task RunAsync(CancellationToken cancellationToken = default) - { - return RunInternalAsync(new JobContext(cancellationToken)); - } - - protected abstract Task RunInternalAsync(JobContext context); -} diff --git a/src/Foundatio/Jobs/JobClaim.cs b/src/Foundatio/Jobs/JobClaim.cs new file mode 100644 index 000000000..dd68e01fc --- /dev/null +++ b/src/Foundatio/Jobs/JobClaim.cs @@ -0,0 +1,35 @@ +using System; +using System.Collections.Generic; + +namespace Foundatio.Jobs; + +/// Eligibility and ownership for one atomic job claim. +public sealed record JobClaimRequest +{ + /// Diagnostic worker identity. Ownership is fenced by a fresh claim token for every run. + public required string NodeId { get; init; } + + /// Registered wire names this worker can execute. + public required IReadOnlyCollection JobTypes { get; init; } + + /// Renewable execution lease. Default five minutes. + public TimeSpan Lease { get; init; } = TimeSpan.FromMinutes(5); +} + +/// How an owned execution ended. +public enum JobCompletionKind +{ + Succeeded, + Failed, + Cancelled, + Interrupted +} + +/// Completion input for an atomic, claim-guarded job transition. +public sealed record JobCompletion +{ + public required JobCompletionKind Kind { get; init; } + public string? Error { get; init; } + public string? Message { get; init; } + public bool Retryable { get; init; } = true; +} diff --git a/src/Foundatio/Jobs/JobClaimValidation.cs b/src/Foundatio/Jobs/JobClaimValidation.cs new file mode 100644 index 000000000..58ff32ea5 --- /dev/null +++ b/src/Foundatio/Jobs/JobClaimValidation.cs @@ -0,0 +1,17 @@ +using System; +using System.Linq; + +namespace Foundatio.Jobs; + +internal static class JobClaimValidation +{ + public static void Validate(JobClaimRequest request) + { + ArgumentNullException.ThrowIfNull(request); + ArgumentException.ThrowIfNullOrWhiteSpace(request.NodeId); + ArgumentNullException.ThrowIfNull(request.JobTypes); + if (request.JobTypes.Count == 0 || request.JobTypes.Any(String.IsNullOrWhiteSpace)) + throw new ArgumentException("Register the job types this worker can execute.", nameof(request)); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(request.Lease, TimeSpan.Zero); + } +} diff --git a/src/Foundatio/Jobs/JobContext.cs b/src/Foundatio/Jobs/JobContext.cs deleted file mode 100644 index c955f2657..000000000 --- a/src/Foundatio/Jobs/JobContext.cs +++ /dev/null @@ -1,25 +0,0 @@ -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Lock; - -namespace Foundatio.Jobs; - -public class JobContext -{ - public JobContext(CancellationToken cancellationToken, ILock? lck = null) - { - Lock = lck; - CancellationToken = cancellationToken; - } - - public ILock? Lock { get; } - public CancellationToken CancellationToken { get; } - - public virtual Task RenewLockAsync() - { - if (Lock != null) - return Lock.RenewAsync(); - - return Task.CompletedTask; - } -} diff --git a/src/Foundatio/Jobs/JobCounterStats.cs b/src/Foundatio/Jobs/JobCounterStats.cs new file mode 100644 index 000000000..e1215d7fb --- /dev/null +++ b/src/Foundatio/Jobs/JobCounterStats.cs @@ -0,0 +1,40 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +namespace Foundatio.Jobs; + +/// +/// Counter statistics for a queue, including totals and per-hour buckets for sparkline rendering. +/// +public sealed record JobCounterStats +{ + /// + /// Sum of all counters across the requested time window. + /// Keys are counter names (e.g., "processed", "failed", "dead_lettered"). + /// + public required IReadOnlyDictionary Totals { get; init; } + + /// + /// Per-hour counter values ordered oldest to newest, suitable for sparkline rendering. + /// Each bucket represents one UTC hour. + /// + public required IReadOnlyList Buckets { get; init; } +} + +/// +/// Counter values for a single hour. +/// +public sealed record JobCounterBucket +{ + /// + /// The UTC hour this bucket represents (truncated to the hour). + /// + public required DateTimeOffset Hour { get; init; } + + /// + /// Counter values for this hour. Keys are counter names. + /// + public required IReadOnlyDictionary Counters { get; init; } +} diff --git a/src/Foundatio/Jobs/JobExceptions.cs b/src/Foundatio/Jobs/JobExceptions.cs new file mode 100644 index 000000000..051cc3a48 --- /dev/null +++ b/src/Foundatio/Jobs/JobExceptions.cs @@ -0,0 +1,40 @@ +using System; + +namespace Foundatio.Jobs; + +/// +/// Base exception for job-runtime errors (unresolvable job types, untriggerable schedules). Derives from +/// so catch blocks written against the general type keep working. +/// +public class JobException : InvalidOperationException +{ + public JobException() { } + + public JobException(string message) : base(message) { } + + public JobException(string message, Exception innerException) : base(message, innerException) { } +} + +/// Thrown when a scheduled-job operation addresses a schedule name that is not registered. +public sealed class ScheduledJobNotFoundException : JobException +{ + public ScheduledJobNotFoundException(string name) : base($"No scheduled job named \"{name}\" is registered.") + { + Name = name; + } + + /// The schedule name that could not be found. + public string Name { get; } +} + +/// Thrown when a scheduled job is triggered while its schedule is disabled. +public sealed class ScheduledJobDisabledException : JobException +{ + public ScheduledJobDisabledException(string name) : base($"Scheduled job \"{name}\" is disabled. Enable it before triggering (SetEnabledAsync(\"{name}\", true)).") + { + Name = name; + } + + /// The name of the disabled schedule. + public string Name { get; } +} diff --git a/src/Foundatio/Jobs/JobMonitorExtensions.cs b/src/Foundatio/Jobs/JobMonitorExtensions.cs new file mode 100644 index 000000000..5d73ae8a7 --- /dev/null +++ b/src/Foundatio/Jobs/JobMonitorExtensions.cs @@ -0,0 +1,24 @@ +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Jobs; + +/// Observation helpers for runtime jobs and tracked broker work. +public static class JobMonitorExtensions +{ + /// Waits for terminal state. Cancelling observation does not cancel the job. + public static async Task WaitForCompletionAsync(this IJobMonitor monitor, string jobId, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(monitor); + ArgumentException.ThrowIfNullOrWhiteSpace(jobId); + while (true) + { + var state = await monitor.GetAsync(jobId, cancellationToken).ConfigureAwait(false) + ?? throw new InvalidOperationException($"Job '{jobId}' was not found or has expired."); + if (state.Status is JobStatus.Completed or JobStatus.Failed or JobStatus.Cancelled or JobStatus.DeadLettered) + return state; + await Task.Delay(TimeSpan.FromMilliseconds(100), cancellationToken).ConfigureAwait(false); + } + } +} diff --git a/src/Foundatio/Jobs/JobOptions.cs b/src/Foundatio/Jobs/JobOptions.cs deleted file mode 100644 index 8960a0fbd..000000000 --- a/src/Foundatio/Jobs/JobOptions.cs +++ /dev/null @@ -1,104 +0,0 @@ -using System; -using System.Reflection; -using Foundatio.Extensions; -using Foundatio.Utility; - -namespace Foundatio.Jobs; - -public class JobOptions -{ - public string? Name { get; set; } - public string? Description { get; set; } - public Func? JobFactory { get; set; } - public bool RunContinuous { get; set; } = true; - public TimeSpan? Interval { get; set; } - public TimeSpan? InitialDelay { get; set; } - public int IterationLimit { get; set; } = -1; - public int InstanceCount { get; set; } = 1; - - public static JobOptions GetDefaults(Type jobType) - { - var jobOptions = new JobOptions(); - ApplyDefaults(jobOptions, jobType); - return jobOptions; - } - - public static void ApplyDefaults(JobOptions jobOptions, Type jobType) - { - var jobAttribute = jobType.GetCustomAttribute() ?? new JobAttribute(); - - jobOptions.Name = jobAttribute.Name; - if (String.IsNullOrEmpty(jobOptions.Name)) - jobOptions.Name = GetDefaultJobName(jobType); - - jobOptions.Description = jobAttribute.Description; - jobOptions.RunContinuous = jobAttribute.IsContinuous; - - if (!String.IsNullOrEmpty(jobAttribute.Interval)) - { - TimeSpan? interval; - if (TimeUnit.TryParse(jobAttribute.Interval, out interval)) - jobOptions.Interval = interval; - } - - if (!String.IsNullOrEmpty(jobAttribute.InitialDelay)) - { - TimeSpan? delay; - if (TimeUnit.TryParse(jobAttribute.InitialDelay, out delay)) - jobOptions.InitialDelay = delay; - } - - jobOptions.IterationLimit = jobAttribute.IterationLimit; - jobOptions.InstanceCount = jobAttribute.InstanceCount; - } - - public static JobOptions GetDefaults() where T : IJob - { - return GetDefaults(typeof(T)); - } - - public static JobOptions GetDefaults(IJob instance) - { - var jobOptions = GetDefaults(instance.GetType()); - jobOptions.JobFactory = _ => instance; - return jobOptions; - } - - public static JobOptions GetDefaults(IJob instance) where T : IJob - { - var jobOptions = GetDefaults(); - jobOptions.JobFactory = _ => instance; - return jobOptions; - } - - public static JobOptions GetDefaults(Type jobType, Func jobFactory) - { - var jobOptions = GetDefaults(jobType); - jobOptions.JobFactory = jobFactory; - return jobOptions; - } - - public static JobOptions GetDefaults(Func jobFactory) where T : IJob - { - var jobOptions = GetDefaults(); - jobOptions.JobFactory = jobFactory; - return jobOptions; - } - - public static string GetDefaultJobName(Type type) - { - string jobName = type.Name; - if (jobName.EndsWith("Job")) - jobName = jobName.Substring(0, jobName.Length - 3); - - return jobName.ToSpacedWords(); - } -} - -public static class JobOptionExtensions -{ - public static void ApplyDefaults(this JobOptions jobOptions) - { - JobOptions.ApplyDefaults(jobOptions, typeof(T)); - } -} diff --git a/src/Foundatio/Jobs/JobPage.cs b/src/Foundatio/Jobs/JobPage.cs new file mode 100644 index 000000000..ef94819aa --- /dev/null +++ b/src/Foundatio/Jobs/JobPage.cs @@ -0,0 +1,14 @@ +using System.Collections; +using System.Collections.Generic; + +namespace Foundatio.Jobs; + +/// A bounded page ordered by job ID. Continue until ContinuationToken is null, including after an empty filtered page. +public sealed class JobPage(IReadOnlyList items, string? continuationToken) : IReadOnlyList +{ + public string? ContinuationToken { get; } = continuationToken; + public int Count => items.Count; + public JobState this[int index] => items[index]; + public IEnumerator GetEnumerator() => items.GetEnumerator(); + IEnumerator IEnumerable.GetEnumerator() => GetEnumerator(); +} diff --git a/src/Foundatio/Jobs/JobResult.cs b/src/Foundatio/Jobs/JobResult.cs index 42ada5859..eb59278c8 100644 --- a/src/Foundatio/Jobs/JobResult.cs +++ b/src/Foundatio/Jobs/JobResult.cs @@ -3,17 +3,19 @@ namespace Foundatio.Jobs; -public class JobResult +/// +/// The outcome of one job run. Immutable — the shared / instances are +/// safe to return from any job; use the *WithMessage/ factories (or a +/// with-expression) to attach details. +/// +public sealed record JobResult { - public bool IsCancelled { get; set; } - public Exception? Error { get; set; } - public string Message { get; set; } = String.Empty; - public bool IsSuccess { get; set; } - - public static readonly JobResult None = new() - { - IsSuccess = true - }; + public bool IsCancelled { get; init; } + public Exception? Error { get; init; } + public string Message { get; init; } = String.Empty; + public bool IsSuccess { get; init; } + /// False makes a failure terminal without consuming the remaining retry budget. + public bool Retryable { get; init; } = true; public static readonly JobResult Cancelled = new() { diff --git a/src/Foundatio/Jobs/JobRetryPolicy.cs b/src/Foundatio/Jobs/JobRetryPolicy.cs new file mode 100644 index 000000000..bef96a95b --- /dev/null +++ b/src/Foundatio/Jobs/JobRetryPolicy.cs @@ -0,0 +1,31 @@ +using System; + +namespace Foundatio.Jobs; + +/// Serializable retry curve persisted with each job, independent of worker configuration. +public sealed record JobRetryPolicy +{ + public TimeSpan InitialDelay { get; init; } = TimeSpan.FromSeconds(10); + public TimeSpan MaxDelay { get; init; } = TimeSpan.FromMinutes(5); + public double Multiplier { get; init; } = 2; + public double JitterFactor { get; init; } = 0.2; + + public void Validate() + { + ArgumentOutOfRangeException.ThrowIfLessThan(InitialDelay, TimeSpan.Zero); + ArgumentOutOfRangeException.ThrowIfLessThan(MaxDelay, InitialDelay); + if (!Double.IsFinite(Multiplier) || Multiplier < 1) throw new ArgumentOutOfRangeException(nameof(Multiplier)); + if (!Double.IsFinite(JitterFactor) || JitterFactor is < 0 or > 1) throw new ArgumentOutOfRangeException(nameof(JitterFactor)); + } + + public TimeSpan GetDelay(int attempt) + { + Validate(); + ArgumentOutOfRangeException.ThrowIfLessThan(attempt, 1); + if (InitialDelay == TimeSpan.Zero) + return TimeSpan.Zero; + double seconds = Math.Min(MaxDelay.TotalSeconds, InitialDelay.TotalSeconds * Math.Pow(Multiplier, Math.Min(100, attempt - 1))); + double jitter = 1 + JitterFactor * (2 * Random.Shared.NextDouble() - 1); + return TimeSpan.FromSeconds(Math.Min(MaxDelay.TotalSeconds, seconds * jitter)); + } +} diff --git a/src/Foundatio/Jobs/JobRunner.cs b/src/Foundatio/Jobs/JobRunner.cs deleted file mode 100644 index 0437123f4..000000000 --- a/src/Foundatio/Jobs/JobRunner.cs +++ /dev/null @@ -1,286 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Diagnostics; -using System.IO; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Utility; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Logging.Abstractions; - -namespace Foundatio.Jobs; - -public class JobRunner -{ - private readonly TimeProvider _timeProvider; - private readonly ILogger _logger; - private readonly JobOptions _options; - private readonly IServiceProvider _serviceProvider; - - public JobRunner(JobOptions options, IServiceProvider serviceProvider, ILoggerFactory? loggerFactory = null) - { - _timeProvider = serviceProvider.GetService() ?? TimeProvider.System; - _logger = loggerFactory?.CreateLogger() ?? NullLogger.Instance; - _options = options; - _serviceProvider = serviceProvider; - } - - public JobRunner(IJob instance, IServiceProvider serviceProvider, ILoggerFactory? loggerFactory = null, TimeSpan? initialDelay = null, int instanceCount = 1, bool runContinuous = true, int iterationLimit = -1, TimeSpan? interval = null) - : this(new JobOptions - { - JobFactory = _ => instance, - InitialDelay = initialDelay, - InstanceCount = instanceCount, - IterationLimit = iterationLimit, - RunContinuous = runContinuous, - Interval = interval - }, serviceProvider, loggerFactory) - { - } - - public JobRunner(Func jobFactory, IServiceProvider serviceProvider, - ILoggerFactory? loggerFactory = null, TimeSpan? initialDelay = null, int instanceCount = 1, - bool runContinuous = true, int iterationLimit = -1, TimeSpan? interval = null) - : this(new JobOptions - { - JobFactory = jobFactory, - InitialDelay = initialDelay, - InstanceCount = instanceCount, - IterationLimit = iterationLimit, - RunContinuous = runContinuous, - Interval = interval - }, serviceProvider, loggerFactory) - { - } - - public string Id { get; } = Guid.NewGuid().ToString("N").Substring(0, 10); - public CancellationTokenSource? CancellationTokenSource { get; private set; } - - public async Task RunInConsoleAsync() - { - int result; - try - { - CancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(GetShutdownCancellationToken(_logger)); - bool success = await RunAsync(CancellationTokenSource.Token).AnyContext(); - result = success ? 0 : -1; - - if (Debugger.IsAttached) - Console.ReadKey(); - } - catch (TaskCanceledException) - { - return 0; - } - catch (FileNotFoundException ex) - { - _logger.LogError(ex, "Job {JobName} error: {Message} ({FileName})", _options.Name, ex.GetMessage(), ex.FileName); - if (Debugger.IsAttached) - Console.ReadKey(); - - return 1; - } - catch (Exception ex) - { - _logger.LogError(ex, "Job {JobName} error: {Message}", _options.Name, ex.GetMessage()); - - if (Debugger.IsAttached) - Console.ReadKey(); - - return 1; - } - - return result; - } - - public void RunInBackground(CancellationToken cancellationToken = default) - { - if (_options.InstanceCount == 1) - { - _ = Task.Run(async () => - { - try - { - await RunAsync(cancellationToken).AnyContext(); - } - catch (TaskCanceledException) - { - // Ignore cancellation - } - catch (Exception ex) - { - _logger.LogError(ex, "Error running job {JobName} in background: {Message}", _options.Name, ex.Message); - throw; - } - }, cancellationToken); - } - else - { - var ignored = RunAsync(cancellationToken); - } - } - - public async Task RunAsync(CancellationToken cancellationToken = default) - { - if (_options.JobFactory is null) - { - _logger.LogError("JobFactory must be specified"); - return false; - } - - IJob job; - try - { - job = _options.JobFactory(_serviceProvider); - if (job is IJobWithOptions jobWithOptions) - jobWithOptions.Options = _options; - } - catch (Exception ex) - { - _logger.LogError(ex, "Error creating job instance from JobFactory"); - return false; - } - - if (job is null) - { - _logger.LogError("JobFactory returned null job instance"); - return false; - } - - using var _ = _logger.BeginScope(s => s.Property("job.name", _options.Name ?? String.Empty).Property("job.id", Id)); - - _logger.LogInformation("Starting job type {JobName} on machine {MachineName}...", _options.Name, Environment.MachineName); - - if (job is IAsyncLifetime jobLifetime) - { - _logger.LogInformation("Initializing job lifetime {JobName} on machine {MachineName}...", _options.Name, Environment.MachineName); - await jobLifetime.InitializeAsync().AnyContext(); - _logger.LogInformation("Finished initializing job lifetime {JobName} on machine {MachineName}", _options.Name, Environment.MachineName); - } - - try - { - if (_options.InitialDelay.HasValue && _options.InitialDelay.Value > TimeSpan.Zero) - await _timeProvider.SafeDelay(_options.InitialDelay.Value, cancellationToken).AnyContext(); - - if (_options.RunContinuous && _options.InstanceCount > 1) - { - try - { - var tasks = new List(_options.InstanceCount); - for (int i = 0; i < _options.InstanceCount; i++) - { - tasks.Add(Task.Run(async () => - { - try - { - await using var scope = _serviceProvider.CreateAsyncScope(); - var jobInstance = _options.JobFactory(scope.ServiceProvider); - if (jobInstance is IJobWithOptions jobWithOptions) - jobWithOptions.Options = _options; - - await jobInstance.RunContinuousAsync(_options, cancellationToken).AnyContext(); - } - catch (TaskCanceledException) - { - // Ignore cancellation - } - catch (Exception ex) - { - _logger.LogError(ex, "Error running job instance: {Message}", ex.Message); - throw; - } - }, cancellationToken)); - } - - await Task.WhenAll(tasks).AnyContext(); - } - catch (OperationCanceledException) - { - // Ignore cancellation - } - } - else if (_options.RunContinuous && _options.InstanceCount == 1) - { - await job.RunContinuousAsync(_options, cancellationToken).AnyContext(); - } - else - { - using var activity = FoundatioDiagnostics.ActivitySource.StartActivity("Job: " + _options.Name); - - var result = await job.TryRunAsync(cancellationToken).AnyContext(); - _logger.LogJobResult(result, _options.Name); - - return result.IsSuccess; - } - } - finally - { - if (job is IAsyncDisposable jobDisposable) - { - _logger.LogInformation("Disposing job lifetime {JobName} on machine {MachineName}...", _options.Name, Environment.MachineName); - await jobDisposable.DisposeAsync().AnyContext(); - _logger.LogInformation("Finished disposing job lifetime {JobName} on machine {MachineName}", _options.Name, Environment.MachineName); - } - } - - return true; - } - - private static CancellationTokenSource? _jobShutdownCancellationTokenSource; - private static FileSystemWatcher? _shutdownFileWatcher; - private static readonly object _lock = new(); - public static CancellationToken GetShutdownCancellationToken(ILogger? logger = null) - { - if (_jobShutdownCancellationTokenSource != null) - return _jobShutdownCancellationTokenSource.Token; - - lock (_lock) - { - if (_jobShutdownCancellationTokenSource != null) - return _jobShutdownCancellationTokenSource.Token; - - _jobShutdownCancellationTokenSource = new(); - Console.CancelKeyPress += (sender, args) => - { - _jobShutdownCancellationTokenSource.Cancel(); - logger?.LogInformation("Job shutdown event signaled: {SpecialKey}", args.SpecialKey); - args.Cancel = true; - }; - - string? webJobsShutdownFile = Environment.GetEnvironmentVariable("WEBJOBS_SHUTDOWN_FILE"); - if (String.IsNullOrEmpty(webJobsShutdownFile)) - return _jobShutdownCancellationTokenSource.Token; - - var handler = new FileSystemEventHandler((s, e) => - { - if (e.FullPath.IndexOf(Path.GetFileName(webJobsShutdownFile)!, StringComparison.OrdinalIgnoreCase) < 0) - return; - - _jobShutdownCancellationTokenSource.Cancel(); - logger?.LogInformation("Job shutdown signaled"); - }); - - _shutdownFileWatcher = new FileSystemWatcher(Path.GetDirectoryName(webJobsShutdownFile) ?? "."); - _shutdownFileWatcher.Created += handler; - _shutdownFileWatcher.Changed += handler; - _shutdownFileWatcher.NotifyFilter = NotifyFilters.CreationTime | NotifyFilters.FileName | NotifyFilters.LastWrite; - _shutdownFileWatcher.IncludeSubdirectories = false; - _shutdownFileWatcher.EnableRaisingEvents = true; - - _jobShutdownCancellationTokenSource.Token.Register(() => - { - if (_shutdownFileWatcher is not null) - { - _shutdownFileWatcher.Created -= handler; - _shutdownFileWatcher.Changed -= handler; - _shutdownFileWatcher.Dispose(); - _shutdownFileWatcher = null; - } - }); - - return _jobShutdownCancellationTokenSource.Token; - } - } -} diff --git a/src/Foundatio/Jobs/JobRuntime.cs b/src/Foundatio/Jobs/JobRuntime.cs new file mode 100644 index 000000000..c0915ac06 --- /dev/null +++ b/src/Foundatio/Jobs/JobRuntime.cs @@ -0,0 +1,933 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Diagnostics.Metrics; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Foundatio.Serializer; +using Foundatio.Utility; + +namespace Foundatio.Jobs; + +/// +/// Core-owned durable-job instruments, shared by every so job throughput and run latency are +/// observable independent of the runtime store implementation. +/// +internal static class JobInstruments +{ + public static readonly Counter CapacityRejected = FoundatioDiagnostics.Meter.CreateCounter("foundatio.jobs.capacity_rejected", description: "Job store admissions rejected by a configured resource budget"); + public static readonly Counter Started = FoundatioDiagnostics.Meter.CreateCounter("foundatio.jobs.started", description: "Number of durable jobs started"); + public static readonly Counter Completed = FoundatioDiagnostics.Meter.CreateCounter("foundatio.jobs.completed", description: "Number of durable jobs completed successfully"); + public static readonly Counter Failed = FoundatioDiagnostics.Meter.CreateCounter("foundatio.jobs.failed", description: "Number of durable jobs that failed"); + public static readonly Counter Cancelled = FoundatioDiagnostics.Meter.CreateCounter("foundatio.jobs.cancelled", description: "Number of durable jobs that were cancelled"); + public static readonly Histogram RunTime = FoundatioDiagnostics.Meter.CreateHistogram("foundatio.jobs.runtime", unit: "ms", description: "Durable job execution time"); +} + +public enum JobStatus +{ + Queued, + Scheduled, + Processing, + Completed, + Failed, + Cancelled, + DeadLettered, + /// A broker delivery may recur; the job runtime must not schedule it. + RetryPending, + /// Sending did not confirm whether the broker accepted the work. + EnqueueUnknown +} + +/// Identifies the system responsible for delivering and recovering work. +public enum JobExecutionOwner { Runtime, Broker } + +public enum ScheduledDispatchKind +{ + QueueMessage, + PubSubMessage +} + +public sealed record JobState +{ + public required string JobId { get; init; } + public required string Name { get; init; } + /// Runtime is the default. Broker records are never claimed by the job worker. + public JobExecutionOwner ExecutionOwner { get; init; } + /// Queue destination for broker-delivered work. + public string? QueueName { get; init; } + /// Optional operational metadata, shared by runtime and broker jobs. + public IReadOnlyDictionary? Metadata { get; init; } + /// Last progress report or explicit execution heartbeat. + public DateTimeOffset? LastHeartbeatUtc { get; init; } + /// Optional retention of broker history since its last update. Expiry never removes broker work. + public TimeSpan? HistoryRetention { get; init; } + /// Store-maintained expiration of optional broker history. + public DateTimeOffset? HistoryExpiresUtc { get; init; } + public string? JobType { get; init; } + + /// Serialized per-invocation arguments (see ); null when the job takes none. + public ReadOnlyMemory? Payload { get; init; } + + /// Discriminator for the payload type (the argument type's full name), stored for forensics and mismatch diagnostics. + public string? PayloadType { get; init; } + public JobStatus Status { get; init; } = JobStatus.Queued; + public int? Progress { get; init; } + public string? ProgressMessage { get; init; } + public int Attempt { get; init; } + /// Total execution attempts allowed, including retries and crash recovery. + public int MaxAttempts { get; init; } = 3; + public JobRetryPolicy RetryPolicy { get; init; } = new(); + /// Earliest execution time, including persisted retry delays. + public DateTimeOffset? AvailableUtc { get; init; } + /// Unique ownership token for the current execution; changes on every claim. + public string? ClaimToken { get; init; } + /// Optional node affinity for per-node scheduled work. + public string? RequiredNodeId { get; init; } + /// Schedule that created this occurrence; null for ad hoc jobs. + public string? ScheduleName { get; init; } + public string? NodeId { get; init; } + public DateTimeOffset CreatedUtc { get; init; } + public DateTimeOffset LastUpdatedUtc { get; init; } + public DateTimeOffset? StartedUtc { get; init; } + public DateTimeOffset? CompletedUtc { get; init; } + public DateTimeOffset? LeaseExpiresUtc { get; init; } + public string? Error { get; init; } + /// Informational outcome, independent of failure diagnostics. + public string? ResultMessage { get; init; } + public bool CancellationRequested { get; init; } + public DateTimeOffset? ScheduledForUtc { get; init; } + /// Expires an unclaimed per-node occurrence when its intended node never returns. + public DateTimeOffset? ExpiresUtc { get; init; } + /// Validates execution ownership and retention settings. + public void Validate() + { + if (!Enum.IsDefined(ExecutionOwner)) throw new ArgumentOutOfRangeException(nameof(ExecutionOwner)); + if (HistoryRetention <= TimeSpan.Zero) throw new ArgumentOutOfRangeException(nameof(HistoryRetention)); + if (ExecutionOwner == JobExecutionOwner.Broker) + { + ArgumentException.ThrowIfNullOrWhiteSpace(QueueName); + if (ScheduleName is not null || RequiredNodeId is not null || LeaseExpiresUtc is not null) + throw new ArgumentException("Broker jobs cannot carry runtime scheduling or lease ownership."); + } + } + +} + +public sealed record JobQuery +{ + public string? Name { get; init; } + public JobStatus? Status { get; init; } + public int Limit { get; init; } = 100; + /// Restricts monitoring to a queue destination. + public string? QueueName { get; init; } + /// Orders by creation time descending instead of the default job-ID cursor order. + public bool NewestFirst { get; init; } + /// Offset for creation-ordered monitoring pages. + public int Skip { get; init; } + + + /// Continue after the token returned by the preceding page, using the same filters. + public string? AfterJobId { get; init; } + + public void Validate() + { + ArgumentOutOfRangeException.ThrowIfNegative(Skip); + ArgumentOutOfRangeException.ThrowIfGreaterThan(Skip, 1000); + if (NewestFirst && AfterJobId is not null || !NewestFirst && Skip != 0) + throw new ArgumentException("Use Skip with NewestFirst, or AfterJobId with job-ID ordering."); + ArgumentOutOfRangeException.ThrowIfLessThan(Limit, 1); + ArgumentOutOfRangeException.ThrowIfGreaterThan(Limit, 1000); + } + +} + +public sealed record ScheduledDispatchState +{ + public required string DispatchId { get; init; } + public ScheduledDispatchKind Kind { get; init; } + + /// The transport destination for a queue message or publication. + public DestinationAddress? Destination { get; init; } + + public required ReadOnlyMemory Body { get; init; } + public MessageHeaders Headers { get; init; } = MessageHeaders.Empty; + public TransportSendOptions Options { get; init; } = new(); + public DateTimeOffset DueUtc { get; init; } + public string? ClaimOwner { get; init; } + public DateTimeOffset? ClaimExpiresUtc { get; init; } + public int Attempts { get; init; } +} + +/// Distinguishes a confirmed occurrence from overlap that may become eligible later. +public enum JobOccurrenceResult { AlreadyExists, Created, OverlapBlocked } + +public sealed record JobRequestOptions +{ + /// Relative initial delay. Mutually exclusive with RunAt. + public TimeSpan? Delay { get; init; } + /// Absolute earliest execution time. Mutually exclusive with Delay. + public DateTimeOffset? RunAt { get; init; } + /// Total execution attempts, including retries. Default three. + public int MaxAttempts { get; init; } = 3; + public JobRetryPolicy RetryPolicy { get; init; } = new(); + public string? JobId { get; init; } + public string? Name { get; init; } +} + +public sealed record JobTypeRegistration(string Name, Type JobType); + +public interface IJobTypeRegistry +{ + IReadOnlyCollection Names { get; } + string GetName(Type jobType); + Type Resolve(string name); +} + +public sealed class JobTypeRegistry : IJobTypeRegistry +{ + private readonly Dictionary _nameToType; + private readonly Dictionary _typeToName; + + public JobTypeRegistry(IEnumerable? registrations = null) + { + _nameToType = new Dictionary(StringComparer.Ordinal); + _typeToName = new Dictionary(); + + foreach (var registration in registrations ?? []) + Add(registration); + } + + public IReadOnlyCollection Names => _nameToType.Keys; + + public string GetName(Type jobType) + { + ArgumentNullException.ThrowIfNull(jobType); + if (!typeof(IJob).IsAssignableFrom(jobType)) + throw new ArgumentException("Job type must implement IJob.", nameof(jobType)); + + return _typeToName.TryGetValue(jobType, out string? name) + ? name + : jobType.FullName ?? jobType.Name; + } + + public Type Resolve(string name) + { + ArgumentException.ThrowIfNullOrEmpty(name); + + if (_nameToType.TryGetValue(name, out var registered)) + return registered; + + throw new JobException($"Job type \"{name}\" is not registered. Register each worker job with AddFoundatio().Jobs.AddJobType()."); + } + + private void Add(JobTypeRegistration registration) + { + ArgumentNullException.ThrowIfNull(registration); + ArgumentException.ThrowIfNullOrEmpty(registration.Name); + ArgumentNullException.ThrowIfNull(registration.JobType); + + if (!typeof(IJob).IsAssignableFrom(registration.JobType)) + throw new ArgumentException("Job type must implement IJob.", nameof(registration)); + + if (_nameToType.TryGetValue(registration.Name, out var existing) && existing != registration.JobType) + throw new InvalidOperationException($"Job type name \"{registration.Name}\" is already registered for \"{existing.FullName}\"."); + + _nameToType[registration.Name] = registration.JobType; + _typeToName[registration.JobType] = registration.Name; + } +} + +public sealed class JobHandle +{ + private readonly IJobMonitor _monitor; + private readonly TimeProvider _timeProvider; + private readonly Func> _requestCancellation; + + internal JobHandle(string jobId, IJobMonitor monitor, Func> requestCancellation, TimeProvider? timeProvider = null) + { + _timeProvider = timeProvider ?? TimeProvider.System; + JobId = jobId; + _monitor = monitor; + _requestCancellation = requestCancellation; + } + + public string JobId { get; } + + public Task GetStateAsync(CancellationToken cancellationToken = default) + { + return _monitor.GetAsync(JobId, cancellationToken); + } + + /// Waits for a terminal state. Timeout or cancellation stops this wait without cancelling the job. + public async Task WaitForCompletionAsync(TimeSpan? timeout = null, TimeSpan? pollInterval = null, CancellationToken cancellationToken = default) + { + var duration = timeout ?? TimeSpan.FromMinutes(5); + var interval = pollInterval ?? TimeSpan.FromMilliseconds(250); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(duration, TimeSpan.Zero); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(interval, TimeSpan.Zero); + using var deadline = new CancellationTokenSource(duration, _timeProvider); + using var waiting = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, deadline.Token); + try + { + while (true) + { + waiting.Token.ThrowIfCancellationRequested(); + var state = await _monitor.GetAsync(JobId, waiting.Token).WaitAsync(waiting.Token).AnyContext() + ?? throw new JobException($"Job {JobId} is unavailable; its retained history may have expired."); + if (state.Status is JobStatus.Completed or JobStatus.Failed or JobStatus.Cancelled or JobStatus.DeadLettered) + return state; + await Task.Delay(interval, _timeProvider, waiting.Token).AnyContext(); + } + } + catch (OperationCanceledException) when (deadline.IsCancellationRequested && !cancellationToken.IsCancellationRequested) + { + throw new TimeoutException($"Job {JobId} did not finish within {duration}."); + } + } + + public Task RequestCancellationAsync(CancellationToken cancellationToken = default) + { + return _requestCancellation(JobId, cancellationToken); + } +} + +/// +/// Passed to a job on each run. Gives the running job its identity and attempt number, plus store-backed progress +/// reporting, lease heartbeat (for long runs), and cooperative cancellation checks — the parts of +/// that are useful from inside job code. When a job is run outside the durable runtime +/// (for example directly in a test), the store-backed helpers are no-ops and cancellation reflects the supplied token. +/// +public sealed class JobExecutionContext +{ + private readonly IJobRuntimeStore? _store; + private readonly string _claimToken; + private readonly TimeSpan _lease; + private readonly ReadOnlyMemory? _payload; + private readonly string? _payloadType; + private readonly ISerializer? _serializer; + private readonly object? _detachedArguments; + + internal JobExecutionContext(string jobId, int attempt, CancellationToken cancellationToken, IJobRuntimeStore store, string claimToken, TimeSpan lease, ReadOnlyMemory? payload = null, string? payloadType = null, ISerializer? serializer = null) + { + JobId = jobId; + Attempt = attempt; + CancellationToken = cancellationToken; + _store = store; + _claimToken = claimToken; + _lease = lease; + _payload = payload; + _payloadType = payloadType; + _serializer = serializer; + } + + /// + /// Creates a detached context for running a job outside the durable runtime (tests or one-off invocations). + /// Progress reporting and lease renewal are no-ops; cancellation reflects ; + /// surfaces through without serialization. + /// + public JobExecutionContext(CancellationToken cancellationToken = default, string? jobId = null, int attempt = 1, object? arguments = null) + { + JobId = jobId ?? Guid.NewGuid().ToString("N"); + Attempt = attempt; + CancellationToken = cancellationToken; + _store = null; + _claimToken = String.Empty; + _lease = TimeSpan.Zero; + _detachedArguments = arguments; + } + + public string JobId { get; } + public int Attempt { get; } + public CancellationToken CancellationToken { get; } + + /// Whether this invocation carries typed arguments (see ). + public bool HasArguments => _detachedArguments is not null || _payload is not null; + + /// + /// The typed per-invocation arguments this job was enqueued with. Throws a descriptive + /// when the job was enqueued without arguments or the payload cannot be + /// read as (the stored discriminator is included for triage). + /// + public TArgs GetArguments() where TArgs : class + { + if (_detachedArguments is not null) + { + return _detachedArguments as TArgs + ?? throw new InvalidOperationException($"Job \"{JobId}\" arguments are of type \"{_detachedArguments.GetType().FullName}\", not the requested \"{typeof(TArgs).FullName}\"."); + } + + if (_payload is not { } payload) + throw new InvalidOperationException($"Job \"{JobId}\" was enqueued without arguments. Use EnqueueAsync(args) to supply a typed payload."); + + // The stored discriminator is a guard, not just forensics: deserializing type A's payload as a structurally + // similar type B usually SUCCEEDS with silently-wrong data, so a mismatch must fail before deserialization. + if (_payloadType is not null && !String.Equals(_payloadType, typeof(TArgs).FullName, StringComparison.Ordinal)) + throw new InvalidOperationException($"Job \"{JobId}\" arguments were stored as \"{_payloadType}\" but were requested as \"{typeof(TArgs).FullName}\". Request the type the job was enqueued with."); + + var serializer = _serializer ?? DefaultSerializer.Instance; + TArgs? args; + try + { + args = serializer.Deserialize(payload, typeof(TArgs)) as TArgs; + } + catch (Exception ex) + { + throw new InvalidOperationException($"Unable to deserialize job \"{JobId}\" arguments (stored type \"{_payloadType}\") as \"{typeof(TArgs).FullName}\".", ex); + } + + return args ?? throw new InvalidOperationException($"Job \"{JobId}\" arguments (stored type \"{_payloadType}\") deserialized to null as \"{typeof(TArgs).FullName}\"."); + } + + public async Task ReportProgressAsync(int? percent = null, string? message = null, CancellationToken cancellationToken = default) + { + CancellationToken.ThrowIfCancellationRequested(); + if (_store is not null && !await _store.ReportJobProgressAsync(JobId, _claimToken, percent, message, cancellationToken == default ? CancellationToken : cancellationToken).AnyContext()) + throw new JobException($"Job {JobId} no longer owns its execution lease."); + } + + /// + /// Forces an immediate lease renewal. Long-running jobs do NOT need to call this — the worker renews the lease + /// automatically on a supervised loop for the entire run (and cancels the run if the lease is lost). Use it only + /// to observe lease health explicitly (a false return means another node now owns the job). + /// + public Task RenewLeaseAsync(CancellationToken cancellationToken = default) + => _store?.RenewJobLeaseAsync(JobId, _claimToken, _lease, cancellationToken == default ? CancellationToken : cancellationToken) ?? Task.FromResult(true); + + public Task IsCancellationRequestedAsync(CancellationToken cancellationToken = default) + => _store?.IsCancellationRequestedAsync(JobId, cancellationToken == default ? CancellationToken : cancellationToken) ?? Task.FromResult(CancellationToken.IsCancellationRequested); +} + +public interface IJobMonitor +{ + Task GetAsync(string jobId, CancellationToken cancellationToken = default); + Task QueryAsync(JobQuery query, CancellationToken cancellationToken = default); + /// Counts retained jobs matching the query filters; pagination is ignored. + Task CountAsync(JobQuery query, CancellationToken cancellationToken = default); +} + +public interface IJobClient +{ + Task EnqueueAsync(JobRequestOptions? options = null, CancellationToken cancellationToken = default) where TJob : IJob; + + /// + /// Enqueues a job with typed per-invocation arguments. The args are serialized into the durable + /// via the runtime's serializer and surface to the job through + /// . + /// + Task EnqueueAsync(TArgs args, JobRequestOptions? options = null, CancellationToken cancellationToken = default) where TJob : IJob where TArgs : class; + + Task EnqueueAsync(Type jobType, JobRequestOptions? options = null, CancellationToken cancellationToken = default); + Task RequestCancellationAsync(string jobId, CancellationToken cancellationToken = default); +} + +public interface IJobWorker +{ + /// False while any execution slot is recovering from an infrastructure failure. + bool IsHealthy { get; } + Task RunAsync(string jobId, CancellationToken cancellationToken = default); + Task RunQueuedAsync(int limit = 100, CancellationToken cancellationToken = default); + /// Continuously replenishes independent execution slots until shutdown. + Task RunContinuouslyAsync(CancellationToken cancellationToken = default); +} + +/// +/// Durable storage for time-gated dispatches: delayed messages beyond a transport's native ceiling, store-parked +/// retry delays, and delayed publication. This is the only store contract the messaging client depends on — +/// a provider that offers durable scheduling without the full job runtime implements just this. +/// +public interface IScheduledDispatchStore +{ + Task ScheduleDispatchAsync(ScheduledDispatchState dispatch, CancellationToken cancellationToken = default); + // Claiming must be atomic per dispatch — in a relational store, a single conditional statement (e.g. + // SELECT ... FOR UPDATE SKIP LOCKED, or UPDATE ... WHERE due and unclaimed/lease-expired), never read-then-write — + // so concurrent nodes never claim the same dispatch. + Task> ClaimDueDispatchesAsync(DateTimeOffset now, int limit, string nodeId, TimeSpan lease, CancellationToken cancellationToken = default); + Task CompleteDispatchAsync(string dispatchId, string nodeId, CancellationToken cancellationToken = default); + Task ReleaseDispatchAsync(string dispatchId, string nodeId, DateTimeOffset nextDueUtc, CancellationToken cancellationToken = default); +} + +/// +/// The full job runtime store: job state persistence, queries, lease/ownership management, cancellation signaling, +/// and scheduled-dispatch storage. The state/lease/cancellation members are deliberately one contract — transitions +/// verify current ownership tokens and runtime lease expiry atomically, so +/// splitting them would break the compare-and-set semantics correctness depends on. +/// +public interface IJobRuntimeStore : IJobMonitor, IScheduledDispatchStore, IScheduledJobStore +{ + /// Whether state and cancellation are shared across processes. + bool IsShared { get; } + /// Records a broker attempt with a new ownership token. Missing, terminal, and stale attempts return null; this never schedules work. + Task BeginBrokerAttemptAsync(string jobId, int attempt, string nodeId, CancellationToken cancellationToken = default); + /// Records uncertain acceptance only before any broker attempt begins. + Task MarkEnqueueUnknownAsync(string jobId, string error, CancellationToken cancellationToken = default); + /// Refreshes history only for the current processing owner. It does not renew a broker delivery lease. + Task HeartbeatJobAsync(string jobId, string claimToken, CancellationToken cancellationToken = default); + /// Removes operational history. Active jobs owned by the runtime cannot be removed. + Task RemoveAsync(string jobId, CancellationToken cancellationToken = default); + /// Adds a named operational counter to the current hourly bucket. + Task IncrementCounterAsync(string name, string counterName, long value = 1, CancellationToken cancellationToken = default); + /// Reads retained hourly operational counters. + Task GetCounterStatsAsync(string name, TimeSpan? window = null, CancellationToken cancellationToken = default); + + /// Atomically creates an occurrence, enforcing its unique ID and optional overlap exclusion. + Task CreateOccurrenceAsync(JobState initial, bool allowOverlap = false, CancellationToken cancellationToken = default); + /// Atomically claims the oldest eligible due job, including recoverable expired executions. + Task ClaimNextAsync(JobClaimRequest request, CancellationToken cancellationToken = default); + /// Atomically claims a specific eligible job. + Task ClaimJobAsync(string jobId, JobClaimRequest request, CancellationToken cancellationToken = default); + /// Completes, retries, cancels, or returns work only while the supplied claim is still valid. + Task CompleteJobAsync(string jobId, string claimToken, JobCompletion completion, CancellationToken cancellationToken = default); + /// Renews only the current, unexpired execution claim. + Task RenewJobLeaseAsync(string jobId, string claimToken, TimeSpan lease, CancellationToken cancellationToken = default); + /// Updates progress only for the current owner, requiring an unexpired lease for runtime-owned jobs. + Task ReportJobProgressAsync(string jobId, string claimToken, int? percent = null, string? message = null, CancellationToken cancellationToken = default); + Task GetStatsAsync(CancellationToken cancellationToken = default); + /// Applies configured history, idempotency and unclaimed occurrence retention in bounded batches. + Task CleanupAsync(int limit = 1000, CancellationToken cancellationToken = default); + Task CreateIfAbsentAsync(JobState initial, CancellationToken cancellationToken = default); + Task RequestCancellationAsync(string jobId, CancellationToken cancellationToken = default); + Task IsCancellationRequestedAsync(string jobId, CancellationToken cancellationToken = default); +} + +public sealed partial class InMemoryJobRuntimeStore : IJobRuntimeStore +{ + private readonly Dictionary _jobs = new(StringComparer.Ordinal); + private readonly ConcurrentDictionary _dispatches = new(StringComparer.Ordinal); + private readonly TimeProvider _timeProvider; + private readonly object _lock = new(); + private readonly JobRuntimeStoreOptions _options; + private readonly Dictionary _deduplication = new(StringComparer.Ordinal); + private readonly PriorityQueue<(string Id, DateTimeOffset Expires), DateTimeOffset> _deduplicationExpiry = new(); + private readonly PriorityQueue<(string Id, DateTimeOffset Completed), DateTimeOffset> _history = new(); + private int _activeJobs; + private readonly Dictionary _active = new(StringComparer.Ordinal); + + public InMemoryJobRuntimeStore(TimeProvider? timeProvider = null, int maxJobs = 100000) + : this(new JobRuntimeStoreOptions { MaxActiveJobs = maxJobs }, timeProvider) { } + + public InMemoryJobRuntimeStore(JobRuntimeStoreOptions options, TimeProvider? timeProvider = null) + { + ArgumentNullException.ThrowIfNull(options); + options.Validate(); + _options = options; + _timeProvider = timeProvider ?? TimeProvider.System; + } + + public Task CreateIfAbsentAsync(JobState initial, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(initial); + cancellationToken.ThrowIfCancellationRequested(); + + lock (_lock) + { + ValidatePayload(initial.Payload?.Length ?? 0); + initial.Validate(); + initial.RetryPolicy.Validate(); + PurgeBrokerHistory(); + PurgeDeduplication(); + if (_jobs.ContainsKey(initial.JobId) || _deduplication.ContainsKey(initial.JobId)) + return Task.CompletedTask; + EnsureCapacity(); + var now = _timeProvider.GetUtcNow(); + StoreJob(initial with + { + CreatedUtc = initial.CreatedUtc == default ? now : initial.CreatedUtc, + LastUpdatedUtc = initial.LastUpdatedUtc == default ? now : initial.LastUpdatedUtc + }); + } + + return Task.CompletedTask; + } + + public Task GetAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + PurgeBrokerHistory(); + _jobs.TryGetValue(jobId, out var state); + return Task.FromResult(state); + } + } + + private IEnumerable MatchingJobs(JobQuery query) => _jobs.Values.Where(s => + (query.Name is null || s.Name == query.Name) && (query.Status is null || s.Status == query.Status) + && (query.QueueName is null || s.QueueName == query.QueueName)); + + public Task QueryAsync(JobQuery query, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(query); + query.Validate(); + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + PurgeBrokerHistory(); + var matching = MatchingJobs(query); + var candidates = (query.NewestFirst + ? matching.OrderByDescending(s => s.CreatedUtc).ThenByDescending(s => s.JobId, StringComparer.Ordinal).Skip(query.Skip) + : matching.Where(s => query.AfterJobId is null || StringComparer.Ordinal.Compare(s.JobId, query.AfterJobId) > 0).OrderBy(s => s.JobId, StringComparer.Ordinal)) + .Take(query.Limit + 1).ToArray(); + return Task.FromResult(new JobPage(candidates.Take(query.Limit).ToArray(), !query.NewestFirst && candidates.Length > query.Limit ? candidates[query.Limit - 1].JobId : null)); + } + } + + public Task CountAsync(JobQuery query, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(query); + query.Validate(); + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) { PurgeBrokerHistory(); return Task.FromResult(MatchingJobs(query).LongCount()); } + } + + private void EnsureCapacity() + { + PurgeBrokerHistory(); + PurgeDeduplication(); + if (_activeJobs >= _options.MaxActiveJobs) + throw CapacityExceeded("active jobs", _options.MaxActiveJobs); + if (_deduplication.Count >= _options.MaxDeduplicationRecords) + throw CapacityExceeded("idempotency records", _options.MaxDeduplicationRecords); + } + + private static JobException CapacityExceeded(string budget, int maximum) + { + JobInstruments.CapacityRejected.Add(1, new KeyValuePair("budget", budget)); + return new JobException($"Job store {budget} capacity ({maximum}) reached. Configure JobRuntimeStoreOptions to increase this budget."); + } + + private void ValidatePayload(long bytes) + { + if (bytes > _options.MaxPayloadBytes) + throw new JobException($"Payload exceeds the configured {_options.MaxPayloadBytes} byte limit."); + } + + private static bool IsActive(JobState state) => state.Status is JobStatus.Queued or JobStatus.Scheduled or JobStatus.Processing or JobStatus.RetryPending or JobStatus.EnqueueUnknown; + + private void StoreJob(JobState state) + { + _jobs.TryGetValue(state.JobId, out var previous); + bool active = IsActive(state); + _activeJobs += (active ? 1 : 0) - (previous is not null && IsActive(previous) ? 1 : 0); + if (!active && state.CompletedUtc is null) + state = state with { CompletedUtc = _timeProvider.GetUtcNow() }; + if (state.ExecutionOwner == JobExecutionOwner.Broker && state.HistoryRetention is { } retention) + { + if (previous?.HistoryExpiresUtc is { } previousExpiry) _brokerExpiry.Remove((previousExpiry, state.JobId)); + state = state with { HistoryExpiresUtc = _timeProvider.GetUtcNow().Add(retention) }; + _brokerExpiry.Add((state.HistoryExpiresUtc.Value, state.JobId)); + } + _jobs[state.JobId] = state; + if (active) _active[state.JobId] = state; + else _active.Remove(state.JobId); + if (previous is null) + _deduplication[state.JobId] = DateTimeOffset.MaxValue; + if (!active && (previous is null || previous.CompletedUtc != state.CompletedUtc || IsActive(previous))) + { + var completed = state.CompletedUtc!.Value; + var expires = completed.Add(_options.DeduplicationRetention); + if (state.ExecutionOwner == JobExecutionOwner.Runtime) + { + _deduplication[state.JobId] = expires; + _deduplicationExpiry.Enqueue((state.JobId, expires), expires); + } + _history.Enqueue((state.JobId, completed), completed); + TrimHistory(1000, pressureOnly: true); + } + } + + private void PurgeDeduplication() + { + var now = _timeProvider.GetUtcNow(); + while (_deduplicationExpiry.TryPeek(out var item, out var expires) && expires <= now) + { + _deduplicationExpiry.Dequeue(); + if (_deduplication.TryGetValue(item.Id, out var current) && current == item.Expires) + _deduplication.Remove(item.Id); + } + } + + private int TrimHistory(int limit, bool pressureOnly = false) + { + int removed = 0; + var cutoff = _timeProvider.GetUtcNow().Subtract(_options.HistoryRetention); + while (removed < limit && _history.TryPeek(out var item, out var completed)) + { + if (!_jobs.TryGetValue(item.Id, out var retained) || retained.CompletedUtc != item.Completed) + { + _history.Dequeue(); + continue; + } + if (_jobs.Count - _activeJobs <= _options.MaxHistoryJobs && (pressureOnly || completed > cutoff)) + break; + _history.Dequeue(); + if (!IsActive(retained)) + { + ForgetJob(retained); + removed++; + } + } + return removed; + } + + public Task GetStatsAsync(CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + PurgeBrokerHistory(); + return Task.FromResult(new JobRuntimeStoreStats(_activeJobs, _jobs.Count - _activeJobs, _deduplication.Count, _dispatches.Count)); + } + } + + public Task CleanupAsync(int limit = 1000, CancellationToken cancellationToken = default) + { + ArgumentOutOfRangeException.ThrowIfLessThan(limit, 1); + ArgumentOutOfRangeException.ThrowIfGreaterThan(limit, 1000); + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + PurgeBrokerHistory(); + PurgeDeduplication(); + var now = _timeProvider.GetUtcNow(); + foreach (var state in _jobs.Values.Where(s => s.ExecutionOwner == JobExecutionOwner.Runtime && s.RequiredNodeId is not null && s.Attempt == 0 && s.Status is JobStatus.Queued or JobStatus.Scheduled && s.ExpiresUtc <= now).Take(limit).ToArray()) + StoreJob(state with { Status = JobStatus.Cancelled, CompletedUtc = now, LastUpdatedUtc = now, ResultMessage = "Unclaimed per-node occurrence expired." }); + return Task.FromResult(TrimHistory(limit)); + } + } + + public Task RequestCancellationAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + return Task.FromResult(UpdateJob(jobId, state => state with + { + CancellationRequested = true, + Status = state.ExecutionOwner == JobExecutionOwner.Runtime && state.Status is JobStatus.Queued or JobStatus.Scheduled ? JobStatus.Cancelled : state.Status, + CompletedUtc = state.ExecutionOwner == JobExecutionOwner.Runtime && state.Status is JobStatus.Queued or JobStatus.Scheduled ? _timeProvider.GetUtcNow() : state.CompletedUtc, + LastUpdatedUtc = _timeProvider.GetUtcNow() + })); + } + + public Task IsCancellationRequestedAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + PurgeBrokerHistory(); + return Task.FromResult(_jobs.TryGetValue(jobId, out var state) && state.CancellationRequested); + } + } + + public Task ScheduleDispatchAsync(ScheduledDispatchState dispatch, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(dispatch); + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + if (_dispatches.ContainsKey(dispatch.DispatchId)) + return Task.CompletedTask; + ValidatePayload(dispatch.Body.Length + dispatch.Headers.Sum(h => (long)System.Text.Encoding.UTF8.GetByteCount(h.Key) + System.Text.Encoding.UTF8.GetByteCount(h.Value))); + if (_dispatches.Count >= _options.MaxScheduledDispatches) + throw CapacityExceeded("scheduled dispatches", _options.MaxScheduledDispatches); + _dispatches.TryAdd(dispatch.DispatchId, dispatch with { Body = dispatch.Body.ToArray() }); + } + return Task.CompletedTask; + } + + public Task> ClaimDueDispatchesAsync(DateTimeOffset now, int limit, string nodeId, TimeSpan lease, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + ArgumentException.ThrowIfNullOrEmpty(nodeId); + + lock (_lock) + { + var due = _dispatches.Values + .Where(d => d.DueUtc <= now && (String.IsNullOrEmpty(d.ClaimOwner) || d.ClaimExpiresUtc <= _timeProvider.GetUtcNow())) + .OrderBy(d => d.DueUtc) + .Take(Math.Max(1, limit)) + .ToArray(); + + for (int index = 0; index < due.Length; index++) + { + var claimed = due[index] with + { + ClaimOwner = nodeId, + ClaimExpiresUtc = _timeProvider.GetUtcNow().Add(lease), + Attempts = due[index].Attempts + 1 + }; + _dispatches[claimed.DispatchId] = claimed; + due[index] = claimed; + } + + return Task.FromResult>(due); + } + } + + public Task CompleteDispatchAsync(string dispatchId, string nodeId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + lock (_lock) + { + if (_dispatches.TryGetValue(dispatchId, out var dispatch) && dispatch.ClaimOwner == nodeId && dispatch.ClaimExpiresUtc > _timeProvider.GetUtcNow()) + return Task.FromResult(_dispatches.TryRemove(dispatchId, out _)); + return Task.FromResult(false); + } + } + + public Task ReleaseDispatchAsync(string dispatchId, string nodeId, DateTimeOffset nextDueUtc, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + lock (_lock) + { + if (_dispatches.TryGetValue(dispatchId, out var dispatch) && dispatch.ClaimOwner == nodeId && dispatch.ClaimExpiresUtc > _timeProvider.GetUtcNow()) + { + _dispatches[dispatchId] = dispatch with + { + DueUtc = nextDueUtc, + ClaimOwner = null, + ClaimExpiresUtc = null + }; + } + } + + return Task.CompletedTask; + } + + private bool UpdateJob(string jobId, Func update) + { + lock (_lock) + { + PurgeBrokerHistory(); + if (!_jobs.TryGetValue(jobId, out var current)) + return false; + if (current.ExecutionOwner == JobExecutionOwner.Broker && !IsActive(current)) + return false; + + StoreJob(update(current)); + return true; + } + } + + +} + +public sealed class JobClient : IJobClient +{ + private readonly IJobRuntimeStore _store; + private readonly TimeProvider _timeProvider; + private readonly IJobTypeRegistry _jobTypes; + private readonly bool _requireRegisteredTypes; + private readonly ISerializer _serializer; + + public JobClient(IJobRuntimeStore store, TimeProvider? timeProvider = null, IJobTypeRegistry? jobTypes = null, ISerializer? serializer = null) + { + _store = store ?? throw new ArgumentNullException(nameof(store)); + _timeProvider = timeProvider ?? TimeProvider.System; + _jobTypes = jobTypes ?? new JobTypeRegistry(); + _requireRegisteredTypes = jobTypes is not null; + _serializer = serializer ?? DefaultSerializer.Instance; + } + + public Task EnqueueAsync(JobRequestOptions? options = null, CancellationToken cancellationToken = default) where TJob : IJob + { + return EnqueueCoreAsync(typeof(TJob), args: null, options, cancellationToken); + } + + public Task EnqueueAsync(TArgs args, JobRequestOptions? options = null, CancellationToken cancellationToken = default) where TJob : IJob where TArgs : class + { + ArgumentNullException.ThrowIfNull(args); + return EnqueueCoreAsync(typeof(TJob), args, options, cancellationToken); + } + + public Task EnqueueAsync(Type jobType, JobRequestOptions? options = null, CancellationToken cancellationToken = default) + { + return EnqueueCoreAsync(jobType, args: null, options, cancellationToken); + } + + private async Task EnqueueCoreAsync(Type jobType, object? args, JobRequestOptions? options, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(jobType); + if (!typeof(IJob).IsAssignableFrom(jobType)) + throw new ArgumentException("Job type must implement IJob.", nameof(jobType)); + + JobArgumentContract.Validate(jobType, args); + options ??= new JobRequestOptions(); + ArgumentOutOfRangeException.ThrowIfLessThan(options.MaxAttempts, 1); + options.RetryPolicy.Validate(); + if (options.Delay is not null && options.RunAt is not null) + throw new ArgumentException("Specify either Delay or RunAt, not both.", nameof(options)); + if (options.Delay is { } delay) + ArgumentOutOfRangeException.ThrowIfLessThan(delay, TimeSpan.Zero); + string jobId = options.JobId ?? Guid.NewGuid().ToString("N"); + string name = options.Name ?? jobType.Name; + var now = _timeProvider.GetUtcNow(); + + string typeName = _jobTypes.GetName(jobType); + if (_requireRegisteredTypes) _jobTypes.Resolve(typeName); + await _store.CreateIfAbsentAsync(new JobState + { + JobId = jobId, + Name = name, + JobType = typeName, + MaxAttempts = options.MaxAttempts, + RetryPolicy = options.RetryPolicy, + AvailableUtc = options.RunAt ?? (options.Delay is { } delayValue ? now.Add(delayValue) : now), + // Explicitly typed: the byte[] -> ReadOnlyMemory conversion maps a null array to an EMPTY memory, which + // would make an argless job look like it carries a zero-byte payload. + Payload = args is null ? null : (ReadOnlyMemory?)_serializer.SerializeToBytes(args), + PayloadType = args?.GetType().FullName, + Status = JobStatus.Queued, + CreatedUtc = now, + LastUpdatedUtc = now + }, cancellationToken).ConfigureAwait(false); + + return new JobHandle(jobId, _store, RequestCancellationAsync, _timeProvider); + } + + public Task RequestCancellationAsync(string jobId, CancellationToken cancellationToken = default) + { + return _store.RequestCancellationAsync(jobId, cancellationToken); + } +} + +/// +/// Resolves a stable, process-unique node identity used for job claims and per-node scheduling. +/// Honors the FOUNDATIO_NODE_ID environment variable when set; otherwise combines machine name, +/// process id, and a process-lifetime token so co-located worker processes do not collapse to one identity. +/// +internal static class NodeIdentity +{ + public static string? Configured => Environment.GetEnvironmentVariable("FOUNDATIO_NODE_ID") is { } value && !String.IsNullOrWhiteSpace(value) ? value : null; + public static string RequireStable(string? nodeId) + => !String.IsNullOrWhiteSpace(nodeId) ? nodeId : Configured + ?? throw new JobException("PerNode schedules require a stable NodeId. Configure Jobs.ConfigureWorker(o => o with { NodeId = ... }) or FOUNDATIO_NODE_ID; use Global for fleet-wide work."); + public static string Current { get; } = Resolve(); + + private static string Resolve() + { + string? configured = Environment.GetEnvironmentVariable("FOUNDATIO_NODE_ID"); + if (!String.IsNullOrEmpty(configured)) + return configured; + + return $"{Environment.MachineName}:{Environment.ProcessId}:{Guid.NewGuid().ToString("N")[..8]}"; + } +} + +/// +/// Optional dependencies and tuning for . Prefer the options-taking constructor when +/// hand-wiring a worker; unset properties fall back to the same defaults as the full constructor. +/// +public sealed record JobWorkerOptions +{ + public Microsoft.Extensions.Logging.ILoggerFactory? LoggerFactory { get; init; } + public TimeProvider? TimeProvider { get; init; } + public string? NodeId { get; init; } + public TimeSpan? Lease { get; init; } + public IJobTypeRegistry? JobTypes { get; init; } + public TimeSpan? CancellationPollInterval { get; init; } + public ISerializer? Serializer { get; init; } + public int MaxConcurrency { get; init; } = 1; +} diff --git a/src/Foundatio/Jobs/JobRuntimeStoreOptions.cs b/src/Foundatio/Jobs/JobRuntimeStoreOptions.cs new file mode 100644 index 000000000..238afa6a3 --- /dev/null +++ b/src/Foundatio/Jobs/JobRuntimeStoreOptions.cs @@ -0,0 +1,30 @@ +using System; + +namespace Foundatio.Jobs; + +/// Independent budgets for executable work, history, idempotency and delayed messaging. +public sealed record JobRuntimeStoreOptions +{ + public int MaxActiveJobs { get; init; } = 100000; + public int MaxHistoryJobs { get; init; } = 100000; + public TimeSpan HistoryRetention { get; init; } = TimeSpan.FromDays(7); + /// Minimum time an ID remains reserved after completion, including after history eviction. + public TimeSpan DeduplicationRetention { get; init; } = TimeSpan.FromDays(7); + public int MaxDeduplicationRecords { get; init; } = 1000000; + public int MaxScheduledDispatches { get; init; } = 100000; + public int MaxPayloadBytes { get; init; } = 1048576; + + public void Validate() + { + ArgumentOutOfRangeException.ThrowIfLessThan(MaxActiveJobs, 1); + ArgumentOutOfRangeException.ThrowIfNegative(MaxHistoryJobs); + ArgumentOutOfRangeException.ThrowIfLessThan(MaxDeduplicationRecords, MaxActiveJobs); + ArgumentOutOfRangeException.ThrowIfLessThan(MaxScheduledDispatches, 1); + ArgumentOutOfRangeException.ThrowIfLessThan(MaxPayloadBytes, 1); + ArgumentOutOfRangeException.ThrowIfLessThan(HistoryRetention, TimeSpan.Zero); + ArgumentOutOfRangeException.ThrowIfLessThan(DeduplicationRetention, HistoryRetention); + } +} + +/// Current usage for admission and retention monitoring. +public sealed record JobRuntimeStoreStats(long ActiveJobs, long HistoryJobs, long DeduplicationRecords, long ScheduledDispatches); diff --git a/src/Foundatio/Jobs/JobScheduler.cs b/src/Foundatio/Jobs/JobScheduler.cs new file mode 100644 index 000000000..8c0a2dcd5 --- /dev/null +++ b/src/Foundatio/Jobs/JobScheduler.cs @@ -0,0 +1,481 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Linq; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Cronos; +using Foundatio.Serializer; + +namespace Foundatio.Jobs; + +public enum ScheduledJobScope +{ + Global, + PerNode +} + +public enum OverlapPolicy +{ + SkipIfRunning, + AllowConcurrent +} + +public sealed record ScheduledJobDefinition +{ + /// + /// The schedule name a job type gets when none is given explicitly (the type's simple name). This is the single + /// home of the convention shared by AddCronJob<TJob> and the generic + /// overloads, so type-addressed management always finds type-registered schedules. + /// + public static string DefaultNameFor(Type jobType) + { + ArgumentNullException.ThrowIfNull(jobType); + return jobType.Name; + } + + public required string Name { get; init; } + public required string Cron { get; init; } + public required string JobType { get; init; } + public string TimeZoneId { get; init; } = "UTC"; + public ScheduledJobScope Scope { get; init; } = ScheduledJobScope.Global; + public OverlapPolicy Overlap { get; init; } = OverlapPolicy.SkipIfRunning; + public TimeSpan? MisfireWindow { get; init; } + /// Maximum TOTAL run attempts for a failed occurrence before it ends in Failed. Default 3. + public int MaxAttempts { get; init; } = 3; + public JobRetryPolicy RetryPolicy { get; init; } = new(); + /// Retires unclaimed per-node occurrences after this interval; active executions are unaffected. + public TimeSpan UnclaimedLifetime { get; init; } = TimeSpan.FromDays(1); + + /// Serialized arguments copied into each occurrence. + public ReadOnlyMemory? Payload { get; init; } + public string? PayloadType { get; init; } + /// Store revision. Read the latest definition before editing an existing schedule. + public long Revision { get; init; } + /// Increase to intentionally replace persisted schedule settings from declarative configuration. + public int ConfigurationVersion { get; init; } = 1; + + public bool Enabled { get; init; } = true; + + /// Validates a serializable schedule before persisting it. + public void Validate() + { + ArgumentException.ThrowIfNullOrWhiteSpace(Name); + ArgumentException.ThrowIfNullOrWhiteSpace(JobType); + ArgumentOutOfRangeException.ThrowIfLessThan(MaxAttempts, 1); + RetryPolicy.Validate(); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(UnclaimedLifetime, TimeSpan.Zero); + if (!Enum.IsDefined(Scope)) throw new ArgumentOutOfRangeException(nameof(Scope)); + if (!Enum.IsDefined(Overlap)) throw new ArgumentOutOfRangeException(nameof(Overlap)); + ArgumentOutOfRangeException.ThrowIfNegative(Revision); + ArgumentOutOfRangeException.ThrowIfNegative(ConfigurationVersion); + if (MisfireWindow is { } window && (window < TimeSpan.Zero || window > TimeSpan.FromDays(1))) + throw new ArgumentOutOfRangeException(nameof(MisfireWindow), "MisfireWindow must be between zero and one day."); + JobScheduleProcessor.ValidateCron(Cron); + _ = TimeZoneInfo.FindSystemTimeZoneById(TimeZoneId); + } + +} + +/// +/// Options for a declaratively-registered CRON job — AddFoundatio().Jobs.AddCronJob<TJob>(cron, o => ...). +/// The registered definitions are scheduled automatically when the explicitly registered job scheduler starts. +/// +public sealed class CronJobOptions +{ + /// Schedule name (must be unique across scheduled jobs). Defaults to the job type name. + public string? Name { get; set; } + + /// Global (one instance per tick, the default) or PerNode (every instance runs it per tick). + public ScheduledJobScope Scope { get; set; } = ScheduledJobScope.Global; + + /// Whether a new occurrence is skipped while a prior one is still running. Default SkipIfRunning. + public OverlapPolicy Overlap { get; set; } = OverlapPolicy.SkipIfRunning; + + /// How late a missed occurrence may still fire. Null uses the scheduler default. + public TimeSpan? MisfireWindow { get; set; } + + /// Maximum TOTAL run attempts for a failed occurrence before reaching Failed. Default 3. + public int MaxAttempts { get; set; } = 3; + public JobRetryPolicy RetryPolicy { get; set; } = new(); + public TimeSpan UnclaimedLifetime { get; set; } = TimeSpan.FromDays(1); + + /// Whether the schedule is active. Default true. + public bool Enabled { get; set; } = true; + + /// Time zone the CRON expression is evaluated in. Null uses the scheduler default (UTC). + public TimeZoneInfo? TimeZone { get; set; } + + /// Increase when deploying an intentional change to this declared schedule. + public int ConfigurationVersion { get; set; } = 1; +} + +/// +/// Storage contract for scheduled (CRON) job definitions. Implementations persist the definitions themselves; +/// is the user-facing management API layered on top of this store. +/// +public interface IScheduledJobStore +{ + /// Creates or updates a schedule, requiring the supplied Revision to match the stored revision. + Task ScheduleAsync(ScheduledJobDefinition definition, CancellationToken cancellationToken = default); + /// Applies a newer declared configuration; repeated or older deployments preserve persisted edits. + Task ReconcileAsync(ScheduledJobDefinition definition, CancellationToken cancellationToken = default); + Task GetScheduleAsync(string name, CancellationToken cancellationToken = default); + Task UnscheduleAsync(string name, CancellationToken cancellationToken = default); + Task> GetSchedulesAsync(ScheduleQuery? query = null, CancellationToken cancellationToken = default); +} + +/// +/// Runtime management surface for scheduled (CRON) jobs: list and inspect schedules, add or replace definitions, +/// change a schedule's cron expression, enable/disable, and trigger an immediate occurrence. Declaratively-registered +/// jobs (AddCronJob<TJob>) and definitions added here share the same store, +/// so both are manageable through this interface. The DI-configured manager requires job types to be registered +/// with Jobs.AddJobType<TJob>() before adding schedules. +/// +public interface IScheduledJobManager +{ + Task> GetSchedulesAsync(ScheduleQuery? query = null, CancellationToken cancellationToken = default); + Task GetScheduleAsync(string name, CancellationToken cancellationToken = default); + + /// Adds a new schedule or replaces the existing definition with the same name. + Task ScheduleAsync(ScheduledJobDefinition definition, CancellationToken cancellationToken = default); + + /// Creates a schedule for an argument-free job. + Task ScheduleAsync(string cron, Action? configure = null, CancellationToken cancellationToken = default) where TJob : IJob; + /// Creates a schedule with arguments constrained to the job contract. + Task ScheduleAsync(string cron, TArgs arguments, Action? configure = null, CancellationToken cancellationToken = default) + where TJob : IJob where TArgs : class; + + Task UnscheduleAsync(string name, CancellationToken cancellationToken = default); + + /// Changes an existing schedule's cron expression (validated). Returns false when no schedule has that name. + Task RescheduleAsync(string name, string cronSchedule, CancellationToken cancellationToken = default); + + /// + /// Enables or disables a schedule. A disabled schedule materializes no occurrences (and cannot be triggered) + /// until re-enabled. Returns false when no schedule has that name. + /// + Task SetEnabledAsync(string name, bool enabled, CancellationToken cancellationToken = default); + + /// + /// Triggers an immediate occurrence of the named schedule, independent of its cron expression, and returns a + /// for watching or cancelling the run. The occurrence is durable (materialized into the + /// runtime store and executed by a job worker) and uses the definition's retry budget and + /// serialized arguments. Manual occurrences respect the configured overlap policy. + /// Throws when the schedule does not exist, is disabled, or already has active work that excludes overlap. + /// + Task TriggerAsync(string name, CancellationToken cancellationToken = default); +} + +/// +/// Type-addressed conveniences over : they resolve the schedule name from the job +/// type via — the same default AddCronJob<TJob> uses — +/// so a schedule registered without an explicit name is manageable by its type alone. Schedules registered under a +/// custom are addressed with the string overloads. +/// +public static class ScheduledJobManagerExtensions +{ + public static Task GetScheduleAsync(this IScheduledJobManager manager, CancellationToken cancellationToken = default) where TJob : IJob + => Manager(manager).GetScheduleAsync(ScheduledJobDefinition.DefaultNameFor(typeof(TJob)), cancellationToken); + + public static Task TriggerAsync(this IScheduledJobManager manager, CancellationToken cancellationToken = default) where TJob : IJob + => Manager(manager).TriggerAsync(ScheduledJobDefinition.DefaultNameFor(typeof(TJob)), cancellationToken); + + public static Task RescheduleAsync(this IScheduledJobManager manager, string cronSchedule, CancellationToken cancellationToken = default) where TJob : IJob + => Manager(manager).RescheduleAsync(ScheduledJobDefinition.DefaultNameFor(typeof(TJob)), cronSchedule, cancellationToken); + + public static Task SetEnabledAsync(this IScheduledJobManager manager, bool enabled, CancellationToken cancellationToken = default) where TJob : IJob + => Manager(manager).SetEnabledAsync(ScheduledJobDefinition.DefaultNameFor(typeof(TJob)), enabled, cancellationToken); + + public static Task UnscheduleAsync(this IScheduledJobManager manager, CancellationToken cancellationToken = default) where TJob : IJob + => Manager(manager).UnscheduleAsync(ScheduledJobDefinition.DefaultNameFor(typeof(TJob)), cancellationToken); + + private static IScheduledJobManager Manager(IScheduledJobManager manager) + { + ArgumentNullException.ThrowIfNull(manager); + return manager; + } +} + +public sealed class ScheduledJobManager : IScheduledJobManager +{ + private readonly string? _nodeId; + private readonly IScheduledJobStore _scheduleStore; + private readonly IJobRuntimeStore _store; + private readonly IJobTypeRegistry _jobTypes; + private readonly bool _requireRegisteredTypes; + private readonly ISerializer _serializer; + private readonly TimeProvider _timeProvider; + + public ScheduledJobManager(IScheduledJobStore scheduleStore, IJobRuntimeStore store, IJobTypeRegistry? jobTypes = null, ISerializer? serializer = null, TimeProvider? timeProvider = null, string? nodeId = null) + { + _nodeId = nodeId; + _scheduleStore = scheduleStore ?? throw new ArgumentNullException(nameof(scheduleStore)); + _store = store ?? throw new ArgumentNullException(nameof(store)); + _jobTypes = jobTypes ?? new JobTypeRegistry(); + _requireRegisteredTypes = jobTypes is not null; + _serializer = serializer ?? DefaultSerializer.Instance; + _timeProvider = timeProvider ?? TimeProvider.System; + } + + public Task> GetSchedulesAsync(ScheduleQuery? query = null, CancellationToken cancellationToken = default) + => _scheduleStore.GetSchedulesAsync(query, cancellationToken); + + public Task GetScheduleAsync(string name, CancellationToken cancellationToken = default) + => _scheduleStore.GetScheduleAsync(name, cancellationToken); + + public Task ScheduleAsync(ScheduledJobDefinition definition, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(definition); + if (_requireRegisteredTypes && definition.JobType is not null) + _jobTypes.Resolve(definition.JobType); + return _scheduleStore.ScheduleAsync(definition, cancellationToken); + } + + public Task ScheduleAsync(string cron, Action? configure = null, CancellationToken cancellationToken = default) where TJob : IJob + => ScheduleAsync(typeof(TJob), cron, null, configure, cancellationToken); + + public Task ScheduleAsync(string cron, TArgs arguments, Action? configure = null, CancellationToken cancellationToken = default) + where TJob : IJob where TArgs : class + => ScheduleAsync(typeof(TJob), cron, arguments, configure, cancellationToken); + + private Task ScheduleAsync(Type jobType, string cron, object? arguments, Action? configure, CancellationToken cancellationToken) + { + var options = new CronJobOptions(); + configure?.Invoke(options); + return ScheduleAsync(new ScheduledJobRegistration(jobType, cron, options, arguments).Create(_jobTypes, _serializer), cancellationToken); + } + + public Task UnscheduleAsync(string name, CancellationToken cancellationToken = default) + => _scheduleStore.UnscheduleAsync(name, cancellationToken); + + public async Task RescheduleAsync(string name, string cronSchedule, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrEmpty(cronSchedule); + JobScheduleProcessor.ValidateCron(cronSchedule); + + var definition = await GetScheduleAsync(name, cancellationToken).ConfigureAwait(false); + if (definition is null) + return false; + + await _scheduleStore.ScheduleAsync(definition with { Cron = cronSchedule }, cancellationToken).ConfigureAwait(false); + return true; + } + + public async Task SetEnabledAsync(string name, bool enabled, CancellationToken cancellationToken = default) + { + var definition = await GetScheduleAsync(name, cancellationToken).ConfigureAwait(false); + if (definition is null) + return false; + + if (definition.Enabled != enabled) + await _scheduleStore.ScheduleAsync(definition with { Enabled = enabled }, cancellationToken).ConfigureAwait(false); + + return true; + } + + public async Task TriggerAsync(string name, CancellationToken cancellationToken = default) + { + var definition = await GetScheduleAsync(name, cancellationToken).ConfigureAwait(false) + ?? throw new ScheduledJobNotFoundException(name); + + if (definition.JobType is null) + throw new JobException($"Scheduled job \"{name}\" has no job type and cannot be triggered."); + + // The occurrence-run path releases (and endlessly re-claims) dispatches whose definition is disabled, so a + // trigger of a disabled schedule would park forever rather than run — refuse it up front instead. + if (!definition.Enabled) + throw new ScheduledJobDisabledException(name); + + var now = _timeProvider.GetUtcNow(); + + // Unique id: manual runs are deliberate, so they never dedupe against each other or against cron occurrences + // (whose deterministic "{name}:{timestamp}:{scope}" ids exist precisely to dedupe scheduler ticks). + string jobId = $"{name}:manual:{Guid.NewGuid():N}"; + + var occurrence = new JobState + { + JobId = jobId, + Name = definition.Name, + ScheduleName = definition.Name, + JobType = definition.JobType, + MaxAttempts = definition.MaxAttempts, + RequiredNodeId = definition.Scope == ScheduledJobScope.PerNode ? NodeIdentity.RequireStable(_nodeId) : null, + ExpiresUtc = definition.Scope == ScheduledJobScope.PerNode ? now.Add(definition.UnclaimedLifetime) : null, + RetryPolicy = definition.RetryPolicy, + Payload = definition.Payload, + PayloadType = definition.PayloadType, + Status = JobStatus.Queued, + CreatedUtc = now, + LastUpdatedUtc = now, + ScheduledForUtc = now + }; + if (await _store.CreateOccurrenceAsync(occurrence, definition.Overlap == OverlapPolicy.AllowConcurrent, cancellationToken).ConfigureAwait(false) != JobOccurrenceResult.Created) + throw new JobException($"Scheduled job {name} already has pending or running work."); + + return new JobHandle(jobId, _store, _store.RequestCancellationAsync, _timeProvider); + } +} + +/// +/// Optional dependencies for . Prefer the options-taking constructor when +/// hand-wiring a processor; unset properties fall back to the same defaults as the full constructor. +/// +public sealed record JobScheduleProcessorOptions +{ + public TimeProvider? TimeProvider { get; init; } + public string? NodeId { get; init; } +} + +public sealed class JobScheduleProcessor +{ + private static readonly TimeSpan DefaultMisfireWindow = TimeSpan.FromMinutes(1); + + private readonly IScheduledJobStore _scheduleStore; + private readonly IJobRuntimeStore _store; + private readonly TimeProvider _timeProvider; + private readonly string? _nodeId; + private readonly ConcurrentDictionary _cache = new(StringComparer.Ordinal); + private sealed class CachedSchedule(ScheduledJobDefinition definition) + { + public ScheduledJobDefinition Definition { get; } = definition; + public CronExpression Cron { get; } = ParseCron(definition.Cron); + public TimeZoneInfo TimeZone { get; } = TimeZoneInfo.FindSystemTimeZoneById(definition.TimeZoneId); + private long _lastConfirmedTicks; + public DateTimeOffset LastConfirmed => new(Interlocked.Read(ref _lastConfirmedTicks), TimeSpan.Zero); + public void Confirm(DateTimeOffset occurrence) + { + long ticks = occurrence.UtcTicks; + long previous; + do { previous = Interlocked.Read(ref _lastConfirmedTicks); if (previous >= ticks) return; } + while (Interlocked.CompareExchange(ref _lastConfirmedTicks, ticks, previous) != previous); + } + } + + public JobScheduleProcessor(IScheduledJobStore scheduleStore, IJobRuntimeStore store, JobScheduleProcessorOptions? options = null) + { + _scheduleStore = scheduleStore ?? throw new ArgumentNullException(nameof(scheduleStore)); + _store = store ?? throw new ArgumentNullException(nameof(store)); + _timeProvider = options?.TimeProvider ?? TimeProvider.System; + _nodeId = options?.NodeId ?? NodeIdentity.Configured; + } + + public Task> EnqueueDueOccurrencesAsync(CancellationToken cancellationToken = default) + { + return EnqueueDueOccurrencesAsync(_timeProvider.GetUtcNow(), cancellationToken); + } + + public async Task> EnqueueDueOccurrencesAsync(DateTimeOffset utcNow, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + var scheduled = new List(); + var seen = new HashSet(StringComparer.Ordinal); + await foreach (var definition in EnumerateSchedulesAsync(cancellationToken).ConfigureAwait(false)) + { + seen.Add(definition.Name); + if (!definition.Enabled) + continue; + + var cached = _cache.AddOrUpdate(definition.Name, _ => new CachedSchedule(definition), + (_, previous) => previous.Definition.Revision == definition.Revision && previous.Definition.Cron == definition.Cron && previous.Definition.TimeZoneId == definition.TimeZoneId ? previous : new CachedSchedule(definition)); + var cron = cached.Cron; + var timeZone = cached.TimeZone; + var window = definition.MisfireWindow ?? DefaultMisfireWindow; + if (window < TimeSpan.Zero) + throw new ArgumentOutOfRangeException(nameof(definition), window, "MisfireWindow must be greater than or equal to zero."); + + string scopeKey = GetScopeKey(definition); + + // Materialize every occurrence that fell due within the misfire window, not just the most recent, so a + // scheduler that lagged behind the cadence does not silently drop intermediate ticks. Deterministic + // occurrence ids dedupe across overlapping windows and across nodes ticking simultaneously. + var occurrences = cron.GetOccurrences(utcNow - window, utcNow, timeZone, fromInclusive: true, toInclusive: true).ToList(); + if (occurrences.Count == 0) + continue; + + if (definition.Overlap == OverlapPolicy.SkipIfRunning) + occurrences = [occurrences[^1]]; + + foreach (var occurrence in occurrences) + { + if (occurrence <= cached.LastConfirmed) + continue; + var state = new JobState + { + JobId = CreateOccurrenceId(definition.Name, occurrence, scopeKey), + Name = definition.Name, + ScheduleName = definition.Name, + JobType = definition.JobType, + MaxAttempts = definition.MaxAttempts, + RequiredNodeId = definition.Scope == ScheduledJobScope.PerNode ? NodeIdentity.RequireStable(_nodeId) : null, + ExpiresUtc = definition.Scope == ScheduledJobScope.PerNode ? utcNow.Add(definition.UnclaimedLifetime) : null, + RetryPolicy = definition.RetryPolicy, + Payload = definition.Payload, + PayloadType = definition.PayloadType, + Status = JobStatus.Queued, + CreatedUtc = utcNow, + LastUpdatedUtc = utcNow, + ScheduledForUtc = occurrence + }; + var result = await _store.CreateOccurrenceAsync(state, definition.Overlap == OverlapPolicy.AllowConcurrent, cancellationToken).ConfigureAwait(false); + if (result != JobOccurrenceResult.OverlapBlocked) + cached.Confirm(occurrence); + if (result == JobOccurrenceResult.Created) + scheduled.Add(state); + } + } + + foreach (string name in _cache.Keys) + if (!seen.Contains(name)) _cache.TryRemove(name, out _); + return scheduled; + } + + private async IAsyncEnumerable EnumerateSchedulesAsync([EnumeratorCancellation] CancellationToken cancellationToken) + { + string? afterName = null; + while (true) + { + var page = await _scheduleStore.GetSchedulesAsync(new ScheduleQuery { AfterName = afterName }, cancellationToken).ConfigureAwait(false); + foreach (var definition in page) + yield return definition; + if (page.Count < 100) + yield break; + afterName = page[^1].Name; + } + } + + private string GetScopeKey(ScheduledJobDefinition definition) + { + return definition.Scope == ScheduledJobScope.PerNode ? NodeIdentity.RequireStable(_nodeId) : "global"; + } + + private static string CreateOccurrenceId(string name, DateTimeOffset scheduledForUtc, string scopeKey) + { + return $"{name}:{scheduledForUtc.UtcDateTime:yyyyMMddHHmmss}:{scopeKey}"; + } + + internal static void ValidateCron(string expression) + { + ParseCron(expression); + } + + /// + /// Parses a 5- or 6-field cron expression using the vendored Cronos parser. Six fields are interpreted as + /// seconds-first (); five fields use the standard format. Cronos + /// supports the full grammar (ranges, steps, lists, L/W/#, named months/days, and macros + /// such as @daily). + /// + private static CronExpression ParseCron(string expression) + { + ArgumentException.ThrowIfNullOrWhiteSpace(expression); + + if (expression.StartsWith('@')) + return CronExpression.Parse(expression, CronFormat.IncludeSeconds); + + int fieldCount = expression.Split(' ', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries).Length; + var format = fieldCount == 6 ? CronFormat.IncludeSeconds : CronFormat.Standard; + return CronExpression.Parse(expression, format); + } +} diff --git a/src/Foundatio/Jobs/JobWithLockBase.cs b/src/Foundatio/Jobs/JobWithLockBase.cs deleted file mode 100644 index 8af43866d..000000000 --- a/src/Foundatio/Jobs/JobWithLockBase.cs +++ /dev/null @@ -1,77 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Lock; -using Foundatio.Resilience; -using Foundatio.Utility; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Logging.Abstractions; - -namespace Foundatio.Jobs; - -public abstract class JobWithLockBase : IJobWithOptions, IHaveLogger, IHaveLoggerFactory, IHaveTimeProvider, IHaveResiliencePolicyProvider -{ - protected readonly TimeProvider _timeProvider; - protected readonly IResiliencePolicyProvider _resiliencePolicyProvider; - protected readonly ILoggerFactory _loggerFactory; - protected readonly ILogger _logger; - private readonly string _jobName; - - public JobWithLockBase(ILoggerFactory? loggerFactory = null) : this(null, null, loggerFactory) - { - } - - public JobWithLockBase(TimeProvider? timeProvider, IResiliencePolicyProvider? resiliencePolicyProvider, ILoggerFactory? loggerFactory = null) - { - _jobName = GetType().Name; - _timeProvider = timeProvider ?? TimeProvider.System; - _resiliencePolicyProvider = resiliencePolicyProvider ?? DefaultResiliencePolicyProvider.Instance; - _loggerFactory = loggerFactory ?? NullLoggerFactory.Instance; - _logger = _loggerFactory.CreateLogger(GetType()); - } - - public string JobId { get; } = Guid.NewGuid().ToString("N").Substring(0, 10); - ILogger IHaveLogger.Logger => _logger; - ILoggerFactory IHaveLoggerFactory.LoggerFactory => _loggerFactory; - TimeProvider IHaveTimeProvider.TimeProvider => _timeProvider; - IResiliencePolicyProvider IHaveResiliencePolicyProvider.ResiliencePolicyProvider => _resiliencePolicyProvider; - - public JobOptions? Options { get; set; } - - public virtual async Task RunAsync(CancellationToken cancellationToken = default) - { - ILock? lockValue; - using (var lockActivity = FoundatioDiagnostics.ActivitySource.StartActivity($"Job Lock: {Options?.Name ?? _jobName}")) - { - lockActivity?.AddTag("job.id", JobId); - - try - { - lockValue = await GetLockAsync(cancellationToken).AnyContext(); - } - catch (Exception ex) - { - lockActivity?.SetErrorStatus(ex); - throw; - } - - if (lockValue is null) - { - return JobResult.CancelledWithMessage("Unable to acquire job lock"); - } - } - - try - { - return await RunInternalAsync(new JobContext(cancellationToken, lockValue)).AnyContext(); - } - finally - { - await lockValue.ReleaseAsync().AnyContext(); - } - } - - protected abstract Task RunInternalAsync(JobContext context); - - protected abstract Task GetLockAsync(CancellationToken cancellationToken = default); -} diff --git a/src/Foundatio/Jobs/JobWorker.cs b/src/Foundatio/Jobs/JobWorker.cs new file mode 100644 index 000000000..10c9ccb4c --- /dev/null +++ b/src/Foundatio/Jobs/JobWorker.cs @@ -0,0 +1,282 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Serializer; +using Foundatio.Utility; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Foundatio.Jobs; + +/// Executes registered job types using atomic claims and one state machine for scheduled and ad hoc work. +public sealed class JobWorker : IJobWorker, IDisposable +{ + private readonly IJobRuntimeStore _store; + private readonly IServiceProvider _services; + private readonly TimeProvider _time; + private readonly IJobTypeRegistry _types; + private readonly ISerializer _serializer; + private readonly JobClaimRequest _request; + private readonly TimeSpan _cancellationPollInterval; + private readonly SemaphoreSlim _slots; + private readonly int _concurrency; + private readonly ILogger _logger; + private int _failingSlots; + public bool IsHealthy => Volatile.Read(ref _failingSlots) == 0; + + public JobWorker(IJobRuntimeStore store, IServiceProvider serviceProvider, JobWorkerOptions? options = null) + { + ArgumentNullException.ThrowIfNull(store); + ArgumentNullException.ThrowIfNull(serviceProvider); + options ??= new JobWorkerOptions(); + ArgumentOutOfRangeException.ThrowIfLessThan(options.MaxConcurrency, 1); + _logger = (options.LoggerFactory ?? serviceProvider.GetService() ?? NullLoggerFactory.Instance).CreateLogger(); + _store = store; + _services = serviceProvider; + _time = options.TimeProvider ?? TimeProvider.System; + _types = options.JobTypes ?? serviceProvider.GetService() ?? new JobTypeRegistry(); + _serializer = options.Serializer ?? DefaultSerializer.Instance; + _request = new JobClaimRequest + { + NodeId = options.NodeId ?? NodeIdentity.Current, + JobTypes = _types.Names.ToArray(), + Lease = options.Lease ?? TimeSpan.FromMinutes(5) + }; + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(_request.Lease, TimeSpan.Zero); + _cancellationPollInterval = options.CancellationPollInterval ?? TimeSpan.FromSeconds(1); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(_cancellationPollInterval, TimeSpan.Zero); + _concurrency = options.MaxConcurrency; + _slots = new SemaphoreSlim(_concurrency, _concurrency); + } + + public async Task RunQueuedAsync(int limit = 100, CancellationToken cancellationToken = default) + { + ArgumentOutOfRangeException.ThrowIfLessThan(limit, 1); + if (_request.JobTypes.Count == 0) + return 0; + + int reserved = 0; + int executed = 0; + async Task RunSlotAsync() + { + while (!cancellationToken.IsCancellationRequested && Interlocked.Increment(ref reserved) <= limit) + { + await _slots.WaitAsync(cancellationToken).AnyContext(); + try + { + var claim = await _store.ClaimNextAsync(_request, cancellationToken).AnyContext(); + if (claim is null) + return; + await RunClaimedAsync(claim, cancellationToken).AnyContext(); + Interlocked.Increment(ref executed); + } + finally + { + _slots.Release(); + } + } + } + + await Task.WhenAll(Enumerable.Range(0, Math.Min(_concurrency, limit)).Select(_ => RunSlotAsync())).AnyContext(); + return executed; + } + + public Task RunContinuouslyAsync(CancellationToken cancellationToken = default) + { + async Task RunSlotAsync() + { + bool failed = false; + try + { + while (!cancellationToken.IsCancellationRequested) + { + try + { + int executed = await RunQueuedAsync(1, cancellationToken).AnyContext(); + if (failed) { failed = false; Interlocked.Decrement(ref _failingSlots); } + if (executed == 0) + await Task.Delay(TimeSpan.FromMilliseconds(100), _time, cancellationToken).AnyContext(); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + break; + } + catch (Exception ex) + { + if (!failed) { failed = true; Interlocked.Increment(ref _failingSlots); } + _logger.LogError(ex, "Job worker failed to claim or settle work; retrying"); + await _time.SafeDelay(TimeSpan.FromSeconds(1), cancellationToken).AnyContext(); + } + } + } + finally + { + if (failed) Interlocked.Decrement(ref _failingSlots); + } + } + + return Task.WhenAll(Enumerable.Range(0, _concurrency).Select(_ => RunSlotAsync())); + } + + public async Task RunAsync(string jobId, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(jobId); + JobClaimValidation.Validate(_request); + await _slots.WaitAsync(cancellationToken).AnyContext(); + try + { + var claim = await _store.ClaimJobAsync(jobId, _request, cancellationToken).AnyContext(); + if (claim is null) + return false; + await RunClaimedAsync(claim, cancellationToken).AnyContext(); + return true; + } + finally + { + _slots.Release(); + } + } + + private async Task RunClaimedAsync(JobState claim, CancellationToken stoppingToken) + { + var tag = new KeyValuePair("job", claim.Name); + JobInstruments.Started.Add(1, tag); + long started = _time.GetTimestamp(); + using var execution = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken); + using var supervision = new CancellationTokenSource(); + int leaseLost = 0; + var leaseLoop = RenewLeaseAsync(claim, execution, () => Interlocked.Exchange(ref leaseLost, 1), supervision.Token); + var cancellationLoop = PollCancellationAsync(claim.JobId, execution, supervision.Token); + try + { + JobResult result; + try + { + var context = new JobExecutionContext(claim.JobId, claim.Attempt, execution.Token, _store, claim.ClaimToken!, + _request.Lease, claim.Payload, claim.PayloadType, _serializer); + execution.Token.ThrowIfCancellationRequested(); + var type = _types.Resolve(claim.JobType!); + await using var scope = _services.CreateAsyncScope(); + var registered = scope.ServiceProvider.GetService(type); + var job = (IJob)(registered ?? ActivatorUtilities.CreateInstance(scope.ServiceProvider, type)); + try + { + result = await job.TryRunAsync(context).AnyContext(); + } + finally + { + if (registered is null) + { + if (job is IAsyncDisposable asyncDisposable) + await asyncDisposable.DisposeAsync().AnyContext(); + else if (job is IDisposable disposable) + disposable.Dispose(); + } + } + } + catch (OperationCanceledException) when (execution.IsCancellationRequested) + { + result = JobResult.Cancelled; + } + catch (Exception ex) + { + result = JobResult.FromException(ex); + } + + if (Volatile.Read(ref leaseLost) != 0) + return; + + var kind = stoppingToken.IsCancellationRequested ? JobCompletionKind.Interrupted + : result.IsCancelled ? JobCompletionKind.Cancelled + : result.IsSuccess ? JobCompletionKind.Succeeded : JobCompletionKind.Failed; + if (kind == JobCompletionKind.Failed) + _logger.LogError(result.Error, "Job {JobId} ({JobType}) failed on attempt {Attempt} of {MaxAttempts}: {Message}", + claim.JobId, claim.JobType, claim.Attempt, claim.MaxAttempts, result.Message); + using var settlement = new CancellationTokenSource(TimeSpan.FromSeconds(5), _time); + if (await _store.CompleteJobAsync(claim.JobId, claim.ClaimToken!, new JobCompletion { Kind = kind, Retryable = result.Retryable, Message = result.Message, Error = kind == JobCompletionKind.Failed ? BoundError(result) : null }, settlement.Token) + .WaitAsync(_request.Lease, _time, settlement.Token).AnyContext()) + { + if (kind == JobCompletionKind.Succeeded) JobInstruments.Completed.Add(1, tag); + else if (kind == JobCompletionKind.Failed) JobInstruments.Failed.Add(1, tag); + else if (kind == JobCompletionKind.Cancelled) JobInstruments.Cancelled.Add(1, tag); + } + } + finally + { + await supervision.CancelAsync().AnyContext(); + await Task.WhenAll(leaseLoop, cancellationLoop).AnyContext(); + JobInstruments.RunTime.Record(_time.GetElapsedTime(started).TotalMilliseconds, tag); + } + } + + private async Task RenewLeaseAsync(JobState claim, CancellationTokenSource execution, Action lost, CancellationToken supervision) + { + var expires = claim.LeaseExpiresUtc!.Value; + try + { + while (!supervision.IsCancellationRequested) + { + var remaining = expires - _time.GetUtcNow(); + if (remaining <= TimeSpan.Zero) + throw new TimeoutException("Execution lease expired."); + await Task.Delay(remaining / 3, _time, supervision).AnyContext(); + var renewalStarted = _time.GetUtcNow(); + remaining = expires - renewalStarted; + if (remaining <= TimeSpan.Zero) + throw new TimeoutException("Execution lease expired."); + using var deadline = new CancellationTokenSource(remaining, _time); + using var renewal = CancellationTokenSource.CreateLinkedTokenSource(supervision, deadline.Token); + bool renewed = await _store.RenewJobLeaseAsync(claim.JobId, claim.ClaimToken!, _request.Lease, renewal.Token) + .WaitAsync(remaining, _time, supervision).AnyContext(); + if (!renewed) + throw new JobException("Execution lease was lost."); + expires = renewalStarted.Add(_request.Lease); + } + } + catch (OperationCanceledException) when (supervision.IsCancellationRequested) + { + } + catch (Exception ex) + { + _logger.LogWarning(ex, "Execution lease lost for job {JobId}", claim.JobId); + lost(); + await execution.CancelAsync().AnyContext(); + } + } + + private async Task PollCancellationAsync(string jobId, CancellationTokenSource execution, CancellationToken supervision) + { + while (!supervision.IsCancellationRequested) + { + try + { + if (await _store.IsCancellationRequestedAsync(jobId, supervision) + .WaitAsync(_cancellationPollInterval, _time, supervision).AnyContext()) + { + await execution.CancelAsync().AnyContext(); + return; + } + } + catch (OperationCanceledException) when (supervision.IsCancellationRequested) + { + return; + } + catch (Exception) + { + } + + await _time.SafeDelay(_cancellationPollInterval, supervision).AnyContext(); + } + } + + private static string? BoundError(JobResult result) + { + string? error = result.Error?.ToString() ?? result.Message; + return error?.Length > 8192 ? error[..8192] : error; + } + + public void Dispose() => _slots.Dispose(); +} diff --git a/src/Foundatio/Jobs/QueueEntryContext.cs b/src/Foundatio/Jobs/QueueEntryContext.cs deleted file mode 100644 index 04ba268a8..000000000 --- a/src/Foundatio/Jobs/QueueEntryContext.cs +++ /dev/null @@ -1,25 +0,0 @@ -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Lock; -using Foundatio.Queues; -using Foundatio.Utility; - -namespace Foundatio.Jobs; - -public class QueueEntryContext : JobContext where T : class -{ - public QueueEntryContext(IQueueEntry queueEntry, ILock queueEntryLock, CancellationToken cancellationToken = default) : base(cancellationToken, queueEntryLock) - { - QueueEntry = queueEntry; - } - - public IQueueEntry QueueEntry { get; private set; } - - public override async Task RenewLockAsync() - { - if (QueueEntry != null) - await QueueEntry.RenewLockAsync().AnyContext(); - - await base.RenewLockAsync().AnyContext(); - } -} diff --git a/src/Foundatio/Jobs/QueueJobBase.cs b/src/Foundatio/Jobs/QueueJobBase.cs deleted file mode 100644 index fb1f520d4..000000000 --- a/src/Foundatio/Jobs/QueueJobBase.cs +++ /dev/null @@ -1,251 +0,0 @@ -using System; -using System.Diagnostics; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Lock; -using Foundatio.Queues; -using Foundatio.Resilience; -using Foundatio.Utility; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Logging.Abstractions; - -namespace Foundatio.Jobs; - -public abstract class QueueJobBase : IQueueJob, IHaveLogger, IHaveLoggerFactory, IHaveTimeProvider, IHaveResiliencePolicyProvider where T : class -{ - protected readonly ILogger _logger; - protected readonly ILoggerFactory _loggerFactory; - protected readonly Lazy> _queue; - protected readonly TimeProvider _timeProvider; - protected readonly IResiliencePolicyProvider _resiliencePolicyProvider; - protected readonly string _queueName = typeof(T).Name; - - public QueueJobBase( - IQueue queue, - TimeProvider? timeProvider = null, - IResiliencePolicyProvider? resiliencePolicyProvider = null, - ILoggerFactory? loggerFactory = null - ) : this( - new Lazy>(() => queue), timeProvider, resiliencePolicyProvider, loggerFactory) - { - } - - public QueueJobBase(Lazy> queue, TimeProvider? timeProvider = null, IResiliencePolicyProvider? resiliencePolicyProvider = null, ILoggerFactory? loggerFactory = null) - { - _queue = queue; - _timeProvider = timeProvider ?? TimeProvider.System; - _resiliencePolicyProvider = resiliencePolicyProvider ?? DefaultResiliencePolicyProvider.Instance; - _loggerFactory = loggerFactory ?? NullLoggerFactory.Instance; - _logger = _loggerFactory.CreateLogger(GetType()); - AutoComplete = true; - } - - protected bool AutoComplete { get; set; } - public string JobId { get; } = Guid.NewGuid().ToString("N").Substring(0, 10); - IQueue IQueueJob.Queue => _queue.Value; - ILogger IHaveLogger.Logger => _logger; - ILoggerFactory IHaveLoggerFactory.LoggerFactory => _loggerFactory; - TimeProvider IHaveTimeProvider.TimeProvider => _timeProvider; - IResiliencePolicyProvider IHaveResiliencePolicyProvider.ResiliencePolicyProvider => _resiliencePolicyProvider; - - public virtual async Task RunAsync(CancellationToken cancellationToken = default) - { - IQueueEntry? queueEntry; - - using var linkedCancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); - linkedCancellationTokenSource.CancelAfter(TimeSpan.FromSeconds(30)); - - using var dequeueActivity = StartDequeueActivity(); - try - { - queueEntry = await _queue.Value.DequeueAsync(linkedCancellationTokenSource.Token).AnyContext(); - EnrichDequeueActivity(dequeueActivity, queueEntry); - } - catch (OperationCanceledException) - { - return JobResult.Cancelled; - } - catch (Exception ex) - { - dequeueActivity?.SetErrorStatus(ex, $"Error trying to dequeue message: {ex.Message}"); - return JobResult.FromException(ex, $"Error trying to dequeue message: {ex.Message}"); - } - - if (cancellationToken.IsCancellationRequested && queueEntry is null) - return JobResult.Cancelled; - - if (queueEntry is null) - return JobResult.SuccessWithMessage("No queue entry to process."); - - return await ProcessAsync(queueEntry, cancellationToken).AnyContext(); - } - - public async Task ProcessAsync(IQueueEntry queueEntry, CancellationToken cancellationToken) - { - using var activity = StartProcessQueueEntryActivity(queueEntry); - using var _ = _logger.BeginScope(s => s - .Property("JobId", JobId) - .Property("QueueName", _queueName) - .Property("QueueEntryId", queueEntry.Id) - .PropertyIf("CorrelationId", queueEntry.CorrelationId, !String.IsNullOrEmpty(queueEntry.CorrelationId))); - - _logger.LogInformation("Processing queue entry: id={QueueEntryId} type={QueueName} attempt={QueueEntryAttempt}", queueEntry.Id, _queueName, queueEntry.Attempts); - - if (cancellationToken.IsCancellationRequested) - { - _logger.LogInformation("Job was cancelled. Abandoning {QueueName} queue entry: {QueueEntryId}", _queueName, queueEntry.Id); - - await queueEntry.AbandonAsync().AnyContext(); - return JobResult.CancelledWithMessage($"Abandoning {_queueName} queue entry: {queueEntry.Id}"); - } - - // Safety net: poison messages have null Value at runtime despite the non-nullable type. - if (queueEntry.Value is null) - { - _logger.LogWarning("Null queue entry value (poison message) in {QueueName}: {EntryId}. Abandoning.", _queueName, queueEntry.Id); - await queueEntry.AbandonAsync().AnyContext(); - return JobResult.SuccessWithMessage($"Abandoned poison message in {_queueName}: {queueEntry.Id}"); - } - - ILock? lockValue; - try - { - lockValue = await GetQueueEntryLockAsync(queueEntry, cancellationToken).AnyContext(); - } - catch (Exception ex) - { - _logger.LogError(ex, "Error acquiring lock for {QueueName} queue entry {QueueEntryId}: {Message}", _queueName, queueEntry.Id, ex.Message); - await queueEntry.AbandonAsync().AnyContext(); - return JobResult.FromException(ex, $"Error acquiring lock for {_queueName} queue entry {queueEntry.Id}: {ex.Message}"); - } - - if (lockValue is null) - { - await queueEntry.AbandonAsync().AnyContext(); - return JobResult.CancelledWithMessage($"Unable to acquire queue entry lock. Abandoning {_queueName} queue entry: {queueEntry.Id}"); - } - - try - { - LogProcessingQueueEntry(queueEntry); - var result = await ProcessQueueEntryAsync(new QueueEntryContext(queueEntry, lockValue, cancellationToken)).AnyContext(); - - if (!AutoComplete || queueEntry.IsCompleted || queueEntry.IsAbandoned) - return result; - - if (result.IsSuccess) - { - await queueEntry.CompleteAsync().AnyContext(); - LogAutoCompletedQueueEntry(queueEntry); - } - else - { - string? message = !String.IsNullOrEmpty(result.Message) ? result.Message : result.Error?.Message; - if (result.Error != null || !String.IsNullOrEmpty(message)) - _logger.LogError(result.Error, "{QueueName} queue entry {QueueEntryId} returned an unsuccessful response: {Message}", _queueName, queueEntry.Id, message); - - _logger.LogTrace("Processing was not successful. Auto Abandoning {QueueName} queue entry: {QueueEntryId}", _queueName, queueEntry.Id); - await queueEntry.AbandonAsync().AnyContext(); - _logger.LogWarning("Auto abandoned {QueueName} queue entry: {QueueEntryId}", _queueName, queueEntry.Id); - } - - return result; - } - catch (Exception ex) - { - activity?.SetErrorStatus(ex); - _logger.LogError(ex, "Error processing {QueueName} queue entry: {QueueEntryId}", _queueName, queueEntry.Id); - - if (!queueEntry.IsCompleted && !queueEntry.IsAbandoned) - await queueEntry.AbandonAsync().AnyContext(); - - throw; - } - finally - { - _logger.LogTrace("Releasing Lock for {QueueName} queue entry: {QueueEntryId}", _queueName, queueEntry.Id); - await lockValue.ReleaseAsync().AnyContext(); - _logger.LogTrace("Released Lock for {QueueName} queue entry: {QueueEntryId}", _queueName, queueEntry.Id); - } - } - - protected virtual Activity? StartDequeueActivity() - { - var activity = FoundatioDiagnostics.ActivitySource.StartActivity("DequeueQueueEntry"); - if (activity is null) - return null; - - activity.DisplayName = $"Dequeue: {_queueName}"; - activity.AddTag("QueueName", _queueName); - activity.AddTag("JobId", JobId); - - return activity; - } - - protected virtual void EnrichDequeueActivity(Activity? activity, IQueueEntry? entry) - { - if (activity is null || !activity.IsAllDataRequested) - return; - - if (entry is null) - return; - - activity.AddTag("EntryType", entry.EntryType?.FullName); - activity.AddTag("Id", entry.Id); - activity.AddTag("CorrelationId", entry.CorrelationId); - } - - protected virtual Activity? StartProcessQueueEntryActivity(IQueueEntry entry) - { - var activity = FoundatioDiagnostics.ActivitySource.StartActivity("ProcessQueueEntry", ActivityKind.Internal, entry.CorrelationId); - if (activity is null) - return null; - - if (entry.Properties is not null && entry.Properties.TryGetValue("TraceState", out string? traceState)) - activity.TraceStateString = traceState; - - activity.DisplayName = $"Queue: {entry.EntryType?.Name}"; - - EnrichProcessQueueEntryActivity(activity, entry); - - return activity; - } - - protected virtual void EnrichProcessQueueEntryActivity(Activity activity, IQueueEntry entry) - { - if (!activity.IsAllDataRequested) - return; - - activity.AddTag("EntryType", entry.EntryType?.FullName); - activity.AddTag("Id", entry.Id); - activity.AddTag("CorrelationId", entry.CorrelationId); - - if (entry.Properties is not { Count: > 0 }) - return; - - foreach (var p in entry.Properties) - { - if (p.Key != "TraceState") - activity.AddTag(p.Key, p.Value); - } - } - - protected virtual void LogProcessingQueueEntry(IQueueEntry queueEntry) - { - _logger.LogInformation("Processing {QueueName} queue entry: {QueueEntryId}", _queueName, queueEntry.Id); - } - - protected virtual void LogAutoCompletedQueueEntry(IQueueEntry queueEntry) - { - _logger.LogInformation("Auto completed {QueueName} queue entry: {QueueEntryId}", _queueName, queueEntry.Id); - } - - protected abstract Task ProcessQueueEntryAsync(QueueEntryContext context); - - protected virtual Task GetQueueEntryLockAsync(IQueueEntry queueEntry, CancellationToken cancellationToken = default) - { - _logger.LogTrace("Returning Empty Lock for {QueueName} queue entry: {QueueEntryId}", _queueName, queueEntry.Id); - - return Task.FromResult(Disposable.EmptyLock); - } -} diff --git a/src/Foundatio/Jobs/ScheduleQuery.cs b/src/Foundatio/Jobs/ScheduleQuery.cs new file mode 100644 index 000000000..63c1aadd3 --- /dev/null +++ b/src/Foundatio/Jobs/ScheduleQuery.cs @@ -0,0 +1,17 @@ +using System; + +namespace Foundatio.Jobs; + +/// A bounded page of schedules ordered by name. Use the last name as AfterName for the next page. +public sealed record ScheduleQuery +{ + public string? AfterName { get; init; } + public int Limit { get; init; } = 100; + + /// Checks the page size before querying storage. + public void Validate() + { + ArgumentOutOfRangeException.ThrowIfLessThan(Limit, 1); + ArgumentOutOfRangeException.ThrowIfGreaterThan(Limit, 1000); + } +} diff --git a/src/Foundatio/Jobs/ScheduledJobRegistration.cs b/src/Foundatio/Jobs/ScheduledJobRegistration.cs new file mode 100644 index 000000000..6bf1d6f58 --- /dev/null +++ b/src/Foundatio/Jobs/ScheduledJobRegistration.cs @@ -0,0 +1,52 @@ +using System; +using Foundatio.Serializer; + +namespace Foundatio.Jobs; + +internal sealed record ScheduledJobRegistration(Type JobType, string Cron, CronJobOptions Options, object? Arguments) +{ + public string Name => Options.Name ?? ScheduledJobDefinition.DefaultNameFor(JobType); + + public void Validate() + { + JobArgumentContract.Validate(JobType, Arguments); + new ScheduledJobDefinition + { + Name = Name, + Cron = Cron, + JobType = JobType.FullName ?? JobType.Name, + TimeZoneId = Options.TimeZone?.Id ?? "UTC", + Scope = Options.Scope, + Overlap = Options.Overlap, + MisfireWindow = Options.MisfireWindow, + MaxAttempts = Options.MaxAttempts, + RetryPolicy = Options.RetryPolicy, + UnclaimedLifetime = Options.UnclaimedLifetime, + ConfigurationVersion = Options.ConfigurationVersion + }.Validate(); + } + + public ScheduledJobDefinition Create(IJobTypeRegistry jobTypes, ISerializer serializer) + { + JobArgumentContract.Validate(JobType, Arguments); + var definition = new ScheduledJobDefinition + { + Name = Name, + Cron = Cron, + JobType = jobTypes.GetName(JobType), + TimeZoneId = Options.TimeZone?.Id ?? "UTC", + Scope = Options.Scope, + Overlap = Options.Overlap, + MisfireWindow = Options.MisfireWindow, + MaxAttempts = Options.MaxAttempts, + RetryPolicy = Options.RetryPolicy, + UnclaimedLifetime = Options.UnclaimedLifetime, + Enabled = Options.Enabled, + ConfigurationVersion = Options.ConfigurationVersion, + Payload = Arguments is null ? null : (ReadOnlyMemory?)serializer.SerializeToBytes(Arguments), + PayloadType = Arguments?.GetType().FullName + }; + definition.Validate(); + return definition; + } +} diff --git a/src/Foundatio/Jobs/WorkItemJob/WorkItemContext.cs b/src/Foundatio/Jobs/WorkItemJob/WorkItemContext.cs deleted file mode 100644 index 355d779ed..000000000 --- a/src/Foundatio/Jobs/WorkItemJob/WorkItemContext.cs +++ /dev/null @@ -1,44 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Lock; - -namespace Foundatio.Jobs; - -public class WorkItemContext -{ - private readonly Func _progressCallback; - - public WorkItemContext(object data, string jobId, ILock? workItemLock, CancellationToken cancellationToken, Func progressCallback) - { - Data = data; - JobId = jobId; - WorkItemLock = workItemLock; - CancellationToken = cancellationToken; - _progressCallback = progressCallback; - } - - public object Data { get; private set; } - public string JobId { get; private set; } - public ILock? WorkItemLock { get; private set; } - public JobResult Result { get; set; } = JobResult.Success; - public CancellationToken CancellationToken { get; private set; } - - public Task ReportProgressAsync(int progress, string? message = null) - { - return _progressCallback(progress, message); - } - - public Task RenewLockAsync() - { - if (WorkItemLock != null) - return WorkItemLock.RenewAsync(); - - return Task.CompletedTask; - } - - public T? GetData() where T : class - { - return Data as T; - } -} diff --git a/src/Foundatio/Jobs/WorkItemJob/WorkItemData.cs b/src/Foundatio/Jobs/WorkItemJob/WorkItemData.cs deleted file mode 100644 index 987ee5d32..000000000 --- a/src/Foundatio/Jobs/WorkItemJob/WorkItemData.cs +++ /dev/null @@ -1,14 +0,0 @@ -using Foundatio.Metrics; -using Foundatio.Queues; - -namespace Foundatio.Jobs; - -public class WorkItemData : IHaveSubMetricName, IHaveUniqueIdentifier -{ - public required string WorkItemId { get; set; } - public required string Type { get; set; } - public required byte[] Data { get; set; } - public bool SendProgressReports { get; set; } - public string? UniqueIdentifier { get; set; } - public string? SubMetricName { get; set; } -} diff --git a/src/Foundatio/Jobs/WorkItemJob/WorkItemHandlers.cs b/src/Foundatio/Jobs/WorkItemJob/WorkItemHandlers.cs deleted file mode 100644 index 0b9082673..000000000 --- a/src/Foundatio/Jobs/WorkItemJob/WorkItemHandlers.cs +++ /dev/null @@ -1,176 +0,0 @@ -using System; -using System.Collections.Concurrent; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Lock; -using Foundatio.Queues; -using Foundatio.Utility; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Logging.Abstractions; - -namespace Foundatio.Jobs; - -public class WorkItemHandlers -{ - private readonly ConcurrentDictionary> _handlers; - - public WorkItemHandlers() - { - _handlers = new ConcurrentDictionary>(); - } - - public void Register(IWorkItemHandler handler) - { - _handlers.TryAdd(typeof(T), new Lazy(() => handler)); - } - - public void Register(Func handler) - { - _handlers.TryAdd(typeof(T), new Lazy(handler)); - } - - public void Register(Func handler, ILogger? logger = null, Action, Type, object>? logProcessingWorkItem = null, Action, Type, object>? logAutoCompletedWorkItem = null) where T : class - { - _handlers.TryAdd(typeof(T), new Lazy(() => new DelegateWorkItemHandler(handler, logger, logProcessingWorkItem, logAutoCompletedWorkItem))); - } - - public IWorkItemHandler? GetHandler(Type jobDataType) - { - if (!_handlers.TryGetValue(jobDataType, out var handler)) - return null; - - return handler.Value; - } -} - -/// -/// Defines a handler that processes a specific type of work item dequeued by . -/// Register handlers with to map work item types to processing logic. -/// For simple cases, use ; for complex scenarios, extend . -/// -public interface IWorkItemHandler -{ - /// - /// Acquires a lock for the given work item to prevent concurrent processing. - /// - /// The deserialized work item payload. - /// Token to cancel the lock acquisition. - /// - /// An if the lock was acquired, or null if the work item - /// should be abandoned (e.g., another instance is already processing it). - /// The default implementation returns (always succeeds). - /// - Task GetWorkItemLockAsync(object workItem, CancellationToken cancellationToken = default); - - /// - /// Processes a single work item. The provides access to the - /// deserialized payload via and supports reporting - /// progress via . - /// If this method completes without calling , - /// the entry is auto-completed when is true. - /// - /// Context containing the queue entry, work item data, and progress reporting. - Task HandleItemAsync(WorkItemContext context); - - /// - /// When true, the lock on the queue entry is automatically renewed each time - /// the handler reports progress. This prevents long-running work items from timing out. - /// - bool AutoRenewLockOnProgress { get; set; } - - /// - /// The logger used by this handler for diagnostic output. - /// - ILogger Log { get; set; } - - /// - /// Called when a work item begins processing. Override to customize logging behavior. - /// - /// The raw queue entry being processed. - /// The CLR type of the deserialized work item payload. - /// The deserialized work item payload. - void LogProcessingQueueEntry(IQueueEntry queueEntry, Type workItemDataType, object workItem); - - /// - /// Called when a work item is auto-completed after the handler returns without - /// explicitly completing or abandoning the entry. Override to customize logging behavior. - /// - /// The raw queue entry that was auto-completed. - /// The CLR type of the deserialized work item payload. - /// The deserialized work item payload. - void LogAutoCompletedQueueEntry(IQueueEntry queueEntry, Type workItemDataType, object workItem); -} - -public abstract class WorkItemHandlerBase : IWorkItemHandler -{ - public WorkItemHandlerBase(ILoggerFactory? loggerFactory = null) - { - Log = loggerFactory?.CreateLogger(GetType()) ?? NullLogger.Instance; - } - public WorkItemHandlerBase(ILogger? logger) - { - Log = logger ?? NullLogger.Instance; - } - - public virtual Task GetWorkItemLockAsync(object workItem, CancellationToken cancellationToken = default) - { - return Task.FromResult(Disposable.EmptyLock); - } - - public bool AutoRenewLockOnProgress { get; set; } - public ILogger Log { get; set; } - - public virtual void LogProcessingQueueEntry(IQueueEntry queueEntry, Type workItemDataType, object workItem) - { - Log.LogInformation("Processing {TypeName} work item queue entry: {QueueEntryId}", workItemDataType.Name, queueEntry.Id); - } - - public virtual void LogAutoCompletedQueueEntry(IQueueEntry queueEntry, Type workItemDataType, object workItem) - { - Log.LogInformation("Auto completed {TypeName} work item queue entry: {QueueEntryId}", workItemDataType.Name, queueEntry.Id); - } - - public abstract Task HandleItemAsync(WorkItemContext context); - - protected int CalculateProgress(long total, long completed, int startProgress = 0, int endProgress = 100) - { - return startProgress + (int)((100 * (double)completed / total) * (((double)endProgress - startProgress) / 100)); - } -} - -public class DelegateWorkItemHandler : WorkItemHandlerBase -{ - private readonly Func _handler; - private readonly Action, Type, object>? _logProcessingWorkItem; - private readonly Action, Type, object>? _logAutoCompletedWorkItem; - - public DelegateWorkItemHandler(Func handler, ILogger? logger = null, Action, Type, object>? logProcessingWorkItem = null, Action, Type, object>? logAutoCompletedWorkItem = null) : base(logger) - { - ArgumentNullException.ThrowIfNull(handler); - - _handler = handler; - _logProcessingWorkItem = logProcessingWorkItem; - _logAutoCompletedWorkItem = logAutoCompletedWorkItem; - } - - public override Task HandleItemAsync(WorkItemContext context) - { - return _handler(context); - } - - public override void LogProcessingQueueEntry(IQueueEntry queueEntry, Type workItemDataType, object workItem) - { - if (_logProcessingWorkItem != null) - _logProcessingWorkItem(queueEntry, workItemDataType, workItem); - else - base.LogProcessingQueueEntry(queueEntry, workItemDataType, workItem); - } - - public override void LogAutoCompletedQueueEntry(IQueueEntry queueEntry, Type workItemDataType, object workItem) - { - if (_logAutoCompletedWorkItem != null) - _logAutoCompletedWorkItem(queueEntry, workItemDataType, workItem); - else - base.LogAutoCompletedQueueEntry(queueEntry, workItemDataType, workItem); - } -} diff --git a/src/Foundatio/Jobs/WorkItemJob/WorkItemJob.cs b/src/Foundatio/Jobs/WorkItemJob/WorkItemJob.cs deleted file mode 100644 index 5d37547e6..000000000 --- a/src/Foundatio/Jobs/WorkItemJob/WorkItemJob.cs +++ /dev/null @@ -1,288 +0,0 @@ -using System; -using System.Collections.Concurrent; -using System.Diagnostics; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Messaging; -using Foundatio.Queues; -using Foundatio.Serializer; -using Foundatio.Utility; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Logging.Abstractions; - -namespace Foundatio.Jobs; - -[Job(Description = "Processes adhoc work item queues entries")] -public class WorkItemJob : IQueueJob, IHaveLogger, IHaveLoggerFactory -{ - protected readonly IMessagePublisher _publisher; - protected readonly WorkItemHandlers _handlers; - protected readonly IQueue _queue; - protected readonly ILogger _logger; - protected readonly ILoggerFactory _loggerFactory; - - public WorkItemJob(IQueue queue, IMessagePublisher publisher, WorkItemHandlers handlers, ILoggerFactory? loggerFactory = null) - { - _publisher = publisher; - _handlers = handlers; - _queue = queue; - _loggerFactory = loggerFactory ?? NullLoggerFactory.Instance; - _logger = _loggerFactory.CreateLogger(GetType()); - } - - public string JobId { get; } = Guid.NewGuid().ToString("N").Substring(0, 10); - IQueue IQueueJob.Queue => _queue; - ILogger IHaveLogger.Logger => _logger; - ILoggerFactory IHaveLoggerFactory.LoggerFactory => _loggerFactory; - - public virtual async Task RunAsync(CancellationToken cancellationToken = default) - { - IQueueEntry? queueEntry; - - using var linkedCancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); - linkedCancellationTokenSource.CancelAfter(TimeSpan.FromSeconds(30)); - - try - { - queueEntry = await _queue.DequeueAsync(linkedCancellationTokenSource.Token).AnyContext(); - } - catch (OperationCanceledException) - { - return JobResult.Cancelled; - } - catch (Exception ex) - { - return JobResult.FromException(ex, $"Error trying to dequeue work item: {ex.Message}"); - } - - if (cancellationToken.IsCancellationRequested && queueEntry is null) - return JobResult.Cancelled; - - if (queueEntry is null) - return JobResult.SuccessWithMessage("No queue entry to process."); - - return await ProcessAsync(queueEntry, cancellationToken).AnyContext(); - } - - public async Task ProcessAsync(IQueueEntry queueEntry, CancellationToken cancellationToken) - { - if (cancellationToken.IsCancellationRequested) - { - await queueEntry.AbandonAsync().AnyContext(); - return JobResult.CancelledWithMessage($"Abandoning {queueEntry.Value?.Type} work item: {queueEntry.Id}"); - } - - var workItemDataType = GetWorkItemType(queueEntry.Value?.Type); - if (workItemDataType is null) - { - await queueEntry.AbandonAsync().AnyContext(); - return JobResult.FailedWithMessage($"Abandoning {queueEntry.Value?.Type} work item: {queueEntry.Id}: Could not resolve work item data type"); - } - - using var activity = StartProcessWorkItemActivity(queueEntry, workItemDataType); - using var _ = _logger.BeginScope(s => s - .Property("JobId", JobId) - .Property("QueueEntryId", queueEntry.Id) - .PropertyIf("CorrelationId", queueEntry.CorrelationId, !String.IsNullOrEmpty(queueEntry.CorrelationId)) - .Property("QueueEntryName", workItemDataType.Name)); - - object? workItemData; - try - { - workItemData = _queue.Serializer.Deserialize(queueEntry.Value!.Data, workItemDataType); - } - catch (Exception ex) - { - activity?.SetErrorStatus(ex, $"Abandoning {queueEntry.Value!.Type} work item: {queueEntry.Id}: Failed to parse {workItemDataType.Name} work item data"); - await queueEntry.AbandonAsync().AnyContext(); - return JobResult.FromException(ex, $"Abandoning {queueEntry.Value!.Type} work item: {queueEntry.Id}: Failed to parse {workItemDataType.Name} work item data"); - } - - if (workItemData is null) - { - _logger.LogWarning("Abandoning {TypeName} work item: {Id}: Deserialization returned null for {WorkItemDataType}", queueEntry.Value.Type, queueEntry.Id, workItemDataType.Name); - await queueEntry.AbandonAsync().AnyContext(); - return JobResult.FailedWithMessage($"Abandoning {queueEntry.Value.Type} work item: {queueEntry.Id}: Deserialization returned null for {workItemDataType.Name}"); - } - - var handler = _handlers.GetHandler(workItemDataType); - if (handler is null) - { - await queueEntry.CompleteAsync().AnyContext(); - var result = JobResult.FailedWithMessage($"Completing {queueEntry.Value.Type} work item: {queueEntry.Id}: Handler for type {workItemDataType.Name} not registered"); - activity?.SetErrorStatus(message: result.Message); - return result; - } - - if (queueEntry.Value.SendProgressReports) - await ReportProgressAsync(handler, queueEntry).AnyContext(); - - var lockValue = await handler.GetWorkItemLockAsync(workItemData, cancellationToken).AnyContext(); - if (lockValue is null) - { - handler.Log.LogInformation("Abandoning {TypeName} work item: {Id}: Unable to acquire work item lock", queueEntry.Value.Type, queueEntry.Id); - - await queueEntry.AbandonAsync().AnyContext(); - return JobResult.CancelledWithMessage($"Unable to acquire work item lock. Abandoning {queueEntry.Value.Type} queue entry: {queueEntry.Id}"); - } - - var progressCallback = new Func(async (progress, message) => - { - if (handler.AutoRenewLockOnProgress) - { - try - { - await Task.WhenAll( - queueEntry.RenewLockAsync(), - lockValue.RenewAsync() - ).AnyContext(); - } - catch (Exception ex) - { - handler.Log.LogError(ex, "Error renewing work item locks: {Message}", ex.Message); - } - } - - await ReportProgressAsync(handler, queueEntry, progress, message).AnyContext(); - handler.Log.LogInformation("{TypeName} Progress {Progress}%: {Message}", workItemDataType.Name, progress, message); - }); - - try - { - handler.LogProcessingQueueEntry(queueEntry, workItemDataType, workItemData); - var workItemContext = new WorkItemContext(workItemData, JobId, lockValue, cancellationToken, progressCallback); - await handler.HandleItemAsync(workItemContext).AnyContext(); - - if (!workItemContext.Result.IsSuccess) - { - if (!queueEntry.IsAbandoned && !queueEntry.IsCompleted) - { - await queueEntry.AbandonAsync().AnyContext(); - return workItemContext.Result; - } - } - - if (!queueEntry.IsAbandoned && !queueEntry.IsCompleted) - { - await queueEntry.CompleteAsync().AnyContext(); - handler.LogAutoCompletedQueueEntry(queueEntry, workItemDataType, workItemData); - } - - if (queueEntry.Value.SendProgressReports) - await ReportProgressAsync(handler, queueEntry, 100).AnyContext(); - - return JobResult.Success; - } - catch (Exception ex) - { - activity?.SetErrorStatus(ex); - - if (queueEntry.Value.SendProgressReports) - await ReportProgressAsync(handler, queueEntry, -1, $"Failed: {ex.Message}").AnyContext(); - - if (!queueEntry.IsAbandoned && !queueEntry.IsCompleted) - { - await queueEntry.AbandonAsync().AnyContext(); - return JobResult.FromException(ex, $"Abandoning {queueEntry.Value.Type} work item: {queueEntry.Id}: Error in handler {workItemDataType.Name}"); - } - - return JobResult.FromException(ex, $"Error processing {queueEntry.Value.Type} work item: {queueEntry.Id} in handler: {workItemDataType.Name}"); - } - finally - { - await lockValue.ReleaseAsync().AnyContext(); - } - } - - protected virtual Activity? StartProcessWorkItemActivity(IQueueEntry entry, Type workItemDataType) - { - var activity = FoundatioDiagnostics.ActivitySource.StartActivity("ProcessQueueEntry", ActivityKind.Internal, entry.CorrelationId); - if (activity is null) - return null; - - if (entry.Properties is not null && entry.Properties.TryGetValue("TraceState", out string? traceState)) - activity.TraceStateString = traceState; - - activity.DisplayName = $"Work Item: {entry.Value?.SubMetricName ?? workItemDataType.Name}"; - - EnrichProcessWorkItemActivity(activity, entry, workItemDataType); - - return activity; - } - - protected virtual void EnrichProcessWorkItemActivity(Activity activity, IQueueEntry entry, Type workItemDataType) - { - if (!activity.IsAllDataRequested) - return; - - activity.AddTag("WorkItemType", entry.Value?.Type); - activity.AddTag("Id", entry.Id); - activity.AddTag("CorrelationId", entry.CorrelationId); - - if (entry.Properties is null || entry.Properties.Count <= 0) - return; - - foreach (var p in entry.Properties) - { - if (p.Key != "TraceState") - activity.AddTag(p.Key, p.Value); - } - } - - private readonly ConcurrentDictionary _knownTypesCache = new(); - protected virtual Type? GetWorkItemType(string? workItemType) - { - if (String.IsNullOrWhiteSpace(workItemType)) - return null; - - if (_knownTypesCache.TryGetValue(workItemType, out var cachedType)) - return cachedType; - - Type? resolvedType = null; - - try - { - resolvedType = Type.GetType(workItemType); - } - catch (Exception) - { - try - { - // try resolve type without version - string[] typeParts = workItemType.Split(','); - string shortType = typeParts.Length >= 2 - ? String.Join(",", typeParts[0], typeParts[1]) - : workItemType; - - resolvedType = Type.GetType(shortType); - } - catch (Exception ex) - { - _logger.LogWarning(ex, "Error getting work item type: {WorkItemType}", workItemType); - } - } - - if (resolvedType is not null) - _knownTypesCache.TryAdd(workItemType, resolvedType); - - return resolvedType; - } - - protected async Task ReportProgressAsync(IWorkItemHandler handler, IQueueEntry queueEntry, int progress = 0, string? message = null) - { - try - { - await _publisher.PublishAsync(new WorkItemStatus - { - WorkItemId = queueEntry.Value?.WorkItemId, - Type = queueEntry.Value?.Type, - Progress = progress, - Message = message - }).AnyContext(); - } - catch (Exception ex) - { - handler.Log.LogError(ex, "Error sending progress report: {Message}", ex.Message); - } - } -} diff --git a/src/Foundatio/Jobs/WorkItemJob/WorkItemQueueExtensions.cs b/src/Foundatio/Jobs/WorkItemJob/WorkItemQueueExtensions.cs deleted file mode 100644 index 72f3704b8..000000000 --- a/src/Foundatio/Jobs/WorkItemJob/WorkItemQueueExtensions.cs +++ /dev/null @@ -1,62 +0,0 @@ -using System; -using System.Threading.Tasks; -using Foundatio.Metrics; -using Foundatio.Queues; -using Foundatio.Serializer; -using Foundatio.Utility; - -namespace Foundatio.Jobs; - -public static class WorkItemQueueExtensions -{ - public static async Task EnqueueAsync(this IQueue queue, T workItemData, bool includeProgressReporting = false) - { - string jobId = Guid.NewGuid().ToString("N"); - var bytes = queue.Serializer.SerializeToBytes(workItemData); - string typeName = typeof(T).AssemblyQualifiedName - ?? throw new InvalidOperationException($"Type {typeof(T).Name} does not have an assembly-qualified name"); - - var data = new WorkItemData - { - Data = bytes, - WorkItemId = jobId, - Type = typeName, - SendProgressReports = includeProgressReporting - }; - - if (workItemData is IHaveUniqueIdentifier haveUniqueIdentifier) - data.UniqueIdentifier = haveUniqueIdentifier.UniqueIdentifier; - - if (workItemData is IHaveSubMetricName haveSubMetricName && haveSubMetricName.SubMetricName != null) - data.SubMetricName = haveSubMetricName.SubMetricName; - else - data.SubMetricName = GetDefaultSubMetricName(data); - - await queue.EnqueueAsync(data).AnyContext(); - - return jobId; - } - - private static string? GetDefaultSubMetricName(WorkItemData data) - { - if (String.IsNullOrEmpty(data.Type)) - return null; - - string? type = GetTypeName(data.Type); - if (type != null && type.EndsWith("WorkItem")) - type = type.Substring(0, type.Length - 8); - - return type?.ToLowerInvariant(); - } - - private static string? GetTypeName(string assemblyQualifiedName) - { - if (String.IsNullOrEmpty(assemblyQualifiedName)) - return null; - - var parts = assemblyQualifiedName.Split(','); - int i = parts[0].LastIndexOf('.'); - - return i < 0 ? null : parts[0].Substring(i + 1); - } -} diff --git a/src/Foundatio/Jobs/WorkItemJob/WorkItemStatus.cs b/src/Foundatio/Jobs/WorkItemJob/WorkItemStatus.cs deleted file mode 100644 index 1ddad0248..000000000 --- a/src/Foundatio/Jobs/WorkItemJob/WorkItemStatus.cs +++ /dev/null @@ -1,9 +0,0 @@ -namespace Foundatio.Jobs; - -public class WorkItemStatus -{ - public string? WorkItemId { get; set; } - public int Progress { get; set; } - public string? Message { get; set; } - public string? Type { get; set; } -} diff --git a/src/Foundatio/Lock/CacheLockProvider.cs b/src/Foundatio/Lock/CacheLockProvider.cs index aae3d4d14..cf7a4e82a 100644 --- a/src/Foundatio/Lock/CacheLockProvider.cs +++ b/src/Foundatio/Lock/CacheLockProvider.cs @@ -60,7 +60,7 @@ public CacheLockProvider(ICacheClient cacheClient, IMessageBus? messageBus, Time private async Task EnsureTopicSubscriptionAsync() { - if (_isSubscribed || _messageBus is null) + if (_isSubscribed || _messageBus is null || !_messageBus.SupportsTemporarySubscriptions) return; using (await _lock.LockAsync().AnyContext()) @@ -69,7 +69,10 @@ private async Task EnsureTopicSubscriptionAsync() return; _logger.LogTrace("Subscribing to cache lock released"); - await _messageBus.SubscribeAsync(OnLockReleasedAsync).AnyContext(); + // Lock-released notifications are events every waiting node must see: published-only and per-instance. + await _messageBus.SubscribeAsync( + (context, token) => OnLockReleasedAsync(context.Message, token), + new MessageSubscriptionOptions()).AnyContext(); _isSubscribed = true; _logger.LogTrace("Subscribed to cache lock released"); } diff --git a/src/Foundatio/Lock/LockOwnershipLostException.cs b/src/Foundatio/Lock/LockOwnershipLostException.cs new file mode 100644 index 000000000..0c34e1f79 --- /dev/null +++ b/src/Foundatio/Lock/LockOwnershipLostException.cs @@ -0,0 +1,6 @@ +using System; + +namespace Foundatio.Lock; + +/// The resource lock is no longer owned by this lease holder. +public sealed class LockOwnershipLostException(string message) : InvalidOperationException(message); diff --git a/src/Foundatio/Messaging/DeadLetterQuery.cs b/src/Foundatio/Messaging/DeadLetterQuery.cs new file mode 100644 index 000000000..986711fad --- /dev/null +++ b/src/Foundatio/Messaging/DeadLetterQuery.cs @@ -0,0 +1,16 @@ +using System; + +namespace Foundatio.Messaging; + +/// A bounded page of dead letters. Pass the last entry ID as AfterId to continue. +public sealed record DeadLetterQuery +{ + public string? AfterId { get; init; } + public int Limit { get; init; } = 100; + + public void Validate() + { + ArgumentOutOfRangeException.ThrowIfLessThan(Limit, 1); + ArgumentOutOfRangeException.ThrowIfGreaterThan(Limit, 1000); + } +} diff --git a/src/Foundatio/Messaging/IMessageBus.cs b/src/Foundatio/Messaging/IMessageBus.cs index 1567da0d5..90cc8e056 100644 --- a/src/Foundatio/Messaging/IMessageBus.cs +++ b/src/Foundatio/Messaging/IMessageBus.cs @@ -2,7 +2,7 @@ using System.Collections.Generic; using System.Diagnostics.CodeAnalysis; -namespace Foundatio.Messaging; +namespace Foundatio.Messaging.Legacy; /// /// Represents a message bus that supports both publishing and subscribing to messages. diff --git a/src/Foundatio/Messaging/IMessageContext.cs b/src/Foundatio/Messaging/IMessageContext.cs new file mode 100644 index 000000000..8b172363f --- /dev/null +++ b/src/Foundatio/Messaging/IMessageContext.cs @@ -0,0 +1,145 @@ +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Messaging; + +public enum AckMode +{ + Auto, + Manual +} + +/// +/// Core-owned retry and dead-letter policy. Foundatio always owns redelivery and dead-lettering so the behavior is +/// identical across transports; transports stay simple and only provide the underlying primitives (redelivery and an +/// optional dead-letter sink). Configure a default on ; a subscription can override +/// /backoff per subscription. +/// +public sealed record RetryPolicy +{ + /// Maximum delivery attempts for a failing handler before the message is dead-lettered. Default 5. + public int MaxAttempts { get; init; } = 5; + + /// + /// Delay before each redelivery given the 1-based attempt number. Defaults to + /// (immediate first retry, then 10s/20s/30s with jitter). Set null to defer to the transport's own redelivery timing. + /// + public Func? Backoff { get; init; } = DefaultBackoff; + + /// + /// Marks a handler failure as unrecoverable: when the predicate returns true the message is dead-lettered + /// immediately instead of retried (a poison message should not burn its attempt budget). Deserialization failures + /// are always unrecoverable regardless of this predicate. A subscription's + /// overrides this default. + /// + public Func? DeadLetterWhen { get; init; } + + /// + /// Destination terminal (dead-lettered) messages are sent to when the transport has no native dead-letter sink. + /// Null (default) derives "{source}.deadletter" per source. Ignored when the transport supports native dead-lettering. + /// + public string? DeadLetterDestination { get; init; } + + /// + /// The default redelivery curve: an immediate first retry, then 10s/20s/30s (capped) with ±20% jitter — the delay + /// shape mature messaging stacks converged on. The attempt number is 1-based: the value is the delay applied after + /// that attempt failed. + /// + public static readonly Func DefaultBackoff = attempt => + { + if (attempt <= 1) + return TimeSpan.Zero; + + double seconds = Math.Min((attempt - 1) * 10, 30); + double jitter = 1 + (Random.Shared.NextDouble() * 0.4 - 0.2); + return TimeSpan.FromSeconds(seconds * jitter); + }; + + /// Maximum attempts for a message whose type has no registered consumer before it is dead-lettered as "no-handler". Default 50. + public int UnmatchedMaxAttempts { get; init; } = 50; + + /// Delay before redelivering an unmatched-type message. Default five seconds with jitter. Null defers to the transport's own redelivery timing. + public Func? UnmatchedBackoff { get; init; } = _ => TimeSpan.FromSeconds(4 + Random.Shared.NextDouble() * 2); +} + +/// +/// Thrown by the consumer loop when a message arrives on a shared destination whose type has no registered consumer +/// on this node (for example a newer message type mid rolling-deploy, or a misconfiguration). It is surfaced loudly +/// per message and isolated to that message — the receive loop and the other type handlers keep running. +/// +public sealed class UnhandledMessageTypeException : Exception +{ + public UnhandledMessageTypeException(string? messageType, string source) + : base($"No consumer is registered for message type \"{messageType ?? "(unknown)"}\" received on source \"{source}\".") + { + MessageType = messageType; + SourceName = source; + } + + public string? MessageType { get; } + public string SourceName { get; } +} + +public sealed record RejectOptions +{ + /// + /// When false (default) the message is returned for redelivery (a retry). When true the message is terminal: it + /// is moved to the transport's native dead-letter sink where one exists, otherwise sent to the configured or + /// derived ("{source}.deadletter") dead-letter destination. Terminal messages are never redelivered. + /// + public bool Terminal { get; init; } + + /// Reason carried to the dead-letter sink for a terminal reject. + public string? Reason { get; init; } + + /// + /// The failure behind a terminal reject; its type/message/stack are stamped as forensics headers on the + /// dead-lettered message so a dead message is triageable with plain transport tooling. + /// + public Exception? Exception { get; init; } + + /// + /// An explicit delay before the message is redelivered. Honored only for a non-terminal reject, served natively + /// when the transport supports redelivery delay within its advertised maximum, otherwise through the runtime store. + /// When null the transport's own redelivery timing applies. + /// + public TimeSpan? RedeliveryDelay { get; init; } + + /// + /// When true, a the transport cannot honor (no native support and no runtime store) + /// degrades to immediate redelivery instead of failing. The core's retry policy rejects with best-effort delays; + /// an explicit caller-specified delay defaults to strict. + /// + public bool BestEffortDelay { get; init; } +} + +public interface IMessageContext +{ + /// The receiving destination, when supplied by the bus. + DestinationAddress? Destination => null; + /// Original broker enqueue time, when available. + DateTimeOffset? EnqueuedUtc => null; + /// Whether this delivery lost ownership. A stale worker must not settle it. + bool IsLeaseLost => false; + string Id { get; } + + /// Broker-assigned ID for diagnostics; may change when a message is re-sent. + string BrokerMessageId { get; } + ReadOnlyMemory Body { get; } + MessageHeaders Headers { get; } + string? CorrelationId { get; } + string? MessageType { get; } + MessagePriority Priority { get; } + int Attempts { get; } + bool IsHandled { get; } + CancellationToken CancellationToken { get; } + Task CompleteAsync(CancellationToken cancellationToken = default); + Task RejectAsync(RejectOptions? options = null, CancellationToken cancellationToken = default); + Task RenewLockAsync(TimeSpan? duration = null, CancellationToken cancellationToken = default); +} + +public interface IMessageContext : IMessageContext where T : class +{ + T Message { get; } +} diff --git a/src/Foundatio/Messaging/IMessageHandler.cs b/src/Foundatio/Messaging/IMessageHandler.cs new file mode 100644 index 000000000..120e5a0d6 --- /dev/null +++ b/src/Foundatio/Messaging/IMessageHandler.cs @@ -0,0 +1,13 @@ +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Messaging; + +/// +/// Handles one message in a DI scope. Register with AddConsumer for queued work or AddSubscriber for published events. +/// Throw to apply the endpoint retry policy; return to acknowledge successfully processed messages. +/// +public interface IMessageHandler where T : class +{ + Task HandleAsync(IMessageContext context, CancellationToken cancellationToken); +} diff --git a/src/Foundatio/Messaging/IMessageProcessingObserver.cs b/src/Foundatio/Messaging/IMessageProcessingObserver.cs new file mode 100644 index 000000000..8f428265f --- /dev/null +++ b/src/Foundatio/Messaging/IMessageProcessingObserver.cs @@ -0,0 +1,11 @@ +namespace Foundatio.Messaging; + +/// Optional transport decoration for deterministic drains through processing, settlement, and observer updates. +public interface IMessageProcessingObserver +{ + /// Called when hosted processing begins. Implementations must not throw. + void ProcessingStarted(TransportEntry entry) { } + + /// Called once when core releases a delivered entry. Implementations must not throw. + void ProcessingFinished(TransportEntry entry); +} diff --git a/src/Foundatio/Messaging/IMessagePublisher.cs b/src/Foundatio/Messaging/IMessagePublisher.cs index 1961d85f0..531a4efcd 100644 --- a/src/Foundatio/Messaging/IMessagePublisher.cs +++ b/src/Foundatio/Messaging/IMessagePublisher.cs @@ -2,7 +2,7 @@ using System.Threading; using System.Threading.Tasks; -namespace Foundatio.Messaging; +namespace Foundatio.Messaging.Legacy; /// /// Publishes messages to all subscribers listening for the message type. diff --git a/src/Foundatio/Messaging/IMessageSubscriber.cs b/src/Foundatio/Messaging/IMessageSubscriber.cs index a5c107d01..f6a57fe1c 100644 --- a/src/Foundatio/Messaging/IMessageSubscriber.cs +++ b/src/Foundatio/Messaging/IMessageSubscriber.cs @@ -2,7 +2,7 @@ using System.Threading; using System.Threading.Tasks; -namespace Foundatio.Messaging; +namespace Foundatio.Messaging.Legacy; /// /// Subscribes to messages published on the message bus. @@ -38,23 +38,4 @@ public static Task SubscribeAsync(this IMessageSubscriber subscriber, Action< return Task.CompletedTask; }, cancellationToken); } - - public static Task SubscribeAsync(this IMessageSubscriber subscriber, Func handler, CancellationToken cancellationToken = default) - { - return subscriber.SubscribeAsync((msg, token) => handler(msg, token), cancellationToken); - } - - public static Task SubscribeAsync(this IMessageSubscriber subscriber, Func handler, CancellationToken cancellationToken = default) - { - return subscriber.SubscribeAsync((msg, token) => handler(msg), cancellationToken); - } - - public static Task SubscribeAsync(this IMessageSubscriber subscriber, Action handler, CancellationToken cancellationToken = default) - { - return subscriber.SubscribeAsync((msg, token) => - { - handler(msg); - return Task.CompletedTask; - }, cancellationToken); - } } diff --git a/src/Foundatio/Messaging/InMemoryMessageBus.cs b/src/Foundatio/Messaging/InMemoryMessageBus.cs deleted file mode 100644 index 7cd2132e4..000000000 --- a/src/Foundatio/Messaging/InMemoryMessageBus.cs +++ /dev/null @@ -1,101 +0,0 @@ -using System; -using System.Collections.Concurrent; -using System.Collections.Generic; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Utility; -using Microsoft.Extensions.Logging; - -namespace Foundatio.Messaging; - -public class InMemoryMessageBus : MessageBusBase -{ - private readonly ConcurrentDictionary _messageCounts = new(); - private long _messagesSent; - - public InMemoryMessageBus() : this(o => o) { } - - public InMemoryMessageBus(InMemoryMessageBusOptions options) : base(options) { } - - public InMemoryMessageBus(Builder config) - : this(config(new InMemoryMessageBusOptionsBuilder()).Build()) { } - - public long MessagesSent => _messagesSent; - - public long GetMessagesSent(Type messageType) - { - return _messageCounts.GetValueOrDefault(GetMappedMessageType(messageType), 0); - } - - public long GetMessagesSent() - { - return _messageCounts.GetValueOrDefault(GetMappedMessageType(typeof(T)), 0); - } - - public void ResetMessagesSent() - { - Interlocked.Exchange(ref _messagesSent, 0); - _messageCounts.Clear(); - } - - protected override async Task PublishImplAsync(string messageType, object message, MessageOptions options, CancellationToken cancellationToken) - { - Interlocked.Increment(ref _messagesSent); - _messageCounts.AddOrUpdate(messageType, _ => 1, (_, c) => c + 1); - var mappedType = GetMappedMessageType(messageType); - - if (_subscribers.IsEmpty) - return; - - if (options.DeliveryDelay.HasValue && options.DeliveryDelay.Value > TimeSpan.Zero) - { - if (mappedType is null) - throw new MessageBusException($"Unable to resolve CLR type for delayed message: {messageType}"); - - _logger.LogTrace("Schedule delayed message: {MessageType} ({Delay}ms)", messageType, options.DeliveryDelay.Value.TotalMilliseconds); - SendDelayedMessage(mappedType, message, options); - return; - } - - byte[] body = SerializeMessageBody(messageType, message); - var messageData = new Message(body, DeserializeMessageBody) - { - CorrelationId = options.CorrelationId, - UniqueId = options.UniqueId, - Type = messageType, - ClrType = mappedType - }; - - foreach (var property in options.Properties) - messageData.Properties[property.Key] = property.Value; - - try - { - await SendMessageToSubscribersAsync(messageData).AnyContext(); - } - catch (MessageBusException) - { - // Swallow handler errors to match distributed message bus behavior. - // In distributed buses (Redis, RabbitMQ, etc.), subscriber errors occur - // in a separate process and never propagate to the publisher. - // Note: SendMessageToSubscribersAsync already logged the error, so we don't log again. - } - catch (Exception ex) - { - // Catch any other unexpected exceptions for defensive purposes - _logger.LogError(ex, "Error sending message to subscribers: {Message}", ex.Message); - } - } - - public override void Dispose() - { - _messageCounts.Clear(); - base.Dispose(); - } - - public override ValueTask DisposeAsync() - { - _messageCounts.Clear(); - return base.DisposeAsync(); - } -} diff --git a/src/Foundatio/Messaging/InMemoryMessageBusOptions.cs b/src/Foundatio/Messaging/InMemoryMessageBusOptions.cs deleted file mode 100644 index 1387eee31..000000000 --- a/src/Foundatio/Messaging/InMemoryMessageBusOptions.cs +++ /dev/null @@ -1,5 +0,0 @@ -namespace Foundatio.Messaging; - -public class InMemoryMessageBusOptions : SharedMessageBusOptions { } - -public class InMemoryMessageBusOptionsBuilder : SharedMessageBusOptionsBuilder { } diff --git a/src/Foundatio/Messaging/InMemoryMessageTransport.Subscriptions.cs b/src/Foundatio/Messaging/InMemoryMessageTransport.Subscriptions.cs new file mode 100644 index 000000000..b610d1e2a --- /dev/null +++ b/src/Foundatio/Messaging/InMemoryMessageTransport.Subscriptions.cs @@ -0,0 +1,75 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Messaging; + +public sealed partial class InMemoryMessageTransport +{ + private sealed class TemporarySubscription + { + public required DateTimeOffset ExpiresUtc { get; set; } + public required ITimer Timer { get; init; } + } + + private readonly Dictionary _temporarySubscriptions = new(StringComparer.OrdinalIgnoreCase); + + private void CreateTemporarySubscription(DestinationAddress source, TimeSpan lease) + { + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(lease, TimeSpan.Zero); + if (source.Role != DestinationRole.Subscription) + throw new ArgumentException("Only subscriptions can have an expiration lease.", nameof(source)); + string key = StorageKey(source); + if (_temporarySubscriptions.TryGetValue(key, out var existing)) + { + existing.ExpiresUtc = _timeProvider.GetUtcNow().Add(lease); + existing.Timer.Change(lease, Timeout.InfiniteTimeSpan); + return; + } + var timer = _timeProvider.CreateTimer(_ => ExpireTemporarySubscription(key), null, lease, Timeout.InfiniteTimeSpan); + _temporarySubscriptions[key] = new TemporarySubscription { ExpiresUtc = _timeProvider.GetUtcNow().Add(lease), Timer = timer }; + } + + private void ExpireTemporarySubscription(string key) + { + lock (_temporarySubscriptions) + { + if (!_temporarySubscriptions.TryGetValue(key, out var subscription) || subscription.ExpiresUtc > _timeProvider.GetUtcNow()) + return; + _temporarySubscriptions.Remove(key); + subscription.Timer.Dispose(); + DeleteDestination(key); + } + } + + public Task RenewSubscriptionAsync(DestinationAddress source, TimeSpan lease, CancellationToken cancellationToken = default) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(source); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(lease, TimeSpan.Zero); + cancellationToken.ThrowIfCancellationRequested(); + string key = StorageKey(source); + lock (_temporarySubscriptions) + { + if (!_temporarySubscriptions.TryGetValue(key, out var subscription) || subscription.ExpiresUtc <= _timeProvider.GetUtcNow()) + { + ExpireTemporarySubscription(key); + return Task.FromResult(false); + } + subscription.ExpiresUtc = _timeProvider.GetUtcNow().Add(lease); + subscription.Timer.Change(lease, Timeout.InfiniteTimeSpan); + return Task.FromResult(true); + } + } + + private void DisposeTemporarySubscriptions() + { + lock (_temporarySubscriptions) + { + foreach (var subscription in _temporarySubscriptions.Values) + subscription.Timer.Dispose(); + _temporarySubscriptions.Clear(); + } + } +} diff --git a/src/Foundatio/Messaging/InMemoryMessageTransport.cs b/src/Foundatio/Messaging/InMemoryMessageTransport.cs new file mode 100644 index 000000000..31f710876 --- /dev/null +++ b/src/Foundatio/Messaging/InMemoryMessageTransport.cs @@ -0,0 +1,831 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Threading; +using System.Threading.Channels; +using System.Threading.Tasks; +using Foundatio.AsyncEx; +using Foundatio.Utility; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Foundatio.Messaging; + +public sealed partial class InMemoryMessageTransport : IMessageTransport, ISupportsPull, ISupportsVisibilityTimeout, ISupportsDeadLetter, ISupportsRedeliveryDelay, ISupportsLockRenewal, ISupportsStats, ISupportsEphemeralSubscriptions, ITransportInfo +{ + private static readonly TimeSpan _defaultLockRenewal = TimeSpan.FromMinutes(1); + private static readonly TimeSpan _reclaimInterval = TimeSpan.FromMilliseconds(50); + + // Priority and expiration are honored on every role; there is no native delayed delivery (delays route through + // the runtime-store fallback) and no broker-imposed size or batch limits. + private static readonly TransportCapabilities _capabilities = new() + { + Priority = true, + Expiration = true, + Ordering = OrderingGuarantee.Fifo + }; + + private static readonly IReadOnlySet _supportedRoles = new HashSet + { + DestinationRole.Queue, + DestinationRole.Topic, + DestinationRole.Subscription, + DestinationRole.Binding + }; + + private readonly ConcurrentDictionary _destinations = new(StringComparer.OrdinalIgnoreCase); + private readonly ConcurrentDictionary _roles = new(StringComparer.OrdinalIgnoreCase); + private readonly ConcurrentDictionary> _topicSubscriptions = new(StringComparer.OrdinalIgnoreCase); + private readonly ConcurrentDictionary _redeliveryTimers = new(); + private readonly ConcurrentDictionary _warnedDroppedTopics = new(StringComparer.OrdinalIgnoreCase); + private readonly TimeProvider _timeProvider; + private readonly ILogger _logger; + private readonly CancellationTokenSource _disposeCancellationTokenSource = new(); + private readonly object _reclaimGate = new(); + private readonly ITimer _reclaimTimer; + private int _reclaimActive; + private int _reclaimRunning; + private int _isDisposed; + + public InMemoryMessageTransport(TimeProvider? timeProvider = null, ILoggerFactory? loggerFactory = null) + { + _timeProvider = timeProvider ?? TimeProvider.System; + _logger = loggerFactory?.CreateLogger() ?? NullLogger.Instance; + _reclaimTimer = _timeProvider.CreateTimer(ReclaimExpired, null, Timeout.InfiniteTimeSpan, Timeout.InfiniteTimeSpan); + } + + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => _supportedRoles; + + public TransportCapabilities GetCapabilities(DestinationAddress destination) => _capabilities; + + // The in-memory transport has no broker-imposed ceiling on visibility or redelivery delay. + public TimeSpan? MaxVisibilityTimeout => null; + public TimeSpan? MaxRedeliveryDelay => null; + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(destination); + ArgumentNullException.ThrowIfNull(messages); + + if (options.DeliverAt is { } deliverAt && deliverAt > _timeProvider.GetUtcNow()) + throw new NotSupportedException($"Transport \"{GetType().Name}\" does not support native delayed delivery. Use the runtime-store scheduled dispatch fallback."); + + // The address role picks the physical namespace, so a queue and a topic can share a route name (a message + // type that is both sent and published) without colliding or cross-delivering. + string key = StorageKey(destination); + + var results = new SendItemResult[messages.Count]; + for (int index = 0; index < messages.Count; index++) + { + var message = messages[index]; + // Each message gets a unique id so per-message settlement never aliases across distinct messages. + string messageId = Guid.NewGuid().ToString("N"); + var stored = CreateStoredMessage(key, messageId, message, options); + EnqueueForDestination(key, stored); + + results[index] = new SendItemResult { MessageId = messageId }; + } + + return Task.FromResult(new SendResult { Items = results }); + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + { + return ReceiveAsync(source, request, visibility: TimeSpan.FromMinutes(1), ct); + } + + public async Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, TimeSpan visibility, CancellationToken ct) + { + return await ReceiveAsync(source, request, (TimeSpan?)visibility, ct).AnyContext(); + } + + private async Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, TimeSpan? visibility, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(source); + + int maxMessages = request.MaxMessages <= 0 ? 1 : request.MaxMessages; + if (!_destinations.TryGetValue(ReceivableKey(source), out var state)) + { + if (source.Role == DestinationRole.Subscription) + throw new MessageDestinationNotFoundException(source, new InvalidOperationException("The subscription must be provisioned before receiving.")); + state = GetOrAddDestination(ReceivableKey(source)); + } + var entries = new List(maxMessages); + DateTimeOffset? waitUntil = request.MaxWaitTime is { } waitTime && waitTime > TimeSpan.Zero + ? _timeProvider.GetUtcNow().Add(waitTime) + : null; + + // Return any messages whose visibility window lapsed (consumer crashed without settling) to the queue so + // they are redelivered with an incremented delivery count — honoring the advertised at-least-once contract. + state.ReclaimExpired(_timeProvider.GetUtcNow()); + + while (entries.Count < maxMessages) + { + if (TryReceive(source, state, visibility, out var entry)) + { + entries.Add(entry); + continue; + } + + if (entries.Count > 0 || waitUntil is null) + break; + + TimeSpan remaining = waitUntil.Value - _timeProvider.GetUtcNow(); + if (remaining <= TimeSpan.Zero) + break; + + using var waitCancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(ct, _disposeCancellationTokenSource.Token); + waitCancellationTokenSource.CancelAfter(remaining); + + try + { + if (!await state.WaitToReadAsync(waitCancellationTokenSource.Token).ConfigureAwait(false)) + break; + } + catch (OperationCanceledException) when (!ct.IsCancellationRequested && !_disposeCancellationTokenSource.IsCancellationRequested) + { + break; + } + } + + return entries; + } + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(entry); + + var receipt = GetReceipt(entry); + var state = GetExistingDestination(receipt.Destination); + + if (!state.InFlight.TryRemove(receipt, out var inFlight) || !String.Equals(inFlight.Message.Id, entry.Id, StringComparison.Ordinal)) + throw new ReceiptExpiredException(); + + Interlocked.Increment(ref state.Completed); + return Task.CompletedTask; + } + + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(entry); + + var receipt = GetReceipt(entry); + var state = GetExistingDestination(receipt.Destination); + + if (!state.InFlight.TryRemove(receipt, out var inFlight) || !String.Equals(inFlight.Message.Id, entry.Id, StringComparison.Ordinal)) + throw new ReceiptExpiredException(); + + Interlocked.Increment(ref state.Abandoned); + var redelivered = inFlight.Message with { DeliveryCount = entry.DeliveryCount + 1 }; + EnqueueStoredMessage(receipt.Destination, redelivered); + + return Task.CompletedTask; + } + + public Task AbandonAsync(TransportEntry entry, TimeSpan redeliveryDelay, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(entry); + + if (redeliveryDelay <= TimeSpan.Zero) + return AbandonAsync(entry, ct); + + var receipt = GetReceipt(entry); + var state = GetExistingDestination(receipt.Destination); + + if (!state.InFlight.TryRemove(receipt, out var inFlight) || !String.Equals(inFlight.Message.Id, entry.Id, StringComparison.Ordinal)) + throw new ReceiptExpiredException(); + + Interlocked.Increment(ref state.Abandoned); + var redelivered = inFlight.Message with { DeliveryCount = entry.DeliveryCount + 1 }; + ScheduleRedelivery(receipt.Destination, redelivered, redeliveryDelay); + + return Task.CompletedTask; + } + + public Task RenewLockAsync(TransportEntry entry, TimeSpan? duration, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(entry); + + var receipt = GetReceipt(entry); + var state = GetExistingDestination(receipt.Destination); + + if (!state.InFlight.TryGetValue(receipt, out var inFlight) || !String.Equals(inFlight.Message.Id, entry.Id, StringComparison.Ordinal)) + throw new ReceiptExpiredException(); + + // Renewal only extends a finite visibility window. A message received without a window holds an indefinite + // lock, so there is nothing to extend — leave it as-is rather than imposing a window that could reclaim it. + if (inFlight.VisibilityExpiresUtc is null) + return Task.CompletedTask; + + var renewed = inFlight with { VisibilityExpiresUtc = _timeProvider.GetUtcNow().Add(duration ?? _defaultLockRenewal) }; + if (!state.InFlight.TryUpdate(receipt, renewed, inFlight)) + throw new ReceiptExpiredException(); + + return Task.CompletedTask; + } + + public Task DeadLetterAsync(TransportEntry entry, string? reason, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(entry); + + var receipt = GetReceipt(entry); + var state = GetExistingDestination(receipt.Destination); + + if (!state.InFlight.TryRemove(receipt, out var inFlight) || !String.Equals(inFlight.Message.Id, entry.Id, StringComparison.Ordinal)) + throw new ReceiptExpiredException(); + + // Dead-letter with the caller's entry headers (which may carry forensics stamped by the core), not the + // originally-stored ones. + DeadLetter(state, inFlight.Message with { Headers = entry.Headers }, reason); + return Task.CompletedTask; + } + + public Task> PeekDeadLetteredAsync(DestinationAddress destination, DeadLetterQuery? query = null, CancellationToken cancellationToken = default) + { + ThrowIfDisposed(); + cancellationToken.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(destination); + query ??= new DeadLetterQuery(); + query.Validate(); + if (!_destinations.TryGetValue(ReceivableKey(destination), out var state)) + return Task.FromResult>([]); + var entries = state.Deadletters.Values.Where(m => query.AfterId is null || StringComparer.Ordinal.Compare(m.Id, query.AfterId) > 0) + .OrderBy(m => m.Id, StringComparer.Ordinal).Take(query.Limit).Select(message => new TransportEntry + { + Id = message.Id, + ApplicationMessageId = message.ApplicationMessageId, + ContentType = message.ContentType, + Destination = destination, + Body = message.Body, + Headers = message.Headers, + DeliveryCount = message.DeliveryCount, + EnqueuedUtc = message.EnqueuedUtc, + Receipt = default + }).ToArray(); + return Task.FromResult>(entries); + } + + public Task DeleteDeadLetteredAsync(DestinationAddress destination, string id, CancellationToken cancellationToken = default) + { + ThrowIfDisposed(); + cancellationToken.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(destination); + ArgumentException.ThrowIfNullOrWhiteSpace(id); + if (!_destinations.TryGetValue(ReceivableKey(destination), out var state)) + return Task.FromResult(false); + lock (state.Deadletters) + return Task.FromResult(state.Deadletters.TryRemove(id, out _)); + } + + public Task ReplayDeadLetteredAsync(DestinationAddress source, string id, DestinationAddress target, CancellationToken cancellationToken = default) + { + ThrowIfDisposed(); + cancellationToken.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(source); + ArgumentNullException.ThrowIfNull(target); + ArgumentException.ThrowIfNullOrWhiteSpace(id); + if (target.Role is not (DestinationRole.Queue or DestinationRole.Topic)) + throw new ArgumentException("Replay targets must be a queue or topic.", nameof(target)); + if (!_destinations.TryGetValue(ReceivableKey(source), out var state)) + return Task.FromResult(false); + lock (state.Deadletters) + { + if (!state.Deadletters.TryGetValue(id, out var message)) + return Task.FromResult(false); + var headers = MessageHeaders.Create(message.Headers.Where(h => !h.Key.Equals(KnownHeaders.Attempts, StringComparison.OrdinalIgnoreCase) + && !h.Key.Equals(KnownHeaders.Expiration, StringComparison.OrdinalIgnoreCase) + && !h.Key.StartsWith("message.dead_letter.", StringComparison.OrdinalIgnoreCase))); + string key = StorageKey(target); + var replayed = CreateStoredMessage(key, Guid.NewGuid().ToString("N"), new TransportMessage + { + MessageId = message.ApplicationMessageId, + Body = message.Body, + Headers = headers, + ContentType = message.ContentType + }, new TransportSendOptions()); + EnqueueForDestination(key, replayed); + state.Deadletters.TryRemove(id, out _); + return Task.FromResult(true); + } + } + + public Task GetStatsAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(destination); + + if (!_destinations.TryGetValue(ReceivableKey(destination), out var state)) + return Task.FromResult(new MessageDestinationStats()); + + return Task.FromResult(new MessageDestinationStats + { + Queued = state.QueuedCount, + Working = state.InFlight.Count, + Delayed = _redeliveryTimers.Values.LongCount(key => key == ReceivableKey(destination)), + Deadletter = state.DeadletterCount, + Enqueued = Volatile.Read(ref state.Enqueued), + Dequeued = Volatile.Read(ref state.Dequeued), + Completed = Volatile.Read(ref state.Completed), + Abandoned = Volatile.Read(ref state.Abandoned) + }); + } + + public Task EnsureAsync(IReadOnlyList declarations, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(declarations); + + foreach (var declaration in declarations) + { + var address = declaration.Address; + ArgumentNullException.ThrowIfNull(address); + + lock (_temporarySubscriptions) + { + switch (address.Role) + { + case DestinationRole.Queue: + GetOrAddDestination(StorageKey(address)); + break; + case DestinationRole.Topic: + _roles.TryAdd(StorageKey(address), DestinationRole.Topic); + _topicSubscriptions.GetOrAdd(StorageKey(address), static _ => new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase)); + break; + case DestinationRole.Subscription: + if (String.IsNullOrEmpty(address.Topic)) + throw new ArgumentException("A subscription declaration must specify its owning topic.", nameof(declarations)); + + AddTopicSubscription(address.Topic, StorageKey(address)); + break; + case DestinationRole.Binding: + if (String.IsNullOrEmpty(address.Topic)) + throw new ArgumentException("A binding declaration must specify a source topic.", nameof(declarations)); + + AddTopicSubscription(address.Topic, StorageKey(address)); + break; + default: + throw new ArgumentOutOfRangeException(nameof(declarations), address.Role, "Unsupported destination role."); + } + if (declaration.AutoDeleteAfter is { } lease) + CreateTemporarySubscription(address, lease); + } + } + + return Task.CompletedTask; + } + + public Task DeleteAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(destination); + + lock (_temporarySubscriptions) + { + string key = StorageKey(destination); + if (_temporarySubscriptions.Remove(key, out var lease)) + lease.Timer.Dispose(); + DeleteDestination(key); + } + + return Task.CompletedTask; + } + + private void DeleteDestination(string key) + { + _roles.TryRemove(key, out _); + if (_destinations.TryRemove(key, out var removed)) + removed.Complete(); + _topicSubscriptions.TryRemove(key, out _); + + foreach (var subscriptions in _topicSubscriptions.Values) + subscriptions.TryRemove(key, out _); + + } + + public Task ExistsAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(destination); + + return Task.FromResult(_roles.ContainsKey(StorageKey(destination))); + } + + public ValueTask DisposeAsync() + { + DisposeTemporarySubscriptions(); + if (Interlocked.Exchange(ref _isDisposed, 1) == 1) + return ValueTask.CompletedTask; + + _disposeCancellationTokenSource.Cancel(); + _disposeCancellationTokenSource.Dispose(); + lock (_reclaimGate) + _reclaimTimer.Dispose(); + + foreach (var timer in _redeliveryTimers.Keys) + { + if (_redeliveryTimers.TryRemove(timer, out _)) + timer.Dispose(); + } + + _destinations.Clear(); + _roles.Clear(); + _topicSubscriptions.Clear(); + return ValueTask.CompletedTask; + } + + private void EnqueueForDestination(string key, StoredMessage message) + { + // Topic sends fan out one copy per subscription; a topic with no subscriptions drops the message (real + // pub/sub semantics — subscriptions must exist before a publish can reach them). The drop is the single + // most confusing beginner outcome ("I published and nothing happened"), so it is warned once per topic. + if (key.StartsWith("t:", StringComparison.Ordinal)) + { + if (!_topicSubscriptions.TryGetValue(key, out var subscriptions) || subscriptions.IsEmpty) + { + if (_warnedDroppedTopics.TryAdd(key, 0)) + _logger.LogWarning("Message published to topic \"{Topic}\" was dropped: no subscriptions exist. Subscriptions are created when a handler subscribes (or via topology provisioning); publish after they exist. Further drops on this topic will not be logged", key[2..]); + return; + } + + foreach (string subscription in subscriptions.Keys) + EnqueueStoredMessage(subscription, message with { Destination = subscription }); + + return; + } + + EnqueueStoredMessage(key, message); + } + + private void EnqueueStoredMessage(string key, StoredMessage message) + { + var state = GetOrAddDestination(key); + state.Enqueue(message with { Destination = key }); + } + + // Make an abandoned message invisible for the redelivery delay, then re-enqueue it. Re-enqueueing releases the + // destination's availability semaphore, so a consumer blocked in a long receive wait wakes immediately when the + // message becomes due. The one-shot timer is tracked so it can be disposed if the transport is torn down first. + private void ScheduleRedelivery(string destination, StoredMessage message, TimeSpan delay) + { + ITimer? timer = null; + timer = _timeProvider.CreateTimer(timerState => + { + if (timer is not null && _redeliveryTimers.TryRemove(timer, out _)) + timer.Dispose(); + + if (Volatile.Read(ref _isDisposed) == 1) + return; + + try + { + EnqueueStoredMessage(destination, message); + } + catch (ObjectDisposedException) { } + catch (InvalidOperationException) { } // destination was deleted / completed between scheduling and firing + }, null, Timeout.InfiniteTimeSpan, Timeout.InfiniteTimeSpan); + + _redeliveryTimers[timer] = destination; + timer.Change(delay, Timeout.InfiniteTimeSpan); + + // A redelivery scheduled right as the transport disposes could otherwise leak its timer; clean up the race. + if (Volatile.Read(ref _isDisposed) == 1 && _redeliveryTimers.TryRemove(timer, out _)) + timer.Dispose(); + } + + private void EnsureReclaimTimer() + { + if (Volatile.Read(ref _reclaimActive) == 1) + return; + lock (_reclaimGate) + { + if (_reclaimActive == 1 || Volatile.Read(ref _isDisposed) == 1) + return; + Volatile.Write(ref _reclaimActive, 1); + _reclaimTimer.Change(_reclaimInterval, _reclaimInterval); + } + } + + private void ReclaimExpired(object? timerState) + { + if (Volatile.Read(ref _isDisposed) == 1 || Interlocked.CompareExchange(ref _reclaimRunning, 1, 0) != 0) + return; + try + { + var now = _timeProvider.GetUtcNow(); + foreach (var destination in _destinations) + { + try { destination.Value.ReclaimExpired(now); } + catch (InvalidOperationException) { } // Destination was completed/deleted during reclamation. + } + + if (HasInFlightMessages()) + return; + lock (_reclaimGate) + { + if (Volatile.Read(ref _isDisposed) == 1) + return; + // New receivers must observe the inactive flag before the final emptiness check. + Volatile.Write(ref _reclaimActive, 0); + if (HasInFlightMessages()) + Volatile.Write(ref _reclaimActive, 1); + else + _reclaimTimer.Change(Timeout.InfiniteTimeSpan, Timeout.InfiniteTimeSpan); + } + } + finally + { + Volatile.Write(ref _reclaimRunning, 0); + } + } + + private bool HasInFlightMessages() + { + foreach (var destination in _destinations) + if (!destination.Value.InFlight.IsEmpty) + return true; + return false; + } + + private bool TryReceive(DestinationAddress source, DestinationState state, TimeSpan? visibility, out TransportEntry entry) + { + while (state.TryDequeue(out var message)) + { + if (IsExpired(message)) + { + DeadLetter(state, message, "expired"); + continue; + } + + // The receipt carries the internal (role-qualified) key so settlement resolves the same state; the entry's + // Destination stays the caller-facing source address. + var receipt = new InMemoryReceipt(state.Key); + DateTimeOffset? visibilityExpiresUtc = visibility is { } window ? _timeProvider.GetUtcNow().Add(window) : null; + state.InFlight[receipt] = new InFlightMessage(message, receipt, visibilityExpiresUtc); + Interlocked.Increment(ref state.Dequeued); + + if (visibility is not null) + EnsureReclaimTimer(); + + entry = new TransportEntry + { + Id = message.Id, + ApplicationMessageId = message.ApplicationMessageId, + ContentType = message.ContentType, + Destination = source, + LockExpiresUtc = visibilityExpiresUtc, + Body = message.Body, + Headers = message.Headers, + DeliveryCount = message.DeliveryCount, + EnqueuedUtc = message.EnqueuedUtc, + Receipt = new Receipt { TransportState = receipt } + }; + return true; + } + + entry = null!; + return false; + } + + private bool IsExpired(StoredMessage message) + { + string? expiration = message.Headers.GetValueOrDefault(KnownHeaders.Expiration); + if (expiration is null) + return false; + + return DateTimeOffset.TryParse(expiration, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal, out var expiresAt) + && expiresAt <= _timeProvider.GetUtcNow(); + } + + private void DeadLetter(DestinationState state, StoredMessage message, string? reason) + { + if (!String.IsNullOrEmpty(reason)) + message = message with { Headers = message.Headers.ToBuilder().Set(KnownHeaders.DeadLetterReason, reason).Build() }; + + state.Deadletter(message); + } + + private StoredMessage CreateStoredMessage(string destination, string messageId, TransportMessage message, TransportSendOptions options) + { + var headers = message.Headers.ToBuilder() + .SetIfMissing(KnownHeaders.Priority, options.Priority.ToString()) + .Build(); + int deliveryCount = Int32.TryParse(headers.GetValueOrDefault(KnownHeaders.Attempts), NumberStyles.Integer, CultureInfo.InvariantCulture, out int attempts) && attempts > 0 + ? attempts + : 1; + + return new StoredMessage( + messageId, + message.MessageId, + message.ContentType, + destination, + message.Body.ToArray(), + headers, + NormalizePriority(options.Priority), + DeliveryCount: deliveryCount, + EnqueuedUtc: _timeProvider.GetUtcNow()); + } + + // Internal state is keyed by role-qualified names derived from the canonical address: "t:" for topics, "q:" for + // every receivable destination (queues AND subscriptions — a subscription is a queue-shaped destination a topic + // fans into, exactly like an SNS-bound SQS queue, keyed by its topic-qualified address key). This gives a + // queue/subscription and a topic sharing a route name distinct namespaces, as real brokers do. + private static string StorageKey(DestinationAddress address) => + address.Role == DestinationRole.Topic ? "t:" + address.Name : "q:" + address.Key; + + // Receive-path keys are always queue-shaped; receive/stats/dead-letter reads never target a topic. + private static string ReceivableKey(DestinationAddress address) => "q:" + address.Key; + + private static DestinationRole RoleForKey(string key) => key[0] == 't' ? DestinationRole.Topic : DestinationRole.Queue; + + private DestinationState GetOrAddDestination(string key) + { + _roles.TryAdd(key, RoleForKey(key)); + return _destinations.GetOrAdd(key, static name => new DestinationState(name)); + } + + private DestinationState GetExistingDestination(string key) + { + if (_destinations.TryGetValue(key, out var destination)) + return destination; + + throw new ReceiptExpiredException($"The destination \"{key}\" no longer exists."); + } + + private void AddTopicSubscription(string topic, string subscriptionStorageKey) + { + string topicKey = "t:" + topic; + _roles.TryAdd(topicKey, DestinationRole.Topic); + GetOrAddDestination(subscriptionStorageKey); + var subscriptions = _topicSubscriptions.GetOrAdd(topicKey, static _ => new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase)); + subscriptions[subscriptionStorageKey] = 0; + } + + private static MessagePriority NormalizePriority(MessagePriority priority) + { + return priority switch + { + MessagePriority.Low => MessagePriority.Low, + MessagePriority.Normal => MessagePriority.Normal, + MessagePriority.High => MessagePriority.High, + _ => MessagePriority.Normal + }; + } + + private static InMemoryReceipt GetReceipt(TransportEntry entry) + { + return entry.Receipt.TransportState as InMemoryReceipt ?? throw new ReceiptExpiredException(); + } + + private void ThrowIfDisposed() + { + ObjectDisposedException.ThrowIf(Volatile.Read(ref _isDisposed) == 1, this); + } + + private sealed record StoredMessage( + string Id, + string? ApplicationMessageId, + string? ContentType, + string Destination, + ReadOnlyMemory Body, + MessageHeaders Headers, + MessagePriority Priority, + int DeliveryCount, + DateTimeOffset EnqueuedUtc); + + private sealed record InFlightMessage(StoredMessage Message, InMemoryReceipt Receipt, DateTimeOffset? VisibilityExpiresUtc); + + // Receipt identity is the lock token: each delivery gets a fresh object, so a stale receipt + // cannot settle a redelivery, even when it carries the same message id. + private sealed class InMemoryReceipt(string destination) + { + public string Destination { get; } = destination; + } + + private sealed class DestinationState(string key) + { + public string Key { get; } = key; + + private readonly Channel[] _channels = + [ + Channel.CreateUnbounded(CreateChannelOptions()), + Channel.CreateUnbounded(CreateChannelOptions()), + Channel.CreateUnbounded(CreateChannelOptions()) + ]; + + public ConcurrentDictionary Deadletters { get; } = new(StringComparer.Ordinal); + private readonly SemaphoreSlim _availableMessages = new(0); + private long _queuedCount; + private int _isCompleted; + + public ConcurrentDictionary InFlight { get; } = new(); + public long Enqueued; + public long Dequeued; + public long Completed; + public long Abandoned; + public long Deadlettered; + + public long QueuedCount => Volatile.Read(ref _queuedCount); + public long DeadletterCount => Deadletters.Count; + + public void Enqueue(StoredMessage message) + { + if (!_channels[(int)message.Priority].Writer.TryWrite(message)) + throw new InvalidOperationException("The destination is no longer accepting messages."); + + Interlocked.Increment(ref _queuedCount); + Interlocked.Increment(ref Enqueued); + _availableMessages.Release(); + } + + public bool TryDequeue(out StoredMessage message) + { + for (int index = (int)MessagePriority.High; index >= (int)MessagePriority.Low; index--) + { + if (_channels[index].Reader.TryRead(out message!)) + { + _availableMessages.Wait(0); + Interlocked.Decrement(ref _queuedCount); + return true; + } + } + + message = null!; + return false; + } + + public async ValueTask WaitToReadAsync(CancellationToken cancellationToken) + { + if (QueuedCount > 0) + return true; + + await _availableMessages.WaitAsync(cancellationToken).ConfigureAwait(false); + return true; + } + + public void Deadletter(StoredMessage message) + { + if (Volatile.Read(ref _isCompleted) == 1) + throw new InvalidOperationException("The destination is no longer accepting dead-letter messages."); + Deadletters[message.Id] = message; + Interlocked.Increment(ref Deadlettered); + } + + public void ReclaimExpired(DateTimeOffset now) + { + if (Volatile.Read(ref _isCompleted) == 1) + return; + + foreach (var kvp in InFlight) + { + if (kvp.Value.VisibilityExpiresUtc is { } expiry && expiry <= now && InFlight.TryRemove(kvp)) + { + Interlocked.Increment(ref Abandoned); + Enqueue(kvp.Value.Message with { DeliveryCount = kvp.Value.Message.DeliveryCount + 1 }); + } + } + } + + public void Complete() + { + if (Interlocked.Exchange(ref _isCompleted, 1) == 1) + return; + + foreach (var channel in _channels) + channel.Writer.TryComplete(); + + Deadletters.Clear(); + } + + private static UnboundedChannelOptions CreateChannelOptions() + { + return new UnboundedChannelOptions + { + AllowSynchronousContinuations = false, + SingleReader = false, + SingleWriter = false + }; + } + } + +} diff --git a/src/Foundatio/Messaging/KnownHeaders.cs b/src/Foundatio/Messaging/KnownHeaders.cs new file mode 100644 index 000000000..60dc719f2 --- /dev/null +++ b/src/Foundatio/Messaging/KnownHeaders.cs @@ -0,0 +1,24 @@ +namespace Foundatio.Messaging; + +public static class KnownHeaders +{ + public const string MessageId = "message.id"; + public const string MessageType = "message.type"; + public const string ContentType = "message.content_type"; + public const string CorrelationId = "message.correlation_id"; + public const string TraceParent = "traceparent"; + public const string TraceState = "tracestate"; + public const string Priority = "message.priority"; + public const string Expiration = "message.expiration"; + public const string Attempts = "message.attempts"; + public const string DeadLetterReason = "message.dead_letter.reason"; + + // Forensics stamped by the core when a message is dead-lettered, so a dead message is triageable with plain + // transport tooling. These names are a compatibility contract; values are truncated to fit transport limits. + public const string DeadLetterAttempts = "message.dead_letter.attempts"; + public const string DeadLetterExceptionType = "message.dead_letter.exception_type"; + public const string DeadLetterExceptionMessage = "message.dead_letter.exception_message"; + public const string DeadLetterExceptionStackTrace = "message.dead_letter.exception_stack"; + public const string DeadLetterFailedAt = "message.dead_letter.failed_at"; + public const string DeadLetterOriginalDestination = "message.dead_letter.original_destination"; +} diff --git a/src/Foundatio/Messaging/LegacyMessageBusAdapter.cs b/src/Foundatio/Messaging/LegacyMessageBusAdapter.cs new file mode 100644 index 000000000..af7173ac0 --- /dev/null +++ b/src/Foundatio/Messaging/LegacyMessageBusAdapter.cs @@ -0,0 +1,82 @@ +using System; +using System.Collections.Concurrent; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Utility; + +namespace Foundatio.Messaging.Legacy; + +/// +/// Migration adapter: implements the legacy publish/subscribe interfaces over the redesigned +/// so existing consuming code keeps compiling while it migrates. +/// Register with AddFoundatio().Messaging.AddLegacyAdapter() and delete the call once call sites are on the +/// new API — there is no legacy bus implementation behind this, only the mapping. +/// +/// +/// Semantics map as follows. Every legacy subscription is per-instance and published-only, matching the old bus's +/// fan-out of every message to every subscriber in every process. maps to +/// a delayed publish (durable through the runtime store when one is configured — an upgrade over the old in-memory +/// timer). and map to the +/// correlation id and headers. has no equivalent (broker deduplication does not +/// exist in the new contract) and is ignored. Messages route by their runtime type through the new routing +/// conventions, so a subscriber of a base/interface type only sees derived messages when routing maps them to the +/// same topic (MapTopic/UseDefaultTopic) — the old bus was one implicit shared channel; the new bus is +/// destination-scoped. For the same reason the old raw-envelope (IMessage) tap has no adapter path: subscribe +/// to concrete types, or use the new bus's untyped SubscribeAsync on an explicitly routed topic. +/// +public sealed class LegacyMessageBusAdapter : IMessageBus +{ + private readonly Foundatio.Messaging.IMessageBus _bus; + private readonly ConcurrentQueue _subscriptions = new(); + private int _isDisposed; + + public LegacyMessageBusAdapter(Foundatio.Messaging.IMessageBus bus) + { + _bus = bus ?? throw new ArgumentNullException(nameof(bus)); + } + + public Task PublishAsync(Type messageType, object message, MessageOptions? options = null, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(messageType); + ArgumentNullException.ThrowIfNull(message); + + var publishOptions = new MessagePublishOptions + { + Delay = options?.DeliveryDelay, + CorrelationId = options?.CorrelationId, + Headers = options?.Properties is { Count: > 0 } properties ? MessageHeaders.Create(properties) : null + }; + + return _bus.PublishBatchAsync([message], publishOptions, cancellationToken); + } + + public async Task SubscribeAsync(Func handler, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(handler); + + // The old bus delivered every published message to every subscriber in every process: per-instance, + // events only. Auto-ack on return, retry on throw now come from the core policy instead of being swallowed. + var options = new MessageSubscriptionOptions(); + + var subscription = await _bus.SubscribeAsync((context, token) => handler(context.Message, token), options, cancellationToken).AnyContext(); + + _subscriptions.Enqueue(subscription); + if (cancellationToken.CanBeCanceled) + cancellationToken.Register(() => _ = subscription.DisposeAsync()); + } + + public void Dispose() + { + DisposeAsync().AsTask().GetAwaiter().GetResult(); + } + + // Disposes only the subscriptions this adapter created; the underlying bus is owned by whoever registered it. + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _isDisposed, 1) == 1) + return; + + while (_subscriptions.TryDequeue(out var subscription)) + await subscription.DisposeAsync().AnyContext(); + } +} diff --git a/src/Foundatio/Messaging/LegacyMessageBusException.cs b/src/Foundatio/Messaging/LegacyMessageBusException.cs new file mode 100644 index 000000000..aef314810 --- /dev/null +++ b/src/Foundatio/Messaging/LegacyMessageBusException.cs @@ -0,0 +1,17 @@ +using System; + +namespace Foundatio.Messaging.Legacy; + +/// +/// Exception thrown when a legacy message bus operation fails. +/// +public class MessageBusException : Exception +{ + public MessageBusException(string message) : base(message) + { + } + + public MessageBusException(string message, Exception innerException) : base(message, innerException) + { + } +} diff --git a/src/Foundatio/Messaging/Message.cs b/src/Foundatio/Messaging/Message.cs deleted file mode 100644 index eeaad8fc5..000000000 --- a/src/Foundatio/Messaging/Message.cs +++ /dev/null @@ -1,115 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Diagnostics; -using System.Diagnostics.CodeAnalysis; - -namespace Foundatio.Messaging; - -/// -/// Represents a message received from the message bus with metadata and raw payload. -/// Subscribe to to receive all message types. -/// -public interface IMessage -{ - /// - /// Gets the unique identifier for this message instance. - /// - string? UniqueId { get; } - - /// - /// Gets the correlation identifier for distributed tracing. - /// - string? CorrelationId { get; } - - /// - /// Gets the message type name used for routing, or null if the message was received from - /// an external publisher that did not provide type metadata. - /// - string? Type { get; } - - /// - /// Gets the CLR type of the message payload, or null if the type cannot be resolved. - /// - Type? ClrType { get; } - - /// - /// Gets the raw serialized message payload. - /// - /// - /// Returned as a to avoid forcing an array copy on providers - /// whose transport buffers are already memory-backed (e.g. Azure Service Bus, RabbitMQ). The buffer - /// is only guaranteed valid for the duration of message handling: some providers (e.g. RabbitMQ) - /// expose a pooled transport buffer that is reclaimed once the handler returns. Consumers that need - /// to retain the payload beyond the current handler invocation must copy it via ToArray(). - /// - ReadOnlyMemory Data { get; } - - /// - /// Deserializes and returns the message payload. - /// - object? GetBody(); - - /// - /// Gets custom properties attached to this message. - /// - IDictionary Properties { get; } -} - -/// -/// A typed message providing strongly-typed access to the message payload. -/// -/// The type of message payload. -public interface IMessage : IMessage where T : class -{ - /// - /// Gets the deserialized message payload. - /// - T Body { get; } -} - -[DebuggerDisplay("Type: {Type}")] -public class Message : IMessage -{ - private readonly Func _getBody; - - public Message(ReadOnlyMemory data, Func getBody) - { - Data = data; - _getBody = getBody; - } - - public string? UniqueId { get; set; } - public string? CorrelationId { get; set; } - public string? Type { get; set; } - public Type? ClrType { get; set; } - [DisallowNull] - public IDictionary Properties { get => field; set => field = value ?? new Dictionary(); } = new Dictionary(); - public ReadOnlyMemory Data { get; set; } - public object? GetBody() => _getBody(this); -} - -public class Message : IMessage where T : class -{ - private readonly IMessage _message; - - public Message(IMessage message) - { - _message = message; - } - - public ReadOnlyMemory Data => _message.Data; - - public T Body => GetBody() as T ?? throw new MessageBusException("Message body is null or not of expected type"); - - public string? UniqueId => _message.UniqueId; - - public string? CorrelationId => _message.CorrelationId; - - public string? Type => _message.Type; - - public Type? ClrType => _message.ClrType; - - public IDictionary Properties => _message.Properties; - - public object? GetBody() => _message.GetBody(); -} diff --git a/src/Foundatio/Messaging/MessageAdministration.cs b/src/Foundatio/Messaging/MessageAdministration.cs new file mode 100644 index 000000000..92b35afb7 --- /dev/null +++ b/src/Foundatio/Messaging/MessageAdministration.cs @@ -0,0 +1,139 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Foundatio.Messaging; + +/// Provider-neutral queue statistics and bounded dead-letter inspection and recovery. +public sealed class MessageAdministration(IMessageTransport transport, TimeProvider? timeProvider = null, ILogger? logger = null) +{ + private readonly ILogger _logger = logger ?? NullLogger.Instance; + private readonly TimeProvider _time = timeProvider ?? TimeProvider.System; + private static DestinationAddress DeadLetters(DestinationAddress source) => DestinationAddress.ForQueue(source.Name + ".deadletter"); + + public async Task GetStatsAsync(DestinationAddress source, CancellationToken cancellationToken = default) + { + if (transport is not ISupportsStats stats) throw new NotSupportedException("This transport does not report queue statistics."); + var value = await stats.GetStatsAsync(source, cancellationToken).ConfigureAwait(false); + if (transport is ISupportsDeadLetterSink) return value; + if (transport is ISupportsProvisioning provisioning && !await provisioning.ExistsAsync(DeadLetters(source), cancellationToken).ConfigureAwait(false)) return value; + var dead = await stats.GetStatsAsync(DeadLetters(source), cancellationToken).ConfigureAwait(false); + return value with { Deadletter = dead.Queued + dead.Working }; + } + + /// Returns a bounded snapshot. Inspection leaves the messages available for subsequent recovery. + public async Task> PeekDeadLettersAsync(DestinationAddress source, int limit = 20, CancellationToken cancellationToken = default) + { + ArgumentOutOfRangeException.ThrowIfLessThan(limit, 1); + ArgumentOutOfRangeException.ThrowIfGreaterThan(limit, 1000); + if (transport is ISupportsDeadLetter native) + return await native.PeekDeadLetteredAsync(source, new DeadLetterQuery { Limit = limit }, cancellationToken).ConfigureAwait(false); + var found = new List(limit); + await VisitAsync(source, limit, (entry, _) => { found.Add(entry); return Task.FromResult(false); }, cancellationToken).ConfigureAwait(false); + return found; + } + + public async Task DeleteDeadLetterAsync(DestinationAddress source, string id, CancellationToken cancellationToken = default) + { + if (transport is ISupportsDeadLetter native) return await native.DeleteDeadLetteredAsync(source, id, cancellationToken).ConfigureAwait(false); + bool deleted = false; + await VisitAsync(source, 1000, async (entry, token) => + { + if (entry.Id != id) return false; + await transport.CompleteAsync(entry, token).ConfigureAwait(false); + deleted = true; + return true; + }, cancellationToken).ConfigureAwait(false); + return deleted; + } + + /// Replays one message with optional new execution metadata. A confirmed send precedes deleting its dead letter. + public async Task ReplayDeadLetterAsync(DestinationAddress source, string id, + Func>? prepare = null, CancellationToken cancellationToken = default) + { + if (transport is ISupportsDeadLetter native) + { + if (prepare is null) return await native.ReplayDeadLetteredAsync(source, id, source, cancellationToken).ConfigureAwait(false); + string? after = null; + for (int inspected = 0; inspected < 1000; inspected += 100) + { + var page = await native.PeekDeadLetteredAsync(source, new DeadLetterQuery { Limit = 100, AfterId = after }, cancellationToken).ConfigureAwait(false); + if (page.FirstOrDefault(entry => entry.Id == id) is { } selected) + { + await ReplayAsync(selected, cancellationToken).ConfigureAwait(false); + if (!await native.DeleteDeadLetteredAsync(source, id, cancellationToken).ConfigureAwait(false)) + throw new MessageBusException("Replay was accepted, but the dead letter was already removed by another operation."); + return true; + } + if (page.Count < 100) return false; + after = page[^1].Id; + } + return false; + } + bool replayed = false; + await VisitAsync(source, 1000, async (entry, token) => + { + if (entry.Id != id) return false; + await ReplayAsync(entry, token).ConfigureAwait(false); + await transport.CompleteAsync(entry, token).ConfigureAwait(false); + replayed = true; + return true; + }, cancellationToken).ConfigureAwait(false); + return replayed; + + async Task ReplayAsync(TransportEntry entry, CancellationToken token) + { + var message = prepare is null + ? new TransportMessage { Body = entry.Body, ContentType = entry.ContentType, Headers = MessageHeaders.Create(entry.Headers.Where(pair => pair.Key != KnownHeaders.Attempts && !pair.Key.StartsWith("message.dead_letter.", StringComparison.Ordinal)).ToDictionary()), MessageId = entry.ApplicationMessageId } + : await prepare(entry, token).ConfigureAwait(false); + var result = await transport.SendAsync(source, [message], new TransportSendOptions(), token).ConfigureAwait(false); + if (result.Items.Count != 1 || result.Items[0].Status != MessageSendStatus.Accepted) + throw new MessageBusException("Replay was not confirmed. The original dead letter was retained; an unknown send may still have been accepted."); + } + } + + // Providers without native peeking use bounded receive/hold/release. Hold non-matches for the whole scan so + // a broker cannot repeatedly return the same first ten entries and hide a selected message further back. + private async Task VisitAsync(DestinationAddress source, int limit, Func> visit, CancellationToken cancellationToken) + { + if (transport is not ISupportsPull pull) throw new NotSupportedException("This transport cannot inspect dead letters."); + var destination = DeadLetters(source); + if (transport is ISupportsProvisioning provisioning && !await provisioning.ExistsAsync(destination, cancellationToken).ConfigureAwait(false)) return; + var held = new List(); + var seen = new HashSet(StringComparer.Ordinal); + long started = Stopwatch.GetTimestamp(); + try + { + while (seen.Count < limit && Stopwatch.GetElapsedTime(started) < TimeSpan.FromSeconds(10)) + { + var request = new ReceiveRequest { MaxMessages = Math.Min(10, limit - seen.Count), MaxWaitTime = TimeSpan.FromSeconds(1) }; + var batch = transport is ISupportsVisibilityTimeout visibility + ? await visibility.ReceiveAsync(destination, request, TimeSpan.FromSeconds(30), cancellationToken).ConfigureAwait(false) + : await pull.ReceiveAsync(destination, request, cancellationToken).ConfigureAwait(false); + if (batch.Count == 0) return; + held.AddRange(batch); + foreach (var entry in batch) + { + if (!seen.Add(entry.Id)) return; + if (await visit(entry, cancellationToken).ConfigureAwait(false)) { held.Remove(entry); return; } + } + } + } + finally + { + using var cleanup = new CancellationTokenSource(TimeSpan.FromSeconds(5), _time); + await Task.WhenAll(held.Select(async entry => + { + try { await transport.AbandonAsync(entry, cleanup.Token).WaitAsync(cleanup.Token).ConfigureAwait(false); } + catch (ReceiptExpiredException) { } + catch (OperationCanceledException) when (cleanup.IsCancellationRequested) { } + catch (Exception exception) { _logger.LogWarning(exception, "Unable to return inspected dead letter {MessageId}; its visibility lease will expire", entry.Id); } + })).ConfigureAwait(false); + } + } +} diff --git a/src/Foundatio/Messaging/MessageBus.cs b/src/Foundatio/Messaging/MessageBus.cs new file mode 100644 index 000000000..07b47a767 --- /dev/null +++ b/src/Foundatio/Messaging/MessageBus.cs @@ -0,0 +1,527 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Serializer; +using Foundatio.Utility; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Foundatio.Messaging; + +public sealed record MessageSendOptions +{ + /// Stable application ID for correlation and consumer deduplication. Null generates an ID. Does not make sends exactly once. + public string? MessageId { get; init; } + public MessagePriority Priority { get; init; } = MessagePriority.Normal; + public TimeSpan? Delay { get; init; } + public DateTimeOffset? DeliverAt { get; init; } + public TimeSpan? TimeToLive { get; init; } + public string? CorrelationId { get; init; } + /// Overrides the routed destination for this send. + public string? Destination { get; init; } + public MessageHeaders? Headers { get; init; } +} + +public sealed record MessagePublishOptions +{ + /// Stable application ID. Null generates an ID. Consumers remain responsible for deduplication. + public string? MessageId { get; init; } + public MessagePriority Priority { get; init; } = MessagePriority.Normal; + public TimeSpan? Delay { get; init; } + public DateTimeOffset? DeliverAt { get; init; } + public TimeSpan? TimeToLive { get; init; } + public string? CorrelationId { get; init; } + /// Overrides the routed topic for this publish. + public string? Topic { get; init; } + public MessageHeaders? Headers { get; init; } +} + +/// Failure handling and concurrency for one receiving endpoint. +public abstract class MessageHandlerOptions +{ + /// Optional stable wire name bound with a declarative handler registration. + public string? MessageTypeName { get; set; } + /// Maximum in-flight messages across this endpoint's handlers on this process. Default 1. + public int MaxConcurrency { get; set; } = 1; + + /// Maximum entries requested in one receive, additionally bounded by free concurrency and provider limits. + public int? PrefetchCount { get; set; } + + /// Optional bounded delay for collecting capacity after a partially filled batch. Null uses the provider policy. + public TimeSpan? ReceiveBatchDelay { get; set; } + + /// Initial and automatically renewed delivery lease. Default one minute. + public TimeSpan VisibilityTimeout { get; set; } = TimeSpan.FromMinutes(1); + + /// Renew active delivery leases automatically. Lease loss still cancels processing when disabled. + public bool AutoRenewLock { get; set; } = true; + + /// Time admitted handlers may finish after receiving stops. Zero cancels immediately. + public TimeSpan ShutdownTimeout { get; set; } + + /// Keep manual deliveries active until explicit settlement. False releases processing on handler return. + public bool WaitForManualSettlement { get; set; } = true; + + /// Maximum delivery attempts. Null uses the bus retry policy. + public int? MaxAttempts { get; set; } + + /// Delay after a failed delivery. Null uses the bus retry policy. + public Func? RedeliveryBackoff { get; set; } + + /// Identifies failures that must be dead-lettered immediately. + public Func? DeadLetterWhen { get; set; } + + /// Automatically acknowledge successful handlers, or require explicit settlement. + public AckMode AckMode { get; set; } = AckMode.Auto; + + internal void Validate() + { + ArgumentOutOfRangeException.ThrowIfLessThan(MaxConcurrency, 1); + if (PrefetchCount is { } prefetch) + ArgumentOutOfRangeException.ThrowIfLessThan(prefetch, 1); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(VisibilityTimeout, TimeSpan.Zero); + ArgumentOutOfRangeException.ThrowIfLessThan(ShutdownTimeout, TimeSpan.Zero); + if (ReceiveBatchDelay is { } batchDelay) ArgumentOutOfRangeException.ThrowIfLessThan(batchDelay, TimeSpan.Zero); + if (MaxAttempts is { } attempts) + ArgumentOutOfRangeException.ThrowIfEqual(attempts, 0); + if (!Enum.IsDefined(AckMode)) + throw new ArgumentOutOfRangeException(nameof(AckMode)); + if (this is MessageConsumerOptions { Destination: { } destination }) + ArgumentException.ThrowIfNullOrWhiteSpace(destination); + if (this is MessageSubscriptionOptions { Topic: { } topic }) + ArgumentException.ThrowIfNullOrWhiteSpace(topic); + } + + /// Dead-letters this exception type without retrying. Multiple calls compose. + public void DeadLetterOn() where TException : Exception + { + var existing = DeadLetterWhen; + DeadLetterWhen = existing is null ? static ex => ex is TException : ex => existing(ex) || ex is TException; + } +} + +/// Options for a competing consumer of queued work. +public sealed class MessageConsumerOptions : MessageHandlerOptions +{ + /// Queue name. Null uses the message type's configured route. + public string? Destination { get; set; } +} + +/// Options for receiving published events. +public sealed class MessageSubscriptionOptions : MessageHandlerOptions +{ + /// Topic name. Null uses the message type's configured route. + public string? Topic { get; set; } + + /// + /// Stable durable subscription name. Replicas using the same name compete for that subscription's events. + /// Null creates a temporary subscription with a renewable expiration lease; disposal removes its backlog. + /// + public string? Subscription { get; set; } + + internal MessageSubscriptionOptions Copy() => (MessageSubscriptionOptions)MemberwiseClone(); +} + +/// The observable state of a supervised listener. +public enum MessageSubscriptionStatus { Starting, Healthy, Recovering, Stopped } + +/// A running consumer. Disposal stops receiving and releases the listener's resources. +public interface IMessageSubscription : IAsyncDisposable +{ + /// The queue or topic subscription this consumer receives from. + DestinationAddress Source { get; } + /// Recovering listeners must not be reported as healthy. + MessageSubscriptionStatus Status { get; } + /// Increases after a possible delivery gap. Derived local state must be resynchronized. + long RecoveryVersion { get; } + /// Waits until receiving resumes; cancellation stops only the wait. + Task WaitUntilReadyAsync(CancellationToken cancellationToken = default); +} + +/// A batch payload with a stable application ID for selective retry. +public sealed record MessageBatchItem(T Message, string? MessageId = null, MessageHeaders? Headers = null) : IMessageBatchItem where T : class +{ + object IMessageBatchItem.Value => Message; +} + +internal interface IMessageBatchItem +{ + object Value { get; } + string? MessageId { get; } + MessageHeaders? Headers { get; } +} + +public interface IMessageBus : IAsyncDisposable +{ + /// Consumes work with an explicit outcome; core applies the endpoint retry and settlement policy. + Task ConsumeWithOutcomeAsync(Func, CancellationToken, ValueTask> handler, MessageConsumerOptions? options = null, CancellationToken cancellationToken = default) where T : class + => throw new NotSupportedException("This message bus does not support outcome handlers."); + + /// Consumes raw deliveries with an explicit outcome and core-owned retry and settlement. + Task ConsumeWithOutcomeAsync(Func> handler, MessageConsumerOptions options, CancellationToken cancellationToken = default) + => throw new NotSupportedException("This message bus does not support outcome handlers."); + /// Whether per-instance, automatically expiring event subscriptions are available. + bool SupportsTemporarySubscriptions => false; + /// Receives queued work directly. Dispose an unsettled delivery to return it for redelivery. + Task?> ReceiveAsync(MessageReceiveOptions? options = null, CancellationToken cancellationToken = default) where T : class; + + /// Receives a raw message from an explicit queue without deserializing its body. + Task ReceiveAsync(MessageReceiveOptions options, CancellationToken cancellationToken = default); + + /// Receives one best-effort copy on this node, acknowledging before callbacks. Independent nodes do not compete. + Task SubscribeNodeAsync(Func handler, MessageNodeSubscriptionOptions options, CancellationToken cancellationToken = default) + => throw new NotSupportedException("This bus does not support node subscriptions."); + + /// Enqueues work for a competing consumer. Returns its application message ID. + Task SendAsync(T message, MessageSendOptions? options = null, CancellationToken cancellationToken = default) where T : class; + + /// Enqueues work in input order. Batches are not atomic. + Task> SendBatchAsync(IEnumerable messages, MessageSendOptions? options = null, CancellationToken cancellationToken = default) where T : class; + /// Sends per-input application IDs and headers, preserving outcome order. + Task> SendBatchAsync(IEnumerable> messages, MessageSendOptions? options = null, CancellationToken cancellationToken = default) where T : class; + Task> SendBatchAsync(IEnumerable messages, MessageSendOptions? options = null, CancellationToken cancellationToken = default); + + /// Publishes to existing subscriptions. Events without subscriptions are dropped. + Task PublishAsync(T message, MessagePublishOptions? options = null, CancellationToken cancellationToken = default) where T : class; + + /// Publishes events and returns their IDs in input order. Batches are not atomic. + Task> PublishBatchAsync(IEnumerable messages, MessagePublishOptions? options = null, CancellationToken cancellationToken = default) where T : class; + /// Publishes per-input application IDs and headers, preserving outcome order. + Task> PublishBatchAsync(IEnumerable> messages, MessagePublishOptions? options = null, CancellationToken cancellationToken = default) where T : class; + Task> PublishBatchAsync(IEnumerable messages, MessagePublishOptions? options = null, CancellationToken cancellationToken = default); + + /// Consumes queued work. Only one handler per message type may be registered on an endpoint in this bus. + Task ConsumeAsync(Func, CancellationToken, Task> handler, MessageConsumerOptions? options = null, CancellationToken cancellationToken = default) where T : class; + Task ConsumeAsync(Func handler, MessageConsumerOptions options, CancellationToken cancellationToken = default); + + /// Receives published events. An unnamed subscription is temporary; a named subscription is durable. + Task SubscribeAsync(Func, CancellationToken, Task> handler, MessageSubscriptionOptions? options = null, CancellationToken cancellationToken = default) where T : class; + Task SubscribeAsync(Func handler, MessageSubscriptionOptions options, CancellationToken cancellationToken = default); +} + +/// +/// Governs the messaging client's topology-administration behavior. Publishing and subscribing never implicitly grant +/// themselves more than this mode allows, so an app on a locked-down broker can state "validate only" or "never touch +/// topology" instead of hoping implicit creation fails gracefully. +/// +public enum TopologyMode +{ + /// Create missing destinations on first use and at handler-host startup (default). + Ensure, + + /// Never create. Verify each destination exists on first use (cached) and throw when missing. + Validate, + + /// No topology calls at all; destinations are assumed pre-provisioned out of band. + None +} + +public sealed record MessageBusOptions +{ + /// How the client administers topology (create on use, validate-only, or never touch). Default . + public TopologyMode Topology { get; init; } = TopologyMode.Ensure; + + public ISerializer Serializer { get; init; } = DefaultSerializer.Instance; + /// Media type produced by the serializer. Defaults to JSON for SystemTextJsonSerializer, otherwise byte-safe application/octet-stream. + public string? ContentType { get; init; } + public IMessageRouter Router { get; init; } = DefaultMessageRouter.Instance; + public IMessageTypeRegistry MessageTypes { get; init; } = new MessageTypeRegistry(); + /// + /// Stores delayed sends and retries beyond native transport limits. Start a ScheduledMessageDispatcher + /// explicitly (AddScheduledMessageDispatcher in hosted apps). Messaging needs only IScheduledDispatchStore; + /// an IJobRuntimeStore can also supply this contract. Registering storage starts no background services. + /// + public IScheduledDispatchStore? RuntimeStore { get; init; } + public RetryPolicy RetryPolicy { get; init; } = new(); + + /// + /// Whether disposing this bus also disposes the transport. True (default) for a transport the bus solely uses; set + /// false when the transport is a shared/externally-owned instance (e.g. a DI singleton). + /// + public bool OwnsTransport { get; init; } = true; + public TimeProvider TimeProvider { get; init; } = TimeProvider.System; + public ILoggerFactory? LoggerFactory { get; init; } +} + +/// +/// The one messaging client over the transport. Routing, serialization, settlement, scheduling, and the consumer loop +/// live in ; this type maps the two delivery verbs and subscriptions onto that core. +/// +public sealed class MessageBus : IMessageBus +{ + private readonly MessageClientCore _core; + private readonly IMessageTransport _transport; + private readonly ILogger _logger; + + public MessageBus(IMessageTransport transport, MessageBusOptions? options = null) + { + ArgumentNullException.ThrowIfNull(transport); + _transport = transport; + options ??= new MessageBusOptions(); + _logger = (options.LoggerFactory ?? NullLoggerFactory.Instance).CreateLogger(); + _core = new MessageClientCore(transport, options.Serializer, options.Router, options.RuntimeStore, options.TimeProvider, _logger, + static (message, inner) => inner is null ? new MessageBusException(message) : new MessageBusException(message, inner), options.RetryPolicy, options.OwnsTransport, options.MessageTypes, options.ContentType, options.Topology); + } + + /// + public async Task SubscribeNodeAsync(Func handler, MessageNodeSubscriptionOptions options, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(handler); + ArgumentNullException.ThrowIfNull(options); + options.Validate(); + IManagedNodeSubscription? owner = null; + if (_transport is ISupportsManagedNodeSubscriptions managed) + owner = await managed.OpenNodeSubscriptionAsync(options, cancellationToken).ConfigureAwait(false); + else if (_transport is not ISupportsEphemeralSubscriptions) + throw new NotSupportedException("The transport supports neither expiring nor managed node subscriptions."); + try + { + var consumer = await SubscribeAsync(async (message, token) => + { + await message.CompleteAsync(token).ConfigureAwait(false); + await handler(message, token).ConfigureAwait(false); + }, new MessageSubscriptionOptions + { + Topic = options.Topic, + Subscription = owner?.Source.Name, + MaxConcurrency = options.MaxConcurrency, + AckMode = AckMode.Manual + }, cancellationToken).ConfigureAwait(false); + return owner is null ? consumer : new NodeMessageSubscription(consumer, owner); + } + catch + { + if (owner is not null) await owner.DisposeAsync().ConfigureAwait(false); + throw; + } + } + + public Task SendAsync(T message, MessageSendOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(message); + options ??= new MessageSendOptions(); + return _core.SendAsync(ScheduledDispatchKind.QueueMessage, typeof(T), message, ToEnvelope(options), GetDestination(typeof(T), options.Destination), EnsureDestinationAsync, cancellationToken); + } + + public Task?> ReceiveAsync(MessageReceiveOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + options ??= new MessageReceiveOptions(); + return _core.ReceiveAsync(GetDestination(typeof(T), options.Destination), options.WaitTime, cancellationToken); + } + + public Task ReceiveAsync(MessageReceiveOptions options, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(options); + ArgumentException.ThrowIfNullOrWhiteSpace(options.Destination); + return _core.ReceiveAsync(DestinationAddress.ForQueue(options.Destination), options.WaitTime, cancellationToken); + } + + public Task> SendBatchAsync(IEnumerable messages, MessageSendOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(messages); + options ??= new MessageSendOptions(); + return _core.SendBatchAsync(ScheduledDispatchKind.QueueMessage, messages.Cast(), typeof(T), ToEnvelope(options), type => GetDestination(type, options.Destination), EnsureDestinationAsync, cancellationToken); + } + + public Task> SendBatchAsync(IEnumerable> messages, MessageSendOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(messages); + options ??= new MessageSendOptions(); + return _core.SendBatchAsync(ScheduledDispatchKind.QueueMessage, messages.Cast(), typeof(T), ToEnvelope(options), type => GetDestination(type, options.Destination), EnsureDestinationAsync, cancellationToken); + } + + public Task> SendBatchAsync(IEnumerable messages, MessageSendOptions? options = null, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(messages); + options ??= new MessageSendOptions(); + return _core.SendBatchAsync(ScheduledDispatchKind.QueueMessage, messages, null, ToEnvelope(options), type => GetDestination(type, options.Destination), EnsureDestinationAsync, cancellationToken); + } + + public Task PublishAsync(T message, MessagePublishOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(message); + options ??= new MessagePublishOptions(); + return _core.SendAsync(ScheduledDispatchKind.PubSubMessage, typeof(T), message, ToEnvelope(options), GetTopic(typeof(T), options.Topic), EnsureDestinationAsync, cancellationToken); + } + + public Task> PublishBatchAsync(IEnumerable messages, MessagePublishOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(messages); + options ??= new MessagePublishOptions(); + return _core.SendBatchAsync(ScheduledDispatchKind.PubSubMessage, messages.Cast(), typeof(T), ToEnvelope(options), type => GetTopic(type, options.Topic), EnsureDestinationAsync, cancellationToken); + } + + public Task> PublishBatchAsync(IEnumerable> messages, MessagePublishOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(messages); + options ??= new MessagePublishOptions(); + return _core.SendBatchAsync(ScheduledDispatchKind.PubSubMessage, messages.Cast(), typeof(T), ToEnvelope(options), type => GetTopic(type, options.Topic), EnsureDestinationAsync, cancellationToken); + } + + public Task> PublishBatchAsync(IEnumerable messages, MessagePublishOptions? options = null, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(messages); + options ??= new MessagePublishOptions(); + return _core.SendBatchAsync(ScheduledDispatchKind.PubSubMessage, messages, null, ToEnvelope(options), type => GetTopic(type, options.Topic), EnsureDestinationAsync, cancellationToken); + } + + public bool SupportsTemporarySubscriptions => _core.SupportsTemporarySubscriptions; + + public Task ConsumeWithOutcomeAsync(Func, CancellationToken, ValueTask> handler, MessageConsumerOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(handler); + return ConsumeCoreAsync(options ?? new(), typeof(T), (config, token) => _core.StartOutcomeListenerAsync(config, handler, token), cancellationToken); + } + + public Task ConsumeWithOutcomeAsync(Func> handler, MessageConsumerOptions options, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(handler); + ArgumentNullException.ThrowIfNull(options); + ArgumentException.ThrowIfNullOrWhiteSpace(options.Destination); + return ConsumeCoreAsync(options, typeof(object), (config, token) => _core.StartOutcomeListenerAsync(config, handler, token), cancellationToken); + } + + public Task ConsumeAsync(Func, CancellationToken, Task> handler, MessageConsumerOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(handler); + return ConsumeCoreAsync(options ?? new(), typeof(T), (config, token) => _core.StartListenerAsync(config, handler, token), cancellationToken); + } + + public Task ConsumeAsync(Func handler, MessageConsumerOptions options, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(handler); + ArgumentNullException.ThrowIfNull(options); + ArgumentException.ThrowIfNullOrWhiteSpace(options.Destination); + return ConsumeCoreAsync(options, typeof(object), (config, token) => _core.StartListenerAsync(config, handler, token), cancellationToken); + } + + public Task SubscribeAsync(Func, CancellationToken, Task> handler, MessageSubscriptionOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(handler); + return SubscribeCoreAsync(options ?? new(), typeof(T), (config, token) => _core.StartListenerAsync(config, handler, token), cancellationToken); + } + + public Task SubscribeAsync(Func handler, MessageSubscriptionOptions options, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(handler); + ArgumentNullException.ThrowIfNull(options); + ArgumentException.ThrowIfNullOrWhiteSpace(options.Topic); + return SubscribeCoreAsync(options, typeof(object), (config, token) => _core.StartListenerAsync(config, handler, token), cancellationToken); + } + + private async Task ConsumeCoreAsync(MessageConsumerOptions options, Type messageType, Func> start, CancellationToken cancellationToken) + { + RequireRole(DestinationRole.Queue); + var source = GetDestination(messageType, options.Destination); + var config = CreateListener(options, messageType, source); + await EnsureDestinationAsync(source, cancellationToken).AnyContext(); + return await start(config, cancellationToken).AnyContext(); + } + + private async Task SubscribeCoreAsync(MessageSubscriptionOptions options, Type messageType, Func> start, CancellationToken cancellationToken) + { + RequireRole(DestinationRole.Topic); + RequireRole(DestinationRole.Subscription); + var topic = GetTopic(messageType, options.Topic); + bool ephemeral = options.Subscription is null; + if (!ephemeral) + ArgumentException.ThrowIfNullOrWhiteSpace(options.Subscription); + string subscription = options.Subscription ?? $"temporary-{Guid.NewGuid():N}"; + var source = DestinationAddress.ForSubscription(topic.Name, subscription); + var config = CreateListener(options, messageType, source) with { Ephemeral = ephemeral }; + if (ephemeral) + _core.RequireEphemeralSubscriptions(); + await _core.EnsureAsync([ + new DestinationDeclaration { Address = topic }, + new DestinationDeclaration { Address = source, AutoDeleteAfter = ephemeral ? TimeSpan.FromMinutes(2) : null } + ], cancellationToken).AnyContext(); + return await start(config, cancellationToken).AnyContext(); + } + + private static ListenerConfig CreateListener(MessageHandlerOptions options, Type messageType, DestinationAddress source) + { + options.Validate(); + return new ListenerConfig + { + Source = source, + Key = messageType.FullName ?? messageType.Name, + MessageType = messageType, + AckMode = options.AckMode, + MaxConcurrency = options.MaxConcurrency, + PrefetchCount = options.PrefetchCount, + ReceiveBatchDelay = options.ReceiveBatchDelay, + VisibilityTimeout = options.VisibilityTimeout, + AutoRenewLock = options.AutoRenewLock, + ShutdownTimeout = options.ShutdownTimeout, + WaitForManualSettlement = options.WaitForManualSettlement, + MaxAttempts = options.MaxAttempts, + RedeliveryBackoff = options.RedeliveryBackoff, + DeadLetterWhen = options.DeadLetterWhen + }; + } + + private void RequireRole(DestinationRole role) + { + if (!_core.SupportsRole(role)) + throw new NotSupportedException($"The transport does not support {role} destinations."); + } + + public ValueTask DisposeAsync() => _core.DisposeAsync(); + + private Task EnsureDestinationAsync(DestinationAddress destination, CancellationToken cancellationToken) + { + return _core.EnsureAsync([new DestinationDeclaration { Address = destination }], cancellationToken); + } + + private DestinationAddress GetDestination(Type messageType, string? destination) + { + return DestinationAddress.ForQueue(_core.Router.ResolveRoute(new MessageRouteContext + { + MessageType = messageType, + Role = MessageRouteRole.QueueDestination, + OperationOverride = destination + })); + } + + private DestinationAddress GetTopic(Type messageType, string? topic) + { + return DestinationAddress.ForTopic(_core.Router.ResolveRoute(new MessageRouteContext + { + MessageType = messageType, + Role = MessageRouteRole.PubSubTopic, + OperationOverride = topic + })); + } + + private static MessageEnvelopeOptions ToEnvelope(MessageSendOptions options) + { + return new MessageEnvelopeOptions + { + MessageId = options.MessageId, + Priority = options.Priority, + Delay = options.Delay, + DeliverAt = options.DeliverAt, + TimeToLive = options.TimeToLive, + CorrelationId = options.CorrelationId, + Headers = options.Headers + }; + } + + private static MessageEnvelopeOptions ToEnvelope(MessagePublishOptions options) + { + return new MessageEnvelopeOptions + { + MessageId = options.MessageId, + Priority = options.Priority, + Delay = options.Delay, + DeliverAt = options.DeliverAt, + TimeToLive = options.TimeToLive, + CorrelationId = options.CorrelationId, + Headers = options.Headers + }; + } + +} diff --git a/src/Foundatio/Messaging/MessageBusBase.cs b/src/Foundatio/Messaging/MessageBusBase.cs deleted file mode 100644 index aff97ce4b..000000000 --- a/src/Foundatio/Messaging/MessageBusBase.cs +++ /dev/null @@ -1,678 +0,0 @@ -using System; -using System.Collections.Concurrent; -using System.Collections.Generic; -using System.Diagnostics; -using System.Linq; -using System.Reflection; -using System.Runtime.InteropServices; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Resilience; -using Foundatio.Serializer; -using Foundatio.Utility; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Logging.Abstractions; - -namespace Foundatio.Messaging; - -public abstract class MessageBusBase : IMessageBus, IHaveLogger, IHaveLoggerFactory, IHaveTimeProvider, IHaveResiliencePolicyProvider, IDisposable, IAsyncDisposable where TOptions : SharedMessageBusOptions -{ - protected readonly ConcurrentDictionary _subscribers = new(); - protected readonly TOptions _options; - protected readonly ILogger _logger; - protected readonly ILoggerFactory _loggerFactory; - protected readonly TimeProvider _timeProvider; - protected readonly IResiliencePolicyProvider _resiliencePolicyProvider; - protected readonly IResiliencePolicy _resiliencePolicy; - protected readonly ISerializer _serializer; - private readonly CancellationTokenSource _disposedCancellationTokenSource = new(); - private int _disposeState; - protected bool IsDisposed => Volatile.Read(ref _disposeState) != 0; - - /// - /// Signals that this instance is being disposed by setting . - /// Unlike , does not cancel the token immediately - /// because cancels after shutdown completes. - /// - /// true if this is the first caller; false if already signaled. - protected bool SignalDispose() - { - return Interlocked.CompareExchange(ref _disposeState, 1, 0) == 0; - } - - public MessageBusBase(TOptions options) - { - ArgumentNullException.ThrowIfNull(options); - - _options = options; - _loggerFactory = options.LoggerFactory ?? NullLoggerFactory.Instance; - _logger = _loggerFactory.CreateLogger(GetType()); - _timeProvider = options.TimeProvider ?? TimeProvider.System; - - _resiliencePolicyProvider = options.ResiliencePolicyProvider; - _resiliencePolicy = _resiliencePolicyProvider.GetPolicy, IMessageBus>( - builder => builder.WithUnhandledException(), - _logger, _timeProvider); - - _serializer = options.Serializer ?? DefaultSerializer.Instance; - MessageBusId = _options.Topic + Guid.NewGuid().ToString("N").Substring(10); - } - - /// - /// Gets a cancellation token that is canceled when this instance is disposed. - /// Use this token to cancel background operations during shutdown. - /// - protected CancellationToken DisposedCancellationToken => _disposedCancellationTokenSource.Token; - - /// - /// Creates a linked cancellation token source that combines the provided token with the disposal token. - /// This allows operations to be cancelled by either the caller or when this instance is disposed. - /// - /// The caller's cancellation token to link. - /// A new that should be disposed by the caller. - protected CancellationTokenSource GetLinkedDisposableCancellationTokenSource(CancellationToken cancellationToken) - { - return CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, DisposedCancellationToken); - } - - ILogger IHaveLogger.Logger => _logger; - ILoggerFactory IHaveLoggerFactory.LoggerFactory => _loggerFactory; - TimeProvider IHaveTimeProvider.TimeProvider => _timeProvider; - IResiliencePolicyProvider IHaveResiliencePolicyProvider.ResiliencePolicyProvider => _resiliencePolicyProvider; - - /// - /// Called before publishing to ensure the topic exists. The - /// is always ; topic creation should only - /// abort when the message bus is being disposed, never due to an individual caller's cancellation. - /// - protected virtual Task EnsureTopicCreatedAsync(CancellationToken cancellationToken) => Task.CompletedTask; - - protected abstract Task PublishImplAsync(string messageType, object message, MessageOptions options, CancellationToken cancellationToken); - - public async Task PublishAsync(Type messageType, object message, MessageOptions? options = null, CancellationToken cancellationToken = default) - { - ArgumentNullException.ThrowIfNull(messageType); - ArgumentNullException.ThrowIfNull(message); - cancellationToken.ThrowIfCancellationRequested(); - if (IsDisposed) - throw new MessageBusException($"Cannot publish: message bus has been disposed (MessageBusId: {MessageBusId})."); - - options ??= new MessageOptions(); - - if (String.IsNullOrEmpty(options.CorrelationId)) - { - options.CorrelationId = Activity.Current?.Id; - if (!String.IsNullOrEmpty(Activity.Current?.TraceStateString)) - options.Properties.Add("TraceState", Activity.Current.TraceStateString); - } - - try - { - // Use DisposedCancellationToken for setup: topic creation should only abort on disposal, - // not due to an individual caller's cancellation token. - await EnsureTopicCreatedAsync(DisposedCancellationToken).AnyContext(); - - using var linkedCancellationTokenSource = GetLinkedDisposableCancellationTokenSource(cancellationToken); - await PublishImplAsync(GetMappedMessageType(messageType), message, options, linkedCancellationTokenSource.Token).AnyContext(); - } - catch (Exception ex) when (ex is not OperationCanceledException and not MessageBusException) - { - throw new MessageBusException($"Error publishing {messageType.Name}: {ex.Message}", ex); - } - } - - private readonly ConcurrentDictionary _mappedMessageTypesCache = new(); - protected string GetMappedMessageType(Type messageType) - { - return _mappedMessageTypesCache.GetOrAdd(messageType, type => - { - var reversedMap = _options.MessageTypeMappings.ToDictionary(kvp => kvp.Value, kvp => kvp.Key); - if (reversedMap.ContainsKey(type)) - return reversedMap[type]; - - return String.Concat(messageType.FullName, ", ", messageType.Assembly.GetName().Name); - }); - } - - private readonly ConcurrentDictionary _knownMessageTypesCache = new(); - protected virtual Type? GetMappedMessageType(string? messageType) - { - if (String.IsNullOrEmpty(messageType)) - return null; - - if (_knownMessageTypesCache.TryGetValue(messageType, out var cachedType)) - return cachedType; - - Type? resolvedType = null; - - if (_options.MessageTypeMappings.TryGetValue(messageType, out Type? typeMapping)) - { - if (typeMapping is not null) - resolvedType = typeMapping; - else - _logger.LogWarning("Message type mapping for {MessageType} resolved to null; falling back to Type.GetType", messageType); - } - - if (resolvedType is null) - { - try - { - resolvedType = Type.GetType(messageType); - } - catch (Exception) - { - try - { - // try resolve type without version - string[] typeParts = messageType.Split(','); - string shortType = typeParts.Length >= 2 - ? String.Join(",", typeParts[0], typeParts[1]) - : messageType; - - resolvedType = Type.GetType(shortType); - } - catch (Exception ex) - { - _logger.LogError(ex, "Error getting message body type: {MessageType}", messageType); - } - } - } - - if (resolvedType is not null) - _knownMessageTypesCache.TryAdd(messageType, resolvedType); - - return resolvedType; - } - - /// - /// Called during the first phase of disposal, before - /// is cancelled and before subscribers are cleared. - /// - /// - /// - /// Override this method to gracefully drain in-flight work (stop processors, close consumer - /// groups, flush buffers). Subscribers are still registered and the cancellation token is - /// still active, so handlers can finish processing normally. - /// - /// - /// The base implementation delegates to so that - /// existing provider overrides continue to work without changes. - /// - /// - protected virtual Task ShutdownAsync() => RemoveTopicSubscriptionAsync(); - - /// - /// Called during the second phase of disposal, after - /// is cancelled and after subscribers are cleared. - /// - /// - /// Override this method to tear down transport infrastructure — close connections, - /// dispose clients, await background listener tasks. No subscribers remain at this point - /// and the cancellation token has been signaled, so background loops should have exited. - /// - protected virtual Task CleanupAsync() => Task.CompletedTask; - - /// - /// Called during the first phase of disposal to remove transport-level topic subscriptions. - /// New providers should prefer overriding instead, which calls - /// this method by default. - /// - protected virtual Task RemoveTopicSubscriptionAsync() => Task.CompletedTask; - - /// - /// Called after subscribing to ensure the topic subscription infrastructure exists. The - /// is always ; - /// subscription setup should only abort when the message bus is being disposed, never due to - /// an individual caller's cancellation. - /// - protected virtual Task EnsureTopicSubscriptionAsync(CancellationToken cancellationToken) => Task.CompletedTask; - - protected virtual Task SubscribeImplAsync(Func handler, CancellationToken cancellationToken) where T : class - { - var subscriber = new Subscriber - { - CancellationToken = cancellationToken, - Type = typeof(T), - Action = (message, token) => - { - if (message is T typedMessage) - return handler(typedMessage, token); - - if (message is null) - { - _logger.LogWarning("Subscriber action skipped: message body is null (likely a deserialization failure) for subscriber type {SubscriberType}", typeof(T)); - return Task.CompletedTask; - } - - _logger.LogTrace("Unable to call subscriber action: {MessageType} cannot be safely casted to {SubscriberType}", message.GetType(), typeof(T)); - return Task.CompletedTask; - } - }; - - if (cancellationToken != CancellationToken.None) - { - // CancellationToken.Register only accepts synchronous callbacks, so we cannot safely - // call RemoveTopicSubscriptionAsync here. The async-capable CancelAsync was added in - // .NET 8 but still does not support async callbacks — see: - // https://github.com/dotnet/runtime/issues/31315 - // Topic subscription teardown is handled during DisposeAsync via ShutdownAsync/CleanupAsync. - cancellationToken.Register(() => - { - _subscribers.TryRemove(subscriber.Id, out _); - }); - } - - if (subscriber.Type.Name == "IMessage`1" && subscriber.Type.GenericTypeArguments.Length == 1) - { - var modelType = subscriber.Type.GenericTypeArguments.Single(); - subscriber.GenericType = typeof(Message<>).MakeGenericType(modelType); - } - - if (!_subscribers.TryAdd(subscriber.Id, subscriber)) - _logger.LogError("Unable to add subscriber {SubscriberId}", subscriber.Id); - - return Task.CompletedTask; - } - - public async Task SubscribeAsync(Func handler, CancellationToken cancellationToken = default) where T : class - { - cancellationToken.ThrowIfCancellationRequested(); - if (IsDisposed) - throw new MessageBusException($"Cannot subscribe: message bus has been disposed (MessageBusId: {MessageBusId})."); - _logger.LogTrace("Adding subscriber for {MessageType}", typeof(T).FullName); - - await SubscribeImplAsync(handler, cancellationToken).AnyContext(); - // Use DisposedCancellationToken for setup: subscription infrastructure should only abort on disposal, - // not due to the caller's cancellation token. - await EnsureTopicSubscriptionAsync(DisposedCancellationToken).AnyContext(); - } - - protected List GetMessageSubscribers(IMessage message) - { - return _subscribers.Values.Where(s => SubscriberHandlesMessage(s, message)).ToList(); - } - - protected virtual bool SubscriberHandlesMessage(Subscriber subscriber, IMessage message) - { - if (subscriber.Type == typeof(IMessage)) - return true; - - var clrType = message.ClrType ?? GetMappedMessageType(message.Type); - if (clrType is null) - { - _logger.LogWarning("Unable to resolve CLR type for message body type: ClrType={MessageClrType} Type={MessageType}", message.ClrType, message.Type); - return false; - } - - if (subscriber.IsAssignableFrom(clrType)) - return true; - - return false; - } - - protected virtual byte[] SerializeMessageBody(string messageType, object body) - { - if (body is null) - return []; - - return _serializer.SerializeToBytes(body); - } - - protected virtual object? DeserializeMessageBody(IMessage message) - { - if (message.Data.IsEmpty) - return null; - - object? body; - try - { - var clrType = message.ClrType ?? GetMappedMessageType(message.Type); - body = clrType != null ? _serializer.Deserialize(message.Data, clrType) : GetRawBody(message.Data); - } - catch (Exception ex) - { - _logger.LogError(ex, "Error deserializing message body: {Message}", ex.Message); - return null; - } - - return body; - } - - /// - /// Returns the raw payload as a array for subscribers that consume the body - /// without a mapped CLR type (e.g. Subscribe<byte[]>()). - /// - /// - /// When the payload already wraps a full-length managed array (offset 0, count equal to the array - /// length) the underlying array is returned directly, preserving the no-copy behavior that existed - /// before became a . Otherwise the memory - /// is copied to honor the byte[] contract. - /// - private static byte[] GetRawBody(ReadOnlyMemory data) - { - if (MemoryMarshal.TryGetArray(data, out ArraySegment segment) && segment.Array is not null - && segment.Offset == 0 && segment.Count == segment.Array.Length) - { - return segment.Array; - } - - return data.ToArray(); - } - - protected async Task SendMessageToSubscribersAsync(IMessage message) - { - if (IsDisposed) - { - _logger.LogTrace("Message bus {MessageBusId} is disposed, skipping message delivery for type {MessageType}", MessageBusId, message.Type); - return; - } - - var subscribers = GetMessageSubscribers(message); - - _logger.LogTrace("Found {SubscriberCount} subscribers for message type: ClrType={MessageClrType} Type={MessageType}", subscribers.Count, message.ClrType, message.Type); - - if (subscribers.Count == 0) - return; - - var subscriberHandlers = subscribers.Select(subscriber => - { - if (subscriber.CancellationToken.IsCancellationRequested) - { - if (_subscribers.TryRemove(subscriber.Id, out _)) - { - _logger.LogTrace("Removed cancelled subscriber: {SubscriberId}", subscriber.Id); - } - else - { - _logger.LogTrace("Unable to remove cancelled subscriber: {SubscriberId}", subscriber.Id); - } - - return Task.CompletedTask; - } - - return Task.Run(async () => - { - if (DisposedCancellationToken.IsCancellationRequested || subscriber.CancellationToken.IsCancellationRequested) - { - _logger.LogTrace("The cancelled subscriber action will not be called: {SubscriberId}", subscriber.Id); - return; - } - - _logger.LogTrace("Calling subscriber action: {SubscriberId}", subscriber.Id); - using var activity = StartHandleMessageActivity(message); - - try - { - using (_logger.BeginScope(s => s - .PropertyIf("UniqueId", message.UniqueId, !String.IsNullOrEmpty(message.UniqueId)) - .PropertyIf("CorrelationId", message.CorrelationId, !String.IsNullOrEmpty(message.CorrelationId)))) - { - if (subscriber.Type == typeof(IMessage)) - { - await subscriber.Action(message, subscriber.CancellationToken).AnyContext(); - } - else if (subscriber.GenericType is not null) - { - object? typedMessage = Activator.CreateInstance(subscriber.GenericType, message); - if (typedMessage is null) - { - _logger.LogError("Skipping subscriber {SubscriberId}: failed to create typed message wrapper for type {MessageType}", subscriber.Id, message.Type); - return; - } - - await subscriber.Action(typedMessage, subscriber.CancellationToken).AnyContext(); - } - else - { - object? body = message.GetBody(); - if (body is null) - { - _logger.LogWarning("Skipping subscriber {SubscriberId}: message body deserialization returned null for type {MessageType}", subscriber.Id, message.Type); - return; - } - - await subscriber.Action(body, subscriber.CancellationToken).AnyContext(); - } - } - - _logger.LogTrace("Finished calling subscriber action: {SubscriberId}", subscriber.Id); - } - catch (Exception ex) - { - activity?.SetErrorStatus(ex); - throw; - } - }, DisposedCancellationToken); - }); - - try - { - await Task.WhenAll(subscriberHandlers.ToArray()); - } - catch (Exception ex) - { - _logger.LogError(ex, "Error sending message to subscribers: {Message}", ex.Message); - throw new MessageBusException($"Error sending message to subscribers: {ex.Message}", ex); - } - - _logger.LogTrace("Done enqueueing message to {SubscriberCount} subscribers for message type {MessageType}", subscribers.Count, message.Type); - } - - protected virtual Activity? StartHandleMessageActivity(IMessage message) - { - var activity = FoundatioDiagnostics.ActivitySource.StartActivity("HandleMessage", ActivityKind.Internal, message.CorrelationId); - if (activity is null) - return null; - - if (message.Properties is not null && message.Properties.TryGetValue("TraceState", out string? traceState)) - activity.TraceStateString = traceState; - - activity.DisplayName = $"Message: {message.ClrType?.Name ?? message.Type}"; - - EnrichHandleMessageActivity(activity, message); - return activity; - } - - protected virtual void EnrichHandleMessageActivity(Activity activity, IMessage message) - { - if (!activity.IsAllDataRequested) - return; - - activity.AddTag("MessageType", message.Type); - activity.AddTag("ClrType", message.ClrType?.FullName); - activity.AddTag("UniqueId", message.UniqueId); - activity.AddTag("CorrelationId", message.CorrelationId); - - if (message.Properties is not { Count: > 0 }) - return; - - foreach (var p in message.Properties) - { - if (p.Key != "TraceState") - activity.AddTag(p.Key, p.Value); - } - } - - /// - /// Schedules a message for delayed delivery using an in-memory timer. - /// - /// - /// This method calls (not ) to ensure - /// topic infrastructure is re-established if needed (e.g., after reconnection for external providers). - /// The is cleared via record copy to prevent infinite recursion. - /// The Properties dictionary is cloned to avoid shared mutable state between the caller and delayed delivery. - /// - protected void SendDelayedMessage(Type messageType, object message, MessageOptions options) - { - ArgumentNullException.ThrowIfNull(messageType); - ArgumentNullException.ThrowIfNull(message); - ArgumentNullException.ThrowIfNull(options); - - var delay = options.DeliveryDelay.GetValueOrDefault(); - ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(delay, TimeSpan.Zero); - - // Clone options to capture current state and avoid shared mutable state - var clonedOptions = options with - { - DeliveryDelay = null, // Clear to prevent infinite recursion - Properties = new Dictionary(options.Properties) - }; - - var sendTime = _timeProvider.GetUtcNow().UtcDateTime.SafeAdd(delay); - Task.Factory.StartNew(async () => - { - await _timeProvider.SafeDelay(delay, _disposedCancellationTokenSource.Token).AnyContext(); - if (_disposedCancellationTokenSource.IsCancellationRequested) - { - _logger.LogTrace("Discarding delayed message scheduled for {SendTime:O} for type {MessageType}", sendTime, messageType); - return; - } - - _logger.LogTrace("Sending delayed message scheduled for {SendTime:O} for type {MessageType}", sendTime, messageType); - - try - { - await _resiliencePolicy.ExecuteAsync(async ct => - { - await PublishAsync(messageType, message, clonedOptions, ct).AnyContext(); - }, _disposedCancellationTokenSource.Token).AnyContext(); - } - catch (Exception ex) - { - _logger.LogError(ex, "Failed to publish delayed message for type {MessageType}: {Message}", messageType, ex.Message); - } - }, _disposedCancellationTokenSource.Token); - } - - public string MessageBusId { get; init; } - - public virtual async ValueTask DisposeAsync() - { - if (!SignalDispose()) - { - _logger.LogTrace("MessageBus {MessageBusId} async dispose was already called", MessageBusId); - return; - } - - _logger.LogTrace("MessageBus {MessageBusId} async dispose", MessageBusId); - - try - { - await ShutdownAsync().AnyContext(); - } - catch (OperationCanceledException ex) - { - _logger.LogTrace(ex, "Shutdown cancelled for {MessageBusId}", MessageBusId); - } - catch (ObjectDisposedException ex) - { - _logger.LogDebug(ex, "Resource already disposed during shutdown for {MessageBusId}", MessageBusId); - } - catch (Exception ex) - { - _logger.LogError(ex, "Error during shutdown for {MessageBusId}: {Message}", MessageBusId, ex.Message); - } - - _subscribers?.Clear(); - _disposedCancellationTokenSource.Cancel(); - - try - { - await CleanupAsync().AnyContext(); - } - catch (OperationCanceledException ex) - { - _logger.LogTrace(ex, "Cleanup cancelled for {MessageBusId}", MessageBusId); - } - catch (ObjectDisposedException ex) - { - _logger.LogDebug(ex, "Resource already disposed during cleanup for {MessageBusId}", MessageBusId); - } - catch (Exception ex) - { - _logger.LogError(ex, "Error during cleanup for {MessageBusId}: {Message}", MessageBusId, ex.Message); - } - - _disposedCancellationTokenSource.Dispose(); - } - - public virtual void Dispose() - { - if (!SignalDispose()) - { - _logger.LogTrace("MessageBus {MessageBusId} dispose was already called", MessageBusId); - return; - } - - _logger.LogTrace("MessageBus {MessageBusId} dispose", MessageBusId); - - try - { - ShutdownAsync().AnyContext().GetAwaiter().GetResult(); - } - catch (OperationCanceledException ex) - { - _logger.LogTrace(ex, "Shutdown cancelled for {MessageBusId}", MessageBusId); - } - catch (ObjectDisposedException ex) - { - _logger.LogDebug(ex, "Resource already disposed during shutdown for {MessageBusId}", MessageBusId); - } - catch (Exception ex) - { - _logger.LogError(ex, "Error during shutdown for {MessageBusId}: {Message}", MessageBusId, ex.Message); - } - - _subscribers?.Clear(); - _disposedCancellationTokenSource.Cancel(); - - try - { - CleanupAsync().AnyContext().GetAwaiter().GetResult(); - } - catch (OperationCanceledException ex) - { - _logger.LogTrace(ex, "Cleanup cancelled for {MessageBusId}", MessageBusId); - } - catch (ObjectDisposedException ex) - { - _logger.LogDebug(ex, "Resource already disposed during cleanup for {MessageBusId}", MessageBusId); - } - catch (Exception ex) - { - _logger.LogError(ex, "Error during cleanup for {MessageBusId}: {Message}", MessageBusId, ex.Message); - } - - _disposedCancellationTokenSource.Dispose(); - } - - [DebuggerDisplay("Id: {Id} Type: {Type} CancellationToken: {CancellationToken}")] - protected class Subscriber - { - private readonly ConcurrentDictionary _assignableTypesCache = new(); - - public string Id { get; private set; } = Guid.NewGuid().ToString("N"); - public CancellationToken CancellationToken { get; set; } - public required Type Type { get; set; } - public Type? GenericType { get; set; } - public required Func Action { get; set; } - - public bool IsAssignableFrom(Type type) - { - if (type is null) - return false; - - return _assignableTypesCache.GetOrAdd(type, t => - { - if (t.IsClass) - { - var typedMessageType = typeof(IMessage<>).MakeGenericType(t); - if (Type == typedMessageType) - return true; - } - - return Type.GetTypeInfo().IsAssignableFrom(t); - }); - } - } -} diff --git a/src/Foundatio/Messaging/MessageClientCore.cs b/src/Foundatio/Messaging/MessageClientCore.cs new file mode 100644 index 000000000..d685e4c75 --- /dev/null +++ b/src/Foundatio/Messaging/MessageClientCore.cs @@ -0,0 +1,1799 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Diagnostics; +using System.Diagnostics.Metrics; +using System.Globalization; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Serializer; +using Foundatio.Utility; +using Microsoft.Extensions.Logging; + +namespace Foundatio.Messaging; + +/// +/// Core-owned messaging instruments. Counters and histograms are transport-agnostic and shared by every +/// instance so that send/receive/settlement volume and handler +/// latency are observable regardless of which transport is plugged in. +/// +internal static class MessagingInstruments +{ + public static readonly Counter Sent = FoundatioDiagnostics.Meter.CreateCounter("foundatio.messaging.sent", description: "Number of messages sent to a destination"); + public static readonly Counter Received = FoundatioDiagnostics.Meter.CreateCounter("foundatio.messaging.received", description: "Number of messages received from a source"); + public static readonly Counter Completed = FoundatioDiagnostics.Meter.CreateCounter("foundatio.messaging.completed", description: "Number of messages completed"); + public static readonly Counter Abandoned = FoundatioDiagnostics.Meter.CreateCounter("foundatio.messaging.abandoned", description: "Number of messages abandoned"); + public static readonly Counter DeadLettered = FoundatioDiagnostics.Meter.CreateCounter("foundatio.messaging.deadlettered", description: "Number of messages dead-lettered"); + public static readonly Counter Unhandled = FoundatioDiagnostics.Meter.CreateCounter("foundatio.messaging.unhandled", description: "Number of received messages with no registered consumer for their type"); + public static readonly Histogram HandlerTime = FoundatioDiagnostics.Meter.CreateHistogram("foundatio.messaging.handlertime", unit: "ms", description: "Message handler execution time"); +} + +/// +/// Transport-neutral envelope options shared by queue send and pub/sub publish operations. +/// +internal sealed record MessageEnvelopeOptions +{ + public string? MessageId { get; init; } + public MessagePriority Priority { get; init; } = MessagePriority.Normal; + public TimeSpan? Delay { get; init; } + public DateTimeOffset? DeliverAt { get; init; } + public TimeSpan? TimeToLive { get; init; } + public string? CorrelationId { get; init; } + public MessageHeaders? Headers { get; init; } +} + +/// +/// Describes a consumer/subscription listener independent of whether it is backed by a queue or a pub/sub subscription. +/// +internal sealed record ListenerConfig +{ + public required DestinationAddress Source { get; init; } + public required string Key { get; init; } + public required Type MessageType { get; init; } + public bool Ephemeral { get; init; } + public AckMode AckMode { get; init; } = AckMode.Auto; + public int MaxConcurrency { get; init; } = 1; + public int? PrefetchCount { get; init; } + public TimeSpan? ReceiveBatchDelay { get; init; } + public TimeSpan VisibilityTimeout { get; init; } = TimeSpan.FromMinutes(1); + public bool AutoRenewLock { get; init; } = true; + public TimeSpan ShutdownTimeout { get; init; } + public bool WaitForManualSettlement { get; init; } = true; + // Null falls back to the client's default RetryPolicy. + public int? MaxAttempts { get; init; } + public Func? RedeliveryBackoff { get; init; } + public Func? DeadLetterWhen { get; init; } +} + +/// +/// Shared implementation behind : serialization, header/trace +/// construction, routing-agnostic send (with batch chunking and runtime-store scheduled dispatch), received-message +/// creation with poison handling, auto/manual ack settlement, and the resilient consumer/subscription loop. +/// +internal sealed class MessageClientCore : IAsyncDisposable +{ + private readonly IMessageTransport _transport; + private readonly ISerializer _serializer; + private readonly IMessageRouter _router; + private readonly IScheduledDispatchStore? _runtimeStore; + private readonly TimeProvider _timeProvider; + private readonly ILogger _logger; + private readonly Func _exceptionFactory; + private readonly RetryPolicy _retryPolicy; + private readonly IMessageTypeRegistry _typeRegistry; + private readonly string _contentType; + private readonly bool _ownsTransport; + private readonly ConcurrentDictionary _sources = new(); + private readonly TopologyMode _topologyMode; + private readonly ConcurrentDictionary _validatedDestinations = new(); + private readonly CancellationTokenSource _lifetimeCancellation = new(); + private int _isDisposed; + + public MessageClientCore(IMessageTransport transport, ISerializer serializer, IMessageRouter router, + IScheduledDispatchStore? runtimeStore, TimeProvider timeProvider, ILogger logger, Func exceptionFactory, RetryPolicy? retryPolicy = null, bool ownsTransport = true, IMessageTypeRegistry? typeRegistry = null, string? contentType = null, TopologyMode topologyMode = TopologyMode.Ensure) + { + _topologyMode = topologyMode; + _transport = transport ?? throw new ArgumentNullException(nameof(transport)); + _serializer = serializer; + _router = router; + _runtimeStore = runtimeStore; + _timeProvider = timeProvider; + _logger = logger; + _exceptionFactory = exceptionFactory; + _retryPolicy = retryPolicy ?? new RetryPolicy(); + _typeRegistry = typeRegistry ?? new MessageTypeRegistry(); + _contentType = contentType ?? (serializer is SystemTextJsonSerializer ? "application/json" : "application/octet-stream"); + _ownsTransport = ownsTransport; + } + + public IMessageRouter Router => _router; + + // A transport that advertises ITransportInfo is held to its declaration. One that does not is assumed to be a + // minimal QUEUE-ONLY transport — assuming every role would let a queue-only provider silently accept topic + // publishes it can never fan out, which contradicts the "anything not advertised is unsupported" capability + // philosophy. Real providers should implement ITransportInfo and state their roles. + public bool SupportsRole(DestinationRole role) + { + return _transport is ITransportInfo info ? info.SupportedRoles.Contains(role) : role == DestinationRole.Queue; + } + + public bool SupportsTemporarySubscriptions => _topologyMode == TopologyMode.Ensure && _transport is ISupportsEphemeralSubscriptions; + + public void RequireEphemeralSubscriptions() + { + if (_topologyMode != TopologyMode.Ensure || _transport is not ISupportsEphemeralSubscriptions) + throw new NotSupportedException("Temporary subscriptions require a transport with expiring subscription leases and TopologyMode.Ensure. Use an explicitly named, pre-provisioned subscription with this transport or topology mode."); + } + + private readonly ConcurrentDictionary _ensuredDestinations = new(); + private readonly SemaphoreSlim _provisioning = new(1, 1); + + public Task EnsureAsync(IReadOnlyList declarations, CancellationToken cancellationToken) + => _topologyMode switch + { + TopologyMode.None => Task.CompletedTask, + TopologyMode.Validate => ValidateDeclarationsAsync(declarations, cancellationToken), + _ => EnsureDeclarationsAsync(declarations, cancellationToken) + }; + + private async Task EnsureDeclarationsAsync(IReadOnlyList declarations, CancellationToken cancellationToken) + { + if (_transport is not ISupportsProvisioning provisioning) return; + foreach (var declaration in declarations) + { + if (declaration.AutoDeleteAfter is not null) + { + await provisioning.EnsureAsync([declaration], cancellationToken).AnyContext(); + continue; + } + if (_ensuredDestinations.TryGetValue(declaration.Address, out var expires) && expires > _timeProvider.GetUtcNow()) continue; + await _provisioning.WaitAsync(cancellationToken).AnyContext(); + try + { + if (_ensuredDestinations.TryGetValue(declaration.Address, out expires) && expires > _timeProvider.GetUtcNow()) continue; + await provisioning.EnsureAsync([declaration], cancellationToken).AnyContext(); + _ensuredDestinations[declaration.Address] = _timeProvider.GetUtcNow().AddSeconds(30); + } + finally { _provisioning.Release(); } + } + } + + private void InvalidateProvisioning(DestinationAddress address) + { + _ensuredDestinations.TryRemove(address, out _); + _validatedDestinations.TryRemove(address, out _); + } + + // Validate never creates: each destination is checked once (successes are cached so steady-state publishes pay no + // exists round-trip) and a missing one fails loudly instead of being silently created on a broker the app is not + // supposed to administer. + private async Task ValidateDeclarationsAsync(IReadOnlyList declarations, CancellationToken cancellationToken) + { + if (_transport is not ISupportsProvisioning provisioning) + throw new NotSupportedException($"{nameof(TopologyMode)}.{nameof(TopologyMode.Validate)} requires a transport that can check destination existence; \"{_transport.GetType().Name}\" does not support provisioning. Use {nameof(TopologyMode)}.{nameof(TopologyMode.None)} when the transport cannot inspect a pre-provisioned broker."); + + foreach (var declaration in declarations) + { + if (_validatedDestinations.ContainsKey(declaration.Address)) + continue; + + if (!await provisioning.ExistsAsync(declaration.Address, cancellationToken).AnyContext()) + throw _exceptionFactory($"Message topology destination {declaration.Address} does not exist and {nameof(TopologyMode)}.{nameof(TopologyMode.Validate)} never creates topology. Provision it out of band or use {nameof(TopologyMode)}.{nameof(TopologyMode.Ensure)}.", null); + + _validatedDestinations.TryAdd(declaration.Address, 0); + } + } + + public async Task SendAsync(ScheduledDispatchKind kind, Type messageType, object message, MessageEnvelopeOptions options, DestinationAddress destination, Func? ensureDestination, CancellationToken cancellationToken) + { + ThrowIfDisposed(); + ValidateCapabilities(destination, options.Priority, options.TimeToLive); + + if (options.MessageId is not null) + ArgumentException.ThrowIfNullOrWhiteSpace(options.MessageId); + + var sendOptions = BuildSendOptions(options); + string messageId = options.MessageId ?? Guid.NewGuid().ToString("N"); + var transportMessage = CreateTransportMessage(message, options, messageId); + + // Produce-side routing visibility: the consume side logs its effective topology at subscribe time, and this + // is its counterpart for "where did my message actually go" debugging. + if (_logger.IsEnabled(LogLevel.Debug)) + _logger.LogDebug("Sending {MessageType} to {Destination}", messageType.Name, destination); + + if (ensureDestination is not null) + await ensureDestination(destination, cancellationToken).AnyContext(); + + if (await TryScheduleAsync(kind, destination, [transportMessage], sendOptions, cancellationToken).AnyContext()) + return messageId; + + await SendOneAsync(destination, transportMessage, sendOptions, cancellationToken).AnyContext(); + return messageId; + } + + public async Task> SendBatchAsync(ScheduledDispatchKind kind, IEnumerable messages, Type? declaredType, MessageEnvelopeOptions options, Func resolveDestination, Func? ensureDestination, CancellationToken cancellationToken) + { + ThrowIfDisposed(); + + var sendOptions = BuildSendOptions(options); + if (options.MessageId is not null) + throw new ArgumentException("A batch cannot share one message ID. Use MessageBatchItem to supply per-input application IDs.", nameof(options)); + + var grouped = new Dictionary>(); + var messageIds = new List(); + + foreach (var input in messages) + { + ArgumentNullException.ThrowIfNull(input); + var item = input as IMessageBatchItem; + var message = item?.Value ?? input; + ArgumentNullException.ThrowIfNull(message); + Type messageType = declaredType ?? message.GetType(); + var destination = resolveDestination(messageType); + + if (!grouped.TryGetValue(destination, out var transportMessages)) + { + transportMessages = []; + grouped.Add(destination, transportMessages); + } + + // Application IDs stay in input order even when messages route to different destinations. + string messageId = item?.MessageId ?? Guid.NewGuid().ToString("N"); + ArgumentException.ThrowIfNullOrWhiteSpace(messageId); + messageIds.Add(messageId); + transportMessages.Add((messageIds.Count - 1, CreateTransportMessage(message, options with { Headers = item?.Headers ?? options.Headers }, messageId))); + } + + var outcomes = messageIds.Select(id => new MessageSendOutcome(id, MessageSendStatus.NotAttempted)).ToArray(); + foreach (var group in grouped) + { + try + { + ValidateCapabilities(group.Key, options.Priority, options.TimeToLive); + if (ensureDestination is not null) + await ensureDestination(group.Key, cancellationToken).AnyContext(); + + var transportMessages = group.Value.Select(item => item.Message).ToArray(); + if (!await TryScheduleAsync(kind, group.Key, transportMessages, sendOptions, cancellationToken).AnyContext()) + await SendChunkedAsync(group.Key, transportMessages, sendOptions, cancellationToken).AnyContext(); + + foreach (var item in group.Value) + outcomes[item.InputIndex] = outcomes[item.InputIndex] with { Status = MessageSendStatus.Accepted }; + } + catch (Exception ex) + { + if (ex is MessageSendException failed) + { + for (int index = 0; index < group.Value.Count; index++) + outcomes[group.Value[index].InputIndex] = failed.Outcomes[index]; + } + + throw new MessageSendException(outcomes, ex); + } + } + + return messageIds; + } + + public Task StartListenerAsync(ListenerConfig config, Func handler, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(handler); + return RegisterConsumerAsync(config, async (entry, lease, token) => + { + var received = CreateMessageContext(entry, token, lease); + await HandleMessageAsync(received, config, handler, token).AnyContext(); + }, cancellationToken); + } + + public Task StartOutcomeListenerAsync(ListenerConfig config, Func> handler, CancellationToken cancellationToken) + => RegisterConsumerAsync(config, (entry, lease, token) => HandleMessageOutcomeAsync(CreateMessageContext(entry, token, lease), config, handler, token), cancellationToken); + + public Task StartOutcomeListenerAsync(ListenerConfig config, Func, CancellationToken, ValueTask> handler, CancellationToken cancellationToken) where T : class + => RegisterConsumerAsync(config, async (entry, lease, token) => + { + var received = await CreateMessageContextAsync(entry, token, lease).AnyContext(); + await HandleMessageOutcomeAsync(received, config, handler, token).AnyContext(); + }, cancellationToken); + + public Task?> ReceiveAsync(DestinationAddress source, TimeSpan wait, CancellationToken cancellationToken) where T : class + { + return ReceiveCoreAsync>(source, wait, async (entry, cancellation, supervision) => + new ReceivedMessage(await CreateMessageContextAsync(entry, cancellation.Token, supervision).AnyContext(), cancellation, supervision, ct => ReturnUnsettledAsync(entry, ct)), cancellationToken); + } + + public Task ReceiveAsync(DestinationAddress source, TimeSpan wait, CancellationToken cancellationToken) + { + return ReceiveCoreAsync(source, wait, (entry, cancellation, supervision) => + Task.FromResult(new ReceivedMessage(CreateMessageContext(entry, cancellation.Token, supervision), cancellation, supervision, ct => ReturnUnsettledAsync(entry, ct))), cancellationToken); + } + + private async Task ReceiveCoreAsync(DestinationAddress source, TimeSpan wait, + Func> create, CancellationToken cancellationToken) where T : class + { + ThrowIfDisposed(); + ArgumentOutOfRangeException.ThrowIfLessThan(wait, TimeSpan.Zero); + var pull = RequirePull(); + var cancellation = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, _lifetimeCancellation.Token); + MessageDeliveryLease? supervision = null; + bool transferred = false; + try + { + await EnsureAsync([new DestinationDeclaration { Address = source }], cancellation.Token).AnyContext(); + var request = new ReceiveRequest { MaxMessages = 1, MaxWaitTime = wait }; + var entries = _transport is ISupportsVisibilityTimeout visibility + ? await visibility.ReceiveAsync(source, request, TimeSpan.FromMinutes(1), cancellation.Token).AnyContext() + : await pull.ReceiveAsync(source, request, cancellation.Token).AnyContext(); + if (entries.Count == 0) + return null; + + supervision = new MessageDeliveryLease(_transport, entries[0], TimeSpan.FromMinutes(1), true, _timeProvider, _logger, cancellation); + var received = await create(entries[0], cancellation, supervision).AnyContext(); + transferred = true; + return received; + } + finally + { + if (!transferred) + { + await cancellation.CancelAsync().AnyContext(); + if (supervision is not null) await supervision.DisposeAsync().AnyContext(); + cancellation.Dispose(); + } + } + } + + public Task StartListenerAsync(ListenerConfig config, Func, CancellationToken, Task> handler, CancellationToken cancellationToken) where T : class + { + ArgumentNullException.ThrowIfNull(handler); + return RegisterConsumerAsync(config, async (entry, lease, token) => + { + var received = await CreateMessageContextAsync(entry, token, lease).AnyContext(); + await HandleMessageAsync(received, config, handler, token).AnyContext(); + }, cancellationToken); + } + + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _isDisposed, 1) == 1) + return; + + await _lifetimeCancellation.CancelAsync().AnyContext(); + + foreach (var listener in _sources.Values.ToArray()) + await listener.DisposeAsync().AnyContext(); + + // Only dispose the transport when this client owns it. In DI the transport is a shared singleton owned by the + // container, so neither the queue nor the pub/sub client should dispose it (that would double-dispose the one + // the other still depends on). + if (_ownsTransport) + await _transport.DisposeAsync().AnyContext(); + _lifetimeCancellation.Dispose(); + } + + // Multiple typed consumers can share one destination. They attach to a single per-source listener whose loop + // demultiplexes each message to the consumer registered for its type; a type with no registered consumer is + // handled by HandleUnmatchedAsync. Duplicate registration on one endpoint is rejected. + private async Task RegisterConsumerAsync(ListenerConfig config, Func dispatch, CancellationToken cancellationToken) + { + ThrowIfDisposed(); + cancellationToken.ThrowIfCancellationRequested(); + + bool catchAll = IsCatchAll(config.MessageType); + var registration = new ConsumerRegistration + { + Key = config.Key, + Config = config, + Dispatch = dispatch, + IsCatchAll = catchAll, + TypeName = catchAll ? null : _typeRegistry.GetName(config.MessageType) + }; + + while (true) + { + var listener = _sources.GetOrAdd(config.Source, source => new SourceListener(this, source)); + if (listener.TryAddConsumer(registration, out var handle, out bool created)) + { + if (created) + { + try + { + await listener.StartAsync(cancellationToken).AnyContext(); + } + catch + { + await listener.DisposeAsync().AnyContext(); + throw; + } + } + + return handle; + } + + // The listener was disposing as its last consumer detached; drop our stale reference and retry. + _sources.TryRemove(new KeyValuePair(config.Source, listener)); + } + } + + // A concrete message type binds an exact-type consumer; object/interface/abstract route types are catch-alls that + // receive every message a more specific typed consumer did not claim (the grouped/raw-envelope path). + private static bool IsCatchAll(Type messageType) + { + return messageType == typeof(object) || messageType.IsInterface || messageType.IsAbstract; + } + + private async Task HandleUnmatchedAsync(TransportEntry entry, DestinationAddress source, CancellationToken cancellationToken) + { + MessagingInstruments.Unhandled.Add(1, new KeyValuePair("source", source)); + + var message = CreateMessageContext(entry, cancellationToken); + + // Same WARN-while-retryable / ERROR-when-terminal convention as handler failures. + if (message.Attempts >= _retryPolicy.UnmatchedMaxAttempts) + _logger.LogError("No consumer registered for message type \"{MessageType}\" on \"{Source}\" (attempt {Attempt} of {MaxAttempts}); dead-lettering as no-handler", message.MessageType, source, message.Attempts, _retryPolicy.UnmatchedMaxAttempts); + else + _logger.LogWarning("No consumer registered for message type \"{MessageType}\" on \"{Source}\" (attempt {Attempt} of {MaxAttempts}); will retry", message.MessageType, source, message.Attempts, _retryPolicy.UnmatchedMaxAttempts); + + // Retry so a node that does handle this type can pick it up; dead-letter as "no-handler" once the lenient + // budget is exhausted so a genuinely orphaned type cannot loop forever. + await SettleFailedMessageAsync(message, unrecoverable: false, _retryPolicy.UnmatchedMaxAttempts, _retryPolicy.UnmatchedBackoff, deadLetterReason: "no-handler", exception: null, cancellationToken).AnyContext(); + + // Surface to direct callers. The throw is caught (and not re-logged) by the loop's per-message handling + // (SafeProcessAsync), so it never tears down the receive loop or the other type handlers sharing this source. + throw new UnhandledMessageTypeException(message.MessageType, source.Key); + } + + // MaxConcurrency bounds the number of in-flight messages. A slot is held from receive until the message settles + // and is released the instant that one message finishes — so a single slow message never stalls the other slots + // (no head-of-line blocking) and steady-state utilization stays at the configured concurrency. A failure while + // receiving or while processing a single entry (including a poison message that was already dead-lettered) must + // never tear down the loop, otherwise one bad message or a transient transport blip silently stops consumption. + private async Task RunPullLoopAsync(DestinationAddress source, ISupportsPull pull, Func onMessage, ListenerConfig endpoint, CancellationToken cancellationToken, Action? receivingHealth = null) + { + int maxConcurrency = Math.Max(1, endpoint.MaxConcurrency); + var capabilities = (_transport as ITransportInfo)?.GetCapabilities(source); + int batchSize = Math.Clamp(capabilities?.MaxReceiveBatchSize ?? maxConcurrency, 1, maxConcurrency); + if (endpoint.PrefetchCount is { } prefetch) + batchSize = Math.Min(batchSize, prefetch); + var batchDelay = endpoint.ReceiveBatchDelay.HasValue ? TimeSpan.Zero : capabilities?.ReceiveBatchDelay ?? TimeSpan.Zero; + int largestReceivedBatch = 0; + var slots = new SemaphoreSlim(maxConcurrency, maxConcurrency); + var inFlight = new ConcurrentDictionary(); + var cleanupSlots = new SemaphoreSlim(maxConcurrency, maxConcurrency); + using var collecting = new SemaphoreSlim(1, 1); + using var receivingCancellation = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + var receivingToken = receivingCancellation.Token; + using var processingCancellation = new CancellationTokenSource(); + var processingToken = processingCancellation.Token; + using var stopRegistration = receivingToken.Register(() => + { + if (endpoint.ShutdownTimeout == TimeSpan.Zero) processingCancellation.Cancel(); + else processingCancellation.CancelAfter(endpoint.ShutdownTimeout); + }); + int receiveConcurrency = Math.Clamp(capabilities?.MaxConcurrentReceives ?? 1, 1, (maxConcurrency - 1) / batchSize + 1); + try + { + var receivers = new Task[receiveConcurrency]; + for (int i = 0; i < receivers.Length; i++) + receivers[i] = ReceiveAsync(receivingToken); + await Task.WhenAll(receivers).AnyContext(); + } + finally + { + await Task.WhenAll(inFlight.Keys.ToArray()).AnyContext(); + slots.Dispose(); + cleanupSlots.Dispose(); + } + + async Task ReceiveAsync(CancellationToken cancellationToken) + { + int consecutiveReceiveFailures = 0; + + try + { + while (!cancellationToken.IsCancellationRequested) + { + // Block for a free slot before receiving so we never pull more than we can process concurrently. + int claimed = 0; + bool collectingBatch = false; + try + { + await collecting.WaitAsync(cancellationToken).AnyContext(); + collectingBatch = true; + if (endpoint.ReceiveBatchDelay is { } collectionDelay && collectionDelay > TimeSpan.Zero + && maxConcurrency - slots.CurrentCount >= (maxConcurrency + 1) / 2 + && slots.CurrentCount < largestReceivedBatch) + { + await Task.WhenAny(Task.WhenAll(inFlight.Keys.ToArray()), Task.Delay(collectionDelay, _timeProvider, cancellationToken)).AnyContext(); + cancellationToken.ThrowIfCancellationRequested(); + } + await slots.WaitAsync(cancellationToken).AnyContext(); + claimed = 1; + long batchStart = batchDelay > TimeSpan.Zero ? Stopwatch.GetTimestamp() : 0; + while (claimed < batchSize) + { + if (await slots.WaitAsync(TimeSpan.Zero).AnyContext()) + { + claimed++; + continue; + } + if (batchDelay <= TimeSpan.Zero) + break; + var remaining = batchDelay - Stopwatch.GetElapsedTime(batchStart); + if (remaining <= TimeSpan.Zero || !await slots.WaitAsync(TimeSpan.FromMilliseconds(Math.Ceiling(remaining.TotalMilliseconds)), cancellationToken).AnyContext()) + break; + claimed++; + } + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + ReleaseSlots(slots, claimed); + break; + } + finally + { + if (collectingBatch) collecting.Release(); + } + + var pollWindow = TimeSpan.FromSeconds(1); + long pollStart = _timeProvider.GetTimestamp(); + IReadOnlyList entries; + try + { + var request = new ReceiveRequest + { + MaxMessages = claimed, + MaxWaitTime = pollWindow + }; + var receive = _transport is ISupportsVisibilityTimeout visibility + ? visibility.ReceiveAsync(source, request, endpoint.VisibilityTimeout, cancellationToken) + : pull.ReceiveAsync(source, request, cancellationToken); + try { entries = await receive.WaitAsync(cancellationToken).AnyContext(); } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + var cleanup = ReleaseLateReceiveAsync(receive); + // Join cooperative receive cancellation before disposal completes, while a provider that + // ignores cancellation gets a bounded grace period and releases its late deliveries separately. + await Task.WhenAny(cleanup, Task.Delay(TimeSpan.FromMilliseconds(50))).AnyContext(); + throw; + } + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + ReleaseSlots(slots, claimed); + break; + } + catch (Exception ex) + { + ReleaseSlots(slots, claimed); + InvalidateProvisioning(source); + receivingHealth?.Invoke(false); + if (ex is MessageDestinationNotFoundException) throw; + + // The first failure of an outage is the alert; repeats at 1/s would be a firehose, so they + // de-escalate to WARN (with a running count) until a receive succeeds again. + consecutiveReceiveFailures++; + if (consecutiveReceiveFailures == 1) + _logger.LogError(ex, "Error receiving from \"{Source}\"; retrying: {Message}", source, ex.Message); + else + _logger.LogWarning(ex, "Error receiving from \"{Source}\" ({ConsecutiveFailures} consecutive); retrying: {Message}", source, consecutiveReceiveFailures, ex.Message); + + await _timeProvider.SafeDelay(TimeSpan.FromSeconds(1), cancellationToken).AnyContext(); + continue; + } + + if (consecutiveReceiveFailures > 0) receivingHealth?.Invoke(true); + if (consecutiveReceiveFailures > 1) + _logger.LogInformation("Receiving from \"{Source}\" recovered after {ConsecutiveFailures} consecutive failures", source, consecutiveReceiveFailures); + consecutiveReceiveFailures = 0; + + // We hold exactly `claimed` slots and release one per processed entry, so never process more than we + // claimed: a well-behaved transport returns <= MaxMessages, but a transport that ignores MaxMessages and + // over-returns would otherwise release more slots than acquired (breaching the cap / overflowing the + // semaphore). Any over-returned entries are left unsettled and redeliver after their visibility window. + int toProcess = Math.Min(entries.Count, claimed); + largestReceivedBatch = Math.Max(largestReceivedBatch, toProcess); + ReleaseSlots(slots, claimed - toProcess); // return slots we claimed but won't fill (always >= 0) + + // An empty poll should have blocked for MaxWaitTime; a transport that returns empty early (or + // synchronously) would otherwise hot-spin this loop, so sleep out the remainder of the window. + if (toProcess == 0) + { + var remaining = pollWindow - _timeProvider.GetElapsedTime(pollStart); + if (remaining > TimeSpan.Zero) + await _timeProvider.SafeDelay(remaining, cancellationToken).AnyContext(); + } + + for (int index = 0; index < toProcess; index++) + { + if (cancellationToken.IsCancellationRequested) + { + using var cleanup = new CancellationTokenSource(TimeSpan.FromSeconds(5), _timeProvider); + await ReturnUnsettledAsync(entries[index], cleanup.Token).AnyContext(); + (_transport as IMessageProcessingObserver)?.ProcessingFinished(entries[index]); + slots.Release(); + continue; + } + var task = SafeProcessAsync(entries[index], onMessage, source, processingToken, slots, cleanupSlots, endpoint); + if (!task.IsCompleted) + { + inFlight[task] = 0; + _ = task.ContinueWith(static (t, state) => ((ConcurrentDictionary)state!).TryRemove(t, out _), inFlight, TaskScheduler.Default); + } + } + } + } + catch + { + await receivingCancellation.CancelAsync().AnyContext(); + throw; + } + } + } + + private static void ReleaseSlots(SemaphoreSlim slots, int count) + { + if (count > 0) + slots.Release(count); + } + + private async Task SafeProcessAsync(TransportEntry entry, Func onMessage, DestinationAddress source, CancellationToken cancellationToken, SemaphoreSlim? slots = null, SemaphoreSlim? cleanupSlots = null, ListenerConfig? endpoint = null) + { + (_transport as IMessageProcessingObserver)?.ProcessingStarted(entry); + using var deliveryCancellation = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + await using var lease = new MessageDeliveryLease(_transport, entry, endpoint?.VisibilityTimeout ?? TimeSpan.FromMinutes(1), + endpoint?.AutoRenewLock ?? true, _timeProvider, _logger, deliveryCancellation); + try + { + deliveryCancellation.Token.ThrowIfCancellationRequested(); + await onMessage(entry, lease, deliveryCancellation.Token).AnyContext(); + } + catch (OperationCanceledException) when (deliveryCancellation.IsCancellationRequested) { } + catch (UnhandledMessageTypeException) { } + catch (Exception ex) + { + _logger.LogError(ex, "Error processing message \"{MessageId}\" from \"{Source}\": {Message}", entry.Id, source, ex.Message); + } + finally + { + if (cancellationToken.IsCancellationRequested && !lease.IsLost && !lease.IsSettled) + { + using var cleanup = new CancellationTokenSource(TimeSpan.FromSeconds(5), _timeProvider); + await ReturnUnsettledAsync(entry, cleanup.Token).AnyContext(); + } + lease.Settled(); + var cancellation = deliveryCancellation.CancelAsync(); + if (cleanupSlots is not null) + await cleanupSlots.WaitAsync().AnyContext(); + try + { + slots?.Release(); + await cancellation.AnyContext(); + await lease.Completion.AnyContext(); + } + finally { cleanupSlots?.Release(); } + (_transport as IMessageProcessingObserver)?.ProcessingFinished(entry); + } + } + + private async Task ReturnUnsettledAsync(TransportEntry entry, CancellationToken cancellationToken) + { + try + { + await _transport.AbandonAsync(entry, cancellationToken).WaitAsync(cancellationToken).AnyContext(); + } + catch (ReceiptExpiredException) { } + catch (Exception ex) + { + _logger.LogWarning(ex, "Unable to return interrupted message {MessageId} from {Source}; its lease will expire", entry.Id, entry.Destination); + } + } + + private async Task ReleaseLateReceiveAsync(Task> receive) + { + try + { + foreach (var entry in await receive.AnyContext()) + { + using var cleanup = new CancellationTokenSource(TimeSpan.FromSeconds(5), _timeProvider); + await ReturnUnsettledAsync(entry, cleanup.Token).AnyContext(); + (_transport as IMessageProcessingObserver)?.ProcessingFinished(entry); + } + } + catch (OperationCanceledException) { } + catch (Exception exception) { _logger.LogWarning(exception, "Receive finished after its listener stopped"); } + } + + private async Task HandleMessageAsync(TMessage message, ListenerConfig config, Func handler, CancellationToken cancellationToken) where TMessage : IMessageContext + => await HandleMessageOutcomeAsync(message, config, async (context, token) => + { + await handler(context, token).AnyContext(); + return MessageOutcome.Success; + }, cancellationToken).AnyContext(); + + private async Task HandleMessageOutcomeAsync(TMessage message, ListenerConfig config, Func> handler, CancellationToken cancellationToken) where TMessage : IMessageContext + { + // Re-establish the producer's trace context on the consumer side so a cross-process trace continues here + // instead of breaking at the transport boundary. + using var activity = StartProcessActivity(message, config); + long startTimestamp = Stopwatch.GetTimestamp(); + try + { + var outcome = await handler(message, cancellationToken).AnyContext(); + if (outcome.Kind is MessageOutcomeKind.Retry or MessageOutcomeKind.DeadLetter) + { + await outcome.SettleFailureAsync(message, config.MaxAttempts ?? _retryPolicy.MaxAttempts, config.RedeliveryBackoff ?? _retryPolicy.Backoff, cancellationToken).AnyContext(); + return; + } + if (outcome.Kind == MessageOutcomeKind.Unsettled) + return; + + if (config.AckMode == AckMode.Auto && !message.IsHandled) + await message.CompleteAsync(cancellationToken).AnyContext(); + else if (config.AckMode == AckMode.Manual && config.WaitForManualSettlement && message is MessageContext context) + await context.WaitForSettlementAsync(cancellationToken).AnyContext(); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception ex) + { + activity?.SetErrorStatus(ex); + + // The handler already settled (e.g. terminal-rejected a poison payload, then rethrew): the settle path + // will skip, so don't log a retry/dead-letter that won't happen. + if (message.IsHandled) + { + _logger.LogWarning(ex, "Handler threw after settling message \"{MessageId}\" from \"{Source}\"; no further settlement will occur: {Message}", message.Id, config.Source, ex.Message); + return; + } + + int maxAttempts = config.MaxAttempts ?? _retryPolicy.MaxAttempts; + var backoff = config.RedeliveryBackoff ?? _retryPolicy.Backoff; + + bool unrecoverable = false; + try + { + unrecoverable = (config.DeadLetterWhen ?? _retryPolicy.DeadLetterWhen)?.Invoke(ex) == true; + } + catch (Exception predicateEx) + { + _logger.LogError(predicateEx, "DeadLetterWhen predicate threw for message \"{MessageId}\"; treating the failure as retryable: {Message}", message.Id, predicateEx.Message); + } + + // A retry that can still happen is a warning; the terminal decision (unrecoverable or attempts exhausted) + // is the error worth alerting on. + if (unrecoverable || maxAttempts >= 0 && message.Attempts >= maxAttempts) + _logger.LogError(ex, "Handler failed for message \"{MessageId}\" from \"{Source}\" (attempt {Attempt} of {MaxAttempts}); dead-lettering: {Message}", message.Id, config.Source, message.Attempts, maxAttempts, ex.Message); + else + _logger.LogWarning(ex, "Handler failed for message \"{MessageId}\" from \"{Source}\" (attempt {Attempt} of {MaxAttempts}); will retry: {Message}", message.Id, config.Source, message.Attempts, maxAttempts, ex.Message); + + string reason = unrecoverable ? $"unrecoverable:{ex.GetType().Name}" : "handler-error"; + await SettleFailedMessageAsync(message, unrecoverable, maxAttempts, backoff, reason, ex, cancellationToken).AnyContext(); + } + finally + { + MessagingInstruments.HandlerTime.Record(Stopwatch.GetElapsedTime(startTimestamp).TotalMilliseconds, new KeyValuePair("source", config.Source.Key)); + } + } + + private static Activity? StartProcessActivity(IMessageContext message, ListenerConfig config) + { + string? traceParent = message.Headers.GetValueOrDefault(KnownHeaders.TraceParent); + var activity = FoundatioDiagnostics.ActivitySource.StartActivity("ProcessMessage", ActivityKind.Consumer, traceParent); + if (activity is null) + return null; + + string? traceState = message.Headers.GetValueOrDefault(KnownHeaders.TraceState); + if (!String.IsNullOrEmpty(traceState)) + activity.TraceStateString = traceState; + + activity.DisplayName = $"Process: {message.MessageType ?? config.MessageType.Name}"; + + if (activity.IsAllDataRequested) + { + activity.SetTag("messaging.source", config.Source.Key); + activity.SetTag("messaging.message.id", message.Id); + } + + return activity; + } + + private static Task SettleFailedMessageAsync(IMessageContext message, bool unrecoverable, int maxAttempts, Func? backoff, string deadLetterReason, Exception? exception, CancellationToken cancellationToken) + { + if (message.IsHandled) + return Task.CompletedTask; + + var outcome = unrecoverable ? MessageOutcome.DeadLetter(deadLetterReason, exception) : MessageOutcome.Retry(deadLetterReason, exception); + return outcome.SettleFailureAsync(message, maxAttempts, backoff, cancellationToken); + } + + private MessageContext CreateMessageContext(TransportEntry entry, CancellationToken cancellationToken, MessageDeliveryLease? lease = null) + { + MessagingInstruments.Received.Add(1, new KeyValuePair("source", entry.Destination.Key)); + return new MessageContext(_transport, entry, cancellationToken, _runtimeStore, _timeProvider, _retryPolicy.DeadLetterDestination, _logger, _topologyMode, lease); + } + + private async Task> CreateMessageContextAsync(TransportEntry entry, CancellationToken cancellationToken, MessageDeliveryLease? lease = null) where T : class + { + MessagingInstruments.Received.Add(1, new KeyValuePair("source", entry.Destination.Key)); + + if (entry.EnvelopeError is { } envelopeError) + { + await DeadLetterPoisonMessageAsync(entry, "invalid-envelope", envelopeError, cancellationToken).AnyContext(); + throw _exceptionFactory($"Message {entry.Id} has an invalid transport envelope.", envelopeError); + } + string? contentType = entry.ContentType ?? entry.Headers.GetValueOrDefault(KnownHeaders.ContentType); + if (!String.IsNullOrEmpty(contentType) && !String.Equals(contentType, _contentType, StringComparison.OrdinalIgnoreCase)) + { + await DeadLetterPoisonMessageAsync(entry, "unsupported-content-type", null, cancellationToken).AnyContext(); + throw _exceptionFactory($"Message {entry.Id} uses {contentType}, but this consumer expects {_contentType}. Configure the same serializer on producers and consumers.", null); + } + + // For an interface, abstract or object route, resolve the concrete payload type + // from the message-type header via the registry and deserialize that, then hand it back as T (the concrete + // instance is assignable to T). Exact concrete routes deserialize as T directly. + Type targetType = typeof(T); + string? typeName = entry.Headers.GetValueOrDefault(KnownHeaders.MessageType); + if (IsCatchAll(typeof(T))) + { + var resolved = String.IsNullOrEmpty(typeName) ? null : _typeRegistry.Resolve(typeName); + if (resolved is null || !typeof(T).IsAssignableFrom(resolved)) + { + await DeadLetterPoisonMessageAsync(entry, "unresolved-type", exception: null, cancellationToken).AnyContext(); + throw _exceptionFactory($"Unable to resolve a concrete type \"{typeName}\" assignable to \"{typeof(T).Name}\" for message \"{entry.Id}\".", null); + } + + targetType = resolved; + } + else if (!String.IsNullOrEmpty(typeName) && typeName != _typeRegistry.GetName(targetType)) + { + await DeadLetterPoisonMessageAsync(entry, "unexpected-message-type", null, cancellationToken).AnyContext(); + throw _exceptionFactory($"Message {entry.Id} has type {typeName}, but this receiver expects {_typeRegistry.GetName(targetType)}. Use a raw receiver for a queue carrying multiple message types.", null); + } + + T? message; + try + { + message = _serializer.Deserialize(entry.Body, targetType) as T; + } + catch (Exception ex) + { + await DeadLetterPoisonMessageAsync(entry, "deserialize-failure", ex, cancellationToken).AnyContext(); + throw _exceptionFactory($"Unable to deserialize message \"{entry.Id}\".", ex); + } + + if (message is null) + { + await DeadLetterPoisonMessageAsync(entry, "deserialize-failure", exception: null, cancellationToken).AnyContext(); + throw _exceptionFactory($"Message \"{entry.Id}\" deserialized to null.", null); + } + + return new MessageContext(_transport, entry, message, cancellationToken, _runtimeStore, _timeProvider, _retryPolicy.DeadLetterDestination, _logger, _topologyMode, lease); + } + + private Task DeadLetterPoisonMessageAsync(TransportEntry entry, string reason, Exception? exception, CancellationToken cancellationToken) + { + MessagingInstruments.DeadLettered.Add(1, new KeyValuePair("source", entry.Destination.Key)); + var enriched = entry with { Headers = MessageContext.BuildDeadLetterHeaders(entry, entry.DeliveryCount, exception, _timeProvider) }; + return MessageContext.DeadLetterAsync(_transport, enriched, reason, _retryPolicy.DeadLetterDestination, _logger, cancellationToken, _topologyMode); + } + + + private TransportMessage CreateTransportMessage(object message, MessageEnvelopeOptions options, string messageId) + { + var headers = (options.Headers ?? MessageHeaders.Empty).ToBuilder() + .Set(KnownHeaders.MessageId, messageId) + .Set(KnownHeaders.ContentType, _contentType) + .Set(KnownHeaders.MessageType, _typeRegistry.GetName(message.GetType())) + .Set(KnownHeaders.Priority, options.Priority.ToString()); + + if (!String.IsNullOrEmpty(options.CorrelationId)) + headers.Set(KnownHeaders.CorrelationId, options.CorrelationId); + + if (Activity.Current is { } activity) + { + if (!String.IsNullOrEmpty(activity.Id)) + headers.SetIfMissing(KnownHeaders.TraceParent, activity.Id); + + if (!String.IsNullOrEmpty(activity.TraceStateString)) + headers.SetIfMissing(KnownHeaders.TraceState, activity.TraceStateString); + } + + if (options.TimeToLive is { } ttl) + headers.Set(KnownHeaders.Expiration, _timeProvider.GetUtcNow().Add(ttl).ToString("O", CultureInfo.InvariantCulture)); + + return new TransportMessage + { + Body = _serializer.SerializeToBytes(message), + Headers = headers.Build(), + MessageId = messageId, + ContentType = _contentType + }; + } + + // A pub/sub publish targets a topic; everything else targets a queue. Stating the role lets the transport route + // without inferring (e.g. SNS publish vs. SQS send). + private static DestinationRole RoleFor(ScheduledDispatchKind kind) + { + return kind == ScheduledDispatchKind.PubSubMessage ? DestinationRole.Topic : DestinationRole.Queue; + } + + private TransportSendOptions BuildSendOptions(MessageEnvelopeOptions options) + { + return new TransportSendOptions + { + Priority = options.Priority, + DeliverAt = options.DeliverAt ?? (options.Delay is { } delay ? _timeProvider.GetUtcNow().Add(delay) : null) + }; + } + + // Capabilities are destination-aware: the same transport can honor a feature on queues but not topics (SQS + // DelaySeconds vs. SNS publish) — and a routing/composite transport can differ per destination — so every + // send-path decision asks for the destination it is actually targeting. + private TransportCapabilities CapabilitiesFor(DestinationAddress destination) + { + return _transport is ITransportInfo info ? info.GetCapabilities(destination) : TransportCapabilities.None; + } + + private void ValidateCapabilities(DestinationAddress destination, MessagePriority priority, TimeSpan? timeToLive) + { + // Sends are role-enforced too: a topic publish on a queue-only transport must fail loudly here rather than + // be accepted into a namespace nothing can ever fan out. + if (!SupportsRole(destination.Role)) + throw new NotSupportedException($"Transport \"{_transport.GetType().Name}\" does not support {destination.Role} destinations."); + + var capabilities = CapabilitiesFor(destination); + + if (priority != MessagePriority.Normal && !capabilities.Priority) + throw new NotSupportedException($"Transport \"{_transport.GetType().Name}\" does not support message priority for {destination.Role} destinations."); + + if (timeToLive is not null && !capabilities.Expiration) + throw new NotSupportedException($"Transport \"{_transport.GetType().Name}\" does not support message expiration for {destination.Role} destinations."); + } + + private async Task TryScheduleAsync(ScheduledDispatchKind kind, DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken cancellationToken) + { + if (!ShouldScheduleThroughRuntimeStore(destination, options, out var dueUtc)) + return false; + + var outcomes = messages.Select(m => new MessageSendOutcome(m.MessageId!, MessageSendStatus.NotAttempted)).ToArray(); + for (int index = 0; index < messages.Count; index++) + { + var message = messages[index]; + outcomes[index] = outcomes[index] with { Status = MessageSendStatus.Unknown }; + try + { + await _runtimeStore!.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = Guid.NewGuid().ToString("N"), + Kind = kind, + Destination = destination, + Body = message.Body, + Headers = message.Headers, + Options = options with { DeliverAt = null }, + DueUtc = dueUtc + }, cancellationToken).AnyContext(); + outcomes[index] = outcomes[index] with { Status = MessageSendStatus.Accepted }; + } + catch (Exception ex) + { + throw new MessageSendException(outcomes, ex); + } + } + + return true; + } + + private bool ShouldScheduleThroughRuntimeStore(DestinationAddress destination, TransportSendOptions options, out DateTimeOffset dueUtc) + { + dueUtc = options.DeliverAt.GetValueOrDefault(); + var now = _timeProvider.GetUtcNow(); + if (options.DeliverAt is null || dueUtc <= now) + return false; + + // A destination can deliver natively only up to its advertised maximum; a delay longer than the broker supports + // (e.g. SQS caps DelaySeconds at 15 minutes) must route through the durable runtime store rather than be + // silently truncated to the broker's ceiling. The check is per destination role: a transport whose queues take + // a native delay may still have topics that cannot (SQS vs. SNS), and those publishes must fall back too. + var capabilities = CapabilitiesFor(destination); + if (capabilities.DelayedDelivery && (capabilities.MaxDeliveryDelay is not { } max || dueUtc - now <= max)) + return false; + + if (_runtimeStore is null) + throw _exceptionFactory($"Delayed delivery requires either native delayed-delivery support from transport \"{_transport.GetType().Name}\" for {destination.Role} destinations (within its supported maximum) or a registered job runtime store.", null); + + return true; + } + + private async Task SendOneAsync(DestinationAddress destination, TransportMessage message, TransportSendOptions options, CancellationToken cancellationToken) + { + if (CapabilitiesFor(destination).MaxMessageBytes is { } maximum && message.Body.Length > maximum) + throw _exceptionFactory($"Message of {message.Body.Length} bytes exceeds transport \"{_transport.GetType().Name}\" maximum of {maximum} bytes for destination \"{destination}\".", null); + + bool attempted = false; + IReadOnlyList? reported = null; + try + { + cancellationToken.ThrowIfCancellationRequested(); + attempted = true; + var result = await _transport.SendAsync(destination, [message], options, cancellationToken).AnyContext(); + if (result.Items.Count != 1) + throw new MessageBusException("The transport did not return one acceptance result per message."); + var item = result.Items[0]; + if (item.Index is not (null or 0) || !Enum.IsDefined(item.Status)) + throw new MessageBusException("Transport returned invalid or duplicate result indexes."); + reported = result.Items; + if (item.Status != MessageSendStatus.Accepted) + throw new MessageBusException("The provider rejected or could not confirm part of the batch."); + RecordSent(destination, result.Items); + } + catch (Exception exception) + { + InvalidateProvisioning(destination); + if (exception is MessageSendException) throw; + MessageSendOutcome[] outcomes = [new(message.MessageId!, attempted ? MessageSendStatus.Unknown : MessageSendStatus.NotAttempted)]; + if (reported is not null) + ApplyOutcomes(reported, outcomes, 0, 1); + if (exception is TransportSendException { Items: { } indexed }) + ApplyOutcomes(indexed, outcomes, 0, 1); + else if (exception is TransportSendException { AcceptedCount: < 1 } partial) + outcomes[0] = outcomes[0] with { Status = partial.AcceptedCount == 0 ? MessageSendStatus.Unknown : MessageSendStatus.NotAttempted }; + throw new MessageSendException(outcomes, exception); + } + } + + private async Task> SendChunkedAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken cancellationToken) + { + var capabilities = CapabilitiesFor(destination); + + // Enforce a transport-declared maximum message size up front with a clear error, rather than letting an opaque + // broker rejection surface mid-send (the limit is advertised, so honor it). + if (capabilities.MaxMessageBytes is { } maxBytes) + { + foreach (var message in messages) + { + if (message.Body.Length > maxBytes) + throw _exceptionFactory($"Message of {message.Body.Length} bytes exceeds transport \"{_transport.GetType().Name}\" maximum of {maxBytes} bytes for destination \"{destination}\".", null); + } + } + + int limit = capabilities.MaxBatchSize is > 0 ? capabilities.MaxBatchSize.Value : Math.Max(1, messages.Count); + var items = new List(messages.Count); + var outcomes = messages.Select(m => new MessageSendOutcome(m.MessageId!, MessageSendStatus.NotAttempted)).ToArray(); + for (int offset = 0; offset < messages.Count; offset += limit) + { + var chunk = new TransportMessage[Math.Min(limit, messages.Count - offset)]; + for (int index = 0; index < chunk.Length; index++) chunk[index] = messages[offset + index]; + try + { + cancellationToken.ThrowIfCancellationRequested(); + for (int index = 0; index < chunk.Length; index++) + outcomes[offset + index] = outcomes[offset + index] with { Status = MessageSendStatus.Unknown }; + + var result = await _transport.SendAsync(destination, chunk, options, cancellationToken).AnyContext(); + if (result.Items.Count != chunk.Length) + throw new MessageBusException("The transport did not return one acceptance result per message."); + + ApplyOutcomes(result.Items, outcomes, offset, chunk.Length); + var accepted = result.Items.Where(i => i.Status == MessageSendStatus.Accepted).ToArray(); + RecordSent(destination, accepted); + items.AddRange(accepted); + if (accepted.Length != chunk.Length) + throw new MessageSendException(outcomes, new MessageBusException("The provider rejected or could not confirm part of the batch.")); + } + catch (Exception ex) + { + InvalidateProvisioning(destination); + if (ex is MessageSendException) throw; + if (ex is TransportSendException { Items: { } indexed }) + ApplyOutcomes(indexed, outcomes, offset, chunk.Length); + else if (ex is TransportSendException partial && partial.AcceptedCount < chunk.Length) + { + for (int index = 0; index < chunk.Length; index++) + { + var status = index < partial.AcceptedCount ? MessageSendStatus.Accepted + : index == partial.AcceptedCount ? MessageSendStatus.Unknown : MessageSendStatus.NotAttempted; + outcomes[offset + index] = outcomes[offset + index] with { Status = status }; + } + } + + throw new MessageSendException(outcomes, ex); + } + } + + return items; + } + + private static void RecordSent(DestinationAddress destination, IReadOnlyList items) + { + if (items.Count > 0) + MessagingInstruments.Sent.Add(items.Count, new KeyValuePair("destination", destination.Key)); + } + + private static void ApplyOutcomes(IReadOnlyList items, MessageSendOutcome[] outcomes, int offset, int count) + { + if (items.Count != count) throw new MessageBusException("Transport must report every input outcome."); + var seen = new bool[count]; + for (int position = 0; position < items.Count; position++) + { + int index = items[position].Index ?? position; + if (index < 0 || index >= count || seen[index] || !Enum.IsDefined(items[position].Status)) + throw new MessageBusException("Transport returned invalid or duplicate result indexes."); + seen[index] = true; + } + for (int position = 0; position < items.Count; position++) + { + var item = items[position]; + int index = offset + (item.Index ?? position); + outcomes[index] = outcomes[index] with { Status = item.Status, ErrorCode = item.ErrorCode, ErrorMessage = item.ErrorMessage, Retryable = item.Retryable }; + } + } + + private ISupportsPull RequirePull() + { + return _transport as ISupportsPull + ?? throw _exceptionFactory($"Transport \"{_transport.GetType().Name}\" does not support pull receive.", null); + } + + private void RemoveSource(DestinationAddress source, SourceListener listener) + { + _sources.TryRemove(new KeyValuePair(source, listener)); + } + + private void ThrowIfDisposed() + { + ObjectDisposedException.ThrowIf(Volatile.Read(ref _isDisposed) == 1, this); + } + + private sealed class ConsumerRegistration + { + public required string Key { get; init; } + public required ListenerConfig Config { get; init; } + public required Func Dispatch { get; init; } + public required bool IsCatchAll { get; init; } + public required string? TypeName { get; init; } + } + + // One receive loop per source. Consumers register by message type; the loop reads the message-type header and + // dispatches each entry to its exact-type consumer, one fallback for unmapped types, or HandleUnmatchedAsync. + // The loop runs while at least one consumer is attached and shuts down when the last one detaches. + private sealed class SourceListener + { + private readonly MessageClientCore _core; + private readonly DestinationAddress _source; + private readonly object _lock = new(); + private readonly CancellationTokenSource _cancellationTokenSource = new(); + private readonly ConcurrentDictionary _consumers = new(StringComparer.Ordinal); + private readonly ConcurrentDictionary _byType = new(StringComparer.Ordinal); + private readonly ConsumerGroup _catchAll = new(); + private int _maxConcurrency = 1; + private ListenerConfig? _endpoint; + private bool _ephemeral; + private Task? _loop; + private int _status = (int)MessageSubscriptionStatus.Starting; + private long _recoveryVersion; + public MessageSubscriptionStatus Status => (MessageSubscriptionStatus)Volatile.Read(ref _status); + public long RecoveryVersion => Interlocked.Read(ref _recoveryVersion); + + public async Task WaitUntilReadyAsync(CancellationToken cancellationToken) + { + while (Status != MessageSubscriptionStatus.Healthy) + { + if (Status == MessageSubscriptionStatus.Stopped) + throw new ObjectDisposedException(nameof(IMessageSubscription)); + await Task.Delay(TimeSpan.FromMilliseconds(50), cancellationToken).AnyContext(); + } + } + private bool _isDisposed; + + public SourceListener(MessageClientCore core, DestinationAddress source) + { + _core = core; + _source = source; + } + + public bool TryAddConsumer(ConsumerRegistration registration, out MessageListenerHandle handle, out bool created) + { + handle = null!; + created = false; + + lock (_lock) + { + if (_isDisposed) + return false; + + var group = GroupFor(registration); + if (_consumers.ContainsKey(registration.Key) || !group.IsEmpty) + throw new InvalidOperationException($"A handler for {registration.Config.MessageType.Name} is already registered on {_source}. Register one handler per concrete type and at most one interface/raw fallback per endpoint; use separate named subscriptions for independent event handlers."); + + int desired = Math.Max(1, registration.Config.MaxConcurrency); + if (_consumers.IsEmpty) + { + _maxConcurrency = desired; + _endpoint = registration.Config; + _ephemeral = registration.Config.Ephemeral; + created = true; + } + else if (desired != _maxConcurrency) + { + throw new InvalidOperationException($"Source \"{_source}\" is already consumed with MaxConcurrency {_maxConcurrency}; a conflicting MaxConcurrency {desired} was requested. Consumers sharing a destination must use the same MaxConcurrency."); + } + else if (_endpoint is { } endpoint && (endpoint.PrefetchCount != registration.Config.PrefetchCount + || endpoint.ReceiveBatchDelay != registration.Config.ReceiveBatchDelay + || endpoint.VisibilityTimeout != registration.Config.VisibilityTimeout + || endpoint.AutoRenewLock != registration.Config.AutoRenewLock + || endpoint.ShutdownTimeout != registration.Config.ShutdownTimeout)) + { + throw new InvalidOperationException($"Consumers sharing '{_source}' must agree on receive, lease, and shutdown settings."); + } + + handle = new MessageListenerHandle(_source, registration.Key, () => RemoveConsumerAsync(registration.Key), () => Status, () => RecoveryVersion, WaitUntilReadyAsync); + _consumers[registration.Key] = new Registered(registration, handle); + group.Add(registration); + + return true; + } + } + + private ConsumerGroup GroupFor(ConsumerRegistration registration) + { + return registration.IsCatchAll ? _catchAll : _byType.GetOrAdd(registration.TypeName!, _ => new ConsumerGroup()); + } + + public Task StartAsync(CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + if (_core._transport is not ISupportsPull && _core._transport is not ISupportsPush) + throw _core._exceptionFactory($"Transport {_core._transport.GetType().Name} does not support receiving messages.", null); + _loop = Task.Run(() => RunSupervisedAsync(_cancellationTokenSource.Token), CancellationToken.None); + return Task.CompletedTask; + } + + private async Task RunSupervisedAsync(CancellationToken cancellationToken) + { + bool recreate = false; + try + { + while (!cancellationToken.IsCancellationRequested) + { + using var receiving = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + Task? receiver = null; + Task? lease = null; + try + { + if (recreate) + await _core.EnsureAsync([new DestinationDeclaration { Address = _source, AutoDeleteAfter = _ephemeral ? TimeSpan.FromMinutes(2) : null }], cancellationToken).AnyContext(); + Volatile.Write(ref _status, (int)MessageSubscriptionStatus.Healthy); + receiver = RunReceiverAsync(receiving.Token); + lease = _ephemeral ? SuperviseSubscriptionAsync(receiving.Token) : Task.Delay(Timeout.Infinite, receiving.Token); + var completed = await Task.WhenAny(receiver, lease).AnyContext(); + await completed.AnyContext(); + cancellationToken.ThrowIfCancellationRequested(); + throw new InvalidOperationException($"Listener {_source} stopped unexpectedly."); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) { break; } + catch (Exception ex) + { + Volatile.Write(ref _status, (int)MessageSubscriptionStatus.Recovering); + Interlocked.Increment(ref _recoveryVersion); + _core._logger.LogWarning(ex, "Listener {Source} interrupted; recovering subscription", _source); + recreate = true; + } + finally + { + await receiving.CancelAsync().AnyContext(); + try { await Task.WhenAll(receiver ?? Task.CompletedTask, lease ?? Task.CompletedTask).AnyContext(); } + catch (Exception) { } + } + await _core._timeProvider.SafeDelay(TimeSpan.FromSeconds(1), cancellationToken).AnyContext(); + } + } + finally { Volatile.Write(ref _status, (int)MessageSubscriptionStatus.Stopped); } + } + + private async Task RunReceiverAsync(CancellationToken cancellationToken) + { + if (_core._transport is ISupportsPull pull) + { + await _core.RunPullLoopAsync(_source, pull, DispatchAsync, _endpoint!, cancellationToken, healthy => + { + int previous = Interlocked.Exchange(ref _status, (int)(healthy ? MessageSubscriptionStatus.Healthy : MessageSubscriptionStatus.Recovering)); + if (!healthy && previous == (int)MessageSubscriptionStatus.Healthy) + Interlocked.Increment(ref _recoveryVersion); + }).AnyContext(); + return; + } + await using var push = await ((ISupportsPush)_core._transport).SubscribeAsync(_source, + (entry, token) => _core.SafeProcessAsync(entry, DispatchAsync, _source, token, endpoint: _endpoint), + new PushOptions { MaxConcurrentMessages = _maxConcurrency }, cancellationToken).AnyContext(); + await Task.Delay(Timeout.Infinite, cancellationToken).AnyContext(); + } + + private async Task SuperviseSubscriptionAsync(CancellationToken cancellationToken) + { + var expires = _core._timeProvider.GetUtcNow().AddMinutes(2); + var delay = TimeSpan.FromSeconds(30); + while (!cancellationToken.IsCancellationRequested) + { + await Task.Delay(delay, _core._timeProvider, cancellationToken).AnyContext(); + var started = _core._timeProvider.GetUtcNow(); + var remaining = expires - started; + if (remaining <= TimeSpan.Zero) + throw new ReceiptExpiredException("The temporary subscription lease expired."); + try + { + using var timeout = new CancellationTokenSource(remaining < TimeSpan.FromSeconds(10) ? remaining : TimeSpan.FromSeconds(10), _core._timeProvider); + using var operation = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeout.Token); + bool renewed = await ((ISupportsEphemeralSubscriptions)_core._transport).RenewSubscriptionAsync(_source, TimeSpan.FromMinutes(2), operation.Token) + .WaitAsync(operation.Token).AnyContext(); + if (!renewed) + throw new ReceiptExpiredException("The temporary subscription lease was lost."); + expires = started.AddMinutes(2); + delay = TimeSpan.FromSeconds(30); + } + catch (ReceiptExpiredException) { throw; } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) { throw; } + catch (Exception ex) + { + _core._logger.LogWarning(ex, "Unable to renew temporary subscription {Source}; retrying within its lease", _source); + delay = TimeSpan.FromSeconds(1); + } + } + } + + public async ValueTask DisposeAsync() + { + lock (_lock) + { + if (_isDisposed) + return; + + _isDisposed = true; + } + + await ShutdownAsync().AnyContext(); + } + + private async ValueTask RemoveConsumerAsync(string key) + { + bool shutdown = false; + lock (_lock) + { + if (!_consumers.TryRemove(key, out var registered)) + return; + + var registration = registered.Registration; + if (registration.IsCatchAll) + { + _catchAll.Remove(registration); + } + else if (registration.TypeName is { } typeName && _byType.TryGetValue(typeName, out var group)) + { + group.Remove(registration); + if (group.IsEmpty) + _byType.TryRemove(new KeyValuePair(typeName, group)); + } + + if (_consumers.IsEmpty && !_isDisposed) + { + _isDisposed = true; + shutdown = true; + } + } + + if (shutdown) + await ShutdownAsync().AnyContext(); + } + + private async Task ShutdownAsync() + { + await _cancellationTokenSource.CancelAsync().AnyContext(); + + if (_loop is not null) + { + try + { + await _loop.AnyContext(); + } + catch (OperationCanceledException) { } + } + + _cancellationTokenSource.Dispose(); + _core.RemoveSource(_source, this); + if (_ephemeral && _core._transport is ISupportsProvisioning provisioning) + { + using var cleanup = new CancellationTokenSource(TimeSpan.FromSeconds(5), _core._timeProvider); + try { await provisioning.DeleteAsync(_source, cleanup.Token).WaitAsync(cleanup.Token).AnyContext(); } + catch (Exception ex) { _core._logger.LogWarning(ex, "Unable to remove temporary subscription {Source}; its lease will expire", _source); } + } + } + + private async Task DispatchAsync(TransportEntry entry, MessageDeliveryLease lease, CancellationToken token) + { + if (entry.EnvelopeError is { } error) + { + await _core.DeadLetterPoisonMessageAsync(entry, "invalid-envelope", error, token).AnyContext(); + return; + } + var registration = Resolve(entry); + if (registration is null) + { + await _core.HandleUnmatchedAsync(entry, _source, token).AnyContext(); + return; + } + + await registration.Dispatch(entry, lease, token).AnyContext(); + } + + private ConsumerRegistration? Resolve(TransportEntry entry) + { + string? typeName = entry.Headers.GetValueOrDefault(KnownHeaders.MessageType); + if (typeName is not null && _byType.TryGetValue(typeName, out var group) && group.Next() is { } typed) + return typed; + + return _catchAll.Next(); + } + + private sealed record Registered(ConsumerRegistration Registration, MessageListenerHandle Handle); + + private sealed class ConsumerGroup + { + private ConsumerRegistration? _registration; + public bool IsEmpty => Volatile.Read(ref _registration) is null; + public void Add(ConsumerRegistration registration) => Volatile.Write(ref _registration, registration); + public void Remove(ConsumerRegistration registration) => Interlocked.CompareExchange(ref _registration, null, registration); + public ConsumerRegistration? Next() => Volatile.Read(ref _registration); + } + + } +} + +internal class MessageContext : IMessageContext +{ + private readonly IMessageTransport _transport; + private readonly TransportEntry _entry; + private readonly IScheduledDispatchStore? _runtimeStore; + private readonly TimeProvider _timeProvider; + private readonly string? _deadLetterDestination; + private readonly ILogger _logger; + private readonly TopologyMode _topologyMode; + private int _isHandled; + private TaskCompletionSource? _settled; + private readonly MessageDeliveryLease? _lease; + + public MessageContext(IMessageTransport transport, TransportEntry entry, CancellationToken cancellationToken, IScheduledDispatchStore? runtimeStore = null, TimeProvider? timeProvider = null, string? deadLetterDestination = null, ILogger? logger = null, TopologyMode topologyMode = TopologyMode.Ensure, MessageDeliveryLease? lease = null) + { + _lease = lease; + _transport = transport; + _entry = entry; + _runtimeStore = runtimeStore; + _timeProvider = timeProvider ?? TimeProvider.System; + _deadLetterDestination = deadLetterDestination; + _logger = logger ?? Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance; + _topologyMode = topologyMode; + CancellationToken = cancellationToken; + } + + public string Id => _entry.ApplicationMessageId ?? _entry.Headers.GetValueOrDefault(KnownHeaders.MessageId) ?? _entry.Id; + public DestinationAddress Destination => _entry.Destination; + public DateTimeOffset? EnqueuedUtc => _entry.EnqueuedUtc; + public bool IsLeaseLost => _lease?.IsLost == true; + public string BrokerMessageId => _entry.Id; + public ReadOnlyMemory Body => _entry.Body; + public MessageHeaders Headers => _entry.Headers; + public string? CorrelationId => Headers.GetValueOrDefault(KnownHeaders.CorrelationId); + public string? MessageType => Headers.GetValueOrDefault(KnownHeaders.MessageType); + public MessagePriority Priority => Enum.TryParse(Headers.GetValueOrDefault(KnownHeaders.Priority), ignoreCase: true, out MessagePriority priority) ? priority : MessagePriority.Normal; + + // Reconcile the transport-reported delivery count with the message.attempts header. When redelivery-delay is + // served through the runtime-store fallback (transports without native ISupportsRedeliveryDelay), the message is + // re-sent as a brand-new transport message, so its DeliveryCount resets to 1; the carried-over attempt count + // lives in the header. Taking the max keeps MaxAttempts/dead-letter correct regardless of whether the transport + // honors the header, so the counter never silently resets and redelivery can't loop forever. + public int Attempts => Math.Max(_entry.DeliveryCount, ParseAttemptsHeader(_entry.Headers)); + public bool IsHandled => Volatile.Read(ref _isHandled) == 2; + public CancellationToken CancellationToken { get; } + + public async Task CompleteAsync(CancellationToken cancellationToken = default) + { + if (cancellationToken == default) cancellationToken = CancellationToken; + if (IsHandled) + return; + CancellationToken.ThrowIfCancellationRequested(); + if (!TryBeginSettlement()) + return; + try + { + await _transport.CompleteAsync(_entry, cancellationToken).AnyContext(); + Volatile.Write(ref _isHandled, 2); + _lease?.Settled(); + Volatile.Read(ref _settled)?.TrySetResult(); + MessagingInstruments.Completed.Add(1, new KeyValuePair("source", _entry.Destination.Key)); + } + catch + { + Volatile.Write(ref _isHandled, 0); + throw; + } + } + + public async Task RejectAsync(RejectOptions? options = null, CancellationToken cancellationToken = default) + { + if (cancellationToken == default) cancellationToken = CancellationToken; + if (IsHandled) + return; + CancellationToken.ThrowIfCancellationRequested(); + if (!TryBeginSettlement()) + return; + try + { + await RejectCoreAsync(options, cancellationToken).AnyContext(); + Volatile.Write(ref _isHandled, 2); + _lease?.Settled(); + Volatile.Read(ref _settled)?.TrySetResult(); + var counter = options?.Terminal == true ? MessagingInstruments.DeadLettered : MessagingInstruments.Abandoned; + counter.Add(1, new KeyValuePair("source", _entry.Destination.Key)); + } + catch + { + Volatile.Write(ref _isHandled, 0); + throw; + } + } + + private async Task RejectCoreAsync(RejectOptions? options, CancellationToken cancellationToken) + { + options ??= new RejectOptions(); + + if (options.Terminal) + { + var enriched = _entry with { Headers = BuildDeadLetterHeaders(_entry, Attempts, options.Exception, _timeProvider) }; + await DeadLetterAsync(_transport, enriched, options.Reason, _deadLetterDestination, _logger, cancellationToken, _topologyMode).AnyContext(); + return; + } + + if (options.RedeliveryDelay is not { } redeliveryDelay || redeliveryDelay <= TimeSpan.Zero) + { + await _transport.AbandonAsync(_entry, cancellationToken).AnyContext(); + return; + } + + // Honor an explicit redelivery delay natively when the transport can (within its advertised maximum); otherwise + // re-schedule the message through the runtime store and complete the original so the delay survives transports + // without native delayed redelivery. + if (_transport is ISupportsRedeliveryDelay redelivery && (redelivery.MaxRedeliveryDelay is not { } max || redeliveryDelay <= max)) + { + await redelivery.AbandonAsync(_entry, redeliveryDelay, cancellationToken).AnyContext(); + return; + } + + // The runtime-store fallback re-sends the message as a plain queue send, which only makes sense for a + // queue-channel entry: a subscription-channel entry would need to be re-sent into its subscription group, and + // a queue send to that address would land where no subscription group reads. + bool isSubscriptionSource = _entry.Destination.Role == DestinationRole.Subscription; + if (_runtimeStore is null || isSubscriptionSource) + { + // A best-effort delay (the core retry policy) degrades to immediate redelivery; an explicit caller delay + // stays strict because the caller is depending on the timing. + if (options.BestEffortDelay) + { + await _transport.AbandonAsync(_entry, cancellationToken).AnyContext(); + return; + } + + throw new MessageBusException($"Delayed redelivery of \"{_entry.Destination.Key}\" requires native redelivery-delay support from transport \"{_transport.GetType().Name}\" (within its supported maximum){(isSubscriptionSource ? "" : " or a registered job runtime store")}."); + } + + // Advance from the reconciled attempt count, not the raw transport DeliveryCount: the re-send produces a new + // transport message whose native DeliveryCount resets to 1, so basing the next attempt on DeliveryCount would + // pin it at 2 and redeliver forever. Attempts already takes the max of DeliveryCount and the carried header. + int nextAttempt = Attempts + 1; + var headers = _entry.Headers.ToBuilder() + .Set(KnownHeaders.Attempts, nextAttempt.ToString(CultureInfo.InvariantCulture)) + .Build(); + + await _runtimeStore.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = $"{_entry.Id}:retry:{nextAttempt}", + Kind = ScheduledDispatchKind.QueueMessage, + Destination = _entry.Destination, + Body = _entry.Body, + Headers = headers, + Options = new TransportSendOptions { Priority = Priority }, + DueUtc = _timeProvider.GetUtcNow().Add(redeliveryDelay) + }, cancellationToken).AnyContext(); + + await _transport.CompleteAsync(_entry, cancellationToken).AnyContext(); + } + + public Task RenewLockAsync(TimeSpan? duration = null, CancellationToken cancellationToken = default) + { + if (_lease is not null) + return _lease.RenewAsync(duration, cancellationToken == default ? CancellationToken : cancellationToken); + return _transport is ISupportsLockRenewal lockRenewal + ? lockRenewal.RenewLockAsync(_entry, duration, cancellationToken == default ? CancellationToken : cancellationToken) + : throw new NotSupportedException($"Transport \"{_transport.GetType().Name}\" does not support lock renewal."); + } + + internal static async Task DeadLetterAsync(IMessageTransport transport, TransportEntry entry, string? reason, string? deadLetterDestination, ILogger logger, CancellationToken cancellationToken, TopologyMode topologyMode = TopologyMode.Ensure) + { + if (transport is ISupportsDeadLetterSink deadLetter) + { + await deadLetter.DeadLetterAsync(entry, reason, cancellationToken).AnyContext(); + return; + } + + var destination = !String.IsNullOrEmpty(deadLetterDestination) + ? DestinationAddress.ForQueue(deadLetterDestination) + : DestinationAddress.ForQueue($"{entry.Destination.Key}.deadletter"); + if (topologyMode == TopologyMode.Validate && transport is not ISupportsProvisioning) + throw new NotSupportedException("Dead-letter topology validation requires a provisioning-capable transport."); + if (transport is ISupportsProvisioning provisioning) + { + if (topologyMode == TopologyMode.Ensure) + await provisioning.EnsureAsync([new DestinationDeclaration { Address = destination }], cancellationToken).AnyContext(); + else if (topologyMode == TopologyMode.Validate && !await provisioning.ExistsAsync(destination, cancellationToken).AnyContext()) + throw new MessageBusException($"Dead-letter destination {destination} does not exist. Provision it before using Validate mode."); + } + var headers = String.IsNullOrEmpty(reason) + ? entry.Headers + : entry.Headers.ToBuilder().Set(KnownHeaders.DeadLetterReason, reason).Build(); + + try + { + var result = await transport.SendAsync(destination, [new TransportMessage { Body = entry.Body, Headers = headers, MessageId = entry.ApplicationMessageId ?? entry.Headers.GetValueOrDefault(KnownHeaders.MessageId) ?? entry.Id, ContentType = entry.ContentType ?? entry.Headers.GetValueOrDefault(KnownHeaders.ContentType) }], new TransportSendOptions(), cancellationToken).AnyContext(); + result.EnsureAccepted(1); + } + catch (Exception ex) + { + logger.LogError(ex, "Failed to park dead-lettered message \"{MessageId}\" at \"{Destination}\"; original delivery remains unsettled: {Message}", entry.Id, destination.Key, ex.Message); + throw; + } + + await transport.CompleteAsync(entry, cancellationToken).AnyContext(); + } + + // Stamps the dead-letter forensics contract (see KnownHeaders) so a dead message is triageable — exception details, + // reconciled attempt count, where it was consumed from, and when it died. The attempt count goes in a forensics + // header (never message.attempts) so a replayed message starts with a fresh retry budget. + internal static MessageHeaders BuildDeadLetterHeaders(TransportEntry entry, int attempts, Exception? exception, TimeProvider timeProvider) + { + var headers = entry.Headers.ToBuilder() + .Set(KnownHeaders.DeadLetterAttempts, attempts.ToString(CultureInfo.InvariantCulture)) + .Set(KnownHeaders.DeadLetterFailedAt, timeProvider.GetUtcNow().ToString("O", CultureInfo.InvariantCulture)) + .Set(KnownHeaders.DeadLetterOriginalDestination, entry.Destination.Key); + + if (exception is not null) + { + headers.Set(KnownHeaders.DeadLetterExceptionType, exception.GetType().FullName ?? exception.GetType().Name); + headers.Set(KnownHeaders.DeadLetterExceptionMessage, Truncate(exception.Message, 1024)); + if (exception.StackTrace is { } stack) + headers.Set(KnownHeaders.DeadLetterExceptionStackTrace, Truncate(stack, 4096)); + } + else + { + // A death with no exception (no-handler, unresolved-type) must not carry stale forensics from a previous one. + headers.Remove(KnownHeaders.DeadLetterExceptionType); + headers.Remove(KnownHeaders.DeadLetterExceptionMessage); + headers.Remove(KnownHeaders.DeadLetterExceptionStackTrace); + } + + return headers.Build(); + } + + private static string Truncate(string value, int maxLength) + { + return value.Length <= maxLength ? value : value[..maxLength]; + } + + private bool TryBeginSettlement() + { + int state = Interlocked.CompareExchange(ref _isHandled, 1, 0); + if (state == 1) + throw new InvalidOperationException("A settlement operation is already in progress for this message."); + return state == 0; + } + + internal Task WaitForSettlementAsync(CancellationToken cancellationToken) + { + if (IsHandled) + return Task.CompletedTask; + + var completion = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + completion = Interlocked.CompareExchange(ref _settled, completion, null) ?? completion; + if (IsHandled) + completion.TrySetResult(); + return completion.Task.WaitAsync(cancellationToken); + } + + private static int ParseAttemptsHeader(MessageHeaders headers) + { + return Int32.TryParse(headers.GetValueOrDefault(KnownHeaders.Attempts), NumberStyles.Integer, CultureInfo.InvariantCulture, out int attempts) && attempts > 0 + ? attempts + : 0; + } +} + +internal sealed class MessageContext : MessageContext, IMessageContext where T : class +{ + public MessageContext(IMessageTransport transport, TransportEntry entry, T message, CancellationToken cancellationToken, IScheduledDispatchStore? runtimeStore = null, TimeProvider? timeProvider = null, string? deadLetterDestination = null, ILogger? logger = null, TopologyMode topologyMode = TopologyMode.Ensure, MessageDeliveryLease? lease = null) + : base(transport, entry, cancellationToken, runtimeStore, timeProvider, deadLetterDestination, logger, topologyMode, lease) + { + Message = message; + } + + public T Message { get; } +} + +internal static class MessageRoutingConventions +{ + public static string ToKebabCase(string value) + { + if (String.IsNullOrEmpty(value)) + return value; + + Span buffer = stackalloc char[value.Length * 2]; + int position = 0; + for (int index = 0; index < value.Length; index++) + { + char current = value[index]; + if (Char.IsUpper(current)) + { + if (index > 0) + buffer[position++] = '-'; + + buffer[position++] = Char.ToLowerInvariant(current); + } + else + { + buffer[position++] = current; + } + } + + return new String(buffer[..position]); + } +} + +/// +/// A started listener handle for one channel (a send destination or a topic subscription); the bus composes one per +/// channel into the it returns. +/// +internal sealed class MessageListenerHandle : IMessageSubscription +{ + private readonly Func _dispose; + private readonly Func _status; + private readonly Func _recoveryVersion; + private readonly Func _waitUntilReady; + private int _isDisposed; + + public MessageListenerHandle(DestinationAddress source, string key, Func dispose, Func status, Func recoveryVersion, Func waitUntilReady) + { + _status = status; + _recoveryVersion = recoveryVersion; + _waitUntilReady = waitUntilReady; + Source = source; + Key = key; + _dispose = dispose; + } + + public DestinationAddress Source { get; } + public MessageSubscriptionStatus Status => Volatile.Read(ref _isDisposed) == 1 ? MessageSubscriptionStatus.Stopped : _status(); + public long RecoveryVersion => _recoveryVersion(); + public Task WaitUntilReadyAsync(CancellationToken cancellationToken = default) => _waitUntilReady(cancellationToken); + public string Topic => Source.Topic ?? ""; + public string Subscription => Source.Role == DestinationRole.Subscription ? Source.Name : ""; + public string Key { get; } + + // Disposing a single consumer handle detaches just that consumer from its source listener; the underlying receive + // loop keeps running until its last consumer detaches. + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _isDisposed, 1) == 1) + return; + + await _dispose().AnyContext(); + } +} diff --git a/src/Foundatio/Messaging/MessageDeliveryLease.cs b/src/Foundatio/Messaging/MessageDeliveryLease.cs new file mode 100644 index 000000000..604c270d0 --- /dev/null +++ b/src/Foundatio/Messaging/MessageDeliveryLease.cs @@ -0,0 +1,173 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Utility; +using Microsoft.Extensions.Logging; + +namespace Foundatio.Messaging; + +/// Coordinates automatic and explicit renewal against one conservative delivery deadline. +internal sealed class MessageDeliveryLease : IAsyncDisposable +{ + private readonly IMessageTransport _transport; + private readonly TransportEntry _entry; + private readonly TimeSpan _duration; + private readonly TimeProvider _time; + private readonly ILogger _logger; + private readonly CancellationTokenSource _processing; + private readonly object _monitorGate = new(); + private readonly ITimer? _firstCheck; + private readonly bool _autoRenew; + private CancellationTokenSource? _renewal; + private Task _completion = Task.CompletedTask; + private Task _stopping = Task.CompletedTask; + private bool _stopped; + private SemaphoreSlim? _gate; + private long _expiresTicks; + private int _lost; + private int _settled; + + public MessageDeliveryLease(IMessageTransport transport, TransportEntry entry, TimeSpan duration, bool autoRenew, + TimeProvider time, ILogger logger, CancellationTokenSource processing) + { + _transport = transport; + _entry = entry; + _duration = transport is ISupportsVisibilityTimeout { MaxVisibilityTimeout: { } maximum } && duration > maximum ? maximum : duration; + _time = time; + _logger = logger; + _processing = processing; + _expiresTicks = entry.LockExpiresUtc?.UtcTicks ?? DateTimeOffset.MaxValue.UtcTicks; + _autoRenew = autoRenew; + if (entry.LockExpiresUtc is not null) + { + // Most deliveries settle before their first lease check. A timer keeps supervision active + // even for a blocking handler, without starting an async loop for every short delivery. + _firstCheck = time.CreateTimer(static state => ((MessageDeliveryLease)state!).StartMonitor(), this, + Timeout.InfiniteTimeSpan, Timeout.InfiniteTimeSpan); + var delay = autoRenew && transport is ISupportsLockRenewal ? Remaining / 2 : Remaining; + _firstCheck.Change(delay > TimeSpan.Zero ? delay : TimeSpan.Zero, Timeout.InfiniteTimeSpan); + } + } + + public Task Completion { get { lock (_monitorGate) return _completion; } } + public bool IsLost => Volatile.Read(ref _lost) != 0 || !IsSettled && Remaining <= TimeSpan.Zero; + public bool IsSettled => Volatile.Read(ref _settled) != 0; + private TimeSpan Remaining => new(Interlocked.Read(ref _expiresTicks) - _time.GetUtcNow().UtcTicks); + + public void Settled() + { + Interlocked.Exchange(ref _settled, 1); + StopMonitoring(); + } + + private void StopMonitoring() + { + lock (_monitorGate) + { + if (_stopped) return; + _stopped = true; + _firstCheck?.Dispose(); + _stopping = _renewal?.CancelAsync() ?? Task.CompletedTask; + } + } + + private void StartMonitor() + { + TaskCompletionSource completion; + CancellationToken token; + lock (_monitorGate) + { + if (_stopped) return; + _renewal = new CancellationTokenSource(); + token = _renewal.Token; + completion = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + _completion = completion.Task; + } + _ = RunMonitorAsync(completion, token); + } + + private async Task RunMonitorAsync(TaskCompletionSource completion, CancellationToken token) + { + try + { + await MonitorAsync(token).AnyContext(); + completion.TrySetResult(); + } + catch (Exception exception) { completion.TrySetException(exception); } + } + + public async Task RenewAsync(TimeSpan? duration, CancellationToken cancellationToken) + { + var extension = duration ?? _duration; + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(extension, TimeSpan.Zero); + if (_transport is ISupportsVisibilityTimeout { MaxVisibilityTimeout: { } maximum } && extension > maximum) + throw new ArgumentOutOfRangeException(nameof(duration), $"The transport supports a maximum lease of {maximum}."); + if (_transport is not ISupportsLockRenewal renewal) + throw new NotSupportedException($"Transport {_transport.GetType().Name} does not support delivery lease renewal."); + var gate = LazyInitializer.EnsureInitialized(ref _gate, static () => new SemaphoreSlim(1)); + await gate.WaitAsync(cancellationToken).AnyContext(); + try + { + if (IsSettled || IsLost || Remaining <= TimeSpan.Zero) + throw new ReceiptExpiredException(); + var remaining = Remaining; + var started = _time.GetUtcNow(); + using var deadline = new CancellationTokenSource(remaining > TimeSpan.FromSeconds(30) ? TimeSpan.FromSeconds(30) : remaining, _time); + using var operation = CancellationTokenSource.CreateLinkedTokenSource(deadline.Token, cancellationToken); + await renewal.RenewLockAsync(_entry, extension, operation.Token).WaitAsync(operation.Token).AnyContext(); + Interlocked.Exchange(ref _expiresTicks, started.Add(extension).UtcTicks); + } + finally { gate.Release(); } + } + + private async Task MonitorAsync(CancellationToken token) + { + bool retry = false; + bool firstCheck = true; + try + { + while (!token.IsCancellationRequested) + { + var remaining = Remaining; + if (remaining <= TimeSpan.Zero) break; + bool canRenew = _autoRenew && _transport is ISupportsLockRenewal; + var delay = !canRenew ? remaining : retry + ? TimeSpan.FromTicks(Math.Min(TimeSpan.TicksPerSecond, remaining.Ticks / 4)) : remaining / 2; + if (!firstCheck) + await Task.Delay(delay, _time, token).ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing); + firstCheck = false; + if (token.IsCancellationRequested) return; + if (Remaining <= TimeSpan.Zero) break; + if (!canRenew) continue; + try + { + await RenewAsync(_duration, token).AnyContext(); + retry = false; + } + catch (ReceiptExpiredException) { break; } + catch (OperationCanceledException) when (token.IsCancellationRequested) { return; } + catch (Exception exception) + { + retry = true; + _logger.LogWarning(exception, "Unable to renew delivery {MessageId} at {Source}; retrying within its lease", _entry.Id, _entry.Destination); + } + } + if (!token.IsCancellationRequested && !IsSettled) + { + Interlocked.Exchange(ref _lost, 1); + _logger.LogWarning("Delivery lease lost for {MessageId} at {Source}; cancelling processing", _entry.Id, _entry.Destination); + await _processing.CancelAsync().AnyContext(); + } + } + catch (OperationCanceledException) when (token.IsCancellationRequested) { } + } + + public async ValueTask DisposeAsync() + { + StopMonitoring(); + await _stopping.AnyContext(); + await Completion.AnyContext(); + _renewal?.Dispose(); + // Explicit renewal may still be unwinding after settlement. SemaphoreSlim owns no wait handle here. + } +} diff --git a/src/Foundatio/Messaging/MessageDestinationNotFoundException.cs b/src/Foundatio/Messaging/MessageDestinationNotFoundException.cs new file mode 100644 index 000000000..5e8cb0465 --- /dev/null +++ b/src/Foundatio/Messaging/MessageDestinationNotFoundException.cs @@ -0,0 +1,10 @@ +using System; + +namespace Foundatio.Messaging; + +/// A destination disappeared. Ensure-mode listeners can recreate it under supervision. +public sealed class MessageDestinationNotFoundException(DestinationAddress destination, Exception innerException) + : MessageBusException($"Message destination {destination} no longer exists.", innerException) +{ + public DestinationAddress Destination { get; } = destination; +} diff --git a/src/Foundatio/Messaging/MessageHandlerRegistration.cs b/src/Foundatio/Messaging/MessageHandlerRegistration.cs new file mode 100644 index 000000000..2bd5c728c --- /dev/null +++ b/src/Foundatio/Messaging/MessageHandlerRegistration.cs @@ -0,0 +1,20 @@ +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Messaging; + +/// +/// One declarative message-handler registration: a description for logging and a factory that starts the underlying +/// queue consumer or pub/sub subscription and returns it for disposal on shutdown. Built by the consumer/subscriber +/// builder methods, which bind the message type at compile time (one registration per delivery verb). +/// +internal sealed class MessageHandlerRegistration +{ + public required string Description { get; init; } + public required Func> StartAsync { get; init; } +} + +/// The DI-selected , applied at startup and by the message clients on use. +/// The effective topology policy selected for this messaging client. +public sealed record MessagingTopologyOptions(TopologyMode Mode); diff --git a/src/Foundatio/Messaging/MessageHeaders.cs b/src/Foundatio/Messaging/MessageHeaders.cs new file mode 100644 index 000000000..6216dc437 --- /dev/null +++ b/src/Foundatio/Messaging/MessageHeaders.cs @@ -0,0 +1,157 @@ +using System; +using System.Collections; +using System.Collections.Generic; +using System.Text.Json; + +namespace Foundatio.Messaging; + +public sealed class MessageHeaders : IReadOnlyDictionary +{ + public static MessageHeaders Empty { get; } = new(new Dictionary(StringComparer.OrdinalIgnoreCase)); + + private readonly Dictionary _headers; + + private MessageHeaders(Dictionary headers) + { + _headers = headers; + } + + public string this[string key] => _headers[key]; + public IEnumerable Keys => _headers.Keys; + public IEnumerable Values => _headers.Values; + public int Count => _headers.Count; + + public static MessageHeaders Create(IEnumerable> headers) + { + ArgumentNullException.ThrowIfNull(headers); + + if (headers is MessageHeaders messageHeaders) + return messageHeaders; + + var values = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var header in headers) + { + ArgumentException.ThrowIfNullOrEmpty(header.Key); + ArgumentNullException.ThrowIfNull(header.Value); + values[header.Key] = header.Value; + } + + return values.Count == 0 + ? Empty + : new MessageHeaders(values); + } + + /// + /// The canonical on-the-wire encoding for headers (a JSON object). Transports should use this rather than rolling + /// their own so the round-trip semantics — notably case-insensitive keys (see ) — are identical + /// and contractually guaranteed across providers. + /// + public static string SerializeToJson(MessageHeaders headers) + { + ArgumentNullException.ThrowIfNull(headers); + return JsonSerializer.Serialize(headers._headers); + } + + /// Reads headers from the canonical encoding produced by . + public static MessageHeaders DeserializeFromJson(string? json) + { + if (String.IsNullOrEmpty(json)) + return Empty; + var map = JsonSerializer.Deserialize>(json); + return map is null ? Empty : Create(map); + } + + public bool ContainsKey(string key) + { + return _headers.ContainsKey(key); + } + + public bool TryGetValue(string key, out string value) + { + return _headers.TryGetValue(key, out value!); + } + + public string? GetValueOrDefault(string key) + { + return _headers.GetValueOrDefault(key); + } + + public Builder ToBuilder() + { + return new Builder(this); + } + + public IEnumerator> GetEnumerator() + { + return _headers.GetEnumerator(); + } + + IEnumerator IEnumerable.GetEnumerator() + { + return GetEnumerator(); + } + + public sealed class Builder + { + private Dictionary _headers; + private MessageHeaders? _snapshot; + + internal Builder(MessageHeaders headers) + { + _headers = headers._headers; + _snapshot = headers; + } + + public Builder Add(string key, string value) + { + ArgumentException.ThrowIfNullOrEmpty(key); + ArgumentNullException.ThrowIfNull(value); + EnsureWritable(); + _headers.Add(key, value); + return this; + } + + public Builder Set(string key, string value) + { + ArgumentException.ThrowIfNullOrEmpty(key); + ArgumentNullException.ThrowIfNull(value); + EnsureWritable(); + _headers[key] = value; + return this; + } + + public Builder SetIfMissing(string key, string value) + { + ArgumentException.ThrowIfNullOrEmpty(key); + ArgumentNullException.ThrowIfNull(value); + if (!_headers.ContainsKey(key)) + { + EnsureWritable(); + _headers.Add(key, value); + } + return this; + } + + public bool Remove(string key) + { + ArgumentException.ThrowIfNullOrEmpty(key); + if (!_headers.ContainsKey(key)) + return false; + EnsureWritable(); + return _headers.Remove(key); + } + + public MessageHeaders Build() + { + return _headers.Count == 0 ? Empty : _snapshot ??= new MessageHeaders(_headers); + } + + private void EnsureWritable() + { + if (_snapshot is null) + return; + _headers = new Dictionary(_headers, StringComparer.OrdinalIgnoreCase); + _snapshot = null; + } + } +} diff --git a/src/Foundatio/Messaging/MessageNodeSubscription.cs b/src/Foundatio/Messaging/MessageNodeSubscription.cs new file mode 100644 index 000000000..d79780f13 --- /dev/null +++ b/src/Foundatio/Messaging/MessageNodeSubscription.cs @@ -0,0 +1,59 @@ +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Messaging; + +/// A best-effort subscription for one running node, independent of durable service subscriptions. +public sealed record MessageNodeSubscriptionOptions +{ + /// Topic broadcast to the running nodes. + public required string Topic { get; init; } + /// Diagnostic node identity; providers may add a unique resource suffix. + public string NodeId { get; init; } = Guid.NewGuid().ToString("N"); + /// Maximum callbacks running concurrently on this node. + public int MaxConcurrency { get; init; } = 10; + /// Interval for renewing managed node ownership. + public TimeSpan HeartbeatInterval { get; init; } = TimeSpan.FromMinutes(2); + /// Time without a heartbeat before another node may clean up resources. + public TimeSpan StaleAfter { get; init; } = TimeSpan.FromMinutes(10); + /// Maximum retained backlog on managed node resources. + public TimeSpan MessageRetention { get; init; } = TimeSpan.FromMinutes(5); + + /// Rejects invalid lifecycle and receiving settings. + public void Validate() + { + ArgumentException.ThrowIfNullOrWhiteSpace(Topic); + ArgumentException.ThrowIfNullOrWhiteSpace(NodeId); + ArgumentOutOfRangeException.ThrowIfLessThan(MaxConcurrency, 1); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(HeartbeatInterval, TimeSpan.Zero); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(StaleAfter, HeartbeatInterval); + } +} + +/// A provider-managed node subscription. Disposal removes its infrastructure. +public interface IManagedNodeSubscription : IAsyncDisposable +{ + /// Native subscription destination owned by this node. + DestinationAddress Source { get; } +} + +/// Manages node resources on brokers without natively expiring subscriptions. Does not advertise native expiration. +public interface ISupportsManagedNodeSubscriptions : IMessageTransport +{ + /// Creates a node subscription and starts renewing its managed ownership. + Task OpenNodeSubscriptionAsync(MessageNodeSubscriptionOptions options, CancellationToken cancellationToken = default); +} + +internal sealed class NodeMessageSubscription(IMessageSubscription consumer, IManagedNodeSubscription owner) : IMessageSubscription +{ + public DestinationAddress Source => consumer.Source; + public MessageSubscriptionStatus Status => consumer.Status; + public long RecoveryVersion => consumer.RecoveryVersion; + public Task WaitUntilReadyAsync(CancellationToken cancellationToken = default) => consumer.WaitUntilReadyAsync(cancellationToken); + public async ValueTask DisposeAsync() + { + try { await consumer.DisposeAsync().ConfigureAwait(false); } + finally { await owner.DisposeAsync().ConfigureAwait(false); } + } +} diff --git a/src/Foundatio/Messaging/MessageOutcome.cs b/src/Foundatio/Messaging/MessageOutcome.cs new file mode 100644 index 000000000..283157275 --- /dev/null +++ b/src/Foundatio/Messaging/MessageOutcome.cs @@ -0,0 +1,55 @@ +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Messaging; + +/// A handler's delivery decision. Returning an expected failure does not require throwing an exception. +public readonly record struct MessageOutcome +{ + private MessageOutcome(MessageOutcomeKind kind, string? reason, Exception? exception) + { + Kind = kind; + Reason = reason; + Exception = exception; + } + + /// The requested settlement. An already confirmed explicit settlement takes precedence. + public MessageOutcomeKind Kind { get; } + /// A diagnostic reason retained when dead-lettering. + public string? Reason { get; } + /// Optional exception retained as dead-letter evidence. + public Exception? Exception { get; } + /// Processing succeeded; automatic acknowledgement applies. + public static MessageOutcome Success => default; + /// Leave the delivery unsettled for its lease to expire. + public static MessageOutcome Unsettled => new(MessageOutcomeKind.Unsettled, null, null); + /// Retry using the endpoint policy, or dead-letter when its attempt budget is exhausted. + public static MessageOutcome Retry(string? reason = null, Exception? exception = null) => new(MessageOutcomeKind.Retry, reason, exception); + /// Dead-letter immediately with the supplied reason. + public static MessageOutcome DeadLetter(string reason, Exception? exception = null) => new(MessageOutcomeKind.DeadLetter, reason, exception); + + internal Task SettleFailureAsync(IMessageContext context, int maxAttempts, Func? backoff, CancellationToken cancellationToken) + { + if (context.IsHandled) + return Task.CompletedTask; + bool terminal = Kind == MessageOutcomeKind.DeadLetter || maxAttempts >= 0 && context.Attempts >= maxAttempts; + return context.RejectAsync(new RejectOptions + { + Terminal = terminal, + Reason = Reason, + Exception = Exception, + RedeliveryDelay = terminal ? null : backoff?.Invoke(context.Attempts), + BestEffortDelay = true + }, cancellationToken); + } +} + +/// The delivery decision returned by an outcome handler. +public enum MessageOutcomeKind +{ + Success, + Retry, + DeadLetter, + Unsettled +} diff --git a/src/Foundatio/Messaging/MessageRouteAttribute.cs b/src/Foundatio/Messaging/MessageRouteAttribute.cs new file mode 100644 index 000000000..fe90fd0e8 --- /dev/null +++ b/src/Foundatio/Messaging/MessageRouteAttribute.cs @@ -0,0 +1,20 @@ +using System; + +namespace Foundatio.Messaging; + +[AttributeUsage(AttributeTargets.Class | AttributeTargets.Struct, Inherited = false)] +public sealed class MessageRouteAttribute : Attribute +{ + public MessageRouteAttribute() + { + } + + public MessageRouteAttribute(string name) + { + Destination = name; + Topic = name; + } + + public string? Destination { get; set; } + public string? Topic { get; set; } +} diff --git a/src/Foundatio/Messaging/MessageRouting.cs b/src/Foundatio/Messaging/MessageRouting.cs new file mode 100644 index 000000000..3c6854fe5 --- /dev/null +++ b/src/Foundatio/Messaging/MessageRouting.cs @@ -0,0 +1,243 @@ +using System; +using System.Collections.Generic; +using System.Collections.Concurrent; +using System.Linq; +using System.Reflection; + +namespace Foundatio.Messaging; + +public enum MessageRouteRole +{ + QueueDestination, + PubSubTopic +} + +public sealed record MessageRouteContext +{ + public required Type MessageType { get; init; } + public required MessageRouteRole Role { get; init; } + public string? OperationOverride { get; init; } +} + +public interface IMessageRouter +{ + string ResolveRoute(MessageRouteContext context); +} + +public sealed record MessageRouteMap +{ + public required Type MessageType { get; init; } + public required MessageRouteRole Role { get; init; } + public required string Route { get; init; } +} + +public sealed class MessageRoutingOptions +{ + internal List RouteMaps { get; } = []; + internal List TopologyDeclarations { get; } = []; + + public string? DefaultQueueDestination { get; set; } + public string? DefaultPubSubTopic { get; set; } + public Func? Convention { get; set; } + + /// Returns the declared type-to-route mappings for configuration diagnostics. + public IReadOnlyList GetRouteMaps() => RouteMaps.ToArray(); + + public IReadOnlyList GetTopologyDeclarations() + { + return TopologyDeclarations.ToArray(); + } + + internal void Declare(DestinationDeclaration declaration) + { + ArgumentNullException.ThrowIfNull(declaration); + + if (!TopologyDeclarations.Any(d => d.Address == declaration.Address)) + TopologyDeclarations.Add(declaration); + } + + internal void RemoveDeclarations(Predicate match) + { + TopologyDeclarations.RemoveAll(match); + } +} + +public sealed class MessageRoutingOptionsBuilder +{ + private readonly MessageRoutingOptions _options; + + public MessageRoutingOptionsBuilder() + : this(new MessageRoutingOptions()) + { + } + + internal MessageRoutingOptionsBuilder(MessageRoutingOptions options) + { + _options = options; + } + + public MessageRoutingOptionsBuilder UseDefaultQueue(string destination) + { + ArgumentException.ThrowIfNullOrEmpty(destination); + _options.DefaultQueueDestination = destination; + DeclareQueue(destination); + return this; + } + + public MessageRoutingOptionsBuilder UseDefaultTopic(string topic) + { + ArgumentException.ThrowIfNullOrEmpty(topic); + _options.DefaultPubSubTopic = topic; + DeclareTopic(topic); + return this; + } + + public MessageRoutingOptionsBuilder MapQueue(string destination) + { + return MapQueue(typeof(T), destination); + } + + public MessageRoutingOptionsBuilder MapQueue(Type messageType, string destination) + { + return Map(MessageRouteRole.QueueDestination, destination, messageType); + } + + public MessageRoutingOptionsBuilder MapQueue(string destination, params Type[] messageTypes) + { + return Map(MessageRouteRole.QueueDestination, destination, messageTypes); + } + + public MessageRoutingOptionsBuilder MapTopic(string topic) + { + return MapTopic(typeof(T), topic); + } + + public MessageRoutingOptionsBuilder MapTopic(Type messageType, string topic) + { + return Map(MessageRouteRole.PubSubTopic, topic, messageType); + } + + public MessageRoutingOptionsBuilder MapTopic(string topic, params Type[] messageTypes) + { + return Map(MessageRouteRole.PubSubTopic, topic, messageTypes); + } + + public MessageRoutingOptionsBuilder UseConvention(Func convention) + { + _options.Convention = convention ?? throw new ArgumentNullException(nameof(convention)); + return this; + } + + + public MessageRoutingOptions Build() + { + return _options; + } + + private MessageRoutingOptionsBuilder Map(MessageRouteRole role, string route, params Type[] messageTypes) + { + ArgumentException.ThrowIfNullOrEmpty(route); + ArgumentNullException.ThrowIfNull(messageTypes); + + if (messageTypes.Length == 0) + throw new ArgumentException("At least one message type is required.", nameof(messageTypes)); + + foreach (var messageType in messageTypes) + { + ArgumentNullException.ThrowIfNull(messageType); + _options.RouteMaps.Add(new MessageRouteMap + { + MessageType = messageType, + Role = role, + Route = route + }); + } + + if (role == MessageRouteRole.QueueDestination) + DeclareQueue(route); + else + DeclareTopic(route); + + return this; + } + + private void DeclareQueue(string destination) + { + _options.Declare(new DestinationDeclaration { Address = DestinationAddress.ForQueue(destination) }); + } + + private void DeclareTopic(string topic) + { + _options.Declare(new DestinationDeclaration { Address = DestinationAddress.ForTopic(topic) }); + } + +} + +public sealed class DefaultMessageRouter : IMessageRouter +{ + public static DefaultMessageRouter Instance { get; } = new(new MessageRoutingOptions()); + + private readonly MessageRoutingOptions _options; + private readonly ConcurrentDictionary<(Type Type, MessageRouteRole Role), string> _routes = new(); + + public DefaultMessageRouter(MessageRoutingOptions options) + { + ArgumentNullException.ThrowIfNull(options); + _options = new MessageRoutingOptions + { + DefaultQueueDestination = options.DefaultQueueDestination, + DefaultPubSubTopic = options.DefaultPubSubTopic, + Convention = options.Convention + }; + _options.RouteMaps.AddRange(options.RouteMaps); + } + + public string ResolveRoute(MessageRouteContext context) + { + ArgumentNullException.ThrowIfNull(context); + ArgumentNullException.ThrowIfNull(context.MessageType); + + if (!String.IsNullOrEmpty(context.OperationOverride)) + return context.OperationOverride; + + if (_options.Convention is not null) + return ResolveUncached(context); + return _routes.GetOrAdd((context.MessageType, context.Role), key => ResolveUncached(new MessageRouteContext { MessageType = key.Type, Role = key.Role })); + } + + private string ResolveUncached(MessageRouteContext context) + { + var exact = _options.RouteMaps.LastOrDefault(m => m.Role == context.Role && m.MessageType == context.MessageType); + if (exact is not null) + return exact.Route; + + var assignable = _options.RouteMaps.LastOrDefault(m => m.Role == context.Role && m.MessageType != context.MessageType && m.MessageType.IsAssignableFrom(context.MessageType)); + if (assignable is not null) + return assignable.Route; + + var attribute = context.MessageType.GetCustomAttribute(); + string? attributedRoute = context.Role == MessageRouteRole.QueueDestination + ? attribute?.Destination + : attribute?.Topic ?? attribute?.Destination; + + if (!String.IsNullOrEmpty(attributedRoute)) + return attributedRoute; + + string? configuredDefault = context.Role == MessageRouteRole.QueueDestination + ? _options.DefaultQueueDestination + : _options.DefaultPubSubTopic; + + if (!String.IsNullOrEmpty(configuredDefault)) + return configuredDefault; + + if (_options.Convention is not null) + { + string convention = _options.Convention(context); + if (!String.IsNullOrEmpty(convention)) + return convention; + } + + return MessageRoutingConventions.ToKebabCase(context.MessageType.Name); + } + +} diff --git a/src/Foundatio/Messaging/MessageSendException.cs b/src/Foundatio/Messaging/MessageSendException.cs new file mode 100644 index 000000000..3c1a281d7 --- /dev/null +++ b/src/Foundatio/Messaging/MessageSendException.cs @@ -0,0 +1,65 @@ +using System; +using System.Collections.Generic; + +namespace Foundatio.Messaging; + +/// The known outcome of one message in a failed send. +public enum MessageSendStatus +{ + /// No send or scheduling operation was attempted. + NotAttempted, + /// The transport or scheduling store confirmed acceptance. + Accepted, + /// The operation failed without confirming whether the message was accepted. Retrying may duplicate delivery. + Unknown, + /// The provider confirmed rejection; no message was accepted for this input. + Rejected +} + +/// An application message ID and its send outcome. +public sealed record MessageSendOutcome(string MessageId, MessageSendStatus Status) +{ + public string? ErrorCode { get; init; } + public string? ErrorMessage { get; init; } + public bool? Retryable { get; init; } +} + +/// +/// A send failed. Outcomes cover every input message in input order, including messages not attempted. +/// Retry with the same application IDs and deduplicate at the consumer; sends are not transactions. +/// +public sealed class MessageSendException : MessageBusException +{ + public MessageSendException(IReadOnlyList outcomes, Exception innerException) + : base("Message sending failed. Inspect Outcomes before retrying; messages with an unknown outcome may already have been accepted.", innerException) + { + ArgumentNullException.ThrowIfNull(outcomes); + Outcomes = outcomes; + } + + public IReadOnlyList Outcomes { get; } +} + +/// +/// A sequential transport batch failed after accepting a prefix. The next message has an unknown outcome; +/// later messages were not attempted. Concurrent providers use the indexed Items constructor to retain every known outcome. +/// +public sealed class TransportSendException : MessageBusException +{ + public TransportSendException(int acceptedCount, Exception innerException) + : base("The transport accepted part of a batch before sending failed.", innerException) + { + ArgumentOutOfRangeException.ThrowIfNegative(acceptedCount); + AcceptedCount = acceptedCount; + } + + public TransportSendException(IReadOnlyList items, Exception innerException) + : base("The transport reported incomplete acceptance. Inspect indexed outcomes before retrying.", innerException) + { + ArgumentNullException.ThrowIfNull(items); + Items = items; + } + + public IReadOnlyList? Items { get; } + public int AcceptedCount { get; } +} diff --git a/src/Foundatio/Messaging/MessageTopology.cs b/src/Foundatio/Messaging/MessageTopology.cs new file mode 100644 index 000000000..b5ad04702 --- /dev/null +++ b/src/Foundatio/Messaging/MessageTopology.cs @@ -0,0 +1,64 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Utility; + +namespace Foundatio.Messaging; + +public interface IMessageTopology +{ + IReadOnlyList GetDeclarations(); + Task EnsureAsync(CancellationToken cancellationToken = default); + Task ValidateAsync(CancellationToken cancellationToken = default); +} + +public sealed class MessageTopology : IMessageTopology +{ + private readonly IMessageTransport _transport; + private readonly MessageRoutingOptions _options; + + public MessageTopology(IMessageTransport transport, MessageRoutingOptions options) + { + _transport = transport ?? throw new ArgumentNullException(nameof(transport)); + _options = options ?? throw new ArgumentNullException(nameof(options)); + } + + public IReadOnlyList GetDeclarations() + { + return _options.GetTopologyDeclarations(); + } + + public async Task EnsureAsync(CancellationToken cancellationToken = default) + { + var declarations = GetDeclarations(); + if (declarations.Count == 0) + return; + + if (_transport is not ISupportsProvisioning provisioning) + throw new NotSupportedException($"Transport \"{_transport.GetType().Name}\" does not support topology provisioning."); + + await provisioning.EnsureAsync(declarations, cancellationToken).AnyContext(); + } + + public async Task ValidateAsync(CancellationToken cancellationToken = default) + { + var declarations = GetDeclarations(); + if (declarations.Count == 0) + return; + + if (_transport is not ISupportsProvisioning provisioning) + throw new NotSupportedException($"Transport \"{_transport.GetType().Name}\" does not support topology validation."); + + var missing = new List(); + foreach (var declaration in declarations) + { + if (!await provisioning.ExistsAsync(declaration.Address, cancellationToken).AnyContext()) + missing.Add(declaration); + } + + if (missing.Count > 0) + throw new InvalidOperationException($"Message topology is missing: {String.Join(", ", missing.Select(d => d.Address.ToString()))}."); + } +} diff --git a/src/Foundatio/Messaging/MessageTransport.cs b/src/Foundatio/Messaging/MessageTransport.cs new file mode 100644 index 000000000..88a90c7f2 --- /dev/null +++ b/src/Foundatio/Messaging/MessageTransport.cs @@ -0,0 +1,432 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Messaging; + +public enum MessagePriority +{ + Low = 0, + Normal = 1, + High = 2 +} + +public enum DeliveryGuarantee +{ + AtMostOnce, + AtLeastOnce +} + +public enum OrderingGuarantee +{ + None, + Fifo, + PerPartition +} + +public enum DestinationRole +{ + Queue, + Topic, + Subscription, + Binding +} + +/// +/// The canonical identity of a transport destination: a name, the role that names the physical namespace it lives in, +/// and — for subscriptions — the owning topic. Every transport API (send, receive, subscribe, stats, settlement, +/// provisioning) uses this one value, so the same logical destination can never be spelled two ways on two paths. +/// +/// +/// is the destination's opaque string form ("{topic}/{name}" for subscriptions, Name +/// otherwise) for logging, metrics tags, and dictionary keys. Because a subscription key contains '/', a +/// transport must NOT assume it is a legal broker resource name (e.g. an SQS queue name) — map it to native resources +/// during and treat it as a lookup key thereafter. Topic and +/// subscription names must not contain '/'. +/// +public sealed record DestinationAddress +{ + public required string Name { get; init; } + public DestinationRole Role { get; init; } = DestinationRole.Queue; + + /// The owning topic when is ; null otherwise. + public string? Topic { get; init; } + + /// The canonical opaque string form: "{topic}/{name}" for subscriptions, Name otherwise. + public string Key => Topic is { Length: > 0 } topic ? $"{topic}/{Name}" : Name; + + public override string ToString() => $"{Role}:{Key}"; + + public static DestinationAddress ForQueue(string name) + { + ArgumentException.ThrowIfNullOrEmpty(name); + return new DestinationAddress { Name = name, Role = DestinationRole.Queue }; + } + + public static DestinationAddress ForTopic(string name) + { + ArgumentException.ThrowIfNullOrEmpty(name); + return new DestinationAddress { Name = name, Role = DestinationRole.Topic }; + } + + public static DestinationAddress ForSubscription(string topic, string subscription) + { + ArgumentException.ThrowIfNullOrEmpty(topic); + ArgumentException.ThrowIfNullOrEmpty(subscription); + return new DestinationAddress { Name = subscription, Role = DestinationRole.Subscription, Topic = topic }; + } +} + +public sealed record TransportMessage +{ + public required ReadOnlyMemory Body { get; init; } + public MessageHeaders Headers { get; init; } = MessageHeaders.Empty; + public string? MessageId { get; init; } + + /// + /// Content type of (e.g. application/json). A transport whose native wire format is text + /// (such as SQS/SNS) can store a text body directly when this indicates text, avoiding base64 overhead; null means + /// unknown, so a byte-safe encoding should be used. + /// + public string? ContentType { get; init; } +} + +public sealed record TransportSendOptions +{ + public MessagePriority Priority { get; init; } = MessagePriority.Normal; + public DateTimeOffset? DeliverAt { get; init; } +} + +/// +/// One delivered message: the payload and metadata a receive/subscribe hands to the consumer, plus the +/// that settles it ( / +/// ). +/// +/// +/// Provider authors: future contract growth only ever adds OPTIONAL init members to this record (never new required +/// ones), so provider code constructing entries stays source-compatible across core upgrades. +/// +public sealed record TransportEntry +{ + /// Per-entry envelope decoding failure. Body and headers retain raw evidence for quarantine. + public Exception? EnvelopeError { get; init; } + /// The broker-assigned message id — stable across redeliveries of the same message. + public required string Id { get; init; } + + /// The caller-supplied TransportMessage.MessageId, preserved independently of the broker ID. + public string? ApplicationMessageId { get; init; } + + /// The media type of the original serialized body. + public string? ContentType { get; init; } + + /// The source address the entry was received from (the queue or subscription, never the owning topic). + public required DestinationAddress Destination { get; init; } + + public required ReadOnlyMemory Body { get; init; } + + /// The sent message's headers, which must round-trip byte-for-byte through the transport. + public MessageHeaders Headers { get; init; } = MessageHeaders.Empty; + + /// How many times this message has been delivered, INCLUDING this delivery — starts at 1, never 0. + public int DeliveryCount { get; init; } = 1; + + public DateTimeOffset? EnqueuedUtc { get; init; } + + /// Expiry of this delivery's lease. Null means the delivery has no expiring lease. + public DateTimeOffset? LockExpiresUtc { get; init; } + + /// The settlement token for this delivery; see . + public required Receipt Receipt { get; init; } +} + +/// +/// The transport's opaque settlement token for one delivery. Everything the transport needs to settle the entry later +/// (complete/abandon/dead-letter) must live in — not in transport instance state keyed by +/// entry identity alone — because the same message can be in flight again (a redelivery) by the time a stale receipt +/// is settled, and per-delivery state is what keeps the two from aliasing. +/// +public readonly struct Receipt +{ + public object? TransportState { get; init; } +} + +public sealed record ReceiveRequest +{ + public int MaxMessages { get; init; } = 1; + public TimeSpan? MaxWaitTime { get; init; } +} + +public sealed record MessageDestinationStats +{ + // Point-in-time gauges every transport can report (may be approximate / eventually consistent on real brokers, + // e.g. SQS ApproximateNumberOf*). + public long Queued { get; init; } + public long Working { get; init; } + /// Messages awaiting delayed redelivery, when the provider can report this gauge. + public long? Delayed { get; init; } + public long Deadletter { get; init; } + + // Lifetime counters. Not universally available — a transport that does not track a counter leaves it null (e.g. + // SQS exposes no lifetime "completed" count). Null means "not reported", distinct from a reported zero. + public long? Enqueued { get; init; } + public long? Dequeued { get; init; } + public long? Completed { get; init; } + public long? Abandoned { get; init; } + public long? Errors { get; init; } + public long? Timeouts { get; init; } +} + +public sealed record SendItemResult +{ + /// Zero-based index in this transport call. Null uses the result's position for sequential providers. + public int? Index { get; init; } + public MessageSendStatus Status { get; init; } = MessageSendStatus.Accepted; + public string? ErrorCode { get; init; } + public string? ErrorMessage { get; init; } + public bool? Retryable { get; init; } + /// The broker-assigned id of the accepted message. + public string? MessageId { get; init; } +} + +/// One indexed outcome for every input. Providers may mix acceptance, rejection and unknown outcomes. +public sealed record SendResult +{ + public required IReadOnlyList Items { get; init; } + + /// Verifies acceptance before discarding a durable source record. + public void EnsureAccepted(int expectedCount) + { + if (Items.Count != expectedCount) + throw new MessageBusException("Transport returned an incomplete result."); + var seen = new HashSet(); + for (int position = 0; position < Items.Count; position++) + { + var item = Items[position]; + int index = item.Index ?? position; + if (index < 0 || index >= expectedCount || !seen.Add(index)) + throw new MessageBusException("Transport returned an invalid or duplicate input index."); + if (item.Status != MessageSendStatus.Accepted) + throw new TransportSendException(Items, new MessageBusException(item.ErrorMessage ?? "Transport did not accept every input.")); + } + } +} + +/// +/// Thrown when a transport settle operation is given a receipt that has expired or was already settled. Strict receipt +/// validation is transport-specific: some brokers (e.g. SQS) treat settling with a stale receipt as idempotent and do +/// not raise, so callers must not depend on this exception for correctness — it is a best-effort safety signal. +/// +public sealed class ReceiptExpiredException : Exception +{ + public ReceiptExpiredException() : base("The transport receipt has expired or has already been settled.") { } + + public ReceiptExpiredException(string message) : base(message) { } + + public ReceiptExpiredException(string message, Exception innerException) : base(message, innerException) { } +} + +public sealed record DestinationDeclaration +{ + /// The canonical identity of the destination to provision — the SAME address the runtime later sends to, + /// receives from, and asks stats for, so provisioning and runtime can never disagree on a destination's identity. + public required DestinationAddress Address { get; init; } + + /// For temporary subscriptions, the lease after which the subscription and backlog expire without renewal. + public TimeSpan? AutoDeleteAfter { get; init; } + + // Provider-specific creation arguments for transports that provision destinations (e.g. RabbitMQ queue arguments). + // Retry and dead-letter behavior is owned by the core RetryPolicy, not declared here, so destinations stay simple. + public IReadOnlyDictionary? ProviderArguments { get; init; } +} + +public sealed record PushOptions +{ + public int MaxConcurrentMessages { get; init; } = 1; + public TimeSpan PollInterval { get; init; } = TimeSpan.FromSeconds(1); +} + +/// +/// The capability and limit facts a transport advertises for one . Capabilities vary by +/// role on real brokers (SQS queues take DelaySeconds; SNS topics have no native delay), so the core asks per role +/// via rather than reading transport-wide flags. Anything not advertised +/// here is treated as unsupported: the core validates, falls back, or throws instead of letting the broker silently +/// drop a requested behavior. +/// +public sealed record TransportCapabilities +{ + /// Capabilities of a transport (or role) that advertises nothing: every feature routes through core fallbacks or fails validation. + public static readonly TransportCapabilities None = new(); + + /// The destination honors natively. + public bool DelayedDelivery { get; init; } + + /// + /// The longest delivery delay honored natively when is true (e.g. SQS caps + /// DelaySeconds at 15 minutes); null means unbounded. A send scheduled further out is routed through the + /// runtime-store fallback instead of being silently truncated to the broker's ceiling. + /// + public TimeSpan? MaxDeliveryDelay { get; init; } + + /// The destination honors . + public bool Priority { get; init; } + + /// The destination honors per-message expiration (). + public bool Expiration { get; init; } + + public OrderingGuarantee Ordering { get; init; } = OrderingGuarantee.None; + + /// Maximum messages per call; null means unbounded. The core chunks larger sends. + public int? MaxBatchSize { get; init; } + + /// Maximum entries per pull receive; null means no transport-specific limit. + public int? MaxReceiveBatchSize { get; init; } + + /// Maximum parallel pull requests per source. All requests share the consumer concurrency budget. + public int MaxConcurrentReceives { get; init; } = 1; + + /// Optional brief wait for concurrently settling deliveries to free a fuller receive batch. Defaults to no wait. + public TimeSpan ReceiveBatchDelay { get; init; } + + /// Maximum message body size in bytes; null means unbounded. The core rejects oversized messages up front. + public long? MaxMessageBytes { get; init; } +} + +public interface ITransportInfo +{ + DeliveryGuarantee DeliveryGuarantee { get; } + IReadOnlySet SupportedRoles { get; } + + /// + /// The capabilities and limits this transport honors for the given destination. Most transports vary only by + /// (SQS queues take a native delay; SNS topics do not), but the full address + /// is the key so a routing/composite transport can answer per destination. Must be side-effect free and cheap; + /// the core consults it on every send-path decision (native delay vs. runtime-store fallback, priority/expiration + /// validation, size and batch limits). + /// + TransportCapabilities GetCapabilities(DestinationAddress destination); +} + +/// +/// The provider SPI every transport implements: send messages and settle deliveries. Everything else (pull, push, +/// dead-letter, delays, stats, provisioning) is an optional ISupports* capability interface the core detects +/// at runtime — implement only what the broker actually offers and the core validates or falls back for the rest. +/// +public interface IMessageTransport : IAsyncDisposable +{ + /// + /// Delivers the messages to the destination and reports one indexed outcome per input, including rejected or + /// unknown outcomes. A multi-message send is not atomic. If an exception interrupts a partially accepted batch, + /// use to preserve known outcomes; an ordinary exception leaves acceptance unknown. + /// + /// + /// A future the transport cannot honor natively must be refused with + /// , never accepted and delivered immediately (a silently dropped delay); the + /// core only routes a delayed send here when the destination advertises the + /// capability. A topic send with zero subscriptions is + /// dropped — real pub/sub semantics: subscriptions must exist before a publish can reach them. + /// + Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default); + + /// + /// Permanently removes the delivered entry — the terminal success settlement. Settling with a stale or + /// already-settled receipt SHOULD throw , but that signal is best-effort: + /// some brokers (e.g. SQS) treat stale settlement as idempotent, so callers must not depend on it for correctness. + /// + Task CompleteAsync(TransportEntry entry, CancellationToken ct = default); + + /// + /// Returns the delivered entry to its source for redelivery with + /// incremented. Same stale-receipt semantics as . + /// + Task AbandonAsync(TransportEntry entry, CancellationToken ct = default); +} + +public interface ISupportsPull : IMessageTransport +{ + /// + /// Receives up to entries (a ceiling — fewer, including zero, is valid). + /// is a long-poll window: return as soon as any messages arrive, block up + /// to the window when none are available, and return empty when it lapses. Returned entries carry the source + /// address as their . + /// + Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct = default); +} + +public interface ISupportsPush : IMessageTransport +{ + /// + /// Attaches a callback that is invoked for each entry delivered from the source until the returned subscription is + /// disposed. The callback (or the core wrapping it) settles each entry; a callback that throws without settling + /// must result in the entry being abandoned for redelivery, never lost. At most + /// callbacks run concurrently per subscription. + /// + Task SubscribeAsync(DestinationAddress source, Func onMessage, PushOptions options, CancellationToken ct = default); +} + +public interface ISupportsRedeliveryDelay : IMessageTransport +{ + // The longest redelivery delay the transport can honor natively (e.g. SQS serves this via ChangeMessageVisibility, + // capped at 12 hours). Null means unbounded. A requested delay longer than this is routed through the runtime-store + // fallback instead of being silently clamped by the broker. + TimeSpan? MaxRedeliveryDelay { get; } + + Task AbandonAsync(TransportEntry entry, TimeSpan redeliveryDelay, CancellationToken ct = default); +} + +/// A native dead-letter destination, independently of non-destructive administration support. +public interface ISupportsDeadLetterSink : IMessageTransport +{ + Task DeadLetterAsync(TransportEntry entry, string? reason, CancellationToken ct = default); +} + +public interface ISupportsDeadLetter : ISupportsDeadLetterSink +{ + + /// Inspects raw dead letters without consuming them. + Task> PeekDeadLetteredAsync(DestinationAddress destination, DeadLetterQuery? query = null, CancellationToken cancellationToken = default); + /// Explicitly removes one dead letter by its broker ID. Returns false if it no longer exists. + Task DeleteDeadLetteredAsync(DestinationAddress destination, string id, CancellationToken cancellationToken = default); + /// Replays a dead letter to an explicit queue or topic, preserving its application ID and resetting retry headers. + Task ReplayDeadLetteredAsync(DestinationAddress source, string id, DestinationAddress target, CancellationToken cancellationToken = default); + +} + +public interface ISupportsLockRenewal : IMessageTransport +{ + Task RenewLockAsync(TransportEntry entry, TimeSpan? duration, CancellationToken ct = default); +} + +public interface ISupportsVisibilityTimeout : IMessageTransport +{ + // The longest receive visibility timeout the transport can honor natively (e.g. SQS caps visibility at 12 hours). + // Null means unbounded. Callers requesting a longer visibility than the broker supports should treat that as + // unsatisfiable rather than relying on a silently clamped value. + TimeSpan? MaxVisibilityTimeout { get; } + + Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, TimeSpan visibility, CancellationToken ct = default); +} + +public interface ISupportsStats : IMessageTransport +{ + Task GetStatsAsync(DestinationAddress destination, CancellationToken ct = default); +} + +public interface ISupportsProvisioning : IMessageTransport +{ + Task EnsureAsync(IReadOnlyList declarations, CancellationToken ct = default); + Task DeleteAsync(DestinationAddress destination, CancellationToken ct = default); + Task ExistsAsync(DestinationAddress destination, CancellationToken ct = default); +} + +public interface IPushSubscription : IAsyncDisposable +{ + DestinationAddress Source { get; } +} + +/// Temporary subscriptions whose ownership expires after a listener crashes. +public interface ISupportsEphemeralSubscriptions : ISupportsProvisioning +{ + /// Extends an existing unexpired subscription lease. Returns false after ownership expires. + Task RenewSubscriptionAsync(DestinationAddress source, TimeSpan lease, CancellationToken cancellationToken = default); +} diff --git a/src/Foundatio/Messaging/MessageTypeRegistry.cs b/src/Foundatio/Messaging/MessageTypeRegistry.cs new file mode 100644 index 000000000..891e4848d --- /dev/null +++ b/src/Foundatio/Messaging/MessageTypeRegistry.cs @@ -0,0 +1,61 @@ +using System; +using System.Collections.Generic; + +namespace Foundatio.Messaging; + +/// +/// Resolves the stable wire discriminator written to the message.type header in both directions: a CLR type to +/// its name (for sending) and a name back to its CLR type (so a grouped/interface consumer can deserialize the actual +/// payload type). Register stable names for types that may move between assemblies or namespaces; unregistered types +/// fall back to (never AssemblyQualifiedName). +/// +public interface IMessageTypeRegistry +{ + string GetName(Type messageType); + Type? Resolve(string name); +} + +public sealed record MessageTypeRegistration(string Name, Type MessageType); + +public sealed class MessageTypeRegistry : IMessageTypeRegistry +{ + private readonly Dictionary _nameToType = new(StringComparer.Ordinal); + private readonly Dictionary _typeToName = []; + + public MessageTypeRegistry(IEnumerable? registrations = null) + { + foreach (var registration in registrations ?? []) + Add(registration); + } + + public string GetName(Type messageType) + { + ArgumentNullException.ThrowIfNull(messageType); + return _typeToName.TryGetValue(messageType, out string? name) + ? name + : messageType.FullName ?? messageType.Name; + } + + public Type? Resolve(string name) + { + ArgumentException.ThrowIfNullOrEmpty(name); + + if (_nameToType.TryGetValue(name, out var registered)) + return registered; + + return null; + } + + private void Add(MessageTypeRegistration registration) + { + ArgumentNullException.ThrowIfNull(registration); + ArgumentException.ThrowIfNullOrEmpty(registration.Name); + ArgumentNullException.ThrowIfNull(registration.MessageType); + + if (_nameToType.TryGetValue(registration.Name, out var existing) && existing != registration.MessageType) + throw new InvalidOperationException($"Message type name \"{registration.Name}\" is already registered for \"{existing.FullName}\"."); + + _nameToType[registration.Name] = registration.MessageType; + _typeToName[registration.MessageType] = registration.Name; + } +} diff --git a/src/Foundatio/Messaging/NullMessageBus.cs b/src/Foundatio/Messaging/NullMessageBus.cs deleted file mode 100644 index cee2fa42a..000000000 --- a/src/Foundatio/Messaging/NullMessageBus.cs +++ /dev/null @@ -1,24 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; - -namespace Foundatio.Messaging; - -public class NullMessageBus : IMessageBus -{ - public static readonly NullMessageBus Instance = new(); - - public Task PublishAsync(Type messageType, object message, MessageOptions? options = null, CancellationToken cancellationToken = default) - { - return Task.CompletedTask; - } - - public Task SubscribeAsync(Func handler, CancellationToken cancellationToken = default) where T : class - { - return Task.CompletedTask; - } - - public void Dispose() { } - - public ValueTask DisposeAsync() => default; -} diff --git a/src/Foundatio/Messaging/ReceivedMessage.cs b/src/Foundatio/Messaging/ReceivedMessage.cs new file mode 100644 index 000000000..ff0016a46 --- /dev/null +++ b/src/Foundatio/Messaging/ReceivedMessage.cs @@ -0,0 +1,84 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Utility; + +namespace Foundatio.Messaging; + +/// A directly received delivery. Disposal returns unsettled work for redelivery and stops lease renewal. +public interface IReceivedMessage : IMessageContext, IAsyncDisposable; + +/// A directly received typed delivery. Use await using, then complete or reject it explicitly. +public interface IReceivedMessage : IReceivedMessage, IMessageContext where T : class; + +/// Options for receiving one queued message without registering a handler. +public sealed record MessageReceiveOptions +{ + /// Queue name. Null uses the message type's configured route. + public string? Destination { get; init; } + + /// Maximum wait for a message. Zero checks for immediately available work. + public TimeSpan WaitTime { get; init; } +} + +internal class ReceivedMessage(IMessageContext context, CancellationTokenSource cancellation, MessageDeliveryLease supervision, Func abandon) : IReceivedMessage +{ + private int _disposed; + public string Id => context.Id; + public DestinationAddress? Destination => context.Destination; + public DateTimeOffset? EnqueuedUtc => context.EnqueuedUtc; + public bool IsLeaseLost => context.IsLeaseLost; + public string BrokerMessageId => context.BrokerMessageId; + public ReadOnlyMemory Body => context.Body; + public MessageHeaders Headers => context.Headers; + public string? CorrelationId => context.CorrelationId; + public string? MessageType => context.MessageType; + public MessagePriority Priority => context.Priority; + public int Attempts => context.Attempts; + public bool IsHandled => context.IsHandled; + public CancellationToken CancellationToken => context.CancellationToken; + + public async Task CompleteAsync(CancellationToken cancellationToken = default) + { + await context.CompleteAsync(cancellationToken).AnyContext(); + await DisposeAsync().AnyContext(); + } + + public async Task RejectAsync(RejectOptions? options = null, CancellationToken cancellationToken = default) + { + await context.RejectAsync(options, cancellationToken).AnyContext(); + await DisposeAsync().AnyContext(); + } + + public Task RenewLockAsync(TimeSpan? duration = null, CancellationToken cancellationToken = default) + => context.RenewLockAsync(duration, cancellationToken); + + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _disposed, 1) != 0) + return; + + try + { + await cancellation.CancelAsync().AnyContext(); + await supervision.DisposeAsync().AnyContext(); + bool leaseLost = supervision.IsLost; + if (!context.IsHandled && !leaseLost) + { + using var cleanup = new CancellationTokenSource(TimeSpan.FromSeconds(5)); + await abandon(cleanup.Token).AnyContext(); + } + } + finally + { + await cancellation.CancelAsync().AnyContext(); + cancellation.Dispose(); + } + } +} + +internal sealed class ReceivedMessage(IMessageContext context, CancellationTokenSource cancellation, MessageDeliveryLease supervision, Func abandon) + : ReceivedMessage(context, cancellation, supervision, abandon), IReceivedMessage where T : class +{ + public T Message => context.Message; +} diff --git a/src/Foundatio/Messaging/ScheduledMessageDispatcher.cs b/src/Foundatio/Messaging/ScheduledMessageDispatcher.cs new file mode 100644 index 000000000..4c05c2354 --- /dev/null +++ b/src/Foundatio/Messaging/ScheduledMessageDispatcher.cs @@ -0,0 +1,111 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Utility; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Foundatio.Messaging; + +/// Dependencies and topology policy for dispatching persisted delayed messages. +public sealed record ScheduledMessageDispatcherOptions +{ + public TimeProvider? TimeProvider { get; init; } + public ILoggerFactory? LoggerFactory { get; init; } + public TopologyMode TopologyMode { get; init; } = TopologyMode.Ensure; +} + +/// +/// Sends due messages from a scheduling store. Independent of the job worker and scheduler. +/// Delivery is at least once: a crash after sending but before settlement can resend the same application message ID. +/// +public sealed class ScheduledMessageDispatcher +{ + private static readonly TimeSpan Lease = TimeSpan.FromMinutes(1); + private static readonly TimeSpan SendTimeout = TimeSpan.FromSeconds(30); + private readonly IScheduledDispatchStore _store; + private readonly IMessageTransport _transport; + private readonly TimeProvider _timeProvider; + private readonly TopologyMode _topologyMode; + private readonly ILogger _logger; + private Exception? _lastFailure; + public Exception? LastFailure => Volatile.Read(ref _lastFailure); + + public ScheduledMessageDispatcher(IScheduledDispatchStore store, IMessageTransport transport, ScheduledMessageDispatcherOptions? options = null) + { + _store = store ?? throw new ArgumentNullException(nameof(store)); + _transport = transport ?? throw new ArgumentNullException(nameof(transport)); + _timeProvider = options?.TimeProvider ?? TimeProvider.System; + _topologyMode = options?.TopologyMode ?? TopologyMode.Ensure; + _logger = (options?.LoggerFactory ?? NullLoggerFactory.Instance).CreateLogger(); + } + + /// Dispatches up to due messages, claiming each only when ready to send it. + public Task DispatchDueAsync(int limit = 100, CancellationToken cancellationToken = default) + => DispatchDueAsync(_timeProvider.GetUtcNow(), limit, cancellationToken); + + /// Dispatches messages due by the specified UTC time. + public async Task DispatchDueAsync(DateTimeOffset utcNow, int limit = 100, CancellationToken cancellationToken = default) + { + ArgumentOutOfRangeException.ThrowIfLessThan(limit, 1); + Volatile.Write(ref _lastFailure, null); + int completed = 0; + for (int index = 0; index < limit; index++) + { + cancellationToken.ThrowIfCancellationRequested(); + string claim = Guid.NewGuid().ToString("N"); + var dispatches = await _store.ClaimDueDispatchesAsync(utcNow, 1, claim, Lease, cancellationToken).AnyContext(); + if (dispatches.Count == 0) + break; + + var dispatch = dispatches[0]; + using var timeout = new CancellationTokenSource(SendTimeout, _timeProvider); + using var operation = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeout.Token); + try + { + await SendAsync(dispatch, operation.Token).WaitAsync(operation.Token).AnyContext(); + if (await _store.CompleteDispatchAsync(dispatch.DispatchId, claim, operation.Token).WaitAsync(operation.Token).AnyContext()) + completed++; + else + _logger.LogWarning("Scheduled dispatch {DispatchId} lost its claim before completion", dispatch.DispatchId); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception ex) + { + Volatile.Write(ref _lastFailure, ex); + _logger.LogError(ex, "Failed to dispatch scheduled message {DispatchId}", dispatch.DispatchId); + using var settlement = new CancellationTokenSource(TimeSpan.FromSeconds(5), _timeProvider); + await _store.ReleaseDispatchAsync(dispatch.DispatchId, claim, _timeProvider.GetUtcNow().AddSeconds(30), settlement.Token) + .WaitAsync(settlement.Token).AnyContext(); + } + } + + return completed; + } + + private async Task SendAsync(ScheduledDispatchState dispatch, CancellationToken cancellationToken) + { + var destination = dispatch.Destination ?? throw new InvalidOperationException($"Scheduled message {dispatch.DispatchId} has no destination."); + if (_topologyMode == TopologyMode.Validate && _transport is not ISupportsProvisioning) + throw new NotSupportedException($"Transport {_transport.GetType().Name} cannot validate destinations."); + if (_topologyMode != TopologyMode.None && _transport is ISupportsProvisioning provisioning) + { + if (_topologyMode == TopologyMode.Ensure) + await provisioning.EnsureAsync([new DestinationDeclaration { Address = destination }], cancellationToken).AnyContext(); + else if (!await provisioning.ExistsAsync(destination, cancellationToken).AnyContext()) + throw new InvalidOperationException($"Scheduled message destination {destination} does not exist."); + } + + var result = await _transport.SendAsync(destination, [new TransportMessage + { + MessageId = dispatch.Headers.GetValueOrDefault(KnownHeaders.MessageId) ?? dispatch.DispatchId, + Body = dispatch.Body, Headers = dispatch.Headers, + ContentType = dispatch.Headers.GetValueOrDefault(KnownHeaders.ContentType) + }], dispatch.Options with { DeliverAt = null }, cancellationToken).AnyContext(); + result.EnsureAccepted(1); + } +} diff --git a/src/Foundatio/Messaging/SharedMessageBusOptions.cs b/src/Foundatio/Messaging/SharedMessageBusOptions.cs deleted file mode 100644 index 94a15423c..000000000 --- a/src/Foundatio/Messaging/SharedMessageBusOptions.cs +++ /dev/null @@ -1,44 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Diagnostics.CodeAnalysis; - -namespace Foundatio.Messaging; - -public class SharedMessageBusOptions : SharedOptions -{ - /// - /// The topic name - /// - public string Topic { get; set; } = "messages"; - - /// - /// Controls which types messages are mapped to. - /// - [DisallowNull] - public Dictionary MessageTypeMappings { get => field; set => field = value ?? new(); } = new(); -} - -public class SharedMessageBusOptionsBuilder : SharedOptionsBuilder - where TOptions : SharedMessageBusOptions, new() - where TBuilder : SharedMessageBusOptionsBuilder, new() -{ - public TBuilder Topic(string topic) - { - ArgumentException.ThrowIfNullOrEmpty(topic); - - Target.Topic = topic; - return (TBuilder)this; - } - - public TBuilder MapMessageType(string name) - { - Target.MessageTypeMappings[name] = typeof(T); - return (TBuilder)this; - } - - public TBuilder MapMessageTypeToClassName() - { - Target.MessageTypeMappings[typeof(T).Name] = typeof(T); - return (TBuilder)this; - } -} diff --git a/src/Foundatio/Messaging/Tracking/ExecutionHeaders.cs b/src/Foundatio/Messaging/Tracking/ExecutionHeaders.cs new file mode 100644 index 000000000..1905e67cc --- /dev/null +++ b/src/Foundatio/Messaging/Tracking/ExecutionHeaders.cs @@ -0,0 +1,11 @@ +namespace Foundatio.Messaging; + +/// Optional execution metadata carried by ordinary Foundatio messages. +public static class ExecutionHeaders +{ + public const string ExecutionId = "message.execution.id"; + public const string OriginalExecutionId = "message.execution.original_id"; + public const string EnqueuedAt = "message.enqueued_at"; + public const string ReplayedAt = "message.replayed_at"; + public const string OriginNode = "message.origin_node"; +} diff --git a/src/Foundatio/Messaging/Tracking/MessageExecutionOptions.cs b/src/Foundatio/Messaging/Tracking/MessageExecutionOptions.cs new file mode 100644 index 000000000..1a4b0e0aa --- /dev/null +++ b/src/Foundatio/Messaging/Tracking/MessageExecutionOptions.cs @@ -0,0 +1,30 @@ +using System; + +namespace Foundatio.Messaging; + +/// Optional execution tracking for broker-delivered work. This never schedules a second job-store worker. +public sealed record MessageExecutionOptions +{ + /// Logical queue represented by this execution. + public required string QueueName { get; init; } + /// Application message type used for diagnostics. + public Type? MessageType { get; init; } + /// Total delivery budget, including the first attempt; negative means unlimited. + public int MaxAttempts { get; init; } = 3; + /// Delay for a failed attempt, using its one-based attempt number. + public Func RetryBackoff { get; init; } = RetryPolicy.DefaultBackoff; + /// Delivery duration used by explicit progress heartbeats. + public TimeSpan VisibilityTimeout { get; init; } = TimeSpan.FromMinutes(1); + /// Settle returned outcomes automatically; explicit settlement takes precedence. + public bool AutoComplete { get; init; } = true; + /// Read and update broker execution history in the supplied store. + public bool TrackProgress { get; init; } + /// Header containing the producer-created execution identifier. + public string ExecutionIdHeader { get; init; } = "message.execution.id"; + /// Interval between cooperative cancellation checks and execution heartbeats. + public TimeSpan CancellationPollInterval { get; init; } = TimeSpan.FromSeconds(5); + /// Process identity recorded when an attempt starts. + public string WorkerId { get; init; } = $"{Environment.MachineName}:{Environment.ProcessId}"; + /// Receives success, failed-attempt, and dead-letter measurements. Must not throw. + public Action? OnProcessed { get; init; } +} diff --git a/src/Foundatio/Messaging/Tracking/MessageExecutionPipeline.cs b/src/Foundatio/Messaging/Tracking/MessageExecutionPipeline.cs new file mode 100644 index 000000000..69de5feef --- /dev/null +++ b/src/Foundatio/Messaging/Tracking/MessageExecutionPipeline.cs @@ -0,0 +1,243 @@ +using System; +using System.Diagnostics; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Utility; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Foundatio.Messaging; + +/// +/// Executes broker deliveries with optional job progress, cancellation, and history. +/// The broker owns delivery leases and retries; the job store records each attempt without scheduling it. +/// +public sealed class MessageExecutionPipeline +{ + private static readonly TimeSpan OperationTimeout = TimeSpan.FromSeconds(30); + private readonly MessageExecutionOptions _options; + private readonly IJobRuntimeStore? _store; + private readonly TimeProvider _time; + private readonly ILogger _logger; + + public MessageExecutionPipeline(MessageExecutionOptions options, IJobRuntimeStore? store = null, TimeProvider? timeProvider = null, ILogger? logger = null) + { + ArgumentNullException.ThrowIfNull(options); + ArgumentException.ThrowIfNullOrWhiteSpace(options.QueueName); + ArgumentOutOfRangeException.ThrowIfEqual(options.MaxAttempts, 0); + ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(options.CancellationPollInterval, TimeSpan.Zero); + if (options.TrackProgress && store is null) + throw new ArgumentException("Execution tracking requires a job store. Configure AddFoundatio().Jobs.UseInMemory() or Jobs.UseRedis().", nameof(store)); + _options = options; + _store = store; + _time = timeProvider ?? TimeProvider.System; + _logger = logger ?? NullLogger.Instance; + } + + /// Processes a delivery, recording terminal job state only after confirmed broker settlement. + public async Task ProcessAsync(IMessageContext delivery, Func> handler, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(delivery); + ArgumentNullException.ThrowIfNull(handler); + string? jobId = _options.TrackProgress ? delivery.Headers.GetValueOrDefault(_options.ExecutionIdHeader) : null; + JobState? attempt = null; + if (jobId is not null) + { + attempt = await RunAsync(ct => _store!.BeginBrokerAttemptAsync(jobId, delivery.Attempts, _options.WorkerId, ct), cancellationToken).AnyContext(); + if (attempt is null) + { + var state = await RunAsync(ct => _store!.GetAsync(jobId, ct), cancellationToken).AnyContext(); + if (state is not null) + { + if (state.Status is JobStatus.Completed or JobStatus.Failed or JobStatus.Cancelled or JobStatus.DeadLettered) + await SettleAsync(delivery.CompleteAsync, delivery).AnyContext(); + else + await SettleAsync(ct => delivery.RejectAsync(new RejectOptions { RedeliveryDelay = _options.RetryBackoff(delivery.Attempts) }, ct), delivery).AnyContext(); + return; + } + _logger.LogWarning("Job history {JobId} expired before delivery; processing continues without retained history", jobId); + } + } + + using var processing = CancellationTokenSource.CreateLinkedTokenSource(delivery.CancellationToken, cancellationToken); + var token = processing.Token; + long started = Stopwatch.GetTimestamp(); + Task? poll = attempt is null ? null : PollCancellationAsync(attempt, processing); + var context = new MessageProcessingContext + { + Body = delivery.Body, + QueueName = _options.QueueName, + MessageId = delivery.BrokerMessageId, + MessageType = _options.MessageType, + DequeueCount = delivery.Attempts, + MaxAttempts = _options.MaxAttempts, + VisibilityTimeout = _options.VisibilityTimeout, + EnqueuedAt = delivery.EnqueuedUtc ?? _time.GetUtcNow(), + JobId = jobId, + Headers = delivery.Headers, + OnCancelProcessing = processing.Cancel, + OnRenewTimeout = (duration, ct) => delivery.RenewLockAsync(duration, ct), + OnComplete = ct => RunAsync(delivery.CompleteAsync, ct), + OnAbandon = (delay, ct) => RunAsync(t => delivery.RejectAsync(new RejectOptions { RedeliveryDelay = delay }, t), ct), + OnReportProgress = async ct => + { + await delivery.RenewLockAsync(_options.VisibilityTimeout, ct).AnyContext(); + if (attempt is not null) + await RecordAsync(t => _store!.HeartbeatJobAsync(attempt.JobId, attempt.ClaimToken!, t), ct).AnyContext(); + }, + OnReportDetailedProgress = attempt is null ? null : async (percent, message, ct) => + { + if (await IsCancelledAsync(attempt.JobId, ct).AnyContext()) + throw new OperationCanceledException("Job cancellation was requested."); + await RecordAsync(t => _store!.ReportJobProgressAsync(attempt.JobId, attempt.ClaimToken!, Math.Clamp(percent, 0, 100), message, t), ct).AnyContext(); + } + }; + JobCompletion completion = new() { Kind = JobCompletionKind.Interrupted }; + try + { + if (attempt?.CancellationRequested == true) + { + if (await SettleAsync(delivery.CompleteAsync, delivery).AnyContext()) + completion = new() { Kind = JobCompletionKind.Cancelled }; + } + else if (_options.MaxAttempts > 0 && delivery.Attempts > _options.MaxAttempts) + completion = await DeadLetterAsync(delivery, $"Exceeded max attempts ({_options.MaxAttempts})").AnyContext(); + else + { + var outcome = await handler(context, token).AnyContext(); + if (!context.IsCompleted && !context.IsAbandoned) + { + if (outcome.Kind == MessageOutcomeKind.Retry) + completion = await FailureAsync(delivery, outcome.Reason ?? "Processing failed").AnyContext(); + else if (outcome.Kind == MessageOutcomeKind.DeadLetter) + completion = await DeadLetterAsync(delivery, outcome.Reason ?? "Processing rejected").AnyContext(); + else if (_options.AutoComplete && outcome.Kind != MessageOutcomeKind.Unsettled) + { + token.ThrowIfCancellationRequested(); + await SettleAsync(context.CompleteAsync, delivery).AnyContext(); + } + } + } + } + catch (OperationCanceledException) when (delivery.IsLeaseLost) { } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + if (!context.IsCompleted && !context.IsAbandoned) + await SettleAsync(ct => delivery.RejectAsync(new RejectOptions { RedeliveryDelay = TimeSpan.Zero }, ct), delivery).AnyContext(); + } + catch (OperationCanceledException) + { + if (!context.IsCompleted && !context.IsAbandoned) + { + if (attempt is not null && await IsCancelledAsync(attempt.JobId, CancellationToken.None).AnyContext()) + { + if (await SettleAsync(delivery.CompleteAsync, delivery).AnyContext()) + completion = new() { Kind = JobCompletionKind.Cancelled }; + } + else completion = await FailureAsync(delivery, "Processing was cancelled").AnyContext(); + } + } + catch (Exception exception) + { + _logger.LogError(exception, "Processing failed for {MessageId} at {Queue} on attempt {Attempt}", delivery.Id, _options.QueueName, delivery.Attempts); + if (!delivery.IsLeaseLost && !context.IsCompleted && !context.IsAbandoned) + completion = await FailureAsync(delivery, exception.Message).AnyContext(); + } + finally + { + await processing.CancelAsync().AnyContext(); + if (poll is not null) await poll.AnyContext(); + if (context.IsCompleted) completion = new() { Kind = JobCompletionKind.Succeeded }; + if (attempt is not null) + await RecordAsync(ct => _store!.CompleteJobAsync(attempt.JobId, attempt.ClaimToken!, completion, ct)).AnyContext(); + var outcome = completion.Kind switch + { + JobCompletionKind.Succeeded => MessageOutcomeKind.Success, + JobCompletionKind.Failed when !completion.Retryable => MessageOutcomeKind.DeadLetter, + JobCompletionKind.Failed => MessageOutcomeKind.Retry, + _ => (MessageOutcomeKind?)null + }; + if (outcome is { } kind) + { + _options.OnProcessed?.Invoke(kind, Stopwatch.GetElapsedTime(started)); + if (_store is not null) + await RecordAsync(ct => _store.IncrementCounterAsync(_options.QueueName, kind == MessageOutcomeKind.Success ? "processed" : kind == MessageOutcomeKind.DeadLetter ? "dead_lettered" : "failed", 1, ct)).AnyContext(); + } + } + } + + private async Task FailureAsync(IMessageContext delivery, string reason) + { + if (_options.AutoComplete && _options.MaxAttempts > 0 && delivery.Attempts >= _options.MaxAttempts) + return await DeadLetterAsync(delivery, reason).AnyContext(); + if (_options.AutoComplete) + await SettleAsync(ct => delivery.RejectAsync(new RejectOptions { RedeliveryDelay = _options.RetryBackoff(delivery.Attempts) }, ct), delivery).AnyContext(); + return new() { Kind = JobCompletionKind.Failed, Error = reason }; + } + + private async Task DeadLetterAsync(IMessageContext delivery, string reason) + => await SettleAsync(ct => MessageOutcome.DeadLetter(reason).SettleFailureAsync(delivery, _options.MaxAttempts, _options.RetryBackoff, ct), delivery).AnyContext() + ? new() { Kind = JobCompletionKind.Failed, Retryable = false, Error = reason } + : new() { Kind = JobCompletionKind.Interrupted, Error = reason }; + + private Task IsCancelledAsync(string jobId, CancellationToken token) => RunAsync(ct => _store!.IsCancellationRequestedAsync(jobId, ct), token); + + private async Task PollCancellationAsync(JobState attempt, CancellationTokenSource processing) + { + var token = processing.Token; + while (!token.IsCancellationRequested) + { + try + { + await Task.Delay(_options.CancellationPollInterval, _time, token).ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing); + if (token.IsCancellationRequested) + return; + if (await IsCancelledAsync(attempt.JobId, token).AnyContext()) + { + await processing.CancelAsync().AnyContext(); + return; + } + await RecordAsync(ct => _store!.HeartbeatJobAsync(attempt.JobId, attempt.ClaimToken!, ct), token).AnyContext(); + } + catch (OperationCanceledException) when (token.IsCancellationRequested) { return; } + catch (Exception exception) { _logger.LogWarning(exception, "Unable to poll job {JobId}; retrying", attempt.JobId); } + } + } + + private async Task SettleAsync(Func operation, IMessageContext delivery) + { + try { await RunAsync(operation).AnyContext(); return true; } + catch (Exception exception) + { + _logger.LogError(exception, "Settlement was not confirmed for {MessageId} at {Queue}; delivery may recur", delivery.Id, _options.QueueName); + return false; + } + } + + private Task RecordAsync(Func> operation, CancellationToken cancellationToken = default) + => RecordAsync(async ct => { _ = await operation(ct).AnyContext(); }, cancellationToken); + + private async Task RecordAsync(Func operation, CancellationToken cancellationToken = default) + { + try { await RunAsync(operation, cancellationToken).AnyContext(); } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) { throw; } + catch (Exception exception) { _logger.LogWarning(exception, "Unable to persist job history for {Queue}", _options.QueueName); } + } + + private async Task RunAsync(Func operation, CancellationToken cancellationToken = default) + { + using var deadline = new CancellationTokenSource(OperationTimeout, _time); + using var linked = cancellationToken.CanBeCanceled ? CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, deadline.Token) : null; + var token = linked?.Token ?? deadline.Token; + await operation(token).WaitAsync(token).AnyContext(); + } + + private async Task RunAsync(Func> operation, CancellationToken cancellationToken = default) + { + using var deadline = new CancellationTokenSource(OperationTimeout, _time); + using var linked = cancellationToken.CanBeCanceled ? CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, deadline.Token) : null; + var token = linked?.Token ?? deadline.Token; + return await operation(token).WaitAsync(token).AnyContext(); + } +} diff --git a/src/Foundatio/Messaging/Tracking/MessageProcessingContext.cs b/src/Foundatio/Messaging/Tracking/MessageProcessingContext.cs new file mode 100644 index 000000000..59509a74b --- /dev/null +++ b/src/Foundatio/Messaging/Tracking/MessageProcessingContext.cs @@ -0,0 +1,198 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +namespace Foundatio.Messaging; + +/// Execution progress, cancellation and explicit settlement for a broker-delivered message. +public class MessageProcessingContext +{ + private SemaphoreSlim? _settlementGate; + private int _settlement; + + /// The serialized application payload. + public ReadOnlyMemory Body { get; init; } + + internal Action? OnSettled { get; init; } + internal Action? OnCancelProcessing { get; init; } + + /// Cancel cooperative processing after losing an application resource lock. + public void CancelProcessing() => OnCancelProcessing?.Invoke(); + + /// + /// The name of the queue this message was received from. + /// + public string QueueName { get; init; } = string.Empty; + + /// + /// The transport-assigned id of the message being processed. + /// + public string MessageId { get; init; } = string.Empty; + + /// + /// The visibility timeout the worker requested for this message. + /// + public TimeSpan VisibilityTimeout { get; init; } + + /// + /// The message type being processed. + /// + public Type? MessageType { get; init; } + + /// + /// The number of times this message has been dequeued (including the current attempt). + /// Useful for detecting poison messages or implementing backoff strategies. + /// + public int DequeueCount { get; init; } + + /// + /// The maximum number of attempts configured for this queue. + /// After this many attempts, the message will be dead-lettered. + /// + public int MaxAttempts { get; init; } + + /// + /// When the message was originally enqueued. + /// + public DateTimeOffset EnqueuedAt { get; init; } + + /// + /// The unique job identifier for progress tracking, or null if tracking is not enabled. + /// + public string? JobId { get; init; } + + /// Message headers, including correlation, propagated context, and replay lineage. + public IReadOnlyDictionary Headers { get; init; } = MessageHeaders.Empty; + + /// + /// Delegate invoked by to signal that the handler + /// is still actively working. This acts as a heartbeat keep-alive that extends the + /// message visibility by the configured timeout. Set by the worker infrastructure. + /// + internal Func? OnReportProgress { get; init; } + + /// + /// Delegate invoked by + /// to update progress percentage and message in the state store. + /// Set by the worker infrastructure when progress tracking is enabled. + /// + internal Func? OnReportDetailedProgress { get; init; } + + /// + /// Delegate invoked by to extend the message lock + /// or visibility timeout by a specific duration. Set by the worker infrastructure. + /// + internal Func? OnRenewTimeout { get; init; } + + /// + /// Delegate invoked by to remove the message from the queue. + /// Set by the worker infrastructure. + /// + internal Func? OnComplete { get; init; } + + /// + /// Delegate invoked by to + /// return the message to the queue for redelivery. Set by the worker infrastructure. + /// + internal Func? OnAbandon { get; init; } + + /// + /// Indicates whether the handler explicitly completed the message via . + /// When true, the worker infrastructure will skip automatic completion. + /// + public bool IsCompleted => Volatile.Read(ref _settlement) == 1; + + /// + /// Indicates whether the handler explicitly abandoned the message via + /// or . + /// When true, the worker infrastructure will skip automatic abandonment. + /// + public bool IsAbandoned => Volatile.Read(ref _settlement) == 2; + + /// + /// Reports that the handler is still actively processing the message. + /// For transports that support it, this extends the visibility timeout + /// by the configured default duration, preventing the message from being + /// redelivered during long-running operations. + /// + public Task ReportProgressAsync(CancellationToken cancellationToken = default) + => OnReportProgress?.Invoke(cancellationToken) ?? Task.CompletedTask; + + /// + /// Reports progress with a percentage and optional message. + /// When progress tracking is enabled, this updates the job state store + /// and checks for cancellation. If cancellation has been requested, + /// an is thrown. + /// Also acts as a heartbeat to extend the message visibility timeout. + /// + /// Progress percentage (0–100). + /// Optional description of current work. + /// A cancellation token. + public async Task ReportProgressAsync(int progressPercent, string? message = null, CancellationToken cancellationToken = default) + { + // Always renew the visibility timeout as a heartbeat + if (OnReportProgress is not null) + await OnReportProgress(cancellationToken).ConfigureAwait(false); + + // Update state store and check for cancellation + if (OnReportDetailedProgress is not null) + await OnReportDetailedProgress(progressPercent, message, cancellationToken).ConfigureAwait(false); + } + + /// + /// Extends the message lock or visibility timeout by the specified duration. + /// Use this for long-running handlers to prevent the message from being + /// redelivered to another consumer. + /// + public Task RenewTimeoutAsync(TimeSpan extension, CancellationToken cancellationToken = default) + => OnRenewTimeout?.Invoke(extension, cancellationToken) ?? Task.CompletedTask; + + /// + /// Completes the message, removing it from the queue permanently. + /// Use this when AutoComplete is disabled and the handler has finished + /// processing successfully. If AutoComplete is enabled, the worker + /// infrastructure will skip its own completion when this has been called. + /// + public Task CompleteAsync(CancellationToken cancellationToken = default) + => SettleAsync(1, OnComplete, cancellationToken); + + private async Task SettleAsync(int outcome, Func? operation, CancellationToken cancellationToken) + { + var gate = LazyInitializer.EnsureInitialized(ref _settlementGate, static () => new SemaphoreSlim(1)); + await gate.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + if (_settlement == outcome) + return; + if (_settlement != 0) + throw new InvalidOperationException("This delivery has already been settled with a different outcome."); + if (operation is not null) + await operation(cancellationToken).ConfigureAwait(false); + Volatile.Write(ref _settlement, outcome); + OnSettled?.Invoke(); + } + finally + { + gate.Release(); + } + } + + /// + /// Abandons the message so it becomes immediately visible for redelivery. + /// Use this when AutoComplete is disabled and the handler cannot + /// process the message successfully. + /// + public Task AbandonAsync(CancellationToken cancellationToken = default) + => AbandonAsync(TimeSpan.Zero, cancellationToken); + + /// + /// Abandons the message so it becomes visible for redelivery after the specified delay. + /// Use this when AutoComplete is disabled and the handler wants to retry + /// the message after a backoff period. + /// + /// How long before the message becomes visible again. Use for immediate redelivery. + /// A cancellation token. + public Task AbandonAsync(TimeSpan delay, CancellationToken cancellationToken = default) + => SettleAsync(2, ct => OnAbandon?.Invoke(delay, ct) ?? Task.CompletedTask, cancellationToken); +} diff --git a/src/Foundatio/Properties/AssemblyInfo.cs b/src/Foundatio/Properties/AssemblyInfo.cs index ad9840441..1895ba706 100644 --- a/src/Foundatio/Properties/AssemblyInfo.cs +++ b/src/Foundatio/Properties/AssemblyInfo.cs @@ -3,3 +3,5 @@ [assembly: InternalsVisibleTo("Foundatio.Tests, PublicKey=0024000004800000940000000602000000240000525341310004000001000100a9357232b9bcad78fd310297fdb41bf42816ee2ca9ccdace999889de2badb6f06df2de1d9f2c8cb17b21f5311f11d6bb328d55e0dd9fe8adc5e2dc4610028c1bdacb3355d2e239b81d0bb0ac83e615fc641f8a3ec49e4fad8e305994953d448ef7b38e8c256601e54af19c035b562e3e5e5461c2a93b8dd11936e451b05034a2")] [assembly: InternalsVisibleTo("Foundatio.TestHarness, PublicKey=0024000004800000940000000602000000240000525341310004000001000100a9357232b9bcad78fd310297fdb41bf42816ee2ca9ccdace999889de2badb6f06df2de1d9f2c8cb17b21f5311f11d6bb328d55e0dd9fe8adc5e2dc4610028c1bdacb3355d2e239b81d0bb0ac83e615fc641f8a3ec49e4fad8e305994953d448ef7b38e8c256601e54af19c035b562e3e5e5461c2a93b8dd11936e451b05034a2")] [assembly: InternalsVisibleTo("Foundatio.Benchmarks, PublicKey=0024000004800000940000000602000000240000525341310004000001000100a9357232b9bcad78fd310297fdb41bf42816ee2ca9ccdace999889de2badb6f06df2de1d9f2c8cb17b21f5311f11d6bb328d55e0dd9fe8adc5e2dc4610028c1bdacb3355d2e239b81d0bb0ac83e615fc641f8a3ec49e4fad8e305994953d448ef7b38e8c256601e54af19c035b562e3e5e5461c2a93b8dd11936e451b05034a2")] + +[assembly: InternalsVisibleTo("Foundatio.Extensions.Hosting, PublicKey=0024000004800000940000000602000000240000525341310004000001000100a9357232b9bcad78fd310297fdb41bf42816ee2ca9ccdace999889de2badb6f06df2de1d9f2c8cb17b21f5311f11d6bb328d55e0dd9fe8adc5e2dc4610028c1bdacb3355d2e239b81d0bb0ac83e615fc641f8a3ec49e4fad8e305994953d448ef7b38e8c256601e54af19c035b562e3e5e5461c2a93b8dd11936e451b05034a2")] diff --git a/src/Foundatio/Queues/DuplicateDetectionQueueBehavior.cs b/src/Foundatio/Queues/DuplicateDetectionQueueBehavior.cs deleted file mode 100644 index 42cdf078e..000000000 --- a/src/Foundatio/Queues/DuplicateDetectionQueueBehavior.cs +++ /dev/null @@ -1,58 +0,0 @@ -using System; -using System.Threading.Tasks; -using Foundatio.Caching; -using Microsoft.Extensions.Logging; - -namespace Foundatio.Queues; - -/// -/// Automatically detects and discards duplicate entries in a queue based on a unique identifier. -/// -public class DuplicateDetectionQueueBehavior : QueueBehaviorBase where T : class -{ - private readonly ICacheClient _cacheClient; - private readonly ILoggerFactory _loggerFactory; - private readonly TimeSpan _detectionWindow; - - public DuplicateDetectionQueueBehavior(ICacheClient cacheClient, ILoggerFactory loggerFactory, TimeSpan? detectionWindow = null) - { - _cacheClient = cacheClient; - _loggerFactory = loggerFactory; - _detectionWindow = detectionWindow ?? TimeSpan.FromMinutes(10); - } - - protected override async Task OnEnqueuing(object sender, EnqueuingEventArgs enqueuingEventArgs) - { - string? uniqueIdentifier = GetUniqueIdentifier(enqueuingEventArgs.Data); - if (String.IsNullOrEmpty(uniqueIdentifier)) - return; - - bool success = await _cacheClient.AddAsync(uniqueIdentifier, true, _detectionWindow); - if (!success) - { - var logger = _loggerFactory.CreateLogger(); - logger.LogInformation("Discarding queue entry due to duplicate {UniqueIdentifier}", uniqueIdentifier); - enqueuingEventArgs.Cancel = true; - } - } - - protected override async Task OnDequeued(object sender, DequeuedEventArgs dequeuedEventArgs) - { - string? uniqueIdentifier = GetUniqueIdentifier(dequeuedEventArgs.Entry.Value); - if (String.IsNullOrEmpty(uniqueIdentifier)) - return; - - await _cacheClient.RemoveAsync(uniqueIdentifier); - } - - private string? GetUniqueIdentifier(T data) - { - var haveUniqueIdentifier = data as IHaveUniqueIdentifier; - return haveUniqueIdentifier?.UniqueIdentifier; - } -} - -public interface IHaveUniqueIdentifier -{ - string? UniqueIdentifier { get; } -} diff --git a/src/Foundatio/Queues/IQueue.cs b/src/Foundatio/Queues/IQueue.cs deleted file mode 100644 index 2e1e8e3e5..000000000 --- a/src/Foundatio/Queues/IQueue.cs +++ /dev/null @@ -1,358 +0,0 @@ -using System; -using System.Collections.Generic; -using System.ComponentModel; -using System.Diagnostics; -using System.Diagnostics.CodeAnalysis; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Serializer; -using Foundatio.Utility; - -namespace Foundatio.Queues; - -/// -/// A typed message queue that supports enqueue, dequeue, and work item lifecycle management. -/// Entries are processed with at-least-once delivery semantics and must be explicitly completed or abandoned. -/// -/// The type of message payload stored in the queue. -public interface IQueue : IQueue where T : class -{ - /// - /// Raised before an item is enqueued. Set to prevent enqueueing. - /// - AsyncEvent> Enqueuing { get; } - - /// - /// Raised after an item has been successfully enqueued. - /// - AsyncEvent> Enqueued { get; } - - /// - /// Raised after an item has been dequeued and is ready for processing. - /// - AsyncEvent> Dequeued { get; } - - /// - /// Raised after a queue entry's lock has been renewed. - /// - AsyncEvent> LockRenewed { get; } - - /// - /// Raised after a queue entry has been marked as completed. - /// - AsyncEvent> Completed { get; } - - /// - /// Raised after a queue entry has been abandoned and returned to the queue. - /// When a poison message (deserialization failure) is detected, the entry is automatically - /// abandoned and this event is raised with a phantom entry whose - /// is null at runtime. See for details. - /// - AsyncEvent> Abandoned { get; } - - /// - /// Raised after the queue has been deleted. - /// - AsyncEvent> QueueDeleted { get; } - - /// - /// Attaches a behavior that can intercept and modify queue operations. - /// - /// The behavior to attach. - void AttachBehavior(IQueueBehavior behavior); - - /// - /// Adds an item to the queue for processing. - /// - /// The message payload to enqueue. - /// Optional settings for delivery delay, correlation ID, and custom properties. - /// The unique identifier assigned to the queued entry. - Task EnqueueAsync(T data, QueueEntryOptions? options = null); - - /// - /// Retrieves and locks the next available item from the queue. - /// Blocks until an item is available or the cancellation token is triggered. - /// - /// Token to cancel the wait for an item. - /// The dequeued entry, or null if cancelled before an item became available. - Task?> DequeueAsync(CancellationToken cancellationToken); - - /// - /// Retrieves and locks the next available item from the queue. - /// - /// Maximum time to wait for an item. Defaults to 30 seconds. - /// The dequeued entry, or null if no item was available within the timeout. - Task?> DequeueAsync(TimeSpan? timeout = null); - - /// - /// Extends the processing lock on a queue entry to prevent it from being redelivered. - /// Call periodically for long-running work items. - /// - /// The entry whose lock should be renewed. - Task RenewLockAsync(IQueueEntry queueEntry); - - /// - /// Marks a queue entry as successfully processed and removes it from the queue. - /// - /// The entry to complete. - Task CompleteAsync(IQueueEntry queueEntry); - - /// - /// Returns a queue entry to the queue for reprocessing. - /// The entry will be redelivered after a delay, up to the maximum retry limit. - /// - /// The entry to abandon. - Task AbandonAsync(IQueueEntry queueEntry); - - /// - /// Retrieves items that have exceeded the maximum retry attempts. - /// - /// Token to cancel the operation. - /// The collection of dead-lettered message payloads. - Task> GetDeadletterItemsAsync(CancellationToken cancellationToken = default); - - /// - /// Starts a background worker that continuously dequeues and processes items. - /// - /// The async function invoked for each dequeued entry. - /// - /// If true, automatically calls after the handler completes successfully. - /// If false (default), the handler must explicitly complete or abandon the entry. - /// - /// Token to stop the background worker. - Task StartWorkingAsync(Func, CancellationToken, Task> handler, bool autoComplete = false, CancellationToken cancellationToken = default); -} - -/// -/// Base interface for queue operations that are not type-specific. -/// -public interface IQueue : IHaveSerializer, IDisposable -{ - /// - /// Gets current queue statistics including counts for queued, working, and dead-lettered items. - /// - Task GetQueueStatsAsync(); - - /// - /// Permanently deletes the queue and all its contents. - /// - Task DeleteQueueAsync(); - - /// - /// Gets the unique identifier for this queue instance. - /// - string QueueId { get; } -} - -public static class QueueExtensions -{ - public static Task StartWorkingAsync(this IQueue queue, Func, Task> handler, bool autoComplete = false, CancellationToken cancellationToken = default) where T : class - => queue.StartWorkingAsync((entry, token) => handler(entry), autoComplete, cancellationToken); -} - -/// -/// Provides statistics about queue state and processing activity. -/// -[DebuggerDisplay("Queued={Queued}, Working={Working}, Deadletter={Deadletter}, Enqueued={Enqueued}, Dequeued={Dequeued}, Completed={Completed}, Abandoned={Abandoned}, Errors={Errors}, Timeouts={Timeouts}")] -public record QueueStats -{ - /// - /// Number of items waiting to be processed. - /// - public long Queued { get; set; } - - /// - /// Number of items currently being processed (dequeued but not yet completed or abandoned). - /// - public long Working { get; set; } - - /// - /// Number of items that exceeded retry limits and were moved to the dead-letter queue. - /// - public long Deadletter { get; set; } - - /// - /// Total number of items that have been enqueued since queue creation. - /// - public long Enqueued { get; set; } - - /// - /// Total number of items that have been dequeued since queue creation. - /// - public long Dequeued { get; set; } - - /// - /// Total number of items that have been successfully completed since queue creation. - /// - public long Completed { get; set; } - - /// - /// Total number of times items have been abandoned since queue creation. - /// - public long Abandoned { get; set; } - - /// - /// Total number of processing errors since queue creation. - /// - public long Errors { get; set; } - - /// - /// Total number of items that timed out during processing since queue creation. - /// - public long Timeouts { get; set; } -} - -/// -/// Options for customizing how a message is enqueued. -/// -public record QueueEntryOptions -{ - /// - /// A unique identifier for the message. If not specified, one will be generated. - /// Can be used for deduplication or idempotency checks. - /// - public string? UniqueId { get; set; } - - /// - /// A correlation identifier for distributed tracing across services. - /// - public string? CorrelationId { get; set; } - - /// - /// Delay before the message becomes visible for processing. - /// - public TimeSpan? DeliveryDelay { get; set; } - - /// - /// Custom properties to attach to the message. - /// - [DisallowNull] - public IDictionary Properties { get => field; set => field = value ?? new Dictionary(); } = new Dictionary(); -} - -/// -/// Event arguments for the event. -/// -/// The type of message payload. -public class EnqueuingEventArgs : CancelEventArgs where T : class -{ - /// - /// The queue raising the event. - /// - public required IQueue Queue { get; set; } - - /// - /// The message payload being enqueued. - /// - public required T Data { get; set; } - - /// - /// The options for the enqueue operation. - /// - public required QueueEntryOptions Options { get; set; } -} - -/// -/// Event arguments for the event. -/// -/// The type of message payload. -public class EnqueuedEventArgs : EventArgs where T : class -{ - /// - /// The queue raising the event. - /// - public required IQueue Queue { get; set; } - - /// - /// The queue entry that was enqueued. - /// - public required IQueueEntry Entry { get; set; } -} - -/// -/// Event arguments for the event. -/// -/// The type of message payload. -public class DequeuedEventArgs : EventArgs where T : class -{ - /// - /// The queue raising the event. - /// - public required IQueue Queue { get; set; } - - /// - /// The queue entry that was dequeued. - /// - public required IQueueEntry Entry { get; set; } -} - -/// -/// Event arguments for the event. -/// -/// The type of message payload. -public class LockRenewedEventArgs : EventArgs where T : class -{ - /// - /// The queue raising the event. - /// - public required IQueue Queue { get; set; } - - /// - /// The queue entry whose lock was renewed. - /// - public required IQueueEntry Entry { get; set; } -} - -/// -/// Event arguments for the event. -/// -/// The type of message payload. -public class CompletedEventArgs : EventArgs where T : class -{ - /// - /// The queue raising the event. - /// - public required IQueue Queue { get; set; } - - /// - /// The queue entry that was completed. - /// - public required IQueueEntry Entry { get; set; } -} - -/// -/// Event arguments for the event. -/// -/// The type of message payload. -/// -/// When a queue entry is abandoned due to a deserialization failure (poison message), -/// . may be null at runtime -/// even though the property is typed as non-nullable. This occurs because the message -/// payload could not be deserialized, so the queue creates a phantom entry with a null value -/// that is immediately abandoned. Handlers subscribing to the -/// event should check for null before accessing Value if they need the payload. -/// -public class AbandonedEventArgs : EventArgs where T : class -{ - /// - /// The queue raising the event. - /// - public required IQueue Queue { get; set; } - - /// - /// The queue entry that was abandoned. - /// - public required IQueueEntry Entry { get; set; } -} - -/// -/// Event arguments for the event. -/// -/// The type of message payload. -public class QueueDeletedEventArgs : EventArgs where T : class -{ - /// - /// The queue raising the event. - /// - public required IQueue Queue { get; set; } -} diff --git a/src/Foundatio/Queues/IQueueActivity.cs b/src/Foundatio/Queues/IQueueActivity.cs deleted file mode 100644 index 4ded13280..000000000 --- a/src/Foundatio/Queues/IQueueActivity.cs +++ /dev/null @@ -1,19 +0,0 @@ -using System; - -namespace Foundatio.Queues; - -/// -/// Provides activity timestamps for monitoring queue health and detecting idle queues. -/// -public interface IQueueActivity -{ - /// - /// Gets the timestamp of the last enqueue operation, or null if no items have been enqueued. - /// - DateTimeOffset? LastEnqueueActivity { get; } - - /// - /// Gets the timestamp of the last dequeue operation, or null if no items have been dequeued. - /// - DateTimeOffset? LastDequeueActivity { get; } -} diff --git a/src/Foundatio/Queues/IQueueEntry.cs b/src/Foundatio/Queues/IQueueEntry.cs deleted file mode 100644 index 1d1032171..000000000 --- a/src/Foundatio/Queues/IQueueEntry.cs +++ /dev/null @@ -1,113 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Threading.Tasks; - -namespace Foundatio.Queues; - -/// -/// Represents a dequeued item with its processing state and lifecycle methods. -/// Each entry holds a lock that must be renewed for long-running operations. -/// -public interface IQueueEntry -{ - /// - /// Gets the unique identifier for this queue entry. - /// - string Id { get; } - - /// - /// Gets the correlation identifier for distributed tracing. - /// - string? CorrelationId { get; } - - /// - /// Gets custom properties attached to this entry. - /// - IDictionary Properties { get; } - - /// - /// Gets the CLR type of the message payload. - /// - /// - /// For poison messages (deserialization failures), the return value is null. - /// - Type? EntryType { get; } - - /// - /// Gets the message payload as an untyped object. - /// - /// - /// For poison messages (deserialization failures), the return value is null at runtime - /// even though the signature is non-nullable. Poison entries are immediately abandoned and - /// only observable via the event. - /// - object GetValue(); - - /// - /// Gets whether this entry has been marked as completed. - /// - bool IsCompleted { get; } - - /// - /// Gets whether this entry has been marked as abandoned. - /// - bool IsAbandoned { get; } - - /// - /// Gets the number of times this entry has been dequeued, including the current attempt. - /// Useful for implementing retry limits or exponential backoff. - /// - int Attempts { get; } - - /// - /// Marks this entry as abandoned locally without notifying the queue. - /// Use to return the entry to the queue for reprocessing. - /// - void MarkAbandoned(); - - /// - /// Marks this entry as completed locally without notifying the queue. - /// Use to remove the entry from the queue. - /// - void MarkCompleted(); - - /// - /// Extends the processing lock to prevent the entry from being redelivered. - /// Call periodically for long-running work items. - /// - Task RenewLockAsync(); - - /// - /// Returns this entry to the queue for reprocessing. - /// The entry will be redelivered after a delay, up to the maximum retry limit. - /// - Task AbandonAsync(); - - /// - /// Marks this entry as successfully processed and removes it from the queue. - /// - Task CompleteAsync(); - - /// - /// Releases resources associated with this entry. - /// If not completed or abandoned, the entry will be automatically abandoned. - /// - ValueTask DisposeAsync(); -} - -/// -/// A typed queue entry providing strongly-typed access to the message payload. -/// -/// The type of message payload. -public interface IQueueEntry : IQueueEntry where T : class -{ - /// - /// Gets the deserialized message payload. - /// - /// - /// For poison messages (deserialization failures), the return value is null at runtime - /// even though the property is typed as non-nullable. Poison entries are immediately abandoned - /// and only observable via the event. - /// - T Value { get; } -} diff --git a/src/Foundatio/Queues/InMemoryQueue.cs b/src/Foundatio/Queues/InMemoryQueue.cs deleted file mode 100644 index 25780b213..000000000 --- a/src/Foundatio/Queues/InMemoryQueue.cs +++ /dev/null @@ -1,445 +0,0 @@ -using System; -using System.Collections.Concurrent; -using System.Collections.Generic; -using System.Collections.ObjectModel; -using System.Diagnostics; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.AsyncEx; -using Foundatio.Utility; -using Microsoft.Extensions.Logging; - -namespace Foundatio.Queues; - -public class InMemoryQueue : QueueBase> where T : class -{ - private readonly ConcurrentQueue> _queue = new(); - private readonly ConcurrentDictionary> _dequeued = new(); - private readonly ConcurrentQueue> _deadletterQueue = new(); - private readonly ConcurrentQueue> _completedQueue = new(); - private readonly AsyncAutoResetEvent _autoResetEvent = new(); - - private int _enqueuedCount; - private int _dequeuedCount; - private int _completedCount; - private int _abandonedCount; - private int _workerErrorCount; - private int _workerItemTimeoutCount; - private int _pendingRetryCount; - - public InMemoryQueue() : this(o => o) { } - - public InMemoryQueue(InMemoryQueueOptions options) : base(options) - { - InitializeMaintenance(); - } - - public InMemoryQueue(Builder, InMemoryQueueOptions> config) - : this(config(new InMemoryQueueOptionsBuilder()).Build()) { } - - protected override Task EnsureQueueCreatedAsync(CancellationToken cancellationToken = default) - { - return Task.CompletedTask; - } - - protected override Task GetQueueStatsImplAsync() - { - return Task.FromResult(GetMetricsQueueStats()); - } - - protected override QueueStats GetMetricsQueueStats() - { - return new QueueStats - { - Queued = _queue.Count + _pendingRetryCount, - Working = _dequeued.Count, - Deadletter = _deadletterQueue.Count, - Enqueued = _enqueuedCount, - Dequeued = _dequeuedCount, - Completed = _completedCount, - Abandoned = _abandonedCount, - Errors = _workerErrorCount, - Timeouts = _workerItemTimeoutCount - }; - } - - public IReadOnlyCollection> GetEntries() - { - return new ReadOnlyCollection>(_queue.ToList()); - } - - public IReadOnlyCollection> GetDequeuedEntries() - { - return new ReadOnlyCollection>(_dequeued.Values.ToList()); - } - - public IReadOnlyCollection> GetCompletedEntries() - { - return new ReadOnlyCollection>(_completedQueue.ToList()); - } - - public IReadOnlyCollection> GetDeadletterEntries() - { - return new ReadOnlyCollection>(_deadletterQueue.ToList()); - } - - protected override async Task EnqueueImplAsync(T data, QueueEntryOptions options) - { - string id = !String.IsNullOrEmpty(options.UniqueId) ? options.UniqueId : Guid.NewGuid().ToString("N"); - _logger.LogTrace("Queue {QueueName} enqueue item: {QueueEntryId}", _options.Name, id); - - if (!await OnEnqueuingAsync(data, options).AnyContext()) - return null; - - var entry = new QueueEntry(id, options.CorrelationId, data.DeepClone(), this, _timeProvider.GetUtcNow().UtcDateTime, 0); - entry.Properties.AddRange(options.Properties); - - Interlocked.Increment(ref _enqueuedCount); - - if (options.DeliveryDelay is not null && options.DeliveryDelay.Value > TimeSpan.Zero) - { - _ = Run.DelayedAsync(options.DeliveryDelay.Value, async () => - { - _queue.Enqueue(entry); - _logger.LogTrace("Enqueue: Set Event"); - - _autoResetEvent.Set(); - - await OnEnqueuedAsync(entry).AnyContext(); - _logger.LogTrace("Enqueue done"); - }, _timeProvider, DisposedCancellationToken); - return id; - } - - _queue.Enqueue(entry); - _logger.LogTrace("Enqueue: Set Event"); - - _autoResetEvent.Set(); - - await OnEnqueuedAsync(entry).AnyContext(); - _logger.LogTrace("Enqueue done"); - - return id; - } - - private readonly List _workers = new(); - - protected override void StartWorkingImpl(Func, CancellationToken, Task> handler, bool autoComplete, CancellationToken cancellationToken) - { - ArgumentNullException.ThrowIfNull(handler); - - _logger.LogTrace("Queue {QueueName} start working", _options.Name); - - var linkedCancellationTokenSource = GetLinkedDisposableCancellationTokenSource(cancellationToken); - _workers.Add(Task.Run(async () => - { - using var _ = new DisposableAction(linkedCancellationTokenSource.Dispose); - _logger.LogTrace("WorkerLoop Start {QueueName}", _options.Name); - - while (!linkedCancellationTokenSource.IsCancellationRequested) - { - _logger.LogTrace("WorkerLoop Signaled {QueueName}", _options.Name); - - IQueueEntry? queueEntry = null; - try - { - queueEntry = await DequeueImplAsync(linkedCancellationTokenSource.Token).AnyContext(); - } - catch (Exception ex) - { - _logger.LogError(ex, "Error on Dequeue: {Message}", ex.Message); - } - - if (linkedCancellationTokenSource.IsCancellationRequested || queueEntry is null) - return; - - try - { - await handler(queueEntry, linkedCancellationTokenSource.Token).AnyContext(); - } - catch (Exception ex) - { - _logger.LogError(ex, "Worker error: {Message}", ex.Message); - - if (!queueEntry.IsAbandoned && !queueEntry.IsCompleted) - { - try - { - await _resiliencePolicy.ExecuteAsync(async _ => await queueEntry.AbandonAsync(), linkedCancellationTokenSource.Token).AnyContext(); - } - catch (Exception abandonEx) - { - _logger.LogError(abandonEx, "Worker error abandoning queue entry: {Message}", abandonEx.Message); - } - } - - Interlocked.Increment(ref _workerErrorCount); - } - - if (autoComplete && !queueEntry.IsAbandoned && !queueEntry.IsCompleted) - { - try - { - await _resiliencePolicy.ExecuteAsync(async _ => await queueEntry.CompleteAsync(), linkedCancellationTokenSource.Token).AnyContext(); - } - catch (Exception ex) - { - _logger.LogError(ex, "Worker error attempting to auto complete entry: {Message}", ex.Message); - } - } - } - - _logger.LogTrace("Worker exiting: {QueueName} Cancel Requested: {IsCancellationRequested}", _options.Name, linkedCancellationTokenSource.IsCancellationRequested); - }, linkedCancellationTokenSource.Token).ContinueWith(_ => linkedCancellationTokenSource.Dispose())); - } - - protected override async Task?> DequeueImplAsync(CancellationToken linkedCancellationToken) - { - _logger.LogTrace("Queue {QueueName} dequeuing item... Queue count: {Count}", _options.Name, _queue.Count); - - while (true) - { - while (_queue.Count is 0 && !linkedCancellationToken.IsCancellationRequested) - { - _logger.LogTrace("Waiting to dequeue item..."); - var sw = Stopwatch.StartNew(); - - using var dequeueCancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(linkedCancellationToken); - dequeueCancellationTokenSource.CancelAfter(TimeSpan.FromSeconds(10)); - - try - { - await _autoResetEvent.WaitAsync(dequeueCancellationTokenSource.Token).AnyContext(); - } - catch (OperationCanceledException) { } - - sw.Stop(); - _logger.LogTrace("Waited for dequeue: {Elapsed:g}", sw.Elapsed); - } - - if (_queue.Count is 0) - return null; - - _logger.LogTrace("Dequeue: Attempt"); - if (!_queue.TryDequeue(out var entry) || entry is null) - return null; - - ScheduleNextMaintenance(_timeProvider.GetUtcNow().UtcDateTime.Add(_options.WorkItemTimeout)); - - entry.Attempts++; - entry.DequeuedTimeUtc = _timeProvider.GetUtcNow().UtcDateTime; - - if (entry.Attempts > _options.Retries + 1) - { - _logger.LogInformation("Exceeded retry limit ({Attempts}/{Retries}), moving message {QueueEntryId} to dead letter", entry.Attempts, _options.Retries, entry.Id); - _deadletterQueue.Enqueue(entry); - Interlocked.Increment(ref _abandonedCount); - continue; - } - - if (!_dequeued.TryAdd(entry.Id, entry)) - throw new Exception("Unable to add item to the dequeued list"); - - Interlocked.Increment(ref _dequeuedCount); - _logger.LogTrace("Dequeue: Got Item"); - - await entry.RenewLockAsync(); - await OnDequeuedAsync(entry).AnyContext(); - - return entry; - } - } - - public override async Task RenewLockAsync(IQueueEntry queueEntry) - { - _logger.LogDebug("Queue {QueueName} renew lock item: {QueueEntryId}", _options.Name, queueEntry.Id); - - if (!_dequeued.TryGetValue(queueEntry.Id, out var targetEntry)) - return; - - targetEntry.RenewedTimeUtc = _timeProvider.GetUtcNow().UtcDateTime; - - await OnLockRenewedAsync(queueEntry).AnyContext(); - _logger.LogTrace("Renew lock done: {QueueEntryId}", queueEntry.Id); - } - - public override async Task CompleteAsync(IQueueEntry queueEntry) - { - _logger.LogDebug("Queue {QueueName} complete item: {QueueEntryId}", _options.Name, queueEntry.Id); - if (queueEntry.IsAbandoned || queueEntry.IsCompleted) - throw new InvalidOperationException("Queue entry has already been completed or abandoned"); - - if (!_dequeued.TryRemove(queueEntry.Id, out var info) || info is null) - throw new Exception("Unable to remove item from the dequeued list"); - - if (_options.CompletedEntryRetentionLimit > 0) - { - _completedQueue.Enqueue(info); - while (_completedQueue.Count > _options.CompletedEntryRetentionLimit) - _completedQueue.TryDequeue(out _); - } - - queueEntry.MarkCompleted(); - Interlocked.Increment(ref _completedCount); - await OnCompletedAsync(queueEntry).AnyContext(); - _logger.LogTrace("Complete done: {QueueEntryId}", queueEntry.Id); - } - - public override async Task AbandonAsync(IQueueEntry queueEntry) - { - _logger.LogDebug("Queue {QueueName}:{QueueId} abandon item: {QueueEntryId}", _options.Name, QueueId, queueEntry.Id); - - if (queueEntry.IsAbandoned || queueEntry.IsCompleted) - throw new InvalidOperationException("Queue entry has already been completed or abandoned"); - - Interlocked.Increment(ref _pendingRetryCount); - - if (!_dequeued.TryRemove(queueEntry.Id, out var targetEntry) || targetEntry is null) - { - Interlocked.Decrement(ref _pendingRetryCount); - - foreach (var kvp in _queue) - { - if (kvp.Id == queueEntry.Id) - throw new Exception("Unable to remove item from the dequeued list (item is in queue)"); - } - foreach (var kvp in _deadletterQueue) - { - if (kvp.Id == queueEntry.Id) - throw new Exception("Unable to remove item from the dequeued list (item is in dead letter)"); - } - - throw new Exception("Unable to remove item from the dequeued list"); - } - - queueEntry.MarkAbandoned(); - Interlocked.Increment(ref _abandonedCount); - _logger.LogTrace("Abandon complete: {QueueEntryId}", queueEntry.Id); - - try - { - await OnAbandonedAsync(queueEntry).AnyContext(); - } - finally - { - if (targetEntry.Attempts < _options.Retries + 1) - { - var retryEntry = targetEntry.CreateRetryEntry(); - if (_options.RetryDelay > TimeSpan.Zero) - { - Interlocked.Decrement(ref _pendingRetryCount); - _logger.LogTrace("Adding item to wait list for future retry: {QueueEntryId} Attempts: {QueueEntryAttempts}", queueEntry.Id, queueEntry.Attempts); - _ = Run.DelayedAsync(GetRetryDelay(targetEntry.Attempts), () => - { - Retry(retryEntry); - return Task.CompletedTask; - }, _timeProvider, DisposedCancellationToken); - } - else - { - _logger.LogTrace("Adding item back to queue for retry: {QueueEntryId} Attempts: {QueueEntryAttempts}", queueEntry.Id, queueEntry.Attempts); - Retry(retryEntry); - Interlocked.Decrement(ref _pendingRetryCount); - } - } - else - { - _logger.LogInformation("Exceeded retry limit ({Attempts}/{Retries}), moving message {QueueEntryId} to dead letter", targetEntry.Attempts, _options.Retries, queueEntry.Id); - Interlocked.Decrement(ref _pendingRetryCount); - _deadletterQueue.Enqueue(targetEntry); - } - } - } - - private void Retry(QueueEntry entry) - { - _logger.LogTrace("Queue {QueueName} retrying item: {QueueEntryId} Attempts: {QueueEntryAttempts}", _options.Name, entry.Id, entry.Attempts); - _queue.Enqueue(entry); - _autoResetEvent.Set(); - } - - private TimeSpan GetRetryDelay(int attempts) - { - int maxMultiplier = _options.RetryMultipliers.Length > 0 ? _options.RetryMultipliers.Last() : 1; - int multiplier = attempts <= _options.RetryMultipliers.Length ? _options.RetryMultipliers[attempts - 1] : maxMultiplier; - return TimeSpan.FromMilliseconds((int)(_options.RetryDelay.TotalMilliseconds * multiplier)); - } - - protected override Task> GetDeadletterItemsImplAsync(CancellationToken cancellationToken) - { - return Task.FromResult(_deadletterQueue.Select(i => i.Value).Where(v => v is not null).Cast()); - } - - protected override Task DeleteQueueImplAsync() - { - _queue.Clear(); - _deadletterQueue.Clear(); - _dequeued.Clear(); - _enqueuedCount = 0; - _dequeuedCount = 0; - _completedCount = 0; - _abandonedCount = 0; - _workerErrorCount = 0; - _pendingRetryCount = 0; - - return Task.CompletedTask; - } - - protected override async Task DoMaintenanceAsync() - { - var utcNow = _timeProvider.GetUtcNow(); - var minAbandonAt = DateTimeOffset.MaxValue; - - try - { - foreach (var entry in _dequeued.Values.ToList()) - { - var abandonAt = entry.RenewedTimeUtc.Add(_options.WorkItemTimeout); - if (abandonAt < utcNow) - { - _logger.LogInformation("DoMaintenance Abandon: {QueueEntryId}", entry.Id); - - await AbandonAsync(entry).AnyContext(); - Interlocked.Increment(ref _workerItemTimeoutCount); - } - else if (abandonAt < minAbandonAt) - minAbandonAt = abandonAt; - } - } - catch (Exception ex) - { - _logger.LogError(ex, "DoMaintenance Error: {Message}", ex.Message); - } - - // Add a tiny buffer just in case the schedule next timer fires early. - // The system clock typically has a resolution of 10-15 milliseconds, so timers cannot be more accurate than this resolution. - return minAbandonAt.UtcDateTime.SafeAdd(TimeSpan.FromMilliseconds(15)); - } - - public override void Dispose() - { - if (!SignalDispose()) - { - _logger.LogTrace("Queue {QueueName} ({QueueId}) dispose was already called", _options.Name, QueueId); - return; - } - - _queue.Clear(); - _deadletterQueue.Clear(); - _dequeued.Clear(); - - _logger.LogTrace("Got {WorkerCount} workers to cleanup", _workers.Count); - foreach (var worker in _workers) - { - if (worker.IsCompleted) - continue; - - _logger.LogTrace("Attempting to cleanup worker"); - if (!worker.Wait(TimeSpan.FromSeconds(5))) - _logger.LogError("Failed waiting for worker to stop"); - } - - base.Dispose(); - } -} diff --git a/src/Foundatio/Queues/InMemoryQueueOptions.cs b/src/Foundatio/Queues/InMemoryQueueOptions.cs deleted file mode 100644 index ec71f6f33..000000000 --- a/src/Foundatio/Queues/InMemoryQueueOptions.cs +++ /dev/null @@ -1,43 +0,0 @@ -using System; - -namespace Foundatio.Queues; - -public class InMemoryQueueOptions : SharedQueueOptions where T : class -{ - public TimeSpan RetryDelay { get; set; } = TimeSpan.FromMinutes(1); - public int CompletedEntryRetentionLimit { get; set; } = 100; - public int[] RetryMultipliers { get; set; } = { 1, 3, 5, 10 }; -} - -public class InMemoryQueueOptionsBuilder : SharedQueueOptionsBuilder, InMemoryQueueOptionsBuilder> where T : class -{ - public InMemoryQueueOptionsBuilder RetryDelay(TimeSpan retryDelay) - { - ArgumentOutOfRangeException.ThrowIfLessThan(retryDelay, TimeSpan.Zero); - - Target.RetryDelay = retryDelay; - return this; - } - - public InMemoryQueueOptionsBuilder CompletedEntryRetentionLimit(int retentionCount) - { - ArgumentOutOfRangeException.ThrowIfNegative(retentionCount); - - Target.CompletedEntryRetentionLimit = retentionCount; - return this; - } - - public InMemoryQueueOptionsBuilder RetryMultipliers(int[] multipliers) - { - ArgumentNullException.ThrowIfNull(multipliers); - - foreach (int multiplier in multipliers) - { - if (multiplier < 1) - throw new ArgumentOutOfRangeException(nameof(multipliers)); - } - - Target.RetryMultipliers = multipliers; - return this; - } -} diff --git a/src/Foundatio/Queues/QueueBase.cs b/src/Foundatio/Queues/QueueBase.cs deleted file mode 100644 index 7a14d674b..000000000 --- a/src/Foundatio/Queues/QueueBase.cs +++ /dev/null @@ -1,450 +0,0 @@ -using System; -using System.Collections.Concurrent; -using System.Collections.Generic; -using System.Diagnostics; -using System.Diagnostics.Metrics; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Metrics; -using Foundatio.Resilience; -using Foundatio.Serializer; -using Foundatio.Utility; -using Microsoft.Extensions.Logging; - -namespace Foundatio.Queues; - -public abstract class QueueBase : MaintenanceBase, IQueue, IHaveTimeProvider, IQueueActivity where T : class where TOptions : SharedQueueOptions -{ - protected readonly TOptions _options; - private readonly string _metricsPrefix; - protected readonly ISerializer _serializer; - protected readonly IResiliencePolicy _resiliencePolicy; - - private readonly Counter _enqueuedCounter; - private readonly Counter _dequeuedCounter; - private readonly Histogram _queueTimeHistogram; - private readonly Counter _completedCounter; - private readonly Histogram _processTimeHistogram; - private readonly Histogram _totalTimeHistogram; - private readonly Counter _abandonedCounter; -#pragma warning disable IDE0052 // Remove unread private members - private readonly ObservableGauge? _countGauge; - private readonly ObservableGauge? _workingGauge; - private readonly ObservableGauge? _deadletterGauge; -#pragma warning restore IDE0052 // Remove unread private members - private readonly TagList _emptyTags = default; - - private readonly List> _behaviors = new(); - private QueueStats? _queueStats; - private DateTimeOffset _nextQueueStatsUpdate = DateTimeOffset.MinValue; - - protected QueueBase(TOptions options) : base(options?.TimeProvider, options?.LoggerFactory) - { - ArgumentNullException.ThrowIfNull(options); - - _options = options; - _metricsPrefix = $"foundatio.{typeof(T).Name.ToLowerInvariant()}"; - if (!String.IsNullOrWhiteSpace(options.MetricsPrefix)) - _metricsPrefix = $"{_metricsPrefix}.{options.MetricsPrefix.Trim()}"; - - QueueId = $"{options.Name.Trim()}{Guid.NewGuid().ToString("N").Substring(10)}"; - - _serializer = options.Serializer; - options.Behaviors.ForEach(AttachBehavior); - - var resiliencePolicyProvider = _options.GetResiliencePolicyProvider() ?? DefaultResiliencePolicyProvider.Instance; - _resiliencePolicy = resiliencePolicyProvider.GetPolicy, IQueue, IQueue>(_logger, _timeProvider); - - // setup meters - _enqueuedCounter = FoundatioDiagnostics.Meter.CreateCounter(GetFullMetricName("enqueued"), description: "Number of enqueued items"); - _dequeuedCounter = FoundatioDiagnostics.Meter.CreateCounter(GetFullMetricName("dequeued"), description: "Number of dequeued items"); - _queueTimeHistogram = FoundatioDiagnostics.Meter.CreateHistogram(GetFullMetricName("queuetime"), description: "Time in queue", unit: "ms"); - _completedCounter = FoundatioDiagnostics.Meter.CreateCounter(GetFullMetricName("completed"), description: "Number of completed items"); - _processTimeHistogram = FoundatioDiagnostics.Meter.CreateHistogram(GetFullMetricName("processtime"), description: "Time to process items", unit: "ms"); - _totalTimeHistogram = FoundatioDiagnostics.Meter.CreateHistogram(GetFullMetricName("totaltime"), description: "Total time in queue", unit: "ms"); - _abandonedCounter = FoundatioDiagnostics.Meter.CreateCounter(GetFullMetricName("abandoned"), description: "Number of abandoned items"); - - if (!options.MetricsPollingEnabled) - return; - - var queueMetricValues = new InstrumentsValues(() => - { - if (IsDisposed || (options.MetricsPollingInterval > TimeSpan.Zero && _nextQueueStatsUpdate >= _timeProvider.GetUtcNow())) - { - if (_queueStats is not null) - { - _logger.LogTrace("Using cached queue stats for {QueueName} ({QueueId})", _options.Name, QueueId); - return (_queueStats.Queued, _queueStats.Working, _queueStats.Deadletter); - } - - _logger.LogTrace("Returning default queue stats for {QueueName} ({QueueId})", _options.Name, QueueId); - return (0, 0, 0); - } - - _nextQueueStatsUpdate = _timeProvider.GetUtcNow().Add(_options.MetricsPollingInterval); - _logger.LogTrace("Getting metrics queue stats for {QueueName} ({QueueId}): Next update scheduled for {NextQueueStatsUpdate:O}", _options.Name, QueueId, _nextQueueStatsUpdate); - using var activity = FoundatioDiagnostics.ActivitySource.StartActivity("Queue Stats: " + _options.Name); - try - { - _queueStats = GetMetricsQueueStats(); - return (_queueStats.Queued, _queueStats.Working, _queueStats.Deadletter); - } - catch (Exception ex) - { - activity?.SetErrorStatus(ex); - _logger.LogError(ex, "Error getting queue metrics for {QueueName} ({QueueId}): {Message}", _options.Name, QueueId, ex.Message); - return (0, 0, 0); - } - }, _logger); - - _countGauge = FoundatioDiagnostics.Meter.CreateObservableGauge(GetFullMetricName("count"), - () => IsDisposed ? Array.Empty>() : [new Measurement(queueMetricValues.GetValue1())], - description: "Number of items in the queue"); - _workingGauge = FoundatioDiagnostics.Meter.CreateObservableGauge(GetFullMetricName("working"), - () => IsDisposed ? Array.Empty>() : [new Measurement(queueMetricValues.GetValue2())], - description: "Number of items currently being processed"); - _deadletterGauge = FoundatioDiagnostics.Meter.CreateObservableGauge(GetFullMetricName("deadletter"), - () => IsDisposed ? Array.Empty>() : [new Measurement(queueMetricValues.GetValue3())], - description: "Number of items in the deadletter queue"); - } - - public string QueueId { get; init; } - public DateTimeOffset? LastEnqueueActivity { get; protected set; } - public DateTimeOffset? LastDequeueActivity { get; protected set; } - ISerializer IHaveSerializer.Serializer => _serializer; - TimeProvider IHaveTimeProvider.TimeProvider => _timeProvider; - - public void AttachBehavior(IQueueBehavior behavior) - { - ArgumentNullException.ThrowIfNull(behavior); - - _behaviors.Add(behavior); - behavior.Attach(this); - } - - /// - /// Called before queue operations to ensure the queue exists. The - /// is always ; queue creation should only - /// abort when the queue is being disposed, never due to an individual caller's cancellation. - /// - protected abstract Task EnsureQueueCreatedAsync(CancellationToken cancellationToken = default); - - protected abstract Task EnqueueImplAsync(T data, QueueEntryOptions options); - public async Task EnqueueAsync(T data, QueueEntryOptions? options = null) - { - ObjectDisposedException.ThrowIf(IsDisposed, this); - ArgumentNullException.ThrowIfNull(data); - - await EnsureQueueCreatedAsync(DisposedCancellationToken).AnyContext(); - - LastEnqueueActivity = _timeProvider.GetUtcNow(); - options ??= new QueueEntryOptions(); - - return await EnqueueImplAsync(data, options).AnyContext(); - } - - protected abstract Task?> DequeueImplAsync(CancellationToken linkedCancellationToken); - public async Task?> DequeueAsync(CancellationToken cancellationToken) - { - ObjectDisposedException.ThrowIf(IsDisposed, this); - // Use DisposedCancellationToken for setup: callers may pass an already-cancelled token - // (e.g. TimeSpan.Zero timeout) which should skip waiting, not prevent queue creation. - await EnsureQueueCreatedAsync(DisposedCancellationToken).AnyContext(); - - using var linkedCancellationTokenSource = GetLinkedDisposableCancellationTokenSource(cancellationToken); - LastDequeueActivity = _timeProvider.GetUtcNow(); - return await DequeueImplAsync(linkedCancellationTokenSource.Token).AnyContext(); - } - - public virtual async Task?> DequeueAsync(TimeSpan? timeout = null) - { - ObjectDisposedException.ThrowIf(IsDisposed, this); - using var timeoutCancellationTokenSource = timeout.ToCancellationTokenSource(TimeSpan.FromSeconds(30)); - return await DequeueAsync(timeoutCancellationTokenSource.Token).AnyContext(); - } - - public abstract Task RenewLockAsync(IQueueEntry queueEntry); - - public abstract Task CompleteAsync(IQueueEntry queueEntry); - - public abstract Task AbandonAsync(IQueueEntry queueEntry); - - protected abstract Task> GetDeadletterItemsImplAsync(CancellationToken cancellationToken); - public async Task> GetDeadletterItemsAsync(CancellationToken cancellationToken = default) - { - // Use DisposedCancellationToken for setup: queue creation should only abort on disposal, - // not due to the caller's cancellation token. - await EnsureQueueCreatedAsync(DisposedCancellationToken).AnyContext(); - - using var linkedCancellationTokenSource = GetLinkedDisposableCancellationTokenSource(cancellationToken); - return await GetDeadletterItemsImplAsync(linkedCancellationTokenSource.Token).AnyContext(); - } - - protected abstract Task GetQueueStatsImplAsync(); - - public async Task GetQueueStatsAsync() - { - _logger.LogTrace("Getting queue stats for {QueueName} ({QueueId})", _options.Name, QueueId); - _queueStats = await GetQueueStatsImplAsync().AnyContext(); - return _queueStats; - } - - // TODO: sync-over-async — called from ObservableGauge callbacks (which must be synchronous), - // so this blocks a thread-pool thread on async I/O for external providers (Redis, Azure, etc.). - // The MetricsPollingInterval cache above mitigates frequency but doesn't eliminate the risk of - // thread-pool starvation under load. Blocked on async gauge callback support in .NET: - // https://github.com/dotnet/runtime/issues/96850 - protected virtual QueueStats GetMetricsQueueStats() - { - return GetQueueStatsAsync().AnyContext().GetAwaiter().GetResult(); - } - - protected abstract Task DeleteQueueImplAsync(); - - public async Task DeleteQueueAsync() - { - _logger.LogTrace("Deleting queue: {QueueName} ({QueueId})", _options.Name, QueueId); - await DeleteQueueImplAsync().AnyContext(); - await OnQueueDeletedAsync().AnyContext(); - } - - protected abstract void StartWorkingImpl(Func, CancellationToken, Task> handler, bool autoComplete, CancellationToken cancellationToken); - public async Task StartWorkingAsync(Func, CancellationToken, Task> handler, bool autoComplete = false, CancellationToken cancellationToken = default) - { - ObjectDisposedException.ThrowIf(IsDisposed, this); - // Use DisposedCancellationToken for setup: queue creation should only abort on disposal. - // StartWorkingImpl creates its own linked token for the long-running worker loop. - await EnsureQueueCreatedAsync(DisposedCancellationToken).AnyContext(); - StartWorkingImpl(handler, autoComplete, cancellationToken); - } - - public IReadOnlyCollection> Behaviors => _behaviors; - - public AsyncEvent> Enqueuing { get; } = new AsyncEvent>(); - - protected virtual async Task OnEnqueuingAsync(T data, QueueEntryOptions options) - { - if (String.IsNullOrEmpty(options.CorrelationId)) - { - options.CorrelationId = Activity.Current?.Id; - if (!String.IsNullOrEmpty(Activity.Current?.TraceStateString)) - options.Properties.Add("TraceState", Activity.Current.TraceStateString); - } - - var enqueueing = Enqueuing; - if (enqueueing is null) - return false; - - var args = new EnqueuingEventArgs { Queue = this, Data = data, Options = options }; - await enqueueing.InvokeAsync(this, args).AnyContext(); - - return !args.Cancel; - } - - public AsyncEvent> Enqueued { get; } = new AsyncEvent>(true); - - protected virtual Task OnEnqueuedAsync(IQueueEntry entry) - { - LastEnqueueActivity = _timeProvider.GetUtcNow(); - - var tags = GetQueueEntryTags(entry); - _enqueuedCounter.Add(1, tags); - IncrementSubCounter(entry.Value, "enqueued", tags); - - var enqueued = Enqueued; - if (enqueued is null) - return Task.CompletedTask; - - var args = new EnqueuedEventArgs { Queue = this, Entry = entry }; - return enqueued.InvokeAsync(this, args); - } - - public AsyncEvent> Dequeued { get; } = new AsyncEvent>(true); - - protected virtual Task OnDequeuedAsync(IQueueEntry entry) - { - LastDequeueActivity = _timeProvider.GetUtcNow(); - - var tags = GetQueueEntryTags(entry); - _dequeuedCounter.Add(1, tags); - IncrementSubCounter(entry.Value, "dequeued", tags); - - var metadata = entry as IQueueEntryMetadata; - if (metadata != null && (metadata.EnqueuedTimeUtc != DateTime.MinValue || metadata.DequeuedTimeUtc != DateTime.MinValue)) - { - var start = metadata.EnqueuedTimeUtc; - var end = metadata.DequeuedTimeUtc; - double time = (end - start).TotalMilliseconds; - - _queueTimeHistogram.Record(time, tags); - RecordSubHistogram(entry.Value, "queuetime", time, tags); - } - - var dequeued = Dequeued; - if (dequeued is null) - return Task.CompletedTask; - - var args = new DequeuedEventArgs { Queue = this, Entry = entry }; - return dequeued.InvokeAsync(this, args); - } - - protected virtual TagList GetQueueEntryTags(IQueueEntry entry) - { - return _emptyTags; - } - - public AsyncEvent> LockRenewed { get; } = new AsyncEvent>(true); - - protected virtual Task OnLockRenewedAsync(IQueueEntry entry) - { - LastDequeueActivity = _timeProvider.GetUtcNow(); - - var lockRenewed = LockRenewed; - if (lockRenewed is null) - return Task.CompletedTask; - - var args = new LockRenewedEventArgs { Queue = this, Entry = entry }; - return lockRenewed.InvokeAsync(this, args); - } - - public AsyncEvent> Completed { get; } = new AsyncEvent>(true); - - protected virtual async Task OnCompletedAsync(IQueueEntry entry) - { - var utcNow = _timeProvider.GetUtcNow(); - LastDequeueActivity = utcNow; - - var tags = GetQueueEntryTags(entry); - _completedCounter.Add(1, tags); - IncrementSubCounter(entry.Value, "completed", tags); - - if (entry is QueueEntry metadata) - { - if (metadata.EnqueuedTimeUtc > DateTime.MinValue) - { - metadata.TotalTime = utcNow.Subtract(metadata.EnqueuedTimeUtc); - _totalTimeHistogram.Record((int)metadata.TotalTime.TotalMilliseconds, tags); - RecordSubHistogram(entry.Value, "totaltime", (int)metadata.TotalTime.TotalMilliseconds, tags); - } - - if (metadata.DequeuedTimeUtc > DateTime.MinValue) - { - metadata.ProcessingTime = utcNow.Subtract(metadata.DequeuedTimeUtc); - _processTimeHistogram.Record((int)metadata.ProcessingTime.TotalMilliseconds, tags); - RecordSubHistogram(entry.Value, "processtime", (int)metadata.ProcessingTime.TotalMilliseconds, tags); - } - } - - if (Completed != null) - { - var args = new CompletedEventArgs { Queue = this, Entry = entry }; - await Completed.InvokeAsync(this, args).AnyContext(); - } - } - - public AsyncEvent> Abandoned { get; } = new AsyncEvent>(true); - - protected virtual async Task OnAbandonedAsync(IQueueEntry entry) - { - LastDequeueActivity = _timeProvider.GetUtcNow(); - - var tags = GetQueueEntryTags(entry); - _abandonedCounter.Add(1, tags); - IncrementSubCounter(entry.Value, "abandoned", tags); - - if (entry is QueueEntry metadata && metadata.DequeuedTimeUtc > DateTime.MinValue) - { - metadata.ProcessingTime = _timeProvider.GetUtcNow().Subtract(metadata.DequeuedTimeUtc); - _processTimeHistogram.Record((int)metadata.ProcessingTime.TotalMilliseconds, tags); - RecordSubHistogram(entry.Value, "processtime", (int)metadata.ProcessingTime.TotalMilliseconds, tags); - } - - if (Abandoned != null) - { - var args = new AbandonedEventArgs { Queue = this, Entry = entry }; - await Abandoned.InvokeAsync(this, args).AnyContext(); - } - } - - public AsyncEvent> QueueDeleted { get; } = new AsyncEvent>(true); - - protected virtual async Task OnQueueDeletedAsync() - { - if (QueueDeleted is not null) - { - var args = new QueueDeletedEventArgs { Queue = this }; - await QueueDeleted.InvokeAsync(this, args).AnyContext(); - } - } - - protected string? GetSubMetricName(T? data) - { - var haveStatName = data as IHaveSubMetricName; - return haveStatName?.SubMetricName; - } - - protected readonly ConcurrentDictionary> _counters = new(); - private void IncrementSubCounter(T? data, string name, in TagList tags) - { - if (data is not IHaveSubMetricName) - return; - - string? subMetricName = GetSubMetricName(data); - if (String.IsNullOrEmpty(subMetricName)) - return; - - var fullName = GetFullMetricName(subMetricName, name); - _counters.GetOrAdd(fullName, FoundatioDiagnostics.Meter.CreateCounter(fullName)).Add(1, tags); - } - - protected readonly ConcurrentDictionary> _histograms = new(); - private void RecordSubHistogram(T? data, string name, double value, in TagList tags) - { - if (data is not IHaveSubMetricName) - return; - - string? subMetricName = GetSubMetricName(data); - if (String.IsNullOrEmpty(subMetricName)) - return; - - var fullName = GetFullMetricName(subMetricName, name); - _histograms.GetOrAdd(fullName, FoundatioDiagnostics.Meter.CreateHistogram(fullName)).Record(value, tags); - } - - protected string GetFullMetricName(string name) - { - return String.Concat(_metricsPrefix, ".", name); - } - - protected string GetFullMetricName(string customMetricName, string name) - { - return String.IsNullOrEmpty(customMetricName) ? GetFullMetricName(name) : String.Concat(_metricsPrefix, ".", customMetricName.ToLower(), ".", name); - } - - protected CancellationTokenSource GetLinkedDisposableCancellationTokenSource(CancellationToken cancellationToken) - { - return CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, DisposedCancellationToken); - } - - public override void Dispose() - { - _logger.LogTrace("Queue {QueueName} ({QueueId}) dispose", _options.Name, QueueId); - SignalDispose(); - - Abandoned?.Dispose(); - Completed?.Dispose(); - Dequeued?.Dispose(); - Enqueued?.Dispose(); - Enqueuing?.Dispose(); - LockRenewed?.Dispose(); - QueueDeleted?.Dispose(); - - foreach (var behavior in _behaviors.OfType()) - behavior.Dispose(); - - _behaviors.Clear(); - base.Dispose(); - } -} - diff --git a/src/Foundatio/Queues/QueueBehaviour.cs b/src/Foundatio/Queues/QueueBehaviour.cs deleted file mode 100644 index 443aa5a46..000000000 --- a/src/Foundatio/Queues/QueueBehaviour.cs +++ /dev/null @@ -1,77 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Diagnostics.CodeAnalysis; -using System.Threading.Tasks; - -namespace Foundatio.Queues; - -public interface IQueueBehavior where T : class -{ - void Attach(IQueue queue); -} - -public abstract class QueueBehaviorBase : IQueueBehavior, IDisposable where T : class -{ - protected IQueue _queue = null!; // Set in Attach() before any other method is called - private readonly List _disposables = new(); - - [MemberNotNull(nameof(_queue))] - public virtual void Attach(IQueue queue) - { - ArgumentNullException.ThrowIfNull(queue); - - if (_queue is not null) - throw new QueueException("This behavior is already attached to a queue. Create a separate behavior instance for each queue."); - - _queue = queue; - - _disposables.Add(_queue.Enqueuing.AddHandler(OnEnqueuing)); - _disposables.Add(_queue.Enqueued.AddHandler(OnEnqueued)); - _disposables.Add(_queue.Dequeued.AddHandler(OnDequeued)); - _disposables.Add(_queue.LockRenewed.AddHandler(OnLockRenewed)); - _disposables.Add(_queue.Completed.AddHandler(OnCompleted)); - _disposables.Add(_queue.Abandoned.AddHandler(OnAbandoned)); - _disposables.Add(_queue.QueueDeleted.AddHandler(OnQueueDeleted)); - } - - protected virtual Task OnEnqueuing(object sender, EnqueuingEventArgs enqueuingEventArgs) - { - return Task.CompletedTask; - } - - protected virtual Task OnEnqueued(object sender, EnqueuedEventArgs enqueuedEventArgs) - { - return Task.CompletedTask; - } - - protected virtual Task OnDequeued(object sender, DequeuedEventArgs dequeuedEventArgs) - { - return Task.CompletedTask; - } - - protected virtual Task OnLockRenewed(object sender, LockRenewedEventArgs dequeuedEventArgs) - { - return Task.CompletedTask; - } - - protected virtual Task OnCompleted(object sender, CompletedEventArgs completedEventArgs) - { - return Task.CompletedTask; - } - - protected virtual Task OnAbandoned(object sender, AbandonedEventArgs abandonedEventArgs) - { - return Task.CompletedTask; - } - - protected virtual Task OnQueueDeleted(object sender, QueueDeletedEventArgs queueDeletedEventArgs) - { - return Task.CompletedTask; - } - - public virtual void Dispose() - { - foreach (var disposable in _disposables) - disposable.Dispose(); - } -} diff --git a/src/Foundatio/Queues/QueueEntry.cs b/src/Foundatio/Queues/QueueEntry.cs deleted file mode 100644 index f69ab91c5..000000000 --- a/src/Foundatio/Queues/QueueEntry.cs +++ /dev/null @@ -1,94 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Threading.Tasks; -using Foundatio.Utility; - -namespace Foundatio.Queues; - -public class QueueEntry : IQueueEntry, IQueueEntryMetadata, IAsyncDisposable where T : class -{ - private readonly IQueue _queue; - private readonly T _original; - - public QueueEntry(string id, string? correlationId, T value, IQueue queue, DateTime enqueuedTimeUtc, int attempts) - { - Id = id; - CorrelationId = correlationId; - _original = value; - Value = value?.DeepClone()!; - _queue = queue; - EnqueuedTimeUtc = enqueuedTimeUtc; - Attempts = attempts; - DequeuedTimeUtc = RenewedTimeUtc = _queue.GetTimeProvider().GetUtcNow().UtcDateTime; - } - - public string Id { get; } - public string? CorrelationId { get; } - public IDictionary Properties { get; } = new Dictionary(); - public bool IsCompleted { get; private set; } - public bool IsAbandoned { get; private set; } - - public Type? EntryType => Value?.GetType(); - public object GetValue() => Value; - public T Value { get; set; } - public DateTime EnqueuedTimeUtc { get; set; } - public DateTime RenewedTimeUtc { get; set; } - public DateTime DequeuedTimeUtc { get; set; } - public int Attempts { get; set; } - public TimeSpan ProcessingTime { get; set; } - public TimeSpan TotalTime { get; set; } - - void IQueueEntry.MarkCompleted() - { - IsCompleted = true; - } - - void IQueueEntry.MarkAbandoned() - { - IsAbandoned = true; - } - - public Task RenewLockAsync() - { - RenewedTimeUtc = _queue.GetTimeProvider().GetUtcNow().UtcDateTime; - return _queue.RenewLockAsync(this); - } - - public Task CompleteAsync() - { - return _queue.CompleteAsync(this); - } - - public Task AbandonAsync() - { - return _queue.AbandonAsync(this); - } - - public async ValueTask DisposeAsync() - { - if (!IsAbandoned && !IsCompleted) - await AbandonAsync(); - } - - internal QueueEntry CreateRetryEntry() - { - var entry = new QueueEntry(Id, CorrelationId, _original, _queue, EnqueuedTimeUtc, Attempts); - foreach (var kvp in Properties) - entry.Properties[kvp.Key] = kvp.Value; - - return entry; - } -} - -public interface IQueueEntryMetadata -{ - string Id { get; } - string? CorrelationId { get; } - IDictionary Properties { get; } - DateTime EnqueuedTimeUtc { get; } - DateTime RenewedTimeUtc { get; } - DateTime DequeuedTimeUtc { get; } - int Attempts { get; } - TimeSpan ProcessingTime { get; } - TimeSpan TotalTime { get; } -} diff --git a/src/Foundatio/Queues/QueueException.cs b/src/Foundatio/Queues/QueueException.cs deleted file mode 100644 index c7a1de6c4..000000000 --- a/src/Foundatio/Queues/QueueException.cs +++ /dev/null @@ -1,17 +0,0 @@ -using System; - -namespace Foundatio.Queues; - -/// -/// Exception thrown for queue operation errors. -/// -public class QueueException : Exception -{ - public QueueException(string message) : base(message) - { - } - - public QueueException(string message, Exception innerException) : base(message, innerException) - { - } -} diff --git a/src/Foundatio/Queues/SharedQueueOptions.cs b/src/Foundatio/Queues/SharedQueueOptions.cs deleted file mode 100644 index b2a3d7d74..000000000 --- a/src/Foundatio/Queues/SharedQueueOptions.cs +++ /dev/null @@ -1,118 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Diagnostics.CodeAnalysis; - -namespace Foundatio.Queues; - -public class SharedQueueOptions : SharedOptions where T : class -{ - public string Name { get; set; } = typeof(T).Name; - public int Retries { get; set; } = 2; - public TimeSpan WorkItemTimeout { get; set; } = TimeSpan.FromMinutes(5); - [DisallowNull] - public ICollection> Behaviors { get => field; set => field = value ?? new List>(); } = new List>(); - - /// - /// Allows you to set a prefix on queue metrics. This allows you to have unique metrics for keyed queues (e.g., priority queues). - /// - public string? MetricsPrefix { get; set; } - - /// - /// How often to poll queue metrics. These metrics are more expensive to calculate. Defaults to 5 seconds. - /// - public TimeSpan MetricsPollingInterval { get; set; } = TimeSpan.FromSeconds(5); - - /// - /// If metrics that require polling are enabled. These metrics are more expensive to calculate and should be disabled if you are not using them. Defaults to true. - /// - public bool MetricsPollingEnabled { get; set; } = true; -} - -public class SharedQueueOptionsBuilder : SharedOptionsBuilder - where T : class - where TOptions : SharedQueueOptions, new() - where TBuilder : SharedQueueOptionsBuilder, new() -{ - public TBuilder Name(string? name) - { - if (!String.IsNullOrWhiteSpace(name)) - Target.Name = name.Trim(); - - return (TBuilder)this; - } - - public TBuilder Retries(int retries) - { - ArgumentOutOfRangeException.ThrowIfNegative(retries); - - Target.Retries = retries; - return (TBuilder)this; - } - - public TBuilder WorkItemTimeout(TimeSpan timeout) - { - ArgumentOutOfRangeException.ThrowIfLessThan(timeout, TimeSpan.Zero); - - Target.WorkItemTimeout = timeout; - return (TBuilder)this; - } - - public TBuilder Behaviors(params IQueueBehavior[] behaviors) - { - ArgumentNullException.ThrowIfNull(behaviors); - - for (int index = 0; index < behaviors.Length; index++) - ArgumentNullException.ThrowIfNull(behaviors[index], $"behaviors[{index}]"); - - Target.Behaviors = behaviors; - return (TBuilder)this; - } - - public TBuilder AddBehavior(IQueueBehavior behavior) - { - ArgumentNullException.ThrowIfNull(behavior); - - Target.Behaviors.Add(behavior); - return (TBuilder)this; - } - - /// - /// Allows you to set a prefix on queue metrics. This allows you to have unique metrics for keyed queues (e.g., priority queues). - /// - public TBuilder MetricsPrefix(string? prefix) - { - if (!String.IsNullOrWhiteSpace(prefix)) - Target.MetricsPrefix = prefix.Trim(); - - return (TBuilder)this; - } - - /// - /// How often to poll queue metrics. These metrics are more expensive to calculate. Defaults to 5 seconds. - /// - public TBuilder MetricsPollingInterval(TimeSpan interval) - { - ArgumentOutOfRangeException.ThrowIfLessThan(interval, TimeSpan.Zero); - - Target.MetricsPollingInterval = interval; - return (TBuilder)this; - } - - /// - /// If metrics that require polling are enabled. These metrics are more expensive to calculate and should be disabled if you are not using them. Defaults to true. - /// - public TBuilder MetricsPollingEnabled(bool enabled) - { - Target.MetricsPollingEnabled = enabled; - return (TBuilder)this; - } - - /// - /// Disable metrics collection for this queue. - /// - public TBuilder DisableMetricsPolling() - { - Target.MetricsPollingEnabled = false; - return (TBuilder)this; - } -} diff --git a/src/Foundatio/Serializer/IBufferSerializer.cs b/src/Foundatio/Serializer/IBufferSerializer.cs new file mode 100644 index 000000000..31c094d1d --- /dev/null +++ b/src/Foundatio/Serializer/IBufferSerializer.cs @@ -0,0 +1,19 @@ +using System; + +namespace Foundatio.Serializer; + +/// +/// Optional support for serializing directly to and from UTF-8 or binary buffers without intermediate streams. +/// Serializer extension methods use this capability automatically when available. +/// +public interface IBufferSerializer : ISerializer +{ + /// Serializes a value, including null, to an independently owned byte array. + byte[] SerializeToBytes(object? value); + + /// Deserializes a value without retaining or modifying the input buffer. + /// The nonempty serialized data. + /// The type of object to deserialize. + /// The deserialized object, or null if the data represents a null value. + object? Deserialize(ReadOnlyMemory data, Type objectType); +} diff --git a/src/Foundatio/Serializer/ISerializer.cs b/src/Foundatio/Serializer/ISerializer.cs index 749a075e7..fba82d378 100644 --- a/src/Foundatio/Serializer/ISerializer.cs +++ b/src/Foundatio/Serializer/ISerializer.cs @@ -85,8 +85,7 @@ public static T Deserialize(this ISerializer serializer, byte[] data) if (data.Length == 0) throw new ArgumentException("Data cannot be empty.", nameof(data)); - using var stream = new MemoryStream(data); - var result = serializer.Deserialize(stream, typeof(T)); + var result = serializer.Deserialize((ReadOnlyMemory)data, typeof(T)); if (result is T typed) return typed; @@ -104,8 +103,7 @@ public static T Deserialize(this ISerializer serializer, byte[] data) if (data.Length == 0) throw new ArgumentException("Data cannot be empty.", nameof(data)); - using var stream = new MemoryStream(data); - return serializer.Deserialize(stream, objectType); + return serializer.Deserialize((ReadOnlyMemory)data, objectType); } /// @@ -141,6 +139,9 @@ public static T Deserialize(this ISerializer serializer, ReadOnlyMemory if (data.IsEmpty) throw new ArgumentException("Data cannot be empty.", nameof(data)); + if (serializer is IBufferSerializer bufferSerializer) + return bufferSerializer.Deserialize(data, objectType); + // Fast path: if the memory is backed by a managed array we can hand it straight to a // MemoryStream without copying. Otherwise fall back to a stream over the memory. if (MemoryMarshal.TryGetArray(data, out ArraySegment segment) && segment.Array is not null) @@ -204,6 +205,9 @@ public static byte[] SerializeToBytes(this ISerializer serializer, T value) { ArgumentNullException.ThrowIfNull(serializer); + if (serializer is IBufferSerializer bufferSerializer) + return bufferSerializer.SerializeToBytes(value); + // Serialize null values - underlying serializers handle this correctly // (produces "null" for JSON, nil marker for MessagePack) using var stream = new MemoryStream(); diff --git a/src/Foundatio/Serializer/SystemTextJsonSerializer.cs b/src/Foundatio/Serializer/SystemTextJsonSerializer.cs index 281b72f59..2bb35c1e4 100644 --- a/src/Foundatio/Serializer/SystemTextJsonSerializer.cs +++ b/src/Foundatio/Serializer/SystemTextJsonSerializer.cs @@ -4,7 +4,7 @@ namespace Foundatio.Serializer; -public class SystemTextJsonSerializer : ITextSerializer +public class SystemTextJsonSerializer : ITextSerializer, IBufferSerializer { private readonly JsonSerializerOptions _serializeOptions; private readonly JsonSerializerOptions _deserializeOptions; @@ -24,6 +24,27 @@ public void Serialize(object? value, Stream output) JsonSerializer.Serialize(output, value, value?.GetType() ?? typeof(object), _serializeOptions); } + /// + byte[] IBufferSerializer.SerializeToBytes(object? value) + { + return JsonSerializer.SerializeToUtf8Bytes(value, value?.GetType() ?? typeof(object), _serializeOptions); + } + + /// + object? IBufferSerializer.Deserialize(ReadOnlyMemory data, Type objectType) + { + ArgumentNullException.ThrowIfNull(objectType); + if (data.IsEmpty) + throw new ArgumentException("Data cannot be empty.", nameof(data)); + + var utf8 = data.Span; + if (utf8.StartsWith("\uFEFF"u8)) + utf8 = utf8[3..]; + + object? result = JsonSerializer.Deserialize(utf8, objectType, _deserializeOptions); + return result is JsonElement element ? ConvertJsonElement(element) : result; + } + public object? Deserialize(Stream data, Type objectType) { ArgumentNullException.ThrowIfNull(data); diff --git a/tests/Foundatio.Aws.Tests/AwsBatchTests.cs b/tests/Foundatio.Aws.Tests/AwsBatchTests.cs new file mode 100644 index 000000000..28e2a6bea --- /dev/null +++ b/tests/Foundatio.Aws.Tests/AwsBatchTests.cs @@ -0,0 +1,464 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Amazon.SQS; +using Amazon.SQS.Model; +using Amazon.SimpleNotificationService; +using Foundatio.Messaging; +using Moq; +using Xunit; +using Sns = Amazon.SimpleNotificationService.Model; + +namespace Foundatio.Aws.Tests; + +public class AwsBatchTests +{ + [Theory] + [InlineData(true)] + [InlineData(false)] + public async Task SendAsync_OversizedSingleMessage_RejectsBeforeResolvingDestination(bool batching) + { + var sqs = CreateSqs(); + await using var transport = new AwsMessageTransport(new() { EnableBatching = batching }, sqs.Object, Mock.Of()); + + var result = Assert.Single((await transport.SendAsync(DestinationAddress.ForQueue("test"), [Text(new string('é', 600_000))], new(), TestContext.Current.CancellationToken)).Items); + + Assert.Equal(0, result.Index); + Assert.Equal(MessageSendStatus.Rejected, result.Status); + Assert.Equal("MessageTooLarge", result.ErrorCode); + Assert.False(result.Retryable); + sqs.Verify(s => s.GetQueueUrlAsync(It.IsAny(), It.IsAny()), Times.Never); + sqs.Verify(s => s.SendMessageBatchAsync(It.IsAny(), It.IsAny()), Times.Never); + } + + [Theory] + [InlineData(true, "missing")] + [InlineData(false, "missing")] + [InlineData(true, "duplicate")] + [InlineData(false, "duplicate")] + [InlineData(true, "unknown")] + [InlineData(false, "unknown")] + public async Task SendAsync_UnconfirmedSingleResponse_ReportsUnknown(bool batching, string outcome) + { + var sqs = CreateSqs(); + sqs.Setup(s => s.SendMessageBatchAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync(new SendMessageBatchResponse + { + Successful = outcome == "missing" ? [] : outcome == "duplicate" + ? [new() { Id = "0", MessageId = "one" }, new() { Id = "0", MessageId = "two" }] + : [new() { Id = "1", MessageId = "unknown" }] + }); + await using var transport = new AwsMessageTransport(new() { EnableBatching = batching }, sqs.Object, Mock.Of()); + + var result = Assert.Single((await transport.SendAsync(DestinationAddress.ForQueue("test"), [Text("body")], new(), TestContext.Current.CancellationToken)).Items); + + Assert.Equal(0, result.Index); + Assert.Equal(MessageSendStatus.Unknown, result.Status); + Assert.Null(result.MessageId); + } + + [Fact] + public async Task SendAsync_AutomaticBatcher_DoesNotRetainCallerExecutionContext() + { + var caller = new AsyncLocal { Value = "first-request" }; + var observed = new ConcurrentQueue(); + var sqs = CreateSqs(); + sqs.Setup(s => s.SendMessageBatchAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync((SendMessageBatchRequest request, CancellationToken _) => + { + observed.Enqueue(caller.Value); + return Accepted(request); + }); + await using var transport = new AwsMessageTransport(new(), sqs.Object, Mock.Of()); + await transport.SendAsync(DestinationAddress.ForQueue("test"), [Text("one")], new(), TestContext.Current.CancellationToken); + caller.Value = "second-request"; + await transport.SendAsync(DestinationAddress.ForQueue("test"), [Text("two")], new(), TestContext.Current.CancellationToken); + Assert.Equal(2, observed.Count); + Assert.All(observed, Assert.Null); + Assert.Equal("second-request", caller.Value); + } + + [Fact] + public async Task SendAsync_LargerExplicitBatch_PreservesIndicesAcrossRequestsAndCancellation() + { + using var cancellation = CancellationTokenSource.CreateLinkedTokenSource(TestContext.Current.CancellationToken); + var sqs = CreateSqs(); + int calls = 0; + sqs.Setup(s => s.SendMessageBatchAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync((SendMessageBatchRequest request, CancellationToken _) => + { + if (Interlocked.Increment(ref calls) == 2) cancellation.Cancel(); + return Accepted(request); + }); + await using var transport = new AwsMessageTransport(new(), sqs.Object, Mock.Of()); + var result = await transport.SendAsync(DestinationAddress.ForQueue("test"), Enumerable.Range(0, 25).Select(i => Text(i.ToString())).ToArray(), new(), cancellation.Token); + Assert.Equal(2, calls); + for (int i = 0; i < 25; i++) + { + Assert.Equal(i, result.Items[i].Index); + Assert.Equal(i < 20 ? MessageSendStatus.Accepted : MessageSendStatus.NotAttempted, result.Items[i].Status); + if (i < 20) Assert.Equal("broker-" + i, result.Items[i].MessageId); + } + } + + [Fact] + public async Task SendAsync_OversizedEncodedMessage_RejectsOnlyThatInput() + { + var sqs = CreateSqs(); + sqs.Setup(s => s.SendMessageBatchAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync((SendMessageBatchRequest request, CancellationToken _) => Accepted(request)); + await using var transport = new AwsMessageTransport(new(), sqs.Object, Mock.Of()); + var result = await transport.SendAsync(DestinationAddress.ForQueue("test"), [Text(new string('é', 600_000)), Text("valid")], new(), TestContext.Current.CancellationToken); + Assert.Equal(MessageSendStatus.Rejected, result.Items[0].Status); + Assert.Equal("MessageTooLarge", result.Items[0].ErrorCode); + Assert.Equal(MessageSendStatus.Accepted, result.Items[1].Status); + Assert.Equal(1, result.Items[1].Index); + Assert.Equal("broker-valid", result.Items[1].MessageId); + sqs.Verify(s => s.SendMessageBatchAsync(It.Is(r => r.Entries.Count == 1), It.IsAny()), Times.Once); + } + + [Theory] + [InlineData(0, 1, 0, 100)] + [InlineData(1, 0, 0, 100)] + [InlineData(1, 1, -1, 100)] + [InlineData(1, 1, 101, 100)] + [InlineData(1, 1, 0, 0)] + public void Constructor_InvalidBatchLimits_RejectsConfiguration(int concurrency, int pending, int delay, int timeout) + { + Assert.Throws(() => new AwsMessageTransport(new AwsMessageTransportOptions + { + MaxConcurrentBatches = concurrency, + MaxPendingBatchMessages = pending, + BatchDelay = TimeSpan.FromMilliseconds(delay), + BatchTimeout = TimeSpan.FromMilliseconds(timeout) + })); + } + + [Fact] + public async Task SendAsync_ConcurrentTopics_RespectsByteLimitAndPreservesOutcomes() + { + var requests = new ConcurrentBag(); + var sns = new Mock(); + sns.Setup(s => s.ListTopicsAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync(new Sns.ListTopicsResponse { Topics = [new Sns.Topic { TopicArn = "arn:aws:sns:us-east-1:123:test" }] }); + sns.Setup(s => s.PublishBatchAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync((Sns.PublishBatchRequest request, CancellationToken _) => + { + requests.Add(request); + return new Sns.PublishBatchResponse + { + Successful = request.PublishBatchRequestEntries.Where(e => e.Message[0] != 'c').Select(e => new Sns.PublishBatchResultEntry { Id = e.Id, MessageId = "broker-" + e.Message[0] }).ToList(), + Failed = request.PublishBatchRequestEntries.Where(e => e.Message[0] == 'c').Select(e => new Sns.BatchResultErrorEntry { Id = e.Id, Code = "InvalidParameter", SenderFault = true }).ToList() + }; + }); + await using var transport = new AwsMessageTransport(new() { BatchDelay = TimeSpan.FromMilliseconds(20) }, Mock.Of(), sns.Object); + var tasks = Enumerable.Range(0, 6).Select(i => transport.SendAsync(DestinationAddress.ForTopic("test"), + [Text(new string((char)('a' + i), 100_000))], new(), TestContext.Current.CancellationToken)).ToArray(); + var results = await Task.WhenAll(tasks).WaitAsync(TimeSpan.FromSeconds(10), TestContext.Current.CancellationToken); + Assert.InRange(requests.Count, 3, 5); + Assert.All(requests, request => Assert.InRange(request.PublishBatchRequestEntries.Count, 1, 2)); + for (int i = 0; i < results.Length; i++) + { + var item = Assert.Single(results[i].Items); + Assert.Equal(0, item.Index); + Assert.Equal(i == 2 ? MessageSendStatus.Rejected : MessageSendStatus.Accepted, item.Status); + if (i == 2) Assert.False(item.Retryable); + else Assert.Equal("broker-" + (char)('a' + i), item.MessageId); + } + } + + [Fact] + public async Task SendAsync_ConcurrentDestinationsAndDelays_DoesNotMixTheirSettings() + { + var requests = new ConcurrentBag(); + var sqs = CreateSqs(); + sqs.Setup(s => s.SendMessageBatchAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync((SendMessageBatchRequest request, CancellationToken _) => { requests.Add(request); return Accepted(request); }); + await using var transport = new AwsMessageTransport(new() { BatchDelay = TimeSpan.FromMilliseconds(20) }, sqs.Object, Mock.Of()); + var tasks = Enumerable.Range(0, 20).Select(i => transport.SendAsync(DestinationAddress.ForQueue(i % 2 == 0 ? "even" : "odd"), + [Text(i.ToString())], new() { DeliverAt = i % 4 == 0 ? DateTimeOffset.UtcNow.AddSeconds(60) : null }, TestContext.Current.CancellationToken)).ToArray(); + await Task.WhenAll(tasks).WaitAsync(TimeSpan.FromSeconds(10), TestContext.Current.CancellationToken); + Assert.InRange(requests.Count, 2, 6); + foreach (var request in requests) + foreach (var entry in request.Entries) + { + int number = Int32.Parse(entry.MessageBody); + Assert.Equal(number % 2 == 0 ? "http://test/even" : "http://test/odd", request.QueueUrl); + if (number % 4 == 0) Assert.InRange(entry.DelaySeconds.GetValueOrDefault(), 55, 60); + else Assert.Null(entry.DelaySeconds); + } + } + + [Theory] + [InlineData("failed")] + [InlineData("missing")] + [InlineData("duplicate")] + public async Task CompleteAsync_UnconfirmedReceipt_DoesNotReportSuccess(string outcome) + { + var sqs = CreateSqs(); + sqs.Setup(s => s.DeleteMessageBatchAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync((DeleteMessageBatchRequest request, CancellationToken _) => new DeleteMessageBatchResponse + { + Successful = request.Entries.Where(e => e.ReceiptHandle != "bad" || outcome == "duplicate").Select(e => new DeleteMessageBatchResultEntry { Id = e.Id }).ToList(), + Failed = request.Entries.Where(e => e.ReceiptHandle == "bad" && outcome != "missing").Select(e => new BatchResultErrorEntry { Id = e.Id, Code = "ReceiptHandleIsInvalid", SenderFault = true }).ToList() + }); + await using var transport = new AwsMessageTransport(new(), sqs.Object, Mock.Of()); + var bad = transport.CompleteAsync(Entry("bad"), TestContext.Current.CancellationToken); + var good = transport.CompleteAsync(Entry("good"), TestContext.Current.CancellationToken); + await Assert.ThrowsAsync(() => bad); + if (outcome == "duplicate") + { + try { await good; } + catch (MessageBusException) { } + } + else await good; + } + + [Fact] + public async Task SendAsync_CancelOneInFlightCaller_DoesNotCancelOtherMessages() + { + var occupied = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var allowBatch = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + SendMessageBatchRequest? sharedRequest = null; + CancellationToken sharedToken = default; + var sqs = CreateSqs(); + sqs.Setup(s => s.SendMessageBatchAsync(It.IsAny(), It.IsAny())) + .Returns(async (SendMessageBatchRequest request, CancellationToken ct) => + { + if (request.Entries[0].MessageBody == "occupy") + { + occupied.TrySetResult(); + await allowBatch.Task.WaitAsync(ct); + return Accepted(request); + } + sharedRequest = request; + sharedToken = ct; + entered.TrySetResult(); + await release.Task.WaitAsync(ct); + return Accepted(request); + }); + await using var transport = new AwsMessageTransport(new() { MaxConcurrentBatches = 1 }, sqs.Object, Mock.Of()); + using var canceled = CancellationTokenSource.CreateLinkedTokenSource(TestContext.Current.CancellationToken); + var occupying = transport.SendAsync(DestinationAddress.ForQueue("test"), [Text("occupy")], new(), TestContext.Current.CancellationToken); + await occupied.Task.WaitAsync(TimeSpan.FromSeconds(10), TestContext.Current.CancellationToken); + var first = transport.SendAsync(DestinationAddress.ForQueue("test"), [Text("first")], new(), canceled.Token); + var second = transport.SendAsync(DestinationAddress.ForQueue("test"), [Text("second")], new(), TestContext.Current.CancellationToken); + try + { + allowBatch.TrySetResult(); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(10), TestContext.Current.CancellationToken); + Assert.Equal(["first", "second"], sharedRequest!.Entries.Select(e => e.MessageBody)); + await canceled.CancelAsync(); + var canceledResult = await first.WaitAsync(TimeSpan.FromSeconds(10), TestContext.Current.CancellationToken); + Assert.Equal(MessageSendStatus.Unknown, Assert.Single(canceledResult.Items).Status); + Assert.False(second.IsCompleted); + Assert.False(sharedToken.IsCancellationRequested); + } + finally { release.TrySetResult(); } + Assert.Equal(MessageSendStatus.Accepted, Assert.Single((await occupying).Items).Status); + Assert.Equal(MessageSendStatus.Accepted, Assert.Single((await second).Items).Status); + } + + [Fact] + public async Task SendAsync_BoundedQueueAndDisposal_DropsCanceledWorkAndDrainsAcceptedWork() + { + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var sent = new ConcurrentBag(); + var sqs = CreateSqs(); + sqs.Setup(s => s.SendMessageBatchAsync(It.IsAny(), It.IsAny())) + .Returns(async (SendMessageBatchRequest request, CancellationToken ct) => + { + foreach (var entry in request.Entries) sent.Add(entry.MessageBody); + entered.TrySetResult(); + await release.Task.WaitAsync(ct); + return Accepted(request); + }); + await using var transport = new AwsMessageTransport(new() { MaxConcurrentBatches = 1, MaxPendingBatchMessages = 1, BatchDelay = TimeSpan.Zero }, sqs.Object, Mock.Of()); + using var canceled = CancellationTokenSource.CreateLinkedTokenSource(TestContext.Current.CancellationToken); + var first = transport.SendAsync(DestinationAddress.ForQueue("test"), [Text("first")], new(), TestContext.Current.CancellationToken); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(10), TestContext.Current.CancellationToken); + var second = transport.SendAsync(DestinationAddress.ForQueue("test"), [Text("second")], new(), TestContext.Current.CancellationToken); + var third = transport.SendAsync(DestinationAddress.ForQueue("test"), [Text("canceled")], new(), canceled.Token); + Task disposal; + try + { + await canceled.CancelAsync(); + Assert.NotEqual(MessageSendStatus.Accepted, Assert.Single((await third).Items).Status); + Assert.Single(sent); + disposal = transport.DisposeAsync().AsTask(); + Assert.False(disposal.IsCompleted); + } + finally { release.TrySetResult(); } + await disposal.WaitAsync(TimeSpan.FromSeconds(10), TestContext.Current.CancellationToken); + Assert.Equal(MessageSendStatus.Accepted, Assert.Single((await first).Items).Status); + Assert.Equal(MessageSendStatus.Accepted, Assert.Single((await second).Items).Status); + Assert.Equal(new[] { "first", "second" }, sent.Order()); + sqs.Verify(s => s.Dispose(), Times.Never); + } + + [Fact] + public async Task SendAsync_SharedRequestTimeout_ReportsUnknownAndAllowsLaterRequests() + { + var sqs = CreateSqs(); + sqs.Setup(s => s.SendMessageBatchAsync(It.IsAny(), It.IsAny())) + .Returns(async (SendMessageBatchRequest request, CancellationToken ct) => + { + if (request.Entries[0].MessageBody == "timeout") + await Task.Delay(Timeout.InfiniteTimeSpan, ct); + return Accepted(request); + }); + await using var transport = new AwsMessageTransport(new() { BatchTimeout = TimeSpan.FromMilliseconds(100) }, sqs.Object, Mock.Of()); + var first = await transport.SendAsync(DestinationAddress.ForQueue("test"), [Text("timeout")], new(), TestContext.Current.CancellationToken); + Assert.Equal(MessageSendStatus.Unknown, Assert.Single(first.Items).Status); + var second = await transport.SendAsync(DestinationAddress.ForQueue("test"), [Text("success")], new(), TestContext.Current.CancellationToken); + Assert.Equal(MessageSendStatus.Accepted, Assert.Single(second.Items).Status); + } + + [Fact] + public async Task CompleteAsync_ReceiveCapacityIsFilled_FlushesWithoutWaitingForImpossibleEntries() + { + var token = TestContext.Current.CancellationToken; + var sqs = CreateSqs(); + var requests = new ConcurrentBag(); + sqs.Setup(s => s.ReceiveMessageAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync((ReceiveMessageRequest request, CancellationToken _) => new ReceiveMessageResponse + { + Messages = Enumerable.Range(0, request.MaxNumberOfMessages.GetValueOrDefault()).Select(i => new Message + { + MessageId = i.ToString(), + ReceiptHandle = "receipt" + i, + Body = "e30=" + }).ToList() + }); + sqs.Setup(s => s.DeleteMessageBatchAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync((DeleteMessageBatchRequest request, CancellationToken _) => + { + requests.Add(request); + return new DeleteMessageBatchResponse { Successful = request.Entries.Select(e => new DeleteMessageBatchResultEntry { Id = e.Id }).ToList() }; + }); + await using var transport = new AwsMessageTransport(new() { BatchDelay = TimeSpan.FromMilliseconds(100) }, sqs.Object, Mock.Of()); + var entries = await transport.ReceiveAsync(DestinationAddress.ForQueue("test"), new ReceiveRequest { MaxMessages = 2 }, token); + await Task.WhenAll(entries.Select(e => transport.CompleteAsync(e, token))).WaitAsync(TimeSpan.FromMilliseconds(75), token); + Assert.Equal(2, Assert.Single(requests).Entries.Count); + + entries = await transport.ReceiveAsync(DestinationAddress.ForQueue("test"), new ReceiveRequest { MaxMessages = 4 }, token); + var pending = entries.Take(2).Select(e => transport.CompleteAsync(e, token)).ToList(); + await Task.Delay(25, token); + Assert.Single(requests); + pending.AddRange(entries.Skip(2).Select(e => transport.CompleteAsync(e, token))); + await Task.WhenAll(pending).WaitAsync(TimeSpan.FromMilliseconds(75), token); + Assert.Contains(requests, request => request.Entries.Count == 4); + } + + private static Mock CreateSqs() + { + var sqs = new Mock(); + sqs.Setup(s => s.GetQueueUrlAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync((string queue, CancellationToken _) => new GetQueueUrlResponse { QueueUrl = "http://test/" + queue }); + return sqs; + } + + private static TransportMessage Text(string value) => new() { Body = System.Text.Encoding.UTF8.GetBytes(value), ContentType = "text/plain" }; + private static TransportEntry Entry(string receipt) => new() { Id = receipt, Destination = DestinationAddress.ForQueue("test"), Body = ReadOnlyMemory.Empty, Receipt = new Receipt { TransportState = receipt } }; + private static SendMessageBatchResponse Accepted(SendMessageBatchRequest request) => new() + { + Successful = request.Entries.Select(e => new SendMessageBatchResultEntry { Id = e.Id, MessageId = "broker-" + e.MessageBody }).ToList() + }; + + [Fact] + public async Task SendAsync_ConcurrentSingleMessages_CoalescesRequestsAndPreservesEachOutcome() + { + var requests = new ConcurrentBag(); + var sqs = new Mock(); + sqs.Setup(s => s.GetQueueUrlAsync(It.IsAny(), It.IsAny())).ReturnsAsync(new GetQueueUrlResponse { QueueUrl = "http://test/queue" }); + sqs.Setup(s => s.SendMessageBatchAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync((SendMessageBatchRequest request, CancellationToken _) => + { + requests.Add(request); + return new SendMessageBatchResponse + { + Successful = request.Entries.Where(e => e.MessageBody != "7").Select(e => new SendMessageBatchResultEntry { Id = e.Id, MessageId = "broker-" + e.MessageBody }).ToList(), + Failed = request.Entries.Where(e => e.MessageBody == "7").Select(e => new BatchResultErrorEntry { Id = e.Id, Code = "Throttled", SenderFault = false }).ToList() + }; + }); + await using var transport = new AwsMessageTransport(new() { BatchDelay = TimeSpan.FromMilliseconds(20) }, sqs.Object, Mock.Of()); + var tasks = Enumerable.Range(0, 20).Select(i => transport.SendAsync(DestinationAddress.ForQueue("test"), + [new TransportMessage { Body = System.Text.Encoding.UTF8.GetBytes(i.ToString()), ContentType = "text/plain" }], new(), TestContext.Current.CancellationToken)).ToArray(); + var results = await Task.WhenAll(tasks).WaitAsync(TimeSpan.FromSeconds(10), TestContext.Current.CancellationToken); + + Assert.InRange(requests.Count, 2, 5); + Assert.All(requests, request => Assert.InRange(request.Entries.Count, 1, 10)); + for (int i = 0; i < results.Length; i++) + { + var item = Assert.Single(results[i].Items); + Assert.Equal(0, item.Index); + Assert.Equal(i == 7 ? MessageSendStatus.Rejected : MessageSendStatus.Accepted, item.Status); + if (i == 7) Assert.True(item.Retryable); + else Assert.Equal("broker-" + i, item.MessageId); + } + } + + [Fact] + public async Task CompleteAsync_ConcurrentReceipts_WaitsForBatchedBrokerAcknowledgement() + { + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var requests = new ConcurrentBag(); + var sqs = new Mock(); + sqs.Setup(s => s.GetQueueUrlAsync(It.IsAny(), It.IsAny())).ReturnsAsync(new GetQueueUrlResponse { QueueUrl = "http://test/queue" }); + sqs.Setup(s => s.DeleteMessageAsync(It.IsAny(), It.IsAny(), It.IsAny())).ReturnsAsync(new DeleteMessageResponse()); + sqs.Setup(s => s.DeleteMessageBatchAsync(It.IsAny(), It.IsAny())) + .Returns(async (DeleteMessageBatchRequest request, CancellationToken ct) => + { + requests.Add(request); + entered.TrySetResult(); + await release.Task.WaitAsync(ct); + return new DeleteMessageBatchResponse { Successful = request.Entries.Select(e => new DeleteMessageBatchResultEntry { Id = e.Id }).ToList() }; + }); + await using var transport = new AwsMessageTransport(new() { BatchDelay = TimeSpan.FromMilliseconds(20) }, sqs.Object, Mock.Of()); + var tasks = Enumerable.Range(0, 20).Select(i => transport.CompleteAsync(new TransportEntry + { + Id = i.ToString(), + Destination = DestinationAddress.ForQueue("test"), + Body = ReadOnlyMemory.Empty, + Receipt = new Receipt { TransportState = "receipt-" + i } + }, TestContext.Current.CancellationToken)).ToArray(); + try + { + Assert.All(tasks, task => Assert.False(task.IsCompleted)); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(10), TestContext.Current.CancellationToken); + } + finally { release.TrySetResult(); } + await Task.WhenAll(tasks).WaitAsync(TimeSpan.FromSeconds(10), TestContext.Current.CancellationToken); + Assert.InRange(requests.Count, 2, 5); + Assert.All(requests, request => Assert.InRange(request.Entries.Count, 1, 10)); + Assert.Equal(20, requests.Sum(r => r.Entries.Count)); + sqs.Verify(s => s.DeleteMessageAsync(It.IsAny(), It.IsAny(), It.IsAny()), Times.Never); + } + + [Fact] + public async Task SendAsync_NativeBatch_ReportsNoncontiguousFailure() + { + var sqs = new Mock(); + sqs.Setup(s => s.GetQueueUrlAsync(It.IsAny(), It.IsAny())).ReturnsAsync(new GetQueueUrlResponse { QueueUrl = "http://test/queue" }); + sqs.Setup(s => s.SendMessageBatchAsync(It.IsAny(), It.IsAny())).ReturnsAsync(new SendMessageBatchResponse + { + Successful = [new SendMessageBatchResultEntry { Id = "2", MessageId = "broker-c" }, new SendMessageBatchResultEntry { Id = "0", MessageId = "broker-a" }], + Failed = [new BatchResultErrorEntry { Id = "1", Code = "Throttled", SenderFault = false, Message = "Retry later" }] + }); + await using var transport = new AwsMessageTransport(new(), sqs.Object, Mock.Of()); + var result = await transport.SendAsync(DestinationAddress.ForQueue("test"), Enumerable.Range(0, 3).Select(_ => new TransportMessage { Body = "hello"u8.ToArray(), ContentType = "text/plain" }).ToArray(), new(), TestContext.Current.CancellationToken); + Assert.Collection(result.Items, + a => Assert.Equal(MessageSendStatus.Accepted, a.Status), + b => { Assert.Equal(MessageSendStatus.Rejected, b.Status); Assert.Equal(1, b.Index); Assert.True(b.Retryable); }, + c => Assert.Equal(MessageSendStatus.Accepted, c.Status)); + sqs.Verify(s => s.SendMessageBatchAsync(It.Is(r => r.Entries.Count == 3), It.IsAny()), Times.Once); + sqs.Verify(s => s.SendMessageAsync(It.IsAny(), It.IsAny()), Times.Never); + } +} diff --git a/tests/Foundatio.Aws.Tests/AwsEnvelopeTests.cs b/tests/Foundatio.Aws.Tests/AwsEnvelopeTests.cs new file mode 100644 index 000000000..c42a52454 --- /dev/null +++ b/tests/Foundatio.Aws.Tests/AwsEnvelopeTests.cs @@ -0,0 +1,146 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Amazon.SQS; +using Amazon.SQS.Model; +using Amazon.SimpleNotificationService; +using Foundatio.Messaging; +using Moq; +using Xunit; + +namespace Foundatio.Aws.Tests; + +public class AwsEnvelopeTests +{ + [Fact] + public async Task ReceiveAsync_SystemAttributes_RequestsDeliveryCountAndEnqueueTime() + { + var sqs = new Mock(); + sqs.Setup(s => s.GetQueueUrlAsync(It.IsAny(), It.IsAny())).ReturnsAsync(new GetQueueUrlResponse { QueueUrl = "http://test/queue" }); + sqs.Setup(s => s.ReceiveMessageAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync((ReceiveMessageRequest request, CancellationToken _) => + { + Assert.Equal(["ApproximateReceiveCount", "SentTimestamp"], request.MessageSystemAttributeNames); + Assert.Equal(["All"], request.MessageAttributeNames); + return new ReceiveMessageResponse + { + Messages = [new Message { MessageId = "id", ReceiptHandle = "receipt", Body = "e30=", Attributes = new() { ["ApproximateReceiveCount"] = "3" } }] + }; + }); + await using var transport = new AwsMessageTransport(new(), sqs.Object, Mock.Of()); + + var entry = Assert.Single(await transport.ReceiveAsync(DestinationAddress.ForQueue("test"), new(), TestContext.Current.CancellationToken)); + + Assert.Equal(3, entry.DeliveryCount); + } + + [Theory] + [InlineData(null)] + [InlineData("")] + [InlineData("fnd.envelope")] + [InlineData("FND.custom")] + [InlineData("AWS.trace")] + [InlineData("Amazon.id")] + [InlineData(".leading")] + [InlineData("trailing.")] + [InlineData("two..dots")] + [InlineData("bad name")] + [InlineData("résumé")] + public void Constructor_InvalidNativeHeader_FailsBeforeConnecting(string? header) + { + Assert.ThrowsAny(() => new AwsMessageTransport(new AwsMessageTransportOptions { NativeMessageHeaders = [header!] })); + } + + [Fact] + public void Constructor_ExcessiveDuplicateOrMissingNativeHeaders_RejectsConfiguration() + { + Assert.ThrowsAny(() => new AwsMessageTransport(new AwsMessageTransportOptions { NativeMessageHeaders = Enumerable.Range(0, 10).Select(i => "header" + i).ToArray() })); + Assert.ThrowsAny(() => new AwsMessageTransport(new AwsMessageTransportOptions { NativeMessageHeaders = ["header", "header"] })); + Assert.ThrowsAny(() => new AwsMessageTransport(new AwsMessageTransportOptions { NativeMessageHeaders = [new string('a', 257)] })); + Assert.ThrowsAny(() => new AwsMessageTransport(new AwsMessageTransportOptions { NativeMessageHeaders = null! })); + } + + [Theory] + [InlineData("application/json", "{\"name\":\"héllo 世界\"}", false)] + [InlineData("application/json", "{\"name\":\"héllo 世界\"}", true)] + [InlineData("application/octet-stream", "binary", false)] + [InlineData("application/octet-stream", "binary", true)] + [InlineData(null, "unknown", false)] + [InlineData(null, "unknown", true)] + public async Task SendAndReceiveAsync_CompactEnvelope_PreservesPayloadMetadataAndNativeFilters(string? contentType, string text, bool nativeHeaders) + { + var token = TestContext.Current.CancellationToken; + var body = contentType == "application/octet-stream" ? Enumerable.Range(0, 256).Select(i => (byte)i).ToArray() : Encoding.UTF8.GetBytes(text); + var headers = MessageHeaders.Create(new Dictionary + { + [KnownHeaders.MessageType] = "order.v1", + [KnownHeaders.Priority] = "high", + [KnownHeaders.CorrelationId] = "trace-id", + [KnownHeaders.MessageId] = "independent-header-id", + [KnownHeaders.ContentType] = "independent-header-type", + ["Mixed-Case"] = "résumé" + }); + SendMessageBatchRequestEntry? sent = null; + var sqs = new Mock(); + sqs.Setup(s => s.GetQueueUrlAsync(It.IsAny(), It.IsAny())).ReturnsAsync(new GetQueueUrlResponse { QueueUrl = "http://test/queue" }); + sqs.Setup(s => s.SendMessageBatchAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync((SendMessageBatchRequest request, CancellationToken _) => + { + sent = Assert.Single(request.Entries); + return new SendMessageBatchResponse { Successful = [new SendMessageBatchResultEntry { Id = sent.Id, MessageId = "broker-id" }] }; + }); + sqs.Setup(s => s.ReceiveMessageAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync(() => new ReceiveMessageResponse { Messages = [new Message { MessageId = "broker-id", ReceiptHandle = "receipt", Body = sent!.MessageBody, MessageAttributes = sent.MessageAttributes }] }); + string[] nativeNames = nativeHeaders ? [KnownHeaders.MessageType, KnownHeaders.Priority, KnownHeaders.CorrelationId, "Mixed-Case"] : []; + var configuredNames = nativeNames.ToArray(); + await using var transport = new AwsMessageTransport(new AwsMessageTransportOptions { NativeMessageHeaders = configuredNames }, sqs.Object, Mock.Of()); + if (configuredNames.Length > 0) configuredNames[0] = "fnd.envelope"; + await transport.SendAsync(DestinationAddress.ForQueue("test"), [new TransportMessage { Body = body, ContentType = contentType, MessageId = "application-id", Headers = headers }], new(), token); + Assert.Equal(nativeNames.Length + 1, sent!.MessageAttributes.Count); + Assert.Contains("fnd.envelope", sent.MessageAttributes.Keys); + foreach (string key in nativeNames) + Assert.Equal(headers[key], sent.MessageAttributes[key].StringValue); + if (contentType == "application/json") Assert.Equal(text, sent.MessageBody); + var received = Assert.Single(await transport.ReceiveAsync(DestinationAddress.ForQueue("test"), new(), token)); + Assert.Null(received.EnvelopeError); + Assert.Equal(body, received.Body.ToArray()); + Assert.Equal("application-id", received.ApplicationMessageId); + Assert.Equal(contentType, received.ContentType); + Assert.Equal("broker-id", received.Id); + Assert.Equal(headers.Count, received.Headers.Count); + foreach (var header in headers) Assert.Equal(header.Value, received.Headers[header.Key]); + Assert.Equal("résumé", received.Headers["mixed-case"]); + } + + [Theory] + [InlineData("fnd.headers", "{invalid", "original body")] + [InlineData("fnd.encoding", "base64", "!!!")] + [InlineData("fnd.envelope", "{invalid", "dmFsaWQ=")] + [InlineData("fnd.envelope", "{\"Version\":2,\"Encoding\":\"text\",\"Headers\":{}}", "dmFsaWQ=")] + [InlineData("fnd.envelope", "{\"Version\":1,\"Encoding\":\"unknown\",\"Headers\":{}}", "dmFsaWQ=")] + [InlineData("fnd.envelope", "{\"Version\":1,\"Encoding\":\"text\",\"Headers\":null}", "dmFsaWQ=")] + public async Task ReceiveAsync_MalformedEnvelope_PreservesReceiptAndValidEntries(string attribute, string value, string body) + { + var token = TestContext.Current.CancellationToken; + var sqs = new Mock(); + sqs.Setup(s => s.GetQueueUrlAsync(It.IsAny(), It.IsAny())).ReturnsAsync(new GetQueueUrlResponse { QueueUrl = "http://test/queue" }); + sqs.Setup(s => s.ReceiveMessageAsync(It.IsAny(), It.IsAny())).ReturnsAsync(new ReceiveMessageResponse + { + Messages = [ + new Message { MessageId = "bad", ReceiptHandle = "bad-receipt", Body = body, MessageAttributes = new Dictionary { [attribute] = new() { StringValue = value, DataType = "String" } } }, + new Message { MessageId = "good", ReceiptHandle = "good-receipt", Body = "valid", MessageAttributes = new Dictionary { ["fnd.encoding"] = new() { StringValue = "text", DataType = "String" } } } + ] + }); + await using var transport = new AwsMessageTransport(new(), sqs.Object, Mock.Of()); + var entries = await transport.ReceiveAsync(DestinationAddress.ForQueue("test"), new ReceiveRequest { MaxMessages = 2 }, token); + Assert.Equal(2, entries.Count); + Assert.NotNull(entries[0].EnvelopeError); + Assert.Equal(body, Encoding.UTF8.GetString(entries[0].Body.Span)); + Assert.Equal("bad-receipt", entries[0].Receipt.TransportState); + Assert.Null(entries[1].EnvelopeError); + Assert.Equal("valid", Encoding.UTF8.GetString(entries[1].Body.Span)); + } +} diff --git a/tests/Foundatio.Aws.Tests/AwsMessageAdministrationTests.cs b/tests/Foundatio.Aws.Tests/AwsMessageAdministrationTests.cs new file mode 100644 index 000000000..6e5d91da3 --- /dev/null +++ b/tests/Foundatio.Aws.Tests/AwsMessageAdministrationTests.cs @@ -0,0 +1,59 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Xunit; + +namespace Foundatio.Aws.Tests; + +public class AwsMessageAdministrationTests +{ + [Fact] + public async Task ReplayBeyondFirstReceiveBatch_ResetsRetryStateAndPreservesOtherDeadLetters() + { + string? connection = Environment.GetEnvironmentVariable("FOUNDATIO_AWS_CONNECTION_STRING"); + Assert.SkipWhen(String.IsNullOrEmpty(connection), "FOUNDATIO_AWS_CONNECTION_STRING not set."); + var options = AwsMessageTransportOptions.FromConnectionString(connection); + options.ResourcePrefix = $"admin-test-{Guid.NewGuid():N}-"; + await using var transport = new AwsMessageTransport(options); + var token = TestContext.Current.CancellationToken; + var source = DestinationAddress.ForQueue("work"); + var dead = DestinationAddress.ForQueue("work.deadletter"); + await transport.EnsureAsync([new() { Address = source }, new() { Address = dead }], token); + try + { + var messages = Enumerable.Range(0, 15).Select(index => new TransportMessage + { + Body = "{}"u8.ToArray(), + ContentType = "application/json", + MessageId = $"original-{index}", + Headers = MessageHeaders.Create(new Dictionary + { + [KnownHeaders.Attempts] = "9", + [KnownHeaders.DeadLetterReason] = "Original failure" + }) + }).ToArray(); + (await transport.SendAsync(dead, messages, new(), token)).EnsureAccepted(messages.Length); + var administration = new MessageAdministration(transport); + var snapshot = await administration.PeekDeadLettersAsync(source, 15, token); + Assert.Equal(15, snapshot.Count); + var selected = snapshot[^1]; + Assert.True(await administration.ReplayDeadLetterAsync(source, selected.Id, cancellationToken: token)); + var replay = Assert.Single(await transport.ReceiveAsync(source, new() { MaxMessages = 1, MaxWaitTime = TimeSpan.FromSeconds(2) }, token)); + Assert.Equal(selected.ApplicationMessageId, replay.ApplicationMessageId); + Assert.NotNull(replay.EnqueuedUtc); + Assert.False(replay.Headers.ContainsKey(KnownHeaders.Attempts)); + Assert.False(replay.Headers.ContainsKey(KnownHeaders.DeadLetterReason)); + await transport.CompleteAsync(replay, token); + var remaining = await administration.PeekDeadLettersAsync(source, 15, token); + Assert.Equal(14, remaining.Count); + Assert.DoesNotContain(remaining, entry => entry.Id == selected.Id); + } + finally + { + await transport.DeleteAsync(source, token); + await transport.DeleteAsync(dead, token); + } + } +} diff --git a/tests/Foundatio.Aws.Tests/AwsMessageTransportConformanceTests.cs b/tests/Foundatio.Aws.Tests/AwsMessageTransportConformanceTests.cs new file mode 100644 index 000000000..2f84d0227 --- /dev/null +++ b/tests/Foundatio.Aws.Tests/AwsMessageTransportConformanceTests.cs @@ -0,0 +1,44 @@ +using System; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Foundatio.Tests.Messaging; +using Xunit; + +namespace Foundatio.Aws.Tests; + +/// +/// Runs the shared transport conformance suite against AWS SQS/SNS. Set the environment variable +/// FOUNDATIO_AWS_CONNECTION_STRING (e.g. serviceurl=http://localhost:4566;accesskey=test;secretkey=test;region=us-east-1 +/// for LocalStack, or real AWS credentials) to run; when it is not set every test is skipped. Inherits every base +/// [Fact], so a new conformance check runs against SQS/SNS automatically; capabilities SQS/SNS do not support +/// (priority, per-message expiration, push delivery, transport-native dead-letter) self-skip via their +/// ISupports* capability checks in the base suite. +/// +public class AwsMessageTransportConformanceTests : MessageTransportConformanceTests +{ + // One prefix per test run isolates these queues/topics from prior runs and other environments on the same broker. + private static readonly string RunPrefix = "fnd-conf-" + Guid.NewGuid().ToString("N")[..8] + "-"; + + public AwsMessageTransportConformanceTests(ITestOutputHelper output) : base(output) { } + + protected override IMessageTransport? CreateTransport() + { + string? connectionString = Environment.GetEnvironmentVariable("FOUNDATIO_AWS_CONNECTION_STRING"); + if (String.IsNullOrEmpty(connectionString)) + return null; // not configured -> the base suite skips every test + + var options = AwsMessageTransportOptions.FromConnectionString(connectionString); + options.ResourcePrefix = RunPrefix; + return new AwsMessageTransport(options); + } + + [Fact] + public override Task CompleteAsync_WithExpiredReceipt_ThrowsReceiptExpiredExceptionAsync() + { + // Explicit, visible opt-out (not a silent skip): SQS DeleteMessage with a stale/used receipt handle is + // idempotent and does not raise. Strict receipt validation is transport-specific, not part of the shared + // contract, so SQS does not satisfy this check. + Assert.Skip("SQS DeleteMessage is idempotent for a stale receipt handle; strict receipt validation is not part of the shared contract."); + return Task.CompletedTask; + } +} diff --git a/tests/Foundatio.Aws.Tests/AwsMessageTransportTests.cs b/tests/Foundatio.Aws.Tests/AwsMessageTransportTests.cs new file mode 100644 index 000000000..c11323aac --- /dev/null +++ b/tests/Foundatio.Aws.Tests/AwsMessageTransportTests.cs @@ -0,0 +1,154 @@ +using System; +using System.Collections.Generic; +using Amazon.SimpleNotificationService; +using Amazon.SimpleNotificationService.Model; +using System.Text; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Xunit; + +namespace Foundatio.Aws.Tests; + +public class AwsMessageTransportTests +{ + private static AwsMessageTransport? CreateTransport(string testName) + { + string? connectionString = Environment.GetEnvironmentVariable("FOUNDATIO_AWS_CONNECTION_STRING"); + if (String.IsNullOrEmpty(connectionString)) + return null; + + var options = AwsMessageTransportOptions.FromConnectionString(connectionString); + options.ResourcePrefix = $"fnd-{testName}-{Guid.NewGuid():N}"[..24] + "-"; + return new AwsMessageTransport(options); + } + + [Theory] + [InlineData(DestinationRole.Queue)] + [InlineData(DestinationRole.Topic)] + [InlineData(DestinationRole.Subscription)] + public async Task Provisioning_FreshInstanceValidatesAndDeletesExistingResourcesAsync(DestinationRole role) + { + string? connectionString = Environment.GetEnvironmentVariable("FOUNDATIO_AWS_CONNECTION_STRING"); + Assert.SkipWhen(String.IsNullOrEmpty(connectionString), "FOUNDATIO_AWS_CONNECTION_STRING not set."); + var options = AwsMessageTransportOptions.FromConnectionString(connectionString); + options.ResourcePrefix = $"cold-{Guid.NewGuid():N}-"; + var token = TestContext.Current.CancellationToken; + await using var first = new AwsMessageTransport(options); + await using var second = new AwsMessageTransport(options); + var destination = role switch + { + DestinationRole.Queue => DestinationAddress.ForQueue("work"), + DestinationRole.Topic => DestinationAddress.ForTopic("events"), + _ => DestinationAddress.ForSubscription("events", "audit") + }; + await first.EnsureAsync([new DestinationDeclaration { Address = destination }], token); + Assert.True(await second.ExistsAsync(destination, token)); + await second.DeleteAsync(destination, token); + Assert.False(await first.ExistsAsync(destination, token)); + await second.DeleteAsync(destination, token); + if (role == DestinationRole.Subscription) + await first.DeleteAsync(DestinationAddress.ForTopic("events"), token); + } + + [Fact] + public async Task NativeMessageHeaders_CustomTenantFilter_DeliversMatchingMessagesAsync() + { + string? connectionString = Environment.GetEnvironmentVariable("FOUNDATIO_AWS_CONNECTION_STRING"); + Assert.SkipWhen(String.IsNullOrEmpty(connectionString), "FOUNDATIO_AWS_CONNECTION_STRING not set."); + var options = AwsMessageTransportOptions.FromConnectionString(connectionString); + Assert.SkipWhen(String.IsNullOrEmpty(options.ServiceUrl), "This filter propagation smoke test requires LocalStack."); + options.ResourcePrefix = $"filter-{Guid.NewGuid():N}-"; + options.NativeMessageHeaders = ["tenant.id"]; + var config = new AmazonSimpleNotificationServiceConfig + { + ServiceURL = options.ServiceUrl, + AuthenticationRegion = (options.Region ?? Amazon.RegionEndpoint.USEast1).SystemName + }; + using var sns = options.Credentials is { } credentials ? new AmazonSimpleNotificationServiceClient(credentials, config) : new AmazonSimpleNotificationServiceClient(config); + await using var transport = new AwsMessageTransport(options); + var topic = DestinationAddress.ForTopic("events"); + var subscription = DestinationAddress.ForSubscription("events", "audit"); + var token = TestContext.Current.CancellationToken; + try + { + await transport.EnsureAsync([new DestinationDeclaration { Address = subscription }], token); + string topicArn = (await sns.CreateTopicAsync(new CreateTopicRequest { Name = options.ResourcePrefix + "events" }, token)).TopicArn; + var binding = Assert.Single((await sns.ListSubscriptionsByTopicAsync(new ListSubscriptionsByTopicRequest { TopicArn = topicArn }, token)).Subscriptions); + await sns.SetSubscriptionAttributesAsync(new SetSubscriptionAttributesRequest + { + SubscriptionArn = binding.SubscriptionArn, + AttributeName = "FilterPolicy", + AttributeValue = "{\"tenant.id\":[\"allowed\"]}" + }, token); + var sent = await transport.SendAsync(topic, + [Message("allowed"), Message("denied")], new(), token); + Assert.All(sent.Items, item => Assert.Equal(MessageSendStatus.Accepted, item.Status)); + var message = Assert.Single(await transport.ReceiveAsync(subscription, new ReceiveRequest { MaxMessages = 10, MaxWaitTime = TimeSpan.FromSeconds(2) }, token)); + Assert.Equal("allowed", message.Headers["tenant.id"]); + await transport.CompleteAsync(message, token); + Assert.Empty(await transport.ReceiveAsync(subscription, new ReceiveRequest { MaxMessages = 10, MaxWaitTime = TimeSpan.FromSeconds(1) }, token)); + } + finally + { + await transport.DeleteAsync(subscription, token); + await transport.DeleteAsync(topic, token); + } + + static TransportMessage Message(string tenant) => new() + { + Body = Encoding.UTF8.GetBytes("{}"), + ContentType = "application/json", + Headers = MessageHeaders.Create(new Dictionary { ["tenant.id"] = tenant }) + }; + } + + [Fact] + public async Task TextContentBody_RoundTripsThroughSqsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = CreateTransport("text"); + if (transport is null) + { + Assert.Skip("FOUNDATIO_AWS_CONNECTION_STRING not set."); + return; + } + + // Non-ASCII JSON exercises UTF-8 round-trip through the SQS string body (the text-content path that avoids base64). + string json = "{\"greeting\":\"héllo wörld\",\"n\":42}"; + await transport.EnsureAsync([new DestinationDeclaration { Address = DestinationAddress.ForQueue("text-body") }], cancellationToken); + + await transport.SendAsync(DestinationAddress.ForQueue("text-body"), + [new TransportMessage { Body = Encoding.UTF8.GetBytes(json), ContentType = "application/json" }], + new TransportSendOptions(), cancellationToken); + + var entries = await transport.ReceiveAsync(DestinationAddress.ForQueue("text-body"), new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, cancellationToken); + var entry = Assert.Single(entries); + Assert.Equal(json, Encoding.UTF8.GetString(entry.Body.Span)); + await transport.CompleteAsync(entry, cancellationToken); + } + + [Fact] + public async Task BinaryContentBody_RoundTripsThroughSqsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = CreateTransport("binary"); + if (transport is null) + { + Assert.Skip("FOUNDATIO_AWS_CONNECTION_STRING not set."); + return; + } + + // Non-UTF-8 bytes must still round-trip (via base64) when no text content type is declared. + byte[] payload = [0x00, 0x01, 0xFF, 0xFE, 0x10, 0x80]; + await transport.EnsureAsync([new DestinationDeclaration { Address = DestinationAddress.ForQueue("binary-body") }], cancellationToken); + + await transport.SendAsync(DestinationAddress.ForQueue("binary-body"), + [new TransportMessage { Body = payload }], + new TransportSendOptions(), cancellationToken); + + var entries = await transport.ReceiveAsync(DestinationAddress.ForQueue("binary-body"), new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, cancellationToken); + var entry = Assert.Single(entries); + Assert.Equal(payload, entry.Body.ToArray()); + await transport.CompleteAsync(entry, cancellationToken); + } +} diff --git a/tests/Foundatio.Aws.Tests/AwsNodeSubscriptionTests.cs b/tests/Foundatio.Aws.Tests/AwsNodeSubscriptionTests.cs new file mode 100644 index 000000000..543580b90 --- /dev/null +++ b/tests/Foundatio.Aws.Tests/AwsNodeSubscriptionTests.cs @@ -0,0 +1,40 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Xunit; + +namespace Foundatio.Aws.Tests; + +public class AwsNodeSubscriptionTests +{ + [Fact] + public async Task Nodes_ReceiveIndependentCopies_AndDisposeTheirResources() + { + string? connection = Environment.GetEnvironmentVariable("FOUNDATIO_AWS_CONNECTION_STRING"); + Assert.SkipWhen(String.IsNullOrEmpty(connection), "FOUNDATIO_AWS_CONNECTION_STRING not set."); + var options = AwsMessageTransportOptions.FromConnectionString(connection); + options.ResourcePrefix = $"node-test-{Guid.NewGuid():N}-"; + await using var transport = new AwsMessageTransport(options); + await using var bus = new MessageBus(transport, new MessageBusOptions { OwnsTransport = false }); + var token = TestContext.Current.CancellationToken; + var firstReceived = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var secondReceived = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var first = await bus.SubscribeNodeAsync((_, _) => { firstReceived.TrySetResult(); return Task.CompletedTask; }, new() { Topic = "events", NodeId = "first" }, token); + var second = await bus.SubscribeNodeAsync((_, _) => { secondReceived.TrySetResult(); return Task.CompletedTask; }, new() { Topic = "events", NodeId = "second" }, token); + try + { + Assert.NotEqual(first.Source, second.Source); + await bus.PublishAsync("changed", new MessagePublishOptions { Topic = "events" }, token); + await Task.WhenAll(firstReceived.Task, secondReceived.Task).WaitAsync(TimeSpan.FromSeconds(15), token); + } + finally + { + await first.DisposeAsync(); + await second.DisposeAsync(); + await transport.DeleteAsync(DestinationAddress.ForTopic("events"), token); + } + Assert.False(await transport.ExistsAsync(first.Source, token)); + Assert.False(await transport.ExistsAsync(second.Source, token)); + } +} diff --git a/tests/Foundatio.Aws.Tests/Foundatio.Aws.Tests.csproj b/tests/Foundatio.Aws.Tests/Foundatio.Aws.Tests.csproj new file mode 100644 index 000000000..e7889c7b0 --- /dev/null +++ b/tests/Foundatio.Aws.Tests/Foundatio.Aws.Tests.csproj @@ -0,0 +1,11 @@ + + + + $(TestingPlatformCommandLineArguments) --ignore-exit-code 8 + + + + + + + diff --git a/tests/Foundatio.Aws.Tests/README.md b/tests/Foundatio.Aws.Tests/README.md new file mode 100644 index 000000000..be7f56a91 --- /dev/null +++ b/tests/Foundatio.Aws.Tests/README.md @@ -0,0 +1,33 @@ +# Foundatio.Aws.Tests + +Runs the shared transport conformance suite (`MessageTransportConformanceTests`) against the AWS SQS/SNS +`IMessageTransport` (`Foundatio.Aws`). This is a temporary in-repo provider used to validate the redesigned transport +contract against a real broker before it is extracted to its own package. + +## Run against LocalStack + +```sh +# 1. Start LocalStack (SQS + SNS) +docker compose -f tests/Foundatio.Aws.Tests/docker-compose.yml up -d + +# 2. Point the tests at it +export FOUNDATIO_AWS_CONNECTION_STRING="serviceurl=http://localhost:4566;accesskey=test;secretkey=test;region=us-east-1" + +# 3. Run the conformance suite +dotnet test tests/Foundatio.Aws.Tests/Foundatio.Aws.Tests.csproj +``` + +When `FOUNDATIO_AWS_CONNECTION_STRING` is **not** set, broker integration tests are skipped. Mocked batching, envelope and configuration tests still run. + +To run against real AWS, set the connection string to real credentials/region (omit `serviceurl`), e.g. +`accesskey=...;secretkey=...;region=us-east-1`. + +## Capability coverage + +SQS/SNS supports pull receive, visibility timeout, lock renewal, redelivery delay (12h cap), delayed delivery (15-min +cap), provisioning, and stats. It does **not** support per-message priority, per-message TTL/expiration, push delivery, +or transport-native dead-lettering (the core owns retry/dead-lettering). Conformance tests for those capabilities skip +automatically via their `ISupports*` checks. + +Each run uses a unique `ResourcePrefix` so leftover messages from a prior run cannot leak in. LocalStack state is +ephemeral; restart the container to reset. diff --git a/tests/Foundatio.Aws.Tests/docker-compose.yml b/tests/Foundatio.Aws.Tests/docker-compose.yml new file mode 100644 index 000000000..053c8bc0d --- /dev/null +++ b/tests/Foundatio.Aws.Tests/docker-compose.yml @@ -0,0 +1,10 @@ +services: + localstack: + # Pinned to the 3.x community edition: SQS/SNS run free, with no auth token (the rolling `latest`/Pro tag now + # requires a LocalStack license). + image: localstack/localstack:3 + ports: + - "4566:4566" + environment: + - SERVICES=sqs,sns + - DEBUG=0 diff --git a/tests/Foundatio.Redis.Tests/Foundatio.Redis.Tests.csproj b/tests/Foundatio.Redis.Tests/Foundatio.Redis.Tests.csproj new file mode 100644 index 000000000..3d2ab2138 --- /dev/null +++ b/tests/Foundatio.Redis.Tests/Foundatio.Redis.Tests.csproj @@ -0,0 +1,10 @@ + + + + $(TestingPlatformCommandLineArguments) --ignore-exit-code 8 + + + + + + diff --git a/tests/Foundatio.Redis.Tests/README.md b/tests/Foundatio.Redis.Tests/README.md new file mode 100644 index 000000000..cda720aab --- /dev/null +++ b/tests/Foundatio.Redis.Tests/README.md @@ -0,0 +1,25 @@ +# Foundatio.Redis.Tests + +Validates the temporary in-repo Redis providers against a real Redis: + +- **`RedisJobRuntimeStore`** — runs the shared `JobRuntimeStoreConformanceTests` suite (the same assertions the + in-memory reference store passes) plus `RedisJobStoreIntegrationTests` (delayed-send fallback + CRON end-to-end). +- **`RedisStreamsMessageTransport`** — runs the shared `MessageTransportConformanceTests` suite (pull, settlement, + visibility timeout, lock renewal, redelivery delay, dead-letter, provisioning, stats, topic fan-out; push/priority/ + expiration/delayed-delivery skip via capability gates) plus `RedisStreamsTransportIntegrationTests` (cross-instance + crash recovery, the core's retry/dead-letter machinery over Streams, and `PubSub` fan-out). + +## Running + +Start Redis and point the tests at it. Without the connection string every test is skipped. + +```sh +docker compose -f tests/Foundatio.Redis.Tests/docker-compose.yml up -d + +export FOUNDATIO_REDIS_CONNECTION_STRING=localhost:6399 +dotnet run --project tests/Foundatio.Redis.Tests +``` + +Each test runs under a unique key prefix, so concurrent runs and leftover keys never collide. The job-store suite +drives lease/expiry timing with a `FakeTimeProvider` (no real sleeps); the transport suite uses real, whole-second +timing windows (the same cross-transport windows the AWS suite uses). diff --git a/tests/Foundatio.Redis.Tests/RedisJobRuntimeStoreConformanceTests.cs b/tests/Foundatio.Redis.Tests/RedisJobRuntimeStoreConformanceTests.cs new file mode 100644 index 000000000..71247dfdc --- /dev/null +++ b/tests/Foundatio.Redis.Tests/RedisJobRuntimeStoreConformanceTests.cs @@ -0,0 +1,22 @@ +using System; +using Foundatio.Jobs; +using Foundatio.Tests.Jobs; +using Xunit; + +namespace Foundatio.Redis.Tests; + +/// +/// Runs the shared conformance suite against a real Redis. Set +/// FOUNDATIO_REDIS_CONNECTION_STRING (e.g. localhost:6399 for the bundled docker-compose Redis) to run; +/// when it is not set every test is skipped. Each test gets a unique key prefix so runs never collide. Inheriting the +/// base [Fact]s means a new conformance check automatically runs against Redis with no override to forget. +/// +public class RedisJobRuntimeStoreConformanceTests : JobRuntimeStoreConformanceTests +{ + public RedisJobRuntimeStoreConformanceTests(ITestOutputHelper output) : base(output) { } + + protected override IJobRuntimeStore? CreateStore(TimeProvider timeProvider, JobRuntimeStoreOptions? options = null) => + RedisTestConnection.Multiplexer is { } connection + ? new RedisJobRuntimeStore(new RedisJobRuntimeStoreOptions { ConnectionMultiplexer = connection, TimeProvider = timeProvider, KeyPrefix = $"conformance:{Guid.NewGuid():N}:", Runtime = options ?? new() }) + : null; // not configured -> the base suite skips every test +} diff --git a/tests/Foundatio.Redis.Tests/RedisJobStoreIntegrationTests.cs b/tests/Foundatio.Redis.Tests/RedisJobStoreIntegrationTests.cs new file mode 100644 index 000000000..10acd47f4 --- /dev/null +++ b/tests/Foundatio.Redis.Tests/RedisJobStoreIntegrationTests.cs @@ -0,0 +1,384 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Time.Testing; +using Xunit; + +namespace Foundatio.Redis.Tests; + +/// +/// End-to-end tests that wire the real messaging core / CRON scheduler on top of the Redis +/// and exercise the two paths the store exists to support but that the primitive-level conformance suite does not cover: +/// (1) a delayed send whose delay exceeds the transport's being +/// durably stored in Redis and drained by the dispatch pump when due, and (2) CRON occurrences being materialized, run, +/// retried/dead-lettered, and stale-reclaimed through Redis. +/// +/// Gated on FOUNDATIO_REDIS_CONNECTION_STRING; skips when unset. Each test isolates under a unique key prefix. +/// +public class RedisJobStoreIntegrationTests +{ + private static JobTypeRegistry CreateJobRegistry() => new(typeof(RedisJobStoreIntegrationTests).GetNestedTypes(System.Reflection.BindingFlags.NonPublic) + .Where(t => t.IsClass && !t.IsAbstract && typeof(IJob).IsAssignableFrom(t)) + .Select(t => new JobTypeRegistration(t.FullName!, t))); + + [Fact] + public async Task BrokerMonitoring_ExistingRecordsWithoutCachedKeys_KeepTheirIndexesAsync() + { + var connection = RedisTestConnection.Multiplexer; + Assert.SkipWhen(connection is null, "FOUNDATIO_REDIS_CONNECTION_STRING not set."); + var token = TestContext.Current.CancellationToken; + string prefix = $"test:legacy-monitoring:{Guid.NewGuid():N}:"; + var time = new FakeTimeProvider(); + var store = new RedisJobRuntimeStore(new RedisJobRuntimeStoreOptions { ConnectionMultiplexer = connection, KeyPrefix = prefix, TimeProvider = time }); + await store.CreateIfAbsentAsync(new JobState + { + JobId = "legacy", Name = "résumé", QueueName = "exports/日本語", ExecutionOwner = JobExecutionOwner.Broker, + HistoryRetention = TimeSpan.FromMinutes(1), HistoryExpiresUtc = time.GetUtcNow().AddMinutes(1) + }, token); + await connection.GetDatabase().HashDeleteAsync(prefix + "job:legacy", ["monitorName", "monitorQueue", "monitorExpiry"]); + + var attempt = await store.BeginBrokerAttemptAsync("legacy", 1, "worker", token); + Assert.NotNull(attempt); + Assert.Equal(1, await store.CountAsync(new JobQuery { Name = "résumé", Status = JobStatus.Processing }, token)); + Assert.Equal(1, await store.CountAsync(new JobQuery { QueueName = "exports/日本語", Status = JobStatus.Processing }, token)); + Assert.Equal(0, await store.CountAsync(new JobQuery { Status = JobStatus.Queued }, token)); + Assert.Equal("legacy", Assert.Single(await store.QueryAsync(new JobQuery { QueueName = "exports/日本語" }, token)).JobId); + time.Advance(TimeSpan.FromMinutes(1)); + Assert.Null(await store.GetAsync("legacy", token)); + Assert.Equal(0, await store.CountAsync(new JobQuery { Name = "résumé" }, token)); + Assert.Equal(0, await store.CountAsync(new JobQuery { QueueName = "exports/日本語" }, token)); + } + + [Fact] + public async Task CreateIfAbsentAsync_ConcurrentAdmission_EnforcesCapacityAtomicallyAsync() + { + var connection = RedisTestConnection.Multiplexer; + Assert.SkipWhen(connection is null, "FOUNDATIO_REDIS_CONNECTION_STRING not set."); + var token = TestContext.Current.CancellationToken; + var store = new RedisJobRuntimeStore(new RedisJobRuntimeStoreOptions { ConnectionMultiplexer = connection, KeyPrefix = $"test:capacity:{Guid.NewGuid():N}:", MaxJobs = 1 }); + var accepted = await Task.WhenAll(Enumerable.Range(0, 10).Select(async index => + { + try + { + await store.CreateIfAbsentAsync(new JobState { JobId = index.ToString(), Name = "work" }, token); + return true; + } + catch (JobException) { return false; } + })); + Assert.Single(accepted, value => value); + var existing = Assert.Single(await store.QueryAsync(new JobQuery(), token)); + await store.CreateIfAbsentAsync(existing, token); + } + + [Fact] + public async Task QueryAsync_SparseFilter_ContinuesPastEmptyBoundedPageAsync() + { + var connection = RedisTestConnection.Multiplexer; + Assert.SkipWhen(connection is null, "FOUNDATIO_REDIS_CONNECTION_STRING not set."); + var token = TestContext.Current.CancellationToken; + var store = RedisTestConnection.CreateStore(connection); + await Task.WhenAll(Enumerable.Range(0, 1001).Select(index => store.CreateIfAbsentAsync(new JobState + { + JobId = index.ToString("D4"), + Name = "same", + Status = index == 1000 ? JobStatus.Queued : JobStatus.Completed + }, token))); + var page = await store.QueryAsync(new JobQuery { Name = "same", Status = JobStatus.Queued }, token); + Assert.Empty(page); + Assert.NotNull(page.ContinuationToken); + var last = await store.QueryAsync(new JobQuery { Name = "same", Status = JobStatus.Queued, AfterJobId = page.ContinuationToken }, token); + Assert.Equal("1000", Assert.Single(last).JobId); + Assert.Null(last.ContinuationToken); + } + + [Fact] + public async Task Schedules_FreshStoreInstanceReadsPersistedDefinitionAndArgumentsAsync() + { + var connection = RedisTestConnection.Multiplexer; + Assert.SkipWhen(connection is null, "FOUNDATIO_REDIS_CONNECTION_STRING not set."); + var token = TestContext.Current.CancellationToken; + string prefix = $"test:schedules:{Guid.NewGuid():N}:"; + var first = new RedisJobRuntimeStore(connection, prefix); + await first.ReconcileAsync(new ScheduledJobDefinition + { + Name = "persisted", + Cron = "0 3 * * *", + JobType = "report.v2", + TimeZoneId = "America/Chicago", + Payload = "hello"u8.ToArray(), + PayloadType = "report-args.v2" + }, token); + + var second = new RedisJobRuntimeStore(connection, prefix); + var definition = await second.GetScheduleAsync("persisted", token); + Assert.NotNull(definition); + Assert.Equal("report.v2", definition.JobType); + Assert.Equal("America/Chicago", definition.TimeZoneId); + Assert.Equal("hello"u8.ToArray(), definition.Payload!.Value.ToArray()); + Assert.Equal("report-args.v2", definition.PayloadType); + Assert.Equal(1, definition.Revision); + await second.UnscheduleAsync("persisted", token); + Assert.Null(await first.GetScheduleAsync("persisted", token)); + } + + [Fact] + public async Task DelayedQueueSend_BeyondTransportLimit_StoresInRedisAndDrainsWhenDueAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var cancellationToken = TestContext.Current.CancellationToken; + var now = DateTimeOffset.UtcNow; + + // Within the transport's advertised maximum: delivered natively, nothing is parked in Redis. + var nativeStore = RedisTestConnection.CreateStore(connection); + await using var nativeTransport = new CappedDelayTransport(TimeSpan.FromMinutes(15)); + await using var nativeQueue = new MessageBus(nativeTransport, new MessageBusOptions { RuntimeStore = nativeStore }); + var nativeProcessor = new ScheduledMessageDispatcher(nativeStore, nativeTransport); + + await nativeQueue.SendAsync(new PreviewWorkItem { Data = "soon" }, new MessageSendOptions { Delay = TimeSpan.FromMinutes(5) }, cancellationToken); + Assert.Equal(1, nativeTransport.SendCount); + Assert.NotNull(nativeTransport.LastSendOptions?.DeliverAt); + Assert.Equal(0, await nativeProcessor.DispatchDueAsync(now.AddYears(1), cancellationToken: cancellationToken)); + + // Beyond the transport's maximum: routed into the Redis store rather than truncated to the broker ceiling. + var fallbackStore = RedisTestConnection.CreateStore(connection); + await using var fallbackTransport = new CappedDelayTransport(TimeSpan.FromMinutes(15)); + await using var fallbackQueue = new MessageBus(fallbackTransport, new MessageBusOptions { RuntimeStore = fallbackStore }); + var fallbackProcessor = new ScheduledMessageDispatcher(fallbackStore, fallbackTransport); + + await fallbackQueue.SendAsync(new PreviewWorkItem { Data = "later" }, new MessageSendOptions { Delay = TimeSpan.FromHours(1) }, cancellationToken); + Assert.Equal(0, fallbackTransport.SendCount); + + // Durably parked in Redis and time-gated: a drain before the due time claims nothing; only when due does the + // pump pull it from Redis and hand it to the transport. + Assert.Equal(0, await fallbackProcessor.DispatchDueAsync(now, cancellationToken: cancellationToken)); + Assert.Equal(0, fallbackTransport.SendCount); + + Assert.Equal(1, await fallbackProcessor.DispatchDueAsync(now.AddHours(2), cancellationToken: cancellationToken)); + Assert.Equal(1, fallbackTransport.SendCount); + + var deliveredContext = new TaskCompletionSource>(TaskCreationOptions.RunContinuationsAsynchronously); + await using var subscription = await fallbackQueue.ConsumeAsync((context, _) => + { + deliveredContext.TrySetResult(context); + return Task.CompletedTask; + }, new MessageConsumerOptions { AckMode = AckMode.Manual }, cancellationToken); + + var delivered = await deliveredContext.Task.WaitAsync(TimeSpan.FromSeconds(10), cancellationToken); + Assert.Equal("later", delivered.Message.Data); + await delivered.CompleteAsync(cancellationToken); + } + + [Fact] + public async Task CronOccurrence_MaterializesRunsAndDedupesThroughRedisAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var cancellationToken = TestContext.Current.CancellationToken; + var store = RedisTestConnection.CreateStore(connection); + var scheduler = new InMemoryScheduledJobStore(); + var (processor, worker, probe) = CreateProcessor(store, scheduler); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "nightly", + Cron = "* * * * *", + JobType = typeof(ProbeJob).FullName! + }, cancellationToken); + + // Materialize: one occurrence is written to Redis as a Scheduled JobState + a JobOccurrence dispatch. + var first = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + var dispatch = Assert.Single(first); + Assert.Equal("nightly:20260101000000:global", dispatch.JobId); + Assert.Equal("nightly", dispatch.ScheduleName); + Assert.Equal("nightly", dispatch.Name); + + var scheduled = await store.GetAsync(dispatch.JobId!, cancellationToken); + Assert.NotNull(scheduled); + Assert.Equal(JobStatus.Queued, scheduled.Status); + Assert.Equal(new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero), scheduled.ScheduledForUtc); + + // Deterministic occurrence id dedupes against the Redis row: a second materialize pass at the same time is a no-op. + Assert.Empty(await processor.EnqueueDueOccurrencesAsync(now, cancellationToken)); + + // Claim from Redis and run: the occurrence completes and the run is recorded once. + Assert.Equal(1, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + Assert.Equal(1, probe.RunCount); + + var completed = await store.GetAsync(dispatch.JobId!, cancellationToken); + Assert.NotNull(completed); + Assert.Equal(JobStatus.Completed, completed.Status); + Assert.Equal(1, completed.Attempt); + Assert.Equal(100, completed.Progress); + + // The dispatch was completed (removed) in Redis, so a later drain finds nothing. + Assert.Equal(0, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + } + + [Fact] + public async Task CronOccurrence_RetryDeadLetterAndStaleReclaimThroughRedisAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var cancellationToken = TestContext.Current.CancellationToken; + var time = new FakeTimeProvider(new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero)); + var store = RedisTestConnection.CreateStore(connection, time); + var scheduler = new InMemoryScheduledJobStore(); + var (processor, worker, probe) = CreateProcessor(store, scheduler, timeProvider: time); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + // (a) Retry-then-dead-letter: a failing occurrence is rescheduled in Redis until its retry budget is spent. + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "flaky", + Cron = "* * * * *", + JobType = typeof(FailingJob).FullName!, + MaxAttempts = 2 + }, cancellationToken); + var flaky = Assert.Single(await processor.EnqueueDueOccurrencesAsync(now, cancellationToken)); + + Assert.Equal(1, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + var retried = await store.GetAsync(flaky.JobId!, cancellationToken); + Assert.NotNull(retried); + Assert.Equal(JobStatus.Queued, retried.Status); + Assert.Equal(1, retried.Attempt); + + time.Advance(TimeSpan.FromMinutes(2)); + Assert.Equal(1, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + var deadlettered = await store.GetAsync(flaky.JobId!, cancellationToken); + Assert.NotNull(deadlettered); + Assert.Equal(JobStatus.Failed, deadlettered.Status); + Assert.Equal(2, deadlettered.Attempt); + + // (b) Stale reclaim: an occurrence stuck in Processing under a dead node with an expired lease is reclaimed + // (via the Redis CAS reclaim) and run to completion by the live node. + const string jobId = "nightly:20260101000000:global"; + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "nightly", + Cron = "* * * * *", + JobType = typeof(ProbeJob).FullName!, + MaxAttempts = 2 + }, cancellationToken); + await store.CreateIfAbsentAsync(new JobState + { + JobId = jobId, + Name = "nightly", + Status = JobStatus.Processing, + JobType = typeof(ProbeJob).FullName, + MaxAttempts = 2, + Attempt = 1, + NodeId = "node-b", + LeaseExpiresUtc = now.AddMinutes(-1), + ScheduledForUtc = now.AddSeconds(-30) + }, cancellationToken); + + Assert.Equal(1, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + var reclaimed = await store.GetAsync(jobId, cancellationToken); + Assert.NotNull(reclaimed); + Assert.Equal(JobStatus.Completed, reclaimed.Status); + Assert.Equal(2, reclaimed.Attempt); + Assert.Equal(1, probe.RunCount); + } + + private static (JobScheduleProcessor Processor, IJobWorker Worker, Probe Probe) CreateProcessor(IJobRuntimeStore store, IMessageTransport? transport = null) + => CreateProcessor(store, new InMemoryScheduledJobStore(), transport); + + private static (JobScheduleProcessor Processor, IJobWorker Worker, Probe Probe) CreateProcessor(IJobRuntimeStore store, IScheduledJobStore scheduler, IMessageTransport? transport = null, TimeProvider? timeProvider = null) + { + var probe = new Probe(); + var serviceProvider = new ServiceCollection().AddSingleton(probe).BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "node-a", JobTypes = CreateJobRegistry(), TimeProvider = timeProvider }); + return (new JobScheduleProcessor(scheduler, store, new JobScheduleProcessorOptions { NodeId = "node-a", TimeProvider = timeProvider }), worker, probe); + } + + private sealed class Probe + { + private int _runCount; + public int RunCount => Volatile.Read(ref _runCount); + public void Record() => Interlocked.Increment(ref _runCount); + } + + private sealed class ProbeJob(Probe probe) : IJob + { + public Task RunAsync(JobExecutionContext context) + { + context.CancellationToken.ThrowIfCancellationRequested(); + probe.Record(); + return Task.FromResult(JobResult.Success); + } + } + + private sealed class FailingJob : IJob + { + public Task RunAsync(JobExecutionContext context) + { + context.CancellationToken.ThrowIfCancellationRequested(); + return Task.FromResult(JobResult.FromException(new InvalidOperationException("boom"))); + } + } + + private sealed class PreviewWorkItem + { + public string? Data { get; set; } + } + + // Minimal pull transport with a configurable native delayed-delivery ceiling, so a delay beyond the cap is forced + // through the runtime store (mirrors the fixture used by the in-memory MessageBus tests). + private sealed class CappedDelayTransport : IMessageTransport, ISupportsPull, ITransportInfo + { + private readonly Queue _entries = new(); + + public CappedDelayTransport(TimeSpan? maxDeliveryDelay) => MaxDeliveryDelay = maxDeliveryDelay; + + public TimeSpan? MaxDeliveryDelay { get; } + public int SendCount { get; private set; } + public TransportSendOptions? LastSendOptions { get; private set; } + + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => new HashSet { DestinationRole.Queue }; + + public TransportCapabilities GetCapabilities(DestinationAddress destination) => + new() { DelayedDelivery = true, MaxDeliveryDelay = MaxDeliveryDelay }; + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + SendCount += messages.Count; + LastSendOptions = options; + var items = new SendItemResult[messages.Count]; + for (int i = 0; i < messages.Count; i++) + { + string id = messages[i].MessageId ?? Guid.NewGuid().ToString("N"); + _entries.Enqueue(new TransportEntry { Id = id, Destination = destination, Body = messages[i].Body, Headers = messages[i].Headers, Receipt = new Receipt() }); + items[i] = new SendItemResult { MessageId = id }; + } + + return Task.FromResult(new SendResult { Items = items }); + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + => Task.FromResult>(_entries.Count > 0 ? [_entries.Dequeue()] : []); + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public ValueTask DisposeAsync() => ValueTask.CompletedTask; + } +} diff --git a/tests/Foundatio.Redis.Tests/RedisRegistrationTests.cs b/tests/Foundatio.Redis.Tests/RedisRegistrationTests.cs new file mode 100644 index 000000000..ac241bd92 --- /dev/null +++ b/tests/Foundatio.Redis.Tests/RedisRegistrationTests.cs @@ -0,0 +1,41 @@ +using System; +using Microsoft.Extensions.DependencyInjection; +using StackExchange.Redis; +using Xunit; + +namespace Foundatio.Redis.Tests; + +public class RedisRegistrationTests +{ + [Fact] + public void UseRedis_ConflictingExplicitConnections_FailsBeforeConnecting() + { + var builder = new ServiceCollection().AddFoundatio(); + builder.Messaging.UseRedis(connectionString: "localhost:6379"); + var ex = Assert.Throws(() => builder.Jobs.UseRedis(connectionString: "localhost:6380")); + Assert.Contains("share one Redis connection", ex.Message); + Assert.DoesNotContain("6380", ex.Message); + } + + [Fact] + public void UseRedis_ExistingConnection_RejectsIgnoredConnectionString() + { + var services = new ServiceCollection(); + services.AddSingleton(_ => throw new InvalidOperationException("Must not connect during registration.")); + var builder = services.AddFoundatio(); + builder.Messaging.UseRedis(); + var ex = Assert.Throws(() => builder.Jobs.UseRedis(connectionString: "localhost:6380")); + Assert.Contains("already registered", ex.Message); + Assert.DoesNotContain("6380", ex.Message); + } + + [Fact] + public void UseRedis_DefaultThenExplicitConnection_AllowsOneSharedSetting() + { + var builder = new ServiceCollection().AddFoundatio(); + builder.Messaging.UseRedis(); + builder.Jobs.UseRedis(connectionString: "localhost:6380"); + builder.Messaging.UseRedis(connectionString: "localhost:6380"); + Assert.Throws(() => builder.Jobs.UseRedis(connectionString: "localhost:6379")); + } +} diff --git a/tests/Foundatio.Redis.Tests/RedisResourceLockTests.cs b/tests/Foundatio.Redis.Tests/RedisResourceLockTests.cs new file mode 100644 index 000000000..3c0b242c3 --- /dev/null +++ b/tests/Foundatio.Redis.Tests/RedisResourceLockTests.cs @@ -0,0 +1,42 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Lock; +using Xunit; + +namespace Foundatio.Redis.Tests; + +public class RedisResourceLockTests +{ + [Fact] + public async Task ExpiredOwner_CannotReleaseOrRenewANewOwnersLock() + { + var connection = RedisTestConnection.Multiplexer; + Assert.SkipWhen(connection is null, "FOUNDATIO_REDIS_CONNECTION_STRING not set."); + var provider = new RedisLockProvider(connection!, $"native-lock:{Guid.NewGuid():N}:"); + var token = TestContext.Current.CancellationToken; + await using var first = await provider.AcquireAsync("report", TimeSpan.FromMilliseconds(100), cancellationToken: token); + await Task.Delay(150, token); + await using var second = await provider.AcquireAsync("report", TimeSpan.FromSeconds(10), cancellationToken: token); + await Assert.ThrowsAsync(() => first.RenewAsync()); + await first.ReleaseAsync(); + Assert.True(await provider.IsLockedAsync("report")); + await second.RenewAsync(TimeSpan.FromSeconds(20)); + Assert.Equal(1, second.RenewalCount); + await second.ReleaseAsync(); + Assert.False(await provider.IsLockedAsync("report")); + } + + [Fact] + public async Task ContendedAcquisition_StopsWaitingWhenCancelled() + { + var connection = RedisTestConnection.Multiplexer; + Assert.SkipWhen(connection is null, "FOUNDATIO_REDIS_CONNECTION_STRING not set."); + var provider = new RedisLockProvider(connection!, $"native-lock:{Guid.NewGuid():N}:"); + await using var held = await provider.AcquireAsync("report", cancellationToken: TestContext.Current.CancellationToken); + using var timeout = CancellationTokenSource.CreateLinkedTokenSource(TestContext.Current.CancellationToken); + timeout.CancelAfter(TimeSpan.FromMilliseconds(100)); + Assert.Null(await provider.TryAcquireAsync("report", cancellationToken: timeout.Token)); + Assert.True(await provider.IsLockedAsync("report")); + } +} diff --git a/tests/Foundatio.Redis.Tests/RedisStreamsTransportConformanceTests.cs b/tests/Foundatio.Redis.Tests/RedisStreamsTransportConformanceTests.cs new file mode 100644 index 000000000..c935c5b8b --- /dev/null +++ b/tests/Foundatio.Redis.Tests/RedisStreamsTransportConformanceTests.cs @@ -0,0 +1,30 @@ +using System; +using Foundatio.Messaging; +using Foundatio.Tests.Messaging; +using Xunit; + +namespace Foundatio.Redis.Tests; + +/// +/// Runs the shared transport conformance suite against the Redis Streams transport. Set +/// FOUNDATIO_REDIS_CONNECTION_STRING (e.g. localhost:6399) to run; skips when unset. A unique key prefix +/// per transport isolates each test's streams. Capabilities Streams does not provide (push delivery, per-message +/// priority, per-message expiration, native scheduled delivery) are skipped by the base suite's capability gates. +/// +public class RedisStreamsTransportConformanceTests : MessageTransportConformanceTests +{ + public RedisStreamsTransportConformanceTests(ITestOutputHelper output) : base(output) { } + + protected override IMessageTransport? CreateTransport() + { + if (RedisTestConnection.Multiplexer is not { } connection) + return null; // not configured -> the base suite skips every test + + return new RedisStreamsMessageTransport(new RedisStreamsMessageTransportOptions + { + ConnectionMultiplexer = connection, + KeyPrefix = $"fnd-conf:{Guid.NewGuid():N}:" + }); + } + +} diff --git a/tests/Foundatio.Redis.Tests/RedisStreamsTransportIntegrationTests.cs b/tests/Foundatio.Redis.Tests/RedisStreamsTransportIntegrationTests.cs new file mode 100644 index 000000000..598875a7a --- /dev/null +++ b/tests/Foundatio.Redis.Tests/RedisStreamsTransportIntegrationTests.cs @@ -0,0 +1,384 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Foundatio.Jobs; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace Foundatio.Redis.Tests; + +/// +/// End-to-end tests for the Redis Streams transport that the cross-transport conformance suite can't express: at-least-once +/// recovery across two consumer instances, the core's retry/dead-letter machinery driving the transport, and topic +/// fan-out through the facade. Gated on FOUNDATIO_REDIS_CONNECTION_STRING; unique key prefix +/// per test. +/// +public class RedisStreamsTransportIntegrationTests +{ + private static RedisStreamsMessageTransport CreateTransport(StackExchange.Redis.IConnectionMultiplexer connection, string prefix, string? consumer = null) => + new(new RedisStreamsMessageTransportOptions + { + ConnectionMultiplexer = connection, + KeyPrefix = prefix, + ConsumerName = consumer + }); + + private static string NewPrefix() => $"fnd-it:{Guid.NewGuid():N}:"; + + [Fact] + public async Task MessagingScheduling_DifferentTransportPrefixes_IsolatesDispatchesAsync() + { + var connection = RedisTestConnection.Multiplexer; + Assert.SkipWhen(connection is null, "FOUNDATIO_REDIS_CONNECTION_STRING not set."); + var token = TestContext.Current.CancellationToken; + var firstServices = new ServiceCollection(); + firstServices.AddSingleton(connection); + firstServices.AddFoundatio().Messaging.UseRedis(o => o.KeyPrefix = NewPrefix()); + var secondServices = new ServiceCollection(); + secondServices.AddSingleton(connection); + secondServices.AddFoundatio().Messaging.UseRedis(o => o.KeyPrefix = NewPrefix()); + await using var first = firstServices.BuildServiceProvider(); + await using var second = secondServices.BuildServiceProvider(); + var firstStore = first.GetRequiredService(); + var secondStore = second.GetRequiredService(); + string id = Guid.NewGuid().ToString("N"); + await firstStore.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = id, Destination = DestinationAddress.ForQueue("work"), Body = "test"u8.ToArray(), DueUtc = DateTimeOffset.UtcNow + }, token); + var foreignClaims = await secondStore.ClaimDueDispatchesAsync(DateTimeOffset.UtcNow, 100, "second", TimeSpan.FromMinutes(1), token); + foreach (var claim in foreignClaims) + await secondStore.CompleteDispatchAsync(claim.DispatchId, "second", token); + Assert.DoesNotContain(foreignClaims, claim => claim.DispatchId == id); + Assert.Equal(id, Assert.Single(await firstStore.ClaimDueDispatchesAsync(DateTimeOffset.UtcNow, 100, "first", TimeSpan.FromMinutes(1), token)).DispatchId); + Assert.True(await firstStore.CompleteDispatchAsync(id, "first", token)); + } + + [Fact] + public async Task ReceiveAsync_MissingQueue_DoesNotProvisionAsync() + { + var connection = RedisTestConnection.Multiplexer; + Assert.SkipWhen(connection is null, "FOUNDATIO_REDIS_CONNECTION_STRING not set."); + var token = TestContext.Current.CancellationToken; + await using var transport = CreateTransport(connection, NewPrefix()); + var queue = DestinationAddress.ForQueue("missing"); + await Assert.ThrowsAnyAsync(() => transport.ReceiveAsync(queue, new ReceiveRequest(), token)); + Assert.False(await transport.ExistsAsync(queue, token)); + } + + [Fact] + public async Task SendAsync_AtCapacity_PreservesUnreadWorkAndResumesAfterSettlementAsync() + { + var connection = RedisTestConnection.Multiplexer; + Assert.SkipWhen(connection is null, "FOUNDATIO_REDIS_CONNECTION_STRING not set."); + var token = TestContext.Current.CancellationToken; + await using var transport = new RedisStreamsMessageTransport(new RedisStreamsMessageTransportOptions + { + ConnectionMultiplexer = connection, + KeyPrefix = NewPrefix(), + MaxPendingMessages = 1 + }); + var topic = DestinationAddress.ForTopic("bounded"); + var first = DestinationAddress.ForSubscription("bounded", "first"); + var second = DestinationAddress.ForSubscription("bounded", "second"); + await transport.EnsureAsync([new DestinationDeclaration { Address = first }, new DestinationDeclaration { Address = second }], token); + await transport.SendAsync(topic, [Message("one")], new TransportSendOptions(), token); + Assert.Equal(MessageSendStatus.Rejected, Assert.Single((await transport.SendAsync(topic, [Message("two")], new TransportSendOptions(), token)).Items).Status); + await transport.CompleteAsync(Assert.Single(await transport.ReceiveAsync(first, new ReceiveRequest(), token)), token); + Assert.Equal(MessageSendStatus.Rejected, Assert.Single((await transport.SendAsync(topic, [Message("two")], new TransportSendOptions(), token)).Items).Status); + var held = Assert.Single(await transport.ReceiveAsync(second, new ReceiveRequest(), token)); + Assert.Equal("one", System.Text.Encoding.UTF8.GetString(held.Body.Span)); + await transport.CompleteAsync(held, token); + await transport.SendAsync(topic, [Message("two")], new TransportSendOptions(), token); + Assert.Equal("two", System.Text.Encoding.UTF8.GetString(Assert.Single(await transport.ReceiveAsync(first, new ReceiveRequest(), token)).Body.Span)); + } + + [Fact] + public async Task ReceiveAsync_RecoversPendingEntryMissingLeaseMetadataAsync() + { + var connection = RedisTestConnection.Multiplexer; + Assert.SkipWhen(connection is null, "FOUNDATIO_REDIS_CONNECTION_STRING not set."); + var token = TestContext.Current.CancellationToken; + string prefix = NewPrefix(); + await using var transport = new RedisStreamsMessageTransport(new RedisStreamsMessageTransportOptions + { + ConnectionMultiplexer = connection, + KeyPrefix = prefix, + DefaultVisibilityTimeout = TimeSpan.FromMilliseconds(10) + }); + var source = DestinationAddress.ForQueue("orphan"); + await transport.EnsureAsync([new DestinationDeclaration { Address = source }], token); + await transport.SendAsync(source, [Message("orphan")], new TransportSendOptions(), token); + var pending = await connection.GetDatabase().StreamReadGroupAsync(prefix + "q:" + Convert.ToHexString(System.Text.Encoding.UTF8.GetBytes("orphan")), "foundatio", "crashed", ">", 1); + Assert.Single(pending); + await Task.Delay(30, token); + + var recovered = await transport.ReceiveAsync(source, new ReceiveRequest { MaxMessages = 1 }, TimeSpan.FromMinutes(1), token); + Assert.Equal(pending[0].Id.ToString(), Assert.Single(recovered).Id); + await transport.CompleteAsync(recovered[0], token); + } + + [Fact] + public async Task Settlement_AfterLeaseExpires_RejectsEveryStaleMutationAsync() + { + var connection = RedisTestConnection.Multiplexer; + Assert.SkipWhen(connection is null, "FOUNDATIO_REDIS_CONNECTION_STRING not set."); + var token = TestContext.Current.CancellationToken; + var time = new Microsoft.Extensions.Time.Testing.FakeTimeProvider(); + await using var transport = new RedisStreamsMessageTransport(new RedisStreamsMessageTransportOptions + { + ConnectionMultiplexer = connection, + KeyPrefix = NewPrefix(), + TimeProvider = time + }); + var source = DestinationAddress.ForQueue("expired"); + await transport.EnsureAsync([new DestinationDeclaration { Address = source }], token); + await transport.SendAsync(source, [Message("expired")], new TransportSendOptions(), token); + var entry = Assert.Single(await transport.ReceiveAsync(source, new ReceiveRequest(), TimeSpan.FromSeconds(1), token)); + time.Advance(TimeSpan.FromSeconds(2)); + + await Assert.ThrowsAsync(() => transport.CompleteAsync(entry, token)); + await Assert.ThrowsAsync(() => transport.AbandonAsync(entry, token)); + await Assert.ThrowsAsync(() => transport.RenewLockAsync(entry, TimeSpan.FromMinutes(1), token)); + await Assert.ThrowsAsync(() => transport.DeadLetterAsync(entry, "stale", token)); + Assert.Equal(entry.Id, Assert.Single(await transport.ReceiveAsync(source, new ReceiveRequest(), TimeSpan.FromMinutes(1), token)).Id); + } + + [Fact] + public async Task CrashedConsumer_LeaseLapses_AnotherInstanceReclaimsAndCompletesAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var ct = TestContext.Current.CancellationToken; + string prefix = NewPrefix(); + var visibility = TimeSpan.FromSeconds(2); + + // Both instances share the same key prefix so they operate on the same streams (the lease lives in Redis). + await using var nodeA = CreateTransport(connection, prefix, "node-a"); + await using var nodeB = CreateTransport(connection, prefix, "node-b"); + + await nodeA.EnsureAsync([new DestinationDeclaration { Address = DestinationAddress.ForQueue("work") }], ct); + await nodeA.SendAsync(DestinationAddress.ForQueue("work"), [Message("survive-me")], new TransportSendOptions(), ct); + + // node-a receives and then "crashes" — it never settles the message. + var heldByA = Assert.Single(await nodeA.ReceiveAsync(DestinationAddress.ForQueue("work"), new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, visibility, ct)); + Assert.Equal(1, heldByA.DeliveryCount); + + // While node-a's lease is live, node-b must not see it. + Assert.Empty(await nodeB.ReceiveAsync(DestinationAddress.ForQueue("work"), new ReceiveRequest { MaxWaitTime = TimeSpan.FromMilliseconds(100) }, visibility, ct)); + + // After the lease lapses, node-b reclaims the in-flight message (lease state lives in Redis) and completes it. + var reclaimedByB = Assert.Single(await nodeB.ReceiveAsync(DestinationAddress.ForQueue("work"), new ReceiveRequest { MaxWaitTime = visibility + TimeSpan.FromSeconds(5) }, visibility, ct)); + Assert.Equal(heldByA.Id, reclaimedByB.Id); + Assert.Equal(2, reclaimedByB.DeliveryCount); + Assert.Equal("survive-me", System.Text.Encoding.UTF8.GetString(reclaimedByB.Body.Span)); + await nodeB.CompleteAsync(reclaimedByB, ct); + + var stats = await nodeB.GetStatsAsync(DestinationAddress.ForQueue("work"), ct); + Assert.Equal(0, stats.Queued); + Assert.Equal(0, stats.Working); + } + + [Fact] + public async Task Core_RetriesFailedHandler_ThenDeadLettersAfterMaxAttemptsAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var ct = TestContext.Current.CancellationToken; + var transport = CreateTransport(connection, NewPrefix()); + await using var queue = new MessageBus(transport, new MessageBusOptions()); + + // (a) A handler that throws once is redelivered (via the transport) and succeeds on the second attempt — the + // core's retry machinery works unchanged over Streams. + int retryAttempts = 0; + var succeeded = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var retryConsumer = await queue.ConsumeAsync((message, _) => + { + int attempt = Interlocked.Increment(ref retryAttempts); + if (attempt == 1) + throw new InvalidOperationException("first attempt fails"); + + Assert.Equal(2, message.Attempts); + succeeded.TrySetResult(); + return Task.CompletedTask; + }, new MessageConsumerOptions { MaxAttempts = 3, RedeliveryBackoff = _ => TimeSpan.FromMilliseconds(200) }, ct); + + // (b) A handler that always throws is dead-lettered once its attempt budget is spent. + await using var poisonConsumer = await queue.ConsumeAsync((_, _) => + throw new InvalidOperationException("always fails"), + new MessageConsumerOptions { MaxAttempts = 2, RedeliveryBackoff = _ => TimeSpan.FromMilliseconds(100) }, ct); + + await queue.SendAsync(new RetryItem { Data = "retry" }, cancellationToken: ct); + await queue.SendAsync(new PoisonItem { Data = "poison" }, cancellationToken: ct); + + await succeeded.Task.WaitAsync(TimeSpan.FromSeconds(30), ct); + Assert.Equal(2, Volatile.Read(ref retryAttempts)); + + // The poison message lands in the dead-letter stream after exhausting its 2 attempts. + MessageDestinationStats stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("streams-poison"), ct); + for (int i = 0; i < 100 && stats.Deadletter == 0; i++) + { + await Task.Delay(100, ct); + stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("streams-poison"), ct); + } + + Assert.Equal(1, stats.Deadletter); + Assert.Equal(0, stats.Working); + + // The poison payload is inspectable in the dead-letter stream with a reason recorded by the core. + var deadLettered = Assert.Single(await transport.PeekDeadLetteredAsync(DestinationAddress.ForQueue("streams-poison"), new DeadLetterQuery { Limit = 10 }, ct)); + Assert.NotEmpty(deadLettered.Headers[KnownHeaders.DeadLetterReason]); + } + + [Fact] + public async Task PubSub_PublishToTopic_FansOutToEverySubscriptionAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var ct = TestContext.Current.CancellationToken; + var transport = CreateTransport(connection, NewPrefix()); + await using var pubsub = new MessageBus(transport, new MessageBusOptions()); + + var receivedByA = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var receivedByB = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + + await using var subA = await pubsub.SubscribeAsync((message, _) => + { + receivedByA.TrySetResult(message.Message.Data ?? ""); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "sub-a" }, ct); + + await using var subB = await pubsub.SubscribeAsync((message, _) => + { + receivedByB.TrySetResult(message.Message.Data ?? ""); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "sub-b" }, ct); + + await pubsub.PublishAsync(new FanItem { Data = "broadcast" }, cancellationToken: ct); + + // Each named subscription is its own consumer group, so both receive an independent copy. Delivery is + // poll-driven across two subscriptions (the core pull-fallback loop), so allow generous headroom for the whole + // conformance suite hammering the same Redis concurrently. + await Task.WhenAll(receivedByA.Task, receivedByB.Task).WaitAsync(TimeSpan.FromSeconds(30), ct); + Assert.Equal("broadcast", await receivedByA.Task); + Assert.Equal("broadcast", await receivedByB.Task); + } + + [Fact] + public async Task MessageBus_SendAndPublishSameType_StayIsolatedAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var ct = TestContext.Current.CancellationToken; + var transport = CreateTransport(connection, NewPrefix()); + await using var bus = new MessageBus(transport, new MessageBusOptions()); + + // One subscription listens on both of the type's channels. Send targets the queue-role stream and Publish the + // topic-role stream, so the same route name must never cross-deliver: exactly one delivery per verb. (A shared + // stream would deliver each message through BOTH channels — 4 deliveries instead of 2.) + var sent = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var published = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + int deliveries = 0; + + await using var consumer = await bus.ConsumeAsync((message, _) => + { + Interlocked.Increment(ref deliveries); + sent.TrySetResult(message.Message.Data ?? ""); + return Task.CompletedTask; + }, cancellationToken: ct); + await using var subscription = await bus.SubscribeAsync((message, _) => + { + Interlocked.Increment(ref deliveries); + published.TrySetResult(message.Message.Data ?? ""); + return Task.CompletedTask; + }, cancellationToken: ct); + + await bus.SendAsync(new DualItem { Data = "for-one" }, cancellationToken: ct); + await bus.PublishAsync(new DualItem { Data = "for-all" }, cancellationToken: ct); + + await Task.WhenAll(sent.Task, published.Task).WaitAsync(TimeSpan.FromSeconds(30), ct); + Assert.Equal("for-one", await sent.Task); + Assert.Equal("for-all", await published.Task); + + // Give any cross-delivery a moment to surface, then assert exactly one delivery per verb. + await Task.Delay(500, ct); + Assert.Equal(2, Volatile.Read(ref deliveries)); + } + + [Fact] + public async Task Publish_CompletedByOneGroup_StillDeliveredToSlowerGroupAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var ct = TestCancellation(); + await using var transport = CreateTransport(connection, NewPrefix()); + + await transport.EnsureAsync( + [ + new DestinationDeclaration { Address = DestinationAddress.ForTopic("iso-topic") }, + new DestinationDeclaration { Address = DestinationAddress.ForSubscription("iso-topic", "sub-a") }, + new DestinationDeclaration { Address = DestinationAddress.ForSubscription("iso-topic", "sub-b") } + ], ct); + + await transport.SendAsync(DestinationAddress.ForTopic("iso-topic"), [Message("retained")], new TransportSendOptions(), ct); + + // Group A reads and completes FIRST; the entry must remain on the topic stream for group B (completing must + // not delete a shared topic entry other groups haven't read yet). + var byA = Assert.Single(await transport.ReceiveAsync(DestinationAddress.ForSubscription("iso-topic", "sub-a"), new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, ct)); + await transport.CompleteAsync(byA, ct); + + var byB = Assert.Single(await transport.ReceiveAsync(DestinationAddress.ForSubscription("iso-topic", "sub-b"), new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, ct)); + Assert.Equal("retained", System.Text.Encoding.UTF8.GetString(byB.Body.Span)); + await transport.CompleteAsync(byB, ct); + } + + private static CancellationToken TestCancellation() => TestContext.Current.CancellationToken; + + private static TransportMessage Message(string body) => + new() { Body = System.Text.Encoding.UTF8.GetBytes(body) }; + + [MessageRoute("streams-dual")] + private sealed class DualItem + { + public string? Data { get; set; } + } + + [MessageRoute("streams-retry")] + private sealed class RetryItem + { + public string? Data { get; set; } + } + + [MessageRoute("streams-poison")] + private sealed class PoisonItem + { + public string? Data { get; set; } + } + + [MessageRoute("streams-topic")] + private sealed class FanItem + { + public string? Data { get; set; } + } +} diff --git a/tests/Foundatio.Redis.Tests/RedisTestConnection.cs b/tests/Foundatio.Redis.Tests/RedisTestConnection.cs new file mode 100644 index 000000000..41a06dfc5 --- /dev/null +++ b/tests/Foundatio.Redis.Tests/RedisTestConnection.cs @@ -0,0 +1,30 @@ +using System; +using Foundatio.Jobs; +using StackExchange.Redis; + +namespace Foundatio.Redis.Tests; + +/// +/// Shared, lazily-opened Redis connection for the Redis test suites. Gated on +/// FOUNDATIO_REDIS_CONNECTION_STRING (e.g. localhost:6399 for the bundled docker-compose Redis); when it +/// is unset is null and the suites skip every test. +/// +internal static class RedisTestConnection +{ + private static readonly Lazy Shared = new(() => + { + string? connectionString = Environment.GetEnvironmentVariable("FOUNDATIO_REDIS_CONNECTION_STRING"); + return String.IsNullOrEmpty(connectionString) ? null : ConnectionMultiplexer.Connect(connectionString); + }); + + public static IConnectionMultiplexer? Multiplexer => Shared.Value; + + /// Creates a store under a unique key prefix so concurrent tests and leftover keys never collide. + public static RedisJobRuntimeStore CreateStore(IConnectionMultiplexer connection, TimeProvider? timeProvider = null) => + new(new RedisJobRuntimeStoreOptions + { + ConnectionMultiplexer = connection, + KeyPrefix = $"fnd-it:{Guid.NewGuid():N}:", + TimeProvider = timeProvider + }); +} diff --git a/tests/Foundatio.Redis.Tests/docker-compose.yml b/tests/Foundatio.Redis.Tests/docker-compose.yml new file mode 100644 index 000000000..d41de3cdd --- /dev/null +++ b/tests/Foundatio.Redis.Tests/docker-compose.yml @@ -0,0 +1,6 @@ +services: + redis: + image: redis:7-alpine + # Mapped to 6399 to avoid clashing with a local Redis on the default 6379. + ports: + - "6399:6379" diff --git a/tests/Foundatio.Tests/Caching/InMemoryHybridCacheClientTests.cs b/tests/Foundatio.Tests/Caching/InMemoryHybridCacheClientTests.cs index 993cf0f61..f5547d95a 100644 --- a/tests/Foundatio.Tests/Caching/InMemoryHybridCacheClientTests.cs +++ b/tests/Foundatio.Tests/Caching/InMemoryHybridCacheClientTests.cs @@ -741,6 +741,6 @@ public override void Dispose() { base.Dispose(); _distributedCache.Dispose(); - _messageBus.Dispose(); + _messageBus.DisposeAsync().AsTask().GetAwaiter().GetResult(); } } diff --git a/tests/Foundatio.Tests/DeclarativeRegistrationTests.cs b/tests/Foundatio.Tests/DeclarativeRegistrationTests.cs new file mode 100644 index 000000000..4afff05d8 --- /dev/null +++ b/tests/Foundatio.Tests/DeclarativeRegistrationTests.cs @@ -0,0 +1,312 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Extensions.Hosting.Jobs; +using Foundatio.Extensions.Hosting.Messaging; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Xunit; + +namespace Foundatio.Tests; + +public class DeclarativeRegistrationTests +{ + [Fact] + public async Task RegisteringClientsAndHandlers_DoesNotStartConsumersAsync() + { + var services = new ServiceCollection(); + services.AddFoundatio().Messaging.UseInMemory() + .AddConsumer((_, _) => Task.CompletedTask); + await using var provider = services.BuildServiceProvider(); + Assert.Empty(provider.GetServices()); + Assert.NotNull(provider.GetRequiredService()); + } + + [Fact] + public async Task ExplicitConsumersAndSubscribers_DeliverTheirRespectivePatternsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var probe = new HandlerProbe(); + + var services = new ServiceCollection(); + services.AddLogging(); + services.AddSingleton(probe); + services.AddFoundatio() + .Messaging.UseInMemory() + .AddConsumer() + .AddSubscriber("orders") // class handler + .AddConsumer((context, _) => { probe.Record($"task:{context.Message.Id}"); return Task.CompletedTask; }); // delegate handler + + services.AddMessageConsumers(); + await using var provider = services.BuildServiceProvider(); + var hosted = provider.GetServices().ToList(); + // Auto-registered: startup topology, ONE handler host driving every handler, and the misconfiguration validator. + Assert.Equal(2, hosted.Count); + Assert.Single(hosted, service => service.GetType().Name == "MessageHandlerHostedService"); + + foreach (var service in hosted) + await service.StartAsync(cancellationToken); + + try + { + var bus = provider.GetRequiredService(); + + // Queue consumers and event subscribers are registered separately. + await bus.SendAsync(new HandledOrder { Id = "sent" }, cancellationToken: cancellationToken); + await bus.PublishAsync(new HandledOrder { Id = "published" }, cancellationToken: cancellationToken); + await bus.SendAsync(new HandledTask { Id = "t1" }, cancellationToken: cancellationToken); + + Assert.True(await probe.WaitForAsync(3, TimeSpan.FromSeconds(10)), $"handled: {String.Join(",", probe.Events)}"); + Assert.Contains("order:sent", probe.Events); + Assert.Contains("order:published", probe.Events); + Assert.Contains("task:t1", probe.Events); + + // Same type sent AND published: exactly one delivery per verb — the queue and topic namespaces are + // segregated, so a send is never fanned out and a publish is never consumed as queue work. + Assert.Equal(3, probe.Events.Count); + } + finally + { + foreach (var service in hosted) + await service.StopAsync(cancellationToken); + } + } + + [Fact] + public async Task AddSubscriber_NamedSubscriptionsCompeteAndTemporarySubscriptionsBroadcastAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + + // Two service providers sharing one transport simulate two scaled instances of the same service. + var sharedProbe = new HandlerProbe(); + var instanceA = BuildInstance(transport, sharedProbe); + var instanceB = BuildInstance(transport, sharedProbe); + + await using (instanceA.Provider) + await using (instanceB.Provider) + { + await StartAsync(instanceA, cancellationToken); + await StartAsync(instanceB, cancellationToken); + + try + { + var bus = instanceA.Provider.GetRequiredService(); + + // Default subscription = service identity, shared by both instances => they compete: one copy total. + await bus.PublishAsync(new HandledEvent { Id = "shared" }, cancellationToken: cancellationToken); + Assert.True(await sharedProbe.WaitForAsync(1, TimeSpan.FromSeconds(10)), $"handled: {String.Join(",", sharedProbe.Events)}"); + await Task.Delay(250, cancellationToken); + Assert.Single(sharedProbe.Events, e => e.StartsWith("event:", StringComparison.Ordinal)); + + // PerInstance handlers each take a unique subscription => every instance receives its own copy. + await bus.PublishAsync(new HandledBroadcast { Id = "all" }, cancellationToken: cancellationToken); + Assert.True(await sharedProbe.WaitForAsync(3, TimeSpan.FromSeconds(10)), $"handled: {String.Join(",", sharedProbe.Events)}"); + Assert.Equal(2, sharedProbe.Events.Count(e => e.StartsWith("broadcast:", StringComparison.Ordinal))); + } + finally + { + await StopAsync(instanceA, cancellationToken); + await StopAsync(instanceB, cancellationToken); + } + } + } + + [Fact] + public async Task AddSubscriber_IndependentSubscriptionsDoNotCompeteWithQueueConsumerAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var probe = new HandlerProbe(); + + var services = new ServiceCollection(); + services.AddLogging(); + services.AddSingleton(probe); + services.AddFoundatio() + .Messaging.UseInMemory() + .AddSubscriber("events") + .AddSubscriber("second-events") + .AddConsumer(); + + services.AddMessageConsumers(); + await using var provider = services.BuildServiceProvider(); + var hosted = provider.GetServices().ToList(); + foreach (var service in hosted) + await service.StartAsync(cancellationToken); + + try + { + var bus = provider.GetRequiredService(); + + // An event reaches EVERY handler class (each is its own subscriber group). + await bus.PublishAsync(new HandledEvent { Id = "e1" }, cancellationToken: cancellationToken); + Assert.True(await probe.WaitForAsync(2, TimeSpan.FromSeconds(10)), $"handled: {String.Join(",", probe.Events)}"); + Assert.Contains("event:e1", probe.Events); + Assert.Contains("second:e1", probe.Events); + + // A command reaches exactly ONE handler (competing consumers on the type's send channel). + await bus.SendAsync(new HandledEvent { Id = "s1" }, cancellationToken: cancellationToken); + Assert.True(await probe.WaitForAsync(3, TimeSpan.FromSeconds(10)), $"handled: {String.Join(",", probe.Events)}"); + await Task.Delay(250, cancellationToken); + Assert.Equal(1, probe.Events.Count(e => e.EndsWith(":s1", StringComparison.Ordinal))); + } + finally + { + foreach (var service in hosted) + await service.StopAsync(cancellationToken); + } + } + + [Fact] + public async Task AddCronJob_RegistersDefinitionAndSchedulesWhenPumpStartsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + + var services = new ServiceCollection(); + services.AddLogging(); + services.AddFoundatio() + .Jobs.UseInMemory() + .ConfigureWorker(o => o with { NodeId = "cron-probe" }) + .AddCronJob("* * * * *", o => o.Scope = ScheduledJobScope.PerNode); + + services.AddJobScheduler(); + services.AddMessageConsumers(); + await using var provider = services.BuildServiceProvider(); + + // The builder records the schedule as a DI singleton with the requested scope and a type-derived name. + var definition = Assert.Single(provider.GetServices()); + Assert.Equal(typeof(CronProbeJob).FullName, definition.JobType); + Assert.Equal(ScheduledJobScope.PerNode, definition.Scope); + Assert.Equal(nameof(CronProbeJob), definition.Name); + + // Starting the runtime pump schedules registered CRON jobs into the scheduler — no manual ScheduleAsync call. + var hosted = provider.GetServices().ToList(); + foreach (var service in hosted) + await service.StartAsync(cancellationToken); + + try + { + var scheduler = provider.GetRequiredService(); + ScheduledJobDefinition? scheduled = null; + long deadline = Environment.TickCount64 + 10_000; + while (Environment.TickCount64 < deadline) + { + scheduled = (await scheduler.GetSchedulesAsync(cancellationToken: cancellationToken)).FirstOrDefault(s => s.Name == nameof(CronProbeJob)); + if (scheduled is not null) + break; + await Task.Delay(25, cancellationToken); + } + + Assert.NotNull(scheduled); + Assert.Equal(ScheduledJobScope.PerNode, scheduled!.Scope); + } + finally + { + foreach (var service in hosted) + await service.StopAsync(cancellationToken); + } + } + + private static (ServiceProvider Provider, List Hosted) BuildInstance(InMemoryMessageTransport transport, HandlerProbe probe) + { + var services = new ServiceCollection(); + services.AddLogging(); + services.AddSingleton(probe); + services.AddFoundatio() + .Messaging.UseTransport(transport) + .AddSubscriber("events") + .AddTemporarySubscriber(); + + services.AddMessageConsumers(); + var provider = services.BuildServiceProvider(); + return (provider, provider.GetServices().ToList()); + } + + private static async Task StartAsync((ServiceProvider Provider, List Hosted) instance, CancellationToken cancellationToken) + { + foreach (var service in instance.Hosted) + await service.StartAsync(cancellationToken); + } + + private static async Task StopAsync((ServiceProvider Provider, List Hosted) instance, CancellationToken cancellationToken) + { + foreach (var service in instance.Hosted) + await service.StopAsync(cancellationToken); + } + + private sealed class HandlerProbe + { + private readonly ConcurrentBag _events = new(); + public IReadOnlyCollection Events => _events; + public void Record(string value) => _events.Add(value); + + public async Task WaitForAsync(int count, TimeSpan timeout) + { + long deadline = Environment.TickCount64 + (long)timeout.TotalMilliseconds; + while (Environment.TickCount64 < deadline) + { + if (_events.Count >= count) + return true; + await Task.Delay(25); + } + return _events.Count >= count; + } + } + + [MessageRoute("declarative-orders")] + public class HandledOrder { public string Id { get; set; } = ""; } + + [MessageRoute("declarative-tasks")] + public class HandledTask { public string Id { get; set; } = ""; } + + [MessageRoute("declarative-events")] + public class HandledEvent { public string Id { get; set; } = ""; } + + [MessageRoute("declarative-broadcasts")] + public class HandledBroadcast { public string Id { get; set; } = ""; } + + private sealed class OrderHandler(HandlerProbe probe) : IMessageHandler + { + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + probe.Record($"order:{context.Message.Id}"); + return Task.CompletedTask; + } + } + + private sealed class EventHandler(HandlerProbe probe) : IMessageHandler + { + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + probe.Record($"event:{context.Message.Id}"); + return Task.CompletedTask; + } + } + + private sealed class SecondEventHandler(HandlerProbe probe) : IMessageHandler + { + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + probe.Record($"second:{context.Message.Id}"); + return Task.CompletedTask; + } + } + + private sealed class BroadcastHandler(HandlerProbe probe) : IMessageHandler + { + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + probe.Record($"broadcast:{context.Message.Id}"); + return Task.CompletedTask; + } + } + + private sealed class CronProbeJob : IJob + { + public Task RunAsync(JobExecutionContext context) => Task.FromResult(JobResult.Success); + } +} diff --git a/tests/Foundatio.Tests/DeveloperExperienceTests.cs b/tests/Foundatio.Tests/DeveloperExperienceTests.cs new file mode 100644 index 000000000..cdc1f38b3 --- /dev/null +++ b/tests/Foundatio.Tests/DeveloperExperienceTests.cs @@ -0,0 +1,136 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Xunit; + +namespace Foundatio.Tests; + +public class DeveloperExperienceTests +{ + [Theory] + [InlineData(true, false)] + [InlineData(false, true)] + [InlineData(true, true)] + public async Task AddFoundatioWorker_StartsOnlyConfiguredFeaturesAsync(bool messaging, bool jobs) + { + var token = TestContext.Current.CancellationToken; + var handled = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var builder = Host.CreateApplicationBuilder(); + builder.Services.AddFoundatioWorker(foundatio => + { + if (messaging) + foundatio.Messaging.UseInMemory().AddConsumer((_, _) => { handled.TrySetResult(); return Task.CompletedTask; }); + if (jobs) + foundatio.Jobs.UseInMemory().AddCronJob("0 2 * * *"); + }); + using var host = builder.Build(); + await host.StartAsync(token); + try + { + var names = host.Services.GetServices().Select(s => s.GetType().Name).ToArray(); + Assert.Equal(messaging, names.Contains("MessageHandlerHostedService")); + Assert.Equal(jobs, names.Contains("JobWorkerService")); + Assert.Equal(jobs, names.Contains("JobSchedulerService")); + Assert.Equal(messaging, names.Contains("ScheduledMessageDispatcherService")); + if (messaging) + { + await host.Services.GetRequiredService().SendAsync(new Ping(), cancellationToken: token); + await handled.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + } + if (jobs) + { + Assert.NotNull(await host.Services.GetRequiredService().GetScheduleAsync(nameof(NoopJob), token)); + var handle = await host.Services.GetRequiredService().EnqueueAsync(cancellationToken: token); + using var deadline = CancellationTokenSource.CreateLinkedTokenSource(token); + deadline.CancelAfter(TimeSpan.FromSeconds(5)); + while ((await handle.GetStateAsync(deadline.Token))!.Status != JobStatus.Completed) + await Task.Delay(10, deadline.Token); + } + } + finally { await host.StopAsync(token); } + } + + [Fact] + public void AddFoundatioWorker_MissingDependencies_ExplainsTheFix() + { + var ex = Assert.Throws(() => new ServiceCollection().AddFoundatioWorker(f => f.Messaging.AddConsumer((_, _) => Task.CompletedTask))); + Assert.Contains("Messaging.Use", ex.Message); + ex = Assert.Throws(() => new ServiceCollection().AddFoundatioWorker(f => f.Jobs.AddJobType())); + Assert.Contains("Jobs.Use", ex.Message); + } + + [Theory] + [InlineData("")] + [InlineData(" ")] + public void AddSubscriber_InvalidDurableName_FailsAtRegistration(string? name) + { + Assert.ThrowsAny(() => new ServiceCollection().AddFoundatio().Messaging.AddSubscriber((_, _) => Task.CompletedTask, name!)); + } + + [Theory] + [InlineData(0)] + [InlineData(1)] + [InlineData(2)] + [InlineData(3)] + public void AddConsumer_InvalidOptions_FailsAtRegistration(int scenario) + { + Assert.ThrowsAny(() => new ServiceCollection().AddFoundatio().Messaging.AddConsumer((_, _) => Task.CompletedTask, options => + { + if (scenario == 0) options.MaxConcurrency = 0; + if (scenario == 1) options.MaxAttempts = 0; + if (scenario == 2) options.AckMode = (AckMode)99; + if (scenario == 3) options.Destination = " "; + })); + } + + [Theory] + [InlineData(0)] + [InlineData(1)] + [InlineData(2)] + [InlineData(3)] + public void AddCronJob_InvalidOptions_FailsAtRegistration(int scenario) + { + Assert.ThrowsAny(() => new ServiceCollection().AddFoundatio().Jobs.AddCronJob("* * * * *", options => + { + if (scenario == 0) options.MaxAttempts = 0; + if (scenario == 1) options.MisfireWindow = TimeSpan.FromDays(2); + if (scenario == 2) options.Scope = (ScheduledJobScope)99; + if (scenario == 3) options.Name = " "; + })); + } + + [Fact] + public void AddJobType_AbstractJob_FailsAtRegistration() + { + Assert.Throws(() => new ServiceCollection().AddFoundatio().Jobs.AddJobType()); + } + + [Fact] + public async Task AddTemporarySubscriber_StatesItsLifetimeExplicitlyAsync() + { + var token = TestContext.Current.CancellationToken; + var received = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var builder = Host.CreateApplicationBuilder(); + builder.Services.AddFoundatioWorker(f => f.Messaging.UseInMemory() + .AddTemporarySubscriber((_, _) => { received.TrySetResult(); return Task.CompletedTask; })); + using var host = builder.Build(); + await host.StartAsync(token); + try + { + await host.Services.GetRequiredService().PublishAsync(new Ping(), cancellationToken: token); + await received.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + } + finally { await host.StopAsync(token); } + } + + private sealed record Ping; + private sealed class NoopJob : IJob + { + public Task RunAsync(JobExecutionContext context) => Task.FromResult(JobResult.Success); + } +} diff --git a/tests/Foundatio.Tests/Foundatio.Tests.csproj b/tests/Foundatio.Tests/Foundatio.Tests.csproj index 36614c657..24bb2d408 100644 --- a/tests/Foundatio.Tests/Foundatio.Tests.csproj +++ b/tests/Foundatio.Tests/Foundatio.Tests.csproj @@ -9,6 +9,7 @@ + diff --git a/tests/Foundatio.Tests/Jobs/InMemoryJobQueueTests.cs b/tests/Foundatio.Tests/Jobs/InMemoryJobQueueTests.cs deleted file mode 100644 index 05bea7e55..000000000 --- a/tests/Foundatio.Tests/Jobs/InMemoryJobQueueTests.cs +++ /dev/null @@ -1,47 +0,0 @@ -using System; -using System.Threading.Tasks; -using Foundatio.Queues; - -using Xunit; - -namespace Foundatio.Tests.Jobs; - -public class InMemoryJobQueueTests : JobQueueTestsBase -{ - public InMemoryJobQueueTests(ITestOutputHelper output) : base(output) { } - - protected override IQueue GetSampleWorkItemQueue(int retries, TimeSpan retryDelay) - { - return new InMemoryQueue(o => o.RetryDelay(retryDelay).Retries(retries).LoggerFactory(Log)); - } - - [Fact] - public override Task ActivityWillFlowThroughQueueJobAsync() - { - return base.ActivityWillFlowThroughQueueJobAsync(); - } - - [Fact] - public override Task CanRunMultipleQueueJobsAsync() - { - return base.CanRunMultipleQueueJobsAsync(); - } - - [Fact] - public override Task CanRunQueueJobAsync() - { - return base.CanRunQueueJobAsync(); - } - - [Fact] - public override Task CanRunQueueJobWithLockFailAsync() - { - return base.CanRunQueueJobWithLockFailAsync(); - } - - [Fact] - public override Task GetQueueEntryLockAsync_WhenLockThrows_AbandonsQueueEntry() - { - return base.GetQueueEntryLockAsync_WhenLockThrows_AbandonsQueueEntry(); - } -} diff --git a/tests/Foundatio.Tests/Jobs/InMemoryJobRuntimeStoreTests.cs b/tests/Foundatio.Tests/Jobs/InMemoryJobRuntimeStoreTests.cs new file mode 100644 index 000000000..33101b159 --- /dev/null +++ b/tests/Foundatio.Tests/Jobs/InMemoryJobRuntimeStoreTests.cs @@ -0,0 +1,14 @@ +using System; +using Foundatio.Jobs; +using Xunit; + +namespace Foundatio.Tests.Jobs; + +// Inherits every [Fact] from JobRuntimeStoreConformanceTests, so a new conformance check automatically runs here with +// no per-test override to forget. +public class InMemoryJobRuntimeStoreTests : JobRuntimeStoreConformanceTests +{ + public InMemoryJobRuntimeStoreTests(ITestOutputHelper output) : base(output) { } + + protected override IJobRuntimeStore CreateStore(TimeProvider timeProvider, JobRuntimeStoreOptions? options = null) => new InMemoryJobRuntimeStore(options ?? new(), timeProvider); +} diff --git a/tests/Foundatio.Tests/Jobs/JobPolicyTests.cs b/tests/Foundatio.Tests/Jobs/JobPolicyTests.cs new file mode 100644 index 000000000..496687a0a --- /dev/null +++ b/tests/Foundatio.Tests/Jobs/JobPolicyTests.cs @@ -0,0 +1,70 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Microsoft.Extensions.Time.Testing; +using Xunit; + +namespace Foundatio.Tests.Jobs; + +public class JobPolicyTests +{ + [Fact] + public async Task HistoryPressure_PreservesAdmissionAndIdempotency() + { + var token = TestContext.Current.CancellationToken; + var time = new FakeTimeProvider(); + var store = new InMemoryJobRuntimeStore(new JobRuntimeStoreOptions { MaxActiveJobs = 1, MaxHistoryJobs = 1, MaxDeduplicationRecords = 10 }, time); + for (int i = 0; i < 3; i++) + { + await store.CreateIfAbsentAsync(new JobState { JobId = $"job-{i}", Name = "work", JobType = "work" }, token); + var claim = await store.ClaimNextAsync(new JobClaimRequest { JobTypes = ["work"], NodeId = "node" }, token); + Assert.NotNull(claim); + Assert.True(await store.CompleteJobAsync(claim.JobId, claim.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Succeeded, Message = "Done" }, token)); + } + await store.CleanupAsync(cancellationToken: token); + Assert.Single((await store.QueryAsync(new JobQuery(), token))); + await store.CreateIfAbsentAsync(new JobState { JobId = "job-0", Name = "work", JobType = "work" }, token); + Assert.Null(await store.ClaimNextAsync(new JobClaimRequest { JobTypes = ["work"], NodeId = "node" }, token)); + var stats = await store.GetStatsAsync(token); + Assert.Equal(0, stats.ActiveJobs); + Assert.Equal(1, stats.HistoryJobs); + Assert.Equal(3, stats.DeduplicationRecords); + } + + [Fact] + public async Task EnqueueAsync_Delay_DefersClaimAndWaitCancellationDoesNotCancelJob() + { + var token = TestContext.Current.CancellationToken; + var time = new FakeTimeProvider(); + var store = new InMemoryJobRuntimeStore(time); + var client = new JobClient(store, time, new JobTypeRegistry([new("work", typeof(Work))])); + var handle = await client.EnqueueAsync(new JobRequestOptions { Delay = TimeSpan.FromMinutes(5) }, token); + var request = new JobClaimRequest { JobTypes = ["work"], NodeId = "node" }; + Assert.Null(await store.ClaimNextAsync(request, token)); + using var cancelWait = CancellationTokenSource.CreateLinkedTokenSource(token); + await cancelWait.CancelAsync(); + await Assert.ThrowsAnyAsync(() => handle.WaitForCompletionAsync(cancellationToken: cancelWait.Token)); + Assert.False((await handle.GetStateAsync(token))!.CancellationRequested); + time.Advance(TimeSpan.FromMinutes(5)); + var claim = Assert.IsType(await store.ClaimNextAsync(request, token)); + await store.CompleteJobAsync(claim.JobId, claim.ClaimToken!, new JobCompletion { Kind = JobCompletionKind.Succeeded, Message = "Done" }, token); + var final = await handle.WaitForCompletionAsync(cancellationToken: token); + Assert.Null(final.Error); + Assert.Equal("Done", final.ResultMessage); + } + + [Fact] + public async Task EnqueueAsync_ConflictingDelayAndRunAt_FailsBeforePersistence() + { + var store = new InMemoryJobRuntimeStore(); + var client = new JobClient(store); + await Assert.ThrowsAsync(() => client.EnqueueAsync(new JobRequestOptions { Delay = TimeSpan.FromSeconds(1), RunAt = DateTimeOffset.UtcNow }, TestContext.Current.CancellationToken)); + Assert.Empty((await store.QueryAsync(new JobQuery(), TestContext.Current.CancellationToken))); + } + + public sealed class Work : IJob + { + public Task RunAsync(JobExecutionContext context) => Task.FromResult(JobResult.Success); + } +} diff --git a/tests/Foundatio.Tests/Jobs/JobRuntimeTests.cs b/tests/Foundatio.Tests/Jobs/JobRuntimeTests.cs new file mode 100644 index 000000000..5162f8b1c --- /dev/null +++ b/tests/Foundatio.Tests/Jobs/JobRuntimeTests.cs @@ -0,0 +1,529 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace Foundatio.Tests.Jobs; + +public class JobRuntimeTests +{ + private static JobTypeRegistry CreateJobRegistry() => new(typeof(JobRuntimeTests).GetNestedTypes(System.Reflection.BindingFlags.NonPublic) + .Where(t => t.IsClass && !t.IsAbstract && typeof(IJob).IsAssignableFrom(t)) + .Select(t => new JobTypeRegistration(t.FullName!, t))); + + [Fact] + public async Task CreateIfAbsentAsync_AtCapacity_PreservesExistingWorkAndRejectsNewWorkAsync() + { + var token = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(maxJobs: 1); + var state = new JobState { JobId = "one", Name = "work" }; + await store.CreateIfAbsentAsync(state, token); + await store.CreateIfAbsentAsync(state, token); + await Assert.ThrowsAsync(() => store.CreateIfAbsentAsync(state with { JobId = "two" }, token)); + Assert.Equal("one", Assert.Single(await store.QueryAsync(new JobQuery(), token)).JobId); + } + + [Fact] + public async Task RequestCancellationAsync_BeforeClaim_PreventsExecutionAsync() + { + var token = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var probe = new JobRuntimeProbe(); + await using var provider = new ServiceCollection().AddSingleton(probe).BuildServiceProvider(); + var client = new JobClient(store); + var handle = await client.EnqueueAsync(cancellationToken: token); + await handle.RequestCancellationAsync(token); + var worker = new JobWorker(store, provider, new JobWorkerOptions { JobTypes = CreateJobRegistry() }); + Assert.False(await worker.RunAsync(handle.JobId, token)); + var state = await handle.GetStateAsync(token); + Assert.Equal(JobStatus.Cancelled, state!.Status); + Assert.Equal(0, state.Attempt); + } + + [Fact] + public async Task RunAsync_HostStops_LeavesUnfinishedWorkQueuedAsync() + { + var token = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var started = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var provider = new ServiceCollection().AddSingleton(started).BuildServiceProvider(); + var client = new JobClient(store); + var handle = await client.EnqueueAsync(cancellationToken: token); + var worker = new JobWorker(store, provider, new JobWorkerOptions { JobTypes = CreateJobRegistry() }); + using var shutdown = CancellationTokenSource.CreateLinkedTokenSource(token); + var run = worker.RunAsync(handle.JobId, shutdown.Token); + await started.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + await shutdown.CancelAsync(); + await run.WaitAsync(TimeSpan.FromSeconds(5), token); + var state = await handle.GetStateAsync(token); + Assert.Equal(JobStatus.Queued, state!.Status); + Assert.Null(state.CompletedUtc); + Assert.False(state.CancellationRequested); + } + + private sealed class InterruptedJob(TaskCompletionSource started) : IJob + { + public async Task RunAsync(JobExecutionContext context) + { + started.TrySetResult(); + await Task.Delay(Timeout.InfiniteTimeSpan, context.CancellationToken); + return JobResult.Success; + } + } + + [Fact] + public async Task EnqueueAsync_TypedJobWithoutArguments_RejectsBeforePersistingAsync() + { + var token = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var client = new JobClient(store); + await Assert.ThrowsAsync(() => client.EnqueueAsync(cancellationToken: token)); + Assert.Empty(await store.QueryAsync(new JobQuery(), token)); + } + + [Fact] + public async Task RunAsync_WithExecutionContext_ReportsProgressAndIdentityAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + await using var serviceProvider = new ServiceCollection().BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "ctx-node", JobTypes = CreateJobRegistry() }); + + await store.CreateIfAbsentAsync(new JobState + { + JobId = "ctx-job", + Name = "ctx", + JobType = typeof(ProgressJob).FullName, + Status = JobStatus.Queued + }, cancellationToken); + + Assert.True(await worker.RunAsync("ctx-job", cancellationToken)); + + var state = await store.GetAsync("ctx-job", cancellationToken); + Assert.Equal(JobStatus.Completed, state!.Status); + Assert.Equal(100, state.Progress); // a completed job is 100%; the worker sets this on success + // The job wrote its context identity + attempt into the progress message (preserved through completion), + // proving the store-backed context is wired through to job code. + Assert.Equal("ctx-job:1", state.ProgressMessage); + } + + [Fact] + public async Task RunQueuedAsync_RecoversExpiredAdHocAndScheduledJobsAsync() + { + var token = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var probe = new JobRuntimeProbe(); + await using var provider = new ServiceCollection().AddSingleton(probe).BuildServiceProvider(); + using var worker = new JobWorker(store, provider, new JobWorkerOptions { JobTypes = CreateJobRegistry() }); + var expired = DateTimeOffset.UtcNow.AddMinutes(-5); + foreach (var id in new[] { "ad-hoc", "scheduled", "exhausted", "healthy" }) + { + await store.CreateIfAbsentAsync(new JobState + { + JobId = id, + Name = "recovery", + JobType = typeof(SuccessfulTrackedJob).FullName, + Status = JobStatus.Processing, + NodeId = "previous-worker", + ClaimToken = "previous-claim", + LeaseExpiresUtc = id == "healthy" ? DateTimeOffset.UtcNow.AddMinutes(5) : expired, + Attempt = id == "exhausted" ? 3 : 1, + ScheduledForUtc = id == "scheduled" ? expired : null + }, token); + } + + Assert.Equal(2, await worker.RunQueuedAsync(cancellationToken: token)); + Assert.Equal(2, probe.RunCount); + Assert.Equal(JobStatus.Completed, (await store.GetAsync("ad-hoc", token))!.Status); + Assert.Equal(JobStatus.Completed, (await store.GetAsync("scheduled", token))!.Status); + Assert.Equal(JobStatus.Failed, (await store.GetAsync("exhausted", token))!.Status); + Assert.Equal(JobStatus.Processing, (await store.GetAsync("healthy", token))!.Status); + } + + [Fact] + public async Task CreateIfAbsentAsync_WithExistingJob_DoesNotOverwriteStateAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + + await store.CreateIfAbsentAsync(new JobState + { + JobId = "job-1", + Name = "first", + Status = JobStatus.Queued + }, cancellationToken); + + await store.CreateIfAbsentAsync(new JobState + { + JobId = "job-1", + Name = "second", + Status = JobStatus.Failed, + Error = "should not overwrite" + }, cancellationToken); + + var state = await store.GetAsync("job-1", cancellationToken); + + Assert.NotNull(state); + Assert.Equal("first", state.Name); + Assert.Equal(JobStatus.Queued, state.Status); + Assert.Null(state.Error); + } + + [Fact] + public async Task ClaimDueDispatchesAsync_ClaimsReleasesAndCompletesDueDispatchesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var now = DateTimeOffset.UtcNow; + + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = "dispatch-1", + Kind = ScheduledDispatchKind.QueueMessage, + Destination = DestinationAddress.ForQueue("work"), + Body = "hello"u8.ToArray(), + DueUtc = now.AddSeconds(-1) + }, cancellationToken); + + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = "dispatch-2", + Kind = ScheduledDispatchKind.QueueMessage, + Destination = DestinationAddress.ForQueue("work"), + Body = "later"u8.ToArray(), + DueUtc = now.AddHours(1) + }, cancellationToken); + + var claimed = await store.ClaimDueDispatchesAsync(now, 10, "node-a", TimeSpan.FromMinutes(1), cancellationToken); + + var dispatch = Assert.Single(claimed); + Assert.Equal("dispatch-1", dispatch.DispatchId); + Assert.Equal("node-a", dispatch.ClaimOwner); + Assert.Equal(1, dispatch.Attempts); + + var claimedAgain = await store.ClaimDueDispatchesAsync(now, 10, "node-b", TimeSpan.FromMinutes(1), cancellationToken); + Assert.Empty(claimedAgain); + + await store.ReleaseDispatchAsync("dispatch-1", "node-a", now.AddSeconds(-1), cancellationToken); + + var reclaimed = await store.ClaimDueDispatchesAsync(now, 10, "node-b", TimeSpan.FromMinutes(1), cancellationToken); + dispatch = Assert.Single(reclaimed); + Assert.Equal("node-b", dispatch.ClaimOwner); + Assert.Equal(2, dispatch.Attempts); + + await store.CompleteDispatchAsync("dispatch-1", "node-b", cancellationToken); + + var afterComplete = await store.ClaimDueDispatchesAsync(now, 10, "node-c", TimeSpan.FromMinutes(1), cancellationToken); + Assert.Empty(afterComplete); + } + + [Fact] + public async Task RunAsync_WhenJobSucceeds_TracksCompletedStateAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var probe = new JobRuntimeProbe(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "node-a", JobTypes = CreateJobRegistry() }); + + JobHandle handle = await client.EnqueueAsync(new JobRequestOptions { JobId = "job-1" }, cancellationToken); + Assert.True(await worker.RunAsync(handle.JobId, cancellationToken)); + + var state = await handle.GetStateAsync(cancellationToken); + Assert.NotNull(state); + Assert.Equal(1, probe.RunCount); + Assert.Equal(JobStatus.Completed, state.Status); + Assert.Equal(1, state.Attempt); + Assert.Equal(100, state.Progress); + Assert.NotNull(state.StartedUtc); + Assert.NotNull(state.CompletedUtc); + Assert.Null(state.NodeId); + Assert.Null(state.LeaseExpiresUtc); + } + + [Fact] + public async Task EnqueueAsync_WithRegisteredJobType_PersistsStableNameAndWorkerResolvesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var probe = new JobRuntimeProbe(); + var registry = new JobTypeRegistry([new JobTypeRegistration("search.rebuild", typeof(SuccessfulTrackedJob))]); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var client = new JobClient(store, jobTypes: registry); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "node-a", JobTypes = registry }); + + JobHandle handle = await client.EnqueueAsync(new JobRequestOptions { JobId = "job-registered" }, cancellationToken); + var queued = await handle.GetStateAsync(cancellationToken); + + Assert.NotNull(queued); + Assert.Equal("search.rebuild", queued.JobType); + Assert.DoesNotContain(",", queued.JobType); + Assert.True(await worker.RunAsync(handle.JobId, cancellationToken)); + + var completed = await handle.GetStateAsync(cancellationToken); + Assert.NotNull(completed); + Assert.Equal(JobStatus.Completed, completed.Status); + Assert.Equal(1, probe.RunCount); + } + + [Fact] + public async Task RequestCancellationAsync_WhenJobIsRunning_CancelsAndTracksStateAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var probe = new JobRuntimeProbe(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "node-a", JobTypes = CreateJobRegistry() }); + + JobHandle handle = await client.EnqueueAsync(new JobRequestOptions { JobId = "job-1" }, cancellationToken); + var runTask = worker.RunAsync(handle.JobId, cancellationToken); + await probe.Started.Task.WaitAsync(TimeSpan.FromSeconds(5), cancellationToken); + + Assert.True(await handle.RequestCancellationAsync(cancellationToken)); + + await probe.Cancelled.Task.WaitAsync(TimeSpan.FromSeconds(5), cancellationToken); + Assert.True(await runTask.WaitAsync(TimeSpan.FromSeconds(5), cancellationToken)); + var state = await handle.GetStateAsync(cancellationToken); + + Assert.NotNull(state); + Assert.Equal(JobStatus.Cancelled, state.Status); + Assert.True(state.CancellationRequested); + Assert.NotNull(state.CompletedUtc); + Assert.Null(state.NodeId); + Assert.Null(state.LeaseExpiresUtc); + } + + private sealed class JobRuntimeProbe + { + private int _runCount; + + public TaskCompletionSource Started { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public TaskCompletionSource Cancelled { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + + public int RunCount => Volatile.Read(ref _runCount); + public string? LastMessage { get; private set; } + + public void RecordRun(string? message = null) + { + Interlocked.Increment(ref _runCount); + LastMessage = message; + } + } + + [Fact] + public async Task EnqueueAsync_WithTypedArguments_JobReceivesDeserializedPayloadAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var probe = new JobRuntimeProbe(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "node-a", JobTypes = CreateJobRegistry() }); + + var handle = await client.EnqueueAsync(new ResizeArgs { Path = "/img/1.png", Width = 640 }, cancellationToken: cancellationToken); + + // The payload and its discriminator are durable state, not in-process context. + var state = await store.GetAsync(handle.JobId, cancellationToken); + Assert.NotNull(state?.Payload); + Assert.Equal(typeof(ResizeArgs).FullName, state.PayloadType); + + Assert.True(await worker.RunAsync(handle.JobId, cancellationToken)); + Assert.Equal("/img/1.png:640", probe.LastMessage); + Assert.Equal(JobStatus.Completed, (await handle.GetStateAsync(cancellationToken))!.Status); + } + + [Fact] + public async Task RunJob_ResolvesScopedServicesPerExecutionAndDisposesThemAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var tracker = new ScopedLifetimeTracker(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(tracker) + .AddScoped() + .AddTransient() + .BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "node-a", JobTypes = CreateJobRegistry() }); + + var first = await client.EnqueueAsync(cancellationToken: cancellationToken); + var second = await client.EnqueueAsync(cancellationToken: cancellationToken); + Assert.True(await worker.RunAsync(first.JobId, cancellationToken)); + Assert.True(await worker.RunAsync(second.JobId, cancellationToken)); + + // Two runs -> two scoped instances (not one root-container singleton), each disposed when its run ended. + Assert.Equal(2, tracker.Created); + Assert.Equal(2, tracker.Disposed); + } + + [Fact] + public async Task RunQueuedAsync_WithMaxConcurrency_RespectsCapAndRunsInParallelAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var gauge = new ConcurrencyGauge(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(gauge) + .BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "node-a", MaxConcurrency = 2, JobTypes = CreateJobRegistry() }); + + for (int i = 0; i < 6; i++) + await client.EnqueueAsync(cancellationToken: cancellationToken); + + Assert.Equal(6, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + Assert.True(gauge.MaxObserved <= 2, $"expected at most 2 in-flight jobs, observed {gauge.MaxObserved}"); + Assert.True(gauge.MaxObserved > 1, "expected the pool to actually run jobs in parallel"); + } + + private sealed class ScopedLifetimeTracker + { + private int _created; + private int _disposed; + public int Created => Volatile.Read(ref _created); + public int Disposed => Volatile.Read(ref _disposed); + public void RecordCreated() => Interlocked.Increment(ref _created); + public void RecordDisposed() => Interlocked.Increment(ref _disposed); + } + + private sealed class ScopedDependency : IDisposable + { + private readonly ScopedLifetimeTracker _tracker; + + public ScopedDependency(ScopedLifetimeTracker tracker) + { + _tracker = tracker; + _tracker.RecordCreated(); + } + + public void Dispose() => _tracker.RecordDisposed(); + } + + private sealed class ScopedConsumingJob : IJob + { + // The dependency's usefulness is its lifetime tracking; resolving it is the test. + public ScopedConsumingJob(ScopedDependency dependency) => _ = dependency; + + public Task RunAsync(JobExecutionContext context) => Task.FromResult(JobResult.Success); + } + + private sealed class ConcurrencyGauge + { + private int _inFlight; + private int _maxObserved; + + public int MaxObserved => Volatile.Read(ref _maxObserved); + + public async Task TrackAsync() + { + int current = Interlocked.Increment(ref _inFlight); + int max; + while (current > (max = Volatile.Read(ref _maxObserved))) + Interlocked.CompareExchange(ref _maxObserved, current, max); + + await Task.Delay(TimeSpan.FromMilliseconds(100)); + Interlocked.Decrement(ref _inFlight); + } + } + + private sealed class ConcurrencyProbeJob : IJob + { + private readonly ConcurrencyGauge _gauge; + + public ConcurrencyProbeJob(ConcurrencyGauge gauge) => _gauge = gauge; + + public async Task RunAsync(JobExecutionContext context) + { + await _gauge.TrackAsync(); + return JobResult.Success; + } + } + + private sealed class ResizeArgs + { + public string? Path { get; set; } + public int Width { get; set; } + } + + private sealed class ArgsConsumingJob : IJob + { + private readonly JobRuntimeProbe _probe; + + public ArgsConsumingJob(JobRuntimeProbe probe) + { + _probe = probe; + } + + public Task RunAsync(ResizeArgs args, JobExecutionContext context) + { + _probe.RecordRun($"{args.Path}:{args.Width}"); + return Task.FromResult(JobResult.Success); + } + } + + private sealed class SuccessfulTrackedJob : IJob + { + private readonly JobRuntimeProbe _probe; + + public SuccessfulTrackedJob(JobRuntimeProbe probe) + { + _probe = probe; + } + + public Task RunAsync(JobExecutionContext context) + { + context.CancellationToken.ThrowIfCancellationRequested(); + _probe.RecordRun(); + return Task.FromResult(JobResult.Success); + } + } + + private sealed class CancellableTrackedJob : IJob + { + private readonly JobRuntimeProbe _probe; + + public CancellableTrackedJob(JobRuntimeProbe probe) + { + _probe = probe; + } + + public async Task RunAsync(JobExecutionContext context) + { + _probe.Started.TrySetResult(); + + try + { + await Task.Delay(TimeSpan.FromMinutes(1), context.CancellationToken); + return JobResult.Success; + } + catch (OperationCanceledException) + { + _probe.Cancelled.TrySetResult(); + throw; + } + } + } + + private sealed class ProgressJob : IJob + { + public async Task RunAsync(JobExecutionContext context) + { + await context.ReportProgressAsync(75, $"{context.JobId}:{context.Attempt}", context.CancellationToken); + return JobResult.Success; + } + } +} diff --git a/tests/Foundatio.Tests/Jobs/JobSchedulerTests.cs b/tests/Foundatio.Tests/Jobs/JobSchedulerTests.cs new file mode 100644 index 000000000..d4f5133f2 --- /dev/null +++ b/tests/Foundatio.Tests/Jobs/JobSchedulerTests.cs @@ -0,0 +1,417 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Extensions.Hosting.Jobs; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Time.Testing; +using Xunit; + +namespace Foundatio.Tests.Jobs; + +public class JobSchedulerTests +{ + private static JobTypeRegistry CreateJobRegistry() => new(typeof(JobSchedulerTests).GetNestedTypes(System.Reflection.BindingFlags.NonPublic) + .Where(t => t.IsClass && !t.IsAbstract && typeof(IJob).IsAssignableFrom(t)) + .Select(t => new JobTypeRegistration(t.FullName!, t))); + + [Fact] + public async Task EnqueueDueOccurrencesAsync_WhenOccurrenceIsDue_CreatesSingleGlobalOccurrenceAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryScheduledJobStore(); + var store = new InMemoryJobRuntimeStore(); + var processor = CreateProcessor(scheduler, store, "node-a"); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "nightly", + Cron = "* * * * *", + JobType = typeof(ScheduledProbeJob).FullName! + }, cancellationToken); + + var first = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + var second = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + + var dispatch = Assert.Single(first); + Assert.Empty(second); + Assert.Equal("nightly:20260101000000:global", dispatch.JobId); + Assert.Equal("nightly", dispatch.Name); + Assert.Empty(await store.ClaimDueDispatchesAsync(now, 100, "other-node", TimeSpan.FromMinutes(1), cancellationToken)); + + var state = await store.GetAsync(dispatch.JobId!, cancellationToken); + Assert.NotNull(state); + Assert.Equal(JobStatus.Queued, state.Status); + Assert.Equal(new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero), state.ScheduledForUtc); + } + + [Fact] + public async Task EnqueueDueOccurrencesAsync_WithAllowConcurrent_MaterializesEveryMissedOccurrenceAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryScheduledJobStore(); + var store = new InMemoryJobRuntimeStore(); + var processor = CreateProcessor(scheduler, store, "node-a"); + var now = new DateTimeOffset(2026, 1, 1, 0, 5, 30, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "frequent", + Cron = "* * * * *", + JobType = typeof(ScheduledProbeJob).FullName!, + Overlap = OverlapPolicy.AllowConcurrent, + MisfireWindow = TimeSpan.FromMinutes(10) + }, cancellationToken); + + // A scheduler that lagged behind a per-minute cadence must materialize every missed occurrence in the window, + // not just the most recent one. + var first = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + Assert.True(first.Count >= 5, $"Expected multiple missed occurrences, got {first.Count}"); + + // Deterministic occurrence ids dedupe across overlapping windows: a second pass at the same time adds nothing. + var second = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + Assert.Empty(second); + } + + [Fact] + public async Task RunDueOccurrencesAsync_WhenOccurrenceIsDue_RunsConfiguredJobAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryScheduledJobStore(); + var store = new InMemoryJobRuntimeStore(); + var probe = new JobSchedulerProbe(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "node-a", JobTypes = CreateJobRegistry() }); + var processor = new JobScheduleProcessor(scheduler, store, new JobScheduleProcessorOptions { NodeId = "node-a" }); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "nightly", + Cron = "* * * * *", + JobType = typeof(ScheduledProbeJob).FullName! + }, cancellationToken); + var scheduled = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + + int completed = await worker.RunQueuedAsync(cancellationToken: cancellationToken); + + var dispatch = Assert.Single(scheduled); + var state = await store.GetAsync(dispatch.JobId!, cancellationToken); + Assert.Equal(1, completed); + Assert.Equal(1, probe.RunCount); + Assert.NotNull(state); + Assert.Equal(JobStatus.Completed, state.Status); + Assert.Equal(1, state.Attempt); + Assert.Equal(100, state.Progress); + } + + [Fact] + public async Task EnqueueDueOccurrencesAsync_WithPerNodeScope_CreatesOccurrencePerNodeAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryScheduledJobStore(); + var store = new InMemoryJobRuntimeStore(); + var nodeA = CreateProcessor(scheduler, store, "node-a"); + var nodeB = CreateProcessor(scheduler, store, "node-b"); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "per-node", + Cron = "* * * * *", + JobType = typeof(ScheduledProbeJob).FullName!, + Scope = ScheduledJobScope.PerNode + }, cancellationToken); + + var first = await nodeA.EnqueueDueOccurrencesAsync(now, cancellationToken); + var second = await nodeB.EnqueueDueOccurrencesAsync(now, cancellationToken); + + Assert.Equal("per-node:20260101000000:node-a", Assert.Single(first).JobId); + Assert.Equal("per-node:20260101000000:node-b", Assert.Single(second).JobId); + + var states = await store.QueryAsync(new JobQuery { Name = "per-node" }, cancellationToken); + Assert.Equal(2, states.Count); + } + + [Fact] + public async Task EnqueueDueOccurrencesAsync_WithMisfireWindow_CatchesRecentMissedOccurrenceAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryScheduledJobStore(); + var store = new InMemoryJobRuntimeStore(); + var processor = CreateProcessor(scheduler, store, "node-a"); + var now = new DateTimeOffset(2026, 1, 1, 0, 5, 0, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "daily", + Cron = "0 0 * * *", + JobType = typeof(ScheduledProbeJob).FullName!, + MisfireWindow = TimeSpan.FromMinutes(10) + }, cancellationToken); + + var scheduled = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + + var dispatch = Assert.Single(scheduled); + var state = await store.GetAsync(dispatch.JobId!, cancellationToken); + Assert.NotNull(state); + Assert.Equal(new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero), state.ScheduledForUtc); + } + + [Fact] + public async Task RunDueOccurrencesAsync_WhenDispatchIsQueueMessage_MaterializesItAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryScheduledJobStore(); + var store = new InMemoryJobRuntimeStore(); + await using var transport = new InMemoryMessageTransport(); + var dispatcher = new ScheduledMessageDispatcher(store, transport); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero); + + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = "delayed-message", + Kind = ScheduledDispatchKind.QueueMessage, + Destination = DestinationAddress.ForQueue("work"), + Body = "hello"u8.ToArray(), + DueUtc = now + }, cancellationToken); + + int completed = await dispatcher.DispatchDueAsync(now, cancellationToken: cancellationToken); + + Assert.Equal(1, completed); + var pull = Assert.IsAssignableFrom(transport); + var entries = await pull.ReceiveAsync(DestinationAddress.ForQueue("work"), new ReceiveRequest { MaxMessages = 1, MaxWaitTime = TimeSpan.FromMilliseconds(50) }, cancellationToken); + var entry = Assert.Single(entries); + Assert.Equal("delayed-message", entry.ApplicationMessageId); + Assert.Equal("hello"u8.ToArray(), entry.Body.ToArray()); + } + + [Fact] + public async Task RunDueOccurrencesAsync_WhenJobFails_RetriesThenDeadLettersAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryScheduledJobStore(); + var time = new FakeTimeProvider(new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero)); + var store = new InMemoryJobRuntimeStore(time); + var probe = new JobSchedulerProbe(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "node-a", JobTypes = CreateJobRegistry(), TimeProvider = time }); + var processor = new JobScheduleProcessor(scheduler, store, new JobScheduleProcessorOptions { NodeId = "node-a" }); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "nightly", + Cron = "* * * * *", + JobType = typeof(FailingScheduledJob).FullName!, + MaxAttempts = 2 + }, cancellationToken); + var scheduled = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + var dispatch = Assert.Single(scheduled); + + Assert.Equal(1, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + var retried = await store.GetAsync(dispatch.JobId!, cancellationToken); + Assert.NotNull(retried); + Assert.Equal(JobStatus.Queued, retried.Status); + Assert.Equal(1, retried.Attempt); + + time.Advance(TimeSpan.FromMinutes(2)); + Assert.Equal(1, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + var deadlettered = await store.GetAsync(dispatch.JobId!, cancellationToken); + Assert.NotNull(deadlettered); + Assert.Equal(JobStatus.Failed, deadlettered.Status); + Assert.Equal(2, deadlettered.Attempt); + Assert.Equal(2, probe.RunCount); + } + + [Fact] + public async Task RunDueOccurrencesAsync_WhenProcessingLeaseExpired_ReclaimsAndRunsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryScheduledJobStore(); + var store = new InMemoryJobRuntimeStore(); + var probe = new JobSchedulerProbe(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "node-a", JobTypes = CreateJobRegistry() }); + var processor = new JobScheduleProcessor(scheduler, store, new JobScheduleProcessorOptions { NodeId = "node-a" }); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + const string jobId = "nightly:20260101000000:global"; + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "nightly", + Cron = "* * * * *", + JobType = typeof(ScheduledProbeJob).FullName!, + MaxAttempts = 2 + }, cancellationToken); + await store.CreateIfAbsentAsync(new JobState + { + JobId = jobId, + Name = "nightly", + Status = JobStatus.Processing, + JobType = typeof(ScheduledProbeJob).FullName, + MaxAttempts = 2, + Attempt = 1, + NodeId = "node-b", + LeaseExpiresUtc = now.AddMinutes(-1), + ScheduledForUtc = now.AddSeconds(-30) + }, cancellationToken); + + Assert.Equal(1, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + + var state = await store.GetAsync(jobId, cancellationToken); + Assert.NotNull(state); + Assert.Equal(JobStatus.Completed, state.Status); + Assert.Equal(2, state.Attempt); + Assert.Equal(1, probe.RunCount); + } + + [Fact] + public async Task RunQueuedAsync_ClaimsScheduledOccurrencesThroughTheSameWorkerAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var probe = new JobSchedulerProbe(); + await using var serviceProvider = new ServiceCollection().AddSingleton(probe).BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "node-a", JobTypes = CreateJobRegistry() }); + + // A CRON occurrence sitting in Queued (the scheduler transitioned it Scheduled->Queued) must NOT be claimed by + // the generic worker — only the scheduler runs occurrences, with its own retry/dead-letter accounting. + await store.CreateIfAbsentAsync(new JobState + { + JobId = "nightly:20260101000000:global", + Name = "nightly", + JobType = typeof(ScheduledProbeJob).FullName, + Status = JobStatus.Queued, + ScheduledForUtc = new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero) + }, cancellationToken); + + Assert.Equal(1, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + Assert.Equal(1, probe.RunCount); + Assert.Equal(JobStatus.Completed, (await store.GetAsync("nightly:20260101000000:global", cancellationToken))!.Status); + } + + [Fact] + public async Task EnqueueDueOccurrencesAsync_PerNodeScope_WithDelimiterInNodeId_DoesNotCrossMatchAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryScheduledJobStore(); + var store = new InMemoryJobRuntimeStore(); + // Node ids that are suffix-confusable under a naive EndsWith(":{scope}") check — the default NodeIdentity contains ':'. + var nodeXB = CreateProcessor(scheduler, store, "x:b"); + var nodeB = CreateProcessor(scheduler, store, "b"); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "per-node", + Cron = "* * * * *", + JobType = typeof(ScheduledProbeJob).FullName!, + Scope = ScheduledJobScope.PerNode // default Overlap = SkipIfRunning, which runs the active-occurrence check + }, cancellationToken); + + Assert.Single(await nodeXB.EnqueueDueOccurrencesAsync(now, cancellationToken)); // creates "per-node:...:x:b" + // node "b" must still materialize its own occurrence; node "x:b"'s occurrence must not be mistaken for node "b"'s. + Assert.Single(await nodeB.EnqueueDueOccurrencesAsync(now, cancellationToken)); // creates "per-node:...:b" + + var states = await store.QueryAsync(new JobQuery { Name = "per-node", Limit = 100 }, cancellationToken); + Assert.Equal(2, states.Count); + } + + [Fact] + public async Task UseRuntimeStore_RegistersClientsWithoutStartingWorkersAsync() + { + var services = new ServiceCollection(); + services.AddFoundatio().Jobs.UseInMemory(); + await using var provider = services.BuildServiceProvider(); + Assert.Empty(provider.GetServices()); + Assert.NotNull(provider.GetRequiredService()); + } + + [Fact] + public async Task AddJobWorker_ExplicitlyRunsQueuedJobsAndRegistersOnceAsync() + { + var token = TestContext.Current.CancellationToken; + var probe = new JobSchedulerProbe(); + var services = new ServiceCollection().AddLogging().AddSingleton(probe); + services.AddJobWorker(); + services.AddFoundatio().Jobs.UseInMemory().AddJobType("probe"); + services.AddJobWorker(); + await using var provider = services.BuildServiceProvider(); + var hosted = Assert.Single(provider.GetServices()); + await hosted.StartAsync(token); + try + { + var handle = await provider.GetRequiredService().EnqueueAsync(cancellationToken: token); + JobState? state = null; + for (int i = 0; i < 100 && (state = await handle.GetStateAsync(token))?.Status != JobStatus.Completed; i++) + await Task.Delay(50, token); + Assert.Equal(JobStatus.Completed, state?.Status); + Assert.Equal(1, probe.RunCount); + } + finally + { + await hosted.StopAsync(token); + } + } + + private static JobScheduleProcessor CreateProcessor(IScheduledJobStore scheduler, IJobRuntimeStore store, string nodeId) + => new(scheduler, store, new JobScheduleProcessorOptions { NodeId = nodeId }); + + private sealed class JobSchedulerProbe + { + private int _runCount; + + public int RunCount => Volatile.Read(ref _runCount); + + public void RecordRun() + { + Interlocked.Increment(ref _runCount); + } + } + + private sealed class FailingScheduledJob : IJob + { + private readonly JobSchedulerProbe _probe; + + public FailingScheduledJob(JobSchedulerProbe probe) + { + _probe = probe; + } + + public Task RunAsync(JobExecutionContext context) + { + context.CancellationToken.ThrowIfCancellationRequested(); + _probe.RecordRun(); + return Task.FromResult(JobResult.FromException(new InvalidOperationException("failed"))); + } + } + + private sealed class ScheduledProbeJob : IJob + { + private readonly JobSchedulerProbe _probe; + + public ScheduledProbeJob(JobSchedulerProbe probe) + { + _probe = probe; + } + + public Task RunAsync(JobExecutionContext context) + { + context.CancellationToken.ThrowIfCancellationRequested(); + _probe.RecordRun(); + return Task.FromResult(JobResult.Success); + } + } +} diff --git a/tests/Foundatio.Tests/Jobs/JobTests.cs b/tests/Foundatio.Tests/Jobs/JobTests.cs deleted file mode 100644 index 93e12c694..000000000 --- a/tests/Foundatio.Tests/Jobs/JobTests.cs +++ /dev/null @@ -1,234 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Diagnostics; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Caching; -using Foundatio.Jobs; -using Foundatio.Xunit; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Time.Testing; -using Xunit; - -namespace Foundatio.Tests.Jobs; - -public class JobTests : TestWithLoggingBase -{ - public JobTests(ITestOutputHelper output) : base(output) - { - } - - [Fact] - public async Task CanCancelJob() - { - var job = new HelloWorldJob(null, Log); - var sp = new ServiceCollection().BuildServiceProvider(); - using var timeoutCancellationTokenSource = new CancellationTokenSource(1000); - var resultTask = new JobRunner(job, sp, Log).RunAsync(timeoutCancellationTokenSource.Token); - await TimeProvider.System.Delay(TimeSpan.FromSeconds(2), TestCancellationToken); - Assert.True(await resultTask); - } - - [Fact] - public async Task CanStopLongRunningJob() - { - var job = new LongRunningJob(null, Log); - var sp = new ServiceCollection().BuildServiceProvider(); - var runner = new JobRunner(job, sp, Log); - using var cts = new CancellationTokenSource(1000); - bool result = await runner.RunAsync(cts.Token); - - Assert.True(result); - } - - [Fact] - public async Task CanStopLongRunningCronJob() - { - var job = new LongRunningJob(null, Log); - var sp = new ServiceCollection().BuildServiceProvider(); - var runner = new JobRunner(job, sp, Log); - using var cts = new CancellationTokenSource(1000); - bool result = await runner.RunAsync(cts.Token); - - Assert.True(result); - } - - [Fact] - public async Task CanRunJobs() - { - var job = new HelloWorldJob(null, Log); - Assert.Equal(0, job.RunCount); - await job.RunAsync(TestCancellationToken); - Assert.Equal(1, job.RunCount); - - await job.RunContinuousAsync(iterationLimit: 2, cancellationToken: TestCancellationToken); - Assert.Equal(3, job.RunCount); - - var sw = Stopwatch.StartNew(); - using (var timeoutCancellationTokenSource = new CancellationTokenSource(100)) - { - await job.RunContinuousAsync(cancellationToken: timeoutCancellationTokenSource.Token); - } - sw.Stop(); - Assert.InRange(sw.Elapsed, TimeSpan.FromMilliseconds(95), TimeSpan.FromMilliseconds(800)); - - var jobInstance = new HelloWorldJob(null, Log); - Assert.NotNull(jobInstance); - Assert.Equal(0, jobInstance.RunCount); - Assert.Equal(JobResult.Success, await jobInstance.RunAsync(TestCancellationToken)); - Assert.Equal(1, jobInstance.RunCount); - } - - [Fact] - public async Task CanRunMultipleInstances() - { - var job = new HelloWorldJob(null, Log); - var sp = new ServiceCollection().BuildServiceProvider(); - - HelloWorldJob.GlobalRunCount = 0; - using (var timeoutCancellationTokenSource = new CancellationTokenSource(1000)) - { - await new JobRunner(job, sp, Log, instanceCount: 5, iterationLimit: 1).RunAsync(timeoutCancellationTokenSource.Token); - } - - Assert.Equal(5, HelloWorldJob.GlobalRunCount); - - HelloWorldJob.GlobalRunCount = 0; - using (var timeoutCancellationTokenSource = new CancellationTokenSource(50000)) - { - await new JobRunner(job, sp, Log, instanceCount: 5, iterationLimit: 100).RunAsync(timeoutCancellationTokenSource.Token); - } - - Assert.Equal(500, HelloWorldJob.GlobalRunCount); - } - - [Fact] - public async Task CanCancelContinuousJobs() - { - var timeProvider = new FakeTimeProvider { AutoAdvanceAmount = TimeSpan.FromSeconds(1) }; - var job = new HelloWorldJob(timeProvider, Log); - var sp = new ServiceCollection().AddSingleton(_ => timeProvider).BuildServiceProvider(); - var timeoutCancellationTokenSource = new CancellationTokenSource(100); - await job.RunContinuousAsync(TimeSpan.FromSeconds(1), 5, timeoutCancellationTokenSource.Token); - - Assert.Equal(1, job.RunCount); - - timeoutCancellationTokenSource = new CancellationTokenSource(TimeSpan.FromMilliseconds(50), timeProvider); - var runnerTask = new JobRunner(job, sp, Log, instanceCount: 5, iterationLimit: 10000, interval: TimeSpan.FromMilliseconds(1)).RunAsync(timeoutCancellationTokenSource.Token); - timeProvider.Advance(TimeSpan.FromSeconds(1)); - await runnerTask; - } - - [Fact] - public async Task CanRunJobsWithLocks() - { - var job = new WithLockingJob(Log); - Assert.Equal(0, job.RunCount); - await job.RunAsync(TestCancellationToken); - Assert.Equal(1, job.RunCount); - - await job.RunContinuousAsync(iterationLimit: 2, cancellationToken: TestCancellationToken); - Assert.Equal(3, job.RunCount); - - await Parallel.ForEachAsync(Enumerable.Range(1, 2), async (_, ct) => await job.RunAsync(ct)); - Assert.Equal(4, job.RunCount); - } - - [Fact] - public async Task CanRunThrottledJobs() - { - using var client = new InMemoryCacheClient(o => o.LoggerFactory(Log)); - var jobs = new List([ - new ThrottledJob(client, Log), - new ThrottledJob(client, Log), - new ThrottledJob(client, Log) - ]); - - var sw = Stopwatch.StartNew(); - using var timeoutCancellationTokenSource = new CancellationTokenSource(1000); - await Task.WhenAll(jobs.Select(job => job.RunContinuousAsync(TimeSpan.FromMilliseconds(1), cancellationToken: timeoutCancellationTokenSource.Token))); - sw.Stop(); - - Assert.InRange(jobs.Sum(j => j.RunCount), 4, 14); - _logger.LogInformation("Job run count: {RunCount}", jobs.Sum(j => j.RunCount).ToString()); - Assert.InRange(sw.ElapsedMilliseconds, 20, 1500); - } - - [Fact] - public async Task CanRunJobsWithInterval() - { - var time = DateTimeOffset.UnixEpoch; - var timeProvider = new FakeTimeProvider(time); - var interval = TimeSpan.FromHours(.75); - - var job = new HelloWorldJob(timeProvider, Log); - - var jobTask = Task.Run(() => job.RunContinuousAsync(iterationLimit: 2, interval: interval), TestCancellationToken); - while (job.RunCount < 1) - await Task.Delay(10, TestCancellationToken); - timeProvider.Advance(interval); - await jobTask; - - Assert.Equal(2, job.RunCount); - Assert.Equal(interval, (timeProvider.GetUtcNow() - time)); - } - - [Fact] - public async Task CanRunJobsWithIntervalBetweenFailingJob() - { - var time = DateTimeOffset.UnixEpoch; - var interval = TimeSpan.FromHours(.75); - var timeProvider = new FakeTimeProvider(time) { AutoAdvanceAmount = interval }; - - var job = new FailingJob(timeProvider, Log); - - var jobTask = Task.Run(() => job.RunContinuousAsync(iterationLimit: 2, interval: interval), TestCancellationToken); - while (job.RunCount < 1) - await Task.Delay(10, TestCancellationToken); - timeProvider.Advance(interval); - await jobTask; - - Assert.Equal(2, job.RunCount); - Assert.Equal(interval, (timeProvider.GetUtcNow() - time)); - } - - [Fact(Skip = "Meant to be run manually.")] - public async Task JobLoopPerf() - { - const int iterations = 10000; - - var job = new SampleJob(null, Log); - var sw = Stopwatch.StartNew(); - await job.RunContinuousAsync(null, iterations, TestCancellationToken); - sw.Stop(); - } - - [Fact] - public async Task RunContinuousAsync_SuccessfulJob_DoesNotSetActivityErrorStatus() - { - // Arrange - Activity? capturedActivity = null; - using var listener = new ActivityListener - { - ShouldListenTo = s => s.Name == "Foundatio", - Sample = (ref ActivityCreationOptions _) => ActivitySamplingResult.AllDataAndRecorded, - ActivityStopped = a => - { - if (a.OperationName.StartsWith("Job:")) - capturedActivity = a; - } - }; - ActivitySource.AddActivityListener(listener); - - var job = new HelloWorldJob(null, Log); - - // Act - await job.RunContinuousAsync(iterationLimit: 1, cancellationToken: TestCancellationToken); - - // Assert - Assert.NotNull(capturedActivity); - Assert.Equal(ActivityStatusCode.Unset, capturedActivity.Status); - } -} diff --git a/tests/Foundatio.Tests/Jobs/JobsTestHarnessTests.cs b/tests/Foundatio.Tests/Jobs/JobsTestHarnessTests.cs new file mode 100644 index 000000000..ff820d994 --- /dev/null +++ b/tests/Foundatio.Tests/Jobs/JobsTestHarnessTests.cs @@ -0,0 +1,158 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Jobs.Testing; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace Foundatio.Tests.Jobs; + +public class JobsTestHarnessTests +{ + [Fact] + public async Task RunAllQueued_MoreThanOneBatch_DrainsAllReadyJobsAsync() + { + var token = TestContext.Current.CancellationToken; + var (provider, probe) = CreateProvider(); + await using var _ = provider; + var harness = provider.GetRequiredService(); + for (int i = 0; i < 201; i++) + await harness.Client.EnqueueAsync(cancellationToken: token); + Assert.Equal(201, await harness.RunAllQueuedAsync(token)); + Assert.Equal(201, probe.RunCount); + } + + [Fact] + public async Task RunToCompletion_LeavesOtherJobsQueuedAsync() + { + var token = TestContext.Current.CancellationToken; + var (provider, probe) = CreateProvider(); + await using var _ = provider; + var harness = provider.GetRequiredService(); + var other = await harness.Client.EnqueueAsync(cancellationToken: token); + var target = await harness.Client.EnqueueAsync(new GreetingArgs { Name = "ada" }, cancellationToken: token); + Assert.Equal(JobStatus.Completed, (await harness.RunToCompletionAsync(target, token)).Status); + Assert.Equal(JobStatus.Queued, (await other.GetStateAsync(token))!.Status); + Assert.Equal(0, probe.RunCount); + } + + [Fact] + public async Task RunAllQueued_RunsEnqueuedJobsToCompletionAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var (provider, probe) = CreateProvider(); + await using var _ = provider; + var harness = provider.GetRequiredService(); + + // The harness disables the auto pump, so nothing runs until the test says so. + Assert.Empty(provider.GetServices()); + + var handle = await harness.Client.EnqueueAsync(cancellationToken: cancellationToken); + Assert.Equal(JobStatus.Queued, (await harness.Monitor.GetAsync(handle.JobId, cancellationToken))!.Status); + Assert.Equal(0, probe.RunCount); + + Assert.Equal(1, await harness.RunAllQueuedAsync(cancellationToken)); + Assert.Equal(1, probe.RunCount); + Assert.Equal(JobStatus.Completed, (await handle.GetStateAsync(cancellationToken))!.Status); + + // Nothing left queued: a second pass is a no-op, not a re-run. + Assert.Equal(0, await harness.RunAllQueuedAsync(cancellationToken)); + Assert.Equal(1, probe.RunCount); + } + + [Fact] + public async Task RunDue_MaterializesAndRunsTheCronOccurrenceAtAFixedNowAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var (provider, probe) = CreateProvider(); + await using var _ = provider; + var harness = provider.GetRequiredService(); + + await harness.Schedules.ScheduleAsync(new ScheduledJobDefinition + { + Name = "every-minute", + Cron = "* * * * *", + JobType = typeof(CounterJob).FullName! + }, cancellationToken); + + // One deterministic tick at a fixed "now": the 00:00:00 occurrence falls due within the misfire window and + // runs in this call — no pump, no sleeps. + var tick = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + Assert.Equal(1, await harness.RunDueAsync(tick, cancellationToken)); + Assert.Equal(1, probe.RunCount); + + var occurrence = Assert.Single(await harness.Monitor.QueryAsync(new JobQuery { Name = "every-minute" }, cancellationToken)); + Assert.Equal(JobStatus.Completed, occurrence.Status); + Assert.Equal(new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero), occurrence.ScheduledForUtc); + + // The same tick again is a no-op: the occurrence id dedupes and its dispatch was retired. + Assert.Equal(0, await harness.RunDueAsync(tick, cancellationToken)); + Assert.Equal(1, probe.RunCount); + } + + [Fact] + public async Task RunToCompletion_RunsATypedArgsJobToItsTerminalStateAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var (provider, probe) = CreateProvider(); + await using var _ = provider; + var harness = provider.GetRequiredService(); + + var handle = await harness.Client.EnqueueAsync(new GreetingArgs { Name = "ada" }, cancellationToken: cancellationToken); + + var state = await harness.RunToCompletionAsync(handle, cancellationToken); + Assert.Equal(JobStatus.Completed, state.Status); + Assert.Equal("ada", probe.LastGreeted); + } + + private static (ServiceProvider Provider, Probe Probe) CreateProvider() + { + var probe = new Probe(); + var services = new ServiceCollection(); + services.AddSingleton(probe); + services.AddFoundatio().Jobs.UseTestHarness().AddJobType().AddJobType(); + return (services.BuildServiceProvider(), probe); + } + + private sealed class Probe + { + private int _runCount; + public int RunCount => Volatile.Read(ref _runCount); + public string? LastGreeted { get; private set; } + + public void Ran() => Interlocked.Increment(ref _runCount); + public void Greeted(string? name) => LastGreeted = name; + } + + private sealed class CounterJob : IJob + { + private readonly Probe _probe; + + public CounterJob(Probe probe) => _probe = probe; + + public Task RunAsync(JobExecutionContext context) + { + _probe.Ran(); + return Task.FromResult(JobResult.Success); + } + } + + private sealed class GreetingArgs + { + public string? Name { get; set; } + } + + private sealed class GreetingJob : IJob + { + private readonly Probe _probe; + + public GreetingJob(Probe probe) => _probe = probe; + + public Task RunAsync(GreetingArgs arguments, JobExecutionContext context) + { + _probe.Greeted(arguments.Name); + return Task.FromResult(JobResult.Success); + } + } +} diff --git a/tests/Foundatio.Tests/Jobs/LeaseSupervisionTests.cs b/tests/Foundatio.Tests/Jobs/LeaseSupervisionTests.cs new file mode 100644 index 000000000..375fb1b74 --- /dev/null +++ b/tests/Foundatio.Tests/Jobs/LeaseSupervisionTests.cs @@ -0,0 +1,118 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace Foundatio.Tests.Jobs; + +public class LeaseSupervisionTests +{ + private static JobTypeRegistry CreateJobRegistry() => new(typeof(LeaseSupervisionTests).GetNestedTypes(System.Reflection.BindingFlags.NonPublic) + .Where(t => t.IsClass && !t.IsAbstract && typeof(IJob).IsAssignableFrom(t)) + .Select(t => new JobTypeRegistration(t.FullName!, t))); + + [Fact] + public async Task RenewalDenied_CancelsRunningJobAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new LeaseFailingStore(new InMemoryJobRuntimeStore()) { DenyRenewals = true }; + await using var serviceProvider = new ServiceCollection().BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "node-a", Lease = TimeSpan.FromSeconds(1), JobTypes = CreateJobRegistry() }); + + var handle = await client.EnqueueAsync(cancellationToken: cancellationToken); + Assert.True(await worker.RunAsync(handle.JobId, cancellationToken)); + + // A clean "renewal denied" means another node owns the lease: the run must be cancelled, not left executing. + var state = await handle.GetStateAsync(cancellationToken); + Assert.Equal(JobStatus.Processing, state!.Status); + } + + [Fact] + public async Task RenewalThrowingPastLeaseWindow_CancelsRunningJobAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new LeaseFailingStore(new InMemoryJobRuntimeStore()) { ThrowOnRenewals = true }; + await using var serviceProvider = new ServiceCollection().BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "node-a", Lease = TimeSpan.FromSeconds(1), JobTypes = CreateJobRegistry() }); + + var handle = await client.EnqueueAsync(cancellationToken: cancellationToken); + Assert.True(await worker.RunAsync(handle.JobId, cancellationToken)); + + // Renewal that keeps THROWING must not let the run outlive its lease: once the window passes without one + // successful renewal, another node may have reclaimed the job, so continuing would double-run side effects. + var state = await handle.GetStateAsync(cancellationToken); + Assert.Equal(JobStatus.Processing, state!.Status); + } + + private sealed class WaitForCancellationJob : IJob + { + public async Task RunAsync(JobExecutionContext context) + { + try + { + // Runs "forever" unless the supervision loop cancels the run. + await Task.Delay(TimeSpan.FromSeconds(30), context.CancellationToken); + } + catch (OperationCanceledException) + { + return JobResult.CancelledWithMessage("lease lost"); + } + + return JobResult.FailedWithMessage("was never cancelled"); + } + } + + // Delegates everything to the inner store; renewals can be denied (clean lease loss) or made to throw (store outage). + private sealed class LeaseFailingStore : IJobRuntimeStore + { + private readonly IJobRuntimeStore _inner; + + public LeaseFailingStore(IJobRuntimeStore inner) => _inner = inner; + + public bool DenyRenewals { get; set; } + public bool ThrowOnRenewals { get; set; } + + public bool IsShared => _inner.IsShared; + public Task CountAsync(JobQuery query, CancellationToken cancellationToken = default) => _inner.CountAsync(query, cancellationToken); + public Task BeginBrokerAttemptAsync(string jobId, int attempt, string nodeId, CancellationToken cancellationToken = default) => _inner.BeginBrokerAttemptAsync(jobId, attempt, nodeId, cancellationToken); + public Task MarkEnqueueUnknownAsync(string jobId, string error, CancellationToken cancellationToken = default) => _inner.MarkEnqueueUnknownAsync(jobId, error, cancellationToken); + public Task HeartbeatJobAsync(string jobId, string claimToken, CancellationToken cancellationToken = default) => _inner.HeartbeatJobAsync(jobId, claimToken, cancellationToken); + public Task RemoveAsync(string jobId, CancellationToken cancellationToken = default) => _inner.RemoveAsync(jobId, cancellationToken); + public Task IncrementCounterAsync(string name, string counterName, long value = 1, CancellationToken cancellationToken = default) => _inner.IncrementCounterAsync(name, counterName, value, cancellationToken); + public Task GetCounterStatsAsync(string name, TimeSpan? window = null, CancellationToken cancellationToken = default) => _inner.GetCounterStatsAsync(name, window, cancellationToken); + + public Task ScheduleAsync(ScheduledJobDefinition definition, CancellationToken ct = default) => _inner.ScheduleAsync(definition, ct); + public Task ReconcileAsync(ScheduledJobDefinition definition, CancellationToken ct = default) => _inner.ReconcileAsync(definition, ct); + public Task GetScheduleAsync(string name, CancellationToken ct = default) => _inner.GetScheduleAsync(name, ct); + public Task UnscheduleAsync(string name, CancellationToken ct = default) => _inner.UnscheduleAsync(name, ct); + public Task> GetSchedulesAsync(ScheduleQuery? query = null, CancellationToken ct = default) => _inner.GetSchedulesAsync(query, ct); + public Task GetAsync(string jobId, CancellationToken ct = default) => _inner.GetAsync(jobId, ct); + public Task CleanupAsync(int limit = 1000, CancellationToken ct = default) => _inner.CleanupAsync(limit, ct); + public Task QueryAsync(JobQuery query, CancellationToken ct = default) => _inner.QueryAsync(query, ct); + public Task CreateOccurrenceAsync(JobState initial, bool allowOverlap = false, CancellationToken ct = default) => _inner.CreateOccurrenceAsync(initial, allowOverlap, ct); + public Task ClaimNextAsync(JobClaimRequest request, CancellationToken ct = default) => _inner.ClaimNextAsync(request, ct); + public Task ClaimJobAsync(string jobId, JobClaimRequest request, CancellationToken ct = default) => _inner.ClaimJobAsync(jobId, request, ct); + public Task CompleteJobAsync(string jobId, string claimToken, JobCompletion completion, CancellationToken ct = default) => _inner.CompleteJobAsync(jobId, claimToken, completion, ct); + public Task RenewJobLeaseAsync(string jobId, string claimToken, TimeSpan lease, CancellationToken ct = default) + { + if (ThrowOnRenewals) + throw new TimeoutException("store unreachable"); + return DenyRenewals ? Task.FromResult(false) : _inner.RenewJobLeaseAsync(jobId, claimToken, lease, ct); + } + public Task ReportJobProgressAsync(string jobId, string claimToken, int? percent = null, string? message = null, CancellationToken ct = default) => _inner.ReportJobProgressAsync(jobId, claimToken, percent, message, ct); + public Task CreateIfAbsentAsync(JobState initial, CancellationToken ct = default) => _inner.CreateIfAbsentAsync(initial, ct); + public Task RequestCancellationAsync(string jobId, CancellationToken ct = default) => _inner.RequestCancellationAsync(jobId, ct); + public Task IsCancellationRequestedAsync(string jobId, CancellationToken ct = default) => _inner.IsCancellationRequestedAsync(jobId, ct); + public Task ScheduleDispatchAsync(ScheduledDispatchState dispatch, CancellationToken ct = default) => _inner.ScheduleDispatchAsync(dispatch, ct); + public Task> ClaimDueDispatchesAsync(DateTimeOffset now, int limit, string nodeId, TimeSpan lease, CancellationToken ct = default) => _inner.ClaimDueDispatchesAsync(now, limit, nodeId, lease, ct); + public Task GetStatsAsync(CancellationToken ct = default) => _inner.GetStatsAsync(ct); + public Task CompleteDispatchAsync(string dispatchId, string nodeId, CancellationToken ct = default) => _inner.CompleteDispatchAsync(dispatchId, nodeId, ct); + public Task ReleaseDispatchAsync(string dispatchId, string nodeId, DateTimeOffset nextDueUtc, CancellationToken ct = default) => _inner.ReleaseDispatchAsync(dispatchId, nodeId, nextDueUtc, ct); + } +} diff --git a/tests/Foundatio.Tests/Jobs/ScheduledJobManagerTests.cs b/tests/Foundatio.Tests/Jobs/ScheduledJobManagerTests.cs new file mode 100644 index 000000000..3a0c81448 --- /dev/null +++ b/tests/Foundatio.Tests/Jobs/ScheduledJobManagerTests.cs @@ -0,0 +1,212 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace Foundatio.Tests.Jobs; + +public class ScheduledJobManagerTests +{ + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task ScheduleAsync_WithUnregisteredType_RejectsBeforePersisting(bool typed) + { + var token = TestContext.Current.CancellationToken; + var schedules = new InMemoryScheduledJobStore(); + var manager = new ScheduledJobManager(schedules, new InMemoryJobRuntimeStore(), new JobTypeRegistry()); + + var error = await Assert.ThrowsAsync(() => typed + ? manager.ScheduleAsync("* * * * *", cancellationToken: token) + : manager.ScheduleAsync(new ScheduledJobDefinition { Name = "unknown", Cron = "* * * * *", JobType = typeof(ProbeJob).FullName! }, token)); + + Assert.Contains("AddJobType", error.Message); + Assert.Empty(await schedules.GetSchedulesAsync(cancellationToken: token)); + } + + private static JobTypeRegistry CreateJobRegistry() => new(typeof(ScheduledJobManagerTests).GetNestedTypes(System.Reflection.BindingFlags.NonPublic) + .Where(t => t.IsClass && !t.IsAbstract && typeof(IJob).IsAssignableFrom(t)) + .Select(t => new JobTypeRegistration(t.FullName!, t))); + + [Fact] + public async Task ScheduleAsync_AddsAndReplacesByNameAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var (manager, _, _, _) = CreateRuntime(); + + await manager.ScheduleAsync(new ScheduledJobDefinition { Name = "nightly", Cron = "0 3 * * *", JobType = typeof(ProbeJob).FullName! }, cancellationToken); + Assert.Equal("0 3 * * *", (await manager.GetScheduleAsync("nightly", cancellationToken))!.Cron); + + // Re-scheduling the same name replaces the whole definition (runtime add/update, no restart). + await manager.ScheduleAsync((await manager.GetScheduleAsync("nightly", cancellationToken))! with { Cron = "0 4 * * *", MaxAttempts = 7 }, cancellationToken); + var updated = await manager.GetScheduleAsync("nightly", cancellationToken); + Assert.Equal("0 4 * * *", updated!.Cron); + Assert.Equal(7, updated.MaxAttempts); + Assert.Single(await manager.GetSchedulesAsync(cancellationToken: cancellationToken)); + } + + [Fact] + public async Task RescheduleAsync_ChangesCronAndValidatesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var (manager, _, _, _) = CreateRuntime(); + await manager.ScheduleAsync(new ScheduledJobDefinition { Name = "nightly", Cron = "0 3 * * *", JobType = typeof(ProbeJob).FullName!, MaxAttempts = 5 }, cancellationToken); + + Assert.True(await manager.RescheduleAsync("nightly", "*/5 * * * *", cancellationToken)); + var updated = await manager.GetScheduleAsync("nightly", cancellationToken); + Assert.Equal("*/5 * * * *", updated!.Cron); + Assert.Equal(5, updated.MaxAttempts); // only the cron changed; the rest of the definition is preserved + + Assert.False(await manager.RescheduleAsync("unknown", "*/5 * * * *", cancellationToken)); + await Assert.ThrowsAnyAsync(() => manager.RescheduleAsync("nightly", "not-a-cron", cancellationToken)); + Assert.Equal("*/5 * * * *", (await manager.GetScheduleAsync("nightly", cancellationToken))!.Cron); // invalid input changed nothing + } + + [Fact] + public async Task SetEnabledAsync_StopsAndResumesOccurrenceMaterializationAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var (manager, processor, _, _) = CreateRuntime(); + await manager.ScheduleAsync(new ScheduledJobDefinition { Name = "everyminute", Cron = "* * * * *", JobType = typeof(ProbeJob).FullName! }, cancellationToken); + + var tick = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + Assert.True(await manager.SetEnabledAsync("everyminute", false, cancellationToken)); + Assert.Empty(await processor.EnqueueDueOccurrencesAsync(tick, cancellationToken)); // disabled -> nothing materializes + + Assert.True(await manager.SetEnabledAsync("everyminute", true, cancellationToken)); + Assert.Single(await processor.EnqueueDueOccurrencesAsync(tick, cancellationToken)); // re-enabled -> occurrence materializes + + Assert.False(await manager.SetEnabledAsync("unknown", true, cancellationToken)); + } + + [Fact] + public async Task TriggerAsync_RunsImmediatelyWithArgumentsAndReturnsHandleAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var (manager, _, worker, probe) = CreateRuntime(); + + // A schedule that would never fire on its own within the test (yearly), with typed arguments. + await manager.ScheduleAsync("0 0 1 1 *", new ReportArgs { Region = "emea" }, o => o.Name = "yearly-report", cancellationToken); + + var handle = await manager.TriggerAsync("yearly-report", cancellationToken); + Assert.StartsWith("yearly-report:manual:", handle.JobId); + + // The trigger is durable: the pump's normal drain claims and runs it. + Assert.Equal(1, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + Assert.Equal("emea", probe.LastRegion); + + var state = await handle.GetStateAsync(cancellationToken); + Assert.Equal(JobStatus.Completed, state!.Status); + + // A second trigger runs again (manual occurrences never dedupe). + await manager.TriggerAsync("yearly-report", cancellationToken); + Assert.Equal(1, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + Assert.Equal(2, probe.RunCount); + } + + [Fact] + public async Task GenericOverloads_ResolveTheTypeDefaultNameAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var (manager, _, worker, probe) = CreateRuntime(); + + // Registered the way AddCronJob does when no explicit name is given: the type's default name. + await manager.ScheduleAsync(new ScheduledJobDefinition + { + Name = ScheduledJobDefinition.DefaultNameFor(typeof(ProbeJob)), + Cron = "0 0 1 1 *", + JobType = typeof(ProbeJob).FullName! + }, cancellationToken); + + var found = await manager.GetScheduleAsync(cancellationToken); + Assert.NotNull(found); + Assert.Equal(nameof(ProbeJob), found.Name); + + Assert.True(await manager.RescheduleAsync("*/10 * * * *", cancellationToken)); + Assert.Equal("*/10 * * * *", (await manager.GetScheduleAsync(cancellationToken))!.Cron); + + Assert.True(await manager.SetEnabledAsync(false, cancellationToken)); + await Assert.ThrowsAsync(() => manager.TriggerAsync(cancellationToken)); + Assert.True(await manager.SetEnabledAsync(true, cancellationToken)); + + var handle = await manager.TriggerAsync(cancellationToken); + Assert.StartsWith($"{nameof(ProbeJob)}:manual:", handle.JobId); + Assert.Equal(1, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + Assert.Equal(1, probe.RunCount); + Assert.Equal(JobStatus.Completed, (await handle.GetStateAsync(cancellationToken))!.Status); + + await manager.UnscheduleAsync(cancellationToken); + Assert.Null(await manager.GetScheduleAsync(cancellationToken)); + } + + [Fact] + public async Task TriggerAsync_UnknownOrDisabled_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var (manager, _, _, _) = CreateRuntime(); + + var notFound = await Assert.ThrowsAsync(() => manager.TriggerAsync("unknown", cancellationToken)); + Assert.Equal("unknown", notFound.Name); + + await manager.ScheduleAsync(new ScheduledJobDefinition { Name = "off", Cron = "* * * * *", JobType = typeof(ProbeJob).FullName!, Enabled = false }, cancellationToken); + var ex = await Assert.ThrowsAsync(() => manager.TriggerAsync("off", cancellationToken)); + Assert.Contains("disabled", ex.Message); + Assert.Equal("off", ex.Name); + } + + private static (IScheduledJobManager Manager, JobScheduleProcessor Processor, IJobWorker Worker, RegionProbe Probe) CreateRuntime() + { + var store = new InMemoryJobRuntimeStore(); + var scheduler = new InMemoryScheduledJobStore(); + var probe = new RegionProbe(); + var serviceProvider = new ServiceCollection().AddSingleton(probe).BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, new JobWorkerOptions { NodeId = "node-a", JobTypes = CreateJobRegistry() }); + var manager = new ScheduledJobManager(scheduler, store); + return (manager, new JobScheduleProcessor(scheduler, store), worker, probe); + } + + private sealed class TypedProbeJob(RegionProbe probe) : IJob + { + public Task RunAsync(ReportArgs arguments, JobExecutionContext context) + { + context.CancellationToken.ThrowIfCancellationRequested(); + probe.Record(arguments.Region); + return Task.FromResult(JobResult.Success); + } + } + + private sealed class ReportArgs + { + public string? Region { get; set; } + } + + private sealed class RegionProbe + { + private int _runCount; + public int RunCount => Volatile.Read(ref _runCount); + public string? LastRegion { get; private set; } + + public void Record(string? region) + { + Interlocked.Increment(ref _runCount); + LastRegion = region; + } + } + + private sealed class ProbeJob : IJob + { + private readonly RegionProbe _probe; + + public ProbeJob(RegionProbe probe) => _probe = probe; + + public Task RunAsync(JobExecutionContext context) + { + _probe.Record(context.HasArguments ? context.GetArguments().Region : null); + return Task.FromResult(JobResult.Success); + } + } +} diff --git a/tests/Foundatio.Tests/Jobs/WorkItemJobTests.cs b/tests/Foundatio.Tests/Jobs/WorkItemJobTests.cs deleted file mode 100644 index 9b1cd4226..000000000 --- a/tests/Foundatio.Tests/Jobs/WorkItemJobTests.cs +++ /dev/null @@ -1,363 +0,0 @@ -using System; -using System.Collections.Concurrent; -using System.Collections.Generic; -using System.Diagnostics; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; -using Exceptionless; -using Foundatio.AsyncEx; -using Foundatio.Jobs; -using Foundatio.Messaging; -using Foundatio.Queues; -using Foundatio.Tests.Extensions; -using Foundatio.Xunit; -using Microsoft.Extensions.Logging; -using Xunit; - -namespace Foundatio.Tests.Jobs; - -public class WorkItemJobTests : TestWithLoggingBase -{ - public WorkItemJobTests(ITestOutputHelper output) : base(output) { } - - [Fact] - public async Task CanRunWorkItem() - { - using var queue = new InMemoryQueue(o => o.LoggerFactory(Log)); - using var messageBus = new InMemoryMessageBus(o => o.LoggerFactory(Log)); - var handlerRegistry = new WorkItemHandlers(); - var job = new WorkItemJob(queue, messageBus, handlerRegistry, Log); - - handlerRegistry.Register(async ctx => - { - var jobData = ctx.GetData(); - Assert.NotNull(jobData); - Assert.Equal("Test", jobData.SomeData); - - for (int i = 0; i < 10; i++) - { - await Task.Delay(100, TestCancellationToken); - await ctx.ReportProgressAsync(10 * i); - } - }); - - string jobId = await queue.EnqueueAsync(new MyWorkItem - { - SomeData = "Test" - }, true); - - var countdown = new AsyncCountdownEvent(12); - await messageBus.SubscribeAsync(status => - { - _logger.LogInformation("Progress: {Progress}", status.Progress); - Assert.Equal(jobId, status.WorkItemId); - countdown.Signal(); - }, TestCancellationToken); - - await job.RunAsync(TestCancellationToken); - await countdown.WaitAsync(TimeSpan.FromSeconds(2)); - Assert.Equal(0, countdown.CurrentCount); - } - - [Fact] - public async Task CanHandleMultipleWorkItemInstances() - { - const int workItemCount = 1000; - - using var queue = new InMemoryQueue(o => o.RetryDelay(TimeSpan.Zero).Retries(0).LoggerFactory(Log)); - using var messageBus = new InMemoryMessageBus(o => o.LoggerFactory(Log)); - var handlerRegistry = new WorkItemHandlers(); - var j1 = new WorkItemJob(queue, messageBus, handlerRegistry, Log); - var j2 = new WorkItemJob(queue, messageBus, handlerRegistry, Log); - var j3 = new WorkItemJob(queue, messageBus, handlerRegistry, Log); - int errors = 0; - - var jobIds = new ConcurrentDictionary(); - - handlerRegistry.Register(async ctx => - { - var jobData = ctx.GetData(); - Assert.NotNull(jobData); - Assert.Equal("Test", jobData.SomeData); - - int jobWorkTotal = jobIds.AddOrUpdate(ctx.JobId, 1, (key, value) => value + 1); - if (jobData.Index % 100 == 0) - _logger.LogTrace("Job {JobId} processing work item #: {JobWorkTotal}", ctx.JobId, jobWorkTotal); - - for (int i = 0; i < 10; i++) - await ctx.ReportProgressAsync(10 * i); - - if (RandomData.GetBool(1)) - { - Interlocked.Increment(ref errors); - throw new Exception("Boom!"); - } - }); - - for (int i = 0; i < workItemCount; i++) - await queue.EnqueueAsync(new MyWorkItem - { - SomeData = "Test", - Index = i - }, true); - - var completedItems = new List(); - object completedItemsLock = new(); - await messageBus.SubscribeAsync(status => - { - if (status.Progress == 100) - _logger.LogTrace("Progress: {Progress}", status.Progress); - - if (status.Progress < 100) - return; - - lock (completedItemsLock) - { - Assert.NotNull(status.WorkItemId); - completedItems.Add(status.WorkItemId); - } - }, TestCancellationToken); - - using var cancellationTokenSource = new CancellationTokenSource(10000); - List tasks = - [ - Task.Run(async () => - { - await j1.RunUntilEmptyAsync(cancellationTokenSource.Token); - await cancellationTokenSource.CancelAsync(); - }, cancellationTokenSource.Token), - - Task.Run(async () => - { - await j2.RunUntilEmptyAsync(cancellationTokenSource.Token); - await cancellationTokenSource.CancelAsync(); - }, cancellationTokenSource.Token), - - Task.Run(async () => - { - await j3.RunUntilEmptyAsync(cancellationTokenSource.Token); - await cancellationTokenSource.CancelAsync(); - }, cancellationTokenSource.Token) - ]; - - try - { - await Task.WhenAll(tasks); - } - catch (OperationCanceledException ex) - { - _logger.LogError(ex, "One or more tasks were cancelled: {Message}", ex.Message); - } - - await Task.Delay(100, TestCancellationToken); - _logger.LogInformation("Completed: {CompletedItems} Errors: {Errors}", completedItems.Count, errors); - Assert.Equal(workItemCount, completedItems.Count + errors); - Assert.Equal(3, jobIds.Count); - Assert.Equal(workItemCount, jobIds.Sum(kvp => kvp.Value)); - } - - [Fact] - public async Task CanRunWorkItemWithClassHandler() - { - using var queue = new InMemoryQueue(o => o.LoggerFactory(Log)); - using var messageBus = new InMemoryMessageBus(o => o.LoggerFactory(Log)); - var handlerRegistry = new WorkItemHandlers(); - var job = new WorkItemJob(queue, messageBus, handlerRegistry, Log); - - handlerRegistry.Register(new MyWorkItemHandler(Log)); - - string jobId = await queue.EnqueueAsync(new MyWorkItem - { - SomeData = "Test" - }, true); - - var countdown = new AsyncCountdownEvent(11); - await messageBus.SubscribeAsync(status => - { - _logger.LogTrace("Progress: {Progress}", status.Progress); - Assert.Equal(jobId, status.WorkItemId); - countdown.Signal(); - }, TestCancellationToken); - - Assert.Equal(1, await job.RunUntilEmptyAsync(cancellationToken: TestCancellationToken)); - await countdown.WaitAsync(TimeSpan.FromSeconds(2)); - Assert.Equal(0, countdown.CurrentCount); - } - - [Fact] - public async Task CanRunWorkItemWithDelegateHandler() - { - using var queue = new InMemoryQueue(o => o.LoggerFactory(Log)); - using var messageBus = new InMemoryMessageBus(o => o.LoggerFactory(Log)); - var handlerRegistry = new WorkItemHandlers(); - var job = new WorkItemJob(queue, messageBus, handlerRegistry, Log); - - handlerRegistry.Register(async ctx => - { - var jobData = ctx.GetData(); - Assert.NotNull(jobData); - Assert.Equal("Test", jobData.SomeData); - - for (int i = 1; i < 10; i++) - { - await Task.Delay(100, TestCancellationToken); - await ctx.ReportProgressAsync(10 * i); - } - }, Log.CreateLogger("MyWorkItem")); - - string jobId = await queue.EnqueueAsync(new MyWorkItem - { - SomeData = "Test" - }, true); - - var countdown = new AsyncCountdownEvent(11); - await messageBus.SubscribeAsync(status => - { - _logger.LogTrace("Progress: {Progress}", status.Progress); - Assert.Equal(jobId, status.WorkItemId); - countdown.Signal(); - }, TestCancellationToken); - - Assert.Equal(1, await job.RunUntilEmptyAsync(cancellationToken: TestCancellationToken)); - await countdown.WaitAsync(TimeSpan.FromSeconds(2)); - Assert.Equal(0, countdown.CurrentCount); - } - - [Fact] - public async Task CanRunWorkItemJobUntilEmpty() - { - using var queue = new InMemoryQueue(o => o.LoggerFactory(Log)); - using var messageBus = new InMemoryMessageBus(o => o.LoggerFactory(Log)); - var handlerRegistry = new WorkItemHandlers(); - var job = new WorkItemJob(queue, messageBus, handlerRegistry, Log); - - handlerRegistry.Register(new MyWorkItemHandler(Log)); - - await queue.EnqueueAsync(new MyWorkItem - { - SomeData = "Test" - }, true); - - await queue.EnqueueAsync(new MyWorkItem - { - SomeData = "Test" - }, true); - - Assert.Equal(2, await job.RunUntilEmptyAsync(cancellationToken: TestCancellationToken)); - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(2, stats.Enqueued); - Assert.Equal(2, stats.Dequeued); - Assert.Equal(2, stats.Completed); - } - - [Fact] - public async Task CanRunWorkItemJobUntilEmptyWithNoEnqueuedItems() - { - using var queue = new InMemoryQueue(o => o.LoggerFactory(Log)); - using var messageBus = new InMemoryMessageBus(o => o.LoggerFactory(Log)); - var handlerRegistry = new WorkItemHandlers(); - var job = new WorkItemJob(queue, messageBus, handlerRegistry, Log); - - handlerRegistry.Register(new MyWorkItemHandler(Log)); - - var sw = Stopwatch.StartNew(); - Assert.Equal(0, await job.RunUntilEmptyAsync(TimeSpan.FromMilliseconds(100), TestCancellationToken)); - sw.Stop(); - - Assert.True(sw.Elapsed < TimeSpan.FromMilliseconds(250)); - - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(0, stats.Enqueued); - Assert.Equal(0, stats.Dequeued); - Assert.Equal(0, stats.Completed); - } - - [Fact] - public async Task CanRunWorkItemJobUntilEmptyHandlesCancellation() - { - using var queue = new InMemoryQueue(o => o.LoggerFactory(Log)); - using var messageBus = new InMemoryMessageBus(o => o.LoggerFactory(Log)); - var handlerRegistry = new WorkItemHandlers(); - var job = new WorkItemJob(queue, messageBus, handlerRegistry, Log); - - handlerRegistry.Register(new MyWorkItemHandler(Log)); - - await queue.EnqueueAsync(new MyWorkItem - { - SomeData = "Test" - }, true); - - await queue.EnqueueAsync(new MyWorkItem - { - SomeData = "Test" - }, true); - - Assert.Equal(1, await job.RunUntilEmptyAsync(TimeSpan.FromMilliseconds(50), TestCancellationToken)); - - var stats = await queue.GetQueueStatsAsync(); - Assert.Equal(2, stats.Enqueued); - Assert.Equal(1, stats.Dequeued); - Assert.Equal(1, stats.Completed); - } - - [Fact] - public async Task CanRunBadWorkItem() - { - using var queue = new InMemoryQueue(o => o.RetryDelay(TimeSpan.FromMilliseconds(500)).LoggerFactory(Log)); - using var messageBus = new InMemoryMessageBus(o => o.LoggerFactory(Log)); - var handlerRegistry = new WorkItemHandlers(); - var job = new WorkItemJob(queue, messageBus, handlerRegistry, Log); - - handlerRegistry.Register(ctx => - { - var jobData = ctx.GetData(); - Assert.NotNull(jobData); - Assert.Equal("Test", jobData.SomeData); - throw new Exception(); - }); - - string jobId = await queue.EnqueueAsync(new MyWorkItem - { - SomeData = "Test" - }, true); - - var countdown = new AsyncCountdownEvent(2); - await messageBus.SubscribeAsync(status => - { - _logger.LogTrace("Progress: {Progress}", status.Progress); - Assert.Equal(jobId, status.WorkItemId); - countdown.Signal(); - }, TestCancellationToken); - - Assert.Equal(0, await job.RunUntilEmptyAsync(cancellationToken: TestCancellationToken)); - await countdown.WaitAsync(TimeSpan.FromSeconds(2)); - Assert.Equal(0, countdown.CurrentCount); - } -} - -public class MyWorkItem -{ - public required string SomeData { get; set; } - public int Index { get; set; } -} - -public class MyWorkItemHandler : WorkItemHandlerBase -{ - public MyWorkItemHandler(ILoggerFactory? loggerFactory = null) : base(loggerFactory) - { - } - - public override async Task HandleItemAsync(WorkItemContext context) - { - var jobData = context.GetData(); - Assert.NotNull(jobData); - Assert.Equal("Test", jobData.SomeData); - - for (int i = 1; i < 10; i++) - { - await Task.Delay(10); - await context.ReportProgressAsync(10 * i); - } - } -} diff --git a/tests/Foundatio.Tests/Locks/InMemoryLockTests.cs b/tests/Foundatio.Tests/Locks/InMemoryLockTests.cs index de9c2d073..4326e65cd 100644 --- a/tests/Foundatio.Tests/Locks/InMemoryLockTests.cs +++ b/tests/Foundatio.Tests/Locks/InMemoryLockTests.cs @@ -15,7 +15,7 @@ public class InMemoryLockTests : LockTestBase, IDisposable public InMemoryLockTests(ITestOutputHelper output) : base(output) { _cache = new InMemoryCacheClient(o => o.LoggerFactory(Log)); - _messageBus = new InMemoryMessageBus(o => o.LoggerFactory(Log)); + _messageBus = new MessageBus(new InMemoryMessageTransport(), new MessageBusOptions { LoggerFactory = Log }); } protected override ILockProvider GetThrottlingLockProvider(int maxHits, TimeSpan period) @@ -151,6 +151,6 @@ public override Task TryUsingAsync_WithSuccessfulAction_ExecutesAndReleasesLock( public void Dispose() { _cache.Dispose(); - _messageBus.Dispose(); + _messageBus.DisposeAsync().AsTask().GetAwaiter().GetResult(); } } diff --git a/tests/Foundatio.Tests/Messaging/BatchOutcomeTests.cs b/tests/Foundatio.Tests/Messaging/BatchOutcomeTests.cs new file mode 100644 index 000000000..aec2e39f6 --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/BatchOutcomeTests.cs @@ -0,0 +1,77 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Moq; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class BatchOutcomeTests +{ + [Theory] + [InlineData(MessageSendStatus.Rejected, false)] + [InlineData(MessageSendStatus.Unknown, false)] + [InlineData(MessageSendStatus.NotAttempted, false)] + [InlineData(MessageSendStatus.Rejected, true)] + [InlineData(MessageSendStatus.Unknown, true)] + public async Task SendAsync_PartialAcceptance_PreservesTheApplicationIdAndProviderOutcome(MessageSendStatus status, bool throws) + { + var transport = new Mock(); + var item = new SendItemResult { Index = 0, MessageId = "broker-id", Status = status, ErrorCode = "Unavailable", ErrorMessage = "Retry later", Retryable = true }; + var send = transport.Setup(t => t.SendAsync(It.IsAny(), It.IsAny>(), It.IsAny(), It.IsAny())); + if (throws) send.ThrowsAsync(new TransportSendException([item], new TimeoutException())); + else send.ReturnsAsync(new SendResult { Items = [item] }); + await using var bus = new MessageBus(transport.Object); + var failure = await Assert.ThrowsAsync(() => bus.SendAsync(new Event(), new MessageSendOptions { MessageId = "application-id" }, TestContext.Current.CancellationToken)); + var outcome = Assert.Single(failure.Outcomes); + Assert.Equal("application-id", outcome.MessageId); + Assert.Equal(status, outcome.Status); + Assert.Equal("Unavailable", outcome.ErrorCode); + Assert.Equal("Retry later", outcome.ErrorMessage); + Assert.True(outcome.Retryable); + } + + [Theory] + [InlineData(-1)] + [InlineData(1)] + public async Task SendAsync_InvalidProviderIndex_DoesNotReportAcceptance(int index) + { + var transport = new Mock(); + transport.Setup(t => t.SendAsync(It.IsAny(), It.IsAny>(), It.IsAny(), It.IsAny())) + .ReturnsAsync(new SendResult { Items = [new SendItemResult { Index = index, Status = MessageSendStatus.Accepted }] }); + await using var bus = new MessageBus(transport.Object); + var failure = await Assert.ThrowsAsync(() => bus.SendAsync(new Event(), cancellationToken: TestContext.Current.CancellationToken)); + Assert.Equal(MessageSendStatus.Unknown, Assert.Single(failure.Outcomes).Status); + } + + [Theory] + [InlineData(-1)] + [InlineData(1)] + public void EnsureAccepted_InvalidInputIndex_RejectsResult(int index) + { + var result = new SendResult { Items = [new SendItemResult { Index = index }] }; + Assert.Throws(() => result.EnsureAccepted(1)); + } + + [Fact] + public async Task SendBatchAsync_UnorderedPartialResults_PreservesEveryInputOutcome() + { + var transport = new Mock(); + transport.As().SetupGet(t => t.SupportedRoles).Returns(new HashSet { DestinationRole.Queue }); + transport.As().Setup(t => t.GetCapabilities(It.IsAny())).Returns(new TransportCapabilities { MaxBatchSize = 2 }); + transport.Setup(t => t.SendAsync(It.IsAny(), It.IsAny>(), It.IsAny(), It.IsAny())) + .ReturnsAsync(new SendResult { Items = [new SendItemResult { Index = 1, Status = MessageSendStatus.Accepted, MessageId = "broker-b" }, new SendItemResult { Index = 0, Status = MessageSendStatus.Rejected, ErrorCode = "Throttled", Retryable = true }] }); + await using var bus = new MessageBus(transport.Object, new MessageBusOptions { OwnsTransport = false }); + var failure = await Assert.ThrowsAsync(() => bus.SendBatchAsync([ + new MessageBatchItem(new Event(), "a"), new MessageBatchItem(new Event(), "b"), new MessageBatchItem(new Event(), "c") + ], cancellationToken: TestContext.Current.CancellationToken)); + Assert.Collection(failure.Outcomes, + a => { Assert.Equal("a", a.MessageId); Assert.Equal(MessageSendStatus.Rejected, a.Status); Assert.True(a.Retryable); Assert.Equal("Throttled", a.ErrorCode); }, + b => { Assert.Equal("b", b.MessageId); Assert.Equal(MessageSendStatus.Accepted, b.Status); }, + c => { Assert.Equal("c", c.MessageId); Assert.Equal(MessageSendStatus.NotAttempted, c.Status); }); + } + + public sealed record Event; +} diff --git a/tests/Foundatio.Tests/Messaging/ConfigurationExperienceTests.cs b/tests/Foundatio.Tests/Messaging/ConfigurationExperienceTests.cs new file mode 100644 index 000000000..7d8e899f9 --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/ConfigurationExperienceTests.cs @@ -0,0 +1,36 @@ +using System; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class ConfigurationExperienceTests +{ + [Fact] + public async Task MessagingOnly_InMemory_PersistsDelayedMessages() + { + var services = new ServiceCollection(); + services.AddFoundatio().ConfigureMessaging(m => m.UseInMemory().AddMessageType("event.v1", topic: "events")); + await using var provider = services.BuildServiceProvider(); + Assert.Null(provider.GetService()); + Assert.NotNull(provider.GetService()); + await provider.GetRequiredService().PublishAsync(new Event(), new MessagePublishOptions { Delay = TimeSpan.FromHours(1) }, TestContext.Current.CancellationToken); + } + + [Fact] + public async Task ConsumerRegistration_BindsWireNameAndProducerRoute() + { + var services = new ServiceCollection(); + services.AddFoundatio().ConfigureMessaging(m => m.UseInMemory().AddConsumer((_, _) => Task.CompletedTask, + o => { o.MessageTypeName = "event.v1"; o.Destination = "work"; })); + await using var provider = services.BuildServiceProvider(); + var router = provider.GetRequiredService(); + Assert.Equal("work", router.ResolveRoute(new MessageRouteContext { MessageType = typeof(Event), Role = MessageRouteRole.QueueDestination })); + Assert.Equal("event.v1", provider.GetRequiredService().GetName(typeof(Event))); + } + + public sealed record Event; +} diff --git a/tests/Foundatio.Tests/Messaging/DeliveryIntentTests.cs b/tests/Foundatio.Tests/Messaging/DeliveryIntentTests.cs new file mode 100644 index 000000000..c6f660a02 --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/DeliveryIntentTests.cs @@ -0,0 +1,260 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.AsyncEx; +using Foundatio.Messaging; +using Foundatio.Tests.Extensions; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class DeliveryIntentTests +{ + [Fact] + public async Task ConsumeAsync_MultipleFallbackTypesOnSameEndpoint_RejectsAmbiguousDispatchAsync() + { + var token = TestContext.Current.CancellationToken; + await using var bus = new MessageBus(new InMemoryMessageTransport()); + var options = new MessageConsumerOptions { Destination = "shared" }; + await using var first = await bus.ConsumeAsync((_, _) => Task.CompletedTask, options, token); + await Assert.ThrowsAsync(() => bus.ConsumeAsync((_, _) => Task.CompletedTask, options, token)); + } + + [Fact] + public async Task ReceiveAsync_WithoutHandler_CanSettleOrReturnWorkAsync() + { + var token = TestContext.Current.CancellationToken; + await using var bus = new MessageBus(new InMemoryMessageTransport()); + string id = await bus.SendAsync(new IntentEvent(), cancellationToken: token); + await using (var first = await bus.ReceiveAsync(cancellationToken: token)) + { + Assert.NotNull(first); + Assert.Equal(id, first.Id); + } + + await using var second = await bus.ReceiveAsync(cancellationToken: token); + Assert.NotNull(second); + Assert.Equal(id, second.Id); + Assert.Equal(2, second.Attempts); + await second.CompleteAsync(token); + Assert.Null(await bus.ReceiveAsync(cancellationToken: token)); + } + + [Fact] + public async Task ConsumeAsync_IgnoresPublishedMessagesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var received = new ConcurrentQueue(); + var sentSignal = new AsyncCountdownEvent(1); + await using var subscription = await bus.ConsumeAsync((message, _) => + { + received.Enqueue(message.Message.Data); + sentSignal.Signal(); + return Task.CompletedTask; + }, new MessageConsumerOptions(), cts.Token); + + Assert.Equal(DestinationRole.Queue, subscription.Source.Role); + + // A published event must not reach a sent-only handler (its group does not exist), and the command must. + await bus.PublishAsync(new IntentEvent { Data = "event" }, cancellationToken: cancellationToken); + await bus.SendAsync(new IntentEvent { Data = "command" }, cancellationToken: cancellationToken); + + await sentSignal.WaitAsync(TimeSpan.FromSeconds(2)); + await Task.Delay(TimeSpan.FromMilliseconds(150), cancellationToken); // let any (incorrect) event delivery arrive + + Assert.Equal(["command"], received); + } + + [Fact] + public async Task SubscribeAsync_IgnoresSentMessagesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var received = new ConcurrentQueue(); + var publishedSignal = new AsyncCountdownEvent(1); + await using var subscription = await bus.SubscribeAsync((message, _) => + { + received.Enqueue(message.Message.Data); + publishedSignal.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions(), cts.Token); + + Assert.Equal(DestinationRole.Subscription, subscription.Source.Role); + + // The command sits unconsumed on its queue (this handler never attached to it); the event must arrive. + await bus.SendAsync(new IntentEvent { Data = "command" }, cancellationToken: cancellationToken); + await bus.PublishAsync(new IntentEvent { Data = "event" }, cancellationToken: cancellationToken); + + await publishedSignal.WaitAsync(TimeSpan.FromSeconds(2)); + await Task.Delay(TimeSpan.FromMilliseconds(150), cancellationToken); // let any (incorrect) command delivery arrive + + Assert.Equal(["event"], received); + + var queueStats = await transport.GetStatsAsync(DestinationAddress.ForQueue("intent-event"), cancellationToken); + Assert.Equal(1, queueStats.Queued); // the command is still parked on the routed queue, untouched + } + + [Fact] + public async Task SubscribeAsync_OnQueueOnlyTransport_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var bus = new MessageBus(new QueueOnlyTransport()); + + await Assert.ThrowsAsync(() => bus.SubscribeAsync( + (_, _) => Task.CompletedTask, + new MessageSubscriptionOptions(), + cancellationToken)); + } + + [Fact] + public async Task ConsumeAsync_OnQueueOnlyTransport_ReceivesCommandsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new QueueOnlyTransport(); + await using var bus = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var received = new AsyncCountdownEvent(1); + await using var subscription = await bus.ConsumeAsync((message, _) => + { + Assert.Equal("command", message.Message.Data); + received.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + Assert.Equal(DestinationRole.Queue, subscription.Source.Role); + + await bus.SendAsync(new IntentEvent { Data = "command" }, cancellationToken: cancellationToken); + await received.WaitAsync(TimeSpan.FromSeconds(2)); + } + + [Fact] + public async Task ConsumeAsync_DuplicateHandlerForSameQueueAndType_ThrowsAsync() + { + var token = TestContext.Current.CancellationToken; + await using var bus = new MessageBus(new InMemoryMessageTransport()); + await using var consumer = await bus.ConsumeAsync((_, _) => Task.CompletedTask, cancellationToken: token); + await Assert.ThrowsAsync(() => bus.ConsumeAsync((_, _) => Task.CompletedTask, cancellationToken: token)); + } + + [Fact] + public async Task SubscribeAsync_UnnamedSubscription_DisposalDeletesResourceAsync() + { + var token = TestContext.Current.CancellationToken; + var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport); + var subscription = await bus.SubscribeAsync((_, _) => Task.CompletedTask, cancellationToken: token); + var source = subscription.Source; + Assert.True(await transport.ExistsAsync(source, token)); + await subscription.DisposeAsync(); + Assert.False(await transport.ExistsAsync(source, token)); + } + + [Fact] + public async Task SubscribeAsync_NamedSubscription_DisposalPreservesBacklogAsync() + { + var token = TestContext.Current.CancellationToken; + var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport); + var subscription = await bus.SubscribeAsync((_, _) => Task.CompletedTask, new() { Subscription = "billing" }, token); + var source = subscription.Source; + await subscription.DisposeAsync(); + await bus.PublishAsync(new IntentEvent(), cancellationToken: token); + Assert.True(await transport.ExistsAsync(source, token)); + Assert.Equal(1, (await transport.GetStatsAsync(source, token)).Queued); + } + + private sealed class IntentEvent + { + public string? Data { get; set; } + } + + // A transport that truly has no topic/subscription support, so the bus must not wire (or fake) a publish channel. + private sealed class QueueOnlyTransport : IMessageTransport, ISupportsPull, ITransportInfo + { + private readonly ConcurrentDictionary> _queues = new(StringComparer.Ordinal); + private readonly ConcurrentDictionary _inFlight = new(StringComparer.Ordinal); + + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => new HashSet { DestinationRole.Queue }; + public TransportCapabilities GetCapabilities(DestinationAddress destination) => TransportCapabilities.None; + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + if (destination.Role != DestinationRole.Queue) + throw new NotSupportedException("Queues only."); + + var queue = _queues.GetOrAdd(destination.Key, static _ => new ConcurrentQueue()); + var items = new List(messages.Count); + foreach (var message in messages) + { + string id = message.MessageId ?? Guid.NewGuid().ToString("N"); + queue.Enqueue(message with { MessageId = id }); + items.Add(new SendItemResult { MessageId = id }); + } + + return Task.FromResult(new SendResult { Items = items }); + } + + public async Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + { + var queue = _queues.GetOrAdd(source.Key, static _ => new ConcurrentQueue()); + var deadline = request.MaxWaitTime is { } wait && wait > TimeSpan.Zero ? DateTimeOffset.UtcNow.Add(wait) : DateTimeOffset.UtcNow; + var entries = new List(); + int max = Math.Max(1, request.MaxMessages); + + while (true) + { + while (entries.Count < max && queue.TryDequeue(out var message)) + { + string token = Guid.NewGuid().ToString("N"); + _inFlight[token] = message; + entries.Add(new TransportEntry + { + Id = message.MessageId!, + Destination = source, + Body = message.Body, + Headers = message.Headers, + Receipt = new Receipt { TransportState = token } + }); + } + + if (entries.Count > 0 || DateTimeOffset.UtcNow >= deadline) + return entries; + + await Task.Delay(TimeSpan.FromMilliseconds(15), ct).ConfigureAwait(false); + } + } + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) + { + if (entry.Receipt.TransportState is not string token || !_inFlight.TryRemove(token, out _)) + throw new ReceiptExpiredException(); + + return Task.CompletedTask; + } + + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) + { + if (entry.Receipt.TransportState is not string token || !_inFlight.TryRemove(token, out var message)) + throw new ReceiptExpiredException(); + + _queues.GetOrAdd(entry.Destination.Key, static _ => new ConcurrentQueue()).Enqueue(message); + return Task.CompletedTask; + } + + public ValueTask DisposeAsync() => ValueTask.CompletedTask; + } +} diff --git a/tests/Foundatio.Tests/Messaging/FailureHandlingTests.cs b/tests/Foundatio.Tests/Messaging/FailureHandlingTests.cs new file mode 100644 index 000000000..288912992 --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/FailureHandlingTests.cs @@ -0,0 +1,429 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Microsoft.Extensions.Time.Testing; +using Moq; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class FailureHandlingTests +{ + [Fact] + public async Task ConsumeAsync_SettledHandlerCleanup_DoesNotHoldConsumerCapacity() + { + var token = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport); + using var finishCleanup = new ManualResetEventSlim(); + var cleanupStarted = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var second = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var third = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + int calls = 0; + await using var subscription = await bus.ConsumeAsync((_, ct) => + { + int call = Interlocked.Increment(ref calls); + if (call == 1) + ct.Register(() => { cleanupStarted.TrySetResult(); finishCleanup.Wait(token); }); + else if (call == 2) second.TrySetResult(); + else third.TrySetResult(); + return Task.CompletedTask; + }, cancellationToken: token); + try + { + await bus.SendBatchAsync(new[] { new FailingItem(), new FailingItem(), new FailingItem() }, cancellationToken: token); + await cleanupStarted.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + await second.Task.WaitAsync(TimeSpan.FromSeconds(1), token); + await Task.Delay(100, token); + Assert.False(third.Task.IsCompleted); + } + finally { finishCleanup.Set(); } + await third.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + } + + [Fact] + public async Task ConsumeAsync_ConcurrentPulls_SharesCapacityAndCancelsPendingRequests() + { + var token = TestContext.Current.CancellationToken; + var transport = new Mock(); + var info = transport.As(); + info.SetupGet(t => t.SupportedRoles).Returns(new HashSet { DestinationRole.Queue }); + info.Setup(t => t.GetCapabilities(It.IsAny())).Returns(new TransportCapabilities + { + MaxReceiveBatchSize = 2, + MaxConcurrentReceives = 4 + }); + var full = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + int reserved = 0; + int calls = 0; + int cancelled = 0; + transport.Setup(t => t.ReceiveAsync(It.IsAny(), It.IsAny(), It.IsAny())) + .Returns(async (DestinationAddress _, ReceiveRequest request, CancellationToken ct) => + { + Assert.InRange(request.MaxMessages, 1, 2); + Interlocked.Increment(ref calls); + int total = Interlocked.Add(ref reserved, request.MaxMessages); + Assert.InRange(total, 1, 5); + if (total == 5) full.TrySetResult(); + try { await Task.Delay(Timeout.InfiniteTimeSpan, ct); } + finally { Interlocked.Increment(ref cancelled); } + return (IReadOnlyList)Array.Empty(); + }); + await using var bus = new MessageBus(transport.Object); + var subscription = await bus.ConsumeAsync((_, _) => Task.CompletedTask, + new MessageConsumerOptions { Destination = "work", MaxConcurrency = 5 }, token); + try + { + await full.Task.WaitAsync(TimeSpan.FromSeconds(2), token); + Assert.Equal(3, Volatile.Read(ref calls)); + } + finally { await subscription.DisposeAsync(); } + Assert.Equal(3, Volatile.Read(ref cancelled)); + } + + [Fact] + public async Task ConsumeAsync_BatchCapacityReturns_ReceivesWithoutWaitingForCollectionTimeout() + { + var token = TestContext.Current.CancellationToken; + var transport = new Mock(); + var info = transport.As(); + info.SetupGet(t => t.SupportedRoles).Returns(new HashSet { DestinationRole.Queue }); + info.Setup(t => t.GetCapabilities(It.IsAny())).Returns(new TransportCapabilities + { + MaxReceiveBatchSize = 2, + ReceiveBatchDelay = TimeSpan.FromSeconds(10) + }); + var contexts = new ConcurrentDictionary(); + var full = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var replacement = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + int next = 0; + transport.Setup(t => t.CompleteAsync(It.IsAny(), It.IsAny())).Returns(Task.CompletedTask); + transport.Setup(t => t.ReceiveAsync(It.IsAny(), It.IsAny(), It.IsAny())) + .Returns((DestinationAddress source, ReceiveRequest request, CancellationToken _) => + { + var entries = new List(); + for (int i = 0; i < request.MaxMessages && next < 4; i++) + entries.Add(new TransportEntry { Id = (++next).ToString(), Destination = source, Body = ReadOnlyMemory.Empty, Receipt = default }); + return Task.FromResult>(entries); + }); + await using var bus = new MessageBus(transport.Object); + await using var subscription = await bus.ConsumeAsync((context, _) => + { + contexts[context.BrokerMessageId] = context; + if (context.BrokerMessageId == "2") full.TrySetResult(); + if (context.BrokerMessageId == "4") replacement.TrySetResult(); + return Task.CompletedTask; + }, new MessageConsumerOptions { Destination = "work", MaxConcurrency = 2, AckMode = AckMode.Manual }, token); + await full.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + await contexts["1"].CompleteAsync(token); + await Task.Delay(100, token); + Assert.False(replacement.Task.IsCompleted); + await contexts["2"].CompleteAsync(token); + await replacement.Task.WaitAsync(TimeSpan.FromSeconds(1), token); + foreach (var context in contexts.Values) await context.CompleteAsync(token); + } + + [Fact] + public async Task ConsumeAsync_BatchedPulls_FillsConcurrencyAndDoesNotWaitForSlowHandler() + { + var token = TestContext.Current.CancellationToken; + var transport = new Mock(); + var info = transport.As(); + info.SetupGet(t => t.SupportedRoles).Returns(new HashSet { DestinationRole.Queue }); + info.Setup(t => t.GetCapabilities(It.IsAny())).Returns(new TransportCapabilities + { + MaxReceiveBatchSize = 2, + ReceiveBatchDelay = TimeSpan.FromMilliseconds(1) + }); + var contexts = new ConcurrentDictionary(); + var full = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var replacement = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + int next = 0; + transport.Setup(t => t.CompleteAsync(It.IsAny(), It.IsAny())).Returns(Task.CompletedTask); + transport.Setup(t => t.ReceiveAsync(It.IsAny(), It.IsAny(), It.IsAny())) + .Returns((DestinationAddress source, ReceiveRequest request, CancellationToken _) => + { + Assert.InRange(request.MaxMessages, 1, 2); + var entries = new List(); + for (int i = 0; i < request.MaxMessages && next < 9; i++) + entries.Add(new TransportEntry { Id = (++next).ToString(), Destination = source, Body = ReadOnlyMemory.Empty, Receipt = default }); + return Task.FromResult>(entries); + }); + await using var bus = new MessageBus(transport.Object); + await using var subscription = await bus.ConsumeAsync((context, _) => + { + contexts[context.BrokerMessageId] = context; + if (contexts.Count == 8) full.TrySetResult(); + if (context.BrokerMessageId == "9") replacement.TrySetResult(); + return Task.CompletedTask; + }, new MessageConsumerOptions { Destination = "work", MaxConcurrency = 8, AckMode = AckMode.Manual }, token); + await full.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + Assert.Equal(8, next); + Assert.False(replacement.Task.IsCompleted); + await contexts["2"].CompleteAsync(token); + await replacement.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + Assert.False(contexts["1"].IsHandled); + foreach (var context in contexts.Values) + await context.CompleteAsync(token); + } + + [Fact] + public async Task ConsumeAsync_TransportReceiveLimit_CapsEachPull() + { + var token = TestContext.Current.CancellationToken; + var transport = new Mock(); + var info = transport.As(); + info.SetupGet(t => t.SupportedRoles).Returns(new HashSet { DestinationRole.Queue }); + info.Setup(t => t.GetCapabilities(It.IsAny())).Returns(new TransportCapabilities { MaxReceiveBatchSize = 2 }); + var received = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + transport.Setup(t => t.ReceiveAsync(It.IsAny(), It.IsAny(), It.IsAny())) + .Returns((DestinationAddress _, ReceiveRequest request, CancellationToken _) => + { + received.TrySetResult(request.MaxMessages); + return Task.FromResult>(Array.Empty()); + }); + await using var bus = new MessageBus(transport.Object); + await using var subscription = await bus.ConsumeAsync((_, _) => Task.CompletedTask, + new MessageConsumerOptions { Destination = "work", MaxConcurrency = 8 }, token); + Assert.Equal(2, await received.Task.WaitAsync(TimeSpan.FromSeconds(5), token)); + } + + [Theory] + [InlineData(TopologyMode.Ensure)] + [InlineData(TopologyMode.Validate)] + public async Task DeadLetterFallback_HonorsTopologyBeforeSendingAndCompletingAsync(TopologyMode mode) + { + var token = TestContext.Current.CancellationToken; + var transport = new Mock(); + var entry = new TransportEntry { Id = "source", Destination = DestinationAddress.ForQueue("work"), Body = new byte[] { 1 }, Receipt = default }; + transport.Setup(t => t.SendAsync(It.IsAny(), It.IsAny>(), It.IsAny(), token)) + .ReturnsAsync(new SendResult { Items = new[] { new SendItemResult { MessageId = "parked" } } }); + if (mode == TopologyMode.Validate) + { + await Assert.ThrowsAsync(() => MessageContext.DeadLetterAsync(transport.Object, entry, "failure", null, + Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance, token, mode)); + transport.Verify(t => t.SendAsync(It.IsAny(), It.IsAny>(), It.IsAny(), token), Times.Never); + transport.Verify(t => t.CompleteAsync(entry, token), Times.Never); + } + else + { + await MessageContext.DeadLetterAsync(transport.Object, entry, "failure", null, Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance, token, mode); + transport.Verify(t => t.EnsureAsync(It.Is>(d => d[0].Address.Name == "work.deadletter"), token), Times.Once); + transport.Verify(t => t.CompleteAsync(entry, token), Times.Once); + } + } + + [Fact] + public async Task ConsumeAsync_WithManualAcknowledgement_HoldsConcurrencySlotUntilSettlementAsync() + { + var token = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport); + var first = new TaskCompletionSource>(TaskCreationOptions.RunContinuationsAsynchronously); + var second = new TaskCompletionSource>(TaskCreationOptions.RunContinuationsAsynchronously); + int count = 0; + await using var consumer = await bus.ConsumeAsync((context, _) => + { + (Interlocked.Increment(ref count) == 1 ? first : second).TrySetResult(context); + return Task.CompletedTask; + }, new MessageConsumerOptions { AckMode = AckMode.Manual }, token); + await bus.SendBatchAsync(new[] { new FailingItem(), new FailingItem() }, cancellationToken: token); + var message = await first.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + await Task.Delay(100, token); + Assert.False(second.Task.IsCompleted); + await message.CompleteAsync(token); + var next = await second.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + await next.CompleteAsync(token); + } + + [Fact] + public async Task ConsumeAsync_WhenLeaseRenewalFails_CancelsHandlerWithoutSettlingAsync() + { + var time = new FakeTimeProvider(); + var transport = new Mock(); + var renewal = transport.As(); + var entry = new TransportEntry + { + Id = "leased-message", + Destination = DestinationAddress.ForQueue("work"), + Body = new byte[] { 1 }, + LockExpiresUtc = time.GetUtcNow().AddSeconds(10), + Receipt = default + }; + transport.SetupSequence(t => t.ReceiveAsync(It.IsAny(), It.IsAny(), It.IsAny())) + .ReturnsAsync(new[] { entry }) + .ReturnsAsync(Array.Empty()); + renewal.Setup(t => t.RenewLockAsync(entry, It.IsAny(), It.IsAny())) + .ThrowsAsync(new ReceiptExpiredException()); + var started = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var cancelled = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var bus = new MessageBus(transport.Object, new MessageBusOptions { TimeProvider = time }); + await using var consumer = await bus.ConsumeAsync(async (context, token) => + { + started.TrySetResult(); + try + { + await Task.Delay(Timeout.InfiniteTimeSpan, token); + } + catch (OperationCanceledException) when (token.IsCancellationRequested) + { + Assert.True(context.CancellationToken.IsCancellationRequested); + await Assert.ThrowsAnyAsync(() => context.CompleteAsync(TestContext.Current.CancellationToken)); + cancelled.TrySetResult(); + } + }, new MessageConsumerOptions { Destination = "work" }, TestContext.Current.CancellationToken); + await started.Task.WaitAsync(TimeSpan.FromSeconds(5), TestContext.Current.CancellationToken); + time.Advance(TimeSpan.FromSeconds(6)); + await cancelled.Task.WaitAsync(TimeSpan.FromSeconds(5), TestContext.Current.CancellationToken); + transport.Verify(t => t.CompleteAsync(entry, It.IsAny()), Times.Never); + transport.Verify(t => t.AbandonAsync(entry, It.IsAny()), Times.Never); + } + + [Fact] + public async Task CompleteAsync_WhenTransportFails_RemainsUnsettledAndCanRetryAsync() + { + var token = TestContext.Current.CancellationToken; + var transport = new Mock(); + var entry = new TransportEntry { Id = "message", Destination = DestinationAddress.ForQueue("work"), Body = new byte[] { 1 }, Receipt = default }; + transport.SetupSequence(t => t.CompleteAsync(entry, token)) + .ThrowsAsync(new TimeoutException("broker unavailable")) + .Returns(Task.CompletedTask); + var context = new MessageContext(transport.Object, entry, token); + await Assert.ThrowsAsync(() => context.CompleteAsync(token)); + Assert.False(context.IsHandled); + await context.CompleteAsync(token); + Assert.True(context.IsHandled); + await context.CompleteAsync(token); + transport.Verify(t => t.CompleteAsync(entry, token), Times.Exactly(2)); + } + + [Fact] + public async Task RejectAsync_WhenDeadLetterStorageFails_DoesNotDeleteOriginalAsync() + { + var token = TestContext.Current.CancellationToken; + var transport = new Mock(); + var entry = new TransportEntry { Id = "message", Destination = DestinationAddress.ForQueue("work"), Body = new byte[] { 1 }, Receipt = default }; + transport.Setup(t => t.SendAsync(It.IsAny(), It.IsAny>(), It.IsAny(), token)) + .ThrowsAsync(new TimeoutException("DLQ unavailable")); + var context = new MessageContext(transport.Object, entry, token); + await Assert.ThrowsAsync(() => context.RejectAsync(new() { Terminal = true }, token)); + transport.Verify(t => t.CompleteAsync(entry, token), Times.Never); + Assert.False(context.IsHandled); + } + + [Fact] + public async Task DeadLetterOn_MatchingException_DeadLettersOnFirstAttemptAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport); + int attempts = 0; + + var options = new MessageConsumerOptions { MaxAttempts = 5 }; + options.DeadLetterOn(); + + await using var subscription = await bus.ConsumeAsync((_, _) => + { + Interlocked.Increment(ref attempts); + throw new ArgumentException("bad data a retry can never fix"); + }, options, cancellationToken); + + await bus.SendAsync(new FailingItem { Data = "poison" }, cancellationToken: cancellationToken); + + var stats = await WaitForDeadLetterAsync(transport, "failing-item", cancellationToken); + Assert.Equal(1, stats.Deadletter); + Assert.Equal(1, Volatile.Read(ref attempts)); // never retried + + var dead = Assert.Single(await transport.PeekDeadLetteredAsync(DestinationAddress.ForQueue("failing-item"), new DeadLetterQuery { Limit = 10 }, cancellationToken)); + Assert.Equal("unrecoverable:ArgumentException", dead.Headers[KnownHeaders.DeadLetterReason]); + } + + [Fact] + public async Task DeadLetterWhen_GlobalPolicy_AppliesWhenSubscriptionDoesNotOverrideAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport, new MessageBusOptions + { + RetryPolicy = new RetryPolicy { DeadLetterWhen = ex => ex is InvalidOperationException } + }); + int attempts = 0; + + await using var subscription = await bus.ConsumeAsync((_, _) => + { + Interlocked.Increment(ref attempts); + throw new InvalidOperationException("unrecoverable per global policy"); + }, cancellationToken: cancellationToken); + + await bus.SendAsync(new FailingItem { Data = "poison" }, cancellationToken: cancellationToken); + + var stats = await WaitForDeadLetterAsync(transport, "failing-item", cancellationToken); + Assert.Equal(1, stats.Deadletter); + Assert.Equal(1, Volatile.Read(ref attempts)); + } + + [Fact] + public async Task DeadLetter_StampsForensicsHeadersAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport); + + await using var subscription = await bus.ConsumeAsync((_, _) => + throw new InvalidOperationException("the failure detail"), + new MessageConsumerOptions { MaxAttempts = 1 }, cancellationToken); + + await bus.SendAsync(new FailingItem { Data = "doomed" }, cancellationToken: cancellationToken); + + await WaitForDeadLetterAsync(transport, "failing-item", cancellationToken); + var dead = Assert.Single(await transport.PeekDeadLetteredAsync(DestinationAddress.ForQueue("failing-item"), new DeadLetterQuery { Limit = 10 }, cancellationToken)); + + Assert.Equal(typeof(InvalidOperationException).FullName, dead.Headers[KnownHeaders.DeadLetterExceptionType]); + Assert.Equal("the failure detail", dead.Headers[KnownHeaders.DeadLetterExceptionMessage]); + Assert.NotEmpty(dead.Headers[KnownHeaders.DeadLetterExceptionStackTrace]); + Assert.Equal("failing-item", dead.Headers[KnownHeaders.DeadLetterOriginalDestination]); + Assert.NotEmpty(dead.Headers[KnownHeaders.DeadLetterFailedAt]); + // The exhausted count is forensics-only: message.attempts is left alone so a replayed message starts fresh. + Assert.Equal("1", dead.Headers[KnownHeaders.DeadLetterAttempts]); + Assert.False(dead.Headers.ContainsKey(KnownHeaders.Attempts)); + } + + [Fact] + public void DefaultBackoff_MatchesTheConvergedCurve() + { + // Immediate first retry, then 10s/20s/30s (capped) with ±20% jitter. + Assert.Equal(TimeSpan.Zero, RetryPolicy.DefaultBackoff(1)); + + foreach ((int attempt, double expectedSeconds) in new[] { (2, 10d), (3, 20d), (4, 30d), (7, 30d) }) + { + var delay = RetryPolicy.DefaultBackoff(attempt); + Assert.InRange(delay.TotalSeconds, expectedSeconds * 0.8, expectedSeconds * 1.2); + } + + // The default policy uses the curve. + Assert.Same(RetryPolicy.DefaultBackoff, new RetryPolicy().Backoff); + } + + private static async Task WaitForDeadLetterAsync(InMemoryMessageTransport transport, string destination, CancellationToken cancellationToken) + { + var address = DestinationAddress.ForQueue(destination); + var stats = await transport.GetStatsAsync(address, cancellationToken); + long deadline = Environment.TickCount64 + 10_000; + while (stats.Deadletter == 0 && Environment.TickCount64 < deadline) + { + await Task.Delay(25, cancellationToken); + stats = await transport.GetStatsAsync(address, cancellationToken); + } + + return stats; + } + + [MessageRoute("failing-item")] + private sealed class FailingItem + { + public string? Data { get; set; } + } +} diff --git a/tests/Foundatio.Tests/Messaging/InMemoryMessageBusTests.cs b/tests/Foundatio.Tests/Messaging/InMemoryMessageBusTests.cs deleted file mode 100644 index d94a0a9c8..000000000 --- a/tests/Foundatio.Tests/Messaging/InMemoryMessageBusTests.cs +++ /dev/null @@ -1,308 +0,0 @@ -using System; -using System.Threading.Tasks; -using Foundatio.AsyncEx; -using Foundatio.Messaging; -using Foundatio.Tests.Extensions; -using Xunit; - -namespace Foundatio.Tests.Messaging; - -public class InMemoryMessageBusTests : MessageBusTestBase, IDisposable -{ - private IMessageBus? _messageBus; - - public InMemoryMessageBusTests(ITestOutputHelper output) : base(output) { } - - protected override IMessageBus GetMessageBus(Func? config = null) - { - if (_messageBus != null) - return _messageBus; - - _messageBus = new InMemoryMessageBus(o => - { - o.LoggerFactory(Log); - if (config != null) - config(o.Target); - - return o; - }); - return _messageBus; - } - - [Fact] - public override Task CanUseMessageOptionsAsync() - { - return base.CanUseMessageOptionsAsync(); - } - - [Fact] - public override Task CanSendMessageAsync() - { - return base.CanSendMessageAsync(); - } - - [Fact] - public override Task CanHandleNullMessageAsync() - { - return base.CanHandleNullMessageAsync(); - } - - [Fact] - public override Task CanSendDerivedMessageAsync() - { - return base.CanSendDerivedMessageAsync(); - } - - [Fact] - public override Task CanSendMappedMessageAsync() - { - return base.CanSendMappedMessageAsync(); - } - - [Fact] - public override Task CanSendDelayedMessageAsync() - { - return base.CanSendDelayedMessageAsync(); - } - - [Fact] - public override Task CanSubscribeConcurrentlyAsync() - { - return base.CanSubscribeConcurrentlyAsync(); - } - - [Fact] - public override Task CanReceiveMessagesConcurrentlyAsync() - { - return base.CanReceiveMessagesConcurrentlyAsync(); - } - - [Fact] - public override Task CanSendMessageToMultipleSubscribersAsync() - { - return base.CanSendMessageToMultipleSubscribersAsync(); - } - - [Fact] - public override Task CanTolerateSubscriberFailureAsync() - { - return base.CanTolerateSubscriberFailureAsync(); - } - - [Fact] - public override Task WillOnlyReceiveSubscribedMessageTypeAsync() - { - return base.WillOnlyReceiveSubscribedMessageTypeAsync(); - } - - [Fact] - public override Task WillReceiveDerivedMessageTypesAsync() - { - return base.WillReceiveDerivedMessageTypesAsync(); - } - - [Fact] - public override Task CanSubscribeToAllMessageTypesAsync() - { - return base.CanSubscribeToAllMessageTypesAsync(); - } - - [Fact] - public override Task CanSubscribeToRawMessagesAsync() - { - return base.CanSubscribeToRawMessagesAsync(); - } - - [Fact] - public override Task CanCancelSubscriptionAsync() - { - return base.CanCancelSubscriptionAsync(); - } - - [Fact] - public override Task WontKeepMessagesWithNoSubscribersAsync() - { - return base.WontKeepMessagesWithNoSubscribersAsync(); - } - - [Fact] - public override Task CanReceiveFromMultipleSubscribersAsync() - { - return base.CanReceiveFromMultipleSubscribersAsync(); - } - - [Fact] - public override Task CanDisposeWithNoSubscribersOrPublishersAsync() - { - return base.CanDisposeWithNoSubscribersOrPublishersAsync(); - } - - [Fact] - public override Task CanHandlePoisonedMessageAsync() - { - return base.CanHandlePoisonedMessageAsync(); - } - - [Fact] - public override Task DisposeAsync_CalledMultipleTimes_IsIdempotentAsync() - { - return base.DisposeAsync_CalledMultipleTimes_IsIdempotentAsync(); - } - - [Fact] - public override Task DisposeAsync_WhilePublishing_CompletesWithoutDeadlockAsync() - { - return base.DisposeAsync_WhilePublishing_CompletesWithoutDeadlockAsync(); - } - - [Fact] - public override Task DisposeAsync_WithNoSubscribersOrPublishers_CompletesWithoutExceptionAsync() - { - return base.DisposeAsync_WithNoSubscribersOrPublishers_CompletesWithoutExceptionAsync(); - } - - [Fact] - public override Task PublishAsync_AfterDispose_ThrowsMessageBusExceptionAsync() - { - return base.PublishAsync_AfterDispose_ThrowsMessageBusExceptionAsync(); - } - - [Fact] - public override Task PublishAsync_WithCancellation_ThrowsOperationCanceledExceptionAsync() - { - return base.PublishAsync_WithCancellation_ThrowsOperationCanceledExceptionAsync(); - } - - [Fact] - public override Task PublishAsync_WithDelayedMessageAndDisposeBeforeDelivery_DiscardsMessageAsync() - { - return base.PublishAsync_WithDelayedMessageAndDisposeBeforeDelivery_DiscardsMessageAsync(); - } - - [Fact] - public override Task PublishAsync_WithSerializationFailure_ThrowsSerializerExceptionAsync() - { - return base.PublishAsync_WithSerializationFailure_ThrowsSerializerExceptionAsync(); - } - - [Fact] - public override Task SubscribeAsync_AfterDispose_ThrowsMessageBusExceptionAsync() - { - return base.SubscribeAsync_AfterDispose_ThrowsMessageBusExceptionAsync(); - } - - [Fact] - public override Task SubscribeAsync_CancelledToken_DoesNotTearDownInfrastructureAsync() - { - return base.SubscribeAsync_CancelledToken_DoesNotTearDownInfrastructureAsync(); - } - - [Fact] - public override Task SubscribeAsync_WithCancellation_ThrowsOperationCanceledExceptionAsync() - { - return base.SubscribeAsync_WithCancellation_ThrowsOperationCanceledExceptionAsync(); - } - - [Fact] - public override Task SubscribeAsync_WithDeserializationFailure_SkipsMessageAsync() - { - return base.SubscribeAsync_WithDeserializationFailure_SkipsMessageAsync(); - } - - [Fact] - public override Task SubscribeAsync_WithValidThenPoisonedMessage_DeliversOnlyValidMessageAsync() - { - return base.SubscribeAsync_WithValidThenPoisonedMessage_DeliversOnlyValidMessageAsync(); - } - - [Fact] - public override Task PublishAsync_WithDeliveryDelayExtension_DelaysDeliveryAsync() - { - return base.PublishAsync_WithDeliveryDelayExtension_DelaysDeliveryAsync(); - } - - [Fact] - public override Task PublishAsync_WithUniqueId_PropagatesUniqueIdToSubscriberAsync() - { - return base.PublishAsync_WithUniqueId_PropagatesUniqueIdToSubscriberAsync(); - } - - [Fact] - public override Task SubscribeAsync_ToRawIMessage_CanAccessAllPropertiesAsync() - { - return base.SubscribeAsync_ToRawIMessage_CanAccessAllPropertiesAsync(); - } - - [Fact] - public override Task SubscribeAsync_WithCancellationTokenHandler_ReceivesCancellationTokenAsync() - { - return base.SubscribeAsync_WithCancellationTokenHandler_ReceivesCancellationTokenAsync(); - } - - [Fact] - public async Task CanCheckMessageCounts() - { - var messageBus = new InMemoryMessageBus(o => o.LoggerFactory(Log)); - await messageBus.PublishAsync(new SimpleMessageA - { - Data = "Hello" - }, cancellationToken: TestCancellationToken); - Assert.Equal(1, messageBus.MessagesSent); - Assert.Equal(1, messageBus.GetMessagesSent()); - Assert.Equal(0, messageBus.GetMessagesSent()); - } - - [Fact] - public async Task SendMessageToSubscribersAsync_WithNullMessageType_DeliversToRawSubscribersOnly() - { - // Arrange - var messageBus = new TestableInMemoryMessageBus(o => o.LoggerFactory(Log)); - - var rawReceived = new AsyncCountdownEvent(1); - var typedReceived = new AsyncCountdownEvent(1); - - await messageBus.SubscribeAsync(msg => - { - Assert.Null(msg.Type); - Assert.Null(msg.ClrType); - Assert.False(msg.Data.IsEmpty); - rawReceived.Signal(); - }, TestCancellationToken); - - await messageBus.SubscribeAsync(_ => - { - typedReceived.Signal(); - }, TestCancellationToken); - - var message = new Message("test payload"u8.ToArray(), _ => "test payload") - { - Type = null, - ClrType = null - }; - - // Act - await messageBus.TestSendMessageToSubscribersAsync(message); - - // Assert - await rawReceived.WaitAsync(TimeSpan.FromSeconds(2)); - Assert.Equal(0, rawReceived.CurrentCount); - - await Task.Delay(100, TestCancellationToken); - Assert.Equal(1, typedReceived.CurrentCount); - } - - public void Dispose() - { - _messageBus?.Dispose(); - _messageBus = null; - } -} - -internal class TestableInMemoryMessageBus : InMemoryMessageBus -{ - public TestableInMemoryMessageBus(Builder config) - : base(config) { } - - public Task TestSendMessageToSubscribersAsync(IMessage message) - => SendMessageToSubscribersAsync(message); -} diff --git a/tests/Foundatio.Tests/Messaging/InMemoryMessageTransportTests.cs b/tests/Foundatio.Tests/Messaging/InMemoryMessageTransportTests.cs new file mode 100644 index 000000000..f94041da1 --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/InMemoryMessageTransportTests.cs @@ -0,0 +1,162 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Microsoft.Extensions.Time.Testing; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class InMemoryMessageTransportTests : MessageTransportConformanceTests +{ + public InMemoryMessageTransportTests(ITestOutputHelper output) : base(output) { } + + protected override IMessageTransport CreateTransport() + { + return new InMemoryMessageTransport(); + } + + [Fact] + public async Task CompletedDeliveries_KeepVisibilityTimerResourcesBounded() + { + var time = new TrackingTimeProvider(); + await using (var transport = new InMemoryMessageTransport(time)) + { + var queue = DestinationAddress.ForQueue("timer-resources"); + for (int i = 0; i < 100; i++) + { + await transport.SendAsync(queue, [new TransportMessage { Body = new byte[] { 1 } }], new(), TestCancellationToken); + var entry = Assert.Single(await transport.ReceiveAsync(queue, new(), TestCancellationToken)); + await transport.RenewLockAsync(entry, TimeSpan.FromMinutes(2), TestCancellationToken); + await transport.CompleteAsync(entry, TestCancellationToken); + } + + Assert.InRange(time.TimerCount, 0, 2); + time.Clock.Advance(TimeSpan.FromMinutes(2)); + int callbacks = time.CallbackCount; + time.Clock.Advance(TimeSpan.FromDays(1)); + Assert.Equal(callbacks, time.CallbackCount); + } + + Assert.Equal(0, time.TimerCount); + } + + [Fact] + public async Task RenewedVisibility_WakesBlockedReceiverOnlyAfterCurrentLeaseExpires() + { + var time = new TrackingTimeProvider(); + await using var transport = new InMemoryMessageTransport(time); + var queue = DestinationAddress.ForQueue("renewed-visibility"); + for (int iteration = 0; iteration < 2; iteration++) + { + await transport.SendAsync(queue, [new TransportMessage { Body = new byte[] { 1 } }], new(), TestCancellationToken); + var entry = Assert.Single(await transport.ReceiveAsync(queue, new(), TimeSpan.FromSeconds(1), TestCancellationToken)); + await transport.RenewLockAsync(entry, TimeSpan.FromSeconds(2), TestCancellationToken); + + var pending = transport.ReceiveAsync(queue, new() { MaxWaitTime = TimeSpan.FromSeconds(10) }, TestCancellationToken); + time.Clock.Advance(TimeSpan.FromMilliseconds(1200)); + Assert.False(pending.IsCompleted); + time.Clock.Advance(TimeSpan.FromSeconds(1)); + var redelivered = Assert.Single(await pending.WaitAsync(TimeSpan.FromSeconds(5), TestCancellationToken)); + Assert.Equal(entry.Id, redelivered.Id); + Assert.Equal(2, redelivered.DeliveryCount); + await Assert.ThrowsAsync(() => transport.CompleteAsync(entry, TestCancellationToken)); + await Assert.ThrowsAsync(() => transport.AbandonAsync(entry, TestCancellationToken)); + await Assert.ThrowsAsync(() => transport.RenewLockAsync(entry, TimeSpan.FromSeconds(2), TestCancellationToken)); + await transport.CompleteAsync(redelivered, TestCancellationToken); + time.Clock.Advance(TimeSpan.FromSeconds(1)); + } + } + + [Fact] + public void DestinationAddress_KeyEncodesTopicAndSubscription() + { + var destination = DestinationAddress.ForSubscription("orders", "sub-a"); + Assert.Equal("orders/sub-a", destination.Key); + Assert.Equal("orders", destination.Topic); + Assert.Equal("sub-a", destination.Name); + Assert.Equal(DestinationRole.Subscription, destination.Role); + + // A bare (non-subscription) destination has no topic and a bare key. + var bare = DestinationAddress.ForQueue("orders"); + Assert.Null(bare.Topic); + Assert.Equal("orders", bare.Key); + Assert.NotEqual(destination, bare); + } + + [Fact] + public void MessageHeaders_SerializeToJson_RoundTripsCaseInsensitively() + { + var headers = MessageHeaders.Create([ + new KeyValuePair("Message.Type", "order.created"), + new KeyValuePair("tenant", "acme") + ]); + + // The shared codec both transports use preserves the case-insensitive contract across the wire. + var roundTripped = MessageHeaders.DeserializeFromJson(MessageHeaders.SerializeToJson(headers)); + Assert.Equal("order.created", roundTripped["MESSAGE.TYPE"]); + Assert.Equal("acme", roundTripped["tenant"]); + + Assert.Empty(MessageHeaders.DeserializeFromJson(null)); + Assert.Empty(MessageHeaders.DeserializeFromJson("")); + } + + [Fact] + public void MessageHeaders_AreImmutableAndCaseInsensitive() + { + var source = new Dictionary(StringComparer.Ordinal) + { + ["message.type"] = "order.created" + }; + + var headers = MessageHeaders.Create(source); + source["message.type"] = "changed"; + + Assert.Equal("order.created", headers["MESSAGE.TYPE"]); + Assert.Equal("order.created", headers.GetValueOrDefault("Message.Type")); + Assert.True(headers.ContainsKey("MESSAGE.TYPE")); + + var updated = headers.ToBuilder() + .Set("TraceParent", "00-123") + .SetIfMissing("traceparent", "ignored") + .Build(); + + Assert.Equal("00-123", updated["traceparent"]); + Assert.False(headers.ContainsKey("traceparent")); + } + + private sealed class TrackingTimeProvider : TimeProvider + { + private int _timerCount; + private int _callbackCount; + public FakeTimeProvider Clock { get; } = new(); + public int TimerCount => Volatile.Read(ref _timerCount); + public int CallbackCount => Volatile.Read(ref _callbackCount); + public override DateTimeOffset GetUtcNow() => Clock.GetUtcNow(); + public override ITimer CreateTimer(TimerCallback callback, object? state, TimeSpan dueTime, TimeSpan period) + { + Interlocked.Increment(ref _timerCount); + var timer = Clock.CreateTimer(value => { Interlocked.Increment(ref _callbackCount); callback(value); }, state, dueTime, period); + return new TrackedTimer(timer, () => Interlocked.Decrement(ref _timerCount)); + } + } + + private sealed class TrackedTimer(ITimer timer, Action disposed) : ITimer + { + private int _disposed; + public bool Change(TimeSpan dueTime, TimeSpan period) => timer.Change(dueTime, period); + public void Dispose() + { + if (Interlocked.Exchange(ref _disposed, 1) != 0) + return; + timer.Dispose(); + disposed(); + } + public ValueTask DisposeAsync() + { + Dispose(); + return ValueTask.CompletedTask; + } + } +} diff --git a/tests/Foundatio.Tests/Messaging/LegacyMessageBusAdapterTests.cs b/tests/Foundatio.Tests/Messaging/LegacyMessageBusAdapterTests.cs new file mode 100644 index 000000000..d78053812 --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/LegacyMessageBusAdapterTests.cs @@ -0,0 +1,102 @@ +using System; +using System.Collections.Concurrent; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.AsyncEx; +using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; +using Foundatio.Tests.Extensions; +using Microsoft.Extensions.DependencyInjection; +using Xunit; +using IMessageBus = Foundatio.Messaging.IMessageBus; + +namespace Foundatio.Tests.Messaging; + +public class LegacyMessageBusAdapterTests +{ + [Fact] + public async Task OldStyleSubscribeAndPublish_WorkOverTheNewBusAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var bus = new MessageBus(new InMemoryMessageTransport()); + await using var adapter = new LegacyMessageBusAdapter(bus); + var received = new AsyncCountdownEvent(2); + var payloads = new ConcurrentQueue(); + + // Two old-style subscribers: BOTH must receive every published message (the old fan-out semantics). + await adapter.SubscribeAsync((message, _) => + { + payloads.Enqueue(message.Data); + received.Signal(); + return Task.CompletedTask; + }, cancellationToken); + + await adapter.SubscribeAsync((message, _) => + { + payloads.Enqueue(message.Data); + received.Signal(); + return Task.CompletedTask; + }, cancellationToken); + + // Old-style publish extension with MessageOptions. + await adapter.PublishAsync(new LegacyEvent { Data = "hello" }, new MessageOptions { CorrelationId = "abc" }, cancellationToken); + + await received.WaitAsync(TimeSpan.FromSeconds(2)); + Assert.Equal(2, payloads.Count); + Assert.All(payloads, data => Assert.Equal("hello", data)); + } + + [Fact] + public async Task NewBusSubscribers_ReceiveAdapterPublishesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var bus = new MessageBus(new InMemoryMessageTransport()); + await using var adapter = new LegacyMessageBusAdapter(bus); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var received = new AsyncCountdownEvent(1); + + // Migrated code on the NEW api and unmigrated code on the adapter interoperate: same bus, same topics. + await using var subscription = await bus.SubscribeAsync((context, _) => + { + Assert.Equal("bridged", context.Message.Data); + received.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + await adapter.PublishAsync(new LegacyEvent { Data = "bridged" }, cancellationToken: cancellationToken); + await received.WaitAsync(TimeSpan.FromSeconds(2)); + } + + [Fact] + public async Task AddLegacyAdapter_ResolvesOldInterfacesFromDiAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var services = new ServiceCollection(); + services.AddFoundatio() + .Messaging.UseInMemory() + .AddLegacyAdapter(); + + await using var provider = services.BuildServiceProvider(); + var legacyBus = provider.GetRequiredService(); + Assert.IsType(legacyBus); + Assert.Same(legacyBus, provider.GetRequiredService()); + Assert.Same(legacyBus, provider.GetRequiredService()); + + var received = new AsyncCountdownEvent(1); + await legacyBus.SubscribeAsync((message, _) => + { + received.Signal(); + return Task.CompletedTask; + }, cancellationToken); + + // The adapter and the new bus resolved from the SAME container share the transport. + await provider.GetRequiredService().PublishAsync(new LegacyEvent { Data = "di" }, cancellationToken: cancellationToken); + await received.WaitAsync(TimeSpan.FromSeconds(2)); + } + + private sealed class LegacyEvent + { + public string? Data { get; set; } + } +} diff --git a/tests/Foundatio.Tests/Messaging/MessageDeliveryLeaseTests.cs b/tests/Foundatio.Tests/Messaging/MessageDeliveryLeaseTests.cs new file mode 100644 index 000000000..e99d8f4c2 --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/MessageDeliveryLeaseTests.cs @@ -0,0 +1,132 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Time.Testing; +using Moq; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class MessageDeliveryLeaseTests +{ + [Fact] + public async Task SettledBeforeRenewal_DoesNotRenewOrCancelProcessing() + { + var time = new FakeTimeProvider(); + var transport = new Mock(); + using var processing = new CancellationTokenSource(); + await using var lease = CreateLease(transport.Object, time, processing); + lease.Settled(); + time.Advance(TimeSpan.FromMinutes(2)); + await lease.Completion.WaitAsync(TestContext.Current.CancellationToken); + Assert.False(processing.IsCancellationRequested); + Assert.False(lease.IsLost); + transport.Verify(t => t.RenewLockAsync(It.IsAny(), It.IsAny(), It.IsAny()), Times.Never); + } + + [Fact] + public async Task TransientRenewalFailure_RetriesInsideTheOriginalLease() + { + var time = new FakeTimeProvider(); + var transport = new Mock(); + var renewed = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + int calls = 0; + transport.Setup(t => t.RenewLockAsync(It.IsAny(), It.IsAny(), It.IsAny())) + .Returns(() => + { + if (Interlocked.Increment(ref calls) == 1) throw new TimeoutException("Transient broker failure"); + renewed.TrySetResult(); + return Task.CompletedTask; + }); + using var processing = new CancellationTokenSource(); + await using var lease = CreateLease(transport.Object, time, processing); + // Advance one tick at a time until the retry has completed; monitor continuations run asynchronously. + for (int tick = 0; tick < 9 && !renewed.Task.IsCompleted; tick++) + { + time.Advance(TimeSpan.FromSeconds(1)); + await Task.Delay(10, TestContext.Current.CancellationToken); + } + await renewed.Task.WaitAsync(TimeSpan.FromSeconds(5), TestContext.Current.CancellationToken); + lease.Settled(); + Assert.False(processing.IsCancellationRequested); + Assert.True(calls >= 2); + } + + [Fact] + public async Task RenewalThatIgnoresCancellation_DoesNotPreventDisposal() + { + var time = new FakeTimeProvider(); + var transport = new Mock(); + var started = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var broker = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + transport.Setup(t => t.RenewLockAsync(It.IsAny(), It.IsAny(), It.IsAny())) + .Returns((TransportEntry _, TimeSpan? _, CancellationToken ct) => { started.TrySetResult(ct); return broker.Task; }); + using var processing = new CancellationTokenSource(); + var lease = CreateLease(transport.Object, time, processing); + try + { + time.Advance(TimeSpan.FromSeconds(5)); + var renewalToken = await started.Task.WaitAsync(TimeSpan.FromSeconds(5), TestContext.Current.CancellationToken); + await lease.DisposeAsync().AsTask().WaitAsync(TimeSpan.FromSeconds(5), TestContext.Current.CancellationToken); + Assert.True(renewalToken.IsCancellationRequested); + Assert.False(processing.IsCancellationRequested); + } + finally { broker.TrySetResult(); } + } + + [Fact] + public async Task SettlementRacingTheFirstLeaseCheck_DoesNotLeaveRenewalRunning() + { + for (int iteration = 0; iteration < 100; iteration++) + { + var time = new FakeTimeProvider(); + var transport = new Mock(); + int calls = 0; + transport.Setup(t => t.RenewLockAsync(It.IsAny(), It.IsAny(), It.IsAny())) + .Returns(() => { Interlocked.Increment(ref calls); return Task.CompletedTask; }); + using var processing = new CancellationTokenSource(); + var lease = CreateLease(transport.Object, time, processing); + await Task.WhenAll(Task.Run(() => time.Advance(TimeSpan.FromSeconds(5)), TestContext.Current.CancellationToken), + Task.Run(async () => await lease.DisposeAsync(), TestContext.Current.CancellationToken)); + int callsAtDisposal = calls; + time.Advance(TimeSpan.FromMinutes(2)); + Assert.Equal(callsAtDisposal, calls); + Assert.False(processing.IsCancellationRequested); + } + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task DeadlineExpires_CancelsProcessingEvenWhenRenewalCannotFinish(bool autoRenew) + { + var time = new FakeTimeProvider(); + var transport = new Mock(); + var started = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var broker = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + transport.Setup(t => t.RenewLockAsync(It.IsAny(), It.IsAny(), It.IsAny())) + .Returns(() => { started.TrySetResult(); return broker.Task; }); + using var processing = new CancellationTokenSource(); + var cancelled = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + using var registration = processing.Token.Register(() => cancelled.TrySetResult()); + await using var lease = CreateLease(transport.Object, time, processing, autoRenew); + try + { + time.Advance(TimeSpan.FromSeconds(5)); + if (autoRenew) await started.Task.WaitAsync(TimeSpan.FromSeconds(5), TestContext.Current.CancellationToken); + time.Advance(TimeSpan.FromSeconds(6)); + await cancelled.Task.WaitAsync(TimeSpan.FromSeconds(5), TestContext.Current.CancellationToken); + Assert.True(lease.IsLost); + } + finally { broker.TrySetResult(); } + } + + private static MessageDeliveryLease CreateLease(IMessageTransport transport, FakeTimeProvider time, CancellationTokenSource processing, bool autoRenew = true) + => new(transport, new TransportEntry + { + Id = "work", Destination = DestinationAddress.ForQueue("work"), Body = new byte[] { 1 }, Receipt = default, + LockExpiresUtc = time.GetUtcNow().AddSeconds(10) + }, TimeSpan.FromSeconds(10), autoRenew, time, NullLogger.Instance, processing); +} diff --git a/tests/Foundatio.Tests/Messaging/MessageEndpointPolicyTests.cs b/tests/Foundatio.Tests/Messaging/MessageEndpointPolicyTests.cs new file mode 100644 index 000000000..430afbfe9 --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/MessageEndpointPolicyTests.cs @@ -0,0 +1,227 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Foundatio.Jobs; +using Moq; +using Microsoft.Extensions.Time.Testing; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class MessageEndpointPolicyTests +{ + [Fact] + public async Task ConcurrentExplicitCompletion_SettlesTheDeliveryOnce() + { + var token = TestContext.Current.CancellationToken; + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var delivery = new Mock(); + delivery.SetupGet(value => value.Headers).Returns(MessageHeaders.Empty); + delivery.Setup(value => value.CompleteAsync(It.IsAny())).Returns(release.Task); + var pipeline = new MessageExecutionPipeline(new MessageExecutionOptions { QueueName = "exports" }); + await pipeline.ProcessAsync(delivery.Object, async (context, ct) => + { + var first = context.CompleteAsync(ct); + var second = context.CompleteAsync(ct); + Assert.False(context.IsCompleted); + release.TrySetResult(); + await Task.WhenAll(first, second); + Assert.True(context.IsCompleted); + await Assert.ThrowsAsync(() => context.AbandonAsync(ct)); + return MessageOutcome.Success; + }, token); + delivery.Verify(value => value.CompleteAsync(It.IsAny()), Times.Once); + } + + [Fact] + public async Task TrackedCancellation_PollsAndStopsTheRunningHandler() + { + var token = TestContext.Current.CancellationToken; + var time = new FakeTimeProvider(); + var store = new InMemoryJobRuntimeStore(time); + await store.CreateIfAbsentAsync(new JobState { JobId = "cancel", Name = "exports", QueueName = "exports", ExecutionOwner = JobExecutionOwner.Broker }, token); + var delivery = new Mock(); + delivery.SetupGet(value => value.Headers).Returns(MessageHeaders.Empty.ToBuilder().Set(ExecutionHeaders.ExecutionId, "cancel").Build()); + delivery.SetupGet(value => value.Attempts).Returns(1); + delivery.Setup(value => value.CompleteAsync(It.IsAny())).Returns(Task.CompletedTask); + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var pipeline = new MessageExecutionPipeline(new MessageExecutionOptions { QueueName = "exports", TrackProgress = true, CancellationPollInterval = TimeSpan.FromSeconds(1) }, store, time); + var processing = pipeline.ProcessAsync(delivery.Object, async (_, ct) => + { + entered.TrySetResult(); + await Task.Delay(Timeout.InfiniteTimeSpan, ct); + return MessageOutcome.Success; + }, token); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + Assert.True(await store.RequestCancellationAsync("cancel", token)); + time.Advance(TimeSpan.FromSeconds(1)); + await processing.WaitAsync(TimeSpan.FromSeconds(5), token); + Assert.Equal(JobStatus.Cancelled, (await store.GetAsync("cancel", token))!.Status); + delivery.Verify(value => value.CompleteAsync(It.IsAny()), Times.Once); + } + + [Fact] + public async Task ExpiredTrackingHistory_DoesNotPreventBrokerWorkFromRunning() + { + var store = new InMemoryJobRuntimeStore(); + var delivery = new Mock(); + delivery.SetupGet(value => value.Headers).Returns(MessageHeaders.Create(new Dictionary { [ExecutionHeaders.ExecutionId] = "expired" })); + delivery.SetupGet(value => value.Attempts).Returns(1); + delivery.Setup(value => value.CompleteAsync(It.IsAny())).Returns(Task.CompletedTask); + var pipeline = new MessageExecutionPipeline(new MessageExecutionOptions { QueueName = "exports", TrackProgress = true }, store); + bool invoked = false; + await pipeline.ProcessAsync(delivery.Object, (_, _) => + { + invoked = true; + return ValueTask.FromResult(MessageOutcome.Success); + }, TestContext.Current.CancellationToken); + Assert.True(invoked); + delivery.Verify(value => value.CompleteAsync(It.IsAny()), Times.Once); + Assert.Null(await store.GetAsync("expired", TestContext.Current.CancellationToken)); + } + + [Fact] + public async Task ManualRenewal_ExtendsTheSupervisedDeadlineWhenAutoRenewIsDisabled() + { + var time = new FakeTimeProvider(); + await using var transport = new InMemoryMessageTransport(time); + await using var bus = new MessageBus(transport, new MessageBusOptions { TimeProvider = time, OwnsTransport = false }); + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var cancelled = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var consumer = await bus.ConsumeAsync(async (message, token) => + { + entered.TrySetResult(message); + try { await Task.Delay(Timeout.InfiniteTimeSpan, token); } + catch (OperationCanceledException) when (token.IsCancellationRequested) { cancelled.TrySetResult(); } + }, new MessageConsumerOptions { Destination = "manual-lease", AutoRenewLock = false, VisibilityTimeout = TimeSpan.FromSeconds(30) }, TestContext.Current.CancellationToken); + await bus.SendAsync("work", new MessageSendOptions { Destination = "manual-lease" }, TestContext.Current.CancellationToken); + var delivery = await entered.Task.WaitAsync(TimeSpan.FromSeconds(5), TestContext.Current.CancellationToken); + time.Advance(TimeSpan.FromSeconds(10)); + await delivery.RenewLockAsync(TimeSpan.FromSeconds(60), TestContext.Current.CancellationToken); + time.Advance(TimeSpan.FromSeconds(25)); + await Task.Delay(50, TestContext.Current.CancellationToken); + Assert.False(delivery.CancellationToken.IsCancellationRequested); + time.Advance(TimeSpan.FromSeconds(40)); + await cancelled.Task.WaitAsync(TimeSpan.FromSeconds(5), TestContext.Current.CancellationToken); + Assert.True(delivery.IsLeaseLost); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task FailedAcknowledgment_DoesNotPersistCompletion(bool cancellationRequested) + { + var store = new InMemoryJobRuntimeStore(); + await store.CreateIfAbsentAsync(new JobState + { + JobId = "ack-failure", + Name = "exports", + ExecutionOwner = JobExecutionOwner.Broker, + QueueName = "exports", + PayloadType = "Export", + CancellationRequested = cancellationRequested, + Status = JobStatus.Queued, + CreatedUtc = DateTimeOffset.UtcNow, + LastUpdatedUtc = DateTimeOffset.UtcNow + }, cancellationToken: TestContext.Current.CancellationToken); + var delivery = new Mock(); + delivery.SetupGet(value => value.Headers).Returns(MessageHeaders.Create(new Dictionary { [ExecutionHeaders.ExecutionId] = "ack-failure" })); + delivery.SetupGet(value => value.Attempts).Returns(1); + delivery.Setup(value => value.CompleteAsync(It.IsAny())).ThrowsAsync(new TimeoutException("Unknown broker acknowledgment")); + var pipeline = new MessageExecutionPipeline(new MessageExecutionOptions { QueueName = "exports", TrackProgress = true }, store); + await pipeline.ProcessAsync(delivery.Object, (_, _) => ValueTask.FromResult(MessageOutcome.Success), TestContext.Current.CancellationToken); + Assert.Equal(JobStatus.RetryPending, (await store.GetAsync("ack-failure", TestContext.Current.CancellationToken))!.Status); + } + + [Fact] + public async Task ConsumeAsync_EndpointPolicy_ControlsVisibilityAndReceiveCapacity() + { + var token = TestContext.Current.CancellationToken; + var transport = new Mock(); + transport.As(); + var received = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + transport.Setup(t => t.ReceiveAsync(It.IsAny(), It.IsAny(), It.IsAny(), It.IsAny())) + .Returns(async (DestinationAddress source, ReceiveRequest request, TimeSpan visibility, CancellationToken ct) => + { + Assert.Equal("exports", source.Name); + Assert.InRange(request.MaxMessages, 1, 2); + Assert.Equal(TimeSpan.FromSeconds(12), visibility); + received.TrySetResult(); + await Task.Delay(Timeout.InfiniteTimeSpan, ct); + return Array.Empty(); + }); + await using var bus = new MessageBus(transport.Object); + await using var consumer = await bus.ConsumeAsync((_, _) => Task.CompletedTask, new MessageConsumerOptions + { + Destination = "exports", + MaxConcurrency = 5, + PrefetchCount = 2, + VisibilityTimeout = TimeSpan.FromSeconds(12) + }, token); + await received.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + } + + [Fact] + public async Task ConsumeAsync_GracefulShutdown_CompletesAdmittedWorkBeforeReturning() + { + var token = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport); + var started = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var finish = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + CancellationToken handlerToken = default; + var consumer = await bus.ConsumeAsync(async (_, ct) => + { + handlerToken = ct; + started.TrySetResult(); + await finish.Task.WaitAsync(ct); + }, new MessageConsumerOptions { Destination = "exports", ShutdownTimeout = TimeSpan.FromSeconds(5) }, token); + try + { + await bus.SendAsync(new Work(), new MessageSendOptions { Destination = "exports" }, token); + await started.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + var stopping = consumer.DisposeAsync().AsTask(); + Assert.False(handlerToken.IsCancellationRequested); + Assert.False(stopping.IsCompleted); + finish.TrySetResult(); + await stopping.WaitAsync(TimeSpan.FromSeconds(5), token); + var stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("exports"), token); + Assert.Equal(1, stats.Completed); + Assert.Equal(0, stats.Queued); + } + finally + { + finish.TrySetResult(); + await consumer.DisposeAsync(); + } + } + + [Fact] + public async Task ConsumeWithOutcomeAsync_RetryBudget_DeadLettersReturnedFailure() + { + var token = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport); + int calls = 0; + await using var consumer = await bus.ConsumeWithOutcomeAsync((_, _) => + { + Interlocked.Increment(ref calls); + return new ValueTask(MessageOutcome.Retry("service unavailable")); + }, new MessageConsumerOptions { Destination = "exports", MaxAttempts = 2, RedeliveryBackoff = _ => TimeSpan.Zero }, token); + await bus.SendAsync(new Work(), new MessageSendOptions { Destination = "exports" }, token); + using var deadline = CancellationTokenSource.CreateLinkedTokenSource(token); + deadline.CancelAfter(TimeSpan.FromSeconds(5)); + IReadOnlyList deadLetters; + do + { + deadLetters = await transport.PeekDeadLetteredAsync(DestinationAddress.ForQueue("exports"), cancellationToken: deadline.Token); + if (deadLetters.Count == 0) await Task.Delay(10, deadline.Token); + } while (deadLetters.Count == 0); + Assert.Equal(2, calls); + Assert.Equal("service unavailable", Assert.Single(deadLetters).Headers[KnownHeaders.DeadLetterReason]); + } + + private sealed record Work; +} diff --git a/tests/Foundatio.Tests/Messaging/MessageTests.cs b/tests/Foundatio.Tests/Messaging/MessageTests.cs deleted file mode 100644 index 67cc60a76..000000000 --- a/tests/Foundatio.Tests/Messaging/MessageTests.cs +++ /dev/null @@ -1,91 +0,0 @@ -using System; -using System.Runtime.InteropServices; -using Foundatio.Messaging; -using Xunit; - -namespace Foundatio.Tests.Messaging; - -public class MessageTests -{ - [Fact] - public void Constructor_WithByteArray_StoresDataAsReadOnlyMemory() - { - // Arrange - byte[] payload = [1, 2, 3, 4]; - - // Act - var message = new Message(payload, _ => null); - - // Assert - Assert.False(message.Data.IsEmpty); - Assert.Equal(payload.Length, message.Data.Length); - Assert.True(payload.AsSpan().SequenceEqual(message.Data.Span)); - } - - [Fact] - public void Constructor_WithReadOnlyMemory_StoresDataWithoutCopy() - { - // Arrange - byte[] payload = [10, 20, 30]; - var memory = new ReadOnlyMemory(payload); - - // Act - var message = new Message(memory, _ => null); - - // Assert - Assert.Equal(3, message.Data.Length); - Assert.True(payload.AsSpan().SequenceEqual(message.Data.Span)); - Assert.True(MemoryMarshal.TryGetArray(message.Data, out ArraySegment segment)); - Assert.Same(payload, segment.Array); - Assert.Equal(0, segment.Offset); - Assert.Equal(payload.Length, segment.Count); - } - - [Fact] - public void Data_WhenEmptyMemory_IsEmptyReturnsTrue() - { - // Arrange / Act - var message = new Message(ReadOnlyMemory.Empty, _ => null); - - // Assert - Assert.True(message.Data.IsEmpty); - Assert.Equal(0, message.Data.Length); - } - - [Fact] - public void GetBody_WhenDelegateIsProvided_ReturnsDelegateResult() - { - // Arrange - byte[] payload = [1]; - var expected = new object(); - - // Act - var message = new Message(payload, _ => expected); - - // Assert - Assert.Same(expected, message.GetBody()); - } - - [Fact] - public void TypedMessage_WhenWrappingMessage_ForwardsPropertiesAndData() - { - // Arrange - byte[] payload = [5, 6, 7]; - var inner = new Message(payload, _ => "body") - { - Type = "test", - UniqueId = "id", - CorrelationId = "corr" - }; - - // Act - var typed = new Message(inner); - - // Assert - Assert.Equal("body", typed.Body); - Assert.Equal("test", typed.Type); - Assert.Equal("id", typed.UniqueId); - Assert.Equal("corr", typed.CorrelationId); - Assert.True(payload.AsSpan().SequenceEqual(typed.Data.Span)); - } -} diff --git a/tests/Foundatio.Tests/Messaging/MessagingTestHarnessTests.cs b/tests/Foundatio.Tests/Messaging/MessagingTestHarnessTests.cs new file mode 100644 index 000000000..a893bff6c --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/MessagingTestHarnessTests.cs @@ -0,0 +1,292 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Extensions.Hosting.Messaging; +using Foundatio.Messaging; +using Foundatio.Messaging.Testing; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Time.Testing; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class MessagingTestHarnessTests +{ + [Fact] + public async Task Harness_RecordsSendPublishAndHandledWithTypedAccessAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var harness = new MessagingTestHarness(); + await using var bus = new MessageBus(harness.Transport, new MessageBusOptions { OwnsTransport = false }); + + var handled = new List(); + await using var subscription = await bus.ConsumeAsync((context, _) => + { + lock (handled) + handled.Add(context.Message.Id); + return Task.CompletedTask; + }, cancellationToken: cancellationToken); + + await using var events = await bus.SubscribeAsync((context, _) => + { + lock (handled) + handled.Add(context.Message.Id); + return Task.CompletedTask; + }, cancellationToken: cancellationToken); + + await bus.SendAsync(new HarnessOrder { Id = "cmd" }, cancellationToken: cancellationToken); + await bus.PublishAsync(new HarnessOrder { Id = "evt" }, cancellationToken: cancellationToken); + await harness.WaitForIdleAsync(cancellationToken: cancellationToken); + + // Sends and publishes are recorded separately, deserialized back to the message type. + Assert.Equal("cmd", Assert.Single(harness.Sent()).Id); + Assert.Equal("evt", Assert.Single(harness.Published()).Id); + Assert.Equal(2, harness.Handled().Count); + Assert.Equal(2, handled.Count); + + // Raw recordings carry the route and role for topology assertions. + var sent = Assert.Single(harness.SentMessages); + Assert.Equal("harness-orders", sent.Destination); + Assert.Equal(DestinationRole.Queue, sent.Role); + var published = Assert.Single(harness.PublishedMessages); + Assert.Equal("harness-orders", published.Destination); + Assert.Equal(DestinationRole.Topic, published.Role); + + // Negative assertions are immediate once idle. + Assert.Empty(harness.DeadLetteredMessages); + Assert.Empty(harness.AbandonedMessages); + Assert.Empty(harness.Sent()); + } + + [Fact] + public async Task Harness_RetryCycleEndsInDeadLetterAndIsFullyObservableAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var harness = new MessagingTestHarness(); + await using var bus = new MessageBus(harness.Transport, new MessageBusOptions { OwnsTransport = false }); + + int attempts = 0; + await using var subscription = await bus.ConsumeAsync((_, _) => + { + Interlocked.Increment(ref attempts); + throw new InvalidOperationException("always fails"); + }, new MessageConsumerOptions { MaxAttempts = 3, RedeliveryBackoff = _ => TimeSpan.Zero }, cancellationToken); + + await bus.SendAsync(new HarnessOrder { Id = "poison" }, cancellationToken: cancellationToken); + await harness.WaitForIdleAsync(cancellationToken: cancellationToken); + + // The whole failure path is assertable: two retries, then terminal dead-letter with the reason and forensics. + Assert.Equal(3, Volatile.Read(ref attempts)); + Assert.Equal(2, harness.AbandonedMessages.Count); + Assert.All(harness.Abandoned(), m => Assert.Equal("poison", m.Id)); + Assert.Equal(2, harness.Abandoned().Count); + var dead = Assert.Single(harness.DeadLetteredMessages); + Assert.Equal("handler-error", dead.Reason); + Assert.Equal(3, dead.Attempts); + Assert.Equal(typeof(InvalidOperationException).FullName, dead.Headers[KnownHeaders.DeadLetterExceptionType]); + Assert.Equal("poison", Assert.Single(harness.DeadLettered()).Id); + Assert.Empty(harness.HandledMessages); + } + + [Fact] + public async Task WaitForIdle_CoversDelayedRedeliveriesAndTimesOutWithDiagnosticsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var harness = new MessagingTestHarness(); + await using var bus = new MessageBus(harness.Transport, new MessageBusOptions { OwnsTransport = false }); + + int attempts = 0; + await using var subscription = await bus.ConsumeAsync((_, _) => + { + if (Interlocked.Increment(ref attempts) == 1) + throw new InvalidOperationException("fails once"); + return Task.CompletedTask; + }, new MessageConsumerOptions { MaxAttempts = 2, RedeliveryBackoff = _ => TimeSpan.FromMilliseconds(500) }, cancellationToken); + + await bus.SendAsync(new HarnessOrder { Id = "retry-me" }, cancellationToken: cancellationToken); + + // The retry is parked in a redelivery timer (neither queued nor in flight); the harness must still see it. + await harness.WaitForIdleAsync(cancellationToken: cancellationToken); + Assert.Equal(2, Volatile.Read(ref attempts)); + Assert.Single(harness.Handled()); + + // Timeout.InfiniteTimeSpan means wait-until-idle; other negative timeouts are rejected up front. + await harness.WaitForIdleAsync(Timeout.InfiniteTimeSpan, cancellationToken); + await Assert.ThrowsAsync(() => harness.WaitForIdleAsync(TimeSpan.FromMilliseconds(-2), cancellationToken)); + + // A destination that never drains fails with the busy destinations named. + await using var stuck = await bus.ConsumeAsync((_, handlerToken) => Task.Delay(Timeout.Infinite, handlerToken), + cancellationToken: cancellationToken); + await bus.SendAsync(new HarnessOther { Id = "stuck" }, cancellationToken: cancellationToken); + + var timeout = await Assert.ThrowsAsync(() => harness.WaitForIdleAsync(TimeSpan.FromSeconds(2), cancellationToken)); + Assert.Contains("harness-other", timeout.Message); + } + + [Fact] + public async Task WaitForHandled_ReturnsMatchesAndTimesOutWithDiagnosticsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var harness = new MessagingTestHarness(); + await using var bus = new MessageBus(harness.Transport, new MessageBusOptions { OwnsTransport = false }); + + await using var subscription = await bus.ConsumeAsync((_, _) => Task.CompletedTask, cancellationToken: cancellationToken); + + await bus.SendAsync(new HarnessOrder { Id = "one" }, cancellationToken: cancellationToken); + await bus.SendAsync(new HarnessOrder { Id = "two" }, cancellationToken: cancellationToken); + + // Awaits just the outcome under test — no full-bus drain needed before asserting. + var handled = await harness.WaitForHandledAsync(2, cancellationToken: cancellationToken); + Assert.Equal(2, handled.Count); + Assert.Contains(handled, m => m.Id == "one"); + Assert.Contains(handled, m => m.Id == "two"); + + // A type that never settles fails fast, naming everything that WAS recorded. + var timeout = await Assert.ThrowsAsync(() => + harness.WaitForHandledAsync(timeout: TimeSpan.FromMilliseconds(200), cancellationToken: cancellationToken)); + Assert.Contains("sent=2", timeout.Message); + Assert.Contains("handled=2", timeout.Message); + Assert.Contains("deadLettered=0", timeout.Message); + } + + [Fact] + public async Task WaitForDeadLettered_WithZeroBackoff_IsSleepFreeAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var harness = new MessagingTestHarness(); + await using var bus = new MessageBus(harness.Transport, new MessageBusOptions { OwnsTransport = false }); + + // Zero backoff makes the whole retry cycle run without any wall-clock delay — the sleep-free way to test the + // retry/dead-letter path (no fake clock to advance). + int attempts = 0; + await using var subscription = await bus.ConsumeAsync((_, _) => + { + Interlocked.Increment(ref attempts); + throw new InvalidOperationException("always fails"); + }, new MessageConsumerOptions { MaxAttempts = 3, RedeliveryBackoff = _ => TimeSpan.Zero }, cancellationToken); + + await bus.SendAsync(new HarnessOrder { Id = "poison" }, cancellationToken: cancellationToken); + + // The raw records surface the terminal forensics: the reason and the exhausted attempt count. + var dead = Assert.Single(await harness.WaitForDeadLetteredAsync(cancellationToken: cancellationToken)); + Assert.Equal("handler-error", dead.Reason); + Assert.Equal(3, dead.Attempts); + Assert.Equal(3, Volatile.Read(ref attempts)); + Assert.Empty(harness.HandledMessages); + } + + [Fact] + public async Task FakeTimeProvider_AdvancingTheClockFiresDelayedRedeliveryAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + + // With an injected fake TimeProvider the harness still WAITS in real time, but delayed redeliveries execute + // on the fake clock — the test must advance it itself or the retry never fires. + var timeProvider = new FakeTimeProvider(DateTimeOffset.UtcNow); + await using var harness = new MessagingTestHarness(timeProvider: timeProvider); + await using var bus = new MessageBus(harness.Transport, new MessageBusOptions { OwnsTransport = false, TimeProvider = timeProvider }); + + int attempts = 0; + await using var subscription = await bus.ConsumeAsync((_, _) => + { + if (Interlocked.Increment(ref attempts) == 1) + throw new InvalidOperationException("fails once"); + return Task.CompletedTask; + }, new MessageConsumerOptions { MaxAttempts = 2, RedeliveryBackoff = _ => TimeSpan.FromMinutes(5) }, cancellationToken); + + await bus.SendAsync(new HarnessOrder { Id = "clockwork" }, cancellationToken: cancellationToken); + + // Advance only after the failed attempt settles — the redelivery timer is armed by the abandon. + while (harness.AbandonedMessages.Count == 0) + await Task.Delay(10, cancellationToken); + Assert.Empty(harness.HandledMessages); + + timeProvider.Advance(TimeSpan.FromMinutes(5)); + + Assert.Equal("clockwork", Assert.Single(await harness.WaitForHandledAsync(cancellationToken: cancellationToken)).Id); + Assert.Equal(2, Volatile.Read(ref attempts)); + } + + [Fact] + public async Task DestinationsWithNoConsumer_NamesTheDestinationsNothingConsumesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var harness = new MessagingTestHarness(); + await using var bus = new MessageBus(harness.Transport, new MessageBusOptions { OwnsTransport = false }); + + // The newcomer's first failing test: sent/published fine, idle immediately, Handled empty — because nothing + // consumes the destination. This property names the culprit. + await bus.SendAsync(new HarnessOther { Id = "orphan" }, cancellationToken: cancellationToken); + await bus.PublishAsync(new HarnessOrder { Id = "dropped" }, cancellationToken: cancellationToken); + + Assert.Contains("harness-other", harness.DestinationsWithNoConsumer); + Assert.Contains("harness-orders", harness.DestinationsWithNoConsumer); + + // Once a subscriber attaches (and drains the parked command), the queue is no longer unconsumed; the topic + // publish stays listed — it was dropped for having zero subscriptions at publish time. + await using var subscription = await bus.ConsumeAsync((_, _) => Task.CompletedTask, cancellationToken: cancellationToken); + await harness.WaitForIdleAsync(cancellationToken: cancellationToken); + + Assert.Single(harness.Handled()); + Assert.DoesNotContain("harness-other", harness.DestinationsWithNoConsumer); + Assert.Contains("harness-orders", harness.DestinationsWithNoConsumer); + } + + [Fact] + public async Task UseTestHarness_WiresDeclarativeHandlersOverTheRecordingTransportAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + + var services = new ServiceCollection(); + services.AddLogging(); + services.AddFoundatio() + .Messaging.UseTestHarness() + .AddConsumer(); + + services.AddMessageConsumers(); + await using var provider = services.BuildServiceProvider(); + var hosted = provider.GetServices().ToList(); + foreach (var service in hosted) + await service.StartAsync(cancellationToken); + + try + { + var harness = provider.GetRequiredService(); + var bus = provider.GetRequiredService(); + Assert.Same(harness.Transport, provider.GetRequiredService()); + + await bus.SendAsync(new HarnessOrder { Id = "from-di" }, cancellationToken: cancellationToken); + await harness.WaitForIdleAsync(cancellationToken: cancellationToken); + + Assert.Equal("from-di", Assert.Single(harness.Sent()).Id); + Assert.Equal("from-di", Assert.Single(harness.Handled()).Id); + Assert.Equal("from-di", Assert.Single(RecordingOrderHandler.Handled)); + } + finally + { + foreach (var service in hosted) + await service.StopAsync(cancellationToken); + } + } + + [MessageRoute("harness-orders")] + public class HarnessOrder { public string Id { get; set; } = ""; } + + [MessageRoute("harness-other")] + public class HarnessOther { public string Id { get; set; } = ""; } + + private sealed class RecordingOrderHandler : IMessageHandler + { + public static readonly List Handled = []; + + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + lock (Handled) + Handled.Add(context.Message.Id); + return Task.CompletedTask; + } + } +} diff --git a/tests/Foundatio.Tests/Messaging/PubSubTests.cs b/tests/Foundatio.Tests/Messaging/PubSubTests.cs new file mode 100644 index 000000000..98bd432e8 --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/PubSubTests.cs @@ -0,0 +1,458 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.AsyncEx; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Foundatio.Tests.Extensions; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class PubSubTests +{ + [Fact] + public async Task PublishAsync_FansOutToMultipleSubscriptionsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var pubSub = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var firstReceived = new AsyncCountdownEvent(1); + var secondReceived = new AsyncCountdownEvent(1); + + await using var first = await pubSub.SubscribeAsync((message, _) => + { + Assert.Equal("published", message.Message.Data); + firstReceived.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "subscriber-a" }, cts.Token); + + await using var second = await pubSub.SubscribeAsync((message, _) => + { + Assert.Equal("published", message.Message.Data); + secondReceived.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "subscriber-b" }, cts.Token); + + await pubSub.PublishAsync(new PreviewEvent { Data = "published" }, cancellationToken: cancellationToken); + + await firstReceived.WaitAsync(TimeSpan.FromSeconds(2)); + await secondReceived.WaitAsync(TimeSpan.FromSeconds(2)); + var firstStats = await transport.GetStatsAsync(DestinationAddress.ForSubscription(first.Source.Topic!, first.Source.Name), cancellationToken); + var secondStats = await transport.GetStatsAsync(DestinationAddress.ForSubscription(second.Source.Topic!, second.Source.Name), cancellationToken); + Assert.Equal(1, firstStats.Completed); + Assert.Equal(1, secondStats.Completed); + } + + [Fact] + public async Task SubscribeAsync_WithSameSubscriptionOnTwoReplicas_CompetesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var pubSub = new MessageBus(transport, new() { OwnsTransport = false }); + await using var replica = new MessageBus(transport, new() { OwnsTransport = false }); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var received = new AsyncCountdownEvent(2); + var deliveriesByMessageId = new ConcurrentDictionary(StringComparer.Ordinal); + + Func, CancellationToken, Task> handler = (message, _) => + { + deliveriesByMessageId.AddOrUpdate(message.Id, 1, (_, count) => count + 1); + received.Signal(); + return Task.CompletedTask; + }; + + await using var first = await pubSub.SubscribeAsync(handler, new MessageSubscriptionOptions + { + Subscription = "billing-service" + }, cts.Token); + await using var second = await replica.SubscribeAsync(handler, new MessageSubscriptionOptions + { + Subscription = "billing-service" + }, cts.Token); + + await pubSub.PublishBatchAsync([ + new PreviewEvent { Data = "one" }, + new PreviewEvent { Data = "two" } + ], cancellationToken: cancellationToken); + + await received.WaitAsync(TimeSpan.FromSeconds(2)); + await WaitForCompletedAsync(transport, DestinationAddress.ForSubscription(first.Source.Topic!, first.Source.Name), 2, cancellationToken); + + Assert.Equal(first.Source.Topic!, second.Source.Topic!); + Assert.Equal(first.Source.Name, second.Source.Name); + Assert.Equal(first.Source, second.Source); // same topic + subscription -> one shared transport source + + Assert.Equal(2, deliveriesByMessageId.Count); + Assert.All(deliveriesByMessageId.Values, count => Assert.Equal(1, count)); + } + + [Fact] + public async Task SubscribeAsync_SameSubscriptionOnTwoTopics_IsolatesPerTopicAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var pubSub = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var ordersReceived = new List(); + var paymentsReceived = new List(); + var ordersSignal = new AsyncCountdownEvent(1); + var paymentsSignal = new AsyncCountdownEvent(1); + + // The same subscription identity ("shared") on two different topics. + await using var orders = await pubSub.SubscribeAsync((message, _) => + { + lock (ordersReceived) + ordersReceived.Add(message.Message.Data); + ordersSignal.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Topic = "orders", Subscription = "shared" }, cts.Token); + + await using var payments = await pubSub.SubscribeAsync((message, _) => + { + lock (paymentsReceived) + paymentsReceived.Add(message.Message.Data); + paymentsSignal.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Topic = "payments", Subscription = "shared" }, cts.Token); + + Assert.Equal(orders.Source.Name, payments.Source.Name); // same logical subscription identity + Assert.NotEqual(orders.Source, payments.Source); // but distinct topic-qualified transport sources + + // Publish one message to each topic. Each subscriber must receive only its own topic's message — proving both + // subscribers are live (not an always-broken one passing a negative-only assertion) and that they are isolated. + await pubSub.PublishAsync(new PreviewEvent { Data = "to-orders" }, new MessagePublishOptions { Topic = "orders" }, cancellationToken); + await pubSub.PublishAsync(new PreviewEvent { Data = "to-payments" }, new MessagePublishOptions { Topic = "payments" }, cancellationToken); + + await ordersSignal.WaitAsync(TimeSpan.FromSeconds(2)); + await paymentsSignal.WaitAsync(TimeSpan.FromSeconds(2)); + + // Let any (incorrect) cross-topic delivery arrive before asserting each side received only its own message. + await Task.Delay(TimeSpan.FromMilliseconds(150), cancellationToken); + + Assert.Equal(new[] { "to-orders" }, ordersReceived); + Assert.Equal(new[] { "to-payments" }, paymentsReceived); + } + + [Fact] + public async Task PublishBatchAsync_DeliversAllMessagesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var pubSub = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var received = new AsyncCountdownEvent(2); + + await using var subscription = await pubSub.SubscribeAsync((message, _) => + { + Assert.StartsWith("batch-", message.Message.Data); + received.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "batch-subscription" }, cts.Token); + + await pubSub.PublishBatchAsync([ + new PreviewEvent { Data = "batch-one" }, + new PreviewEvent { Data = "batch-two" } + ], cancellationToken: cancellationToken); + + await received.WaitAsync(TimeSpan.FromSeconds(2)); + var stats = await transport.GetStatsAsync(DestinationAddress.ForSubscription(subscription.Source.Topic!, subscription.Source.Name), cancellationToken); + Assert.Equal(2, stats.Completed); + } + + [Fact] + public async Task PublishAsync_WithOptions_PropagatesHeadersAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var pubSub = new MessageBus(new InMemoryMessageTransport()); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var received = new TaskCompletionSource>(TaskCreationOptions.RunContinuationsAsynchronously); + + await using var subscription = await pubSub.SubscribeAsync((message, _) => + { + received.TrySetResult(message); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "metadata-subscription" }, cts.Token); + + await pubSub.PublishAsync(new PreviewEvent { Data = "metadata" }, new MessagePublishOptions + { + CorrelationId = "corr-456", + Priority = MessagePriority.High, + Headers = MessageHeaders.Create([ + new KeyValuePair("tenant", "acme") + ]) + }, cancellationToken); + + var completed = await Task.WhenAny(received.Task, Task.Delay(TimeSpan.FromSeconds(2), cancellationToken)); + Assert.Equal(received.Task, completed); + + var message = await received.Task; + Assert.Equal("metadata", message.Message.Data); + Assert.Equal("corr-456", message.CorrelationId); + Assert.Equal(MessagePriority.High, message.Priority); + Assert.Equal("acme", message.Headers["tenant"]); + Assert.Equal(typeof(PreviewEvent).FullName, message.MessageType); + + } + + [Fact] + public async Task PublishAsync_WithDelay_SchedulesThroughRuntimeStoreAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + await using var transport = new InMemoryMessageTransport(); + await using var pubSub = new MessageBus(transport, new MessageBusOptions { RuntimeStore = store }); + var processor = CreateDispatchProcessor(store, transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var received = new AsyncCountdownEvent(1); + + await using var subscription = await pubSub.SubscribeAsync((_, _) => + { + received.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "delayed-subscription" }, cts.Token); + + await pubSub.PublishAsync(new PreviewEvent { Data = "later" }, new MessagePublishOptions { Delay = TimeSpan.FromMinutes(1) }, cancellationToken); + + await Assert.ThrowsAsync(async () => await received.WaitAsync(TimeSpan.FromMilliseconds(50))); + Assert.Equal(1, await processor.DispatchDueAsync(DateTimeOffset.UtcNow.AddMinutes(2), cancellationToken: cancellationToken)); + await received.WaitAsync(TimeSpan.FromSeconds(2)); + + } + + [Fact] + public async Task SubscribeAsync_WhenHandlerFails_RedeliversAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var pubSub = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var received = new AsyncCountdownEvent(2); + int attempts = 0; + + await using var subscription = await pubSub.SubscribeAsync((message, _) => + { + attempts++; + Assert.Equal(attempts, message.Attempts); + received.Signal(); + + if (attempts == 1) + throw new InvalidOperationException("try again"); + + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "retry-subscription", MaxAttempts = 2 }, cts.Token); + + await pubSub.PublishAsync(new PreviewEvent { Data = "retry" }, cancellationToken: cancellationToken); + + await received.WaitAsync(TimeSpan.FromSeconds(2)); + var stats = await transport.GetStatsAsync(DestinationAddress.ForSubscription(subscription.Source.Topic!, subscription.Source.Name), cancellationToken); + Assert.Equal(1, stats.Completed); + Assert.Equal(1, stats.Abandoned); + } + + + [Fact] + public async Task SubscribeAsync_WithDuplicateRegistration_ThrowsAsync() + { + var token = TestContext.Current.CancellationToken; + await using var bus = new MessageBus(new InMemoryMessageTransport()); + Func, CancellationToken, Task> handler = (_, _) => Task.CompletedTask; + await using var first = await bus.SubscribeAsync(handler, new() { Subscription = "billing" }, token); + await Assert.ThrowsAsync(() => bus.SubscribeAsync(handler, new() { Subscription = "billing" }, token)); + } + + [Fact] + public async Task SubscribeAsync_WithSameKeyAndDifferentHandler_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var pubSub = new MessageBus(new InMemoryMessageTransport()); + + await using var first = await pubSub.SubscribeAsync((_, _) => Task.CompletedTask, new MessageSubscriptionOptions { Subscription = "same-key" }, cancellationToken); + + await Assert.ThrowsAsync(async () => + await pubSub.SubscribeAsync((_, _) => Task.CompletedTask, new MessageSubscriptionOptions { Subscription = "same-key" }, cancellationToken)); + } + + [Fact] + public async Task SubscribeAsync_WithSameKeyAndDifferentFailurePolicy_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var pubSub = new MessageBus(new InMemoryMessageTransport()); + Func, CancellationToken, Task> handler = (_, _) => Task.CompletedTask; + + await using var first = await pubSub.SubscribeAsync(handler, new MessageSubscriptionOptions + { + Subscription = "same-key", + DeadLetterWhen = static ex => ex is InvalidOperationException + }, cancellationToken); + + // Shared-key subscriptions form ONE competing group; members with different retry/dead-letter LOGIC would + // settle the same message differently depending on who received it, so a divergent policy must be rejected — + // by delegate identity, not by mere has-a-policy presence. + await Assert.ThrowsAsync(async () => + await pubSub.SubscribeAsync(handler, new MessageSubscriptionOptions + { + Subscription = "same-key", + DeadLetterWhen = static ex => ex is ArgumentException + }, cancellationToken)); + } + + [Fact] + public async Task SubscribeAsync_WithGroupedTopicAndSubscriptionIdentity_ReceivesRawMessagesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + var routing = new MessageRoutingOptionsBuilder() + .MapTopic("order-events", typeof(IGroupedEvent)) + .Build(); + await using var pubSub = new MessageBus(transport, new MessageBusOptions { Router = new DefaultMessageRouter(routing) }); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var received = new AsyncCountdownEvent(2); + var messageTypes = new List(); + + await using var subscription = await pubSub.SubscribeAsync((message, _) => + { + lock (messageTypes) + messageTypes.Add(message.MessageType!); + + received.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Topic = "order-events", Subscription = "billing-service" }, cts.Token); + + await pubSub.PublishBatchAsync(new object[] + { + new PreviewEvent { Data = "one" }, + new OtherEvent { Data = "two" } + }, cancellationToken: cancellationToken); + + await received.WaitAsync(TimeSpan.FromSeconds(2)); + + Assert.Equal("order-events", subscription.Source.Topic!); + Assert.Equal("billing-service", subscription.Source.Name); + Assert.Equal("order-events/billing-service", subscription.Source.Key); // topic-qualified transport source + Assert.Contains(typeof(PreviewEvent).FullName!, messageTypes); + Assert.Contains(typeof(OtherEvent).FullName!, messageTypes); + + var stats = await transport.GetStatsAsync(DestinationAddress.ForSubscription(subscription.Source.Topic!, subscription.Source.Name), cancellationToken); + Assert.Equal(2, stats.Completed); + } + + + [Fact] + public async Task PublishAsync_WithDelay_OnTopicWithoutNativeDelay_RoutesThroughRuntimeStoreAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + + // The AWS SQS/SNS shape: queues honor a native delay (15-minute cap) but topics have none. A delayed publish + // within the QUEUE ceiling must still route through the runtime store — deciding by transport-wide capability + // would take the native path and the broker would silently drop the delay. + var store = new InMemoryJobRuntimeStore(); + await using var transport = new RoleSplitDelayTransport(queueMaxDelay: TimeSpan.FromMinutes(15)); + await using var pubSub = new MessageBus(transport, new MessageBusOptions { RuntimeStore = store }); + var processor = CreateDispatchProcessor(store, transport); + + await pubSub.PublishAsync(new PreviewEvent { Data = "later" }, new MessagePublishOptions { Delay = TimeSpan.FromMinutes(5) }, cancellationToken); + + Assert.Equal(0, transport.SendCount); + Assert.Equal(1, await processor.DispatchDueAsync(DateTimeOffset.UtcNow.AddMinutes(10), cancellationToken: cancellationToken)); + Assert.Equal(1, transport.SendCount); + Assert.Equal(DestinationRole.Topic, transport.LastDestination?.Role); + Assert.Null(transport.LastSendOptions?.DeliverAt); // the store dispatches it as due; the delay is spent, not forwarded + + // A delayed QUEUE send within the same transport's queue ceiling still uses the native path. + await pubSub.SendAsync(new PreviewEvent { Data = "soon" }, new MessageSendOptions { Delay = TimeSpan.FromMinutes(5) }, cancellationToken); + Assert.Equal(2, transport.SendCount); + Assert.NotNull(transport.LastSendOptions?.DeliverAt); + } + + private static async Task WaitForCompletedAsync(InMemoryMessageTransport transport, DestinationAddress destination, long expected, CancellationToken cancellationToken) + { + var deadline = DateTimeOffset.UtcNow.AddSeconds(2); + while (DateTimeOffset.UtcNow < deadline) + { + var stats = await transport.GetStatsAsync(destination, cancellationToken); + if (stats.Completed == expected) + return; + + await Task.Delay(TimeSpan.FromMilliseconds(10), cancellationToken); + } + + var finalStats = await transport.GetStatsAsync(destination, cancellationToken); + Assert.Equal(expected, finalStats.Completed); + } + + private static ScheduledMessageDispatcher CreateDispatchProcessor(IJobRuntimeStore store, IMessageTransport transport) + { + return new ScheduledMessageDispatcher(store, transport); + } + + // Mirrors AWS SQS/SNS: native delayed delivery on queues only. Topic sends with a future DeliverAt throw, so a + // silent delay drop cannot hide. + private sealed class RoleSplitDelayTransport : IMessageTransport, ISupportsPull, ITransportInfo + { + private readonly TimeSpan _queueMaxDelay; + + public RoleSplitDelayTransport(TimeSpan queueMaxDelay) => _queueMaxDelay = queueMaxDelay; + + public int SendCount { get; private set; } + public TransportSendOptions? LastSendOptions { get; private set; } + public DestinationAddress? LastDestination { get; private set; } + + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => + new HashSet { DestinationRole.Queue, DestinationRole.Topic, DestinationRole.Subscription }; + + public TransportCapabilities GetCapabilities(DestinationAddress destination) => destination.Role == DestinationRole.Topic + ? TransportCapabilities.None + : new TransportCapabilities { DelayedDelivery = true, MaxDeliveryDelay = _queueMaxDelay }; + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + if (destination.Role == DestinationRole.Topic && options.DeliverAt is { } deliverAt && deliverAt > DateTimeOffset.UtcNow) + throw new NotSupportedException("Topics have no native delayed delivery."); + + SendCount += messages.Count; + LastSendOptions = options; + LastDestination = destination; + var items = new SendItemResult[messages.Count]; + for (int i = 0; i < messages.Count; i++) + items[i] = new SendItemResult { MessageId = messages[i].MessageId ?? Guid.NewGuid().ToString("N") }; + + return Task.FromResult(new SendResult { Items = items }); + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + => Task.FromResult>([]); + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public ValueTask DisposeAsync() => ValueTask.CompletedTask; + } + + private interface IGroupedEvent + { + } + + private sealed class PreviewEvent : IGroupedEvent + { + public string? Data { get; set; } + } + + private sealed class OtherEvent : IGroupedEvent + { + public string? Data { get; set; } + } +} diff --git a/tests/Foundatio.Tests/Messaging/RecoveryBehaviorTests.cs b/tests/Foundatio.Tests/Messaging/RecoveryBehaviorTests.cs new file mode 100644 index 000000000..911859f7d --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/RecoveryBehaviorTests.cs @@ -0,0 +1,56 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Caching; +using Foundatio.Messaging; +using Moq; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class RecoveryBehaviorTests +{ + [Fact] + public async Task HybridCache_SubscriptionGap_WaitsAndDiscardsStaleLocalData() + { + var token = TestContext.Current.CancellationToken; + using var distributed = new InMemoryCacheClient(); + var subscription = new Mock(); + long version = 0; + Task ready = Task.CompletedTask; + subscription.SetupGet(s => s.RecoveryVersion).Returns(() => version); + subscription.Setup(s => s.WaitUntilReadyAsync(It.IsAny())).Returns((CancellationToken ct) => ready.WaitAsync(ct)); + var bus = new Mock(); + bus.SetupGet(b => b.SupportsTemporarySubscriptions).Returns(true); + bus.Setup(b => b.SubscribeAsync(It.IsAny, CancellationToken, Task>>(), It.IsAny(), It.IsAny())).ReturnsAsync(subscription.Object); + using var cache = new HybridCacheClient(distributed, bus.Object); + await distributed.SetAsync("key", "old"); + Assert.Equal("old", (await cache.GetAsync("key")).Value); + await distributed.SetAsync("key", "new"); + var restored = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + ready = restored.Task; + version++; + var read = cache.GetAsync("key"); + Assert.False(read.IsCompleted); + restored.SetResult(); + Assert.Equal("new", (await read.WaitAsync(TimeSpan.FromSeconds(5), token)).Value); + } + + [Fact] + public async Task DirectReceive_CancelledBeforeDisposal_ReturnsWorkImmediately() + { + var token = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport, new() { OwnsTransport = false }); + await bus.SendAsync(new Event(), cancellationToken: token); + using var receiving = CancellationTokenSource.CreateLinkedTokenSource(token); + var delivery = await bus.ReceiveAsync(cancellationToken: receiving.Token); + Assert.NotNull(delivery); + await receiving.CancelAsync(); + await delivery.DisposeAsync(); + await using var redelivery = await bus.ReceiveAsync(cancellationToken: token); + Assert.NotNull(redelivery); + } + + public sealed record Event; +} diff --git a/tests/Foundatio.Tests/Messaging/ScheduledMessageDispatcherTests.cs b/tests/Foundatio.Tests/Messaging/ScheduledMessageDispatcherTests.cs new file mode 100644 index 000000000..90cc1eae7 --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/ScheduledMessageDispatcherTests.cs @@ -0,0 +1,105 @@ +using System; +using System.Threading.Tasks; +using Foundatio.Extensions.Hosting.Messaging; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Time.Testing; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class ScheduledMessageDispatcherTests +{ + [Fact] + public async Task HostedDispatcher_WithOnlyDispatchStore_DrainsMessagesAsync() + { + var token = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var services = new ServiceCollection().AddLogging(); + services.AddSingleton(store); + services.AddFoundatio().Messaging.UseInMemory(); + services.AddScheduledMessageDispatcher(); + services.AddScheduledMessageDispatcher(); + await using var provider = services.BuildServiceProvider(); + Assert.Null(provider.GetService()); + var destination = DestinationAddress.ForQueue("hosted-dispatch"); + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = "hosted", + Destination = destination, + DueUtc = DateTimeOffset.UtcNow, + Body = "hello"u8.ToArray() + }, token); + var host = Assert.Single(provider.GetServices()); + await host.StartAsync(token); + try + { + var pull = Assert.IsAssignableFrom(provider.GetRequiredService()); + var received = await pull.ReceiveAsync(destination, new ReceiveRequest { MaxMessages = 1, MaxWaitTime = TimeSpan.FromSeconds(5) }, token); + Assert.Equal("hosted", Assert.Single(received).ApplicationMessageId); + } + finally + { + await host.StopAsync(token); + } + } + + [Fact] + public async Task DispatchDueAsync_WithOnlyMessagingDependencies_SendsAndRetiresDueMessages() + { + var token = TestContext.Current.CancellationToken; + var time = new FakeTimeProvider(); + var store = new InMemoryJobRuntimeStore(time); + await using var transport = new InMemoryMessageTransport(); + var destination = DestinationAddress.ForQueue("scheduled-work"); + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = "dispatch-1", + Destination = destination, + DueUtc = time.GetUtcNow(), + Body = "hello"u8.ToArray(), + Headers = MessageHeaders.Create(new System.Collections.Generic.Dictionary + { + [KnownHeaders.MessageId] = "application-1", + [KnownHeaders.ContentType] = "text/plain" + }) + }, token); + var dispatcher = new ScheduledMessageDispatcher(store, transport, new ScheduledMessageDispatcherOptions { TimeProvider = time }); + + Assert.Equal(1, await dispatcher.DispatchDueAsync(cancellationToken: token)); + Assert.Equal(0, await dispatcher.DispatchDueAsync(cancellationToken: token)); + var entry = Assert.Single(await transport.ReceiveAsync(destination, new ReceiveRequest { MaxMessages = 1 }, token)); + Assert.Equal("application-1", entry.ApplicationMessageId); + Assert.Equal("text/plain", entry.ContentType); + Assert.Equal("hello"u8.ToArray(), entry.Body.ToArray()); + } + + [Fact] + public async Task DispatchDueAsync_InValidateMode_DoesNotCreateMissingDestination() + { + var token = TestContext.Current.CancellationToken; + var time = new FakeTimeProvider(); + var store = new InMemoryJobRuntimeStore(time); + await using var transport = new InMemoryMessageTransport(); + var destination = DestinationAddress.ForQueue("missing"); + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = "missing-destination", + Destination = destination, + Body = ReadOnlyMemory.Empty, + DueUtc = time.GetUtcNow() + }, token); + var dispatcher = new ScheduledMessageDispatcher(store, transport, new ScheduledMessageDispatcherOptions + { + TimeProvider = time, + TopologyMode = TopologyMode.Validate + }); + + Assert.Equal(0, await dispatcher.DispatchDueAsync(cancellationToken: token)); + Assert.False(await transport.ExistsAsync(destination, token)); + time.Advance(TimeSpan.FromMinutes(1)); + Assert.Single(await store.ClaimDueDispatchesAsync(time.GetUtcNow(), 1, "new-claim", TimeSpan.FromMinutes(1), token)); + } +} diff --git a/tests/Foundatio.Tests/Messaging/SubscriptionRecoveryTests.cs b/tests/Foundatio.Tests/Messaging/SubscriptionRecoveryTests.cs new file mode 100644 index 000000000..448a7c169 --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/SubscriptionRecoveryTests.cs @@ -0,0 +1,148 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Microsoft.Extensions.Time.Testing; +using Moq; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class SubscriptionRecoveryTests +{ + [Fact] + public async Task ConcurrentReceives_DestinationDisappears_CancelsSiblingBeforeReprovisioning() + { + var token = TestContext.Current.CancellationToken; + var transport = new Mock(); + transport.As().SetupGet(t => t.SupportedRoles).Returns(new HashSet { DestinationRole.Queue }); + transport.As().Setup(t => t.GetCapabilities(It.IsAny())) + .Returns(new TransportCapabilities { MaxReceiveBatchSize = 1, MaxConcurrentReceives = 2 }); + var siblingStarted = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var siblingCancelled = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var recovered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + int receives = 0; + int provisions = 0; + transport.As().Setup(t => t.EnsureAsync(It.IsAny>(), It.IsAny())) + .Returns(() => + { + if (Interlocked.Increment(ref provisions) > 1) + { + Assert.True(siblingCancelled.Task.IsCompleted); + recovered.TrySetResult(); + } + return Task.CompletedTask; + }); + transport.As().Setup(t => t.ReceiveAsync(It.IsAny(), It.IsAny(), It.IsAny())) + .Returns(async (DestinationAddress source, ReceiveRequest _, CancellationToken ct) => + { + int call = Interlocked.Increment(ref receives); + if (call == 1) + { + await siblingStarted.Task.WaitAsync(ct); + throw new MessageDestinationNotFoundException(source, new InvalidOperationException("Queue deleted")); + } + if (call == 2) + { + siblingStarted.TrySetResult(); + try { await Task.Delay(Timeout.InfiniteTimeSpan, ct); } + finally { siblingCancelled.TrySetResult(); } + } + return (IReadOnlyList)Array.Empty(); + }); + await using var bus = new MessageBus(transport.Object); + await using var subscription = await bus.ConsumeAsync((_, _) => Task.CompletedTask, + new MessageConsumerOptions { Destination = "work", MaxConcurrency = 2 }, token); + await recovered.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + Assert.True(subscription.RecoveryVersion > 0); + } + + [Fact] + public async Task NamedSubscription_DeletedWhileListening_RebindsAndSignalsGap() + { + var token = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport, new() { OwnsTransport = false }); + var received = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var subscription = await bus.SubscribeAsync((_, _) => { received.TrySetResult(); return Task.CompletedTask; }, new() { Topic = "events", Subscription = "audit" }, token); + await subscription.WaitUntilReadyAsync(token); + await transport.DeleteAsync(subscription.Source, token); + + using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(5)); + while (subscription.RecoveryVersion == 0) + await Task.Delay(10, timeout.Token); + await subscription.WaitUntilReadyAsync(timeout.Token); + await bus.PublishAsync(new Event(), new() { Topic = "events" }, token); + + await received.Task.WaitAsync(TimeSpan.FromSeconds(2), token); + } + + [Fact] + public async Task TemporarySubscription_TransientRenewalFailure_RetriesWithinLease() + { + var token = TestContext.Current.CancellationToken; + var time = new FakeTimeProvider(); + var transport = new Mock(); + transport.As().SetupGet(t => t.SupportedRoles).Returns(new HashSet { DestinationRole.Topic, DestinationRole.Subscription }); + transport.As().Setup(t => t.GetCapabilities(It.IsAny())).Returns(new TransportCapabilities()); + transport.As().Setup(t => t.ReceiveAsync(It.IsAny(), It.IsAny(), It.IsAny())) + .ReturnsAsync(Array.Empty()); + transport.As().Setup(t => t.EnsureAsync(It.IsAny>(), It.IsAny())).Returns(Task.CompletedTask); + var retried = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + int renewals = 0; + transport.As().Setup(t => t.RenewSubscriptionAsync(It.IsAny(), It.IsAny(), It.IsAny())) + .Returns(() => + { + if (Interlocked.Increment(ref renewals) == 1) + throw new InvalidOperationException("Temporary outage"); + retried.TrySetResult(); + return Task.FromResult(true); + }); + await using var bus = new MessageBus(transport.Object, new MessageBusOptions { TimeProvider = time, OwnsTransport = false }); + await using var subscription = await bus.SubscribeAsync((_, _) => Task.CompletedTask, cancellationToken: token); + await Task.Delay(30, token); + time.Advance(TimeSpan.FromSeconds(30)); + await Task.Delay(30, token); + time.Advance(TimeSpan.FromSeconds(2)); + await retried.Task.WaitAsync(TimeSpan.FromSeconds(2), token); + } + + [Fact] + public async Task TemporarySubscription_LostLease_RecreatesAndSignalsGap() + { + var token = TestContext.Current.CancellationToken; + var time = new FakeTimeProvider(); + var transport = new Mock(); + transport.As().SetupGet(t => t.SupportedRoles).Returns(new HashSet { DestinationRole.Topic, DestinationRole.Subscription }); + transport.As().Setup(t => t.GetCapabilities(It.IsAny())).Returns(new TransportCapabilities()); + transport.As().Setup(t => t.ReceiveAsync(It.IsAny(), It.IsAny(), It.IsAny())) + .ReturnsAsync(Array.Empty()); + int declarations = 0; + transport.As().Setup(t => t.EnsureAsync(It.IsAny>(), It.IsAny())) + .Callback(() => Interlocked.Increment(ref declarations)).Returns(Task.CompletedTask); + transport.As().Setup(t => t.RenewSubscriptionAsync(It.IsAny(), It.IsAny(), It.IsAny())).ReturnsAsync(false); + await using var bus = new MessageBus(transport.Object, new MessageBusOptions { TimeProvider = time, OwnsTransport = false }); + await using var subscription = await bus.SubscribeAsync((_, _) => Task.CompletedTask, cancellationToken: token); + await subscription.WaitUntilReadyAsync(token); + int initialDeclarations = Volatile.Read(ref declarations); + for (int i = 0; i < 40 && Volatile.Read(ref declarations) == initialDeclarations; i++) + { + time.Advance(TimeSpan.FromSeconds(5)); + await Task.Delay(10, token); + } + Assert.True(Volatile.Read(ref declarations) > initialDeclarations); + await subscription.WaitUntilReadyAsync(token); + Assert.True(subscription.RecoveryVersion > 0); + } + + [Fact] + public void UnmatchedMessage_DefaultRetry_AllowsRollingDeployment() + { + var policy = new RetryPolicy(); + Assert.NotNull(policy.UnmatchedBackoff); + Assert.InRange(policy.UnmatchedBackoff(1), TimeSpan.FromSeconds(4), TimeSpan.FromSeconds(6)); + } + + public sealed record Event; +} diff --git a/tests/Foundatio.Tests/Messaging/TopologyModeTests.cs b/tests/Foundatio.Tests/Messaging/TopologyModeTests.cs new file mode 100644 index 000000000..58554bfcc --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/TopologyModeTests.cs @@ -0,0 +1,93 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.AsyncEx; +using Foundatio.Messaging; +using Foundatio.Tests.Extensions; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class TopologyModeTests +{ + [Fact] + public async Task Validate_WithPreProvisionedTopology_DeliversAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + // Provision out of band (the admin path), then run the bus in validate-only mode. + var topic = DestinationAddress.ForTopic("topology-event"); + var subscription = DestinationAddress.ForSubscription("topology-event", "svc"); + await transport.EnsureAsync([new DestinationDeclaration { Address = topic }, new DestinationDeclaration { Address = subscription }], cancellationToken); + + await using var bus = new MessageBus(transport, new MessageBusOptions { Topology = TopologyMode.Validate, OwnsTransport = false }); + var received = new AsyncCountdownEvent(1); + await using var handle = await bus.SubscribeAsync((message, _) => + { + Assert.Equal("hello", message.Message.Data); + received.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "svc" }, cts.Token); + + await bus.PublishAsync(new TopologyEvent { Data = "hello" }, cancellationToken: cancellationToken); + await received.WaitAsync(TimeSpan.FromSeconds(2)); + } + + [Fact] + public async Task Validate_WithMissingTopology_ThrowsAndCreatesNothingAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport, new MessageBusOptions { Topology = TopologyMode.Validate, OwnsTransport = false }); + + await Assert.ThrowsAsync(() => bus.PublishAsync(new TopologyEvent { Data = "hello" }, cancellationToken: cancellationToken)); + await Assert.ThrowsAsync(() => bus.SubscribeAsync((_, _) => Task.CompletedTask, new() { Subscription = "missing" }, cancellationToken: cancellationToken)); + + Assert.False(await transport.ExistsAsync(DestinationAddress.ForTopic("topology-event"), cancellationToken)); + } + + [Fact] + public async Task None_NeverTouchesTopologyAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport, new MessageBusOptions { Topology = TopologyMode.None, OwnsTransport = false }); + + // Publishing to a topic that was never provisioned must not create it (real pub/sub drop semantics). + await bus.PublishAsync(new TopologyEvent { Data = "dropped" }, cancellationToken: cancellationToken); + Assert.False(await transport.ExistsAsync(DestinationAddress.ForTopic("topology-event"), cancellationToken)); + } + + [Fact] + public async Task None_WithPreProvisionedTopology_DeliversAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var topic = DestinationAddress.ForTopic("topology-event"); + var subscription = DestinationAddress.ForSubscription("topology-event", "svc"); + await transport.EnsureAsync([new DestinationDeclaration { Address = topic }, new DestinationDeclaration { Address = subscription }], cancellationToken); + + await using var bus = new MessageBus(transport, new MessageBusOptions { Topology = TopologyMode.None, OwnsTransport = false }); + var received = new AsyncCountdownEvent(1); + await using var handle = await bus.SubscribeAsync((message, _) => + { + received.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "svc" }, cts.Token); + + await bus.PublishAsync(new TopologyEvent { Data = "hello" }, cancellationToken: cancellationToken); + await received.WaitAsync(TimeSpan.FromSeconds(2)); + } + + [MessageRoute("topology-event")] + private sealed class TopologyEvent + { + public string? Data { get; set; } + } +} diff --git a/tests/Foundatio.Tests/Messaging/WireContractTests.cs b/tests/Foundatio.Tests/Messaging/WireContractTests.cs new file mode 100644 index 000000000..112a66496 --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/WireContractTests.cs @@ -0,0 +1,229 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Foundatio.Serializer; +using Moq; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class WireContractTests +{ + [Fact] + public void HeaderBuilder_ReusedAfterBuild_PreservesEveryPublishedSnapshot() + { + var builder = MessageHeaders.Empty.ToBuilder().Set("tenant", "first"); + var first = builder.Build(); + builder.Set("TENANT", "second").Add("trace", "123"); + var second = builder.Build(); + builder.Remove("tenant"); + builder.SetIfMissing("trace", "ignored").SetIfMissing("extra", "value"); + var third = builder.Build(); + Assert.Equal("first", first["tenant"]); + Assert.Single(first); + Assert.Equal("second", second["tenant"]); + Assert.Equal(2, second.Count); + Assert.False(third.ContainsKey("tenant")); + Assert.Equal("123", third["trace"]); + Assert.Equal("value", third["extra"]); + Assert.Equal(2, third.Count); + + var copy = first.ToBuilder(); + copy.SetIfMissing("TENANT", "ignored"); + Assert.False(copy.Remove("missing")); + Assert.Equal("first", copy.Build()["tenant"]); + copy.Set("tenant", "copy"); + Assert.Equal("first", first["tenant"]); + Assert.Equal("copy", copy.Build()["tenant"]); + } + + [Theory] + [InlineData(false, 0)] + [InlineData(false, 1)] + [InlineData(false, 2)] + [InlineData(true, 0)] + [InlineData(true, 1)] + [InlineData(true, 2)] + public async Task SendAsync_WithInterfaceContract_PreservesConcreteWireType(bool publish, int batchKind) + { + var token = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + var registry = new MessageTypeRegistry([new("changed.v1", typeof(Changed))]); + await using var bus = new MessageBus(transport, new() { MessageTypes = registry, OwnsTransport = false }); + var received = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var listener = publish + ? await bus.SubscribeAsync((m, _) => { received.TrySetResult(m.Message); return Task.CompletedTask; }, new() { Topic = "changes", Subscription = "audit" }, token) + : await bus.ConsumeAsync((m, _) => { received.TrySetResult(m.Message); return Task.CompletedTask; }, new() { Destination = "changes" }, token); + IChange change = new Changed(42); + if (publish) + { + var options = new MessagePublishOptions { Topic = "changes" }; + if (batchKind == 0) await bus.PublishAsync(change, options, token); + else if (batchKind == 1) await bus.PublishBatchAsync([change], options, token); + else await bus.PublishBatchAsync([new MessageBatchItem(change, "change-42")], options, token); + } + else + { + var options = new MessageSendOptions { Destination = "changes" }; + if (batchKind == 0) await bus.SendAsync(change, options, token); + else if (batchKind == 1) await bus.SendBatchAsync([change], options, token); + else await bus.SendBatchAsync([new MessageBatchItem(change, "change-42")], options, token); + } + + Assert.Equal(42, Assert.IsType(await received.Task.WaitAsync(TimeSpan.FromSeconds(2), token)).Id); + } + + public interface IChange { int Id { get; } } + public sealed record Changed(int Id) : IChange; + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task ReceiveAsync_WithObjectContract_ResolvesRegisteredConcreteType(bool listener) + { + var token = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport, new() + { + OwnsTransport = false, + MessageTypes = new MessageTypeRegistry([new("changed.v1", typeof(Changed))]) + }); + await bus.SendAsync(new Changed(42), new() { Destination = "changes" }, token); + if (listener) + { + var received = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var subscription = await bus.ConsumeAsync((m, _) => { received.TrySetResult(m.Message); return Task.CompletedTask; }, new() { Destination = "changes" }, token); + Assert.Equal(42, Assert.IsType(await received.Task.WaitAsync(TimeSpan.FromSeconds(2), token)).Id); + } + else + { + await using var received = await bus.ReceiveAsync(new() { Destination = "changes" }, token); + Assert.Equal(42, Assert.IsType(received!.Message).Id); + await received.CompleteAsync(token); + } + } + + [Fact] + public void MessageTypeRegistry_ResolvesOnlyExplicitlyRegisteredTypes() + { + var registry = new MessageTypeRegistry(new[] { new MessageTypeRegistration("work.v1", typeof(Work)) }); + Assert.Equal(typeof(Work), registry.Resolve("work.v1")); + Assert.Null(registry.Resolve(typeof(Work).AssemblyQualifiedName!)); + Assert.Null(new MessageTypeRegistry().Resolve(typeof(Work).FullName!)); + } + + [Fact] + public async Task ReceiveAsync_WithIncompatibleContentType_DeadLettersWithoutDeserializingAsync() + { + var token = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + var queue = DestinationAddress.ForQueue("work"); + await transport.SendAsync(queue, new[] { new TransportMessage { Body = new byte[] { 0xff, 0x80 }, ContentType = "application/octet-stream" } }, new TransportSendOptions(), token); + await using var bus = new MessageBus(transport); + await Assert.ThrowsAsync(() => bus.ReceiveAsync(new MessageReceiveOptions { Destination = "work" }, token)); + var dead = Assert.Single(await transport.PeekDeadLetteredAsync(queue, new DeadLetterQuery(), token)); + Assert.Equal("unsupported-content-type", dead.Headers[KnownHeaders.DeadLetterReason]); + Assert.Equal("application/octet-stream", dead.ContentType); + } + + [Fact] + public async Task SendBatchAsync_WhenSecondChunkFails_ReportsEveryInputOutcomeAsync() + { + var transport = new Mock(); + transport.As().SetupGet(t => t.SupportedRoles).Returns(new HashSet { DestinationRole.Queue }); + transport.As().Setup(t => t.GetCapabilities(It.IsAny())).Returns(new TransportCapabilities { MaxBatchSize = 1 }); + transport.SetupSequence(t => t.SendAsync(It.IsAny(), It.IsAny>(), It.IsAny(), It.IsAny())) + .ReturnsAsync(new SendResult { Items = new[] { new SendItemResult { MessageId = "broker-id" } } }) + .ThrowsAsync(new TimeoutException("Acceptance is unknown")); + await using var bus = new MessageBus(transport.Object); + var error = await Assert.ThrowsAsync(() => bus.SendBatchAsync(new[] { new Work(), new Work(), new Work() }, cancellationToken: TestContext.Current.CancellationToken)); + Assert.Equal(new[] { MessageSendStatus.Accepted, MessageSendStatus.Unknown, MessageSendStatus.NotAttempted }, error.Outcomes.Select(o => o.Status)); + Assert.Equal(3, error.Outcomes.Select(o => o.MessageId).Distinct().Count()); + Assert.All(error.Outcomes, o => Assert.NotEmpty(o.MessageId)); + transport.Verify(t => t.SendAsync(It.IsAny(), It.IsAny>(), It.IsAny(), It.IsAny()), Times.Exactly(2)); + } + + [Fact] + public async Task SendAsync_WhenBrokerAssignsAnotherId_ReturnsApplicationIdAsync() + { + var sent = new List(); + await using var bus = new MessageBus(CreateTransport(sent)); + string id = await bus.SendAsync(new Work(), cancellationToken: TestContext.Current.CancellationToken); + Assert.Equal(Assert.Single(sent).MessageId, id); + Assert.NotEqual("broker-id", id); + Assert.Equal(id, sent[0].Headers[KnownHeaders.MessageId]); + } + + [Fact] + public async Task SendBatchAsync_WhenBrokerAssignsOtherIds_PreservesApplicationIdsAsync() + { + var sent = new List(); + await using var bus = new MessageBus(CreateTransport(sent)); + var ids = await bus.SendBatchAsync(new[] { new Work(), new Work() }, cancellationToken: TestContext.Current.CancellationToken); + Assert.Equal(sent.Select(m => m.MessageId), ids); + Assert.Equal(2, ids.Distinct().Count()); + } + + [Fact] + public async Task SendAsync_WithCallerId_PreservesIdAcrossAttemptsAsync() + { + var sent = new List(); + await using var bus = new MessageBus(CreateTransport(sent)); + var options = new MessageSendOptions { MessageId = "order-123" }; + Assert.Equal("order-123", await bus.SendAsync(new Work(), options, TestContext.Current.CancellationToken)); + Assert.Equal("order-123", await bus.SendAsync(new Work(), options, TestContext.Current.CancellationToken)); + Assert.All(sent, m => Assert.Equal("order-123", m.MessageId)); + } + + [Fact] + public async Task SendAsync_WithBinarySerializer_AdvertisesBinaryBodyAsync() + { + var sent = new List(); + await using var bus = new MessageBus(CreateTransport(sent), new MessageBusOptions { Serializer = new BinarySerializer() }); + await bus.SendAsync(new Work(), cancellationToken: TestContext.Current.CancellationToken); + var message = Assert.Single(sent); + Assert.Equal("application/octet-stream", message.ContentType); + Assert.Equal(message.ContentType, message.Headers[KnownHeaders.ContentType]); + Assert.Equal(new byte[] { 0xff, 0x80, 0x00 }, message.Body.ToArray()); + } + + [Fact] + public void MessageContext_WithApplicationHeader_SeparatesApplicationAndBrokerIds() + { + var entry = new TransportEntry + { + Id = "broker-id", + Destination = DestinationAddress.ForQueue("work"), + Body = ReadOnlyMemory.Empty, + Headers = MessageHeaders.Empty.ToBuilder().Set(KnownHeaders.MessageId, "order-123").Build(), + Receipt = default + }; + var context = new MessageContext(Mock.Of(), entry, CancellationToken.None); + Assert.Equal("order-123", context.Id); + Assert.Equal("broker-id", context.BrokerMessageId); + } + + private static IMessageTransport CreateTransport(List sent) + { + var transport = new Mock(); + transport.Setup(t => t.SendAsync(It.IsAny(), It.IsAny>(), It.IsAny(), It.IsAny())) + .Returns((DestinationAddress _, IReadOnlyList messages, TransportSendOptions _, CancellationToken _) => + { + sent.AddRange(messages); + return Task.FromResult(new SendResult { Items = messages.Select(_ => new SendItemResult { MessageId = "broker-id" }).ToArray() }); + }); + return transport.Object; + } + + private sealed class Work; + + private sealed class BinarySerializer : ISerializer + { + public object? Deserialize(Stream data, Type objectType) => throw new NotSupportedException(); + public void Serialize(object? value, Stream output) => output.Write(new byte[] { 0xff, 0x80, 0x00 }); + } +} diff --git a/tests/Foundatio.Tests/Queue/BasicQueueTransport.cs b/tests/Foundatio.Tests/Queue/BasicQueueTransport.cs new file mode 100644 index 000000000..2a7d2047c --- /dev/null +++ b/tests/Foundatio.Tests/Queue/BasicQueueTransport.cs @@ -0,0 +1,183 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; + +namespace Foundatio.Tests.Queue; + +/// +/// A deliberately minimal transport for tests: basic competing-consumer pull semantics only — no native redelivery +/// delay, lock renewal, visibility timeout, delayed delivery, priority, or expiration. Headers are treated as opaque +/// (preserved but never interpreted) and the delivery count is owned solely by the transport, so it never seeds the +/// count from the message.attempts header. This models a real provider that lacks time-based capabilities, +/// which exercises the runtime-store fallbacks and proves the core reconciles the attempt count itself rather than +/// relying on the transport to honor the header. +/// +internal sealed class BasicQueueTransport : IMessageTransport, ISupportsPull, ISupportsDeadLetter, ISupportsStats +{ + private readonly ConcurrentDictionary _destinations = new(StringComparer.OrdinalIgnoreCase); + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + var dest = _destinations.GetOrAdd(destination.Key, static _ => new Destination()); + var results = new SendItemResult[messages.Count]; + for (int index = 0; index < messages.Count; index++) + { + var message = messages[index]; + string id = message.MessageId ?? Guid.NewGuid().ToString("N"); + dest.Ready.Enqueue(new StoredEntry(id, message.Body, message.Headers, DeliveryCount: 1)); + Interlocked.Increment(ref dest.Enqueued); + results[index] = new SendItemResult { MessageId = id }; + } + + return Task.FromResult(new SendResult { Items = results }); + } + + public async Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + { + var dest = _destinations.GetOrAdd(source.Key, static _ => new Destination()); + int max = request.MaxMessages <= 0 ? 1 : request.MaxMessages; + DateTimeOffset? deadline = request.MaxWaitTime is { } wait && wait > TimeSpan.Zero ? DateTimeOffset.UtcNow.Add(wait) : null; + var entries = new List(max); + + while (true) + { + while (entries.Count < max && dest.Ready.TryDequeue(out var stored)) + { + string token = Guid.NewGuid().ToString("N"); + dest.InFlight[token] = stored; + Interlocked.Increment(ref dest.Dequeued); + entries.Add(new TransportEntry + { + Id = stored.Id, + Destination = source, + Body = stored.Body, + Headers = stored.Headers, + DeliveryCount = stored.DeliveryCount, + Receipt = new Receipt { TransportState = new BasicReceipt(source.Key, token) } + }); + } + + if (entries.Count > 0 || deadline is null || DateTimeOffset.UtcNow >= deadline) + return entries; + + await Task.Delay(TimeSpan.FromMilliseconds(15), ct).ConfigureAwait(false); + } + } + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) + { + var (dest, token) = Locate(entry); + if (!dest.InFlight.TryRemove(token, out _)) + throw new ReceiptExpiredException(); + + Interlocked.Increment(ref dest.Completed); + return Task.CompletedTask; + } + + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) + { + var (dest, token) = Locate(entry); + if (!dest.InFlight.TryRemove(token, out var stored)) + throw new ReceiptExpiredException(); + + Interlocked.Increment(ref dest.Abandoned); + dest.Ready.Enqueue(stored with { DeliveryCount = stored.DeliveryCount + 1 }); + return Task.CompletedTask; + } + + public Task DeadLetterAsync(TransportEntry entry, string? reason, CancellationToken ct) + { + var (dest, token) = Locate(entry); + if (!dest.InFlight.TryRemove(token, out var stored)) + throw new ReceiptExpiredException(); + + var headers = String.IsNullOrEmpty(reason) ? stored.Headers : stored.Headers.ToBuilder().Set(KnownHeaders.DeadLetterReason, reason).Build(); + dest.Dead[stored.Id] = stored with { Headers = headers }; + return Task.CompletedTask; + } + + public Task> PeekDeadLetteredAsync(DestinationAddress destination, DeadLetterQuery? query = null, CancellationToken cancellationToken = default) + { + var entries = new List(); + if (_destinations.TryGetValue(destination.Key, out var dest)) + { + query ??= new DeadLetterQuery(); + query.Validate(); + foreach (var stored in dest.Dead.Values.Where(v => query.AfterId is null || StringComparer.Ordinal.Compare(v.Id, query.AfterId) > 0).OrderBy(v => v.Id, StringComparer.Ordinal).Take(query.Limit)) + { + entries.Add(new TransportEntry + { + Id = stored.Id, + Destination = destination, + Body = stored.Body, + Headers = stored.Headers, + DeliveryCount = stored.DeliveryCount, + Receipt = new Receipt { TransportState = null } + }); + } + } + + return Task.FromResult>(entries); + } + + public Task DeleteDeadLetteredAsync(DestinationAddress destination, string id, CancellationToken cancellationToken = default) + => Task.FromResult(_destinations.TryGetValue(destination.Key, out var dest) && dest.Dead.TryRemove(id, out _)); + + public async Task ReplayDeadLetteredAsync(DestinationAddress source, string id, DestinationAddress target, CancellationToken cancellationToken = default) + { + if (!_destinations.TryGetValue(source.Key, out var dest) || !dest.Dead.TryGetValue(id, out var stored)) + return false; + await SendAsync(target, [new TransportMessage { Body = stored.Body, Headers = stored.Headers, MessageId = stored.Id }], new TransportSendOptions(), cancellationToken); + return dest.Dead.TryRemove(id, out _); + } + + public Task GetStatsAsync(DestinationAddress destination, CancellationToken ct) + { + if (!_destinations.TryGetValue(destination.Key, out var dest)) + return Task.FromResult(new MessageDestinationStats()); + + return Task.FromResult(new MessageDestinationStats + { + Queued = dest.Ready.Count, + Working = dest.InFlight.Count, + Deadletter = dest.Dead.Count, + Enqueued = Interlocked.Read(ref dest.Enqueued), + Dequeued = Interlocked.Read(ref dest.Dequeued), + Completed = Interlocked.Read(ref dest.Completed), + Abandoned = Interlocked.Read(ref dest.Abandoned) + }); + } + + public ValueTask DisposeAsync() + { + _destinations.Clear(); + return ValueTask.CompletedTask; + } + + private (Destination Destination, string Token) Locate(TransportEntry entry) + { + if (entry.Receipt.TransportState is not BasicReceipt receipt || !_destinations.TryGetValue(receipt.Destination, out var dest)) + throw new ReceiptExpiredException(); + + return (dest, receipt.Token); + } + + private sealed record StoredEntry(string Id, ReadOnlyMemory Body, MessageHeaders Headers, int DeliveryCount); + + private sealed record BasicReceipt(string Destination, string Token); + + private sealed class Destination + { + public readonly ConcurrentQueue Ready = new(); + public readonly ConcurrentDictionary Dead = new(); + public readonly ConcurrentDictionary InFlight = new(StringComparer.Ordinal); + public long Enqueued; + public long Dequeued; + public long Completed; + public long Abandoned; + } +} diff --git a/tests/Foundatio.Tests/Queue/InMemoryQueueTests.cs b/tests/Foundatio.Tests/Queue/InMemoryQueueTests.cs deleted file mode 100644 index 11408d1ca..000000000 --- a/tests/Foundatio.Tests/Queue/InMemoryQueueTests.cs +++ /dev/null @@ -1,594 +0,0 @@ -using System; -using System.Collections.Generic; -using System.Diagnostics; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Queues; -using Foundatio.Serializer; -using Microsoft.Extensions.Logging; -using Xunit; - -namespace Foundatio.Tests.Queue; - -public class InMemoryQueueTests : QueueTestBase -{ - private IQueue? _queue; - - public InMemoryQueueTests(ITestOutputHelper output) : base(output) { } - - protected override IQueue GetQueue(int retries = 1, TimeSpan? workItemTimeout = null, TimeSpan? retryDelay = null, int[]? retryMultipliers = null, int deadLetterMaxItems = 100, bool runQueueMaintenance = true, TimeProvider? timeProvider = null, ISerializer? serializer = null) - { - if (_queue is null) - _queue = new InMemoryQueue(o => o - .RetryDelay(retryDelay.GetValueOrDefault(TimeSpan.FromMinutes(1))) - .Retries(retries) - .RetryMultipliers(retryMultipliers ?? new[] { 1, 3, 5, 10 }) - .WorkItemTimeout(workItemTimeout.GetValueOrDefault(TimeSpan.FromMinutes(5))) - .TimeProvider(timeProvider) - .MetricsPollingInterval(TimeSpan.Zero) - .LoggerFactory(Log)); - _logger.LogDebug("Queue Id: {QueueId}", _queue.QueueId); - return _queue; - } - - protected override async Task CleanupQueueAsync(IQueue queue) - { - if (queue is null) - return; - - try - { - await queue.DeleteQueueAsync(); - } - catch (Exception ex) - { - _logger.LogError(ex, "Error cleaning up queue"); - } - } - - [Fact] - public override Task CanQueueAndDequeueWorkItemAsync() - { - return base.CanQueueAndDequeueWorkItemAsync(); - } - - [Fact] - public override Task CanQueueAndDequeueWorkItemWithDelayAsync() - { - return base.CanQueueAndDequeueWorkItemWithDelayAsync(); - } - - [Fact] - public override Task CanUseQueueOptionsAsync() - { - return base.CanUseQueueOptionsAsync(); - } - - [Fact] - public override Task CanDiscardDuplicateQueueEntriesAsync() - { - return base.CanDiscardDuplicateQueueEntriesAsync(); - } - - [Fact] - public override Task DuplicateDetection_WithDifferentIdentifiers_AcceptsBothItemsAsync() - { - return base.DuplicateDetection_WithDifferentIdentifiers_AcceptsBothItemsAsync(); - } - - [Fact] - public override Task DuplicateDetection_WithExpiredWindow_AcceptsDuplicateAsync() - { - return base.DuplicateDetection_WithExpiredWindow_AcceptsDuplicateAsync(); - } - - [Fact] - public override Task DuplicateDetection_WithNullIdentifier_AcceptsAllItemsAsync() - { - return base.DuplicateDetection_WithNullIdentifier_AcceptsAllItemsAsync(); - } - - [Fact] - public override Task CanDequeueWithCancelledTokenAsync() - { - return base.CanDequeueWithCancelledTokenAsync(); - } - - [Fact] - public override Task CanDequeueEfficientlyAsync() - { - return base.CanDequeueEfficientlyAsync(); - } - - [Fact] - public override Task CanResumeDequeueEfficientlyAsync() - { - return base.CanResumeDequeueEfficientlyAsync(); - } - - [Fact] - public override Task CanQueueAndDequeueMultipleWorkItemsAsync() - { - return base.CanQueueAndDequeueMultipleWorkItemsAsync(); - } - - [Fact] - public override Task WillNotWaitForItemAsync() - { - return base.WillNotWaitForItemAsync(); - } - - [Fact] - public override Task WillWaitForItemAsync() - { - return base.WillWaitForItemAsync(); - } - - [Fact] - public override Task DequeueAsync_AfterAbandonWithMutatedValue_ReturnsOriginalValueAsync() - { - return base.DequeueAsync_AfterAbandonWithMutatedValue_ReturnsOriginalValueAsync(); - } - - [Fact(Skip = "InMemoryQueue does not use serialization")] - public override Task DequeueAsync_WithPoisonMessage_MovesToDeadletterAsync() - { - return base.DequeueAsync_WithPoisonMessage_MovesToDeadletterAsync(); - } - - [Fact(Skip = "InMemoryQueue does not use serialization")] - public override Task EnqueueAsync_WithSerializationError_ThrowsAndLeavesQueueEmptyAsync() - { - return base.EnqueueAsync_WithSerializationError_ThrowsAndLeavesQueueEmptyAsync(); - } - - [Fact] - public override Task DequeueWaitWillGetSignaledAsync() - { - return base.DequeueWaitWillGetSignaledAsync(); - } - - [Fact] - public override Task CanUseQueueWorkerAsync() - { - return base.CanUseQueueWorkerAsync(); - } - - [Fact] - public override Task CanHandleErrorInWorkerAsync() - { - return base.CanHandleErrorInWorkerAsync(); - } - - [Fact] - public override Task WorkItemsWillTimeoutAsync() - { - return base.WorkItemsWillTimeoutAsync(); - } - - [Fact] - public override Task WorkItemsWillGetMovedToDeadletterAsync() - { - return base.WorkItemsWillGetMovedToDeadletterAsync(); - } - - [Fact] - public override Task AbandonAsync_WhenRetriesExceeded_MovesToDeadletterAsync() - { - return base.AbandonAsync_WhenRetriesExceeded_MovesToDeadletterAsync(); - } - - [Fact] - public override Task CanAutoCompleteWorkerAsync() - { - return base.CanAutoCompleteWorkerAsync(); - } - - [Fact] - public override Task CanHaveMultipleQueueInstancesAsync() - { - return base.CanHaveMultipleQueueInstancesAsync(); - } - - [Fact] - public override Task CanDelayRetryAsync() - { - return base.CanDelayRetryAsync(); - } - - [Fact] - public override Task CanRunWorkItemWithMetricsAsync() - { - return base.CanRunWorkItemWithMetricsAsync(); - } - - [Fact] - public override Task CanRenewLockAsync() - { - return base.CanRenewLockAsync(); - } - - [Fact] - public override Task CanAbandonQueueEntryOnceAsync() - { - return base.CanAbandonQueueEntryOnceAsync(); - } - - [Fact] - public override Task CanCompleteQueueEntryOnceAsync() - { - return base.CanCompleteQueueEntryOnceAsync(); - } - - [Fact] - public override Task CanDequeueWithLockingAsync() - { - return base.CanDequeueWithLockingAsync(); - } - - [Fact] - public override Task CanHaveMultipleQueueInstancesWithLockingAsync() - { - return base.CanHaveMultipleQueueInstancesWithLockingAsync(); - } - - [Fact] - public override Task MaintainJobNotAbandon_NotWorkTimeOutEntry() - { - return base.MaintainJobNotAbandon_NotWorkTimeOutEntry(); - } - - [Fact] - public override Task VerifyRetryAttemptsAsync() - { - return base.VerifyRetryAttemptsAsync(); - } - - [Fact] - public override Task VerifyDelayedRetryAttemptsAsync() - { - return base.VerifyDelayedRetryAttemptsAsync(); - } - - [Fact] - public override Task CanHandleAutoAbandonInWorker() - { - return base.CanHandleAutoAbandonInWorker(); - } - - [Fact] - public override Task DequeueAsync_WithDispose_AutoAbandonsEntryAsync() - { - return base.DequeueAsync_WithDispose_AutoAbandonsEntryAsync(); - } - - [Fact] - public override Task Dispose_WithMaintenanceRunning_DoesNotThrowObjectDisposedException() - { - return base.Dispose_WithMaintenanceRunning_DoesNotThrowObjectDisposedException(); - } - - [Fact] - public override Task EnqueueAsync_WithUniqueId_UsesProvidedIdAsync() - { - return base.EnqueueAsync_WithUniqueId_UsesProvidedIdAsync(); - } - - [Fact] - public override Task GetDeadletterItemsAsync_WithDeadletteredEntry_ReturnsItemsAsync() - { - return base.GetDeadletterItemsAsync_WithDeadletteredEntry_ReturnsItemsAsync(); - } - - [Fact] - public override Task GetQueueActivity_AfterEnqueueAndDequeue_ReturnsTimestampsAsync() - { - return base.GetQueueActivity_AfterEnqueueAndDequeue_ReturnsTimestampsAsync(); - } - - [Fact] - public override Task GetQueueEntryMetadata_AfterDequeue_ReturnsValidTimestampsAsync() - { - return base.GetQueueEntryMetadata_AfterDequeue_ReturnsValidTimestampsAsync(); - } - - [Fact] - public override Task QueueEntry_EntryType_ReturnsCorrectTypeAsync() - { - return base.QueueEntry_EntryType_ReturnsCorrectTypeAsync(); - } - - [Fact] - public override Task QueueEntry_GetValue_ReturnsUntypedValueAsync() - { - return base.QueueEntry_GetValue_ReturnsUntypedValueAsync(); - } - - [Fact] - public async Task TestAsyncEvents() - { - using var q = new InMemoryQueue(o => o.LoggerFactory(Log)); - var disposables = new List(5); - try - { - disposables.Add(q.Enqueuing.AddHandler(async (sender, args) => - { - await Task.Delay(250); - _logger.LogInformation("First Enqueuing"); - })); - disposables.Add(q.Enqueuing.AddHandler(async (sender, args) => - { - await Task.Delay(250); - _logger.LogInformation("Second Enqueuing"); - })); - disposables.Add(q.Enqueued.AddHandler(async (sender, args) => - { - await Task.Delay(250); - _logger.LogInformation("First"); - })); - disposables.Add(q.Enqueued.AddHandler(async (sender, args) => - { - await Task.Delay(250); - _logger.LogInformation("Second"); - })); - - var sw = Stopwatch.StartNew(); - await q.EnqueueAsync(new SimpleWorkItem()); - sw.Stop(); - _logger.LogTrace("Time {Elapsed:g}", sw.Elapsed); - - sw.Restart(); - await q.EnqueueAsync(new SimpleWorkItem()); - sw.Stop(); - _logger.LogTrace("Time {Elapsed:g}", sw.Elapsed); - } - finally - { - foreach (var disposable in disposables) - disposable.Dispose(); - } - } - - [Fact] - public async Task CanGetCompletedEntries() - { - using var q = new InMemoryQueue(o => o.LoggerFactory(Log).CompletedEntryRetentionLimit(10)); - - await q.EnqueueAsync(new SimpleWorkItem()); - Assert.Single(q.GetEntries()); - Assert.Empty(q.GetDequeuedEntries()); - Assert.Empty(q.GetCompletedEntries()); - - var item = await q.DequeueAsync(); - Assert.Empty(q.GetEntries()); - Assert.Single(q.GetDequeuedEntries()); - Assert.Empty(q.GetCompletedEntries()); - - Assert.NotNull(item); - await item.CompleteAsync(); - Assert.Empty(q.GetEntries()); - Assert.Empty(q.GetDequeuedEntries()); - Assert.Single(q.GetCompletedEntries()); - - for (int i = 0; i < 100; i++) - { - await q.EnqueueAsync(new SimpleWorkItem()); - item = await q.DequeueAsync(); - Assert.NotNull(item); - await item.CompleteAsync(); - } - - Assert.Empty(q.GetEntries()); - Assert.Empty(q.GetDequeuedEntries()); - Assert.Equal(10, q.GetCompletedEntries().Count); - } - - [Fact] - public async Task DeleteQueueAsync_WithEventHandler_RaisesQueueDeletedEvent() - { - // Arrange - using var q = new InMemoryQueue(o => o.LoggerFactory(Log)); - bool eventFired = false; - - using var handler = q.QueueDeleted.AddHandler((sender, args) => - { - eventFired = true; - Assert.Same(q, args.Queue); - return Task.CompletedTask; - }); - - await q.EnqueueAsync(new SimpleWorkItem()); - - // Act - await q.DeleteQueueAsync(); - - // Assert - Assert.True(eventFired); - } - - [Fact] - public async Task DeleteQueueAsync_WithAttachedBehavior_InvokesBehaviorOnQueueDeleted() - { - // Arrange - using var q = new InMemoryQueue(o => o.LoggerFactory(Log)); - var behavior = new QueueDeletedTestBehavior(); - q.AttachBehavior(behavior); - - await q.EnqueueAsync(new SimpleWorkItem()); - - // Act - await q.DeleteQueueAsync(); - - // Assert - Assert.True(behavior.QueueDeletedCalled); - } - - [Fact] - public void AttachBehavior_WhenAlreadyAttached_ThrowsQueueException() - { - // Arrange - using var q1 = new InMemoryQueue(o => o.LoggerFactory(Log)); - using var q2 = new InMemoryQueue(o => o.LoggerFactory(Log)); - var behavior = new QueueDeletedTestBehavior(); - q1.AttachBehavior(behavior); - - // Act & Assert - var ex = Assert.Throws(() => q2.AttachBehavior(behavior)); - Assert.Contains("already attached", ex.Message); - } - - [Fact] - public void AttachBehavior_WithNullQueue_ThrowsArgumentNullException() - { - // Arrange - var behavior = new QueueDeletedTestBehavior(); - - // Act & Assert - Assert.Throws(() => behavior.Attach(null!)); - } - - private class QueueDeletedTestBehavior : QueueBehaviorBase where T : class - { - public bool QueueDeletedCalled { get; private set; } - - protected override Task OnQueueDeleted(object sender, QueueDeletedEventArgs queueDeletedEventArgs) - { - QueueDeletedCalled = true; - return Task.CompletedTask; - } - } - - class QueueEntry_Issue239 : IQueueEntry where T : class - { - IQueueEntry _queueEntry; - - public QueueEntry_Issue239(IQueueEntry queueEntry) - { - _queueEntry = queueEntry; - } - - public T Value => _queueEntry.Value; - - public string Id => _queueEntry.Id; - - public string? CorrelationId => _queueEntry.CorrelationId; - - public IDictionary Properties => _queueEntry.Properties; - - public Type? EntryType => _queueEntry.EntryType; - - public bool IsCompleted => _queueEntry.IsCompleted; - - public bool IsAbandoned => _queueEntry.IsAbandoned; - - public int Attempts => _queueEntry.Attempts; - - public Task AbandonAsync() - { - return _queueEntry.AbandonAsync(); - } - - public Task CompleteAsync() - { - return _queueEntry.CompleteAsync(); - } - - public ValueTask DisposeAsync() - { - return _queueEntry.DisposeAsync(); - } - - public object GetValue() - { - return _queueEntry.GetValue(); - } - - public void MarkAbandoned() - { - // we want to simulate timing of user complete call between the maintenance abandon call to _dequeued.TryRemove and entry.MarkAbandoned(); - Task.Delay(1500).Wait(); - - _queueEntry.MarkAbandoned(); - } - - public void MarkCompleted() - { - _queueEntry.MarkCompleted(); - } - - public Task RenewLockAsync() - { - return _queueEntry.RenewLockAsync(); - } - } - - class InMemoryQueue_Issue239 : InMemoryQueue where T : class - { - public override Task AbandonAsync(IQueueEntry entry) - { - // delay first abandon from maintenance (simulate timing issues which may occur to demonstrate the problem) - return base.AbandonAsync(new QueueEntry_Issue239(entry)); - } - - public InMemoryQueue_Issue239(ILoggerFactory loggerFactory) - : base(o => o - .RetryDelay(TimeSpan.FromMinutes(1)) - .Retries(1) - .RetryMultipliers(new[] { 1, 3, 5, 10 }) - .LoggerFactory(loggerFactory) - .WorkItemTimeout(TimeSpan.FromMilliseconds(100))) - { - } - } - - [Fact] - // this test reproduce an issue which cause worker task loop to crash and stop processing items when auto abandoned item is ultimately processed and user call complete on - // https://github.com/FoundatioFx/Foundatio/issues/239 - public virtual async Task CompleteOnAutoAbandonedHandledProperly_Issue239() - { - // create queue with short work item timeout, so it will be auto abandoned - var queue = new InMemoryQueue_Issue239(Log); - using var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(30)); - - // completion source to wait for CompleteAsync call before to assert - var taskCompletionSource = new TaskCompletionSource(); - - // start handling items - await queue.StartWorkingAsync(async (item, ct) => - { - // we want to wait for maintenance to be performed and auto abandon our item, we don't have any way for waiting in IQueue so we'll settle for a delay - if (item.Value is { Data: "Delay" }) - { - await Task.Delay(TimeSpan.FromSeconds(1), ct); - } - - try - { - // call complete on the auto abandoned item - await item.CompleteAsync(); - } - finally - { - // completeAsync will currently throw an exception becuase item can not be removed from dequeued list because it was already removed due to auto abandon - // infrastructure handles user exception incorrectly - taskCompletionSource.SetResult(true); - } - }, cancellationToken: cancellationTokenSource.Token); - - // enqueue item which will be processed after it's auto abandoned - await queue.EnqueueAsync(new SimpleWorkItem { Data = "Delay" }); - - // wait for taskCompletionSource.SetResult to be called or timeout after 1 second - bool timedout = (await Task.WhenAny(taskCompletionSource.Task, Task.Delay(TimeSpan.FromSeconds(2), TestCancellationToken))) != taskCompletionSource.Task; - Assert.False(timedout); - - // enqueue another item and make sure it was handled (worker loop didn't crash) - taskCompletionSource = new TaskCompletionSource(); - await queue.EnqueueAsync(new SimpleWorkItem { Data = "No Delay" }); - - // one option to fix this issue is surrounding the AbandonAsync call in StartWorkingImpl exception handler in inner try/catch block - timedout = (await Task.WhenAny(taskCompletionSource.Task, Task.Delay(TimeSpan.FromSeconds(30), TestCancellationToken))) != taskCompletionSource.Task; - Assert.False(timedout); - } - -} diff --git a/tests/Foundatio.Tests/Queue/MessageQueueTests.cs b/tests/Foundatio.Tests/Queue/MessageQueueTests.cs new file mode 100644 index 000000000..c8bb0bab5 --- /dev/null +++ b/tests/Foundatio.Tests/Queue/MessageQueueTests.cs @@ -0,0 +1,1096 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Channels; +using System.Threading.Tasks; +using Foundatio.AsyncEx; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Foundatio.Tests.Extensions; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace Foundatio.Tests.Queue; + +public class MessageQueueTests +{ + [Fact] + public async Task EnqueueAsync_WithOptions_CanReceiveAndCompleteAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport); + + string id = await queue.SendAsync(new PreviewWorkItem { Data = "hello" }, new MessageSendOptions + { + CorrelationId = "corr-123", + Priority = MessagePriority.High, + Headers = MessageHeaders.Create([ + new KeyValuePair("tenant", "acme") + ]) + }, cancellationToken); + + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var received = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + + Assert.NotNull(received); + Assert.Equal(id, received.Id); + Assert.Equal("hello", received.Message.Data); + Assert.Equal("corr-123", received.CorrelationId); + Assert.Equal(MessagePriority.High, received.Priority); + Assert.Equal(1, received.Attempts); + Assert.Equal("acme", received.Headers["tenant"]); + Assert.Equal(typeof(PreviewWorkItem).FullName, received.MessageType); + + await received.CompleteAsync(cancellationToken); + + var stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cancellationToken); + Assert.Equal(1, stats.Completed); + Assert.Equal(0, stats.Working); + } + + [Fact] + public async Task EnqueueBatchAsync_UsesDestinationOverrideAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport); + + await queue.SendBatchAsync([ + new PreviewWorkItem { Data = "one" }, + new PreviewWorkItem { Data = "two" } + ], new MessageSendOptions { Destination = "custom-work" }, cancellationToken); + + await using var collector = await MessageCollector.StartAsync(queue, destination: "custom-work", cancellationToken: cancellationToken); + var first = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(first); + await first.CompleteAsync(cancellationToken); + var second = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + + Assert.NotNull(second); + Assert.Equal("one", first.Message.Data); + Assert.Equal("two", second.Message.Data); + + await second.CompleteAsync(cancellationToken); + } + + [Fact] + public async Task RejectAsync_NonTerminal_RedeliversAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var queue = new MessageBus(new InMemoryMessageTransport()); + await queue.SendAsync(new PreviewWorkItem { Data = "retry" }, cancellationToken: cancellationToken); + + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var first = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(first); + + await first.RejectAsync(cancellationToken: cancellationToken); + + var second = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(second); + Assert.Equal(first.Id, second.Id); + Assert.Equal(2, second.Attempts); + Assert.Equal("retry", second.Message.Data); + + await second.CompleteAsync(cancellationToken); + } + + [Fact] + public async Task RenewLockAsync_WhenUnsupported_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + // BasicQueueTransport intentionally does not implement ISupportsLockRenewal, so the core must surface the + // unsupported capability rather than silently no-op. + await using var queue = new MessageBus(new BasicQueueTransport()); + + await queue.SendAsync(new PreviewWorkItem { Data = "lock" }, cancellationToken: cancellationToken); + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var message = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(message); + + await Assert.ThrowsAsync(async () => await message.RenewLockAsync(cancellationToken: cancellationToken)); + } + + [Fact] + public async Task RejectAsync_Terminal_DeadLettersAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport); + + await queue.SendAsync(new PreviewWorkItem { Data = "bad" }, cancellationToken: cancellationToken); + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var message = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(message); + + await message.RejectAsync(new RejectOptions { Terminal = true, Reason = "validation" }, cancellationToken); + + var stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cancellationToken); + Assert.Equal(1, stats.Deadletter); + Assert.Equal(0, stats.Working); + } + + [Fact] + public async Task StartConsumerAsync_WithAutoAck_CompletesMessageAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var handled = new AsyncCountdownEvent(1); + + await using var consumer = await queue.ConsumeAsync((message, _) => + { + Assert.Equal("work", message.Message.Data); + handled.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + await queue.SendAsync(new PreviewWorkItem { Data = "work" }, cancellationToken: cts.Token); + await handled.WaitAsync(TimeSpan.FromSeconds(2)); + await WaitForCompletedAsync(transport, "preview-work-item", cancellationToken); + } + + [Fact] + public async Task StartConsumerAsync_WithManualAck_DoesNotAutoCompleteAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var handled = new AsyncCountdownEvent(1); + + await using var consumer = await queue.ConsumeAsync((message, _) => + { + handled.Signal(); + return Task.CompletedTask; // intentionally does NOT settle the message + }, new MessageConsumerOptions { AckMode = AckMode.Manual }, cts.Token); + + await queue.SendAsync(new PreviewWorkItem { Data = "manual" }, cancellationToken: cts.Token); + await handled.WaitAsync(TimeSpan.FromSeconds(2)); + await Task.Delay(200, cts.Token); + + // Manual ack: the handler ran but did not settle, so the message stays in flight and is not auto-completed. + var stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cancellationToken); + Assert.Equal(0, stats.Completed); + Assert.Equal(1, stats.Working); + } + + [Fact] + public async Task StartConsumerAsync_WithPoisonMessage_DeadLettersAndKeepsConsumingAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var handled = new AsyncCountdownEvent(1); + + await using var consumer = await queue.ConsumeAsync((message, _) => + { + Assert.Equal("good", message.Message.Data); + handled.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + // A poison (undeserializable) payload must be dead-lettered without tearing down the consumer loop, so the + // subsequent valid message is still delivered. + await transport.SendAsync(DestinationAddress.ForQueue("preview-work-item"), [ + new TransportMessage { Body = System.Text.Encoding.UTF8.GetBytes("}{ not json"), Headers = MessageHeaders.Empty } + ], new TransportSendOptions(), cts.Token); + await queue.SendAsync(new PreviewWorkItem { Data = "good" }, cancellationToken: cts.Token); + + await handled.WaitAsync(TimeSpan.FromSeconds(5)); + Assert.Equal(0, handled.CurrentCount); + } + + [Fact] + public async Task EnqueueBatchAsync_RespectsTransportMaxBatchSizeAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var transport = new BatchLimitTransport(maxBatchSize: 2); + await using var queue = new MessageBus(transport); + + await queue.SendBatchAsync(new[] + { + new PreviewWorkItem { Data = "1" }, + new PreviewWorkItem { Data = "2" }, + new PreviewWorkItem { Data = "3" }, + new PreviewWorkItem { Data = "4" }, + new PreviewWorkItem { Data = "5" } + }, cancellationToken: cancellationToken); + + // Five messages to one destination with MaxBatchSize=2 must be split into chunks of 2, 2, 1. + Assert.Equal(new[] { 2, 2, 1 }, transport.SendBatchSizes); + } + + [Fact] + public async Task EnqueueAsync_WithDelay_SchedulesThroughRuntimeStoreAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport, new MessageBusOptions { RuntimeStore = store }); + var processor = CreateDispatchProcessor(store, transport); + + await queue.SendAsync(new PreviewWorkItem { Data = "later" }, new MessageSendOptions { Delay = TimeSpan.FromMinutes(1) }, cancellationToken); + + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var immediate = await collector.NextAsync(TimeSpan.FromMilliseconds(250), cancellationToken); + Assert.Null(immediate); // parked in the runtime store, not on the transport + + Assert.Equal(1, await processor.DispatchDueAsync(DateTimeOffset.UtcNow.AddMinutes(2), cancellationToken: cancellationToken)); + + var delayed = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(delayed); + Assert.Equal("later", delayed.Message.Data); + await delayed.CompleteAsync(cancellationToken); + } + + [Fact] + public async Task EnqueueAsync_WithDelayAndNoRuntimeStore_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var queue = new MessageBus(new InMemoryMessageTransport()); + + await Assert.ThrowsAsync(async () => + await queue.SendAsync(new PreviewWorkItem { Data = "later" }, new MessageSendOptions { Delay = TimeSpan.FromMinutes(1) }, cancellationToken)); + } + + [Fact] + public async Task EnqueueAsync_WithDelay_RespectsTransportMaxDeliveryDelayAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + + // Within the transport's advertised maximum: delivered natively, never touches the runtime store. + var nativeStore = new InMemoryJobRuntimeStore(); + await using var nativeTransport = new CappedDelayTransport(maxDeliveryDelay: TimeSpan.FromMinutes(15)); + await using var nativeQueue = new MessageBus(nativeTransport, new MessageBusOptions { RuntimeStore = nativeStore }); + var nativeProcessor = CreateDispatchProcessor(nativeStore, nativeTransport); + + await nativeQueue.SendAsync(new PreviewWorkItem { Data = "soon" }, new MessageSendOptions { Delay = TimeSpan.FromMinutes(5) }, cancellationToken); + + Assert.Equal(1, nativeTransport.SendCount); + Assert.NotNull(nativeTransport.LastSendOptions?.DeliverAt); + Assert.Equal(0, await nativeProcessor.DispatchDueAsync(DateTimeOffset.UtcNow.AddYears(1), cancellationToken: cancellationToken)); + + // Beyond the transport's maximum: routed through the runtime store instead of being silently truncated. + var fallbackStore = new InMemoryJobRuntimeStore(); + await using var fallbackTransport = new CappedDelayTransport(maxDeliveryDelay: TimeSpan.FromMinutes(15)); + await using var fallbackQueue = new MessageBus(fallbackTransport, new MessageBusOptions { RuntimeStore = fallbackStore }); + var fallbackProcessor = CreateDispatchProcessor(fallbackStore, fallbackTransport); + + await fallbackQueue.SendAsync(new PreviewWorkItem { Data = "later" }, new MessageSendOptions { Delay = TimeSpan.FromHours(1) }, cancellationToken); + + Assert.Equal(0, fallbackTransport.SendCount); + Assert.Equal(1, await fallbackProcessor.DispatchDueAsync(DateTimeOffset.UtcNow.AddHours(2), cancellationToken: cancellationToken)); + Assert.Equal(1, fallbackTransport.SendCount); + + await using var collector = await MessageCollector.StartAsync(fallbackQueue, cancellationToken: cancellationToken); + var delayed = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(delayed); + Assert.Equal("later", delayed.Message.Data); + await delayed.CompleteAsync(cancellationToken); + } + + [Fact] + public async Task StartConsumerAsync_WithRedeliveryBackoff_SchedulesRetryThroughRuntimeStoreAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + // BasicQueueTransport lacks native redelivery delay, so the backoff routes through the runtime store. It also + // never seeds the delivery count from the message.attempts header, proving the core reconciles the attempt + // count from the header itself (second attempt must observe Attempts == 2, not a reset-to-1 loop). + await using var transport = new BasicQueueTransport(); + await using var queue = new MessageBus(transport, new MessageBusOptions { RuntimeStore = store }); + var processor = CreateDispatchProcessor(store, transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var firstAttempt = new AsyncCountdownEvent(1); + var secondAttempt = new AsyncCountdownEvent(1); + int attempts = 0; + + await using var consumer = await queue.ConsumeAsync((message, _) => + { + attempts++; + if (attempts == 1) + { + Assert.Equal(1, message.Attempts); + firstAttempt.Signal(); + throw new InvalidOperationException("try again later"); + } + + Assert.Equal(2, message.Attempts); + Assert.Equal("retry", message.Message.Data); + secondAttempt.Signal(); + return Task.CompletedTask; + }, new MessageConsumerOptions { RedeliveryBackoff = _ => TimeSpan.FromMinutes(1), MaxAttempts = 3 }, cts.Token); + + await queue.SendAsync(new PreviewWorkItem { Data = "retry" }, cancellationToken: cts.Token); + await firstAttempt.WaitAsync(TimeSpan.FromSeconds(2)); + + // The retry is parked in the runtime store: the still-attached consumer must NOT get a second attempt until + // the dispatch pump drains the store. + await Task.Delay(250, cancellationToken); + Assert.Equal(1, attempts); + + Assert.Equal(1, await processor.DispatchDueAsync(DateTimeOffset.UtcNow.AddMinutes(2), cancellationToken: cancellationToken)); + await secondAttempt.WaitAsync(TimeSpan.FromSeconds(2)); + + } + + [Fact] + public async Task EnqueueAsync_ExceedingTransportMaxMessageBytes_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + // The transport advertises an 8-byte maximum; the core must enforce it up front with a clear error rather than + // let an opaque broker rejection surface mid-send. + await using var transport = new BatchLimitTransport(maxBatchSize: 10, maxMessageBytes: 8); + await using var queue = new MessageBus(transport); + + await Assert.ThrowsAsync(async () => + await queue.SendAsync(new PreviewWorkItem { Data = "a payload well over eight bytes" }, cancellationToken: cancellationToken)); + } + + [Fact] + public async Task RejectAsync_RuntimeStoreRedelivery_AdvancesAttemptCountEachCycleAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + // BasicQueueTransport resets DeliveryCount to 1 on every (re)send and has no native redelivery delay, so each + // delayed reject re-schedules through the runtime store. The reconciled Attempts must keep advancing across + // redeliveries (1 -> 2 -> 3); a regression that bases the next attempt on the reset DeliveryCount would pin it at + // 2 and redeliver forever (never reaching MaxAttempts / dead-letter). + await using var transport = new BasicQueueTransport(); + await using var queue = new MessageBus(transport, new MessageBusOptions { RuntimeStore = store }); + var processor = CreateDispatchProcessor(store, transport); + var now = DateTimeOffset.UtcNow; + + string messageId = await queue.SendAsync(new PreviewWorkItem { Data = "loop" }, cancellationToken: cancellationToken); + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + + for (int expectedAttempt = 1; expectedAttempt <= 3; expectedAttempt++) + { + var received = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(received); + Assert.Equal(expectedAttempt, received.Attempts); + Assert.Equal("loop", received.Message.Data); + Assert.Equal(messageId, received.Id); + + if (expectedAttempt < 3) + { + await received.RejectAsync(new RejectOptions { RedeliveryDelay = TimeSpan.FromMinutes(1) }, cancellationToken); + Assert.Equal(1, await processor.DispatchDueAsync(now.AddMinutes(expectedAttempt * 2d), cancellationToken: cancellationToken)); + } + else + { + await received.CompleteAsync(cancellationToken); + } + } + } + + [Fact] + public async Task SendAsync_WithExpiredMessage_IsDeadLetteredNotDeliveredAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport); + + await queue.SendAsync(new PreviewWorkItem { Data = "expired" }, new MessageSendOptions { TimeToLive = TimeSpan.FromMilliseconds(-1) }, cancellationToken); + + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var received = await collector.NextAsync(TimeSpan.FromMilliseconds(500), cancellationToken); + Assert.Null(received); + + var stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cancellationToken); + Assert.Equal(1, stats.Deadletter); + } + + + [Fact] + public async Task AddFoundatio_WithInMemoryMessagingAndJobs_RegistersAppFacingServices() + { + var services = new ServiceCollection(); + + services.AddFoundatio() + .Messaging.UseInMemory() + .Builder.Jobs.UseInMemory(); + + await using var provider = services.BuildServiceProvider(); + + Assert.NotNull(provider.GetRequiredService()); + Assert.NotNull(provider.GetRequiredService()); + Assert.NotNull(provider.GetRequiredService()); + Assert.NotNull(provider.GetRequiredService()); + Assert.NotNull(provider.GetRequiredService()); + Assert.NotNull(provider.GetRequiredService()); + } + + [Fact] + public async Task AddFoundatio_WithRouting_RegistersRouterAndTopologyAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var services = new ServiceCollection(); + + services.AddFoundatio() + .Messaging.ConfigureRouting(r => r + .UseDefaultQueue("all-work") + .MapTopic("grouped-events", typeof(IGroupedWorkItem))) + .UseInMemory(); + + await using var provider = services.BuildServiceProvider(); + + var router = provider.GetRequiredService(); + Assert.Equal("all-work", router.ResolveRoute(new MessageRouteContext + { + MessageType = typeof(PreviewWorkItem), + Role = MessageRouteRole.QueueDestination + })); + Assert.Equal("grouped-events", router.ResolveRoute(new MessageRouteContext + { + MessageType = typeof(OtherWorkItem), + Role = MessageRouteRole.PubSubTopic + })); + + var topology = provider.GetRequiredService(); + var declarations = topology.GetDeclarations(); + Assert.Contains(declarations, d => d.Address.Role == DestinationRole.Queue && d.Address.Name == "all-work"); + Assert.Contains(declarations, d => d.Address.Role == DestinationRole.Topic && d.Address.Name == "grouped-events"); + Assert.DoesNotContain(declarations, d => d.Address.Role == DestinationRole.Subscription); + + await Assert.ThrowsAsync(async () => await topology.ValidateAsync(cancellationToken)); + await topology.EnsureAsync(cancellationToken); + await topology.ValidateAsync(cancellationToken); + } + + [Fact] + public async Task EnqueueAsync_WithRouteAttribute_UsesAttributedDestinationAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var queue = new MessageBus(new InMemoryMessageTransport()); + + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + Assert.Equal("routed-work", collector.Destination); // the [MessageRoute] attribute names the send destination + + await queue.SendAsync(new RoutedWorkItem { Data = "route" }, cancellationToken: cancellationToken); + + var received = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(received); + Assert.Equal("route", received.Message.Data); + await received.CompleteAsync(cancellationToken); + } + + [Fact] + public async Task ConsumeAsync_WithDuplicateRegistration_ThrowsAsync() + { + var token = TestContext.Current.CancellationToken; + await using var queue = new MessageBus(new InMemoryMessageTransport()); + Func, CancellationToken, Task> handler = (_, _) => Task.CompletedTask; + await using var first = await queue.ConsumeAsync(handler, cancellationToken: token); + await Assert.ThrowsAsync(() => queue.ConsumeAsync(handler, cancellationToken: token)); + } + + [Fact] + public async Task StartConsumerAsync_WithSameKeyAndDifferentHandler_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var queue = new MessageBus(new InMemoryMessageTransport()); + + await using var first = await queue.ConsumeAsync((_, _) => Task.CompletedTask, new MessageConsumerOptions { }, cancellationToken); + + await Assert.ThrowsAsync(async () => + await queue.ConsumeAsync((_, _) => Task.CompletedTask, new MessageConsumerOptions { }, cancellationToken)); + } + + [Fact] + public async Task ReceiveAsync_WithGroupedInterfaceRoute_ReturnsRawMessagesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var routing = new MessageRoutingOptionsBuilder() + .MapQueue("grouped-work", typeof(IGroupedWorkItem)) + .Build(); + await using var queue = new MessageBus(new InMemoryMessageTransport(), new MessageBusOptions { Router = new DefaultMessageRouter(routing) }); + + await queue.SendBatchAsync(new object[] + { + new PreviewWorkItem { Data = "one" }, + new OtherWorkItem { Data = "two" } + }, cancellationToken: cancellationToken); + + await using var collector = await MessageCollector.StartAsync(queue, destination: "grouped-work", cancellationToken: cancellationToken); + var first = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(first); + await first.CompleteAsync(cancellationToken); + var second = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + + Assert.NotNull(second); + Assert.NotEmpty(first.Body.ToArray()); + Assert.Equal(typeof(PreviewWorkItem).FullName, first.MessageType); + Assert.Equal(typeof(OtherWorkItem).FullName, second.MessageType); + + await second.CompleteAsync(cancellationToken); + } + + [Fact] + public async Task StartConsumerAsync_WithGroupedInterfaceRoute_DeserializesConcreteTypeAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var routing = new MessageRoutingOptionsBuilder() + .MapQueue("grouped-work", typeof(IGroupedWorkItem)) + .Build(); + await using var queue = new MessageBus(new InMemoryMessageTransport(), new MessageBusOptions + { + Router = new DefaultMessageRouter(routing), + MessageTypes = new MessageTypeRegistry(new[] + { + new MessageTypeRegistration("preview.v1", typeof(PreviewWorkItem)), + new MessageTypeRegistration("other.v1", typeof(OtherWorkItem)) + }) + }); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var received = new ConcurrentDictionary(); + var signal = new AsyncCountdownEvent(2); + + // An interface-typed consumer receives the concrete payload (assignable to the interface), not raw bytes — + // the core resolves the concrete type from the message-type header and deserializes that. + await using var consumer = await queue.ConsumeAsync((message, _) => + { + string? data = message.Message switch + { + PreviewWorkItem p => p.Data, + OtherWorkItem o => o.Data, + _ => null + }; + received[message.Message.GetType()] = data; + signal.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + await queue.SendBatchAsync(new object[] + { + new PreviewWorkItem { Data = "one" }, + new OtherWorkItem { Data = "two" } + }, cancellationToken: cts.Token); + + await signal.WaitAsync(TimeSpan.FromSeconds(2)); + + Assert.Equal("one", received[typeof(PreviewWorkItem)]); + Assert.Equal("two", received[typeof(OtherWorkItem)]); + } + + [Fact] + public async Task ReceiveAsync_WithDefaultQueueRoute_ReturnsRawMessageAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var routing = new MessageRoutingOptionsBuilder() + .UseDefaultQueue("all-work") + .Build(); + await using var queue = new MessageBus(new InMemoryMessageTransport(), new MessageBusOptions { Router = new DefaultMessageRouter(routing) }); + + await queue.SendAsync(new PreviewWorkItem { Data = "global" }, cancellationToken: cancellationToken); + + await using var collector = await MessageCollector.StartAsync(queue, destination: "all-work", cancellationToken: cancellationToken); + var received = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + + Assert.NotNull(received); + Assert.Equal(typeof(PreviewWorkItem).FullName, received.MessageType); + await received.CompleteAsync(cancellationToken); + } + + + private static async Task WaitForCompletedAsync(InMemoryMessageTransport transport, string destination, CancellationToken cancellationToken) + { + var address = DestinationAddress.ForQueue(destination); + var deadline = DateTimeOffset.UtcNow.AddSeconds(2); + while (DateTimeOffset.UtcNow < deadline) + { + var stats = await transport.GetStatsAsync(address, cancellationToken); + if (stats.Completed == 1) + return; + + await Task.Delay(TimeSpan.FromMilliseconds(10), cancellationToken); + } + + var finalStats = await transport.GetStatsAsync(address, cancellationToken); + Assert.Equal(1, finalStats.Completed); + } + + // Pull-style test helper over the subscription API: collects manually-acked contexts so tests can inspect and + // settle deliveries explicitly, now that the bus surface is subscription-only. + private sealed class MessageCollector : IAsyncDisposable where T : class + { + private readonly Channel> _received = Channel.CreateUnbounded>(); + private IMessageSubscription _subscription = null!; + + public static async Task> StartAsync(IMessageBus bus, string? destination = null, CancellationToken cancellationToken = default) + { + var collector = new MessageCollector(); + collector._subscription = await bus.ConsumeAsync((context, _) => + { + collector._received.Writer.TryWrite(context); + return Task.CompletedTask; + }, new MessageConsumerOptions { AckMode = AckMode.Manual, Destination = destination }, cancellationToken); + return collector; + } + + public string Destination => _subscription.Source.Name; + + public async Task?> NextAsync(TimeSpan maxWait, CancellationToken cancellationToken = default) + { + // WaitToReadAsync + TryRead (not ReadAsync + WaitAsync): a timed-out WaitAsync abandons its ReadAsync, + // which would silently consume the next item. + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(maxWait); + try + { + while (await _received.Reader.WaitToReadAsync(cts.Token)) + { + if (_received.Reader.TryRead(out var context)) + return context; + } + + return null; + } + catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested) + { + return null; + } + } + + public ValueTask DisposeAsync() => _subscription.DisposeAsync(); + } + + private sealed class MessageCollector : IAsyncDisposable + { + private readonly Channel _received = Channel.CreateUnbounded(); + private IMessageSubscription _subscription = null!; + + public static async Task StartAsync(IMessageBus bus, string? destination = null, CancellationToken cancellationToken = default) + { + var collector = new MessageCollector(); + collector._subscription = await bus.ConsumeAsync((context, _) => + { + collector._received.Writer.TryWrite(context); + return Task.CompletedTask; + }, new MessageConsumerOptions { AckMode = AckMode.Manual, Destination = destination }, cancellationToken); + return collector; + } + + public async Task NextAsync(TimeSpan maxWait, CancellationToken cancellationToken = default) + { + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(maxWait); + try + { + while (await _received.Reader.WaitToReadAsync(cts.Token)) + { + if (_received.Reader.TryRead(out var context)) + return context; + } + + return null; + } + catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested) + { + return null; + } + } + + public ValueTask DisposeAsync() => _subscription.DisposeAsync(); + } + + [Fact] + public async Task StartConsumerAsync_MultipleTypesOnOneDestination_DispatchByTypeAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var queue = new MessageBus(new InMemoryMessageTransport()); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var aReceived = new List(); + var bReceived = new List(); + var aSignal = new AsyncCountdownEvent(1); + var bSignal = new AsyncCountdownEvent(1); + + await using var consumerA = await queue.ConsumeAsync((message, _) => + { + lock (aReceived) + aReceived.Add(message.Message.Data); + aSignal.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + await using var consumerB = await queue.ConsumeAsync((message, _) => + { + lock (bReceived) + bReceived.Add(message.Message.Data); + bSignal.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + // Both types route to the same destination, so they share one underlying receive loop that dispatches by type. + Assert.Equal(consumerA.Source, consumerB.Source); + + await queue.SendAsync(new SharedAWorkItem { Data = "a" }, cancellationToken: cts.Token); + await queue.SendAsync(new SharedBWorkItem { Data = "b" }, cancellationToken: cts.Token); + + await aSignal.WaitAsync(TimeSpan.FromSeconds(2)); + await bSignal.WaitAsync(TimeSpan.FromSeconds(2)); + + Assert.Equal(new[] { "a" }, aReceived); + Assert.Equal(new[] { "b" }, bReceived); + } + + [Fact] + public async Task StartConsumerAsync_UnmatchedType_DeadLettersAndKeepsConsumingAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport, new MessageBusOptions { RetryPolicy = new RetryPolicy { UnmatchedMaxAttempts = 3, UnmatchedBackoff = _ => TimeSpan.Zero } }); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(20)); + + var aSignal = new AsyncCountdownEvent(1); + await using var consumerA = await queue.ConsumeAsync((_, _) => + { + aSignal.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + // SharedBWorkItem routes to the same destination but has no registered consumer on this node. + await queue.SendAsync(new SharedBWorkItem { Data = "orphan" }, cancellationToken: cts.Token); + + // It is retried and finally dead-lettered as "no-handler" once the configured unmatched budget is exhausted. + for (int i = 0; i < 400; i++) + { + if ((await transport.GetStatsAsync(DestinationAddress.ForQueue("shared-demux"), cts.Token)).Deadletter == 1) + break; + await Task.Delay(TimeSpan.FromMilliseconds(25), cts.Token); + } + + Assert.Equal(1, (await transport.GetStatsAsync(DestinationAddress.ForQueue("shared-demux"), cts.Token)).Deadletter); + + // The loop survived the unmatched message and keeps consuming the type it does handle. + await queue.SendAsync(new SharedAWorkItem { Data = "ok" }, cancellationToken: cts.Token); + await aSignal.WaitAsync(TimeSpan.FromSeconds(2)); + } + + [Fact] + public async Task RejectAsync_Terminal_WithoutNativeDeadLetter_SendsToConfiguredDestinationAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new NoDeadLetterTransport(); + await using var queue = new MessageBus(transport, new MessageBusOptions { RetryPolicy = new RetryPolicy { DeadLetterDestination = "preview-dead-letter" } }); + + await queue.SendAsync(new PreviewWorkItem { Data = "bad" }, cancellationToken: cancellationToken); + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var message = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(message); + + await message.RejectAsync(new RejectOptions { Terminal = true, Reason = "validation" }, cancellationToken); + + // The transport has no native dead-letter sink, so core routes the terminal message to the configured destination. + await using var deadCollector = await MessageCollector.StartAsync(queue, destination: "preview-dead-letter", cancellationToken: cancellationToken); + var dead = await deadCollector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(dead); + Assert.Equal("validation", dead.Headers.GetValueOrDefault(KnownHeaders.DeadLetterReason)); + } + + [Fact] + public async Task RejectAsync_Terminal_WithoutNativeDeadLetterOrConfig_DerivesDeadLetterDestinationAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new NoDeadLetterTransport(); + // No DeadLetterDestination configured: the terminal message must be parked at "{source}.deadletter", not dropped. + await using var queue = new MessageBus(transport); + + await queue.SendAsync(new PreviewWorkItem { Data = "bad" }, cancellationToken: cancellationToken); + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var message = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(message); + + await message.RejectAsync(new RejectOptions { Terminal = true, Reason = "validation", Exception = new InvalidOperationException("boom") }, cancellationToken); + + await using var deadCollector = await MessageCollector.StartAsync(queue, destination: "preview-work-item.deadletter", cancellationToken: cancellationToken); + var dead = await deadCollector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(dead); + Assert.Equal("validation", dead.Headers.GetValueOrDefault(KnownHeaders.DeadLetterReason)); + Assert.Equal(typeof(InvalidOperationException).FullName, dead.Headers.GetValueOrDefault(KnownHeaders.DeadLetterExceptionType)); + Assert.Equal("preview-work-item", dead.Headers.GetValueOrDefault(KnownHeaders.DeadLetterOriginalDestination)); + } + + [Fact] + public async Task RetryPolicy_BackoffOnTransportWithoutDelaySupport_DegradesToImmediateRedeliveryAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + // BasicQueueTransport has no native redelivery delay and no runtime store is registered, so the default + // backoff curve (10s+ after the second attempt) cannot be honored — the policy retry must degrade to + // immediate redelivery rather than failing the settle, and still reach dead-letter after MaxAttempts. + await using var transport = new BasicQueueTransport(); + await using var queue = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(15)); + int attempts = 0; + + await using var consumer = await queue.ConsumeAsync((_, _) => + { + Interlocked.Increment(ref attempts); + throw new InvalidOperationException("always fails"); + }, new MessageConsumerOptions { MaxAttempts = 3 }, cts.Token); + + await queue.SendAsync(new PreviewWorkItem { Data = "doomed" }, cancellationToken: cts.Token); + + // All three attempts happen without a 10s stall, ending in the transport's native dead-letter sink. + var stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cts.Token); + long deadline = Environment.TickCount64 + 10_000; + while (stats.Deadletter == 0 && Environment.TickCount64 < deadline) + { + await Task.Delay(25, cts.Token); + stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cts.Token); + } + + Assert.Equal(1, stats.Deadletter); + Assert.Equal(3, Volatile.Read(ref attempts)); + } + + [Fact] + public async Task StartConsumerAsync_UsesDefaultRetryPolicyMaxAttempts_WhenConsumerDoesNotOverrideAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport, new MessageBusOptions { RetryPolicy = new RetryPolicy { MaxAttempts = 2 } }); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(20)); + + int attempts = 0; + await using var consumer = await queue.ConsumeAsync((_, _) => + { + Interlocked.Increment(ref attempts); + throw new InvalidOperationException("always fails"); + }, cancellationToken: cts.Token); // no per-consumer MaxAttempts -> default RetryPolicy (2) + + await queue.SendAsync(new PreviewWorkItem { Data = "x" }, cancellationToken: cts.Token); + + for (int i = 0; i < 400; i++) + { + if ((await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cts.Token)).Deadletter == 1) + break; + await Task.Delay(TimeSpan.FromMilliseconds(25), cts.Token); + } + + Assert.Equal(1, (await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cts.Token)).Deadletter); + Assert.Equal(2, attempts); + } + + [Fact] + public async Task DisposeAsync_RespectsTransportOwnershipAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + + // Non-owning client (shared transport): disposing the client leaves the transport usable. + var shared = new InMemoryMessageTransport(); + var nonOwning = new MessageBus(shared, new MessageBusOptions { OwnsTransport = false }); + await nonOwning.DisposeAsync(); + await shared.SendAsync(DestinationAddress.ForQueue("still-alive"), [new TransportMessage { Body = ReadOnlyMemory.Empty }], new TransportSendOptions(), cancellationToken); + await shared.DisposeAsync(); + + // Owning client (default): disposing the client disposes the transport. + var owned = new InMemoryMessageTransport(); + var owning = new MessageBus(owned); + await owning.DisposeAsync(); + await Assert.ThrowsAsync(async () => + await owned.SendAsync(DestinationAddress.ForQueue("dead"), [new TransportMessage { Body = ReadOnlyMemory.Empty }], new TransportSendOptions(), cancellationToken)); + } + + [Fact] + public async Task DiBuiltClients_ShareTransport_DisposedExactlyOnceAsync() + { + var transport = new DisposeCountingTransport(); + var services = new ServiceCollection(); + services.AddFoundatio().Messaging.UseTransport(transport); + await using var provider = services.BuildServiceProvider(); + + _ = provider.GetRequiredService(); + + await provider.DisposeAsync(); + + // The container owns the shared transport singleton; the bus does not dispose it, so it is disposed once. + Assert.Equal(1, transport.DisposeCount); + } + + private static ScheduledMessageDispatcher CreateDispatchProcessor(IJobRuntimeStore store, IMessageTransport transport) + { + return new ScheduledMessageDispatcher(store, transport); + } + + [MessageRoute("routed-work")] + private sealed class RoutedWorkItem + { + public string? Data { get; set; } + } + + [MessageRoute("shared-demux")] + private sealed class SharedAWorkItem + { + public string? Data { get; set; } + } + + [MessageRoute("shared-demux")] + private sealed class SharedBWorkItem + { + public string? Data { get; set; } + } + + private interface IGroupedWorkItem + { + } + + private sealed class PreviewWorkItem : IGroupedWorkItem + { + public string? Data { get; set; } + } + + private sealed class OtherWorkItem : IGroupedWorkItem + { + public string? Data { get; set; } + } + + private sealed class BatchLimitTransport : IMessageTransport, ITransportInfo + { + public BatchLimitTransport(int maxBatchSize, long? maxMessageBytes = null) + { + MaxBatchSize = maxBatchSize; + MaxMessageBytes = maxMessageBytes; + } + + public List SendBatchSizes { get; } = new(); + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => new HashSet { DestinationRole.Queue }; + public int? MaxBatchSize { get; } + public long? MaxMessageBytes { get; } + + public TransportCapabilities GetCapabilities(DestinationAddress destination) => + new() { Ordering = OrderingGuarantee.Fifo, MaxBatchSize = MaxBatchSize, MaxMessageBytes = MaxMessageBytes }; + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + SendBatchSizes.Add(messages.Count); + var items = new SendItemResult[messages.Count]; + for (int i = 0; i < messages.Count; i++) + items[i] = new SendItemResult { MessageId = messages[i].MessageId ?? Guid.NewGuid().ToString("N") }; + + return Task.FromResult(new SendResult { Items = items }); + } + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public ValueTask DisposeAsync() => ValueTask.CompletedTask; + } + + private sealed class CappedDelayTransport : IMessageTransport, ISupportsPull, ITransportInfo + { + private readonly Queue _entries = new(); + + public CappedDelayTransport(TimeSpan? maxDeliveryDelay) + { + MaxDeliveryDelay = maxDeliveryDelay; + } + + public TimeSpan? MaxDeliveryDelay { get; } + public int SendCount { get; private set; } + public TransportSendOptions? LastSendOptions { get; private set; } + + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => new HashSet { DestinationRole.Queue }; + + public TransportCapabilities GetCapabilities(DestinationAddress destination) => + new() { DelayedDelivery = true, MaxDeliveryDelay = MaxDeliveryDelay }; + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + SendCount += messages.Count; + LastSendOptions = options; + var items = new SendItemResult[messages.Count]; + for (int i = 0; i < messages.Count; i++) + { + string id = messages[i].MessageId ?? Guid.NewGuid().ToString("N"); + _entries.Enqueue(new TransportEntry { Id = id, Destination = destination, Body = messages[i].Body, Headers = messages[i].Headers, Receipt = new Receipt() }); + items[i] = new SendItemResult { MessageId = id }; + } + + return Task.FromResult(new SendResult { Items = items }); + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + { + return Task.FromResult>(_entries.Count > 0 ? [_entries.Dequeue()] : []); + } + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public ValueTask DisposeAsync() => ValueTask.CompletedTask; + } + + // A minimal multi-destination pull transport with NO native dead-letter sink, used to prove core-managed + // dead-lettering routes terminal messages to the configured RetryPolicy.DeadLetterDestination. + private sealed class NoDeadLetterTransport : IMessageTransport, ISupportsPull + { + private readonly ConcurrentDictionary> _queues = new(StringComparer.Ordinal); + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + var queue = _queues.GetOrAdd(destination.Key, _ => new ConcurrentQueue()); + var items = new SendItemResult[messages.Count]; + for (int i = 0; i < messages.Count; i++) + { + string id = messages[i].MessageId ?? Guid.NewGuid().ToString("N"); + queue.Enqueue(new TransportEntry { Id = id, Destination = destination, Body = messages[i].Body, Headers = messages[i].Headers, Receipt = new Receipt() }); + items[i] = new SendItemResult { MessageId = id }; + } + + return Task.FromResult(new SendResult { Items = items }); + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + { + if (_queues.TryGetValue(source.Key, out var queue) && queue.TryDequeue(out var entry)) + return Task.FromResult>([entry]); + + return Task.FromResult>([]); + } + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) + { + _queues.GetOrAdd(entry.Destination.Key, _ => new ConcurrentQueue()).Enqueue(entry with { DeliveryCount = entry.DeliveryCount + 1 }); + return Task.CompletedTask; + } + + public ValueTask DisposeAsync() => ValueTask.CompletedTask; + } + + // Counts dispose calls without an idempotency guard, so a double-dispose (the bug item 6 fixes) would show as > 1. + private sealed class DisposeCountingTransport : IMessageTransport + { + public int DisposeCount { get; private set; } + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + => Task.FromResult(new SendResult { Items = Array.Empty() }); + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + + public ValueTask DisposeAsync() + { + DisposeCount++; + return ValueTask.CompletedTask; + } + } +} diff --git a/tests/Foundatio.Tests/Review533RegressionTests.cs b/tests/Foundatio.Tests/Review533RegressionTests.cs new file mode 100644 index 000000000..dc2d470e0 --- /dev/null +++ b/tests/Foundatio.Tests/Review533RegressionTests.cs @@ -0,0 +1,177 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Time.Testing; +using Moq; +using Xunit; + +namespace Foundatio.Tests; + +public class Review533RegressionTests +{ + [Fact] + public async Task InMemoryConsumer_ConcurrencyTwo_StartsTwoHandlers() + { + var token = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport, new MessageBusOptions { OwnsTransport = false }); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var firstStarted = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + int started = 0; + await bus.SendAsync(new Ping(), cancellationToken: token); + await bus.SendAsync(new Ping(), cancellationToken: token); + await using var consumer = await bus.ConsumeAsync(async (_, ct) => + { + Interlocked.Increment(ref started); + firstStarted.TrySetResult(); + await release.Task.WaitAsync(ct); + }, new MessageConsumerOptions { MaxConcurrency = 2 }, token); + try + { + await firstStarted.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + await Task.Delay(200, token); + Assert.Equal(2, Volatile.Read(ref started)); + } + finally { release.TrySetResult(); } + } + + [Fact] + public async Task InMemoryConsumer_DisposedDuringProcessing_ReturnsUnfinishedMessage() + { + var token = TestContext.Current.CancellationToken; + var time = new FakeTimeProvider(); + await using var transport = new InMemoryMessageTransport(time); + await using var bus = new MessageBus(transport, new MessageBusOptions { TimeProvider = time, OwnsTransport = false }); + var started = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await bus.SendAsync(new Ping(), cancellationToken: token); + var consumer = await bus.ConsumeAsync(async (_, ct) => + { + started.TrySetResult(); + await Task.Delay(Timeout.InfiniteTimeSpan, ct); + }, cancellationToken: token); + await started.Task.WaitAsync(TimeSpan.FromSeconds(5), token); + await consumer.DisposeAsync(); + time.Advance(TimeSpan.FromDays(1)); + await using var redelivered = await bus.ReceiveAsync(new MessageReceiveOptions { WaitTime = TimeSpan.Zero }, token); + Assert.NotNull(redelivered); + } + + [Theory] + [InlineData(JobStatus.Completed)] + [InlineData(JobStatus.Failed)] + [InlineData(JobStatus.Cancelled)] + public async Task RegisteredJob_IsDisposedAfterExecution(JobStatus outcome) + { + var services = new ServiceCollection(); + var state = new DisposalState { Outcome = outcome }; + services.AddSingleton(state); + services.AddFoundatio().Jobs.UseInMemory().AddJobType(); + await using var provider = services.BuildServiceProvider(); + var handle = await provider.GetRequiredService().EnqueueAsync(new JobRequestOptions { MaxAttempts = 1 }, cancellationToken: TestContext.Current.CancellationToken); + Assert.True(await provider.GetRequiredService().RunAsync(handle.JobId, TestContext.Current.CancellationToken)); + Assert.Equal(outcome, (await handle.GetStateAsync(TestContext.Current.CancellationToken))!.Status); + Assert.Equal(1, state.Disposed); + } + + [Fact] + public async Task ScheduledDispatch_LongBatch_RetiresAllSuccessfulSends() + { + var time = new FakeTimeProvider(); + var store = new InMemoryJobRuntimeStore(time); + for (int i = 0; i < 4; i++) + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = $"message-{i}", + Destination = DestinationAddress.ForQueue("work"), + DueUtc = time.GetUtcNow(), + Body = "hello"u8.ToArray() + }, TestContext.Current.CancellationToken); + var transport = new Mock(); + transport.Setup(t => t.SendAsync(It.IsAny(), It.IsAny>(), It.IsAny(), It.IsAny())) + .Returns(() => + { + time.Advance(TimeSpan.FromSeconds(20)); + return Task.FromResult(new SendResult { Items = [new SendItemResult { MessageId = "sent" }] }); + }); + var dispatcher = new ScheduledMessageDispatcher(store, transport.Object, new ScheduledMessageDispatcherOptions { TimeProvider = time }); + Assert.Equal(4, await dispatcher.DispatchDueAsync(cancellationToken: TestContext.Current.CancellationToken)); + Assert.Empty(await store.ClaimDueDispatchesAsync(time.GetUtcNow(), 10, "next-worker", TimeSpan.FromMinutes(1), TestContext.Current.CancellationToken)); + } + + [Fact] + public async Task ScheduledSend_SameApplicationIdAcrossDestinations_PreservesBothMessages() + { + var time = new FakeTimeProvider(); + var store = new InMemoryJobRuntimeStore(time); + await using var transport = new InMemoryMessageTransport(time); + await using var bus = new MessageBus(transport, new MessageBusOptions { RuntimeStore = store, TimeProvider = time, OwnsTransport = false }); + await bus.SendAsync(new Ping(), new MessageSendOptions { MessageId = "order-123", Destination = "billing", Delay = TimeSpan.FromMinutes(5) }, TestContext.Current.CancellationToken); + await bus.SendAsync(new Ping(), new MessageSendOptions { MessageId = "order-123", Destination = "shipping", Delay = TimeSpan.FromMinutes(5) }, TestContext.Current.CancellationToken); + time.Advance(TimeSpan.FromMinutes(5)); + Assert.Equal(2, (await store.ClaimDueDispatchesAsync(time.GetUtcNow(), 10, "worker", TimeSpan.FromMinutes(1), TestContext.Current.CancellationToken)).Count); + } + + [Fact] + public async Task JobWorker_IdleSlot_ProcessesNewJobWhileAnotherSlotIsBusy() + { + var services = new ServiceCollection(); + var state = new BlockingState(); + services.AddSingleton(state); + services.AddFoundatio().Jobs.UseInMemory().AddJobType().AddJobType(); + await using var provider = services.BuildServiceProvider(); + var client = provider.GetRequiredService(); + using var worker = new JobWorker(provider.GetRequiredService(), provider, new JobWorkerOptions { MaxConcurrency = 2 }); + await client.EnqueueAsync(cancellationToken: TestContext.Current.CancellationToken); + using var stopping = CancellationTokenSource.CreateLinkedTokenSource(TestContext.Current.CancellationToken); + var batch = worker.RunContinuouslyAsync(stopping.Token); + await state.Started.Task.WaitAsync(TimeSpan.FromSeconds(5), TestContext.Current.CancellationToken); + await client.EnqueueAsync(cancellationToken: TestContext.Current.CancellationToken); + try + { + await Task.Delay(200, TestContext.Current.CancellationToken); + Assert.True(state.QuickStarted.Task.IsCompleted, "The second worker slot is idle, but cannot accept new work until the blocking job completes."); + } + finally + { + state.Release.TrySetResult(); + await stopping.CancelAsync(); + await batch.WaitAsync(TimeSpan.FromSeconds(5), TestContext.Current.CancellationToken); + } + } + + public sealed class DisposalState { public int Disposed; public JobStatus Outcome; } + public sealed class DisposableJob(DisposalState state) : IJob, IAsyncDisposable + { + public Task RunAsync(JobExecutionContext context) => Task.FromResult(state.Outcome switch + { + JobStatus.Completed => JobResult.Success, + JobStatus.Cancelled => JobResult.Cancelled, + _ => JobResult.FromException(new InvalidOperationException("Expected job failure")) + }); + public ValueTask DisposeAsync() { state.Disposed++; return ValueTask.CompletedTask; } + } + public sealed record Ping; + public sealed class BlockingState + { + public TaskCompletionSource Started { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public TaskCompletionSource Release { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public TaskCompletionSource QuickStarted { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + } + public sealed class BlockingJob(BlockingState state) : IJob + { + public async Task RunAsync(JobExecutionContext context) + { + state.Started.TrySetResult(); + await state.Release.Task.WaitAsync(context.CancellationToken); + return JobResult.Success; + } + } + public sealed class QuickJob(BlockingState state) : IJob + { + public Task RunAsync(JobExecutionContext context) { state.QuickStarted.TrySetResult(); return Task.FromResult(JobResult.Success); } + } +} diff --git a/tests/Foundatio.Tests/Serializer/SystemTextJsonSerializerTests.cs b/tests/Foundatio.Tests/Serializer/SystemTextJsonSerializerTests.cs index b3b0c5df5..e4e2137ae 100644 --- a/tests/Foundatio.Tests/Serializer/SystemTextJsonSerializerTests.cs +++ b/tests/Foundatio.Tests/Serializer/SystemTextJsonSerializerTests.cs @@ -1,4 +1,7 @@ -using System.Text.Json; +using System; +using System.IO; +using System.Text; +using System.Text.Json; using Foundatio.Serializer; using Foundatio.TestHarness.Utility; using Microsoft.Extensions.Logging; @@ -15,6 +18,75 @@ protected override ISerializer GetSerializer() return new SystemTextJsonSerializer(); } + [Fact] + public void SerializeToBytes_LargePayload_DoesNotAllocateIntermediatePayloadBuffers() + { + ISerializer serializer = new SystemTextJsonSerializer(); + string value = new('x', 50_000); + for (int i = 0; i < 10; i++) serializer.SerializeToBytes(value); + + long before = GC.GetAllocatedBytesForCurrentThread(); + long length = 0; + for (int i = 0; i < 100; i++) length += serializer.SerializeToBytes(value).Length; + long allocated = GC.GetAllocatedBytesForCurrentThread() - before; + + Assert.Equal(5_000_200, length); + Assert.True(allocated < length * 1.25, $"Allocated {allocated:N0} bytes for {length:N0} bytes of output."); + } + + [Theory] + [InlineData("null")] + [InlineData("42")] + [InlineData("2147483648")] + [InlineData("12.5")] + [InlineData("true")] + [InlineData("\"héllo 世界\"")] + [InlineData("\"2026-09-07T00:00:00+03:00\"")] + [InlineData("{\"value\":42}")] + [InlineData("[1,2,3]")] + [InlineData("\uFEFF42")] + [InlineData("\uFEFF{\"value\":42}")] + public void Deserialize_BytesAndSlicedMemory_MatchesStreamNormalization(string json) + { + ISerializer serializer = new SystemTextJsonSerializer(); + byte[] bytes = Encoding.UTF8.GetBytes(json); + using var stream = new MemoryStream(bytes); + object? expected = serializer.Deserialize(stream, typeof(object)); + byte[] padded = Encoding.UTF8.GetBytes("invalid" + json + "invalid"); + ReadOnlyMemory slice = padded.AsMemory(7, bytes.Length); + + AssertEquivalent(expected, serializer.Deserialize(bytes)); + AssertEquivalent(expected, serializer.Deserialize(bytes, typeof(object))); + AssertEquivalent(expected, serializer.Deserialize(slice)); + AssertEquivalent(expected, serializer.Deserialize(slice, typeof(object))); + + static void AssertEquivalent(object? expected, object? actual) + { + Assert.Equal(expected?.GetType(), actual?.GetType()); + if (expected is JsonElement element) Assert.Equal(element.GetRawText(), ((JsonElement)actual!).GetRawText()); + else Assert.Equal(expected, actual); + } + } + + [Fact] + public void SerializeToBytes_CustomOptionsAndRuntimeType_MatchesStream() + { + var writeOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower, WriteIndented = true }; + var readOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower }; + ISerializer serializer = new SystemTextJsonSerializer(writeOptions, readOptions); + object value = new BufferTestMessage("héllo 世界", 42); + using var stream = new MemoryStream(); + serializer.Serialize(value, stream); + + byte[] bytes = serializer.SerializeToBytes(value); + + Assert.Equal(stream.ToArray(), bytes); + Assert.Equal(value, serializer.Deserialize(bytes)); + Assert.Equal(value, serializer.Deserialize(bytes.AsMemory())); + } + + public sealed record BufferTestMessage(string DisplayName, int MessageCount); + [Fact] public override void Deserialize_WithInvalidArguments_ThrowsArgumentNullException() { diff --git a/tests/Foundatio.Tests/StartupValidationTests.cs b/tests/Foundatio.Tests/StartupValidationTests.cs new file mode 100644 index 000000000..df63d29ce --- /dev/null +++ b/tests/Foundatio.Tests/StartupValidationTests.cs @@ -0,0 +1,123 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Extensions.Hosting.Jobs; +using Foundatio.Extensions.Hosting.Messaging; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Xunit; + +namespace Foundatio.Tests; + +public class StartupValidationTests +{ + [Fact] + public async Task ProducerTopologyNone_DuplicateWireNames_FailsBeforePublishingAsync() + { + var services = new ServiceCollection(); + services.AddFoundatio().ConfigureMessaging(m => m.UseInMemory().ConfigureTopology(TopologyMode.None) + .AddMessageType("event.v1").AddMessageType("event.v1")); + services.AddMessagingTopology(); + await using var provider = services.BuildServiceProvider(); + var error = await Assert.ThrowsAsync(() => StartHostedAsync(provider, TestContext.Current.CancellationToken)); + Assert.Contains("event.v1", error.Message); + } + + private sealed record OtherPing; + + [Fact] + public async Task CronJobWithoutRuntimeStore_FailsStartupWithActionableMessageAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var services = new ServiceCollection(); + services.AddFoundatio().Jobs.AddCronJob("* * * * *"); + + services.AddLogging(); + services.AddMessageConsumers(); + services.AddJobScheduler(); + await using var provider = services.BuildServiceProvider(); + var ex = await Assert.ThrowsAsync(() => StartHostedAsync(provider, cancellationToken)); + Assert.Contains("UseInMemory", ex.Message); + Assert.Contains("never run", ex.Message); + } + + [Fact] + public async Task HandlerWithoutTransport_FailsStartupWithActionableMessageAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var services = new ServiceCollection(); + services.AddFoundatio().Messaging.AddConsumer((_, _) => Task.CompletedTask); + + services.AddLogging(); + services.AddMessageConsumers(); + services.AddJobScheduler(); + await using var provider = services.BuildServiceProvider(); + var ex = await Assert.ThrowsAsync(() => StartHostedAsync(provider, cancellationToken)); + Assert.Contains("no message transport", ex.Message); + Assert.Contains("UseTransport", ex.Message); + } + + [Fact] + public void AddCronJob_WithInvalidCron_ThrowsAtRegistration() + { + var services = new ServiceCollection(); + Assert.ThrowsAny(() => services.AddFoundatio().Jobs.AddCronJob("not-a-cron")); + } + + [Fact] + public void AddCronJob_WithDuplicateName_ThrowsAtRegistration() + { + var services = new ServiceCollection(); + var builder = services.AddFoundatio(); + builder.Jobs.AddCronJob("* * * * *"); + + var ex = Assert.Throws(() => builder.Jobs.AddCronJob("*/5 * * * *")); + Assert.Contains(nameof(NoopJob), ex.Message); + Assert.Contains("Name", ex.Message); + } + + [Fact] + public async Task ValidConfiguration_StartsCleanlyAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var services = new ServiceCollection(); + services.AddFoundatio() + .Messaging.UseInMemory() + .AddConsumer((_, _) => Task.CompletedTask) + .Builder.Jobs.UseInMemory() + .AddCronJob("0 3 * * *"); + + services.AddLogging(); + services.AddMessageConsumers(); + services.AddJobScheduler(); + await using var provider = services.BuildServiceProvider(); + await StartHostedAsync(provider, cancellationToken); + await StopHostedAsync(provider, cancellationToken); + } + + private static async Task StartHostedAsync(ServiceProvider provider, CancellationToken cancellationToken) + { + // Validators and hosts run in registration order, like the generic host would run them. + foreach (var hosted in provider.GetServices()) + await hosted.StartAsync(cancellationToken); + } + + private static async Task StopHostedAsync(ServiceProvider provider, CancellationToken cancellationToken) + { + foreach (var hosted in provider.GetServices().Reverse()) + await hosted.StopAsync(cancellationToken); + } + + private sealed class Ping + { + public string? Data { get; set; } + } + + private sealed class NoopJob : IJob + { + public Task RunAsync(JobExecutionContext context) => Task.FromResult(JobResult.Success); + } +} diff --git a/tests/Foundatio.Tests/Utility/ResiliencePolicyTests.cs b/tests/Foundatio.Tests/Utility/ResiliencePolicyTests.cs index 95ac75e3e..878a0fe54 100644 --- a/tests/Foundatio.Tests/Utility/ResiliencePolicyTests.cs +++ b/tests/Foundatio.Tests/Utility/ResiliencePolicyTests.cs @@ -458,7 +458,7 @@ public async Task CanUsePolly() return Task.FromResult(true); }); - var lockProvider = new CacheLockProvider(mockCacheClient.Object, new InMemoryMessageBus()); + var lockProvider = new CacheLockProvider(mockCacheClient.Object, new MessageBus(new InMemoryMessageTransport())); var l = await lockProvider.TryAcquireAsync("test", TimeSpan.FromSeconds(1), TimeSpan.Zero); Assert.NotNull(l);