Feat/reminder concurrency control

[rkargMsft]

This PR introduces an opt-in reminder delivery admission pipeline for Orleans reminders to protect the silo from bursty reminder work during overload, recovery, and ownership changes. The goal is to add explicit, configurable backpressure around reminder dispatch so reminder ticks do not overwhelm the system when demand spikes.

What this change does

• Adds a reminder-delivery admission pipeline that runs before reminder dispatch.
• Implements the built-in per-silo throttle as a composed sequence of admission gates:
  • overload protection
  • slow-start ramp-up
  • local concurrency limiting
  • local sustained-rate limiting
• Uses a lease-based model so admitted reminder ticks acquire a lease before dispatch and release it when the work completes or fails.
• Skips reminder delivery when the throttle denies admission, so denied ticks never invoke the grain.
• Preserves shared timeout budgeting across sequential waits:
  • if an earlier gate consumes part of a WaitUpTo(...) timeout, later gates only get the remaining time
  • later gates do not restart the timeout budget
• Wires the feature into the reminder service hosting path so it is opt-in and configuration-driven.

Configuration

These settings are opt-in and existing reminder behavior is preserved if nothing is configured.

using System;
using Orleans.Hosting;
using Orleans.Reminders.Concurrency;

var host = new HostBuilder()
    .UseOrleans(siloBuilder =>
    {
        siloBuilder
            .AddReminders()
            .AddReminderConcurrencyControl(cfg =>
            {
                cfg.PerSilo(t => t
                    .MaxConcurrent(
                        50,
                        ThrottleBlockMode.Wait) // Limit in-flight reminder deliveries and wait for a slot

                    .PermitsPerSecond(
                        100,
                        200,
                        ThrottleBlockMode.SkipImmediately) // Cap sustained rate, allow a burst of 200, skip when the bucket is empty

                    .RespectOverload(
                        ThrottleBlockMode.WaitUpTo(TimeSpan.FromSeconds(20))) // Back off while the silo reports overload

                    .SlowStart(
                        initialCapacity: 5,
                        interval: TimeSpan.FromMinutes(1),
                        onCapacityExceeded: ThrottleBlockMode.Wait)); // Start small and ramp up over time
            });
    })
    .Build();

await host.RunAsync();

Configuration notes

• All limiter-style configuration now requires an explicit ThrottleBlockMode.
• MaxConcurrent(...) requires both the limit and the block mode.
• PermitsPerSecond(...) requires the rate, burst size, and block mode.
• RespectOverload(...) and SlowStart(...) also require their own explicit block modes.
• This keeps the admission behavior explicit at each configuration call instead of relying on shared defaults.

Intended structure

The implementation is organized as a small layer above the existing reminder service:

• LocalReminderDeliveryThrottle is the per-silo orchestrator.
• Each built-in behavior lives in its own admission gate implementation.
• The throttle acquires gates in sequence and aggregates their release actions into a single lease.
• The reminder service integrates the throttle around each reminder tick.
• Configuration validates required combinations and rejects silent no-op setups.

This keeps the current per-silo feature working while making the implementation easier to extend later with additional scopes such as global, grain-specific, or reminder-name-specific limiters.

Why

Reminder delivery can become bursty, especially when many reminders become due at once. This change gives the reminder pipeline a way to absorb and shape that load before it reaches grain execution, making overload behavior more predictable and more explicitly configurable.