Skip to content

feat(runtime): add deterministic dissemination broadcast and repair - #10236

Draft
ReubenBond wants to merge 90 commits into
dotnet:mainfrom
ReubenBond:feature/efficient-broadcast-draft
Draft

feat(runtime): add deterministic dissemination broadcast and repair#10236
ReubenBond wants to merge 90 commits into
dotnet:mainfrom
ReubenBond:feature/efficient-broadcast-draft

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Jun 18, 2026 •
edited by dotnet-policy-service Bot
Loading

Copy link
Copy Markdown
Member

Problem

Several silo-to-silo runtime state streams currently rely on high-fanout publication patterns. That scales poorly for high-rate state such as deployment load statistics and makes correctness backstops rely on each individual subsystem.

Solution

This PR adds a shared dissemination substrate for monotonically versioned runtime values. It uses deterministic fixed-tree broadcast for the fast path and bounded anti-entropy repair for convergence, with per-topic membership scopes and status/age/address topology ordering so likely-available members sit higher in the tree.

The implementation wires the substrate into deployment load statistics, membership snapshots, and manifest convergence. It also adds dynamic fanout, membership diff payloads, manifest hash reuse, local frozen collection lookups, and guarded instrumentation paths so unobserved metrics avoid unnecessary work.

Property-based CsCheck tests and Accordant model checks cover the fixed-tree invariants, anti-entropy candidate selection, monotonic apply behavior, and membership diff handling.

Review focus

Please focus on the fixed-tree forwarding semantics, anti-entropy repair behavior, membership diff fallback rules, and the public options/API surface.

Microsoft Reviewers: Open in CodeFlow

@ReubenBond ReubenBond changed the title Add deterministic dissemination broadcast and repair feat(runtime): add deterministic dissemination broadcast and repair Jun 19, 2026
ReubenBond
ReubenBond commented Jun 19, 2026
View reviewed changes
Comment thread src/Orleans.Core/Configuration/Options/DisseminationOptions.cs Outdated
/// <summary>
/// Gets or sets how long peer capability probe results are cached.
/// </summary>
public TimeSpan CapabilityCacheTtl { get; set; } = TimeSpan.FromMinutes(5);

@ReubenBond ReubenBond Jun 19, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: remove the cache altogether, and the capability probing where possible - this will require some design revision

ReubenBond
ReubenBond commented Jun 19, 2026
View reviewed changes
Comment thread src/Orleans.Core/Configuration/Options/DisseminationOptions.cs Outdated
/// <summary>
/// Gets or sets how long failed peers are backed off before retrying.
/// </summary>
public TimeSpan FailureBackoff { get; set; } = TimeSpan.FromSeconds(5);

@ReubenBond ReubenBond Jun 19, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: Use Polly for backoffs instead.

ReubenBond
ReubenBond commented Jun 19, 2026
View reviewed changes
Comment thread src/Orleans.Core/Configuration/Options/DisseminationOptions.cs Outdated
/// <summary>
/// Gets or sets the maximum serialized bytes in one dissemination batch.
/// </summary>
public int MaxBatchBytes { get; set; } = 64 * 1024;

@ReubenBond ReubenBond Jun 19, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this 1MB by default, or something larger than 64K. Do some rough math to figure out a good default based on the payload.

ReubenBond
ReubenBond commented Jun 19, 2026
View reviewed changes
Comment thread src/Orleans.Core/Configuration/Options/DisseminationOptions.cs Outdated
/// <summary>
/// Gets or sets the maximum number of items in one dissemination batch.
/// </summary>
public int MaxBatchItems { get; set; } = 64;

@ReubenBond ReubenBond Jun 19, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be larger, like 16K. No reason for it to be arbitrarily small. A rule of thumb is that we want to include the latest update for each silo in the cluster for a given topic.

ReubenBond
ReubenBond commented Jun 19, 2026
View reviewed changes
Comment thread src/Orleans.Core/Configuration/Options/DisseminationOptions.cs
/// <summary>
/// Gets or sets the interval between anti-entropy repair rounds.
/// </summary>
public TimeSpan AntiEntropyInterval { get; set; } = TimeSpan.FromSeconds(5);

@ReubenBond ReubenBond Jun 19, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rule of thumb: some factor larger than the expected update interval. 5 * 1s seems appropriate already.

ReubenBond
ReubenBond commented Jun 19, 2026
View reviewed changes
Comment thread src/Orleans.Core/Configuration/Options/DisseminationOptions.cs
/// <summary>
/// Gets or sets the maximum number of pending topic items.
/// </summary>
public int MaxPendingItemCount { get; set; } = 1024;

@ReubenBond ReubenBond Jun 19, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to describe the meaning behind this - and rule of thumb for setting it. Seems it could be longer.

@ReubenBond
ReubenBond force-pushed the feature/efficient-broadcast-draft branch 2 times, most recently from 611d5dd to 72bfd57 Compare July 7, 2026 00:16
@ReubenBond
ReubenBond requested a review from Copilot July 9, 2026 20:27
Copilot AI reviewed Jul 9, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new internal dissemination substrate for monotonically versioned runtime state, using deterministic fixed-tree broadcast for the fast path and periodic anti-entropy repair for convergence. It integrates the substrate into deployment load statistics and membership gossip, and improves manifest convergence by reusing content-addressed manifest hashes/caching and peer-assisted fills.

Changes:

  • Added dissemination system-target contracts, runtime implementation (broadcast queue, protocol, membership snapshotting, metrics/events), and configuration options/validators.
  • Wired dissemination into deployment load publishing and membership gossip with opt-in options and legacy fallbacks.
  • Added manifest hash-based fetch/caching and peer-based fill to reduce manifest convergence request fanout; expanded SiloAddress parsing APIs and added new dissemination-focused tests/docs.
Show a summary per file
File Description
test/TestInfrastructure/TestExtensions/Diagnostics/PlacementDiagnosticObserver.cs Uses Equals for SiloAddress comparisons in placement diagnostics helper.
test/Orleans.Runtime.Internal.Tests/Orleans.Runtime.Internal.Tests.csproj Adds test dependencies (Accordant/net10-only, NSubstitute + analyzers).
test/Orleans.Runtime.Internal.Tests/Dissemination/WakeTimerTests.cs Adds unit tests for the new wakeable one-shot timer.
test/Orleans.Runtime.Internal.Tests/Dissemination/DisseminationMembershipSnapshotTests.cs Adds CsCheck property tests for dissemination membership snapshot invariants.
test/Orleans.Core.Tests/General/Identifiertests.cs Adds tests for new SiloAddress parsing interfaces (string + UTF-8).
src/Orleans.Runtime/Scheduler/SchedulerExtensions.cs Adds RunOrQueueTask<TResult> helper for scheduling result-returning tasks.
src/Orleans.Runtime/Placement/Rebalancing/ActivationRebalancerMonitor.cs Uses value equality for SiloAddress.Zero comparisons.
src/Orleans.Runtime/Placement/DeploymentLoadPublisher.cs Publishes runtime stats via dissemination when enabled; adds fallback direct publish path; exposes apply/obsolete helpers.
src/Orleans.Runtime/Networking/SiloConnectionMaintainer.cs Uses value equality for silo address comparisons on status changes.
src/Orleans.Runtime/MembershipService/MembershipGossiper.cs Attempts dissemination publish for membership snapshots, falls back to legacy gossip.
src/Orleans.Runtime/Manifest/ClusterManifestProvider.cs Adds manifest hash cache, peer fill optimization, and hash-based fetch fallback logic.
src/Orleans.Runtime/Hosting/EndpointOptions.cs Uses Equals for IPAddress comparisons; trims trailing whitespace in docs.
src/Orleans.Runtime/Hosting/DefaultSiloServices.cs Registers dissemination services/namespaces and validators; adds formatter for DisseminationOptions.
src/Orleans.Runtime/GrainTypeManager/ClusterManifestSystemTarget.cs Adds manifest-hash APIs (hash summary, fetch-by-hash) and caches local manifest hash.
src/Orleans.Runtime/Dissemination/WakeTimer.cs Adds a thread-safe wakeable timer used for coalescing/flush scheduling.
src/Orleans.Runtime/Dissemination/MembershipDisseminationNamespace.cs Implements membership snapshot dissemination with diff-based repair and bounded history.
src/Orleans.Runtime/Dissemination/ManifestHashCalculator.cs Adds canonical manifest hashing utility for CAS reuse/validation.
src/Orleans.Runtime/Dissemination/IDisseminationService.cs Defines internal publish service abstraction.
src/Orleans.Runtime/Dissemination/IDisseminationNamespace.cs Defines dissemination namespace abstraction (digests, repair materialization, apply).
src/Orleans.Runtime/Dissemination/DisseminationSystemTarget.cs Implements system target endpoint and anti-entropy loop lifecycle.
src/Orleans.Runtime/Dissemination/DisseminationProtocol.cs Implements fixed-tree broadcast routing, anti-entropy exchanges, and apply/forward logic.
src/Orleans.Runtime/Dissemination/DisseminationNamespaceNames.cs Defines namespace IDs for load + membership dissemination.
src/Orleans.Runtime/Dissemination/DisseminationMembershipSnapshot.cs Computes deterministic forwarding targets and anti-entropy peer selection helpers.
src/Orleans.Runtime/Dissemination/DisseminationMembership.cs Builds dissemination membership snapshots from membership manager, ordered by status/age/address.
src/Orleans.Runtime/Dissemination/DisseminationInstruments.cs Adds low-cardinality metrics for dissemination traffic/results.
src/Orleans.Runtime/Dissemination/DisseminationEvents.cs Adds DiagnosticListener events for apply/drop events.
src/Orleans.Runtime/Dissemination/DisseminationBroadcastQueue.cs Adds per-peer coalescing/flush queue for broadcast batches and bounded sending.
src/Orleans.Runtime/Dissemination/DisseminationApplyResult.cs Defines apply result enum for value application semantics.
src/Orleans.Runtime/Dissemination/DeploymentLoadStatisticsDisseminationNamespace.cs Implements dissemination namespace for per-silo runtime load statistics.
src/Orleans.Runtime/Diagnostics/DeploymentLoadPublisherEvents.cs Uses value equality for self-filtering in diagnostics events.
src/Orleans.Runtime/Configuration/Options/DisseminationOptionsValidator.cs Adds options validation for dissemination global/namespace options and related owners.
src/Orleans.Runtime/Configuration/Options/DeploymentLoadPublisherOptions.cs Adds per-namespace dissemination options to deployment load publisher options.
src/Orleans.Core/SystemTargetInterfaces/IDisseminationSystemTarget.cs Adds wire contracts for dissemination broadcast + anti-entropy and related DTOs.
src/Orleans.Core/Runtime/Constants.cs Adds dissemination system target grain type constant and singleton name mapping.
src/Orleans.Core/Placement/Repartitioning/IActivationRepartitionerSystemTarget.cs Updates EdgeVertex.Equals to use SiloAddress.Equals.
src/Orleans.Core/Networking/Shared/SocketConnectionListener.cs Uses value equality for IPv6Any check and adds cancellation to AcceptAsync.
src/Orleans.Core/Manifest/IClusterManifestSystemTarget.cs Adds manifest hash/CAS APIs and DTOs (ManifestHash, ClusterManifestHashSummary).
src/Orleans.Core/Configuration/Options/DisseminationOptions.cs Adds public configuration options for dissemination subsystem + overlay + namespaces.
src/Orleans.Core/Configuration/Options/ClusterMembershipOptions.cs Adds per-namespace dissemination options for membership updates.
src/Orleans.Core.Abstractions/IDs/SiloAddress.cs Adds IParsable/IUtf8SpanParsable support and non-throwing TryParse implementations.
src/api/Orleans.Runtime/Orleans.Runtime.cs Updates public API baseline for DeploymentLoadPublisherOptions dissemination property.
src/api/Orleans.Core/Orleans.Core.cs Updates public API baseline for dissemination options and membership options property.
src/api/Orleans.Core.Abstractions/Orleans.Core.Abstractions.cs Updates public API baseline for SiloAddress parsing interfaces/methods.
efficient-broadcast.md Adds design/branch documentation for the efficient broadcast + repair approach.
dissemination.md Adds detailed design documentation for topic-based dissemination and testing/rollout.
Directory.Packages.props Adds Microsoft.Accordant package version.
agency.toml Adds MCP/agent configuration (appears unrelated to Orleans runtime).

Copilot's findings

  • Files reviewed: 48/48 changed files
  • Comments generated: 4

Comment thread src/Orleans.Core/Placement/Repartitioning/IActivationRepartitionerSystemTarget.cs
Comment on lines 99 to 101
public override bool Equals([NotNullWhen(true)] object obj) => obj is EdgeVertex other && Equals(other);
public bool Equals(EdgeVertex other) => Id == other.Id && Silo == other.Silo && IsMigratable == other.IsMigratable;
public bool Equals(EdgeVertex other) => Id == other.Id && Silo.Equals(other.Silo) && IsMigratable == other.IsMigratable;

Comment thread src/Orleans.Runtime/Manifest/ClusterManifestProvider.cs
Comment on lines 33 to 36
private IInternalGrainFactory? _grainFactory;
private Task? _runTask;
private readonly Dictionary<ManifestHash, GrainManifest> _manifestCache = new();

Comment thread src/Orleans.Runtime/Dissemination/ManifestHashCalculator.cs
Comment on lines +47 to +55
private static void AppendString(IncrementalHash hash, string value)
{
var bytes = Encoding.UTF8.GetBytes(value ?? string.Empty);
var length = Encoding.UTF8.GetBytes(bytes.Length.ToString(CultureInfo.InvariantCulture));
hash.AppendData(length);
hash.AppendData(new byte[] { 0 });
hash.AppendData(bytes);
hash.AppendData(new byte[] { 0xff });
}
Comment thread agency.toml Outdated
Comment on lines +1 to +16
[mcps.builtins]
m365-user = true
m365-copilot = true
teams = true
mail = true
workiq = true
enghub = true
msft-learn = true
sharepoint = true
calendar = true
icm = true

[mcps.builtins.kusto]
type = "kusto"
service_uri = "https://ddtelinsights.kusto.windows.net/"
database = "DDTelInsights"
@ReubenBond
ReubenBond force-pushed the feature/efficient-broadcast-draft branch from 8e4b7d2 to a94b017 Compare July 9, 2026 20:40
ReubenBond and others added 17 commits July 15, 2026 16:06
@ReubenBond
Implement deterministic dissemination broadcast
88350ce 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
Refine dissemination value topology
df2e9e8 
Rename the wire model around digests and values, use string value keys, and select active/all-member dissemination trees per topic. Use natural SiloAddress ordering for tree topology.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
Implement fixed-tree dissemination follow-ups
f93792b 
Use status- and age-ordered dissemination topologies with dynamic fanout, fixed-tree forwarding, and level-aware anti-entropy repair.

Add peer-aware topic materialization, membership diff payloads, manifest peer fill, and property/model tests for the new behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
Optimize dissemination diagnostics and lookups
e568b7a 
Move dissemination metric aggregation into instruments and guard unobserved counters before tag/result allocation.

Use frozen collections for local protocol lookup and topology data while keeping wire DTOs unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
fix(runtime): serialize membership diffs with arrays
68898ef 
Use array fields in the internal membership diff payload so Orleans serialization can encode the repair DTO in the runtime test host. The diff semantics remain unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
feat(runtime): reduce dissemination repair overhead
3e2527a 
Add per-topic expected update cadence so anti-entropy only probes value streams which have gone stale, while still allowing topics to emit low-watermark digests for known missing streams.

Queue outbound tree gossip per peer, coalesce by value stream, and flush on delay or batch limits so routine fast-path sends use fewer envelopes without changing monotonic version semantics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
feat(runtime): remove dissemination capability probing
556576a 
Send dissemination messages directly without preflight capability negotiation. Unsupported or temporarily mismatched peers now fail or reject actual messages, with failure backoff and anti-entropy repair handling convergence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
fix: align dissemination behavior with design doc
db67be6 
Validate publish-time values before queueing and use topic-specific repair peer selection. Clarify the efficient broadcast design doc to match opt-in defaults and payload-byte batching.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
perf: tune dissemination collection and batch defaults
0742f04 
Prefer frozen collections for manifest hash summaries and raise dissemination batch and payload defaults.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
perf(dissemination): mark wire types immutable and use immutable coll…
486e344 
…ections

Mark the newly introduced dissemination wire types as [Immutable] so Orleans
skips deep-copying them during RPC. Replace mutable collection members with
immutable equivalents: ImmutableArray for non-byte arrays and
ReadOnlyMemory<byte> for byte payloads.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
fix(dissemination): simplify wire metadata and prune peer state
cfc807a 
Prune stale dissemination peer state, refresh membership when roots are missing, and simplify dissemination digest identity by moving membership snapshot/diff discrimination into the payload.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor(dissemination): simplify root refresh and anti-entropy topics
1b2bea0 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor(dissemination): streamline anti-entropy request building
f935785 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor(dissemination): reduce anti-entropy state allocations
7572d23 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor(dissemination): streamline participant ordering
e66334f 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor(dissemination): streamline pending gossip queues
ca1edfb 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor(dissemination): preserve pending gossip value order
0a37a45 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
ReubenBond and others added 27 commits July 15, 2026 16:06
@ReubenBond
refactor: simplify dissemination membership routing
1179981 
Move dissemination membership caching and routing state into snapshots, use typed dissemination keys and namespaces, and consolidate anti-entropy orchestration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
test: add dissemination membership snapshot properties
9ddd1c7 
Cover snapshot collection uniqueness, routing invariants, anti-entropy peer selection, and constructor validation using CsCheck.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor: reduce dissemination member set storage
e311fbc 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor: defer dissemination broadcast forwarding
f14a40f 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor: simplify dissemination anti-entropy requests
0fe58ec 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor: simplify dissemination topology
4b365e5 
Remove dissemination groups and use the all-member topology for every namespace. Streamline anti-entropy digest construction and exchange, and expose namespace digests as a flat iterator property.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor: simplify dissemination anti-entropy flow
2e0cbae 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor: remove dissemination transport abstraction
c08377d 
Move dissemination broadcast sending into per-peer senders which resolve and cache system target references directly via IInternalGrainFactory. Use ILocalSiloDetails for the local silo address and update tests to mock the grain factory instead of a transport abstraction.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor: use mutable dissemination broadcast batches
90aead8 
Change DisseminationBroadcastBatch to carry Dictionary/List values so peer senders can avoid frozen dictionary and immutable array conversions. Replace the outgoing batch after each send instead of clearing the mutable batch owned by the sent message.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor: simplify dissemination membership snapshot
689c7f8 
Flatten dissemination membership snapshot target caches into the snapshot itself and expose originator and forwarding targets as cached properties used by the protocol and tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor: use mutable dissemination message collections
0a43721 
Use Dictionary and List for anti-entropy request and response payloads while retaining immutable message contracts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor: simplify anti-entropy peer exchange
225f3ab 
Remove anti-entropy peer failure backoff and replace the Parallel.ForAsync fan-out with direct Task.WhenAll peer exchanges. MaxConcurrentSends now only documents and controls broadcast sends.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor: simplify dissemination broadcast scheduling
86975b7 
Use a reusable wake timer and lifetime peer pumps, remove send backoff, and centralize flushing in the scheduled loop. Simplify anti-entropy fan-out and expand lifecycle and timer coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 86a945d7-e85a-4e17-9918-b91516efbab7
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
fix: preserve wake timer wait semantics
5db6688 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 4cf13ff3-e132-4a1a-bc90-77d986b6933d
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
refactor: improve dissemination repair convergence
83a6a4d 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8aebdd9e-9a17-4dab-b500-68db479f4eab
Copilot-Session: 56caa2ba-bee6-450b-b29b-da02b0541333
@ReubenBond
feat(dissemination): track peer versions for repair
c3e32ce 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e55a7332-a93e-40d5-ba0c-201c68eada38
@ReubenBond
docs(dissemination): document protocol, queue, and namespace invariants
b840271 
Add explanatory comments across the dissemination protocol, broadcast
queue, and namespace types clarifying repair/acknowledgment semantics,
scheduling decisions, and membership fingerprinting invariants.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dcb8582-bd88-4992-963d-3dd2c876d01d
@ReubenBond
fix(dissemination): make anti-entropy loop shutdown race- and leak-free
b6c97cc 
StopAsync disposed the anti-entropy PeriodicTimer before cancelling the
shutdown token, so the loop could race the disposal when refreshing
Timer.Period and fault the loop task with ObjectDisposedException.
Additionally, if draining the protocol threw, StopAsync skipped
cancelling and disposing the shutdown token and never observed the loop
task, leaking the CancellationTokenSource and an unobserved exception.

Cancel the shutdown token before disposing the timer, tolerate a
shutdown-time ObjectDisposedException in the loop, and move the loop
await and token disposal into a finally so cleanup always runs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dcb8582-bd88-4992-963d-3dd2c876d01d
@ReubenBond
fix(dissemination): drop broadcast notifications after queue shutdown
dfbfe10 
DisseminationBroadcastQueue.Notify threw ObjectDisposedException once the
queue was stopped, so an in-flight ReceiveBroadcast or publish arriving
during shutdown surfaced the exception back to the RPC caller. Make
Notify a no-op after shutdown, mirroring ObservePeerVersion, since the
peer pumps are already draining or gone.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dcb8582-bd88-4992-963d-3dd2c876d01d
@ReubenBond
fix(dissemination): dispose the broadcast send gate on shutdown
f4b63a2 
The SemaphoreSlim used to bound concurrent broadcast sends was never
disposed. Dispose it once every peer pump has drained during StopAsync,
where no further send can acquire it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dcb8582-bd88-4992-963d-3dd2c876d01d
@ReubenBond
feat(dissemination): emit broadcast flush-scheduling diagnostics
7bfdf88 
Each per-peer broadcast pump arms its flush timer from several code
paths (immediate fill, coalescing, and post-send retry/backoff), which
tests previously observed only via wall-clock Task.Delay bridges.

Emit a Dissemination.BroadcastScheduled DiagnosticListener event
whenever a pump (re)arms its flush timer, carrying the peer, the reason
(Immediate/Coalesce/Retry), the due time, the retry attempt, and the
notification epoch. The event is gated by Listener.IsEnabled, so it
costs nothing when unsubscribed, and the decision is captured under the
pump lock but written after the lock is released.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dcb8582-bd88-4992-963d-3dd2c876d01d
@ReubenBond
test(dissemination): add deterministic broadcast schedule observer
27e5b32 
The retry/backoff broadcast tests bridged the pump's asynchronous
retry-timer arming with wall-clock Task.Delay(50ms) sleeps before
advancing the FakeTimeProvider, which raced the pump under load and made
the tests flaky.

Add a BroadcastScheduleObserver that subscribes to the new
Dissemination.BroadcastScheduled DiagnosticListener event with buffered,
consume-once semantics, and use it in SendFailureRetriesAutomatically-
WithBoundedBackoff and NewNotificationResetsRetryBackoff to await the
exact retry arming before advancing virtual time. The tests are now
deterministic and no longer depend on real time.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dcb8582-bd88-4992-963d-3dd2c876d01d
@ReubenBond
perf(dissemination): cache cluster manifest hash summary by version
46363b0 
GetClusterManifestHashSummary recomputed a SHA-256 hash for every silo
manifest on each call. Cache the resulting summary keyed by the manifest
version and recompute only when the version changes, mirroring the
existing cluster-manifest-update cache. The system target is turn-based,
so no additional synchronization is required.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dcb8582-bd88-4992-963d-3dd2c876d01d
@ReubenBond
fix(dissemination): bound cluster manifest cache growth
8ab7818 
The per-hash manifest cache accumulated an entry for every distinct
grain manifest ever observed and was never pruned, so rolling upgrades
and silo churn grew it without bound. After building an updated cluster
manifest, drop cached entries whose hash is no longer referenced by any
silo in the manifest, always retaining the local silo's manifest. The
live-hash recomputation is skipped until the cache actually outgrows the
live set, so steady-state clusters pay nothing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dcb8582-bd88-4992-963d-3dd2c876d01d
@ReubenBond
test(dissemination): harden fake time and namespace doubles
2c4fbb0 
Remove the unused WaitUntil polling helper. In FakeNamespace.Digests,
derive the ExpectedKeys fallback from the locked version snapshot instead
of reading _versions outside the lock, which races concurrent publishes.
Guard TestTimeProvider reads and Advance with a lock so a background pump
reading the clock never observes a torn DateTimeOffset/timestamp update,
matching AutoAdvancingTimeProvider.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dcb8582-bd88-4992-963d-3dd2c876d01d
@ReubenBond
feat(dissemination): send high-priority namespace updates immediately
c5e2bce 
Membership updates must propagate as fast as possible. Previously every
namespace shared the per-peer coalescing timer, and a notification could not
pull an already-scheduled flush forward, so a membership update arriving after
other gossip could wait the full coalescing window.

Add a configurable DisseminationNamespaceOptions.Priority (Normal/High).
High-priority namespaces bypass the coalescing window: their notifications arm
the peer flush immediately and pull any pending coalesced flush forward, and
leftover high-priority work re-arms immediately after a flush. High-priority
namespaces no longer contribute to the coalescing/retry minima used by normal
namespaces. Membership defaults to High.

A new Priority broadcast-schedule reason makes the immediacy observable for
deterministic tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dcb8582-bd88-4992-963d-3dd2c876d01d
@ReubenBond
feat(dissemination): order high-priority namespace updates ahead of o…
7e8cb58 
…thers

When a peer pump drains its dirty keys, drain higher-priority namespaces first
so their values fill the earliest batches. This ensures membership updates take
precedence over lower-priority gossip and are not starved behind a large backlog
when a batch is capacity-bound.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dcb8582-bd88-4992-963d-3dd2c876d01d
@ReubenBond
ReubenBond force-pushed the feature/efficient-broadcast-draft branch from a94b017 to 7e8cb58 Compare July 15, 2026 23:09
ReubenBond and others added 2 commits July 15, 2026 17:31
@ReubenBond
fix(manifest): use a serializable dictionary for cluster manifest has…
e65c37f 
…h summary

ClusterManifestHashSummary.SiloManifestHashes used a FrozenDictionary, which has
no serialization codec. GetClusterManifestHashSummary() sends this type over the
wire, so the RPC threw CodecNotFoundException. The caller swallows that exception
and silently falls back to fetching each peer's full manifest, defeating the hash
summary optimization.

Switch SiloManifestHashes to a regular Dictionary, which has a codec, so the
summary serializes correctly. Add a serializer round-trip regression test that
would have caught the missing codec.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dcb8582-bd88-4992-963d-3dd2c876d01d
@ReubenBond
test(dissemination): add in-process cluster integration test
e382206 
Existing dissemination unit tests use a fake transport that passes payloads by
reference, so they never exercise serialization. Add an integration test that
builds a real in-process cluster whose in-memory transport is a byte pipe,
running the full serialization pipeline across silos. It enables dissemination
(disabled by default), grows the cluster, and asserts that membership updates are
serialized, disseminated, and applied on at least two distinct silos. This guards
against wire types that lack serialization codecs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3dcb8582-bd88-4992-963d-3dd2c876d01d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ReubenBond