Add pressure-based durable history retention and metrics - #97
Draft
Tamir Dresher (tamirdresher) wants to merge 1 commit into
Conversation
Tamir Dresher (tamirdresher)
added this pull request to stack #96
September 11, 2026 20:07
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Tamir Dresher (tamirdresher)
force-pushed
the
tamirdresher-microsoft-stack-4-retention-metrics
branch
from
September 12, 2026 03:53
cbb0a7d to
52f607e
Compare
Copilot started reviewing on behalf of
Tamir Dresher (tamirdresher)
September 12, 2026 03:53
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
One or more issues must be addressed before approval.
Get a fresh assessment by requesting another Copilot review.
Review tier: Lite
Findings: 1
Open findings (3)
What changed in this PR
Adds opt-in, pressure-based durable transcript retention with protected schema 2 mailbox state and bounded telemetry.
Changes:
- Adds
KeepAll/Autoretention modes, watermarks, eviction, and capacity failures. - Integrates retention with entity execution, migration, and TTL handling.
- Adds metrics, logging, documentation, and comprehensive tests.
| File | Description |
|---|---|
| dotnet/tests/Microsoft.Agents.AI.DurableTask.UnitTests/DurableAgentTelemetryTests.cs | Updated as part of this pull request. |
| dotnet/tests/Microsoft.Agents.AI.DurableTask.UnitTests/DurableAgentStateRetentionTests.cs | Updated as part of this pull request. |
| dotnet/tests/Microsoft.Agents.AI.DurableTask.UnitTests/AgentEntityTimeToLiveTests.cs | Updated as part of this pull request. |
| dotnet/tests/Microsoft.Agents.AI.DurableTask.UnitTests/AgentEntityHistoryTests.cs | Updated as part of this pull request. |
| dotnet/src/Microsoft.Agents.AI.DurableTask/RetentionResult.cs | Updated as part of this pull request. |
| dotnet/src/Microsoft.Agents.AI.DurableTask/README.md | Updated as part of this pull request. |
| dotnet/src/Microsoft.Agents.AI.DurableTask/Logs.cs | Updated as part of this pull request. |
| dotnet/src/Microsoft.Agents.AI.DurableTask/DurableAgentTelemetry.cs | Updated as part of this pull request. |
| dotnet/src/Microsoft.Agents.AI.DurableTask/DurableAgentStateSizeLimitExceededException.cs | Updated as part of this pull request. |
| dotnet/src/Microsoft.Agents.AI.DurableTask/DurableAgentStateRetention.cs | Updated as part of this pull request. |
| dotnet/src/Microsoft.Agents.AI.DurableTask/DurableAgentsOptions.cs | Updated as part of this pull request. |
| dotnet/src/Microsoft.Agents.AI.DurableTask/DurableAgentHistoryRetentionMode.cs | Updated as part of this pull request. |
| dotnet/src/Microsoft.Agents.AI.DurableTask/CHANGELOG.md | Updated as part of this pull request. |
| dotnet/src/Microsoft.Agents.AI.DurableTask/AgentEntity.cs | Updated as part of this pull request. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| Assert.DoesNotContain(state.Data.ConversationHistory, entry => entry.CorrelationId == "old"); | ||
| Assert.DoesNotContain(state.Data.ConversationHistory, entry => entry.CorrelationId == "completed"); | ||
| Assert.Contains(state.Data.ConversationHistory, entry => entry.CorrelationId == "newest"); |
Comment on lines
+66
to
+69
| while (GetSerializedSize(state) > lowWatermark) | ||
| { | ||
| List<DurableAgentStateEntry>? group = FindOldestEligibleExchange( | ||
| state.Data.ConversationHistory); |
|
|
||
| ## [Unreleased] | ||
|
|
||
| - Added opt-in pressure-based durable transcript retention and low-cardinality operational metrics while protecting schema 2 mailbox and execution-control state. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add opt-in, deterministic pressure retention for durable conversation transcripts and low-cardinality metrics for retention attempts, while preserving schema 2 mailbox and execution-control state.
What changed
KeepAllandAutoretention modes, defaulting toKeepAllso proactive deletion is opt-in.MaxStateBytesbudget and activate it only whenAutois selected.conversationHistorytranscript groups from the 85% high watermark toward the 70% low watermark.DurableAgentStateSizeLimitExceededExceptionwhen eligible transcript eviction cannot reduce complete state below the safe write threshold.DurableAgentTelemetry.MeterNameand record transcript eviction, protected-capacity failure, no-op outcomes, state sizes, entry/message counts, and reclaimed bytes with bounded tags.Retention is not compaction or mailbox expiry
Retention is a destructive durable-storage policy applied only when an application explicitly selects
Auto. It removes persisted transcript groups to keep the complete entity payload within its configured budget.Model-context compaction is a separate concern that changes context presented to a model.
Autois notFollowCompaction, and stateful compaction remains unsupported.Mailbox payload expiry is also separate and remains owned by the layer 2 delivery policy. Pressure retention never removes terminal-result envelopes or completion receipts.
KeepAllperforms no proactive transcript eviction. Backend or provider size limits can still reject writes.Schema 2 and ownership invariants
Autoinitializes genuinely new sessions directly as mailbox-aware schema 2 state. Persisted legacy state, including an empty transcript, is converted only after independently authoritative complete-history migration is explicitly authorized; otherwise new execution and successful duplicate delivery fail closed.Metrics caveat
Retention metrics are attempt-level operational telemetry, not authoritative durable-state truth. They are emitted before the final TaskEntity commit, so later scheduling, persistence, rollback, or retry behavior can produce measurements for an uncommitted attempt or duplicate measurements.
Tags are limited to the registered agent name and bounded outcome/reason values. Session IDs, correlation IDs, message IDs, content, and exception text are never tags. Missing or throwing listeners, including publication and measurement callbacks, cannot change retention behavior.
Validate committed behavior by reloading durable state and checking mailbox outcomes and subsequent model input; emitted counters alone do not prove a successful commit.
Tests
Microsoft.Agents.AI.DurableTask.UnitTestsRelease suite.Microsoft.Agents.AI.Hosting.AzureFunctions.UnitTestsRelease suite.Microsoft.Agents.AI.DurableTask.git diff --check.gpt-5.6-solformal review:ACCEPT.Dependency and stack position
This PR depends on final layer 3,
Add durable chat history ownership and session persistence, atd3872e1e122b1462628793e193eb7409761b4728. Layer 3 preserves the shared opaque history-binding contract while applying the C# fixed-owner profile, and prevents external, service, and opaque owners from mirroring transcripts into entity history.This is product stack layer 4 of 4 and GitHub chain layer 4 of 6. Read the product stack bottom-up:
Not in this PR
FollowCompaction, payload offload, callbacks, outbox behavior, or layer 3 ownership changes.The automatic-retention sample can follow as the next dependent layer; it is not included here.