Skip to content

v4: module timeout silently spans the entire retry chain including backoff; timeouts during backoff log a false "ignored cancellation" warning #3790

Description

@thomhurst

ExecuteWithPolicies wraps the Polly retry policy inside the timeout (executeFunc = ct => retryPolicy.ExecuteAsync(...) is passed to TimeoutHelper), so WithTimeout(5min) + WithRetry(3) gives 5 minutes total for all four attempts plus exponential backoff — not 5 minutes per attempt as most users expect, and neither WithTimeout nor WithRetry documents the interaction.

Additionally, wasCancellationTokenRespected requires the last module attempt to have observed cancellation (moduleAttemptRespondedToCancellation == 1, lines 520-522); when the timeout fires while Polly is sleeping between attempts, the last attempt ended before cancellation, so the flag stays 0 and HandleException logs "Module did not complete within the cancellation grace period" (lines 671-677) even though the module code fully respected its token.

For v4: decide and document the semantics — per-attempt timeout (or a WithAttemptTimeout/WithTotalTimeout pair) is the least surprising — and exclude backoff sleeps from the token-cooperation heuristic.

Evidence: src/ModularPipelines/Engine/ModuleExecutionPipeline.cs:496-533,671-677, src/ModularPipelines/Configuration/ModuleConfigurationBuilder.cs:308-352


Raised by the v4 scoping audit round 3 — umbrella #3239.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium: improvements and polish - after P1breakingBreaking API changeenhancementNew feature or requestin-progressPicked up by an agent

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions