Skip to content

feat(grpc-gcp): penalize retryable channel errors - #14219

Open
rahul2393 wants to merge 1 commit into
mainfrom
fm/dcp-split-4-error-penalty
Open

feat(grpc-gcp): penalize retryable channel errors#14219
rahul2393 wants to merge 1 commit into
mainfrom
fm/dcp-split-4-error-penalty

Conversation

@rahul2393

Copy link
Copy Markdown
Contributor

With dynamic scaling enabled, a channel returning retryable errors (UNAVAILABLE / RESOURCE_EXHAUSTED) kept its low active-stream count and so kept attracting picks: the pool had no signal that a channel was unhealthy until its streams actually piled up, and scale-up never triggered for a pool that was failing rather than busy.

Change

  • Error penalty on retryable failures. A call completing with UNAVAILABLE or RESOURCE_EXHAUSTED adds errorPenaltyStep (new pool option) of synthetic load to that channel for errorPenaltyDuration, capped at maxRpcPerChannel. The picker sees the penalty through getPickerLoad(), steering new picks toward healthy channels; expiry is lazy and lock-free.
  • Scale-up integration. Penalty load counts toward the dynamic scale-up trigger, so a pool that is failing-but-idle can still grow.
  • Single net-delta aggregate accounting. Every update to the shared penalty aggregate is computed under the channel lock and applied as one atomic delta, so lock-free readers never observe an intermediate state; a channel leaving the pool clears its contribution.
  • Off by default (errorPenaltyStep = 0 disables); no allocations added on the per-RPC path.

@rahul2393
rahul2393 requested review from a team as code owners August 31, 2026 17:49

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an error penalty mechanism for channels experiencing retryable errors (such as UNAVAILABLE or RESOURCE_EXHAUSTED). It adds configuration options for the penalty step and duration, tracks individual and aggregate penalty loads, and incorporates these penalties into the channel load calculations and scale-up signaling. Additionally, a comprehensive test suite has been added to verify the correctness of the error penalty logic under various scenarios. I have no feedback to provide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant