Skip to content

fix(azure): bound concurrency in the multi-subscription recommendations fan-out #58

Closed

Description

@cristim

Summary

MultiSubscriptionRecommendationsClient.GetRecommendations (providers/azure/recommendations_multi_subscription.go) fans out to every selected subscription with an unbounded errgroup — one goroutine per subscription, no SetLimit. Each per-subscription client then issues its own service calls (6 services), so a caller with N subscriptions can have roughly N x 6 ARM calls outstanding at once. For a 200-subscription tenant that is ~1200 simultaneous requests.

Latent today: the fan-out is not reachable in production (see LeanerCloud/cloud-commitments-cli#1520's disclosure), and when it is, the scheduler path attaches a CUDLY_MAX_PARALLELISM semaphore to the context, which the per-subscription clients acquire around their outbound calls.

The gap

The bound lives in the caller's context, not in the fan-out layer. A CLI invocation or any library consumer that constructs MultiSubscriptionRecommendationsClient without installing that semaphore gets no bound at all. The fan-out's own doc comment states the semaphore is acquired inside each per-subscription client "so no additional semaphore is needed at this layer" — true only for callers that supply one.

Consequences of the unbounded case: ARM throttling (429s) that the fan-out then reports as per-subscription failures, wasted retries, and a burst profile that can trip subscription-level rate limits for unrelated workloads in the same tenant.

Suggested direction

Give the fan-out a bound that does not depend on the caller remembering to install one:

  • g.SetLimit(n) on the errgroup, with n derived from the same configuration CUDLY_MAX_PARALLELISM feeds (do not hardcode a magic number), or
  • have NewMultiSubscriptionRecommendationsClient install a default semaphore on the context when the caller supplied none, so the caller-provided value still wins when present.

Either way, a test should assert the observed peak concurrency stays within the bound for a large subscription list.

Provenance

Found during the adversarial review of LeanerCloud/cloud-commitments-cli#1520. Deliberately not fixed there — the fan-out is unreachable today, and picking the bound is a configuration decision rather than a defect in the merge/error logic that PR is about.

Activity

  1. self-assigned this
    on Oct 17, 2026
  2. cristim commented on Oct 17, 2026

    @cristim
    MemberAuthor

    Claiming: working on this.

  3. added a commit that references this issue on Oct 17, 2026
    1940dfa
  4. added a commit that references this issue on Oct 18, 2026
    5bb4c18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions