scanServiceQuotas slowed a full all-regions AWS scan by ~3 minutes after
listQuotaDefaultsForCode was added (v0.32.0). Several quota regions are now the
last scanners to return.
Cause
The fan-out concurrency is not at fault. sqWorkers = 30 sits above
sqReqPerSec (10) x ~3s worst-case latency, so the pacer — not the semaphore —
bounds throughput, which is the intended design. newPacer is constructed inside
scanServiceQuotas, so each region gets its own bucket and regions do not
contend.
What changed is the number of calls drawn from that fixed bucket.
listQuotasForCode opens with:
defaults := listQuotaDefaultsForCode(ctx, client, code, pacer)
Four properties compound:
- Unconditional. It runs for every service code before we know whether the
code has any applied quotas in this region. A code unsupported or not
subscribed there returns NoSuchResourceException from ListServiceQuotas
and yields zero rows — but its defaults calls were already spent.
- Serial-before. It runs to completion in the worker slot before applied
paging starts, so each code holds its semaphore slot roughly twice as long.
- Shares one pacer with
ListServiceQuotas, so both operations draw a
single 10 req/s budget even though AWS meters them as separate operations
with separate documented limits.
- Pages wider.
ListAWSDefaultServiceQuotas returns the service's whole
published catalogue; ListServiceQuotas returns only account-applied rows.
Defaults are therefore >= applied in page count, not equal to it.
Per-region calls went from C to at least 2C against a fixed 10 req/s.
ListServices returns on the order of 300-400 codes per region, so the
wall-clock floor moved from roughly 35s to 70s+ per region.
Candidate fixes, cheapest first
A. Give defaults its own pacer. ListAWSDefaultServiceQuotas and
ListServiceQuotas are separate operations with separate documented rate
limits, so sharing one limiter halves the throughput available to each.
A second newPacer(sqReqPerSec, sqBurst) threaded to listQuotaDefaultsForCode
restores the pre-change floor for the applied pass. Confirm against the Service
Quotas docs that the limit is per-operation before landing.
B. Fetch defaults lazily. Reorder listQuotasForCode to page applied quotas
first and fetch defaults only when at least one row came back. Removes every
defaults call for a code unsupported in the region. Highest value if the
unsupported fraction is large; measure it with DISCO_SCAN_RATE_DEBUG before
choosing.
C. Fetch defaults concurrently with the first applied page. Removes the
latency serialization inside the worker slot. Only pays off combined with A —
with one shared pacer the request count is unchanged. Mutually exclusive with B,
which needs the applied result before deciding.
D. Skip defaults for all-global service codes outside the home region.
quotaRow drops GlobalQuota rows in every non-home region, so a service whose
quotas are entirely global pays a defaults fetch per region for rows that are
then discarded. Only reachable with B's ordering.
E. (investigate, not a candidate yet) Cache defaults across regions. Would
cut the defaults cost by a factor of the region count. Requires first proving
AWS default values do not vary by region — they can differ, opt-in regions
especially — so this is unsafe to assume.
A and B are the two simple ones and they compose.
Measuring
DISCO_SCAN_RATE_DEBUG=1 makes reportRateDebug emit calls / elapsed /
observed req/s per region, which is enough to attribute the split between the
two operations before and after any change.
scanServiceQuotasslowed a full all-regions AWS scan by ~3 minutes afterlistQuotaDefaultsForCodewas added (v0.32.0). Several quota regions are now thelast scanners to return.
Cause
The fan-out concurrency is not at fault.
sqWorkers = 30sits abovesqReqPerSec (10) x ~3sworst-case latency, so the pacer — not the semaphore —bounds throughput, which is the intended design.
newPaceris constructed insidescanServiceQuotas, so each region gets its own bucket and regions do notcontend.
What changed is the number of calls drawn from that fixed bucket.
listQuotasForCodeopens with:Four properties compound:
code has any applied quotas in this region. A code unsupported or not
subscribed there returns
NoSuchResourceExceptionfromListServiceQuotasand yields zero rows — but its defaults calls were already spent.
paging starts, so each code holds its semaphore slot roughly twice as long.
ListServiceQuotas, so both operations draw asingle 10 req/s budget even though AWS meters them as separate operations
with separate documented limits.
ListAWSDefaultServiceQuotasreturns the service's wholepublished catalogue;
ListServiceQuotasreturns only account-applied rows.Defaults are therefore >= applied in page count, not equal to it.
Per-region calls went from C to at least 2C against a fixed 10 req/s.
ListServicesreturns on the order of 300-400 codes per region, so thewall-clock floor moved from roughly 35s to 70s+ per region.
Candidate fixes, cheapest first
A. Give defaults its own pacer.
ListAWSDefaultServiceQuotasandListServiceQuotasare separate operations with separate documented ratelimits, so sharing one limiter halves the throughput available to each.
A second
newPacer(sqReqPerSec, sqBurst)threaded tolistQuotaDefaultsForCoderestores the pre-change floor for the applied pass. Confirm against the Service
Quotas docs that the limit is per-operation before landing.
B. Fetch defaults lazily. Reorder
listQuotasForCodeto page applied quotasfirst and fetch defaults only when at least one row came back. Removes every
defaults call for a code unsupported in the region. Highest value if the
unsupported fraction is large; measure it with
DISCO_SCAN_RATE_DEBUGbeforechoosing.
C. Fetch defaults concurrently with the first applied page. Removes the
latency serialization inside the worker slot. Only pays off combined with A —
with one shared pacer the request count is unchanged. Mutually exclusive with B,
which needs the applied result before deciding.
D. Skip defaults for all-global service codes outside the home region.
quotaRowdropsGlobalQuotarows in every non-home region, so a service whosequotas are entirely global pays a defaults fetch per region for rows that are
then discarded. Only reachable with B's ordering.
E. (investigate, not a candidate yet) Cache defaults across regions. Would
cut the defaults cost by a factor of the region count. Requires first proving
AWS default values do not vary by region — they can differ, opt-in regions
especially — so this is unsafe to assume.
A and B are the two simple ones and they compose.
Measuring
DISCO_SCAN_RATE_DEBUG=1makesreportRateDebugemit calls / elapsed /observed req/s per region, which is enough to attribute the split between the
two operations before and after any change.