@W-23735204 Add circuit breaker around data store reads - #639
Draft
npeternel-sf wants to merge 1 commit into
Draft
npeternel-sf wants to merge 1 commit into
npeternel-sf wants to merge 1 commit into
Conversation
Wrap DataStore.getEntry in a per-warm-container in-memory circuit breaker (closed -> open -> half-open) that sheds load when DynamoDB is failing or throttling: after a run of failures (throttles weighted heavier) the breaker opens and fails fast for a cooldown without calling DynamoDB, letting the client's application-level API fallback serve reads, then admits a bounded probe burst to recover. - Misses (DataStoreNotFoundError) count as healthy, never trip the breaker. - Telemetry only on state transitions: open via logMRTError, recovery via a new info-level logMRTEvent so recovery doesn't trip error alerting. - Kill switch: MRT_DATA_STORE_CIRCUIT_BREAKER_DISABLED. Threshold/cooldown/ probe count are internal constants. - CircuitBreaker kept internal (not exported from the package barrel).
clavery
approved these changes
Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wraps
DataStore.getEntryreads in a per-warm-container, in-memory circuit breaker so a container stops hammering DynamoDB when the table is failing or throttling. Builds on the adaptive-retry + throttle-aware work (W-23735205) and the sharding work (W-23735176).Even with adaptive retries and bounded timeouts, storefront-next keeps issuing DAL reads during a sustained throttling event — paying per-call latency/cost and adding load to a saturated table. The breaker converts a run of slow repeated failures into fast, cheap ones and gives the table room to recover.
How it works
closed → open → half-open. Closed counts failures (throttles weighted heavier, reusing the existingisThrottlingErrorsignal). At threshold it opens and fails fast for a cooldown without calling DynamoDB; after the cooldown it admits a bounded half-open probe burst — success closes it, any failure re-opens it.DataStoreServiceErrorimmediately. The DAL client's existing application-level API fallback then serves correct data (the API is the source of truth; DAL is a read-scaling copy).DataStoreNotFoundError(a miss) never trips the breaker; a completed send counts as success even on a miss. Normal miss/hit traffic can't open it.Telemetry
Emitted only on state transitions (log-quota conscious), via MRT internal structured logging:
logMRTError(error level — the backend is failing).logMRTEvent, so recovery doesn't trip error-based alerting.Configuration
One env var kill switch:
MRT_DATA_STORE_CIRCUIT_BREAKER_DISABLED(true/1to disable; unset = active). Threshold, throttle weight, cooldown, and half-open probe count are engineering-tuned internal constants, not incident-time dials.Backward compatibility
getEntry(key)signature and return shape unchanged. Default behavior unchanged when the breaker never trips.CircuitBreakeris kept internal (not exported from the package barrel).Testing
CircuitBreakerunit tests: trip, throttle weighting, open/half-open/recovery, re-open on probe failure, concurrent-probe budget, mixed fail/success non-trip, transition callbacks.getEntryintegration tests: fail-fast with zero DynamoDB calls when open, failures accumulate to open, throttle weighting, miss doesn't trip, recovery closes, concurrent reads straddling half-open admit one probe, telemetry emitted, kill switch bypass (trueand1).@salesforce/mrt-utilities.Work item: W-23735204