-
Notifications
You must be signed in to change notification settings - Fork 33
Add rate limits initial documentation #1450
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
70 changes: 70 additions & 0 deletions
70
docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| --- | ||
| title: "SV Rate Limits" | ||
| sidebarTitle: "SV Rate Limits" | ||
| description: "Rate limiting requirements for Super Validator nodes" | ||
| --- | ||
|
|
||
| SV nodes protect their HTTP and gRPC endpoints against excessive load with two complementary layers: | ||
|
|
||
| 1. **Application-level rate limits**, enforced by the SV app, the Scan app and the sequencer. They are enabled by default and values are tuned based on observed traffic patterns. Temporary overrides can be applied by the operators to react to unexpected traffic spikes. | ||
| 2. **Infrastructure-level rate limits**, enforced by your network infrastructure. These are not shipped with Splice and **must** be configured by every SV operator. | ||
|
|
||
| ## Application-level rate limits | ||
|
|
||
| ### Splice apps | ||
|
|
||
| For each incoming request the SV app and the Scan app evaluate a per-operation per-client-IP limiter, a global per-client-IP limiter, a per-operation limiter and a global limiter. If any of them is exhausted, the request is rejected with `429 Too Many Requests`. | ||
|
|
||
| Each limiter enforces a peak rate that allows short bursts (1s window), combined with a lower rate that is enforced on average over a longer window (60s). Limits can be set globally for an app, per operation, and per client IP. | ||
|
|
||
| #### Client IP header configuration | ||
|
|
||
| The headers used to extract the client IP are configured through the ordered list `rate-limiting.client-ip-headers`: the first configured header that is present and parses as an IP literal is used, taking the first entry of comma-separated values such as `X-Forwarded-For`. IPv6 addresses are grouped by their `/64` prefix. | ||
|
|
||
| <Warning> | ||
| The default headers (`x-forwarded-for`, `x-real-ip`) are client-controlled and can be spoofed. A client rotating the header value bypasses per-client-IP limits entirely. | ||
|
|
||
| The default value for rate-limiting.client-ip-headers is set to ["x-forwarded-for", "x-real-ip"] as a convenience trade-off. Every SV operator must configure set rate-limiting.client-ip-headers to the header fully controlled (rewritten, not passed through or appended to) by their ingress. For example ["x-forwarded-for"] or ["x-envoy-external-address"]. | ||
| </Warning> | ||
|
|
||
| ```hocon | ||
| - name: ADDITIONAL_CONFIG_RATE_LIMIT_HEADERS | ||
| value: | | ||
| canton.scan-apps.scan-app.parameters.rate-limiting.client-ip-headers = ["x-envoy-external-address"] | ||
| canton.sv-apps.sv.parameters.rate-limiting.client-ip-headers = ["x-envoy-external-address"] | ||
| ``` | ||
|
|
||
| Setting `client-ip-headers` to an empty list disables per-client-IP rate limiting entirely and is not recommended. | ||
|
|
||
| #### Shared rate limiter configuration | ||
|
|
||
| Similar to the sequencer traffic caps configuration, a shared rate limiter configuration file will be maintained centrally for each network (DevNet, TestNet, MainNet). The file defines any overrides that must be applied to the defaults. | ||
|
|
||
| #### Monitoring | ||
|
|
||
| - `splice_rate_limiting` — metrics prefix that identifies the rate limiting metrics. | ||
|
|
||
| ## Infrastructure-level rate limits | ||
|
|
||
| ### Requirements | ||
|
|
||
| The following infrastructure level rate limits should be configured. These limits should apply individually to the Scan API and the sequencer gRPC API. | ||
| These limits should be higher than the application level limits, and work as a last line of defense. | ||
|
|
||
| - **Global limits**: a cap on the total request rate reaching a given service. | ||
| - **Per-source-IP limits**: a cap on the request rate of a single client IP, so a single client cannot consume the global budget. | ||
|
|
||
| #### Enforcement window | ||
|
|
||
| Both limits must be enforced over a **fixed window of 60 seconds**. A client is allowed at most `limit` requests per 60s window; once the budget for the current window is exhausted, all further requests from that client (or, for the global limit, to that service) are rejected with `429 Too Many Requests` until the window rolls over. | ||
|
|
||
| For the sequencer, limits must be applied to gRPC(HTTP/2) traffic. | ||
|
|
||
| ### Optional: fine-grained custom limits | ||
|
|
||
| Beyond the global and per-IP limits above, operators are encouraged to keep the **ability to apply more fine-grained, custom rate limits on demand** — for example per URL path or path prefix, per gRPC service or method, or for an individual client IP or IP range. This makes it possible to react quickly to a misbehaving client or an expensive endpoint without taking the whole node offline, and to lift limits temporarily for a specific trusted peer. | ||
|
|
||
|
|
||
| ### Shared rate limiter configuration | ||
|
|
||
| The same shared rate limiter configuration file that defines application-level overrides will also define a number of infrastructure level configurations (global and per-IP limits, the 60s rate limiting interval, per IP overrides, and any ban thresholds) that represent suggested configurations that the SVs should apply to the infra level rate limits. | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it would make sense to note that these limits are last resort and they must be higher than app limits
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added a note