From aa232cb6dfb728d191bdbf2134e22dcbe43234d2 Mon Sep 17 00:00:00 2001 From: Nicu Reut Date: Mon, 24 Aug 2026 08:18:48 +0000 Subject: [PATCH 1/6] Add rate limits initial documentation Signed-off-by: Nicu Reut --- docs-main/docs.json | 1 + .../production-operations/sv-rate-limits.mdx | 72 +++++++++++++++++++ 2 files changed, 73 insertions(+) create mode 100644 docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx diff --git a/docs-main/docs.json b/docs-main/docs.json index 7f1ad3a38..718d6156e 100644 --- a/docs-main/docs.json +++ b/docs-main/docs.json @@ -378,6 +378,7 @@ "global-synchronizer/deployment/kubernetes-deployment", "global-synchronizer/deployment/sv-network-resets", "global-synchronizer/production-operations/sv-security", + "global-synchronizer/production-operations/sv-rate-limits", "global-synchronizer/production-operations/sv-upgrades", "global-synchronizer/production-operations/logical-synchronizer-upgrade", "global-synchronizer/deployment/sv-operations", diff --git a/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx b/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx new file mode 100644 index 000000000..363af72e7 --- /dev/null +++ b/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx @@ -0,0 +1,72 @@ +--- +title: "SV Rate Limits" +sidebarTitle: "SV Rate Limits" +description: "Rate limiting requirements for Super Validator nodes" +--- + +SV nodes must protect their HTTP and gRPC endpoints against excessive load with two complementary layers: + +1. **Application-level rate limits**, enforced by the SV app, the Scan app and the sequencer. They are enabled by default and values are tuned based on observed traffic patterns. Temporary overrides can be applied by the operators to react to unexpected traffic spikes. +2. **Infrastructure-level rate limits**, enforced by your network infrastructure. These are not shipped with Splice and **must** be configured by every SV operator. + +## Application-level rate limits + +### Splice apps + +For each incoming request the SV app and the Scan app evaluate a per-operation per-client-IP limiter, a global per-client-IP limiter, a per-operation limiter and a global limiter. If any of them is exhausted, the request is rejected with `429 Too Many Requests`. + +Each limiter enforces a peak rate that allows short bursts (1s window), combined with a lower rate that is enforced on average over a longer window (60s). Limits can be set globally for an app, per operation, and per client IP. + +#### Client IP header configuration + +The headers used to extract the client IP are configured through the ordered list `rate-limiting.client-ip-headers`: the first configured header that is present and parses as an IP literal is used, taking the first entry of comma-separated values such as `X-Forwarded-For`. IPv6 addresses are grouped by their `/64` prefix. + + +The default headers (`x-forwarded-for`, `x-real-ip`) are client-controlled and can be spoofed. A client rotating the header value bypasses per-client-IP limits entirely. + +Every SV operator must therefore either configure their ingress to **overwrite** (not append to) the header the apps read, or point `client-ip-headers` at a header the client cannot influence. With an Istio/Envoy ingress the recommended value is the non-spoofable `x-envoy-external-address`. Make sure the number of trusted XFF hops on your gateway matches your actual topology, so Envoy derives the real client IP. + + +```hocon +canton.scan-apps.scan-app.parameters.rate-limiting.client-ip-headers = ["x-envoy-external-address"] +canton.sv-apps.sv.parameters.rate-limiting.client-ip-headers = ["x-envoy-external-address"] +``` + +Setting `client-ip-headers` to an empty list disables per-client-IP rate limiting entirely and is not recommended. + +#### Shared rate limiter configuration + +Similar to the sequencer traffic caps configuration, a shared rate limiter configuration file will be maintained centrally for each network (DevNet, TestNet, MainNet). The file defines any overrides that must be applied to the defaults. + +#### Monitoring + +- `splice_splice_rate_limiting` — metrics prefix that identifies the rate limiting metrics. + +## Infrastructure-level rate limits + +### Requirements + +The following infrastructure level rate limits should be configured. These limits should apply individually to the Scan API, public SV app endpoints and the sequencer gRPC API. + +- **Global limits**: a cap on the total request rate reaching a given service. +- **Per-source-IP limits**: a cap on the request rate of a single client IP, so a single client cannot consume the global budget. + +#### Enforcement window + +Both limits must be enforced over a **fixed window of 60 seconds**. A client is allowed at most `limit` requests per 60s window; once the budget for the current window is exhausted, all further requests from that client (or, for the global limit, to that service) are rejected with `429 Too Many Requests` until the window rolls over. + +For the sequencer, limits must be applied to gRPC traffic requests. + +### Optional: banning aggressively retrying clients + +Clients that ignore `429` responses and immediately retry keep consuming ingress capacity even though every request is rejected. Operators are therefore encouraged to additionally **ban clients that retry too quickly**. + +### Optional: fine-grained custom limits + +Beyond the global and per-IP limits above, operators are encouraged to keep the **ability to apply more fine-grained, custom rate limits on demand** — for example per URL path or path prefix, per gRPC service or method, or for an individual client IP or IP range. This makes it possible to react quickly to a misbehaving client or an expensive endpoint without taking the whole node offline, and to lift limits temporarily for a specific trusted peer. + +Since the apps derive the client IP from an HTTP header, the same infrastructure that enforces these limits must also set that header in a way the client cannot influence. + +### Shared rate limiter configuration + +The same shared rate limiter configuration file that defines application-level overrides will also define a number of infrastructure level configurations (global and per-IP limits, the 60s rate limiting interval, per IP overrides, and any ban thresholds) that should be applied to the ingress by the SV operator. From 2f8e6ade4501aa7d3375587a486c60371691809a Mon Sep 17 00:00:00 2001 From: Nicu Reut Date: Wed, 26 Aug 2026 09:11:09 +0200 Subject: [PATCH 2/6] Apply suggestions from code review Co-authored-by: Martin Florian Co-authored-by: Nicu Reut --- .../production-operations/sv-rate-limits.mdx | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx b/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx index 363af72e7..e9f9b45e2 100644 --- a/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx +++ b/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx @@ -4,7 +4,7 @@ sidebarTitle: "SV Rate Limits" description: "Rate limiting requirements for Super Validator nodes" --- -SV nodes must protect their HTTP and gRPC endpoints against excessive load with two complementary layers: +SV nodes protect their HTTP and gRPC endpoints against excessive load with two complementary layers: 1. **Application-level rate limits**, enforced by the SV app, the Scan app and the sequencer. They are enabled by default and values are tuned based on observed traffic patterns. Temporary overrides can be applied by the operators to react to unexpected traffic spikes. 2. **Infrastructure-level rate limits**, enforced by your network infrastructure. These are not shipped with Splice and **must** be configured by every SV operator. @@ -40,13 +40,13 @@ Similar to the sequencer traffic caps configuration, a shared rate limiter confi #### Monitoring -- `splice_splice_rate_limiting` — metrics prefix that identifies the rate limiting metrics. +- `splice_rate_limiting` — metrics prefix that identifies the rate limiting metrics. ## Infrastructure-level rate limits ### Requirements -The following infrastructure level rate limits should be configured. These limits should apply individually to the Scan API, public SV app endpoints and the sequencer gRPC API. +The following infrastructure level rate limits should be configured. These limits should apply individually to the Scan API and the sequencer gRPC API. - **Global limits**: a cap on the total request rate reaching a given service. - **Per-source-IP limits**: a cap on the request rate of a single client IP, so a single client cannot consume the global budget. @@ -57,10 +57,6 @@ Both limits must be enforced over a **fixed window of 60 seconds**. A client is For the sequencer, limits must be applied to gRPC traffic requests. -### Optional: banning aggressively retrying clients - -Clients that ignore `429` responses and immediately retry keep consuming ingress capacity even though every request is rejected. Operators are therefore encouraged to additionally **ban clients that retry too quickly**. - ### Optional: fine-grained custom limits Beyond the global and per-IP limits above, operators are encouraged to keep the **ability to apply more fine-grained, custom rate limits on demand** — for example per URL path or path prefix, per gRPC service or method, or for an individual client IP or IP range. This makes it possible to react quickly to a misbehaving client or an expensive endpoint without taking the whole node offline, and to lift limits temporarily for a specific trusted peer. From c242a60e689c64186ee85c1bda94e660893167ac Mon Sep 17 00:00:00 2001 From: Nicu Reut Date: Wed, 26 Aug 2026 09:11:55 +0200 Subject: [PATCH 3/6] Apply suggestion from @nicu-da --- .../global-synchronizer/production-operations/sv-rate-limits.mdx | 1 - 1 file changed, 1 deletion(-) diff --git a/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx b/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx index e9f9b45e2..55a209388 100644 --- a/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx +++ b/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx @@ -61,7 +61,6 @@ For the sequencer, limits must be applied to gRPC traffic requests. Beyond the global and per-IP limits above, operators are encouraged to keep the **ability to apply more fine-grained, custom rate limits on demand** — for example per URL path or path prefix, per gRPC service or method, or for an individual client IP or IP range. This makes it possible to react quickly to a misbehaving client or an expensive endpoint without taking the whole node offline, and to lift limits temporarily for a specific trusted peer. -Since the apps derive the client IP from an HTTP header, the same infrastructure that enforces these limits must also set that header in a way the client cannot influence. ### Shared rate limiter configuration From 7b6d6db65161a94d85ef167f1e9fc86e4fb9ce74 Mon Sep 17 00:00:00 2001 From: Nicu Reut Date: Wed, 26 Aug 2026 07:17:09 +0000 Subject: [PATCH 4/6] fix docs Signed-off-by: Nicu Reut --- .../production-operations/sv-rate-limits.mdx | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx b/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx index 55a209388..a6dd64939 100644 --- a/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx +++ b/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx @@ -28,8 +28,10 @@ Every SV operator must therefore either configure their ingress to **overwrite** ```hocon -canton.scan-apps.scan-app.parameters.rate-limiting.client-ip-headers = ["x-envoy-external-address"] -canton.sv-apps.sv.parameters.rate-limiting.client-ip-headers = ["x-envoy-external-address"] + - name: ADDITIONAL_CONFIG_RATE_LIMIT_HEADERS + value: | + canton.scan-apps.scan-app.parameters.rate-limiting.client-ip-headers = ["x-envoy-external-address"] + canton.sv-apps.sv.parameters.rate-limiting.client-ip-headers = ["x-envoy-external-address"] ``` Setting `client-ip-headers` to an empty list disables per-client-IP rate limiting entirely and is not recommended. From 65f1dd5a120b2ce78420f5901e2c051651fd7f5e Mon Sep 17 00:00:00 2001 From: Nicu Reut Date: Wed, 26 Aug 2026 07:18:52 +0000 Subject: [PATCH 5/6] . Signed-off-by: Nicu Reut --- .../production-operations/sv-rate-limits.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx b/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx index a6dd64939..5ec2c8ef8 100644 --- a/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx +++ b/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx @@ -57,7 +57,7 @@ The following infrastructure level rate limits should be configured. These limit Both limits must be enforced over a **fixed window of 60 seconds**. A client is allowed at most `limit` requests per 60s window; once the budget for the current window is exhausted, all further requests from that client (or, for the global limit, to that service) are rejected with `429 Too Many Requests` until the window rolls over. -For the sequencer, limits must be applied to gRPC traffic requests. +For the sequencer, limits must be applied to gRPC(HTTP/2) traffic requests. ### Optional: fine-grained custom limits From 0d02b6d0e606a9ca2db7183350fdaf5d3a5a82e1 Mon Sep 17 00:00:00 2001 From: Nicu Reut Date: Wed, 26 Aug 2026 11:35:07 +0000 Subject: [PATCH 6/6] . Signed-off-by: Nicu Reut --- .../production-operations/sv-rate-limits.mdx | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx b/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx index 5ec2c8ef8..82516559c 100644 --- a/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx +++ b/docs-main/global-synchronizer/production-operations/sv-rate-limits.mdx @@ -24,7 +24,7 @@ The headers used to extract the client IP are configured through the ordered lis The default headers (`x-forwarded-for`, `x-real-ip`) are client-controlled and can be spoofed. A client rotating the header value bypasses per-client-IP limits entirely. -Every SV operator must therefore either configure their ingress to **overwrite** (not append to) the header the apps read, or point `client-ip-headers` at a header the client cannot influence. With an Istio/Envoy ingress the recommended value is the non-spoofable `x-envoy-external-address`. Make sure the number of trusted XFF hops on your gateway matches your actual topology, so Envoy derives the real client IP. +The default value for rate-limiting.client-ip-headers is set to ["x-forwarded-for", "x-real-ip"] as a convenience trade-off. Every SV operator must configure set rate-limiting.client-ip-headers to the header fully controlled (rewritten, not passed through or appended to) by their ingress. For example ["x-forwarded-for"] or ["x-envoy-external-address"]. ```hocon @@ -49,6 +49,7 @@ Similar to the sequencer traffic caps configuration, a shared rate limiter confi ### Requirements The following infrastructure level rate limits should be configured. These limits should apply individually to the Scan API and the sequencer gRPC API. +These limits should be higher than the application level limits, and work as a last line of defense. - **Global limits**: a cap on the total request rate reaching a given service. - **Per-source-IP limits**: a cap on the request rate of a single client IP, so a single client cannot consume the global budget. @@ -57,7 +58,7 @@ The following infrastructure level rate limits should be configured. These limit Both limits must be enforced over a **fixed window of 60 seconds**. A client is allowed at most `limit` requests per 60s window; once the budget for the current window is exhausted, all further requests from that client (or, for the global limit, to that service) are rejected with `429 Too Many Requests` until the window rolls over. -For the sequencer, limits must be applied to gRPC(HTTP/2) traffic requests. +For the sequencer, limits must be applied to gRPC(HTTP/2) traffic. ### Optional: fine-grained custom limits @@ -66,4 +67,4 @@ Beyond the global and per-IP limits above, operators are encouraged to keep the ### Shared rate limiter configuration -The same shared rate limiter configuration file that defines application-level overrides will also define a number of infrastructure level configurations (global and per-IP limits, the 60s rate limiting interval, per IP overrides, and any ban thresholds) that should be applied to the ingress by the SV operator. +The same shared rate limiter configuration file that defines application-level overrides will also define a number of infrastructure level configurations (global and per-IP limits, the 60s rate limiting interval, per IP overrides, and any ban thresholds) that represent suggested configurations that the SVs should apply to the infra level rate limits.