Skip to content

Commit ffa3616

Browse files
myftijamyftija
committed
feat(webapp): separate rate limit budget for deployment endpoints
Most deploy-flow API calls (build-time env var resolution and sync, env key exchange, preview branches, whoami) drew from the same general API rate limit bucket as an environment's runtime traffic, so a busy environment could starve its own deployments; the /api/v*/deployments endpoints themselves were fully exempt from rate limits as a stopgap. The whole group now goes through a dedicated limiter with its own token bucket, keyed per environment for environment API keys and per token for PATs/OATs, configurable via DEPLOYMENT_RATE_LIMIT_* env vars. The general API limiter whitelists the group via a shared path list. /api/v1/deployments/current is runtime SDK surface and stays exempt from rate limits as before, outside the deploy budget.
1 parent 820c079 commit ffa3616

10 files changed

Lines changed: 171 additions & 11 deletions

‎.server-changes/deployment-rate-limit-budget.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
area: webapp
3+
type: improvement
4+
---
5+
6+
Deployment-related API endpoints now draw from their own rate limit budget, configurable via the `DEPLOYMENT_RATE_LIMIT_*` environment variables, so heavy runtime API traffic in an environment can no longer make its deployments fail with 429s.

‎apps/webapp/app/entry.server.tsx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -302,6 +302,7 @@ singleton("SentryTenantContextProcessor", () => {
302302
});
303303

304304
export { apiRateLimiter } from "./services/apiRateLimit.server";
305+
export { deploymentRateLimiter } from "./services/deploymentRateLimit.server";
305306
export { engineRateLimiter } from "./services/engineRateLimit.server";
306307
export { otlpRateLimiter } from "./services/otlpRateLimit.server";
307308
export { runWithHttpContext } from "./services/httpAsyncStorage.server";

‎apps/webapp/app/env.server.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -614,6 +614,14 @@ const EnvironmentSchema = z
614614
API_RATE_LIMIT_JWT_WINDOW: z.string().default("1m"),
615615
API_RATE_LIMIT_JWT_TOKENS: z.coerce.number().int().default(60),
616616

617+
// Separate budget for deploy-flow endpoints, see deploymentRateLimit.server.ts
618+
DEPLOYMENT_RATE_LIMIT_REFILL_INTERVAL: z.string().default("10s"),
619+
DEPLOYMENT_RATE_LIMIT_MAX: z.coerce.number().int().default(1500),
620+
DEPLOYMENT_RATE_LIMIT_REFILL_RATE: z.coerce.number().int().default(500),
621+
DEPLOYMENT_RATE_LIMIT_REQUEST_LOGS_ENABLED: z.string().default("0"),
622+
DEPLOYMENT_RATE_LIMIT_REJECTION_LOGS_ENABLED: z.string().default("1"),
623+
DEPLOYMENT_RATE_LIMIT_LIMITER_LOGS_ENABLED: z.string().default("0"),
624+
617625
// Per-IP rate limit for the unauthenticated OTLP ingestion endpoints
618626
// (/otel/*). Bounds unauthenticated request rates. Opt-in
619627
// (disabled by default): because it keys on the source IP, it is only

‎apps/webapp/app/services/apiRateLimit.server.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import { resolvePrivateApiKeyRateLimitScope } from "~/models/runtimeEnvironment.
44
import { batchStreamGrants } from "~/runEngine/concerns/batchStreamGrantsInstance.server";
55
import { authenticateAuthorizationHeader } from "./apiAuth.server";
66
import { authorizationRateLimitMiddleware } from "./authorizationRateLimitMiddleware.server";
7+
import { deploymentApiPaths } from "./deploymentApiPaths.server";
78
import type { Duration } from "./rateLimiter.server";
89

910
const BATCH_STREAM_ITEMS_PATH = /^\/api\/v3\/batches\/([^/]+)\/items$/;
@@ -91,7 +92,8 @@ export const apiRateLimiter = authorizationRateLimitMiddleware({
9192
"/api/v1/auth/jwt/claims",
9293
/^\/api\/v1\/runs\/[^/]+\/attempts$/, // /api/v1/runs/$runFriendlyId/attempts
9394
/^\/api\/v1\/waitpoints\/tokens\/[^/]+\/callback\/[^/]+$/, // /api/v1/waitpoints/tokens/$waitpointFriendlyId/callback/$hash
94-
/^\/api\/v\d+\/deployments/, // /api/v{1,2,3,n}/deployments/*
95+
...deploymentApiPaths, // rate limited separately by deploymentRateLimiter
96+
/^\/api\/v\d+\/deployments\/current$/, // runtime SDK surface, exempt as before the deploy budget split
9597
// Internal SDK plumbing — packets are presigned-URL handshakes for
9698
// payload uploads (v2 PUT) and downloads (v1 GET), authenticated via
9799
// run-scoped JWT, called once per task/turn boundary by the runtime.

‎apps/webapp/app/services/authorizationRateLimitMiddleware.server.ts‎

Lines changed: 8 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,12 @@ async function resolveRateLimit(
169169
};
170170
}
171171

172+
export function matchesAnyPath(path: string, matchers: (RegExp | string)[]): boolean {
173+
return matchers.some((matcher) =>
174+
matcher instanceof RegExp ? matcher.test(path) : path === matcher
175+
);
176+
}
177+
172178
/**
173179
* Creates a Ratelimit limiter from a RateLimiterConfig.
174180
* This function is shared across the codebase to ensure consistent limiter creation.
@@ -231,23 +237,15 @@ export function authorizationRateLimitMiddleware({
231237

232238
//first check if any of the pathMatchers match the request path
233239
const path = req.path;
234-
if (
235-
!pathMatchers.some((matcher) =>
236-
matcher instanceof RegExp ? matcher.test(path) : path === matcher
237-
)
238-
) {
240+
if (!matchesAnyPath(path, pathMatchers)) {
239241
if (log.requests) {
240242
logger.info(`RateLimiter (${keyPrefix}): didn't match ${req.path}`);
241243
}
242244
return next();
243245
}
244246

245247
// Check if the path matches any of the whitelisted paths
246-
if (
247-
pathWhiteList.some((matcher) =>
248-
matcher instanceof RegExp ? matcher.test(path) : path === matcher
249-
)
250-
) {
248+
if (matchesAnyPath(path, pathWhiteList)) {
251249
if (log.requests) {
252250
logger.info(`RateLimiter (${keyPrefix}): whitelisted ${req.path}`);
253251
}

‎apps/webapp/app/services/deploymentApiPaths.server.ts‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
// Deploy-flow endpoints. Deploys are highly sensitive to 429s (some builders
2+
// don't retry them), so these get their own rate limit budget
3+
// (deploymentRateLimiter) instead of sharing the general per-environment
4+
// buckets with runtime API traffic.
5+
export const deploymentApiPaths: (RegExp | string)[] = [
6+
// /current is runtime SDK surface (deployments.retrieveCurrent), not deploy
7+
// flow: it stays out of the deploy budget so it can't drain it
8+
/^\/api\/v\d+\/deployments(?!\/current$)(\/|$)/,
9+
/^\/api\/v1\/projects\/[^/]+\/(dev|staging|prod|preview)$/,
10+
/^\/api\/v1\/projects\/[^/]+\/envvars$/,
11+
/^\/api\/v1\/projects\/[^/]+\/envvars\/[^/]+\/import$/,
12+
/^\/api\/v1\/projects\/[^/]+\/branches$/,
13+
/^\/api\/v1\/projects\/[^/]+\/branches\/archive$/,
14+
"/api/v1/whoami",
15+
"/api/v2/whoami",
16+
"/api/v1/remote-build-provider-status",
17+
"/api/v1/artifacts",
18+
];

‎apps/webapp/app/services/deploymentRateLimit.server.ts‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
import { env } from "~/env.server";
2+
import { resolvePrivateApiKeyRateLimitScope } from "~/models/runtimeEnvironment.server";
3+
import { authorizationRateLimitMiddleware } from "./authorizationRateLimitMiddleware.server";
4+
import { deploymentApiPaths } from "./deploymentApiPaths.server";
5+
import type { Duration } from "./rateLimiter.server";
6+
7+
export const deploymentRateLimiter = authorizationRateLimitMiddleware({
8+
redis: {
9+
port: env.RATE_LIMIT_REDIS_PORT,
10+
host: env.RATE_LIMIT_REDIS_HOST,
11+
username: env.RATE_LIMIT_REDIS_USERNAME,
12+
password: env.RATE_LIMIT_REDIS_PASSWORD,
13+
tlsDisabled: env.RATE_LIMIT_REDIS_TLS_DISABLED === "true",
14+
clusterMode: env.RATE_LIMIT_REDIS_CLUSTER_MODE_ENABLED === "1",
15+
},
16+
keyPrefix: "deployment",
17+
defaultLimiter: {
18+
type: "tokenBucket",
19+
refillRate: env.DEPLOYMENT_RATE_LIMIT_REFILL_RATE,
20+
interval: env.DEPLOYMENT_RATE_LIMIT_REFILL_INTERVAL as Duration,
21+
maxTokens: env.DEPLOYMENT_RATE_LIMIT_MAX,
22+
},
23+
limiterCache: {
24+
fresh: 60_000 * 10,
25+
stale: 60_000 * 20,
26+
maxItems: 1000,
27+
},
28+
limiterConfigOverride: async (authorizationValue) => {
29+
const rawApiKey = authorizationValue.replace(/^Bearer /, "");
30+
31+
if (!rawApiKey.startsWith("tr_")) {
32+
return;
33+
}
34+
35+
const scope = await resolvePrivateApiKeyRateLimitScope(rawApiKey);
36+
37+
if (!scope) {
38+
return;
39+
}
40+
41+
// Identifier only: the org's apiRateLimiterConfig governs the general API
42+
// limiter, not the deploy budget.
43+
return {
44+
identifier: scope.environmentId,
45+
};
46+
},
47+
pathMatchers: deploymentApiPaths,
48+
log: {
49+
rejections: env.DEPLOYMENT_RATE_LIMIT_REJECTION_LOGS_ENABLED === "1",
50+
requests: env.DEPLOYMENT_RATE_LIMIT_REQUEST_LOGS_ENABLED === "1",
51+
limiter: env.DEPLOYMENT_RATE_LIMIT_LIMITER_LOGS_ENABLED === "1",
52+
},
53+
});

‎apps/webapp/server.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -183,6 +183,7 @@ async function startServer() {
183183
const socketIo: { io: IoServer } | undefined = build.entry.module.socketIo;
184184
const wss: WebSocketServer | undefined = build.entry.module.wss;
185185
const apiRateLimiter: RateLimitMiddleware = build.entry.module.apiRateLimiter;
186+
const deploymentRateLimiter: RateLimitMiddleware = build.entry.module.deploymentRateLimiter;
186187
const engineRateLimiter: RateLimitMiddleware = build.entry.module.engineRateLimiter;
187188
const otlpRateLimiter: RequestHandler = build.entry.module.otlpRateLimiter;
188189
const runWithHttpContext: RunWithHttpContextFunction = build.entry.module.runWithHttpContext;
@@ -235,6 +236,7 @@ async function startServer() {
235236
}
236237

237238
app.use(apiRateLimiter);
239+
app.use(deploymentRateLimiter);
238240
app.use(engineRateLimiter);
239241
app.use(otlpRateLimiter);
240242

‎apps/webapp/test/deploymentApiPaths.test.ts‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
import { describe, expect, it } from "vitest";
2+
import { matchesAnyPath } from "../app/services/authorizationRateLimitMiddleware.server.js";
3+
import { deploymentApiPaths } from "../app/services/deploymentApiPaths.server.js";
4+
5+
describe("deploymentApiPaths", () => {
6+
it("matches every endpoint the deploy flow calls", () => {
7+
const deployFlowPaths = [
8+
"/api/v1/deployments",
9+
"/api/v1/deployments/latest",
10+
"/api/v1/deployments/deployment_123",
11+
"/api/v1/deployments/deployment_123/progress",
12+
"/api/v1/deployments/deployment_123/fail",
13+
"/api/v1/deployments/deployment_123/cancel",
14+
"/api/v1/deployments/deployment_123/background-workers",
15+
"/api/v1/deployments/deployment_123/generate-registry-credentials",
16+
"/api/v1/deployments/20260811.1/promote",
17+
"/api/v3/deployments/deployment_123/finalize",
18+
"/api/v1/projects/proj_abc123/dev",
19+
"/api/v1/projects/proj_abc123/staging",
20+
"/api/v1/projects/proj_abc123/prod",
21+
"/api/v1/projects/proj_abc123/preview",
22+
"/api/v1/projects/proj_abc123/envvars",
23+
"/api/v1/projects/proj_abc123/envvars/prod/import",
24+
"/api/v1/projects/proj_abc123/branches",
25+
"/api/v1/projects/proj_abc123/branches/archive",
26+
"/api/v1/whoami",
27+
"/api/v2/whoami",
28+
"/api/v1/remote-build-provider-status",
29+
"/api/v1/artifacts",
30+
];
31+
32+
for (const path of deployFlowPaths) {
33+
expect(
34+
matchesAnyPath(path, deploymentApiPaths),
35+
`expected ${path} to be a deployment API path`
36+
).toBe(true);
37+
}
38+
});
39+
40+
it("does not match runtime API surface", () => {
41+
const runtimePaths = [
42+
"/api/v1/deployments/current",
43+
"/api/v1/deploymentsfoo",
44+
"/api/v1/tasks/my-task/trigger",
45+
"/api/v1/tasks/batch",
46+
"/api/v2/runs/run_123",
47+
"/api/v1/runs/run_123/replay",
48+
"/api/v3/runs/run_123/trace",
49+
"/api/v1/projects",
50+
"/api/v1/projects/proj_abc123",
51+
"/api/v1/projects/proj_abc123/dev-status",
52+
"/api/v1/projects/proj_abc123/prod/jwt",
53+
"/api/v1/projects/proj_abc123/envvars/prod",
54+
"/api/v1/projects/proj_abc123/envvars/prod/MY_VAR",
55+
"/api/v1/schedules",
56+
"/api/v1/queues/queue_123",
57+
];
58+
59+
for (const path of runtimePaths) {
60+
expect(
61+
matchesAnyPath(path, deploymentApiPaths),
62+
`expected ${path} not to be a deployment API path`
63+
).toBe(false);
64+
}
65+
});
66+
});

‎docs/self-hosting/env/webapp.mdx‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,12 @@ mode: "wide"
7676
| `API_RATE_LIMIT_LIMITER_LOGS_ENABLED` | No | 0 | API rate limit limiter logs. |
7777
| `API_RATE_LIMIT_JWT_WINDOW` | No | 1m | API rate limit JWT window. |
7878
| `API_RATE_LIMIT_JWT_TOKENS` | No | 60 | API rate limit JWT tokens. |
79+
| `DEPLOYMENT_RATE_LIMIT_REFILL_INTERVAL` | No | 10s | Deployment endpoints rate limit refill interval. |
80+
| `DEPLOYMENT_RATE_LIMIT_MAX` | No | 1500 | Deployment endpoints rate limit max. |
81+
| `DEPLOYMENT_RATE_LIMIT_REFILL_RATE` | No | 500 | Deployment endpoints rate limit refill rate. |
82+
| `DEPLOYMENT_RATE_LIMIT_REQUEST_LOGS_ENABLED` | No | 0 | Deployment endpoints rate limit request logs. |
83+
| `DEPLOYMENT_RATE_LIMIT_REJECTION_LOGS_ENABLED` | No | 1 | Deployment endpoints rate limit rejection logs. |
84+
| `DEPLOYMENT_RATE_LIMIT_LIMITER_LOGS_ENABLED` | No | 0 | Deployment endpoints rate limit limiter logs. |
7985
| **Deploy & Registry** | | | |
8086
| `DEPLOY_REGISTRY_HOST` | Yes | — | Deploy registry host. |
8187
| `DEPLOY_REGISTRY_USERNAME` | No | — | Deploy registry username. |

0 commit comments

Comments
 (0)