Set rate limit quotas without locking yourself out of removing them

[sethbergman]

The last item on the after-v1.0 list reachable without a cloud account. scripts/bootstrap-quotas.sh configures quotas; tests/quotas exercises the result against a real cluster — 21 assertions.

The feature is two vault write calls. The script exists because two of the three obvious ways to make those calls look like they worked and did not, and because one of them takes away your ability to undo it.

sys/quotas/config replaces, it does not merge

Vault ships seven exempt paths — sys/health, sys/seal-status and sys/unseal among them. Writing a single unrelated field empties the list:

$ vault read -format=json sys/quotas/config | jq '.data.rate_limit_exempt_paths | length'
7
$ vault write sys/quotas/config enable_rate_limit_audit_logging=true
$ vault read -format=json sys/quotas/config | jq '.data.rate_limit_exempt_paths | length'
0

Nothing warns you. The next global quota then applies to your load balancer's health checks and to sys/unseal, and you find out at the next restart.

The list is a list

rate_limit_exempt_paths takes an array, and k=v does not produce one:

$ vault write sys/quotas/config rate_limit_exempt_paths="sys/health,sys/seal-status"
$ vault read -format=json sys/quotas/config | jq -c '.data.rate_limit_exempt_paths'
["sys/health,sys/seal-status"]

One element containing a comma, matching no path that exists. It writes successfully and protects nothing. The script sends JSON, then reads the list back and counts it — the broken form and the working one differ only in the length of an array nobody looks at.

A quota rate limits the endpoint that removes it

sys/quotas/* isn't exempt by default, so a quota set too low answers 429 to the DELETE that would remove it. Standbys don't help — they forward to the leader, which is where the limiter is.

The script always exempts sys/quotas/config and sys/quotas/rate-limit, turning that outage into an inconvenience. And if it's already too late: send nothing for one full interval, then spend the first request of the new window on the delete. It returns 204. Any request before it — including the read you'd naturally run first to see what's happening — consumes the budget and you wait again.

The suite creates that lockout deliberately, then demonstrates the recovery.

429 does not identify a quota

A healthy standby answers 429 on sys/health — which is why terraform/aws matches 200,429 on the target group. A tripped quota also answers 429, so a node refusing traffic stays in the pool, healthy by every signal the infrastructure collects. That's the quorum-less node answering 200 to standbyok=true, arriving from the other direction.

Response headers are on by default because that's what separates them:

	Standby 429	Quota 429
headers off	none	none
headers on	none	retry-after, x-ratelimit-*

Testing

Four assertions are about Vault rather than the script, because the script is shaped around those behaviours — if any stops being true, the suite should say so rather than the script quietly guarding nothing.

Three mutations, all watched to fail:

#	Mutation	Caught by
Q1	Exempt paths written comma-joined	the exempt list has nine or more entries (8 in total)
Q2	sys/quotas left unexempted	the quota config is still readable; the quota can be deleted while tripped (5)
Q3	Response headers left off	a quota's 429 carries x-ratelimit-limit (2)

Q1 is the one worth reading: the mutation isn't a mistake, it's the form the CLI leads you to, and it writes successfully.

Auto-classified slow by the #84 derivation, no Makefile edit. 28 suites, 38 jobs, all wired.

Not covered

Lease count, role- and namespace-scoped quotas are Enterprise. How the limit behaves across three nodes isn't established — what was measured is that a standby returns 429 for a request whose budget the leader had already spent.

🤖 Generated with Claude Code