Skip to content

feat(ratelimit): local capacity from measured CPU use, and a share per controller - #98

Merged
braghettos merged 33 commits into
mainfrom
feat/capacity-model
Nov 14, 2026
Merged

braghettos merged 33 commits into
mainfrom
feat/capacity-model

Conversation

@braghettos

@braghettos braghettos commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

What it adds

Design: docs/design-capacity-model.md. Measurement: docs/configuration.md, "Capacity model, measured 2026-11-13 and 14".

  • A capacity model, opt-in (Saturation.CapacityModel, off by default; needs a CPU limit of 500m or more). Over a window of 10s intervals it scales the reconciles in flight by how far CPU use is from the target: in flight × (target × limit − background) ÷ (CPU used − background), the utilization law the Horizontal Pod Autoscaler applies to replicas. Six rules keep it from moving without evidence.
  • The v1.19.0 CPU pressure step stays as a veto: local capacity is the lower of the two, and a cut pulls the model down.
  • A guaranteed share per controller, whenever saturation is on: a freed slot goes first to a controller under its equal share, otherwise it is lent to any controller with work. Nothing is preempted.
  • Instruments so something outside the controller can size it: per-controller in-flight, waiting and arrivals, cuts, the CPU limit, CPU per reconcile, and a required-CPU estimate.

Measured on a cluster: a trade, not a win on everything

GKE, 32 workers, each Observe 20ms of CPU then a 400ms wait; bars committed before each block's first run.

500m limit (the workload saturates), five runs per arm:

Arm CPU used Observes/s Mean local Wall time per call Cuts
governor off 99.7% 23.22 — 428ms —
the step alone 99.3% 23.06 26.9 366ms 19
the step with the model, target 0.9 91.0% 20.02 8.9 23ms 0
the step with the model, target 0.95 92.9% 20.75 10.1 24ms 4
  • It met four of the five bars and missed the throughput bar (at least 95% of the step alone's): 86.8% at the default target, 89.6% at 0.95. The user decided to ship it as that trade: about 14% less throughput for calls 16 times faster, opt-in.
  • With one slot a tenth of the limit, a target between two slot levels (0.95 here) can settle on either.

1 CPU limit (the workload does not saturate: 84% ungoverned), three runs per arm: in two runs the model settled at the knee (22ms a call against 70ms, no less throughput); in the third CPU use never passed the target and it never moved. Under the target the law lowers nothing.

Not measured on a cluster: the share per controller with several controllers, slow reconciles, any provider other than the harness.

Behaviour with the option off

The model is off by default, and every existing test passes unmodified. The share per controller changes which controller gets a freed slot when several share a governor with saturation on; a single controller behaves as before.

Verification

  • Each rule of the law, removed in turn, fails a test (26 mutations, all killed).
  • go test -race on pkg/ratelimit, pkg/telemetry, pkg/controller, including a multi-queue stress test with a moving capacity.
  • An independent review of the dispatch and queue code found two defects (a cut could pin capacity; lending was not round-robin), both fixed with the reviewer's failing tests added, and no lost slots, deadlocks or races.

Known limits

  • After a cut, recovery with very slow reconciles is one doubling per evaluated window (up to 5 minutes), against the step's +1 per 2s.
  • cpu.required is a lower bound while a backlog stands; read it with local.waiting.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4

braghettos and others added 30 commits November 13, 2026 19:16
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
…r controller

Implements docs/design-capacity-model.md.

The capacity model (SaturationOptions.CapacityModel, off by default): the
sampler reads cpu.max at every sample and, with the model on, cpu.stat's
usage_usec; the gated queues report slot time, completions and arrivals.
Every ModelInterval the law, a pure function in capacitymodel.go, moves the
model's capacity M towards N x (target - B) / (U - B) under its six rules.
Local capacity is min(P, floor(M)), at least 1, with P the CPU pressure
step's own capacity: a cut halves the published local and pulls M down. The
law runs only with a CPU limit of at least MinCPULimit and a readable
cpu.stat; otherwise M is the ceiling and the step alone governs. With the
option off behaviour is v1.19.0's, and every existing test passes unchanged.

The guaranteed share (whenever saturation is on): each gated queue is owed
max(1, local / queues) slots. A freed slot goes first to an owed queue,
round-robin among them, then to any queue with work. Nothing is preempted.

New instruments, from the governor's snapshot: local.model_capacity,
local.in_flight, local.guaranteed, local.waiting, local.arrivals,
local.cuts, cpu.limit, cpu.used, cpu.background, cpu.per_reconcile and
cpu.required.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
configuration.md and api.md gain the four options, the share per
controller and the eleven instruments; index.md links the design, whose
status is now "built, not measured on a cluster".

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
…t grows

Two changes to the law of docs/design-capacity-model.md, decided on the
review of the first build.

The tolerance (rule 3) is on r = (target - B) / (U - B): M is unchanged if
r is within 5% of 1. On M itself, 10% sat exactly on a container pinned at
its limit with no background, where wanted is 0.9 x M.

The law is evaluated over a window of whole ModelIntervals that grows until
it holds 20 completions, so slow reconciles are measured too. A limit
change ends the window at the end of its interval.

The design's text follows what was built: the cpu.required formula, where
the counts are kept, and that any cut by the step pulls M down.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Three changes from an independent review of the first build.

A step cut could pin local capacity for good with the model on: the cut
sets M to what it left, and M then rose only by a law that refused to move
without 20 completions, or with U - B under 5% of the limit. Now the
completions only choose the window's length: a window is evaluated at 30
intervals with what it has. With the CPU barely used the law asks for the
rise cap, twice M, still only if the slots were used. A window with nothing
in flight, or with a limit change, moves nothing.

Lending was not round-robin: one cursor served both dispatch passes, so a
grant to an owed queue reset where lending went on from, and with two deep
queues and a light one every lent slot went to the same queue. Each pass
keeps its own cursor.

The sampler takes the instant its sample is judged at under the Governor's
lock, so the slot time it reads never covers time past that instant.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
An interval with nothing in flight completes nothing, so the window it
began ran on into the first busy interval and was judged over all of it: a
burst pinned at the limit read under the target, and the fall waited one
more window.

An idle interval now feeds the background and nothing else. It neither
starts a window nor joins one in progress, and doesn't count towards the 30.
A window is the sum of the intervals in which something ran.

The design's section 3 also states three limits as they are: the doubling
with the CPU barely used can overshoot by up to 5% of the limit when the
background is large, rule 2 is a high bar at a small capacity, and the cpu
gauges are as old as the last window that ended.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
model_block.sh runs three arms (off, the step alone, the step with the
capacity model) and writes the bars to its settings before run 1;
model_report.py reads the logs against them. The harness takes
CAPACITY_MODEL and CPU_TARGET, prints the model's figures in its STAT
lines, and takes the workers from RLGOV_WORKERS.

The bars are the user's choice of 2026-11-13, a latency win at
near-equal throughput; the design's section 10 now states them.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
One 150s run of the model arm at 500m with 32 workers, before block m5.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Stopped by hand, not by a failure: block m5 missed the throughput bar
at the 0.9 target, and the user chose to rerun 500m at a 0.95 target
before the 1 CPU block. m10-off-1 is a whole run; m10-step-1 was
interrupted and is not kept. The 1 CPU block will run under another
prefix.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
braghettos and others added 3 commits November 14, 2026 09:11
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
At a 500m limit the step with the model gives 86.8% of the step alone's
throughput for calls 16 times faster, with local at the computed knee
and no cut; it missed the bar of 95% set before the run, at the default
target and at 0.95, and met the other four. The user decided to ship it
as that trade, opt-in. At 1 CPU the workload does not saturate, and the
model settled at the knee in two runs of three.

The design and the configuration reference state the two properties the
runs showed: a target between two slot levels can settle on either, and
under the target the law lowers nothing. cpu.required is documented as
a lower bound with a standing backlog.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
@braghettos
braghettos marked this pull request as ready for review November 14, 2026 08:12
@braghettos
braghettos merged commit e0a759e into main Nov 14, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant