Repository navigation
feat(ratelimit): local capacity from measured CPU use, and a share per controller - #98
Merged
Merged
Conversation
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
…r controller Implements docs/design-capacity-model.md. The capacity model (SaturationOptions.CapacityModel, off by default): the sampler reads cpu.max at every sample and, with the model on, cpu.stat's usage_usec; the gated queues report slot time, completions and arrivals. Every ModelInterval the law, a pure function in capacitymodel.go, moves the model's capacity M towards N x (target - B) / (U - B) under its six rules. Local capacity is min(P, floor(M)), at least 1, with P the CPU pressure step's own capacity: a cut halves the published local and pulls M down. The law runs only with a CPU limit of at least MinCPULimit and a readable cpu.stat; otherwise M is the ceiling and the step alone governs. With the option off behaviour is v1.19.0's, and every existing test passes unchanged. The guaranteed share (whenever saturation is on): each gated queue is owed max(1, local / queues) slots. A freed slot goes first to an owed queue, round-robin among them, then to any queue with work. Nothing is preempted. New instruments, from the governor's snapshot: local.model_capacity, local.in_flight, local.guaranteed, local.waiting, local.arrivals, local.cuts, cpu.limit, cpu.used, cpu.background, cpu.per_reconcile and cpu.required. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
configuration.md and api.md gain the four options, the share per controller and the eleven instruments; index.md links the design, whose status is now "built, not measured on a cluster". Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
…t grows Two changes to the law of docs/design-capacity-model.md, decided on the review of the first build. The tolerance (rule 3) is on r = (target - B) / (U - B): M is unchanged if r is within 5% of 1. On M itself, 10% sat exactly on a container pinned at its limit with no background, where wanted is 0.9 x M. The law is evaluated over a window of whole ModelIntervals that grows until it holds 20 completions, so slow reconciles are measured too. A limit change ends the window at the end of its interval. The design's text follows what was built: the cpu.required formula, where the counts are kept, and that any cut by the step pulls M down. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Three changes from an independent review of the first build. A step cut could pin local capacity for good with the model on: the cut sets M to what it left, and M then rose only by a law that refused to move without 20 completions, or with U - B under 5% of the limit. Now the completions only choose the window's length: a window is evaluated at 30 intervals with what it has. With the CPU barely used the law asks for the rise cap, twice M, still only if the slots were used. A window with nothing in flight, or with a limit change, moves nothing. Lending was not round-robin: one cursor served both dispatch passes, so a grant to an owed queue reset where lending went on from, and with two deep queues and a light one every lent slot went to the same queue. Each pass keeps its own cursor. The sampler takes the instant its sample is judged at under the Governor's lock, so the slot time it reads never covers time past that instant. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
An interval with nothing in flight completes nothing, so the window it began ran on into the first busy interval and was judged over all of it: a burst pinned at the limit read under the target, and the fall waited one more window. An idle interval now feeds the background and nothing else. It neither starts a window nor joins one in progress, and doesn't count towards the 30. A window is the sum of the intervals in which something ran. The design's section 3 also states three limits as they are: the doubling with the CPU barely used can overshoot by up to 5% of the limit when the background is large, rule 2 is a high bar at a small capacity, and the cpu gauges are as old as the last window that ended. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
model_block.sh runs three arms (off, the step alone, the step with the capacity model) and writes the bars to its settings before run 1; model_report.py reads the logs against them. The harness takes CAPACITY_MODEL and CPU_TARGET, prints the model's figures in its STAT lines, and takes the workers from RLGOV_WORKERS. The bars are the user's choice of 2026-11-13, a latency win at near-equal throughput; the design's section 10 now states them. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
One 150s run of the model arm at 500m with 32 workers, before block m5. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Stopped by hand, not by a failure: block m5 missed the throughput bar at the 0.9 target, and the user chose to rerun 500m at a 0.95 target before the 1 CPU block. m10-off-1 is a whole run; m10-step-1 was interrupted and is not kept. The 1 CPU block will run under another prefix. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
At a 500m limit the step with the model gives 86.8% of the step alone's throughput for calls 16 times faster, with local at the computed knee and no cut; it missed the bar of 95% set before the run, at the default target and at 0.95, and met the other four. The user decided to ship it as that trade, opt-in. At 1 CPU the workload does not saturate, and the model settled at the knee in two runs of three. The design and the configuration reference state the two properties the runs showed: a target between two slot levels can settle on either, and under the target the law lowers nothing. cpu.required is documented as a lower bound with a standing backlog. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What it adds
Design:
docs/design-capacity-model.md. Measurement:docs/configuration.md, "Capacity model, measured 2026-11-13 and 14".Saturation.CapacityModel, off by default; needs a CPU limit of 500m or more). Over a window of 10s intervals it scales the reconciles in flight by how far CPU use is from the target:in flight × (target × limit − background) ÷ (CPU used − background), the utilization law the Horizontal Pod Autoscaler applies to replicas. Six rules keep it from moving without evidence.Measured on a cluster: a trade, not a win on everything
GKE, 32 workers, each Observe 20ms of CPU then a 400ms wait; bars committed before each block's first run.
500m limit (the workload saturates), five runs per arm:
1 CPU limit (the workload does not saturate: 84% ungoverned), three runs per arm: in two runs the model settled at the knee (22ms a call against 70ms, no less throughput); in the third CPU use never passed the target and it never moved. Under the target the law lowers nothing.
Not measured on a cluster: the share per controller with several controllers, slow reconciles, any provider other than the harness.
Behaviour with the option off
The model is off by default, and every existing test passes unmodified. The share per controller changes which controller gets a freed slot when several share a governor with saturation on; a single controller behaves as before.
Verification
go test -raceonpkg/ratelimit,pkg/telemetry,pkg/controller, including a multi-queue stress test with a moving capacity.Known limits
cpu.requiredis a lower bound while a backlog stands; read it withlocal.waiting.🤖 Generated with Claude Code
https://claude.ai/code/session_01NSM9VYtekkTV1h9dK4WxM4