Skip to content

feat: adopt provider-runtime's automatic rate-limit slow-down (v1.11.0) - #32

Merged
braghettos merged 3 commits into
mainfrom
feat/rate-limit-governor-v1.10.0
Nov 8, 2026
Merged

braghettos merged 3 commits into
mainfrom
feat/rate-limit-governor-v1.10.0

Conversation

@braghettos

@braghettos braghettos commented Nov 5, 2026 •

Copy link
Copy Markdown

Note

Hold lifted on 2026-11-08. krateo-demo is unfrozen and passes installer's check on installer 0.3.528, and git-provider 0.16.0 is healthy there in krateo-system and in the acme and globex tenants. The user decided to bump to provider-runtime v1.11.0, then merge, release, and hand the pin to installer.

What this does

Adopts provider-runtime's automatic slow-down, so reconciles back off by themselves when a git server rate-limits them, per remote and credential. git-provider is the first real consumer of this feature.

  • Bumps provider-runtime to v1.11.0 (72076b4; this PR first used v1.10.0). Its upgrade notes require no migration:
    • v1.9.0 to v1.10.0 are additive, and the governor is off unless enabled.
    • v1.10.1 keeps item priorities when queue telemetry is on.
    • v1.11.0 adds an unused scale-to-zero package.
  • --rate-limit-governor / GIT_PROVIDER_RATE_LIMIT_GOVERNOR: false in the binary, "true" in the chart. The CPU-saturation and experimental memory inputs are neither exposed nor enabled (see below).
  • Calls SetupRateLimitGovernor once and passes WithRateLimitGovernor to both reconcilers.
  • Reads rate-limit signals in go-git's HTTP transport, with httpsignal.NewTransport.

Keys

The key is host|namespace/name/key: the Secret's reference, never its value and no hash of it, so rotating a token doesn't split a quota. With no credential it's the host alone, which is the anonymous quota. A remote that isn't http or https has no key, because it has no response headers to read. ratelimitkey.For builds the key for both the gate and the signals, so the two can't drift apart. It only parses strings and does no I/O.

Repo implements reconciler.RateLimitKeysProvider. Every external call of a Repo reaches both remotes: Observe lists fromRepo and clones toRepo, and a sync clones both and pushes to toRepo. Those can be different servers with different credentials, as in testdata/samples/repo-gitcookies.yaml. One key can't represent that honestly:

  • Either remote's key alone leaves the object calling the other remote while it refuses.
  • Reporting all signals under one key throttles a healthy remote and names the wrong host.
  • A combined key splits a shared quota across every pair of remotes.

So a Repo returns both keys, or one when both remotes are the same host and credential. A call takes a slot in every key plus the local one, all or nothing. This is why the PR needs v1.10.0 or later.

LocalResource implements reconciler.RateLimitKeyer. It has one remote and one credential.

Both keyers are on the ExternalClient, not the connector. Connect makes no git call in either controller, it only reads Secrets from the cluster, so gating it on those keys would count calls that never reach those remotes. Connect is still bounded by saturation. A throttled object is deferred at the Observe gate, before any git call.

go-git integration

Insecure remotes. A plain githttp.NewClient around the wrapper would fail every spec.insecure: true resource. When InsecureSkipTLS is set, go-git asserts the client's transport is an *http.Transport so it can clone and reconfigure it, and fails the session otherwise. In go-git v5.19.2, plumbing/transport/http/common.go:295-298:

tr, ok := c.client.Transport.(*http.Transport)
if !ok {
	return nil, fmt.Errorf("expected underlying client transport to be of type: %s; got: %s",
		reflect.TypeOf(transport), reflect.TypeOf(c.client.Transport))
}

So signalClient (internal/clients/git/transport.go) implements transport.Transport, holds one signal-wrapped client per TLS mode, and clears the option from a copy of the endpoint. git-provider sets no other per-endpoint transport option.

Process-wide install. init() installs the client for http and https before any goroutine exists. ssh, git and file keep go-git's own clients, and a test pins that. The cookie path now uses the signal client too, and restores ours rather than go-git's: restoring go-git's would have dropped every later signal in the process with no error.

Context. A signal arrives only if the runtime's context reaches the HTTP request. Clone, GetLatestCommitRemote, Push, Pull and all three git.Clone calls ran on context.Background(), two of them inside functions that took a ctx. All of them now use go-git's Context forms. IsInGitCommitHistoryContext is folded into IsInGitCommitHistory, so no context-free form is left.

Non-go-git HTTP. There is none to the git server: net/http is used only for the cookie jar.

Behaviour change: remote calls now stop at the reconcile timeout

Clones and pushes that used to run unbounded now fail at the timeout. This follows from threading the context, and it's wanted: a hung clone would otherwise hold a governor slot forever.

  • The timeout is --timeout / GIT_PROVIDER_TIMEOUT, default 4m (main.go), passed to reconciler.WithTimeout.
  • It's cumulative for the whole pass, not per call: Connect, Observe and Create share it. The README said "for each action", which was wrong, and is corrected here.
  • The chart doesn't list it in values.yaml, but the env map carries it: --set-string env.GIT_PROVIDER_TIMEOUT=10m. Repositories that take longer than the timeout to clone and push need this raised before this ships.
  • A sync that overran was already broken in a quieter way: the status write used the expired context, so the push landed and its record was lost. It now fails earlier.
  • A push cut off at the deadline may still have landed on the server, as with any interrupted call. The next pass observes the target and reconciles it.
  • Listing refs keeps go-git's 10s limit, now applied explicitly because ListContext has none.

Known limit: transportMu isn't context-aware. Cookie and Azure DevOps operations hold it exclusively for a whole clone, and a waiter keeps its governor slot while queued and can't be cancelled there. Holders are now bounded by the timeout, so the wait is finite, but it can chain.

Requirement on providers

Expensive work has to run inside gated calls. It does: clones, pushes, rendering and history walks all run inside Observe, Create and Update. Connect only reads Secrets, and Delete is a no-op. Nothing heavy runs outside them.

No CPU-saturation governor

An earlier commit here also enabled the CPU-saturation rule. 1690c88 removes it: flag, env var, chart value and README row.

  • No benefit is possible here: the rule's protective case is a saturated controller failing its liveness probe, and git-provider ships no liveness probe.
  • No benefit was shown elsewhere: in 15 runs at 250m with incident-provider's probe (httpGet /healthz, period 10, timeout 1, failure 3), the rule off had 0 liveness kills.
  • Its cost isn't known yet: provider-runtime's local gate runs after the workqueue pop, so a deferred object can sleep through a freed slot (the shape of unstructured-runtime#72). That is being fixed in provider-runtime and will be re-measured.

It stays available in the runtime, off by default, and can be adopted here once a benefit is shown.

Tests

A real git http-backend behind a switchable throttle (gittest.Serve) lets each go-git request type be refused or hung on its own.

Test What it proves
TestThrottledResponseReachesTheGovernor A 429 with Retry-After and a 403 with X-RateLimit-Remaining: 0, each on every request of every remote operation (list, clone, history, pull, push), reach the governor and defer the key until the server's deadline.
TestThrottledRemoteDefersTheNextPass (both controllers) Real reconciler, connector and go-git: the pass that gets the refusal errors, and the next is deferred with no request to the server, with Synced=ReconcileDeferred naming the key.
TestMixedRepoIsDeferredByEitherRemote A Repo on two servers with two credentials: a 429 from either remote defers the next pass with no further request to either server. The object names the remote that refused, and the healthy remote stays open.
TestSignalsAreReportedPerRemote, TestSyncReportsSignalsPerRemote A refusal lands on the key of the remote that refused, for Observe and for each step of a sync.
TestRateLimitKey / TestRateLimitKeys / TestFor The expected key for each credential shape. The keyer tests run on a client with no kube client, so any read panics, and a live server sees zero requests.
TestInsecureRemotesKeepWorkingAndReportSignals, ...WithACookie spec.insecure still works and reports signals, a certificate is rejected without it, and the cookie path restores our client.
TestHungRemoteStopsAtTheCallersDeadline, TestHungRemoteGivesItsSlotBackAtTheTimeout A server that never answers stops at the deadline, and the pass gives its governor slot back.
TestReportsRateLimits (both controllers) provider-runtime's AssertReportsRateLimits conformance check passes through Observe, including the advisory on a 200 and the healthy control.
TestSetupGovernsEveryReconciler Both reconcilers register with the governor: the ceiling is twice MaxConcurrentReconciles.
TestOnlyHTTPSchemesUseTheSignalClient ssh, git and file keep go-git's clients.

Revert check: 29 single-change reverts on the final code, each failing its own tests. They cover each dropped context separately, the transport install, the insecure handling, gating a Repo on only fromRepo, only toRepo, a joined key, or no key, and each remote's key in a sync. An early round had a surviving revert: a swapped key in a sync went unnoticed because those tests used one scope for both remotes. TestSyncReportsSignalsPerRemote closes that gap.

Results: CI's test command (go test -race -tags=unit,integration -p 1 ...) passes locally on this code, including the Gitea end-to-end suites. go vet is clean. The repo has no lint config or lint CI job, so I ran golangci-lint v2.15.0 (provider-runtime's pin) with default settings and no output caps: 61 findings on both main and this branch, none new. helm lint passes and both flags render into the ConfigMap. Not tested here: a real server's rate limit, and any cluster. Vendor responses are provider-runtime's recorded fixtures, and the rollout is held as stated above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N8HNbCRkKScdEeft3u5kfr

braghettos and others added 3 commits November 5, 2026 23:03
Bump provider-runtime to v1.10.0 and turn on its rate-limit governor, so
reconciles back off by themselves when a git server rate-limits them, per
remote and credential, or when the controller's own container is
CPU-saturated. Both inputs have a flag each, default false in the binary and
true in the chart. The experimental memory input is not exposed.

Signals are read in go-git's HTTP transport. go-git fails a session with
InsecureSkipTLS for any transport that isn't *http.Transport, so signalClient
holds one signal-wrapped client per TLS mode rather than wrapping one client.
It is installed for http and https only; ssh, git and file are untouched.

Every remote operation now takes the runtime's context, because a signal only
arrives if that context reaches the request. So clones and pushes that ran
unbounded now fail at the reconcile timeout (GIT_PROVIDER_TIMEOUT, default
4m), which keeps a hung remote from holding a governor slot forever.

Keys are host|namespace/name/key, from the Secret's reference and never its
value. Repo implements RateLimitKeysProvider, because each of its calls
reaches both remotes and those can be two servers with two credentials;
LocalResource implements RateLimitKey. Both are on the client: Connect makes
no git call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N8HNbCRkKScdEeft3u5kfr
git-provider ships no liveness probe, so it can't get the benefit the CPU
rule exists for, and a block with incident-provider's probe showed none
either (0 liveness kills with the rule off, 15 runs). Its cost is
unknown while provider-runtime's local gate runs after the workqueue pop.
This PR now adopts the rate-limit slow-down only. The flag, the env var,
the chart value and its README row are removed rather than defaulted off,
so nobody turns on a rule with no shown benefit here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N8HNbCRkKScdEeft3u5kfr
Picks up v1.10.1's fix: with queue telemetry on, items keep their priority
instead of all running at priority 0. v1.11.0's scale-to-zero package is
additive and unused here. No code change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N8HNbCRkKScdEeft3u5kfr
@braghettos braghettos changed the title feat: adopt provider-runtime v1.10.0 automatic slow-down (on hold, do not merge) feat: adopt provider-runtime's automatic rate-limit slow-down (v1.11.0) Nov 8, 2026
@braghettos
braghettos merged commit 8bc9e3c into main Nov 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant