Repository navigation
feat: adopt provider-runtime's automatic rate-limit slow-down (v1.11.0) - #32
Merged
Merged
Conversation
Bump provider-runtime to v1.10.0 and turn on its rate-limit governor, so reconciles back off by themselves when a git server rate-limits them, per remote and credential, or when the controller's own container is CPU-saturated. Both inputs have a flag each, default false in the binary and true in the chart. The experimental memory input is not exposed. Signals are read in go-git's HTTP transport. go-git fails a session with InsecureSkipTLS for any transport that isn't *http.Transport, so signalClient holds one signal-wrapped client per TLS mode rather than wrapping one client. It is installed for http and https only; ssh, git and file are untouched. Every remote operation now takes the runtime's context, because a signal only arrives if that context reaches the request. So clones and pushes that ran unbounded now fail at the reconcile timeout (GIT_PROVIDER_TIMEOUT, default 4m), which keeps a hung remote from holding a governor slot forever. Keys are host|namespace/name/key, from the Secret's reference and never its value. Repo implements RateLimitKeysProvider, because each of its calls reaches both remotes and those can be two servers with two credentials; LocalResource implements RateLimitKey. Both are on the client: Connect makes no git call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N8HNbCRkKScdEeft3u5kfr
git-provider ships no liveness probe, so it can't get the benefit the CPU rule exists for, and a block with incident-provider's probe showed none either (0 liveness kills with the rule off, 15 runs). Its cost is unknown while provider-runtime's local gate runs after the workqueue pop. This PR now adopts the rate-limit slow-down only. The flag, the env var, the chart value and its README row are removed rather than defaulted off, so nobody turns on a rule with no shown benefit here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N8HNbCRkKScdEeft3u5kfr
Picks up v1.10.1's fix: with queue telemetry on, items keep their priority instead of all running at priority 0. v1.11.0's scale-to-zero package is additive and unused here. No code change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N8HNbCRkKScdEeft3u5kfr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Hold lifted on 2026-11-08. krateo-demo is unfrozen and passes installer's check on installer 0.3.528, and git-provider 0.16.0 is healthy there in krateo-system and in the acme and globex tenants. The user decided to bump to provider-runtime v1.11.0, then merge, release, and hand the pin to installer.
What this does
Adopts provider-runtime's automatic slow-down, so reconciles back off by themselves when a git server rate-limits them, per remote and credential. git-provider is the first real consumer of this feature.
provider-runtimeto v1.11.0 (72076b4; this PR first used v1.10.0). Its upgrade notes require no migration:--rate-limit-governor/GIT_PROVIDER_RATE_LIMIT_GOVERNOR: false in the binary,"true"in the chart. The CPU-saturation and experimental memory inputs are neither exposed nor enabled (see below).SetupRateLimitGovernoronce and passesWithRateLimitGovernorto both reconcilers.httpsignal.NewTransport.Keys
The key is
host|namespace/name/key: the Secret's reference, never its value and no hash of it, so rotating a token doesn't split a quota. With no credential it's the host alone, which is the anonymous quota. A remote that isn't http or https has no key, because it has no response headers to read.ratelimitkey.Forbuilds the key for both the gate and the signals, so the two can't drift apart. It only parses strings and does no I/O.Repo implements
reconciler.RateLimitKeysProvider. Every external call of a Repo reaches both remotes: Observe listsfromRepoand clonestoRepo, and a sync clones both and pushes totoRepo. Those can be different servers with different credentials, as intestdata/samples/repo-gitcookies.yaml. One key can't represent that honestly:So a Repo returns both keys, or one when both remotes are the same host and credential. A call takes a slot in every key plus the local one, all or nothing. This is why the PR needs v1.10.0 or later.
LocalResource implements
reconciler.RateLimitKeyer. It has one remote and one credential.Both keyers are on the
ExternalClient, not the connector.Connectmakes no git call in either controller, it only reads Secrets from the cluster, so gating it on those keys would count calls that never reach those remotes.Connectis still bounded by saturation. A throttled object is deferred at the Observe gate, before any git call.go-git integration
Insecure remotes. A plain
githttp.NewClientaround the wrapper would fail everyspec.insecure: trueresource. WhenInsecureSkipTLSis set, go-git asserts the client's transport is an*http.Transportso it can clone and reconfigure it, and fails the session otherwise. In go-git v5.19.2,plumbing/transport/http/common.go:295-298:So
signalClient(internal/clients/git/transport.go) implementstransport.Transport, holds one signal-wrapped client per TLS mode, and clears the option from a copy of the endpoint. git-provider sets no other per-endpoint transport option.Process-wide install.
init()installs the client forhttpandhttpsbefore any goroutine exists.ssh,gitandfilekeep go-git's own clients, and a test pins that. The cookie path now uses the signal client too, and restores ours rather than go-git's: restoring go-git's would have dropped every later signal in the process with no error.Context. A signal arrives only if the runtime's context reaches the HTTP request.
Clone,GetLatestCommitRemote,Push,Pulland all threegit.Clonecalls ran oncontext.Background(), two of them inside functions that took actx. All of them now use go-git's Context forms.IsInGitCommitHistoryContextis folded intoIsInGitCommitHistory, so no context-free form is left.Non-go-git HTTP. There is none to the git server:
net/httpis used only for the cookie jar.Behaviour change: remote calls now stop at the reconcile timeout
Clones and pushes that used to run unbounded now fail at the timeout. This follows from threading the context, and it's wanted: a hung clone would otherwise hold a governor slot forever.
--timeout/GIT_PROVIDER_TIMEOUT, default4m(main.go), passed toreconciler.WithTimeout.values.yaml, but theenvmap carries it:--set-string env.GIT_PROVIDER_TIMEOUT=10m. Repositories that take longer than the timeout to clone and push need this raised before this ships.ListContexthas none.Known limit:
transportMuisn't context-aware. Cookie and Azure DevOps operations hold it exclusively for a whole clone, and a waiter keeps its governor slot while queued and can't be cancelled there. Holders are now bounded by the timeout, so the wait is finite, but it can chain.Requirement on providers
Expensive work has to run inside gated calls. It does: clones, pushes, rendering and history walks all run inside
Observe,CreateandUpdate.Connectonly reads Secrets, andDeleteis a no-op. Nothing heavy runs outside them.No CPU-saturation governor
An earlier commit here also enabled the CPU-saturation rule. 1690c88 removes it: flag, env var, chart value and README row.
httpGet /healthz, period 10, timeout 1, failure 3), the rule off had 0 liveness kills.It stays available in the runtime, off by default, and can be adopted here once a benefit is shown.
Tests
A real
git http-backendbehind a switchable throttle (gittest.Serve) lets each go-git request type be refused or hung on its own.TestThrottledResponseReachesTheGovernorRetry-Afterand a 403 withX-RateLimit-Remaining: 0, each on every request of every remote operation (list, clone, history, pull, push), reach the governor and defer the key until the server's deadline.TestThrottledRemoteDefersTheNextPass(both controllers)Synced=ReconcileDeferrednaming the key.TestMixedRepoIsDeferredByEitherRemoteTestSignalsAreReportedPerRemote,TestSyncReportsSignalsPerRemoteTestRateLimitKey/TestRateLimitKeys/TestForTestInsecureRemotesKeepWorkingAndReportSignals,...WithACookiespec.insecurestill works and reports signals, a certificate is rejected without it, and the cookie path restores our client.TestHungRemoteStopsAtTheCallersDeadline,TestHungRemoteGivesItsSlotBackAtTheTimeoutTestReportsRateLimits(both controllers)AssertReportsRateLimitsconformance check passes throughObserve, including the advisory on a 200 and the healthy control.TestSetupGovernsEveryReconcilerMaxConcurrentReconciles.TestOnlyHTTPSchemesUseTheSignalClientssh,gitandfilekeep go-git's clients.Revert check: 29 single-change reverts on the final code, each failing its own tests. They cover each dropped context separately, the transport install, the insecure handling, gating a Repo on only
fromRepo, onlytoRepo, a joined key, or no key, and each remote's key in a sync. An early round had a surviving revert: a swapped key in a sync went unnoticed because those tests used one scope for both remotes.TestSyncReportsSignalsPerRemotecloses that gap.Results: CI's test command (
go test -race -tags=unit,integration -p 1 ...) passes locally on this code, including the Gitea end-to-end suites.go vetis clean. The repo has no lint config or lint CI job, so I ran golangci-lint v2.15.0 (provider-runtime's pin) with default settings and no output caps: 61 findings on bothmainand this branch, none new.helm lintpasses and both flags render into the ConfigMap. Not tested here: a real server's rate limit, and any cluster. Vendor responses are provider-runtime's recorded fixtures, and the rollout is held as stated above.🤖 Generated with Claude Code
https://claude.ai/code/session_01N8HNbCRkKScdEeft3u5kfr