[braghettos]

The wedge

Repo/LocalResource objects stick at Synced=False with:

cannot determine creation result - remove the krateo.io/external-create-pending annotation if it is safe to proceed

…even though the remote push had already succeeded. LocalResource publish-pet-000 sat wedged for ~6 days. Clearing the annotation by hand lets the publish finish immediately.

Root cause — verified from a production crash dump

git-provider mutates go-git's process-global transport registry around every operation, via setDefaultHTTPSClient / setCustomHTTPSClientWithCookieJar.

go-git's plumbing/transport/client.Protocols is a plain map with no locking of its own — InstallProtocol writes it, getTransport reads it on every remote operation. Our clientMutex only guarded our writes; it could not guard go-git's reads.

With MaxConcurrentReconciles defaulting to 5, a write landing while another worker is inside a clone is a fatal runtime error, which kills the process outright — no recover, no deferred cleanup, no error returned to the reconciler.

From the running pod (git-provider:0.15.1, exitCode: 2, reason: Error):

fatal error: concurrent map read and map write

goroutine 178 [running]:
github.com/go-git/go-git/v5/plumbing/transport/client.getTransport(...)
	.../go-git/v5@v5.19.2/plumbing/transport/client/client.go:42
github.com/krateoplatformops/git-provider/internal/clients/git.IsFuncInGitCommitHistory(...)
	/src/internal/clients/git/git.go:376
github.com/krateoplatformops/git-provider/internal/controllers/localresource.(*external).Observe(...)
	/src/internal/controllers/localresource/localresource.go:196
github.com/krateoplatformops/provider-runtime/pkg/reconciler.(*Reconciler).Reconcile(...)
	.../provider-runtime@v1.2.1/pkg/reconciler/reconciler.go:804

Four concurrent IsFuncInGitCommitHistory → Clone goroutines were in flight in that same dump.

How that becomes a permanent wedge

provider-runtime writes external-create-pending, calls Create, then writes external-create-succeeded/-failed (reconciler.go:942 / 977 / 1010). A fatal error between those two writes leaves a bare pending annotation, and every later reconcile refuses to proceed (reconciler.go:766).

This is the discriminator that rules out a "the wrong client got installed" theory: a bad client would make Create return an error, which writes external-create-failed and retries. Only a process death can leave a bare pending. On the cluster, the healthy objects show exactly that failed → retry → succeeded pattern, while publish-pet-000 shows a lone pending.

Note this is not fixable by bumping dependencies: provider-runtime v1.2.1 is the newest tag, and go-git v5.19.2 is the newest v5 (v6 is alpha-only).

The fix

• Purely local operations no longer touch the registry at all — Exists (an fs.Stat!), CurrentBranch, GetLatestCommit, Branch, Commit, UpdateIndex. This removes the large majority of writes. With no git cookie configured, every one of those writes was a no-op restore to begin with.
• Remote operations hold transportMu for the whole operation. Callers needing no custom client take the read lock and share go-git's default client, so concurrency is preserved in the common case; only the cookie-jar and capability paths take the exclusive lock.
• GetLatestCommitRemote is split into a locking wrapper and a non-locking getLatestCommitRemote, because clone calls it while already holding the lock and sync.RWMutex is not reentrant. (Commit→UpdateIndex and clone→Branch nested the same way; those inner calls are now lock-free local ops.)
• transport.UnsupportedCapabilities is a second process-global. It is now read/written only on the Azure DevOps path, under the exclusive lock — and clone now restores it instead of leaking it to every later clone, which it previously never did.
• Releasing restores githttp.DefaultClient, the exact singleton go-git ships, rather than allocating an equivalent client each time.

Incidental bugs this also fixes

The nested calls meant the inner op's defer setDefaultHTTPSClient() restored the default client while the outer op was still running — so a cookie-configured Commit lost its cookie jar partway through, single-threaded. clone had an explicit re-install at line 547 working around exactly that.

Testing

transport_test.go drives readers, cookie writers and capability mutators concurrently; it must stay green under -race. The whole suite passes with go test -race ./....

The pre-fix pattern reproduces the crash locally in isolation (concurrent InstallProtocol + NewClient → fatal error: concurrent map writes), confirming the mechanism rather than just asserting it.

Note for operators

This stops new wedges. Already-wedged objects still need their krateo.io/external-create-pending annotation removed by hand once the push is confirmed — the annotation is deliberately a manual safety gate.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JumEqTGGJ2hf3vjE3xFQsv