Skip to content

Know when a provider credential stops working - #2126

Merged
ppXD merged 1 commit into
mainfrom
fix/know-when-a-provider-credential-stops-working
Oct 18, 2026
Merged

ppXD merged 1 commit into
mainfrom
fix/know-when-a-provider-credential-stops-working

Conversation

@ppXD

@ppXD ppXD commented Oct 18, 2026

Copy link
Copy Markdown
Owner

Summary

  • A credential whose provider has refused it for good is now recorded and named instead of staying Active: a refresh refused for the grant or the client marks it Error with "needs reconnect: " and fails with the typed credential_needs_reconnect; a successful refresh or probe lifts it. A 401 on an access token GitLab retired early is retried once with the token another caller stored (or after one rotation), by NGitLab, the raw calls, and the provider's typed writes. CredentialSummary gains a derived health and the days to expiry.
  • A recurring probe asks each provider about a bounded batch of credentials and records what it says: GitLab's real expiry for personal, project and group tokens (also when a token is linked or added), validation for what it accepts, Error only for a definite 401, and a recheck of an Error credential after an hour so it can heal (an OAuth one also on use). The sweep cannot be held by a provider that stops answering: the identity call honours its deadline, a probe that ignores its token is walked away from at the probe deadline, a pass ends at a budget under the tick interval, and a tick that finds the previous pass running skips.
  • Reconnecting beside a credential its provider refused (OAuth, token link or credentials API) carries its repositories and group hooks to the new credential, as a disconnect's reconnect does; the repair sweep applies the same rule. A rotated pair a process could not store is stored before the fail-fast applies, so another process's refusal of the consumed refresh token cannot strand it. The GitLab members lookup throws instead of answering Unknown, retries a 429 or 5xx once, and admission refuses with credential_rejected or author_standing_unknown naming the status and the scope needed.

No migration: status Error and last_error already exist. Adding a token through POST /api/credentials/pat or /group-access-token now asks the provider about it (bounded by the 30 s probe deadline); a provider that cannot be reached, is silent or refuses the token leaves the credential added as before and due for the sweep.

Test plan

  • Unit: 14550 passed, 1 skipped (resolver on-use recheck window, strategy ordering for both providers, StoreKeptPairAsync, GitLab probe deadline / retry / refusal classification, typed-write resend matrix, pinned deadlines and lock key)
  • Integration: sweep deadlines against a server that never answers (probe deadline, pass budget, token-ignoring and thread-blocking probes, overlap skip); reconnect beside a refused credential over all three connect paths plus sweep and census shapes; two-pod kept pair; writer lifts an Error; pusher standing-unavailable refusals; reverted client edit heals on use; add-time expiry (Pat, group token, offline, silent, refused)
  • E2E (Surface=Http, 414 passed): a token added through the credentials API reads expiring at once; the three tests that asserted no provider traffic after seeding now count from after the seed
  • Each fix was written red-first and mutation-checked (removing the fix turns a test red)
  • Staging: watch credential-health-probe for "ran out of its budget" and "previous pass is still running" warnings after deploy
  • Follow-up (not in this change): show Error credentials in the Providers dialog and render the repository's credential health in the repository list

@ppXD
ppXD force-pushed the fix/know-when-a-provider-credential-stops-working branch from b58f9d8 to acfeec3 Compare October 18, 2026 12:07
@ppXD
ppXD force-pushed the fix/send-gitlab-the-right-auth-on-the-right-host branch from 0afe394 to cd25e9d Compare October 18, 2026 12:09
@ppXD
ppXD changed the base branch from fix/send-gitlab-the-right-auth-on-the-right-host to main October 18, 2026 12:09
A credential whose provider had refused it for good stayed Active with
nothing to say why: a dead OAuth refresh token was re-presented by every
run and webhook and failed as a string, an expired or revoked GitLab
token read as Connected, a 401 on an access token GitLab had retired
early was never recovered, and a failed members lookup refused the
author's own merge request as if they were a stranger.

Credential health is now carried by the existing columns. A refresh the
provider refuses for the grant or the client (invalid_grant and its
kin, not a 5xx or a malformed answer) marks the credential Error with
"needs reconnect: <error>" over the row it judged, and fails with the
typed credential_needs_reconnect naming it; the resolver fails fast on an
Error credential and a successful refresh or probe lifts it. A 401 to an
OAuth credential is sent once more with the token another caller already
stored, or after one rotation (not again within a minute), by NGitLab
and every raw call; a write through the typed client that opens a call is
repeated by the provider once the refusal has put the replacement on the
credential. A recurring probe asks each provider about a bounded batch
of credentials: it validates what GitLab accepts and records a static
token's real expiry (also when a token is linked or added), marks only a
definite 401, and re-asks an Error credential after an hour so it can
heal. The members lookup throws instead of answering Unknown, retries a
429 or 5xx once, and admission refuses with credential_rejected or
author_standing_unknown naming the status and the scope needed.
CredentialSummary gains a derived health and the days to expiry.

A provider that stops answering cannot hold the sweep: the identity call
is asynchronous and honours its deadline, a probe that ignores its token
is walked away from at the probe deadline, a pass ends at a budget under
the tick interval, and a tick that finds the previous pass still running
skips. Reconnecting beside a credential its provider refused, by OAuth,
token link or the credentials API, carries its repositories and group
hooks to the new credential as a disconnect's reconnect does, and the
repair sweep applies the same rule to what an older pod left behind. A
pair a process kept because it could not store it is stored before the
fail-fast applies, so another process's refusal of the consumed refresh
token cannot strand it. An OAuth credential marked Error is asked again
on use once an hour has passed, so a grant that heals recovers without
waiting for the sweep.

No migration: status Error and last_error already exist.
@ppXD
ppXD force-pushed the fix/know-when-a-provider-credential-stops-working branch from acfeec3 to 9f5155d Compare October 18, 2026 12:10
@ppXD
ppXD merged commit 9932741 into main Oct 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant