Intermittent IDP authentication failure: Intent has not succeeded (IDP-nme4gszsvx)

[scotwells]

Problem

Users are intermittently experiencing authentication failures when using external identity providers (Google, GitHub, etc.) with the error:

ConnectError [failed_precondition] Intent has not succeeded (IDP-nme4gszsvx)

Root Cause Analysis

Primary Cause: Zitadel Multi-Replica Race Condition

The error originates from Zitadel's backend (internal/api/grpc/user/v2/intent.go) when retrieveIdentityProviderIntent is called before the intent has transitioned to the Succeeded state.

Known Issue: zitadel/zitadel#10932

When running multiple Zitadel replicas:

1. User completes IDP authentication → callback processed by Replica A
2. Intent transitions to Succeeded state in event store
3. auth-ui calls RetrieveIdentityProviderIntent which hits Replica B
4. Replica B hasn't synced the event yet
5. Intent appears as Started → Error thrown

Fixed in: Zitadel v4.6.2 (PR #11014)

Secondary Cause: Missing Error Handling in auth-ui

In apps/login/src/app/(main)/(boxed)/idp/[provider]/success/page.tsx at line 149:

const intent = await retrieveIDPIntent({
  serviceUrl,
  id,
  token,
});

This call has no try-catch block. When Zitadel returns the failed_precondition error, it propagates as an unhandled exception.

Proposed Solution

Short-term Fix (auth-ui)

1. Add error handling around retrieveIDPIntent() call
2. Implement retry logic with exponential backoff for the race condition
3. Provide user-friendly error message with retry option

let intent;
try {
  intent = await retrieveIDPIntentWithRetry({
    serviceUrl,
    id,
    token,
  });
} catch (error) {
  Sentry.captureException(error, {
    tags: { flow: 'idp_intent_retrieval', provider },
    extra: { intentId: id },
  });
  
  if (error?.message?.includes('IDP-nme4gszsvx')) {
    return loginFailed("Authentication is still processing. Please try again.");
  }
  
  return loginFailed("Authentication failed. Please try again.");
}

Long-term Fix

Upgrade Zitadel from v3.3.2 to v4.6.2+ which contains the race condition fix.

Affected Files

File	Lines	Issue
apps/login/src/app/(main)/(boxed)/idp/[provider]/success/page.tsx	149-153	Missing error handling
apps/login/src/lib/zitadel.ts	1329-1347	retrieveIDPIntent() - no retry logic

Related Issues

• zitadel/zitadel#10932 - Race conditions when running multiple Zitadel replicas
• zitadel/zitadel#11014 - Fix: locking behavior based on configuration

Labels

• bug
• authentication
• idp

[scotwells]

Zitadel Upgrade Analysis: v3.3.2 → v4.12.0

Current State

Component	Version	Location
Zitadel Server	v3.3.2	infra/apps/datum-iam-system/base/zitadel/helm-release.yaml:32
Helm Chart	v8.x	Same file
@zitadel/client	1.2.0	apps/login/package.json
@zitadel/proto	1.2.0	packages/zitadel-proto/package.json
zitadel-go SDK	v3.13.0	auth-provider-zitadel/go.mod
auth-provider-zitadel	v0.7.2 (prod)	infra/.../production/patches/

Deployment Configuration

• Replicas: 3 (production/staging), 1 (preview) — multi-replica is the root cause of the race condition
• Database: PostgreSQL (local in staging, Cloud SQL in production)
• TLS: Enabled with cert-manager
• IDPs: GitHub and Google OAuth configured

---

Upgrade Path

Version Milestones

Version	Key Changes	Impact
v3.4.x	Minor fixes	Low risk
v4.0.0	Login V2 default, Connect RPC required	Medium risk
v4.6.2	Race condition fix (PR #11014)	Fixes this IDP issue
v4.12.0	Token exchange, back-channel logout GA	Latest stable

Breaking Changes Assessment

1. API Protocol Change (v4.0.0)

• v1: REST/OpenAPI supported
• v4: Only gRPC or Connect RPC for v2 APIs
• Status: ✅ COMPATIBLE — auth-ui already uses @connectrpc/connect v2.0.0

2. Contextual Information Location (v4.0.0)

• v1: Headers (x-zitadel-orgid: 123)
• v4: Request body ({"organization_id": "123"})
• Status: ✅ COMPATIBLE — auth-ui uses v2 APIs which pass organization in request body

3. Login V2 Required (v4.0.0)

• Status: ✅ COMPATIBLE — auth-ui IS the Login V2 implementation using Session API

4. Helm Chart v9+ Changes

• Login V2 deployment is now default
• Must create login client user secret for existing installations
• Status: ⚠️ REQUIRES CONFIGURATION

---

Impact by Repository

auth-ui

Component	Compatibility	Notes
@zitadel/client	✅ Compatible	Uses Connect RPC
v2 APIs (Session, User, OIDC)	✅ Compatible	Primary API version
U2F methods	⚠️ Legacy	Consider migrating to WebAuthn-only
Error handling	❌ Needs fix	Missing try-catch on retrieveIDPIntent

auth-provider-zitadel

Component	Compatibility	Notes
zitadel-go SDK v3.13.0	✅ Compatible	Latest SDK version
gRPC Session/User Service v2	✅ Compatible	Already v2
HTTP Management v1 APIs	⚠️ Review	create_machine_user.go uses /management/v1/

infra

Component	Change Required	Notes
Helm release version	✅ Update	v8.x → v9.x
Zitadel image tag	✅ Update	v3.3.2 → v4.12.0
Login secrets	⚠️ Configure	New secret for Login V2 client
Database	✅ Compatible	PostgreSQL supported

---

Recommended Upgrade Plan

Phase 1: Immediate Fixes (No Upgrade Required)

1. Fix IDP intent error handling in apps/login/src/app/(main)/(boxed)/idp/[provider]/success/page.tsx:

   let intent;
   try {
     intent = await retrieveIDPIntentWithRetry({
       serviceUrl,
       id,
       token,
     }, 3, 500); // 3 retries, 500ms backoff
   } catch (error) {
     Sentry.captureException(error, {
       tags: { flow: 'idp_intent_retrieval', provider },
       extra: { intentId: id },
     });
     
     if (error?.message?.includes('IDP-nme4gszsvx')) {
       return loginFailed("Authentication is still processing. Please try again.");
     }
     return loginFailed("Authentication failed. Please try again.");
   }

2. Add retry helper function:

   async function retrieveIDPIntentWithRetry(
     params: { serviceUrl: string; id: string; token: string },
     maxRetries = 3,
     delayMs = 500
   ) {
     for (let attempt = 1; attempt <= maxRetries; attempt++) {
       try {
         return await retrieveIDPIntent(params);
       } catch (error) {
         if (error?.message?.includes('IDP-nme4gszsvx') && attempt < maxRetries) {
           await new Promise(resolve => setTimeout(resolve, delayMs * attempt));
           continue;
         }
         throw error;
       }
     }
   }

Phase 2: Staging Upgrade

1. Update Helm chart to v9.x
2. Update Zitadel image to v4.12.0
3. Configure Login V2 settings:

   ZITADEL_DEFAULTINSTANCE_FEATURES_LOGINV2_REQUIRED: true
   ZITADEL_DEFAULTINSTANCE_FEATURES_LOGINV2_BASEURI: "https://auth.staging.env.datum.net/ui/v2/login"

4. Test IDP flows with multiple replicas

Phase 3: Production Upgrade

1. Backup database
2. Plan maintenance window (setup phase may take several minutes)
3. Deploy v4.12.0
4. Monitor for API errors

---

Files to Update

File	Change
infra/apps/datum-iam-system/base/zitadel/helm-release.yaml	Update chart version to 9.x, image tag to v4.12.0
infra/apps/datum-iam-system/base/zitadel/helm-release.yaml	Add Login V2 feature flags
apps/login/src/app/(main)/(boxed)/idp/[provider]/success/page.tsx	Add error handling + retry logic
auth-provider-zitadel/internal/zitadel/create_machine_user.go	Review v1 API usage for deprecation

---

Risk Assessment

Risk	Severity	Mitigation
Database migration failure	High	Backup before upgrade
v1 API deprecation in auth-provider-zitadel	Medium	Review HTTP client usage
Helm chart breaking changes	Medium	Test in staging first
Passkey domain binding	Low	Not changing domains
IDP race condition	Resolved	Fixed in v4.6.2

---

References

• Zitadel v4.6.2 Release
• Race Condition Fix PR #11014
• Race Condition Issue #10932
• v1 to v2 API Migration Guide
• Zitadel Helm Chart Upgrade Guide

[scotwells]

auth-provider-zitadel Compatibility

Result: ✅ Compatible with v4.12.0

API Status

• 10/11 endpoints use v2 APIs — fully compatible
• 1 endpoint uses v1 API — deprecated but functional in v4.x

Action Required

Before Zitadel v5: Migrate machine user creation from /management/v1/users/machine to /v2/users

File: auth-provider-zitadel/internal/zitadel/create_machine_user.go

Authentication

JWT Profile, token introspection, and zitadel-go/v3 SDK are all fully compatible.

Recommendation

Safe to upgrade to v4.12.0 now. Plan v1→v2 migration before Zitadel v5 release.

[scotwells]

Zitadel v4.12.0 Testing Update

Successfully tested Zitadel v4.12.0 in a fresh preview environment (preview-pr-1813). The login flow works correctly with the new version.

Configuration Changes Required

The following configuration changes were needed for v4.12.0:

1. Login v2 image tag - Updated to match Zitadel version:

   login:
     image:
       tag: v4.12.0

2. Trust bundle for login container - Required for internal TLS:

   login:
     extraVolumes:
       - name: trust-bundle
         configMap:
           name: datum-control-plane-trust-bundle
     extraVolumeMounts:
       - name: trust-bundle
         mountPath: /etc/ssl/certs/ca.crt
         subPath: ca.crt
     env:
       - name: NODE_EXTRA_CA_CERTS
         value: /etc/ssl/certs/ca.crt

3. Trusted IPs for internal communication - Required for login pod to communicate with Zitadel:

   zitadel:
     configmapConfig:
       HTTP:
         Forwarded:
           TrustedIPs:
             - "10.0.0.0/8"
             - "172.16.0.0/12"
             - "192.168.0.0/16"

4. Executions DenyList - Required for actions server (already had Actions.HTTP.DenyList):

   Executions:
     DenyList:
       - "0.0.0.0"
       - "::"

Next Step: Test Upgrade Path

The fresh install works, but we need to verify the upgrade path from v3.3.2 → v4.12.0 simulates how this will roll out to production.

Plan:

1. Delete the current preview namespace
2. Create preview pointing to main branch (deploys v3.3.2)
3. Wait for full initialization with existing data/schema
4. Update GitRepository to point to test/zitadel-v4-upgrade branch
5. Observe the upgrade process and verify functionality

This will ensure the database migrations and configuration changes work correctly during an in-place upgrade.

[scotwells]

May have encountered another race condition with our current version of Zitadel.

[not_found] Auth Request does not exist (QUERY-Thee9)

[scotwells]

Upgrade Path Test Successful ✅

Successfully tested the upgrade path from v3.3.2 → v4.12.0 in preview environment.

Test Procedure

1. Created fresh preview with main branch (deployed Zitadel v3.3.2, chart 8.13.4)
2. Switched GitRepository to test/zitadel-v4-upgrade branch
3. Observed upgrade to v4.12.0 (chart 9.25.0)
4. Verified authentication flow works post-upgrade

Additional Fix Required

During upgrade testing, discovered the Pulumi job wasn't creating the login-client secret expected by the Helm chart. Fixed in commit 8939b4f:

// Create login-client secret expected by the Zitadel Helm chart's login v2 deployment
new k8s.core.v1.Secret("login-client", {
  metadata: {
    name: "login-client",
    namespace: config.namespace,
  },
  stringData: {
    pat: datumLoginPersonalAccessToken.token,
  },
});

Summary of All Changes

Change	Purpose
login.image.tag: v4.12.0	Match login UI version to Zitadel
login.extraVolumes + NODE_EXTRA_CA_CERTS	Trust internal TLS certificates
HTTP.Forwarded.TrustedIPs	Trust internal pod IPs for requests
Executions.DenyList	Allow localhost for actions server
login-client secret creation	Helm chart compatibility

Ready for Production

The upgrade path is validated and ready for production rollout.

[scotwells]

Upgraded to v4.12.2 in production.

[drewr]

Upgrade didn't fix the issue. Waiting on upstream fix.