diff --git a/SKILL.md b/SKILL.md index 9b656c3..4903be5 100644 --- a/SKILL.md +++ b/SKILL.md @@ -170,6 +170,7 @@ Find the row that matches the user's symptom. Start the investigation at the fir | `NondeterminismError`, repeating `WorkflowTaskFailed` | Non-determinism | Identify the last `WorkflowTaskFailed` cause in the Event History | [non-determinism.md#the-wft-failure-signature-of-non-determinism](references/triage/non-determinism.md#the-wft-failure-signature-of-non-determinism) | | Replay fails locally but prod workflow was running | Non-determinism | Fetch the history and run the SDK replayer in one test | [replay.md#step-2--run-the-sdk-replayer-all-supported-sdks](references/triage/replay.md#step-2--run-the-sdk-replayer-all-supported-sdks) | | HA failover did not route traffic to failover region | HA failover | `tcld namespace get --namespace .` vs. DNS CNAME | [ha-failover.md#start-here-establish-ground-truth](references/triage/ha-failover.md#start-here-establish-ground-truth) | +| Serverless Worker (AWS Lambda) stopped processing work after a Namespace failover | HA failover | Confirm a `FailoverNamespace` audit event, then compare the new active region against the Lambda ARN on the Worker Deployment Version | [ha-failover.md#symptom-serverless-workers-kept-running-in-the-old-region-after-failover](references/triage/ha-failover.md#symptom-serverless-workers-kept-running-in-the-old-region-after-failover) | | `context deadline exceeded` (unknown layer) | Runtime errors | Identify which operation and SDK emitted it | [runtime-errors.md#deadline-exceeded](references/triage/runtime-errors.md#deadline-exceeded) | | `Workflow is busy` / `ResourceExhausted` on signal/update/query to one Workflow (BusyWorkflow) | Runtime errors | Rule out account-limit throttling, then split `temporal_cloud_v1_resource_exhausted_error_count` by `operation` | [runtime-errors.md#workflow-lock-contention-busyworkflow](references/triage/runtime-errors.md#workflow-lock-contention-busyworkflow) | | `PAYLOADS_TOO_LARGE`, `exceeds size limit`, payload/gRPC blob size error | Blob size limits | Check whether the issue is payload (2 MB) or gRPC message (4 MB) | [blob-size-limits.md](references/triage/blob-size-limits.md) | @@ -283,7 +284,7 @@ If the layer above the fix is still failing, return to step 4 and continue walki - [non-determinism.md](references/triage/non-determinism.md) — determinism definition, WFT-failure signature, ND-inducing code patterns, per-SDK error shapes, identifying ND from Event History, local replay reproduction, remediation via Worker Versioning / patching / reset. - [worker-health.md](references/triage/worker-health.md) — no-pollers runbook via `temporal task-queue describe`, reachability and versioning, worker-level describe, schedule-to-start latency, worker task slots, sticky execution and sticky cache, worker heartbeating, Cloud namespace-level poller limits, worker log signatures. - [rate-limits.md](references/triage/rate-limits.md) — what `RESOURCE_EXHAUSTED` means (and does not), Cloud APS / RPS / OPS under On-Demand and Provisioned capacity modes, self-hosted `frontend.rps` / `frontend.namespaceRPS` dynamic config, identifying which limit fired via the throttle metrics (Cloud v1) or the `resource_exhausted_cause` label (v0 / self-hosted), and separating account-limit throttling from single-resource exhaustion. -- [ha-failover.md](references/triage/ha-failover.md) — Cloud HA routing via the Namespace Endpoint CNAME, verifying the active region (control-plane `tcld namespace get` vs. DNS view), clients that did not follow the failover, PrivateLink after failover, failover-not-executing, handover-window errors, platform limits, RPO/RTO semantics. +- [ha-failover.md](references/triage/ha-failover.md) — Cloud HA routing via the Namespace Endpoint CNAME, verifying the active region (control-plane `tcld namespace get` vs. DNS view), clients that did not follow the failover, PrivateLink after failover, failover-not-executing, handover-window errors, platform limits, RPO/RTO semantics, and Serverless Workers (AWS Lambda) not following a failover because compute-provider configuration is region-scoped. - [runtime-errors.md](references/triage/runtime-errors.md) — deadline-exceeded disambiguated by operation and by where the call was made, Workflow lock contention (BusyWorkflow) separated from account-limit throttling and confirmed via the `operation` breakdown, routing for `no pollers` / `INVALID_ARGUMENT` / unspecified `UNAVAILABLE`. - [replay.md](references/triage/replay.md) — fetching Event History with the SDK client (CLI export as fallback), running the SDK replayer in every supported SDK (Go, Python, TypeScript, Java, .NET, Ruby, PHP), `TEMPORAL_DEBUG` and the deadlock detector, interpreting divergent and successful replays, and the TypeScript-only VS Code extension. - [blob-size-limits.md](references/triage/blob-size-limits.md) — Payload size limit (2 MB) and gRPC message size limit (4 MB): error messages, per-SDK behavior (Python 1.23.0+ vs. others), claim check pattern, External Storage (Pre-release), batch-size reduction. diff --git a/references/triage/ha-failover.md b/references/triage/ha-failover.md index cf4802a..9bbcc55 100644 --- a/references/triage/ha-failover.md +++ b/references/triage/ha-failover.md @@ -15,6 +15,7 @@ Out of scope here: - [Start here: establish ground truth](#start-here-establish-ground-truth) - [How Cloud HA routing works (minimum needed for triage)](#how-cloud-ha-routing-works-minimum-needed-for-triage) - [Symptom: clients did not follow the failover](#symptom-clients-did-not-follow-the-failover) +- [Symptom: Serverless Workers kept running in the old region after failover](#symptom-serverless-workers-kept-running-in-the-old-region-after-failover) - [Symptom: PrivateLink / PSC stopped working after failover](#symptom-privatelink--psc-stopped-working-after-failover) - [Symptom: failover was requested but never happened](#symptom-failover-was-requested-but-never-happened) - [Symptom: Workflows are rejected during handover](#symptom-workflows-are-rejected-during-handover) @@ -76,6 +77,7 @@ Two timing facts the triage logic depends on: 3. **Application-level address caching.** A caller that resolved the hostname to an IP at startup and reused it won't follow a CNAME swap. Pass the hostname to the client config, never a pre-resolved IP. 4. **GCP Private Service Connect.** PSC has no DNS-based automatic failover — workers must be manually repointed to the new region's PSC endpoint. See [PrivateLink / PSC stopped working](#symptom-privatelink--psc-stopped-working-after-failover). 5. **Private DNS override covers only one region.** Same section. +6. **Serverless Workers (AWS Lambda).** The Worker Controller Instance keeps invoking Workers in the compute provider's originally configured region because compute-provider configuration is region-scoped and the WCI has no failover-detection mechanism. This is a distinct failure mode from long-lived Worker DNS caching — see [Serverless Workers kept running in the old region](#symptom-serverless-workers-kept-running-in-the-old-region-after-failover). **Fix:** clear/await the offending cache, restart wedged workers, or repoint PSC workers per the discriminator that matched. @@ -88,6 +90,20 @@ dig +short ..tmprl.cloud Then re-run the operation that was failing. +## Symptom: Serverless Workers kept running in the old region after failover + +**Symptom:** the Namespace failed over successfully, but Serverless Workers (AWS Lambda, Public Preview) are still being invoked in the old region. Silent while that region is healthy; degraded throughput, latency, or a stall once it is not. + +Nothing in your infrastructure polls, so there is no DNS to re-resolve. The Worker Controller Instance invokes the compute provider configured on a Worker Deployment Version, that configuration is scoped to a single region (for example, a Lambda ARN), and the WCI has no mechanism to detect a failover or redirect invocations into the new active region. Applies to Multi-region and Multi-cloud Replication alike. See `/cloud/high-availability#serverless-workers` and the High Availability row of `/serverless-workers#constraints`. + +**Discriminate:** confirm the new active region (`tcld namespace get --namespace .` plus the `FailoverNamespace` audit entry) and compare it against the Lambda ARN on the Version serving the affected Task Queue. Long-lived Workers on other Task Queues recover on their own, so a mixed fleet recovers partially — which reads like a regional outage rather than a configuration constraint. + +**Fix:** `tcld namespace failover` moves the Namespace only, and tcld has no compute-provider surface. Remediation is to repoint the existing Worker Deployment Version's compute provider at a function in the new active region — an in-place update, not a new Version. Hand it to `skill-temporal-serverless`; it changes where production Workers are invoked, so propose it before running. + +**Prevent:** publish the function in every region the Namespace can fail over to, so the repoint is a single command instead of a provisioning exercise under time pressure. + +**Verify:** the affected Task Queue drains, and invocations land on the new region's function in the provider's logs. + ## Symptom: PrivateLink / PSC stopped working after failover **Symptom:** pre-failover the Namespace was reachable via a private VPC Endpoint; post-failover DNS resolves the Namespace Endpoint to a public IP the VPC can't reach, or to nothing.