From c3bef540c9b303e8feca61d2c54b8341de1664dd Mon Sep 17 00:00:00 2001 From: "skill-sync[bot]" Date: Fri, 7 Aug 2026 20:08:53 +0000 Subject: [PATCH 1/2] Finalize draft for 0002-serverless-ha --- SKILL.md | 2 + references/ops/cloud-ops-api.md | 2 +- references/triage/connectivity.md | 4 +- references/triage/ha-failover.md | 1 + references/triage/serverless-ha.md | 144 +++++++++++++++++++++++++++++ 5 files changed, 150 insertions(+), 3 deletions(-) create mode 100644 references/triage/serverless-ha.md diff --git a/SKILL.md b/SKILL.md index 9b656c3..66d24b9 100644 --- a/SKILL.md +++ b/SKILL.md @@ -170,6 +170,7 @@ Find the row that matches the user's symptom. Start the investigation at the fir | `NondeterminismError`, repeating `WorkflowTaskFailed` | Non-determinism | Identify the last `WorkflowTaskFailed` cause in the Event History | [non-determinism.md#the-wft-failure-signature-of-non-determinism](references/triage/non-determinism.md#the-wft-failure-signature-of-non-determinism) | | Replay fails locally but prod workflow was running | Non-determinism | Fetch the history and run the SDK replayer in one test | [replay.md#step-2--run-the-sdk-replayer-all-supported-sdks](references/triage/replay.md#step-2--run-the-sdk-replayer-all-supported-sdks) | | HA failover did not route traffic to failover region | HA failover | `tcld namespace get --namespace .` vs. DNS CNAME | [ha-failover.md#start-here-establish-ground-truth](references/triage/ha-failover.md#start-here-establish-ground-truth) | +| Serverless Worker (AWS Lambda / GCP Cloud Run) stopped processing work after a Namespace failover | Serverless HA | Confirm a `FailoverNamespace` audit event, then check whether the Worker Deployment Version's compute provider is in the new active region | [serverless-ha.md#diagnostic-signal--did-a-failover-happen](references/triage/serverless-ha.md#diagnostic-signal--did-a-failover-happen) | | `context deadline exceeded` (unknown layer) | Runtime errors | Identify which operation and SDK emitted it | [runtime-errors.md#deadline-exceeded](references/triage/runtime-errors.md#deadline-exceeded) | | `Workflow is busy` / `ResourceExhausted` on signal/update/query to one Workflow (BusyWorkflow) | Runtime errors | Rule out account-limit throttling, then split `temporal_cloud_v1_resource_exhausted_error_count` by `operation` | [runtime-errors.md#workflow-lock-contention-busyworkflow](references/triage/runtime-errors.md#workflow-lock-contention-busyworkflow) | | `PAYLOADS_TOO_LARGE`, `exceeds size limit`, payload/gRPC blob size error | Blob size limits | Check whether the issue is payload (2 MB) or gRPC message (4 MB) | [blob-size-limits.md](references/triage/blob-size-limits.md) | @@ -284,6 +285,7 @@ If the layer above the fix is still failing, return to step 4 and continue walki - [worker-health.md](references/triage/worker-health.md) — no-pollers runbook via `temporal task-queue describe`, reachability and versioning, worker-level describe, schedule-to-start latency, worker task slots, sticky execution and sticky cache, worker heartbeating, Cloud namespace-level poller limits, worker log signatures. - [rate-limits.md](references/triage/rate-limits.md) — what `RESOURCE_EXHAUSTED` means (and does not), Cloud APS / RPS / OPS under On-Demand and Provisioned capacity modes, self-hosted `frontend.rps` / `frontend.namespaceRPS` dynamic config, identifying which limit fired via the throttle metrics (Cloud v1) or the `resource_exhausted_cause` label (v0 / self-hosted), and separating account-limit throttling from single-resource exhaustion. - [ha-failover.md](references/triage/ha-failover.md) — Cloud HA routing via the Namespace Endpoint CNAME, verifying the active region (control-plane `tcld namespace get` vs. DNS view), clients that did not follow the failover, PrivateLink after failover, failover-not-executing, handover-window errors, platform limits, RPO/RTO semantics. +- [serverless-ha.md](references/triage/serverless-ha.md) — Serverless Workers and Namespace failover: the WCI has no cross-region redirect and compute providers are region-scoped, so after a Multi-region or Multi-cloud Replication failover the operator must create a new Worker Deployment Version pointing at the new region's Lambda ARN or Cloud Run Worker Pool and promote it with `set-current-version`. Public Preview / Pre-release. - [runtime-errors.md](references/triage/runtime-errors.md) — deadline-exceeded disambiguated by operation and by where the call was made, Workflow lock contention (BusyWorkflow) separated from account-limit throttling and confirmed via the `operation` breakdown, routing for `no pollers` / `INVALID_ARGUMENT` / unspecified `UNAVAILABLE`. - [replay.md](references/triage/replay.md) — fetching Event History with the SDK client (CLI export as fallback), running the SDK replayer in every supported SDK (Go, Python, TypeScript, Java, .NET, Ruby, PHP), `TEMPORAL_DEBUG` and the deadlock detector, interpreting divergent and successful replays, and the TypeScript-only VS Code extension. - [blob-size-limits.md](references/triage/blob-size-limits.md) — Payload size limit (2 MB) and gRPC message size limit (4 MB): error messages, per-SDK behavior (Python 1.23.0+ vs. others), claim check pattern, External Storage (Pre-release), batch-size reduction. diff --git a/references/ops/cloud-ops-api.md b/references/ops/cloud-ops-api.md index 9fa520e..325a978 100644 --- a/references/ops/cloud-ops-api.md +++ b/references/ops/cloud-ops-api.md @@ -66,7 +66,7 @@ import ( ) ``` -Go samples: [github.com/temporalio/cloud-samples-go](https://github.com/temporalio/cloud-samples-go) +Go samples: [github.com/temporalio/cloud-samples-go](https://github.com/temporalio/cloud-samples-go) Cloud Ops API client setup: [client/api/client.go](https://github.com/temporalio/cloud-samples-go/blob/main/client/api/client.go) --- diff --git a/references/triage/connectivity.md b/references/triage/connectivity.md index ff191d7..eed291e 100644 --- a/references/triage/connectivity.md +++ b/references/triage/connectivity.md @@ -65,7 +65,7 @@ Interpreting results: ## Endpoint formats -Using the wrong endpoint family is one of the most common causes of "cannot connect" reports. +Using the wrong endpoint family is one of the most common causes of "cannot connect" reports. | Purpose | Endpoint pattern | Port | Source | |---|---|---|---| @@ -134,7 +134,7 @@ nc -zvw10 vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com 7233 ## Quick diagnostic scripts -Run from the failing environment (the pod, container, or host where the problem reproduces). These scripts chain the layer-1/2 checks with a final `temporal` call to confirm the whole stack end-to-end. +Run from the failing environment (the pod, container, or host where the problem reproduces). These scripts chain the layer-1/2 checks with a final `temporal` call to confirm the whole stack end-to-end. ### mTLS variant diff --git a/references/triage/ha-failover.md b/references/triage/ha-failover.md index cf4802a..f1f5da2 100644 --- a/references/triage/ha-failover.md +++ b/references/triage/ha-failover.md @@ -76,6 +76,7 @@ Two timing facts the triage logic depends on: 3. **Application-level address caching.** A caller that resolved the hostname to an IP at startup and reused it won't follow a CNAME swap. Pass the hostname to the client config, never a pre-resolved IP. 4. **GCP Private Service Connect.** PSC has no DNS-based automatic failover — workers must be manually repointed to the new region's PSC endpoint. See [PrivateLink / PSC stopped working](#symptom-privatelink--psc-stopped-working-after-failover). 5. **Private DNS override covers only one region.** Same section. +6. **Serverless Workers (AWS Lambda / GCP Cloud Run).** The Worker Controller Instance keeps invoking Workers in the compute provider's originally configured region because compute-provider configuration is region-scoped and the WCI has no failover-detection mechanism. This is a distinct failure mode from long-lived Worker DNS caching — see [`serverless-ha.md`](serverless-ha.md). **Fix:** clear/await the offending cache, restart wedged workers, or repoint PSC workers per the discriminator that matched. diff --git a/references/triage/serverless-ha.md b/references/triage/serverless-ha.md new file mode 100644 index 0000000..65f8938 --- /dev/null +++ b/references/triage/serverless-ha.md @@ -0,0 +1,144 @@ +# Serverless HA + +> [!NOTE] +> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. + +AWS Lambda support for Serverless Workers is in Public Preview; GCP Cloud Run support is in Pre-release. Serverless Workers *can* be used with a Namespace that has Multi-region or Multi-cloud Replication, but they do **not** follow a Namespace failover automatically. When the Namespace endpoint re-routes to the new active region, the Worker Controller Instance (WCI) keeps invoking Workers in the compute provider's originally configured region. Without manual intervention, workloads that failed over will continue to be executed by Workers in the old region — and if that region is degraded, your workloads are degraded with it. + +This file explains why the constraint exists and what an operator must do after a failover to keep processing Workflows. + +## Table of contents + +- [The constraint in one sentence](#the-constraint-in-one-sentence) +- [Why this happens](#why-this-happens) +- [Diagnostic signal — did a failover happen?](#diagnostic-signal--did-a-failover-happen) +- [Manual remediation](#manual-remediation) + - [AWS Lambda](#aws-lambda) + - [GCP Cloud Run](#gcp-cloud-run) +- [Prevention: dual-region compute-provider pre-provisioning](#prevention-dual-region-compute-provider-pre-provisioning) +- [Hand-off pointers](#hand-off-pointers) + +## The constraint in one sentence + +> On failover of a Namespace with Multi-region or Multi-cloud Replication, the WCI keeps invoking Workers in the original region unless you manually repoint the compute provider. Compute provider configuration, such as a Lambda ARN or a Cloud Run Worker Pool, is scoped to a single region. + +The canonical statement in the HA docs is equivalent: "each compute provider configuration (for example, a Lambda ARN) is scoped to a single region. The Worker Controller Instance (WCI) has no mechanism to detect a failover or redirect invocations to a Worker in the new active region. To keep processing Workflows after a failover, you must manually reconfigure the Worker Deployment Version's compute provider to point at a Worker in the new region." + +See the full docs entry at [https://docs.temporal.io/cloud/high-availability#serverless-workers](https://docs.temporal.io/cloud/high-availability#serverless-workers) and the Constraints table at [https://docs.temporal.io/serverless-workers#constraints](https://docs.temporal.io/serverless-workers#constraints). + +## Why this happens + +Two independent facts combine to produce the constraint: + +1. **Compute provider configuration is per-Worker-Deployment-Version and pins a region.** The compute provider "is set on a Worker Deployment Version and specifies the provider type, the invocation target, and the credentials Temporal needs to trigger the invocation." For AWS Lambda, the invocation target is a Lambda function ARN, which embeds the AWS region. For GCP Cloud Run, the target is a Worker Pool whose region is set explicitly by `--gcp-cloud-run-region`. + +2. **The WCI has no failover-detection or cross-region redirect.** "One WCI Workflow runs per Worker Deployment Version that has a compute provider configured. The WCI runs in the same Namespace as your Worker Deployment." The WCI's role is to scale Workers up and down in response to Task Queue conditions by triggering the configured compute provider. It has no branch that swaps compute provider region on failover; the HA docs are explicit that it "has no mechanism to detect a failover or redirect invocations to a Worker in the new active region." + +Contrast with long-lived Workers: for a long-lived fleet, "the DNS redirection orchestrated by Temporal ensures that your existing Workers continue to poll the Namespace without interruption. Temporal Cloud forwards their requests from the passive replica to the active region and the responses back, so Workers keep running through a failover." That mechanism operates at the *Client-to-Namespace* layer: a Worker process resolves the Namespace Endpoint and re-resolves when a connection cycles. Serverless Workers do not have a persistent poller in your infrastructure that could re-resolve DNS; each invocation is created *by Temporal* against the compute provider you configured, and that configuration is regional. + +For the DNS-based failover mechanics that long-lived Workers rely on, see [`references/triage/ha-failover.md`](./ha-failover.md). Do not re-derive that here. + +## Diagnostic signal — did a failover happen? + +You are usually called in because Workflows are stalled, latency is up, or a customer reported an outage. Before doing anything provider-specific, confirm that a Namespace failover actually occurred and note the new active region. + +- **Web UI.** "After any failover, whether triggered by you or by Temporal, an event appears in both the [Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs." Open the Namespace detail page and inspect the failover list. +- **Audit log signal.** "The audit log entry uses the `\"operation\": \"FailoverNamespace\"` event." Temporal Cloud also emails admins on every failover. +- **What long-lived Workers do vs. Serverless Workers.** Long-lived Workers follow the Namespace Endpoint DNS change and, once existing connections cycle (Temporal Cloud enforces a maximum connection lifetime of 5 minutes), re-resolve to the new active region. Serverless Workers do not — the WCI keeps calling the compute provider you configured until you repoint it. If your fleet is mixed, expect long-lived Workers to recover on their own within a few minutes while Serverless-only Task Queues stay wedged until you act. Note also that the Namespace Endpoint DNS change itself "can take a few minutes to fully propagate to all Clients and Workers." + +If a `FailoverNamespace` event has fired and the new active region is not the same region as your compute provider's Lambda ARN or Cloud Run Worker Pool, you are in the constraint. Proceed to remediation. + +## Manual remediation + +`tcld namespace failover --namespace . --region ` fails over the *Namespace*. It does not modify any Worker Deployment Version's compute provider. The documented remediation is to create a new Worker Deployment Version pointing at a Worker in the new active region, then set that Version current. + +The compute provider is configured at Version creation time via `temporal worker deployment create-version` with provider-specific flags. Changing the target region means creating a *new* Version — the two provider flag surfaces are described below. After creating the new Version, promote it with `temporal worker deployment set-current-version`; "without this step, tasks on the Task Queue will not route to the version, and Temporal will not invoke the Lambda function." The equivalent guidance holds for Cloud Run: without setting the version current, "Tasks on the Task Queue will not route to the version, and the WCI will not start any instances." + +### AWS Lambda + +**Prerequisite (AWS side).** A Lambda function must already exist in the new active AWS region, with an IAM trust configuration that lets Temporal assume the invocation role. The full AWS setup is out of scope for this file — hand off to [https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/aws-lambda](https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/aws-lambda) for the deployment guide. + +**Create a new Worker Deployment Version pointing at the new region's Lambda ARN.** The Lambda ARN embeds the region, so pointing at a new-region function is the entire point of the remediation. + +```bash +temporal worker deployment create-version \ + --namespace \ + --deployment-name my-app \ + --build-id build-1- \ + --aws-lambda-function-arn \ + --aws-lambda-assume-role-arn \ + --aws-lambda-assume-role-external-id +``` + +Flags in play (all transcribed from the AWS Lambda doc): + +- `--deployment-name` — Worker Deployment name. Must match `DeploymentName` in your Worker code. +- `--build-id` — Worker Deployment Version build ID. Must match `BuildID` in your Worker code. +- `--aws-lambda-function-arn` — Qualified versioned ARN of the Lambda function Temporal invokes for this version (for example, `function:my-worker:5`). +- `--aws-lambda-assume-role-arn` — IAM role Temporal assumes to invoke the function. +- `--aws-lambda-assume-role-external-id` — External ID configured in the IAM role trust policy. + +**Set the new Version current.** + +```bash +temporal worker deployment set-current-version \ + --deployment-name my-app \ + --build-id build-1- +``` + +### GCP Cloud Run + +**Prerequisite (GCP side).** A Cloud Run Worker Pool must exist in the new active region, in a project Temporal's invoker service account can impersonate. The Cloud Run setup — Worker Pool creation, invoker IAM, Terraform — is out of scope here; hand off to [https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/cloud-run](https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/cloud-run). + +**Create a new Worker Deployment Version with `--gcp-cloud-run-region` set to the new active region and the new pool's name/project/service-account.** + +```bash +temporal worker deployment create-version \ + --namespace \ + --deployment-name my-app \ + --build-id build-1- \ + --gcp-cloud-run-project \ + --gcp-cloud-run-region \ + --gcp-cloud-run-worker-pool \ + --gcp-cloud-run-service-account +``` + +Flags in play (all transcribed from the Cloud Run doc): + +- `--deployment-name` — Worker Deployment name. Must match `deployment_name` in your Worker code. +- `--build-id` — Worker Deployment Version build ID. Must match `build_id` in your Worker code. +- `--gcp-cloud-run-project` — GCP project ID that contains the Worker Pool. +- `--gcp-cloud-run-region` — Region of the Worker Pool. +- `--gcp-cloud-run-worker-pool` — Name of the Worker Pool. +- `--gcp-cloud-run-service-account` — The invoker service account Temporal impersonates to read and scale the pool. + +**Set the new Version current.** + +```bash +temporal worker deployment set-current-version \ + --namespace \ + --deployment-name my-app \ + --build-id build-1- +``` + +The Cloud Run `set-current-version` command asks for confirmation because it changes which version new Tasks route to; pass `--yes` to skip the prompt in an incident. + +## Prevention: dual-region compute-provider pre-provisioning + +The docs establish that a compute provider is per-Version and per-region, and that the mutation verbs available are `create-version` and `set-current-version`. From that alone, the operator-facing implication is clear: if you keep a Worker Deployment Version already configured for each region *in advance*, then failover-time work reduces to a single `set-current-version` call against the pre-existing new-region Version. You skip the AWS-side or GCP-side provisioning under time pressure. + +This means, for each Worker Deployment: + +- Publish the underlying compute artifact (Lambda function, Cloud Run Worker Pool) in every region you might fail over to, before you need it. +- Pre-create one Worker Deployment Version per region using `temporal worker deployment create-version` with that region's provider flags. Use distinct `--build-id` values so each Version is addressable. +- On failover, promote the Version whose compute provider is in the new active region with `temporal worker deployment set-current-version --deployment-name ... --build-id ...`. + +The docs stop short of prescribing a specific automation pattern (Terraform module, runbook script, or CI hook) for this pre-provisioning. Do not invent one on the operator's behalf. The Temporal Cloud Terraform provider does not support triggering failovers — that fact is documented for the Namespace-failover path, not the Worker-Deployment-Version path — so treat any Terraform-provider claim about `create-version`/`set-current-version` as something to look up in the CLI reference rather than assert from this file. + +## Hand-off pointers + +- **Worker placement architecture (Active-Passive vs. Active-Active, latency/cost trade-offs across regions):** see the [`skill-temporal-deploy`](https://docs.temporal.io/cloud/high-availability/architecture-patterns) skill's deployment-pattern references and [https://docs.temporal.io/cloud/high-availability/architecture-patterns](https://docs.temporal.io/cloud/high-availability/architecture-patterns). +- **`tcld` and `temporal` flag semantics in depth (all flags on `namespace failover`, `worker deployment create-version`, `worker deployment set-current-version`):** hand off to the `skill-temporal-cli` skill; that skill owns exhaustive CLI reference and stays authoritative when flag surfaces change. +- **Serverless Worker SDK code (registering Workflows/Activities, using the serverless Worker package for your language SDK, Worker Versioning `AutoUpgrade` vs `Pinned` behavior):** hand off to the `skill-temporal-developer` skill. +- **Long-lived Worker failover behavior (DNS/Namespace Endpoint propagation, the 5-minute connection lifetime, Regional Endpoint escape hatch):** see [`references/triage/ha-failover.md`](./ha-failover.md) in this skill. +- **How to trigger the Namespace failover itself (`tcld`, Web UI, Cloud Ops API):** [https://docs.temporal.io/cloud/high-availability/failovers/manage#trigger-failover](https://docs.temporal.io/cloud/high-availability/failovers/manage#trigger-failover). This file assumes the failover already happened. From 83a9e2bfc67806f59fe3cd0cb4b3c56258758006 Mon Sep 17 00:00:00 2001 From: starfleeth <128422269+starfleeth@users.noreply.github.com> Date: Tue, 11 Aug 2026 16:38:45 -0700 Subject: [PATCH 2/2] docs(ha-failover): fold serverless-HA constraint into ha-failover, tcld only Replace the standalone serverless-ha.md with a symptom section in ha-failover.md, which already owns failover triage. The remediation commands are unified-CLI only, so they are described in prose and handed to skill-temporal-serverless rather than printed here. Corrections against the docs while folding: the fix is an in-place compute-provider repoint on the existing Worker Deployment Version, not a new Version promoted to current; old-region invocations continue after failover and bite only when that region is degraded, rather than stalling the Task Queue outright; drop the pre-create-a-Version-per-region prevention advice, which no doc prescribes. Also revert the whitespace-only edits to cloud-ops-api.md and connectivity.md. Co-Authored-By: Claude Opus 5 (1M context) --- SKILL.md | 5 +- references/ops/cloud-ops-api.md | 2 +- references/triage/connectivity.md | 4 +- references/triage/ha-failover.md | 17 +++- references/triage/serverless-ha.md | 144 ----------------------------- 5 files changed, 21 insertions(+), 151 deletions(-) delete mode 100644 references/triage/serverless-ha.md diff --git a/SKILL.md b/SKILL.md index 66d24b9..4903be5 100644 --- a/SKILL.md +++ b/SKILL.md @@ -170,7 +170,7 @@ Find the row that matches the user's symptom. Start the investigation at the fir | `NondeterminismError`, repeating `WorkflowTaskFailed` | Non-determinism | Identify the last `WorkflowTaskFailed` cause in the Event History | [non-determinism.md#the-wft-failure-signature-of-non-determinism](references/triage/non-determinism.md#the-wft-failure-signature-of-non-determinism) | | Replay fails locally but prod workflow was running | Non-determinism | Fetch the history and run the SDK replayer in one test | [replay.md#step-2--run-the-sdk-replayer-all-supported-sdks](references/triage/replay.md#step-2--run-the-sdk-replayer-all-supported-sdks) | | HA failover did not route traffic to failover region | HA failover | `tcld namespace get --namespace .` vs. DNS CNAME | [ha-failover.md#start-here-establish-ground-truth](references/triage/ha-failover.md#start-here-establish-ground-truth) | -| Serverless Worker (AWS Lambda / GCP Cloud Run) stopped processing work after a Namespace failover | Serverless HA | Confirm a `FailoverNamespace` audit event, then check whether the Worker Deployment Version's compute provider is in the new active region | [serverless-ha.md#diagnostic-signal--did-a-failover-happen](references/triage/serverless-ha.md#diagnostic-signal--did-a-failover-happen) | +| Serverless Worker (AWS Lambda) stopped processing work after a Namespace failover | HA failover | Confirm a `FailoverNamespace` audit event, then compare the new active region against the Lambda ARN on the Worker Deployment Version | [ha-failover.md#symptom-serverless-workers-kept-running-in-the-old-region-after-failover](references/triage/ha-failover.md#symptom-serverless-workers-kept-running-in-the-old-region-after-failover) | | `context deadline exceeded` (unknown layer) | Runtime errors | Identify which operation and SDK emitted it | [runtime-errors.md#deadline-exceeded](references/triage/runtime-errors.md#deadline-exceeded) | | `Workflow is busy` / `ResourceExhausted` on signal/update/query to one Workflow (BusyWorkflow) | Runtime errors | Rule out account-limit throttling, then split `temporal_cloud_v1_resource_exhausted_error_count` by `operation` | [runtime-errors.md#workflow-lock-contention-busyworkflow](references/triage/runtime-errors.md#workflow-lock-contention-busyworkflow) | | `PAYLOADS_TOO_LARGE`, `exceeds size limit`, payload/gRPC blob size error | Blob size limits | Check whether the issue is payload (2 MB) or gRPC message (4 MB) | [blob-size-limits.md](references/triage/blob-size-limits.md) | @@ -284,8 +284,7 @@ If the layer above the fix is still failing, return to step 4 and continue walki - [non-determinism.md](references/triage/non-determinism.md) — determinism definition, WFT-failure signature, ND-inducing code patterns, per-SDK error shapes, identifying ND from Event History, local replay reproduction, remediation via Worker Versioning / patching / reset. - [worker-health.md](references/triage/worker-health.md) — no-pollers runbook via `temporal task-queue describe`, reachability and versioning, worker-level describe, schedule-to-start latency, worker task slots, sticky execution and sticky cache, worker heartbeating, Cloud namespace-level poller limits, worker log signatures. - [rate-limits.md](references/triage/rate-limits.md) — what `RESOURCE_EXHAUSTED` means (and does not), Cloud APS / RPS / OPS under On-Demand and Provisioned capacity modes, self-hosted `frontend.rps` / `frontend.namespaceRPS` dynamic config, identifying which limit fired via the throttle metrics (Cloud v1) or the `resource_exhausted_cause` label (v0 / self-hosted), and separating account-limit throttling from single-resource exhaustion. -- [ha-failover.md](references/triage/ha-failover.md) — Cloud HA routing via the Namespace Endpoint CNAME, verifying the active region (control-plane `tcld namespace get` vs. DNS view), clients that did not follow the failover, PrivateLink after failover, failover-not-executing, handover-window errors, platform limits, RPO/RTO semantics. -- [serverless-ha.md](references/triage/serverless-ha.md) — Serverless Workers and Namespace failover: the WCI has no cross-region redirect and compute providers are region-scoped, so after a Multi-region or Multi-cloud Replication failover the operator must create a new Worker Deployment Version pointing at the new region's Lambda ARN or Cloud Run Worker Pool and promote it with `set-current-version`. Public Preview / Pre-release. +- [ha-failover.md](references/triage/ha-failover.md) — Cloud HA routing via the Namespace Endpoint CNAME, verifying the active region (control-plane `tcld namespace get` vs. DNS view), clients that did not follow the failover, PrivateLink after failover, failover-not-executing, handover-window errors, platform limits, RPO/RTO semantics, and Serverless Workers (AWS Lambda) not following a failover because compute-provider configuration is region-scoped. - [runtime-errors.md](references/triage/runtime-errors.md) — deadline-exceeded disambiguated by operation and by where the call was made, Workflow lock contention (BusyWorkflow) separated from account-limit throttling and confirmed via the `operation` breakdown, routing for `no pollers` / `INVALID_ARGUMENT` / unspecified `UNAVAILABLE`. - [replay.md](references/triage/replay.md) — fetching Event History with the SDK client (CLI export as fallback), running the SDK replayer in every supported SDK (Go, Python, TypeScript, Java, .NET, Ruby, PHP), `TEMPORAL_DEBUG` and the deadlock detector, interpreting divergent and successful replays, and the TypeScript-only VS Code extension. - [blob-size-limits.md](references/triage/blob-size-limits.md) — Payload size limit (2 MB) and gRPC message size limit (4 MB): error messages, per-SDK behavior (Python 1.23.0+ vs. others), claim check pattern, External Storage (Pre-release), batch-size reduction. diff --git a/references/ops/cloud-ops-api.md b/references/ops/cloud-ops-api.md index 325a978..9fa520e 100644 --- a/references/ops/cloud-ops-api.md +++ b/references/ops/cloud-ops-api.md @@ -66,7 +66,7 @@ import ( ) ``` -Go samples: [github.com/temporalio/cloud-samples-go](https://github.com/temporalio/cloud-samples-go) +Go samples: [github.com/temporalio/cloud-samples-go](https://github.com/temporalio/cloud-samples-go) Cloud Ops API client setup: [client/api/client.go](https://github.com/temporalio/cloud-samples-go/blob/main/client/api/client.go) --- diff --git a/references/triage/connectivity.md b/references/triage/connectivity.md index eed291e..ff191d7 100644 --- a/references/triage/connectivity.md +++ b/references/triage/connectivity.md @@ -65,7 +65,7 @@ Interpreting results: ## Endpoint formats -Using the wrong endpoint family is one of the most common causes of "cannot connect" reports. +Using the wrong endpoint family is one of the most common causes of "cannot connect" reports. | Purpose | Endpoint pattern | Port | Source | |---|---|---|---| @@ -134,7 +134,7 @@ nc -zvw10 vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com 7233 ## Quick diagnostic scripts -Run from the failing environment (the pod, container, or host where the problem reproduces). These scripts chain the layer-1/2 checks with a final `temporal` call to confirm the whole stack end-to-end. +Run from the failing environment (the pod, container, or host where the problem reproduces). These scripts chain the layer-1/2 checks with a final `temporal` call to confirm the whole stack end-to-end. ### mTLS variant diff --git a/references/triage/ha-failover.md b/references/triage/ha-failover.md index f1f5da2..9bbcc55 100644 --- a/references/triage/ha-failover.md +++ b/references/triage/ha-failover.md @@ -15,6 +15,7 @@ Out of scope here: - [Start here: establish ground truth](#start-here-establish-ground-truth) - [How Cloud HA routing works (minimum needed for triage)](#how-cloud-ha-routing-works-minimum-needed-for-triage) - [Symptom: clients did not follow the failover](#symptom-clients-did-not-follow-the-failover) +- [Symptom: Serverless Workers kept running in the old region after failover](#symptom-serverless-workers-kept-running-in-the-old-region-after-failover) - [Symptom: PrivateLink / PSC stopped working after failover](#symptom-privatelink--psc-stopped-working-after-failover) - [Symptom: failover was requested but never happened](#symptom-failover-was-requested-but-never-happened) - [Symptom: Workflows are rejected during handover](#symptom-workflows-are-rejected-during-handover) @@ -76,7 +77,7 @@ Two timing facts the triage logic depends on: 3. **Application-level address caching.** A caller that resolved the hostname to an IP at startup and reused it won't follow a CNAME swap. Pass the hostname to the client config, never a pre-resolved IP. 4. **GCP Private Service Connect.** PSC has no DNS-based automatic failover — workers must be manually repointed to the new region's PSC endpoint. See [PrivateLink / PSC stopped working](#symptom-privatelink--psc-stopped-working-after-failover). 5. **Private DNS override covers only one region.** Same section. -6. **Serverless Workers (AWS Lambda / GCP Cloud Run).** The Worker Controller Instance keeps invoking Workers in the compute provider's originally configured region because compute-provider configuration is region-scoped and the WCI has no failover-detection mechanism. This is a distinct failure mode from long-lived Worker DNS caching — see [`serverless-ha.md`](serverless-ha.md). +6. **Serverless Workers (AWS Lambda).** The Worker Controller Instance keeps invoking Workers in the compute provider's originally configured region because compute-provider configuration is region-scoped and the WCI has no failover-detection mechanism. This is a distinct failure mode from long-lived Worker DNS caching — see [Serverless Workers kept running in the old region](#symptom-serverless-workers-kept-running-in-the-old-region-after-failover). **Fix:** clear/await the offending cache, restart wedged workers, or repoint PSC workers per the discriminator that matched. @@ -89,6 +90,20 @@ dig +short ..tmprl.cloud Then re-run the operation that was failing. +## Symptom: Serverless Workers kept running in the old region after failover + +**Symptom:** the Namespace failed over successfully, but Serverless Workers (AWS Lambda, Public Preview) are still being invoked in the old region. Silent while that region is healthy; degraded throughput, latency, or a stall once it is not. + +Nothing in your infrastructure polls, so there is no DNS to re-resolve. The Worker Controller Instance invokes the compute provider configured on a Worker Deployment Version, that configuration is scoped to a single region (for example, a Lambda ARN), and the WCI has no mechanism to detect a failover or redirect invocations into the new active region. Applies to Multi-region and Multi-cloud Replication alike. See `/cloud/high-availability#serverless-workers` and the High Availability row of `/serverless-workers#constraints`. + +**Discriminate:** confirm the new active region (`tcld namespace get --namespace .` plus the `FailoverNamespace` audit entry) and compare it against the Lambda ARN on the Version serving the affected Task Queue. Long-lived Workers on other Task Queues recover on their own, so a mixed fleet recovers partially — which reads like a regional outage rather than a configuration constraint. + +**Fix:** `tcld namespace failover` moves the Namespace only, and tcld has no compute-provider surface. Remediation is to repoint the existing Worker Deployment Version's compute provider at a function in the new active region — an in-place update, not a new Version. Hand it to `skill-temporal-serverless`; it changes where production Workers are invoked, so propose it before running. + +**Prevent:** publish the function in every region the Namespace can fail over to, so the repoint is a single command instead of a provisioning exercise under time pressure. + +**Verify:** the affected Task Queue drains, and invocations land on the new region's function in the provider's logs. + ## Symptom: PrivateLink / PSC stopped working after failover **Symptom:** pre-failover the Namespace was reachable via a private VPC Endpoint; post-failover DNS resolves the Namespace Endpoint to a public IP the VPC can't reach, or to nothing. diff --git a/references/triage/serverless-ha.md b/references/triage/serverless-ha.md deleted file mode 100644 index 65f8938..0000000 --- a/references/triage/serverless-ha.md +++ /dev/null @@ -1,144 +0,0 @@ -# Serverless HA - -> [!NOTE] -> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. - -AWS Lambda support for Serverless Workers is in Public Preview; GCP Cloud Run support is in Pre-release. Serverless Workers *can* be used with a Namespace that has Multi-region or Multi-cloud Replication, but they do **not** follow a Namespace failover automatically. When the Namespace endpoint re-routes to the new active region, the Worker Controller Instance (WCI) keeps invoking Workers in the compute provider's originally configured region. Without manual intervention, workloads that failed over will continue to be executed by Workers in the old region — and if that region is degraded, your workloads are degraded with it. - -This file explains why the constraint exists and what an operator must do after a failover to keep processing Workflows. - -## Table of contents - -- [The constraint in one sentence](#the-constraint-in-one-sentence) -- [Why this happens](#why-this-happens) -- [Diagnostic signal — did a failover happen?](#diagnostic-signal--did-a-failover-happen) -- [Manual remediation](#manual-remediation) - - [AWS Lambda](#aws-lambda) - - [GCP Cloud Run](#gcp-cloud-run) -- [Prevention: dual-region compute-provider pre-provisioning](#prevention-dual-region-compute-provider-pre-provisioning) -- [Hand-off pointers](#hand-off-pointers) - -## The constraint in one sentence - -> On failover of a Namespace with Multi-region or Multi-cloud Replication, the WCI keeps invoking Workers in the original region unless you manually repoint the compute provider. Compute provider configuration, such as a Lambda ARN or a Cloud Run Worker Pool, is scoped to a single region. - -The canonical statement in the HA docs is equivalent: "each compute provider configuration (for example, a Lambda ARN) is scoped to a single region. The Worker Controller Instance (WCI) has no mechanism to detect a failover or redirect invocations to a Worker in the new active region. To keep processing Workflows after a failover, you must manually reconfigure the Worker Deployment Version's compute provider to point at a Worker in the new region." - -See the full docs entry at [https://docs.temporal.io/cloud/high-availability#serverless-workers](https://docs.temporal.io/cloud/high-availability#serverless-workers) and the Constraints table at [https://docs.temporal.io/serverless-workers#constraints](https://docs.temporal.io/serverless-workers#constraints). - -## Why this happens - -Two independent facts combine to produce the constraint: - -1. **Compute provider configuration is per-Worker-Deployment-Version and pins a region.** The compute provider "is set on a Worker Deployment Version and specifies the provider type, the invocation target, and the credentials Temporal needs to trigger the invocation." For AWS Lambda, the invocation target is a Lambda function ARN, which embeds the AWS region. For GCP Cloud Run, the target is a Worker Pool whose region is set explicitly by `--gcp-cloud-run-region`. - -2. **The WCI has no failover-detection or cross-region redirect.** "One WCI Workflow runs per Worker Deployment Version that has a compute provider configured. The WCI runs in the same Namespace as your Worker Deployment." The WCI's role is to scale Workers up and down in response to Task Queue conditions by triggering the configured compute provider. It has no branch that swaps compute provider region on failover; the HA docs are explicit that it "has no mechanism to detect a failover or redirect invocations to a Worker in the new active region." - -Contrast with long-lived Workers: for a long-lived fleet, "the DNS redirection orchestrated by Temporal ensures that your existing Workers continue to poll the Namespace without interruption. Temporal Cloud forwards their requests from the passive replica to the active region and the responses back, so Workers keep running through a failover." That mechanism operates at the *Client-to-Namespace* layer: a Worker process resolves the Namespace Endpoint and re-resolves when a connection cycles. Serverless Workers do not have a persistent poller in your infrastructure that could re-resolve DNS; each invocation is created *by Temporal* against the compute provider you configured, and that configuration is regional. - -For the DNS-based failover mechanics that long-lived Workers rely on, see [`references/triage/ha-failover.md`](./ha-failover.md). Do not re-derive that here. - -## Diagnostic signal — did a failover happen? - -You are usually called in because Workflows are stalled, latency is up, or a customer reported an outage. Before doing anything provider-specific, confirm that a Namespace failover actually occurred and note the new active region. - -- **Web UI.** "After any failover, whether triggered by you or by Temporal, an event appears in both the [Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs." Open the Namespace detail page and inspect the failover list. -- **Audit log signal.** "The audit log entry uses the `\"operation\": \"FailoverNamespace\"` event." Temporal Cloud also emails admins on every failover. -- **What long-lived Workers do vs. Serverless Workers.** Long-lived Workers follow the Namespace Endpoint DNS change and, once existing connections cycle (Temporal Cloud enforces a maximum connection lifetime of 5 minutes), re-resolve to the new active region. Serverless Workers do not — the WCI keeps calling the compute provider you configured until you repoint it. If your fleet is mixed, expect long-lived Workers to recover on their own within a few minutes while Serverless-only Task Queues stay wedged until you act. Note also that the Namespace Endpoint DNS change itself "can take a few minutes to fully propagate to all Clients and Workers." - -If a `FailoverNamespace` event has fired and the new active region is not the same region as your compute provider's Lambda ARN or Cloud Run Worker Pool, you are in the constraint. Proceed to remediation. - -## Manual remediation - -`tcld namespace failover --namespace . --region ` fails over the *Namespace*. It does not modify any Worker Deployment Version's compute provider. The documented remediation is to create a new Worker Deployment Version pointing at a Worker in the new active region, then set that Version current. - -The compute provider is configured at Version creation time via `temporal worker deployment create-version` with provider-specific flags. Changing the target region means creating a *new* Version — the two provider flag surfaces are described below. After creating the new Version, promote it with `temporal worker deployment set-current-version`; "without this step, tasks on the Task Queue will not route to the version, and Temporal will not invoke the Lambda function." The equivalent guidance holds for Cloud Run: without setting the version current, "Tasks on the Task Queue will not route to the version, and the WCI will not start any instances." - -### AWS Lambda - -**Prerequisite (AWS side).** A Lambda function must already exist in the new active AWS region, with an IAM trust configuration that lets Temporal assume the invocation role. The full AWS setup is out of scope for this file — hand off to [https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/aws-lambda](https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/aws-lambda) for the deployment guide. - -**Create a new Worker Deployment Version pointing at the new region's Lambda ARN.** The Lambda ARN embeds the region, so pointing at a new-region function is the entire point of the remediation. - -```bash -temporal worker deployment create-version \ - --namespace \ - --deployment-name my-app \ - --build-id build-1- \ - --aws-lambda-function-arn \ - --aws-lambda-assume-role-arn \ - --aws-lambda-assume-role-external-id -``` - -Flags in play (all transcribed from the AWS Lambda doc): - -- `--deployment-name` — Worker Deployment name. Must match `DeploymentName` in your Worker code. -- `--build-id` — Worker Deployment Version build ID. Must match `BuildID` in your Worker code. -- `--aws-lambda-function-arn` — Qualified versioned ARN of the Lambda function Temporal invokes for this version (for example, `function:my-worker:5`). -- `--aws-lambda-assume-role-arn` — IAM role Temporal assumes to invoke the function. -- `--aws-lambda-assume-role-external-id` — External ID configured in the IAM role trust policy. - -**Set the new Version current.** - -```bash -temporal worker deployment set-current-version \ - --deployment-name my-app \ - --build-id build-1- -``` - -### GCP Cloud Run - -**Prerequisite (GCP side).** A Cloud Run Worker Pool must exist in the new active region, in a project Temporal's invoker service account can impersonate. The Cloud Run setup — Worker Pool creation, invoker IAM, Terraform — is out of scope here; hand off to [https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/cloud-run](https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/cloud-run). - -**Create a new Worker Deployment Version with `--gcp-cloud-run-region` set to the new active region and the new pool's name/project/service-account.** - -```bash -temporal worker deployment create-version \ - --namespace \ - --deployment-name my-app \ - --build-id build-1- \ - --gcp-cloud-run-project \ - --gcp-cloud-run-region \ - --gcp-cloud-run-worker-pool \ - --gcp-cloud-run-service-account -``` - -Flags in play (all transcribed from the Cloud Run doc): - -- `--deployment-name` — Worker Deployment name. Must match `deployment_name` in your Worker code. -- `--build-id` — Worker Deployment Version build ID. Must match `build_id` in your Worker code. -- `--gcp-cloud-run-project` — GCP project ID that contains the Worker Pool. -- `--gcp-cloud-run-region` — Region of the Worker Pool. -- `--gcp-cloud-run-worker-pool` — Name of the Worker Pool. -- `--gcp-cloud-run-service-account` — The invoker service account Temporal impersonates to read and scale the pool. - -**Set the new Version current.** - -```bash -temporal worker deployment set-current-version \ - --namespace \ - --deployment-name my-app \ - --build-id build-1- -``` - -The Cloud Run `set-current-version` command asks for confirmation because it changes which version new Tasks route to; pass `--yes` to skip the prompt in an incident. - -## Prevention: dual-region compute-provider pre-provisioning - -The docs establish that a compute provider is per-Version and per-region, and that the mutation verbs available are `create-version` and `set-current-version`. From that alone, the operator-facing implication is clear: if you keep a Worker Deployment Version already configured for each region *in advance*, then failover-time work reduces to a single `set-current-version` call against the pre-existing new-region Version. You skip the AWS-side or GCP-side provisioning under time pressure. - -This means, for each Worker Deployment: - -- Publish the underlying compute artifact (Lambda function, Cloud Run Worker Pool) in every region you might fail over to, before you need it. -- Pre-create one Worker Deployment Version per region using `temporal worker deployment create-version` with that region's provider flags. Use distinct `--build-id` values so each Version is addressable. -- On failover, promote the Version whose compute provider is in the new active region with `temporal worker deployment set-current-version --deployment-name ... --build-id ...`. - -The docs stop short of prescribing a specific automation pattern (Terraform module, runbook script, or CI hook) for this pre-provisioning. Do not invent one on the operator's behalf. The Temporal Cloud Terraform provider does not support triggering failovers — that fact is documented for the Namespace-failover path, not the Worker-Deployment-Version path — so treat any Terraform-provider claim about `create-version`/`set-current-version` as something to look up in the CLI reference rather than assert from this file. - -## Hand-off pointers - -- **Worker placement architecture (Active-Passive vs. Active-Active, latency/cost trade-offs across regions):** see the [`skill-temporal-deploy`](https://docs.temporal.io/cloud/high-availability/architecture-patterns) skill's deployment-pattern references and [https://docs.temporal.io/cloud/high-availability/architecture-patterns](https://docs.temporal.io/cloud/high-availability/architecture-patterns). -- **`tcld` and `temporal` flag semantics in depth (all flags on `namespace failover`, `worker deployment create-version`, `worker deployment set-current-version`):** hand off to the `skill-temporal-cli` skill; that skill owns exhaustive CLI reference and stays authoritative when flag surfaces change. -- **Serverless Worker SDK code (registering Workflows/Activities, using the serverless Worker package for your language SDK, Worker Versioning `AutoUpgrade` vs `Pinned` behavior):** hand off to the `skill-temporal-developer` skill. -- **Long-lived Worker failover behavior (DNS/Namespace Endpoint propagation, the 5-minute connection lifetime, Regional Endpoint escape hatch):** see [`references/triage/ha-failover.md`](./ha-failover.md) in this skill. -- **How to trigger the Namespace failover itself (`tcld`, Web UI, Cloud Ops API):** [https://docs.temporal.io/cloud/high-availability/failovers/manage#trigger-failover](https://docs.temporal.io/cloud/high-availability/failovers/manage#trigger-failover). This file assumes the failover already happened.