diff --git a/README.md b/README.md index 2567adb..c50af81 100644 --- a/README.md +++ b/README.md @@ -1,44 +1,49 @@ # Temporal Serverless Workers Skill -Deploy and operate [Temporal](https://temporal.io/) Workers on serverless compute with help from a coding agent. The skill guides an agent through the complete AWS Lambda lifecycle: scoping, access checks, Worker implementation, packaging, deployment, Temporal registration, verification, troubleshooting, updates, and rollback. +Deploy and operate [Temporal](https://temporal.io/) Workers on serverless compute with help from a coding agent. The skill guides an agent through the complete lifecycle on AWS Lambda and GCP Cloud Run: scoping, access checks, Worker implementation, packaging, deployment, Temporal registration, verification, troubleshooting, updates, and rollback. > [!WARNING] > This skill is in Public Preview and will continue to evolve. Pin the Temporal SDK, serverless Worker package, and CLI versions for long-lived projects. > [!NOTE] -> Temporal Serverless Workers on AWS Lambda are in Public Preview and are available to all Temporal Cloud customers without an access request. AWS Lambda is currently the only compute provider supported by this skill. +> **AWS Lambda** is in Public Preview and available to all Temporal Cloud customers without an access request. +> **GCP Cloud Run** is in Pre-release, its APIs may change in backwards-incompatible ways, and access is granted on request — create a support ticket or contact your account team. + +> [!IMPORTANT] +> The two providers have different execution models. Lambda invokes a function per unit of work and the Worker exits when the invocation ends. Cloud Run resizes a pool of long-lived instances, scaling to zero when idle. That changes what the Worker code is, what bounds an Activity, what there is to tune, and how failures present — guidance does not transfer between them. Each provider directory carries a `constraints.md` describing what follows from its model. ## What the skill can do -- Build Serverless Workers with the Go, Python, TypeScript, Java, or .NET SDK. -- Package and deploy Workers to AWS Lambda with the correct architecture, timeout, and shutdown settings. -- Configure the separate AWS roles used by the Lambda function and by Temporal. +- Build Serverless Workers with the Go, Python, TypeScript, Java, or .NET SDK on either provider, plus Ruby and Rust on Cloud Run. +- Package and deploy to AWS Lambda with the correct architecture, timeout, and shutdown settings — or containerize and deploy to a Cloud Run Worker Pool. +- Configure the two distinct identities each provider needs: execution and invocation roles on AWS, runner and invoker service accounts on GCP. - Register a Worker Deployment Version, validate its Task Queue binding, and set it current. -- Verify a deployment from both Temporal Workflow history and Lambda logs. -- Diagnose Workers that are not invoked or do not complete Tasks. -- Publish immutable Lambda versions, update deployments, and roll back safely. -- Add OpenTelemetry observability with the AWS Distro for OpenTelemetry. +- Verify a deployment from both Temporal Workflow history and the provider's logs. +- Diagnose Workers that are never started, or that start but do not complete Tasks. +- Keep each build immutable, update deployments, and roll back safely. +- Add OpenTelemetry observability — the AWS Distro for OpenTelemetry on Lambda, the SDK's standard setup plus Cloud Logging on Cloud Run. - Configure self-hosted Temporal deployments that meet the serverless prerequisites. ## Support | Area | Supported | |---|---| -| Compute | AWS Lambda — Public Preview | +| Compute | AWS Lambda — Public Preview; GCP Cloud Run — Pre-release, access-gated | | Temporal | Temporal Cloud and self-hosted Temporal Service | -| SDKs | Go, Python, TypeScript, Java, .NET | +| SDKs | Lambda: Go, Python, TypeScript, Java, .NET. Cloud Run: those plus Ruby and Rust | | Other compute providers | Not currently supported | -For Temporal Cloud, the Namespace must be hosted on AWS. The Namespace and Lambda function may be in different AWS regions. +For Temporal Cloud, the Namespace must be hosted on the same cloud provider as the compute — AWS for Lambda, GCP for Cloud Run. Regions need not match. ## Before you start Before starting, make sure you can sign in to: -- An AWS account with permission to inspect and create the required Lambda, IAM, CloudFormation, and logging resources. -- A Temporal Cloud Namespace hosted on AWS, or a compatible self-hosted Temporal Service. +- For AWS Lambda: an AWS account with permission to inspect and create the required Lambda, IAM, CloudFormation, and logging resources. +- For GCP Cloud Run: a GCP project with the Cloud Run and Artifact Registry APIs enabled, and permission to create Worker Pools, service accounts, and Secret Manager secrets. Cloud Run access must also be enabled on your Temporal Cloud account. +- A Temporal Cloud Namespace hosted on the same cloud provider as your compute, or a compatible self-hosted Temporal Service. -You do not need to install or configure the AWS CLI, `tcld`, or the Temporal CLI before you begin. The skill checks what is already available and can help set up the tools and supported login flows needed for the task. If you prefer not to install a CLI, or a login method is unavailable, it can guide you through the corresponding Temporal Cloud UI or AWS console steps instead. It never asks you to paste credentials or secrets into the conversation. +You do not need to install or configure the AWS CLI, `gcloud`, Terraform, `tcld`, or the Temporal CLI before you begin. The skill checks what is already available and can help set up the tools and supported login flows needed for the task. If you prefer not to install a CLI, or a login method is unavailable, it can guide you through the corresponding Temporal Cloud UI, AWS console, or Google Cloud console steps instead. It never asks you to paste credentials or secrets into the conversation. ## Installation @@ -98,23 +103,31 @@ Package this Java Worker as a shaded jar and deploy it to Lambda. Deploy this .NET Worker to Lambda with a runtime-specific publish. ``` +```text +Deploy this Go Worker to a GCP Cloud Run Worker Pool. +``` + +```text +My Cloud Run Worker Pool is stuck at zero instances. Find out why. +``` + For a new deployment, the skill follows five stages: 1. **Scope** — confirm the SDK, compute provider, Namespace, region, and resource-naming prefix. -2. **Access** — verify AWS and Temporal identities and permissions, then present the exact billable resources for approval. -3. **Build** — install the serverless Worker package, inspect its current API, author the Worker, and deploy it. -4. **Connect** — configure Temporal's invocation role, register the Worker Deployment Version, validate the Task Queue binding, and set the version current. +2. **Access** — verify cloud-provider and Temporal identities and permissions, then present the exact billable resources for approval. +3. **Build** — author the Worker and deploy it. On Lambda that means installing the serverless Worker package and inspecting its current API; on Cloud Run it means an ordinary long-lived Worker in a container image. +4. **Connect** — grant Temporal access to the compute (an invocation role on AWS, an impersonated invoker service account on GCP), register the Worker Deployment Version, confirm it is reachable, and set the version current. 5. **Verify and hand back** — run a Workflow, confirm two independent health signals, inventory every created resource, and offer teardown. Nothing is created before you approve the resource list. Troubleshooting and inspection requests skip the deployment walkthrough and begin with read-only diagnostics. ## Important operating constraints -- Serverless Workers and their APIs are Public Preview, not generally available. +- Serverless Workers are not generally available: AWS Lambda is Public Preview, GCP Cloud Run is Pre-release and access-gated. - Every Workflow must use a Worker Versioning behavior: `Pinned` or `AutoUpgrade`. - The deployment name and build ID in Worker code must exactly match the registered Worker Deployment Version. -- Production releases should map each build ID to one immutable Lambda version. -- Activities must finish within the Lambda invocation limit and configured shutdown buffer; Workflow duration remains unbounded. +- Production releases should map each build ID to one immutable build: a published Lambda version, or a dedicated Cloud Run Worker Pool. +- Activities must finish within the compute provider's execution bound and configured shutdown buffer; Workflow duration remains unbounded. On AWS Lambda that bound is the invocation limit — see [`references/aws-lambda/constraints.md`](references/aws-lambda/constraints.md). - Secrets belong in a secret store for shared or production deployments, not plaintext environment variables. - Temporal creates and manages the Worker Controller Instance (WCI); this skill never creates or manages it directly. @@ -131,10 +144,23 @@ Nothing is created before you approve the resource list. Troubleshooting and ins | [`references/aws-lambda/sdk-dotnet.md`](references/aws-lambda/sdk-dotnet.md) | .NET package, API, handler, build, RID-specific publish, Lambda deployment values, tuned defaults, connection configuration, OpenTelemetry integration, logging, and diagnostics | | [`references/aws-lambda/setup.md`](references/aws-lambda/setup.md) | Shared AWS and Temporal deployment lifecycle, verification, and teardown workflow | | [`references/aws-lambda/iam.md`](references/aws-lambda/iam.md) | Operator permissions, Lambda execution role, and Temporal invocation role | +| [`references/aws-lambda/constraints.md`](references/aws-lambda/constraints.md) | What follows from Lambda's per-invocation execution model — Worker lifetime, invocation deadline, timeout triple, Activity duration bounds — and what does not generalize to other providers | | [`references/aws-lambda/diagnostics.md`](references/aws-lambda/diagnostics.md) | Diagnostic decision tree and WCI inspection | | [`references/aws-lambda/versioning.md`](references/aws-lambda/versioning.md) | Immutable releases, updates, and rollback | | [`references/aws-lambda/observability.md`](references/aws-lambda/observability.md) | Shared ADOT Collector configuration, X-Ray enablement, and IAM permissions | | [`references/aws-lambda/self-hosted.md`](references/aws-lambda/self-hosted.md) | Self-hosted Temporal prerequisites and configuration | +| [`references/gcp-cloud-run/sdk-go.md`](references/gcp-cloud-run/sdk-go.md) | Go versioned Worker, versioning behavior, connection configuration, image packaging, scale-in safety, and observability on Cloud Run | +| [`references/gcp-cloud-run/sdk-python.md`](references/gcp-cloud-run/sdk-python.md) | Python versioned Worker, versioning behavior, connection configuration, image packaging, scale-in safety, and observability on Cloud Run | +| [`references/gcp-cloud-run/sdk-typescript.md`](references/gcp-cloud-run/sdk-typescript.md) | TypeScript versioned Worker, versioning behavior, connection configuration, image packaging, scale-in safety, and observability on Cloud Run | +| [`references/gcp-cloud-run/sdk-java.md`](references/gcp-cloud-run/sdk-java.md) | Java versioned Worker, versioning behavior, connection configuration, image packaging, scale-in safety, and observability on Cloud Run | +| [`references/gcp-cloud-run/sdk-dotnet.md`](references/gcp-cloud-run/sdk-dotnet.md) | .NET versioned Worker, versioning behavior, connection configuration, image packaging, scale-in safety, and observability on Cloud Run | +| [`references/gcp-cloud-run/setup.md`](references/gcp-cloud-run/setup.md) | End-to-end Cloud Run deployment: container image, Worker Pool, registration, verification, teardown | +| [`references/gcp-cloud-run/iam.md`](references/gcp-cloud-run/iam.md) | Operator permissions, runner vs invoker service accounts, and the Terraform module | +| [`references/gcp-cloud-run/constraints.md`](references/gcp-cloud-run/constraints.md) | What follows from Cloud Run's pool-of-instances model — instance lifetime, autoscaling, scale-in interrupting Activities — and what does not generalize | +| [`references/gcp-cloud-run/versioning.md`](references/gcp-cloud-run/versioning.md) | One Worker Pool per build ID, the redeploy-into-a-live-pool hazard, and rollback | +| [`references/gcp-cloud-run/diagnostics.md`](references/gcp-cloud-run/diagnostics.md) | Pool annotations, scaling failures, and Worker-side errors | +| [`references/gcp-cloud-run/observability.md`](references/gcp-cloud-run/observability.md) | Logs, memory sizing per runtime, and the scaling signals to watch | +| [`references/gcp-cloud-run/self-hosted.md`](references/gcp-cloud-run/self-hosted.md) | Self-hosted prerequisites: dynamic config, the server's GCP identity, invoker creation | | [`assets/`](assets/) | CloudFormation templates for Temporal invocation roles | ## Feedback diff --git a/SKILL.md b/SKILL.md index dcee00d..1b985b4 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,34 +1,41 @@ --- name: temporal-serverless -description: 'Deploy and operate Temporal Workers on serverless compute (AWS Lambda) driven by the Worker Controller Instance (WCI). Use when the user mentions: "serverless worker", "Temporal serverless", "Worker Controller Instance", "WCI", "deploy Temporal worker on Lambda", "Lambda packaging", "Lambda timeout", "WCI inspection", "CloudFormation Temporal".' +description: 'Deploy and operate Temporal Workers on serverless compute (AWS Lambda, GCP Cloud Run) driven by the Worker Controller Instance (WCI). Use when the user mentions: "serverless worker", "Temporal serverless", "Worker Controller Instance", "WCI", "deploy Temporal worker on Lambda", "Lambda packaging", "Lambda timeout", "WCI inspection", "CloudFormation Temporal", "Cloud Run worker", "Worker Pool", "deploy Temporal worker on Cloud Run", "gcloud run worker-pools", "invoker service account".' version: 0.6.2 -disable-model-invocation: true --- # Skill: temporal-serverless ## Overview -This skill helps users deploy and operate Temporal Workers on serverless compute. Instead of a long-lived process, Temporal invokes the Worker on demand through the Worker Controller Instance (WCI); the Worker processes available Tasks and shuts down, scaling to zero when idle. The skill produces Worker code, deployment configuration, connection configs, and packaging steps for the chosen SDK, and walks users through troubleshooting when serverless Workers aren't picking up Tasks. +This skill helps users deploy and operate Temporal Workers on serverless compute controlled by the Worker Controller Instance (WCI). On Lambda, the WCI invokes short-lived Workers on demand; on Cloud Run, it resizes a pool of ordinary long-lived Workers. Both models scale to zero when idle. The skill produces Worker code, deployment configuration, connection configs, and packaging steps for the chosen SDK, and walks users through troubleshooting when serverless Workers aren't picking up Tasks. ## Supported compute providers | Cloud provider | Compute service | Support | Reference directory | |---|---|---|---| | AWS | Lambda | Supported — Public Preview, open to all Temporal Cloud customers | `references/aws-lambda/` | -| GCP | Cloud Run | Not supported | — | +| GCP | Cloud Run | Supported — **Pre-release, access-gated**; APIs may change incompatibly | `references/gcp-cloud-run/` | Only a provider marked Supported is covered. If a request names another, say it is not supported and stop; do not adapt a supported provider's material to it. **Never let the provider be an unstated assumption:** when the request does not name one, it is confirmed in the step 1 questions, not silently defaulted. -Every supported provider's directory carries the same shared layout — `setup.md`, `iam.md`, `versioning.md`, `diagnostics.md`, `observability.md`, `self-hosted.md` — plus one `sdk-.md` file for each supported SDK. Paths below are written `references//…`; substitute the directory from the table. Provider-specific commands, templates, permissions, SDK APIs, and defaults live there — this file stays at the workflow level. When a step needs concrete commands or SDK details, go to the reference file named at the end of that step. +**The two providers have different execution models.** Lambda invokes a function per unit of work and the Worker exits when the invocation ends. Cloud Run resizes a pool of long-lived instances, scaling to zero when idle. That changes what the Worker code is (a handler against a provider package, versus an ordinary long-lived Worker), what bounds an Activity, what there is to tune, and how failures present. Read the chosen provider's `constraints.md` before advising on any of it, and never carry a fact from one provider to the other. -| SDK language | AWS Lambda reference | -|---|---| -| Go | `references/aws-lambda/sdk-go.md` | -| Python | `references/aws-lambda/sdk-python.md` | -| TypeScript | `references/aws-lambda/sdk-typescript.md` | -| Java | `references/aws-lambda/sdk-java.md` | -| .NET | `references/aws-lambda/sdk-dotnet.md` | +**Cloud Run access is gated.** It is not open to all customers — the user creates a support ticket or contacts their account team. Confirm access before planning a Cloud Run deployment; no amount of correct configuration substitutes for it. + +Every supported provider's directory carries the same layout — `setup.md`, `iam.md`, `constraints.md`, `versioning.md`, `diagnostics.md`, `observability.md`, `self-hosted.md` — plus one `sdk-.md` file for each supported SDK. `constraints.md` is the provider "diff surface": what follows from that provider's execution model, and therefore what does *not* carry across to another one. Paths below are written `references//…`; substitute the directory from the table. Provider-specific commands, templates, permissions, SDK APIs, and defaults live there — this file stays at the workflow level. When a step needs concrete commands or SDK details, go to the reference file named at the end of that step. + +| SDK language | AWS Lambda reference | GCP Cloud Run reference | +|---|---|---| +| Go | `references/aws-lambda/sdk-go.md` | `references/gcp-cloud-run/sdk-go.md` | +| Python | `references/aws-lambda/sdk-python.md` | `references/gcp-cloud-run/sdk-python.md` | +| TypeScript | `references/aws-lambda/sdk-typescript.md` | `references/gcp-cloud-run/sdk-typescript.md` | +| Java | `references/aws-lambda/sdk-java.md` | `references/gcp-cloud-run/sdk-java.md` | +| .NET | `references/aws-lambda/sdk-dotnet.md` | `references/gcp-cloud-run/sdk-dotnet.md` | + +The two columns are not interchangeable. A Lambda SDK reference describes a provider Worker package and its handler; the Cloud Run counterpart describes an ordinary long-lived Worker with Worker Versioning enabled, and there is no Cloud Run Worker package. Read the one for the confirmed provider. + +**Ruby and Rust run on Cloud Run and have no Lambda packages**, so they have no row above. There is no `sdk-ruby.md` or `sdk-rust.md`; for those two, follow `references/gcp-cloud-run/setup.md` and the SDK's own Cloud Run guide, and say that the skill carries no per-SDK reference for them. **Public Preview is not GA.** The APIs are still evolving and may change: pin SDK and CLI versions for anything long-lived, and read the installed package's actual API surface rather than writing from memory. @@ -42,7 +49,7 @@ Follow these steps in order. Each step is provider-neutral; the concrete command > Here's what's about to happen, before I ask anything: > -> - This creates real resources in your cloud account — the compute unit that runs your Worker, roles, an infrastructure stack, logs. They're live and billable for as long as they exist. +> - This creates real resources in your cloud account — the compute unit that runs your Worker, identities and access grants, infrastructure state, and logs. They're live and billable for as long as they exist. > - **How it goes.** Five stages: > - **Scope** — a handful of questions, below. > - **Access** — check credentials and permissions on both sides, then show you an exact list of what I'm about to create and wait for your approval. @@ -54,7 +61,7 @@ Follow these steps in order. Each step is provider-neutral; the concrete command **Write the summary provider-neutral, because at that point you do not know the provider.** It is one of the things step 1 asks. Say "your cloud account", never the name of a provider you have not been told. The same applies to the account, Namespace, and region: if a cheap read-only call has already told you (see step 1), name what you actually found; otherwise leave it out rather than filling it in with a plausible guess. -Skip the summary for troubleshooting, inspection, and configuration-change tasks. Someone whose Worker is not being invoked does not need an overview of a deployment they have already done. +Skip the summary for troubleshooting, inspection, and configuration-change tasks. Someone whose Worker is not receiving Tasks does not need an overview of a deployment they have already done. **Then track the run on a checklist, and reprint it every time a step completes.** The eight steps group into the five stages below. Create one item per step, grouped under its stage, and build the checklist as soon as step 1's answers land, so items can name the confirmed provider and the agreed prefix instead of hedging. @@ -76,7 +83,7 @@ Where the harness has a todo list, use it *in addition to* the printed checklist > > **Connect** > ⬜ Create the role Temporal assumes to invoke the Worker -> ⬜ Register the Worker Deployment Version, confirm the validation invocation bound the Task Queue, set it current +> ⬜ Register the Worker Deployment Version, confirm its registration bootstrap bound the Task Queue, set it current > > **Verify and hand back** > ⬜ Start a Workflow and confirm it executes, from both the Temporal side and the provider's logs @@ -92,9 +99,11 @@ Where the harness has a todo list, use it *in addition to* the printed checklist **A step is complete when its verification passed — not when its command exited zero.** Several commands in this workflow exit clean having done nothing: the traffic-shifting and key-revocation commands no-op when their confirmation prompt goes unanswered, and providers return from create and update calls while the resource is still settling. Check an item off against state you read back, not against an exit code. When a step's verification fails, say which step you are on and what it is blocked on rather than moving down the list. -1. **Scope the task.** Identify the SDK language (Go, Python, TypeScript, Java, or .NET), the deployment target (Temporal Cloud or self-hosted — self-hosted has its own server prerequisites), the compute provider, and whether this is a new setup, a configuration change, or troubleshooting. Confirm the deployment target is compatible with the chosen provider — see "A Namespace on the target cloud provider is required" under Provider-neutral principles. Ensure a Temporal client/CLI is available and authenticated to the target. Each changes the specifics. → `references/concepts.md` for what the user is building; `references//setup.md` for the compatibility and client-setup details. +1. **Scope the task.** Identify the SDK language (Go, Python, TypeScript, Java, .NET, and on Cloud Run also Ruby or Rust — **SDK support differs by provider**), the deployment target (Temporal Cloud or self-hosted — self-hosted has its own server prerequisites), the compute provider, and whether this is a new setup, a configuration change, or troubleshooting. Confirm the deployment target is compatible with the chosen provider — see "A Namespace on the target cloud provider is required" under Provider-neutral principles. Ensure a Temporal client/CLI is available and authenticated to the target. Each changes the specifics. → `references/concepts.md` for what the user is building; `references//setup.md` for the compatibility and client-setup details. - **Put the compute provider in that batch of questions as a confirmable default, not a free choice.** Pre-select the supported provider from the table above and carry its support status in the option's description. The user confirms rather than chooses, so it costs no extra turn, but the provider is never something they were assumed into. Skip the question only when the request already names a provider. Do not restate any of this in a paragraph before the questions; the option description is where it belongs. + **Ask the compute provider as a real question now that there are two, and carry each option's support status in its description** — AWS Lambda is Public Preview and open to everyone; GCP Cloud Run is Pre-release, access-gated, and its APIs may change incompatibly. Skip the question only when the request already names a provider. Do not restate any of this in a paragraph before the questions; the option description is where it belongs. + + **The Namespace usually settles it, so ask them together.** A Serverless Worker runs only on the cloud provider hosting its Namespace, so a user with only AWS Namespaces has no Cloud Run option. Where the user genuinely has both, the deciding factors are: **Activity duration** (anything over Lambda's 15-minute ceiling rules Lambda out), **SDK** (Ruby and Rust are Cloud Run only), and **tolerance for Pre-release APIs**. Say which factor decided it rather than presenting the choice as arbitrary. **Let the user pick the Namespace from a list; never make them retype one.** Namespace names are long and error-prone — a generated suffix on an account ID, `-.`. Where control-plane access is available, `tcld namespace list` returns the full Namespace objects, so one call gives every name with its region — and a region ID is provider-prefixed (`aws-…`, `gcp-…`), so the same response tells you each Namespace's provider. Only the prefix carries meaning; the region itself imposes no constraint. @@ -139,19 +148,38 @@ Where the harness has a todo list, use it *in addition to* the printed checklist **Do not self-select a row.** Drop to a lower one only after the choice above has been put to the user and the browser path chosen, or the login attempted and failed. When you hand off a runbook, say the offer stands — if the user authenticates and comes back, take the work over rather than leaving them to run the steps by hand. - **Before the first account-mutating command, list what you are about to create — with final names — and get approval.** Name the target account and region, then every resource: compute unit, execution role, infrastructure stack, log group, deployment name, and Task Queue. Say plainly that they are live and billable. This is the mirror of the inventory in step 8, and it is worth more here than there: it makes the naming prefix concrete while changing it is still free, and the deployment name, build ID, and Task Queue become expensive to change once step 3 compiles them into the Worker. Skip it only when nothing will be created — a troubleshooting or inspection task. + **Before the first account-mutating command, list what you are about to create — with final names — and get approval.** Name the target account or project and region, then every resource. For Lambda this includes the function, execution and invocation roles, infrastructure stack, and log group. For Cloud Run this includes the image repository and image, Worker Pool, runner and invoker service accounts, Terraform state, logs, and any secrets. For either provider include the Temporal deployment name, build ID, and Task Queue. Say plainly that the resources are live and billable. This is the mirror of the inventory in step 8, and it is worth more here than there: it makes the naming prefix concrete while changing it is still free, and the deployment name, build ID, and Task Queue become expensive to change once step 3 compiles them into the Worker. Skip it only when nothing will be created — a troubleshooting or inspection task. + +3. **Author the Worker.** Follow the selected provider's Worker model; do not infer one from the phrase "serverless Worker." -3. **Author the Worker.** *Install the SDK's serverless Worker package before writing any code* — it is usually shipped separately from the main SDK — sometimes on its own version line, sometimes in lockstep with it, and in one SDK not separately at all — so having the base SDK installed does not mean it is importable. Then read the installed package's actual API surface and write against that; these are Public Preview APIs that drift between versions, and generating code from memory costs a build cycle. Entry-point names are not consistent between SDKs, so inspect first rather than pattern-matching from another language. Every Workflow must declare a versioning behavior (`Pinned` or `AutoUpgrade`), per-Workflow or as a Worker-level default — code without it fails at runtime. → `references//sdk-.md` (package, install, API inspection, entry point, handler shape, versioning behavior, tuned defaults). + - **AWS Lambda:** install the SDK's serverless Worker package before writing code. It is usually shipped separately from the main SDK, and its Public Preview API and handler shape can drift. Read the installed package's actual API before writing the entry point. + - **GCP Cloud Run:** write an ordinary long-lived Worker that starts polling when the container starts. There is no Cloud Run serverless Worker package or per-invocation handler. For Go, Python, TypeScript, Java, and .NET, use the selected `references/gcp-cloud-run/sdk-.md`; for Ruby or Rust, use `references/gcp-cloud-run/setup.md` and the SDK's current Cloud Run guide. -4. **Package and deploy the compute unit.** Build and package per SDK, deploy the compute unit, and set the invocation deadline high enough for the Worker to start, connect, register the Task Queue, and shut down gracefully. Match the build's target architecture to the deployed compute unit's — a mismatch fails only at invocation time, not at build time. After a create or update, wait for the compute unit to reach a ready state before the next step; providers return from these calls while the unit is still settling. → `references//sdk-.md` (build, packaging, runtime, handler, architecture, and SDK-specific deployment values) and `references//setup.md` (shared deployment lifecycle). + Every Workflow must declare a versioning behavior (`Pinned` or `AutoUpgrade`), per Workflow or as a Worker-level default. The deployment name and build ID in the Worker must match the version that will be registered. → `references//constraints.md` and the applicable SDK reference. -5. **Grant Temporal permission to invoke the Worker.** Configure the compute provider's access so Temporal can invoke and inspect the Worker. This access is separate from the compute unit's own execution role — do not confuse the two. Two things to get right before you create anything: (a) this grant is **shared, account-wide infrastructure** that a previous deployment may already have created — look for an existing one and extend it to cover your new Worker rather than creating a parallel copy, and never delete or repurpose one you did not create without asking; (b) scope the grant so that *future* immutable builds are covered, not just today's — a grant pinned to one build breaks the next release in a way that surfaces later as an unrelated-looking invocation failure. → `references//iam.md`. +4. **Package and deploy the compute unit.** Build for the target runtime and architecture, deploy an immutable unit for this build ID, and wait for the provider to report it ready before registering it; provider create and update calls may return while the resource is still settling. -6. **Register the Worker Deployment Version, verify the validation invocation, then set it current.** Create the Worker Deployment Version with the compute provider configured; the deployment name and build ID must exactly match the values in the Worker code. Creating it triggers one validation invocation — **check that it bound the Task Queue before going further.** If the Task Queue is bound, the permission grant, package, config, and deadline are all provably correct, and any later failure is downstream; if it is not, setting the version current will not fix it. Then set it current: through the UI this happens automatically, through the CLI it is a separate step, without which Tasks never route to the version. → `references//setup.md`. + - **AWS Lambda:** package the SDK-specific handler, configure the function architecture and runtime, and set the invocation deadline and shutdown settings together so the Worker can initialize, process Tasks, and drain safely. + - **GCP Cloud Run:** build and push a container image, then create a dedicated Worker Pool for the build ID at zero instances. The container entry point runs the Worker continuously. There is no invocation deadline or Lambda shutdown buffer to configure. -7. **Verify.** Start a Workflow on the Task Queue and confirm Temporal invokes the Worker — check the Workflow history in the Temporal UI and the compute provider's logs. If it does not progress, → `references//diagnostics.md`. + → `references//sdk-.md`, when one exists, and `references//setup.md`. -8. **Hand back the inventory first; offer teardown as the closing note.** The order is inventory → offer, never the reverse. Close with what now exists — compute unit and published build identifiers, roles, infrastructure stacks, region, deployment name and build ID — and what the run actually did, including anything you worked around or deviated from. Say plainly that it is live and billable. These names are only knowable from the run that created them, and reconstructing them later means scanning the user's account. +5. **Grant Temporal permission to control the compute.** Keep the identity Temporal uses separate from the identity the compute runs as. + + - **AWS Lambda:** configure the invocation role Temporal assumes to inspect and invoke the function; it is distinct from the function's execution role. + - **GCP Cloud Run:** configure the invoker service account Temporal impersonates with permission to read and update Worker Pools; it is distinct from the runner service account attached to pool instances. + + Treat this grant as shared account- or project-level infrastructure: look for an existing compatible grant before creating another, cover future immutable builds rather than only today's target, and never delete or repurpose infrastructure you did not create without asking. → `references//iam.md`. + +6. **Register the Worker Deployment Version, verify its registration bootstrap, then set it current.** Create the version with compute configured; the deployment name and build ID must exactly match the Worker. The WCI first validates the provider configuration, then starts enough compute for the Worker to poll and register its Task Queues: Lambda emits an invoke action, while Cloud Run resizes the Worker Pool to the scaler's planned count, with a minimum of one for registration even when `initial_count` is zero. This bootstrap happens during version registration and does not require the version to be current. + + Wait until `describe-version` shows the expected Task Queue types before shifting traffic. A binding proves that a Worker started, connected, and polled under the registered deployment name and build ID. If it is absent, setting the version current does not repair the bootstrap; inspect the provider-specific failure path. Do not confuse this with the UI's **Validate Connection** action: for Cloud Run that action reads the pool but neither starts an instance nor tests the update permission used for scaling. + + Then set the version current. The UI does this automatically during its creation flow; the CLI requires a separate command. → `references//setup.md` and `references//diagnostics.md`. + +7. **Verify.** Start a Workflow on the Task Queue and confirm it progresses in Temporal, then confirm the matching provider-side signal: a Lambda invocation and Worker logs, or a Cloud Run pool resize followed by instance startup and Worker logs. If the Workflow does not progress, follow `references//diagnostics.md` rather than translating the other provider's symptoms. + +8. **Hand back the inventory first; offer teardown as the closing note.** The order is inventory → offer, never the reverse. Close with what now exists — compute unit and immutable build identifiers, cloud identities and access grants, infrastructure stack or Terraform state, region, deployment name, and build ID — and what the run actually did, including anything you worked around or deviated from. Say plainly that it is live and billable. These names are only knowable from the run that created them, and reconstructing them later means scanning the user's account or project. **Do not write a teardown script before the user asks for one.** Generating it unprompted buries the inventory under a file they did not request, and the inventory is what they need in order to decide. End with a single line — *"Let me know if you want a teardown script to remove these resources"* — and stop there. Write the script, or run the teardown, when they take you up on it. → `references//setup.md` (Teardown). @@ -169,18 +197,18 @@ How to move through the workflow above. > > **Creating the invocation role Temporal assumes, with a generated External ID** - For anything that creates, updates, or deletes, name the resource and the target account or Namespace explicitly — an approval prompt should arrive with its justification already on screen, not after it. + For anything that creates, updates, or deletes, name the resource and the target account, project, or Namespace explicitly — an approval prompt should arrive with its justification already on screen, not after it. - **Read the current state instead of recalling it.** Check the installed package's API, the CLI's own `--help` for the flags you are about to pass, the compute unit's reported state, and the CLI version. Each of these has drifted in practice: a Public Preview SDK whose fields moved, a CLI too old to have the serverless subcommand at all, a resource that reports success while still settling. - **Do not chain `cd` with commands that create or modify files.** A compound `cd && ` triggers a manual approval prompt no matter how the user's permissions are configured, so scaffolding a project this way asks for approval on every run. Use absolute paths, or the tool's own directory flag (`go -C …`), and rely on the shell's working directory persisting between calls — the `cd` buys nothing and costs a prompt. Keep the command count down for the same reason: one `go get` covering both packages beats two. -- **Verify each step before building the next on top of it.** Compile the Worker before packaging it, confirm the package's target architecture before uploading, wait for the compute unit to be ready before publishing a build, and confirm the Task Queue is bound before shifting traffic. Deployment failures here surface far from their cause — an architecture or dependency mismatch appears only at first invocation, and a first-invocation failure appears as "the Worker is never invoked", several steps later. -- **When something fails, read the actual error before changing anything.** Fetch the failure reason from the provider (deployment events, logs, status fields) and fix that. Do not retry the same command with variations, and do not start editing permissions or trust policies on the theory that the problem might be access — most first-invocation failures are not permission problems, and some failures are on Temporal's side and will reproduce no matter what you change. -- **Treat the user's account as shared and pre-existing.** Assume other deployments, roles, and stacks are already there. Look before creating, extend rather than duplicate, and never delete or repurpose something you did not create without asking. When you do work around existing infrastructure — a different name, a reused role — say so explicitly in your summary rather than leaving it as a silent deviation. +- **Verify each step before building the next on top of it.** Compile the Worker before packaging it, confirm the artifact or image architecture before deploying it, wait for the compute unit to be ready before registering the build, and confirm the Task Queue is bound before shifting traffic. Deployment failures surface far from their cause: a bad Lambda package appears at first invocation, while a bad Cloud Run image appears after the pool resize when its instance tries to start. +- **When something fails, read the actual error before changing anything.** Fetch the failure reason from the provider (deployment events, logs, status fields) and fix that. Do not retry the same command with variations, and do not start editing permissions or trust policies on the theory that the problem might be access — startup failures are often in the artifact, image, or Worker configuration, and Temporal-side failures reproduce no matter what you change in the provider. +- **Treat the user's account or project as shared and pre-existing.** Assume other deployments, identities, and infrastructure state are already there. Look before creating, extend rather than duplicate, and never delete or repurpose something you did not create without asking. When you do work around existing infrastructure — a different name or a reused identity — say so explicitly in your summary rather than leaving it as a silent deviation. - **Confirm the end state from two independent signals.** A Workflow that completes in the Temporal UI *and* the Worker's own logs showing startup, Task Queue registration, and Task execution. One signal alone can mislead: a system Workflow that exists and is running proves nothing about invocation health, and a command that exits zero may have done nothing at all if it was waiting on a confirmation prompt. - **Account for what you created.** Keep the inventory as you go rather than reconstructing it at the end, say plainly that the resources are live and billable, and offer to tear them down (step 8). ## Never create or manage the WCI -Temporal creates the WCI automatically once a Worker Deployment Version has a compute provider. You never create, start, or manage it. A WCI that exists or is running is *not* evidence that invocation works — it continue-as-news and keeps running even while its Activities fail. Diagnose from Temporal's own signals: read the WCI Workflow history and look for Activity failures. Do not enumerate compute resources across regions or scan the account to reverse-engineer state. → `references/concepts.md`, `references//diagnostics.md`. +Temporal creates the WCI automatically once a Worker Deployment Version has a compute provider. You never create, start, or manage it. A WCI that exists or is running is *not* evidence that its provider action works — it continues-as-new and keeps running even while invoke or resize Activities fail. Diagnose from Temporal's own signals: read the WCI Workflow history and look for Activity failures. Do not enumerate compute resources across regions or scan the account or project to reverse-engineer state. → `references/concepts.md`, `references//diagnostics.md`. ## Provider-neutral principles @@ -189,32 +217,29 @@ Surface these early — they apply regardless of compute provider: - **A Namespace on the target cloud provider is required.** A Serverless Worker runs only on the cloud provider that hosts its Temporal Cloud Namespace — there is no cross-cloud pairing. Confirm the user has a Namespace on the provider they intend to run compute on *before* building anything; without one, the work stops there and they need either a Namespace on that provider or a different provider. A mismatch is not caught at deploy time — it fails later, at connection time. **Regions do not have to match:** a Namespace in one region can drive a compute unit in another, so never tell a user to move or re-create a Namespace to line up regions. - **Use `tcld` for every Temporal Cloud control-plane operation** — accounts, Namespaces, API keys, users, service accounts. Do not use the unified CLI's `temporal cloud …` subcommands for them. Worker Deployments and Workflows are *not* control-plane operations: they live on the Namespace frontend, have no `tcld` equivalent, and use `temporal worker deployment …`. → `references//setup.md`. - **Versioning behavior is mandatory.** Every Workflow needs `Pinned` or `AutoUpgrade`, or the Worker sets a default. -- **Deployment name and build ID must match exactly** between the Worker code and the Worker Deployment Version. A mismatch causes an invocation loop (Temporal invokes → Worker polls with the wrong version → Task not processed → invoke again). Signature: rapid repeated invocations with no Workflow progress. -- **Set the invocation deadline high enough.** Providers often default to a very short timeout. If the first invocation times out before the Worker registers the Task Queue, the binding is never created and the Worker is never invoked again. → `references//setup.md` for the exact default. +- **Deployment name and build ID must match exactly** between the Worker code and the Worker Deployment Version. The symptom follows the provider: Lambda repeatedly invokes Workers that poll under the wrong version, while Cloud Run shows running instances that poll under the wrong version and make no progress on the intended Tasks. → `references//diagnostics.md`. - **Use an immutable, versioned build per Build ID in production.** Pointing the provider at a mutable "latest" target lets code change under in-flight Workflows and cause non-determinism errors, even for Pinned Workflows. Keep a 1-to-1 mapping between each Build ID and one immutable build. → `references//versioning.md`. -- **Tune the timeout triple together for long-running Activities:** (1) worker stop timeout > longest Activity runtime, (2) shutdown deadline buffer > worker stop timeout + shutdown hook time, (3) invocation deadline > longest Activity runtime + shutdown deadline buffer. Raising one alone does not help. If the longest Activity exceeds half the maximum invocation deadline, recommend Activity Heartbeats. → `references/concepts.md`, `references//sdk-.md`. -- **Eager Activities are always disabled** — serverless invocations don't maintain persistent connections. Don't suggest them as an optimization. -- **Activities are bounded by the invocation limit** (minus the shutdown deadline buffer); Workflow duration is unbounded and can span many invocations. Flag Activities that approach the provider's limit early. → `references/concepts.md`. -- **Mixed serverless + long-lived Workers on one Task Queue:** do not enable dynamic scaling on the long-lived Workers — the two groups can't coordinate scaling and will cause unnecessary invocations. +- **Respect the compute provider's execution model, and do not assume it.** How long a Worker lives, what bounds an Activity's duration, which Worker options the provider pins, and how much timeout tuning is needed all follow from the provider's model — and they differ enough between providers that carrying an assumption across is a real source of wrong answers. Read them before advising on Activity duration, timeouts, or Worker options. → `references//constraints.md`. +- **Mixed serverless + independently managed long-lived Workers require provider-specific treatment.** Lambda can use a fixed long-lived fleet as spillover capacity, but do not dynamically scale that fleet. Do not share a Cloud Run serverless Worker's Task Queue with another long-lived fleet: the rate-based WCI scaler sees the full queue workload and cannot subtract work handled by the other fleet, so it provisions duplicate capacity. → `references//constraints.md`. - **Secrets belong in a secret store**, not plaintext environment variables. Provider docs and quickstarts commonly pass the API key or TLS key as a plaintext environment variable; that is acceptable in a throwaway development walkthrough *only if you say so explicitly at the time*. Anything the user describes as production, shared, or long-lived gets the secret store, loaded at cold start. Either way, keep key material out of shell history and command echoes. +- **Never put a Temporal API key in `--api-key`, an inline environment assignment, generated scripts, or an echoed command.** Namespace commands inherit an already-populated `TEMPORAL_API_KEY` or use an existing Temporal CLI profile. If neither is available, pause and have the user set the variable privately in their own terminal; never ask for, print, inspect, or capture its value. This rule applies even to a one-off verification command. - **Both CLIs prompt for confirmation before mutating state, and their flags differ.** Setting the current or ramping version, and revoking an API key, all ask interactively; run non-interactively without the flag, the command exits having done nothing, which reads as success. `temporal worker deployment …` takes `--yes`; `tcld` takes the global `--auto_confirm`. Pass the right one in scripts, CI, and agent shells, and confirm the resulting state rather than trusting the exit code. → `references//setup.md`. ## Troubleshooting -Start by determining whether the Worker is being invoked at all. Then, in priority order: (1) **Validate Connection** in the Temporal UI (Workers > Deployments > select > Actions > Validate Connection) — checks credentials, role assumption, and reachability in one step; (2) check whether the version's **Task Queue is bound** — if it is, invocation and Worker startup provably work and the fault is downstream, which rules out most of the surface in one command; (3) confirm the version is **current** (CLI-created versions are not automatic, and a confirmation-prompted command may have silently done nothing); (4) check the compute provider's logs for connection, auth, or TLS errors; (5) if rapid repeated invocations show no progress, check the deployment name/build ID match. Distinguish a Temporal-side failure (reproduces no matter what you change on the provider side) from a genuine user-permission problem before editing anything. → `references//diagnostics.md`, `references/concepts.md`. +Start by identifying the provider and asking the corresponding lifecycle question: **did Lambda invoke the function, or did Cloud Run receive a pool resize and start an instance?** Then, in priority order: (1) use **Validate Connection** in the Temporal UI, interpreting only what the selected provider says it proves; (2) inspect the registration bootstrap and check whether the version's expected **Task Queue types are bound**; (3) confirm the version is **current**; (4) read the provider's logs for startup, connection, authentication, or TLS errors; and (5) check the deployment name/build ID match, using the provider-specific symptom rather than assuming an invocation loop. Distinguish a Temporal-side failure from a genuine provider-permission problem before editing anything. → `references//diagnostics.md`, `references/concepts.md`. ## Common Pitfalls High-impact mistakes — warn the user proactively. Each is a symptom → cause → fix. -1. **Deployment name / build ID mismatch → invocation loop.** *Symptom:* rapid, repeated invocations with no Workflow progress. *Cause:* the name or build ID in the Worker code doesn't match the Worker Deployment Version, so the Worker polls with the wrong version, the Task isn't processed, and Temporal invokes again. *Fix:* make the values in code exactly match the version configuration. -2. **Version not set as current.** A version created through the CLI is not automatically current; without it, Tasks don't route to the version and the Worker is never invoked. *Fix:* set it current as a separate step (the UI does this automatically). -3. **Failed first invocation.** When a version is created, the WCI invokes the Worker once to validate. If that invocation fails — missing env vars, bad TLS/auth config, missing dependencies, or an invocation deadline too short for the Worker to start and register the Task Queue — the Worker never connects, never polls, the binding is never created, and the Worker is never automatically invoked again. *Fix:* diagnose by manually invoking the compute unit, and confirm the invocation deadline is set high. -4. **Confusing the two roles.** The compute unit's execution role (grants the function permission to run) is separate from the access Temporal uses to invoke it. Never describe one as the other. → `references//iam.md`. -5. **Timeout tuning mismatch.** Raising only the shutdown deadline buffer makes the Worker stop polling earlier but gives in-flight Activities no more time; raising only the worker stop timeout doesn't make it stop polling earlier, so the provider may terminate the Worker first. *Fix:* tune the three values together (see the timeout triple above). -6. **Mutable "latest" build reference in production.** Pointing the provider at a mutable/unqualified target means the code changes on every redeploy; deploying replay-unsafe code then causes non-determinism errors for in-flight Workflows, even Pinned ones. *Fix:* publish an immutable versioned build and keep a 1-to-1 mapping between each Build ID and one build. → `references//versioning.md`. -7. **Re-creating shared permission infrastructure that already exists.** *Symptom:* the infrastructure deployment fails outright and rolls back, or it succeeds and leaves a second, redundant grant behind. *Cause:* the permission grant Temporal assumes is account-wide with a fixed default name, so a previous serverless deployment already owns it. *Fix:* check whether it exists and what owns it *before* creating; extend the existing one to cover the new Worker, and fall back to a distinctly named parallel one only when the existing infrastructure is not yours to change — saying why when you do. A failed-and-rolled-back deployment must be deleted before the name can be reused; a successful one is live infrastructure and must not be. → `references//iam.md`. -8. **Invoke permission scoped to a single build.** *Symptom:* the deployment works, then the *next* release cannot be invoked, with an error that looks like a connection or configuration problem rather than a permissions one. *Cause:* the grant named one immutable build, and the new release is a different resource. *Fix:* scope the grant to cover the base resource and all its published builds. → `references//iam.md`. +1. **Deployment name / build ID mismatch.** *Symptom:* Lambda rapidly invokes without Workflow progress; Cloud Run starts instances that look healthy but do not process the intended Tasks. *Cause:* the Worker polls under a different version from the one the WCI controls. *Fix:* make both values in code exactly match the version configuration. +2. **Version not set as current.** A version created through the CLI is not automatically current; without it, new traffic does not route to the version. *Fix:* set it current as a separate step (the UI does this automatically). +3. **Confusing the two identities.** The compute unit's own identity (which lets it run — a Lambda execution role, a Cloud Run runner service account) is separate from the identity Temporal uses to reach it. Never describe one as the other. → `references//iam.md`. +4. **Mutable "latest" build reference in production.** Pointing the provider at a mutable/unqualified target means the code changes on every redeploy; deploying replay-unsafe code then causes non-determinism errors for in-flight Workflows, even Pinned ones. *Fix:* publish an immutable versioned build and keep a 1-to-1 mapping between each Build ID and one build. → `references//versioning.md`. +5. **Re-creating shared permission infrastructure that already exists.** *Symptom:* the infrastructure deployment fails and rolls back, or succeeds but leaves a redundant grant. *Cause:* a previous serverless deployment already owns the account- or project-level role/service account under the default name. *Fix:* check ownership before creating, extend a compatible shared grant to cover the new Worker, and create a distinctly named parallel grant only when the existing infrastructure is not yours to change. A failed-and-rolled-back deployment must be deleted before its name can be reused; a successful one is live infrastructure and must not be. → `references//iam.md`. + +**Provider-specific pitfalls live with their provider.** The five above apply to any provider; the ones that follow from an execution model — a failed first invocation and timeout-triple mismatch on Lambda, scale-in interrupting Activities on Cloud Run — are in `references//constraints.md`. Read both lists. ## Routing to reference files @@ -224,17 +249,18 @@ Most questions need 2–3 reference files. |---|---| | What is a Serverless Worker / the WCI? How do invocation and autoscaling work? What are the constraints? Serverless vs long-lived Workers? | `references/concepts.md` | | Deploy a Serverless Worker (happy path): write code, package, deploy, register + set-current version, verify, tear down. | `references//setup.md` + the selected `references//sdk-.md` (+ `references/concepts.md`) | -| Operator permissions and preflight; execution role vs Temporal invocation role; CloudFormation (Cloud + self-hosted). | `references//iam.md` | -| Update or redeploy; version the build, use a qualified ARN, roll back. | `references//versioning.md` (+ `references/concepts.md`) | -| Self-hosted server enablement (dynamic config, WCI, server AWS credentials). | `references//self-hosted.md` (+ `references//iam.md`) | -| Go SDK-specific options and tuned defaults, package and import, API inspection, handler, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, OpenTelemetry integration. | `references//sdk-go.md` | -| Python SDK-specific options and tuned defaults, package and import, API inspection, handler, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, OpenTelemetry integration, diagnostic signatures. | `references//sdk-python.md` | -| TypeScript SDK-specific options and tuned defaults, package and import, API inspection, handler, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, pre-bundled Workflow code, OpenTelemetry integration. | `references//sdk-typescript.md` | -| Java SDK-specific options and tuned defaults, artifact and imports, API inspection, handler, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, OpenTelemetry integration, logging and diagnostic signatures. | `references//sdk-java.md` | -| .NET SDK-specific options and tuned defaults, package and imports, API inspection, handler, RID-specific publish and packaging, runtime and deployment values, versioning-behavior configuration, connection config and `SSL_CERT_FILE`, OpenTelemetry integration, logging and diagnostic signatures. | `references//sdk-dotnet.md` | -| Add OpenTelemetry observability, Collector config, X-Ray, and IAM. | `references//observability.md` + the selected `references//sdk-.md` | -| Worker not invoked, Workflows not progressing, inspect the WCI. | `references//diagnostics.md` + the selected `references//sdk-.md` (+ `references/concepts.md`) | -| Long-running Activities and timeout relationships. Isolate Activities from resource exhaustion. | `references/concepts.md` (+ the selected `references//sdk-.md`) | +| Operator permissions and preflight; the compute unit's own identity vs the identity Temporal uses; infrastructure-as-code (CloudFormation on Lambda, Terraform on Cloud Run). | `references//iam.md` | +| Update or redeploy; make each build immutable, roll back. | `references//versioning.md` (+ `references/concepts.md`) | +| Self-hosted server enablement (dynamic config, WCI, the server's cloud credentials). | `references//self-hosted.md` (+ `references//iam.md`) | +| Go SDK-specific Worker construction and options, package and import, API inspection, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, graceful shutdown, OpenTelemetry integration. | `references//sdk-go.md` | +| Python SDK-specific Worker construction and options, package and import, API inspection, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, graceful shutdown, logging, OpenTelemetry integration, and diagnostic signatures. | `references//sdk-python.md` | +| TypeScript SDK-specific Worker construction and options, package and import, API inspection, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, graceful shutdown, pre-bundled Workflow code, OpenTelemetry integration. | `references//sdk-typescript.md` | +| Java SDK-specific Worker construction and options, artifact and imports, API inspection, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, graceful shutdown, OpenTelemetry integration, logging, and diagnostic signatures. | `references//sdk-java.md` | +| .NET SDK-specific Worker construction and options, package and imports, API inspection, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, graceful shutdown, OpenTelemetry integration, logging, and diagnostic signatures. | `references//sdk-dotnet.md` | +| Add OpenTelemetry observability, collector config, tracing, and the permissions it needs. | `references//observability.md` + the selected `references//sdk-.md` | +| Worker not started, pool not resized, or Workflows not progressing; inspect the WCI. | `references//diagnostics.md` + the selected `references//sdk-.md` (+ `references/concepts.md`) | +| Long-running Activities and timeout relationships. Isolate Activities from resource exhaustion. | `references//constraints.md` (+ `references/concepts.md`, the selected `references//sdk-.md`) | +| How long does a Worker live? What bounds an Activity? What does this provider pin or disable? What differs from another provider? | `references//constraints.md` | ## Out of Scope diff --git a/references/aws-lambda/constraints.md b/references/aws-lambda/constraints.md new file mode 100644 index 0000000..27dff8a --- /dev/null +++ b/references/aws-lambda/constraints.md @@ -0,0 +1,57 @@ +# AWS Lambda — execution-model constraints + +Consequences of Lambda's execution model: **Temporal invokes a function per unit of work, and the Worker exits when the invocation ends.** Everything here follows from that, and none of it generalizes to a provider that keeps long-lived instances alive. + +This file is the provider "diff surface." To understand what changes when moving between compute providers, compare this file with the equivalent one for the other provider — the workflow in `SKILL.md` stays the same; these constraints do not. + +## Worker lifetime is one invocation + +A Worker starts, connects, polls until its shutdown deadline buffer, drains, and exits. The invocation is the Worker's *lifetime*, not one unit of work: a single invocation can serve many Workflow and Activity Tasks from however many executions arrive in its window. Billing follows the invocation, not the work done inside it. + +## Set the invocation deadline high enough + +Providers often default to a very short timeout — Lambda's default is 3 seconds. If the first invocation times out before the Worker registers the Task Queue, the binding is never created and the Worker is never invoked again. → `setup.md` for the exact defaults, the per-SDK examples, and the GB-seconds trade-off behind choosing a value. + +## Tune the timeout triple together for long-running Activities + +The worker stop timeout controls how long the Worker waits for in-flight Tasks after it stops polling; the shutdown deadline buffer controls how long before the invocation deadline it stops polling. + +1. worker stop timeout > longest Activity runtime +2. shutdown deadline buffer > worker stop timeout + shutdown hook time +3. invocation deadline > longest Activity runtime + shutdown deadline buffer + +Raising one alone does not help. Raising only the shutdown deadline buffer makes the Worker stop polling earlier but gives in-flight Activities no more time; raising only the worker stop timeout doesn't make it stop polling earlier, so the provider may terminate the Worker first. + +Worked example: a longest Activity runtime of 5 minutes with 3 seconds of shutdown hooks means a worker stop timeout above 5 minutes, a shutdown deadline buffer above 303 seconds, and an invocation deadline of at least 10 minutes 3 seconds. + +*Symptom of getting this wrong:* Activities abandoned mid-execution and retried on a later invocation. + +If the longest Activity exceeds half the maximum invocation deadline, recommend Activity Heartbeats. → `../concepts.md`, `sdk-.md`. + +## Activities are bounded by the invocation limit + +An Activity must finish within the invocation deadline minus the shutdown deadline buffer. Workflow duration is unbounded and can span many invocations. Flag Activities that approach the provider's limit early — Lambda's ceiling is 15 minutes. + +**An Activity that cannot fit needs a different hosting strategy**, not a larger timeout: a long-lived Worker on a separate Task Queue, or a compute provider without a per-invocation ceiling. → `../concepts.md`. + +## Eager Activities are always disabled + +Every SDK's Lambda Worker package sets this and it cannot be overridden, because eager Activity execution requires a persistent connection that per-invocation Workers don't maintain. Don't suggest it as an optimization. → `sdk-.md` for the per-SDK setting names. + +## Pitfalls specific to this execution model + +These are the invocation-shaped members of the pitfall list in `SKILL.md`; the rest apply to any provider. + +1. **Failed first invocation.** When a version is created, the WCI invokes the Worker once to validate. If that invocation fails — missing env vars, bad TLS/auth config, missing dependencies, or an invocation deadline too short for the Worker to start and register the Task Queue — the Worker never connects, never polls, the binding is never created, and the Worker is never automatically invoked again. *Fix:* diagnose by manually invoking the function, and confirm the invocation deadline is set high. A successful manual invoke also establishes the binding. → `diagnostics.md`. + +2. **Timeout tuning mismatch.** See the timeout triple above. *Fix:* tune the three values together. + +3. **Invoke permission scoped to a single build.** *Symptom:* the deployment works, then the *next* release cannot be invoked, with an error that looks like a connection or configuration problem rather than a permissions one. *Cause:* the grant named one immutable build, and the new release is a different resource. *Fix:* scope the grant to cover the base function ARN **and** its published versions (`function:name` and `function:name:*` — the wildcard form does not cover the unqualified ARN). → `iam.md`. + +## What does *not* follow from this model + +Stated explicitly, because these are easy to over-generalize from Lambda: + +- **"Serverless Worker" does not imply a per-invocation lifetime.** A provider that scales a pool of long-lived instances is still a Serverless Worker driven by the same WCI, with the same Worker Deployment Versioning, and none of the constraints above apply to it in the same form. +- **The shutdown deadline buffer is a property of the Lambda Worker packages**, not of Temporal. +- **Connection-per-invocation is a Lambda property.** Anything reasoning from "the connection is not persistent" — eager Activities being the example above — needs rechecking against a provider that holds the connection for an instance's lifetime. diff --git a/references/aws-lambda/setup.md b/references/aws-lambda/setup.md index 30083bd..94016b4 100644 --- a/references/aws-lambda/setup.md +++ b/references/aws-lambda/setup.md @@ -30,32 +30,23 @@ Steps 4–6 and the CLI troubleshooting paths use the `temporal` CLI. Install it ```bash export TEMPORAL_ADDRESS="..tmprl.cloud:7233" export TEMPORAL_NAMESPACE="." -export TEMPORAL_API_KEY="" +printf 'Temporal API key: ' >&2 +IFS= read -r -s TEMPORAL_API_KEY +printf '\n' >&2 +export TEMPORAL_API_KEY ``` -or configure a profile and pass `--profile prod` on each command: +An existing profile or environment is also valid; pass `--profile prod` or `--env prod` on each command. Do not create or update its API key with `config set --value` or `env set --value` — the value would be exposed in shell history and process arguments. If no credential is already configured, use the private environment-variable prompt above. -```bash -temporal --profile prod config set --prop address --value "..tmprl.cloud:7233" -temporal --profile prod config set --prop namespace --value "." -temporal --profile prod config set --prop api_key --value "" -``` - - -or configure an environment and pass `--env prod` (or set `TEMPORAL_ENV`): - -```bash -temporal env set --env prod --key address --value "..tmprl.cloud:7233" -temporal env set --env prod --key namespace --value "." -temporal env set --env prod --key api-key --value "" -``` - -**Do not assume which of the three a user has, and do not migrate them.** `--env` (YAML, `temporal env`) is the long-standing mechanism; `--profile` (TOML, `temporal config`) is newer and the CLI still marks it EXPERIMENTAL. Both are supported — work with whichever is already configured. Read the existing values rather than asking the user to re-enter them: +**Do not assume which mechanism a user has, and do not migrate them.** `--env` (YAML, `temporal env`) is the long-standing mechanism; `--profile` (TOML, `temporal config`) is newer and the CLI still marks it EXPERIMENTAL. Inspect only the non-secret properties needed for the deployment; a broad `env get` or `config get` can print stored credentials: ```bash -temporal env get --env prod # --env mechanism -temporal config get --prop address # --profile mechanism +temporal env get --env prod --key address +temporal env get --env prod --key namespace +temporal --profile prod config get --prop address +temporal --profile prod config get --prop namespace ``` + - For Temporal Cloud the Namespace is the fully-qualified `.`, not the bare name. - Supplying an API key auto-enables TLS; no cert flags are needed for API-key auth. @@ -72,7 +63,7 @@ If this fails with an auth error, note first that this is a **frontend** call | | Control plane (accounts, Namespaces, API keys) | Namespace frontend (Workflows, Worker Deployments) | |---|---|---| | Interactive | `tcld login` | `temporal ...` with address + namespace | -| Headless | `--api-key` / `TEMPORAL_CLOUD_API_KEY` | `TEMPORAL_API_KEY` | +| Headless | `TEMPORAL_CLOUD_API_KEY` | `TEMPORAL_API_KEY` | **Use `tcld` for every Temporal Cloud control-plane operation** — accounts, Namespaces, API keys, users, service accounts. Do not use the unified CLI's `temporal cloud …` subcommands for them. @@ -85,13 +76,7 @@ Two `tcld` mechanics worth knowing before you run it in an agent shell: - `tcld login --disable-pop-up` prints the URL instead of opening a browser. Auto-open is unreliable over SSH, in containers, and in remote sessions, and the user needs the URL in the conversation either way. - `tcld` prompts for confirmation before mutating operations. Non-interactively, pass the global `--auto_confirm` (note the underscore) or set `AUTO_CONFIRM=true`, then read the resulting state back — without it the command exits clean having changed nothing. -**Go to the API key first.** It requires no CLI login, no browser handshake, and works on every account type: - -```bash -export TEMPORAL_ADDRESS="..tmprl.cloud:7233" -export TEMPORAL_NAMESPACE="." -export TEMPORAL_API_KEY="" -``` +**Go to the API key first.** It requires no CLI login, no browser handshake, and works on every account type. Use the private environment-variable prompt above. Have the user create the key in the Cloud UI, signing in however they normally do, and confirm the address against the endpoint shown on the Namespace page — some Namespaces have regional endpoints that do not follow the pattern above. Never ask them to paste the key into the conversation. @@ -100,10 +85,9 @@ Have the user create the key in the Cloud UI, signing in however they normally d ```bash tcld namespace list # full Namespace objects — every name with its region and endpoint tcld namespace get -n # one Namespace -tcld apikey create --name --duration ``` -`apikey create` mints a key for the calling user and creates a long-lived credential in their account — offer it and get explicit approval, never silently. `tcld` is not guaranteed present: check `command -v tcld`, and read `tcld --help` for the flags you are about to pass. +`tcld` is not guaranteed present: check `command -v tcld`, and read `tcld --help` for the flags you are about to pass. Create API keys in the Cloud UI so their values never enter the agent transcript. **When a control-plane login fails, stop — do not debug it, retry it, or install another CLI.** Some accounts cannot complete a `tcld` login at all, and no flag, plugin upgrade, or alternate CLI changes that. Retrying burns turns without converging, and the browser path below reaches the same end state anyway. @@ -139,9 +123,9 @@ See the selected SDK reference's **Build and package** section. **A freshly created execution role may not be assumable immediately.** `create-function` can fail with an assume-role / "cannot be assumed by Lambda" error because of IAM propagation delay. Wait a few seconds and retry; it is not a policy error, so do not start rewriting the trust policy. -**Operator CLI config does not reach the function.** All three CLI mechanisms above — exported `TEMPORAL_*` variables, `--profile`, and `--env` — configure the `temporal` CLI on the operator's machine only. The function reads its own environment, set by the `--environment` block below (or a secret store). A user with a working `--env prod` or `--profile prod` still needs every value written into that block; nothing is inherited. Treat their CLI configuration as the *source* of the values, not a substitute for setting them. +**Operator CLI config does not reach the function.** All three CLI mechanisms above — exported `TEMPORAL_*` variables, `--profile`, and `--env` — configure the `temporal` CLI on the operator's machine only. The function reads its own environment, set by the `--environment` block below (or a secret store). A user with a working `--env prod` or `--profile prod` still needs every value written into that block; nothing is inherited. Use the CLI configuration only for non-secret values, and collect the API key through the private prompt above. -**Resolve the values before building the block, and check they are not empty.** The heredoc below expands shell variables, which hold values only under the env-var mechanism. Under `--env` or `--profile` they are unset, and an unset variable expands to an empty string: the JSON stays valid, `create-function` succeeds, and the function deploys with `"TEMPORAL_ADDRESS":""` — failing at first invocation with a connection error that looks nothing like its cause. Populate them from whichever mechanism the user actually has (`temporal env get --env prod`, `temporal config get --prop address`), then guard: +**Resolve the values before building the block, and check they are not empty.** The heredoc below expands shell variables, which hold values only under the env-var mechanism. Under `--env` or `--profile` they are unset, and an unset variable expands to an empty string: the JSON stays valid, `create-function` succeeds, and the function deploys with `"TEMPORAL_ADDRESS":""` — failing at first invocation with a connection error that looks nothing like its cause. Populate the address and Namespace from the non-secret lookups above, and have the user set `TEMPORAL_API_KEY` with the private prompt; never retrieve a stored key into the agent transcript. Then guard: ```bash : "${TEMPORAL_ADDRESS:?resolve this before deploying}" diff --git a/references/aws-lambda/versioning.md b/references/aws-lambda/versioning.md index 1959d63..2d4f99d 100644 --- a/references/aws-lambda/versioning.md +++ b/references/aws-lambda/versioning.md @@ -54,6 +54,18 @@ The command prints the `FunctionArn` for the new version, for example `arn:aws:l For development or non-critical workloads, you can skip `publish-version` and use an unqualified ARN to iterate faster. +### What an unqualified ARN costs + +An unqualified ARN (no version suffix) points at `$LATEST`, which changes on every redeploy. Deploying replay-unsafe code that way causes non-determinism errors for in-flight Workflows, **even ones annotated Pinned**. + +Pinned and Auto-Upgrade control how Workflows move between Worker Deployment Versions in Temporal; neither changes how a version targets Lambda. Both expect a qualified ARN naming one immutable function version. + +| Versioning Behavior | With versioned Lambda ARN | Without versioned Lambda ARN | +|---|---|---| +| **Pinned** | Existing Workflows stay on their original Lambda function version until they complete. | Existing Workflows stay on their original Worker Deployment Version, but the underlying Lambda code has already changed since `$LATEST` updated at redeploy. The new code must be replay-compatible. | +| **Auto-Upgrade** | Existing Workflows move to the new Worker Deployment Version and its new Lambda function version at the next Workflow Task after you move the Current Version. | The Lambda redeploy already changed the code for all versions. Setting the Current Version only changes routing, not which code runs. | + + To roll back, revert the Temporal Current Version with `temporal worker deployment set-current-version`. The previous Worker Deployment Version still points at its original Lambda function version and is ready to receive traffic again. ```bash diff --git a/references/concepts.md b/references/concepts.md index f5d141d..d7b58f4 100644 --- a/references/concepts.md +++ b/references/concepts.md @@ -6,22 +6,30 @@ **AWS Lambda — Public Preview since July 30, 2026.** Open to all Temporal Cloud customers. There is no access request, no support ticket, and no manual toggle to enable: a customer selects "AWS Lambda (Public Preview)" as the compute provider in the UI and sets up their Worker Deployment directly. Never route a user to support to "get access" for Lambda. -AWS Lambda is the only compute provider this skill supports. Do not adapt the Lambda material to any other provider. +**GCP Cloud Run — Pre-release.** Its APIs may change in backwards-incompatible ways, and **access is gated**: the customer creates a support ticket or contacts their account team. Unlike Lambda, routing a user to support *is* correct here. + +Those are the two supported providers. Do not adapt either one's material to a third. + +**Do not carry facts between them.** Anything about Worker lifetime, Activity duration bounds, timeouts, packaging, or tuning is provider-specific — see `/constraints.md`. This page names the provider whenever the action differs: Lambda is invoked; Cloud Run is resized. Public Preview is not General Availability. APIs are still evolving and may be subject to backwards-incompatible changes between versions — pin SDK and CLI versions for anything long-lived, and read the installed package's real API surface rather than writing from memory. ## What is a Serverless Worker? -A Serverless Worker is a Temporal Worker that runs on serverless compute instead of a long-lived process. -There is no always-on infrastructure to provision or scale. Temporal invokes the Worker when Tasks arrive on a Task Queue, and the Worker shuts down when the work is done. +A Serverless Worker is a Temporal Worker whose compute lifecycle is controlled by Temporal instead of by an independently operated Worker fleet. +There is no always-on compute capacity to maintain: Temporal starts capacity when needed and can return it to zero when idle. The provider resource itself—a Lambda function or Cloud Run Worker Pool—continues to exist. + +**"Starts" means different things per provider.** On AWS Lambda, Temporal invokes a function per unit of work and the Worker exits when that invocation ends. On GCP Cloud Run, Temporal resizes a pool of long-lived instances, each running an ordinary Worker that polls for its whole lifetime. Both scale to zero when idle; almost nothing else about their lifecycles is the same. -A Serverless Worker uses the same Temporal SDKs as a traditional long-lived Worker. It registers Workflows and Activities the same way. The difference is in the lifecycle: instead of the Worker starting and polling continuously, Temporal invokes the Serverless Worker on demand, the Worker starts, processes available Tasks, and then shuts down. +A Serverless Worker uses the same Temporal SDKs as a traditional long-lived Worker. It registers Workflows and Activities the same way. + +What changes is the lifecycle, and only on Lambda does it change much: instead of polling continuously, the Worker is invoked on demand, starts, processes available Tasks, and shuts down — which is why Lambda needs a dedicated serverless Worker package (`aws-lambda/sdk-.md`). **On Cloud Run the Worker code is unchanged from a long-lived Worker**; the only addition is Worker Versioning, and there is no Cloud Run Worker package at all. Serverless Workers require Worker Versioning. Each Serverless Worker must be associated with a Worker Deployment Version that has a compute provider configured. Each Workflow must have an `AutoUpgrade` or `Pinned` versioning behavior, set per-Workflow or as a Worker-level default. -## How Serverless invocation works +## How Temporal controls Serverless Workers With long-lived Workers, the Worker process starts, connects to Temporal, and polls a Task Queue for work. Temporal does not need to know anything about the Worker's infrastructure. @@ -32,7 +40,7 @@ With Serverless Workers, Temporal starts the Worker. One WCI Workflow runs per Worker Deployment Version that has a compute provider configured. The WCI runs in the same Namespace as your Worker Deployment. -The WCI responds to two triggers: sync match failures and Task Queue backlog. When either trigger fires, the WCI produces a scaling action, such as invoking the configured compute provider (for example, calling AWS Lambda's `InvokeFunction` API) to start new Workers. +The WCI responds to sync match failures and periodically reads Task Queue metrics. It turns those inputs into an action compatible with the provider: invoke a Lambda function, or update a Cloud Run Worker Pool's manual instance count. You can list WCI Workflows in your Namespace: @@ -54,43 +62,41 @@ temporal workflow show \ ``` -### Invocation flow +### Shared Task routing -The invocation flow works as follows: +1. A Task is submitted, for example by `StartWorkflow` or `ScheduleActivity`. +2. Matching attempts to route it directly to an available Worker in a sync match. +3. If no Worker is available, Matching adds the Task to the backlog and signals the WCI for that Worker Deployment Version. +4. The WCI asks the version's scaling algorithm for an action and applies it through the configured provider. -1. A Task is submitted (for example, `StartWorkflow` or `ScheduleActivity`). -2. The Matching Service attempts to route the Task directly to an available Worker (a sync match). -3. If a Worker is available, the Task is routed to that Worker. -4. If no Worker is available (sync match fails), the Matching Service pushes a signal to the WCI, and the WCI invokes the configured compute provider. -5. The Serverless Worker starts, creates a Temporal Client, and begins polling the Task Queue. -6. The Worker processes available Tasks until it exits (see Worker lifecycle). +After that point the providers diverge: -Each invocation is independent. The Worker creates a fresh client connection on every invocation. There is no connection reuse or shared state across invocations. +- **AWS Lambda:** the WCI invokes the function. Each invocation creates a Worker and client connection, processes available Tasks, and shuts down. Invocations do not share a connection or in-memory state. +- **GCP Cloud Run:** the WCI increases the Worker Pool's manual instance count. Cloud Run starts an instance whose ordinary Worker connects once and polls for the instance's lifetime. The WCI later lowers the count as demand drains. ## Autoscaling -The WCI automatically scales Serverless Workers based on Task Queue signals. When Tasks arrive and no Worker is available, the WCI invokes new Workers. When the Tasks are done, Workers exit and scale to zero. - -The WCI uses two signals to decide when to invoke new Workers: +The WCI automatically scales Serverless Workers from Task Queue signals and metrics. The resulting action depends on the provider. ### Sync match failure -When a Task is submitted, the Matching Service attempts to route it directly to an available Worker. If no Worker is available, the sync match fails, and the Matching Service pushes a signal to the WCI. The WCI then invokes a new Worker. This is the primary scaling path. +When a Task is submitted, Matching attempts to route it directly to an available Worker. If no Worker is available, the sync match fails and Matching signals the WCI. Lambda's no-sync algorithm can invoke another function; Cloud Run's rate-based algorithm can immediately increase the planned pool size, subject to its cooldown and maximum. Because the Matching Service pushes match failures to the WCI as they happen rather than the WCI polling on a timer, latency stays low and scaling is responsive. ### Task Queue backlog -The WCI monitors Task Queue metadata to determine whether pending Tasks exist without enough Workers to process them. If there are Tasks on the queue and not enough Workers, the WCI invokes additional Workers. +The WCI periodically reads version-level Task Queue arrival rate, dispatch rate, and backlog. Cloud Run's rate-based algorithm uses these metrics to calculate a desired instance count and explicitly resizes the pool; the periodic path also scales the pool down, including to zero. Lambda Workers instead end with their invocations and do not use this worker-set sizing model. ## Scaling with long-lived Workers -Serverless Workers can share a Task Queue with long-lived Workers. Because Serverless Workers are only invoked on sync match failure, Serverless Workers only pick up Tasks that no long-lived Worker was available to handle. In practice, the Serverless Workers act as spillover capacity for the long-lived fleet. - -**Warning:** If you configure Serverless and long-lived Workers on the same Task Queue, do not enable dynamic scaling on the long-lived Workers. The two groups cannot coordinate their scaling behavior. If both scale dynamically, the long-lived Workers may scale up to handle the same Tasks that Temporal is simultaneously invoking Serverless Workers for, leading to unnecessary invocations and unpredictable scaling. +- **AWS Lambda:** a Lambda Worker can share a Task Queue with a fixed long-lived fleet and act as spillover when sync matching finds no available poller. Do not dynamically scale the long-lived fleet as well; the two scaling systems cannot coordinate. +- **GCP Cloud Run:** use a separate Task Queue from any independently managed long-lived fleet. The rate-based WCI scaler reads the full version-level arrival, dispatch, and backlog metrics and cannot subtract work handled by the other fleet, so sharing provisions duplicate capacity. → `gcp-cloud-run/constraints.md`. ## Worker lifecycle +**This section describes providers that invoke per unit of work, such as AWS Lambda.** On a provider that scales a pool of long-lived instances, such as GCP Cloud Run, an instance connects once and polls for its whole lifetime: there are no per-invocation phases, and none of the tuning below applies. → `/constraints.md`. + A single Serverless Worker invocation has three phases: init, work, and shutdown. ### Init phase @@ -107,52 +113,23 @@ The Worker stops polling, waits for in-flight Tasks to finish, and runs any shut ### Tuning for long-running Activities -If your Worker handles long-running Activities, set these three values together: - -- **Worker stop timeout > longest Activity runtime.** Gives in-flight Activities enough time to finish after polling stops. -- **Shutdown deadline buffer > Worker stop timeout + shutdown hook time.** Ensures the drain and any shutdown hooks complete before the compute provider terminates the environment. -- **Invocation deadline > longest Activity runtime + shutdown deadline buffer.** Set on the compute provider to give each invocation enough total runtime. - -If your longest-running Activity runs longer than half the maximum invocation deadline, use Activity Heartbeats to record the state of the Activity execution so that the next retry can pick up where it left off. - -Example: if your longest Activity runtime is 5 minutes, and your shutdown hooks take 3 seconds, set the Worker stop timeout to more than 5 minutes, and the shutdown deadline buffer to more than 303 seconds (5 minutes + 3 seconds). Set your invocation deadline to at least 10 minutes and 3 seconds. - -The Worker stop timeout controls how long the Worker waits for in-flight Tasks to finish after it stops polling. The shutdown deadline buffer controls how much time before the invocation deadline the Worker stops polling for Tasks. - -Raising only the shutdown deadline buffer makes the Worker stop polling earlier, but does not give in-flight Tasks any more time to complete. - -Raising only the Worker stop timeout does not make the Worker stop polling earlier, which means the compute provider might terminate the Worker before the full stop timeout completes. +Three values must be tuned together — worker stop timeout, shutdown deadline buffer, and invocation deadline — and raising one alone does not help. The exact relationships, a worked example, the failure symptom, and the Activity Heartbeat threshold are provider-specific. → `/constraints.md`. ## Failure handling Serverless Workers rely on Temporal's standard retry and timeout semantics to recover from failures. -### Worker crash - -If a Worker invocation crashes (out of memory, unhandled exception, etc.): +### Worker crash or instance termination -- The Activity Timeout fires after the configured duration. -- Temporal retries the Activity on a different Worker invocation. -- No manual intervention is required. +If a Worker crashes or its compute is terminated, the in-flight Task is not acknowledged. Temporal applies the configured timeout and retry policy, and another Worker can receive the retry. On Lambda that means another invocation; on Cloud Run it means another running or replacement pool instance. Activity Heartbeats preserve progress for long-running work that can be interrupted. -### Provider concurrency limit +### Provider capacity limit -If the compute provider's concurrency limit is reached (for example, AWS Lambda account concurrency): - -- Further invocations from the WCI fail. -- Tasks remain in the Task Queue backlog. No data loss occurs. -- Processing slows until concurrency frees up. +If the provider cannot add capacity—Lambda account concurrency, Cloud Run regional quotas, or the configured Cloud Run `max_count`—Tasks remain in the Task Queue backlog without data loss and processing slows until capacity becomes available. The provider-side symptom differs: Lambda invocations are throttled or rejected, while a Cloud Run pool stops growing or its update fails. ### Resource exhaustion across Activity slots -By default, a single Worker invocation may run multiple Activity slots. A crash or resource exhaustion in one Activity can affect other Activities running in the same invocation. - -To isolate Activities from each other: - -- Split Workflow and Activity Workers into separate compute functions. -- Set Activity slots to 1 per invocation. - -With single-slot configuration, each Activity gets a dedicated execution environment. +A Worker process may run multiple Activity slots, so a crash or resource exhaustion in one Activity can affect other Activities in that same process. On Lambda, split Workflow and Activity Workers into separate functions or use one Activity slot per invocation for execution-environment isolation. On Cloud Run, size instance resources and Worker concurrency together; one slot limits concurrency within an instance but does not turn the long-lived instance into a per-Activity environment. → `/constraints.md`. ## Constraints @@ -160,39 +137,27 @@ With single-slot configuration, each Activity gets a dedicated execution environ | Constraint | Detail | |---|---| -| Activity duration | Must complete within the compute provider's invocation limit (minus shutdown deadline buffer). For AWS Lambda, the maximum is 15 minutes. | -| Workflow duration | No limit. Workflows of any duration work, regardless of the invocation timeout. A Workflow runs across as many invocations as needed. | -| Worker code | Same Temporal SDK Worker code, using the serverless Worker package for your SDK. | +| Activity duration | On a provider that invokes per unit of work, must complete within its invocation limit minus the shutdown deadline buffer — Lambda's ceiling is 15 minutes. A pool-based provider such as Cloud Run imposes no per-invocation ceiling. → `/constraints.md`. | +| Workflow duration | No compute-provider limit. Workflow state is durable in Temporal and does not depend on one Lambda invocation or Cloud Run instance remaining alive. | +| Worker code | Same Temporal SDK Worker code. On Lambda it runs through that SDK's serverless Worker package; on Cloud Run it is an ordinary long-lived Worker with no extra package. | | Versioning | Worker Versioning is required. Each Workflow must have an `AutoUpgrade` or `Pinned` behavior, set per-Workflow or as a Worker-level default. | ## Worker Versioning with Serverless Workers -Serverless Workers require Worker Versioning, and the compute provider must invoke a stable, immutable build for each Worker Deployment Version. With AWS Lambda, this means aligning two versioning systems: +Serverless Workers require Worker Versioning, and the compute provider must target a **stable, immutable build** for each Worker Deployment Version. That means aligning two versioning systems: - **Temporal Worker Deployment Versions** — identified by deployment name and Build ID. Each Workflow runs against a specific Worker Deployment Version (Pinned) or moves between them on routing changes (Auto-Upgrade). -- **AWS Lambda function versions** — immutable numbered snapshots of your Lambda function code (`1`, `2`, `3`, ...). - -For production workloads, map each Worker Deployment Version to exactly one Lambda function version, and configure the compute provider with the qualified versioned ARN for that Lambda version (for example, `arn:aws:lambda:us-east-1:123:function:my-worker:5`). - -For development or non-critical workloads, you can use an unqualified ARN to iterate without publishing a new Lambda function version each time. - -**Caution:** An unqualified ARN (no version suffix) points at `$LATEST`, which changes on every redeploy. Without a versioned ARN, deploying replay-unsafe code causes non-determinism errors for in-flight Workflows, even for Workflows annotated as Pinned. - -The choice of Pinned or Auto-Upgrade controls how Workflows move between Worker Deployment Versions in Temporal. It does not change how a Worker Deployment Version targets Lambda. Both behaviors expect a versioned ARN that points at one immutable Lambda function version. +- **The provider's own unit of immutability** — a published Lambda function version, pinned by a qualified ARN; or on Cloud Run a dedicated Worker Pool per Build ID, because the compute configuration names a pool and not a revision. Keep a one-to-one mapping between it and the Build ID. -| Versioning Behavior | With versioned Lambda ARN | Without versioned Lambda ARN | -|---|---|---| -| **Pinned** | Existing Workflows stay on their original Lambda function version until they complete. | Existing Workflows stay on their original Worker Deployment Version, but the underlying Lambda code has already changed since `$LATEST` updated at redeploy. The new code must be replay-compatible. | -| **Auto-Upgrade** | Existing Workflows move to the new Worker Deployment Version and its new Lambda function version at the next Workflow Task after you move the Current Version. | The Lambda redeploy already changed the code for all versions. Setting the Current Version only changes routing, not which code runs. | - +**Pointing a Worker Deployment Version at a mutable target causes non-determinism errors for in-flight Workflows, including Pinned ones.** Pinned routes Workflows to a version; it cannot pin code that changed underneath that version. The failure is the same on both providers but is reached differently — on Lambda you have to choose it by registering an unqualified ARN, while on Cloud Run a plain redeploy into a live pool does it — so read the provider file for which action is the dangerous one. → `aws-lambda/versioning.md`, `gcp-cloud-run/versioning.md`. -See `aws-lambda/versioning.md` for the step-by-step `aws lambda publish-version` workflow and `aws-lambda/setup.md` (Step 4) for how to configure the compute provider with a versioned ARN. +Pinned or Auto-Upgrade controls how Workflows move between Worker Deployment Versions in Temporal. It does not change how a Worker Deployment Version targets the provider; both behaviors expect one immutable build per version. ## Compute providers -A compute provider is the configuration that tells Temporal how to invoke a Serverless Worker. The compute provider is set on a Worker Deployment Version and specifies the provider type, the invocation target, and the credentials Temporal needs to trigger the invocation. +A compute provider is the configuration that tells Temporal how to control Serverless Worker capacity. It is set on a Worker Deployment Version and specifies the provider type, compute target, and credentials Temporal needs to invoke a function or resize a worker set. -For example, an AWS Lambda compute provider includes the Lambda function ARN and the IAM role that Temporal assumes to invoke the function. +For example, an AWS Lambda compute provider includes the Lambda function ARN and the IAM role that Temporal assumes to invoke the function; a Cloud Run compute provider names the project, region, and Worker Pool, plus the service account Temporal impersonates to scale it. Compute providers are only needed for Serverless Workers. Traditional long-lived Workers do not require a compute provider because the Worker process lifecycle is not managed by the Temporal server. @@ -203,15 +168,16 @@ Compute providers are only needed for Serverless Workers. Traditional long-lived | Provider | Description | |---|---| | AWS Lambda | Temporal assumes an IAM role in your AWS account to invoke a Lambda function. | +| GCP Cloud Run | Temporal impersonates a service account in your Google Cloud project to scale a Worker Pool. | ## Why use Serverless Workers? -- **Reduce operational overhead.** No always-on infrastructure to manage and no autoscaling policies to tune. Temporal and the compute provider handle invocation and scaling. -- **Get started faster.** Deploying a Worker is as simple as deploying a function. No Kubernetes, container orchestration, or scaling strategy required. -- **Scale automatically.** The compute provider handles scaling natively. When traffic drops, instances scale down. When there is no work, there is no compute running. -- **Pay only for what you use.** Workers run only when Tasks are available. For low or intermittent volume workloads, this pay-per-invocation model can significantly reduce compute costs. +- **Reduce operational overhead.** Temporal drives capacity from Task Queue demand: invoking Lambda or setting the Cloud Run Worker Pool size. +- **Avoid managing a continuously provisioned fleet.** Deploy a function or a Worker Pool without building a separate autoscaling control plane. +- **Scale automatically.** Capacity grows with demand and can return to zero when idle. +- **Pay only while compute runs.** Lambda bills invocations; Cloud Run bills running pool instances. Both can reduce idle cost for low or intermittent workloads. ## When to use Serverless Workers @@ -229,16 +195,17 @@ May not be ideal when: -- Activities are long-running and cannot be interrupted. AWS Lambda has a 15-minute execution limit. Activities that run longer and cannot be broken into smaller steps need a different hosting strategy or a provider with longer limits. +- Activities are long-running and cannot be interrupted, on a provider with a per-invocation ceiling — Lambda's is 15 minutes. Activities that run longer and cannot be broken into smaller steps need a different hosting strategy, or a provider without that ceiling. - Workloads require sustained high throughput. Long-lived Workers on dedicated compute may be more cost-effective and performant. -- You need persistent connections. Some features require a persistent connection between the Worker and Temporal, which serverless invocations do not maintain. +- You need persistent connections and the provider invokes per unit of work. Some features require a persistent connection between the Worker and Temporal, which per-invocation Workers do not maintain; a pool-based provider holds one for the instance's lifetime. ## How Serverless Workers compare to long-lived Workers -| | Long-lived Worker | Serverless Worker | -|---|---|---| -| **Lifecycle** | Long-lived process that runs continuously. | Invoked on demand. Starts and stops per invocation. | -| **Scaling** | You manage scaling (Kubernetes HPA, instance count, etc.). | Temporal invokes additional instances as needed, within the compute provider's concurrency limits. | -| **Connection** | Persistent connection to Temporal. | Fresh connection on each invocation. | +| | Independently managed Worker | AWS Lambda Serverless Worker | GCP Cloud Run Serverless Worker | +|---|---|---|---| +| **Lifecycle** | Long-lived process; you decide when it runs. | Short-lived Worker created for each function invocation. | Ordinary long-lived Worker inside each pool instance; WCI controls the instance count. | +| **Scaling** | You manage replicas or instances. | WCI invokes functions after Task Queue signals. | WCI explicitly resizes the Worker Pool from signals and periodic queue metrics. | +| **Connection** | Persistent for the process lifetime. | Fresh connection per invocation. | Persistent for the pool instance lifetime. | +| **Scale to zero** | Only if you build and operate it. | Invocations end when work drains. | WCI lowers the pool's manual instance count to zero. | diff --git a/references/gcp-cloud-run/constraints.md b/references/gcp-cloud-run/constraints.md new file mode 100644 index 0000000..178f503 --- /dev/null +++ b/references/gcp-cloud-run/constraints.md @@ -0,0 +1,81 @@ +# GCP Cloud Run — execution-model constraints + + + +Consequences of Cloud Run's execution model: **Temporal resizes a pool of long-lived instances, scaling it to zero when there is no work.** Temporal does *not* invoke your Worker per Task. + +This file is the provider "diff surface." Compare it with `../aws-lambda/constraints.md` to see what changes between providers — the workflow in `SKILL.md` is the same; these constraints are not. + +## Worker lifetime is an instance, not an invocation + +Each pool instance runs **standard long-lived Worker code**: it connects, registers Workflows and Activities, and polls the Task Queue for its whole lifetime. There is no handler, no per-Task lifecycle, and **no serverless Worker package** — the SDK-specific packages in `../aws-lambda/sdk-.md` are AWS Lambda only. Some SDKs add optional Cloud Run conveniences; none are required. + +The WCI controls how many instances run; each instance manages its own polling and Task processing. + +## None of Lambda's timing constraints apply + +Explicitly, so they are not carried across: + +- **No invocation deadline.** Nothing bounds how long a Worker lives except scale-in. +- **No shutdown deadline buffer.** That is a property of the Lambda Worker packages, not of Temporal. +- **No timeout triple to tune.** Two of its three values do not exist here. +- **Activities are not bounded by an invocation limit.** An Activity too long for Lambda's 15-minute ceiling is a reason to choose Cloud Run. +- **Eager Activities are not disabled by the platform.** Lambda's packages force this off because a per-invocation Worker holds no persistent connection; a pool instance holds its connection for its lifetime, so that reasoning does not transfer. Confirm against your SDK rather than assuming either way. + +Cloud Run still sends `SIGTERM` during scale-in and can send `SIGKILL` ten seconds later. Handle `SIGTERM` and configure the SDK's graceful-shutdown timeout below that window; this is separate from Lambda's deadline buffer and timeout triple. → `sdk-.md`. + +## What bounds an Activity instead: scale-in + +**The WCI decides when to remove an instance from Task Queue activity, not from what any individual instance is doing.** It does not track how long an instance has been running or whether it is mid-Activity, so the instance Cloud Run stops may be one that is still executing work. + +Graceful shutdown lets short work drain but cannot guarantee an Activity will finish. **Use Activity Heartbeats** so interrupted work resumes from its last recorded progress instead of restarting. + +*Symptom of ignoring this:* Activities failing partway and retrying from the beginning, correlated with the pool shrinking. + +## Autoscaling behavior + + + +The WCI combines two mechanisms: + +- **Immediate** — bring up instances when a Task arrives and no Worker is free to take it (a sync match failure). Absorbs bursts without waiting for the evaluation cycle. +- **Periodic** — rate-based re-sizing. The WCI measures how fast Tasks arrive and how fast one Worker processes them, computes the instance count needed, and applies it through the Cloud Run admin API. + +It sizes to a **target utilization of 80% by default** rather than loading every Worker fully, so there is headroom to take new Tasks immediately, and adds instances on top when a backlog exists. + +**Scale-in is deliberately more conservative than scale-out:** it holds capacity while sync match failures are still occurring and applies a cooldown before reducing the pool. With no work, it can scale to zero; the next sync match failure or backlog scales it back up. + +The scaler defaults are **minimum `0`, maximum `30`, initial count `0`, and target utilization `0.8`**. Configure them in the version's Scaling and Lifecycle settings or with the Temporal CLI. The four CLI flags are coupled: omit all four to use the defaults, or provide `--gcp-cloud-run-min-instances`, `--gcp-cloud-run-max-instances`, `--gcp-cloud-run-initial-instances`, and `--gcp-cloud-run-utilization-target` together. A partial group is rejected. + +An initial count and minimum of zero do not suppress registration. The rate-based algorithm temporarily requests at least one instance when the version is registered so its Task Queues can bind, then normal scaling can return the pool to zero. A pool that later stops growing under backlog is either at its configured maximum or at a regional Cloud Run quota. + +## One Worker Pool per Worker Deployment Version + +**The compute configuration names a project, region, and pool — not a revision.** Temporal runs whichever revision the pool serves at the time, which ties a pool to a single build. A new build needs a **new pool**, and the Build ID belongs in the pool name to keep that mapping visible. + +Keep an older version's pool in place while Pinned Workflows are still running on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. + +**The mutable-build hazard, in its Cloud Run form:** deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run promotes it to every instance by default. The version does not change, but the code behind it does — causing non-determinism errors for in-flight Workflows, **including Pinned ones**. This is the same failure as pointing Lambda at a mutable `$LATEST`, reached through revisions instead of ARNs. → `versioning.md`. + +## Do not share a Task Queue with long-lived Workers + +Stronger than the Lambda guidance, and for a different reason. On Lambda the two scaling loops fight; here **the pool scales up to cover the Task Queue's full workload even when long-lived Workers are already handling all of it**, so you run and pay for duplicate capacity. + +The WCI sizes the pool from the rate of Tasks arriving on the version's Task Queues, and nothing in that measurement accounts for the long-lived Workers. Sync matching to a long-lived Worker suppresses the *immediate* scale-up, but the periodic re-sizing scales the pool up regardless. Fixing poller counts on the long-lived side does not help — use separate Task Queues. + +## Pre-release status + +Cloud Run support is **Pre-release, and its APIs may change in backwards-incompatible ways.** Access is not open: the user must create a support ticket or contact their account team. Confirm access exists before planning a deployment — this is a gate no amount of correct configuration gets past. + +## Provider-side issue worth knowing + +Google currently reports **high deployment latency creating or updating Cloud Run resources in some regions, including `us-central1`**, and recommends deploying elsewhere while it is open. If pool operations are slow, check [Cloud Run known issues](https://cloud.google.com/run/docs/known-issues) before assuming misconfiguration. + +## What does *not* follow from this model + +- **"Serverless Worker" does not imply standard Worker code.** On Lambda you write a handler against a provider-specific package; here you write an ordinary Worker. Advice about handlers, configure callbacks, or tuned package defaults does not transfer. +- **Scale-to-zero is not per-Task.** An idle pool costs nothing, but a running instance is billed for its lifetime, not per unit of work. +- **A passing Validate Connection is weaker here than on Lambda.** It exercises only the read permission and starts no instance. → `diagnostics.md`. diff --git a/references/gcp-cloud-run/diagnostics.md b/references/gcp-cloud-run/diagnostics.md new file mode 100644 index 0000000..573c96d --- /dev/null +++ b/references/gcp-cloud-run/diagnostics.md @@ -0,0 +1,137 @@ +# GCP Cloud Run — Diagnostics & troubleshooting + + + +## Scaling flow (when working correctly) + + + +1. You deploy the Worker image to a Worker Pool at zero instances. +2. You create a Worker Deployment Version pointing at that pool. This starts a WCI Workflow. +3. An instance starts, the Worker polls, and the server **binds the Task Queue** to the version. +4. The WCI monitors that Task Queue; the Matching Service also signals it when a Task arrives with no free Worker. +5. As work arrives the WCI raises the instance count through the Cloud Run admin API; as it drains, lowers it, possibly to zero. + +**Temporal does not invoke your Worker per Task** — it changes how many instances run. Diagnose accordingly: "was it invoked?" is the wrong first question here. + +## Start here: read the pool's annotations + +This is the single highest-value command, and it has no Lambda equivalent: + +```bash +gcloud run worker-pools describe \ + --region --project --format=yaml +``` + +Three fields under `metadata.annotations`: + +| Field | What it tells you | +|---|---| +| `run.googleapis.com/manualInstanceCount` | The instance count currently requested. `0` means no Worker is running. | +| `run.googleapis.com/scalingMode` | Should be `manual` — the WCI scales by writing the manual instance count. | +| `serving.knative.dev/lastModifier` | Who last changed the pool. **If Temporal has ever scaled it, this is the invoker service account.** | + +**`lastModifier` splits the whole problem in two:** + +- Still the account you deployed with → **Temporal has never successfully written to the pool.** Work through "pool is not scaling up" below. +- The invoker service account → **Temporal is reaching the pool**; the fault is in the Worker. Skip to "instances running but Tasks not completing." + +This is the Cloud Run counterpart of Lambda's "is the Task Queue bound?" checkpoint — one read that eliminates most of the surface. + +## The Worker Pool is not scaling up + +### 1. Validate Connection — and know what it does *not* prove + +Workers → Deployments → select deployment → Actions → **Validate Connection**. For Cloud Run this impersonates the invoker and reads the pool, confirming three things: the compute configuration names a pool that exists, Temporal can impersonate the invoker, and the invoker can read. + +**It starts no instance and does not exercise `run.workerPools.update`.** Version registration is different: its Task Queue bootstrap does update the pool. An invoker with read but not update permission can therefore pass this manual validation, but version registration or a later resize fails. If validation succeeds but registration never changes `lastModifier`, **check the update permission**. + +On failure, check each part of the compute configuration against the pool: + +- **Project, region, pool name.** Temporal addresses the pool as `projects//locations//workerPools/`. **A wrong region reports the pool as not found, identical to a wrong name** — so a "not found" does not tell you which field is wrong. +- **Impersonation.** Temporal's identity needs `roles/iam.serviceAccountTokenCreator` on the invoker. The Terraform module grants this on Cloud; self-hosted grants it to the server's GCP identity. +- **Invoker permissions.** `run.workerPools.get` to read, `run.workerPools.update` to scale. + +→ `iam.md`. + +### 2. Did the registration bootstrap bind the Task Queue? + +```bash +temporal worker deployment describe-version \ + --namespace --deployment-name --build-id --report-task-queue-stats +``` + +The server creates the binding when a Worker running that version connects and polls. **No Task Queues listed means no Worker has polled successfully under this version.** + +Registration is supposed to bootstrap this: the WCI reads the pool, updates its manual instance count to at least one, and Cloud Run starts an instance. An absent binding means that sequence failed or the instance started but did not connect under the expected deployment name and build ID. Inspect the WCI's `ValidateSpec`/registration Activity failure, the pool's `lastModifier` and instance count, and then the pool logs. Do not wait for a first Workflow to repair registration. + +### 3. Is the version current? + +The registration bootstrap does not make the version current. New traffic routes only after the version is current, and a CLI-created version is not current automatically. Verify with `temporal worker deployment describe`. + +**A `set-current-version` run without `--yes` may have done nothing** — it prompts, and non-interactively exits without applying the change, which reads as success. + +### 4. Is the pool at its ceiling? + +If instances are running but the count stops growing while backlog builds: + +- **The maximum defaults to 30.** Raise it in the version's Scaling and Lifecycle settings, or update the existing version from the CLI: + + ```bash + temporal worker deployment update-version-compute-config \ + --namespace \ + --deployment-name \ + --build-id \ + --gcp-cloud-run-min-instances 0 \ + --gcp-cloud-run-max-instances \ + --gcp-cloud-run-initial-instances 0 \ + --gcp-cloud-run-utilization-target 0.8 + ``` + + The four scaler flags must be supplied together, even when changing only the maximum. Choose values appropriate for the version rather than blindly copying this default-shaped example; the initial count must be between the minimum and maximum. +- If the count stalls *below the configured maximum*, check the project's [Cloud Run quotas](https://cloud.google.com/run/quotas) for that region. Cloud Run caps instances and CPU per region regardless of what the WCI requests. + +## Instances are running but Tasks are not completing + +### Read the pool logs + +```bash +gcloud run worker-pools logs read --region --project +``` + +**The pool produces no logs while scaled to zero** — read them while an instance is up. An empty log is not evidence of failure. + +Common errors: + +- **Connection failures** — check `TEMPORAL_ADDRESS` and `TEMPORAL_NAMESPACE` on the pool. Self-hosted: verify network reachability from Cloud Run to the frontend. +- **Missing secrets** — the instance cannot read the API key or TLS material. The **runner** service account needs `roles/secretmanager.secretAccessor` on the secret. That is the account in `spec.template.spec.serviceAccountName`, **not the invoker.** This is the most common consequence of confusing the two. +- **Authentication errors** — key invalid, expired, or without access to the Namespace. +- **`TransportError: … NativeCertsNotFound`** — a Rust-core SDK in a minimal base image with no CA certificates. Documented for TypeScript on Cloud Run; the same class of failure as .NET's `SSL_CERT_FILE` problem on Lambda. Install `ca-certificates` in the image. → `setup.md`. + +### Deployment name and build ID + +Instances start and poll but no Task is ever processed → the name or build ID in the code does not match the version. The Worker polls under a version the WCI does not manage, **so its polls never satisfy the Tasks the WCI is scaling for.** + +Note the different signature from Lambda: there, a mismatch causes rapid repeated invocations. Here there are no invocations to count — you see a running pool, healthy-looking logs, and no progress. + +## Activities interrupted mid-execution + +Activities failing partway and retrying from the beginning, correlated with the pool shrinking, means **scale-in is stopping instances that are still working.** The WCI does not track whether the instance Cloud Run stops is mid-Activity. + +This is expected behavior, not a misconfiguration. Confirm the Worker handles `SIGTERM` and has a non-zero graceful-shutdown timeout below Cloud Run's ten-second termination window; this lets short work drain. Long-running work still needs **Activity Heartbeats** so a retry resumes from its last recorded progress. → `constraints.md`, `sdk-.md`. + +## Rule out a GCP-side cause + +If every check passes, the cause may be in Cloud Run rather than your configuration: + +- [Cloud Run known issues](https://cloud.google.com/run/docs/known-issues) — includes issues affecting how long pool operations take. +- [Google Cloud Service Health](https://status.cloud.google.com/) — active incidents by product and region. + +The usual fix is to wait it out or move region. **Moving region means creating a new pool and updating the compute configuration**, since Temporal addresses a pool by project, region, and name. + +## Never create or manage the WCI + +Unchanged from Lambda: Temporal creates one per Worker Deployment Version with a compute provider, and a running WCI is not evidence that scaling works—it continues-as-new while its Activities fail. Read its history for Activity failures, and prefer the pool's `lastModifier` annotation as the cheapest proof of a successful write. Do not enumerate Cloud Run resources across regions to reverse-engineer state. diff --git a/references/gcp-cloud-run/iam.md b/references/gcp-cloud-run/iam.md new file mode 100644 index 0000000..08cb307 --- /dev/null +++ b/references/gcp-cloud-run/iam.md @@ -0,0 +1,126 @@ +# GCP Cloud Run — IAM & permissions + + + +Three identities, as on AWS Lambda — the **operator** (whose credentials run the commands), the **runner service account** (what the pool runs as), and the **invoker service account** (what Temporal impersonates). The mapping to Lambda's roles is close enough to be useful and different enough to be dangerous if assumed. + +| Concept | AWS Lambda | GCP Cloud Run | +|---|---|---| +| The compute's own identity | Execution role, trusted by `lambda.amazonaws.com` | **Runner service account**, set with `--service-account` | +| Temporal's identity | Invocation role, assumed via `sts:AssumeRole` + External ID | **Invoker service account**, reached by **impersonation** | +| Mechanism | `sts:AssumeRole` with a confused-deputy guard | `roles/iam.serviceAccountTokenCreator` — **no External ID equivalent** | +| What Temporal does with it | Invokes the function | Reads and **updates the pool's instance count** via the Cloud Run admin API | +| Infrastructure as code | CloudFormation template (shipped in `assets/`) | **Terraform module**, `serverless-workers/gcp/cloud-run` | + +**The two service accounts are not interchangeable**, and confusing them is the single most likely IAM mistake here. The runner runs the pool and never scales it; the invoker scales the pool and never runs it. + +## Runner service account + +The runtime identity the pool's instances use to reach other Google Cloud services. Set in `setup.md` Step 4 with `gcloud run worker-pools deploy --service-account`. It may be an account that already exists; a dedicated one is preferred. + +**It needs no baseline role to run the Worker.** Cloud Run collects `stdout` and `stderr` into Cloud Logging through its own infrastructure, and the Cloud Run *service agent* — not the runner — pulls the container image. Grant only what your code actually reaches: + +- `roles/secretmanager.secretAccessor` on each secret you mount, including the Temporal API key. +- `roles/logging.logWriter` **only if** the Worker writes through the Cloud Logging API rather than stdout/stderr. +- Whatever else your Workflows and Activities call. + +This differs from Lambda, where `AWSLambdaBasicExecutionRole` is effectively mandatory because the execution role is what creates the log group. Here, logging works with no grant at all. + +## Invoker service account + +The identity Temporal Cloud impersonates to read and scale the pool. Two grants make it work: + +- Temporal's identity receives **`roles/iam.serviceAccountTokenCreator`** on the invoker, so it can impersonate it. +- The invoker receives a project-level Cloud Run role with at least **`run.workerPools.get`** (read) and **`run.workerPools.update`** (scale). `roles/run.developer` includes both. + +The invoker also needs **`roles/iam.serviceAccountUser` on the runner service account**, which Cloud Run requires in order to attach that identity when it scales the pool. The Terraform module applies this. + +### The read/update split is a real trap + +`run.workerPools.get` alone is enough for the UI's **Validate Connection** action to pass. That manual action never exercises `run.workerPools.update`. Version registration does: the WCI reads the pool and then updates its manual instance count to bootstrap Task Queue registration. An invoker that can read but not update therefore passes manual validation but fails version registration or a later resize. + +There is no Lambda analogue: there, the registration invocation uses the same invoke permission as real traffic. For Cloud Run, verify the registration bootstrap and `lastModifier` rather than trusting the separate green Validate Connection result. → `diagnostics.md`. + +## The Terraform module + +Temporal publishes [`serverless-workers/gcp/cloud-run`](https://github.com/temporalio/terraform-modules/tree/main/modules/serverless-workers/gcp/cloud-run), which creates the invoker service account and applies the grants. + +**Get the template from the Cloud UI, not from here.** Under **Workers → Create Worker Deployment → Access**, Temporal Cloud emits a template with `impersonator_service_account_emails` already filled in for your account. Those values are account-specific, which is why every published snippet shows a placeholder. + +```hcl +module "serverless-worker-cloud-run" { + source = "github.com/temporalio/terraform-modules//modules/serverless-workers/gcp/cloud-run" + + project_id = "" + invoker_account_id = "temporal-worker-pool-invoker" + + impersonator_service_account_emails = [ + "", + ] + + runner_service_account_email = "temporal-worker-pool-runner@.iam.gserviceaccount.com" +} +``` + +| Variable | Required | Description | +|---|---|---| +| `project_id` | Yes | Project hosting the pool and the invoker. | +| `invoker_account_id` | Yes | Name for the invoker the module creates; email becomes `@.iam.gserviceaccount.com`. The template supplies one. | +| `impersonator_service_account_emails` | Yes | Temporal Cloud's service accounts, granted `serviceAccountTokenCreator` on the invoker. **From the UI template.** For self-hosted, the GCP identity the server runs as. | +| `runner_service_account_email` | Yes | The runner from `setup.md` Step 4. The module grants the invoker `roles/iam.serviceAccountUser` on it. | +| `invoker_display_name` | No | Defaults to `Temporal Serverless Worker Pool Invoker`. | +| `deploy_roles` | No | Project-level Cloud Run roles for the invoker. Defaults to `roles/run.developer`. A substitute must include `run.workerPools.get` and `run.workerPools.update`. | + +```bash +terraform init +terraform apply +``` + +Use the **`invoker_email`** output as `--gcp-cloud-run-service-account` when registering the version. + +### Treat the module as shared, pre-existing infrastructure + +Same discipline as Lambda's CloudFormation stack, different tool. One invoker can serve several pools, so before creating a second one, look for an existing account and consider reusing it. **Do not `terraform destroy` state you did not create**, and note that the module's default `invoker_account_id` collides the same way Lambda's default `RoleName` does — an earlier deployment in the project may already own it. + +## Operator GCP permissions + +The identity running the `gcloud`/Terraform commands needs, at minimum: + +| Step | Operator needs | +|---|---| +| Submit the image build (`setup.md` Step 3) | `cloudbuild.builds.create`; permission to use the selected build service account when one is specified | +| Push the built image | The Cloud Build execution service account needs Artifact Registry Writer on the repository when it is user-specified, cross-project, or lacks the same-project default access | +| Create the Worker Pool (Step 4) | `run.workerPools.create`/`update`, and `iam.serviceAccounts.actAs` on the **runner** to attach it | +| Create secrets | Secret Manager admin on the secrets used | +| Apply the Terraform module (Step 5) | Service-account creation plus IAM policy binding on the project and on the runner | +| Read pool state and logs (verify, diagnose) | `run.workerPools.get`, Cloud Logging read | + +`iam.serviceAccounts.actAs` on the runner is the Cloud Run counterpart of Lambda's `iam:PassRole` on the execution role, and fails the same way — a pool create that is denied despite having Cloud Run permissions. + +### Preflight + +Run before anything that creates or modifies GCP resources. Confirm the five required APIs appear in the enabled-service output, then inspect the names the deployment intends to use: + +```bash +gcloud auth list # which identity +gcloud config get-value project # which project +gcloud services list --enabled --project --format='value(config.name)' +gcloud run worker-pools list --region >/dev/null && echo "cloud run: ok" +gcloud iam service-accounts list >/dev/null && echo "iam read: ok" +gcloud artifacts repositories describe --location --project +gcloud iam service-accounts describe --project +gcloud secrets describe --project +terraform version +``` + +Required services: `run.googleapis.com`, `artifactregistry.googleapis.com`, `cloudbuild.googleapis.com`, `secretmanager.googleapis.com`, `iam.googleapis.com`, `iamcredentials.googleapis.com`, and `cloudresourcemanager.googleapis.com`. A `describe` returning Not Found is acceptable for a clean project; record that the named resource will be created and include it in the approval list. A permission error is not the same as absence—stop and resolve access before creating anything. + +For a same-project build using Cloud Build's default service account, Artifact Registry access is normally provided automatically. If the build uses a user-specified service account, the repository is in another project, or an organization policy removed the default grant, inspect that account and grant `roles/artifactregistry.writer` on this repository before submitting the build. + +**Classify an authentication failure before acting on it.** An absent or expired credential (`gcloud auth login`, or `gcloud auth application-default login` for Terraform) is recoverable in a minute; an identity that resolves but is denied a specific action is a real permissions problem. Never collect credentials in conversation and never ask the user to paste a service account key — Google recommends against long-lived keys outright. + +**Confirm the project explicitly before creating anything.** `gcloud config get-value project` is ambient state that is easy to be wrong about, exactly like an AWS profile pointing at an unintended account. Name the project in the approval list and verify it, rather than trusting the default. diff --git a/references/gcp-cloud-run/observability.md b/references/gcp-cloud-run/observability.md new file mode 100644 index 0000000..489a0f0 --- /dev/null +++ b/references/gcp-cloud-run/observability.md @@ -0,0 +1,54 @@ +# GCP Cloud Run — observability + + + +## There is nothing serverless-specific to configure + +**A Cloud Run Serverless Worker emits the same traces and metrics as a Worker anywhere else.** It is an ordinary long-lived Worker, so the SDK's normal metrics and OpenTelemetry tracing setup applies unchanged, and each SDK's general observability guide is the right reference. + +This is a genuine simplification over AWS Lambda, and the contrast is worth keeping in mind: + +| | AWS Lambda | GCP Cloud Run | +|---|---|---| +| OTel wiring | Per-SDK helper in the serverless package (`ApplyDefaults`, `apply_defaults`, `OtelLambdaWorkerConfigurationHelper`, a separate `…Aws.Lambda.OpenTelemetry` package for .NET) | **None — use the SDK's standard setup** | +| Collector | ADOT Lambda layer, plus a custom `otel-collector-config.yaml` because the default does not route OTLP to the traces pipeline | No layer; export as you would from any container | +| Env var | `OPENTELEMETRY_COLLECTOR_CONFIG_URI` / `_FILE`, differing by SDK | n/a | +| Flush timing | Must flush before the invocation deadline, or telemetry is lost | No deadline to beat | +| IAM | Execution role needs X-Ray and CloudWatch permissions | Runner needs nothing for stdout/stderr logging | + +Some SDKs add optional Cloud Run conveniences, such as OpenTelemetry helpers. **They are optional**, and where they exist they are documented in that SDK's Cloud Run guide. + +## Logs + +Cloud Run collects `stdout` and `stderr` into [Cloud Logging](https://cloud.google.com/run/docs/logging) through its own infrastructure. **The runner service account needs no grant for this** — `roles/logging.logWriter` is required only if the Worker writes through the Cloud Logging API instead. → `iam.md`. + +Read a pool's logs: + +```bash +gcloud run worker-pools logs read --region --project +``` + +**The pool produces no logs while scaled to zero.** Read them while an instance is up; an empty log is not evidence of a problem. This is the most common source of confusion when checking a Cloud Run Worker's health, and it has no Lambda equivalent — a Lambda log group retains history after the invocation ends. + +## Memory: give the runtime the instance, not the host + +A Worker Pool instance defaults to **512 MiB**; raise `--memory` when creating the pool if the Worker needs more. Two runtimes need to be told about the container limit explicitly, or they size themselves to a fraction of it: + +| SDK | Setting | Why | +|---|---|---| +| Java | `-XX:MaxRAMPercentage=75` | The JVM reads the container limit but defaults max heap to 25% of it, leaving most of a small instance unused. | +| TypeScript | `NODE_OPTIONS=--max-old-space-size=`, ~80% of the instance limit | Node's default heap is unrelated to the container limit. | + +Both are container-sizing concerns rather than Temporal ones, but they surface as Worker instability under load and are easy to miss. → `setup.md`. + +## What to watch that is specific to this provider + +Standard Worker metrics tell you about the Worker. Two Cloud Run-specific signals tell you about the *scaling*, and neither comes from the SDK: + +- **`run.googleapis.com/manualInstanceCount`** on the pool — what the WCI has asked for. Persistently `0` while a backlog exists is the headline symptom of a scaling failure. +- **`serving.knative.dev/lastModifier`** on the pool — whether Temporal has ever successfully written to it. The cheapest proof that impersonation and the update permission both work. + +Both are read with `gcloud run worker-pools describe … --format=yaml`. → `diagnostics.md`. diff --git a/references/gcp-cloud-run/sdk-dotnet.md b/references/gcp-cloud-run/sdk-dotnet.md new file mode 100644 index 0000000..757bdc8 --- /dev/null +++ b/references/gcp-cloud-run/sdk-dotnet.md @@ -0,0 +1,186 @@ +# .NET SDK on GCP Cloud Run + + + +Use this reference for .NET-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For the shared Cloud Run deployment lifecycle, permissions, versioning model, observability, and diagnostics, see `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. + +**There is no Cloud Run Worker package.** This is an ordinary long-lived .NET Worker plus Worker Versioning, which Serverless Workers require. Nothing in `../aws-lambda/sdk-dotnet.md` applies. + +## Inspect the versioning API before generating code + +Worker Versioning is a Public Preview surface and the option names differ between SDKs. Read the installed version's API rather than writing from memory: + +```bash +dotnet list package +unzip -p ~/.nuget/packages/temporalio//temporalio..nupkg \ + 'lib/net*/Temporalio.xml' | grep -A3 'WorkerDeploymentOptions\|WorkerDeploymentVersion' +``` + +## Versioned Worker + +Set `DeploymentOptions` on `TemporalWorkerOptions`. The Worker reads its connection settings and Task Queue from the environment so one image runs against any Namespace: + +```csharp +using System.Runtime.InteropServices; +using Temporalio.Client; +using Temporalio.Common; +using Temporalio.Worker; + +var client = await TemporalClient.ConnectAsync( + new(Environment.GetEnvironmentVariable("TEMPORAL_ADDRESS")!) + { + Namespace = Environment.GetEnvironmentVariable("TEMPORAL_NAMESPACE")!, + ApiKey = Environment.GetEnvironmentVariable("TEMPORAL_API_KEY"), + Tls = new(), + }); + +var options = new TemporalWorkerOptions( + Environment.GetEnvironmentVariable("TEMPORAL_TASK_QUEUE")!) +{ + DeploymentOptions = new(new("my-app", "build-1"), useWorkerVersioning: true) + { + DefaultVersioningBehavior = VersioningBehavior.Pinned, + }, + GracefulShutdownTimeout = TimeSpan.FromSeconds(8), +}; +options.AddWorkflow(); +options.AddAllActivities(typeof(GreetingActivities), null); + +using var shutdown = new CancellationTokenSource(); +using var sigterm = PosixSignalRegistration.Create( + PosixSignal.SIGTERM, + context => + { + context.Cancel = true; + shutdown.Cancel(); + }); +Console.CancelKeyPress += (_, eventArgs) => +{ + eventArgs.Cancel = true; + shutdown.Cancel(); +}; + +using var worker = new TemporalWorker(client, options); +try +{ + await worker.ExecuteAsync(shutdown.Token); +} +catch (OperationCanceledException) when (shutdown.IsCancellationRequested) +{ + // Expected during Cloud Run scale-in or an interactive stop. +} +``` + +`WorkerDeploymentVersion`'s two arguments are the deployment name and the build ID, and both must match the version created with `temporal worker deployment create-version` exactly. → `setup.md` Step 6. + +No `TemporalLambdaWorker`, no `CreateHandler`, and no RID-specific publish — the native Rust bridge comes from the ordinary publish for the image's platform. + +## Versioning behavior + +Every Workflow needs `VersioningBehavior.Pinned` or `AutoUpgrade`. `DefaultVersioningBehavior` covers every Workflow; to set it per Workflow, set it on the `[Workflow]` attribute. Unlike Lambda, no package supplies a Worker-level default here, so one of the two must be set explicitly. + +```csharp +using Temporalio.Common; +using Temporalio.Workflows; + +[Workflow(VersioningBehavior = VersioningBehavior.Pinned)] +public class GreetingWorkflow +{ + [WorkflowRun] + public async Task RunAsync(string name) => // ... +} +``` + +**A Version set with no behavior fails at runtime**, not at build time. + +## Connection configuration + +Read `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, `TEMPORAL_API_KEY`, and `TEMPORAL_TASK_QUEUE` from the environment set on the pool, and mount the key from Secret Manager rather than passing it in plaintext. → `setup.md` Step 4. + +To load them through the shared config format instead, use `ClientEnvConfig.LoadClientConnectOptions()` from `Temporalio.Common.EnvConfig`. + +**No `SSL_CERT_FILE` override is needed here.** That requirement is specific to AWS's .NET 8 Lambda images, which force-override the variable. The Debian-based `mcr.microsoft.com/dotnet/runtime` images ship a certificate store the Rust core can read. + +## Image packaging + +Publish and run on a .NET runtime image: + +```dockerfile +FROM mcr.microsoft.com/dotnet/sdk:9.0 AS build + +WORKDIR /src +COPY *.csproj ./ +RUN dotnet restore +COPY . . +RUN dotnet publish -c Release -o /out + +FROM mcr.microsoft.com/dotnet/runtime:9.0 + +WORKDIR /app +COPY --from=build /out ./ +CMD ["dotnet", "MyWorker.dll"] +``` + +The Rust core reads TLS roots from the OS certificate store, and the Debian-based runtime images include one. A distroless or Alpine base does not. → `setup.md` Step 2. + +## Graceful shutdown on scale-in + +Cloud Run sends `SIGTERM`, which is distinct from the `SIGINT` raised by Ctrl+C. The example registers both paths and cancels the token passed to `ExecuteAsync`. That stops polling and starts the Temporal Worker's shutdown sequence. + +`GracefulShutdownTimeout` defaults to zero. Keep a non-zero value below Cloud Run's ten-second termination window, leaving time for cancellation and final completions to propagate. Activities should observe `ActivityExecutionContext.Current.WorkerShutdownToken` or `CancellationToken` and record Heartbeats; Cloud Run can still send `SIGKILL` before a long Activity finishes. + +## Keep Activities safe across scale-in + +The WCI removes instances based on Task Queue activity, not on what an individual instance is doing, so **an instance running a long Activity can be stopped mid-execution.** Record Heartbeats so a retry resumes from the last recorded progress: + +```csharp +[Activity] +public static string Process(IReadOnlyList items) +{ + for (var i = 0; i < items.Count; i++) + { + ActivityExecutionContext.Current.Heartbeat(i); + // ... process items[i] + } + return "done"; +} +``` + +→ `constraints.md` for what else follows from the pool model. + +## Logging and diagnostic signatures + +The .NET SDK defaults to `NullLoggerFactory`, so configure a console provider explicitly. Add `Microsoft.Extensions.Logging.Console`, create the factory before connecting, and assign it to the client options: + +```csharp +using Microsoft.Extensions.Logging; + +using var loggerFactory = LoggerFactory.Create(builder => +{ + builder.AddSimpleConsole(options => options.SingleLine = true); + builder.SetMinimumLevel(LogLevel.Information); + builder.AddFilter("Grpc", LogLevel.Warning); +}); + +var client = await TemporalClient.ConnectAsync( + new(Environment.GetEnvironmentVariable("TEMPORAL_ADDRESS")!) + { + Namespace = Environment.GetEnvironmentVariable("TEMPORAL_NAMESPACE")!, + ApiKey = Environment.GetEnvironmentVariable("TEMPORAL_API_KEY"), + Tls = new(), + LoggerFactory = loggerFactory, + }); +``` + +Do not enable DEBUG logging globally in production without first verifying that dependency logs cannot contain credentials or payloads. + +| Log signature | Meaning / action | +|---|---| +| No SDK logs | The default null logger is still in use; pass an `ILoggerFactory` to the client. | +| `NativeCertsNotFound` | The runtime image lacks a readable CA store. Use the Debian runtime image or install CA certificates. | +| `OperationCanceledException` immediately after SIGTERM | Expected when it is caught by the shutdown path shown above. | +| Worker starts but the intended Workflow does not progress | Check the deployment name, build ID, Task Queue, and that the version is current. | + +## Observability + +A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else — no Cloud Run-specific wiring, and none of Lambda's ADOT layer or collector configuration. Use the SDK's normal metrics export and OpenTelemetry tracing interceptors. → `observability.md`, and `docs/develop/dotnet/platform/observability`. diff --git a/references/gcp-cloud-run/sdk-go.md b/references/gcp-cloud-run/sdk-go.md new file mode 100644 index 0000000..dfe73a6 --- /dev/null +++ b/references/gcp-cloud-run/sdk-go.md @@ -0,0 +1,116 @@ +# Go SDK on GCP Cloud Run + + + +Use this reference for Go-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For the shared Cloud Run deployment lifecycle, permissions, versioning model, observability, and diagnostics, see `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. + +**There is no Cloud Run Worker package.** This is an ordinary long-lived Go Worker plus Worker Versioning, which Serverless Workers require. Nothing in `../aws-lambda/sdk-go.md` applies. + +## Inspect the versioning API before generating code + +Worker Versioning is a Public Preview surface and the option names differ between SDKs. Read the installed version's API rather than writing from memory: + +```bash +go doc go.temporal.io/sdk/worker.DeploymentOptions +go doc go.temporal.io/sdk/worker.WorkerDeploymentVersion +go doc go.temporal.io/sdk/workflow.RegisterOptions +``` + +## Versioned Worker + +Set `DeploymentOptions` in `worker.Options`. The Worker reads its connection settings and Task Queue from the environment so one image runs against any Namespace: + +```go +package main + +import ( + "log" + "os" + "time" + + "go.temporal.io/sdk/client" + "go.temporal.io/sdk/contrib/envconfig" + "go.temporal.io/sdk/worker" + "go.temporal.io/sdk/workflow" + + "example.com/myapp" +) + +func main() { + c, err := client.Dial(envconfig.MustLoadDefaultClientOptions()) + if err != nil { + log.Fatalln("Unable to create client", err) + } + defer c.Close() + + w := worker.New(c, os.Getenv("TEMPORAL_TASK_QUEUE"), worker.Options{ + WorkerStopTimeout: 8 * time.Second, + DeploymentOptions: worker.DeploymentOptions{ + UseVersioning: true, + Version: worker.WorkerDeploymentVersion{ + DeploymentName: "my-app", + BuildID: "build-1", + }, + }, + }) + + w.RegisterWorkflowWithOptions(myapp.MyWorkflow, workflow.RegisterOptions{ + VersioningBehavior: workflow.VersioningBehaviorPinned, + }) + w.RegisterActivity(myapp.MyActivity) + + if err := w.Run(worker.InterruptCh()); err != nil { + log.Fatalln("Unable to start worker", err) + } +} +``` + +`DeploymentName` and `BuildID` must match the version created with `temporal worker deployment create-version` exactly, or the Worker polls under a version the WCI does not manage. → `setup.md` Step 6. + +## Versioning behavior + +Every Workflow needs `workflow.VersioningBehaviorPinned` or `VersioningBehaviorAutoUpgrade`. Set it per Workflow at registration as above, or set `DefaultVersioningBehavior` in `DeploymentOptions` to cover every Workflow. Registration **panics** with `workflow type does not have a versioning behavior` if a Version is set and neither is given. + +```go +w.RegisterWorkflowWithOptions(myapp.MyWorkflow, workflow.RegisterOptions{ + VersioningBehavior: workflow.VersioningBehaviorPinned, +}) +``` + +**A Version set with no behavior fails at runtime**, not at build time. + +## Connection configuration + +`go.temporal.io/sdk/contrib/envconfig` loads client configuration from environment variables and an optional TOML file, so the Worker carries no Namespace or credentials. Set non-secret values with `--set-env-vars` on the pool and mount the API key or TLS material from Secret Manager with `--set-secrets`. → `setup.md` Step 4. + +`MustLoadDefaultClientOptions` **panics** on invalid configuration. Use `envconfig.LoadDefaultClientOptions` and check the error to fail with a readable message instead. + +## Image packaging + +Use `CGO_ENABLED=0` with a `distroless/static` base. Go still reads system CA roots; that image includes them. → `setup.md` Step 2. + +## Graceful shutdown on scale-in + +The versioned Worker example uses `w.Run(worker.InterruptCh())` and gives received Tasks up to eight seconds through `WorkerStopTimeout`. `InterruptCh` receives both `SIGINT` and `SIGTERM`, so Cloud Run's `SIGTERM` makes the Worker stop polling and begin its normal shutdown. Do not replace it with an unhandled blocking channel, and do not leave `WorkerStopTimeout` at its zero default when draining is required. + +Cloud Run can send `SIGKILL` ten seconds later. Shutdown therefore improves draining but cannot guarantee that a long Activity finishes; Activities must cooperate with cancellation and record Heartbeats as shown below. + +## Keep Activities safe across scale-in + +The WCI removes instances based on Task Queue activity, not on what an individual instance is doing, so **an instance running a long Activity can be stopped mid-execution.** Record Heartbeats so a retry resumes from the last recorded progress: + +```go +func MyActivity(ctx context.Context, input MyInput) (string, error) { + for i := range input.Items { + activity.RecordHeartbeat(ctx, i) + // ... process input.Items[i] + } + return "done", nil +} +``` + +→ `constraints.md` for what else follows from the pool model. + +## Observability + +A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else — no Cloud Run-specific wiring, and none of Lambda's ADOT layer or collector configuration. Use the SDK's normal metrics export and OpenTelemetry tracing interceptors. → `observability.md`, and `docs/develop/go/platform/observability`. diff --git a/references/gcp-cloud-run/sdk-java.md b/references/gcp-cloud-run/sdk-java.md new file mode 100644 index 0000000..f0dfa3d --- /dev/null +++ b/references/gcp-cloud-run/sdk-java.md @@ -0,0 +1,181 @@ +# Java SDK on GCP Cloud Run + + + +Use this reference for Java-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For the shared Cloud Run deployment lifecycle, permissions, versioning model, observability, and diagnostics, see `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. + +**There is no Cloud Run Worker package.** This is an ordinary long-lived Java Worker plus Worker Versioning, which Serverless Workers require. Nothing in `../aws-lambda/sdk-java.md` applies. + +## Inspect the versioning API before generating code + +Worker Versioning is a Public Preview surface and the option names differ between SDKs. Read the installed version's API rather than writing from memory: + +```bash +mvn -q dependency:get -Dartifact=io.temporal:temporal-sdk::jar:sources +unzip -o ~/.m2/repository/io/temporal/temporal-sdk//temporal-sdk--sources.jar \ + 'io/temporal/worker/WorkerDeploymentOptions.java' 'io/temporal/common/WorkerDeploymentVersion.java' -d /tmp/src +``` + +## Versioned Worker + +Set `WorkerDeploymentOptions` on `WorkerOptions`. The Worker reads its connection settings and Task Queue from the environment so one image runs against any Namespace: + +```java +package example; + +import io.temporal.client.WorkflowClient; +import io.temporal.client.WorkflowClientOptions; +import io.temporal.common.VersioningBehavior; +import io.temporal.common.WorkerDeploymentVersion; +import io.temporal.serviceclient.WorkflowServiceStubs; +import io.temporal.serviceclient.WorkflowServiceStubsOptions; +import io.temporal.worker.Worker; +import io.temporal.worker.WorkerDeploymentOptions; +import io.temporal.worker.WorkerFactory; +import io.temporal.worker.WorkerOptions; +import java.util.concurrent.TimeUnit; + +public final class Main { + public static void main(String[] args) { + String apiKey = System.getenv("TEMPORAL_API_KEY"); + + WorkflowServiceStubs service = + WorkflowServiceStubs.newServiceStubs( + WorkflowServiceStubsOptions.newBuilder() + .setTarget(System.getenv("TEMPORAL_ADDRESS")) + .setEnableHttps(true) + .addApiKey(() -> apiKey) + .build()); + + WorkflowClient client = + WorkflowClient.newInstance( + service, + WorkflowClientOptions.newBuilder() + .setNamespace(System.getenv("TEMPORAL_NAMESPACE")) + .build()); + + WorkerFactory factory = WorkerFactory.newInstance(client); + Worker worker = + factory.newWorker( + System.getenv("TEMPORAL_TASK_QUEUE"), + WorkerOptions.newBuilder() + .setDeploymentOptions( + WorkerDeploymentOptions.newBuilder() + .setUseVersioning(true) + .setVersion(new WorkerDeploymentVersion("my-app", "build-1")) + .setDefaultVersioningBehavior(VersioningBehavior.PINNED) + .build()) + .build()); + + worker.registerWorkflowImplementationTypes(GreetingWorkflowImpl.class); + worker.registerActivitiesImplementations(new GreetingActivitiesImpl()); + + Runtime.getRuntime().addShutdownHook(new Thread(() -> { + factory.shutdown(); + factory.awaitTermination(8, TimeUnit.SECONDS); + })); + factory.start(); + } +} +``` + +`WorkerDeploymentVersion`'s two arguments are the deployment name and the build ID, and both must match the version created with `temporal worker deployment create-version` exactly. → `setup.md` Step 6. + +No `LambdaWorker`, no `define`, no shaded uber-jar requirement — this is `WorkerFactory` as in any long-lived Java Worker. + +## Versioning behavior + +Every Workflow needs `VersioningBehavior.PINNED` or `AUTO_UPGRADE`. `setDefaultVersioningBehavior` covers every Workflow; to set it per Workflow, annotate the Workflow method. + +```java +import io.temporal.common.VersioningBehavior; +import io.temporal.workflow.WorkflowVersioningBehavior; + +public class GreetingWorkflowImpl implements GreetingWorkflow { + @Override + @WorkflowVersioningBehavior(VersioningBehavior.PINNED) + public String run(String name) { + // ... + } +} +``` + +**A Version set with no behavior fails at runtime**, not at build time. + +## Connection configuration + +Read `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, `TEMPORAL_API_KEY`, and `TEMPORAL_TASK_QUEUE` from the environment set on the pool, and mount the key from Secret Manager rather than passing it in plaintext. → `setup.md` Step 4. + +`addApiKey` takes a **supplier**, called on every request, so a key can be rotated by returning a new value instead of restarting the Worker — worth using on a long-lived pool instance, where a restart is not free. + +Java uses gRPC/Netty and the JVM truststore, so it is unaffected by the `NativeCertsNotFound` failure the Rust-core SDKs hit. + +## Image packaging + +Fat jar on a JRE image, with the heap sized to the instance: + +```dockerfile +CMD ["java", "-XX:MaxRAMPercentage=75", "-jar", "/app/worker.jar"] +``` + +The JVM reads the container memory limit but defaults the maximum heap to a quarter of it, leaving most of a small instance unused. A pool defaults to 512 MiB per instance, so raise `--memory` when creating it if the Worker needs more. → `setup.md` Step 2. + +## Graceful shutdown on scale-in + +Register the JVM shutdown hook before `factory.start()`, as in the versioned Worker example. Cloud Run's `SIGTERM` starts the hook; `factory.shutdown()` stops polling, and `awaitTermination` keeps the hook alive while received Tasks drain. `shutdown()` alone is asynchronous, so omitting the wait lets the JVM exit before draining. Do not use `shutdownNow()` as the normal signal path. + +Cloud Run can send `SIGKILL` ten seconds later. Shutdown therefore improves draining but cannot guarantee that a long Activity finishes; Activities must finish promptly or record Heartbeats so a retry can resume. + +## Keep Activities safe across scale-in + +The WCI removes instances based on Task Queue activity, not on what an individual instance is doing, so **an instance running a long Activity can be stopped mid-execution.** Record Heartbeats so a retry resumes from the last recorded progress: + +```java +public class GreetingActivitiesImpl implements GreetingActivities { + @Override + public String process(List items) { + for (int i = 0; i < items.size(); i++) { + Activity.getExecutionContext().heartbeat(i); + // ... process items.get(i) + } + return "done"; + } +} +``` + +→ `constraints.md` for what else follows from the pool model. + +## Logging and diagnostic signatures + +If the application uses `logback-classic`, include an explicit `src/main/resources/logback.xml`. Without one, Logback's basic configuration sets the root logger to DEBUG; grpc-java's Netty transport has a DEBUG frame logger that can emit outbound HTTP/2 headers. Keep transport categories above DEBUG wherever bearer credentials are used: + +```xml + + + + %date %-5level %logger{36} - %msg%n + + + + + + + + + + + +``` + +Use a logging provider compatible with the SLF4J API version selected by the installed Temporal SDK. If Cloud Logging ever contains an `authorization` or `Bearer` value, restrict the transport logger immediately and rotate the exposed Temporal API key. + +| Log signature | Meaning / action | +|---|---| +| No application or SDK logs | No compatible SLF4J provider is bound, or the configuration was not packaged in the jar. | +| `NettyClientHandler ... OUTBOUND HEADERS` | Unsafe transport DEBUG logging is enabled. Raise `io.grpc` and `io.netty` to WARN and inspect for credential exposure. | +| `UNAUTHENTICATED` | Check the Secret Manager mount and the key's Namespace permissions. | +| Worker starts but the intended Workflow does not progress | Check the deployment name, build ID, Task Queue, and that the version is current. | + +## Observability + +A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else — no Cloud Run-specific wiring, and none of Lambda's ADOT layer or collector configuration. Use the SDK's normal metrics export and OpenTelemetry tracing interceptors. → `observability.md`, and `docs/develop/java/platform/observability`. diff --git a/references/gcp-cloud-run/sdk-python.md b/references/gcp-cloud-run/sdk-python.md new file mode 100644 index 0000000..24fd566 --- /dev/null +++ b/references/gcp-cloud-run/sdk-python.md @@ -0,0 +1,149 @@ +# Python SDK on GCP Cloud Run + + + +Use this reference for Python-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For the shared Cloud Run deployment lifecycle, permissions, versioning model, observability, and diagnostics, see `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. + +**There is no Cloud Run Worker package.** This is an ordinary long-lived Python Worker plus Worker Versioning, which Serverless Workers require. Nothing in `../aws-lambda/sdk-python.md` applies. + +## Inspect the versioning API before generating code + +Worker Versioning is a Public Preview surface and the option names differ between SDKs. Read the installed version's API rather than writing from memory: + +```bash +python -c "import temporalio.worker as w; print([n for n in dir(w) if 'Deployment' in n])" +python -c "from temporalio.worker import WorkerDeploymentConfig; help(WorkerDeploymentConfig)" +python -c "from temporalio.common import VersioningBehavior; print(list(VersioningBehavior))" +``` + +## Versioned Worker + +Pass `deployment_config` to `Worker()`. The Worker reads its connection settings and Task Queue from the environment so one image runs against any Namespace: + +```python +import asyncio +import os +import signal +from datetime import timedelta + +from temporalio.client import Client +from temporalio.common import VersioningBehavior, WorkerDeploymentVersion +from temporalio.envconfig import ClientConfig +from temporalio.worker import Worker, WorkerDeploymentConfig + +from my_activities import my_activity +from my_workflows import MyWorkflow + + +async def main() -> None: + client = await Client.connect(**ClientConfig.load_client_connect_config()) + + worker = Worker( + client, + task_queue=os.environ["TEMPORAL_TASK_QUEUE"], + workflows=[MyWorkflow], + activities=[my_activity], + deployment_config=WorkerDeploymentConfig( + version=WorkerDeploymentVersion( + deployment_name="my-app", + build_id="build-1", + ), + use_worker_versioning=True, + default_versioning_behavior=VersioningBehavior.PINNED, + ), + graceful_shutdown_timeout=timedelta(seconds=8), + ) + stop = asyncio.Event() + loop = asyncio.get_running_loop() + for sig in (signal.SIGTERM, signal.SIGINT): + loop.add_signal_handler(sig, stop.set) + + async with worker: + await stop.wait() + + +if __name__ == "__main__": + asyncio.run(main()) +``` + +`deployment_name` and `build_id` must match the version created with `temporal worker deployment create-version` exactly, or the Worker polls under a version the WCI does not manage. → `setup.md` Step 6. + +## Versioning behavior + +Every Workflow needs `VersioningBehavior.PINNED` or `AUTO_UPGRADE`. `default_versioning_behavior` on `WorkerDeploymentConfig` covers every Workflow; to set it per Workflow, pass `versioning_behavior` to the decorator. + +```python +@workflow.defn(versioning_behavior=VersioningBehavior.PINNED) +class MyWorkflow: + @workflow.run + async def run(self, name: str) -> str: + ... +``` + +**A Version set with no behavior fails at runtime**, not at build time. + +## Connection configuration + +`temporalio.envconfig` loads client configuration from environment variables and an optional TOML file. Set non-secret values with `--set-env-vars` on the pool and mount the API key or TLS material from Secret Manager with `--set-secrets`. → `setup.md` Step 4. + +`ClientConfig.load_client_connect_config()` returns keyword arguments for `Client.connect`, which is why it is unpacked with `**`. To inspect or override values first, load the profile instead: + +```python +from temporalio.envconfig import ClientConfigProfile + +profile = ClientConfigProfile.load() +connect_config = profile.to_client_connect_config() +client = await Client.connect(**connect_config) +``` + +## Image packaging + +`pip install "temporalio>=1.30.0,<2"` and run the Worker module as the entrypoint. Python shares the Rust core, so a minimal base image needs `ca-certificates` present. → `setup.md` Step 2. + +## Graceful shutdown on scale-in + +Cloud Run sends `SIGTERM` before stopping an instance. The example converts it into an `asyncio.Event`; leaving the `async with worker` block calls `worker.shutdown()` and waits for the SDK's shutdown sequence. `graceful_shutdown_timeout` gives received Activities up to eight seconds before cancellation. Prefer this explicit path over cancelling `worker.run()`, because cancellation can also cancel the shutdown operation. + +Cloud Run can send `SIGKILL` ten seconds later. Shutdown therefore improves draining but cannot guarantee that a long Activity finishes; Activities must cooperate with cancellation and record Heartbeats as shown below. + +## Keep Activities safe across scale-in + +The WCI removes instances based on Task Queue activity, not on what an individual instance is doing, so **an instance running a long Activity can be stopped mid-execution.** Record Heartbeats so a retry resumes from the last recorded progress: + +```python +@activity.defn +async def my_activity(items: list[str]) -> str: + for i, item in enumerate(items): + activity.heartbeat(i) + # ... process item + return "done" +``` + +→ `constraints.md` for what else follows from the pool model. + +## Logging and diagnostic signatures + +Configure application logging before constructing the client or Worker. Cloud Run captures stdout and stderr automatically: + +```python +import logging + +logging.basicConfig( + level=logging.INFO, + format="%(asctime)s %(levelname)s %(name)s %(message)s", +) +logging.getLogger("temporalio").setLevel(logging.INFO) +``` + +Do not enable DEBUG logging globally in production without first verifying that dependency logs cannot contain credentials or payloads. + +| Log signature | Meaning / action | +|---|---| +| `NativeCertsNotFound` | The runtime image lacks CA certificates. Install `ca-certificates` and rebuild. | +| `TransportError` during startup | Check the mounted API key, address including port, TLS, and Namespace. | +| Worker starts but the intended Workflow does not progress | Check the deployment name, build ID, Task Queue, and that the version is current. | +| No application `INFO` records | Configure the root logger before Worker construction; do not rely on an implicit handler. | + +## Observability + +A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else — no Cloud Run-specific wiring, and none of Lambda's ADOT layer or collector configuration. Use the SDK's normal metrics export and OpenTelemetry tracing interceptors. → `observability.md`, and `docs/develop/python/platform/observability`. diff --git a/references/gcp-cloud-run/sdk-typescript.md b/references/gcp-cloud-run/sdk-typescript.md new file mode 100644 index 0000000..183dad2 --- /dev/null +++ b/references/gcp-cloud-run/sdk-typescript.md @@ -0,0 +1,116 @@ +# TypeScript SDK on GCP Cloud Run + + + +Use this reference for TypeScript-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For the shared Cloud Run deployment lifecycle, permissions, versioning model, observability, and diagnostics, see `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. + +**There is no Cloud Run Worker package.** This is an ordinary long-lived TypeScript Worker plus Worker Versioning, which Serverless Workers require. Nothing in `../aws-lambda/sdk-typescript.md` applies. + +## Inspect the versioning API before generating code + +Worker Versioning is a Public Preview surface and the option names differ between SDKs. Read the installed version's API rather than writing from memory: + +```bash +npm ls @temporalio/worker +grep -rn "workerDeploymentOptions\|WorkerDeploymentOptions" node_modules/@temporalio/worker/lib/*.d.ts +``` + +## Versioned Worker + +Pass `workerDeploymentOptions` to `Worker.create()`. The Worker reads its connection settings and Task Queue from the environment so one image runs against any Namespace: + +```ts +import { NativeConnection, Worker } from '@temporalio/worker'; + +async function main(): Promise { + const connection = await NativeConnection.connect({ + address: process.env.TEMPORAL_ADDRESS, + apiKey: process.env.TEMPORAL_API_KEY, + tls: true, + }); + + const worker = await Worker.create({ + connection, + namespace: process.env.TEMPORAL_NAMESPACE!, + taskQueue: process.env.TEMPORAL_TASK_QUEUE!, + workflowsPath: require.resolve('./workflows'), + workerDeploymentOptions: { + version: { deploymentName: 'my-app', buildId: 'build-1' }, + useWorkerVersioning: true, + defaultVersioningBehavior: 'PINNED', + }, + shutdownGraceTime: '8s', + shutdownForceTime: '9s', + }); + + await worker.run(); +} + +main().catch((err) => { + console.error(err); + process.exit(1); +}); +``` + +`deploymentName` and `buildId` must match the version created with `temporal worker deployment create-version` exactly, or the Worker polls under a version the WCI does not manage. → `setup.md` Step 6. + +Unlike Lambda, Cloud Run needs no `workflowBundle` — `workflowsPath` is fine, because the instance is long-lived and bundling only buys startup time. + +## Versioning behavior + +Every Workflow needs `'PINNED'` or `'AUTO_UPGRADE'`. `defaultVersioningBehavior` covers every Workflow; to set it per Workflow, use `setWorkflowOptions()` from `@temporalio/workflow`. + +```ts +import { setWorkflowOptions } from '@temporalio/workflow'; + +setWorkflowOptions({ versioningBehavior: 'PINNED' }, myWorkflow); +export async function myWorkflow(): Promise { + // ... +} +``` + +**A Version set with no behavior fails at runtime**, not at build time. + +## Connection configuration + +Read the Namespace, address, and Task Queue from environment variables set on the pool, and mount the API key or TLS material from Secret Manager rather than passing it in plaintext. → `setup.md` Step 4. + +To load them through the shared config format and profiles instead, use `loadClientConnectConfig()` from `@temporalio/envconfig` and pass its `connectionOptions` and `namespace` to `NativeConnection.connect()` and `Worker.create()`. + +## Image packaging + +Three things, all of which fail at startup rather than at build time: + +- **Install `ca-certificates` in the runtime stage.** The Rust core reads TLS roots from the OS store and the slim Node images ship without one; on `node:22-slim` a TLS connection fails with `TransportError: tonic::transport::Error(Transport, NativeCertsNotFound)`. +- **Use a glibc image such as `node:22-slim`, not Alpine.** Alpine's musl is unsupported by the Rust core. +- **Set `NODE_OPTIONS=--max-old-space-size=`** to about 80% of the instance memory limit. Node sizes its heap from the host, not the container. A pool defaults to 512 MiB per instance, so raise `--memory` if the Worker needs more. + +→ `setup.md` Step 2, `diagnostics.md`. + +## Graceful shutdown on scale-in + +The versioned Worker example uses `await worker.run()`. The TypeScript SDK Runtime registers `SIGINT`, `SIGTERM`, `SIGQUIT`, and `SIGUSR2` as shutdown signals by default, so Cloud Run's `SIGTERM` starts the Worker's normal shutdown without an application-level signal handler. `shutdownGraceTime` gives received Activities eight seconds before cancellation; `shutdownForceTime` prevents a non-cooperative Activity from keeping the process alive until Cloud Run kills it. If the application installs a custom Runtime, preserve `SIGTERM` in its `shutdownSignals`. + +Cloud Run can send `SIGKILL` ten seconds later. Shutdown therefore improves draining but cannot guarantee that a long or non-cooperative Activity finishes; Activities must react to cancellation and record Heartbeats as shown below. + +## Keep Activities safe across scale-in + +The WCI removes instances based on Task Queue activity, not on what an individual instance is doing, so **an instance running a long Activity can be stopped mid-execution.** Record Heartbeats so a retry resumes from the last recorded progress: + +```ts +import { heartbeat } from '@temporalio/activity'; + +export async function myActivity(items: string[]): Promise { + for (let i = 0; i < items.length; i++) { + heartbeat(i); + // ... process items[i] + } + return 'done'; +} +``` + +→ `constraints.md` for what else follows from the pool model. + +## Observability + +A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else — no Cloud Run-specific wiring, and none of Lambda's ADOT layer or collector configuration. Use the SDK's normal metrics export and OpenTelemetry tracing interceptors. → `observability.md`, and `docs/develop/typescript/platform/observability`. diff --git a/references/gcp-cloud-run/self-hosted.md b/references/gcp-cloud-run/self-hosted.md new file mode 100644 index 0000000..282e5bc --- /dev/null +++ b/references/gcp-cloud-run/self-hosted.md @@ -0,0 +1,94 @@ +# GCP Cloud Run — self-hosted Temporal Service setup + + + +Serverless Workers require **Temporal Service v1.31.0 or later**. Complete this page before following `setup.md`. + +Four prerequisites: network reachability, enable the WCI, give the server a GCP identity, create the invoker service account. + +## 1. Cloud Run instances must reach the Temporal Service + +The frontend must be reachable **from the Worker Pool instances**. If the Service runs on a private network, that likely means [Direct VPC egress](https://cloud.google.com/run/docs/configuring/vpc-direct-vpc) or a [Serverless VPC Access connector](https://cloud.google.com/run/docs/configuring/vpc-connectors). + +Note the direction: instances dial *out* to Temporal. Nothing needs to reach *into* Cloud Run — Temporal drives the pool through the Cloud Run admin API, not by connecting to your instances. + +## 2. Enable the Worker Controller Instance + +The WCI is **disabled by default** and enabled through [Temporal Service dynamic configuration](https://docs.temporal.io/references/dynamic-configuration): + +```yaml +workercontroller.enabled: + - value: true + +workercontroller.compute_providers.enabled: + - value: + - gcp-cloud-run + +workercontroller.scaling_algorithms.enabled: + - value: + - rate-based +``` + +**Cloud Run requires the `rate-based` algorithm.** Because a pool is a set of long-lived instances, the WCI resizes it from arrival and backlog rates. The `no-sync` algorithm applies only to providers invoked once per Task, and **pairing it with `gcp-cloud-run` is rejected** — a concrete way the two providers' execution models surface in server configuration. + +To enable per Namespace instead of globally: + +```yaml +workercontroller.enabled: + - value: true + constraints: + namespace: 'your-namespace' +``` + +The Service watches the file and applies updates **without a restart**. + +Two optional global keys cover multi-hop impersonation: + +| Key | Purpose | +|---|---| +| `workercontroller.compute_providers.gcp.intermediary_service_accounts` | Service accounts to impersonate in sequence before the invoker. Unset means the server impersonates the invoker directly. | +| `workercontroller.compute_providers.gcp.first_delegate_as_base` | When `true`, the first entry is the identity the server's ambient credentials impersonate directly and the rest are passed as token-creator delegates. Defaults to `false`, passing the whole chain as delegates. | + +## 3. Give the Temporal Service a GCP identity + +The server impersonates the invoker service account, so it must first run as a GCP identity permitted to do so. + +**On GCP (GCE, GKE):** the attached service account is used automatically through [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). No extra credential configuration — you grant *that* account impersonation rights in step 4. + +**Outside GCP:** use [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) and point `GOOGLE_APPLICATION_CREDENTIALS` at the credential configuration file it produces. That variable also accepts a service account key file, but **Google recommends against long-lived keys** — prefer federation, and never ask a user to paste a key. + +## 4. Create the invoker service account + +The Service scales the pool as an **invoker** service account, which only reads and scales the pool. The identity instances *run as* is the separate **runner** service account set on the pool in `setup.md`. → `iam.md` for the full distinction. + +Two grants: + +- The GCP identity the Service runs as (step 3) gets **`roles/iam.serviceAccountTokenCreator`** on the invoker. +- The invoker gets a project-level Cloud Run role with at least **`run.workerPools.get`** and **`run.workerPools.update`**. `roles/run.developer` includes both. + +The same Terraform module works — pass the server's GCP identity as the impersonator instead of Temporal Cloud's accounts: + +```hcl +module "serverless-worker-cloud-run" { + source = "github.com/temporalio/terraform-modules//modules/serverless-workers/gcp/cloud-run" + + project_id = "" + invoker_account_id = "temporal-serverless-worker" + + runner_service_account_email = "" + + impersonator_service_account_emails = [ + "", + ] +} +``` + +Use the module's `invoker_email` output as `--gcp-cloud-run-service-account` when registering the Worker Deployment Version. + +**This is the one place self-hosted is simpler than Cloud:** there is no UI-provided template to copy, because you already know the impersonating identity — it is your own server's. + +## Then + +Follow `setup.md` from Step 1. The read/update permission trap in `iam.md` applies identically, and `diagnostics.md`'s `lastModifier` check is still the fastest way to confirm the server can actually scale the pool. diff --git a/references/gcp-cloud-run/setup.md b/references/gcp-cloud-run/setup.md new file mode 100644 index 0000000..298badd --- /dev/null +++ b/references/gcp-cloud-run/setup.md @@ -0,0 +1,281 @@ +# GCP Cloud Run — Setup (happy path) + + + +End-to-end: write a standard Worker, containerize it, push the image, create a Worker Pool at zero instances, grant Temporal permission to scale it, register a Worker Deployment Version, set it current, verify. For the two service accounts and the Terraform module, see `iam.md`. For what the execution model does and does not bound, see `constraints.md`. For new builds and rollback, see `versioning.md`. If it doesn't work, see `diagnostics.md`. + +## Prerequisites + + + +- **Cloud Run support is Pre-release and access-gated.** The user creates a support ticket or contacts their account team. Confirm this before anything else. +- A Temporal Cloud account with a **GCP-hosted Namespace**, or self-hosted Temporal Service v1.31.0+. The Namespace's cloud provider must match the compute provider — an AWS-hosted Namespace cannot drive Cloud Run. Regions need not match. +- For self-hosted, complete `self-hosted.md` first. +- Every Workflow must declare a versioning behavior, or the Worker must set a default. +- A GCP project with billing enabled and permission to enable service APIs and create Worker Pools, Artifact Registry repositories, Cloud Build jobs, service accounts, and Secret Manager secrets. +- `gcloud` CLI installed and authenticated. The Google Cloud console or Terraform also work. +- **Terraform** installed — Temporal ships the IAM setup as a Terraform module. +- A Temporal SDK. Supported on Cloud Run: Go, Python, TypeScript, Java, .NET, **Ruby, and Rust** — the last two are Cloud Run only and unavailable on Lambda. + +The `temporal` CLI commands in Steps 6 and 7 must inherit authentication from an existing profile or from the process environment. **Never append `--api-key ` or put the key in an inline assignment.** If `TEMPORAL_API_KEY` is not already populated, set it privately in the user's own terminal without putting the value in shell history: + +```bash +export TEMPORAL_ADDRESS="
:7233" +export TEMPORAL_NAMESPACE="" +printf 'Temporal API key: ' >&2 +IFS= read -r -s TEMPORAL_API_KEY +printf '\n' >&2 +export TEMPORAL_API_KEY +``` + +Do not run the secret-reading commands through an agent shell, ask the user to paste the key into conversation, or inspect the resulting variable. A configured Temporal CLI profile is equally valid and avoids a session environment variable. + +**Check the region before deploying.** Google reports high deployment latency creating or updating Cloud Run resources in some regions, including `us-central1`, and recommends another region while the issue is open. → `constraints.md`. + +## Prepare a clean GCP project + +After the resource list is approved, create only what is missing. First enable every API used by the commands below: + +```bash +gcloud services enable \ + run.googleapis.com \ + artifactregistry.googleapis.com \ + cloudbuild.googleapis.com \ + secretmanager.googleapis.com \ + iam.googleapis.com \ + iamcredentials.googleapis.com \ + cloudresourcemanager.googleapis.com \ + --project +``` + +Create a regional Docker repository for the Worker image: + +```bash +gcloud artifacts repositories create \ + --repository-format docker \ + --location \ + --project \ + --description "Temporal Serverless Worker images" +``` + +Create the runner service account that pool instances use: + +```bash +gcloud iam service-accounts create \ + --display-name "Temporal Cloud Run Worker runner" \ + --project +``` + +Create the Secret Manager secret before deploying the pool: + +```bash +gcloud secrets create \ + --replication-policy automatic \ + --project + +gcloud secrets versions add \ + --data-file=- \ + --project +``` + +Run the `versions add` command only in the user's own terminal, provide the Temporal API key on standard input, and then send EOF. Never ask for the value in conversation or run it through an agent shell where input or output may be captured. + +Grant only the runner access to that secret: + +```bash +gcloud secrets add-iam-policy-binding \ + --member="serviceAccount:@.iam.gserviceaccount.com" \ + --role roles/secretmanager.secretAccessor \ + --project +``` + +Before each create, use the corresponding `describe` command from `iam.md` to avoid colliding with shared resources. The invoker service account is created later by Temporal's Terraform module; do not substitute it for the runner. + +## Step 1: Write Worker code + +**There is no Cloud Run Worker package.** Write an ordinary long-lived Worker — same client, same `Worker`/`WorkerFactory`, same registration — and add Worker Versioning, which Serverless Workers require. Do not reach for anything in `../aws-lambda/sdk-.md`; those files are AWS Lambda only. The Cloud Run counterpart is `sdk-.md` in this directory. + +Two things the Worker must do: + +1. **Declare its Worker Deployment Version and enable versioning**, with a deployment name and build ID that exactly match the version you register in Step 6. +2. **Read its configuration from the environment** — address, Namespace, Task Queue, credentials — so one image can run against any Namespace. The pool supplies these via `--set-env-vars` and `--set-secrets`. + +Per-SDK code lives in `sdk-.md` in this directory, one file per SDK. Each covers the versioned Worker, connection, image packaging, graceful shutdown, scale-in safety, and observability. The Java, Python, and .NET references also include their logging setup and diagnostic signatures. + +**The entrypoint must start the Worker process**, so an instance begins polling as soon as it starts. + +## Step 2: Containerize the Worker + + + +Per-runtime notes that matter, from the deployment guide: + +| SDK | Notes | +|---|---| +| Go | Multi-stage; `CGO_ENABLED=0` for a static binary, which is what a `distroless/static` base expects. | +| Python | `pip install "temporalio>=1.30.0,<2"`; entrypoint runs the Worker module. | +| TypeScript | **Keep `ca-certificates` installed** — without it the Worker fails at startup with `TransportError: tonic::transport::Error(Transport, NativeCertsNotFound)`. Use a **glibc** image, not Alpine. Set `NODE_OPTIONS=--max-old-space-size=` to ~80% of the instance memory limit. | +| Java | Fat jar on a JRE image; set `-XX:MaxRAMPercentage=75` — the JVM reads the container limit but defaults max heap to 25% of it. | +| .NET | `dotnet publish` in a build stage, run on the .NET runtime image. | + +**The `NativeCertsNotFound` error is the same root cause as the .NET one on Lambda** — a Rust-core SDK that cannot find system root CAs — reached here through a slim base image rather than an overridden `SSL_CERT_FILE`. Any Rust-core SDK (TypeScript, Python, .NET, Ruby) in a minimal image needs CA certificates present. + +## Step 3: Build and push the image + +```bash +gcloud builds submit \ + --tag -docker.pkg.dev///my-temporal-worker:build-1 \ + --project +``` + +Tag the image with the build ID. It keeps image, pool, and Worker Deployment Version aligned, which matters because the compute configuration cannot pin a revision (→ `constraints.md`). + +The happy path deliberately uses Cloud Build's global endpoint. Supplying `--region` can require additional regional build and staging-bucket setup, depending on the project's Cloud Build bucket policy. If regional builds are required for a private pool or data-residency policy, pre-create or select the regional source/log buckets, grant the build identity access, and then add `--region `. + +## Step 4: Create the Worker Pool + +**Create one pool per Worker Deployment Version, initially at zero instances.** Registering the version starts the WCI, which temporarily raises the pool to the scaler's planned count—at least one instance—to register its Task Queues. The version does not need to be current for this bootstrap. + +```bash +gcloud run worker-pools deploy my-temporal-worker-pool-build-1 \ + --image -docker.pkg.dev///my-temporal-worker:build-1 \ + --region \ + --project \ + --service-account \ + --instances 0 \ + --set-env-vars TEMPORAL_ADDRESS=
:7233,TEMPORAL_NAMESPACE=,TEMPORAL_TASK_QUEUE=my-task-queue \ + --set-secrets TEMPORAL_API_KEY=:latest +``` + +| Parameter | Description | +|---|---| +| `--image` | The image pushed in Step 3. | +| `--service-account` | The **runner** service account instances run as. **Not** the invoker Temporal impersonates. → `iam.md`. | +| `--instances` | Set to `0`; the WCI takes ownership of the count when the Worker Deployment Version is registered. | +| `--set-env-vars` | Non-secret configuration. | +| `--set-secrets` | Maps a Secret Manager secret to an env var — use it for `TEMPORAL_API_KEY` or TLS material. | + +**Secrets are the documented default here, not an upgrade.** Unlike Lambda's guide, the Cloud Run path puts the API key in Secret Manager from the start, so there is no "acceptable for development only" plaintext step to warn about. + +## Step 5: Grant Temporal permission to scale the pool + +Cloud Run has no invocation grant. Temporal **impersonates an invoker service account** and drives the Cloud Run admin API. Create it with Temporal's Terraform module — the Cloud UI supplies a filled-in template under **Workers → Create Worker Deployment → Access**. → `iam.md` for the module, its variables, and the two-service-account distinction. + +Terraform's `invoker_email` output is what Step 6 needs. + +## Step 6: Register the Worker Deployment Version + +```bash +temporal worker deployment create --namespace --name my-app + +temporal worker deployment create-version \ + --namespace \ + --deployment-name my-app \ + --build-id build-1 \ + --gcp-cloud-run-project \ + --gcp-cloud-run-region \ + --gcp-cloud-run-worker-pool my-temporal-worker-pool-build-1 \ + --gcp-cloud-run-service-account \ + --gcp-cloud-run-min-instances 0 \ + --gcp-cloud-run-max-instances 30 \ + --gcp-cloud-run-initial-instances 0 \ + --gcp-cloud-run-utilization-target 0.8 +``` + +| Flag | Description | +|---|---| +| `--deployment-name` / `--build-id` | Must match the Worker code exactly. | +| `--gcp-cloud-run-project` | Project containing the pool. | +| `--gcp-cloud-run-region` | Pool region. | +| `--gcp-cloud-run-worker-pool` | Pool name from Step 4. | +| `--gcp-cloud-run-service-account` | The **invoker** — Terraform's `invoker_email`. | +| `--gcp-cloud-run-min-instances` | Floor the scaler maintains; `0` allows scale-to-zero. | +| `--gcp-cloud-run-max-instances` | Ceiling the scaler may request; defaults to `30`. | +| `--gcp-cloud-run-initial-instances` | Initial planned count; must be between min and max. | +| `--gcp-cloud-run-utilization-target` | Target average utilization in `(0, 1]`; defaults to `0.8`. | + +The four scaler flags are a coupled group: **either omit all four and accept the defaults (`0`, `30`, `0`, `0.8`), or provide all four together.** Supplying only one—even only a higher maximum—fails CLI validation. + +Through the UI, the version is set current automatically; through the CLI it is a separate step. + +### Checkpoint: distinguish registration from Validate Connection + +Creating the Worker Deployment Version starts its WCI. The WCI validates the configuration by reading the Worker Pool, then asks the rate-based algorithm for its Task Queue registration action. For a new Cloud Run version, that action updates the pool's manual instance count to at least one. This special registration floor applies even when `--gcp-cloud-run-min-instances` and `--gcp-cloud-run-initial-instances` are both `0`; those values govern the scaler's normal operating range and initial plan, not whether it can bootstrap Task Queue registration. This exercises both `run.workerPools.get` and the `run.workerPools.update` path used by real scaling. + +The update completing proves that Temporal reached Cloud Run, not that the container started or the Worker connected. Wait for the expected Task Queue types to appear; that binding is proof the instance started and polled under the registered deployment name and build ID. + +The separate UI **Validate Connection** action (Workers → Deployments → select → Actions) only impersonates the invoker and reads the pool. It starts no instance and does not exercise `run.workerPools.update`, so a green manual validation is weaker than a successful registration bootstrap. + +Use both views when diagnosing the checkpoint: + +```bash +# has any Worker ever polled under this version? +temporal worker deployment describe-version \ + --namespace --deployment-name my-app --build-id build-1 --report-task-queue-stats + +# has Temporal ever actually written to the pool? +gcloud run worker-pools describe my-temporal-worker-pool-build-1 \ + --region --project --format=yaml +``` + +In the pool's `metadata.annotations`, `serving.knative.dev/lastModifier` becoming the invoker service account is the real proof that scaling works. → `diagnostics.md`. + +## Step 7: Set the version current + +```bash +temporal worker deployment set-current-version \ + --namespace --deployment-name my-app --build-id build-1 --yes +``` + +Without this, new traffic does not route to the version. The registration instance may already have started and bound the Task Queue, but that bootstrap does not make the version current. The command prompts for confirmation; **run non-interactively without `--yes` it exits having changed nothing**, which reads as success. Read the state back with `temporal worker deployment describe`. + +## Step 8: Verify + +```bash +temporal workflow start \ + --namespace --task-queue my-task-queue \ + --type MyWorkflow --input '"Hello, serverless!"' +``` + +Tasks arriving with no active pollers cause the WCI to raise the instance count; Cloud Run starts an instance, the Worker connects and processes the Task. + +Confirm from two independent signals: + +- **Temporal** — Task completions in the Workflow's event history. +- **Cloud Run** — pool logs showing Worker startup and Task processing: + ```bash + gcloud run worker-pools logs read my-temporal-worker-pool-build-1 \ + --region --project + ``` + **The pool produces no logs while scaled to zero**, so read them while an instance is up. An empty log is not evidence of failure. + +## Teardown + +Record what you create as you go: pool name, image tag and Artifact Registry repository, runner and invoker service accounts, the Terraform state, secrets, deployment name and build ID, project and region. + +**The ordering problem is milder than Lambda's**, where the function had to go before the version or the delete deadlocked on active pollers. Here, scaling the pool to zero stops the pollers without destroying anything. + +1. Unset the current version — a Current version cannot be deleted: + ```bash + temporal worker deployment set-current-version \ + --namespace --deployment-name my-app --unversioned --yes + ``` +2. Scale the pool to zero, which ends polling: + ```bash + gcloud run worker-pools update --instances 0 --region --project + ``` +3. Wait for drainage, then delete the version, then the deployment: + ```bash + temporal worker deployment describe-version --namespace --deployment-name my-app --build-id build-1 + temporal worker deployment delete-version --namespace --deployment-name my-app --build-id build-1 + temporal worker deployment delete --namespace --name my-app + ``` +4. Delete the Worker Pool: + ```bash + gcloud run worker-pools delete --region --project + ``` +5. `terraform destroy` the IAM module — **only if this deployment created it.** One invoker service account can serve several pools, so a shared one may still be in use. → `iam.md`. +6. Delete the container image from Artifact Registry, and any Secret Manager secrets created for this deployment. Ask before revoking a Temporal Cloud API key: it is account-scoped, not deployment-scoped. diff --git a/references/gcp-cloud-run/versioning.md b/references/gcp-cloud-run/versioning.md new file mode 100644 index 0000000..eb7686e --- /dev/null +++ b/references/gcp-cloud-run/versioning.md @@ -0,0 +1,73 @@ +# GCP Cloud Run — versioning, updates, and rollback + + + +## One Worker Pool per Build ID + +**The compute configuration names a project, region, and Worker Pool — it does not name a [revision](https://cloud.google.com/run/docs/managing/revisions).** Temporal runs whichever revision the pool happens to serve. That ties a pool to a single build, so **a new build needs a new pool.** Carry the Build ID in the pool name (`my-worker-pool-build-1`) so the mapping stays visible. + +This is the structural difference from Lambda, where one function holds many published versions and the ARN pins which one Temporal invokes. Temporal's Cloud Run compute configuration has no revision selector, so **durable isolation has to come from creating separate pools.** A pool-level instance split can temporarily hold a particular revision, but that split remains mutable state outside Temporal. + +## The hazard: redeploying into a live pool + +> Deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run promotes it to every instance by default. The version does not change, but the code behind it does. Deploying replay-unsafe code this way causes non-determinism errors for in-flight Workflows, **including Pinned ones**. + +Same failure as pointing a Lambda Worker Deployment Version at a mutable `$LATEST`, reached through a different mechanism. Worth stating precisely because the intuition differs: on Lambda you have to *choose* the mutable path by registering an unqualified ARN. **On Cloud Run the mutable path is what a normal `gcloud run worker-pools deploy` does** — the durable Temporal-aligned path is a new pool, while `--no-promote` is an explicit same-pool guardrail. + +`Pinned` does not protect you. Pinning routes Workflows to a *version*; it cannot pin the code behind a pool whose revision moved underneath it. + +## Guardrail and recovery when a pool is reused + +Pool-per-build remains the production rule because Temporal identifies a Worker Pool, not one of its revisions. If an exceptional workflow must deploy a new revision into a pool that still serves a live version, `--no-promote` prevents the new revision from receiving the pool's instances automatically: + +```bash +gcloud run worker-pools deploy \ + --image -docker.pkg.dev///: \ + --region \ + --project \ + --no-promote +``` + +This is a guardrail, not immutable versioning: the pool's revision split remains mutable state outside Temporal. If a new revision was already promoted accidentally, send all instances back to the known-good revision explicitly: + +```bash +gcloud run worker-pools update-instance-split \ + --to-revisions==100 \ + --region \ + --project +``` + +`--to-latest` is **not** that rollback: it assigns instances to the current and future `LATEST` revision. Use it only when deliberately removing the sticky `--no-promote` behavior and restoring automatic promotion of future revisions: + +```bash +gcloud run worker-pools update-instance-split \ + --to-latest \ + --region \ + --project +``` + +After emergency recovery, return to one pool per Build ID for the next release so Temporal version routing and deployed code cannot drift independently. + +## Rolling out a new build + +1. Build and push a new image tagged with the new build ID. +2. **Create a new Worker Pool** at zero instances for that build ID, with the same runner service account. +3. Register a new Worker Deployment Version pointing at the new pool, with a build ID matching the new Worker code. +4. Confirm registration bootstrapped the pool: its `lastModifier` shows the invoker and the expected Task Queue types are bound. Treat the separate UI Validate Connection action as a read-only check. → `diagnostics.md`. +5. Set the new version current, or ramp to it. +6. **Leave the old pool in place** while Pinned Workflows still run on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. + +Only the invoker's permissions are shared across pools, so a new pool usually needs no IAM change — provided the invoker's `deploy_roles` are project-level, which is the module's default. Check `iam.md` if you scoped them to individual pools instead. + +## Rollback + +Set the previous version current again. Its pool is still there (step 6 above), and its WCI scales it back up on the next Task. Nothing needs rebuilding, and no image or revision has to be reverted — which is the payoff for keeping one pool per build. + +Rolling back through Temporal is therefore straightforward if you followed the pool-per-build discipline. If you redeployed into the same pool, setting the previous Temporal version current is not enough because both versions still address mutable pool state. Restore the known-good Cloud Run revision split as described above, provided that revision still exists; if it was deleted, there is nothing left to route back to and the old image must be redeployed deliberately. + +## Cost of the discipline + +A pool per build means pools accumulate. They cost nothing while at zero instances, but they are real resources with real names, and stale ones make the project harder to reason about. Delete a pool once its version is deleted and no Pinned Workflow can route to it — that ordering is in `setup.md`'s teardown section.