From b284912284c58785f32a1235e318eb4cae9253cb Mon Sep 17 00:00:00 2001 From: Harish Narayanappa Date: Fri, 4 Sep 2026 10:47:29 -0700 Subject: [PATCH 1/4] Add Cloud Run provider configuration references --- references/aws-lambda/constraints.md | 57 +++++ references/aws-lambda/setup.md | 48 ++-- references/aws-lambda/versioning.md | 12 + references/gcp-cloud-run/constraints.md | 81 +++++++ references/gcp-cloud-run/diagnostics.md | 137 +++++++++++ references/gcp-cloud-run/iam.md | 126 ++++++++++ references/gcp-cloud-run/observability.md | 54 +++++ references/gcp-cloud-run/self-hosted.md | 94 ++++++++ references/gcp-cloud-run/setup.md | 281 ++++++++++++++++++++++ references/gcp-cloud-run/versioning.md | 73 ++++++ 10 files changed, 931 insertions(+), 32 deletions(-) create mode 100644 references/aws-lambda/constraints.md create mode 100644 references/gcp-cloud-run/constraints.md create mode 100644 references/gcp-cloud-run/diagnostics.md create mode 100644 references/gcp-cloud-run/iam.md create mode 100644 references/gcp-cloud-run/observability.md create mode 100644 references/gcp-cloud-run/self-hosted.md create mode 100644 references/gcp-cloud-run/setup.md create mode 100644 references/gcp-cloud-run/versioning.md diff --git a/references/aws-lambda/constraints.md b/references/aws-lambda/constraints.md new file mode 100644 index 0000000..27dff8a --- /dev/null +++ b/references/aws-lambda/constraints.md @@ -0,0 +1,57 @@ +# AWS Lambda — execution-model constraints + +Consequences of Lambda's execution model: **Temporal invokes a function per unit of work, and the Worker exits when the invocation ends.** Everything here follows from that, and none of it generalizes to a provider that keeps long-lived instances alive. + +This file is the provider "diff surface." To understand what changes when moving between compute providers, compare this file with the equivalent one for the other provider — the workflow in `SKILL.md` stays the same; these constraints do not. + +## Worker lifetime is one invocation + +A Worker starts, connects, polls until its shutdown deadline buffer, drains, and exits. The invocation is the Worker's *lifetime*, not one unit of work: a single invocation can serve many Workflow and Activity Tasks from however many executions arrive in its window. Billing follows the invocation, not the work done inside it. + +## Set the invocation deadline high enough + +Providers often default to a very short timeout — Lambda's default is 3 seconds. If the first invocation times out before the Worker registers the Task Queue, the binding is never created and the Worker is never invoked again. → `setup.md` for the exact defaults, the per-SDK examples, and the GB-seconds trade-off behind choosing a value. + +## Tune the timeout triple together for long-running Activities + +The worker stop timeout controls how long the Worker waits for in-flight Tasks after it stops polling; the shutdown deadline buffer controls how long before the invocation deadline it stops polling. + +1. worker stop timeout > longest Activity runtime +2. shutdown deadline buffer > worker stop timeout + shutdown hook time +3. invocation deadline > longest Activity runtime + shutdown deadline buffer + +Raising one alone does not help. Raising only the shutdown deadline buffer makes the Worker stop polling earlier but gives in-flight Activities no more time; raising only the worker stop timeout doesn't make it stop polling earlier, so the provider may terminate the Worker first. + +Worked example: a longest Activity runtime of 5 minutes with 3 seconds of shutdown hooks means a worker stop timeout above 5 minutes, a shutdown deadline buffer above 303 seconds, and an invocation deadline of at least 10 minutes 3 seconds. + +*Symptom of getting this wrong:* Activities abandoned mid-execution and retried on a later invocation. + +If the longest Activity exceeds half the maximum invocation deadline, recommend Activity Heartbeats. → `../concepts.md`, `sdk-.md`. + +## Activities are bounded by the invocation limit + +An Activity must finish within the invocation deadline minus the shutdown deadline buffer. Workflow duration is unbounded and can span many invocations. Flag Activities that approach the provider's limit early — Lambda's ceiling is 15 minutes. + +**An Activity that cannot fit needs a different hosting strategy**, not a larger timeout: a long-lived Worker on a separate Task Queue, or a compute provider without a per-invocation ceiling. → `../concepts.md`. + +## Eager Activities are always disabled + +Every SDK's Lambda Worker package sets this and it cannot be overridden, because eager Activity execution requires a persistent connection that per-invocation Workers don't maintain. Don't suggest it as an optimization. → `sdk-.md` for the per-SDK setting names. + +## Pitfalls specific to this execution model + +These are the invocation-shaped members of the pitfall list in `SKILL.md`; the rest apply to any provider. + +1. **Failed first invocation.** When a version is created, the WCI invokes the Worker once to validate. If that invocation fails — missing env vars, bad TLS/auth config, missing dependencies, or an invocation deadline too short for the Worker to start and register the Task Queue — the Worker never connects, never polls, the binding is never created, and the Worker is never automatically invoked again. *Fix:* diagnose by manually invoking the function, and confirm the invocation deadline is set high. A successful manual invoke also establishes the binding. → `diagnostics.md`. + +2. **Timeout tuning mismatch.** See the timeout triple above. *Fix:* tune the three values together. + +3. **Invoke permission scoped to a single build.** *Symptom:* the deployment works, then the *next* release cannot be invoked, with an error that looks like a connection or configuration problem rather than a permissions one. *Cause:* the grant named one immutable build, and the new release is a different resource. *Fix:* scope the grant to cover the base function ARN **and** its published versions (`function:name` and `function:name:*` — the wildcard form does not cover the unqualified ARN). → `iam.md`. + +## What does *not* follow from this model + +Stated explicitly, because these are easy to over-generalize from Lambda: + +- **"Serverless Worker" does not imply a per-invocation lifetime.** A provider that scales a pool of long-lived instances is still a Serverless Worker driven by the same WCI, with the same Worker Deployment Versioning, and none of the constraints above apply to it in the same form. +- **The shutdown deadline buffer is a property of the Lambda Worker packages**, not of Temporal. +- **Connection-per-invocation is a Lambda property.** Anything reasoning from "the connection is not persistent" — eager Activities being the example above — needs rechecking against a provider that holds the connection for an instance's lifetime. diff --git a/references/aws-lambda/setup.md b/references/aws-lambda/setup.md index 30083bd..94016b4 100644 --- a/references/aws-lambda/setup.md +++ b/references/aws-lambda/setup.md @@ -30,32 +30,23 @@ Steps 4–6 and the CLI troubleshooting paths use the `temporal` CLI. Install it ```bash export TEMPORAL_ADDRESS="..tmprl.cloud:7233" export TEMPORAL_NAMESPACE="." -export TEMPORAL_API_KEY="" +printf 'Temporal API key: ' >&2 +IFS= read -r -s TEMPORAL_API_KEY +printf '\n' >&2 +export TEMPORAL_API_KEY ``` -or configure a profile and pass `--profile prod` on each command: +An existing profile or environment is also valid; pass `--profile prod` or `--env prod` on each command. Do not create or update its API key with `config set --value` or `env set --value` — the value would be exposed in shell history and process arguments. If no credential is already configured, use the private environment-variable prompt above. -```bash -temporal --profile prod config set --prop address --value "..tmprl.cloud:7233" -temporal --profile prod config set --prop namespace --value "." -temporal --profile prod config set --prop api_key --value "" -``` - - -or configure an environment and pass `--env prod` (or set `TEMPORAL_ENV`): - -```bash -temporal env set --env prod --key address --value "..tmprl.cloud:7233" -temporal env set --env prod --key namespace --value "." -temporal env set --env prod --key api-key --value "" -``` - -**Do not assume which of the three a user has, and do not migrate them.** `--env` (YAML, `temporal env`) is the long-standing mechanism; `--profile` (TOML, `temporal config`) is newer and the CLI still marks it EXPERIMENTAL. Both are supported — work with whichever is already configured. Read the existing values rather than asking the user to re-enter them: +**Do not assume which mechanism a user has, and do not migrate them.** `--env` (YAML, `temporal env`) is the long-standing mechanism; `--profile` (TOML, `temporal config`) is newer and the CLI still marks it EXPERIMENTAL. Inspect only the non-secret properties needed for the deployment; a broad `env get` or `config get` can print stored credentials: ```bash -temporal env get --env prod # --env mechanism -temporal config get --prop address # --profile mechanism +temporal env get --env prod --key address +temporal env get --env prod --key namespace +temporal --profile prod config get --prop address +temporal --profile prod config get --prop namespace ``` + - For Temporal Cloud the Namespace is the fully-qualified `.`, not the bare name. - Supplying an API key auto-enables TLS; no cert flags are needed for API-key auth. @@ -72,7 +63,7 @@ If this fails with an auth error, note first that this is a **frontend** call | | Control plane (accounts, Namespaces, API keys) | Namespace frontend (Workflows, Worker Deployments) | |---|---|---| | Interactive | `tcld login` | `temporal ...` with address + namespace | -| Headless | `--api-key` / `TEMPORAL_CLOUD_API_KEY` | `TEMPORAL_API_KEY` | +| Headless | `TEMPORAL_CLOUD_API_KEY` | `TEMPORAL_API_KEY` | **Use `tcld` for every Temporal Cloud control-plane operation** — accounts, Namespaces, API keys, users, service accounts. Do not use the unified CLI's `temporal cloud …` subcommands for them. @@ -85,13 +76,7 @@ Two `tcld` mechanics worth knowing before you run it in an agent shell: - `tcld login --disable-pop-up` prints the URL instead of opening a browser. Auto-open is unreliable over SSH, in containers, and in remote sessions, and the user needs the URL in the conversation either way. - `tcld` prompts for confirmation before mutating operations. Non-interactively, pass the global `--auto_confirm` (note the underscore) or set `AUTO_CONFIRM=true`, then read the resulting state back — without it the command exits clean having changed nothing. -**Go to the API key first.** It requires no CLI login, no browser handshake, and works on every account type: - -```bash -export TEMPORAL_ADDRESS="..tmprl.cloud:7233" -export TEMPORAL_NAMESPACE="." -export TEMPORAL_API_KEY="" -``` +**Go to the API key first.** It requires no CLI login, no browser handshake, and works on every account type. Use the private environment-variable prompt above. Have the user create the key in the Cloud UI, signing in however they normally do, and confirm the address against the endpoint shown on the Namespace page — some Namespaces have regional endpoints that do not follow the pattern above. Never ask them to paste the key into the conversation. @@ -100,10 +85,9 @@ Have the user create the key in the Cloud UI, signing in however they normally d ```bash tcld namespace list # full Namespace objects — every name with its region and endpoint tcld namespace get -n # one Namespace -tcld apikey create --name --duration ``` -`apikey create` mints a key for the calling user and creates a long-lived credential in their account — offer it and get explicit approval, never silently. `tcld` is not guaranteed present: check `command -v tcld`, and read `tcld --help` for the flags you are about to pass. +`tcld` is not guaranteed present: check `command -v tcld`, and read `tcld --help` for the flags you are about to pass. Create API keys in the Cloud UI so their values never enter the agent transcript. **When a control-plane login fails, stop — do not debug it, retry it, or install another CLI.** Some accounts cannot complete a `tcld` login at all, and no flag, plugin upgrade, or alternate CLI changes that. Retrying burns turns without converging, and the browser path below reaches the same end state anyway. @@ -139,9 +123,9 @@ See the selected SDK reference's **Build and package** section. **A freshly created execution role may not be assumable immediately.** `create-function` can fail with an assume-role / "cannot be assumed by Lambda" error because of IAM propagation delay. Wait a few seconds and retry; it is not a policy error, so do not start rewriting the trust policy. -**Operator CLI config does not reach the function.** All three CLI mechanisms above — exported `TEMPORAL_*` variables, `--profile`, and `--env` — configure the `temporal` CLI on the operator's machine only. The function reads its own environment, set by the `--environment` block below (or a secret store). A user with a working `--env prod` or `--profile prod` still needs every value written into that block; nothing is inherited. Treat their CLI configuration as the *source* of the values, not a substitute for setting them. +**Operator CLI config does not reach the function.** All three CLI mechanisms above — exported `TEMPORAL_*` variables, `--profile`, and `--env` — configure the `temporal` CLI on the operator's machine only. The function reads its own environment, set by the `--environment` block below (or a secret store). A user with a working `--env prod` or `--profile prod` still needs every value written into that block; nothing is inherited. Use the CLI configuration only for non-secret values, and collect the API key through the private prompt above. -**Resolve the values before building the block, and check they are not empty.** The heredoc below expands shell variables, which hold values only under the env-var mechanism. Under `--env` or `--profile` they are unset, and an unset variable expands to an empty string: the JSON stays valid, `create-function` succeeds, and the function deploys with `"TEMPORAL_ADDRESS":""` — failing at first invocation with a connection error that looks nothing like its cause. Populate them from whichever mechanism the user actually has (`temporal env get --env prod`, `temporal config get --prop address`), then guard: +**Resolve the values before building the block, and check they are not empty.** The heredoc below expands shell variables, which hold values only under the env-var mechanism. Under `--env` or `--profile` they are unset, and an unset variable expands to an empty string: the JSON stays valid, `create-function` succeeds, and the function deploys with `"TEMPORAL_ADDRESS":""` — failing at first invocation with a connection error that looks nothing like its cause. Populate the address and Namespace from the non-secret lookups above, and have the user set `TEMPORAL_API_KEY` with the private prompt; never retrieve a stored key into the agent transcript. Then guard: ```bash : "${TEMPORAL_ADDRESS:?resolve this before deploying}" diff --git a/references/aws-lambda/versioning.md b/references/aws-lambda/versioning.md index 1959d63..2d4f99d 100644 --- a/references/aws-lambda/versioning.md +++ b/references/aws-lambda/versioning.md @@ -54,6 +54,18 @@ The command prints the `FunctionArn` for the new version, for example `arn:aws:l For development or non-critical workloads, you can skip `publish-version` and use an unqualified ARN to iterate faster. +### What an unqualified ARN costs + +An unqualified ARN (no version suffix) points at `$LATEST`, which changes on every redeploy. Deploying replay-unsafe code that way causes non-determinism errors for in-flight Workflows, **even ones annotated Pinned**. + +Pinned and Auto-Upgrade control how Workflows move between Worker Deployment Versions in Temporal; neither changes how a version targets Lambda. Both expect a qualified ARN naming one immutable function version. + +| Versioning Behavior | With versioned Lambda ARN | Without versioned Lambda ARN | +|---|---|---| +| **Pinned** | Existing Workflows stay on their original Lambda function version until they complete. | Existing Workflows stay on their original Worker Deployment Version, but the underlying Lambda code has already changed since `$LATEST` updated at redeploy. The new code must be replay-compatible. | +| **Auto-Upgrade** | Existing Workflows move to the new Worker Deployment Version and its new Lambda function version at the next Workflow Task after you move the Current Version. | The Lambda redeploy already changed the code for all versions. Setting the Current Version only changes routing, not which code runs. | + + To roll back, revert the Temporal Current Version with `temporal worker deployment set-current-version`. The previous Worker Deployment Version still points at its original Lambda function version and is ready to receive traffic again. ```bash diff --git a/references/gcp-cloud-run/constraints.md b/references/gcp-cloud-run/constraints.md new file mode 100644 index 0000000..178f503 --- /dev/null +++ b/references/gcp-cloud-run/constraints.md @@ -0,0 +1,81 @@ +# GCP Cloud Run — execution-model constraints + + + +Consequences of Cloud Run's execution model: **Temporal resizes a pool of long-lived instances, scaling it to zero when there is no work.** Temporal does *not* invoke your Worker per Task. + +This file is the provider "diff surface." Compare it with `../aws-lambda/constraints.md` to see what changes between providers — the workflow in `SKILL.md` is the same; these constraints are not. + +## Worker lifetime is an instance, not an invocation + +Each pool instance runs **standard long-lived Worker code**: it connects, registers Workflows and Activities, and polls the Task Queue for its whole lifetime. There is no handler, no per-Task lifecycle, and **no serverless Worker package** — the SDK-specific packages in `../aws-lambda/sdk-.md` are AWS Lambda only. Some SDKs add optional Cloud Run conveniences; none are required. + +The WCI controls how many instances run; each instance manages its own polling and Task processing. + +## None of Lambda's timing constraints apply + +Explicitly, so they are not carried across: + +- **No invocation deadline.** Nothing bounds how long a Worker lives except scale-in. +- **No shutdown deadline buffer.** That is a property of the Lambda Worker packages, not of Temporal. +- **No timeout triple to tune.** Two of its three values do not exist here. +- **Activities are not bounded by an invocation limit.** An Activity too long for Lambda's 15-minute ceiling is a reason to choose Cloud Run. +- **Eager Activities are not disabled by the platform.** Lambda's packages force this off because a per-invocation Worker holds no persistent connection; a pool instance holds its connection for its lifetime, so that reasoning does not transfer. Confirm against your SDK rather than assuming either way. + +Cloud Run still sends `SIGTERM` during scale-in and can send `SIGKILL` ten seconds later. Handle `SIGTERM` and configure the SDK's graceful-shutdown timeout below that window; this is separate from Lambda's deadline buffer and timeout triple. → `sdk-.md`. + +## What bounds an Activity instead: scale-in + +**The WCI decides when to remove an instance from Task Queue activity, not from what any individual instance is doing.** It does not track how long an instance has been running or whether it is mid-Activity, so the instance Cloud Run stops may be one that is still executing work. + +Graceful shutdown lets short work drain but cannot guarantee an Activity will finish. **Use Activity Heartbeats** so interrupted work resumes from its last recorded progress instead of restarting. + +*Symptom of ignoring this:* Activities failing partway and retrying from the beginning, correlated with the pool shrinking. + +## Autoscaling behavior + + + +The WCI combines two mechanisms: + +- **Immediate** — bring up instances when a Task arrives and no Worker is free to take it (a sync match failure). Absorbs bursts without waiting for the evaluation cycle. +- **Periodic** — rate-based re-sizing. The WCI measures how fast Tasks arrive and how fast one Worker processes them, computes the instance count needed, and applies it through the Cloud Run admin API. + +It sizes to a **target utilization of 80% by default** rather than loading every Worker fully, so there is headroom to take new Tasks immediately, and adds instances on top when a backlog exists. + +**Scale-in is deliberately more conservative than scale-out:** it holds capacity while sync match failures are still occurring and applies a cooldown before reducing the pool. With no work, it can scale to zero; the next sync match failure or backlog scales it back up. + +The scaler defaults are **minimum `0`, maximum `30`, initial count `0`, and target utilization `0.8`**. Configure them in the version's Scaling and Lifecycle settings or with the Temporal CLI. The four CLI flags are coupled: omit all four to use the defaults, or provide `--gcp-cloud-run-min-instances`, `--gcp-cloud-run-max-instances`, `--gcp-cloud-run-initial-instances`, and `--gcp-cloud-run-utilization-target` together. A partial group is rejected. + +An initial count and minimum of zero do not suppress registration. The rate-based algorithm temporarily requests at least one instance when the version is registered so its Task Queues can bind, then normal scaling can return the pool to zero. A pool that later stops growing under backlog is either at its configured maximum or at a regional Cloud Run quota. + +## One Worker Pool per Worker Deployment Version + +**The compute configuration names a project, region, and pool — not a revision.** Temporal runs whichever revision the pool serves at the time, which ties a pool to a single build. A new build needs a **new pool**, and the Build ID belongs in the pool name to keep that mapping visible. + +Keep an older version's pool in place while Pinned Workflows are still running on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. + +**The mutable-build hazard, in its Cloud Run form:** deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run promotes it to every instance by default. The version does not change, but the code behind it does — causing non-determinism errors for in-flight Workflows, **including Pinned ones**. This is the same failure as pointing Lambda at a mutable `$LATEST`, reached through revisions instead of ARNs. → `versioning.md`. + +## Do not share a Task Queue with long-lived Workers + +Stronger than the Lambda guidance, and for a different reason. On Lambda the two scaling loops fight; here **the pool scales up to cover the Task Queue's full workload even when long-lived Workers are already handling all of it**, so you run and pay for duplicate capacity. + +The WCI sizes the pool from the rate of Tasks arriving on the version's Task Queues, and nothing in that measurement accounts for the long-lived Workers. Sync matching to a long-lived Worker suppresses the *immediate* scale-up, but the periodic re-sizing scales the pool up regardless. Fixing poller counts on the long-lived side does not help — use separate Task Queues. + +## Pre-release status + +Cloud Run support is **Pre-release, and its APIs may change in backwards-incompatible ways.** Access is not open: the user must create a support ticket or contact their account team. Confirm access exists before planning a deployment — this is a gate no amount of correct configuration gets past. + +## Provider-side issue worth knowing + +Google currently reports **high deployment latency creating or updating Cloud Run resources in some regions, including `us-central1`**, and recommends deploying elsewhere while it is open. If pool operations are slow, check [Cloud Run known issues](https://cloud.google.com/run/docs/known-issues) before assuming misconfiguration. + +## What does *not* follow from this model + +- **"Serverless Worker" does not imply standard Worker code.** On Lambda you write a handler against a provider-specific package; here you write an ordinary Worker. Advice about handlers, configure callbacks, or tuned package defaults does not transfer. +- **Scale-to-zero is not per-Task.** An idle pool costs nothing, but a running instance is billed for its lifetime, not per unit of work. +- **A passing Validate Connection is weaker here than on Lambda.** It exercises only the read permission and starts no instance. → `diagnostics.md`. diff --git a/references/gcp-cloud-run/diagnostics.md b/references/gcp-cloud-run/diagnostics.md new file mode 100644 index 0000000..573c96d --- /dev/null +++ b/references/gcp-cloud-run/diagnostics.md @@ -0,0 +1,137 @@ +# GCP Cloud Run — Diagnostics & troubleshooting + + + +## Scaling flow (when working correctly) + + + +1. You deploy the Worker image to a Worker Pool at zero instances. +2. You create a Worker Deployment Version pointing at that pool. This starts a WCI Workflow. +3. An instance starts, the Worker polls, and the server **binds the Task Queue** to the version. +4. The WCI monitors that Task Queue; the Matching Service also signals it when a Task arrives with no free Worker. +5. As work arrives the WCI raises the instance count through the Cloud Run admin API; as it drains, lowers it, possibly to zero. + +**Temporal does not invoke your Worker per Task** — it changes how many instances run. Diagnose accordingly: "was it invoked?" is the wrong first question here. + +## Start here: read the pool's annotations + +This is the single highest-value command, and it has no Lambda equivalent: + +```bash +gcloud run worker-pools describe \ + --region --project --format=yaml +``` + +Three fields under `metadata.annotations`: + +| Field | What it tells you | +|---|---| +| `run.googleapis.com/manualInstanceCount` | The instance count currently requested. `0` means no Worker is running. | +| `run.googleapis.com/scalingMode` | Should be `manual` — the WCI scales by writing the manual instance count. | +| `serving.knative.dev/lastModifier` | Who last changed the pool. **If Temporal has ever scaled it, this is the invoker service account.** | + +**`lastModifier` splits the whole problem in two:** + +- Still the account you deployed with → **Temporal has never successfully written to the pool.** Work through "pool is not scaling up" below. +- The invoker service account → **Temporal is reaching the pool**; the fault is in the Worker. Skip to "instances running but Tasks not completing." + +This is the Cloud Run counterpart of Lambda's "is the Task Queue bound?" checkpoint — one read that eliminates most of the surface. + +## The Worker Pool is not scaling up + +### 1. Validate Connection — and know what it does *not* prove + +Workers → Deployments → select deployment → Actions → **Validate Connection**. For Cloud Run this impersonates the invoker and reads the pool, confirming three things: the compute configuration names a pool that exists, Temporal can impersonate the invoker, and the invoker can read. + +**It starts no instance and does not exercise `run.workerPools.update`.** Version registration is different: its Task Queue bootstrap does update the pool. An invoker with read but not update permission can therefore pass this manual validation, but version registration or a later resize fails. If validation succeeds but registration never changes `lastModifier`, **check the update permission**. + +On failure, check each part of the compute configuration against the pool: + +- **Project, region, pool name.** Temporal addresses the pool as `projects//locations//workerPools/`. **A wrong region reports the pool as not found, identical to a wrong name** — so a "not found" does not tell you which field is wrong. +- **Impersonation.** Temporal's identity needs `roles/iam.serviceAccountTokenCreator` on the invoker. The Terraform module grants this on Cloud; self-hosted grants it to the server's GCP identity. +- **Invoker permissions.** `run.workerPools.get` to read, `run.workerPools.update` to scale. + +→ `iam.md`. + +### 2. Did the registration bootstrap bind the Task Queue? + +```bash +temporal worker deployment describe-version \ + --namespace --deployment-name --build-id --report-task-queue-stats +``` + +The server creates the binding when a Worker running that version connects and polls. **No Task Queues listed means no Worker has polled successfully under this version.** + +Registration is supposed to bootstrap this: the WCI reads the pool, updates its manual instance count to at least one, and Cloud Run starts an instance. An absent binding means that sequence failed or the instance started but did not connect under the expected deployment name and build ID. Inspect the WCI's `ValidateSpec`/registration Activity failure, the pool's `lastModifier` and instance count, and then the pool logs. Do not wait for a first Workflow to repair registration. + +### 3. Is the version current? + +The registration bootstrap does not make the version current. New traffic routes only after the version is current, and a CLI-created version is not current automatically. Verify with `temporal worker deployment describe`. + +**A `set-current-version` run without `--yes` may have done nothing** — it prompts, and non-interactively exits without applying the change, which reads as success. + +### 4. Is the pool at its ceiling? + +If instances are running but the count stops growing while backlog builds: + +- **The maximum defaults to 30.** Raise it in the version's Scaling and Lifecycle settings, or update the existing version from the CLI: + + ```bash + temporal worker deployment update-version-compute-config \ + --namespace \ + --deployment-name \ + --build-id \ + --gcp-cloud-run-min-instances 0 \ + --gcp-cloud-run-max-instances \ + --gcp-cloud-run-initial-instances 0 \ + --gcp-cloud-run-utilization-target 0.8 + ``` + + The four scaler flags must be supplied together, even when changing only the maximum. Choose values appropriate for the version rather than blindly copying this default-shaped example; the initial count must be between the minimum and maximum. +- If the count stalls *below the configured maximum*, check the project's [Cloud Run quotas](https://cloud.google.com/run/quotas) for that region. Cloud Run caps instances and CPU per region regardless of what the WCI requests. + +## Instances are running but Tasks are not completing + +### Read the pool logs + +```bash +gcloud run worker-pools logs read --region --project +``` + +**The pool produces no logs while scaled to zero** — read them while an instance is up. An empty log is not evidence of failure. + +Common errors: + +- **Connection failures** — check `TEMPORAL_ADDRESS` and `TEMPORAL_NAMESPACE` on the pool. Self-hosted: verify network reachability from Cloud Run to the frontend. +- **Missing secrets** — the instance cannot read the API key or TLS material. The **runner** service account needs `roles/secretmanager.secretAccessor` on the secret. That is the account in `spec.template.spec.serviceAccountName`, **not the invoker.** This is the most common consequence of confusing the two. +- **Authentication errors** — key invalid, expired, or without access to the Namespace. +- **`TransportError: … NativeCertsNotFound`** — a Rust-core SDK in a minimal base image with no CA certificates. Documented for TypeScript on Cloud Run; the same class of failure as .NET's `SSL_CERT_FILE` problem on Lambda. Install `ca-certificates` in the image. → `setup.md`. + +### Deployment name and build ID + +Instances start and poll but no Task is ever processed → the name or build ID in the code does not match the version. The Worker polls under a version the WCI does not manage, **so its polls never satisfy the Tasks the WCI is scaling for.** + +Note the different signature from Lambda: there, a mismatch causes rapid repeated invocations. Here there are no invocations to count — you see a running pool, healthy-looking logs, and no progress. + +## Activities interrupted mid-execution + +Activities failing partway and retrying from the beginning, correlated with the pool shrinking, means **scale-in is stopping instances that are still working.** The WCI does not track whether the instance Cloud Run stops is mid-Activity. + +This is expected behavior, not a misconfiguration. Confirm the Worker handles `SIGTERM` and has a non-zero graceful-shutdown timeout below Cloud Run's ten-second termination window; this lets short work drain. Long-running work still needs **Activity Heartbeats** so a retry resumes from its last recorded progress. → `constraints.md`, `sdk-.md`. + +## Rule out a GCP-side cause + +If every check passes, the cause may be in Cloud Run rather than your configuration: + +- [Cloud Run known issues](https://cloud.google.com/run/docs/known-issues) — includes issues affecting how long pool operations take. +- [Google Cloud Service Health](https://status.cloud.google.com/) — active incidents by product and region. + +The usual fix is to wait it out or move region. **Moving region means creating a new pool and updating the compute configuration**, since Temporal addresses a pool by project, region, and name. + +## Never create or manage the WCI + +Unchanged from Lambda: Temporal creates one per Worker Deployment Version with a compute provider, and a running WCI is not evidence that scaling works—it continues-as-new while its Activities fail. Read its history for Activity failures, and prefer the pool's `lastModifier` annotation as the cheapest proof of a successful write. Do not enumerate Cloud Run resources across regions to reverse-engineer state. diff --git a/references/gcp-cloud-run/iam.md b/references/gcp-cloud-run/iam.md new file mode 100644 index 0000000..08cb307 --- /dev/null +++ b/references/gcp-cloud-run/iam.md @@ -0,0 +1,126 @@ +# GCP Cloud Run — IAM & permissions + + + +Three identities, as on AWS Lambda — the **operator** (whose credentials run the commands), the **runner service account** (what the pool runs as), and the **invoker service account** (what Temporal impersonates). The mapping to Lambda's roles is close enough to be useful and different enough to be dangerous if assumed. + +| Concept | AWS Lambda | GCP Cloud Run | +|---|---|---| +| The compute's own identity | Execution role, trusted by `lambda.amazonaws.com` | **Runner service account**, set with `--service-account` | +| Temporal's identity | Invocation role, assumed via `sts:AssumeRole` + External ID | **Invoker service account**, reached by **impersonation** | +| Mechanism | `sts:AssumeRole` with a confused-deputy guard | `roles/iam.serviceAccountTokenCreator` — **no External ID equivalent** | +| What Temporal does with it | Invokes the function | Reads and **updates the pool's instance count** via the Cloud Run admin API | +| Infrastructure as code | CloudFormation template (shipped in `assets/`) | **Terraform module**, `serverless-workers/gcp/cloud-run` | + +**The two service accounts are not interchangeable**, and confusing them is the single most likely IAM mistake here. The runner runs the pool and never scales it; the invoker scales the pool and never runs it. + +## Runner service account + +The runtime identity the pool's instances use to reach other Google Cloud services. Set in `setup.md` Step 4 with `gcloud run worker-pools deploy --service-account`. It may be an account that already exists; a dedicated one is preferred. + +**It needs no baseline role to run the Worker.** Cloud Run collects `stdout` and `stderr` into Cloud Logging through its own infrastructure, and the Cloud Run *service agent* — not the runner — pulls the container image. Grant only what your code actually reaches: + +- `roles/secretmanager.secretAccessor` on each secret you mount, including the Temporal API key. +- `roles/logging.logWriter` **only if** the Worker writes through the Cloud Logging API rather than stdout/stderr. +- Whatever else your Workflows and Activities call. + +This differs from Lambda, where `AWSLambdaBasicExecutionRole` is effectively mandatory because the execution role is what creates the log group. Here, logging works with no grant at all. + +## Invoker service account + +The identity Temporal Cloud impersonates to read and scale the pool. Two grants make it work: + +- Temporal's identity receives **`roles/iam.serviceAccountTokenCreator`** on the invoker, so it can impersonate it. +- The invoker receives a project-level Cloud Run role with at least **`run.workerPools.get`** (read) and **`run.workerPools.update`** (scale). `roles/run.developer` includes both. + +The invoker also needs **`roles/iam.serviceAccountUser` on the runner service account**, which Cloud Run requires in order to attach that identity when it scales the pool. The Terraform module applies this. + +### The read/update split is a real trap + +`run.workerPools.get` alone is enough for the UI's **Validate Connection** action to pass. That manual action never exercises `run.workerPools.update`. Version registration does: the WCI reads the pool and then updates its manual instance count to bootstrap Task Queue registration. An invoker that can read but not update therefore passes manual validation but fails version registration or a later resize. + +There is no Lambda analogue: there, the registration invocation uses the same invoke permission as real traffic. For Cloud Run, verify the registration bootstrap and `lastModifier` rather than trusting the separate green Validate Connection result. → `diagnostics.md`. + +## The Terraform module + +Temporal publishes [`serverless-workers/gcp/cloud-run`](https://github.com/temporalio/terraform-modules/tree/main/modules/serverless-workers/gcp/cloud-run), which creates the invoker service account and applies the grants. + +**Get the template from the Cloud UI, not from here.** Under **Workers → Create Worker Deployment → Access**, Temporal Cloud emits a template with `impersonator_service_account_emails` already filled in for your account. Those values are account-specific, which is why every published snippet shows a placeholder. + +```hcl +module "serverless-worker-cloud-run" { + source = "github.com/temporalio/terraform-modules//modules/serverless-workers/gcp/cloud-run" + + project_id = "" + invoker_account_id = "temporal-worker-pool-invoker" + + impersonator_service_account_emails = [ + "", + ] + + runner_service_account_email = "temporal-worker-pool-runner@.iam.gserviceaccount.com" +} +``` + +| Variable | Required | Description | +|---|---|---| +| `project_id` | Yes | Project hosting the pool and the invoker. | +| `invoker_account_id` | Yes | Name for the invoker the module creates; email becomes `@.iam.gserviceaccount.com`. The template supplies one. | +| `impersonator_service_account_emails` | Yes | Temporal Cloud's service accounts, granted `serviceAccountTokenCreator` on the invoker. **From the UI template.** For self-hosted, the GCP identity the server runs as. | +| `runner_service_account_email` | Yes | The runner from `setup.md` Step 4. The module grants the invoker `roles/iam.serviceAccountUser` on it. | +| `invoker_display_name` | No | Defaults to `Temporal Serverless Worker Pool Invoker`. | +| `deploy_roles` | No | Project-level Cloud Run roles for the invoker. Defaults to `roles/run.developer`. A substitute must include `run.workerPools.get` and `run.workerPools.update`. | + +```bash +terraform init +terraform apply +``` + +Use the **`invoker_email`** output as `--gcp-cloud-run-service-account` when registering the version. + +### Treat the module as shared, pre-existing infrastructure + +Same discipline as Lambda's CloudFormation stack, different tool. One invoker can serve several pools, so before creating a second one, look for an existing account and consider reusing it. **Do not `terraform destroy` state you did not create**, and note that the module's default `invoker_account_id` collides the same way Lambda's default `RoleName` does — an earlier deployment in the project may already own it. + +## Operator GCP permissions + +The identity running the `gcloud`/Terraform commands needs, at minimum: + +| Step | Operator needs | +|---|---| +| Submit the image build (`setup.md` Step 3) | `cloudbuild.builds.create`; permission to use the selected build service account when one is specified | +| Push the built image | The Cloud Build execution service account needs Artifact Registry Writer on the repository when it is user-specified, cross-project, or lacks the same-project default access | +| Create the Worker Pool (Step 4) | `run.workerPools.create`/`update`, and `iam.serviceAccounts.actAs` on the **runner** to attach it | +| Create secrets | Secret Manager admin on the secrets used | +| Apply the Terraform module (Step 5) | Service-account creation plus IAM policy binding on the project and on the runner | +| Read pool state and logs (verify, diagnose) | `run.workerPools.get`, Cloud Logging read | + +`iam.serviceAccounts.actAs` on the runner is the Cloud Run counterpart of Lambda's `iam:PassRole` on the execution role, and fails the same way — a pool create that is denied despite having Cloud Run permissions. + +### Preflight + +Run before anything that creates or modifies GCP resources. Confirm the five required APIs appear in the enabled-service output, then inspect the names the deployment intends to use: + +```bash +gcloud auth list # which identity +gcloud config get-value project # which project +gcloud services list --enabled --project --format='value(config.name)' +gcloud run worker-pools list --region >/dev/null && echo "cloud run: ok" +gcloud iam service-accounts list >/dev/null && echo "iam read: ok" +gcloud artifacts repositories describe --location --project +gcloud iam service-accounts describe --project +gcloud secrets describe --project +terraform version +``` + +Required services: `run.googleapis.com`, `artifactregistry.googleapis.com`, `cloudbuild.googleapis.com`, `secretmanager.googleapis.com`, `iam.googleapis.com`, `iamcredentials.googleapis.com`, and `cloudresourcemanager.googleapis.com`. A `describe` returning Not Found is acceptable for a clean project; record that the named resource will be created and include it in the approval list. A permission error is not the same as absence—stop and resolve access before creating anything. + +For a same-project build using Cloud Build's default service account, Artifact Registry access is normally provided automatically. If the build uses a user-specified service account, the repository is in another project, or an organization policy removed the default grant, inspect that account and grant `roles/artifactregistry.writer` on this repository before submitting the build. + +**Classify an authentication failure before acting on it.** An absent or expired credential (`gcloud auth login`, or `gcloud auth application-default login` for Terraform) is recoverable in a minute; an identity that resolves but is denied a specific action is a real permissions problem. Never collect credentials in conversation and never ask the user to paste a service account key — Google recommends against long-lived keys outright. + +**Confirm the project explicitly before creating anything.** `gcloud config get-value project` is ambient state that is easy to be wrong about, exactly like an AWS profile pointing at an unintended account. Name the project in the approval list and verify it, rather than trusting the default. diff --git a/references/gcp-cloud-run/observability.md b/references/gcp-cloud-run/observability.md new file mode 100644 index 0000000..489a0f0 --- /dev/null +++ b/references/gcp-cloud-run/observability.md @@ -0,0 +1,54 @@ +# GCP Cloud Run — observability + + + +## There is nothing serverless-specific to configure + +**A Cloud Run Serverless Worker emits the same traces and metrics as a Worker anywhere else.** It is an ordinary long-lived Worker, so the SDK's normal metrics and OpenTelemetry tracing setup applies unchanged, and each SDK's general observability guide is the right reference. + +This is a genuine simplification over AWS Lambda, and the contrast is worth keeping in mind: + +| | AWS Lambda | GCP Cloud Run | +|---|---|---| +| OTel wiring | Per-SDK helper in the serverless package (`ApplyDefaults`, `apply_defaults`, `OtelLambdaWorkerConfigurationHelper`, a separate `…Aws.Lambda.OpenTelemetry` package for .NET) | **None — use the SDK's standard setup** | +| Collector | ADOT Lambda layer, plus a custom `otel-collector-config.yaml` because the default does not route OTLP to the traces pipeline | No layer; export as you would from any container | +| Env var | `OPENTELEMETRY_COLLECTOR_CONFIG_URI` / `_FILE`, differing by SDK | n/a | +| Flush timing | Must flush before the invocation deadline, or telemetry is lost | No deadline to beat | +| IAM | Execution role needs X-Ray and CloudWatch permissions | Runner needs nothing for stdout/stderr logging | + +Some SDKs add optional Cloud Run conveniences, such as OpenTelemetry helpers. **They are optional**, and where they exist they are documented in that SDK's Cloud Run guide. + +## Logs + +Cloud Run collects `stdout` and `stderr` into [Cloud Logging](https://cloud.google.com/run/docs/logging) through its own infrastructure. **The runner service account needs no grant for this** — `roles/logging.logWriter` is required only if the Worker writes through the Cloud Logging API instead. → `iam.md`. + +Read a pool's logs: + +```bash +gcloud run worker-pools logs read --region --project +``` + +**The pool produces no logs while scaled to zero.** Read them while an instance is up; an empty log is not evidence of a problem. This is the most common source of confusion when checking a Cloud Run Worker's health, and it has no Lambda equivalent — a Lambda log group retains history after the invocation ends. + +## Memory: give the runtime the instance, not the host + +A Worker Pool instance defaults to **512 MiB**; raise `--memory` when creating the pool if the Worker needs more. Two runtimes need to be told about the container limit explicitly, or they size themselves to a fraction of it: + +| SDK | Setting | Why | +|---|---|---| +| Java | `-XX:MaxRAMPercentage=75` | The JVM reads the container limit but defaults max heap to 25% of it, leaving most of a small instance unused. | +| TypeScript | `NODE_OPTIONS=--max-old-space-size=`, ~80% of the instance limit | Node's default heap is unrelated to the container limit. | + +Both are container-sizing concerns rather than Temporal ones, but they surface as Worker instability under load and are easy to miss. → `setup.md`. + +## What to watch that is specific to this provider + +Standard Worker metrics tell you about the Worker. Two Cloud Run-specific signals tell you about the *scaling*, and neither comes from the SDK: + +- **`run.googleapis.com/manualInstanceCount`** on the pool — what the WCI has asked for. Persistently `0` while a backlog exists is the headline symptom of a scaling failure. +- **`serving.knative.dev/lastModifier`** on the pool — whether Temporal has ever successfully written to it. The cheapest proof that impersonation and the update permission both work. + +Both are read with `gcloud run worker-pools describe … --format=yaml`. → `diagnostics.md`. diff --git a/references/gcp-cloud-run/self-hosted.md b/references/gcp-cloud-run/self-hosted.md new file mode 100644 index 0000000..282e5bc --- /dev/null +++ b/references/gcp-cloud-run/self-hosted.md @@ -0,0 +1,94 @@ +# GCP Cloud Run — self-hosted Temporal Service setup + + + +Serverless Workers require **Temporal Service v1.31.0 or later**. Complete this page before following `setup.md`. + +Four prerequisites: network reachability, enable the WCI, give the server a GCP identity, create the invoker service account. + +## 1. Cloud Run instances must reach the Temporal Service + +The frontend must be reachable **from the Worker Pool instances**. If the Service runs on a private network, that likely means [Direct VPC egress](https://cloud.google.com/run/docs/configuring/vpc-direct-vpc) or a [Serverless VPC Access connector](https://cloud.google.com/run/docs/configuring/vpc-connectors). + +Note the direction: instances dial *out* to Temporal. Nothing needs to reach *into* Cloud Run — Temporal drives the pool through the Cloud Run admin API, not by connecting to your instances. + +## 2. Enable the Worker Controller Instance + +The WCI is **disabled by default** and enabled through [Temporal Service dynamic configuration](https://docs.temporal.io/references/dynamic-configuration): + +```yaml +workercontroller.enabled: + - value: true + +workercontroller.compute_providers.enabled: + - value: + - gcp-cloud-run + +workercontroller.scaling_algorithms.enabled: + - value: + - rate-based +``` + +**Cloud Run requires the `rate-based` algorithm.** Because a pool is a set of long-lived instances, the WCI resizes it from arrival and backlog rates. The `no-sync` algorithm applies only to providers invoked once per Task, and **pairing it with `gcp-cloud-run` is rejected** — a concrete way the two providers' execution models surface in server configuration. + +To enable per Namespace instead of globally: + +```yaml +workercontroller.enabled: + - value: true + constraints: + namespace: 'your-namespace' +``` + +The Service watches the file and applies updates **without a restart**. + +Two optional global keys cover multi-hop impersonation: + +| Key | Purpose | +|---|---| +| `workercontroller.compute_providers.gcp.intermediary_service_accounts` | Service accounts to impersonate in sequence before the invoker. Unset means the server impersonates the invoker directly. | +| `workercontroller.compute_providers.gcp.first_delegate_as_base` | When `true`, the first entry is the identity the server's ambient credentials impersonate directly and the rest are passed as token-creator delegates. Defaults to `false`, passing the whole chain as delegates. | + +## 3. Give the Temporal Service a GCP identity + +The server impersonates the invoker service account, so it must first run as a GCP identity permitted to do so. + +**On GCP (GCE, GKE):** the attached service account is used automatically through [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). No extra credential configuration — you grant *that* account impersonation rights in step 4. + +**Outside GCP:** use [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) and point `GOOGLE_APPLICATION_CREDENTIALS` at the credential configuration file it produces. That variable also accepts a service account key file, but **Google recommends against long-lived keys** — prefer federation, and never ask a user to paste a key. + +## 4. Create the invoker service account + +The Service scales the pool as an **invoker** service account, which only reads and scales the pool. The identity instances *run as* is the separate **runner** service account set on the pool in `setup.md`. → `iam.md` for the full distinction. + +Two grants: + +- The GCP identity the Service runs as (step 3) gets **`roles/iam.serviceAccountTokenCreator`** on the invoker. +- The invoker gets a project-level Cloud Run role with at least **`run.workerPools.get`** and **`run.workerPools.update`**. `roles/run.developer` includes both. + +The same Terraform module works — pass the server's GCP identity as the impersonator instead of Temporal Cloud's accounts: + +```hcl +module "serverless-worker-cloud-run" { + source = "github.com/temporalio/terraform-modules//modules/serverless-workers/gcp/cloud-run" + + project_id = "" + invoker_account_id = "temporal-serverless-worker" + + runner_service_account_email = "" + + impersonator_service_account_emails = [ + "", + ] +} +``` + +Use the module's `invoker_email` output as `--gcp-cloud-run-service-account` when registering the Worker Deployment Version. + +**This is the one place self-hosted is simpler than Cloud:** there is no UI-provided template to copy, because you already know the impersonating identity — it is your own server's. + +## Then + +Follow `setup.md` from Step 1. The read/update permission trap in `iam.md` applies identically, and `diagnostics.md`'s `lastModifier` check is still the fastest way to confirm the server can actually scale the pool. diff --git a/references/gcp-cloud-run/setup.md b/references/gcp-cloud-run/setup.md new file mode 100644 index 0000000..298badd --- /dev/null +++ b/references/gcp-cloud-run/setup.md @@ -0,0 +1,281 @@ +# GCP Cloud Run — Setup (happy path) + + + +End-to-end: write a standard Worker, containerize it, push the image, create a Worker Pool at zero instances, grant Temporal permission to scale it, register a Worker Deployment Version, set it current, verify. For the two service accounts and the Terraform module, see `iam.md`. For what the execution model does and does not bound, see `constraints.md`. For new builds and rollback, see `versioning.md`. If it doesn't work, see `diagnostics.md`. + +## Prerequisites + + + +- **Cloud Run support is Pre-release and access-gated.** The user creates a support ticket or contacts their account team. Confirm this before anything else. +- A Temporal Cloud account with a **GCP-hosted Namespace**, or self-hosted Temporal Service v1.31.0+. The Namespace's cloud provider must match the compute provider — an AWS-hosted Namespace cannot drive Cloud Run. Regions need not match. +- For self-hosted, complete `self-hosted.md` first. +- Every Workflow must declare a versioning behavior, or the Worker must set a default. +- A GCP project with billing enabled and permission to enable service APIs and create Worker Pools, Artifact Registry repositories, Cloud Build jobs, service accounts, and Secret Manager secrets. +- `gcloud` CLI installed and authenticated. The Google Cloud console or Terraform also work. +- **Terraform** installed — Temporal ships the IAM setup as a Terraform module. +- A Temporal SDK. Supported on Cloud Run: Go, Python, TypeScript, Java, .NET, **Ruby, and Rust** — the last two are Cloud Run only and unavailable on Lambda. + +The `temporal` CLI commands in Steps 6 and 7 must inherit authentication from an existing profile or from the process environment. **Never append `--api-key ` or put the key in an inline assignment.** If `TEMPORAL_API_KEY` is not already populated, set it privately in the user's own terminal without putting the value in shell history: + +```bash +export TEMPORAL_ADDRESS="
:7233" +export TEMPORAL_NAMESPACE="" +printf 'Temporal API key: ' >&2 +IFS= read -r -s TEMPORAL_API_KEY +printf '\n' >&2 +export TEMPORAL_API_KEY +``` + +Do not run the secret-reading commands through an agent shell, ask the user to paste the key into conversation, or inspect the resulting variable. A configured Temporal CLI profile is equally valid and avoids a session environment variable. + +**Check the region before deploying.** Google reports high deployment latency creating or updating Cloud Run resources in some regions, including `us-central1`, and recommends another region while the issue is open. → `constraints.md`. + +## Prepare a clean GCP project + +After the resource list is approved, create only what is missing. First enable every API used by the commands below: + +```bash +gcloud services enable \ + run.googleapis.com \ + artifactregistry.googleapis.com \ + cloudbuild.googleapis.com \ + secretmanager.googleapis.com \ + iam.googleapis.com \ + iamcredentials.googleapis.com \ + cloudresourcemanager.googleapis.com \ + --project +``` + +Create a regional Docker repository for the Worker image: + +```bash +gcloud artifacts repositories create \ + --repository-format docker \ + --location \ + --project \ + --description "Temporal Serverless Worker images" +``` + +Create the runner service account that pool instances use: + +```bash +gcloud iam service-accounts create \ + --display-name "Temporal Cloud Run Worker runner" \ + --project +``` + +Create the Secret Manager secret before deploying the pool: + +```bash +gcloud secrets create \ + --replication-policy automatic \ + --project + +gcloud secrets versions add \ + --data-file=- \ + --project +``` + +Run the `versions add` command only in the user's own terminal, provide the Temporal API key on standard input, and then send EOF. Never ask for the value in conversation or run it through an agent shell where input or output may be captured. + +Grant only the runner access to that secret: + +```bash +gcloud secrets add-iam-policy-binding \ + --member="serviceAccount:@.iam.gserviceaccount.com" \ + --role roles/secretmanager.secretAccessor \ + --project +``` + +Before each create, use the corresponding `describe` command from `iam.md` to avoid colliding with shared resources. The invoker service account is created later by Temporal's Terraform module; do not substitute it for the runner. + +## Step 1: Write Worker code + +**There is no Cloud Run Worker package.** Write an ordinary long-lived Worker — same client, same `Worker`/`WorkerFactory`, same registration — and add Worker Versioning, which Serverless Workers require. Do not reach for anything in `../aws-lambda/sdk-.md`; those files are AWS Lambda only. The Cloud Run counterpart is `sdk-.md` in this directory. + +Two things the Worker must do: + +1. **Declare its Worker Deployment Version and enable versioning**, with a deployment name and build ID that exactly match the version you register in Step 6. +2. **Read its configuration from the environment** — address, Namespace, Task Queue, credentials — so one image can run against any Namespace. The pool supplies these via `--set-env-vars` and `--set-secrets`. + +Per-SDK code lives in `sdk-.md` in this directory, one file per SDK. Each covers the versioned Worker, connection, image packaging, graceful shutdown, scale-in safety, and observability. The Java, Python, and .NET references also include their logging setup and diagnostic signatures. + +**The entrypoint must start the Worker process**, so an instance begins polling as soon as it starts. + +## Step 2: Containerize the Worker + + + +Per-runtime notes that matter, from the deployment guide: + +| SDK | Notes | +|---|---| +| Go | Multi-stage; `CGO_ENABLED=0` for a static binary, which is what a `distroless/static` base expects. | +| Python | `pip install "temporalio>=1.30.0,<2"`; entrypoint runs the Worker module. | +| TypeScript | **Keep `ca-certificates` installed** — without it the Worker fails at startup with `TransportError: tonic::transport::Error(Transport, NativeCertsNotFound)`. Use a **glibc** image, not Alpine. Set `NODE_OPTIONS=--max-old-space-size=` to ~80% of the instance memory limit. | +| Java | Fat jar on a JRE image; set `-XX:MaxRAMPercentage=75` — the JVM reads the container limit but defaults max heap to 25% of it. | +| .NET | `dotnet publish` in a build stage, run on the .NET runtime image. | + +**The `NativeCertsNotFound` error is the same root cause as the .NET one on Lambda** — a Rust-core SDK that cannot find system root CAs — reached here through a slim base image rather than an overridden `SSL_CERT_FILE`. Any Rust-core SDK (TypeScript, Python, .NET, Ruby) in a minimal image needs CA certificates present. + +## Step 3: Build and push the image + +```bash +gcloud builds submit \ + --tag -docker.pkg.dev///my-temporal-worker:build-1 \ + --project +``` + +Tag the image with the build ID. It keeps image, pool, and Worker Deployment Version aligned, which matters because the compute configuration cannot pin a revision (→ `constraints.md`). + +The happy path deliberately uses Cloud Build's global endpoint. Supplying `--region` can require additional regional build and staging-bucket setup, depending on the project's Cloud Build bucket policy. If regional builds are required for a private pool or data-residency policy, pre-create or select the regional source/log buckets, grant the build identity access, and then add `--region `. + +## Step 4: Create the Worker Pool + +**Create one pool per Worker Deployment Version, initially at zero instances.** Registering the version starts the WCI, which temporarily raises the pool to the scaler's planned count—at least one instance—to register its Task Queues. The version does not need to be current for this bootstrap. + +```bash +gcloud run worker-pools deploy my-temporal-worker-pool-build-1 \ + --image -docker.pkg.dev///my-temporal-worker:build-1 \ + --region \ + --project \ + --service-account \ + --instances 0 \ + --set-env-vars TEMPORAL_ADDRESS=
:7233,TEMPORAL_NAMESPACE=,TEMPORAL_TASK_QUEUE=my-task-queue \ + --set-secrets TEMPORAL_API_KEY=:latest +``` + +| Parameter | Description | +|---|---| +| `--image` | The image pushed in Step 3. | +| `--service-account` | The **runner** service account instances run as. **Not** the invoker Temporal impersonates. → `iam.md`. | +| `--instances` | Set to `0`; the WCI takes ownership of the count when the Worker Deployment Version is registered. | +| `--set-env-vars` | Non-secret configuration. | +| `--set-secrets` | Maps a Secret Manager secret to an env var — use it for `TEMPORAL_API_KEY` or TLS material. | + +**Secrets are the documented default here, not an upgrade.** Unlike Lambda's guide, the Cloud Run path puts the API key in Secret Manager from the start, so there is no "acceptable for development only" plaintext step to warn about. + +## Step 5: Grant Temporal permission to scale the pool + +Cloud Run has no invocation grant. Temporal **impersonates an invoker service account** and drives the Cloud Run admin API. Create it with Temporal's Terraform module — the Cloud UI supplies a filled-in template under **Workers → Create Worker Deployment → Access**. → `iam.md` for the module, its variables, and the two-service-account distinction. + +Terraform's `invoker_email` output is what Step 6 needs. + +## Step 6: Register the Worker Deployment Version + +```bash +temporal worker deployment create --namespace --name my-app + +temporal worker deployment create-version \ + --namespace \ + --deployment-name my-app \ + --build-id build-1 \ + --gcp-cloud-run-project \ + --gcp-cloud-run-region \ + --gcp-cloud-run-worker-pool my-temporal-worker-pool-build-1 \ + --gcp-cloud-run-service-account \ + --gcp-cloud-run-min-instances 0 \ + --gcp-cloud-run-max-instances 30 \ + --gcp-cloud-run-initial-instances 0 \ + --gcp-cloud-run-utilization-target 0.8 +``` + +| Flag | Description | +|---|---| +| `--deployment-name` / `--build-id` | Must match the Worker code exactly. | +| `--gcp-cloud-run-project` | Project containing the pool. | +| `--gcp-cloud-run-region` | Pool region. | +| `--gcp-cloud-run-worker-pool` | Pool name from Step 4. | +| `--gcp-cloud-run-service-account` | The **invoker** — Terraform's `invoker_email`. | +| `--gcp-cloud-run-min-instances` | Floor the scaler maintains; `0` allows scale-to-zero. | +| `--gcp-cloud-run-max-instances` | Ceiling the scaler may request; defaults to `30`. | +| `--gcp-cloud-run-initial-instances` | Initial planned count; must be between min and max. | +| `--gcp-cloud-run-utilization-target` | Target average utilization in `(0, 1]`; defaults to `0.8`. | + +The four scaler flags are a coupled group: **either omit all four and accept the defaults (`0`, `30`, `0`, `0.8`), or provide all four together.** Supplying only one—even only a higher maximum—fails CLI validation. + +Through the UI, the version is set current automatically; through the CLI it is a separate step. + +### Checkpoint: distinguish registration from Validate Connection + +Creating the Worker Deployment Version starts its WCI. The WCI validates the configuration by reading the Worker Pool, then asks the rate-based algorithm for its Task Queue registration action. For a new Cloud Run version, that action updates the pool's manual instance count to at least one. This special registration floor applies even when `--gcp-cloud-run-min-instances` and `--gcp-cloud-run-initial-instances` are both `0`; those values govern the scaler's normal operating range and initial plan, not whether it can bootstrap Task Queue registration. This exercises both `run.workerPools.get` and the `run.workerPools.update` path used by real scaling. + +The update completing proves that Temporal reached Cloud Run, not that the container started or the Worker connected. Wait for the expected Task Queue types to appear; that binding is proof the instance started and polled under the registered deployment name and build ID. + +The separate UI **Validate Connection** action (Workers → Deployments → select → Actions) only impersonates the invoker and reads the pool. It starts no instance and does not exercise `run.workerPools.update`, so a green manual validation is weaker than a successful registration bootstrap. + +Use both views when diagnosing the checkpoint: + +```bash +# has any Worker ever polled under this version? +temporal worker deployment describe-version \ + --namespace --deployment-name my-app --build-id build-1 --report-task-queue-stats + +# has Temporal ever actually written to the pool? +gcloud run worker-pools describe my-temporal-worker-pool-build-1 \ + --region --project --format=yaml +``` + +In the pool's `metadata.annotations`, `serving.knative.dev/lastModifier` becoming the invoker service account is the real proof that scaling works. → `diagnostics.md`. + +## Step 7: Set the version current + +```bash +temporal worker deployment set-current-version \ + --namespace --deployment-name my-app --build-id build-1 --yes +``` + +Without this, new traffic does not route to the version. The registration instance may already have started and bound the Task Queue, but that bootstrap does not make the version current. The command prompts for confirmation; **run non-interactively without `--yes` it exits having changed nothing**, which reads as success. Read the state back with `temporal worker deployment describe`. + +## Step 8: Verify + +```bash +temporal workflow start \ + --namespace --task-queue my-task-queue \ + --type MyWorkflow --input '"Hello, serverless!"' +``` + +Tasks arriving with no active pollers cause the WCI to raise the instance count; Cloud Run starts an instance, the Worker connects and processes the Task. + +Confirm from two independent signals: + +- **Temporal** — Task completions in the Workflow's event history. +- **Cloud Run** — pool logs showing Worker startup and Task processing: + ```bash + gcloud run worker-pools logs read my-temporal-worker-pool-build-1 \ + --region --project + ``` + **The pool produces no logs while scaled to zero**, so read them while an instance is up. An empty log is not evidence of failure. + +## Teardown + +Record what you create as you go: pool name, image tag and Artifact Registry repository, runner and invoker service accounts, the Terraform state, secrets, deployment name and build ID, project and region. + +**The ordering problem is milder than Lambda's**, where the function had to go before the version or the delete deadlocked on active pollers. Here, scaling the pool to zero stops the pollers without destroying anything. + +1. Unset the current version — a Current version cannot be deleted: + ```bash + temporal worker deployment set-current-version \ + --namespace --deployment-name my-app --unversioned --yes + ``` +2. Scale the pool to zero, which ends polling: + ```bash + gcloud run worker-pools update --instances 0 --region --project + ``` +3. Wait for drainage, then delete the version, then the deployment: + ```bash + temporal worker deployment describe-version --namespace --deployment-name my-app --build-id build-1 + temporal worker deployment delete-version --namespace --deployment-name my-app --build-id build-1 + temporal worker deployment delete --namespace --name my-app + ``` +4. Delete the Worker Pool: + ```bash + gcloud run worker-pools delete --region --project + ``` +5. `terraform destroy` the IAM module — **only if this deployment created it.** One invoker service account can serve several pools, so a shared one may still be in use. → `iam.md`. +6. Delete the container image from Artifact Registry, and any Secret Manager secrets created for this deployment. Ask before revoking a Temporal Cloud API key: it is account-scoped, not deployment-scoped. diff --git a/references/gcp-cloud-run/versioning.md b/references/gcp-cloud-run/versioning.md new file mode 100644 index 0000000..eb7686e --- /dev/null +++ b/references/gcp-cloud-run/versioning.md @@ -0,0 +1,73 @@ +# GCP Cloud Run — versioning, updates, and rollback + + + +## One Worker Pool per Build ID + +**The compute configuration names a project, region, and Worker Pool — it does not name a [revision](https://cloud.google.com/run/docs/managing/revisions).** Temporal runs whichever revision the pool happens to serve. That ties a pool to a single build, so **a new build needs a new pool.** Carry the Build ID in the pool name (`my-worker-pool-build-1`) so the mapping stays visible. + +This is the structural difference from Lambda, where one function holds many published versions and the ARN pins which one Temporal invokes. Temporal's Cloud Run compute configuration has no revision selector, so **durable isolation has to come from creating separate pools.** A pool-level instance split can temporarily hold a particular revision, but that split remains mutable state outside Temporal. + +## The hazard: redeploying into a live pool + +> Deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run promotes it to every instance by default. The version does not change, but the code behind it does. Deploying replay-unsafe code this way causes non-determinism errors for in-flight Workflows, **including Pinned ones**. + +Same failure as pointing a Lambda Worker Deployment Version at a mutable `$LATEST`, reached through a different mechanism. Worth stating precisely because the intuition differs: on Lambda you have to *choose* the mutable path by registering an unqualified ARN. **On Cloud Run the mutable path is what a normal `gcloud run worker-pools deploy` does** — the durable Temporal-aligned path is a new pool, while `--no-promote` is an explicit same-pool guardrail. + +`Pinned` does not protect you. Pinning routes Workflows to a *version*; it cannot pin the code behind a pool whose revision moved underneath it. + +## Guardrail and recovery when a pool is reused + +Pool-per-build remains the production rule because Temporal identifies a Worker Pool, not one of its revisions. If an exceptional workflow must deploy a new revision into a pool that still serves a live version, `--no-promote` prevents the new revision from receiving the pool's instances automatically: + +```bash +gcloud run worker-pools deploy \ + --image -docker.pkg.dev///: \ + --region \ + --project \ + --no-promote +``` + +This is a guardrail, not immutable versioning: the pool's revision split remains mutable state outside Temporal. If a new revision was already promoted accidentally, send all instances back to the known-good revision explicitly: + +```bash +gcloud run worker-pools update-instance-split \ + --to-revisions==100 \ + --region \ + --project +``` + +`--to-latest` is **not** that rollback: it assigns instances to the current and future `LATEST` revision. Use it only when deliberately removing the sticky `--no-promote` behavior and restoring automatic promotion of future revisions: + +```bash +gcloud run worker-pools update-instance-split \ + --to-latest \ + --region \ + --project +``` + +After emergency recovery, return to one pool per Build ID for the next release so Temporal version routing and deployed code cannot drift independently. + +## Rolling out a new build + +1. Build and push a new image tagged with the new build ID. +2. **Create a new Worker Pool** at zero instances for that build ID, with the same runner service account. +3. Register a new Worker Deployment Version pointing at the new pool, with a build ID matching the new Worker code. +4. Confirm registration bootstrapped the pool: its `lastModifier` shows the invoker and the expected Task Queue types are bound. Treat the separate UI Validate Connection action as a read-only check. → `diagnostics.md`. +5. Set the new version current, or ramp to it. +6. **Leave the old pool in place** while Pinned Workflows still run on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. + +Only the invoker's permissions are shared across pools, so a new pool usually needs no IAM change — provided the invoker's `deploy_roles` are project-level, which is the module's default. Check `iam.md` if you scoped them to individual pools instead. + +## Rollback + +Set the previous version current again. Its pool is still there (step 6 above), and its WCI scales it back up on the next Task. Nothing needs rebuilding, and no image or revision has to be reverted — which is the payoff for keeping one pool per build. + +Rolling back through Temporal is therefore straightforward if you followed the pool-per-build discipline. If you redeployed into the same pool, setting the previous Temporal version current is not enough because both versions still address mutable pool state. Restore the known-good Cloud Run revision split as described above, provided that revision still exists; if it was deleted, there is nothing left to route back to and the old image must be redeployed deliberately. + +## Cost of the discipline + +A pool per build means pools accumulate. They cost nothing while at zero instances, but they are real resources with real names, and stale ones make the project harder to reason about. Delete a pool once its version is deleted and no Pinned Workflow can route to it — that ordering is in `setup.md`'s teardown section. From 674163be359ef9547bc946e8f3dc206b70874c4a Mon Sep 17 00:00:00 2001 From: Harish Narayanappa Date: Fri, 4 Sep 2026 11:13:01 -0700 Subject: [PATCH 2/4] Remove redundant Lambda comparisons from Cloud Run references --- references/gcp-cloud-run/constraints.md | 24 ++++++++++----------- references/gcp-cloud-run/diagnostics.md | 10 ++++----- references/gcp-cloud-run/iam.md | 26 +++++++++++------------ references/gcp-cloud-run/observability.md | 12 ++--------- references/gcp-cloud-run/setup.md | 12 +++++------ references/gcp-cloud-run/versioning.md | 4 ++-- 6 files changed, 37 insertions(+), 51 deletions(-) diff --git a/references/gcp-cloud-run/constraints.md b/references/gcp-cloud-run/constraints.md index 178f503..8d8635d 100644 --- a/references/gcp-cloud-run/constraints.md +++ b/references/gcp-cloud-run/constraints.md @@ -7,25 +7,23 @@ Consequences of Cloud Run's execution model: **Temporal resizes a pool of long-lived instances, scaling it to zero when there is no work.** Temporal does *not* invoke your Worker per Task. -This file is the provider "diff surface." Compare it with `../aws-lambda/constraints.md` to see what changes between providers — the workflow in `SKILL.md` is the same; these constraints are not. +For a deliberate cross-provider comparison, see `../aws-lambda/constraints.md`. ## Worker lifetime is an instance, not an invocation -Each pool instance runs **standard long-lived Worker code**: it connects, registers Workflows and Activities, and polls the Task Queue for its whole lifetime. There is no handler, no per-Task lifecycle, and **no serverless Worker package** — the SDK-specific packages in `../aws-lambda/sdk-.md` are AWS Lambda only. Some SDKs add optional Cloud Run conveniences; none are required. +Each pool instance runs **standard long-lived Worker code**: it connects, registers Workflows and Activities, and polls the Task Queue for its whole lifetime. There is no handler, no per-Task lifecycle, and **no serverless Worker package**. Use the Cloud Run SDK reference in this directory; some SDKs add optional conveniences, but none are required. The WCI controls how many instances run; each instance manages its own polling and Task processing. -## None of Lambda's timing constraints apply - -Explicitly, so they are not carried across: +## Timing and Activity limits - **No invocation deadline.** Nothing bounds how long a Worker lives except scale-in. -- **No shutdown deadline buffer.** That is a property of the Lambda Worker packages, not of Temporal. +- **No shutdown deadline buffer.** Cloud Run terminates instances through its own scale-in lifecycle. - **No timeout triple to tune.** Two of its three values do not exist here. -- **Activities are not bounded by an invocation limit.** An Activity too long for Lambda's 15-minute ceiling is a reason to choose Cloud Run. -- **Eager Activities are not disabled by the platform.** Lambda's packages force this off because a per-invocation Worker holds no persistent connection; a pool instance holds its connection for its lifetime, so that reasoning does not transfer. Confirm against your SDK rather than assuming either way. +- **Activities are not bounded by an invocation limit.** Their practical interruption boundary is scale-in. +- **Eager Activities are not disabled by the platform.** A pool instance holds its connection for its lifetime; confirm support against the selected SDK. -Cloud Run still sends `SIGTERM` during scale-in and can send `SIGKILL` ten seconds later. Handle `SIGTERM` and configure the SDK's graceful-shutdown timeout below that window; this is separate from Lambda's deadline buffer and timeout triple. → `sdk-.md`. +Cloud Run sends `SIGTERM` during scale-in and can send `SIGKILL` ten seconds later. Handle `SIGTERM` and configure the SDK's graceful-shutdown timeout below that window. → `sdk-.md`. ## What bounds an Activity instead: scale-in @@ -58,11 +56,11 @@ An initial count and minimum of zero do not suppress registration. The rate-base Keep an older version's pool in place while Pinned Workflows are still running on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. -**The mutable-build hazard, in its Cloud Run form:** deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run promotes it to every instance by default. The version does not change, but the code behind it does — causing non-determinism errors for in-flight Workflows, **including Pinned ones**. This is the same failure as pointing Lambda at a mutable `$LATEST`, reached through revisions instead of ARNs. → `versioning.md`. +**The mutable-build hazard:** deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run promotes it to every instance by default. The version does not change, but the code behind it does — causing non-determinism errors for in-flight Workflows, **including Pinned ones**. → `versioning.md`. ## Do not share a Task Queue with long-lived Workers -Stronger than the Lambda guidance, and for a different reason. On Lambda the two scaling loops fight; here **the pool scales up to cover the Task Queue's full workload even when long-lived Workers are already handling all of it**, so you run and pay for duplicate capacity. +**The pool scales up to cover the Task Queue's full workload even when independently managed Workers are already handling all of it**, so you run and pay for duplicate capacity. The WCI sizes the pool from the rate of Tasks arriving on the version's Task Queues, and nothing in that measurement accounts for the long-lived Workers. Sync matching to a long-lived Worker suppresses the *immediate* scale-up, but the periodic re-sizing scales the pool up regardless. Fixing poller counts on the long-lived side does not help — use separate Task Queues. @@ -76,6 +74,6 @@ Google currently reports **high deployment latency creating or updating Cloud Ru ## What does *not* follow from this model -- **"Serverless Worker" does not imply standard Worker code.** On Lambda you write a handler against a provider-specific package; here you write an ordinary Worker. Advice about handlers, configure callbacks, or tuned package defaults does not transfer. +- **Use ordinary Worker code.** Handler, configure-callback, and provider-package patterns do not apply. - **Scale-to-zero is not per-Task.** An idle pool costs nothing, but a running instance is billed for its lifetime, not per unit of work. -- **A passing Validate Connection is weaker here than on Lambda.** It exercises only the read permission and starts no instance. → `diagnostics.md`. +- **A passing Validate Connection exercises only the read permission and starts no instance.** → `diagnostics.md`. diff --git a/references/gcp-cloud-run/diagnostics.md b/references/gcp-cloud-run/diagnostics.md index 573c96d..71dc9f7 100644 --- a/references/gcp-cloud-run/diagnostics.md +++ b/references/gcp-cloud-run/diagnostics.md @@ -19,7 +19,7 @@ ## Start here: read the pool's annotations -This is the single highest-value command, and it has no Lambda equivalent: +This is the single highest-value command: ```bash gcloud run worker-pools describe \ @@ -39,8 +39,6 @@ Three fields under `metadata.annotations`: -Three identities, as on AWS Lambda — the **operator** (whose credentials run the commands), the **runner service account** (what the pool runs as), and the **invoker service account** (what Temporal impersonates). The mapping to Lambda's roles is close enough to be useful and different enough to be dangerous if assumed. +Cloud Run uses three identities: -| Concept | AWS Lambda | GCP Cloud Run | -|---|---|---| -| The compute's own identity | Execution role, trusted by `lambda.amazonaws.com` | **Runner service account**, set with `--service-account` | -| Temporal's identity | Invocation role, assumed via `sts:AssumeRole` + External ID | **Invoker service account**, reached by **impersonation** | -| Mechanism | `sts:AssumeRole` with a confused-deputy guard | `roles/iam.serviceAccountTokenCreator` — **no External ID equivalent** | -| What Temporal does with it | Invokes the function | Reads and **updates the pool's instance count** via the Cloud Run admin API | -| Infrastructure as code | CloudFormation template (shipped in `assets/`) | **Terraform module**, `serverless-workers/gcp/cloud-run` | +| Identity | Purpose | +|---|---| +| **Operator** | The credentials that run `gcloud` and Terraform commands. | +| **Runner service account** | The identity attached to pool instances with `--service-account`. | +| **Invoker service account** | The identity Temporal impersonates to read and update the pool's instance count. | + +Temporal reaches the invoker through `roles/iam.serviceAccountTokenCreator`. The Terraform module described below creates the invoker and applies its grants. **The two service accounts are not interchangeable**, and confusing them is the single most likely IAM mistake here. The runner runs the pool and never scales it; the invoker scales the pool and never runs it. @@ -28,8 +28,6 @@ The runtime identity the pool's instances use to reach other Google Cloud servic - `roles/logging.logWriter` **only if** the Worker writes through the Cloud Logging API rather than stdout/stderr. - Whatever else your Workflows and Activities call. -This differs from Lambda, where `AWSLambdaBasicExecutionRole` is effectively mandatory because the execution role is what creates the log group. Here, logging works with no grant at all. - ## Invoker service account The identity Temporal Cloud impersonates to read and scale the pool. Two grants make it work: @@ -43,7 +41,7 @@ The invoker also needs **`roles/iam.serviceAccountUser` on the runner service ac `run.workerPools.get` alone is enough for the UI's **Validate Connection** action to pass. That manual action never exercises `run.workerPools.update`. Version registration does: the WCI reads the pool and then updates its manual instance count to bootstrap Task Queue registration. An invoker that can read but not update therefore passes manual validation but fails version registration or a later resize. -There is no Lambda analogue: there, the registration invocation uses the same invoke permission as real traffic. For Cloud Run, verify the registration bootstrap and `lastModifier` rather than trusting the separate green Validate Connection result. → `diagnostics.md`. +Verify the registration bootstrap and `lastModifier` rather than trusting the separate green Validate Connection result. → `diagnostics.md`. ## The Terraform module @@ -84,7 +82,7 @@ Use the **`invoker_email`** output as `--gcp-cloud-run-service-account` when reg ### Treat the module as shared, pre-existing infrastructure -Same discipline as Lambda's CloudFormation stack, different tool. One invoker can serve several pools, so before creating a second one, look for an existing account and consider reusing it. **Do not `terraform destroy` state you did not create**, and note that the module's default `invoker_account_id` collides the same way Lambda's default `RoleName` does — an earlier deployment in the project may already own it. +One invoker can serve several pools, so before creating a second one, look for an existing account and consider reusing it. **Do not `terraform destroy` state you did not create.** The module's default `invoker_account_id` may already be owned by an earlier deployment in the project. ## Operator GCP permissions @@ -99,7 +97,7 @@ The identity running the `gcloud`/Terraform commands needs, at minimum: | Apply the Terraform module (Step 5) | Service-account creation plus IAM policy binding on the project and on the runner | | Read pool state and logs (verify, diagnose) | `run.workerPools.get`, Cloud Logging read | -`iam.serviceAccounts.actAs` on the runner is the Cloud Run counterpart of Lambda's `iam:PassRole` on the execution role, and fails the same way — a pool create that is denied despite having Cloud Run permissions. +The operator needs `iam.serviceAccounts.actAs` on the runner to attach it to the pool. Without it, pool creation is denied even when the operator otherwise has Cloud Run permissions. ### Preflight @@ -123,4 +121,4 @@ For a same-project build using Cloud Build's default service account, Artifact R **Classify an authentication failure before acting on it.** An absent or expired credential (`gcloud auth login`, or `gcloud auth application-default login` for Terraform) is recoverable in a minute; an identity that resolves but is denied a specific action is a real permissions problem. Never collect credentials in conversation and never ask the user to paste a service account key — Google recommends against long-lived keys outright. -**Confirm the project explicitly before creating anything.** `gcloud config get-value project` is ambient state that is easy to be wrong about, exactly like an AWS profile pointing at an unintended account. Name the project in the approval list and verify it, rather than trusting the default. +**Confirm the project explicitly before creating anything.** `gcloud config get-value project` is ambient state that is easy to be wrong about. Name the project in the approval list and verify it rather than trusting the default. diff --git a/references/gcp-cloud-run/observability.md b/references/gcp-cloud-run/observability.md index 489a0f0..cb0bb8e 100644 --- a/references/gcp-cloud-run/observability.md +++ b/references/gcp-cloud-run/observability.md @@ -9,15 +9,7 @@ **A Cloud Run Serverless Worker emits the same traces and metrics as a Worker anywhere else.** It is an ordinary long-lived Worker, so the SDK's normal metrics and OpenTelemetry tracing setup applies unchanged, and each SDK's general observability guide is the right reference. -This is a genuine simplification over AWS Lambda, and the contrast is worth keeping in mind: - -| | AWS Lambda | GCP Cloud Run | -|---|---|---| -| OTel wiring | Per-SDK helper in the serverless package (`ApplyDefaults`, `apply_defaults`, `OtelLambdaWorkerConfigurationHelper`, a separate `…Aws.Lambda.OpenTelemetry` package for .NET) | **None — use the SDK's standard setup** | -| Collector | ADOT Lambda layer, plus a custom `otel-collector-config.yaml` because the default does not route OTLP to the traces pipeline | No layer; export as you would from any container | -| Env var | `OPENTELEMETRY_COLLECTOR_CONFIG_URI` / `_FILE`, differing by SDK | n/a | -| Flush timing | Must flush before the invocation deadline, or telemetry is lost | No deadline to beat | -| IAM | Execution role needs X-Ray and CloudWatch permissions | Runner needs nothing for stdout/stderr logging | +Do not add provider-specific helper layers, collector environment variables, or invocation-deadline flush logic. Export telemetry as you would from any long-lived container. Some SDKs add optional Cloud Run conveniences, such as OpenTelemetry helpers. **They are optional**, and where they exist they are documented in that SDK's Cloud Run guide. @@ -31,7 +23,7 @@ Read a pool's logs: gcloud run worker-pools logs read --region --project ``` -**The pool produces no logs while scaled to zero.** Read them while an instance is up; an empty log is not evidence of a problem. This is the most common source of confusion when checking a Cloud Run Worker's health, and it has no Lambda equivalent — a Lambda log group retains history after the invocation ends. +**The pool produces no logs while scaled to zero.** Read them while an instance is up; an empty log is not evidence of a problem. ## Memory: give the runtime the instance, not the host diff --git a/references/gcp-cloud-run/setup.md b/references/gcp-cloud-run/setup.md index 298badd..dee3d23 100644 --- a/references/gcp-cloud-run/setup.md +++ b/references/gcp-cloud-run/setup.md @@ -12,13 +12,13 @@ End-to-end: write a standard Worker, containerize it, push the image, create a W - **Cloud Run support is Pre-release and access-gated.** The user creates a support ticket or contacts their account team. Confirm this before anything else. -- A Temporal Cloud account with a **GCP-hosted Namespace**, or self-hosted Temporal Service v1.31.0+. The Namespace's cloud provider must match the compute provider — an AWS-hosted Namespace cannot drive Cloud Run. Regions need not match. +- A Temporal Cloud account with a **GCP-hosted Namespace**, or self-hosted Temporal Service v1.31.0+. The Namespace must be hosted on GCP; its region need not match the pool's. - For self-hosted, complete `self-hosted.md` first. - Every Workflow must declare a versioning behavior, or the Worker must set a default. - A GCP project with billing enabled and permission to enable service APIs and create Worker Pools, Artifact Registry repositories, Cloud Build jobs, service accounts, and Secret Manager secrets. - `gcloud` CLI installed and authenticated. The Google Cloud console or Terraform also work. - **Terraform** installed — Temporal ships the IAM setup as a Terraform module. -- A Temporal SDK. Supported on Cloud Run: Go, Python, TypeScript, Java, .NET, **Ruby, and Rust** — the last two are Cloud Run only and unavailable on Lambda. +- A Temporal SDK. Supported on Cloud Run: Go, Python, TypeScript, Java, .NET, **Ruby, and Rust**. The `temporal` CLI commands in Steps 6 and 7 must inherit authentication from an existing profile or from the process environment. **Never append `--api-key ` or put the key in an inline assignment.** If `TEMPORAL_API_KEY` is not already populated, set it privately in the user's own terminal without putting the value in shell history: @@ -96,7 +96,7 @@ Before each create, use the corresponding `describe` command from `iam.md` to av ## Step 1: Write Worker code -**There is no Cloud Run Worker package.** Write an ordinary long-lived Worker — same client, same `Worker`/`WorkerFactory`, same registration — and add Worker Versioning, which Serverless Workers require. Do not reach for anything in `../aws-lambda/sdk-.md`; those files are AWS Lambda only. The Cloud Run counterpart is `sdk-.md` in this directory. +**There is no Cloud Run Worker package.** Write an ordinary long-lived Worker — same client, same `Worker`/`WorkerFactory`, same registration — and add Worker Versioning, which Serverless Workers require. Use `sdk-.md` in this directory. Two things the Worker must do: @@ -121,7 +121,7 @@ Per-runtime notes that matter, from the deployment guide: | Java | Fat jar on a JRE image; set `-XX:MaxRAMPercentage=75` — the JVM reads the container limit but defaults max heap to 25% of it. | | .NET | `dotnet publish` in a build stage, run on the .NET runtime image. | -**The `NativeCertsNotFound` error is the same root cause as the .NET one on Lambda** — a Rust-core SDK that cannot find system root CAs — reached here through a slim base image rather than an overridden `SSL_CERT_FILE`. Any Rust-core SDK (TypeScript, Python, .NET, Ruby) in a minimal image needs CA certificates present. +**`NativeCertsNotFound` means a Rust-core SDK cannot find system root CAs.** Any Rust-core SDK (TypeScript, Python, .NET, Ruby) in a minimal image needs CA certificates present. ## Step 3: Build and push the image @@ -158,7 +158,7 @@ gcloud run worker-pools deploy my-temporal-worker-pool-build-1 \ | `--set-env-vars` | Non-secret configuration. | | `--set-secrets` | Maps a Secret Manager secret to an env var — use it for `TEMPORAL_API_KEY` or TLS material. | -**Secrets are the documented default here, not an upgrade.** Unlike Lambda's guide, the Cloud Run path puts the API key in Secret Manager from the start, so there is no "acceptable for development only" plaintext step to warn about. +**Put the API key in Secret Manager from the start.** Do not introduce a plaintext environment-variable deployment step. ## Step 5: Grant Temporal permission to scale the pool @@ -256,7 +256,7 @@ Confirm from two independent signals: Record what you create as you go: pool name, image tag and Artifact Registry repository, runner and invoker service accounts, the Terraform state, secrets, deployment name and build ID, project and region. -**The ordering problem is milder than Lambda's**, where the function had to go before the version or the delete deadlocked on active pollers. Here, scaling the pool to zero stops the pollers without destroying anything. +Scale the pool to zero before deleting the version so its pollers stop without destroying the pool prematurely. 1. Unset the current version — a Current version cannot be deleted: ```bash diff --git a/references/gcp-cloud-run/versioning.md b/references/gcp-cloud-run/versioning.md index eb7686e..bd16258 100644 --- a/references/gcp-cloud-run/versioning.md +++ b/references/gcp-cloud-run/versioning.md @@ -9,13 +9,13 @@ **The compute configuration names a project, region, and Worker Pool — it does not name a [revision](https://cloud.google.com/run/docs/managing/revisions).** Temporal runs whichever revision the pool happens to serve. That ties a pool to a single build, so **a new build needs a new pool.** Carry the Build ID in the pool name (`my-worker-pool-build-1`) so the mapping stays visible. -This is the structural difference from Lambda, where one function holds many published versions and the ARN pins which one Temporal invokes. Temporal's Cloud Run compute configuration has no revision selector, so **durable isolation has to come from creating separate pools.** A pool-level instance split can temporarily hold a particular revision, but that split remains mutable state outside Temporal. +Temporal's Cloud Run compute configuration has no revision selector, so **durable isolation requires a separate pool per build.** A pool-level instance split can temporarily hold a particular revision, but that split remains mutable state outside Temporal. ## The hazard: redeploying into a live pool > Deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run promotes it to every instance by default. The version does not change, but the code behind it does. Deploying replay-unsafe code this way causes non-determinism errors for in-flight Workflows, **including Pinned ones**. -Same failure as pointing a Lambda Worker Deployment Version at a mutable `$LATEST`, reached through a different mechanism. Worth stating precisely because the intuition differs: on Lambda you have to *choose* the mutable path by registering an unqualified ARN. **On Cloud Run the mutable path is what a normal `gcloud run worker-pools deploy` does** — the durable Temporal-aligned path is a new pool, while `--no-promote` is an explicit same-pool guardrail. +**A normal `gcloud run worker-pools deploy` takes the mutable path by default.** The durable Temporal-aligned path is a new pool; `--no-promote` is only a same-pool guardrail. `Pinned` does not protect you. Pinning routes Workflows to a *version*; it cannot pin the code behind a pool whose revision moved underneath it. From de8504806d64e8dd1f5ab93d936c27ada2b55fa1 Mon Sep 17 00:00:00 2001 From: Harish Narayanappa Date: Fri, 4 Sep 2026 11:27:15 -0700 Subject: [PATCH 3/4] Trim Cloud Run provider configuration references --- references/aws-lambda/constraints.md | 57 ----------------------- references/aws-lambda/setup.md | 48 ++++++++++++------- references/aws-lambda/versioning.md | 12 ----- references/gcp-cloud-run/constraints.md | 29 +++--------- references/gcp-cloud-run/diagnostics.md | 17 ++----- references/gcp-cloud-run/iam.md | 18 +++---- references/gcp-cloud-run/observability.md | 22 ++------- references/gcp-cloud-run/self-hosted.md | 29 +----------- references/gcp-cloud-run/setup.md | 29 ++---------- references/gcp-cloud-run/versioning.md | 11 ++--- 10 files changed, 61 insertions(+), 211 deletions(-) delete mode 100644 references/aws-lambda/constraints.md diff --git a/references/aws-lambda/constraints.md b/references/aws-lambda/constraints.md deleted file mode 100644 index 27dff8a..0000000 --- a/references/aws-lambda/constraints.md +++ /dev/null @@ -1,57 +0,0 @@ -# AWS Lambda — execution-model constraints - -Consequences of Lambda's execution model: **Temporal invokes a function per unit of work, and the Worker exits when the invocation ends.** Everything here follows from that, and none of it generalizes to a provider that keeps long-lived instances alive. - -This file is the provider "diff surface." To understand what changes when moving between compute providers, compare this file with the equivalent one for the other provider — the workflow in `SKILL.md` stays the same; these constraints do not. - -## Worker lifetime is one invocation - -A Worker starts, connects, polls until its shutdown deadline buffer, drains, and exits. The invocation is the Worker's *lifetime*, not one unit of work: a single invocation can serve many Workflow and Activity Tasks from however many executions arrive in its window. Billing follows the invocation, not the work done inside it. - -## Set the invocation deadline high enough - -Providers often default to a very short timeout — Lambda's default is 3 seconds. If the first invocation times out before the Worker registers the Task Queue, the binding is never created and the Worker is never invoked again. → `setup.md` for the exact defaults, the per-SDK examples, and the GB-seconds trade-off behind choosing a value. - -## Tune the timeout triple together for long-running Activities - -The worker stop timeout controls how long the Worker waits for in-flight Tasks after it stops polling; the shutdown deadline buffer controls how long before the invocation deadline it stops polling. - -1. worker stop timeout > longest Activity runtime -2. shutdown deadline buffer > worker stop timeout + shutdown hook time -3. invocation deadline > longest Activity runtime + shutdown deadline buffer - -Raising one alone does not help. Raising only the shutdown deadline buffer makes the Worker stop polling earlier but gives in-flight Activities no more time; raising only the worker stop timeout doesn't make it stop polling earlier, so the provider may terminate the Worker first. - -Worked example: a longest Activity runtime of 5 minutes with 3 seconds of shutdown hooks means a worker stop timeout above 5 minutes, a shutdown deadline buffer above 303 seconds, and an invocation deadline of at least 10 minutes 3 seconds. - -*Symptom of getting this wrong:* Activities abandoned mid-execution and retried on a later invocation. - -If the longest Activity exceeds half the maximum invocation deadline, recommend Activity Heartbeats. → `../concepts.md`, `sdk-.md`. - -## Activities are bounded by the invocation limit - -An Activity must finish within the invocation deadline minus the shutdown deadline buffer. Workflow duration is unbounded and can span many invocations. Flag Activities that approach the provider's limit early — Lambda's ceiling is 15 minutes. - -**An Activity that cannot fit needs a different hosting strategy**, not a larger timeout: a long-lived Worker on a separate Task Queue, or a compute provider without a per-invocation ceiling. → `../concepts.md`. - -## Eager Activities are always disabled - -Every SDK's Lambda Worker package sets this and it cannot be overridden, because eager Activity execution requires a persistent connection that per-invocation Workers don't maintain. Don't suggest it as an optimization. → `sdk-.md` for the per-SDK setting names. - -## Pitfalls specific to this execution model - -These are the invocation-shaped members of the pitfall list in `SKILL.md`; the rest apply to any provider. - -1. **Failed first invocation.** When a version is created, the WCI invokes the Worker once to validate. If that invocation fails — missing env vars, bad TLS/auth config, missing dependencies, or an invocation deadline too short for the Worker to start and register the Task Queue — the Worker never connects, never polls, the binding is never created, and the Worker is never automatically invoked again. *Fix:* diagnose by manually invoking the function, and confirm the invocation deadline is set high. A successful manual invoke also establishes the binding. → `diagnostics.md`. - -2. **Timeout tuning mismatch.** See the timeout triple above. *Fix:* tune the three values together. - -3. **Invoke permission scoped to a single build.** *Symptom:* the deployment works, then the *next* release cannot be invoked, with an error that looks like a connection or configuration problem rather than a permissions one. *Cause:* the grant named one immutable build, and the new release is a different resource. *Fix:* scope the grant to cover the base function ARN **and** its published versions (`function:name` and `function:name:*` — the wildcard form does not cover the unqualified ARN). → `iam.md`. - -## What does *not* follow from this model - -Stated explicitly, because these are easy to over-generalize from Lambda: - -- **"Serverless Worker" does not imply a per-invocation lifetime.** A provider that scales a pool of long-lived instances is still a Serverless Worker driven by the same WCI, with the same Worker Deployment Versioning, and none of the constraints above apply to it in the same form. -- **The shutdown deadline buffer is a property of the Lambda Worker packages**, not of Temporal. -- **Connection-per-invocation is a Lambda property.** Anything reasoning from "the connection is not persistent" — eager Activities being the example above — needs rechecking against a provider that holds the connection for an instance's lifetime. diff --git a/references/aws-lambda/setup.md b/references/aws-lambda/setup.md index 94016b4..30083bd 100644 --- a/references/aws-lambda/setup.md +++ b/references/aws-lambda/setup.md @@ -30,24 +30,33 @@ Steps 4–6 and the CLI troubleshooting paths use the `temporal` CLI. Install it ```bash export TEMPORAL_ADDRESS="..tmprl.cloud:7233" export TEMPORAL_NAMESPACE="." -printf 'Temporal API key: ' >&2 -IFS= read -r -s TEMPORAL_API_KEY -printf '\n' >&2 -export TEMPORAL_API_KEY +export TEMPORAL_API_KEY="" ``` -An existing profile or environment is also valid; pass `--profile prod` or `--env prod` on each command. Do not create or update its API key with `config set --value` or `env set --value` — the value would be exposed in shell history and process arguments. If no credential is already configured, use the private environment-variable prompt above. - -**Do not assume which mechanism a user has, and do not migrate them.** `--env` (YAML, `temporal env`) is the long-standing mechanism; `--profile` (TOML, `temporal config`) is newer and the CLI still marks it EXPERIMENTAL. Inspect only the non-secret properties needed for the deployment; a broad `env get` or `config get` can print stored credentials: +or configure a profile and pass `--profile prod` on each command: ```bash -temporal env get --env prod --key address -temporal env get --env prod --key namespace -temporal --profile prod config get --prop address -temporal --profile prod config get --prop namespace +temporal --profile prod config set --prop address --value "..tmprl.cloud:7233" +temporal --profile prod config set --prop namespace --value "." +temporal --profile prod config set --prop api_key --value "" ``` +or configure an environment and pass `--env prod` (or set `TEMPORAL_ENV`): + +```bash +temporal env set --env prod --key address --value "..tmprl.cloud:7233" +temporal env set --env prod --key namespace --value "." +temporal env set --env prod --key api-key --value "" +``` + +**Do not assume which of the three a user has, and do not migrate them.** `--env` (YAML, `temporal env`) is the long-standing mechanism; `--profile` (TOML, `temporal config`) is newer and the CLI still marks it EXPERIMENTAL. Both are supported — work with whichever is already configured. Read the existing values rather than asking the user to re-enter them: + +```bash +temporal env get --env prod # --env mechanism +temporal config get --prop address # --profile mechanism +``` + - For Temporal Cloud the Namespace is the fully-qualified `.`, not the bare name. - Supplying an API key auto-enables TLS; no cert flags are needed for API-key auth. - The `temporal ...` commands in Steps 4–6 assume this is configured. To create an API key, see `skill-temporal-ops`. @@ -63,7 +72,7 @@ If this fails with an auth error, note first that this is a **frontend** call | | Control plane (accounts, Namespaces, API keys) | Namespace frontend (Workflows, Worker Deployments) | |---|---|---| | Interactive | `tcld login` | `temporal ...` with address + namespace | -| Headless | `TEMPORAL_CLOUD_API_KEY` | `TEMPORAL_API_KEY` | +| Headless | `--api-key` / `TEMPORAL_CLOUD_API_KEY` | `TEMPORAL_API_KEY` | **Use `tcld` for every Temporal Cloud control-plane operation** — accounts, Namespaces, API keys, users, service accounts. Do not use the unified CLI's `temporal cloud …` subcommands for them. @@ -76,7 +85,13 @@ Two `tcld` mechanics worth knowing before you run it in an agent shell: - `tcld login --disable-pop-up` prints the URL instead of opening a browser. Auto-open is unreliable over SSH, in containers, and in remote sessions, and the user needs the URL in the conversation either way. - `tcld` prompts for confirmation before mutating operations. Non-interactively, pass the global `--auto_confirm` (note the underscore) or set `AUTO_CONFIRM=true`, then read the resulting state back — without it the command exits clean having changed nothing. -**Go to the API key first.** It requires no CLI login, no browser handshake, and works on every account type. Use the private environment-variable prompt above. +**Go to the API key first.** It requires no CLI login, no browser handshake, and works on every account type: + +```bash +export TEMPORAL_ADDRESS="..tmprl.cloud:7233" +export TEMPORAL_NAMESPACE="." +export TEMPORAL_API_KEY="" +``` Have the user create the key in the Cloud UI, signing in however they normally do, and confirm the address against the endpoint shown on the Namespace page — some Namespaces have regional endpoints that do not follow the pattern above. Never ask them to paste the key into the conversation. @@ -85,9 +100,10 @@ Have the user create the key in the Cloud UI, signing in however they normally d ```bash tcld namespace list # full Namespace objects — every name with its region and endpoint tcld namespace get -n # one Namespace +tcld apikey create --name --duration ``` -`tcld` is not guaranteed present: check `command -v tcld`, and read `tcld --help` for the flags you are about to pass. Create API keys in the Cloud UI so their values never enter the agent transcript. +`apikey create` mints a key for the calling user and creates a long-lived credential in their account — offer it and get explicit approval, never silently. `tcld` is not guaranteed present: check `command -v tcld`, and read `tcld --help` for the flags you are about to pass. **When a control-plane login fails, stop — do not debug it, retry it, or install another CLI.** Some accounts cannot complete a `tcld` login at all, and no flag, plugin upgrade, or alternate CLI changes that. Retrying burns turns without converging, and the browser path below reaches the same end state anyway. @@ -123,9 +139,9 @@ See the selected SDK reference's **Build and package** section. **A freshly created execution role may not be assumable immediately.** `create-function` can fail with an assume-role / "cannot be assumed by Lambda" error because of IAM propagation delay. Wait a few seconds and retry; it is not a policy error, so do not start rewriting the trust policy. -**Operator CLI config does not reach the function.** All three CLI mechanisms above — exported `TEMPORAL_*` variables, `--profile`, and `--env` — configure the `temporal` CLI on the operator's machine only. The function reads its own environment, set by the `--environment` block below (or a secret store). A user with a working `--env prod` or `--profile prod` still needs every value written into that block; nothing is inherited. Use the CLI configuration only for non-secret values, and collect the API key through the private prompt above. +**Operator CLI config does not reach the function.** All three CLI mechanisms above — exported `TEMPORAL_*` variables, `--profile`, and `--env` — configure the `temporal` CLI on the operator's machine only. The function reads its own environment, set by the `--environment` block below (or a secret store). A user with a working `--env prod` or `--profile prod` still needs every value written into that block; nothing is inherited. Treat their CLI configuration as the *source* of the values, not a substitute for setting them. -**Resolve the values before building the block, and check they are not empty.** The heredoc below expands shell variables, which hold values only under the env-var mechanism. Under `--env` or `--profile` they are unset, and an unset variable expands to an empty string: the JSON stays valid, `create-function` succeeds, and the function deploys with `"TEMPORAL_ADDRESS":""` — failing at first invocation with a connection error that looks nothing like its cause. Populate the address and Namespace from the non-secret lookups above, and have the user set `TEMPORAL_API_KEY` with the private prompt; never retrieve a stored key into the agent transcript. Then guard: +**Resolve the values before building the block, and check they are not empty.** The heredoc below expands shell variables, which hold values only under the env-var mechanism. Under `--env` or `--profile` they are unset, and an unset variable expands to an empty string: the JSON stays valid, `create-function` succeeds, and the function deploys with `"TEMPORAL_ADDRESS":""` — failing at first invocation with a connection error that looks nothing like its cause. Populate them from whichever mechanism the user actually has (`temporal env get --env prod`, `temporal config get --prop address`), then guard: ```bash : "${TEMPORAL_ADDRESS:?resolve this before deploying}" diff --git a/references/aws-lambda/versioning.md b/references/aws-lambda/versioning.md index 2d4f99d..1959d63 100644 --- a/references/aws-lambda/versioning.md +++ b/references/aws-lambda/versioning.md @@ -54,18 +54,6 @@ The command prints the `FunctionArn` for the new version, for example `arn:aws:l For development or non-critical workloads, you can skip `publish-version` and use an unqualified ARN to iterate faster. -### What an unqualified ARN costs - -An unqualified ARN (no version suffix) points at `$LATEST`, which changes on every redeploy. Deploying replay-unsafe code that way causes non-determinism errors for in-flight Workflows, **even ones annotated Pinned**. - -Pinned and Auto-Upgrade control how Workflows move between Worker Deployment Versions in Temporal; neither changes how a version targets Lambda. Both expect a qualified ARN naming one immutable function version. - -| Versioning Behavior | With versioned Lambda ARN | Without versioned Lambda ARN | -|---|---|---| -| **Pinned** | Existing Workflows stay on their original Lambda function version until they complete. | Existing Workflows stay on their original Worker Deployment Version, but the underlying Lambda code has already changed since `$LATEST` updated at redeploy. The new code must be replay-compatible. | -| **Auto-Upgrade** | Existing Workflows move to the new Worker Deployment Version and its new Lambda function version at the next Workflow Task after you move the Current Version. | The Lambda redeploy already changed the code for all versions. Setting the Current Version only changes routing, not which code runs. | - - To roll back, revert the Temporal Current Version with `temporal worker deployment set-current-version`. The previous Worker Deployment Version still points at its original Lambda function version and is ready to receive traffic again. ```bash diff --git a/references/gcp-cloud-run/constraints.md b/references/gcp-cloud-run/constraints.md index 8d8635d..520b521 100644 --- a/references/gcp-cloud-run/constraints.md +++ b/references/gcp-cloud-run/constraints.md @@ -1,17 +1,10 @@ # GCP Cloud Run — execution-model constraints - - Consequences of Cloud Run's execution model: **Temporal resizes a pool of long-lived instances, scaling it to zero when there is no work.** Temporal does *not* invoke your Worker per Task. -For a deliberate cross-provider comparison, see `../aws-lambda/constraints.md`. - ## Worker lifetime is an instance, not an invocation -Each pool instance runs **standard long-lived Worker code**: it connects, registers Workflows and Activities, and polls the Task Queue for its whole lifetime. There is no handler, no per-Task lifecycle, and **no serverless Worker package**. Use the Cloud Run SDK reference in this directory; some SDKs add optional conveniences, but none are required. +Each pool instance runs **standard long-lived Worker code**: it connects, registers Workflows and Activities, and polls the Task Queue for its whole lifetime. There is no handler, no per-Task lifecycle, and **no serverless Worker package**. Use the Cloud Run SDK reference in this directory; some SDKs add optional conveniences, but none are required. The WCI controls how many instances run; each instance manages its own polling and Task processing. @@ -21,13 +14,13 @@ The WCI controls how many instances run; each instance manages its own polling a - **No shutdown deadline buffer.** Cloud Run terminates instances through its own scale-in lifecycle. - **No timeout triple to tune.** Two of its three values do not exist here. - **Activities are not bounded by an invocation limit.** Their practical interruption boundary is scale-in. -- **Eager Activities are not disabled by the platform.** A pool instance holds its connection for its lifetime; confirm support against the selected SDK. +- **Eager Activities are not disabled by the platform.** A pool instance holds its connection for its lifetime; confirm support against the selected SDK. Cloud Run sends `SIGTERM` during scale-in and can send `SIGKILL` ten seconds later. Handle `SIGTERM` and configure the SDK's graceful-shutdown timeout below that window. → `sdk-.md`. ## What bounds an Activity instead: scale-in -**The WCI decides when to remove an instance from Task Queue activity, not from what any individual instance is doing.** It does not track how long an instance has been running or whether it is mid-Activity, so the instance Cloud Run stops may be one that is still executing work. +**The WCI decides when to remove an instance from Task Queue activity, not from what any individual instance is doing.** It does not track how long an instance has been running or whether it is mid-Activity, so the instance Cloud Run stops may be one that is still executing work. Graceful shutdown lets short work drain but cannot guarantee an Activity will finish. **Use Activity Heartbeats** so interrupted work resumes from its last recorded progress instead of restarting. @@ -35,8 +28,6 @@ Graceful shutdown lets short work drain but cannot guarantee an Activity will fi ## Autoscaling behavior - - The WCI combines two mechanisms: - **Immediate** — bring up instances when a Task arrives and no Worker is free to take it (a sync match failure). Absorbs bursts without waiting for the evaluation cycle. @@ -46,13 +37,13 @@ It sizes to a **target utilization of 80% by default** rather than loading every **Scale-in is deliberately more conservative than scale-out:** it holds capacity while sync match failures are still occurring and applies a cooldown before reducing the pool. With no work, it can scale to zero; the next sync match failure or backlog scales it back up. -The scaler defaults are **minimum `0`, maximum `30`, initial count `0`, and target utilization `0.8`**. Configure them in the version's Scaling and Lifecycle settings or with the Temporal CLI. The four CLI flags are coupled: omit all four to use the defaults, or provide `--gcp-cloud-run-min-instances`, `--gcp-cloud-run-max-instances`, `--gcp-cloud-run-initial-instances`, and `--gcp-cloud-run-utilization-target` together. A partial group is rejected. +The scaler defaults are **minimum `0`, maximum `30`, initial count `0`, and target utilization `0.8`**. Configure them in the version's Scaling and Lifecycle settings or with the Temporal CLI. The four CLI flags are coupled: omit all four to use the defaults, or provide `--gcp-cloud-run-min-instances`, `--gcp-cloud-run-max-instances`, `--gcp-cloud-run-initial-instances`, and `--gcp-cloud-run-utilization-target` together. A partial group is rejected. An initial count and minimum of zero do not suppress registration. The rate-based algorithm temporarily requests at least one instance when the version is registered so its Task Queues can bind, then normal scaling can return the pool to zero. A pool that later stops growing under backlog is either at its configured maximum or at a regional Cloud Run quota. ## One Worker Pool per Worker Deployment Version -**The compute configuration names a project, region, and pool — not a revision.** Temporal runs whichever revision the pool serves at the time, which ties a pool to a single build. A new build needs a **new pool**, and the Build ID belongs in the pool name to keep that mapping visible. +**The compute configuration names a project, region, and pool — not a revision.** Temporal runs whichever revision the pool serves at the time, which ties a pool to a single build. A new build needs a **new pool**, and the Build ID belongs in the pool name to keep that mapping visible. Keep an older version's pool in place while Pinned Workflows are still running on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. @@ -60,18 +51,10 @@ Keep an older version's pool in place while Pinned Workflows are still running o ## Do not share a Task Queue with long-lived Workers -**The pool scales up to cover the Task Queue's full workload even when independently managed Workers are already handling all of it**, so you run and pay for duplicate capacity. +**The pool scales up to cover the Task Queue's full workload even when independently managed Workers are already handling all of it**, so you run and pay for duplicate capacity. The WCI sizes the pool from the rate of Tasks arriving on the version's Task Queues, and nothing in that measurement accounts for the long-lived Workers. Sync matching to a long-lived Worker suppresses the *immediate* scale-up, but the periodic re-sizing scales the pool up regardless. Fixing poller counts on the long-lived side does not help — use separate Task Queues. -## Pre-release status - -Cloud Run support is **Pre-release, and its APIs may change in backwards-incompatible ways.** Access is not open: the user must create a support ticket or contact their account team. Confirm access exists before planning a deployment — this is a gate no amount of correct configuration gets past. - -## Provider-side issue worth knowing - -Google currently reports **high deployment latency creating or updating Cloud Run resources in some regions, including `us-central1`**, and recommends deploying elsewhere while it is open. If pool operations are slow, check [Cloud Run known issues](https://cloud.google.com/run/docs/known-issues) before assuming misconfiguration. - ## What does *not* follow from this model - **Use ordinary Worker code.** Handler, configure-callback, and provider-package patterns do not apply. diff --git a/references/gcp-cloud-run/diagnostics.md b/references/gcp-cloud-run/diagnostics.md index 71dc9f7..6747bf6 100644 --- a/references/gcp-cloud-run/diagnostics.md +++ b/references/gcp-cloud-run/diagnostics.md @@ -1,14 +1,7 @@ # GCP Cloud Run — Diagnostics & troubleshooting - - ## Scaling flow (when working correctly) - - 1. You deploy the Worker image to a Worker Pool at zero instances. 2. You create a Worker Deployment Version pointing at that pool. This starts a WCI Workflow. 3. An instance starts, the Worker polls, and the server **binds the Task Queue** to the version. @@ -26,7 +19,7 @@ gcloud run worker-pools describe \ --region --project --format=yaml ``` -Three fields under `metadata.annotations`: +Three fields under `metadata.annotations`: | Field | What it tells you | |---|---| @@ -45,7 +38,7 @@ Three fields under `metadata.annotations`: +**It starts no instance and does not exercise `run.workerPools.update`.** Version registration is different: its Task Queue bootstrap does update the pool. An invoker with read but not update permission can therefore pass this manual validation, but version registration or a later resize fails. If validation succeeds but registration never changes `lastModifier`, **check the update permission**. On failure, check each part of the compute configuration against the pool: @@ -100,9 +93,9 @@ If instances are running but the count stops growing while backlog builds: gcloud run worker-pools logs read --region --project ``` -**The pool produces no logs while scaled to zero** — read them while an instance is up. An empty log is not evidence of failure. +**A scaled-to-zero pool emits no new logs.** `logs read` still returns historical entries; use `logs tail` only while an instance is running. An empty result may simply mean the pool has never started. -Common errors: +Common errors: - **Connection failures** — check `TEMPORAL_ADDRESS` and `TEMPORAL_NAMESPACE` on the pool. Self-hosted: verify network reachability from Cloud Run to the frontend. - **Missing secrets** — the instance cannot read the API key or TLS material. The **runner** service account needs `roles/secretmanager.secretAccessor` on the secret. That is the account in `spec.template.spec.serviceAccountName`, **not the invoker.** This is the most common consequence of confusing the two. @@ -123,7 +116,7 @@ This is expected behavior, not a misconfiguration. Confirm the Worker handles `S ## Rule out a GCP-side cause -If every check passes, the cause may be in Cloud Run rather than your configuration: +If every check passes, the cause may be in Cloud Run rather than your configuration: - [Cloud Run known issues](https://cloud.google.com/run/docs/known-issues) — includes issues affecting how long pool operations take. - [Google Cloud Service Health](https://status.cloud.google.com/) — active incidents by product and region. diff --git a/references/gcp-cloud-run/iam.md b/references/gcp-cloud-run/iam.md index 9a94ff7..4d9d800 100644 --- a/references/gcp-cloud-run/iam.md +++ b/references/gcp-cloud-run/iam.md @@ -1,11 +1,5 @@ # GCP Cloud Run — IAM & permissions - - Cloud Run uses three identities: | Identity | Purpose | @@ -16,13 +10,13 @@ Cloud Run uses three identities: Temporal reaches the invoker through `roles/iam.serviceAccountTokenCreator`. The Terraform module described below creates the invoker and applies its grants. -**The two service accounts are not interchangeable**, and confusing them is the single most likely IAM mistake here. The runner runs the pool and never scales it; the invoker scales the pool and never runs it. +**The two service accounts are not interchangeable**, and confusing them is the single most likely IAM mistake here. The runner runs the pool and never scales it; the invoker scales the pool and never runs it. ## Runner service account The runtime identity the pool's instances use to reach other Google Cloud services. Set in `setup.md` Step 4 with `gcloud run worker-pools deploy --service-account`. It may be an account that already exists; a dedicated one is preferred. -**It needs no baseline role to run the Worker.** Cloud Run collects `stdout` and `stderr` into Cloud Logging through its own infrastructure, and the Cloud Run *service agent* — not the runner — pulls the container image. Grant only what your code actually reaches: +**It needs no baseline role to run the Worker.** Cloud Run collects `stdout` and `stderr` into Cloud Logging through its own infrastructure, and the Cloud Run *service agent* — not the runner — pulls the container image. Grant only what your code actually reaches: - `roles/secretmanager.secretAccessor` on each secret you mount, including the Temporal API key. - `roles/logging.logWriter` **only if** the Worker writes through the Cloud Logging API rather than stdout/stderr. @@ -30,7 +24,7 @@ The runtime identity the pool's instances use to reach other Google Cloud servic ## Invoker service account -The identity Temporal Cloud impersonates to read and scale the pool. Two grants make it work: +The identity Temporal Cloud impersonates to read and scale the pool. Two grants make it work: - Temporal's identity receives **`roles/iam.serviceAccountTokenCreator`** on the invoker, so it can impersonate it. - The invoker receives a project-level Cloud Run role with at least **`run.workerPools.get`** (read) and **`run.workerPools.update`** (scale). `roles/run.developer` includes both. @@ -39,7 +33,7 @@ The invoker also needs **`roles/iam.serviceAccountUser` on the runner service ac ### The read/update split is a real trap -`run.workerPools.get` alone is enough for the UI's **Validate Connection** action to pass. That manual action never exercises `run.workerPools.update`. Version registration does: the WCI reads the pool and then updates its manual instance count to bootstrap Task Queue registration. An invoker that can read but not update therefore passes manual validation but fails version registration or a later resize. +`run.workerPools.get` alone is enough for the UI's **Validate Connection** action to pass. That manual action never exercises `run.workerPools.update`. Version registration does: the WCI reads the pool and then updates its manual instance count to bootstrap Task Queue registration. An invoker that can read but not update therefore passes manual validation but fails version registration or a later resize. Verify the registration bootstrap and `lastModifier` rather than trusting the separate green Validate Connection result. → `diagnostics.md`. @@ -47,7 +41,7 @@ Verify the registration bootstrap and `lastModifier` rather than trusting the se Temporal publishes [`serverless-workers/gcp/cloud-run`](https://github.com/temporalio/terraform-modules/tree/main/modules/serverless-workers/gcp/cloud-run), which creates the invoker service account and applies the grants. -**Get the template from the Cloud UI, not from here.** Under **Workers → Create Worker Deployment → Access**, Temporal Cloud emits a template with `impersonator_service_account_emails` already filled in for your account. Those values are account-specific, which is why every published snippet shows a placeholder. +Start from the template under **Workers → Create Worker Deployment → Access** because Temporal Cloud fills in the account-specific `impersonator_service_account_emails`. Its shape is: ```hcl module "serverless-worker-cloud-run" { @@ -101,7 +95,7 @@ The operator needs `iam.serviceAccounts.actAs` on the runner to attach it to the ### Preflight -Run before anything that creates or modifies GCP resources. Confirm the five required APIs appear in the enabled-service output, then inspect the names the deployment intends to use: +Run before anything that creates or modifies GCP resources. Confirm the required APIs appear in the enabled-service output, then inspect the names the deployment intends to use: ```bash gcloud auth list # which identity diff --git a/references/gcp-cloud-run/observability.md b/references/gcp-cloud-run/observability.md index cb0bb8e..b045314 100644 --- a/references/gcp-cloud-run/observability.md +++ b/references/gcp-cloud-run/observability.md @@ -1,17 +1,12 @@ # GCP Cloud Run — observability - - ## There is nothing serverless-specific to configure -**A Cloud Run Serverless Worker emits the same traces and metrics as a Worker anywhere else.** It is an ordinary long-lived Worker, so the SDK's normal metrics and OpenTelemetry tracing setup applies unchanged, and each SDK's general observability guide is the right reference. +**A Cloud Run Serverless Worker emits the same traces and metrics as a Worker anywhere else.** It is an ordinary long-lived Worker, so the SDK's normal metrics and OpenTelemetry tracing setup applies unchanged, and each SDK's general observability guide is the right reference. Do not add provider-specific helper layers, collector environment variables, or invocation-deadline flush logic. Export telemetry as you would from any long-lived container. -Some SDKs add optional Cloud Run conveniences, such as OpenTelemetry helpers. **They are optional**, and where they exist they are documented in that SDK's Cloud Run guide. +Some SDKs add optional Cloud Run conveniences, such as OpenTelemetry helpers. **They are optional**, and where they exist they are documented in that SDK's Cloud Run guide. ## Logs @@ -23,18 +18,7 @@ Read a pool's logs: gcloud run worker-pools logs read --region --project ``` -**The pool produces no logs while scaled to zero.** Read them while an instance is up; an empty log is not evidence of a problem. - -## Memory: give the runtime the instance, not the host - -A Worker Pool instance defaults to **512 MiB**; raise `--memory` when creating the pool if the Worker needs more. Two runtimes need to be told about the container limit explicitly, or they size themselves to a fraction of it: - -| SDK | Setting | Why | -|---|---|---| -| Java | `-XX:MaxRAMPercentage=75` | The JVM reads the container limit but defaults max heap to 25% of it, leaving most of a small instance unused. | -| TypeScript | `NODE_OPTIONS=--max-old-space-size=`, ~80% of the instance limit | Node's default heap is unrelated to the container limit. | - -Both are container-sizing concerns rather than Temporal ones, but they surface as Worker instability under load and are easy to miss. → `setup.md`. +**A scaled-to-zero pool emits no new logs.** `logs read` returns historical entries; `logs tail` shows new output only while an instance is running. ## What to watch that is specific to this provider diff --git a/references/gcp-cloud-run/self-hosted.md b/references/gcp-cloud-run/self-hosted.md index 282e5bc..33c4212 100644 --- a/references/gcp-cloud-run/self-hosted.md +++ b/references/gcp-cloud-run/self-hosted.md @@ -1,13 +1,7 @@ # GCP Cloud Run — self-hosted Temporal Service setup - - Serverless Workers require **Temporal Service v1.31.0 or later**. Complete this page before following `setup.md`. -Four prerequisites: network reachability, enable the WCI, give the server a GCP identity, create the invoker service account. - ## 1. Cloud Run instances must reach the Temporal Service The frontend must be reachable **from the Worker Pool instances**. If the Service runs on a private network, that likely means [Direct VPC egress](https://cloud.google.com/run/docs/configuring/vpc-direct-vpc) or a [Serverless VPC Access connector](https://cloud.google.com/run/docs/configuring/vpc-connectors). @@ -31,7 +25,7 @@ workercontroller.scaling_algorithms.enabled: - rate-based ``` -**Cloud Run requires the `rate-based` algorithm.** Because a pool is a set of long-lived instances, the WCI resizes it from arrival and backlog rates. The `no-sync` algorithm applies only to providers invoked once per Task, and **pairing it with `gcp-cloud-run` is rejected** — a concrete way the two providers' execution models surface in server configuration. +**Cloud Run requires the `rate-based` algorithm.** Because a pool is a set of long-lived instances, the WCI resizes it from arrival and backlog rates. Pairing `no-sync` with `gcp-cloud-run` is rejected. To enable per Namespace instead of globally: @@ -68,26 +62,7 @@ Two grants: - The GCP identity the Service runs as (step 3) gets **`roles/iam.serviceAccountTokenCreator`** on the invoker. - The invoker gets a project-level Cloud Run role with at least **`run.workerPools.get`** and **`run.workerPools.update`**. `roles/run.developer` includes both. -The same Terraform module works — pass the server's GCP identity as the impersonator instead of Temporal Cloud's accounts: - -```hcl -module "serverless-worker-cloud-run" { - source = "github.com/temporalio/terraform-modules//modules/serverless-workers/gcp/cloud-run" - - project_id = "" - invoker_account_id = "temporal-serverless-worker" - - runner_service_account_email = "" - - impersonator_service_account_emails = [ - "", - ] -} -``` - -Use the module's `invoker_email` output as `--gcp-cloud-run-service-account` when registering the Worker Deployment Version. - -**This is the one place self-hosted is simpler than Cloud:** there is no UI-provided template to copy, because you already know the impersonating identity — it is your own server's. +Use the Terraform module in `iam.md`, setting `impersonator_service_account_emails` to the GCP identity used by the Temporal Service. Use the module's `invoker_email` output as `--gcp-cloud-run-service-account` when registering the Worker Deployment Version. ## Then diff --git a/references/gcp-cloud-run/setup.md b/references/gcp-cloud-run/setup.md index dee3d23..ed2f72a 100644 --- a/references/gcp-cloud-run/setup.md +++ b/references/gcp-cloud-run/setup.md @@ -1,16 +1,9 @@ # GCP Cloud Run — Setup (happy path) - - End-to-end: write a standard Worker, containerize it, push the image, create a Worker Pool at zero instances, grant Temporal permission to scale it, register a Worker Deployment Version, set it current, verify. For the two service accounts and the Terraform module, see `iam.md`. For what the execution model does and does not bound, see `constraints.md`. For new builds and rollback, see `versioning.md`. If it doesn't work, see `diagnostics.md`. ## Prerequisites - - - **Cloud Run support is Pre-release and access-gated.** The user creates a support ticket or contacts their account team. Confirm this before anything else. - A Temporal Cloud account with a **GCP-hosted Namespace**, or self-hosted Temporal Service v1.31.0+. The Namespace must be hosted on GCP; its region need not match the pool's. - For self-hosted, complete `self-hosted.md` first. @@ -33,8 +26,6 @@ export TEMPORAL_API_KEY Do not run the secret-reading commands through an agent shell, ask the user to paste the key into conversation, or inspect the resulting variable. A configured Temporal CLI profile is equally valid and avoids a session environment variable. -**Check the region before deploying.** Google reports high deployment latency creating or updating Cloud Run resources in some regions, including `us-central1`, and recommends another region while the issue is open. → `constraints.md`. - ## Prepare a clean GCP project After the resource list is approved, create only what is missing. First enable every API used by the commands below: @@ -105,23 +96,11 @@ Two things the Worker must do: Per-SDK code lives in `sdk-.md` in this directory, one file per SDK. Each covers the versioned Worker, connection, image packaging, graceful shutdown, scale-in safety, and observability. The Java, Python, and .NET references also include their logging setup and diagnostic signatures. -**The entrypoint must start the Worker process**, so an instance begins polling as soon as it starts. +**The entrypoint must start the Worker process**, so an instance begins polling as soon as it starts. ## Step 2: Containerize the Worker - - -Per-runtime notes that matter, from the deployment guide: - -| SDK | Notes | -|---|---| -| Go | Multi-stage; `CGO_ENABLED=0` for a static binary, which is what a `distroless/static` base expects. | -| Python | `pip install "temporalio>=1.30.0,<2"`; entrypoint runs the Worker module. | -| TypeScript | **Keep `ca-certificates` installed** — without it the Worker fails at startup with `TransportError: tonic::transport::Error(Transport, NativeCertsNotFound)`. Use a **glibc** image, not Alpine. Set `NODE_OPTIONS=--max-old-space-size=` to ~80% of the instance memory limit. | -| Java | Fat jar on a JRE image; set `-XX:MaxRAMPercentage=75` — the JVM reads the container limit but defaults max heap to 25% of it. | -| .NET | `dotnet publish` in a build stage, run on the .NET runtime image. | - -**`NativeCertsNotFound` means a Rust-core SDK cannot find system root CAs.** Any Rust-core SDK (TypeScript, Python, .NET, Ruby) in a minimal image needs CA certificates present. +Follow the selected `sdk-.md` reference for the container image, runtime, entrypoint, certificate requirements, and memory settings. ## Step 3: Build and push the image @@ -207,7 +186,7 @@ Creating the Worker Deployment Version starts its WCI. The WCI validates the con The update completing proves that Temporal reached Cloud Run, not that the container started or the Worker connected. Wait for the expected Task Queue types to appear; that binding is proof the instance started and polled under the registered deployment name and build ID. -The separate UI **Validate Connection** action (Workers → Deployments → select → Actions) only impersonates the invoker and reads the pool. It starts no instance and does not exercise `run.workerPools.update`, so a green manual validation is weaker than a successful registration bootstrap. +The separate UI **Validate Connection** action (Workers → Deployments → select → Actions) only impersonates the invoker and reads the pool. It starts no instance and does not exercise `run.workerPools.update`, so a green manual validation is weaker than a successful registration bootstrap. Use both views when diagnosing the checkpoint: @@ -250,7 +229,7 @@ Confirm from two independent signals: gcloud run worker-pools logs read my-temporal-worker-pool-build-1 \ --region --project ``` - **The pool produces no logs while scaled to zero**, so read them while an instance is up. An empty log is not evidence of failure. + **A scaled-to-zero pool emits no new logs.** Use `logs read` for historical entries and `logs tail` while an instance is running. ## Teardown diff --git a/references/gcp-cloud-run/versioning.md b/references/gcp-cloud-run/versioning.md index bd16258..a673d6a 100644 --- a/references/gcp-cloud-run/versioning.md +++ b/references/gcp-cloud-run/versioning.md @@ -1,19 +1,14 @@ # GCP Cloud Run — versioning, updates, and rollback - - ## One Worker Pool per Build ID -**The compute configuration names a project, region, and Worker Pool — it does not name a [revision](https://cloud.google.com/run/docs/managing/revisions).** Temporal runs whichever revision the pool happens to serve. That ties a pool to a single build, so **a new build needs a new pool.** Carry the Build ID in the pool name (`my-worker-pool-build-1`) so the mapping stays visible. +**The compute configuration names a project, region, and Worker Pool — it does not name a [revision](https://cloud.google.com/run/docs/managing/revisions).** Temporal runs whichever revision the pool happens to serve. That ties a pool to a single build, so **a new build needs a new pool.** Carry the Build ID in the pool name (`my-worker-pool-build-1`) so the mapping stays visible. Temporal's Cloud Run compute configuration has no revision selector, so **durable isolation requires a separate pool per build.** A pool-level instance split can temporarily hold a particular revision, but that split remains mutable state outside Temporal. ## The hazard: redeploying into a live pool -> Deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run promotes it to every instance by default. The version does not change, but the code behind it does. Deploying replay-unsafe code this way causes non-determinism errors for in-flight Workflows, **including Pinned ones**. +> Deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run promotes it to every instance by default. The version does not change, but the code behind it does. Deploying replay-unsafe code this way causes non-determinism errors for in-flight Workflows, **including Pinned ones**. **A normal `gcloud run worker-pools deploy` takes the mutable path by default.** The durable Temporal-aligned path is a new pool; `--no-promote` is only a same-pool guardrail. @@ -58,7 +53,7 @@ After emergency recovery, return to one pool per Build ID for the next release s 3. Register a new Worker Deployment Version pointing at the new pool, with a build ID matching the new Worker code. 4. Confirm registration bootstrapped the pool: its `lastModifier` shows the invoker and the expected Task Queue types are bound. Treat the separate UI Validate Connection action as a read-only check. → `diagnostics.md`. 5. Set the new version current, or ramp to it. -6. **Leave the old pool in place** while Pinned Workflows still run on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. +6. **Leave the old pool in place** while Pinned Workflows still run on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. Only the invoker's permissions are shared across pools, so a new pool usually needs no IAM change — provided the invoker's `deploy_roles` are project-level, which is the module's default. Check `iam.md` if you scoped them to individual pools instead. From 28b5aa95c22ba7d86b3b1692a102ba5a42ddae4f Mon Sep 17 00:00:00 2001 From: Harish Narayanappa Date: Fri, 4 Sep 2026 11:47:39 -0700 Subject: [PATCH 4/4] Restore Cloud Run source citations --- references/gcp-cloud-run/constraints.md | 12 +++++++----- references/gcp-cloud-run/diagnostics.md | 10 ++++++---- references/gcp-cloud-run/iam.md | 10 +++++----- references/gcp-cloud-run/observability.md | 4 ++-- references/gcp-cloud-run/self-hosted.md | 14 ++++++++------ references/gcp-cloud-run/setup.md | 6 ++++-- references/gcp-cloud-run/versioning.md | 6 +++--- 7 files changed, 35 insertions(+), 27 deletions(-) diff --git a/references/gcp-cloud-run/constraints.md b/references/gcp-cloud-run/constraints.md index 520b521..82d9aeb 100644 --- a/references/gcp-cloud-run/constraints.md +++ b/references/gcp-cloud-run/constraints.md @@ -4,7 +4,7 @@ Consequences of Cloud Run's execution model: **Temporal resizes a pool of long-l ## Worker lifetime is an instance, not an invocation -Each pool instance runs **standard long-lived Worker code**: it connects, registers Workflows and Activities, and polls the Task Queue for its whole lifetime. There is no handler, no per-Task lifecycle, and **no serverless Worker package**. Use the Cloud Run SDK reference in this directory; some SDKs add optional conveniences, but none are required. +Each pool instance runs **standard long-lived Worker code**: it connects, registers Workflows and Activities, and polls the Task Queue for its whole lifetime. There is no handler, no per-Task lifecycle, and **no serverless Worker package**. Use the Cloud Run SDK reference in this directory; some SDKs add optional conveniences, but none are required. The WCI controls how many instances run; each instance manages its own polling and Task processing. @@ -20,7 +20,7 @@ Cloud Run sends `SIGTERM` during scale-in and can send `SIGKILL` ten seconds lat ## What bounds an Activity instead: scale-in -**The WCI decides when to remove an instance from Task Queue activity, not from what any individual instance is doing.** It does not track how long an instance has been running or whether it is mid-Activity, so the instance Cloud Run stops may be one that is still executing work. +**The WCI decides when to remove an instance from Task Queue activity, not from what any individual instance is doing.** It does not track how long an instance has been running or whether it is mid-Activity, so the instance Cloud Run stops may be one that is still executing work. Graceful shutdown lets short work drain but cannot guarantee an Activity will finish. **Use Activity Heartbeats** so interrupted work resumes from its last recorded progress instead of restarting. @@ -28,6 +28,8 @@ Graceful shutdown lets short work drain but cannot guarantee an Activity will fi ## Autoscaling behavior + + The WCI combines two mechanisms: - **Immediate** — bring up instances when a Task arrives and no Worker is free to take it (a sync match failure). Absorbs bursts without waiting for the evaluation cycle. @@ -37,13 +39,13 @@ It sizes to a **target utilization of 80% by default** rather than loading every **Scale-in is deliberately more conservative than scale-out:** it holds capacity while sync match failures are still occurring and applies a cooldown before reducing the pool. With no work, it can scale to zero; the next sync match failure or backlog scales it back up. -The scaler defaults are **minimum `0`, maximum `30`, initial count `0`, and target utilization `0.8`**. Configure them in the version's Scaling and Lifecycle settings or with the Temporal CLI. The four CLI flags are coupled: omit all four to use the defaults, or provide `--gcp-cloud-run-min-instances`, `--gcp-cloud-run-max-instances`, `--gcp-cloud-run-initial-instances`, and `--gcp-cloud-run-utilization-target` together. A partial group is rejected. +The scaler defaults are **minimum `0`, maximum `30`, initial count `0`, and target utilization `0.8`**. Configure them in the version's Scaling and Lifecycle settings or with the Temporal CLI. The four CLI flags are coupled: omit all four to use the defaults, or provide `--gcp-cloud-run-min-instances`, `--gcp-cloud-run-max-instances`, `--gcp-cloud-run-initial-instances`, and `--gcp-cloud-run-utilization-target` together. A partial group is rejected. An initial count and minimum of zero do not suppress registration. The rate-based algorithm temporarily requests at least one instance when the version is registered so its Task Queues can bind, then normal scaling can return the pool to zero. A pool that later stops growing under backlog is either at its configured maximum or at a regional Cloud Run quota. ## One Worker Pool per Worker Deployment Version -**The compute configuration names a project, region, and pool — not a revision.** Temporal runs whichever revision the pool serves at the time, which ties a pool to a single build. A new build needs a **new pool**, and the Build ID belongs in the pool name to keep that mapping visible. +**The compute configuration names a project, region, and pool — not a revision.** Temporal runs whichever revision the pool serves at the time, which ties a pool to a single build. A new build needs a **new pool**, and the Build ID belongs in the pool name to keep that mapping visible. Keep an older version's pool in place while Pinned Workflows are still running on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. @@ -51,7 +53,7 @@ Keep an older version's pool in place while Pinned Workflows are still running o ## Do not share a Task Queue with long-lived Workers -**The pool scales up to cover the Task Queue's full workload even when independently managed Workers are already handling all of it**, so you run and pay for duplicate capacity. +**The pool scales up to cover the Task Queue's full workload even when independently managed Workers are already handling all of it**, so you run and pay for duplicate capacity. The WCI sizes the pool from the rate of Tasks arriving on the version's Task Queues, and nothing in that measurement accounts for the long-lived Workers. Sync matching to a long-lived Worker suppresses the *immediate* scale-up, but the periodic re-sizing scales the pool up regardless. Fixing poller counts on the long-lived side does not help — use separate Task Queues. diff --git a/references/gcp-cloud-run/diagnostics.md b/references/gcp-cloud-run/diagnostics.md index 6747bf6..5b360c3 100644 --- a/references/gcp-cloud-run/diagnostics.md +++ b/references/gcp-cloud-run/diagnostics.md @@ -2,6 +2,8 @@ ## Scaling flow (when working correctly) + + 1. You deploy the Worker image to a Worker Pool at zero instances. 2. You create a Worker Deployment Version pointing at that pool. This starts a WCI Workflow. 3. An instance starts, the Worker polls, and the server **binds the Task Queue** to the version. @@ -19,7 +21,7 @@ gcloud run worker-pools describe \ --region --project --format=yaml ``` -Three fields under `metadata.annotations`: +Three fields under `metadata.annotations`: | Field | What it tells you | |---|---| @@ -38,7 +40,7 @@ Three fields under `metadata.annotations`: Workers → Deployments → select deployment → Actions → **Validate Connection**. For Cloud Run this impersonates the invoker and reads the pool, confirming three things: the compute configuration names a pool that exists, Temporal can impersonate the invoker, and the invoker can read. -**It starts no instance and does not exercise `run.workerPools.update`.** Version registration is different: its Task Queue bootstrap does update the pool. An invoker with read but not update permission can therefore pass this manual validation, but version registration or a later resize fails. If validation succeeds but registration never changes `lastModifier`, **check the update permission**. +**It starts no instance and does not exercise `run.workerPools.update`.** Version registration is different: its Task Queue bootstrap does update the pool. An invoker with read but not update permission can therefore pass this manual validation, but version registration or a later resize fails. If validation succeeds but registration never changes `lastModifier`, **check the update permission**. On failure, check each part of the compute configuration against the pool: @@ -95,7 +97,7 @@ gcloud run worker-pools logs read --region --project - **Connection failures** — check `TEMPORAL_ADDRESS` and `TEMPORAL_NAMESPACE` on the pool. Self-hosted: verify network reachability from Cloud Run to the frontend. - **Missing secrets** — the instance cannot read the API key or TLS material. The **runner** service account needs `roles/secretmanager.secretAccessor` on the secret. That is the account in `spec.template.spec.serviceAccountName`, **not the invoker.** This is the most common consequence of confusing the two. @@ -116,7 +118,7 @@ This is expected behavior, not a misconfiguration. Confirm the Worker handles `S ## Rule out a GCP-side cause -If every check passes, the cause may be in Cloud Run rather than your configuration: +If every check passes, the cause may be in Cloud Run rather than your configuration: - [Cloud Run known issues](https://cloud.google.com/run/docs/known-issues) — includes issues affecting how long pool operations take. - [Google Cloud Service Health](https://status.cloud.google.com/) — active incidents by product and region. diff --git a/references/gcp-cloud-run/iam.md b/references/gcp-cloud-run/iam.md index 4d9d800..a6d1de2 100644 --- a/references/gcp-cloud-run/iam.md +++ b/references/gcp-cloud-run/iam.md @@ -10,13 +10,13 @@ Cloud Run uses three identities: Temporal reaches the invoker through `roles/iam.serviceAccountTokenCreator`. The Terraform module described below creates the invoker and applies its grants. -**The two service accounts are not interchangeable**, and confusing them is the single most likely IAM mistake here. The runner runs the pool and never scales it; the invoker scales the pool and never runs it. +**The two service accounts are not interchangeable**, and confusing them is the single most likely IAM mistake here. The runner runs the pool and never scales it; the invoker scales the pool and never runs it. ## Runner service account The runtime identity the pool's instances use to reach other Google Cloud services. Set in `setup.md` Step 4 with `gcloud run worker-pools deploy --service-account`. It may be an account that already exists; a dedicated one is preferred. -**It needs no baseline role to run the Worker.** Cloud Run collects `stdout` and `stderr` into Cloud Logging through its own infrastructure, and the Cloud Run *service agent* — not the runner — pulls the container image. Grant only what your code actually reaches: +**It needs no baseline role to run the Worker.** Cloud Run collects `stdout` and `stderr` into Cloud Logging through its own infrastructure, and the Cloud Run *service agent* — not the runner — pulls the container image. Grant only what your code actually reaches: - `roles/secretmanager.secretAccessor` on each secret you mount, including the Temporal API key. - `roles/logging.logWriter` **only if** the Worker writes through the Cloud Logging API rather than stdout/stderr. @@ -24,7 +24,7 @@ The runtime identity the pool's instances use to reach other Google Cloud servic ## Invoker service account -The identity Temporal Cloud impersonates to read and scale the pool. Two grants make it work: +The identity Temporal Cloud impersonates to read and scale the pool. Two grants make it work: - Temporal's identity receives **`roles/iam.serviceAccountTokenCreator`** on the invoker, so it can impersonate it. - The invoker receives a project-level Cloud Run role with at least **`run.workerPools.get`** (read) and **`run.workerPools.update`** (scale). `roles/run.developer` includes both. @@ -33,7 +33,7 @@ The invoker also needs **`roles/iam.serviceAccountUser` on the runner service ac ### The read/update split is a real trap -`run.workerPools.get` alone is enough for the UI's **Validate Connection** action to pass. That manual action never exercises `run.workerPools.update`. Version registration does: the WCI reads the pool and then updates its manual instance count to bootstrap Task Queue registration. An invoker that can read but not update therefore passes manual validation but fails version registration or a later resize. +`run.workerPools.get` alone is enough for the UI's **Validate Connection** action to pass. That manual action never exercises `run.workerPools.update`. Version registration does: the WCI reads the pool and then updates its manual instance count to bootstrap Task Queue registration. An invoker that can read but not update therefore passes manual validation but fails version registration or a later resize. Verify the registration bootstrap and `lastModifier` rather than trusting the separate green Validate Connection result. → `diagnostics.md`. @@ -41,7 +41,7 @@ Verify the registration bootstrap and `lastModifier` rather than trusting the se Temporal publishes [`serverless-workers/gcp/cloud-run`](https://github.com/temporalio/terraform-modules/tree/main/modules/serverless-workers/gcp/cloud-run), which creates the invoker service account and applies the grants. -Start from the template under **Workers → Create Worker Deployment → Access** because Temporal Cloud fills in the account-specific `impersonator_service_account_emails`. Its shape is: +Start from the template under **Workers → Create Worker Deployment → Access** because Temporal Cloud fills in the account-specific `impersonator_service_account_emails`. Its shape is: ```hcl module "serverless-worker-cloud-run" { diff --git a/references/gcp-cloud-run/observability.md b/references/gcp-cloud-run/observability.md index b045314..968cdb0 100644 --- a/references/gcp-cloud-run/observability.md +++ b/references/gcp-cloud-run/observability.md @@ -2,11 +2,11 @@ ## There is nothing serverless-specific to configure -**A Cloud Run Serverless Worker emits the same traces and metrics as a Worker anywhere else.** It is an ordinary long-lived Worker, so the SDK's normal metrics and OpenTelemetry tracing setup applies unchanged, and each SDK's general observability guide is the right reference. +**A Cloud Run Serverless Worker emits the same traces and metrics as a Worker anywhere else.** It is an ordinary long-lived Worker, so the SDK's normal metrics and OpenTelemetry tracing setup applies unchanged, and each SDK's general observability guide is the right reference. Do not add provider-specific helper layers, collector environment variables, or invocation-deadline flush logic. Export telemetry as you would from any long-lived container. -Some SDKs add optional Cloud Run conveniences, such as OpenTelemetry helpers. **They are optional**, and where they exist they are documented in that SDK's Cloud Run guide. +Some SDKs add optional Cloud Run conveniences, such as OpenTelemetry helpers. **They are optional**, and where they exist they are documented in that SDK's Cloud Run guide. ## Logs diff --git a/references/gcp-cloud-run/self-hosted.md b/references/gcp-cloud-run/self-hosted.md index 33c4212..b4e02d5 100644 --- a/references/gcp-cloud-run/self-hosted.md +++ b/references/gcp-cloud-run/self-hosted.md @@ -1,10 +1,10 @@ # GCP Cloud Run — self-hosted Temporal Service setup -Serverless Workers require **Temporal Service v1.31.0 or later**. Complete this page before following `setup.md`. +Serverless Workers require **Temporal Service v1.31.0 or later**. Complete this page before following `setup.md`. ## 1. Cloud Run instances must reach the Temporal Service -The frontend must be reachable **from the Worker Pool instances**. If the Service runs on a private network, that likely means [Direct VPC egress](https://cloud.google.com/run/docs/configuring/vpc-direct-vpc) or a [Serverless VPC Access connector](https://cloud.google.com/run/docs/configuring/vpc-connectors). +The frontend must be reachable **from the Worker Pool instances**. If the Service runs on a private network, that likely means [Direct VPC egress](https://cloud.google.com/run/docs/configuring/vpc-direct-vpc) or a [Serverless VPC Access connector](https://cloud.google.com/run/docs/configuring/vpc-connectors). Note the direction: instances dial *out* to Temporal. Nothing needs to reach *into* Cloud Run — Temporal drives the pool through the Cloud Run admin API, not by connecting to your instances. @@ -25,7 +25,7 @@ workercontroller.scaling_algorithms.enabled: - rate-based ``` -**Cloud Run requires the `rate-based` algorithm.** Because a pool is a set of long-lived instances, the WCI resizes it from arrival and backlog rates. Pairing `no-sync` with `gcp-cloud-run` is rejected. +**Cloud Run requires the `rate-based` algorithm.** Because a pool is a set of long-lived instances, the WCI resizes it from arrival and backlog rates. Pairing `no-sync` with `gcp-cloud-run` is rejected. To enable per Namespace instead of globally: @@ -36,7 +36,7 @@ workercontroller.enabled: namespace: 'your-namespace' ``` -The Service watches the file and applies updates **without a restart**. +The Service watches the file and applies updates **without a restart**. Two optional global keys cover multi-hop impersonation: @@ -45,13 +45,15 @@ Two optional global keys cover multi-hop impersonation: | `workercontroller.compute_providers.gcp.intermediary_service_accounts` | Service accounts to impersonate in sequence before the invoker. Unset means the server impersonates the invoker directly. | | `workercontroller.compute_providers.gcp.first_delegate_as_base` | When `true`, the first entry is the identity the server's ambient credentials impersonate directly and the rest are passed as token-creator delegates. Defaults to `false`, passing the whole chain as delegates. | + + ## 3. Give the Temporal Service a GCP identity The server impersonates the invoker service account, so it must first run as a GCP identity permitted to do so. **On GCP (GCE, GKE):** the attached service account is used automatically through [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). No extra credential configuration — you grant *that* account impersonation rights in step 4. -**Outside GCP:** use [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) and point `GOOGLE_APPLICATION_CREDENTIALS` at the credential configuration file it produces. That variable also accepts a service account key file, but **Google recommends against long-lived keys** — prefer federation, and never ask a user to paste a key. +**Outside GCP:** use [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) and point `GOOGLE_APPLICATION_CREDENTIALS` at the credential configuration file it produces. That variable also accepts a service account key file, but **Google recommends against long-lived keys** — prefer federation, and never ask a user to paste a key. ## 4. Create the invoker service account @@ -62,7 +64,7 @@ Two grants: - The GCP identity the Service runs as (step 3) gets **`roles/iam.serviceAccountTokenCreator`** on the invoker. - The invoker gets a project-level Cloud Run role with at least **`run.workerPools.get`** and **`run.workerPools.update`**. `roles/run.developer` includes both. -Use the Terraform module in `iam.md`, setting `impersonator_service_account_emails` to the GCP identity used by the Temporal Service. Use the module's `invoker_email` output as `--gcp-cloud-run-service-account` when registering the Worker Deployment Version. +Use the Terraform module in `iam.md`, setting `impersonator_service_account_emails` to the GCP identity used by the Temporal Service. Use the module's `invoker_email` output as `--gcp-cloud-run-service-account` when registering the Worker Deployment Version. ## Then diff --git a/references/gcp-cloud-run/setup.md b/references/gcp-cloud-run/setup.md index ed2f72a..7651d90 100644 --- a/references/gcp-cloud-run/setup.md +++ b/references/gcp-cloud-run/setup.md @@ -13,6 +13,8 @@ End-to-end: write a standard Worker, containerize it, push the image, create a W - **Terraform** installed — Temporal ships the IAM setup as a Terraform module. - A Temporal SDK. Supported on Cloud Run: Go, Python, TypeScript, Java, .NET, **Ruby, and Rust**. + + The `temporal` CLI commands in Steps 6 and 7 must inherit authentication from an existing profile or from the process environment. **Never append `--api-key ` or put the key in an inline assignment.** If `TEMPORAL_API_KEY` is not already populated, set it privately in the user's own terminal without putting the value in shell history: ```bash @@ -96,7 +98,7 @@ Two things the Worker must do: Per-SDK code lives in `sdk-.md` in this directory, one file per SDK. Each covers the versioned Worker, connection, image packaging, graceful shutdown, scale-in safety, and observability. The Java, Python, and .NET references also include their logging setup and diagnostic signatures. -**The entrypoint must start the Worker process**, so an instance begins polling as soon as it starts. +**The entrypoint must start the Worker process**, so an instance begins polling as soon as it starts. ## Step 2: Containerize the Worker @@ -186,7 +188,7 @@ Creating the Worker Deployment Version starts its WCI. The WCI validates the con The update completing proves that Temporal reached Cloud Run, not that the container started or the Worker connected. Wait for the expected Task Queue types to appear; that binding is proof the instance started and polled under the registered deployment name and build ID. -The separate UI **Validate Connection** action (Workers → Deployments → select → Actions) only impersonates the invoker and reads the pool. It starts no instance and does not exercise `run.workerPools.update`, so a green manual validation is weaker than a successful registration bootstrap. +The separate UI **Validate Connection** action (Workers → Deployments → select → Actions) only impersonates the invoker and reads the pool. It starts no instance and does not exercise `run.workerPools.update`, so a green manual validation is weaker than a successful registration bootstrap. Use both views when diagnosing the checkpoint: diff --git a/references/gcp-cloud-run/versioning.md b/references/gcp-cloud-run/versioning.md index a673d6a..526989d 100644 --- a/references/gcp-cloud-run/versioning.md +++ b/references/gcp-cloud-run/versioning.md @@ -2,13 +2,13 @@ ## One Worker Pool per Build ID -**The compute configuration names a project, region, and Worker Pool — it does not name a [revision](https://cloud.google.com/run/docs/managing/revisions).** Temporal runs whichever revision the pool happens to serve. That ties a pool to a single build, so **a new build needs a new pool.** Carry the Build ID in the pool name (`my-worker-pool-build-1`) so the mapping stays visible. +**The compute configuration names a project, region, and Worker Pool — it does not name a [revision](https://cloud.google.com/run/docs/managing/revisions).** Temporal runs whichever revision the pool happens to serve. That ties a pool to a single build, so **a new build needs a new pool.** Carry the Build ID in the pool name (`my-worker-pool-build-1`) so the mapping stays visible. Temporal's Cloud Run compute configuration has no revision selector, so **durable isolation requires a separate pool per build.** A pool-level instance split can temporarily hold a particular revision, but that split remains mutable state outside Temporal. ## The hazard: redeploying into a live pool -> Deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run promotes it to every instance by default. The version does not change, but the code behind it does. Deploying replay-unsafe code this way causes non-determinism errors for in-flight Workflows, **including Pinned ones**. +> Deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run promotes it to every instance by default. The version does not change, but the code behind it does. Deploying replay-unsafe code this way causes non-determinism errors for in-flight Workflows, **including Pinned ones**. **A normal `gcloud run worker-pools deploy` takes the mutable path by default.** The durable Temporal-aligned path is a new pool; `--no-promote` is only a same-pool guardrail. @@ -53,7 +53,7 @@ After emergency recovery, return to one pool per Build ID for the next release s 3. Register a new Worker Deployment Version pointing at the new pool, with a build ID matching the new Worker code. 4. Confirm registration bootstrapped the pool: its `lastModifier` shows the invoker and the expected Task Queue types are bound. Treat the separate UI Validate Connection action as a read-only check. → `diagnostics.md`. 5. Set the new version current, or ramp to it. -6. **Leave the old pool in place** while Pinned Workflows still run on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. +6. **Leave the old pool in place** while Pinned Workflows still run on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. Only the invoker's permissions are shared across pools, so a new pool usually needs no IAM change — provided the invoker's `deploy_roles` are project-level, which is the module's default. Check `iam.md` if you scoped them to individual pools instead.