diff --git a/references/gcp-cloud-run/constraints.md b/references/gcp-cloud-run/constraints.md new file mode 100644 index 0000000..82d9aeb --- /dev/null +++ b/references/gcp-cloud-run/constraints.md @@ -0,0 +1,64 @@ +# GCP Cloud Run — execution-model constraints + +Consequences of Cloud Run's execution model: **Temporal resizes a pool of long-lived instances, scaling it to zero when there is no work.** Temporal does *not* invoke your Worker per Task. + +## Worker lifetime is an instance, not an invocation + +Each pool instance runs **standard long-lived Worker code**: it connects, registers Workflows and Activities, and polls the Task Queue for its whole lifetime. There is no handler, no per-Task lifecycle, and **no serverless Worker package**. Use the Cloud Run SDK reference in this directory; some SDKs add optional conveniences, but none are required. + +The WCI controls how many instances run; each instance manages its own polling and Task processing. + +## Timing and Activity limits + +- **No invocation deadline.** Nothing bounds how long a Worker lives except scale-in. +- **No shutdown deadline buffer.** Cloud Run terminates instances through its own scale-in lifecycle. +- **No timeout triple to tune.** Two of its three values do not exist here. +- **Activities are not bounded by an invocation limit.** Their practical interruption boundary is scale-in. +- **Eager Activities are not disabled by the platform.** A pool instance holds its connection for its lifetime; confirm support against the selected SDK. + +Cloud Run sends `SIGTERM` during scale-in and can send `SIGKILL` ten seconds later. Handle `SIGTERM` and configure the SDK's graceful-shutdown timeout below that window. → `sdk-.md`. + +## What bounds an Activity instead: scale-in + +**The WCI decides when to remove an instance from Task Queue activity, not from what any individual instance is doing.** It does not track how long an instance has been running or whether it is mid-Activity, so the instance Cloud Run stops may be one that is still executing work. + +Graceful shutdown lets short work drain but cannot guarantee an Activity will finish. **Use Activity Heartbeats** so interrupted work resumes from its last recorded progress instead of restarting. + +*Symptom of ignoring this:* Activities failing partway and retrying from the beginning, correlated with the pool shrinking. + +## Autoscaling behavior + + + +The WCI combines two mechanisms: + +- **Immediate** — bring up instances when a Task arrives and no Worker is free to take it (a sync match failure). Absorbs bursts without waiting for the evaluation cycle. +- **Periodic** — rate-based re-sizing. The WCI measures how fast Tasks arrive and how fast one Worker processes them, computes the instance count needed, and applies it through the Cloud Run admin API. + +It sizes to a **target utilization of 80% by default** rather than loading every Worker fully, so there is headroom to take new Tasks immediately, and adds instances on top when a backlog exists. + +**Scale-in is deliberately more conservative than scale-out:** it holds capacity while sync match failures are still occurring and applies a cooldown before reducing the pool. With no work, it can scale to zero; the next sync match failure or backlog scales it back up. + +The scaler defaults are **minimum `0`, maximum `30`, initial count `0`, and target utilization `0.8`**. Configure them in the version's Scaling and Lifecycle settings or with the Temporal CLI. The four CLI flags are coupled: omit all four to use the defaults, or provide `--gcp-cloud-run-min-instances`, `--gcp-cloud-run-max-instances`, `--gcp-cloud-run-initial-instances`, and `--gcp-cloud-run-utilization-target` together. A partial group is rejected. + +An initial count and minimum of zero do not suppress registration. The rate-based algorithm temporarily requests at least one instance when the version is registered so its Task Queues can bind, then normal scaling can return the pool to zero. A pool that later stops growing under backlog is either at its configured maximum or at a regional Cloud Run quota. + +## One Worker Pool per Worker Deployment Version + +**The compute configuration names a project, region, and pool — not a revision.** Temporal runs whichever revision the pool serves at the time, which ties a pool to a single build. A new build needs a **new pool**, and the Build ID belongs in the pool name to keep that mapping visible. + +Keep an older version's pool in place while Pinned Workflows are still running on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. + +**The mutable-build hazard:** deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run promotes it to every instance by default. The version does not change, but the code behind it does — causing non-determinism errors for in-flight Workflows, **including Pinned ones**. → `versioning.md`. + +## Do not share a Task Queue with long-lived Workers + +**The pool scales up to cover the Task Queue's full workload even when independently managed Workers are already handling all of it**, so you run and pay for duplicate capacity. + +The WCI sizes the pool from the rate of Tasks arriving on the version's Task Queues, and nothing in that measurement accounts for the long-lived Workers. Sync matching to a long-lived Worker suppresses the *immediate* scale-up, but the periodic re-sizing scales the pool up regardless. Fixing poller counts on the long-lived side does not help — use separate Task Queues. + +## What does *not* follow from this model + +- **Use ordinary Worker code.** Handler, configure-callback, and provider-package patterns do not apply. +- **Scale-to-zero is not per-Task.** An idle pool costs nothing, but a running instance is billed for its lifetime, not per unit of work. +- **A passing Validate Connection exercises only the read permission and starts no instance.** → `diagnostics.md`. diff --git a/references/gcp-cloud-run/diagnostics.md b/references/gcp-cloud-run/diagnostics.md new file mode 100644 index 0000000..5b360c3 --- /dev/null +++ b/references/gcp-cloud-run/diagnostics.md @@ -0,0 +1,130 @@ +# GCP Cloud Run — Diagnostics & troubleshooting + +## Scaling flow (when working correctly) + + + +1. You deploy the Worker image to a Worker Pool at zero instances. +2. You create a Worker Deployment Version pointing at that pool. This starts a WCI Workflow. +3. An instance starts, the Worker polls, and the server **binds the Task Queue** to the version. +4. The WCI monitors that Task Queue; the Matching Service also signals it when a Task arrives with no free Worker. +5. As work arrives the WCI raises the instance count through the Cloud Run admin API; as it drains, lowers it, possibly to zero. + +**Temporal does not invoke your Worker per Task** — it changes how many instances run. Diagnose accordingly: "was it invoked?" is the wrong first question here. + +## Start here: read the pool's annotations + +This is the single highest-value command: + +```bash +gcloud run worker-pools describe \ + --region --project --format=yaml +``` + +Three fields under `metadata.annotations`: + +| Field | What it tells you | +|---|---| +| `run.googleapis.com/manualInstanceCount` | The instance count currently requested. `0` means no Worker is running. | +| `run.googleapis.com/scalingMode` | Should be `manual` — the WCI scales by writing the manual instance count. | +| `serving.knative.dev/lastModifier` | Who last changed the pool. **If Temporal has ever scaled it, this is the invoker service account.** | + +**`lastModifier` splits the whole problem in two:** + +- Still the account you deployed with → **Temporal has never successfully written to the pool.** Work through "pool is not scaling up" below. +- The invoker service account → **Temporal is reaching the pool**; the fault is in the Worker. Skip to "instances running but Tasks not completing." + +## The Worker Pool is not scaling up + +### 1. Validate Connection — and know what it does *not* prove + +Workers → Deployments → select deployment → Actions → **Validate Connection**. For Cloud Run this impersonates the invoker and reads the pool, confirming three things: the compute configuration names a pool that exists, Temporal can impersonate the invoker, and the invoker can read. + +**It starts no instance and does not exercise `run.workerPools.update`.** Version registration is different: its Task Queue bootstrap does update the pool. An invoker with read but not update permission can therefore pass this manual validation, but version registration or a later resize fails. If validation succeeds but registration never changes `lastModifier`, **check the update permission**. + +On failure, check each part of the compute configuration against the pool: + +- **Project, region, pool name.** Temporal addresses the pool as `projects//locations//workerPools/`. **A wrong region reports the pool as not found, identical to a wrong name** — so a "not found" does not tell you which field is wrong. +- **Impersonation.** Temporal's identity needs `roles/iam.serviceAccountTokenCreator` on the invoker. The Terraform module grants this on Cloud; self-hosted grants it to the server's GCP identity. +- **Invoker permissions.** `run.workerPools.get` to read, `run.workerPools.update` to scale. + +→ `iam.md`. + +### 2. Did the registration bootstrap bind the Task Queue? + +```bash +temporal worker deployment describe-version \ + --namespace --deployment-name --build-id --report-task-queue-stats +``` + +The server creates the binding when a Worker running that version connects and polls. **No Task Queues listed means no Worker has polled successfully under this version.** + +Registration is supposed to bootstrap this: the WCI reads the pool, updates its manual instance count to at least one, and Cloud Run starts an instance. An absent binding means that sequence failed or the instance started but did not connect under the expected deployment name and build ID. Inspect the WCI's `ValidateSpec`/registration Activity failure, the pool's `lastModifier` and instance count, and then the pool logs. Do not wait for a first Workflow to repair registration. + +### 3. Is the version current? + +The registration bootstrap does not make the version current. New traffic routes only after the version is current, and a CLI-created version is not current automatically. Verify with `temporal worker deployment describe`. + +**A `set-current-version` run without `--yes` may have done nothing** — it prompts, and non-interactively exits without applying the change, which reads as success. + +### 4. Is the pool at its ceiling? + +If instances are running but the count stops growing while backlog builds: + +- **The maximum defaults to 30.** Raise it in the version's Scaling and Lifecycle settings, or update the existing version from the CLI: + + ```bash + temporal worker deployment update-version-compute-config \ + --namespace \ + --deployment-name \ + --build-id \ + --gcp-cloud-run-min-instances 0 \ + --gcp-cloud-run-max-instances \ + --gcp-cloud-run-initial-instances 0 \ + --gcp-cloud-run-utilization-target 0.8 + ``` + + The four scaler flags must be supplied together, even when changing only the maximum. Choose values appropriate for the version rather than blindly copying this default-shaped example; the initial count must be between the minimum and maximum. +- If the count stalls *below the configured maximum*, check the project's [Cloud Run quotas](https://cloud.google.com/run/quotas) for that region. Cloud Run caps instances and CPU per region regardless of what the WCI requests. + +## Instances are running but Tasks are not completing + +### Read the pool logs + +```bash +gcloud run worker-pools logs read --region --project +``` + +**A scaled-to-zero pool emits no new logs.** `logs read` still returns historical entries; use `logs tail` only while an instance is running. An empty result may simply mean the pool has never started. + +Common errors: + +- **Connection failures** — check `TEMPORAL_ADDRESS` and `TEMPORAL_NAMESPACE` on the pool. Self-hosted: verify network reachability from Cloud Run to the frontend. +- **Missing secrets** — the instance cannot read the API key or TLS material. The **runner** service account needs `roles/secretmanager.secretAccessor` on the secret. That is the account in `spec.template.spec.serviceAccountName`, **not the invoker.** This is the most common consequence of confusing the two. +- **Authentication errors** — key invalid, expired, or without access to the Namespace. +- **`TransportError: … NativeCertsNotFound`** — a Rust-core SDK in a minimal base image with no CA certificates. Install `ca-certificates` in the image. → `setup.md`. + +### Deployment name and build ID + +Instances start and poll but no Task is ever processed → the name or build ID in the code does not match the version. The Worker polls under a version the WCI does not manage, **so its polls never satisfy the Tasks the WCI is scaling for.** + +The Cloud Run signature is a running pool, healthy-looking logs, and no Workflow progress. + +## Activities interrupted mid-execution + +Activities failing partway and retrying from the beginning, correlated with the pool shrinking, means **scale-in is stopping instances that are still working.** The WCI does not track whether the instance Cloud Run stops is mid-Activity. + +This is expected behavior, not a misconfiguration. Confirm the Worker handles `SIGTERM` and has a non-zero graceful-shutdown timeout below Cloud Run's ten-second termination window; this lets short work drain. Long-running work still needs **Activity Heartbeats** so a retry resumes from its last recorded progress. → `constraints.md`, `sdk-.md`. + +## Rule out a GCP-side cause + +If every check passes, the cause may be in Cloud Run rather than your configuration: + +- [Cloud Run known issues](https://cloud.google.com/run/docs/known-issues) — includes issues affecting how long pool operations take. +- [Google Cloud Service Health](https://status.cloud.google.com/) — active incidents by product and region. + +The usual fix is to wait it out or move region. **Moving region means creating a new pool and updating the compute configuration**, since Temporal addresses a pool by project, region, and name. + +## Never create or manage the WCI + +Temporal creates one WCI per Worker Deployment Version with a compute provider, and a running WCI is not evidence that scaling works—it continues-as-new while its Activities fail. Read its history for Activity failures, and prefer the pool's `lastModifier` annotation as the cheapest proof of a successful write. Do not enumerate Cloud Run resources across regions to reverse-engineer state. diff --git a/references/gcp-cloud-run/iam.md b/references/gcp-cloud-run/iam.md new file mode 100644 index 0000000..a6d1de2 --- /dev/null +++ b/references/gcp-cloud-run/iam.md @@ -0,0 +1,118 @@ +# GCP Cloud Run — IAM & permissions + +Cloud Run uses three identities: + +| Identity | Purpose | +|---|---| +| **Operator** | The credentials that run `gcloud` and Terraform commands. | +| **Runner service account** | The identity attached to pool instances with `--service-account`. | +| **Invoker service account** | The identity Temporal impersonates to read and update the pool's instance count. | + +Temporal reaches the invoker through `roles/iam.serviceAccountTokenCreator`. The Terraform module described below creates the invoker and applies its grants. + +**The two service accounts are not interchangeable**, and confusing them is the single most likely IAM mistake here. The runner runs the pool and never scales it; the invoker scales the pool and never runs it. + +## Runner service account + +The runtime identity the pool's instances use to reach other Google Cloud services. Set in `setup.md` Step 4 with `gcloud run worker-pools deploy --service-account`. It may be an account that already exists; a dedicated one is preferred. + +**It needs no baseline role to run the Worker.** Cloud Run collects `stdout` and `stderr` into Cloud Logging through its own infrastructure, and the Cloud Run *service agent* — not the runner — pulls the container image. Grant only what your code actually reaches: + +- `roles/secretmanager.secretAccessor` on each secret you mount, including the Temporal API key. +- `roles/logging.logWriter` **only if** the Worker writes through the Cloud Logging API rather than stdout/stderr. +- Whatever else your Workflows and Activities call. + +## Invoker service account + +The identity Temporal Cloud impersonates to read and scale the pool. Two grants make it work: + +- Temporal's identity receives **`roles/iam.serviceAccountTokenCreator`** on the invoker, so it can impersonate it. +- The invoker receives a project-level Cloud Run role with at least **`run.workerPools.get`** (read) and **`run.workerPools.update`** (scale). `roles/run.developer` includes both. + +The invoker also needs **`roles/iam.serviceAccountUser` on the runner service account**, which Cloud Run requires in order to attach that identity when it scales the pool. The Terraform module applies this. + +### The read/update split is a real trap + +`run.workerPools.get` alone is enough for the UI's **Validate Connection** action to pass. That manual action never exercises `run.workerPools.update`. Version registration does: the WCI reads the pool and then updates its manual instance count to bootstrap Task Queue registration. An invoker that can read but not update therefore passes manual validation but fails version registration or a later resize. + +Verify the registration bootstrap and `lastModifier` rather than trusting the separate green Validate Connection result. → `diagnostics.md`. + +## The Terraform module + +Temporal publishes [`serverless-workers/gcp/cloud-run`](https://github.com/temporalio/terraform-modules/tree/main/modules/serverless-workers/gcp/cloud-run), which creates the invoker service account and applies the grants. + +Start from the template under **Workers → Create Worker Deployment → Access** because Temporal Cloud fills in the account-specific `impersonator_service_account_emails`. Its shape is: + +```hcl +module "serverless-worker-cloud-run" { + source = "github.com/temporalio/terraform-modules//modules/serverless-workers/gcp/cloud-run" + + project_id = "" + invoker_account_id = "temporal-worker-pool-invoker" + + impersonator_service_account_emails = [ + "", + ] + + runner_service_account_email = "temporal-worker-pool-runner@.iam.gserviceaccount.com" +} +``` + +| Variable | Required | Description | +|---|---|---| +| `project_id` | Yes | Project hosting the pool and the invoker. | +| `invoker_account_id` | Yes | Name for the invoker the module creates; email becomes `@.iam.gserviceaccount.com`. The template supplies one. | +| `impersonator_service_account_emails` | Yes | Temporal Cloud's service accounts, granted `serviceAccountTokenCreator` on the invoker. **From the UI template.** For self-hosted, the GCP identity the server runs as. | +| `runner_service_account_email` | Yes | The runner from `setup.md` Step 4. The module grants the invoker `roles/iam.serviceAccountUser` on it. | +| `invoker_display_name` | No | Defaults to `Temporal Serverless Worker Pool Invoker`. | +| `deploy_roles` | No | Project-level Cloud Run roles for the invoker. Defaults to `roles/run.developer`. A substitute must include `run.workerPools.get` and `run.workerPools.update`. | + +```bash +terraform init +terraform apply +``` + +Use the **`invoker_email`** output as `--gcp-cloud-run-service-account` when registering the version. + +### Treat the module as shared, pre-existing infrastructure + +One invoker can serve several pools, so before creating a second one, look for an existing account and consider reusing it. **Do not `terraform destroy` state you did not create.** The module's default `invoker_account_id` may already be owned by an earlier deployment in the project. + +## Operator GCP permissions + +The identity running the `gcloud`/Terraform commands needs, at minimum: + +| Step | Operator needs | +|---|---| +| Submit the image build (`setup.md` Step 3) | `cloudbuild.builds.create`; permission to use the selected build service account when one is specified | +| Push the built image | The Cloud Build execution service account needs Artifact Registry Writer on the repository when it is user-specified, cross-project, or lacks the same-project default access | +| Create the Worker Pool (Step 4) | `run.workerPools.create`/`update`, and `iam.serviceAccounts.actAs` on the **runner** to attach it | +| Create secrets | Secret Manager admin on the secrets used | +| Apply the Terraform module (Step 5) | Service-account creation plus IAM policy binding on the project and on the runner | +| Read pool state and logs (verify, diagnose) | `run.workerPools.get`, Cloud Logging read | + +The operator needs `iam.serviceAccounts.actAs` on the runner to attach it to the pool. Without it, pool creation is denied even when the operator otherwise has Cloud Run permissions. + +### Preflight + +Run before anything that creates or modifies GCP resources. Confirm the required APIs appear in the enabled-service output, then inspect the names the deployment intends to use: + +```bash +gcloud auth list # which identity +gcloud config get-value project # which project +gcloud services list --enabled --project --format='value(config.name)' +gcloud run worker-pools list --region >/dev/null && echo "cloud run: ok" +gcloud iam service-accounts list >/dev/null && echo "iam read: ok" +gcloud artifacts repositories describe --location --project +gcloud iam service-accounts describe --project +gcloud secrets describe --project +terraform version +``` + +Required services: `run.googleapis.com`, `artifactregistry.googleapis.com`, `cloudbuild.googleapis.com`, `secretmanager.googleapis.com`, `iam.googleapis.com`, `iamcredentials.googleapis.com`, and `cloudresourcemanager.googleapis.com`. A `describe` returning Not Found is acceptable for a clean project; record that the named resource will be created and include it in the approval list. A permission error is not the same as absence—stop and resolve access before creating anything. + +For a same-project build using Cloud Build's default service account, Artifact Registry access is normally provided automatically. If the build uses a user-specified service account, the repository is in another project, or an organization policy removed the default grant, inspect that account and grant `roles/artifactregistry.writer` on this repository before submitting the build. + +**Classify an authentication failure before acting on it.** An absent or expired credential (`gcloud auth login`, or `gcloud auth application-default login` for Terraform) is recoverable in a minute; an identity that resolves but is denied a specific action is a real permissions problem. Never collect credentials in conversation and never ask the user to paste a service account key — Google recommends against long-lived keys outright. + +**Confirm the project explicitly before creating anything.** `gcloud config get-value project` is ambient state that is easy to be wrong about. Name the project in the approval list and verify it rather than trusting the default. diff --git a/references/gcp-cloud-run/observability.md b/references/gcp-cloud-run/observability.md new file mode 100644 index 0000000..968cdb0 --- /dev/null +++ b/references/gcp-cloud-run/observability.md @@ -0,0 +1,30 @@ +# GCP Cloud Run — observability + +## There is nothing serverless-specific to configure + +**A Cloud Run Serverless Worker emits the same traces and metrics as a Worker anywhere else.** It is an ordinary long-lived Worker, so the SDK's normal metrics and OpenTelemetry tracing setup applies unchanged, and each SDK's general observability guide is the right reference. + +Do not add provider-specific helper layers, collector environment variables, or invocation-deadline flush logic. Export telemetry as you would from any long-lived container. + +Some SDKs add optional Cloud Run conveniences, such as OpenTelemetry helpers. **They are optional**, and where they exist they are documented in that SDK's Cloud Run guide. + +## Logs + +Cloud Run collects `stdout` and `stderr` into [Cloud Logging](https://cloud.google.com/run/docs/logging) through its own infrastructure. **The runner service account needs no grant for this** — `roles/logging.logWriter` is required only if the Worker writes through the Cloud Logging API instead. → `iam.md`. + +Read a pool's logs: + +```bash +gcloud run worker-pools logs read --region --project +``` + +**A scaled-to-zero pool emits no new logs.** `logs read` returns historical entries; `logs tail` shows new output only while an instance is running. + +## What to watch that is specific to this provider + +Standard Worker metrics tell you about the Worker. Two Cloud Run-specific signals tell you about the *scaling*, and neither comes from the SDK: + +- **`run.googleapis.com/manualInstanceCount`** on the pool — what the WCI has asked for. Persistently `0` while a backlog exists is the headline symptom of a scaling failure. +- **`serving.knative.dev/lastModifier`** on the pool — whether Temporal has ever successfully written to it. The cheapest proof that impersonation and the update permission both work. + +Both are read with `gcloud run worker-pools describe … --format=yaml`. → `diagnostics.md`. diff --git a/references/gcp-cloud-run/self-hosted.md b/references/gcp-cloud-run/self-hosted.md new file mode 100644 index 0000000..b4e02d5 --- /dev/null +++ b/references/gcp-cloud-run/self-hosted.md @@ -0,0 +1,71 @@ +# GCP Cloud Run — self-hosted Temporal Service setup + +Serverless Workers require **Temporal Service v1.31.0 or later**. Complete this page before following `setup.md`. + +## 1. Cloud Run instances must reach the Temporal Service + +The frontend must be reachable **from the Worker Pool instances**. If the Service runs on a private network, that likely means [Direct VPC egress](https://cloud.google.com/run/docs/configuring/vpc-direct-vpc) or a [Serverless VPC Access connector](https://cloud.google.com/run/docs/configuring/vpc-connectors). + +Note the direction: instances dial *out* to Temporal. Nothing needs to reach *into* Cloud Run — Temporal drives the pool through the Cloud Run admin API, not by connecting to your instances. + +## 2. Enable the Worker Controller Instance + +The WCI is **disabled by default** and enabled through [Temporal Service dynamic configuration](https://docs.temporal.io/references/dynamic-configuration): + +```yaml +workercontroller.enabled: + - value: true + +workercontroller.compute_providers.enabled: + - value: + - gcp-cloud-run + +workercontroller.scaling_algorithms.enabled: + - value: + - rate-based +``` + +**Cloud Run requires the `rate-based` algorithm.** Because a pool is a set of long-lived instances, the WCI resizes it from arrival and backlog rates. Pairing `no-sync` with `gcp-cloud-run` is rejected. + +To enable per Namespace instead of globally: + +```yaml +workercontroller.enabled: + - value: true + constraints: + namespace: 'your-namespace' +``` + +The Service watches the file and applies updates **without a restart**. + +Two optional global keys cover multi-hop impersonation: + +| Key | Purpose | +|---|---| +| `workercontroller.compute_providers.gcp.intermediary_service_accounts` | Service accounts to impersonate in sequence before the invoker. Unset means the server impersonates the invoker directly. | +| `workercontroller.compute_providers.gcp.first_delegate_as_base` | When `true`, the first entry is the identity the server's ambient credentials impersonate directly and the rest are passed as token-creator delegates. Defaults to `false`, passing the whole chain as delegates. | + + + +## 3. Give the Temporal Service a GCP identity + +The server impersonates the invoker service account, so it must first run as a GCP identity permitted to do so. + +**On GCP (GCE, GKE):** the attached service account is used automatically through [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). No extra credential configuration — you grant *that* account impersonation rights in step 4. + +**Outside GCP:** use [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) and point `GOOGLE_APPLICATION_CREDENTIALS` at the credential configuration file it produces. That variable also accepts a service account key file, but **Google recommends against long-lived keys** — prefer federation, and never ask a user to paste a key. + +## 4. Create the invoker service account + +The Service scales the pool as an **invoker** service account, which only reads and scales the pool. The identity instances *run as* is the separate **runner** service account set on the pool in `setup.md`. → `iam.md` for the full distinction. + +Two grants: + +- The GCP identity the Service runs as (step 3) gets **`roles/iam.serviceAccountTokenCreator`** on the invoker. +- The invoker gets a project-level Cloud Run role with at least **`run.workerPools.get`** and **`run.workerPools.update`**. `roles/run.developer` includes both. + +Use the Terraform module in `iam.md`, setting `impersonator_service_account_emails` to the GCP identity used by the Temporal Service. Use the module's `invoker_email` output as `--gcp-cloud-run-service-account` when registering the Worker Deployment Version. + +## Then + +Follow `setup.md` from Step 1. The read/update permission trap in `iam.md` applies identically, and `diagnostics.md`'s `lastModifier` check is still the fastest way to confirm the server can actually scale the pool. diff --git a/references/gcp-cloud-run/setup.md b/references/gcp-cloud-run/setup.md new file mode 100644 index 0000000..7651d90 --- /dev/null +++ b/references/gcp-cloud-run/setup.md @@ -0,0 +1,262 @@ +# GCP Cloud Run — Setup (happy path) + +End-to-end: write a standard Worker, containerize it, push the image, create a Worker Pool at zero instances, grant Temporal permission to scale it, register a Worker Deployment Version, set it current, verify. For the two service accounts and the Terraform module, see `iam.md`. For what the execution model does and does not bound, see `constraints.md`. For new builds and rollback, see `versioning.md`. If it doesn't work, see `diagnostics.md`. + +## Prerequisites + +- **Cloud Run support is Pre-release and access-gated.** The user creates a support ticket or contacts their account team. Confirm this before anything else. +- A Temporal Cloud account with a **GCP-hosted Namespace**, or self-hosted Temporal Service v1.31.0+. The Namespace must be hosted on GCP; its region need not match the pool's. +- For self-hosted, complete `self-hosted.md` first. +- Every Workflow must declare a versioning behavior, or the Worker must set a default. +- A GCP project with billing enabled and permission to enable service APIs and create Worker Pools, Artifact Registry repositories, Cloud Build jobs, service accounts, and Secret Manager secrets. +- `gcloud` CLI installed and authenticated. The Google Cloud console or Terraform also work. +- **Terraform** installed — Temporal ships the IAM setup as a Terraform module. +- A Temporal SDK. Supported on Cloud Run: Go, Python, TypeScript, Java, .NET, **Ruby, and Rust**. + + + +The `temporal` CLI commands in Steps 6 and 7 must inherit authentication from an existing profile or from the process environment. **Never append `--api-key ` or put the key in an inline assignment.** If `TEMPORAL_API_KEY` is not already populated, set it privately in the user's own terminal without putting the value in shell history: + +```bash +export TEMPORAL_ADDRESS="
:7233" +export TEMPORAL_NAMESPACE="" +printf 'Temporal API key: ' >&2 +IFS= read -r -s TEMPORAL_API_KEY +printf '\n' >&2 +export TEMPORAL_API_KEY +``` + +Do not run the secret-reading commands through an agent shell, ask the user to paste the key into conversation, or inspect the resulting variable. A configured Temporal CLI profile is equally valid and avoids a session environment variable. + +## Prepare a clean GCP project + +After the resource list is approved, create only what is missing. First enable every API used by the commands below: + +```bash +gcloud services enable \ + run.googleapis.com \ + artifactregistry.googleapis.com \ + cloudbuild.googleapis.com \ + secretmanager.googleapis.com \ + iam.googleapis.com \ + iamcredentials.googleapis.com \ + cloudresourcemanager.googleapis.com \ + --project +``` + +Create a regional Docker repository for the Worker image: + +```bash +gcloud artifacts repositories create \ + --repository-format docker \ + --location \ + --project \ + --description "Temporal Serverless Worker images" +``` + +Create the runner service account that pool instances use: + +```bash +gcloud iam service-accounts create \ + --display-name "Temporal Cloud Run Worker runner" \ + --project +``` + +Create the Secret Manager secret before deploying the pool: + +```bash +gcloud secrets create \ + --replication-policy automatic \ + --project + +gcloud secrets versions add \ + --data-file=- \ + --project +``` + +Run the `versions add` command only in the user's own terminal, provide the Temporal API key on standard input, and then send EOF. Never ask for the value in conversation or run it through an agent shell where input or output may be captured. + +Grant only the runner access to that secret: + +```bash +gcloud secrets add-iam-policy-binding \ + --member="serviceAccount:@.iam.gserviceaccount.com" \ + --role roles/secretmanager.secretAccessor \ + --project +``` + +Before each create, use the corresponding `describe` command from `iam.md` to avoid colliding with shared resources. The invoker service account is created later by Temporal's Terraform module; do not substitute it for the runner. + +## Step 1: Write Worker code + +**There is no Cloud Run Worker package.** Write an ordinary long-lived Worker — same client, same `Worker`/`WorkerFactory`, same registration — and add Worker Versioning, which Serverless Workers require. Use `sdk-.md` in this directory. + +Two things the Worker must do: + +1. **Declare its Worker Deployment Version and enable versioning**, with a deployment name and build ID that exactly match the version you register in Step 6. +2. **Read its configuration from the environment** — address, Namespace, Task Queue, credentials — so one image can run against any Namespace. The pool supplies these via `--set-env-vars` and `--set-secrets`. + +Per-SDK code lives in `sdk-.md` in this directory, one file per SDK. Each covers the versioned Worker, connection, image packaging, graceful shutdown, scale-in safety, and observability. The Java, Python, and .NET references also include their logging setup and diagnostic signatures. + +**The entrypoint must start the Worker process**, so an instance begins polling as soon as it starts. + +## Step 2: Containerize the Worker + +Follow the selected `sdk-.md` reference for the container image, runtime, entrypoint, certificate requirements, and memory settings. + +## Step 3: Build and push the image + +```bash +gcloud builds submit \ + --tag -docker.pkg.dev///my-temporal-worker:build-1 \ + --project +``` + +Tag the image with the build ID. It keeps image, pool, and Worker Deployment Version aligned, which matters because the compute configuration cannot pin a revision (→ `constraints.md`). + +The happy path deliberately uses Cloud Build's global endpoint. Supplying `--region` can require additional regional build and staging-bucket setup, depending on the project's Cloud Build bucket policy. If regional builds are required for a private pool or data-residency policy, pre-create or select the regional source/log buckets, grant the build identity access, and then add `--region `. + +## Step 4: Create the Worker Pool + +**Create one pool per Worker Deployment Version, initially at zero instances.** Registering the version starts the WCI, which temporarily raises the pool to the scaler's planned count—at least one instance—to register its Task Queues. The version does not need to be current for this bootstrap. + +```bash +gcloud run worker-pools deploy my-temporal-worker-pool-build-1 \ + --image -docker.pkg.dev///my-temporal-worker:build-1 \ + --region \ + --project \ + --service-account \ + --instances 0 \ + --set-env-vars TEMPORAL_ADDRESS=
:7233,TEMPORAL_NAMESPACE=,TEMPORAL_TASK_QUEUE=my-task-queue \ + --set-secrets TEMPORAL_API_KEY=:latest +``` + +| Parameter | Description | +|---|---| +| `--image` | The image pushed in Step 3. | +| `--service-account` | The **runner** service account instances run as. **Not** the invoker Temporal impersonates. → `iam.md`. | +| `--instances` | Set to `0`; the WCI takes ownership of the count when the Worker Deployment Version is registered. | +| `--set-env-vars` | Non-secret configuration. | +| `--set-secrets` | Maps a Secret Manager secret to an env var — use it for `TEMPORAL_API_KEY` or TLS material. | + +**Put the API key in Secret Manager from the start.** Do not introduce a plaintext environment-variable deployment step. + +## Step 5: Grant Temporal permission to scale the pool + +Cloud Run has no invocation grant. Temporal **impersonates an invoker service account** and drives the Cloud Run admin API. Create it with Temporal's Terraform module — the Cloud UI supplies a filled-in template under **Workers → Create Worker Deployment → Access**. → `iam.md` for the module, its variables, and the two-service-account distinction. + +Terraform's `invoker_email` output is what Step 6 needs. + +## Step 6: Register the Worker Deployment Version + +```bash +temporal worker deployment create --namespace --name my-app + +temporal worker deployment create-version \ + --namespace \ + --deployment-name my-app \ + --build-id build-1 \ + --gcp-cloud-run-project \ + --gcp-cloud-run-region \ + --gcp-cloud-run-worker-pool my-temporal-worker-pool-build-1 \ + --gcp-cloud-run-service-account \ + --gcp-cloud-run-min-instances 0 \ + --gcp-cloud-run-max-instances 30 \ + --gcp-cloud-run-initial-instances 0 \ + --gcp-cloud-run-utilization-target 0.8 +``` + +| Flag | Description | +|---|---| +| `--deployment-name` / `--build-id` | Must match the Worker code exactly. | +| `--gcp-cloud-run-project` | Project containing the pool. | +| `--gcp-cloud-run-region` | Pool region. | +| `--gcp-cloud-run-worker-pool` | Pool name from Step 4. | +| `--gcp-cloud-run-service-account` | The **invoker** — Terraform's `invoker_email`. | +| `--gcp-cloud-run-min-instances` | Floor the scaler maintains; `0` allows scale-to-zero. | +| `--gcp-cloud-run-max-instances` | Ceiling the scaler may request; defaults to `30`. | +| `--gcp-cloud-run-initial-instances` | Initial planned count; must be between min and max. | +| `--gcp-cloud-run-utilization-target` | Target average utilization in `(0, 1]`; defaults to `0.8`. | + +The four scaler flags are a coupled group: **either omit all four and accept the defaults (`0`, `30`, `0`, `0.8`), or provide all four together.** Supplying only one—even only a higher maximum—fails CLI validation. + +Through the UI, the version is set current automatically; through the CLI it is a separate step. + +### Checkpoint: distinguish registration from Validate Connection + +Creating the Worker Deployment Version starts its WCI. The WCI validates the configuration by reading the Worker Pool, then asks the rate-based algorithm for its Task Queue registration action. For a new Cloud Run version, that action updates the pool's manual instance count to at least one. This special registration floor applies even when `--gcp-cloud-run-min-instances` and `--gcp-cloud-run-initial-instances` are both `0`; those values govern the scaler's normal operating range and initial plan, not whether it can bootstrap Task Queue registration. This exercises both `run.workerPools.get` and the `run.workerPools.update` path used by real scaling. + +The update completing proves that Temporal reached Cloud Run, not that the container started or the Worker connected. Wait for the expected Task Queue types to appear; that binding is proof the instance started and polled under the registered deployment name and build ID. + +The separate UI **Validate Connection** action (Workers → Deployments → select → Actions) only impersonates the invoker and reads the pool. It starts no instance and does not exercise `run.workerPools.update`, so a green manual validation is weaker than a successful registration bootstrap. + +Use both views when diagnosing the checkpoint: + +```bash +# has any Worker ever polled under this version? +temporal worker deployment describe-version \ + --namespace --deployment-name my-app --build-id build-1 --report-task-queue-stats + +# has Temporal ever actually written to the pool? +gcloud run worker-pools describe my-temporal-worker-pool-build-1 \ + --region --project --format=yaml +``` + +In the pool's `metadata.annotations`, `serving.knative.dev/lastModifier` becoming the invoker service account is the real proof that scaling works. → `diagnostics.md`. + +## Step 7: Set the version current + +```bash +temporal worker deployment set-current-version \ + --namespace --deployment-name my-app --build-id build-1 --yes +``` + +Without this, new traffic does not route to the version. The registration instance may already have started and bound the Task Queue, but that bootstrap does not make the version current. The command prompts for confirmation; **run non-interactively without `--yes` it exits having changed nothing**, which reads as success. Read the state back with `temporal worker deployment describe`. + +## Step 8: Verify + +```bash +temporal workflow start \ + --namespace --task-queue my-task-queue \ + --type MyWorkflow --input '"Hello, serverless!"' +``` + +Tasks arriving with no active pollers cause the WCI to raise the instance count; Cloud Run starts an instance, the Worker connects and processes the Task. + +Confirm from two independent signals: + +- **Temporal** — Task completions in the Workflow's event history. +- **Cloud Run** — pool logs showing Worker startup and Task processing: + ```bash + gcloud run worker-pools logs read my-temporal-worker-pool-build-1 \ + --region --project + ``` + **A scaled-to-zero pool emits no new logs.** Use `logs read` for historical entries and `logs tail` while an instance is running. + +## Teardown + +Record what you create as you go: pool name, image tag and Artifact Registry repository, runner and invoker service accounts, the Terraform state, secrets, deployment name and build ID, project and region. + +Scale the pool to zero before deleting the version so its pollers stop without destroying the pool prematurely. + +1. Unset the current version — a Current version cannot be deleted: + ```bash + temporal worker deployment set-current-version \ + --namespace --deployment-name my-app --unversioned --yes + ``` +2. Scale the pool to zero, which ends polling: + ```bash + gcloud run worker-pools update --instances 0 --region --project + ``` +3. Wait for drainage, then delete the version, then the deployment: + ```bash + temporal worker deployment describe-version --namespace --deployment-name my-app --build-id build-1 + temporal worker deployment delete-version --namespace --deployment-name my-app --build-id build-1 + temporal worker deployment delete --namespace --name my-app + ``` +4. Delete the Worker Pool: + ```bash + gcloud run worker-pools delete --region --project + ``` +5. `terraform destroy` the IAM module — **only if this deployment created it.** One invoker service account can serve several pools, so a shared one may still be in use. → `iam.md`. +6. Delete the container image from Artifact Registry, and any Secret Manager secrets created for this deployment. Ask before revoking a Temporal Cloud API key: it is account-scoped, not deployment-scoped. diff --git a/references/gcp-cloud-run/versioning.md b/references/gcp-cloud-run/versioning.md new file mode 100644 index 0000000..526989d --- /dev/null +++ b/references/gcp-cloud-run/versioning.md @@ -0,0 +1,68 @@ +# GCP Cloud Run — versioning, updates, and rollback + +## One Worker Pool per Build ID + +**The compute configuration names a project, region, and Worker Pool — it does not name a [revision](https://cloud.google.com/run/docs/managing/revisions).** Temporal runs whichever revision the pool happens to serve. That ties a pool to a single build, so **a new build needs a new pool.** Carry the Build ID in the pool name (`my-worker-pool-build-1`) so the mapping stays visible. + +Temporal's Cloud Run compute configuration has no revision selector, so **durable isolation requires a separate pool per build.** A pool-level instance split can temporarily hold a particular revision, but that split remains mutable state outside Temporal. + +## The hazard: redeploying into a live pool + +> Deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run promotes it to every instance by default. The version does not change, but the code behind it does. Deploying replay-unsafe code this way causes non-determinism errors for in-flight Workflows, **including Pinned ones**. + +**A normal `gcloud run worker-pools deploy` takes the mutable path by default.** The durable Temporal-aligned path is a new pool; `--no-promote` is only a same-pool guardrail. + +`Pinned` does not protect you. Pinning routes Workflows to a *version*; it cannot pin the code behind a pool whose revision moved underneath it. + +## Guardrail and recovery when a pool is reused + +Pool-per-build remains the production rule because Temporal identifies a Worker Pool, not one of its revisions. If an exceptional workflow must deploy a new revision into a pool that still serves a live version, `--no-promote` prevents the new revision from receiving the pool's instances automatically: + +```bash +gcloud run worker-pools deploy \ + --image -docker.pkg.dev///: \ + --region \ + --project \ + --no-promote +``` + +This is a guardrail, not immutable versioning: the pool's revision split remains mutable state outside Temporal. If a new revision was already promoted accidentally, send all instances back to the known-good revision explicitly: + +```bash +gcloud run worker-pools update-instance-split \ + --to-revisions==100 \ + --region \ + --project +``` + +`--to-latest` is **not** that rollback: it assigns instances to the current and future `LATEST` revision. Use it only when deliberately removing the sticky `--no-promote` behavior and restoring automatic promotion of future revisions: + +```bash +gcloud run worker-pools update-instance-split \ + --to-latest \ + --region \ + --project +``` + +After emergency recovery, return to one pool per Build ID for the next release so Temporal version routing and deployed code cannot drift independently. + +## Rolling out a new build + +1. Build and push a new image tagged with the new build ID. +2. **Create a new Worker Pool** at zero instances for that build ID, with the same runner service account. +3. Register a new Worker Deployment Version pointing at the new pool, with a build ID matching the new Worker code. +4. Confirm registration bootstrapped the pool: its `lastModifier` shows the invoker and the expected Task Queue types are bound. Treat the separate UI Validate Connection action as a read-only check. → `diagnostics.md`. +5. Set the new version current, or ramp to it. +6. **Leave the old pool in place** while Pinned Workflows still run on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. + +Only the invoker's permissions are shared across pools, so a new pool usually needs no IAM change — provided the invoker's `deploy_roles` are project-level, which is the module's default. Check `iam.md` if you scoped them to individual pools instead. + +## Rollback + +Set the previous version current again. Its pool is still there (step 6 above), and its WCI scales it back up on the next Task. Nothing needs rebuilding, and no image or revision has to be reverted — which is the payoff for keeping one pool per build. + +Rolling back through Temporal is therefore straightforward if you followed the pool-per-build discipline. If you redeployed into the same pool, setting the previous Temporal version current is not enough because both versions still address mutable pool state. Restore the known-good Cloud Run revision split as described above, provided that revision still exists; if it was deleted, there is nothing left to route back to and the old image must be redeployed deliberately. + +## Cost of the discipline + +A pool per build means pools accumulate. They cost nothing while at zero instances, but they are real resources with real names, and stale ones make the project harder to reason about. Delete a pool once its version is deleted and no Pinned Workflow can route to it — that ordering is in `setup.md`'s teardown section.