refactor: convert AWS and GCP Terraform stacks into reusable modules …

[yassin-berriai]

…with examples/default entry point

• Remove provider blocks from both AWS and GCP stack roots so the modules can be consumed with count, for_each, depends_on, assumed-role or aliased providers; patterns that are forbidden when a module owns its own provider configuration
• Add examples/default/ thin-root wrappers for both stacks that wire the provider (AWS) / providers (google + google-beta) and call the module with a curated variable surface, preserving the one-command deploy experience
• Move terraform.tfvars.example files into examples/default/ alongside the new roots; update example comments to reflect the curated variable surface
• Thread local.tags (containing litellm:stack, managed-by, and var.tags) explicitly onto every taggable AWS resource since the module no longer controls the provider's default_tags; GCP resource labels already flow through the module's labels input
• Add examples/default/variables.tf and outputs.tf for both stacks, exposing the most-used knobs and re-exporting all module outputs
• Commit provider lock files for both examples so terraform init is reproducible without a network fetch
• Update top-level and per-stack READMEs to document the module-first design, the for_each multi-tenant pattern, and the examples/default/ quick-start path

Review follow-ups (Greptile)

• P2 — redundant tag: removed the managed-by tag from the AWS example providers.tf default_tags (the module already stamps it via local.tags); reserved that block for pure org-wide tags.
• P2 — for_each provider limitation (GCP): documented that the module declares no configuration_aliases, so a for_each over it runs every instance against the same project/region/credentials; noted in the GCP README and the example main.tf.
• P1 — state migration: Greptile suggested a terraform state mv upgrade guide (and flagged that the "No changes" assertion wouldn't hold once the managed SSL cert is renamed). These stacks have never been published/tagged, so there's no existing deployment on the old root-module addresses to migrate from; the upgrade path doesn't exist. Rather than document a hypothetical (and partly inaccurate) migration, the guides were dropped from both READMEs and the docs PR.

Subsequent additions

Two follow-on commits land matching proxy_config mount + OpenTelemetry v2 work on both stacks; they sit on top of the refactor so the module-first layout above is what callers see.

feat(terraform/aws): mount proxy_config from S3 and wire OpenTelemetry v2 (ab2c928) drops the inline LITELLM_PROXY_CONFIG_B64 env var, uploads the YAML to S3 at config/litellm-config.yaml, and has the gateway and backend entrypoints pull it to /tmp/litellm-config.yaml via boto3 before exec'ing uvicorn. The S3 object etag is wired into the task definition so a config-only edit forces a new task-def revision and a rolling redeploy.

feat(terraform/gcp): mount proxy_config from GCS and wire OpenTelemetry v2 (c1492a5) does the same on the GCP side but uses a true Cloud Run v2 gcsfuse volume mount instead of an entrypoint download. The YAML is uploaded to a dedicated GCS bucket and mounted read-only into both the gateway and backend at /etc/litellm; CONFIG_FILE_PATH points at the mount path. An md5 of the YAML rides along as PROXY_CONFIG_HASH so a config-only edit forces a new Cloud Run revision (gcsfuse only surfaces a new object on container restart, so without the hash an updated proxy_config would sit in the bucket unread). The config bucket is separate from the data-plane bucket so the runtime SA can hold objectViewer here while keeping objectAdmin on the data-plane bucket.

AWS / GCP consistency pass

refactor(terraform): make AWS and GCP stacks behave identically (9c04aa1) brings both modules to the same surface and runtime shape so swapping clouds (or reading either README) is symmetric.

Labels and tags. GCP previously stamped var.labels onto only the two GCS buckets, leaving Cloud Run, Cloud SQL, Memorystore, Secret Manager, and the LB resources unlabeled. The module now computes local.labels (litellm-stack + managed-by + var.labels, mirroring AWS's local.tags) and threads it onto every label-supporting resource: Cloud Run services and the migrations job, Cloud SQL writer and reader (via user_labels), Memorystore, every Secret Manager entry, both GCS buckets, the LB global address, and the http/https forwarding rules. GCP uses litellm-stack instead of AWS's litellm:stack because GCP label keys forbid colons.

OTel is opt-in on both stacks. AWS already gated everything on otel_endpoint; GCP was stamping LITELLM_OTEL_V2=true into the shared env unconditionally. Both stacks now do the same thing: leave otel_endpoint empty and nothing OTel-related lands in the container env; set it and gateway and backend get LITELLM_OTEL_V2=true plus OTEL_EXPORTER, OTEL_ENDPOINT, OTEL_ENVIRONMENT_NAME, OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, and a per-component OTEL_SERVICE_NAME (${tenant}-litellm-${env}-{gateway,backend}). AWS gains the richer GCP surface: otel_environment_name (defaults to var.env), otel_capture_message_content (defaults to no_content), and *_extra_env override filtering so a caller-set OTEL_* key wins over the default for that service. var.otel_service_name on AWS is gone, replaced by the per-component naming.

uvicorn workers. GCP gains gateway_num_workers, matching AWS; threads into the gateway args as --workers ${var.gateway_num_workers}.

Docs reflect the parity: each README's OTel section, the GCP "Using as a module" Labels paragraph, and a new feature-parity table in the top-level README (and in the docs PR) that lays out the AWS/GCP input mapping side by side.

AWS 1-click end-to-end verification

fix(terraform/aws): expose skip_final_snapshot through the default example (1edeba1) ships a small follow-up surfaced by an end-to-end test of the AWS 1-click against a fresh account.

The example wrapper already exposed s3_force_destroy so a trial stack could be torn down without pre-emptying the bucket, but the matching Aurora knob (skip_final_snapshot) was buried inside the module call, so destroying a trial stack always left a <cluster>-final-<short-sha> snapshot behind. Both knobs guard the same kind of accidental data loss; both should be reachable from the example. The fix adds var.skip_final_snapshot to the example wrapper (default false, tripwire stays intact) and threads it into the module input, mirroring how s3_force_destroy is wired. Tfvars example updated.

The verification itself: stood up the full stack from examples/default/ on a clean AWS account in us-east-1 with allow_plaintext_alb = true / s3_force_destroy = true / skip_final_snapshot = true and the rest at defaults. terraform apply provisioned 93 resources in ~20 minutes (Aurora cluster + writer + reader, ElastiCache, ALB, three ECS services, S3, NAT, the works); the auto-bootstrap created the litellm_app IAM-authed Postgres user; the migration task applied the full Prisma schema; the three ECS services all reached rolloutState: COMPLETED / "steady state".

Live curl against the ALB (master key fetched from Secrets Manager, no proxy_config / no extra secrets so it's the pure OSS / trial path):

===gateway: /health/readiness===
HTTP 200
{"status":"healthy","db":"connected"}
===backend: /key/info===
HTTP 404
{"error":{"message":"Key not found in database",...}}
===backend: /user/info===
HTTP 404
{"error":{"message":"User default_user_id not found",...}}
===ui: /===
HTTP 200
<!DOCTYPE html>... (Next.js index page)
===ui: /ui===
HTTP 200
<!DOCTYPE html>... (Next.js index page)
===gateway: /v1/models===
HTTP 200
{"data":[],"object":"list"}

db: connected on /health/readiness confirms the Aurora-IAM chain (generate_iam_auth_token → DatabaseURLSettings.apply_to_env → Prisma) works end-to-end. /v1/models returning 200 with the master key confirms gateway auth and ALB path-routing (LLM data plane → gateway target group). /key/info returning 404 "Key not found" instead of 401 confirms the backend is reachable through its own path-routing rule; the 404 is correct semantics, the master key is not a DB row. / and /ui returning Next.js HTML confirms the UI service is healthy and the static-asset routing rules (/, /ui, /_next/*, /litellm-asset-prefix/*) work.

terraform destroy then cleanly tore down all 93 resources (no errors, no leftover Aurora snapshot or S3 objects).

Relevant issues

Linear ticket

Resolves LIT-3504

Docs

Companion docs PR: BerriAI/litellm-docs#275

Pre-Submission checklist

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement; see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Type

🧹 Refactoring
📖 Documentation
🚄 Infrastructure

Changes

Infra/docs-only change to the terraform/litellm/ stacks and the deploy docs. No Python runtime code is touched.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 3 committers have signed the CLA.

❌ yassin-berriai
❌ yassinkortam
❌ claude
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR refactors the AWS and GCP Terraform stacks into provider-free reusable modules, adds examples/default/ thin-root wrappers that own the provider config, threads local.tags explicitly onto every taggable AWS resource, adds a lifecycle { ignore_changes = [settings[0].disk_size] } fix to prevent Cloud SQL autoresize-driven destroy/recreate, and renames the GCP managed SSL cert to include a domain hash (with create_before_destroy) to fix the resourceInUseByAnotherResource failure on domain changes.

• AWS module: providers.tf deleted; local.tags added to every taggable resource across alb.tf, ecs.tf, iam.tf, network.tf, rds.tf, redis.tf, s3.tf, secrets.tf, bootstrap.tf, and migrations.tf; state migration guide included in README.
• GCP module: providers.tf deleted; Cloud SQL disk_size ignore_changes fix added; managed cert renamed with domain hash and create_before_destroy; state migration guide included in README — but the guide's "expect: No changes" claim is incorrect for TLS-enabled stacks because the cert rename triggers a replacement even after address migration.
• Both stacks: New examples/default/ roots (provider config, curated variables, outputs, lock files) and updated READMEs documenting the module-first design and for_each multi-tenant pattern.

Confidence Score: 4/5

The refactor is structurally sound, but the GCP state migration guide tells operators to expect no plan changes when there will actually be a managed SSL certificate replacement for any TLS-enabled stack.

The AWS changes are clean: every taggable resource receives local.tags, the provider deletion is correct, and the migration guide accurately claims No Changes after address migration. The GCP Cloud SQL ignore_changes fix and the cert create_before_destroy rename are both correct improvements. The sole defect is that the GCP README migration guide claims terraform plan expect No Changes, which will not hold for any deployment with TLS enabled.

terraform/litellm/gcp/README.md — the state migration guide needs a note that one managed SSL certificate replacement is expected for TLS-enabled stacks and that it is safe due to create_before_destroy.

Important Files Changed

Filename	Overview
terraform/litellm/aws/locals.tf	Adds local.tags combining litellm:stack, managed-by, and var.tags; replaces what the now-deleted provider default_tags was doing.
terraform/litellm/aws/providers.tf	Deleted — provider block moved to examples/default/providers.tf so the module can be consumed with count/for_each/aliased providers.
terraform/litellm/aws/examples/default/main.tf	New thin-root wrapper that wires the AWS provider to the module with a curated variable surface; all looks correct.
terraform/litellm/aws/examples/default/providers.tf	Root-level AWS provider with commented-out default_tags slot; managed-by redundancy from previous thread is addressed.
terraform/litellm/gcp/load_balancer.tf	Adds hashed cert name and create_before_destroy to fix resourceInUseByAnotherResource on domain changes; the name change is a breaking rename for existing TLS deployments that the migration guide's "No changes" claim does not account for.
terraform/litellm/gcp/cloudsql.tf	Adds lifecycle { ignore_changes = [settings[0].disk_size] } to both writer and reader instances to prevent disk autoresize triggering a destroy/recreate; correct fix.
terraform/litellm/gcp/providers.tf	Deleted — google/google-beta provider blocks moved to gcp/examples/default/providers.tf.
terraform/litellm/gcp/examples/default/main.tf	New thin-root wrapper for GCP; correctly notes the configuration_aliases limitation for for_each multi-project deployments.
terraform/litellm/gcp/README.md	Adds module-first design docs, for_each multi-tenant pattern, and state migration guide; the guide's "expect: No changes" assertion is incorrect for TLS-enabled stacks due to the cert rename.
terraform/litellm/aws/README.md	Adds module-first docs, for_each multi-tenant pattern, and AWS state migration guide; terraform plan # expect: No changes is accurate for AWS (no managed certs are renamed here).
terraform/litellm/aws/alb.tf	Threads local.tags onto all ALB resources (lb, target groups, listeners, listener rules); coverage is complete.
terraform/litellm/aws/network.tf	Replaces bare { Name = ... } tags with merge(local.tags, { Name = ... }) and adds local.tags to security groups; correct approach.

_{Reviews (2): Last reviewed commit: "docs(terraform): address review — state-..." | Re-trigger Greptile}

[yassin-berriai]

@greptileai

---

Generated by Claude Code

[greptile-apps]

GCP state migration "No changes" claim is incorrect for TLS-enabled stacks

This PR renames google_compute_managed_ssl_certificate from ${local.name}-cert to ${local.name}-cert-${substr(sha1(join(",", var.lb_domains)), 0, 8)} (in load_balancer.tf). After running the scripted state mv loop, Terraform will still see this resource as needing replacement because the name stored in state (acme-litellm-prod-cert) doesn't match the newly computed name. The create_before_destroy lifecycle makes the replacement safe, but the guide's terraform plan # expect: No changes assertion will fail for anyone who has lb_domains set — leaving operators unsure whether the replacement is expected or a sign that the migration went wrong.

The migration guide should note that one resource replacement is expected for TLS-enabled stacks, and that the create_before_destroy lifecycle handles it safely.

[yassin-berriai]

Good catch on the cert-rename edge case. Rather than patch the wording, I've removed the state-migration guides from both the AWS and GCP READMEs entirely (and the corresponding note from the docs PR).

Rationale: these Terraform stacks have never been published or tagged for external consumption, so there are no existing deployments sitting on the old root-module addresses to migrate from. The terraform state mv guidance — and therefore the inaccurate "expect: No changes" assertion that wouldn't hold once the managed SSL cert is renamed — only applied to a hypothetical upgrade path that doesn't exist. Dropping it removes the incorrect claim at the source.

Done in d28283a (cert note) → superseded by 4c00811 (guides removed).

---

Generated by Claude Code

[veria-ai]

High: Plaintext load balancer by default

allow_plaintext_lb defaults to true for the one-click GCP deploy, which publishes the proxy and UI over http://<lb-ip> and the tutorial has users sign in with the master key on that URL. A network-path attacker can capture the master key or bearer tokens from users who follow the default DeployStack flow; make TLS the default path by collecting lb_domains before apply, or require an explicit opt-in for plaintext rather than defaulting to it.

[yassin-berriai]

Thanks — this is a deliberate, documented tradeoff for the one-click DeployStack path, so we're keeping the allow_plaintext_lb = "true" default here rather than flipping it.

Rationale: GCP's external HTTPS LB uses a Google-managed SSL certificate, which is only issued once the domain in lb_domains resolves to the LB's anycast IP. That IP doesn't exist until after the first apply, so TLS is inherently a two-step flow (apply → point DNS at lb_ip → set lb_domains and re-apply). A true single-click deploy can't provision managed TLS up front, so the installer defaults to HTTP to keep the zero-input path working for trial/dev.

Guardrails already in place:

• The module is TLS-first by default outside DeployStack: terraform plan refuses to build an HTTP-only LB unless allow_plaintext_lb = true is set explicitly (precondition in load_balancer.tf). The plaintext default lives only in the one-click deploystack.json, which is explicitly the trial/dev entry point.
• The setting is surfaced as a first-class toggle in the installer with the description telling users to leave it off and set lb_domains for production.
• The README "TLS" section documents the production two-step flow and the 301→HTTPS upgrade behavior.

We'll make the production-hardening path more prominent in the DeployStack docs, but the HTTP default for the one-click trial flow is intended. Flagging the master-key-over-HTTP exposure for the trial path is fair, and that's why it's gated behind an explicit dev/trial opt-in everywhere except this installer.

---

Generated by Claude Code

[yassin-berriai]

This is working as intended. The DeployStack one-click is a trial/dev convenience path, not the production entry point; that's why the prompt is explicitly labeled "Set true for trial/dev. For production, leave false and add lb_domains to terraform.tfvars after the first apply", and the tutorial has a dedicated "Going to TLS" step covering DNS + a managed cert.

The module and the examples/default root both default allow_plaintext_lb to false, so the production path fails closed: a plan without lb_domains errors out until the operator either provides a managed-cert domain or consciously opts into plaintext. The end user owns their own infrastructure and is responsible for standing it up securely in prod; we give them a secure-by-default module plus a clearly-scoped trial shortcut, which is the right split.

Not making a change here.

---

Generated by Claude Code

[veria-ai]

PR overview

This PR refactors the AWS and GCP Terraform stacks into reusable modules, including example deployment configuration for the LiteLLM GCP stack. The changed GCP DeployStack example defines the default one-click deployment inputs for publishing the proxy and UI behind a load balancer.

There is one open security issue: the default GCP one-click deployment currently enables a plaintext load balancer, causing users who follow the default flow to access the proxy and UI over HTTP. Because the tutorial flow involves signing in with the master key, a network-path attacker could capture credentials or bearer tokens unless TLS is made the default or plaintext is made an explicit opt-in. No issues have been addressed yet, so the PR still carries a clear deployment-default exposure.

Open issues (1)

• High: Plaintext load balancer by default — terraform/litellm/gcp/examples/default/deploystack.json:33

Fixed/addressed: 0 · PR risk: 7/10