refactor(terraform): modernize customer data platform guide infrastructure - #273
Merged
Merged
Conversation
- Remove monolithic project and VPC networking resources in favor of assuming pre-existing project and networking infrastructure. - Prune legacy variables (billing_account, organization, project_create, internet_access, network_prefix) and add standardized variables (subnetwork, bucket_name, service_account_name, create_bucket, destroy_all_resources, bq_dataset, bq_table). - Enable required Google Cloud APIs natively via google_project_service. - Standardize dedicated Dataflow worker service account (cdp-dataflow-sa) with least-privilege IAM roles. - Automate subnetwork IAM (roles/compute.networkUser) for workers when subnetwork is configured. - Provision BigQuery destination table (unified_data) declaratively in Terraform. - Configure local_file.variables_script to generate 00_set_environment.sh. - Refactor pipeline runner scripts into single-responsibility modular scripts: 01_build_and_push_container.sh and 02_run_dataflow.sh (with private IP enforcement and dynamic subnetwork options); delete legacy run.sh. - Update generate_transaction_data.py to use environment variables and CLI arguments with fallback to local sample data. - Update documentation, use case guides, and agent deployment skills.
iht
commented
Sep 7, 2026
iht
commented
Sep 7, 2026
…rraform BQ module to cdp_dataset
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Modernizes the Customer Data Platform (CDP) solution guide infrastructure and pipeline scripts according to the architectural patterns, security guardrails, and quality checklist specified in
terraform/TERRAFORM_REVAMP_GUIDE.mdandAGENTS.md.Key Changes
Infrastructure Modernization (
terraform/cdp/):module.google_cloud_project,module.vpc_network,module.firewall_rules,module.regional_nat).billing_account,organization,project_create,internet_access,network_prefix).subnetwork,bucket_name,service_account_name(default:cdp-dataflow-sa),create_bucket,destroy_all_resources,bq_dataset, andbq_table.google_project_service(dataflow,cloudbuild,artifactregistry,pubsub,bigquery,monitoring,storage).cdp-dataflow-sa) with least-privilege IAM roles.roles/compute.networkUser) for workers when a subnetwork is configured.google_bigquery_table.unified_data) in Terraform with schema matchingunified_table.jsonand lifecycle tied tovar.destroy_all_resources.local_file.variables_scriptto generatepipelines/cdp/scripts/00_set_environment.sh.Pipeline Runner Scripts (
pipelines/cdp/scripts/):01_build_and_push_container.sh: Dedicated container build script using Cloud Build.02_run_dataflow.sh: Dedicated Dataflow job launcher enforcing--no_use_public_ipsand dynamic subnetwork argument injection ($SUBNET_OPT).run.sh,01_cloudbuild_and_push_container.sh,02_run_dataflow_job.sh)..gitignoreto ignore the generated00_set_environment.sh.generate_transaction_data.pyto support environment variables, CLI arguments, and automatic fallback to local sample data.Documentation & Skill Registries:
terraform/cdp/README.md,pipelines/cdp/README.md, anduse_cases/CDP.md..agents/skills/use-case-deployment/SKILL.md,terraform/AGENTS.md, and rootAGENTS.md.(Done)interraform/TERRAFORM_REVAMP_GUIDE.md.Verification
terraform fmt -checkpassed cleanly.terraform init -backend=false -upgradesucceeded with provider lock resolution.terraform validatepassed (Success! The configuration is valid.).bash -n *.shpassed for all scripts inpipelines/cdp/scripts/.python3 -m py_compilepassed forgenerate_transaction_data.py.