diff --git a/skills/bigtable-basics/SKILL.md b/skills/bigtable-basics/SKILL.md index e1bf9c27..df9fbb9d 100644 --- a/skills/bigtable-basics/SKILL.md +++ b/skills/bigtable-basics/SKILL.md @@ -1,9 +1,9 @@ --- name: bigtable-basics description: >- - Assists in provisioning instances/tables, designing performant schemas, and querying data in Bigtable. Use when designing Bigtable row keys, configuring column families, writing SQL queries or client library code (Java, Go, Python) for Bigtable, or diagnosing performance/hotspotting issues. Also use when provisioning Bigtable clusters using gcloud or cbt CLIs. Don't use for generic Cloud SQL administration. + Provisions instances and tables, designs performant schemas, and queries data in Bigtable. Use when designing Bigtable row keys, configuring column families, writing SQL queries or client library code (Java, Go, Python) for Bigtable, or diagnosing performance/hotspotting issues. Also use when provisioning Bigtable clusters using gcloud or cbt CLIs. Don't use for generic Cloud SQL administration. metadata: - version: v1 + version: v2 publisher: google category: Databases license: Apache-2.0 @@ -116,6 +116,6 @@ defined as SQL queries but they must be created using gcloud CLI. ## External Resources * [Cloud Bigtable Documentation](https://cloud.google.com/bigtable/docs) -* [Bigtable SQL Reference](https://cloud.google.com/bigtable/docs/reference/sql) +* [Bigtable SQL Reference](https://cloud.google.com/bigtable/docs/googlesql-overview) * [cbt CLI Reference](https://cloud.google.com/bigtable/docs/cbt-reference) * [gcloud bigtable Reference](https://cloud.google.com/sdk/gcloud/reference/bigtable) diff --git a/skills/cloud-databases-onboarding/SKILL.md b/skills/cloud-databases-onboarding/SKILL.md new file mode 100644 index 00000000..13a534f1 --- /dev/null +++ b/skills/cloud-databases-onboarding/SKILL.md @@ -0,0 +1,95 @@ +--- +name: cloud-databases-onboarding +description: >- + Guides users through discovering their database requirements, recommends a + Google Cloud database product or migration destination based on a recommendation + matrix, and assists in database creation. Use when a user asks 'What database + service should I use?', 'Help me pick a database', asks which Google Cloud + database to migrate or onboard a workload to, or wants to create a new database + on Google Cloud. Don't use for general Google Cloud maintenance, managing + existing databases, or troubleshooting active Database Migration Service (DMS) jobs. +license: Apache-2.0 +metadata: + version: v1 + publisher: google + category: Databases +--- + +# Google Cloud Database Onboarding Skill + +This skill provides domain instructions, decision matrices, and +Infrastructure-as-Code workflows to guide users through discovering their exact +database requirements, selecting an optimal Google Cloud database service, and +drafting starter resource provisioning code for user review. + +## Progressive Disclosure + +* **Reading / Progressive Disclosure:** When interacting with a user during a + conversation, load reference files progressively. Follow the Just-in-Time + (JiT) loading instructions outlined in the phases below. + +-------------------------------------------------------------------------------- + +## Workflow & Just-in-Time (JiT) Instructions + +This workflow operates in three distinct sequential phases. Evaluate the active +conversation history to determine the current phase and follow the corresponding +instructions: + +### Phase 1: Requirement Discovery & Information Gathering + +When a user asks `"What database should I use?"` or requires guidance on Google +Cloud database selection without providing sufficient source or workload +context, you must initiate the Discovery phase. + +1. **Load Discovery Instructions (JiT):** Read the complete contents of + `references/onboarding_prompts.md` using `view_file`. +2. **Execute Discovery:** Optionally check `get_active_gcp_connection` and + `get_active_editor_context` (when available in the IDE) to prefill context, + and follow the detailed Phase 1 instructions in `onboarding_prompts.md` to + gather core requirements (data model, workload, scale, and migration + context) using user-friendly phrasing and enforcing constraints (such as the + 90% confidence rule) before proposing any recommendation. + +### Phase 2: Recommendation Analysis & Matrix Consultation + +Once you have gathered sufficient explicit discovery context, you must determine +the optimal Google Cloud database recommendation. + +1. **Consult Matrix & Formulate Recommendation (JiT):** Follow the Phase 2 + instructions in `references/onboarding_prompts.md` (or, when the user's + prompt already specifies the source database and workload requirements, skip + reading `references/onboarding_prompts.md` and read only the matching + source section in `references/recommendation_matrix.txt` via `view_file` + `StartLine`/`EndLine` — MySQL: `44-118`, PostgreSQL: `119-236`, SQL Server: + `237-305`, Oracle: `306-463`, HBase/DynamoDB/Cassandra: `464-616`, + Redis/Valkey/Memcached/Aerospike: `617-751`, MongoDB/DocumentDB/CosmosDB: + `752-797`, Neo4j/Neptune/Graph: `798-836` — to formulate a single primary + recommendation). +2. **Deliver Recommendation:** Deliver the recommendation to the user, mapping + destination codes to plain English, explaining the reasoning, and offering + to help with provisioning. + +### Phase 3: Implementation & Provisioning (Plan-Validate-Execute Pattern) + +When the user accepts the recommendation and requests to provision or modify +cloud resources, you must initiate the Implementation & Provisioning phase. + +1. **Load Provisioning Instructions (JiT):** Read the Phase 3 section of + `references/onboarding_prompts.md` (if not already loaded) using + `view_file`. +2. **Execute Plan-Validate-Execute Workflow:** Follow the detailed Phase 3 + instructions in `references/onboarding_prompts.md` to analyze the workspace, + confirm target file paths with the user, draft and validate Terraform or + `gcloud` provisioning scripts according to the required formatting and + tagging rules, and prepare a Change List (CL) or Pull Request for user + review without applying live infrastructure changes directly. + +-------------------------------------------------------------------------------- + +## Supporting Resources & Documentation + +- [Onboarding Prompts & Phase Rules](references/onboarding_prompts.md) +- [Database Recommendation Matrix](references/recommendation_matrix.txt) +- [Google Cloud Databases Overview](https://cloud.google.com/products/databases) +- [Google Cloud SDK (gcloud) Reference](https://cloud.google.com/sdk/gcloud/reference) diff --git a/skills/cloud-databases-onboarding/references/onboarding_prompts.md b/skills/cloud-databases-onboarding/references/onboarding_prompts.md new file mode 100644 index 00000000..dc3c5278 --- /dev/null +++ b/skills/cloud-databases-onboarding/references/onboarding_prompts.md @@ -0,0 +1,261 @@ +# Database onboarding agent instructions + +Your primary role is to assist users in selecting the best GCP database service +for their use case from the Google Cloud (GCP) portfolio. + +You operate in one of three distinct phases: **Phase 1 (Discovery)**, **Phase 2 +(Recommendation)**, or **Phase 3 (Implementation & Provisioning)**. Evaluate the +active conversation history to determine your current phase and strictly follow +the instructions for that phase. + +## CRITICAL CONSTRAINTS + +**NEVER provide a premature database recommendation or guess a database choice +during Phase 1 when the user's prompt is open-ended and lacks source/workload +context.** *Why this matters:* Recommending a target database without knowing +the data model, transaction consistency (ACID vs. eventual), and workload/scale +leads to incompatible architectural choices, failed migrations, and costly +re-architecting. However, when the user's prompt **already specifies** their +source database (or application context) and workload/migration requirements +matching `references/recommendation_matrix.txt`, immediately proceed to +**Phase 2 (Recommendation)** in the same turn. + +**GLOBAL OVERRIDE: Instance Creation** **Condition:** At ANY point in the +conversation (Phase 1, Phase 2, or Phase 3), if the user explicitly requests to +create or provision a database instance (e.g., "Create a PostgreSQL instance" or +"Provision Cloud Spanner"). **Your Action:** + +1. IMMEDIATELY proceed to **Phase 3: Implementation & Provisioning + (Plan-Validate-Execute)** using the requested database type and any + requirements gathered so far in the conversation. + +-------------------------------------------------------------------------------- + +## PHASE 1: DISCOVERY (Information Gathering) + +**Condition:** +**DO NOT JUMP TO RECOMMENDATION WITHOUT ASKING FOR THE BELOW INFORMATION.** +You are in this phase when the user's request is open-ended and does not yet +provide enough information to match a scenario in +`references/recommendation_matrix.txt`. *A "complete picture" generally requires +knowing: 1) Data model (e.g., relational, document, key-value, vector), 2) +Primary workload (OLTP, OLAP, HTAP, vector search), 3) Scale, latency, and +throughput requirements (e.g., QPS, vector count, number of dimensions, index +size), 4) On-prem requirements and 5) Current database/migration context.* + +**Stay in this phase until you have gathered enough +information to confidently recommend a database product with 90% confidence.** + +**Your Actions & Rules:** + +1. **Optional IDE & Connection Pre-check:** When available (such as in the Data + Cloud VS Code extension), optionally call `get_active_gcp_connection` (to + resolve the active GCP project and region) and `get_active_editor_context` + (to inspect open files or database resources) so you can skip redundant + discovery questions and prefill context for Phase 3. Treat these tools as + strictly optional and proceed normally if they are unavailable. + +2. **Efficiency:** Gather requirements efficiently by asking multiple relevant + follow-up questions in a single turn. Your goal is to get a "complete + picture" (data model, workload, scale, and context) as quickly as possible. + +3. **Constraint:** Aim to gather enough information to make a confident + recommendation within **1-2 turns maximum**. + +4. **User-Friendly Phrasing:** Frame your questions in plain, accessible + language. If you must use technical jargon (e.g., "OLTP," "HTAP," + "relational," "horizontal scaling"), you MUST provide a brief, + easy-to-understand definition in plain English. + +5. **Provide Examples:** Along with each question, provide 3 or 4 options for + the users to choose from. Users may also respond with free text. + + * *Example:* "Are you dealing with highly structured data (like strict + accounting tables where every row looks the same), flexible/unstructured + data (like user profiles with varying attributes, JSON documents, or IoT + device logs), or both?" + + * *Example:* "What does the database need to do with the stored + information? For example, will you primarily be adding to and updating + it (like a shopping cart or user profile), searching and transforming it + (like running reports or analytics), or both?" + +6. If the user mentions a specific source database, consult the + `discovery_questions` under the corresponding `# Source:` section in + `references/recommendation_matrix.txt` as a baseline, adapting them to fit + these user-friendly rules. + +7. Maintain a conversational, empathetic, and helpful tone. + +8. Ask the user if they have any additional context or requirements to add + about their workflow. + +**CRITICAL CONSTRAINT:** Do NOT consult `references/recommendation_matrix.txt` +to output a database recommendation during this phase. *Why this matters:* +Proposing a target database without a complete discovery context results in +inaccurate or generic recommendations that fail to account for critical +migration complexity, schema constraints, or IOPS requirements. + +-------------------------------------------------------------------------------- + +## PHASE 2: RECOMMENDATION (Analysis & Delivery) + +**Condition:** You are in this phase when the user has provided enough explicit +information (either in their initial prompt or across 1-2 discovery turns) to +identify the matching recommendation in `references/recommendation_matrix.txt`. + +**Your Actions (Execute in strict order):** + +### **Step A: Requirement Summary & Matrix Lookup** + +1. Distill all the user's requirements from the conversation history into a + single, comprehensive summary sentence. + +2. **CONSULT THE RECOMMENDATION MATRIX:** Search for and read the relevant + `# Source:` section in `references/recommendation_matrix.txt` to match the + distilled requirements against the `selection_criteria`, `best_when`, + `migration_complexity`, and `benefits` fields. + +*💡 Examples of effective requirement summaries for matching against the matrix:* + +* "I need a low-cost, 100% open-source compatible, fully managed relational + database for a small internal web app. It will have low traffic and doesn't + need to scale much. MySQL or PostgreSQL is fine. We are on MySQL 5.7." + +* "We're migrating a large (10TB+) on-prem PostgreSQL 15 database to GCP. We + need a managed service with significantly better performance for mixed OLTP + and analytical (HTAP) workloads than standard PostgreSQL, plus low-lag read + replicas for reporting." + +* "We are modernizing from a self-hosted Oracle 19c database and want to move + to an open-source compatible database on GCP to reduce licensing costs. Key + needs are robust migration tooling, high availability, HTAP capabilities, + and a fully managed service." + +* "For our core transaction system, I need a globally distributed relational + database capable of horizontal write scaling with strong consistency, and it + must offer at least 99.999% availability. We're migrating from a sharded + MySQL setup." + +* "I'm looking for a managed NoSQL database on GCP for a high-throughput (1M+ + OPS), low-latency (<10ms) key-value store, primarily for time-series data + from IoT devices. We're migrating from on-prem HBase." + +* "I'm building a new mobile app and need a serverless NoSQL document database + that's easy to use, handles real-time data synchronization with clients, + offers offline support, and has a generous free tier to start." + +### **Step B: The Final Response & Recommendation Delivery** + +1. **CRITICAL INSTRUCTION:** After evaluating + `references/recommendation_matrix.txt`, you must generate a final, + user-facing response explaining the recommendation. + +2. Provide **exactly ONE** top database recommendation based on the matrix + output (do not provide a menu of multiple primary recommendations). + +3. **Map Destination Codes to Plain English:** The matrix uses destination + codes (enums). You MUST translate them to plain-English product names in + your response to the user according to this mapping: + + * `ALLOYDB_FOR_POSTGRESQL` -> AlloyDB for PostgreSQL + * `ALLOYDB_OMNI` -> AlloyDB Omni + * `BIGTABLE` -> Cloud Bigtable + * `CLOUD_SQL_FOR_MYSQL_ENTERPRISE` -> Cloud SQL for MySQL Enterprise + * `CLOUD_SQL_FOR_MYSQL_ENTERPRISE_PLUS` -> Cloud SQL for MySQL Enterprise + Plus + * `CLOUD_SQL_FOR_POSTGRESQL_ENTERPRISE` -> Cloud SQL for PostgreSQL + Enterprise + * `CLOUD_SQL_FOR_POSTGRESQL_ENTERPRISE_PLUS` -> Cloud SQL for PostgreSQL + Enterprise Plus + * `CLOUD_SQL_FOR_SQL_SERVER` -> Cloud SQL for SQL Server + * `FIRESTORE` -> Google Cloud Firestore + * `MEMORYSTORE` -> Memorystore + * `MEMORYSTORE_FOR_MEMCACHED` -> Memorystore for Memcached + * `MEMORYSTORE_FOR_REDIS` -> Memorystore for Redis + * `MEMORYSTORE_FOR_VALKEY` -> Memorystore for Valkey + * `SPANNER` -> Cloud Spanner + * `SPANNER_GRAPH` -> Cloud Spanner Graph + * `ORACLE_AT_DATABASES` -> Oracle Database@Google Cloud + * `ORACLE_SELF_MANAGED` -> Oracle (Self-Managed) + * `SPANNER_OMNI` -> Spanner Omni + +4. Clearly explain your reasoning by mapping the GCP database features, + selection criteria, migration complexity (including specific protocols, + APIs, or migration tools), and benefits from + `references/recommendation_matrix.txt` to the user's stated requirements. + Keep the explanation accessible, just like in the discovery phase. + +5. Offer to help the customer create a starter instance of the recommended + database (transitioning to Phase 3 upon user acceptance). + +-------------------------------------------------------------------------------- + +## PHASE 3: IMPLEMENTATION & PROVISIONING (Plan-Validate-Execute) + +**Condition:** You are in this phase when the user accepts the recommendation +and requests to provision, create, or modify database resources, or triggers the +global override. + +**Your Actions & Rules (Follow strict Plan-Validate-Execute):** + +Follow the **Plan-Validate-Execute Pattern** to draft the infrastructure +modifications. Do NOT apply changes to production directly: + +0. **Environment Verification:** Check if required tools (e.g., `gcloud`, + `terraform`, `python3`) are installed and properly authenticated on the + system. If tools are missing or not authenticated, guide the user on how to + install or log in before proceeding. + +1. **Analyze the Workspace & Active Context:** If not already retrieved during + discovery, optionally call `get_active_gcp_connection` (to prefill the + default `` and region) and `get_active_editor_context` (when + available in the IDE). Scan the user's workspace, open files, and related + directories for existing database resource scripts (e.g., Terraform + configuration files). + +2. **Obtain User Confirmation:** If the target infrastructure files or + directory are not clear, ask the user explicitly to confirm the file + paths or target directory before modifying anything. + +3. **Draft Infrastructure Plan (Plan):** Create or edit the necessary Terraform + configuration files or shell scripts to provision the resources. + + - When generating `gcloud` CLI commands or shell scripts, you MUST follow + and apply these `gcloud` CLI rules: + * Always use `gcloud beta` command group for database provisioning + (e.g., `gcloud beta create`). + * Validate leaf-level command syntax using `gcloud help + ` prior to proposing commands. + * Explicitly append `--project=` and explicit location + flags (`--region`, `--zone`, `--location`, or `--config`). + * Include `--dry-run` or `--validate-only` for pre-execution + verification if supported by the command. + * Include custom label/tag flags (e.g., + `--labels=resource_generated_by=cloud_db_onboarding_skill`) on + generated `gcloud` provisioning commands. + * Do NOT include `--quiet` (`-q`) flags in generated `gcloud` commands + (commands are drafted for interactive user review). + * The skill MUST ONLY draft provisioning commands or code for user + review and MUST NOT execute mutating/write infrastructure operations + directly. + - When creating or editing Terraform files or shell scripts, you MUST: + * Add a stamped header comment at the top of every generated file + (e.g., `# Generated with cloud onboarding skills selector + @timestamp`). + * Add a custom default label/tag `resource_generated_by = + "cloud_db_onboarding_skill"` under the `default_labels` / `labels` + block or as a resource label (GCP label values only allow lowercase + letters, numeric characters, underscores, and hyphens—no spaces). + +4. **Validate Infrastructure Code (Validate):** Before finalizing, validate + the drafted infrastructure code to verify syntax and configuration + correctness. *Why this matters:* Validating Terraform code ensures that + configuration blocks, IAM bindings, and instance sizing are + syntax-error-free and strictly enforceable before code review. + +5. **Create Pull Request / Change List (Execute):** After validation succeeds + with zero errors, prepare the changes as a **Change List (CL)** or Pull + Request for user review. Never apply live infrastructure changes + (`terraform apply` or mutating `gcloud` commands) directly to production + yourself. diff --git a/skills/cloud-databases-onboarding/references/recommendation_matrix.txt b/skills/cloud-databases-onboarding/references/recommendation_matrix.txt new file mode 100644 index 00000000..c034b6c5 --- /dev/null +++ b/skills/cloud-databases-onboarding/references/recommendation_matrix.txt @@ -0,0 +1,836 @@ +# Recommendation Matrix for Database Migrations to Google Cloud + +# =================================================================================================== +# TABLE OF CONTENTS (SUPPORTED SOURCE DATABASES) +# Search for the corresponding `# Source:` header below to read a specific section: +# +# - Source Database: MySQL +# - Primary GCP Destinations: Cloud SQL for MySQL Enterprise / Plus, Spanner +# +# - Source Database: PostgreSQL +# - Primary GCP Destinations: Cloud SQL for PostgreSQL, AlloyDB, Spanner +# +# - Source Database: Microsoft SQL Server +# - Primary GCP Destinations: Cloud SQL for SQL Server, AlloyDB for PostgreSQL, Spanner +# +# - Source Database: Oracle +# - Primary GCP Destinations: Oracle Database@Google Cloud, Oracle Self-Managed, AlloyDB for PostgreSQL, Spanner +# +# - Source Database: HBase +# - Primary GCP Destinations: Bigtable +# +# - Source Database: Amazon DynamoDB +# - Primary GCP Destinations: Bigtable, Spanner +# +# - Source Database: Apache Cassandra +# - Primary GCP Destinations: Bigtable, Spanner +# +# - Source Database: Redis / Valkey +# - Primary GCP Destinations: Memorystore for Valkey +# +# - Source Database: Memcached +# - Primary GCP Destinations: Memorystore for Valkey +# +# - Source Database: Aerospike +# - Primary GCP Destinations: Bigtable, Memorystore for Valkey +# +# - Source Database: MongoDB / Amazon DocumentDB / Azure CosmosDB +# - Primary GCP Destinations: Firestore, Spanner +# +# - Source Database: Neo4j / Amazon Neptune / Aerospike Graph +# - Primary GCP Destinations: Spanner Graph +# =================================================================================================== + +# Source: MySQL +source_recommendations { + source: MYSQL + discovery_questions: [ + "What MySQL version?", + "What is the size of the database?", + "What are the availability requirements?", + "Is it a sharded MySQL deployment?", + "Are you willing to refactor the application?", + "What is the planned growth rate in the next 6-12 months?", + "Does your application require temporary scaling up to handle workload surges?" + ] + recommendations { + selection_criteria: [ + "100% open source compatible", + "Standard production performance and availability (99.95% SLA)", + "Also includes versions predating MySQL 8" + ] + primary_destination: CLOUD_SQL_FOR_MYSQL_ENTERPRISE + migration_complexity: [ + "Low complexity. Easy with Database Migration Service" + ] + best_when: "Existing MySQL customers are looking to stay on MySQL. Cloud SQL for MySQL is a great choice for straightforward migrations, offering an easy and accessible path to a fully managed, secure, and highly available cloud database." + benefits: [ + "Fully managed", + "Secure", + "H/A & DR", + "99.95% SLA", + "Broad MySQL version support", + "Lower entry point" + ] + } + recommendations { + selection_criteria: [ + "100% open source compatible", + "High performance (up to 3x faster reads, 2x faster writes) and high availability (99.99% SLA, near-zero maintenance downtime)", + "Continuous availability is needed", + "MySQL 8+ only" + ] + primary_destination: CLOUD_SQL_FOR_MYSQL_ENTERPRISE_PLUS + migration_complexity: [ + "Low complexity. Easy with Database Migration Service" + ] + best_when: "Cloud SQL Enterprise Plus is the premier choice for mission-critical MySQL workloads, delivering dramatic performance gains with up to 3x faster reads and 2x improved write latency, while ensuring maximum availability through near-zero maintenance downtime and a 99.99% SLA." + benefits: [ + "All the benefits of Cloud SQL for MySQL Enterprise", + "Optimized performance (3x faster)", + "Near zero maintenance downtime", + "99.99% SLA" + ] + } + recommendations { + selection_criteria: [ + "Sharded MySQL workloads (e.g. Vitess, app sharding) or large scale workloads", + "Ability to handle extremely large databases", + "Customers looking for highest scale and availability (e.g. payments use-cases)", + "Customer pain point around scale and management of existing sharded MySQL environment", + "Workloads with highly spiky or unpredictable traffic patterns that benefit from autoscaling to optimize costs" + ] + primary_destination: SPANNER + migration_complexity: [ + "Medium to Higher Complexity. Requires query and transaction handling changes, but Google SQL similarity to MySQL limits the rewrite effort. We offer free Spanner Migration Tool and Black Belt team expertise" + ] + best_when: "The source DB is sharded MySQL - Spanner transforms the operational burden of sharding. Instead of managing individual database shards, Spanner operates as a single, logical database that automatically handles horizontal scaling, delivering virtually infinite capacity and multi-region consistent writes. This shift eliminates the pain of re-sharding and cross-shard management, replacing it with a system built for the highest reliability, backed by an up to 99.999% SLA and no maintenance downtime. Spanner's native autoscaling also makes it a highly cost-effective solution for spiky workloads." + benefits: [ + "Virtually infinite scale", + "Multi-region consistent writes", + "No maintenance downtime", + "99.999% SLA", + "Online schema changes", + "Automatic scaling to handle spiky workloads and optimize costs" + ] + } +} + +# Source: PostgreSQL +source_recommendations { + source: POSTGRESQL + discovery_questions: [ + "What PostgreSQL version?", + "What is the size of the database?", + "Is it an analytical/HTAP workload?", + "Is it a sharded PostgreSQL deployment?", + "Do you need to leave a portion of your workload on-premises?", + "Do you need a multi cloud solution?", + "Do you need to build Gen AI applications with pgvector?", + "Does your application require temporary scaling up to handle workload surges?" + ] + recommendations { + selection_criteria: [ + "100% open source compatible", + "Standard production performance and availability (99.95% SLA)", + "Supports PostgreSQL 9.6 and above (including legacy versions 9.6, 10, 11)" + ] + primary_destination: CLOUD_SQL_FOR_POSTGRESQL_ENTERPRISE + migration_complexity: [ + "Low complexity. Easy with Database Migration Service" + ] + best_when: "Existing PostgreSQL customers who are looking to stay on PostgreSQL. Cloud SQL is a great choice for straightforward migrations, offering an easy and accessible path to a fully managed, secure, and highly available cloud database." + benefits: [ + "Fully managed", + "Secure", + "H/A & DR", + "99.95% SLA", + "Broadest PostgreSQL version support", + "Up to 64TB storage", + "pgvector support for Gen AI apps" + ] + } + recommendations { + selection_criteria: [ + "100% open source compatible", + "High performance (up to 4x reads, 2x writes) and high availability (99.99% SLA, near-zero maintenance downtime)", + "Supports PostgreSQL 12 and above" + ] + primary_destination: CLOUD_SQL_FOR_POSTGRESQL_ENTERPRISE_PLUS + migration_complexity: [ + "Low complexity. Easy with Database Migration Service" + ] + best_when: "Existing PostgreSQL customers who are looking to stay on PostgreSQL. Cloud SQL Enterprise Plus delivers dramatic performance gains, while ensuring maximum availability through a 99.99% SLA." + benefits: [ + "All the features and capabilities of Enterprise edition with up to 2x improved write throughput and up to 4x improved read throughput compared to Enterprise edition", + "Near zero maintenance downtime", + "99.99% SLA", + "pgvector support for Gen AI apps" + ] + } + recommendations { + selection_criteria: [ + "Higher level of performance and manageability than open source", + "Fully open source compatible", + "Extreme performance (4x transactional, 100x analytical HTAP) and high availability (99.99% SLA, near-zero maintenance downtime)", + "Hybrid Transactional and Analytical (HTAP) workloads", + "Do you need to scale out for reads?" + ] + primary_destination: ALLOYDB_FOR_POSTGRESQL + migration_complexity: [ + "Low complexity. Easy with Database Migration Service" + ] + best_when: "AlloyDB is a next-generation, PostgreSQL-compatible database that modernizes the most demanding enterprise workloads by delivering extreme performance for transactions and analytics (HTAP), a cloud-native elastic architecture, and a fully integrated platform for generative AI, all backed by enterprise-grade availability." + benefits: [ + "Superior performance, columnar engine, improved manageability, autopilot optimizations", + "4x performance for transactions and 100x faster analytical queries compared to stock PostgreSQL", + "Up to 64TB storage (>64TB coming soon)", + "Cloud-first architecture with complete disaggregation of compute and storage", + "Read replicas and read replica pools", + "Vertex AI integration for real-time model inferencing, BQ integration with Datastream", + "pgvector support for Gen AI apps", + "Near zero maintenance downtime" + ] + } + recommendations { + selection_criteria: [ + "Need to run PostgreSQL workloads on-premises and/or in any public cloud", + "Need to break free from legacy databases faster than you can replatform in the Cloud", + "Developers looking for a local / on-premises database" + ] + primary_destination: ALLOYDB_OMNI + migration_complexity: [ + "Low complexity" + ] + best_when: "PostgreSQL customers who are looking to stay on PostgreSQL. Looking to run on-prem, in any cloud, or in an air-gapped environment." + benefits: [ + "Run AlloyDB anywhere you run Databases", + "2x performance for OLTP and 100x for analytical queries compared to stock PostgreSQL", + "Support for edge computing, hybrid and multi-cloud customers who have regulatory restrictions or data sovereignty requirements", + "No cost for development and testing", + "Fully compatible with AlloyDB in the Cloud" + ] + } + recommendations { + selection_criteria: [ + "Ability to handle extremely large databases or sharded PostgreSQL workloads", + "Customers looking for highest scale and availability (e.g. payments use-cases)", + "Workloads with highly spiky or unpredictable traffic patterns that benefit from autoscaling to optimize costs" + ] + primary_destination: SPANNER + migration_complexity: [ + "Medium complexity. While Spanner is a distributed database, the Spanner PostgreSQL dialect significantly reduces the migration effort. We offer free Spanner Migration Tool and Black Belt team expertise" + ] + best_when: "The source is a large or sharded PostgreSQL - Spanner transforms the operational burden of sharding. Instead of managing individual database shards, Spanner operates as a single, logical database that automatically handles horizontal scaling, delivering virtually infinite capacity and multi-region consistent writes. This shift eliminates the pain of re-sharding and cross-shard management, replacing it with a system built for the highest reliability, backed by an up to 99.999% SLA and no maintenance downtime. Spanner's native autoscaling also makes it a highly cost-effective solution for spiky workloads." + benefits: [ + "Virtually infinite scale", + "Multi-region consistent writes", + "No maintenance downtime", + "99.999% SLA", + "Online schema changes", + "Analyze operational data isolated from transactional workloads with BigQuery", + "Automatic scaling to handle spiky workloads and optimize costs" + ] + } +} + +# Source: SQL Server +source_recommendations { + source: SQL_SERVER + discovery_questions: [ + "What SQL Server version?", + "Are you concerned about licensing costs and do you have an existing ELA with Microsoft?", + "SQL Server-specific options needed?", + "Is this OLTP, analytical, or mixed workload?", + "Are you willing to refactor the application?", + "Are you interested in moving to open source?" + ] + recommendations { + selection_criteria: [ + "100% SQL Server compatible", + "Supports legacy or packaged apps that need to run on SQL Server", + "Data center exit" + ] + primary_destination: CLOUD_SQL_FOR_SQL_SERVER + migration_complexity: [ + "Easy homogeneous migrations with Database Migration Service coming soon; Current solution is to use SQL Server technologies" + ] + best_when: "Existing SQL Server users are looking to stay on SQL Server, Cloud SQL for SQL Server is a great choice for straightforward migrations, offering an easy and accessible path to a fully managed, secure, and highly available cloud database." + benefits: [ + "Fully managed", + "Secure", + "H/A & DR", + "99.95% SLA", + "Multi-regional DR" + ] + } + recommendations { + selection_criteria: [ + "Looking to break free from legacy, proprietary database with expensive licensing and modernize to open source", + "Support for Hybrid Transactional and Analytical (HTAP) workloads via an open (PostgreSQL) interface" + ] + primary_destination: ALLOYDB_FOR_POSTGRESQL + migration_complexity: [ + "Heterogeneous migration required with application code changes; No cost Database Migration Assessment via first-party tool" + ] + best_when: "(Modernization opportunity) For SQL Server customers looking to break free from legacy licensing costs, AlloyDB for PostgreSQL offers the premier path to modernization. It provides a fully managed, PostgreSQL-compatible experience that combines the freedom of open-source with performance far exceeding the standard version. AlloyDB is built to handle the most demanding enterprise workloads with its extreme speed for transactions and analytics (HTAP), a cloud-native elastic architecture, and a powerful, integrated AI platform." + benefits: [ + "Fully PostgreSQL compatible, with no licensing costs or vendor lock-in", + "Superior performance, columnar engine, improved manageability, autopilot optimizations", + "Cloud-first architecture with complete disaggregation of compute and storage", + "Vertex AI integration for real-time model inferencing, BigQuery integration with Datastream", + "99.99% SLA", + "Monetize your operational data with AI-driven insights: Build transformative applications with vector search, foundation models integration, and generative AI, while maintaining open-source compatibility", + "Near zero maintenance downtime" + ] + } + recommendations { + selection_criteria: [ + "Highest scale and availability requirements (e.g. global payments workloads)", + "Willing to modernize and migrate to a distributed database to eliminate legacy SQL Server licensing costs" + ] + primary_destination: SPANNER + migration_complexity: [ + "Higher complexity. Requires heterogeneous migration and application rewrite from T-SQL to Google SQL or PostgreSQL dialect. We offer database migration assessment tools and expertise" + ] + best_when: "SQL Server customers with extreme scale, global availability, and strong consistency requirements that exceed the capabilities of traditional relational databases. Spanner provides virtually infinite horizontal scale, multi-region consistent writes, and an up to 99.999% SLA, making it the premier choice for mission-critical modernizations." + benefits: [ + "Virtually infinite scale and global consistency", + "99.999% SLA with no maintenance downtime", + "Eliminate expensive SQL Server licensing costs", + "Support for Google SQL and PostgreSQL dialects" + ] + } +} + +# Source: Oracle +source_recommendations { + source: ORACLE + discovery_questions: [ + "What Oracle Database version and do you have an existing ULA?", + "What is the size of the database?", + "Oracle Database options used?", + "Are you willing to do some refactoring of the application?", + "Are you running Oracle Apps and/or other COTS apps that use Oracle DB?", + "Need to run databases on-premises and/or in any public cloud?" + ] + # source: ORACLE - recommendation #1 + recommendations { + selection_criteria: [ + "Mission critical databases", + "Runs Exadata today", + "Highest performance or availability requirements", + "Dedicated infrastructure requirements", + "Needs to run DW or mixed workload with Oracle" + ] + primary_destination: ORACLE_AT_DATABASES + migration_complexity: [ + "Low complexity, supports physical and logical migrations" + ] + benefits: [ + "Highest performance and availability of any Oracle database platform", + "Deploy Oracle Database 19c or 23ai with automation from Oracle", + "License included or BYOL pricing for flexible compliance", + "Dedicated infrastructure provides highest levels of security and consistent performance", + "Run the latest version of Oracle Database, including 23ai", + "Take advantage of Google Cloud’s differentiated AI and analytics capabilities" + ] + } + # source: ORACLE - recommendation #2 + recommendations { + selection_criteria: [ + "Runs Oracle RAC today", + "Needs access to operating system", + "Has a limited number of Oracle licenses available", + "Wants to run RAC without negotiating a contract with Oracle" + ] + primary_destination: ORACLE_AT_DATABASES + migration_complexity: [ + "Low complexity, supports physical and logical migrations" + ] + benefits: [ + "Deploy Oracle Database 19c or 23ai with automation from Oracle", + "High availability with the lowest cost of entry", + "License included or BYOL pricing for flexible compliance", + "Scalable deployments take advantage of cloud economics", + "Run the latest version of Oracle Database, including 23ai", + "Take advantage of Google Cloud’s differentiated AI and analytics capabilities", + "Available with public offer to run month-to-month without long term contracts" + ] + } + # source: ORACLE - recommendation #3 + recommendations { + selection_criteria: [ + "Smaller departmental or dev/test databases", + "Needs access to operating system", + "Has a limited number of Oracle licenses available" + ] + primary_destination: ORACLE_AT_DATABASES + migration_complexity: [ + "Low complexity, supports physical and logical migrations" + ] + benefits: [ + "Lowest cost of entry for Oracle Database", + "License included or BYOL pricing for flexible compliance", + "Run the latest version of Oracle Database, including 23ai", + "Take advantage of Google Cloud’s differentiated AI and analytics capabilities" + ] + } + # source: ORACLE - recommendation #4 + recommendations { + selection_criteria: [ + "Smaller departmental or dev/test databases", + "Looking for a fully managed database", + "Has a limited number of Oracle licenses available" + ] + primary_destination: ORACLE_AT_DATABASES + migration_complexity: [ + "Low complexity, supports physical and logical migrations" + ] + benefits: [ + "Fully managed deployment of Oracle database, including backup and recovery, patching, and system management", + "License included or BYOL pricing for flexible compliance", + "Scalable deployments ranging from 1 CPU to hundreds of cores", + "Run the latest version of Oracle Database, including 23ai", + "Take advantage of Google Cloud’s differentiated AI and analytics capabilities", + "Available with public offer to run month-to-month without long term contracts" + ] + } + # source: ORACLE - recommendation #5 + recommendations { + selection_criteria: [ + "Lift and shift your Oracle database and associated workload to GCE/GKE", + "Data center exit", + "In-house and department app workloads running on Oracle database" + ] + primary_destination: ORACLE_SELF_MANAGED + migration_complexity: [ + "Low complexity, lift and shift" + ] + benefits: [ + "Run Oracle in any Google Cloud region globally", + "Maintain full control over database and OS versions, enabling support for critical legacy Oracle instances or non-Oracle Linux operating systems", + "Elastic, on-demand infrastructure lets you scale compute and storage independently", + "Ideal for applications moving to microservice architecture with GKE" + ] + } + # source: ORACLE - recommendation #6 + recommendations { + selection_criteria: [ + "Mission critical databases", + "Highest performance or availability requirements", + "100% open source compatible (need to reduce proprietary database license expenditures)", + "Future proofing apps for AI", + "Hybrid Transactional and Analytical (HTAP) workloads" + ] + primary_destination: ALLOYDB_FOR_POSTGRESQL + migration_complexity: [ + "Heterogeneous migration with schema and data migration with Database Migration Service;", + "Application code changes required and support for automated code conversion coming soon" + ] + best_when: "Oracle customers looking to move away from the expensive licensing and vendor lock-in technology to an open standards based database with a rich set of capabilities to meet their enterprise workloads needs. AlloyDB is built to handle the most demanding enterprise workloads with its extreme speed for transactions and analytics (HTAP), a cloud-native elastic architecture, and a powerful, integrated AI platform." + benefits: [ + "Fully PostgreSQL compatible, with no vendor lock-in", + "Superior performance, columnar engine, improved manageability, autopilot optimizations", + "4x transaction throughput and up to 100x faster analytical queries compared to stock PostgreSQL", + "Cloud-first architecture with complete disaggregation of compute and storage", + "Vertex AI integration for real-time model inferencing, BigQuery integration with Datastream", + "Monetize your operational data with AI-driven insights. Build transformative applications with vector search, foundation models integration, and generative AI, while maintaining open-source compatibility.", + "Near zero maintenance downtime", + "Run AlloyDB anywhere you run databases with AlloyDB Omni", + "Suitable for edge computing, hybrid and multi-cloud, customers who have regulatory restrictions or data sovereignty requirements" + ] + } + # source: ORACLE - recommendation #7 + recommendations { + selection_criteria: [ + "Highest scale and availability requirements (e.g. global payments workloads)", + "Looking to modernize and eliminate expensive Oracle licensing and vendor lock-in" + ] + primary_destination: SPANNER + migration_complexity: [ + "Higher complexity. Requires heterogeneous migration and application rewrite from PL/SQL to Google SQL or PostgreSQL dialect. We offer database migration assessment tools and expertise" + ] + best_when: "Oracle customers with extreme scale, global availability, and strong consistency requirements that exceed the capabilities of traditional relational databases. Spanner provides virtually infinite horizontal scale, multi-region consistent writes, and an up to 99.999% SLA, making it the premier choice for mission-critical modernizations." + benefits: [ + "Virtually infinite scale and global consistency", + "99.999% SLA with no maintenance downtime", + "Eliminate Oracle licensing costs and vendor lock-in", + "Support for Google SQL and PostgreSQL dialects" + ] + } +} + +# Source: HBase +source_recommendations { + source: HBASE + discovery_questions: [ + "What distribution of HBase?", + "What is the database size, number of nodes, CPU and disk per cluster, and hardware configuration?", + "Are coprocessors being used, and for what purpose?" + ] + recommendations { + selection_criteria: [ + "Looking to move from self managed to fully managed", + "HBase compatible key-value store suitable for all your application", + "Cost sensitive app with high write throughput & large data volumes", + "Storage-driven apps needing low cost storage with moderate to high read/write throughput" + ] + primary_destination: BIGTABLE + migration_complexity: [ + "Tools for assessment and migration.", + "Support for live & offline migrations", + "Direct import tools from HBase to Bigtable", + "Ability to migrate coprocessor functionality" + ] + best_when: "For customers running HBase, Bigtable is the ideal migration destination, offering a seamless path to modernization with its native support for the HBase API. This allows customers the ability to migrate existing applications with minimal code changes while eliminating the operational burden of managing complex HBase clusters. By moving to a fully managed platform, you gain virtually infinite scaling with single-digit millisecond latency, an up to 99.999% SLA with no maintenance downtime, and a lower TCO at any scale, driven by a native autoscaler and flexible SSD/HDD storage options." + benefits: [ + "Fully managed key-value DB", + "Support for HBase APIs", + "Lower TCO at any scale", + "Single digit millisecond read and write latencies", + "Multi-writer support", + "Virtually infinite scaling including writes", + "Global replication and 99.999% SLA", + "Autoscaler to reduce costs", + "No maintenance downtime", + "Low storage costs with both SSD and HDD" + ] + } +} + +# Source: DynamoDB +source_recommendations { + source: DYNAMODB + discovery_questions: [ + "What is the read/write workload ratio?", + "What are the latency requirements?", + "What capacity mode: on-demand or provisioned capacity?", + "Are secondary indexes being used?", + "Are multi-row transactions being used?", + "Are Global Tables being used?" + ] + recommendations { + selection_criteria: [ + "High write ratio (for Clickstream or Time Series style workloads)", + "Large storage volume, needing low cost storage", + "Flexible schema", + "Multi-primary eventually consistent writes", + "Multi-region write latency sensitive and using DynamoDB Global Tables" + ] + primary_destination: BIGTABLE + migration_complexity: [ + "Heterogeneous migration", + "Application code conversion required" + ] + best_when: "Customers are looking for similar access patterns including retrieving data via a unique key that is identical in both systems. Application logic designed for DynamoDB's key-based lookups translates naturally to Bigtable's row key model. This conceptual alignment means the migration effort is less about re-architecting the application's core logic and more about adapting to a platform designed for superior scale and performance, building upon the familiar principles of a wide-column, key-value data store." + benefits: [ + "Virtually infinite scaling including writes", + "Single digit millisecond read and write latencies", + "Flexible single or multi-replica deployment", + "Low storage costs with both SSD and HDD", + "Lower TCO for most workloads" + ] + } + recommendations { + selection_criteria: [ + "High read ratio", + "Application leverages advanced database capabilities (indexes, composite keys, typed values, constraints, JSON, views, ACID transactions, SQL)", + "Need strong and/or eventual consistency options for reads", + "Need replication with strong consistency guarantees" + ] + primary_destination: SPANNER + migration_complexity: [ + "Heterogeneous migration", + "Application code conversion required" + ] + best_when: "For DynamoDB users, migrating to Spanner is a strategic modernization that provides a significant upgrade in database capabilities for roughly half the cost. This move allows customers the ability to evolve from a NoSQL K/V store to a globally distributed multi-model database that unlocks advanced features not native to DynamoDB." + benefits: [ + "Virtually infinite scaling including writes", + "Advanced database capabilities (e.g. SQL query, indexes, transactions, ACID, and typed values)", + "Single digit milliseconds regional read and write latencies (strongly consistent)", + "Single-region and multi-region strong consistency", + "1/2 cost of DynamoDB" + ] + } +} + +# Source: Cassandra +source_recommendations { + source: CASSANDRA + discovery_questions: [ + "What is the read/write workload ratio?", + "What are the latency requirements?", + "Are secondary indexes being used?", + "Are strong consistency reads needed?", + "How many replicas are needed?", + "What is the quorum consistency?", + "What is the data volume?", + "Do you need CQL querying dialect?" + ] + recommendations { + selection_criteria: [ + "High write ratio (for Clickstream or Time Series style workloads)", + "Large storage volume, needing low cost storage", + "Flexible schema", + "Multi-primary eventually consistent writes", + "Multi-region write latency sensitive" + ] + primary_destination: BIGTABLE + migration_complexity: [ + "Heterogeneous migration", + "Application code conversion required" + ] + best_when: "When users want to maintain a familiar architecture. Both are wide-column NoSQL databases, meaning the core concepts of schema design and key-based access patterns are highly transferable, simplifying the transition. The primary driver for migrating is to move from a self-managed cluster to a fully managed, auto-scaling service. This allows you to offload the complexities of maintenance and tuning, while gaining Bigtable’s consistent single-digit millisecond latency and a lower total cost of ownership at virtually any scale." + benefits: [ + "Virtually infinite scaling including writes", + "Single digit milliseconds read and write latency", + "Flexible single or multi-replica deployment", + "Low storage costs with both SSD and HDD", + "Lower TCO for most workloads" + ] + } + recommendations { + selection_criteria: [ + "High read ratio", + "Application leverages advanced database capabilities (indexes, composite keys, typed values, constraints, JSON, views, ACID transactions, SQL)", + "Use SQL API", + "Need strong and/or eventual consistency options for reads", + "Need replication with strong consistency guarantees" + ] + primary_destination: SPANNER + migration_complexity: [ + "Low complexity", + "Near lift-and-shift migration using Spanner's native Cassandra-compatible interface (CQL and wire protocol support) with minimal application changes" + ] + best_when: "When users are looking to gain capabilities that are fundamentally impossible with a NoSQL architecture. The primary driver is upgrading from eventual consistency to Spanner's global, fully ACID-compliant transactions, which ensures absolute data integrity and simplifies application development. A key advantage of this modernization is Spanner's unified multi-model platform, allowing you to combine relational (SQL), full-text search, graph, and vector search queries directly on your live transactional data. This eliminates the need for separate specialized databases and complex data pipelines. Furthermore, the columnar storage engine in Spanner, which enables true HTAP (Hybrid Transactional/Analytical Processing) allows you to run fast, real-time analytical queries directly on your transactional data, a capability far beyond Cassandra's scope. The transition is also more practical than ever, as Spanner's native Cassandra interface allows developers to reuse their CQL queries and client libraries, minimizing code refactoring while eliminating the operational burden of Cassandra with a fully managed service." + benefits: [ + "Virtually infinite scaling including writes", + "Single digit milliseconds regional read and write latencies", + "Single-region and multi-region strong consistency", + "Advanced database capabilities (e.g. SQL query, indexes, transactions, ACID, and typed values)", + "Lower TCO for most workloads" + ] + } +} + +# Source: Redis, Valkey +source_recommendations { + source: [REDIS, VALKEY] + discovery_questions: [ + "What is the primary use case (caching, session store, pub/sub, real-time data)?", + "What are the latency and throughput requirements?", + "Is high availability and automatic failover critical?", + "Are you looking for a fully managed service to reduce operational overhead?", + "Which version of Redis/Valkey?", + "Are you using clustering?", + "What is the read/write throughput?", + "Size of data stored in cache?" + ] + recommendations { + selection_criteria: [ + "Need for a fully managed, in-memory data store.", + "High throughput and ultra-low latency required.", + "Compatibility with existing Redis commands and clients.", + "High availability (automatic failover, replication) is essential.", + "Use cases include caching, session management, real-time analytics, leaderboards, message queues.", + "Moving from Redis to Valkey for OSS or for better performance", + "Moving from self managed to fully managed", + "Looking for an integrated experience with IAM Auth and other services" + ] + primary_destination: MEMORYSTORE_FOR_VALKEY + migration_complexity: [ + "Low complexity migrations.", + "Compatible with existing Redis and Valkey APIs.", + "Data migration typically using RIOT or import a RDB file using RIOT tooling.", + "No application code changes required." + ] + best_when: "For customers running Redis or Valkey, Memorystore for Valkey is the best choice. Valkey is fully compatible with Redis 7.2, making migration easy and seamless with no application code changes required. It offers a fully open-source solution with no lock-in, while delivering substantial performance enhancements (such as 3x throughput improvements compared to Memorystore for Redis Cluster in Valkey 9.0) and extreme performance with sub-millisecond latency, high availability (99.99% SLA), and zero-downtime scaling." + benefits: [ + "Fully managed service, reducing operational burden.", + "Fully managed experience to accelerate innovation and reduce risk and TCO (usually customers save >40-50% from Redis Enterprise)", + "Full open-source compatibility with no lock-in", + "Extreme performance with sub-millisecond latency and millions of operations per second.", + "High availability (99.99% SLA) with automatic failover and replicas.", + "Scalable (zero-downtime scaling).", + "Zero downtime maintenance", + "Integrated with Google Cloud ecosystem (monitoring, logging, security).", + "Open-source compatible (Redis 7.2 and below, Valkey 7.2, 8.0, and 9.0).", + "Supports persistence (RDB snapshot, AOF logging). And backups", + "Support cross regional replication", + "Valkey supports both clustering and noncluster mode.", + "Valkey 9.0 supports 3x throughput improvements compared to Memorystore for Redis Cluster.", + "Valkey supports up to 14.5TB data in cache" + ] + } +} + +# Source: Memcached +source_recommendations { + source: MEMCACHED + discovery_questions: [ + "What is the read/write throughput?", + "How many nodes?", + "Size of data stored in Memcached?" + ] + recommendations { + selection_criteria: [ + "Moving to the cloud?", + "Moving from self managed to fully managed?", + "Looking for a next-gen engine to replace deprecated Memcached" + ] + primary_destination: MEMORYSTORE_FOR_VALKEY + migration_complexity: [ + "Medium complexity migration.", + "Requires updating application code to use a Valkey- or Redis-compatible client library (RESP protocol) instead of Memcached protocol." + ] + best_when: "For customers running Memcached, Memorystore for Valkey is the recommended migration path as the Memcached service is being deprecated. Valkey serves as our next-generation engine, providing a modern, fully open-source in-memory solution." + benefits: [ + "Fully open source engine with no vendor lock-in.", + "Zero-downtime horizontal and vertical scaling (from 1.25 GB instances to 27.5TB clusters).", + "High availability architecture with a 99.99% SLA.", + "Rich and flexible data structures (beyond simple key-value).", + "Supports multi-command transactions and Lua scripting.", + "Access to advanced modules.", + "Persistence options including RDB snapshots and AOF (Append Only File)." + ] + } +} + +# Source: Aerospike +source_recommendations { + source: AEROSPIKE + discovery_questions: [ + "How large is the data?", + "What are latency requirements?", + "Is multi-region or multi-writer deployment required?", + "What are availability requirements?" + ] + recommendations { + selection_criteria: [ + "Looking to move from self managed to fully managed", + "Data needs to be persisted", + "Data volume makes in-memory options cost-prohibitive", + "Does not require sub-millisecond latencies" + ] + primary_destination: BIGTABLE + migration_complexity: [ + "Heterogeneous migration", + "Application code conversion required" + ] + best_when: "Bigtable is a great destination for Aerospike workloads as it delivers single-digit millisecond latency and with significantly lower TCO and reduced risk of a fully managed, auto-scaling platform. You get matching performance while completely eliminating the operational overhead of managing complex clusters." + benefits: [ + "Fully managed experience and reduced risk and TCO", + "Open-source compatibility with no lock-in", + "Single digit millisecond read and write latency", + "Globally replicated and up to 99.999% SLA", + "Low storage cost and native autoscaler to reduce cost" + ] + } + recommendations { + selection_criteria: [ + "Moving from self managed to fully managed", + "Non-volatile storage is not required", + "Requires sub-millisecond latencies", + "Dataset is relatively small" + ] + primary_destination: MEMORYSTORE_FOR_VALKEY + migration_complexity: [ + "Heterogeneous migration", + "Application code conversion required" + ] + best_when: "For Aerospike workloads used primarily as a high-speed cache where durable, persistent storage is not a requirement, Memorystore for Valkey is an excellent choice for modernization and cost reduction." + benefits: [ + "Fully managed experience and reduced risk and TCO", + "Open-source compatibility with no lock-in", + "Sub-millisecond read and write latencies", + "Clustering support for increased scale and throughput with 99.99% SLA" + ] + } +} + +# Source: MongoDB, Amazon DocumentDB, Azure CosmosDB +source_recommendations { + source: [MONGODB, AMAZON_DOCUMENTDB, AZURE_COSMOSDB] + discovery_questions: [ + "What is the size of your DocumentDB deployment?", + "What is the read/write throughput?" + ] + recommendations { + selection_criteria: [ + "Large instance or large storage requirements", + "Lower combined costs for storage and I/O", + "High availability" + ] + primary_destination: FIRESTORE + migration_complexity: [ + "Heterogeneous migration", + "Application code conversion required" + ] + best_when: "For customers with MongoDB workloads, Firestore is the best choice for modernizing your application and eliminating operational overhead. It provides a truly serverless platform that scales elastically to any size, which significantly lowers the total cost of ownership through a pay-for-what-you-use model. This simplicity is backed by enterprise-grade power, including an industry-leading 99.999% SLA and the ability to perform global, multi-document ACID transactions with strong consistency. With deep integration for event-driven architectures, Firestore allows you to build highly scalable and reliable applications without managing servers." + benefits: [ + "Serverless, elastic capacity with no scaling limits", + "Lower storage costs and no additional throughput (I/O) fees", + "Up to 99.999% SLA with automated failure recovery through built-in data replication and zero RPO/zero RTO", + "Global, ACID-compliant, multi-document transactions with strong consistency", + "Event triggering support for GKE, Cloud Run and Cloud Functions" + ] + } + recommendations { + selection_criteria: [ + "Highest scale and availability requirements", + "Complex query requirements, including indexes and JSON handling, at extreme scale" + ] + primary_destination: SPANNER + migration_complexity: [ + "Higher complexity. Requires heterogeneous migration from NoSQL document model to relational/hybrid model, and application rewrite" + ] + best_when: "MongoDB customers with extreme scaling requirements, demanding strong consistency, high availability (up to 99.999% SLA), and complex query capabilities (including JSON support and indexing) that exceed MongoDB's capabilities at scale." + benefits: [ + "Virtually infinite scale with global consistency", + "99.999% SLA with no maintenance downtime", + "Native support for JSON data type and indexing", + "Strong consistency for multi-region deployments" + ] + } +} + +# Source: Neo4j, Amazon Neptune, Aerospike Graph +source_recommendations { + source: [NEO4J, AEROSPIKE_GRAPH, AMAZON_NEPTUNE] + discovery_questions: [ + "What is the size of your Neo4j/ Neptune deployment?", + "Are you running into scaling/ performance issues with your existing graph database?", + "What graph query language are you using?", + "Are strong consistency reads needed?", + "Are you looking for a fully managed solution?", + "What are availability requirements?", + "Are you looking for FTS and vector search capabilities to augment your Graph use-case?", + "What is the source of your graph data?" + ] + recommendations { + selection_criteria: [ + "Need fully managed solution that doesn't require downtime for upgrades/maintenance", + "Want to use a graph query language (e.g. GQL; ISO standard) vs. a proprietary language like Cypher", + "Need a solution whose QPS scales linearly and is virtually unlimited.", + "Simpler solution that avoids managing pipelines from relational stores to build a graph", + "Application leverages advanced database capabilities (indexes, composite keys, typed values, constraints, JSON, views, ACID transactions, SQL)", + "Need strong and/or eventual consistency options for reads", + "Need replication with strong consistency guarantees", + "Need a graph solution without expensive licensing with a pay-as-you-go model." + ] + primary_destination: SPANNER_GRAPH + migration_complexity: [ + "Medium complexity migration. Some work required for schema conversion and data transfer. (WIP utility to migrate data from Neo4j to Spanner Graph). Query conversion is more straightforward.", + "Neo4j APOCs require work to be translated to GQL" + ] + best_when: "Spanner simplifies your data architecture by providing a high-performance, fully managed graph database that operates directly on your existing relational data. This zero-ETL approach eliminates the need to manage separate graph systems, which reduces cost, prevents data duplication, and allows you to run real-time graph analytics on live transactional data." + benefits: [ + "Leverage a unified, zero-ETL architecture by running graph queries directly on your existing relational data, which eliminates the need for separate databases, complex ETL pipelines, and data duplication.", + "Achieve massive scale and performance by efficiently querying graphs with hundreds of billions of nodes, with optimizations for high-speed traversal even on highly connected data.", + "Execute seamless multi-model queries by combining graph analytics with relational (SQL), full-text search, and vector search on a single, unified platform.", + "Built on an enterprise-grade foundation that includes all of Spanner’s core benefits, such as its industry-leading 99.999% availability, zero-downtime operations, and deep integration with GCP services like IAM and BigQuery." + ] + } +} + diff --git a/skills/gcp-job-monitoring/SKILL.md b/skills/gcp-job-monitoring/SKILL.md new file mode 100644 index 00000000..5f8958f8 --- /dev/null +++ b/skills/gcp-job-monitoring/SKILL.md @@ -0,0 +1,336 @@ +--- +name: gcp-job-monitoring +description: "Lists, monitors, ranks, aggregates, and inspects Google Cloud BigQuery jobs, slot reservations, hot tables, and Dataproc Spark workloads (Dataproc Serverless batches, interactive sessions, and Dataproc clusters/jobs) across cost, slot usage, duration, status, and execution stages. Use when the user asks to show their BigQuery or Spark jobs, find which jobs or tables cost the most money or billed the most bytes/slots, get a breakdown of spend or errors by user or reservation, inspect slot reservations and capacity commitments, list active Dataproc clusters or interactive sessions, identify slow or stuck queries, or get detailed execution breakdowns for a specific job ID." +license: Apache-2.0 +metadata: + version: v1 + publisher: google +--- + +# BigQuery & Spark Job Monitoring Skill + +This skill provides a fast, zero-dependency workflow for answering admin and +developer job-monitoring questions across **BigQuery** and **Dataproc Spark** +(both **Dataproc Serverless Batches** and **Dataproc Cluster Jobs**): + +1. **"Show me my BigQuery jobs"** (recent jobs, filterable by user or state) +2. **"What jobs cost the most money?"** (ranked by bytes billed, estimated + on-demand USD cost, and slot time) +3. **"What took long to run?"** (ranked by wall-clock elapsed time and slot + contention) +4. **"What Spark jobs have I run?"** (unified view across Dataproc Serverless + batches and Dataproc cluster jobs) +5. **"Give me details about a specific job"** (full SQL text, execution stages, + bytes/slots/DCUs, referenced tables, errors, and console/log links) + +-------------------------------------------------------------------------------- + +## Critical Rules & Real-World Gotchas (MUST FOLLOW) + +1. **Use `scripts/gcp_job_monitor.py` FIRST for BigQuery and Spark job + queries:** + - Do **NOT** start by running `SELECT ... FROM INFORMATION_SCHEMA.JOBS_BY_PROJECT` + or `JOBS_BY_USER` when shell execution is available. + - **Why:** In many enterprise GCP projects, users with `roles/bigquery.jobUser` + get `403 Access Denied` on both `INFORMATION_SCHEMA.JOBS_BY_PROJECT` + (`bigquery.jobs.listAll`) and `INFORMATION_SCHEMA.JOBS_BY_USER`, whereas + the BigQuery `jobs.list` (`projection=full`) and `jobs.get` REST API used + by `scripts/gcp_job_monitor.py` **succeeds across all regions in a single + call** and **costs $0** (no query job is billed). + - `scripts/gcp_job_monitor.py` automatically tries `allUsers=true` (project + admin view) and falls back to `allUsers=false` (current user's jobs) if the + caller lacks `bigquery.jobs.listAll`. +2. **Dual-Transport Auth (Cloudtop/Remote VMs + GE Sobi VMaaS `remote_cli mcp`):** + - **Cloudtop & Remote VMs (Context-Aware Access)**: Bare `bq ls -j` or + `gcloud auth print-access-token` often fails with `Access was blocked by + Context Aware Access`. `scripts/gcp_job_monitor.py` automatically + authenticates via Application Default Credentials (`gcloud auth + application-default print-access-token --quiet`) first and falls back to + user credentials (`--quiet`), never hanging on interactive `(y/N)?` + prompts. + - **Gemini Enterprise (GE) Sobi VMaaS / ACE Sandboxes (`/usr/local/bin/remote_cli`)**: + In the Sobi VMaaS sandbox, local `gcloud` has no token and all + authenticated egress goes through `remote_cli mcp` (`bigquery`, + `google-managed-apache-spark`, and `google-cloud-cli`). + `scripts/gcp_job_monitor.py` automatically detects when `gcloud` has no + token and `remote_cli` is on `PATH` (or you can pass + `--transport=remote_cli`) and routes every action through + `RemoteCliMcpClient` (`remote_cli mcp bigquery` and `remote_cli mcp + google-managed-apache-spark`). + - If authentication is missing or expired on a local VM, run `gcloud auth + application-default login` or configure Google Application Default Credentials. +3. **Always Check Both Dataproc Serverless Batches AND Cluster Jobs for Spark:** + - Users asking *"What Spark jobs have I run?"* rarely distinguish between + **Dataproc Serverless Batches** (`projects.locations.batches`) and + **Dataproc Cluster Jobs** (`projects.regions.jobs`). + - `scripts/gcp_job_monitor.py` queries **both** APIs in one call and merges + them into a unified table with workload type (`SERVERLESS_BATCH` vs. + `CLUSTER_JOB`), engine (`PYSPARK`, `SPARK`, `SPARK_SQL`), duration, and DCU + usage. +4. **Deep-Dive Troubleshooting Handoff:** + - When `spark_describe` or `spark_list` reveals a `FAILED` Spark batch or + cluster job and the user wants root-cause analysis (driver logs, GCS event + logs, JVM bytecode, or stage skew/spill diagnostics), hand off to + `@skill:gcp-spark-troubleshooting`. + +-------------------------------------------------------------------------------- + +## Workflow 1: BigQuery Job Monitoring (`scripts/gcp_job_monitor.py`) + +Run `scripts/gcp_job_monitor.py` using `python3`. It requires **only the Python +standard library** (do not `pip install` anything). If `--project` (or +`--project_id`) is omitted, it auto-detects the active project from `gcloud +config get-value project`. + +### 1. List Recent BigQuery Jobs ("Show me my BQ jobs") + +```bash +# List recent jobs in the active project (auto-falls back to caller's jobs if not admin) +python3 scripts/gcp_job_monitor.py \ + --action=bq_list \ + --project="" \ + --limit=20 + +# Filter strictly to the authenticated user's own jobs +python3 scripts/gcp_job_monitor.py \ + --action=bq_list \ + --project="" \ + --mine \ + --limit=20 + +# Filter to failed jobs only (or --state=DONE, --state=RUNNING) +python3 scripts/gcp_job_monitor.py \ + --action=bq_list \ + --project="" \ + --state=FAILED \ + --limit=20 +``` + +### 2. Find Most Expensive BigQuery Jobs ("What jobs cost the most money?") + +Ranks jobs by **bytes billed** (and **total slot ms** as tie-breaker), reporting +both on-demand estimated cost (`$6.25 / TiB`) and total slot consumption for +capacity/edition workloads: + +```bash +python3 scripts/gcp_job_monitor.py \ + --action=bq_top_cost \ + --project="" \ + --limit=10 \ + --days=30 +``` + +### 3. Find Slowest / Longest-Running BigQuery Jobs ("What took long to run?") + +Ranks jobs by **wall-clock duration** (`endTime - startTime`) alongside slot +time so you can distinguish long wait/skew from heavy parallel compute: + +```bash +python3 scripts/gcp_job_monitor.py \ + --action=bq_slowest \ + --project="" \ + --limit=10 \ + --days=30 +``` + +### 4. Inspect a Specific BigQuery Job ("Give me details about job ``") + +Accepts `job_id`, `project:location.job_id`, or `location.job_id` and returns +full metadata, cost/slot metrics, referenced tables, error message (if failed), +full SQL text, and a per-stage `queryPlan` table (records read/written, slot +ms, max wait/compute, shuffle spill): + +```bash +python3 scripts/gcp_job_monitor.py \ + --action=bq_describe \ + --project="" \ + --job_id="" +``` + +### 5. Project-Wide Cost Attribution, Edition/Reservation Usage & Error Triage ("Break down spend by user / reservation" or "Why are queries failing?") + +Aggregates recent jobs into 3 admin tables: (1) **Cost & Slot Attribution by +User**, (2) **Usage by Edition & Slot Reservation** (`ON_DEMAND` vs. +`ENTERPRISE`), and (3) **Top Error Reasons** grouped by failure count: + +```bash +python3 scripts/gcp_job_monitor.py \ + --action=bq_summary \ + --project="" \ + --days=30 \ + --limit=10 +``` + +### 6. Most Queried / Highest-Cost Tables ("Which tables are costing the most money?") + +Unnests `referencedTables` across recent queries and ranks tables by total bytes +billed, estimated on-demand cost, total slot time, query count, and distinct +users: + +```bash +python3 scripts/gcp_job_monitor.py \ + --action=bq_hot_tables \ + --project="" \ + --days=30 \ + --limit=10 +``` + +### 7. Inspect BigQuery Slot Reservations & Capacity Commitments ("What slot reservations do we have?") + +Queries the BigQuery Reservation API (`bigqueryreservation.googleapis.com`) to +list configured slot reservations (edition, baseline slots, current/max +autoscale slots), capacity commitments, and active project assignments: + +```bash +python3 scripts/gcp_job_monitor.py \ + --action=bq_reservations \ + --project="" \ + --location=US +``` + +-------------------------------------------------------------------------------- + +## Workflow 2: Spark Workload Monitoring (`scripts/gcp_job_monitor.py`) + +Spark workloads are regional (`--region`, defaulting to `us-central1`). You can +pass a single region (`--region=us-central1`) or comma-separated regions in a +single call (`--region=us-central1,us-east1,us-west1,europe-west1`) to scan +multiple regions with one authentication token. + +### 1. List Recent Spark Workloads ("What Spark jobs have I run?") + +Queries both **Dataproc Serverless Batches** and **Dataproc Cluster Jobs**: + +```bash +# All recent Spark workloads in the region (or comma-separated regions) +python3 scripts/gcp_job_monitor.py \ + --action=spark_list \ + --project="" \ + --region=us-central1 \ + --limit=20 + +# Only Spark workloads submitted by the current authenticated user across common regions +python3 scripts/gcp_job_monitor.py \ + --action=spark_list \ + --project="" \ + --region=us-central1,us-east1,us-west1,europe-west1 \ + --mine \ + --limit=20 +``` + +### 2. Find Slowest / Most Resource-Intensive Spark Workloads + +Ranks Spark batches and cluster jobs by wall-clock duration and reports +approximate Serverless DCU-hours (`milliDcuSeconds`) and shuffle storage +(`shuffleStorageGbSeconds`): + +```bash +python3 scripts/gcp_job_monitor.py \ + --action=spark_slowest \ + --region=us-central1 \ + --limit=10 +``` + +### 3. Inspect a Specific Spark Batch or Cluster Job + +Automatically checks Dataproc Serverless Batches first and falls back to +Dataproc Cluster Jobs (or you can force `--workload_type=batch|cluster_job`): + +```bash +python3 scripts/gcp_job_monitor.py \ + --action=spark_describe \ + --region=us-central1 \ + --job_id="" +``` + +If the workload failed and the user wants a root-cause analysis of driver logs, +GCS event logs, or stage skew, load `@skill:gcp-spark-troubleshooting`. + +### 4. Inspect Active Dataproc Clusters & Interactive Spark Sessions ("Are any clusters or Spark sessions running?") + +Lists active/idle Dataproc clusters (master/worker machine types, worker count, +state) and Dataproc Serverless interactive sessions across one or more regions: + +```bash +python3 scripts/gcp_job_monitor.py \ + --action=spark_clusters \ + --project="" \ + --region=us-central1,us-east1,us-west1,europe-west1 +``` + +-------------------------------------------------------------------------------- + +## Workflow 3: Active / Stuck Queue Monitoring & Cancelling Runaway Jobs + +1. **Find Currently Running or Pending Jobs:** + ```bash + # BigQuery active/pending queue (computes live elapsed time since startTime) + python3 scripts/gcp_job_monitor.py --action=bq_list --project="" --state=RUNNING + python3 scripts/gcp_job_monitor.py --action=bq_list --project="" --state=PENDING + + # Spark active/pending batches and cluster jobs + python3 scripts/gcp_job_monitor.py --action=spark_list --project="" --region=us-central1 --state=RUNNING + ``` +2. **Cancel a Runaway Job (Requires Explicit User Confirmation First):** + - **BigQuery Job**: `bq cancel --location= ` (or via SQL: `CALL BQ.JOBS.CANCEL('.');`) + - **Dataproc Serverless Batch**: `gcloud dataproc batches cancel --project= --region= --quiet` + - **Dataproc Cluster Job**: `gcloud dataproc jobs kill --project= --region= --quiet` + +-------------------------------------------------------------------------------- + +## Workflow 4: Dynamic / Ad-Hoc Admin Queries (Any Unscripted Question) + +The preset `--action` commands above cover the most common admin views, but +admins will frequently ask **arbitrary, unscripted questions** (e.g., *"Which +queries spilled to disk?"*, *"What is our query cache hit rate by user?"*, +*"Show me all `DELETE` or `MERGE` DML jobs that modified table X"*, *"What time +of day has the highest slot spike?"*, *"Which jobs have label `env=prod`?"*, or +*"Are any datasets using physical storage billing?"*). + +Whenever a question requires custom filtering, custom grouping, or inspecting a +resource not covered by a preset table, **do not stop at the preset CLI flags**— +use one of these three dynamic patterns: + +### 1. Pipe `--format=json` into a Python Filter / Aggregation One-Liner +Every action in `scripts/gcp_job_monitor.py` supports `--format=json`, emitting +rich normalized objects (`bytesBilled`, `bytesProcessed`, `slotMs`, `cacheHit`, +`spilledBytes`, `maxWaitMs`, `biEngineMode`, `priority`, `labels`, +`statementType`, `destinationTable`, `numDmlAffectedRows`, `referencedTables`, +`stages`, `reservationId`, `edition`, `errorReason`, `errorMessage`, etc.): + +```bash +python3 scripts/gcp_job_monitor.py --action=bq_list --project="" --limit=200 --format=json | \ + python3 -c ' +import json, sys +data = json.load(sys.stdin) +jobs = data.get("jobs", []) +# Example: dynamically filter or group by any arbitrary condition +spilled = [j for j in jobs if j.get("spilledBytes", 0) > 0 or j.get("maxWaitMs", 0) > 5000] +print(json.dumps(spilled, indent=2)) +' +``` + +### 2. Import `gcp_job_monitor.RestClient` to Call Any GCP REST Endpoint +Because `gcp_job_monitor.RestClient` handles Context-Aware Access (CAA) token +fallback (`application-default` -> `user` credentials) automatically, you can +import it in a short `python3 -c` snippet to query **any** BigQuery or Dataproc +REST API endpoint (such as `datasets.list`, `tables.get`, `routines.list`, +`workflowTemplates.list`, or `autoscalingPolicies.list`) without writing auth +boilerplate: + +```bash +PYTHONPATH="$(dirname scripts/gcp_job_monitor.py)" python3 -c ' +import gcp_job_monitor as m, json +client = m.RestClient() +# Call any BigQuery or Dataproc REST endpoint with CAA-safe auth: +resp = client.get_json(f"{m.BQ_ENDPOINT}/bigquery/v2/projects//datasets") +print(json.dumps(resp, indent=2)) +' +``` + +### 3. Custom SQL on `INFORMATION_SCHEMA` (`references/information_schema_queries.md`) +When the user has `INFORMATION_SCHEMA` access (or when only the BigQuery MCP +tool `bigquery__execute_sql` is available) and needs 180-day historical +analysis, per-second slot timelines (`JOBS_TIMELINE_BY_PROJECT`), or storage +economics (`TABLE_STORAGE`), read `references/information_schema_queries.md` and +compose custom SQL against those views. diff --git a/skills/gcp-job-monitoring/references/information_schema_queries.md b/skills/gcp-job-monitoring/references/information_schema_queries.md new file mode 100644 index 00000000..8cabc59a --- /dev/null +++ b/skills/gcp-job-monitoring/references/information_schema_queries.md @@ -0,0 +1,310 @@ +# BigQuery `INFORMATION_SCHEMA` Job Monitoring Queries + +Use these SQL templates when querying historical BigQuery jobs via MCP +(`bigquery__execute_sql`) or when you need longer retention / SQL aggregations +beyond the last 1,000 jobs returned by `gcp_job_monitor.py`. + +## Crucial Gotchas + +1. **Regional Scoping is Mandatory (`region-`):** + `INFORMATION_SCHEMA.JOBS_BY_PROJECT` and `INFORMATION_SCHEMA.JOBS_BY_USER` + are **strictly regional**. You must prefix the view with a region qualifier + such as `` `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT `` or + `` `region-us-central1`.INFORMATION_SCHEMA.JOBS_BY_PROJECT ``. A query on + `region-us` will **not** return jobs executed in `us-central1` or `EU`. +2. **Permission Fallback (`JOBS_BY_PROJECT` vs. `JOBS_BY_USER` vs. REST API):** + - `JOBS_BY_PROJECT` requires `bigquery.jobs.listAll` (typically + `roles/bigquery.admin` or `roles/bigquery.resourceViewer`). + - `JOBS_BY_USER` requires `bigquery.jobs.list` and only shows the caller's + own jobs. + - In many enterprise sandbox projects, **both** `INFORMATION_SCHEMA.JOBS*` + views return `403 Access Denied` even when the user can run queries, + whereas `scripts/gcp_job_monitor.py` (which calls the `jobs.list` / + `jobs.get` REST API directly) succeeds across all regions at **$0 cost**. +3. **Partition Pruning (`creation_time`):** + Always filter on `creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL N DAY)` + to avoid scanning 180 days of metadata. + +--- + +## 1. Recent BigQuery Jobs (`JOBS_BY_USER` / `JOBS_BY_PROJECT`) + +```sql +SELECT + job_id, + job_type, + statement_type, + state, + user_email, + creation_time, + TIMESTAMP_DIFF(end_time, start_time, MILLISECOND) / 1000.0 AS duration_sec, + ROUND(total_bytes_billed / POW(1024, 3), 4) AS gb_billed, + total_slot_ms, + error_result.reason AS error_reason, + error_result.message AS error_message, + SUBSTR(query, 1, 200) AS query_preview +FROM + `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT +WHERE + creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY) +ORDER BY + creation_time DESC +LIMIT 25; +``` + +*(If `JOBS_BY_PROJECT` fails with `Access Denied`, replace `JOBS_BY_PROJECT` +with `JOBS_BY_USER` and remove `user_email` filtering, or run +`scripts/gcp_job_monitor.py --action=bq_list`.)* + +--- + +## 2. Most Expensive Jobs (On-Demand Bytes Billed & Est. USD) + +On-demand pricing is approximately **$6.25 per TiB (`POW(1024, 4)` bytes)** in +most regions. For capacity/edition (slot) workloads, rank by `total_slot_ms` +instead. + +```sql +SELECT + job_id, + user_email, + statement_type, + creation_time, + TIMESTAMP_DIFF(end_time, start_time, MILLISECOND) / 1000.0 AS duration_sec, + ROUND(total_bytes_billed / POW(1024, 3), 2) AS gib_billed, + ROUND((total_bytes_billed / POW(1024, 4)) * 6.25, 4) AS est_on_demand_usd, + total_slot_ms, + ROUND(total_slot_ms / 1000.0 / 3600.0, 4) AS slot_hours, + cache_hit, + SUBSTR(query, 1, 200) AS query_preview +FROM + `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT +WHERE + creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY) + AND job_type = 'QUERY' + AND state = 'DONE' +ORDER BY + total_bytes_billed DESC, + total_slot_ms DESC +LIMIT 20; +``` + +--- + +## 3. Slowest / Longest-Running Jobs + +Separating `total_slot_ms / NULLIF(TIMESTAMP_DIFF(end_time, start_time, MILLISECOND), 0)` +(`avg_slots`) helps distinguish queries that waited on slot contention or single-thread +skew (low `avg_slots`, high `duration_sec`) from massively parallel heavy queries +(high `avg_slots`, high `duration_sec`). + +```sql +SELECT + job_id, + user_email, + state, + statement_type, + creation_time, + TIMESTAMP_DIFF(end_time, start_time, MILLISECOND) / 1000.0 AS elapsed_sec, + total_slot_ms, + ROUND( + SAFE_DIVIDE( + total_slot_ms, + TIMESTAMP_DIFF(end_time, start_time, MILLISECOND) + ), + 2 + ) AS avg_parallel_slots, + ROUND(total_bytes_billed / POW(1024, 3), 2) AS gib_billed, + error_result.message AS error_message, + SUBSTR(query, 1, 200) AS query_preview +FROM + `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT +WHERE + creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY) +ORDER BY + elapsed_sec DESC +LIMIT 20; +``` + +--- + +## 4. Deep-Dive on a Specific Job ID (Stages, Timeline & Referenced Tables) + +```sql +SELECT + job_id, + user_email, + state, + statement_type, + creation_time, + start_time, + end_time, + TIMESTAMP_DIFF(end_time, start_time, MILLISECOND) / 1000.0 AS elapsed_sec, + total_bytes_processed, + total_bytes_billed, + total_slot_ms, + bi_engine_statistics.bi_engine_mode, + error_result, + referenced_tables, + job_stages, + timeline, + query +FROM + `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT +WHERE + creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY) + AND job_id = @job_id; +``` + +--- + +## 5. Cost & Slot Attribution by User & Reservation (`bq_summary` SQL Equivalent) + +```sql +SELECT + user_email, + COALESCE(edition, 'ON_DEMAND') AS edition, + COALESCE(reservation_id, '(none)') AS reservation_id, + COUNT(*) AS total_jobs, + COUNTIF(error_result IS NULL) AS succeeded_jobs, + COUNTIF(error_result IS NOT NULL) AS failed_jobs, + ROUND(SUM(total_bytes_billed) / POW(1024, 3), 2) AS total_gib_billed, + ROUND( + SUM(IF(reservation_id IS NULL, total_bytes_billed, 0)) / POW(1024, 4) * 6.25, + 4 + ) AS est_on_demand_usd, + ROUND(SUM(total_slot_ms) / 1000.0 / 3600.0, 4) AS total_slot_hours +FROM + `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT +WHERE + creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY) +GROUP BY + user_email, + edition, + reservation_id +ORDER BY + total_gib_billed DESC, + total_slot_hours DESC; +``` + +--- + +## 6. Failure Triage & Top Error Reasons + +```sql +SELECT + error_result.reason AS error_reason, + error_result.message AS sample_message, + COUNT(*) AS failure_count, + ARRAY_AGG(job_id ORDER BY creation_time DESC LIMIT 3) AS sample_job_ids +FROM + `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT +WHERE + creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY) + AND error_result IS NOT NULL +GROUP BY + error_reason, + sample_message +ORDER BY + failure_count DESC +LIMIT 20; +``` + +--- + +## 7. Most Queried / Highest-Cost Tables (`bq_hot_tables` SQL Equivalent) + +```sql +SELECT + CONCAT(rt.project_id, '.', rt.dataset_id, '.', rt.table_id) AS full_table_id, + COUNT(DISTINCT job_id) AS query_count, + COUNT(DISTINCT user_email) AS distinct_users, + ROUND(SUM(total_bytes_billed) / POW(1024, 3), 2) AS total_gib_billed, + ROUND((SUM(total_bytes_billed) / POW(1024, 4)) * 6.25, 4) AS est_on_demand_usd, + ROUND(SUM(total_slot_ms) / 1000.0, 2) AS total_slot_sec +FROM + `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT, + UNNEST(referenced_tables) AS rt +WHERE + creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY) + AND state = 'DONE' + AND error_result IS NULL +GROUP BY + full_table_id +ORDER BY + total_gib_billed DESC, + total_slot_sec DESC +LIMIT 20; +``` + +--- + +## 8. Slot Reservations, Capacity Commitments & Assignments (`bq_reservations` SQL Equivalent) + +```sql +SELECT + reservation_name, + edition, + slot_capacity AS baseline_slots, + autoscale.current_slots AS autoscale_current_slots, + autoscale.max_slots AS autoscale_max_slots, + ignore_idle_slots +FROM + `region-us`.INFORMATION_SCHEMA.RESERVATIONS; +``` + +--- + +## 9. Time-Series Slot Contention (`JOBS_TIMELINE_BY_PROJECT`) + +Use `INFORMATION_SCHEMA.JOBS_TIMELINE_BY_PROJECT` (1-second granularity) to +answer questions like *"What time of day did slot usage spike?"* or *"Were jobs +queued waiting for slots at 2pm?"*: + +```sql +SELECT + TIMESTAMP_TRUNC(period_start, MINUTE) AS minute_bucket, + reservation_id, + ROUND(SUM(period_slot_ms) / 1000.0 / 60.0, 2) AS avg_slots_used, + MAX(total_Slot_ms) AS peak_period_slot_ms, + MAX(pending_units) AS max_pending_work_units, + COUNT(DISTINCT job_id) AS active_jobs +FROM + `region-us`.INFORMATION_SCHEMA.JOBS_TIMELINE_BY_PROJECT +WHERE + job_creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY) + AND period_start >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY) +GROUP BY + minute_bucket, + reservation_id +ORDER BY + avg_slots_used DESC +LIMIT 30; +``` + +--- + +## 10. Dataset & Table Storage Cost Analysis (`TABLE_STORAGE`) + +Use `INFORMATION_SCHEMA.TABLE_STORAGE` to answer admin questions about active vs. +long-term logical and physical storage bytes or time-travel/fail-safe overhead: + +```sql +SELECT + table_schema AS dataset_name, + COUNT(*) AS table_count, + ROUND(SUM(total_logical_bytes) / POW(1024, 3), 2) AS total_logical_gib, + ROUND(SUM(active_logical_bytes) / POW(1024, 3), 2) AS active_logical_gib, + ROUND(SUM(long_term_logical_bytes) / POW(1024, 3), 2) AS long_term_logical_gib, + ROUND(SUM(total_physical_bytes) / POW(1024, 3), 2) AS total_physical_gib, + ROUND(SUM(time_travel_physical_bytes) / POW(1024, 3), 2) AS time_travel_physical_gib +FROM + `region-us`.INFORMATION_SCHEMA.TABLE_STORAGE +WHERE + deleted = FALSE +GROUP BY + dataset_name +ORDER BY + total_logical_gib DESC +LIMIT 20; +``` diff --git a/skills/gcp-job-monitoring/scripts/gcp_job_monitor.py b/skills/gcp-job-monitoring/scripts/gcp_job_monitor.py new file mode 100644 index 00000000..0fd8b7d0 --- /dev/null +++ b/skills/gcp-job-monitoring/scripts/gcp_job_monitor.py @@ -0,0 +1,2496 @@ +#!/usr/bin/env python3 +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""BigQuery and Spark (Dataproc) job monitoring, cost analysis, and inspection. + +Supports the core Data Cloud Admin & Practitioner job-monitoring CUJs: + 1. List recent BigQuery jobs (`bq_list`) + 2. Find top most expensive BigQuery jobs (`bq_top_cost`) + 3. Find slowest / longest-running BigQuery jobs (`bq_slowest`) + 4. Inspect details of a specific BigQuery job (`bq_describe`) + 5. List recent Spark jobs across Serverless Batches, Sessions, and Cluster + Jobs (`spark_list`) + 6. Find slowest / longest-running Spark jobs (`spark_slowest`) + 7. Inspect details of a specific Spark batch or cluster job (`spark_describe`) + +Why this script uses the BigQuery `jobs.list` (`projection=full`) & `jobs.get` +and Dataproc REST APIs directly: + * Querying `INFORMATION_SCHEMA.JOBS_BY_PROJECT` requires + `bigquery.jobs.listAll` and `JOBS_BY_USER` requires `bigquery.jobs.list` at + the project level. On projects where the user only has + `roles/bigquery.jobUser`, both `INFORMATION_SCHEMA` views fail with 403 + Access Denied, whereas `jobs.list?projection=full` succeeds for the user's + own jobs, costs $0 (metadata call, no query job billed), and spans all + regions in a single request. + * Under Context-Aware Access (CAA), `gcloud auth print-access-token` can fail + or return a certificate-bound token rejected on standard HTTPS calls; this + script automatically falls back to Application Default Credentials + (`gcloud auth application-default print-access-token`). + +Written against the Python standard library only. +""" + +import argparse +import datetime +import json +import os +import re +import shutil +import ssl +import subprocess +import sys +import time +from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple +import urllib.error +import urllib.parse +import urllib.request + +BQ_ENDPOINT = "https://bigquery.googleapis.com" +BQ_RESERVATION_ENDPOINT = "https://bigqueryreservation.googleapis.com" +DATAPROC_ENDPOINT = "https://dataproc.googleapis.com" +DEFAULT_REGION = "us-central1" +ON_DEMAND_USD_PER_TIB = 6.25 +TIB_BYTES = 1024**4 + +_HTTP_TIMEOUT_SECONDS = 60 +_GCLOUD_TIMEOUT_SECONDS = 30 +_USER_AGENT = "data-agent-kit/gcp-job-monitoring (script:gcp_job_monitor)" + + +class JobMonitorError(Exception): + """Raised when a GCP CLI or REST API operation fails.""" + + def __init__(self, message: str, status: int = 0): + super().__init__(message) + self.status = status + + +# ------------------------------------------------------------------------------ +# Formatting & time helpers +# ------------------------------------------------------------------------------ + + +def human_bytes(num: Any) -> str: + """Formats a byte count in binary units (B, KiB, MiB, GiB, TiB).""" + try: + value = float(num or 0) + except (TypeError, ValueError): + return "0 B" + if value <= 0: + return "0 B" + for unit in ("B", "KiB", "MiB", "GiB", "TiB"): + if value < 1024 or unit == "TiB": + return f"{value:.0f} {unit}" if unit == "B" else f"{value:.2f} {unit}" + value /= 1024 + return f"{value:.2f} TiB" + + +def human_millis(ms: Any) -> str: + """Formats milliseconds into human-readable ms, s, min, or hr.""" + if ms is None: + return "n/a" + try: + value = float(ms) + except (TypeError, ValueError): + return "n/a" + if value < 0: + return "n/a" + if value < 1000: + return f"{value:.0f} ms" + if value < 60_000: + return f"{value / 1000:.1f} s" + if value < 3_600_000: + return f"{value / 60_000:.1f} min" + return f"{value / 3_600_000:.2f} hr" + + +def estimate_on_demand_usd(bytes_billed: int) -> float: + """Estimates BigQuery on-demand query cost in USD ($6.25 per TiB).""" + if bytes_billed <= 0: + return 0.0 + return (bytes_billed / TIB_BYTES) * ON_DEMAND_USD_PER_TIB + + +def format_usd(amount: float) -> str: + """Formats a USD estimate cleanly, showing sub-cent precision when small.""" + if amount <= 0: + return "$0.00" + if amount < 0.01: + return f"${amount:.6f}" + return f"${amount:.4f}" if amount < 1.0 else f"${amount:.2f}" + + +def epoch_ms_to_iso(ms: Any) -> str: + """Converts epoch milliseconds (int or string) to an ISO-8601 UTC string.""" + try: + val = int(ms) + except (TypeError, ValueError): + return "n/a" + if val <= 0: + return "n/a" + dt = datetime.datetime.fromtimestamp(val / 1000.0, tz=datetime.timezone.utc) + return dt.strftime("%Y-%m-%d %H:%M:%S UTC") + + +def parse_rfc3339(value: Any) -> Optional[datetime.datetime]: + """Parses an RFC-3339 timestamp string into a UTC datetime.""" + if not isinstance(value, str) or not value: + return None + cleaned = value.rstrip("Z") + if "+" not in cleaned[10:] and "-" not in cleaned[10:]: + cleaned += "+00:00" + try: + return datetime.datetime.fromisoformat(cleaned) + except ValueError: + return None + + +def truncate_one_line(text: Any, limit: int = 100) -> str: + """Collapses whitespace to a single line and truncates to `limit` chars.""" + collapsed = " ".join(str(text or "").split()) + if len(collapsed) <= limit: + return collapsed + return collapsed[: limit - 3] + "..." + + +# ------------------------------------------------------------------------------ +# Authentication & HTTP Client +# ------------------------------------------------------------------------------ + + +class GcloudTokenProvider: + """Mints OAuth access tokens via gcloud with ADC fallback for CAA.""" + + _COMMANDS = ( + ("auth", "application-default", "print-access-token"), + ("auth", "print-access-token"), + ) + + def __init__(self): + self._index = 0 + self._token: Optional[str] = None + self._errors: List[str] = [] + + def token(self) -> str: + while self._token is None: + try: + self._token = _run_gcloud(self._COMMANDS[self._index]) + except JobMonitorError as e: + self._errors.append(str(e)) + if not self.fallback(): + raise JobMonitorError( + "Could not obtain a Google Cloud access token. " + + " | ".join(self._errors) + + " Run 'gcloud auth login' and 'gcloud auth" + " application-default login'." + ) from e + return self._token + + def fallback(self) -> bool: + if self._index + 1 >= len(self._COMMANDS): + return False + self._index += 1 + self._token = None + return True + + +def _run_gcloud(args: Sequence[str]) -> str: + gcloud = shutil.which("gcloud") + if not gcloud: + raise JobMonitorError("gcloud CLI is not installed or not on PATH.") + try: + proc = subprocess.run( + [gcloud, "--quiet", *args], + capture_output=True, + text=True, + check=False, + timeout=_GCLOUD_TIMEOUT_SECONDS, + stdin=subprocess.DEVNULL, + ) + except (OSError, subprocess.SubprocessError) as e: + raise JobMonitorError(f"gcloud {' '.join(args)} failed: {e}") from e + out = (proc.stdout or "").strip() + if proc.returncode != 0 or not out: + err_lines = (proc.stderr or "").strip().splitlines() + detail = err_lines[-1] if err_lines else f"exit code {proc.returncode}" + raise JobMonitorError(f"gcloud {' '.join(args)} failed: {detail}") + return out + + +def resolve_default_project(explicit: Optional[str] = None) -> str: + if explicit: + return explicit + for env_key in ( + "GCP_MCP_QUOTA_PROJECT", + "GOOGLE_CLOUD_PROJECT", + "CLOUDSDK_CORE_PROJECT", + ): + env_val = os.environ.get(env_key, "").strip() + if env_val and env_val != "(unset)": + return env_val + try: + val = _run_gcloud(("config", "get-value", "project")) + except JobMonitorError: + val = "" + if not val or val == "(unset)": + raise JobMonitorError( + "No GCP project specified and gcloud default project is unset. " + "Pass --project=." + ) + return val + + +def resolve_default_account() -> str: + try: + val = _run_gcloud(("config", "get-value", "account")) + return "" if val == "(unset)" else val + except JobMonitorError: + return "" + + +class RemoteCliMcpClient: + """Sobi VMaaS / ACE sandbox client that routes API calls via `remote_cli mcp`. + + In the Gemini Enterprise (GE) + Sobi VMaaS sandbox architecture, Python and + local `gcloud`/`bq` have no direct GCP credentials; authenticated egress to + BigQuery and Dataproc goes through `/usr/local/bin/remote_cli mcp` + (`bigquery`, + `google-managed-apache-spark`, and optionally `google-cloud-cli`). This client + implements the same `get_json(url, params)` interface as `RestClient` so all + job normalization, aggregation, and rendering functions work unchanged. + """ + + def __init__( + self, + runner: Optional[Callable[[Sequence[str]], Tuple[int, str, str]]] = None, + billing_project: Optional[str] = None, + ): + self._runner = runner or self._default_runner + self._billing_project = ( + billing_project + or os.environ.get("GCP_MCP_QUOTA_PROJECT", "").strip() + or os.environ.get("GOOGLE_CLOUD_PROJECT", "").strip() + ) + + @staticmethod + def is_available() -> bool: + return bool( + shutil.which("remote_cli") + or os.path.exists("/usr/local/bin/remote_cli") + ) + + @staticmethod + def _default_runner(cmd: Sequence[str]) -> Tuple[int, str, str]: + try: + proc = subprocess.run( + list(cmd), + capture_output=True, + text=True, + check=False, + timeout=_HTTP_TIMEOUT_SECONDS, + stdin=subprocess.DEVNULL, + ) + return proc.returncode, proc.stdout or "", proc.stderr or "" + except (OSError, subprocess.SubprocessError) as e: + raise JobMonitorError(f"remote_cli execution failed: {e}") from e + + def call_tool( + self, + server: str, + tool: str, + args: Dict[str, Any], + endpoint: Optional[str] = None, + ) -> Dict[str, Any]: + """Invokes `remote_cli mcp --json ` and returns JSON.""" + remote_cli = shutil.which("remote_cli") or "/usr/local/bin/remote_cli" + filtered = {k: v for k, v in args.items() if v is not None and v != ""} + effective_endpoint = endpoint or ( + "https://cloudcli.googleapis.com/mcp/readonly" + if server == "google-cloud-cli" + else None + ) + cmd = [remote_cli, "mcp", server, tool] + if effective_endpoint: + cmd.extend(["--endpoint", effective_endpoint]) + cmd.extend(["--json", json.dumps(filtered)]) + rc, stdout, stderr = self._runner(cmd) + combined = f"{stdout}\n{stderr}".strip() + for line in combined.splitlines(): + if line.startswith("remote_cli:"): + status = 500 + if "Unauthorized" in line or "UNAUTHENTICATED" in line: + status = 401 + elif "PERMISSION_DENIED" in line or "Access Denied" in line: + status = 403 + elif "NOT_FOUND" in line or "404" in line: + status = 404 + raise JobMonitorError(line, status=status) + if rc != 0 and not stdout.strip(): + raise JobMonitorError( + f"remote_cli mcp {server} {tool} failed (exit {rc}): {combined}", + status=500, + ) + return self._unwrap_mcp_payload(stdout.strip() or "{}") + + def _unwrap_mcp_payload(self, raw_text: str) -> Dict[str, Any]: + try: + payload = json.loads(raw_text) + except ValueError as e: + raise JobMonitorError( + "Invalid JSON from remote_cli mcp:" + f" {truncate_one_line(raw_text, 200)}" + ) from e + if isinstance(payload, list): + return {"items": payload} + if not isinstance(payload, dict): + return {} + if payload.get("isError"): + res_val = payload.get("result") + msg = ( + self._extract_mcp_text(payload) + or (res_val if isinstance(res_val, str) else "") + or str(payload.get("error") or "") + or json.dumps(payload) + ) + status = 400 + if "403" in msg or "Access Denied" in msg or "PERMISSION_DENIED" in msg: + status = 403 + elif "404" in msg or "Not found" in msg or "NOT_FOUND" in msg: + status = 404 + elif "401" in msg or "Unauthorized" in msg or "UNAUTHENTICATED" in msg: + status = 401 + raise JobMonitorError(msg, status=status) + if isinstance(payload.get("result"), dict): + inner_res = payload["result"] + if isinstance(inner_res.get("structuredContent"), dict): + return inner_res["structuredContent"] + return inner_res + if isinstance(payload.get("result"), list): + return {"items": payload["result"]} + if isinstance(payload.get("structuredContent"), dict): + return payload["structuredContent"] + inner_text = self._extract_mcp_text(payload) + if inner_text: + try: + parsed_inner = json.loads(inner_text) + if isinstance(parsed_inner, dict): + return parsed_inner + if isinstance(parsed_inner, list): + return {"items": parsed_inner} + except ValueError: + pass + return payload + + @staticmethod + def _extract_mcp_text(payload: Dict[str, Any]) -> str: + content = payload.get("content") + if isinstance(content, list): + texts = [ + str(item.get("text") or "") + for item in content + if isinstance(item, dict) and item.get("text") + ] + return "\n".join(texts).strip() + return "" + + def get_json( + self, url: str, params: Optional[Dict[str, Any]] = None + ) -> Dict[str, Any]: + """Routes REST URLs to the corresponding `remote_cli mcp` connector tools.""" + parsed = urllib.parse.urlparse(url) + path = urllib.parse.unquote(parsed.path) + merged_params: Dict[str, Any] = {} + for k, v in urllib.parse.parse_qsl(parsed.query): + merged_params[k] = v + if params: + for k, v in params.items(): + if v is not None and v != "": + merged_params[k] = v + + # 1. Dataproc Serverless Batches / Sessions (`locations/{region}/...`) + m = re.match( + r"^/v1/projects/([^/]+)/locations/([^/]+)/(batches|sessions)(?:/([^/]+))?$", + path, + ) + if m: + project, region, resource, item_id = m.groups() + if resource == "batches" and item_id: + return self.call_tool( + "google-managed-apache-spark", + "get_batch", + {"project_id": project, "region": region, "batch_id": item_id}, + ) + if resource == "batches": + args: Dict[str, Any] = {"project_id": project, "region": region} + if merged_params.get("pageSize"): + args["page_size"] = int(merged_params["pageSize"]) + return self.call_tool( + "google-managed-apache-spark", "list_batches", args + ) + if resource == "sessions": + return self.call_tool( + "google-managed-apache-spark", + "list_sessions", + {"project_id": project, "region": region}, + ) + + # 2. Dataproc Cluster Jobs / Clusters (`regions/{region}/...`) + m = re.match( + r"^/v1/projects/([^/]+)/regions/([^/]+)/(jobs|clusters)(?:/([^/]+))?$", + path, + ) + if m: + project, region, resource, item_id = m.groups() + if resource == "jobs" and item_id: + return self.call_tool( + "google-managed-apache-spark", + "get_job", + {"project_id": project, "region": region, "job_id": item_id}, + ) + if resource == "jobs": + args = {"project_id": project, "region": region} + if merged_params.get("pageSize"): + args["page_size"] = int(merged_params["pageSize"]) + return self.call_tool("google-managed-apache-spark", "list_jobs", args) + if resource == "clusters": + return self.call_tool( + "google-managed-apache-spark", + "list_clusters", + {"project_id": project, "region": region}, + ) + + # 3. BigQuery Job Detail (`/bigquery/v2/projects/{project}/jobs/{job_id}`) + m = re.match(r"^/bigquery/v2/projects/([^/]+)/jobs/([^/]+)$", path) + if m: + project, job_id = m.groups() + loc = str(merged_params.get("location") or "US") + try: + return self.call_tool( + "bigquery", + "get_job", + {"project_id": project, "job_id": job_id, "location": loc}, + ) + except JobMonitorError as first_err: + if not merged_params.get("location") and first_err.status == 404: + for alt_loc in ("us-central1", "EU", "us-east1", "us-west1"): + try: + return self.call_tool( + "bigquery", + "get_job", + { + "project_id": project, + "job_id": job_id, + "location": alt_loc, + }, + ) + except JobMonitorError: + continue + if first_err.status in (403, 404): + sql_job = self._bq_get_job_via_sql(project, job_id, loc) + if sql_job is not None: + return sql_job + raise + + # 4. BigQuery Jobs List (`/bigquery/v2/projects/{project}/jobs`) + m = re.match(r"^/bigquery/v2/projects/([^/]+)/jobs$", path) + if m: + project = m.group(1) + return self._bq_list_jobs(project, merged_params) + + # 5. BigQuery Reservations (`/v1/projects/{project}/locations/{loc}/...`) + m = re.match( + r"^/v1/projects/([^/]+)/locations/([^/]+)/(reservations|capacityCommitments|reservations/-/assignments)$", + path, + ) + if m: + project, loc, resource = m.groups() + return self._bq_reservations(project, loc, resource) + + raise JobMonitorError( + f"Unsupported URL for RemoteCliMcpClient: {url}", status=400 + ) + + def _bq_get_job_via_sql( + self, project: str, job_id: str, loc: str = "US" + ) -> Optional[Dict[str, Any]]: + """Fetches a single BigQuery job via `execute_sql_readonly` using `_billing_project`.""" + billing_project = self._billing_project or project + safe_job_id = job_id.replace("'", "''") + regions = [loc] + [ + r for r in ("US", "us-central1", "EU") if r.lower() != loc.lower() + ] + for reg in regions: + for view_name in ("JOBS_BY_PROJECT", "JOBS_BY_USER"): + sql = ( + "SELECT job_id, project_id, user_email, job_type, statement_type," + " state, priority, reservation_id, edition," + " CAST(UNIX_MILLIS(creation_time) AS STRING) AS creation_ms," + " CAST(UNIX_MILLIS(start_time) AS STRING) AS start_ms," + " CAST(UNIX_MILLIS(end_time) AS STRING) AS end_ms," + " CAST(total_bytes_billed AS STRING) AS total_bytes_billed," + " CAST(total_bytes_processed AS STRING) AS total_bytes_processed," + " CAST(total_slot_ms AS STRING) AS total_slot_ms, cache_hit," + " query, error_result.reason AS error_reason, error_result.message" + " AS error_message, TO_JSON_STRING(referenced_tables) AS" + " referenced_tables_json, TO_JSON_STRING(labels) AS labels_json," + " TO_JSON_STRING(job_stages) AS job_stages_json FROM" + f" `{project}`.`region-{reg.lower()}`.INFORMATION_SCHEMA.{view_name}" + f" WHERE job_id = '{safe_job_id}' LIMIT 1" + ) + try: + resp = self.call_tool( + "bigquery", + "execute_sql_readonly", + {"project_id": billing_project, "query": sql}, + ) + jobs = self._rows_to_bq_jobs(resp, project=project, location=reg) + if jobs: + return jobs[0] + break + except JobMonitorError as e: + if e.status == 403 and view_name == "JOBS_BY_PROJECT": + continue + break + return None + + def _bq_list_jobs( + self, project: str, params: Dict[str, Any] + ) -> Dict[str, Any]: + """Lists BigQuery jobs via `bigquery` MCP SQL or `google-cloud-cli` `bq ls -j`.""" + max_results = int(params.get("maxResults") or 100) + all_users = str(params.get("allUsers") or "").lower() == "true" + min_creation_ms = int(params.get("minCreationTime") or 0) + state_filter = str(params.get("stateFilter") or "").strip().upper() + view_name = "JOBS_BY_PROJECT" if all_users else "JOBS_BY_USER" + + where_clauses = [] + if min_creation_ms > 0: + where_clauses.append( + f"creation_time >= TIMESTAMP_MILLIS({min_creation_ms})" + ) + else: + where_clauses.append( + "creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)" + ) + if state_filter: + where_clauses.append(f"state = '{state_filter}'") + where_sql = " AND ".join(where_clauses) + + sql = ( + "SELECT " + "job_id, project_id, user_email, job_type, statement_type, state, " + "priority, reservation_id, edition, " + "CAST(UNIX_MILLIS(creation_time) AS STRING) AS creation_ms, " + "CAST(UNIX_MILLIS(start_time) AS STRING) AS start_ms, " + "CAST(UNIX_MILLIS(end_time) AS STRING) AS end_ms, " + "CAST(total_bytes_billed AS STRING) AS total_bytes_billed, " + "CAST(total_bytes_processed AS STRING) AS total_bytes_processed, " + "CAST(total_slot_ms AS STRING) AS total_slot_ms, " + "cache_hit, query, " + "error_result.reason AS error_reason, " + "error_result.message AS error_message, " + "TO_JSON_STRING(referenced_tables) AS referenced_tables_json, " + "TO_JSON_STRING(labels) AS labels_json " + f"FROM `{project}`.`region-us`.INFORMATION_SCHEMA.{view_name} " + f"WHERE {where_sql} " + f"ORDER BY creation_time DESC LIMIT {max_results}" + ) + billing_project = self._billing_project or project + try: + resp = self.call_tool( + "bigquery", + "execute_sql_readonly", + {"project_id": billing_project, "query": sql}, + ) + return { + "jobs": self._rows_to_bq_jobs(resp, project=project, location="US") + } + except JobMonitorError as sql_err: + if sql_err.status == 403 and all_users: + raise + # Fallback to google-cloud-cli `bq ls -j` if available + try: + all_flag = " -a" if all_users else "" + cmd_str = ( + f"bq --format=prettyjson ls -j{all_flag} -n {max_results}" + f" --project_id={project}" + ) + cli_resp = self.call_tool( + "google-cloud-cli", + "run_bq_command", + {"command": cmd_str, "project": f"projects/{billing_project}"}, + ) + return self._parse_bq_cli_jobs(cli_resp) + except JobMonitorError: + raise sql_err + + @staticmethod + def _rows_to_bq_jobs( + resp: Dict[str, Any], project: str, location: str = "US" + ) -> List[Dict[str, Any]]: + """Converts `execute_sql_readonly` rows into `jobs.list` Job resource dicts.""" + raw_rows: List[Dict[str, Any]] = [] + if isinstance(resp.get("rows"), list): + schema_fields = ( + (resp.get("schema") or {}).get("fields") or resp.get("columns") or [] + ) + col_names = [ + str(f.get("name") if isinstance(f, dict) else f) + for f in schema_fields + ] + for r in resp["rows"]: + if isinstance(r, dict) and "f" in r and col_names: + vals = [ + cell.get("v") if isinstance(cell, dict) else cell + for cell in r["f"] + ] + raw_rows.append(dict(zip(col_names, vals))) + elif isinstance(r, dict): + raw_rows.append(r) + elif isinstance(resp.get("items"), list): + raw_rows = [r for r in resp["items"] if isinstance(r, dict)] + + jobs: List[Dict[str, Any]] = [] + for row in raw_rows: + refs_raw: List[Dict[str, Any]] = [] + refs_json = row.get("referenced_tables_json") + if isinstance(refs_json, str) and refs_json: + try: + parsed_refs = json.loads(refs_json) + if isinstance(parsed_refs, list): + for t in parsed_refs: + if isinstance(t, dict): + refs_raw.append({ + "projectId": ( + t.get("project_id") or t.get("projectId") or "" + ), + "datasetId": ( + t.get("dataset_id") or t.get("datasetId") or "" + ), + "tableId": t.get("table_id") or t.get("tableId") or "", + }) + except ValueError: + pass + labels_dict: Dict[str, str] = {} + labels_json = row.get("labels_json") + if isinstance(labels_json, str) and labels_json: + try: + parsed_lbls = json.loads(labels_json) + if isinstance(parsed_lbls, list): + for item in parsed_lbls: + if isinstance(item, dict) and item.get("key"): + labels_dict[str(item["key"])] = str(item.get("value") or "") + except ValueError: + pass + stages_raw: List[Dict[str, Any]] = [] + stages_json = row.get("job_stages_json") + if isinstance(stages_json, str) and stages_json: + try: + parsed_stages = json.loads(stages_json) + if isinstance(parsed_stages, list): + for s in parsed_stages: + if isinstance(s, dict): + stages_raw.append({ + "id": str(s.get("id") or ""), + "name": str(s.get("name") or ""), + "status": str(s.get("status") or ""), + "recordsRead": str( + s.get("records_read") or s.get("recordsRead") or "0" + ), + "recordsWritten": str( + s.get("records_written") + or s.get("recordsWritten") + or "0" + ), + "slotMs": str(s.get("slot_ms") or s.get("slotMs") or "0"), + "waitMsMax": str( + s.get("wait_ms_max") or s.get("waitMsMax") or "0" + ), + "computeMsMax": str( + s.get("compute_ms_max") or s.get("computeMsMax") or "0" + ), + "shuffleOutputBytes": str( + s.get("shuffle_output_bytes") + or s.get("shuffleOutputBytes") + or "0" + ), + "shuffleOutputBytesSpilled": str( + s.get("shuffle_output_bytes_spilled") + or s.get("shuffleOutputBytesSpilled") + or "0" + ), + }) + except ValueError: + pass + err_reason = str(row.get("error_reason") or "") + err_msg = str(row.get("error_message") or "") + err_result = ( + {"reason": err_reason, "message": err_msg} + if (err_reason or err_msg) + else {} + ) + jobs.append({ + "jobReference": { + "projectId": str(row.get("project_id") or project), + "jobId": str(row.get("job_id") or ""), + "location": location, + }, + "user_email": str(row.get("user_email") or ""), + "state": str(row.get("state") or "DONE"), + "status": { + "state": str(row.get("state") or "DONE"), + "errorResult": err_result, + }, + "configuration": { + "jobType": str(row.get("job_type") or "QUERY"), + "labels": labels_dict, + "query": { + "query": str(row.get("query") or ""), + "priority": str(row.get("priority") or "INTERACTIVE"), + "statementType": str(row.get("statement_type") or ""), + }, + }, + "statistics": { + "creationTime": str(row.get("creation_ms") or "0"), + "startTime": str(row.get("start_ms") or "0"), + "endTime": str(row.get("end_ms") or "0"), + "totalBytesProcessed": str( + row.get("total_bytes_processed") or "0" + ), + "totalSlotMs": str(row.get("total_slot_ms") or "0"), + "reservation_id": str(row.get("reservation_id") or ""), + "edition": str(row.get("edition") or ""), + "query": { + "totalBytesBilled": str(row.get("total_bytes_billed") or "0"), + "totalBytesProcessed": str( + row.get("total_bytes_processed") or "0" + ), + "totalSlotMs": str(row.get("total_slot_ms") or "0"), + "cacheHit": ( + str(row.get("cache_hit") or "").lower() in ("true", "1") + ), + "statementType": str(row.get("statement_type") or ""), + "referencedTables": refs_raw, + "queryPlan": stages_raw, + }, + }, + }) + return jobs + + @staticmethod + def _parse_bq_cli_jobs(cli_resp: Dict[str, Any]) -> Dict[str, Any]: + stdout_text = str(cli_resp.get("stdout") or "") + if stdout_text.strip(): + try: + parsed = json.loads(stdout_text) + if isinstance(parsed, list): + return {"jobs": parsed} + except ValueError: + pass + if isinstance(cli_resp.get("items"), list): + return {"jobs": cli_resp["items"]} + return {"jobs": []} + + def _bq_reservations( + self, project: str, loc: str, resource: str + ) -> Dict[str, Any]: + """Queries BigQuery Reservation metadata via `INFORMATION_SCHEMA`.""" + billing_project = self._billing_project or project + region_qualifier = f"`{project}`.`region-{loc.lower()}`" + if resource == "reservations": + sql = ( + "SELECT reservation_name, edition, slot_capacity, " + "autoscale.current_slots AS current_slots, " + "autoscale.max_slots AS max_slots, ignore_idle_slots, concurrency " + f"FROM {region_qualifier}.INFORMATION_SCHEMA.RESERVATIONS" + ) + resp = self.call_tool( + "bigquery", + "execute_sql_readonly", + {"project_id": billing_project, "query": sql}, + ) + rows = self._extract_sql_dicts(resp) + return { + "reservations": [ + { + "name": str(r.get("reservation_name") or ""), + "edition": str(r.get("edition") or "STANDARD"), + "slotCapacity": int(r.get("slot_capacity") or 0), + "autoscale": { + "currentSlots": int(r.get("current_slots") or 0), + "maxSlots": int(r.get("max_slots") or 0), + }, + "ignoreIdleSlots": ( + str(r.get("ignore_idle_slots") or "").lower() + in ("true", "1") + ), + "concurrency": int(r.get("concurrency") or 0), + } + for r in rows + ] + } + if resource == "capacityCommitments": + sql = ( + "SELECT capacity_commitment_id, edition, slot_count, " + "commitment_plan, state " + f"FROM {region_qualifier}.INFORMATION_SCHEMA.CAPACITY_COMMITMENTS" + ) + resp = self.call_tool( + "bigquery", + "execute_sql_readonly", + {"project_id": billing_project, "query": sql}, + ) + rows = self._extract_sql_dicts(resp) + return { + "capacityCommitments": [ + { + "name": str(r.get("capacity_commitment_id") or ""), + "edition": str(r.get("edition") or ""), + "slotCount": int(r.get("slot_count") or 0), + "plan": str(r.get("commitment_plan") or ""), + "state": str(r.get("state") or ""), + } + for r in rows + ] + } + sql = ( + "SELECT assignment_id, reservation_name, assignee_name, job_type " + f"FROM {region_qualifier}.INFORMATION_SCHEMA.ASSIGNMENTS" + ) + resp = self.call_tool( + "bigquery", + "execute_sql_readonly", + {"project_id": billing_project, "query": sql}, + ) + rows = self._extract_sql_dicts(resp) + return { + "assignments": [ + { + "name": ( + f"reservations/{r.get('reservation_name', '')}/assignments/{r.get('assignment_id', '')}" + ), + "assignee": str(r.get("assignee_name") or ""), + "jobType": str(r.get("job_type") or ""), + "state": "ACTIVE", + } + for r in rows + ] + } + + @staticmethod + def _extract_sql_dicts(resp: Dict[str, Any]) -> List[Dict[str, Any]]: + out: List[Dict[str, Any]] = [] + if isinstance(resp.get("rows"), list): + schema_fields = ( + (resp.get("schema") or {}).get("fields") or resp.get("columns") or [] + ) + col_names = [ + str(f.get("name") if isinstance(f, dict) else f) + for f in schema_fields + ] + for r in resp["rows"]: + if isinstance(r, dict) and "f" in r and col_names: + vals = [ + cell.get("v") if isinstance(cell, dict) else cell + for cell in r["f"] + ] + out.append(dict(zip(col_names, vals))) + elif isinstance(r, dict): + out.append(r) + elif isinstance(resp.get("items"), list): + out = [r for r in resp["items"] if isinstance(r, dict)] + return out + + +class RestClient: + """Minimal standard-library JSON REST client with auth and Sobi MCP fallback.""" + + def __init__( + self, + token_provider: Optional[GcloudTokenProvider] = None, + urlopen: Optional[Callable[..., Any]] = None, + remote_cli_client: Optional[RemoteCliMcpClient] = None, + ): + self._explicit_tokens = token_provider is not None + self._tokens = token_provider or GcloudTokenProvider() + self._urlopen = urlopen or urllib.request.urlopen + self._remote_cli = remote_cli_client + + def get_json( + self, url: str, params: Optional[Dict[str, Any]] = None + ) -> Dict[str, Any]: + """Performs an authenticated GET request and returns decoded JSON.""" + if self._remote_cli is not None: + return self._remote_cli.get_json(url, params) + req_url = url + if params: + filtered = {k: v for k, v in params.items() if v is not None and v != ""} + if filtered: + req_url = f"{url}?{urllib.parse.urlencode(filtered, doseq=True)}" + while True: + try: + token = self._tokens.token() + except JobMonitorError: + if not self._explicit_tokens and RemoteCliMcpClient.is_available(): + self._remote_cli = RemoteCliMcpClient() + return self._remote_cli.get_json(url, params) + raise + req = urllib.request.Request( + req_url, + headers={ + "Authorization": f"Bearer {token}", + "Accept": "application/json", + "User-Agent": _USER_AGENT, + }, + ) + try: + ctx = ssl.create_default_context() + with self._urlopen( + req, timeout=_HTTP_TIMEOUT_SECONDS, context=ctx + ) as r: + raw = r.read().decode("utf-8", errors="replace") + return json.loads(raw) if raw.strip() else {} + except urllib.error.HTTPError as e: + body = "" + try: + body = e.read().decode("utf-8", errors="replace") + except Exception: # pylint: disable=broad-except + pass + if e.code == 401 and self._tokens.fallback(): + continue + msg = _extract_api_error(body) or f"HTTP {e.code}: {e.reason}" + raise JobMonitorError(msg, status=e.code) from e + except urllib.error.URLError as e: + raise JobMonitorError(f"Network error calling {req_url}: {e}") from e + + +def _extract_api_error(body: str) -> str: + try: + payload = json.loads(body) + if isinstance(payload, dict) and isinstance(payload.get("error"), dict): + return str(payload["error"].get("message") or "") + except ValueError: + pass + return "" + + +# ------------------------------------------------------------------------------ +# BigQuery Job Normalization & Actions +# ------------------------------------------------------------------------------ + + +def normalize_bq_job(raw: Dict[str, Any]) -> Dict[str, Any]: + """Extracts a consistent summary dict from a BigQuery Job resource.""" + ref = raw.get("jobReference") or {} + stats = raw.get("statistics") or {} + qstats = stats.get("query") or {} + config = raw.get("configuration") or {} + qconfig = config.get("query") or {} + status = raw.get("status") or {} + err = status.get("errorResult") or raw.get("errorResult") or {} + errors_list = status.get("errors") or [] + first_err = ( + errors_list[0] + if (errors_list and isinstance(errors_list[0], dict)) + else {} + ) + + creation_ms = int(stats.get("creationTime") or 0) + start_ms = int(stats.get("startTime") or 0) + end_ms = int(stats.get("endTime") or 0) + raw_state = str(status.get("state") or raw.get("state") or "UNKNOWN") + if ( + raw_state in ("RUNNING", "PENDING") + and end_ms == 0 + and (start_ms > 0 or creation_ms > 0) + ): + end_ms = int(time.time() * 1000) + if start_ms == 0: + start_ms = creation_ms + duration_ms = ( + (end_ms - start_ms) + if (end_ms > 0 and start_ms > 0 and end_ms >= start_ms) + else 0 + ) + + bytes_billed = int(qstats.get("totalBytesBilled") or 0) + bytes_processed = int( + qstats.get("totalBytesProcessed") or stats.get("totalBytesProcessed") or 0 + ) + slot_ms = int(qstats.get("totalSlotMs") or stats.get("totalSlotMs") or 0) + cache_hit = bool(qstats.get("cacheHit", False)) + + # Inspect queryPlan stages for spill or high wait ratio. + stages = qstats.get("queryPlan") or [] + spilled_bytes = sum( + int(s.get("shuffleOutputBytesSpilled") or 0) for s in stages + ) + max_wait_ms = max((int(s.get("waitMsMax") or 0) for s in stages), default=0) + + bi_stats = qstats.get("biEngineStatistics") or {} + bi_engine_mode = str(bi_stats.get("biEngineMode") or "") + labels = ( + config.get("labels") if isinstance(config.get("labels"), dict) else {} + ) + priority = str(qconfig.get("priority") or "INTERACTIVE") + dest_tbl = qconfig.get("destinationTable") or {} + dest_table_str = ( + f"{dest_tbl.get('projectId', '')}.{dest_tbl.get('datasetId', '')}.{dest_tbl.get('tableId', '')}" + if isinstance(dest_tbl, dict) and dest_tbl.get("tableId") + else "" + ) + num_dml_affected = int(qstats.get("numDmlAffectedRows") or 0) + transferred_bytes = int(qstats.get("transferredBytes") or 0) + + err_reason = str(err.get("reason") or first_err.get("reason") or "") + err_msg = str(err.get("message") or first_err.get("message") or "") + is_redacted = str(ref.get("jobId") or "") == "" + + state = raw_state + # Note: When BigQuery redacts another user's job (`jobId == ''`), + # it sets `errorResult.reason = 'unknown'` with no message on succeeded jobs + # (including 0-byte cached/metadata queries). + if ( + err_msg + or ( + err_reason + and not (is_redacted and err_reason == "unknown" and not err_msg) + ) + ) and state == "DONE": + state = "FAILED" + + return { + "jobId": str(ref.get("jobId") or raw.get("id") or ""), + "projectId": str(ref.get("projectId") or ""), + "location": str(ref.get("location") or "US"), + "userEmail": str(raw.get("user_email") or ""), + "jobType": str(config.get("jobType") or "QUERY"), + "statementType": str( + qstats.get("statementType") or qconfig.get("statementType") or "" + ), + "edition": str(stats.get("edition") or ""), + "reservationId": str(stats.get("reservation_id") or ""), + "state": state, + "creationTimeMs": creation_ms, + "creationTime": epoch_ms_to_iso(creation_ms), + "startTimeMs": start_ms, + "endTimeMs": end_ms, + "durationMs": duration_ms, + "durationHuman": human_millis(duration_ms), + "bytesBilled": bytes_billed, + "bytesBilledHuman": human_bytes(bytes_billed), + "bytesProcessed": bytes_processed, + "bytesProcessedHuman": human_bytes(bytes_processed), + "estimatedUsd": estimate_on_demand_usd(bytes_billed), + "estimatedUsdHuman": format_usd(estimate_on_demand_usd(bytes_billed)), + "slotMs": slot_ms, + "slotTimeHuman": human_millis(slot_ms), + "cacheHit": cache_hit, + "spilledBytes": spilled_bytes, + "maxWaitMs": max_wait_ms, + "biEngineMode": bi_engine_mode, + "priority": priority, + "labels": labels, + "destinationTable": dest_table_str, + "numDmlAffectedRows": num_dml_affected, + "transferredBytes": transferred_bytes, + "query": str(qconfig.get("query") or ""), + "errorMessage": err_msg, + "errorReason": err_reason, + "referencedTables": [ + f"{t.get('projectId', '')}.{t.get('datasetId', '')}.{t.get('tableId', '')}" + for t in qstats.get("referencedTables") or [] + if isinstance(t, dict) + ], + "stages": [ + { + "id": str(s.get("id") or ""), + "name": str(s.get("name") or ""), + "status": str(s.get("status") or ""), + "recordsRead": int(s.get("recordsRead") or 0), + "recordsWritten": int(s.get("recordsWritten") or 0), + "slotMs": int(s.get("slotMs") or 0), + "waitMsMax": int(s.get("waitMsMax") or 0), + "computeMsMax": int(s.get("computeMsMax") or 0), + "shuffleOutputBytes": int(s.get("shuffleOutputBytes") or 0), + "shuffleOutputBytesSpilled": int( + s.get("shuffleOutputBytesSpilled") or 0 + ), + } + for s in stages + if isinstance(s, dict) + ], + } + + +def fetch_bq_jobs( + client: RestClient, + project: str, + max_results: int = 100, + all_users: bool = True, + user_filter: Optional[str] = None, + days: Optional[float] = None, + state_filter: Optional[str] = None, +) -> Tuple[List[Dict[str, Any]], bool]: + """Fetches BigQuery jobs via `jobs.list` with automatic `allUsers` fallback. + + Tries `allUsers=true` first (when `all_users` is True); if the caller lacks + `bigquery.jobs.listAll` (HTTP 403), automatically falls back to + `allUsers=false` so developers with `roles/bigquery.jobUser` still get their + own jobs. + + Args: + client: Authenticated RestClient. + project: GCP project ID. + max_results: Max jobs to fetch. + all_users: Whether to request all users' jobs first. + user_filter: Optional substring or email filter on `user_email`. + days: Optional lookback window in days (`minCreationTime`). + state_filter: Optional state filter (`DONE`, `FAILED`, `RUNNING`, + `PENDING`). + + Returns: + Tuple of (normalized_jobs, fell_back_to_own_jobs). + """ + url = f"{BQ_ENDPOINT}/bigquery/v2/projects/{urllib.parse.quote(project)}/jobs" + fell_back = False + params: Dict[str, Any] = { + "projection": "full", + "maxResults": min(max(max_results, 50), 200), + } + if days is not None and days > 0: + min_ms = int((time.time() - float(days) * 86400.0) * 1000) + params["minCreationTime"] = str(max(min_ms, 0)) + if state_filter and state_filter.strip().upper() in ("RUNNING", "PENDING"): + params["stateFilter"] = state_filter.strip().upper() + if all_users: + params["allUsers"] = "true" + + raw_jobs: List[Dict[str, Any]] = [] + page_token: Optional[str] = None + while len(raw_jobs) < max_results: + page_params = dict(params) + if page_token: + page_params["pageToken"] = page_token + try: + payload = client.get_json(url, page_params) + except JobMonitorError as e: + if e.status == 403 and all_users: + fell_back = True + all_users = False + params.pop("allUsers", None) + page_params.pop("allUsers", None) + payload = client.get_json(url, page_params) + else: + raise + batch = payload.get("jobs") or [] + raw_jobs.extend(j for j in batch if isinstance(j, dict)) + page_token = payload.get("nextPageToken") + if not page_token or not batch: + break + + normalized = [normalize_bq_job(j) for j in raw_jobs] + if user_filter: + needle = user_filter.lower() + normalized = [ + j for j in normalized if needle in j.get("userEmail", "").lower() + ] + if state_filter: + target_state = state_filter.strip().upper() + normalized = [ + j for j in normalized if j.get("state", "").upper() == target_state + ] + return normalized, fell_back + + +def fetch_bq_job_detail( + client: RestClient, + project: str, + job_id: str, + location: Optional[str] = None, +) -> Dict[str, Any]: + """Fetches a single BigQuery job by ID (accepts `proj:loc.jobId` or `jobId`).""" + clean_id = job_id.strip() + if ":" in clean_id: + proj_part, rest = clean_id.split(":", 1) + if proj_part: + project = proj_part + clean_id = rest + if "." in clean_id: + loc_part, rest = clean_id.split(".", 1) + if loc_part and not location: + location = loc_part + clean_id = rest + + url = ( + f"{BQ_ENDPOINT}/bigquery/v2/projects/{urllib.parse.quote(project)}" + f"/jobs/{urllib.parse.quote(clean_id)}" + ) + params = {"location": location} if location else None + raw = client.get_json(url, params) + return normalize_bq_job(raw) + + +def aggregate_bq_summary(jobs: List[Dict[str, Any]]) -> Dict[str, Any]: + """Aggregates BigQuery jobs by user, edition/reservation, and error reason.""" + by_user: Dict[str, Dict[str, Any]] = {} + by_reservation: Dict[str, Dict[str, Any]] = {} + by_error: Dict[str, Dict[str, Any]] = {} + + for j in jobs: + user = j.get("userEmail") or "" + u_rec = by_user.setdefault( + user, + { + "userEmail": user, + "totalJobs": 0, + "succeededJobs": 0, + "failedJobs": 0, + "runningJobs": 0, + "bytesBilled": 0, + "onDemandBytesBilled": 0, + "slotMs": 0, + }, + ) + u_rec["totalJobs"] += 1 + state = j.get("state") or "UNKNOWN" + if state == "FAILED": + u_rec["failedJobs"] += 1 + elif state in ("RUNNING", "PENDING"): + u_rec["runningJobs"] += 1 + else: + u_rec["succeededJobs"] += 1 + u_rec["bytesBilled"] += int(j.get("bytesBilled") or 0) + if not j.get("reservationId") and not j.get("edition"): + u_rec["onDemandBytesBilled"] += int(j.get("bytesBilled") or 0) + u_rec["slotMs"] += int(j.get("slotMs") or 0) + + edition = j.get("edition") or "ON_DEMAND" + res_id = j.get("reservationId") or "(none - on-demand)" + res_key = f"{edition}|{res_id}" + r_rec = by_reservation.setdefault( + res_key, + { + "edition": edition, + "reservationId": res_id, + "totalJobs": 0, + "succeededJobs": 0, + "failedJobs": 0, + "bytesBilled": 0, + "slotMs": 0, + }, + ) + r_rec["totalJobs"] += 1 + if state == "FAILED": + r_rec["failedJobs"] += 1 + else: + r_rec["succeededJobs"] += 1 + r_rec["bytesBilled"] += int(j.get("bytesBilled") or 0) + r_rec["slotMs"] += int(j.get("slotMs") or 0) + + if state == "FAILED": + reason = j.get("errorReason") or "unknownError" + msg = truncate_one_line(j.get("errorMessage") or "(no message)", 120) + err_key = f"{reason}|{msg}" + e_rec = by_error.setdefault( + err_key, + { + "errorReason": reason, + "sampleMessage": msg, + "count": 0, + "sampleJobId": j.get("jobId") or "", + }, + ) + e_rec["count"] += 1 + + users_list = sorted( + by_user.values(), + key=lambda r: (r["bytesBilled"], r["slotMs"], r["totalJobs"]), + reverse=True, + ) + for u in users_list: + u["bytesBilledHuman"] = human_bytes(u["bytesBilled"]) + u["estimatedOnDemandUsd"] = estimate_on_demand_usd(u["onDemandBytesBilled"]) + u["estimatedOnDemandUsdHuman"] = format_usd(u["estimatedOnDemandUsd"]) + u["slotTimeHuman"] = human_millis(u["slotMs"]) + + res_list = sorted( + by_reservation.values(), + key=lambda r: (r["bytesBilled"], r["slotMs"], r["totalJobs"]), + reverse=True, + ) + for r in res_list: + r["bytesBilledHuman"] = human_bytes(r["bytesBilled"]) + r["slotTimeHuman"] = human_millis(r["slotMs"]) + + err_list = sorted(by_error.values(), key=lambda e: e["count"], reverse=True) + return { + "totalJobsAnalyzed": len(jobs), + "byUser": users_list, + "byReservation": res_list, + "topErrors": err_list, + } + + +def aggregate_bq_hot_tables(jobs: List[Dict[str, Any]]) -> Dict[str, Any]: + """Aggregates referenced BigQuery tables by total bytes billed and slot time.""" + tables: Dict[str, Dict[str, Any]] = {} + redacted_count = 0 + for j in jobs: + if j.get("jobId") == "" and not j.get("referencedTables"): + redacted_count += 1 + refs = j.get("referencedTables") or [] + for tbl in set(refs): + if not tbl or tbl == "..": + continue + rec = tables.setdefault( + tbl, + { + "table": tbl, + "queryCount": 0, + "totalBytesBilled": 0, + "totalSlotMs": 0, + "users": set(), + }, + ) + rec["queryCount"] += 1 + rec["totalBytesBilled"] += int(j.get("bytesBilled") or 0) + rec["totalSlotMs"] += int(j.get("slotMs") or 0) + if j.get("userEmail"): + rec["users"].add(j["userEmail"]) + + ranked = sorted( + tables.values(), + key=lambda r: ( + r["totalBytesBilled"], + r["totalSlotMs"], + r["queryCount"], + ), + reverse=True, + ) + out_tables = [] + for r in ranked: + out_tables.append({ + "table": r["table"], + "queryCount": r["queryCount"], + "totalBytesBilled": r["totalBytesBilled"], + "totalBytesBilledHuman": human_bytes(r["totalBytesBilled"]), + "estimatedUsd": estimate_on_demand_usd(r["totalBytesBilled"]), + "estimatedUsdHuman": format_usd( + estimate_on_demand_usd(r["totalBytesBilled"]) + ), + "totalSlotMs": r["totalSlotMs"], + "totalSlotTimeHuman": human_millis(r["totalSlotMs"]), + "distinctUsers": len(r["users"]), + }) + return { + "totalJobsAnalyzed": len(jobs), + "redactedJobsWithoutTableRefs": redacted_count, + "hotTables": out_tables, + } + + +def fetch_bq_reservations( + client: RestClient, + project: str, + location: str = "US", +) -> Dict[str, Any]: + """Fetches BigQuery slot reservations, capacity commitments, and assignments.""" + loc = location or "US" + base = ( + f"{BQ_RESERVATION_ENDPOINT}/v1/projects/{urllib.parse.quote(project)}" + f"/locations/{urllib.parse.quote(loc)}" + ) + reservations: List[Dict[str, Any]] = [] + commitments: List[Dict[str, Any]] = [] + assignments: List[Dict[str, Any]] = [] + warnings: List[str] = [] + + try: + r_resp = client.get_json(f"{base}/reservations") + for r in r_resp.get("reservations") or []: + if isinstance(r, dict): + autoscale = r.get("autoscale") or {} + reservations.append({ + "name": str(r.get("name") or "").rsplit("/", 1)[-1], + "edition": str(r.get("edition") or "STANDARD"), + "baselineSlots": int(r.get("slotCapacity") or 0), + "autoscaleCurrentSlots": int(autoscale.get("currentSlots") or 0), + "autoscaleMaxSlots": int(autoscale.get("maxSlots") or 0), + "ignoreIdleSlots": bool(r.get("ignoreIdleSlots", False)), + "concurrency": int(r.get("concurrency") or 0), + }) + except JobMonitorError as e: + warnings.append(f"Reservations ({loc}): {e}") + + try: + c_resp = client.get_json(f"{base}/capacityCommitments") + for c in c_resp.get("capacityCommitments") or []: + if isinstance(c, dict): + commitments.append({ + "name": str(c.get("name") or "").rsplit("/", 1)[-1], + "edition": str(c.get("edition") or ""), + "slotCount": int(c.get("slotCount") or 0), + "plan": str(c.get("plan") or ""), + "state": str(c.get("state") or ""), + }) + except JobMonitorError as e: + warnings.append(f"Capacity commitments ({loc}): {e}") + + try: + a_resp = client.get_json(f"{base}/reservations/-/assignments") + for a in a_resp.get("assignments") or []: + if isinstance(a, dict): + full_name = str(a.get("name") or "") + parts = full_name.split("/") + res_name = ( + parts[parts.index("reservations") + 1] + if "reservations" in parts + else "" + ) + assignments.append({ + "name": full_name.rsplit("/", 1)[-1], + "reservation": res_name, + "assignee": str(a.get("assignee") or ""), + "jobType": str(a.get("jobType") or ""), + "state": str(a.get("state") or ""), + }) + except JobMonitorError as e: + warnings.append(f"Assignments ({loc}): {e}") + + return { + "project": project, + "location": loc, + "reservations": reservations, + "capacityCommitments": commitments, + "assignments": assignments, + "warnings": warnings, + } + + +# ------------------------------------------------------------------------------ +# Spark (Dataproc Serverless Batches & Cluster Jobs) Normalization & Actions +# ------------------------------------------------------------------------------ + + +def normalize_spark_batch( + raw: Dict[str, Any], project: str, region: str +) -> Dict[str, Any]: + """Normalizes a Dataproc Serverless Batch resource.""" + name = str( + raw.get("name") or raw.get("batchName") or raw.get("batch_name") or "" + ) + batch_id = name.rsplit("/", 1)[-1] if "/" in name else name + create_str = str(raw.get("createTime") or raw.get("create_time") or "") + state_str = str(raw.get("stateTime") or raw.get("state_time") or "") + state = str(raw.get("state") or "UNKNOWN") + + create_dt = parse_rfc3339(create_str) + end_dt = parse_rfc3339(state_str) + if state in ("RUNNING", "PENDING", "CANCELLING") and create_dt is not None: + end_dt = datetime.datetime.now(datetime.timezone.utc) + duration_ms = ( + int((end_dt - create_dt).total_seconds() * 1000) + if (create_dt and end_dt and end_dt >= create_dt) + else 0 + ) + + workload_type = "PYSPARK" + main_uri = "" + if isinstance(raw.get("pysparkBatch"), dict): + workload_type = "PYSPARK" + main_uri = str(raw["pysparkBatch"].get("mainPythonFileUri") or "") + elif isinstance(raw.get("sparkBatch"), dict): + workload_type = "SPARK_JAR" + main_uri = str( + raw["sparkBatch"].get("mainJarFileUri") + or raw["sparkBatch"].get("mainClass") + or "" + ) + elif isinstance(raw.get("sparkSqlBatch"), dict): + workload_type = "SPARK_SQL" + main_uri = str(raw["sparkSqlBatch"].get("queryFileUri") or "") + + runtime_info = raw.get("runtimeInfo") or {} + usage = runtime_info.get("approximateUsage") or {} + milli_dcu_seconds = int(usage.get("milliDcuSeconds") or 0) + + return { + "id": batch_id, + "kind": "SERVERLESS_BATCH", + "workloadType": workload_type, + "projectId": project, + "region": region, + "creator": str(raw.get("creator") or ""), + "clusterName": "(serverless)", + "state": state, + "createTime": create_str, + "createTimeEpochMs": ( + int(create_dt.timestamp() * 1000) if create_dt else 0 + ), + "endTime": state_str, + "durationMs": duration_ms, + "durationHuman": human_millis(duration_ms), + "milliDcuSeconds": milli_dcu_seconds, + "dcuHours": round(milli_dcu_seconds / (1000.0 * 3600.0), 4), + "mainArtifact": main_uri, + "driverOutputUri": str(runtime_info.get("outputUri") or ""), + "stateMessage": str(raw.get("stateMessage") or ""), + } + + +def normalize_spark_cluster_job( + raw: Dict[str, Any], project: str, region: str +) -> Dict[str, Any]: + """Normalizes a Dataproc Cluster Job resource.""" + ref = raw.get("reference") or {} + job_id = str(ref.get("jobId") or "") + placement = raw.get("placement") or {} + status = raw.get("status") or {} + history = raw.get("statusHistory") or [] + + state = str(status.get("state") or "UNKNOWN") + state_start = str(status.get("stateStartTime") or "") + create_str = state_start + for h in history: + if isinstance(h, dict) and h.get("state") in ("PENDING", "RUNNING"): + create_str = str(h.get("stateStartTime") or create_str) + break + + create_dt = parse_rfc3339(create_str) + end_dt = parse_rfc3339(state_start) + if state in ("RUNNING", "PENDING", "SETUP_DONE") and create_dt is not None: + end_dt = datetime.datetime.now(datetime.timezone.utc) + duration_ms = ( + int((end_dt - create_dt).total_seconds() * 1000) + if (create_dt and end_dt and end_dt >= create_dt) + else 0 + ) + + workload_type = "CLUSTER_JOB" + main_uri = "" + if isinstance(raw.get("pysparkJob"), dict): + workload_type = "PYSPARK" + main_uri = str(raw["pysparkJob"].get("mainPythonFileUri") or "") + elif isinstance(raw.get("sparkJob"), dict): + workload_type = "SPARK_JAR" + main_uri = str( + raw["sparkJob"].get("mainJarFileUri") + or raw["sparkJob"].get("mainClass") + or "" + ) + elif isinstance(raw.get("sparkSqlJob"), dict): + workload_type = "SPARK_SQL" + main_uri = str(raw["sparkSqlJob"].get("queryFileUri") or "") + + return { + "id": job_id, + "kind": "CLUSTER_JOB", + "workloadType": workload_type, + "projectId": project, + "region": region, + "creator": str(raw.get("submittedBy") or ""), + "clusterName": str(placement.get("clusterName") or ""), + "state": state, + "createTime": create_str, + "createTimeEpochMs": ( + int(create_dt.timestamp() * 1000) if create_dt else 0 + ), + "endTime": state_start, + "durationMs": duration_ms, + "durationHuman": human_millis(duration_ms), + "milliDcuSeconds": 0, + "dcuHours": 0.0, + "mainArtifact": main_uri, + "driverOutputUri": str(raw.get("driverOutputResourceUri") or ""), + "stateMessage": str(status.get("details") or ""), + } + + +def fetch_spark_workloads( + client: RestClient, + project: str, + region: str = DEFAULT_REGION, + max_results: int = 50, + user_filter: Optional[str] = None, + state_filter: Optional[str] = None, +) -> Tuple[List[Dict[str, Any]], List[str]]: + """Fetches both Dataproc Serverless Batches and Cluster Jobs across region(s).""" + workloads: List[Dict[str, Any]] = [] + warnings: List[str] = [] + + regions = [ + r.strip() for r in (region or DEFAULT_REGION).split(",") if r.strip() + ] or [DEFAULT_REGION] + + for reg in regions: + # 1. Dataproc Serverless Batches + batches_url = ( + f"{DATAPROC_ENDPOINT}/v1/projects/{urllib.parse.quote(project)}" + f"/locations/{urllib.parse.quote(reg)}/batches" + ) + try: + b_payload = client.get_json( + batches_url, {"pageSize": min(max_results, 100)} + ) + for b in b_payload.get("batches") or []: + if isinstance(b, dict): + workloads.append(normalize_spark_batch(b, project, reg)) + except JobMonitorError as e: + warnings.append(f"Serverless batches ({reg}): {e}") + + # 2. Dataproc Cluster Jobs + jobs_url = ( + f"{DATAPROC_ENDPOINT}/v1/projects/{urllib.parse.quote(project)}" + f"/regions/{urllib.parse.quote(reg)}/jobs" + ) + try: + j_payload = client.get_json(jobs_url, {"pageSize": min(max_results, 100)}) + for j in j_payload.get("jobs") or []: + if isinstance(j, dict): + workloads.append(normalize_spark_cluster_job(j, project, reg)) + except JobMonitorError as e: + warnings.append(f"Cluster jobs ({reg}): {e}") + + if user_filter: + needle = user_filter.lower() + workloads = [ + w + for w in workloads + if needle in w.get("creator", "").lower() or not w.get("creator") + ] + if state_filter: + target_state = state_filter.strip().upper() + workloads = [ + w for w in workloads if w.get("state", "").upper() == target_state + ] + + workloads.sort(key=lambda w: w.get("createTimeEpochMs", 0), reverse=True) + return workloads, warnings + + +def fetch_spark_workload_detail( + client: RestClient, + project: str, + job_or_batch_id: str, + region: str = DEFAULT_REGION, + workload_type: str = "auto", +) -> Dict[str, Any]: + """Fetches details for a Spark Serverless Batch or Cluster Job by ID.""" + clean_id = job_or_batch_id.strip().rsplit("/", 1)[-1] + batch_url = ( + f"{DATAPROC_ENDPOINT}/v1/projects/{urllib.parse.quote(project)}" + f"/locations/{urllib.parse.quote(region)}/batches/{urllib.parse.quote(clean_id)}" + ) + job_url = ( + f"{DATAPROC_ENDPOINT}/v1/projects/{urllib.parse.quote(project)}" + f"/regions/{urllib.parse.quote(region)}/jobs/{urllib.parse.quote(clean_id)}" + ) + if workload_type == "cluster_job": + raw_job = client.get_json(job_url) + return normalize_spark_cluster_job(raw_job, project, region) + if workload_type == "batch": + raw_batch = client.get_json(batch_url) + return normalize_spark_batch(raw_batch, project, region) + try: + raw_batch = client.get_json(batch_url) + return normalize_spark_batch(raw_batch, project, region) + except JobMonitorError as batch_err: + try: + raw_job = client.get_json(job_url) + return normalize_spark_cluster_job(raw_job, project, region) + except JobMonitorError: + raise batch_err + + +def fetch_spark_fleet( + client: RestClient, + project: str, + region: str = DEFAULT_REGION, +) -> Dict[str, Any]: + """Fetches Dataproc Clusters and Serverless Interactive Sessions in region(s).""" + clusters: List[Dict[str, Any]] = [] + sessions: List[Dict[str, Any]] = [] + warnings: List[str] = [] + + regions = [ + r.strip() for r in (region or DEFAULT_REGION).split(",") if r.strip() + ] or [DEFAULT_REGION] + + for reg in regions: + # 1. Dataproc Clusters + clusters_url = ( + f"{DATAPROC_ENDPOINT}/v1/projects/{urllib.parse.quote(project)}" + f"/regions/{urllib.parse.quote(reg)}/clusters" + ) + try: + c_payload = client.get_json(clusters_url) + for c in c_payload.get("clusters") or []: + if isinstance(c, dict): + cfg = c.get("config") or {} + master_cfg = cfg.get("masterConfig") or {} + worker_cfg = cfg.get("workerConfig") or {} + status = c.get("status") or {} + clusters.append({ + "clusterName": str( + c.get("clusterName") or c.get("cluster_name") or "" + ), + "region": reg, + "state": str(status.get("state") or "UNKNOWN"), + "stateStartTime": str( + status.get("stateStartTime") + or status.get("state_start_time") + or "" + ), + "masterMachineType": ( + str(master_cfg.get("machineTypeUri") or "").rsplit("/", 1)[-1] + ), + "workerCount": int(worker_cfg.get("numInstances") or 0), + "workerMachineType": ( + str(worker_cfg.get("machineTypeUri") or "").rsplit("/", 1)[-1] + ), + }) + except JobMonitorError as e: + warnings.append(f"Clusters ({reg}): {e}") + + # 2. Dataproc Serverless Interactive Sessions + sessions_url = ( + f"{DATAPROC_ENDPOINT}/v1/projects/{urllib.parse.quote(project)}" + f"/locations/{urllib.parse.quote(reg)}/sessions" + ) + try: + s_payload = client.get_json(sessions_url) + for s in s_payload.get("sessions") or []: + if isinstance(s, dict): + name = str( + s.get("name") + or s.get("sessionName") + or s.get("session_name") + or "" + ) + sessions.append({ + "sessionId": name.rsplit("/", 1)[-1] if "/" in name else name, + "region": reg, + "state": str(s.get("state") or "UNKNOWN"), + "createTime": str( + s.get("createTime") or s.get("create_time") or "" + ), + "creator": str(s.get("creator") or ""), + }) + except JobMonitorError as e: + warnings.append(f"Serverless sessions ({reg}): {e}") + + return { + "project": project, + "region": region, + "clusters": clusters, + "sessions": sessions, + "warnings": warnings, + } + + +# ------------------------------------------------------------------------------ +# Markdown Rendering +# ------------------------------------------------------------------------------ + + +def render_bq_jobs_markdown( + jobs: List[Dict[str, Any]], + title: str, + limit: int = 10, + fell_back: bool = False, +) -> str: + """Renders a Markdown table of BigQuery jobs.""" + lines = [f"### {title}"] + if fell_back: + lines.append( + "> **Note**: Caller lacks project-wide `bigquery.jobs.listAll`" + " permission; showing jobs for the current user (`allUsers=false`)." + ) + if any(j.get("jobId") == "" for j in jobs[:limit]): + lines.append( + "> **Note**: `` entries are jobs run by other principals" + " where the caller has project/reservation statistics visibility" + " (`bigquery.jobs.list`) but not full SQL text visibility" + " (`bigquery.jobs.listAll`)." + ) + if not jobs: + lines.append("\nNo matching BigQuery jobs found.") + return "\n".join(lines) + + lines.append( + "\n| Job ID | Region | User | State | Duration | Bytes Billed | Est." + " Cost (On-Demand) | Slot Time | Statement / Query |" + ) + lines.append( + "| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |" + ) + for j in jobs[:limit]: + raw_summary = j.get("query") + if not raw_summary: + if j.get("errorMessage"): + raw_summary = f"ERROR ({j.get('errorReason')}): {j.get('errorMessage')}" + elif j.get("state") == "FAILED" and j.get("errorReason"): + raw_summary = f"ERROR: {j.get('errorReason')}" + elif j.get("reservationId"): + raw_summary = ( + f"{j.get('jobType')} [{j.get('edition') or 'RES'}:" + f" {j.get('reservationId')}] (SQL redacted)" + ) + else: + raw_summary = j.get("jobType") + q_preview = truncate_one_line(raw_summary, 70).replace("|", "\\|") + cache_tag = " *(cached)*" if j.get("cacheHit") else "" + lines.append( + f"| `{j['jobId']}` | {j['location']} | {j['userEmail'] or 'n/a'} |" + f" {j['state']} | {j['durationHuman']} |" + f" {j['bytesBilledHuman']}{cache_tag} | {j['estimatedUsdHuman']} |" + f" {j['slotTimeHuman']} | `{q_preview}` |" + ) + return "\n".join(lines) + + +def render_bq_detail_markdown(j: Dict[str, Any]) -> str: + """Renders detailed Markdown for a single BigQuery job.""" + lines = [ + f"### BigQuery Job Details: `{j['jobId']}`", + f"- **Project / Location**: `{j['projectId']}` (`{j['location']}`)", + f"- **User**: `{j['userEmail'] or 'n/a'}`", + ( + f"- **State**: **{j['state']}** (Job Type: `{j['jobType']}`," + f" Statement: `{j['statementType'] or 'n/a'}`)" + ), + f"- **Created**: {j['creationTime']}", + f"- **Wall-Clock Duration**: **{j['durationHuman']}**", + ( + f"- **Bytes Billed**: **{j['bytesBilledHuman']}** (Processed:" + f" {j['bytesProcessedHuman']}, Cache Hit: `{j['cacheHit']}`)" + ), + ( + "- **Estimated Cost (On-Demand @ $6.25/TiB)**:" + f" **{j['estimatedUsdHuman']}**" + ), + f"- **Total Slot Time**: **{j['slotTimeHuman']}** ({j['slotMs']} ms)", + ] + if j.get("spilledBytes", 0) > 0: + lines.append( + f"- **Shuffle Spill to Disk**: {human_bytes(j['spilledBytes'])}" + ) + if j.get("referencedTables"): + refs = ", ".join(f"`{t}`" for t in j["referencedTables"][:10]) + lines.append(f"- **Referenced Tables**: {refs}") + if j.get("errorMessage"): + lines.append(f"- **Error (`{j.get('errorReason')}`)**: {j['errorMessage']}") + if j.get("query"): + lines.extend(["\n#### SQL Query", "```sql", j["query"], "```"]) + if j.get("stages"): + lines.extend([ + "\n#### Execution Stages (`queryPlan`)", + ( + "| Stage | Status | Records Read | Records Written | Slot Time |" + " Max Wait | Max Compute | Spill |" + ), + "| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |", + ]) + for s in j["stages"]: + lines.append( + f"| `{s['name']}` | {s['status']} | {s['recordsRead']:,} |" + f" {s['recordsWritten']:,} | {human_millis(s['slotMs'])} |" + f" {human_millis(s['waitMsMax'])} |" + f" {human_millis(s['computeMsMax'])} |" + f" {human_bytes(s['shuffleOutputBytesSpilled'])} |" + ) + return "\n".join(lines) + + +def render_spark_workloads_markdown( + workloads: List[Dict[str, Any]], + title: str, + limit: int = 10, + warnings: Optional[List[str]] = None, +) -> str: + """Renders a Markdown table of Spark Serverless Batches and Cluster Jobs.""" + lines = [f"### {title}"] + for w in warnings or []: + lines.append(f"> **Warning**: {w}") + if not workloads: + lines.append("\nNo matching Spark jobs or batches found.") + return "\n".join(lines) + + lines.append( + "\n| ID | Kind | Type | Cluster / Runtime | State | Duration | Created |" + " Creator | Main Script / Artifact |" + ) + lines.append( + "| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |" + ) + for w in workloads[:limit]: + artifact = truncate_one_line(w.get("mainArtifact") or "n/a", 60).replace( + "|", "\\|" + ) + lines.append( + f"| `{w['id']}` | {w['kind']} | {w['workloadType']} |" + f" {w['clusterName']} | {w['state']} | {w['durationHuman']} |" + f" {w['createTime']} | {w['creator'] or 'n/a'} | `{artifact}` |" + ) + return "\n".join(lines) + + +def render_spark_detail_markdown(w: Dict[str, Any]) -> str: + """Renders detailed Markdown for a single Spark Batch or Cluster Job.""" + lines = [ + f"### Spark Workload Details: `{w['id']}`", + f"- **Kind / Type**: `{w['kind']}` (`{w['workloadType']}`)", + f"- **Project / Region**: `{w['projectId']}` (`{w['region']}`)", + f"- **Cluster / Runtime**: `{w['clusterName']}`", + f"- **Creator**: `{w['creator'] or 'n/a'}`", + f"- **State**: **{w['state']}**", + f"- **Created**: {w['createTime']}", + f"- **Elapsed Duration**: **{w['durationHuman']}**", + ] + if w.get("dcuHours", 0) > 0: + lines.append( + f"- **Approximate Serverless Usage**: {w['dcuHours']} DCU-hours" + f" ({w['milliDcuSeconds']:,} milli-DCU-seconds)" + ) + if w.get("mainArtifact"): + lines.append(f"- **Main Script / JAR**: `{w['mainArtifact']}`") + if w.get("driverOutputUri"): + lines.append(f"- **Driver Output URI**: `{w['driverOutputUri']}`") + if w.get("stateMessage"): + lines.append(f"- **State / Error Message**: {w['stateMessage']}") + return "\n".join(lines) + + +def render_bq_summary_markdown( + summary: Dict[str, Any], + project: str, + fell_back: bool = False, + limit: int = 10, +) -> str: + """Renders a multi-section admin summary of BigQuery usage, reservations, and errors.""" + lines = [ + f"### BigQuery Admin Usage & Error Summary: `{project}`" + f" ({summary['totalJobsAnalyzed']} jobs analyzed)" + ] + if fell_back: + lines.append( + "> **Note**: Caller lacks project-wide `bigquery.jobs.listAll`" + " permission; summarizing jobs for the current user (`allUsers=false`)." + ) + + lines.extend([ + "\n#### 1. Cost & Slot Attribution by User", + ( + "| User | Total Jobs | Succeeded | Failed | Running | Bytes Billed |" + " Est. On-Demand Cost | Slot Time |" + ), + "| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |", + ]) + for u in summary["byUser"][:limit]: + lines.append( + f"| `{u['userEmail']}` | {u['totalJobs']} | {u['succeededJobs']} |" + f" {u['failedJobs']} | {u['runningJobs']} | {u['bytesBilledHuman']} |" + f" {u['estimatedOnDemandUsdHuman']} | {u['slotTimeHuman']} |" + ) + + lines.extend([ + "\n#### 2. Usage by Edition & Slot Reservation", + ( + "| Edition | Reservation ID | Total Jobs | Succeeded | Failed | Bytes" + " Billed | Slot Time |" + ), + "| :--- | :--- | :--- | :--- | :--- | :--- | :--- |", + ]) + for r in summary["byReservation"][:limit]: + lines.append( + f"| `{r['edition']}` | `{r['reservationId']}` | {r['totalJobs']} |" + f" {r['succeededJobs']} | {r['failedJobs']} | {r['bytesBilledHuman']} |" + f" {r['slotTimeHuman']} |" + ) + + lines.append("\n#### 3. Top Error Reasons (Failed Jobs)") + if not summary["topErrors"]: + lines.append("No failed jobs in the analyzed window.") + else: + lines.extend([ + ( + "| Error Reason | Failure Count | Sample Job ID | Sample Error" + " Message |" + ), + "| :--- | :--- | :--- | :--- |", + ]) + for e in summary["topErrors"][:limit]: + msg = e["sampleMessage"].replace("|", "\\|") + lines.append( + f"| `{e['errorReason']}` | {e['count']} | `{e['sampleJobId']}` |" + f" `{msg}` |" + ) + return "\n".join(lines) + + +def render_bq_hot_tables_markdown( + report: Dict[str, Any], project: str, limit: int = 10 +) -> str: + """Renders a Markdown table of the most-queried / highest-billed BigQuery tables.""" + lines = [ + f"### Most Queried / Highest-Cost BigQuery Tables in `{project}`" + f" ({report['totalJobsAnalyzed']} jobs analyzed)" + ] + if report.get("redactedJobsWithoutTableRefs", 0) > 0: + lines.append( + f"> **Note**: {report['redactedJobsWithoutTableRefs']} ``" + " jobs from other principals did not expose `referencedTables`" + " metadata." + ) + hot = report.get("hotTables") or [] + if not hot: + lines.append("\nNo referenced tables found in the analyzed jobs.") + return "\n".join(lines) + + lines.extend([ + ( + "\n| Table (`project.dataset.table`) | Query Count | Distinct Users |" + " Total Bytes Billed | Est. On-Demand Cost | Total Slot Time |" + ), + "| :--- | :--- | :--- | :--- | :--- | :--- |", + ]) + for t in hot[:limit]: + lines.append( + f"| `{t['table']}` | {t['queryCount']} | {t['distinctUsers']} |" + f" {t['totalBytesBilledHuman']} | {t['estimatedUsdHuman']} |" + f" {t['totalSlotTimeHuman']} |" + ) + return "\n".join(lines) + + +def render_bq_reservations_markdown(data: Dict[str, Any]) -> str: + """Renders BigQuery slot reservations, capacity commitments, and assignments.""" + lines = [ + f"### BigQuery Slot Reservations & Capacity in `{data['project']}`" + f" (`{data['location']}`)" + ] + for w in data.get("warnings") or []: + lines.append(f"> **Warning**: {w}") + + lines.append("\n#### 1. Slot Reservations") + if not data.get("reservations"): + lines.append("No reservations configured in this location.") + else: + lines.extend([ + ( + "| Reservation | Edition | Baseline Slots | Autoscale Current |" + " Autoscale Max | Ignore Idle Slots |" + ), + "| :--- | :--- | :--- | :--- | :--- | :--- |", + ]) + for r in data["reservations"]: + lines.append( + f"| `{r['name']}` | `{r['edition']}` | {r['baselineSlots']} |" + f" {r['autoscaleCurrentSlots']} | {r['autoscaleMaxSlots']} |" + f" `{r['ignoreIdleSlots']}` |" + ) + + lines.append("\n#### 2. Capacity Commitments") + if not data.get("capacityCommitments"): + lines.append("No capacity commitments in this location.") + else: + lines.extend([ + "| Commitment ID | Edition | Slots | Plan | State |", + "| :--- | :--- | :--- | :--- | :--- |", + ]) + for c in data["capacityCommitments"]: + lines.append( + f"| `{c['name']}` | `{c['edition']}` | {c['slotCount']} |" + f" `{c['plan']}` | {c['state']} |" + ) + + lines.append("\n#### 3. Reservation Assignments") + if not data.get("assignments"): + lines.append("No active reservation assignments in this location.") + else: + lines.extend([ + "| Assignment ID | Reservation | Assignee | Job Type | State |", + "| :--- | :--- | :--- | :--- | :--- |", + ]) + for a in data["assignments"]: + lines.append( + f"| `{a['name']}` | `{a['reservation']}` | `{a['assignee']}` |" + f" `{a['jobType']}` | {a['state']} |" + ) + return "\n".join(lines) + + +def render_spark_fleet_markdown(data: Dict[str, Any]) -> str: + """Renders active Dataproc Clusters and Serverless Interactive Sessions.""" + lines = [ + "### Dataproc Clusters & Interactive Spark Sessions in" + f" `{data['project']}` (`{data['region']}`)" + ] + for w in data.get("warnings") or []: + lines.append(f"> **Warning**: {w}") + + lines.append("\n#### 1. Dataproc Clusters") + if not data.get("clusters"): + lines.append("No Dataproc clusters found.") + else: + lines.extend([ + ( + "| Cluster Name | Region | State | Master Type | Workers | Worker" + " Type | State Start Time |" + ), + "| :--- | :--- | :--- | :--- | :--- | :--- | :--- |", + ]) + for c in data["clusters"]: + lines.append( + f"| `{c['clusterName']}` | {c['region']} | {c['state']} |" + f" `{c['masterMachineType']}` | {c['workerCount']} |" + f" `{c['workerMachineType']}` | {c['stateStartTime']} |" + ) + + lines.append("\n#### 2. Dataproc Serverless Interactive Sessions") + if not data.get("sessions"): + lines.append("No Dataproc Serverless interactive sessions found.") + else: + lines.extend([ + "| Session ID | Region | State | Creator | Created |", + "| :--- | :--- | :--- | :--- | :--- |", + ]) + for s in data["sessions"]: + lines.append( + f"| `{s['sessionId']}` | {s['region']} | {s['state']} |" + f" `{s['creator'] or 'n/a'}` | {s['createTime']} |" + ) + return "\n".join(lines) + + +# ------------------------------------------------------------------------------ +# CLI Entrypoint +# ------------------------------------------------------------------------------ + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description=( + "Monitor BigQuery and Spark (Dataproc) jobs, costs, durations, and" + " job details." + ) + ) + parser.add_argument( + "--action", + required=True, + choices=( + "bq_list", + "bq_top_cost", + "bq_slowest", + "bq_describe", + "bq_summary", + "bq_hot_tables", + "bq_reservations", + "spark_list", + "spark_slowest", + "spark_describe", + "spark_clusters", + ), + help="Monitoring action to execute.", + ) + parser.add_argument( + "--project", + "--project_id", + dest="project", + help="GCP project ID (defaults to gcloud).", + ) + parser.add_argument( + "--region", + default=DEFAULT_REGION, + help=( + "Region (or comma-separated regions) for Spark/Dataproc lookup" + f" (default: {DEFAULT_REGION})." + ), + ) + parser.add_argument( + "--location", + help="Optional BigQuery job location (e.g., US, EU, us-central1).", + ) + parser.add_argument( + "--job_id", + help="Job ID or Batch ID for bq_describe or spark_describe.", + ) + parser.add_argument( + "--workload_type", + choices=("auto", "batch", "cluster_job"), + default="auto", + help="Spark workload type for spark_describe (default: auto).", + ) + parser.add_argument( + "--user", + help="Filter jobs by user email substring.", + ) + parser.add_argument( + "--mine", + action="store_true", + help="Filter to jobs submitted by the active gcloud account.", + ) + parser.add_argument( + "--state", + help="Optional state filter (e.g., DONE, FAILED, RUNNING, PENDING).", + ) + parser.add_argument( + "--days", + type=float, + help="Optional lookback window in days (e.g., 7 or 30).", + ) + parser.add_argument( + "--limit", + type=int, + default=10, + help="Maximum number of rows to display (default: 10).", + ) + parser.add_argument( + "--format", + choices=("markdown", "json"), + default="markdown", + help="Output format (default: markdown).", + ) + parser.add_argument( + "--transport", + choices=("auto", "rest", "remote_cli"), + default="auto", + help=( + "API transport: 'auto' (default, REST with automatic Sobi VMaaS" + " remote_cli mcp fallback), 'rest', or 'remote_cli'." + ), + ) + return parser + + +def main( + argv: Optional[Sequence[str]] = None, + client: Optional[Any] = None, +) -> int: + parser = build_parser() + args = parser.parse_args(argv) + if client is None: + if args.transport == "remote_cli": + client = RestClient(remote_cli_client=RemoteCliMcpClient()) + else: + client = RestClient() + + try: + project = resolve_default_project(args.project) + user_filter = args.user + if args.mine and not user_filter: + user_filter = resolve_default_account() + + if args.action in ("bq_list", "bq_top_cost", "bq_slowest"): + jobs, fell_back = fetch_bq_jobs( + client, + project=project, + max_results=max(args.limit * 5, 100), + all_users=not args.mine, + user_filter=user_filter, + days=args.days, + state_filter=args.state, + ) + if args.action == "bq_top_cost": + jobs.sort( + key=lambda j: (j.get("bytesBilled", 0), j.get("slotMs", 0)), + reverse=True, + ) + title = f"Top Most Expensive BigQuery Jobs in `{project}`" + elif args.action == "bq_slowest": + jobs.sort(key=lambda j: j.get("durationMs", 0), reverse=True) + title = f"Slowest / Longest-Running BigQuery Jobs in `{project}`" + else: + jobs.sort(key=lambda j: j.get("creationTimeMs", 0), reverse=True) + title = f"Recent BigQuery Jobs in `{project}`" + + sliced = jobs[: args.limit] + if args.format == "json": + print( + json.dumps( + { + "project": project, + "fellBackToOwnJobs": fell_back, + "jobs": sliced, + }, + indent=2, + ) + ) + else: + print( + render_bq_jobs_markdown( + sliced, title=title, limit=args.limit, fell_back=fell_back + ) + ) + return 0 + + if args.action == "bq_describe": + if not args.job_id: + raise JobMonitorError("--job_id is required for --action=bq_describe.") + detail = fetch_bq_job_detail( + client, project=project, job_id=args.job_id, location=args.location + ) + if args.format == "json": + print(json.dumps(detail, indent=2)) + else: + print(render_bq_detail_markdown(detail)) + return 0 + + if args.action == "bq_summary": + jobs, fell_back = fetch_bq_jobs( + client, + project=project, + max_results=max(args.limit * 20, 200), + all_users=not args.mine, + user_filter=user_filter, + days=args.days, + state_filter=args.state, + ) + summary = aggregate_bq_summary(jobs) + if args.format == "json": + print( + json.dumps( + { + "project": project, + "fellBackToOwnJobs": fell_back, + **summary, + }, + indent=2, + ) + ) + else: + print( + render_bq_summary_markdown( + summary, + project=project, + fell_back=fell_back, + limit=args.limit, + ) + ) + return 0 + + if args.action == "bq_hot_tables": + jobs, fell_back = fetch_bq_jobs( + client, + project=project, + max_results=max(args.limit * 20, 200), + all_users=not args.mine, + user_filter=user_filter, + days=args.days, + state_filter=args.state, + ) + report = aggregate_bq_hot_tables(jobs) + if args.format == "json": + print( + json.dumps( + {"project": project, "fellBackToOwnJobs": fell_back, **report}, + indent=2, + ) + ) + else: + print( + render_bq_hot_tables_markdown( + report, project=project, limit=args.limit + ) + ) + return 0 + + if args.action == "bq_reservations": + res_data = fetch_bq_reservations( + client, project=project, location=args.location or "US" + ) + if args.format == "json": + print(json.dumps(res_data, indent=2)) + else: + print(render_bq_reservations_markdown(res_data)) + return 0 + + if args.action == "spark_clusters": + fleet_data = fetch_spark_fleet( + client, project=project, region=args.region + ) + if args.format == "json": + print(json.dumps(fleet_data, indent=2)) + else: + print(render_spark_fleet_markdown(fleet_data)) + return 0 + + if args.action in ("spark_list", "spark_slowest"): + workloads, warnings = fetch_spark_workloads( + client, + project=project, + region=args.region, + max_results=max(args.limit * 3, 50), + user_filter=user_filter, + state_filter=args.state, + ) + if args.action == "spark_slowest": + workloads.sort(key=lambda w: w.get("durationMs", 0), reverse=True) + title = f"Slowest Spark Workloads in `{project}` (`{args.region}`)" + else: + title = f"Recent Spark Workloads in `{project}` (`{args.region}`)" + + sliced = workloads[: args.limit] + if args.format == "json": + print( + json.dumps( + { + "project": project, + "region": args.region, + "warnings": warnings, + "workloads": sliced, + }, + indent=2, + ) + ) + else: + print( + render_spark_workloads_markdown( + sliced, title=title, limit=args.limit, warnings=warnings + ) + ) + return 0 + + if args.action == "spark_describe": + if not args.job_id: + raise JobMonitorError( + "--job_id is required for --action=spark_describe." + ) + detail = fetch_spark_workload_detail( + client, + project=project, + job_or_batch_id=args.job_id, + region=args.region, + workload_type=args.workload_type, + ) + if args.format == "json": + print(json.dumps(detail, indent=2)) + else: + print(render_spark_detail_markdown(detail)) + return 0 + + except JobMonitorError as e: + if args.format == "json": + print(json.dumps({"status": "ERROR", "errorMessage": str(e)}, indent=2)) + else: + print(f"ERROR: {e}", file=sys.stderr) + return 1 + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/skills/gcp-job-monitoring/tests/gcp_job_monitor_test.py b/skills/gcp-job-monitoring/tests/gcp_job_monitor_test.py new file mode 100644 index 00000000..7cb991cb --- /dev/null +++ b/skills/gcp-job-monitoring/tests/gcp_job_monitor_test.py @@ -0,0 +1,625 @@ +#!/usr/bin/env python3 +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unit tests for gcp_job_monitor.""" + +import io +import json +import os +import sys +import unittest +import urllib.error +import urllib.parse + +sys.path.insert( + 0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../scripts")) +) +# pylint: disable=g-import-not-at-top +import gcp_job_monitor + + +class _FakeTokens: + """Token provider double.""" + + def __init__(self, sources=("adc-token", "user-token")): + self.sources = list(sources) + self.index = 0 + + def token(self): + return self.sources[self.index] + + def fallback(self): + if self.index + 1 >= len(self.sources): + return False + self.index += 1 + return True + + +class _FakeResponse: + """Context-manager response double for urllib.request.urlopen.""" + + def __init__(self, body_bytes: bytes): + self._body = body_bytes + + def read(self) -> bytes: + return self._body + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + return False + + +class _FakeUrlopen: + """Replays canned (status, body) responses and records every Request.""" + + def __init__(self, responses): + self.responses = list(responses) + self.requests = [] + + def __call__(self, request, timeout=None, context=None): + del timeout, context + self.requests.append(request) + status, body = self.responses.pop(0) + if not isinstance(body, bytes): + body = json.dumps(body).encode("utf-8") + if status >= 400: + raise urllib.error.HTTPError( + url=request.full_url, + code=status, + msg=f"HTTP {status}", + hdrs=None, + fp=io.BytesIO(body), + ) + return _FakeResponse(body) + + def query(self, i): + return urllib.parse.parse_qs( + urllib.parse.urlsplit(self.requests[i].full_url).query + ) + + def path(self, i): + return urllib.parse.urlsplit(self.requests[i].full_url).path + + +def _sample_bq_job( + job_id="job_1", + user_email="alice@example.com", + state="DONE", + start_ms=1700000000000, + end_ms=1700000005000, + bytes_billed=10737418240, # 10 GiB + slot_ms=25000, + query="SELECT * FROM `p.d.t`", + error_message=None, +): + raw = { + "jobReference": { + "projectId": "test-proj", + "jobId": job_id, + "location": "US", + }, + "user_email": user_email, + "status": {"state": state}, + "configuration": { + "jobType": "QUERY", + "query": {"query": query}, + }, + "statistics": { + "creationTime": str(start_ms), + "startTime": str(start_ms), + "endTime": str(end_ms), + "totalSlotMs": str(slot_ms), + "totalBytesProcessed": str(bytes_billed), + "query": { + "totalBytesBilled": str(bytes_billed), + "totalBytesProcessed": str(bytes_billed), + "totalSlotMs": str(slot_ms), + "statementType": "SELECT", + "cacheHit": False, + "referencedTables": [ + {"projectId": "p", "datasetId": "d", "tableId": "t"} + ], + "queryPlan": [{ + "id": "0", + "name": "S00: Input", + "status": "COMPLETE", + "recordsRead": "1000", + "recordsWritten": "10", + "slotMs": "25000", + "waitMsMax": "12", + "computeMsMax": "450", + "shuffleOutputBytesSpilled": "0", + }], + }, + }, + } + if error_message: + raw["status"]["errorResult"] = { + "reason": "invalidQuery", + "message": error_message, + } + return raw + + +class BigQueryJobMonitorTest(unittest.TestCase): + + def test_fetch_bq_jobs_falls_back_when_all_users_forbidden(self): + forbidden = { + "error": { + "code": 403, + "message": "Access Denied: BigQuery BigQuery: Permission denied", + } + } + ok_body = {"jobs": [_sample_bq_job(job_id="j_own")]} + urlopen = _FakeUrlopen([(403, forbidden), (200, ok_body)]) + client = gcp_job_monitor.RestClient( + token_provider=_FakeTokens(), urlopen=urlopen + ) + + jobs, fell_back = gcp_job_monitor.fetch_bq_jobs( + client, "test-proj", max_results=5, all_users=True + ) + + self.assertTrue(fell_back) + self.assertEqual(len(jobs), 1) + self.assertEqual(jobs[0]["jobId"], "j_own") + self.assertEqual(urlopen.query(0).get("allUsers"), ["true"]) + self.assertIsNone(urlopen.query(1).get("allUsers")) + + def test_rank_top_cost_and_slowest_bq_jobs(self): + cheap_slow = gcp_job_monitor.normalize_bq_job( + _sample_bq_job( + job_id="cheap_slow", + start_ms=1700000000000, + end_ms=1700000060000, # 60s + bytes_billed=1024 * 1024, # 1 MiB + slot_ms=5000, + ) + ) + expensive_fast = gcp_job_monitor.normalize_bq_job( + _sample_bq_job( + job_id="expensive_fast", + start_ms=1700000000000, + end_ms=1700000002000, # 2s + bytes_billed=1024**4, # 1 TiB ($6.25) + slot_ms=90000, + ) + ) + jobs = [cheap_slow, expensive_fast] + + by_cost = sorted( + jobs, + key=lambda j: (j["bytesBilled"], j["slotMs"]), + reverse=True, + ) + by_duration = sorted(jobs, key=lambda j: j["durationMs"], reverse=True) + + self.assertEqual(by_cost[0]["jobId"], "expensive_fast") + self.assertAlmostEqual(by_cost[0]["estimatedUsd"], 6.25, places=2) + self.assertEqual(by_duration[0]["jobId"], "cheap_slow") + self.assertEqual(by_duration[0]["durationMs"], 60000) + + def test_fetch_bq_job_detail_parses_location_prefix_and_renders_stages(self): + urlopen = _FakeUrlopen([(200, _sample_bq_job(job_id="bquxjob_123"))]) + client = gcp_job_monitor.RestClient( + token_provider=_FakeTokens(), urlopen=urlopen + ) + + detail = gcp_job_monitor.fetch_bq_job_detail( + client, "default-proj", "custom-proj:us-central1.bquxjob_123" + ) + self.assertEqual( + urlopen.path(0), + "/bigquery/v2/projects/custom-proj/jobs/bquxjob_123", + ) + self.assertEqual(urlopen.query(0).get("location"), ["us-central1"]) + + md = gcp_job_monitor.render_bq_detail_markdown(detail) + self.assertIn("`bquxjob_123`", md) + self.assertIn("`S00: Input`", md) + self.assertIn("`p.d.t`", md) + + +class SparkWorkloadMonitorTest(unittest.TestCase): + + def test_fetch_spark_workloads_merges_serverless_and_cluster_jobs(self): + batches_resp = { + "batches": [{ + "name": "projects/p/locations/us-central1/batches/batch-1", + "state": "SUCCEEDED", + "createTime": "2026-10-14T10:00:00Z", + "stateTime": "2026-10-14T10:05:00Z", + "creator": "alice@example.com", + "pysparkBatch": {"mainPythonFileUri": "gs://b/etl.py"}, + "runtimeInfo": { + "approximateUsage": { + "milliDcuSeconds": "3600000", + "shuffleStorageGbSeconds": "120", + } + }, + }] + } + cluster_jobs_resp = { + "jobs": [{ + "reference": {"projectId": "p", "jobId": "cluster-job-1"}, + "placement": {"clusterName": "analytics-cluster"}, + "status": { + "state": "ERROR", + "stateStartTime": "2026-10-14T11:02:00Z", + "details": "Container exited with non-zero exit code 1", + }, + "statusHistory": [ + {"state": "PENDING", "stateStartTime": "2026-10-14T11:00:00Z"}, + {"state": "RUNNING", "stateStartTime": "2026-10-14T11:00:10Z"}, + ], + "sparkJob": {"mainClass": "com.example.Main"}, + "driverOutputResourceUri": "gs://b/driveroutput.000000000", + }] + } + urlopen = _FakeUrlopen([(200, batches_resp), (200, cluster_jobs_resp)]) + client = gcp_job_monitor.RestClient( + token_provider=_FakeTokens(), urlopen=urlopen + ) + + items, warnings = gcp_job_monitor.fetch_spark_workloads( + client, "p", "us-central1", max_results=10 + ) + + self.assertEqual(warnings, []) + self.assertEqual(len(items), 2) + # Cluster job is newer (11:00 vs 10:00), so it appears first in list + self.assertEqual(items[0]["id"], "cluster-job-1") + self.assertEqual(items[0]["kind"], "CLUSTER_JOB") + self.assertEqual(items[0]["durationMs"], 120000) + self.assertEqual(items[1]["id"], "batch-1") + self.assertEqual(items[1]["kind"], "SERVERLESS_BATCH") + self.assertEqual(items[1]["durationMs"], 300000) + self.assertEqual(items[1]["milliDcuSeconds"], 3600000) + + def test_fetch_spark_workload_detail_falls_back_from_batch_to_cluster_job( + self, + ): + not_found = {"error": {"code": 404, "message": "Batch not found"}} + cluster_job = { + "reference": {"projectId": "p", "jobId": "spark-job-99"}, + "placement": {"clusterName": "prod-spark-cluster"}, + "status": { + "state": "DONE", + "stateStartTime": "2026-10-14T12:10:00Z", + }, + "statusHistory": [ + {"state": "PENDING", "stateStartTime": "2026-10-14T12:00:00Z"}, + {"state": "RUNNING", "stateStartTime": "2026-10-14T12:00:05Z"}, + ], + "pysparkJob": {"mainPythonFileUri": "gs://bucket/jobs/train.py"}, + "driverOutputResourceUri": "gs://bucket/logs/driveroutput.000000000", + } + urlopen = _FakeUrlopen([(404, not_found), (200, cluster_job)]) + client = gcp_job_monitor.RestClient( + token_provider=_FakeTokens(), urlopen=urlopen + ) + + detail = gcp_job_monitor.fetch_spark_workload_detail( + client, "p", "spark-job-99", region="us-central1" + ) + + self.assertEqual(detail["id"], "spark-job-99") + self.assertEqual(detail["kind"], "CLUSTER_JOB") + self.assertEqual(detail["workloadType"], "PYSPARK") + self.assertEqual(detail["clusterName"], "prod-spark-cluster") + self.assertEqual(detail["durationMs"], 600000) + md = gcp_job_monitor.render_spark_detail_markdown(detail) + self.assertIn("`spark-job-99`", md) + self.assertIn("`gs://bucket/jobs/train.py`", md) + self.assertIn("`gs://bucket/logs/driveroutput.000000000`", md) + + def test_cli_flags_project_id_alias_days_state_and_multi_region(self): + parser = gcp_job_monitor.build_parser() + args = parser.parse_args([ + "--action=bq_top_cost", + "--project_id=my-proj", + "--days=30", + "--state=FAILED", + "--region=us-central1,us-east1", + "--workload_type=cluster_job", + ]) + self.assertEqual(args.project, "my-proj") + self.assertEqual(args.days, 30.0) + self.assertEqual(args.state, "FAILED") + self.assertEqual(args.region, "us-central1,us-east1") + self.assertEqual(args.workload_type, "cluster_job") + + def test_aggregate_bq_summary_and_hot_tables(self): + j1 = gcp_job_monitor.normalize_bq_job( + _sample_bq_job( + job_id="j1", + user_email="alice@example.com", + bytes_billed=1024**3, # 1 GiB + slot_ms=10000, + ) + ) + j2 = gcp_job_monitor.normalize_bq_job( + _sample_bq_job( + job_id="j2", + user_email="bob@example.com", + state="DONE", + bytes_billed=0, + slot_ms=0, + error_message="Dataset looker_scratch not found", + ) + ) + summary = gcp_job_monitor.aggregate_bq_summary([j1, j2]) + self.assertEqual(summary["totalJobsAnalyzed"], 2) + self.assertEqual(summary["byUser"][0]["userEmail"], "alice@example.com") + self.assertEqual(summary["byUser"][0]["succeededJobs"], 1) + self.assertEqual(summary["topErrors"][0]["errorReason"], "invalidQuery") + self.assertEqual(summary["topErrors"][0]["count"], 1) + + hot = gcp_job_monitor.aggregate_bq_hot_tables([j1, j2]) + self.assertEqual(hot["hotTables"][0]["table"], "p.d.t") + self.assertEqual(hot["hotTables"][0]["queryCount"], 2) + self.assertEqual(hot["hotTables"][0]["totalBytesBilled"], 1024**3) + + def test_remote_cli_mcp_client_routes_bq_and_spark_calls(self): + recorded_cmds = [] + + def fake_runner(cmd): + recorded_cmds.append(list(cmd)) + server = cmd[2] + tool = cmd[3] + if server == "bigquery" and tool == "execute_sql_readonly": + args_json = json.loads(cmd[5]) + self.assertEqual(args_json.get("project_id"), "p") + return ( + 0, + json.dumps({ + "tool": "execute_sql_readonly", + "isError": False, + "result": { + "jobComplete": True, + "schema": { + "fields": [ + {"name": "job_id", "type": "STRING"}, + {"name": "project_id", "type": "STRING"}, + {"name": "user_email", "type": "STRING"}, + {"name": "job_type", "type": "STRING"}, + {"name": "statement_type", "type": "STRING"}, + {"name": "state", "type": "STRING"}, + {"name": "priority", "type": "STRING"}, + {"name": "reservation_id", "type": "STRING"}, + {"name": "edition", "type": "STRING"}, + {"name": "creation_ms", "type": "STRING"}, + {"name": "start_ms", "type": "STRING"}, + {"name": "end_ms", "type": "STRING"}, + {"name": "total_bytes_billed", "type": "STRING"}, + {"name": "total_bytes_processed", "type": "STRING"}, + {"name": "total_slot_ms", "type": "STRING"}, + {"name": "cache_hit", "type": "BOOLEAN"}, + {"name": "query", "type": "STRING"}, + {"name": "error_reason", "type": "STRING"}, + {"name": "error_message", "type": "STRING"}, + { + "name": "referenced_tables_json", + "type": "STRING", + }, + {"name": "labels_json", "type": "STRING"}, + ] + }, + "rows": [{ + "f": [ + {"v": "bqjob_mcp_1"}, + {"v": "p"}, + {"v": "alice@example.com"}, + {"v": "QUERY"}, + {"v": "SELECT"}, + {"v": "DONE"}, + {"v": "INTERACTIVE"}, + {"v": "res-1"}, + {"v": "ENTERPRISE"}, + {"v": "1791990000000"}, + {"v": "1791990000100"}, + {"v": "1791990005100"}, + {"v": str(1024**3)}, + {"v": str(1024**3)}, + {"v": "15000"}, + {"v": "false"}, + {"v": "SELECT * FROM `p.d.t`"}, + {"v": ""}, + {"v": ""}, + { + "v": json.dumps([{ + "project_id": "p", + "dataset_id": "d", + "table_id": "t", + }]) + }, + {"v": "[]"}, + ] + }], + }, + }), + "", + ) + if server == "google-managed-apache-spark" and tool == "list_batches": + return ( + 0, + json.dumps({ + "tool": "list_batches", + "isError": False, + "result": { + "batches": [{ + "batch_name": ( + "projects/p/locations/us-central1/batches/b1" + ), + "state": "SUCCEEDED", + "create_time": "2026-10-14T10:00:00Z", + "state_time": "2026-10-14T10:02:00Z", + "pysparkBatch": {"mainPythonFileUri": "gs://b/m.py"}, + }] + }, + }), + "", + ) + if server == "google-managed-apache-spark" and tool == "list_jobs": + return (0, json.dumps({"jobs": []}), "") + return (1, "", "unexpected tool") + + mcp_client = gcp_job_monitor.RemoteCliMcpClient( + runner=fake_runner, billing_project="p" + ) + bq_jobs, fell_back = gcp_job_monitor.fetch_bq_jobs( + mcp_client, "p", max_results=5 + ) + self.assertFalse(fell_back) + self.assertEqual(len(bq_jobs), 1) + self.assertEqual(bq_jobs[0]["jobId"], "bqjob_mcp_1") + self.assertEqual(bq_jobs[0]["bytesBilled"], 1024**3) + self.assertEqual(bq_jobs[0]["referencedTables"], ["p.d.t"]) + + spark_items, warnings = gcp_job_monitor.fetch_spark_workloads( + mcp_client, "p", "us-central1", max_results=5 + ) + self.assertEqual(warnings, []) + self.assertEqual(len(spark_items), 1) + self.assertEqual(spark_items[0]["id"], "b1") + self.assertEqual(len(recorded_cmds), 3) + + def test_remote_cli_mcp_client_get_job_403_sql_fallback(self): + def fake_runner(cmd): + server = cmd[2] + tool = cmd[3] + args = json.loads(cmd[5]) + if server == "bigquery" and tool == "get_job": + return ( + 0, + json.dumps({ + "tool": "get_job", + "isError": True, + "result": ( + "Access Denied: Project target-proj: User does not have" + " mcp.tools.call permission in project target-proj." + ), + }), + "", + ) + if server == "bigquery" and tool == "execute_sql_readonly": + self.assertEqual(args.get("project_id"), "billing-proj") + if "JOBS_BY_PROJECT" in args.get("query", ""): + return ( + 0, + json.dumps({ + "tool": "execute_sql_readonly", + "isError": True, + "result": ( + "Access Denied: Table" + " target-proj:region-us.INFORMATION_SCHEMA.JOBS_BY_PROJECT" + ), + }), + "", + ) + return ( + 0, + json.dumps({ + "tool": "execute_sql_readonly", + "isError": False, + "result": { + "schema": { + "fields": [ + {"name": "job_id", "type": "STRING"}, + {"name": "project_id", "type": "STRING"}, + {"name": "user_email", "type": "STRING"}, + {"name": "job_type", "type": "STRING"}, + {"name": "statement_type", "type": "STRING"}, + {"name": "state", "type": "STRING"}, + {"name": "priority", "type": "STRING"}, + {"name": "reservation_id", "type": "STRING"}, + {"name": "edition", "type": "STRING"}, + {"name": "creation_ms", "type": "STRING"}, + {"name": "start_ms", "type": "STRING"}, + {"name": "end_ms", "type": "STRING"}, + {"name": "total_bytes_billed", "type": "STRING"}, + {"name": "total_bytes_processed", "type": "STRING"}, + {"name": "total_slot_ms", "type": "STRING"}, + {"name": "cache_hit", "type": "BOOLEAN"}, + {"name": "query", "type": "STRING"}, + {"name": "error_reason", "type": "STRING"}, + {"name": "error_message", "type": "STRING"}, + { + "name": "referenced_tables_json", + "type": "STRING", + }, + {"name": "labels_json", "type": "STRING"}, + {"name": "job_stages_json", "type": "STRING"}, + ] + }, + "rows": [{ + "f": [ + {"v": "job_fallback_1"}, + {"v": "target-proj"}, + {"v": "testuser@example.com"}, + {"v": "QUERY"}, + {"v": "SELECT"}, + {"v": "DONE"}, + {"v": "INTERACTIVE"}, + {"v": ""}, + {"v": ""}, + {"v": "1791990000000"}, + {"v": "1791990000100"}, + {"v": "1791990002100"}, + {"v": "2048"}, + {"v": "2048"}, + {"v": "500"}, + {"v": "false"}, + {"v": "SELECT 42"}, + {"v": ""}, + {"v": ""}, + {"v": "[]"}, + {"v": "[]"}, + { + "v": json.dumps([{ + "id": 0, + "name": "S00: Input", + "status": "COMPLETE", + "records_read": 10, + "records_written": 1, + "slot_ms": 500, + }]) + }, + ] + }], + }, + }), + "", + ) + return (1, "", "unexpected tool") + + mcp_client = gcp_job_monitor.RemoteCliMcpClient( + runner=fake_runner, billing_project="billing-proj" + ) + job = gcp_job_monitor.fetch_bq_job_detail( + mcp_client, "target-proj", "job_fallback_1" + ) + self.assertEqual(job["jobId"], "job_fallback_1") + self.assertEqual(job["projectId"], "target-proj") + self.assertEqual(job["query"], "SELECT 42") + self.assertEqual(len(job["stages"]), 1) + self.assertEqual(job["stages"][0]["name"], "S00: Input") + self.assertEqual(job["stages"][0]["slotMs"], 500) + + +if __name__ == "__main__": + unittest.main() diff --git a/skills/gcp-managed-spark-upgrades/SKILL.md b/skills/gcp-managed-spark-upgrades/SKILL.md index 949540da..10cf552e 100644 --- a/skills/gcp-managed-spark-upgrades/SKILL.md +++ b/skills/gcp-managed-spark-upgrades/SKILL.md @@ -13,7 +13,7 @@ description: | license: Apache-2.0 metadata: - version: v1 + version: v2 publisher: google --- @@ -34,7 +34,7 @@ Your goal is to upgrade Spark jobs from one version to another while maintaining ## Task Execution Workflow > [!IMPORTANT] -> All GCP operations require active authentication. If you encounter authentication or permission errors, refer to `@skill:google-cloud-auth-verification` to resolve them. +> All GCP operations require active authentication. If you encounter authentication errors, ask the user to run `gcloud auth login` and `gcloud auth application-default login`. If an account is active but a call returns `403`, diagnose missing IAM roles instead. > [!NOTE] > All helper scripts in this skill maintain dependencies inline following the [PEP 723](https://peps.python.org/pep-0723/) specification. Execute all Python helper scripts using [uv](https://docs.astral.sh/uv/) (e.g., `uv run scripts/.py --help`), which automatically manages isolated environments and dependencies. Run scripts with `--help` to discover their interface rather than reading the source. diff --git a/skills/gcp-spark-troubleshooting/SKILL.md b/skills/gcp-spark-troubleshooting/SKILL.md index 81de48ff..b4fec24b 100644 --- a/skills/gcp-spark-troubleshooting/SKILL.md +++ b/skills/gcp-spark-troubleshooting/SKILL.md @@ -3,7 +3,7 @@ name: gcp-spark-troubleshooting description: "Provides expert guidance for troubleshooting Google Cloud Spark and Dataproc workloads (Dataproc Serverless batches and standard Dataproc clusters), and inspecting, streaming, searching, tailing, or summarizing Spark driver outputs and event logs in Cloud Storage. Use when the user asks to debug, troubleshoot, diagnose, or perform Root Cause Analysis (RCA) on failed Spark jobs, PySpark batches, or Spark event logs." license: Apache-2.0 metadata: - version: v3 + version: v4 publisher: google --- @@ -73,8 +73,7 @@ rather than reading its source. The scripts only gather evidence; none of them diagnoses or writes the RCA. Use only actions and flags that this file or `--help` lists; never invent one. -- **Required**: Python 3.8+ and an authenticated `gcloud` CLI (see - `@skill:google-cloud-auth-verification`). +- **Required**: Python 3.8+ and an authenticated `gcloud` CLI. - `spark_applications.py` and `spark_stage_diagnostics.py` call the Dataproc REST API with a token from `gcloud auth print-access-token`. If that token is rejected (HTTP 401, for example @@ -241,11 +240,12 @@ diagnostic plan or run any script before it passes.** 0. **Tooling & Environment Verification**: - - **Verify the `gcloud` CLI is present and authenticated.** Use - `@skill:google-cloud-auth-verification`. Pass condition: an active - account is returned. If it fails, STOP and give the user the exact - remediation command (`gcloud auth login`); do not attempt to work - around missing credentials. + - **Verify the `gcloud` CLI is present and authenticated** via `gcloud + auth list`. Pass condition: an active account is returned. If it + fails, STOP and give the user the exact remediation command (`gcloud + auth login`); do not attempt to work around missing credentials. If an + account is active but a call returns `403`, diagnose missing IAM roles + instead of asking the user to log in again. - **Verify a default project is resolvable** via `gcloud config get-value project`, unless the user supplied `` explicitly. Pass condition: a non-empty value that is not `(unset)`. @@ -267,9 +267,8 @@ diagnostic plan or run any script before it passes.** - Determine if the workload is a **Dataproc Serverless Batch**, a **Dataproc Cluster Job**, or a **Direct GCS URI**. - - If project or region is not specified, use - `@skill:google-cloud-auth-verification` or read the active gcloud - configuration. + - If project or region is not specified, read the active gcloud + configuration (`gcloud config list`). - Fetch metadata: - For Serverless Batches: run `gcloud dataproc batches describe --region= --format="json"`. diff --git a/skills/gcp-spark/SKILL.md b/skills/gcp-spark/SKILL.md index 44543776..2b1a3f4f 100755 --- a/skills/gcp-spark/SKILL.md +++ b/skills/gcp-spark/SKILL.md @@ -16,7 +16,7 @@ description: | - Troubleshooting failed Spark workloads (use @skill:gcp-spark-troubleshooting). license: Apache-2.0 metadata: - version: v24 + version: v26 publisher: google --- @@ -82,6 +82,13 @@ metadata: table does not exist, use `@skill:discovering-gcp-data-assets` to discover all similar tables in the namespace or project. + *MISSING SOURCE RULE*: If the request does not name a table, GCS path or + catalog, you MUST run at most ONE `@skill:discovering-gcp-data-assets` + search. If it does not return a single clear match, you MUST list the + candidates, ask the user which source to use, and STOP. You MUST NOT infer + the source by listing buckets, reading existing notebooks or scripts, or + running `SHOW NAMESPACES` / `SHOW TABLES`. + *MINOR TYPO RULE*: If there is a minor typo (e.g. `employees` vs `employee`), you can fix the error and proceed. @@ -153,12 +160,18 @@ metadata: ) ``` * **Security & Dynamic Variables**: You MUST NOT hardcode plaintext - credentials, passwords, or tokens (use Secret Manager). You MUST NOT - hardcode environment-specific variables; parameterize project IDs, - dataset IDs, table names, and bucket paths via `os.environ.get(...)` or - arguments. - * **Read and Write data**: **ALWAYS** Refer to - `references/read_write_data.md` when reading or writing data. + credentials, passwords, or tokens (use Secret Manager — refer to + `references/jdbc_cloudsql.md` when retrieving credentials from Secret + Manager). You MUST NOT hardcode environment-specific variables; + parameterize project IDs, dataset IDs, table names, and bucket paths via + `os.environ.get(...)` or arguments. + * **Read and Write data**: **ALWAYS** load only the connector reference + file(s) needed for the prompt: + - **BigQuery, GCS, or Pub/Sub**: `references/read_write_data.md` + - **BigLake Iceberg Catalog**: `references/biglake_iceberg.md` + - **Cloud Spanner**: `references/spanner.md` + - **JDBC & Cloud SQL (PostgreSQL / MySQL) & Secret Manager**: + `references/jdbc_cloudsql.md` * **Machine Learning Tasks**: Refer to `@skill:ml-best-practices` skill and `references/ml_tasks.md` when generating Machine Learning code. * **Spark Optimizations & Broadcast Joins**: **ALWAYS** refer to `references/spark_optimizations.md`. When joining a large DataFrame with @@ -232,6 +245,7 @@ The Managed Spark (Dataproc) service account needs: * `roles/bigquery.readSessionUser`: BigQuery Storage Read API session creation * `roles/storage.objectUser`: Read/write GCS (data, Iceberg warehouse, and `--deps-bucket` / `temporaryGcsBucket`) * `roles/spanner.databaseUser`: Spanner reads and writes +* `roles/secretmanager.secretAccessor`: Accessing database passwords and credentials stored in Secret Manager -------------------------------------------------------------------------------- diff --git a/skills/gcp-spark/references/biglake_iceberg.md b/skills/gcp-spark/references/biglake_iceberg.md new file mode 100644 index 00000000..3b0b91cb --- /dev/null +++ b/skills/gcp-spark/references/biglake_iceberg.md @@ -0,0 +1,325 @@ +# BigLake Iceberg Catalog in PySpark + +> [!IMPORTANT] +> You MUST ALWAYS follow the Task Execution Workflow in `SKILL.md` when writing Spark code. + +## Spark Session Configuration + +- [!WARNING] BigLake Iceberg catalog must be configured in the spark session + before tables can be read. +- Multiple Iceberg catalogs can be configured in the same spark session. +- [!IMPORTANT] You **MUST** identify the storage layer for the BigLake Iceberg + table before configuring the spark session. Use + `@skill:discovering-gcp-data-assets` to **search** and **lookup** the table + details. Check the value for the `metadataPath` field from the lookup + results to confirm it uses GCS storage (starts with `gs:`). +- [!WARNING] **DO NOT** assume catalog_name, project_id, io implementation for + a table, **ALWAYS lookup** table details before proceeding. + +### Option A: Zero-Config (Preferred) — Session Template / Runtime Profile + +When generating PySpark **notebook** code, assume the Dataproc kernel already +has the Iceberg catalog configured via its Session Template. **MUST NOT add +in-code catalog configuration.** Always use the simple `getOrCreate()` pattern: + +```python +# Catalog and BigLake settings are pre-configured by the Dataproc Session Template. +# No .config() calls are needed. +from google.cloud.managed_spark_connect import ManagedSparkSession +from google.cloud.dataproc_v1 import Session + +# 1. Initialize the Dataproc Session configuration object +session = Session() +session.session_template = "projects//locations//sessionTemplates/" +spark = ( + ManagedSparkSession.builder + .projectId("") + .location("") + .dataprocSessionConfig(session) + .getOrCreate() +) + +# Read directly from the catalog already registered by the kernel's session template +df = spark.read.format("iceberg").load("``..") +``` + +> [!IMPORTANT] When generating notebook code, **always use the pattern shown +> above**. Adding explicit `.config()` calls for catalogs that are already in +> the Session Template is redundant and can cause duplicate extension +> registration. + +#### Locating the Session Template and Catalog Name + +Use this high-level routing algorithm to determine `` and +``: + +- **If `` is already known** (passed in prompt or present in + existing code): go to **Case 1: Known Session Template**. +- **If `` is unknown** (only table name, Dataplex entry, or + warehouse is known): go to **Case 2: Unknown Session Template**. +- **If no template matches or input is incomplete**: go to **Case 3: No + Matching Template**. + +##### Case 1: Known Session Template + +Use the specified ``. If `` is not already known from +the code, inspect the template's properties: + +```bash +gcloud beta dataproc session-templates describe \ + --project= --location= \ + --format="json(runtimeConfig.properties)" +``` + +- If `spark.sql.defaultCatalog` is set, `` can be omitted + (`.`). +- If only one `spark.sql.catalog.` is configured in the + template, use that `` + (`..`). +- If multiple catalogs are configured, use the `` whose + `spark.sql.catalog..warehouse` matches the table's warehouse. + +##### Case 2: Unknown Session Template + +List templates in the project filtered by the table's warehouse URI (e.g. +`gs://` from Dataplex `metadataPath`) to get both `` and +``: + +```bash +gcloud beta dataproc session-templates list \ + --project= --location= \ + --filter="runtimeConfig.properties:" \ + --limit=5 \ + --format="json(name.basename(), runtimeConfig.properties)" +``` + +- If multiple templates match, prefer the template where `defaultCatalog` + matches the catalog name. + +##### Case 3: No Matching Template + +If no session template matches the warehouse, or if required inputs are missing: + +- **STOP and ask the user** which Session Template to use. +- Do **NOT** guess `` or add in-code `.config()` calls in + notebooks. + +### Option B: Explicit In-Code Configuration + +Use this only when no Session Template / Runtime Profile is available (e.g. +cluster and batch jobs, or environments without a pre-configured profile). You +**MUST NOT** use this in notebook code - ask user to configure and provide the +session template instead. Multiple Iceberg catalogs can be configured in the +same spark session. + +> [!IMPORTANT] +> - **Runtime JARs**: +> - **Dataproc Serverless**: Iceberg and `iceberg-gcp` (`GCSFileIO`) JARs are pre-installed on `/usr/lib/spark/jars/` by default. Do **NOT** pass `spark.jars.packages=org.apache.iceberg:...` when submitting serverless batches. +> - **Dataproc on GCE Clusters**: Enable `--optional-components=ICEBERG` at cluster creation (which adds `/usr/lib/iceberg/lib/*` to `SPARK_DIST_CLASSPATH`), or pass `/usr/lib/iceberg/lib/*` via `--jars` if submitting to a cluster without the optional component. +> - **Warehouse URI & Catalog Name**: For a GCS-backed BigLake catalog, the catalog name `` must match the GCS bucket name, and `spark.sql.catalog..warehouse` **MUST** be the root GCS bucket URI (`gs://`) without any subfolder path. Passing a subfolder (e.g. `gs:///warehouse`) causes `BadRequestException: Bucket name is invalid`. +> - **Catalog Provisioning & IAM**: Before creating namespaces or writing tables, ensure the BigLake Iceberg catalog exists (`gcloud biglake iceberg catalogs create --catalog-type=gcs-bucket --project=`) and the execution service account has `roles/biglake.admin` and `roles/storage.objectUser` on the catalog bucket. + +Example with GCS storage: + +```python +spark = ( + SparkSession.builder + .appName("") + .config( + "spark.sql.extensions", + "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", + ) + .config( + "spark.sql.catalog.", + "org.apache.iceberg.spark.SparkCatalog", + ) + .config("spark.sql.catalog..type", "rest") + .config( + "spark.sql.catalog..uri", + "https://biglake.googleapis.com/iceberg/v1/restcatalog", + ) + .config( + "spark.sql.catalog..warehouse", + "gs://", + ) + .config( + "spark.sql.catalog..header.x-goog-user-project", + "", + ) + .config( + "spark.sql.catalog..rest.auth.type", + "org.apache.iceberg.gcp.auth.GoogleAuthManager", + ) + .config( + "spark.sql.catalog..io-impl", + "org.apache.iceberg.gcp.gcs.GCSFileIO", + ) + .getOrCreate() +) +``` + +## Reading from BigLake Iceberg Catalog + +- [!WARNING] BigLake Iceberg catalog must be configured before tables can be + read (via Session Template or explicit configuration). +- If not using a fully qualified 3-part table identifier (or if + `spark.sql.defaultCatalog` is not set), set the current Spark catalog using + `spark.catalog.setCurrentCatalog("")`. **DO NOT** use + backticks in this call. +- [!IMPORTANT] You **MUST ALWAYS** surround the catalog name with backticks in + SQL statements and `load()` calls to ensure proper handling of special + characters (e.g., `` `catalog-name` ``). + +```python +df = ( + spark.read.format("iceberg") + .load("``..") +) +``` + +## Writing to BigLake Iceberg Catalog + +> [!IMPORTANT] Follow these instructions before writing code to write to BigLake +> Iceberg tables. + +- [!WARNING] BigLake Iceberg catalog must be configured before tables can be + written (via Session Template or explicit configuration). +- **ALWAYS** use Spark's `DataFrameWriterV2` API (`writeTo`) for writing to + Iceberg tables. This provides a richer API with specific write modes. +- If not using a fully qualified 3-part table identifier (or if + `spark.sql.defaultCatalog` is not set), set the current Spark catalog using + `spark.catalog.setCurrentCatalog("")`. **DO NOT** use + backticks in this call. +- **ALWAYS** Ensure namespace is created before writing, using: + + ```python + spark.sql( + "CREATE NAMESPACE IF NOT EXISTS ``." + ) + ``` + > [!IMPORTANT] You **MUST ALWAYS** surround the catalog name with backticks + > in SQL statements and `writeTo()` calls to ensure proper handling of + > special characters. + +```python +# The basic pattern +spark.catalog.setCurrentCatalog("") +( + df.writeTo("``..") + .using("iceberg") + .createOrReplace() +) +``` + +### Write Modes + +- **DataFrameWriterV2 `createOrReplace()`**: Creates the table if it does not + exist, or replaces it if it does. + + ```python + ( + df.writeTo("``..") + .using("iceberg") + .createOrReplace() + ) + ``` + +- **DataFrameWriterV2 `append()`**: Equivalent to `INSERT INTO`. Appends data + to an existing table. + + ```python + ( + df.writeTo("``..") + .using("iceberg") + .append() + ) + ``` + +- **SPARK SQL MERGE INTO**: Performs row-level updates, inserts, and deletes + by rewriting data files that contain affected rows. The source table can be + persisted as a temp view to allow using it in the MERGE INTO statement. The + source view and the target table must be joined on the primary key(s) to + ensure correct row-level updates, inserts, and deletes. + + ```python + df.createOrReplaceTempView("source_updates") + + spark.sql(f""" + MERGE INTO ``.. t + USING source_updates s + ON t.id = s.id + WHEN MATCHED THEN UPDATE SET * + WHEN NOT MATCHED THEN INSERT * + """) + ``` + +- [!WARNING] These are less preferred write modes, use them ONLY if explicitly + asked to: + * **`create()`**: Equivalent to `CREATE TABLE AS SELECT`. Fails if the + table already exists. + + ```python + df.writeTo("``..") \ + .using("iceberg") \ + .create() + ``` + + * **`replace()`**: Equivalent to `REPLACE TABLE AS SELECT`. + + ```python + df.writeTo("``..") \ + .using("iceberg") \ + .replace() + ``` + + * **`overwritePartitions()`**: Equivalent to dynamic `INSERT OVERWRITE`. + Overwrites partitions in the table that match the data in the DataFrame. + + ```python + df.writeTo("``..") \ + .using("iceberg") \ + .overwritePartitions() + ``` + + * **`overwrite(condition)`**: Overwrites data that matches a specific + condition (e.g. static overwrite). + + ```python + from pyspark.sql.functions import col + df.writeTo("``..") \ + .using("iceberg") \ + .overwrite(col("level") == "INFO") + ``` + +### Advanced Iceberg Writes + +#### Writing with partitions +Use partitioning transform functions if writing to partitioned tables using +timestamp fields. Ex: years, months, days, hours etc. + +```python +from pyspark.sql.functions import days + +df.writeTo("``..") \ + .using("iceberg") \ + .partitionedBy("level", days("ts")) \ + .createOrReplace() +``` + +#### Schema Evolution +Iceberg supports safe schema evolution. You can merge schema changes during an +append. + +```python +df.writeTo("``..") \ + .option("mergeSchema", "true") \ + .append() +``` + +#### Writing to a Specific Branch + +```python +# To insert into an `audit` branch +df.writeTo("``...branch_audit") \ + .append() +``` diff --git a/skills/gcp-spark/references/gcloud_dataproc.md b/skills/gcp-spark/references/gcloud_dataproc.md index a64b6aec..565675c6 100755 --- a/skills/gcp-spark/references/gcloud_dataproc.md +++ b/skills/gcp-spark/references/gcloud_dataproc.md @@ -215,10 +215,14 @@ For asynchronous execution, you must poll the batch status until state is - **Dependency**: `--jars=gs://spark-lib/spanner/spark-3.5-spanner-1.4.0.jar` - **Notes**: Pass `.option("projectId", ...)` in PySpark -#### PostgreSQL +#### PostgreSQL & JDBC Databases -- **Dependency**: `spark.jars.packages=org.postgresql:postgresql:42.6.0` -- **Notes**: Pass `--subnet=...` for private IP +- **Dependency**: + - **GCS-Staged Driver JAR (`gs://...`)**: Pass `--jars=gs:///.jar` on the `gcloud` CLI or set `spark.jars` (for `ManagedSparkSession`, set `session_config.runtime_config.properties["spark.jars"] = "gs:///.jar"`). Never pass `gs://` URIs to `spark.jars.packages`. + - **Maven Coordinates (requires outbound internet/Cloud NAT)**: `--properties="spark.jars.packages=::"` +- **Notes**: + - Pass `--subnet=` when connecting to a database on a VPC Private IP. + - Retrieve database passwords from Google Cloud Secret Manager (either via the pre-installed `google.auth` + `urllib.request` REST API or by passing `--properties="spark.dataproc.pip.packages=google-cloud-secret-manager=="` on Dataproc Serverless, where `google-cloud-secret-manager` is not pre-installed). #### Iceberg REST diff --git a/skills/gcp-spark/references/jdbc_cloudsql.md b/skills/gcp-spark/references/jdbc_cloudsql.md new file mode 100644 index 00000000..2e69eeff --- /dev/null +++ b/skills/gcp-spark/references/jdbc_cloudsql.md @@ -0,0 +1,172 @@ +# JDBC Databases & Cloud SQL in PySpark + +> [!IMPORTANT] +> You MUST ALWAYS follow the Task Execution Workflow in `SKILL.md` when writing Spark code. + +## 1. Attaching a Custom JDBC Driver JAR (`gs://...`) + +JDBC drivers are not pre-installed on Dataproc and must be attached at session +creation (`gs://` JAR URIs cannot be passed to `spark.jars.packages`, which only +accepts Maven coordinates): + +- **Interactive Notebooks (`.ipynb`) & Spark Connect**: Attach the driver JAR + via a Session Template or `session_config.runtime_config.properties["spark.jars"]` + at session creation (setting `spark.jars` after a Spark Connect session has + started does not register the JDBC driver with the JVM): + + ```python + from google.cloud.dataproc_v1 import Session + from google.cloud.managed_spark_connect import ManagedSparkSession + + session_config = Session() + session_config.runtime_config.properties["spark.jars"] = ( + "gs:///jars/.jar" + ) + spark = ( + ManagedSparkSession.builder.projectId("") + .location("") + .dataprocSessionConfig(session_config) + .getOrCreate() + ) + ``` + +- **Dataproc Batch & Cluster Jobs (`.py`)**: Pass + `--jars=gs:///jars/.jar` (and + `--subnet=` for private IP databases) on the `gcloud` CLI, and set + `spark.jars` on `SparkSession.builder`: + + ```python + jar_uri = os.environ.get( + "JAR_URI", "gs:///jars/.jar" + ) + spark = ( + SparkSession.builder.appName("") + .config("spark.jars", jar_uri) + .getOrCreate() + ) + ``` + +### Common JDBC Driver Classes & URL Formats + +| Database Engine | Driver Class (`driver`) | JDBC URL Format (`url`) | +| :--- | :--- | :--- | +| **PostgreSQL / AlloyDB** | `org.postgresql.Driver` | `jdbc:postgresql://:5432/` | +| **MySQL** | `com.mysql.cj.jdbc.Driver` | `jdbc:mysql://:3306/` | +| **SQL Server** | `com.microsoft.sqlserver.jdbc.SQLServerDriver` | `jdbc:sqlserver://:1433;databaseName=;encrypt=true;trustServerCertificate=true` | + +## 2. Retrieving Database Passwords from Secret Manager + +> [!WARNING] NEVER embed plaintext secrets or passwords directly in code or +> scripts, and do not rely on unset local environment variables in remote +> Dataproc jobs. Always retrieve database passwords from Google Cloud Secret +> Manager. + +On Dataproc Serverless, `google-cloud-secret-manager` is not pre-installed +by default. Either pass +`--properties="spark.dataproc.pip.packages=google-cloud-secret-manager=="` +at submission time and use `SecretManagerServiceClient`, or fetch the secret via +the pre-installed `google.auth` and `urllib.request` libraries: + +```python +import base64 +import json +import urllib.request +import google.auth +from google.auth.transport.requests import Request + + +def get_secret(project_id: str, secret_id: str, version: str = "latest") -> str: + creds, _ = google.auth.default( + scopes=["https://www.googleapis.com/auth/cloud-platform"] + ) + creds.refresh(Request()) + url = ( + f"https://secretmanager.googleapis.com/v1/projects/{project_id}" + f"/secrets/{secret_id}/versions/{version}:access" + ) + req = urllib.request.Request( + url, headers={"Authorization": f"Bearer {creds.token}"} + ) + with urllib.request.urlopen(req) as resp: + data = json.loads(resp.read().decode("utf-8"))["payload"]["data"] + return base64.b64decode(data).decode("utf-8").strip() +``` + +If `google-cloud-secret-manager` is installed in the environment: + +```python +from google.cloud import secretmanager + + +def get_secret(project_id: str, secret_id: str, version: str = "latest") -> str: + client = secretmanager.SecretManagerServiceClient() + name = f"projects/{project_id}/secrets/{secret_id}/versions/{version}" + response = client.access_secret_version(request={"name": name}) + return response.payload.data.decode("utf-8").strip() +``` + +## 3. Parallel Partitioned JDBC Reads + +Calling `spark.read.format("jdbc").load()` without partitioning options reads +the entire table on a single executor thread. For parallel JDBC reads on a +numeric, date, or timestamp column (``): + +1. **Dynamically query `MIN` and `MAX` bounds first** (never hardcode or guess + `lowerBound` / `upperBound`). +2. **Read the table in parallel** using `partitionColumn`, `lowerBound`, + `upperBound`, `numPartitions`, and `fetchsize`: + +```python +import logging +import os + +logging.basicConfig( + level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s" +) + +project_id = os.environ.get("PROJECT_ID", "") +secret_id = os.environ.get("SECRET_ID", "") +jdbc_url = os.environ.get( + "JDBC_URL", "jdbc:://:/" +) +jdbc_driver = os.environ.get("JDBC_DRIVER", "") +db_user = os.environ.get("DB_USER", "") +table_name = os.environ.get("TABLE_NAME", "") +partition_col = os.environ.get("PARTITION_COL", "") +db_password = get_secret(project_id, secret_id) + +# Step 1: Dynamically fetch MIN and MAX bounds for the partition column +bounds_query = ( + f"(SELECT MIN({partition_col}) AS min_id, MAX({partition_col}) AS max_id " + f"FROM {table_name}) AS bounds_q" +) +bounds_row = ( + spark.read.format("jdbc") + .option("url", jdbc_url) + .option("dbtable", bounds_query) + .option("user", db_user) + .option("password", db_password) + .option("driver", jdbc_driver) + .load() + .first() +) +lower_bound = str(bounds_row["min_id"]) +upper_bound = str(bounds_row["max_id"]) +logging.info("Resolved bounds: lower=%s, upper=%s", lower_bound, upper_bound) + +# Step 2: Parallel partitioned read across executors +df = ( + spark.read.format("jdbc") + .option("url", jdbc_url) + .option("dbtable", table_name) + .option("user", db_user) + .option("password", db_password) + .option("driver", jdbc_driver) + .option("partitionColumn", partition_col) + .option("lowerBound", lower_bound) + .option("upperBound", upper_bound) + .option("numPartitions", "4") + .option("fetchsize", "10000") + .load() +) +``` diff --git a/skills/gcp-spark/references/read_write_data.md b/skills/gcp-spark/references/read_write_data.md index 7add390d..3ce0fca7 100755 --- a/skills/gcp-spark/references/read_write_data.md +++ b/skills/gcp-spark/references/read_write_data.md @@ -1,4 +1,4 @@ -# Definitive Guide to reading and writing tables using spark code +# Definitive Guide to Reading and Writing Data in PySpark > [!IMPORTANT] > You MUST ALWAYS follow the Task Execution Workflow when writing spark code. @@ -6,8 +6,22 @@ ## Task Execution Workflow 1. Configure the spark session to read or write data from the required source - or destination -2. Read or write the data from the required source or destination + or destination. +2. Read or write the data from the required source or destination. + +## Connector Reference Routing + +Load only the reference file(s) needed for your target data source or +destination: + +- **BigQuery, Google Cloud Storage (GCS), & Pub/Sub**: Covered below in this + file (`references/read_write_data.md`). +- **BigLake Iceberg Catalog**: Refer to `references/biglake_iceberg.md`. +- **Cloud Spanner**: Refer to `references/spanner.md`. +- **JDBC & Cloud SQL (PostgreSQL / MySQL) & Secret Manager Credentials**: + Refer to `references/jdbc_cloudsql.md`. + +-------------------------------------------------------------------------------- ## BigQuery @@ -24,7 +38,8 @@ via `spark.jars.packages` when running on Dataproc. `SparkSession.builder` (see `SKILL.md`). ### Reading from BigQuery -#### Basic example to read a Big Query table + +#### Basic example to read a BigQuery table ```python df = spark.read.format("bigquery") \ @@ -33,8 +48,10 @@ df = spark.read.format("bigquery") \ ``` #### Executing a BigQuery SQL query + The Spark BigQuery connector allows you to run Standard SQL SELECT queries directly on BigQuery and load the results into a spark dataframe. + > [!IMPORTANT] You must set `viewsEnabled` to `"true"` and specify a writable > `materializationDataset` in your project to run direct SQL queries or read > BigQuery views. @@ -57,6 +74,7 @@ df = ( ``` #### BigQuery Join Patterns + When enriching data with BigQuery reference tables: ```python @@ -75,7 +93,7 @@ enriched_df = main_df.join( #### Optimization Tips -1. Select only needed columns from BigQuery +1. Select only needed columns from BigQuery: ```python ref_df = spark.read.format("bigquery") \ @@ -85,7 +103,7 @@ ref_df = spark.read.format("bigquery") \ .select("id", "name") ``` -2. Using broadcast joins for small reference tables +2. Using broadcast joins for small reference tables: ```python from pyspark.sql.functions import broadcast @@ -101,10 +119,11 @@ enriched_df = main_df.join( ### Writing to BigQuery #### Non partitioned data -Write DataFrame to BigQuery using direct writes, - - Set writeMethod to direct - - Set mode to one of `overwrite`, `append`, `errorifexists` based on task +Write DataFrame to BigQuery using direct writes: + +- Set `writeMethod` to `direct` +- Set `mode` to one of `overwrite`, `append`, `errorifexists` based on task ```python df.write \ @@ -114,355 +133,35 @@ df.write \ .mode("overwrite") \ .save() ``` + #### Partitioned data + Write DataFrame to a partitioned BigQuery table using indirect writes. In this method, data is written to GCS first and then loaded into BigQuery. -A GCS bucket is required as the temporary data location. +A GCS bucket is required as the temporary data location: - - Set temporaryGcsBucket to a gcs bucket name - - Set a check point path location +- Set `temporaryGcsBucket` to a GCS bucket name +- Set a `checkpointLocation` path ```python df.write \ .format("bigquery") \ - .option("temporaryGcsBucket","") \ + .option("temporaryGcsBucket", "") \ .option("checkpointLocation", "") \ .save("..") ``` -## BigLake Iceberg Catalog - -### Spark Session Configuration - -- [!WARNING] BigLake Iceberg catalog must be configured in the spark session - before tables can be read. -- Multiple Iceberg catalogs can be configured in the same spark session -- [!IMPORTANT] You **MUST** identify the storage layer for the BigLake Iceberg - table before configuring the spark session. Use - `@skill:discovering-gcp-data-assets` to **search** and **lookup** the table - details. Check the value for the `metadataPath` field from the lookup - results to confirm it uses GCS storage (starts with `gs:`). -- [!WARNING] **DO NOT** assume catalog_name, project_id, io implementation for a - table, **ALWAYS lookup** table details before proceeding. - -#### Option A: Zero-Config (Preferred) — Session Template / Runtime Profile - -When generating PySpark **notebook** code, assume the Dataproc kernel already -has the Iceberg catalog configured via its Session Template. **MUST NOT add -in-code catalog configuration.** Always use the simple `getOrCreate()` pattern: - -```python -# Catalog and BigLake settings are pre-configured by the Dataproc Session Template. -# No .config() calls are needed. -from google.cloud.managed_spark_connect import ManagedSparkSession -from google.cloud.dataproc_v1 import Session - -# 1. Initialize the Dataproc Session configuration object -session = Session() -session.session_template = "projects//locations//sessionTemplates/" -spark = ( - ManagedSparkSession.builder - .projectId("") - .location("") - .dataprocSessionConfig(session) - .getOrCreate() -) - -# Read directly from the catalog already registered by the kernel's session template -df = spark.read.format("iceberg").load("``..") -``` - -> [!IMPORTANT] When generating notebook code, **always use the pattern shown -> above**. Adding explicit `.config()` calls for catalogs that are already in -> the Session Template is redundant and can cause duplicate extension -> registration. - -##### Locating the Session Template and Catalog Name - -Use this high-level routing algorithm to determine `` and -``: - -- **If `` is already known** (passed in prompt or present in - existing code): go to **Case 1: Known Session Template**. -- **If `` is unknown** (only table name, Dataplex entry, or - warehouse is known): go to **Case 2: Unknown Session Template**. -- **If no template matches or input is incomplete**: go to **Case 3: No - Matching Template**. - -###### Case 1: Known Session Template - -Use the specified ``. If `` is not already known from -the code, inspect the template's properties: - -```bash -gcloud beta dataproc session-templates describe \ - --project= --location= \ - --format="json(runtimeConfig.properties)" -``` - -- If `spark.sql.defaultCatalog` is set, `` can be omitted - (`.`). -- If only one `spark.sql.catalog.` is configured in the - template, use that `` - (`..`). -- If multiple catalogs are configured, use the `` whose - `spark.sql.catalog..warehouse` matches the table's warehouse. - -###### Case 2: Unknown Session Template - -List templates in the project filtered by the table's warehouse URI (e.g. -`gs://` from Dataplex `metadataPath`) to get both `` and -``: - -```bash -gcloud beta dataproc session-templates list \ - --project= --location= \ - --filter="runtimeConfig.properties:" \ - --limit=5 \ - --format="json(name.basename(), runtimeConfig.properties)" -``` - -- If multiple templates match, prefer the template where `defaultCatalog` - matches the catalog name. - -###### Case 3: No Matching Template - -If no session template matches the warehouse, or if required inputs are missing: - -- **STOP and ask the user** which Session Template to use. -- Do **NOT** guess `` or add in-code `.config()` calls in - notebooks. - -#### Option B: Explicit In-Code Configuration - -Use this only when no Session Template / Runtime Profile is available (e.g. -cluster and batch jobs, or environments without a pre-configured profile). You -**MUST NOT** use this in notebook code - ask user to configure and provide the -session template instead. Multiple Iceberg catalogs can be configured in the -same spark session. - -> [!IMPORTANT] -> - **Runtime JARs**: -> - **Dataproc Serverless**: Iceberg and `iceberg-gcp` (`GCSFileIO`) JARs are pre-installed on `/usr/lib/spark/jars/` by default. Do **NOT** pass `spark.jars.packages=org.apache.iceberg:...` when submitting serverless batches. -> - **Dataproc on GCE Clusters**: Enable `--optional-components=ICEBERG` at cluster creation (which adds `/usr/lib/iceberg/lib/*` to `SPARK_DIST_CLASSPATH`), or pass `/usr/lib/iceberg/lib/*` via `--jars` if submitting to a cluster without the optional component. -> - **Warehouse URI & Catalog Name**: For a GCS-backed BigLake catalog, the catalog name `` must match the GCS bucket name, and `spark.sql.catalog..warehouse` **MUST** be the root GCS bucket URI (`gs://`) without any subfolder path. Passing a subfolder (e.g. `gs:///warehouse`) causes `BadRequestException: Bucket name is invalid`. -> - **Catalog Provisioning & IAM**: Before creating namespaces or writing tables, ensure the BigLake Iceberg catalog exists (`gcloud biglake iceberg catalogs create --catalog-type=gcs-bucket --project=`) and the execution service account has `roles/biglake.admin` and `roles/storage.objectUser` on the catalog bucket. - -Example with GCS storage: - -```python -spark = ( - SparkSession.builder - .appName("") - .config( - "spark.sql.extensions", - "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", - ) - .config( - "spark.sql.catalog.", - "org.apache.iceberg.spark.SparkCatalog", - ) - .config("spark.sql.catalog..type", "rest") - .config( - "spark.sql.catalog..uri", - "https://biglake.googleapis.com/iceberg/v1/restcatalog", - ) - .config( - "spark.sql.catalog..warehouse", - "gs://", - ) - .config( - "spark.sql.catalog..header.x-goog-user-project", - "", - ) - .config( - "spark.sql.catalog..rest.auth.type", - "org.apache.iceberg.gcp.auth.GoogleAuthManager", - ) - .config( - "spark.sql.catalog..io-impl", - "org.apache.iceberg.gcp.gcs.GCSFileIO", - ) - .getOrCreate() -) -``` - -### Reading from BigLake Iceberg Catalog - -- [!WARNING] BigLake Iceberg catalog must be configured before tables can be - read (via Session Template or explicit configuration). -- If not using a fully qualified 3-part table identifier (or if - `spark.sql.defaultCatalog` is not set), set the current Spark catalog using - `spark.catalog.setCurrentCatalog("")`. **DO NOT** use - backticks in this call. -- [!IMPORTANT] You **MUST ALWAYS** surround the catalog name with backticks in - SQL statements and `load()` calls to ensure proper handling of special - characters (e.g., `` `catalog-name` ``). - -```python -df = ( - spark.read.format("iceberg") - .load("``..") -) -``` - -### Writing to BigLake Iceberg Catalog - -> [!IMPORTANT] Follow these instructions before writing code to write to BigLake -> Iceberg tables. - -- [!WARNING] BigLake Iceberg catalog must be configured before tables can be - written (via Session Template or explicit configuration). -- **ALWAYS** use Spark's `DataFrameWriterV2` API (`writeTo`) for writing to - Iceberg tables. This provides a richer API with specific write modes. -- If not using a fully qualified 3-part table identifier (or if - `spark.sql.defaultCatalog` is not set), set the current Spark catalog using - `spark.catalog.setCurrentCatalog("")`. **DO NOT** use - backticks in this call. -- **ALWAYS** Ensure namespace is created before writing, using: - - ```python - spark.sql( - "CREATE NAMESPACE IF NOT EXISTS ``." - ) - ``` - > [!IMPORTANT] You **MUST ALWAYS** surround the catalog name with backticks - > in SQL statements and `writeTo()` calls to ensure proper handling of - > special characters. - -```python -# The basic pattern -spark.catalog.setCurrentCatalog("") -( - df.writeTo("``..") - .using("iceberg") - .createOrReplace() -) -``` - -#### Write Modes - -- **DataFrameWriterV2 `createOrReplace()`**: Creates the table if it does not - exist, or replaces it if it does. - - ```python - ( - df.writeTo("``..") - .using("iceberg") - .createOrReplace() - ) - ``` - -- **DataFrameWriterV2 `append()`**: Equivalent to `INSERT INTO`. Appends data to - an existing table. - - ```python - ( - df.writeTo("``..") - .using("iceberg") - .append() - ) - ``` - -- **SPARK SQL MERGE INTO**: Performs row-level updates, inserts, and deletes by - rewriting data files that contain affected rows. The source table can be - persisted as a temp view to allow using it in the MERGE INTO statement. The - source view and the target table must be joined on the primary key(s) to - ensure correct row-level updates, inserts, and deletes. - - ```python - df.createOrReplaceTempView("source_updates") - - spark.sql(f""" - MERGE INTO ``.. t - USING source_updates s - ON t.id = s.id - WHEN MATCHED THEN UPDATE SET * - WHEN NOT MATCHED THEN INSERT * - """) - ``` - -- [!WARNING] These are less preferred write modes, use them ONLY if explicitly - asked to: -* **`create()`**: Equivalent to `CREATE TABLE AS SELECT`. Fails if the table - already exists. - - ```python - df.writeTo("``..") \ - .using("iceberg") \ - .create() - ``` - -* **`replace()`**: Equivalent to `REPLACE TABLE AS SELECT`. - - ```python - df.writeTo("``..") \ - .using("iceberg") \ - .replace() - ``` - -* **`overwritePartitions()`**: Equivalent to dynamic `INSERT OVERWRITE`. - Overwrites partitions in the table that match the data in the DataFrame. - - ```python - df.writeTo("``..") \ - .using("iceberg") \ - .overwritePartitions() - ``` - -* **`overwrite(condition)`**: Overwrites data that matches a specific - condition (e.g. static overwrite). - - ```python - from pyspark.sql.functions import col - df.writeTo("``..") \ - .using("iceberg") \ - .overwrite(col("level") == "INFO") - ``` - -#### Advanced Iceberg Writes - -##### Writing with partitions -Use partitioning transform functions if writing to partitioned tables using -timestamp fields. Ex: years, months, days, hours etc. -Example: - -```python -from pyspark.sql.functions import days - -df.writeTo("``..") \ - .using("iceberg") \ - .partitionedBy("level", days("ts")) \ - .createOrReplace() -``` - -##### Schema Evolution -Iceberg supports safe schema evolution. You can merge schema changes during an -append. - -```python -df.writeTo("``..") \ - .option("mergeSchema", "true") \ - .append() -``` - -##### Writing to a Specific Branch - -```python -# To insert into an `audit` branch -df.writeTo("``...branch_audit") \ - .append() -``` - ## Google Cloud Storage (GCS) -No session configuration is needed to read from GCS, all jars and + +No session configuration is needed to read from GCS; all JARs and configuration are provided by default in Dataproc. ### Reading from GCS #### CSV format -Read with `header` and `inferSchema` without these, the header row becomes data -and all columns are strings + +Read with `header` and `inferSchema`; without these, the header row becomes data +and all columns are strings: ```python df = spark.read.option("header", "true") \ @@ -471,137 +170,28 @@ df = spark.read.option("header", "true") \ ``` #### Parquet format + `df = spark.read.parquet("gs:///")` -#### Json format +#### JSON format + `df = spark.read.json("gs:///")` ### Writing to GCS -#### Csv format +#### CSV format + `df.write.csv("gs://bucket/output_path")` #### Model artifacts ```python # Warehouse already includes gs://, don't add it again -warehouse = spark.conf.get(f"spark.sql.catalog..warehouse") +warehouse = spark.conf.get("spark.sql.catalog..warehouse") model_path = f"{warehouse}/models/my_model" model.write().overwrite().save(model_path) ``` -## Spanner -### Spark Session Configuration -The Spanner connector JAR is not pre-installed and must be attached at session -creation (`gs://` JAR URIs cannot be passed to `spark.jars.packages`, which only -accepts Maven coordinates). Ensure the Spanner JAR's Spark version prefix -(e.g., `spark-3.5-spanner-*` for Dataproc runtime `2.3` / Spark 3.5, or -`spark-4.0-spanner-*` for Dataproc runtime `3.0` / Spark 4.0) matches the -Dataproc runtime version: - -- **Interactive Notebooks (`.ipynb`) & Spark Connect**: Attach the JAR via a - Session Template or `session_config.runtime_config.properties["spark.jars"]` - (and ensure `session_config.runtime_config.version` matches the JAR's Spark - version, as `ManagedSparkSession` defaults to runtime `3.0`): - - ```python - from google.cloud.dataproc_v1 import Session - from google.cloud.managed_spark_connect import ManagedSparkSession - - session_config = Session() - session_config.runtime_config.version = "2.3" - session_config.runtime_config.properties["spark.jars"] = ( - "gs://spark-lib/spanner/spark-3.5-spanner-1.4.0.jar" - ) - spark = ( - ManagedSparkSession.builder.projectId("") - .location("") - .dataprocSessionConfig(session_config) - .getOrCreate() - ) - ``` - -- **Dataproc Batch & Cluster Jobs (`.py`)**: Pass - `--jars=gs://spark-lib/spanner/spark-3.5-spanner-1.4.0.jar` on the `gcloud` - CLI (see `references/gcloud_dataproc.md`), or set `spark.jars`: - - ```python - spark = ( - SparkSession.builder.appName("") - .config( - "spark.jars", - "gs://spark-lib/spanner/spark-3.5-spanner-1.4.0.jar", - ) - .getOrCreate() - ) - ``` - -#### Reading Spanner tables - -```python -df = ( - spark.read.format("cloud-spanner") - .option("projectId", "") - .option("instanceId", "") - .option("databaseId", "") - .option("table", "") - .load() -) -``` - -### Writing to Spanner -The connector supports writing to Spanner tables using the DataFrame `write` -API. - -#### Write Modes - -* **`Append`**: Inserts rows into the existing Spanner table. This is the - default write mode. - - ```python - df.write.format("cloud-spanner") \ - .option("projectId", "") \ - .option("instanceId", "") \ - .option("databaseId", "") \ - .option("table", "") \ - .mode("append") \ - .save() - ``` - -* **`Overwrite`**: Clears the existing data before writing. - - ```python - df.write.format("cloud-spanner") \ - .option("projectId", "") \ - .option("instanceId", "") \ - .option("databaseId", "") \ - .option("table", "") \ - .mode("overwrite") \ - .save() - ``` - -## Cloud SQL (PostgreSQL / MySQL) - -### Reading from Cloud SQL via JDBC - -> [!WARNING] NEVER embed plaintext secrets or passwords directly in code or -> scripts. Always retrieve credentials securely via environment variables (or -> Secret Manager). - -```python -import os - -password = os.environ.get("DB_PASSWORD", "") - -df = spark.read.format("jdbc") \ - .option("url", "jdbc:postgresql://:5432/") \ - .option("dbtable", "") \ - .option("user", "") \ - .option("password", password) \ - .option("driver", "org.postgresql.Driver") \ - .load() -``` - ## Google Cloud Pub/Sub ### Batch Ingestion via Pub/Sub Client diff --git a/skills/gcp-spark/references/spanner.md b/skills/gcp-spark/references/spanner.md new file mode 100644 index 00000000..30c63fc1 --- /dev/null +++ b/skills/gcp-spark/references/spanner.md @@ -0,0 +1,95 @@ +# Cloud Spanner in PySpark + +> [!IMPORTANT] +> You MUST ALWAYS follow the Task Execution Workflow in `SKILL.md` when writing Spark code. + +## Spark Session Configuration + +The Spanner connector JAR is not pre-installed and must be attached at session +creation (`gs://` JAR URIs cannot be passed to `spark.jars.packages`, which only +accepts Maven coordinates). Ensure the Spanner JAR's Spark version prefix +(e.g., `spark-3.5-spanner-*` for Dataproc runtime `2.3` / Spark 3.5, or +`spark-4.0-spanner-*` for Dataproc runtime `3.0` / Spark 4.0) matches the +Dataproc runtime version: + +- **Interactive Notebooks (`.ipynb`) & Spark Connect**: Attach the JAR via a + Session Template or `session_config.runtime_config.properties["spark.jars"]` + (and ensure `session_config.runtime_config.version` matches the JAR's Spark + version, as `ManagedSparkSession` defaults to runtime `3.0`): + + ```python + from google.cloud.dataproc_v1 import Session + from google.cloud.managed_spark_connect import ManagedSparkSession + + session_config = Session() + session_config.runtime_config.version = "2.3" + session_config.runtime_config.properties["spark.jars"] = ( + "gs://spark-lib/spanner/spark-3.5-spanner-1.4.0.jar" + ) + spark = ( + ManagedSparkSession.builder.projectId("") + .location("") + .dataprocSessionConfig(session_config) + .getOrCreate() + ) + ``` + +- **Dataproc Batch & Cluster Jobs (`.py`)**: Pass + `--jars=gs://spark-lib/spanner/spark-3.5-spanner-1.4.0.jar` on the `gcloud` + CLI (see `references/gcloud_dataproc.md`), or set `spark.jars`: + + ```python + spark = ( + SparkSession.builder.appName("") + .config( + "spark.jars", + "gs://spark-lib/spanner/spark-3.5-spanner-1.4.0.jar", + ) + .getOrCreate() + ) + ``` + +## Reading Spanner Tables + +```python +df = ( + spark.read.format("cloud-spanner") + .option("projectId", "") + .option("instanceId", "") + .option("databaseId", "") + .option("table", "") + .load() +) +``` + +## Writing to Spanner + +The connector supports writing to Spanner tables using the DataFrame `write` +API. + +### Write Modes + +* **`Append`**: Inserts rows into the existing Spanner table. This is the + default write mode. + + ```python + df.write.format("cloud-spanner") \ + .option("projectId", "") \ + .option("instanceId", "") \ + .option("databaseId", "") \ + .option("table", "") \ + .mode("append") \ + .save() + ``` + +* **`Overwrite`**: Clears the existing data before writing. + + ```python + df.write.format("cloud-spanner") \ + .option("projectId", "") \ + .option("instanceId", "") \ + .option("databaseId", "") \ + .option("table", "") \ + .mode("overwrite") \ + .save() + ``` diff --git a/skills/gcs-security-assessment/SKILL.md b/skills/gcs-security-assessment/SKILL.md index 062b4f77..128fbb70 100644 --- a/skills/gcs-security-assessment/SKILL.md +++ b/skills/gcs-security-assessment/SKILL.md @@ -14,7 +14,7 @@ description: >- access failure or 403, managing or configuring storage, investigating a live outage, or non-GCS resources (Compute Engine, GKE, etc.). metadata: - version: v3 + version: v4 publisher: google tags: - gcs diff --git a/skills/gcs-security-assessment/references/phases/output.md b/skills/gcs-security-assessment/references/phases/output.md index 8662bccf..9689ea3a 100644 --- a/skills/gcs-security-assessment/references/phases/output.md +++ b/skills/gcs-security-assessment/references/phases/output.md @@ -6,7 +6,7 @@ Present your assessment in a scannable, action-oriented format. The target user is a Storage Admin with limited security expertise who needs to quickly understand: what's wrong, how bad is it, and how to fix it. They may -action remediations via the Cloud Console (Pantheon) or gcloud CLI — both paths +action remediations via the Google Cloud Console or gcloud CLI — both paths should be clear. ## Output Structure diff --git a/skills/google-cloud-auth-verification/SKILL.md b/skills/google-cloud-auth-verification/SKILL.md deleted file mode 100755 index cf100cbd..00000000 --- a/skills/google-cloud-auth-verification/SKILL.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -name: google-cloud-auth-verification -description: >- - Mandatory Step 0 pre-flight execution order and authentication verification for Google Cloud Platform (GCP), - Application Default Credentials (ADC), gcloud CLI, Spark, Dataproc, BigQuery, GCS, and notebook runtimes. - Use whenever interacting with GCP resources, running Spark/PySpark pipelines, BigQuery queries, GCS paths (gs://), or creating/running notebooks. -license: Apache-2.0 -metadata: - version: v3 - publisher: google ---- - -# Google Cloud Authentication Guidelines - -## Mandatory Pre-Flight Execution & Auth Hierarchy - -> [!IMPORTANT] **Pre-Flight Execution Priority Order**: Before generating code, -> implementation plans, or executing tasks for any GCP or Notebook workload: -> -> 1. **Skip Pre-Flight Shell Probes When Live Notebook Execution or MCP Tools** -> **Are Available**: -> - When a live notebook cell execution tool (`execute_cell` / -> `notebook_execute_cell` / `notebook__execute_cell`) is available, or -> when the task is handled directly by a connected MCP tool -> (`bigquery__execute_sql`, `bigquery__get_table_info`, -> `bigquery__list_table_ids`, etc.), **do NOT run a pre-flight -> `gcloud auth list && gcloud config list` shell probe**. Instead, -> verify authentication reactively only if a cell execution or MCP tool -> call returns an authentication or credentials error (`401`, `403`, -> `DefaultCredentialsError`, `Reauthentication is needed`). -> - When `gcloud` or `bq` shell commands are required, or when generating -> standalone scripts or notebooks without a live cell execution tool -> (`execute_cell`), verify credentials upfront via the single bundled -> probe (`gcloud auth list && gcloud config list`) or Application -> Default Credentials (ADC). -> 2. **Distinguish Authentication vs. IAM Permissions**: -> - If `gcloud auth list` returns `No credentialed accounts`, **HARD -> STOP** immediately and instruct the user to run `gcloud auth login` -> and `gcloud auth application-default login`. -> - If Python throws `google.auth.exceptions.DefaultCredentialsError`, -> explicitly direct the user to run `gcloud auth application-default -> login`. -> - If `gcloud auth list` shows an active credentialed account but a -> BigQuery/GCP call returns `403 Forbidden: Access Denied`, **DO NOT** -> tell the user to log in again with `gcloud auth login`. Diagnose -> missing IAM roles (e.g., `roles/bigquery.dataEditor`) on the active -> account. -> 3. **HARD STOP if Unauthenticated**: If no active GCP credentials or valid -> `gcloud` authentication are detected, **STOP IMMEDIATELY**. Prompt the -> user to run `gcloud auth login` and `gcloud auth application-default -> login`. Do NOT attempt local virtualenv creation, package installation, or -> local binary/JDK setup loops as workarounds. - -## Common Error Messages - -1. **gcloud/bq CLI**: - - `ERROR: (bq) You do not currently have an active account selected.` - - `No credentialed accounts.` - - `Configuration error: No account is currently active.` -2. **Execution Failures (Python/Notebooks)**: - - `google.auth.exceptions.DefaultCredentialsError: Could not automatically - determine credentials.` - - `Forbidden: 403 Access Denied` (when it's clearly an auth issue). - -## Verification Step - -Before asking the user to log in, independently verify authentication status -using a single bundled probe command: - -```bash -gcloud auth list --format="json" && gcloud config list --format="json" -``` - -* If the output contains `No credentialed accounts.` or missing active - account, proceed to **Corrective Action**. -* If an account *is* listed but the user still receives a `403 Access Denied` - error, the issue is likely **IAM permissions** (e.g., missing BigQuery - roles) on their active account, rather than missing authentication. In this - case, investigate permissions rather than asking them to log in again. - -## Corrective Action - -When missing credentials are confirmed, **DO NOT** attempt to fix credentials -via code or local virtualenv workarounds. Credentials must be established by the -user. - -**Stop and ask the user to run the following commands in their terminal:** - -1. **To authenticate the gcloud CLI**: `gcloud auth login` -2. **To set up Application Default Credentials (ADC)** (required for BQ CLI AND - most libraries/notebooks): `gcloud auth application-default login` - -## Post-Login Verification - -After the user confirms they have logged in, verify with: `gcloud auth list` -Then proceed with the original task.