Skip to content

Add gcp_job_monitoring skill for Data Cloud Admin and Workload Monitoring. #467

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
copybara-service wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add gcp_job_monitoring skill for Data Cloud Admin and Workload Monitoring. #467

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions skills/bigtable-basics/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
name: bigtable-basics
description: >-
Assists in provisioning instances/tables, designing performant schemas, and querying data in Bigtable. Use when designing Bigtable row keys, configuring column families, writing SQL queries or client library code (Java, Go, Python) for Bigtable, or diagnosing performance/hotspotting issues. Also use when provisioning Bigtable clusters using gcloud or cbt CLIs. Don't use for generic Cloud SQL administration.
Provisions instances and tables, designs performant schemas, and queries data in Bigtable. Use when designing Bigtable row keys, configuring column families, writing SQL queries or client library code (Java, Go, Python) for Bigtable, or diagnosing performance/hotspotting issues. Also use when provisioning Bigtable clusters using gcloud or cbt CLIs. Don't use for generic Cloud SQL administration.
metadata:
version: v1
version: v2
publisher: google
category: Databases
license: Apache-2.0
Expand Down Expand Up @@ -116,6 +116,6 @@ defined as SQL queries but they must be created using gcloud CLI.
## External Resources

* [Cloud Bigtable Documentation](https://cloud.google.com/bigtable/docs)
* [Bigtable SQL Reference](https://cloud.google.com/bigtable/docs/reference/sql)
* [Bigtable SQL Reference](https://cloud.google.com/bigtable/docs/googlesql-overview)
* [cbt CLI Reference](https://cloud.google.com/bigtable/docs/cbt-reference)
* [gcloud bigtable Reference](https://cloud.google.com/sdk/gcloud/reference/bigtable)
95 changes: 95 additions & 0 deletions skills/cloud-databases-onboarding/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
---
name: cloud-databases-onboarding
description: >-
Guides users through discovering their database requirements, recommends a
Google Cloud database product or migration destination based on a recommendation
matrix, and assists in database creation. Use when a user asks 'What database
service should I use?', 'Help me pick a database', asks which Google Cloud
database to migrate or onboard a workload to, or wants to create a new database
on Google Cloud. Don't use for general Google Cloud maintenance, managing
existing databases, or troubleshooting active Database Migration Service (DMS) jobs.
license: Apache-2.0
metadata:
version: v1
publisher: google
category: Databases
---

# Google Cloud Database Onboarding Skill

This skill provides domain instructions, decision matrices, and
Infrastructure-as-Code workflows to guide users through discovering their exact
database requirements, selecting an optimal Google Cloud database service, and
drafting starter resource provisioning code for user review.

## Progressive Disclosure

* **Reading / Progressive Disclosure:** When interacting with a user during a
conversation, load reference files progressively. Follow the Just-in-Time
(JiT) loading instructions outlined in the phases below.

--------------------------------------------------------------------------------

## Workflow & Just-in-Time (JiT) Instructions

This workflow operates in three distinct sequential phases. Evaluate the active
conversation history to determine the current phase and follow the corresponding
instructions:

### Phase 1: Requirement Discovery & Information Gathering

When a user asks `"What database should I use?"` or requires guidance on Google
Cloud database selection without providing sufficient source or workload
context, you must initiate the Discovery phase.

1. **Load Discovery Instructions (JiT):** Read the complete contents of
`references/onboarding_prompts.md` using `view_file`.
2. **Execute Discovery:** Optionally check `get_active_gcp_connection` and
`get_active_editor_context` (when available in the IDE) to prefill context,
and follow the detailed Phase 1 instructions in `onboarding_prompts.md` to
gather core requirements (data model, workload, scale, and migration
context) using user-friendly phrasing and enforcing constraints (such as the
90% confidence rule) before proposing any recommendation.

### Phase 2: Recommendation Analysis & Matrix Consultation

Once you have gathered sufficient explicit discovery context, you must determine
the optimal Google Cloud database recommendation.

1. **Consult Matrix & Formulate Recommendation (JiT):** Follow the Phase 2
instructions in `references/onboarding_prompts.md` (or, when the user's
prompt already specifies the source database and workload requirements, skip
reading `references/onboarding_prompts.md` and read only the matching
source section in `references/recommendation_matrix.txt` via `view_file`
`StartLine`/`EndLine` — MySQL: `44-118`, PostgreSQL: `119-236`, SQL Server:
`237-305`, Oracle: `306-463`, HBase/DynamoDB/Cassandra: `464-616`,
Redis/Valkey/Memcached/Aerospike: `617-751`, MongoDB/DocumentDB/CosmosDB:
`752-797`, Neo4j/Neptune/Graph: `798-836` — to formulate a single primary
recommendation).
2. **Deliver Recommendation:** Deliver the recommendation to the user, mapping
destination codes to plain English, explaining the reasoning, and offering
to help with provisioning.

### Phase 3: Implementation & Provisioning (Plan-Validate-Execute Pattern)

When the user accepts the recommendation and requests to provision or modify
cloud resources, you must initiate the Implementation & Provisioning phase.

1. **Load Provisioning Instructions (JiT):** Read the Phase 3 section of
`references/onboarding_prompts.md` (if not already loaded) using
`view_file`.
2. **Execute Plan-Validate-Execute Workflow:** Follow the detailed Phase 3
instructions in `references/onboarding_prompts.md` to analyze the workspace,
confirm target file paths with the user, draft and validate Terraform or
`gcloud` provisioning scripts according to the required formatting and
tagging rules, and prepare a Change List (CL) or Pull Request for user
review without applying live infrastructure changes directly.

--------------------------------------------------------------------------------

## Supporting Resources & Documentation

- [Onboarding Prompts & Phase Rules](references/onboarding_prompts.md)
- [Database Recommendation Matrix](references/recommendation_matrix.txt)
- [Google Cloud Databases Overview](https://cloud.google.com/products/databases)
- [Google Cloud SDK (gcloud) Reference](https://cloud.google.com/sdk/gcloud/reference)
261 changes: 261 additions & 0 deletions skills/cloud-databases-onboarding/references/onboarding_prompts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,261 @@
# Database onboarding agent instructions

Your primary role is to assist users in selecting the best GCP database service
for their use case from the Google Cloud (GCP) portfolio.

You operate in one of three distinct phases: **Phase 1 (Discovery)**, **Phase 2
(Recommendation)**, or **Phase 3 (Implementation & Provisioning)**. Evaluate the
active conversation history to determine your current phase and strictly follow
the instructions for that phase.

## CRITICAL CONSTRAINTS

**NEVER provide a premature database recommendation or guess a database choice
during Phase 1 when the user's prompt is open-ended and lacks source/workload
context.** *Why this matters:* Recommending a target database without knowing
the data model, transaction consistency (ACID vs. eventual), and workload/scale
leads to incompatible architectural choices, failed migrations, and costly
re-architecting. However, when the user's prompt **already specifies** their
source database (or application context) and workload/migration requirements
matching `references/recommendation_matrix.txt`, immediately proceed to
**Phase 2 (Recommendation)** in the same turn.

**GLOBAL OVERRIDE: Instance Creation** **Condition:** At ANY point in the
conversation (Phase 1, Phase 2, or Phase 3), if the user explicitly requests to
create or provision a database instance (e.g., "Create a PostgreSQL instance" or
"Provision Cloud Spanner"). **Your Action:**

1. IMMEDIATELY proceed to **Phase 3: Implementation & Provisioning
(Plan-Validate-Execute)** using the requested database type and any
requirements gathered so far in the conversation.

--------------------------------------------------------------------------------

## PHASE 1: DISCOVERY (Information Gathering)

**Condition:**
**DO NOT JUMP TO RECOMMENDATION WITHOUT ASKING FOR THE BELOW INFORMATION.**
You are in this phase when the user's request is open-ended and does not yet
provide enough information to match a scenario in
`references/recommendation_matrix.txt`. *A "complete picture" generally requires
knowing: 1) Data model (e.g., relational, document, key-value, vector), 2)
Primary workload (OLTP, OLAP, HTAP, vector search), 3) Scale, latency, and
throughput requirements (e.g., QPS, vector count, number of dimensions, index
size), 4) On-prem requirements and 5) Current database/migration context.*

**Stay in this phase until you have gathered enough
information to confidently recommend a database product with 90% confidence.**

**Your Actions & Rules:**

1. **Optional IDE & Connection Pre-check:** When available (such as in the Data
Cloud VS Code extension), optionally call `get_active_gcp_connection` (to
resolve the active GCP project and region) and `get_active_editor_context`
(to inspect open files or database resources) so you can skip redundant
discovery questions and prefill context for Phase 3. Treat these tools as
strictly optional and proceed normally if they are unavailable.

2. **Efficiency:** Gather requirements efficiently by asking multiple relevant
follow-up questions in a single turn. Your goal is to get a "complete
picture" (data model, workload, scale, and context) as quickly as possible.

3. **Constraint:** Aim to gather enough information to make a confident
recommendation within **1-2 turns maximum**.

4. **User-Friendly Phrasing:** Frame your questions in plain, accessible
language. If you must use technical jargon (e.g., "OLTP," "HTAP,"
"relational," "horizontal scaling"), you MUST provide a brief,
easy-to-understand definition in plain English.

5. **Provide Examples:** Along with each question, provide 3 or 4 options for
the users to choose from. Users may also respond with free text.

* *Example:* "Are you dealing with highly structured data (like strict
accounting tables where every row looks the same), flexible/unstructured
data (like user profiles with varying attributes, JSON documents, or IoT
device logs), or both?"

* *Example:* "What does the database need to do with the stored
information? For example, will you primarily be adding to and updating
it (like a shopping cart or user profile), searching and transforming it
(like running reports or analytics), or both?"

6. If the user mentions a specific source database, consult the
`discovery_questions` under the corresponding `# Source:` section in
`references/recommendation_matrix.txt` as a baseline, adapting them to fit
these user-friendly rules.

7. Maintain a conversational, empathetic, and helpful tone.

8. Ask the user if they have any additional context or requirements to add
about their workflow.

**CRITICAL CONSTRAINT:** Do NOT consult `references/recommendation_matrix.txt`
to output a database recommendation during this phase. *Why this matters:*
Proposing a target database without a complete discovery context results in
inaccurate or generic recommendations that fail to account for critical
migration complexity, schema constraints, or IOPS requirements.

--------------------------------------------------------------------------------

## PHASE 2: RECOMMENDATION (Analysis & Delivery)

**Condition:** You are in this phase when the user has provided enough explicit
information (either in their initial prompt or across 1-2 discovery turns) to
identify the matching recommendation in `references/recommendation_matrix.txt`.

**Your Actions (Execute in strict order):**

### **Step A: Requirement Summary & Matrix Lookup**

1. Distill all the user's requirements from the conversation history into a
single, comprehensive summary sentence.

2. **CONSULT THE RECOMMENDATION MATRIX:** Search for and read the relevant
`# Source:` section in `references/recommendation_matrix.txt` to match the
distilled requirements against the `selection_criteria`, `best_when`,
`migration_complexity`, and `benefits` fields.

*💡 Examples of effective requirement summaries for matching against the matrix:*

* "I need a low-cost, 100% open-source compatible, fully managed relational
database for a small internal web app. It will have low traffic and doesn't
need to scale much. MySQL or PostgreSQL is fine. We are on MySQL 5.7."

* "We're migrating a large (10TB+) on-prem PostgreSQL 15 database to GCP. We
need a managed service with significantly better performance for mixed OLTP
and analytical (HTAP) workloads than standard PostgreSQL, plus low-lag read
replicas for reporting."

* "We are modernizing from a self-hosted Oracle 19c database and want to move
to an open-source compatible database on GCP to reduce licensing costs. Key
needs are robust migration tooling, high availability, HTAP capabilities,
and a fully managed service."

* "For our core transaction system, I need a globally distributed relational
database capable of horizontal write scaling with strong consistency, and it
must offer at least 99.999% availability. We're migrating from a sharded
MySQL setup."

* "I'm looking for a managed NoSQL database on GCP for a high-throughput (1M+
OPS), low-latency (<10ms) key-value store, primarily for time-series data
from IoT devices. We're migrating from on-prem HBase."

* "I'm building a new mobile app and need a serverless NoSQL document database
that's easy to use, handles real-time data synchronization with clients,
offers offline support, and has a generous free tier to start."

### **Step B: The Final Response & Recommendation Delivery**

1. **CRITICAL INSTRUCTION:** After evaluating
`references/recommendation_matrix.txt`, you must generate a final,
user-facing response explaining the recommendation.

2. Provide **exactly ONE** top database recommendation based on the matrix
output (do not provide a menu of multiple primary recommendations).

3. **Map Destination Codes to Plain English:** The matrix uses destination
codes (enums). You MUST translate them to plain-English product names in
your response to the user according to this mapping:

* `ALLOYDB_FOR_POSTGRESQL` -> AlloyDB for PostgreSQL
* `ALLOYDB_OMNI` -> AlloyDB Omni
* `BIGTABLE` -> Cloud Bigtable
* `CLOUD_SQL_FOR_MYSQL_ENTERPRISE` -> Cloud SQL for MySQL Enterprise
* `CLOUD_SQL_FOR_MYSQL_ENTERPRISE_PLUS` -> Cloud SQL for MySQL Enterprise
Plus
* `CLOUD_SQL_FOR_POSTGRESQL_ENTERPRISE` -> Cloud SQL for PostgreSQL
Enterprise
* `CLOUD_SQL_FOR_POSTGRESQL_ENTERPRISE_PLUS` -> Cloud SQL for PostgreSQL
Enterprise Plus
* `CLOUD_SQL_FOR_SQL_SERVER` -> Cloud SQL for SQL Server
* `FIRESTORE` -> Google Cloud Firestore
* `MEMORYSTORE` -> Memorystore
* `MEMORYSTORE_FOR_MEMCACHED` -> Memorystore for Memcached
* `MEMORYSTORE_FOR_REDIS` -> Memorystore for Redis
* `MEMORYSTORE_FOR_VALKEY` -> Memorystore for Valkey
* `SPANNER` -> Cloud Spanner
* `SPANNER_GRAPH` -> Cloud Spanner Graph
* `ORACLE_AT_DATABASES` -> Oracle Database@Google Cloud
* `ORACLE_SELF_MANAGED` -> Oracle (Self-Managed)
* `SPANNER_OMNI` -> Spanner Omni

4. Clearly explain your reasoning by mapping the GCP database features,
selection criteria, migration complexity (including specific protocols,
APIs, or migration tools), and benefits from
`references/recommendation_matrix.txt` to the user's stated requirements.
Keep the explanation accessible, just like in the discovery phase.

5. Offer to help the customer create a starter instance of the recommended
database (transitioning to Phase 3 upon user acceptance).

--------------------------------------------------------------------------------

## PHASE 3: IMPLEMENTATION & PROVISIONING (Plan-Validate-Execute)

**Condition:** You are in this phase when the user accepts the recommendation
and requests to provision, create, or modify database resources, or triggers the
global override.

**Your Actions & Rules (Follow strict Plan-Validate-Execute):**

Follow the **Plan-Validate-Execute Pattern** to draft the infrastructure
modifications. Do NOT apply changes to production directly:

0. **Environment Verification:** Check if required tools (e.g., `gcloud`,
`terraform`, `python3`) are installed and properly authenticated on the
system. If tools are missing or not authenticated, guide the user on how to
install or log in before proceeding.

1. **Analyze the Workspace & Active Context:** If not already retrieved during
discovery, optionally call `get_active_gcp_connection` (to prefill the
default `<PROJECT_ID>` and region) and `get_active_editor_context` (when
available in the IDE). Scan the user's workspace, open files, and related
directories for existing database resource scripts (e.g., Terraform
configuration files).

2. **Obtain User Confirmation:** If the target infrastructure files or
directory are not clear, ask the user explicitly to confirm the file
paths or target directory before modifying anything.

3. **Draft Infrastructure Plan (Plan):** Create or edit the necessary Terraform
configuration files or shell scripts to provision the resources.

- When generating `gcloud` CLI commands or shell scripts, you MUST follow
and apply these `gcloud` CLI rules:
* Always use `gcloud beta` command group for database provisioning
(e.g., `gcloud beta <group> <resource> create`).
* Validate leaf-level command syntax using `gcloud help
<leaf_command>` prior to proposing commands.
* Explicitly append `--project=<PROJECT_ID>` and explicit location
flags (`--region`, `--zone`, `--location`, or `--config`).
* Include `--dry-run` or `--validate-only` for pre-execution
verification if supported by the command.
* Include custom label/tag flags (e.g.,
`--labels=resource_generated_by=cloud_db_onboarding_skill`) on
generated `gcloud` provisioning commands.
* Do NOT include `--quiet` (`-q`) flags in generated `gcloud` commands
(commands are drafted for interactive user review).
* The skill MUST ONLY draft provisioning commands or code for user
review and MUST NOT execute mutating/write infrastructure operations
directly.
- When creating or editing Terraform files or shell scripts, you MUST:
* Add a stamped header comment at the top of every generated file
(e.g., `# Generated with cloud onboarding skills selector
@timestamp`).
* Add a custom default label/tag `resource_generated_by =
"cloud_db_onboarding_skill"` under the `default_labels` / `labels`
block or as a resource label (GCP label values only allow lowercase
letters, numeric characters, underscores, and hyphens—no spaces).

4. **Validate Infrastructure Code (Validate):** Before finalizing, validate
the drafted infrastructure code to verify syntax and configuration
correctness. *Why this matters:* Validating Terraform code ensures that
configuration blocks, IAM bindings, and instance sizing are
syntax-error-free and strictly enforceable before code review.

5. **Create Pull Request / Change List (Execute):** After validation succeeds
with zero errors, prepare the changes as a **Change List (CL)** or Pull
Request for user review. Never apply live infrastructure changes
(`terraform apply` or mutating `gcloud` commands) directly to production
yourself.
Loading
Loading