From 319d8ba176b9890428a844a12575146b33bb8a95 Mon Sep 17 00:00:00 2001 From: Data Cloud Agents Team Date: Thu, 15 Oct 2026 18:42:23 -0700 Subject: [PATCH] feat: Update BigQuery AI skills/references in data_agent_common PiperOrigin-RevId: 1000533955 --- skills/bigquery-ai-ml/SKILL.md | 148 +++- .../references/ai_causal_effect.md | 178 ++++ .../references/ai_detect_anomalies.md | 7 +- .../bigquery-ai-ml/references/ai_evaluate.md | 2 +- .../bigquery-ai-ml/references/ai_forecast.md | 52 +- .../bigquery-ai-ml/references/ai_generate.md | 48 +- .../references/ai_generate_embedding.md | 17 +- .../references/ai_generate_table.md | 26 +- .../references/ai_key_drivers.md | 27 +- .../bigquery-ai-ml/references/ai_predict.md | 231 +++++ skills/bigquery-ai-ml/references/ai_search.md | 2 +- .../references/ml_correlation.md | 185 ++++ .../references/ml_describe_data.md | 73 ++ .../references/ml_detect_change_points.md | 111 +++ .../references/ml_seasonality.md | 124 +++ skills/bigquery-ai-ml/references/ml_trend.md | 82 ++ .../references/remote_models.md | 17 +- skills/bigtable-basics/SKILL.md | 6 +- skills/cloud-databases-onboarding/SKILL.md | 95 ++ .../references/onboarding_prompts.md | 261 ++++++ .../references/recommendation_matrix.txt | 836 ++++++++++++++++++ skills/gcp-managed-spark-upgrades/SKILL.md | 4 +- skills/gcp-spark-troubleshooting/SKILL.md | 21 +- skills/gcp-spark/SKILL.md | 28 +- .../gcp-spark/references/biglake_iceberg.md | 325 +++++++ .../gcp-spark/references/gcloud_dataproc.md | 10 +- skills/gcp-spark/references/jdbc_cloudsql.md | 172 ++++ .../gcp-spark/references/read_write_data.md | 500 +---------- skills/gcp-spark/references/spanner.md | 95 ++ skills/gcs-security-assessment/SKILL.md | 2 +- .../references/phases/output.md | 2 +- .../google-cloud-auth-verification/SKILL.md | 96 -- 32 files changed, 3062 insertions(+), 721 deletions(-) create mode 100644 skills/bigquery-ai-ml/references/ai_causal_effect.md create mode 100644 skills/bigquery-ai-ml/references/ai_predict.md create mode 100644 skills/bigquery-ai-ml/references/ml_correlation.md create mode 100644 skills/bigquery-ai-ml/references/ml_describe_data.md create mode 100644 skills/bigquery-ai-ml/references/ml_detect_change_points.md create mode 100644 skills/bigquery-ai-ml/references/ml_seasonality.md create mode 100644 skills/bigquery-ai-ml/references/ml_trend.md create mode 100644 skills/cloud-databases-onboarding/SKILL.md create mode 100644 skills/cloud-databases-onboarding/references/onboarding_prompts.md create mode 100644 skills/cloud-databases-onboarding/references/recommendation_matrix.txt create mode 100644 skills/gcp-spark/references/biglake_iceberg.md create mode 100644 skills/gcp-spark/references/jdbc_cloudsql.md create mode 100644 skills/gcp-spark/references/spanner.md delete mode 100755 skills/google-cloud-auth-verification/SKILL.md diff --git a/skills/bigquery-ai-ml/SKILL.md b/skills/bigquery-ai-ml/SKILL.md index e945cf42..8c2c1395 100644 --- a/skills/bigquery-ai-ml/SKILL.md +++ b/skills/bigquery-ai-ml/SKILL.md @@ -2,12 +2,19 @@ name: bigquery-ai-ml license: Apache-2.0 metadata: - version: v2 + version: v3 publisher: google description: >- Leverages BigQuery's built-in machine learning and GenAI capabilities for advanced data analytics. Use when you need to write SQL queries - that perform time-series forecasting, detect outliers, find key drivers, or leverage generative AI capabilities in BigQuery. + that perform time-series forecasting, predict values, detect outliers or anomalies, find key drivers, + perform semantic search or vector search, classify text, calculate similarity, + summarize content, translate language, evaluate models, filter by semantic conditions, + measure the causal effect of an intervention, compute correlations between columns, + generate descriptive statistics and table distribution profiles, + detect change points or structural breaks, extract trend or seasonality components, + or leverage generative AI capabilities in BigQuery. Do not use for general + BigQuery dataset, table, or job management requests. --- # BigQuery AI & ML @@ -22,48 +29,99 @@ like `AI.FORECAST`, `AI.KEY_DRIVERS`, `AI.DETECT_ANOMALIES`, and `AI.GENERATE`. ## Reference Directory -- **Best Practices**: - [ai_function_best_practices.md](references/ai_function_best_practices.md) +- **Best Practices**: [ai_function_best_practices.md](references/ai_function_best_practices.md) - Global constraints, connection settings, and mandatory syntax checks for BQML queries in notebooks. -- **Functions Reference**: +### Functions Reference - - **AI.AGG**: [ai_agg.md](references/ai_agg.md) - Multi-row semantic - aggregation and summarization. - - **AI.CLASSIFY**: [ai_classify.md](references/ai_classify.md) - - Classify text. - - **AI.DETECT_ANOMALIES**: - [ai_detect_anomalies.md](references/ai_detect_anomalies.md) - - Detect anomalies. - - **AI.EVALUATE**: [ai_evaluate.md](references/ai_evaluate.md) - - Evaluate models. - - **AI.FORECAST**: [ai_forecast.md](references/ai_forecast.md) - - Time-series forecasting. - - **AI.GENERATE**: [ai_generate.md](references/ai_generate.md) - - Generate text using LLMs. - - **AI.GENERATE_EMBEDDING**: - [ai_generate_embedding.md](references/ai_generate_embedding.md) - - Generate embeddings. - - **AI.GENERATE_TABLE**: - [ai_generate_table.md](references/ai_generate_table.md) - - Table-valued AI generation. - - **AI.IF**: [ai_if.md](references/ai_if.md) - Evaluate semantic - conditions. - - **AI.KEY_DRIVERS**: - [ai_key_drivers.md](references/ai_key_drivers.md) - Identifies key - drivers, this is a TVF. - - **AI.SCORE**: [ai_score.md](references/ai_score.md) - Score data. - - **AI.SEARCH**: [ai_search.md](references/ai_search.md) - Semantic - search. - - **AI.SIMILARITY**: - [ai_similarity.md](references/ai_similarity.md) - Semantic - similarity. - - **Remote Models**: - [remote_models.md](references/remote_models.md) - Working with - remote models (Vertex AI). - - **CONTRIBUTION_ANALYSIS**: - [ml_contribution_analysis.md](references/ml_contribution_analysis.md) - - Finds contributing factors, key drivers of change. Requires creating - a MODEL entity. - - **VECTOR_SEARCH**: - [vector_search.md](references/vector_search.md) - Vector search - best practices. +- **AI.AGG** ([ai_agg.md](references/ai_agg.md)) + - **SQL Shape:** Aggregate + - **Requires CREATE MODEL:** No + - **Usage:** Multi-row semantic aggregation with `GROUP BY`; synthesizes 1 summary string across many rows (unlike per-row `AI.GENERATE`). +- **AI.CAUSAL_EFFECT** ([ai_causal_effect.md](references/ai_causal_effect.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** No + - **Usage:** Quantifies counterfactual post-intervention impact on a time series after a specific treatment timestamp. +- **AI.CLASSIFY** ([ai_classify.md](references/ai_classify.md)) + - **SQL Shape:** Scalar + - **Requires CREATE MODEL:** No + - **Usage:** Semantic classification of text or multimodal unstructured data into predefined category labels. +- **AI.DETECT_ANOMALIES** ([ai_detect_anomalies.md](references/ai_detect_anomalies.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** No + - **Usage:** Zero-shot **TimesFM** detector for isolated time-series point anomalies/outliers (use `ML.DETECT_CHANGE_POINTS` for sustained regime shifts). +- **AI.EVALUATE** ([ai_evaluate.md](references/ai_evaluate.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** No + - **Usage:** Zero-shot **TimesFM** time-series forecast evaluation/backtesting against a historical holdout split. +- **AI.FORECAST** ([ai_forecast.md](references/ai_forecast.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** No + - **Usage:** Zero-shot **TimesFM** time-series forecasting without training a custom model. +- **AI.GENERATE** ([ai_generate.md](references/ai_generate.md)) + - **SQL Shape:** Scalar + - **Requires CREATE MODEL:** No + - **Usage:** Default for per-row LLM text or multimodal structured extraction (`output_schema`); returns `STRUCT`. Prefer over `AI.GENERATE_TABLE`. +- **AI.GENERATE_EMBEDDING** ([ai_generate_embedding.md](references/ai_generate_embedding.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** Yes + - **Usage:** Batch generates `ARRAY` vector embeddings for text or visual content using a remote embedding model. +- **AI.GENERATE_TABLE** ([ai_generate_table.md](references/ai_generate_table.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** Yes + - **Usage:** Table-valued structured extraction (`FROM` clause); only use instead of `AI.GENERATE` when a remote model + TVF is explicitly needed. +- **AI.IF** ([ai_if.md](references/ai_if.md)) + - **SQL Shape:** Scalar + - **Requires CREATE MODEL:** No + - **Usage:** Evaluates boolean semantic conditions (`BOOL`) on text or multimodal data for `WHERE`/`CASE`/`JOIN` filtering (do not use `AI.GENERATE_TABLE`). +- **AI.KEY_DRIVERS** ([ai_key_drivers.md](references/ai_key_drivers.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** No + - **Usage:** Default one-step attribution of dimensional segments driving metric changes (`interest_label_col` `TRUE` vs. `FALSE`). Prefer over `CONTRIBUTION_ANALYSIS`. +- **AI.PREDICT** ([ai_predict.md](references/ai_predict.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** No + - **Usage:** Zero-shot **TabFM** foundation model for tabular classification or regression without training and managing custom models. +- **AI.SCORE** ([ai_score.md](references/ai_score.md)) + - **SQL Shape:** Scalar + - **Requires CREATE MODEL:** No + - **Usage:** Assigns a continuous `FLOAT64` semantic score to text or multimodal data for ranking or sorting (`ORDER BY`). +- **AI.SEARCH** ([ai_search.md](references/ai_search.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** No + - **Usage:** Simplified semantic search using a raw string query on tables with autonomous background embeddings (`AI.EMBED` generated column). +- **AI.SIMILARITY** ([ai_similarity.md](references/ai_similarity.md)) + - **SQL Shape:** Scalar + - **Requires CREATE MODEL:** No + - **Usage:** Computes an on-the-fly semantic similarity score (`FLOAT64`) between two text or multimodal inputs without stored embeddings. +- **Remote Models** ([remote_models.md](references/remote_models.md)) + - **SQL Shape:** CREATE MODEL + - **Requires CREATE MODEL:** Yes + - **Usage:** Creates a Vertex AI `REMOTE WITH CONNECTION DEFAULT` model entity for functions that require one (`AI.GENERATE_TABLE`, `AI.GENERATE_EMBEDDING`, `VECTOR_SEARCH`). +- **CONTRIBUTION_ANALYSIS** ([ml_contribution_analysis.md](references/ml_contribution_analysis.md)) + - **SQL Shape:** CREATE MODEL + - **Requires CREATE MODEL:** Yes + - **Usage:** Model-based driver analysis (`CREATE MODEL ... OPTIONS(MODEL_TYPE='CONTRIBUTION_ANALYSIS')` + `ML.GET_INSIGHTS`). Use `AI.KEY_DRIVERS` unless a `MODEL` entity is required. +- **ML.CORRELATION** ([ml_correlation.md](references/ml_correlation.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** No + - **Usage:** Calculates statistical correlation coefficients between numeric columns, optionally sliced by dimensions. +- **ML.DESCRIBE_DATA** ([ml_describe_data.md](references/ml_describe_data.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** No + - **Usage:** Generates descriptive statistics (e.g., count, mean, median, min, max, stddev, quantiles, nulls, top values) for columns in a table or subquery. +- **ML.DETECT_CHANGE_POINTS** ([ml_detect_change_points.md](references/ml_detect_change_points.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** No + - **Usage:** Detects structural breaks or sustained regime shifts (`[begin_timestamp, end_timestamp]` intervals) in a time series. +- **ML.SEASONALITY** ([ml_seasonality.md](references/ml_seasonality.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** No + - **Usage:** Extracts recurring periodic/seasonal components from a historical time series. +- **ML.TREND** ([ml_trend.md](references/ml_trend.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** No + - **Usage:** Extracts the smoothed long-term trend component from a historical time series. +- **VECTOR_SEARCH** ([vector_search.md](references/vector_search.md)) + - **SQL Shape:** TVF + - **Requires CREATE MODEL:** Yes + - **Usage:** Searches pre-computed `ARRAY` embedding columns between a base table and query table (paired with `AI.GENERATE_EMBEDDING`, unlike `AI.SEARCH`). diff --git a/skills/bigquery-ai-ml/references/ai_causal_effect.md b/skills/bigquery-ai-ml/references/ai_causal_effect.md new file mode 100644 index 00000000..13a23dd3 --- /dev/null +++ b/skills/bigquery-ai-ml/references/ai_causal_effect.md @@ -0,0 +1,178 @@ +# BigQuery AI.CAUSAL_EFFECT + +`AI.CAUSAL_EFFECT` quantifies the impact of specific interventions on time +series data by comparing observed post-intervention values against an +`ARIMA_PLUS` counterfactual baseline. It returns aggregated effect statistics or +detailed pointwise time series evaluations. + +## Syntax Reference + +```sql +SELECT + * +FROM + AI.CAUSAL_EFFECT( + { TABLE TABLE_NAME | (QUERY_STATEMENT) }, + data_col => 'DATA_COL', + timestamp_col => 'TIMESTAMP_COL', + intervention_timestamp => INTERVENTION_TIMESTAMP + [, id_cols => ID_COLS] + [, confidence_level => CONFIDENCE_LEVEL] + [, output_time_series => OUTPUT_TIME_SERIES] + [, num_post_intervention_points => NUM_POST_INTERVENTION_POINTS] + ) +``` + +### Input Arguments + +| Argument | Requirement | Type | Description | +| :--------------------------------- | :----------- | :------------ | :----------------------- | +| **`input_data`** | **Required** | Table / Query | The source table or | +: : : : GoogleSQL query : +: : : : containing time series : +: : : : data from before and : +: : : : after the intervention. : +| **`data_col`** | **Required** | String | The name of the column | +: : : : with time series values : +: : : : to analyze. The target : +: : : : column must be of type : +: : : : `INT64`, `NUMERIC`, : +: : : : `BIGNUMERIC`, or : +: : : : `FLOAT64`. : +| **`timestamp_col`** | **Required** | String | The name of the column | +: : : : that contains the : +: : : : timestamps for the time : +: : : : series. The target : +: : : : column must be of type : +: : : : `TIMESTAMP`, `DATE`, or : +: : : : `DATETIME`. : +| **`intervention_timestamp`** | **Required** | Timestamp / | A timestamp value that | +: : : Date : indicates when the : +: : : : intervention occurred, : +: : : : dividing the data into : +: : : : pre-intervention and : +: : : : post-intervention : +: : : : periods. : +| **`id_cols`** | Optional | Array | The names of columns | +: : : : that identify individual : +: : : : time series for parallel : +: : : : processing. The target : +: : : : columns must be of type : +: : : : `STRING` or `INT64`. : +| **`confidence_level`** | Optional | Float64 | A `FLOAT64` value in the | +: : : : range `[0, 1)` : +: : : : specifying the : +: : : : percentage of future : +: : : : values expected to fall : +: : : : within the prediction : +: : : : interval (default\: : +: : : : `0.95`). : +| **`output_time_series`** | Optional | Bool | Determines the level of | +: : : : detail in the output. : +: : : : `FALSE` (default) : +: : : : returns a summary view; : +: : : : `TRUE` returns the : +: : : : detailed time series : +: : : : view including all : +: : : : pointwise data. : +| **`num_post_intervention_points`** | Optional | Int64 | Specifies the number of | +: : : : time series points after : +: : : : the : +: : : : `intervention_timestamp` : +: : : : to include in the causal : +: : : : effect analysis. If you : +: : : : don't specify a value, : +: : : : the analysis includes : +: : : : all data points from the : +: : : : `intervention_timestamp` : +: : : : to the end of the time : +: : : : series. : + +### Output Schema + +The output includes all columns specified in the `id_cols` argument in addition +to the following columns: + +#### Summary View (`output_time_series => FALSE`, default) + +| Column Name | Type | Description | +| :----------------------- | :-------- | :----------------------------------- | +| **`p_value`** | `FLOAT64` | Two-tailed p-value for the null | +: : : hypothesis across the entire : +: : : post-intervention period. : +| **`prob_causal_effect`** | `FLOAT64` | Probability of a causal effect, | +: : : calculated as `(1 - p_value)`. : +| **`absolute_effect`** | `FLOAT64` | Total cumulative difference between | +: : : observed and predicted values : +: : : post-intervention\: : +: : : `SUM(actual_value - : +: : : expected_value)`. : +| **`relative_effect`** | `FLOAT64` | Relative change of observed values: | +: : : `SUM(actual_value - : +: : : expected_value)/SUM(expected_value)` : +: : : across the post-intervention period. : +| **`status`** | `STRING` | Forecast status; empty string on | +: : : success, or an error string. : + +#### Detailed Pointwise View (`output_time_series => TRUE`) + +Includes the summary statistics columns above, repeated across every row, along +with: + +| Column Name | Type | Description | +| :------------------------- | :---------- | :------------------------ | +| **``** | `TIMESTAMP` | Timestamp of the data | +: : : point from the : +: : : `timestamp_col` input : +: : : (both pre- and : +: : : post-intervention). : +| **`is_post_intervention`** | `BOOL` | `TRUE` for timestamps | +: : : greater than or equal to : +: : : `intervention_timestamp`, : +: : : `FALSE` for timestamps : +: : : prior to the : +: : : intervention. : +| **``** | `FLOAT64` | Observed value from | +: : : `data_col` at the : +: : : specified timestamp. : +| **`predicted_`** : : (counterfactual) value at : +: : : the specified timestamp : +: : : (`NULL` for : +: : : pre-intervention : +: : : timestamps). : +| **`lower_bound`** | `FLOAT64` | Lower bound of the | +: : : prediction result (`NULL` : +: : : for pre-intervention : +: : : timestamps). : +| **`upper_bound`** | `FLOAT64` | Upper bound of the | +: : : prediction result (`NULL` : +: : : for pre-intervention : +: : : timestamps). : + +## Limitations + +- A minimum of 3 historical data points in the pre-intervention period is + required to generate a forecast. + +## Examples + +### Impact of the COVID-19 Pandemic on NYC Taxi Trips (Detailed Pointwise View) + +```sql +SELECT * FROM AI.CAUSAL_EFFECT( + ( + SELECT + DATE(pickup_datetime) AS pickup_date, + COUNT(*) AS trip_count + FROM `bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2020` + WHERE EXTRACT(YEAR FROM pickup_datetime) = 2020 + GROUP BY pickup_date + ), + data_col => 'trip_count', + timestamp_col => 'pickup_date', + intervention_timestamp => '2020-03-11', -- WHO declares COVID-19 a pandemic + num_post_intervention_points => 120, + output_time_series => TRUE +); +``` diff --git a/skills/bigquery-ai-ml/references/ai_detect_anomalies.md b/skills/bigquery-ai-ml/references/ai_detect_anomalies.md index 5fc86a95..3016865a 100644 --- a/skills/bigquery-ai-ml/references/ai_detect_anomalies.md +++ b/skills/bigquery-ai-ml/references/ai_detect_anomalies.md @@ -18,6 +18,7 @@ FROM AI.DETECT_ANOMALIES( [, model => 'MODEL'] [, id_cols => ID_COLS] [, anomaly_prob_threshold => ANOMALY_PROB_THRESHOLD] + [, context_window => CONTEXT_WINDOW] ) ``` @@ -32,7 +33,8 @@ Argument | Requirement | Type | Description **`timestamp_col`** | **Required** | String | The column containing dates/timestamps. **`id_cols`** | Optional | Array | Grouping columns for multiple series (e.g., `['store_id']`). **`anomaly_prob_threshold`** | Optional | Float64 | Threshold for anomaly detection (0 to 1). Defaults to 0.95. -**`model`** | Optional | String | Model version. Defaults to `'TimesFM 2.0'`. +**`model`** | Optional | String | Model version. Allowed options are `'TimesFM 2.0'` and `'TimesFM 2.5'`, with `'TimesFM 2.5'` preferred. +**`context_window`** | Optional | Int64 | The number of historical data points the model uses. The min value is 64 and the max value is 2048 for `'TimesFM 2.0'`. The min value is 64 and the max value is 15360 for `'TimesFM 2.5'`. If not set, the model determines this automatically. ### Output Schema @@ -76,7 +78,8 @@ FROM AI.DETECT_ANOMALIES( -- Target range (Data to inspect for anomalies) (SELECT * FROM bike_trips WHERE date BETWEEN '2016-07-01' AND '2016-09-01'), data_col => 'num_trips', - timestamp_col => 'date' + timestamp_col => 'date', + model => 'TimesFM 2.5' ); ``` diff --git a/skills/bigquery-ai-ml/references/ai_evaluate.md b/skills/bigquery-ai-ml/references/ai_evaluate.md index 4703678a..e5f843a9 100755 --- a/skills/bigquery-ai-ml/references/ai_evaluate.md +++ b/skills/bigquery-ai-ml/references/ai_evaluate.md @@ -50,4 +50,4 @@ SELECT * FROM AI.EVALUATE( timestamp_col => 'date', id_cols => ['store_id'] ); -``` \ No newline at end of file +``` diff --git a/skills/bigquery-ai-ml/references/ai_forecast.md b/skills/bigquery-ai-ml/references/ai_forecast.md index a384b2c3..0ebd4832 100644 --- a/skills/bigquery-ai-ml/references/ai_forecast.md +++ b/skills/bigquery-ai-ml/references/ai_forecast.md @@ -24,47 +24,16 @@ FROM ### Input Arguments -| Argument | Requirement | Type | Description | -| :--------------------- | :----------- | :------------ | :---------------- | -| **`input_data`** | **Required** | | The source table | -: : : : or subquery : -: : : : containing : -: : : : historical data. : -| **`data_col`** | **Required** | String | The numeric | -: : : : column to : -: : : : predict. : -| **`timestamp_col`** | **Required** | String | The column | -: : : : containing : -: : : : dates/timestamps. : -| **`id_cols`** | Optional | Array | Grouping columns | -: : : : for multiple : -: : : : series (e.g., : -: : : : `['store_id']`). : -| **`horizon`** | Optional | Int64 | Number of future | -: : : : points to : -: : : : predict. Defaults : -: : : : to 10. The valid : -: : : : input range is : -: : : : [1, 10,000] : -| **`confidence_level`** | Optional | Float64 | Confidence | -: : : : interval (0 to : -: : : : 1). Defaults to : -: : : : 0.95. : -| **`model`** | Optional | String | Model version. | -: : : : Defaults to : -: : : : `'TimesFM 2.0'`. : -| **`context_window`** | Optional | Int64 | The number of | -: : : : historical data : -: : : : points the model : -: : : : uses to forecast. : -: : : : The min value is : -: : : : 64 and the max : -: : : : value is 2048 for : -: : : : `'TimesFM 2.0'`. : -: : : : If not set, the : -: : : : model determines : -: : : : this : -: : : : automatically. : +Argument | Requirement | Type | Description +:--------------------- | :----------- | :------------ | :---------- +**`input_data`** | **Required** | | The source table or subquery containing historical data. +**`data_col`** | **Required** | String | The numeric column to predict. +**`timestamp_col`** | **Required** | String | The column containing dates/timestamps. +**`id_cols`** | Optional | Array | Grouping columns for multiple series (e.g., `['store_id']`). +**`horizon`** | Optional | Int64 | Number of future points to predict. Defaults to 10. The valid input range is [1, 10,000] +**`confidence_level`** | Optional | Float64 | Confidence interval (0 to 1). Defaults to 0.95. +**`model`** | Optional | String | Model version. Allowed options are `'TimesFM 2.0'` and `'TimesFM 2.5'`, with `'TimesFM 2.5'` preferred. +**`context_window`** | Optional | Int64 | The number of historical data points the model uses to forecast. The min value is 64 and the max value is 2048 for `'TimesFM 2.0'`. The min value is 64 and the max value is 15360 for `'TimesFM 2.5'`. If not set, the model determines this automatically. ### Output Schema @@ -102,5 +71,6 @@ FROM timestamp_col => 'date', id_cols => ['usertype'], horizon => 30, + model => 'TimesFM 2.5', output_historical_time_series => true); ``` diff --git a/skills/bigquery-ai-ml/references/ai_generate.md b/skills/bigquery-ai-ml/references/ai_generate.md index 84c39ac0..0b02a6aa 100644 --- a/skills/bigquery-ai-ml/references/ai_generate.md +++ b/skills/bigquery-ai-ml/references/ai_generate.md @@ -1,6 +1,6 @@ # BigQuery AI.Generate -`AI.GENERATE` is a general-purpose function text and content generation. +`AI.GENERATE` is a general-purpose function for text and content generation. ## Syntax Reference @@ -17,27 +17,25 @@ AI.GENERATE( ### Input Arguments -| Argument | Requirement | Type | Description | -| :------------------ | :----------- | :----- | :-------------------- | -| **`prompt`** | **Required** | String | The prompt text or | -: : : : instruction for the : -: : : : model. : -| **`connection_id`** | Optional | String | The connection ID. | -: : : : Optional if : -: : : : configured via other : -: : : : means or testing. : -| **`endpoint`** | Optional | String | The model name, e.g., | -: : : : `'gemini-2.5-flash'`. : -| **`output_schema`** | Optional | String | Schema definition for | -: : : : structured output, : -: : : : e.g., `'answer BOOL, : -: : : : reason STRING'`. : -| **`request_type`** | Optional | String | `'DEDICATED'` or | -: : : : `'SHARED'`. : -| **`model_params`** | Optional | JSON | JSON object for model | -: : : : parameters (e.g., : -: : : : `temperature`, : -: : : : `max_output_tokens`). : +Argument | Requirement | Type | Description +:------------------ | :----------- | :----- | :---------- +**`prompt`** | **Required** | String | The prompt text or instruction for the model. +**`connection_id`** | Optional | String | The connection ID. Optional if configured via other means or testing. +**`endpoint`** | Optional | String | The model name, e.g., `'gemini-3.5-flash-lite'`. +**`output_schema`** | Optional | String | Schema definition for structured output, e.g., `'answer BOOL, reason STRING'`. Supported data types include `STRING`, `INT64`, `FLOAT64`, `BOOL`, `ARRAY`, and `STRUCT`. +**`request_type`** | Optional | String | `'DEDICATED'` or `'SHARED'`. +**`model_params`** | Optional | JSON | JSON object for model parameters (e.g., `temperature`, `max_output_tokens`). + +### Supported Endpoints + +BigQuery supports the following models: + +* `gemini-3.1-flash-lite` +* `gemini-3.5-flash` +* `gemini-3.5-flash-lite` +* `gemini-3.6-flash` +* `gemini-3.7-flash` +* `gemini-3.8-flash` ### Output Schema @@ -63,9 +61,7 @@ Returns a `STRUCT` with the following fields: ```sql SELECT AI.GENERATE( - 'Summarize this article: ' || article_content, - connection_id => 'my-project.us.my-connection', - endpoint => 'gemini-2.5-flash' + 'Summarize this article: ' || article_content ) as summary FROM `dataset.articles` LIMIT 5; @@ -77,7 +73,7 @@ LIMIT 5; SELECT AI.GENERATE( 'Extract the date and amount from this invoice: ' || invoice_text, - output_schema => 'date DATE, amount FLOAT64' + output_schema => 'date STRING, amount FLOAT64' ) as extracted_data FROM `dataset.invoices`; ``` diff --git a/skills/bigquery-ai-ml/references/ai_generate_embedding.md b/skills/bigquery-ai-ml/references/ai_generate_embedding.md index 55e8c833..fb97f436 100755 --- a/skills/bigquery-ai-ml/references/ai_generate_embedding.md +++ b/skills/bigquery-ai-ml/references/ai_generate_embedding.md @@ -4,6 +4,7 @@ visual content. These embeddings can be used for semantic search, clustering, and recommendation systems. ## Syntax + ```sql SELECT * @@ -18,12 +19,13 @@ FROM ``` ## Input Arguments -| Argument | Requirement | Type | Description | -| :--- | :--- | :--- | :--- | -| **`model`** | **Required** | | The model resource (e.g., `project.dataset.model`). | -| **`input_data`** | **Required*** | | The source table or `SELECT` statement containing the data to embed. | -| **`task_type`** | Optional | String | The intended downstream application (e.g., `RETRIEVAL_QUERY`, `RETRIEVAL_DOCUMENT`). | -| **`output_dimensionality`** | Optional | Int64 | The number of dimensions to use when generating embeddings. | + +Argument | Requirement | Type | Description +:-------------------------- | :------------ | :----- | :---------- +**`model`** | **Required** | | The model resource (e.g., `project.dataset.model`). +**`input_data`** | **Required*** | | The source table or `SELECT` statement containing the data to embed. The text in the column named `content` is sent to the model. If your table doesn't have a `content` column, use a `SELECT` statement to alias an existing table column. An error occurs if no `content` column exists. +**`task_type`** | Optional | String | The intended downstream application (e.g., `RETRIEVAL_QUERY`, `RETRIEVAL_DOCUMENT`). +**`output_dimensionality`** | Optional | Int64 | The number of dimensions to use when generating embeddings. \* *`input_data` is optional for matrix factorization models.* @@ -36,6 +38,7 @@ FROM | **`status`** | STRING | Execution status; contains error messages on failure. | ## Example: Generating Text Embeddings + ```sql -- Generate embeddings for a literal string SELECT * @@ -44,4 +47,4 @@ FROM MODEL `mydataset.text_embedding`, (SELECT "Example text to embed" AS content) ); -``` \ No newline at end of file +``` diff --git a/skills/bigquery-ai-ml/references/ai_generate_table.md b/skills/bigquery-ai-ml/references/ai_generate_table.md index cbe986db..6c75bfe2 100755 --- a/skills/bigquery-ai-ml/references/ai_generate_table.md +++ b/skills/bigquery-ai-ml/references/ai_generate_table.md @@ -3,6 +3,7 @@ Uses Gemini models to extract information into a predefined SQL schema. It is a "Table-Valued Function," meaning it is called in the `FROM` clause. ## Syntax + ```sql SELECT * @@ -22,17 +23,18 @@ FROM ``` ## Input Arguments -| Argument | Requirement | Type | Description | -| :--- | :--- | :--- | :--- | -| **`model`** | **Required** | | The remote model resource (e.g., `project.dataset.model`). | -| **`input_data`** | **Required** | | The source table or `SELECT` statement. Must contain a column named or aliased as `prompt`. | -| **`output_schema`** | **Required** | `STRUCT` | A `STRUCT` containing SQL-style column definitions (e.g., `"name STRING, qty INT64"`). | -| **`max_output_tokens`** | Optional | Int64 | [1, 8192]. Limits response length. | -| **`temperature`** | Optional | Float64 | [0.0, 2.0]. Degree of randomness (0.0 is recommended for deterministic responses). | -| **`top_p`** | Optional | Float64 | [0.0, 1.0]. Changes how the model selects tokens for output. | -| **`stop_sequences`** | Optional | Array | Strings that halt model generation if matched. | -| **`safety_settings`** | Optional | Array | Thresholds to filter hate speech, harassment, etc. | -| **`request_type`** | Optional | String | `SHARED`, `DEDICATED`, or `UNSPECIFIED` (defaults to `UNSPECIFIED`). | + +Argument | Requirement | Type | Description +:---------------------- | :----------- | :------------ | :---------- +**`model`** | **Required** | | The remote model resource (e.g., `project.dataset.model`). +**`input_data`** | **Required** | | The source table or `SELECT` statement. Must contain a column named or aliased as `prompt`. +**`output_schema`** | **Required** | String | A `STRUCT` containing SQL-style column definitions (e.g., `"name STRING, qty INT64"`). Supported data types include `STRING`, `INT64`, `FLOAT64`, `BOOL`, `ARRAY`, and `STRUCT` (do not use `DATE` or `TIMESTAMP`; extract as `STRING` and cast in SQL). +**`max_output_tokens`** | Optional | Int64 | [1, 8192]. Limits response length. +**`temperature`** | Optional | Float64 | [0.0, 2.0]. Degree of randomness (0.0 is recommended for deterministic responses). +**`top_p`** | Optional | Float64 | [0.0, 1.0]. Changes how the model selects tokens for output. +**`stop_sequences`** | Optional | Array | Strings that halt model generation if matched. +**`safety_settings`** | Optional | Array | Thresholds to filter hate speech, harassment, etc. +**`request_type`** | Optional | String | `SHARED`, `DEDICATED`, or `UNSPECIFIED` (defaults to `UNSPECIFIED`). ### Prompt Construction (STRUCT Fields) When using a `QUERY_STATEMENT`, you can build multimodal prompts by combining @@ -66,4 +68,4 @@ FROM AI.GENERATE_TABLE( (SELECT ticket_id, body AS prompt FROM `support.tickets`), STRUCT("priority STRING, issue_summary STRING" AS output_schema) ) -``` \ No newline at end of file +``` diff --git a/skills/bigquery-ai-ml/references/ai_key_drivers.md b/skills/bigquery-ai-ml/references/ai_key_drivers.md index 52c13e00..4cfa566a 100644 --- a/skills/bigquery-ai-ml/references/ai_key_drivers.md +++ b/skills/bigquery-ai-ml/references/ai_key_drivers.md @@ -14,7 +14,7 @@ FROM { TABLE TABLE | (QUERY_STATEMENT) }, metric_col => 'METRIC_COL', dimension_cols => DIMENSION_COLS, - interest_label_col => 'INTEREST_LABEL_COL', + interest_label_col => 'INTEREST_LABEL_COL' [, min_apriori_support => MIN_APRIORI_SUPPORT] [, top_k => TOP_K] [, enable_pruning => ENABLE_PRUNING] @@ -26,8 +26,8 @@ FROM Argument | Requirement | Type | Description :------------------------ | :----------- | :-------------- | :---------- **`input_data`** | **Required** | | The source table or subquery containing the data to analyze. -**`metric_col`** | **Required** | `String` | Metric column name. Must be of type: INT64, NUMERIC, BIGNUMERIC, or FLOAT64. -**`interest_label_col`** | **Required** | `String` | Boolean column name: `TRUE` for interest group, `FALSE` for reference group. +**`metric_col`** | **Required** | `STRING` | Metric column name. Must be of type: INT64, NUMERIC, BIGNUMERIC, or FLOAT64. The values in the metric column that you use in the METRIC_COL option must be non-negative, unless you specify 0 for the MIN_APRIORI_SUPPORT value. +**`interest_label_col`** | **Required** | `STRING` | Boolean column name: `TRUE` for interest group, `FALSE` for reference group. **`dimension_cols`** | **Required** | `ARRAY` | 1-12 dimension columns (INT64, BOOL, STRING); cannot be `metric_col` or `interest_label_col`. **`min_apriori_support`** | Optional | `FLOAT64` | Minimum apriori support threshold [0, 1] for output segments. Default: 0.1. Cannot be used with `top_k`. **`top_k`** | Optional | `INT64` | Return top k insights [1, 1M] by apriori support. If unset, uses `min_apriori_support=0.1`. Cannot be used with `min_apriori_support`. @@ -73,3 +73,24 @@ FROM AI.KEY_DRIVERS( min_apriori_support => 0 ); ``` + +### Finding Top 5 Key Drivers in Bottle Costs + +```sql +SELECT * +FROM AI.KEY_DRIVERS( + ( + SELECT + state_bottle_cost, + city, + category_name, + (EXTRACT(MONTH FROM date) >= 7) AS is_h2 + FROM `bigquery-public-data.iowa_liquor_sales.sales` + WHERE EXTRACT(YEAR FROM date) = 2024 + ), + metric_col => 'state_bottle_cost', + dimension_cols => ['city', 'category_name'], + interest_label_col => 'is_h2', + top_k => 5 +); +``` diff --git a/skills/bigquery-ai-ml/references/ai_predict.md b/skills/bigquery-ai-ml/references/ai_predict.md new file mode 100644 index 00000000..0b8089c3 --- /dev/null +++ b/skills/bigquery-ai-ml/references/ai_predict.md @@ -0,0 +1,231 @@ +# BigQuery AI.Predict + +`AI.PREDICT` leverages the pre-trained **TabFM** foundation model to perform +prediction (classification or regression) tasks on tabular data without the need +to train and manage custom models. Under the hood, this function applies an +In-Context Learning (ICL) approach: it embeds the training data to construct a +temporary model representation (kv cache) and uses it to predict labels for the +prediction data within the lifetime of a single query execution. + +## Syntax Reference + +```sql +SELECT + * +FROM + AI.PREDICT( + { TABLE `project.dataset.training_table` | (TRAINING_QUERY) }, + { TABLE `project.dataset.prediction_table` | (PREDICTION_QUERY) } + [, label_col => 'LABEL_COL'] + [, model => 'MODEL'] + ) +``` + +### Input Arguments + +| Argument | Requirement | Type | Description | +| :--------------------- | :----------- | :------- | :----------------- | +| **`training_table`** | **Required** | Relation | The source table | +: : : : or subquery : +: : : : containing : +: : : : training data. : +: : : : This data MUST : +: : : : contain the label : +: : : : column. Only : +: : : : Numerical (e.g., : +: : : : INT64, FLOAT64), : +: : : : STRING, and BOOL : +: : : : columns are : +: : : : supported. : +| **`prediction_table`** | **Required** | Relation | The source table | +: : : : or subquery : +: : : : containing : +: : : : prediction data. : +: : : : Every column in : +: : : : the : +: : : : `training_table` : +: : : : (except the : +: : : : `label_col`) MUST : +: : : : exist in the : +: : : : `prediction_table` : +: : : : with the exact : +: : : : same : +: : : : case-insensitive : +: : : : name and matching : +: : : : types. Only : +: : : : Numerical (e.g., : +: : : : INT64, FLOAT64), : +: : : : STRING, and BOOL : +: : : : columns are : +: : : : supported. : +| **`label_col`** | Optional | String | The name of the | +: : : : label column in : +: : : : the training data. : +: : : : Defaults to : +: : : : `'label'`. : +| **`model`** | Optional | String | The model version. | +: : : : If specified, the : +: : : : value MUST be the : +: : : : literal `'TabFM : +: : : : 1.0'`. Defaults to : +: : : : `'TabFM 1.0'`. : + +### Constraints & Limitations + +> [!WARNING] **Same Table Split (Avoid Label Leakage)**: If you use the EXACT +> same table (or query) for both the `training_table` and `prediction_table` +> arguments, you MUST split the data into non-overlapping sets to avoid label +> leakage. You can use a helper column with `RAND() <= 0.8` to split the data +> (e.g., 80% for training, 20% for prediction). +> +> Example: +> +> ```sql +> WITH prepared_data AS ( +> SELECT *, RAND() <= 0.8 AS training +> FROM `bigquery-public-data.ml_datasets.penguins` +> WHERE body_mass_g > 0 +> ) +> SELECT +> * +> FROM +> AI.PREDICT( +> # Training data +> (SELECT * EXCEPT(training) FROM prepared_data WHERE training), +> # Prediction data +> (SELECT * EXCEPT(training) FROM prepared_data WHERE NOT training), +> label_col => 'body_mass_g' +> ); +> ``` + +- **Feature Column Limit**: The training and prediction data can include up to + 50 feature columns. +- **Classification Category Limit**: For classification tasks, you can + classify data into at most 10 different categories. +- **Task Type Detection**: + - **CLASSIFICATION**: Automatically triggered if the label column is of + type STRING or BOOL. + - **REGRESSION**: Automatically triggered if the label column is of a + Numerical type. + +### Output Schema + +The output schema includes: + +Column | Type | Description +:-------------------------------------- | :------------------------------------------------------ | :---------- +**`predicted_`** | STRING or BOOL (Classification) or Numeric (Regression) | The predicted label. The data type is the same as the label column in the training data. +**`predicted__probs`** | ARRAY> | (Classification only) An array of structs containing `label` (STRING) and `prob` (FLOAT64) representing the probability of each class. +**All columns from `prediction_table`** | (As Input) | All columns from the input `prediction_table` relation. + +## Examples + +### Classification Example (Inline Data) + +```sql +SELECT * +FROM + AI.PREDICT( + ( + SELECT + 'Alice' AS customer, + 150.0 AS spend, + 2 AS calls, + 24 AS tenure, + TRUE AS churned + UNION ALL + SELECT + 'Bob' AS customer, + 850.0 AS spend, + 1 AS calls, + 48 AS tenure, + FALSE AS churned + ), + ( + SELECT + 'Bob' AS customer, + 850.0 AS spend, + 1 AS calls, + 48 AS tenure + UNION ALL + SELECT + 'Charlie' AS customer, + 200.0 AS spend, + 4 AS calls, + 12 AS tenure + ), + label_col => 'churned' + ); +``` + +### Classification Example (Tables) + +```sql +SELECT * +FROM + AI.PREDICT( + TABLE `my_project.my_dataset.customer_train`, + TABLE `my_project.my_dataset.customer_test`, + label_col => 'churned' + ); +``` + +### Regression Example (Tables) + +```sql +SELECT * +FROM + AI.PREDICT( + TABLE `my_project.my_dataset.housing_train`, + TABLE `my_project.my_dataset.housing_test`, + label_col => 'price' + ); +``` + +### Same Table Split Example (Regression) + +If you must use the same table for both training and prediction, you MUST split +the data to avoid label leakage. This example predicts the numeric `body_mass_g` +column (Regression). + +```sql +WITH prepared_data AS ( + SELECT *, RAND() <= 0.8 AS training + FROM `bigquery-public-data.ml_datasets.penguins` + WHERE body_mass_g > 0 +) +SELECT + * +FROM + AI.PREDICT( + # Training data + (SELECT * EXCEPT(training) FROM prepared_data WHERE training), + # Prediction data + (SELECT * EXCEPT(training) FROM prepared_data WHERE NOT training), + label_col => 'body_mass_g' + ); +``` + +### Same Table Split Example (Classification) + +If you must use the same table for both training and prediction, you MUST split +the data to avoid label leakage. This example predicts the string `species` +column (Classification). + +```sql +WITH prepared_data AS ( + SELECT *, RAND() <= 0.8 AS training + FROM `bigquery-public-data.ml_datasets.penguins` + WHERE species IS NOT NULL +) +SELECT + * +FROM + AI.PREDICT( + # Training data + (SELECT * EXCEPT(training) FROM prepared_data WHERE training), + # Prediction data + (SELECT * EXCEPT(species, training) FROM prepared_data WHERE NOT training), + label_col => 'species' + ); +``` diff --git a/skills/bigquery-ai-ml/references/ai_search.md b/skills/bigquery-ai-ml/references/ai_search.md index a7129775..09aeda60 100644 --- a/skills/bigquery-ai-ml/references/ai_search.md +++ b/skills/bigquery-ai-ml/references/ai_search.md @@ -54,7 +54,7 @@ Column Name | Type | Description ```sql # Create a table with a product column, a description column, and an autonomous -# embedding generation column over the descrption column. +# embedding generation column over the description column. CREATE TABLE mydataset.products ( name STRING, description STRING, diff --git a/skills/bigquery-ai-ml/references/ml_correlation.md b/skills/bigquery-ai-ml/references/ml_correlation.md new file mode 100644 index 00000000..790d0631 --- /dev/null +++ b/skills/bigquery-ai-ml/references/ml_correlation.md @@ -0,0 +1,185 @@ +# BigQuery ML.CORRELATION + +`ML.CORRELATION` is a Table-Valued Function (TVF) that calculates the +statistical correlation between a target column and one or more target +correlation columns, optionally sliced by various dimensions. + +## Syntax Reference + +```sql +ML.CORRELATION( + { TABLE TABLE_NAME | (QUERY_STATEMENT) }, + target_col => TARGET_COL, + target_correlation_cols => TARGET_CORRELATION_COLS + [, dimension_cols => DIMENSION_COLS ] + [, method => METHOD ] +) +``` + +### Input Arguments + +Argument | Requirement | Type | Description +:---------------------------- | :----------- | :-------------------------- | :---------- +**`input_data (positional)`** | **Required** | | The source table or query containing the data to analyze. +**`target_col`** | **Required** | `STRING` | The name of the *numerical* column to be correlated. +**`target_correlation_cols`** | **Required** | `STRING` or `ARRAY` | The name of the numerical columns to correlate against the `target_col`. +**`dimension_cols`** | Optional | `STRING` or `ARRAY` | A value that contains the names of columns to slice the data by. The function calculates correlations for every combination of these dimensions (Max: 12 columns). Supported Types: Any Groupable data type. Unsupported: ARRAY, STRUCT, JSON, GEOGRAPHY. +**`method`** | Optional | `STRING` | The correlation method: `PEARSON` (default), `SPEARMAN`, or `KENDALL`. The `KENDALL` method has O(N^2) performance implications on massive tables. For large datasets, it is highly recommended to use `PEARSON` or `SPEARMAN`. + +**Note on Reserved Column Names:** The column names used in arguments must not +overlap with the reserved output column names: `segment`, `target_col`, +`corr_col`, `correlation`, `segment_size`, `segment_proportion`. + +### Output Schema + +| Column | Type | Description | +| :----------------------- | :-------------------------- | :------------------ | +| **`segment`** | `ARRAY>` : combination of : +: : : dimension names and : +: : : values used to : +: : : group the data for : +: : : this row. : +| *``* | Varies | Columns | +: : : corresponding to : +: : : each dimension in : +: : : `dimension_cols`. : +| **`target_col`** | `STRING` | The name of the | +: : : target column. : +| **`corr_col`** | `STRING` | The name of the | +: : : column being : +: : : correlated against : +: : : the target. : +| **`correlation`** | `FLOAT64` | The calculated | +: : : correlation : +: : : coefficient. : +| **`segment_size`** | `INT64` | The number of rows | +: : : in the segment. : +| **`segment_proportion`** | `FLOAT64` | A value that | +: : : contains the : +: : : fraction of total : +: : : rows in the input : +: : : table that belong : +: : : to this segment. : + +**Note:** Results are sorted by `segment_size` (descending) and then `corr_col` +(ascending) by default. + +## Examples + +The following examples show how to use the `ML.CORRELATION` function and use the +`my_dataset.marketing_sample` table: + +```sql +CREATE OR REPLACE TABLE my_dataset.marketing_sample AS ( + -- New York data + SELECT 'USA' AS country, 'New York' AS city, 'Electronics' AS product_category, 100 AS ad_spend, 150 AS budget, 1000 AS revenue UNION ALL + SELECT 'USA', 'New York', 'Electronics', 150, 200, 1500 UNION ALL + SELECT 'USA', 'New York', 'Apparel', 200, 250, 2000 UNION ALL + + -- Seattle data + SELECT 'USA', 'Seattle', 'Apparel', 200, 250, 2000 UNION ALL + SELECT 'USA', 'Seattle', 'Apparel', 300, 350, 3000 UNION ALL + + -- London data (Genuine NULL country) + SELECT NULL, 'London', 'Electronics', 100, 120, 500 UNION ALL + SELECT NULL, 'London', 'Electronics', 200, 220, 900 UNION ALL + + -- Missing city data (Genuine NULL city) + SELECT NULL, NULL, 'Apparel', 200, 200, 1000 UNION ALL + SELECT NULL, NULL, 'Apparel', 250, 250, 1200 +); + + +/*---------+----------+------------------+----------+---------+---------+ + | country | city | product_category | ad_spend | budget | revenue | + +---------+----------+------------------+----------+---------+---------+ + | USA | New York | Electronics | 100 | 150 | 1000 | + | USA | New York | Electronics | 150 | 200 | 1500 | + | USA | New York | Apparel | 200 | 250 | 2000 | + | USA | Seattle | Apparel | 200 | 250 | 2000 | + | USA | Seattle | Apparel | 300 | 350 | 3000 | + | null | London | Electronics | 100 | 120 | 500 | + | null | London | Electronics | 200 | 220 | 900 | + | null | null | Apparel | 200 | 200 | 1000 | + | null | null | Apparel | 250 | 250 | 1200 | + +---------+----------+------------------+----------+---------+---------*/ +``` + +### Calculate Pearson correlation + +The following example calculates the Pearson correlation between `revenue` and +`ad_spend` from the table `my_dataset.marketing_sample` and uses `country` as a +dimension column: + +```sql +SELECT + country, + segment, + correlation, + segment_size +FROM ML.CORRELATION( + TABLE my_dataset.marketing_sample, + target_col => 'revenue', + target_correlation_cols => 'ad_spend', + dimension_cols => ['country'] +); + +/*---------+---------------------------------------------------------------+-------------+--------------+ + | country | segment | correlation | segment_size | + +---------+---------------------------------------------------------------+-------------+--------------+ + | 'USA' | [{dimension_col: 'country', dimension_value: 'USA'}] | 1.0 | 2 | + | NULL | [{dimension_col: 'country', dimension_value: null}] | 1.0 | 2 | + | NULL | [] | 0.688 | 4 | + +---------+---------------------------------------------------------------+-------------+--------------*/ +``` + +The second row of the result corresponds to a genuine `NULL` in the input data +for country because the `dimension_value` field is `NULL`. The third row of the +result contains `NULL` for country because it corresponds to aggregation over +all countries. + +### Calculate correlation for multiple columns + +The following example calculates the correlation between `revenue` and +`ad_spend` and `budget`, sliced by `city` and `product_category`, from the table +`my_dataset.marketing_sample`: + +```sql +SELECT * +FROM +ML.CORRELATION( + (SELECT * FROM my_dataset.marketing_sample WHERE country = 'USA'), + target_col => 'revenue', + target_correlation_cols => ['ad_spend', 'budget'], + dimension_cols => ['city', 'product_category'] +); +``` + +### Distinguish between global aggregates and missing data + +The following example shows how to use the `segment` column to label your rows +clearly in a report: + +```sql +SELECT + -- Create a clean label for reporting + CASE + -- If 'city' is NULL and not in the segment array, it's a total + WHEN city IS NULL AND NOT EXISTS(SELECT 1 FROM UNNEST(segment) s WHERE s.dimension_col = 'city') + THEN 'ALL CITIES (Global)' + -- If 'city' is NULL and in the segment array, it's missing data + WHEN city IS NULL + THEN 'UNKNOWN CITY' + ELSE city + END AS city_label, + correlation, + segment_size +FROM ML.CORRELATION( + TABLE my_dataset.marketing_sample, + target_col => 'revenue', + target_correlation_cols => 'ad_spend', + dimension_cols => ['city'] +); +``` diff --git a/skills/bigquery-ai-ml/references/ml_describe_data.md b/skills/bigquery-ai-ml/references/ml_describe_data.md new file mode 100644 index 00000000..6386eb84 --- /dev/null +++ b/skills/bigquery-ai-ml/references/ml_describe_data.md @@ -0,0 +1,73 @@ +# BigQuery ML.DESCRIBE_DATA + +`ML.DESCRIBE_DATA` generates descriptive statistics for the columns in a table +or subquery. + +## Syntax Reference + +```sql +SELECT + * +FROM + ML.DESCRIBE_DATA( + { TABLE project.dataset.table | (QUERY_STATEMENT) }, + STRUCT( + [NUM_QUANTILES AS num_quantiles] + [, NUM_ARRAY_LENGTH_QUANTILES AS num_array_length_quantiles] + [, TOP_K AS top_k]) + ) +``` + +### Input Arguments + +Argument | Requirement | Type | Description +:------------------------------- | :----------- | :------ | :---------- +**`input_data`** | **Required** | | The source table or subquery containing the data to analyze. +**`NUM_QUANTILES`** | Optional | `INT64` | Specifies the number of quantiles to return for numerical, `ARRAY`, and `ARRAY>` columns. This affects the number of results shown in the quantiles output column. These quantiles describe the distribution of the data in the column. Specify a lower value for coarser-grained distribution information and a higher value for finer-grained distribution information. The NUM_QUANTILES value must be in the range [1, 100,000]. The default value is 2. +**`NUM_ARRAY_LENGTH_QUANTILES`** | Optional | `INT64` | Specifies the number of quantiles to return for ARRAY columns. This affects the number of results shown in the array_length_quantiles output column. These quantiles describe the distribution of the length of the arrays in the column. Specify a lower value for coarser-grained distribution information and a higher value for finer-grained distribution information. The NUM_ARRAY_LENGTH_QUANTILES value must be in the range [1, 100,000]. The default value is 10. +**`TOP_K`** | Optional | `INT64` | Specifies the number of top values to return for categorical and `ARRAY` columns. This affects the number of results shown in the top_values output column. The top values are the values that are shown most frequently in the column. The TOP_K value must be in the range [1, 10,000]. The default value is 1. + +Input data for the `ML.DESCRIBE_DATA` function can only contain columns of the +following data types: Numeric types, `STRING`, `BOOL`, `BYTES`, `DATE`, +`DATETIME`, `TIME`, `TIMESTAMP`. It also supports `ARRAY` columns or `STRUCT` +columns that contain any of the previous primitive types. Columns with +unsupported data types, such as `GEOGRAPHY`, must be excluded from the input +data. + +### Output Schema + +Returns a table with one row per input column with the following fields: + +Column Name | Type | Description +:--------------------------- | :----------------------------- | :---------- +**`name`** | `STRING` | The name of the input column. +**`num_rows`** | `INT64` | The total number of rows for the input column. +**`num_nulls`** | `INT64` | The number of NULL values found in the column. +**`num_zeros`** | `INT64` | The number of 0 values found in the column. +**`min`** | `STRING` | The MIN value for the numerical column. +**`max`** | `STRING` | The MAX value for the numerical column. +**`mean`** | `FLOAT64` | The Mean value for the numerical column. +**`stddev`** | `FLOAT64` | The standard deviation value calculated for the numerical column. +**`median`** | `FLOAT64` | The median value calculated for the numerical column. +**`quantiles`** | `ARRAY` | The quantiles in an input column. +**`unique`** | `INT64` | The number of unique values in an input column. +**`avg_string_length`** | `FLOAT64` | The average length of the values in the categorical column. +**`num_values`** | `INT64` | The number of array elements for ARRAY columns, and the number of values in the column for other types of columns. +**`top_values`** | `ARRAY>` | The top values and number of occurrences in an input column +**`min_array_length`** | `INT64` | The minimum length of an array in the column. +**`max_array_length`** | `INT64` | The maximum length of an array in the column. +**`avg_array_length`** | `FLOAT64` | The average length of arrays in the column. +**`total_array_length`** | `INT64` | The sum of the size of the arrays in the column. +**`array_length_quantiles`** | `ARRAY` | The quantiles for the array length in an input column, as computed by the APPROX_QUANTILES function. +**`dimension`** | `INT64` | The dimension of array elements for ARRAY columns. + +## Examples + +### Calculate statistics for a table + +```sql +SELECT * +FROM ML.DESCRIBE_DATA( + TABLE `bigquery-public-data.ml_datasets.penguins` +); +``` diff --git a/skills/bigquery-ai-ml/references/ml_detect_change_points.md b/skills/bigquery-ai-ml/references/ml_detect_change_points.md new file mode 100644 index 00000000..640f7da0 --- /dev/null +++ b/skills/bigquery-ai-ml/references/ml_detect_change_points.md @@ -0,0 +1,111 @@ +# BigQuery ML.Detect_Change_Points + +`ML.DETECT_CHANGE_POINTS` is a one-step Table-Valued Function (TVF) that +identifies intervals where the statistical behavior of time series data have +shifted. It distinguishes structural breaks (e.g. "slow bleed" anomalies, +sustained structural shifts) from transient or isolated events. All identified +change points are represented as time windows defined by both a beginning and an +ending timestamp. + +## Syntax Reference + +```sql +SELECT + * +FROM + ML.DETECT_CHANGE_POINTS( + { TABLE `project.dataset.table` | (QUERY_STATEMENT) }, + data_col => 'DATA_COL', + timestamp_col => 'TIMESTAMP_COL' + [, id_cols => ID_COLS] + ) +``` + +### Input Arguments + +| Argument | Requirement | Type | Description | +| :------------------ | :----------- | :------------ | :---------------------- | +| **`input_data`** | **Required** | | The source table or | +: : : : query containing : +: : : : historical time series : +: : : : data. : +| **`data_col`** | **Required** | String | The name of the numeric | +: : : : column in `input_data` : +: : : : to analyze. The data : +: : : : column must use one of : +: : : : the following data : +: : : : types\: `INT64`, : +: : : : `FLOAT64`, `NUMERIC`, : +: : : : `BIGNUMERIC`. : +| **`timestamp_col`** | **Required** | String | The name of the | +: : : : date/timestamp column : +: : : : in `input_data`. The : +: : : : timestamp column must : +: : : : use one of the : +: : : : following data types\: : +: : : : `DATE`, `DATETIME`, : +: : : : `TIMESTAMP`. : +| **`id_cols`** | Optional | Array | The names of the | +: : : : grouping columns for : +: : : : multiple time series : +: : : : (e.g., `['store_id']`). : +: : : : The columns that you : +: : : : specify must use one of : +: : : : the following data : +: : : : types\: `STRING`, : +: : : : `INT64`, : +: : : : `ARRAY`, : +: : : : `ARRAY`. : + +### Output Schema + +| Column | Type | Description | +| :-------------------- | :--------- | :-------------------------------------- | +| **`id_cols`** | (As Input) | Original identifiers for the time | +: : : series. : +| **`begin_timestamp`** | TIMESTAMP | Timestamp corresponding to the start of | +: : : the change point. : +| **`end_timestamp`** | TIMESTAMP | Timestamp corresponding to the end of | +: : : the change point. : +| **`metrics`** | STRUCT | A set of metrics describing the | +: : : statistical behavior. : +| **`metrics.avg`** | FLOAT64 | Calculated average metric value within | +: : : this change point. : +| **`metrics.min`** | FLOAT64 | Minimum metric value observed. | +| **`metrics.max`** | FLOAT64 | Maximum metric value observed. | +| **`metrics.stddev`** | FLOAT64 | Standard deviation within this change | +: : : point. : +| **`metrics.count`** | INT64 | Total count of data points in the | +: : : identified change point. : +| **`status`** | STRING | Error messages or empty string on | +: : : success. : + +## Examples + +### Detecting Change Points in Taxi Trips + +This example detects structural shifts in the number of New York taxi trips +spanning 2019 to 2020: + +```sql +WITH daily_trips AS ( + SELECT + EXTRACT(DATE FROM pickup_datetime) AS trip_date, + COUNT(*) AS total_trips + FROM + `bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_20*` + WHERE + _TABLE_SUFFIX BETWEEN '19' AND '20' + AND EXTRACT(DATE FROM pickup_datetime) BETWEEN '2019-01-01' AND '2020-12-31' + GROUP BY + trip_date +) +SELECT + * +FROM + ML.DETECT_CHANGE_POINTS( + TABLE daily_trips, + data_col => 'total_trips', + timestamp_col => 'trip_date' + ); +``` diff --git a/skills/bigquery-ai-ml/references/ml_seasonality.md b/skills/bigquery-ai-ml/references/ml_seasonality.md new file mode 100644 index 00000000..369edc31 --- /dev/null +++ b/skills/bigquery-ai-ml/references/ml_seasonality.md @@ -0,0 +1,124 @@ +# BigQuery ML.Seasonality + +`ML.SEASONALITY` is a time-series decomposition function that extracts seasonal +components to decouple recurring patterns from the underlying trend and noise. + +## Syntax Reference + +```sql +SELECT + * +FROM + ML.SEASONALITY( + { TABLE `project.dataset.table` | (QUERY_STATEMENT) }, + data_col => 'DATA_COL', + timestamp_col => 'TIMESTAMP_COL' + [, id_cols => ID_COLS] + [, seasonalities => SEASONALITIES] + [, horizon => HORIZON] + ) +``` + +### Input Arguments + +| Argument | Requirement | Type | Description | +| :------------------ | :----------- | :------------ | :---------------------- | +| **`input_data`** | **Required** | | The source table or | +: : : : query containing : +: : : : historical time series : +: : : : data. : +| **`data_col`** | **Required** | String | The name of the numeric | +: : : : column in `input_data`. : +: : : : Supported types\: : +: : : : `INT64`, `FLOAT64`, : +: : : : `NUMERIC`, : +: : : : `BIGNUMERIC`. : +| **`timestamp_col`** | **Required** | String | The name of the | +: : : : date/timestamp column : +: : : : in `input_data`. : +: : : : Supported types\: : +: : : : `DATE`, `DATETIME`, : +: : : : `TIMESTAMP`. : +| **`id_cols`** | Optional | Array | The names of the | +: : : : grouping columns for : +: : : : multiple series (e.g., : +: : : : `['store_id']`). : +: : : : Supported types\: : +: : : : `STRING`, `INT64`, : +: : : : `ARRAY`, : +: : : : `ARRAY`. : +| **`seasonalities`** | Optional | Array | Seasonality types to | +: : : : extract. Valid values\: : +: : : : `Yearly`, `Quarterly`, : +: : : : `Monthly`, `Weekly`, : +: : : : `Daily`. If omitted, : +: : : : the function : +: : : : automatically detects : +: : : : all seasonalities. : +| **`horizon`** | Optional | Int64 | Number of future time | +: : : : points to forecast. : +: : : : Default is `0` (history : +: : : : only). Range `[1, : +: : : : 10,000]`. : + +### Output Schema + +| Column | Type | Description | +| :--------------------- | :--------- | :------------------------------------- | +| **`id_cols`** | (As Input) | Original identifiers for the series. | +| **`[timestamp_col]`** | (As Input) | The column name and type match the | +: : : input column specified in : +: : : *TIMESTAMP_COL*. : +| **`[data_col]`** | FLOAT64 | The column name matches the input | +: : : column specified in *DATA_COL*. For : +: : : 'history' rows, this contains the : +: : : training data (automatically : +: : : interpolated to resolve any missing : +: : : points). For 'forecast' rows, it : +: : : contains the forecast value. : +| **`time_series_type`** | STRING | Label: `'history'` or `'forecast'`. | +| **`yearly`** | FLOAT64 | Calculated yearly seasonal component. | +: : : A NULL value means no yearly : +: : : seasonality detected. : +| **`quarterly`** | FLOAT64 | Calculated quarterly seasonal | +: : : component. A NULL value means no : +: : : quarterly seasonality detected. : +| **`monthly`** | FLOAT64 | Calculated monthly seasonal component. | +: : : A NULL value means no monthly : +: : : seasonality detected. : +| **`weekly`** | FLOAT64 | Calculated weekly seasonal component. | +: : : A NULL value means no weekly : +: : : seasonality detected. : +| **`daily`** | FLOAT64 | Calculated daily seasonal component. A | +: : : NULL value means no daily seasonality : +: : : detected. : +| **`status`** | STRING | Error messages or empty string on | +: : : success. A minimum of 3 data points is : +: : : required. : + +## Examples + +### Analyzing Seasonality in Daily Visits + +This example analyzes the daily website visits from Google Analytics session +data: + +```sql +WITH DailyVisits AS ( + SELECT + PARSE_TIMESTAMP('%Y%m%d', date) AS visit_timestamp, + SUM(totals.visits) AS total_visits + FROM + `bigquery-public-data.google_analytics_sample.ga_sessions_*` + GROUP BY + visit_timestamp +) +SELECT + * +FROM + ML.SEASONALITY( + TABLE DailyVisits, + data_col => 'total_visits', + timestamp_col => 'visit_timestamp' + ); +``` diff --git a/skills/bigquery-ai-ml/references/ml_trend.md b/skills/bigquery-ai-ml/references/ml_trend.md new file mode 100644 index 00000000..8ed4d955 --- /dev/null +++ b/skills/bigquery-ai-ml/references/ml_trend.md @@ -0,0 +1,82 @@ +# BigQuery ML.Trend + +`ML.TREND` is a time-series decomposition function that extracts the underlying +long-term direction of data, isolating the persistent growth or decline from +seasonal fluctuations and random noise without requiring a pre-trained model. + +## Syntax Reference + +```sql +SELECT + * +FROM + ML.TREND( + { TABLE `project.dataset.table` | (QUERY_STATEMENT) }, + data_col => 'DATA_COL', + timestamp_col => 'TIMESTAMP_COL' + [, id_cols => ID_COLS] + [, horizon => HORIZON] + [, smoothing_window_size => SMOOTHING_WINDOW_SIZE] + [, adjust_step_changes => ADJUST_STEP_CHANGES] + ) +``` + +### Input Arguments + +Argument | Requirement | Type | Description +:-------------------------- | :----------- | :------------ | :---------- +**`input_data`** | **Required** | | The source table or query containing historical time series data. +**`data_col`** | **Required** | String | The name of the numeric column in `input_data`. Supported types: `INT64`, `FLOAT64`, `NUMERIC`, `BIGNUMERIC`. +**`timestamp_col`** | **Required** | String | The name of the date/timestamp column in `input_data`. Supported types: `DATE`, `DATETIME`, `TIMESTAMP`. +**`id_cols`** | Optional | Array | The names of the grouping columns for multiple series (e.g., `['store_id']`). Supported types: `STRING`, `INT64`, `ARRAY`, `ARRAY`. +**`horizon`** | Optional | Int64 | Number of future time points to forecast. Default is `0` (historical trend only). Range `[1, 10,000]`. +**`smoothing_window_size`** | Optional | Int64 | Size of the center moving average smoothing window used to smooth out noise before trend extraction. Default is `5`. Must be positive. +**`adjust_step_changes`** | Optional | Bool | Whether to perform automatic step change detection and adjustment. When true, identifies abrupt level shifts and blends their impact into the surrounding data before extracting the trend, generating a smoother, more continuous result. + +### Output Schema + +| Column | Type | Description | +| :--------------------- | :--------- | :------------------------------------- | +| **`id_cols`** | (As Input) | Original identifiers for the series. | +| **`[timestamp_col]`** | (As Input) | The column name and type match the | +: : : input column specified in : +: : : *TIMESTAMP_COL*. : +| **`[data_col]`** | FLOAT64 | The column name matches the input | +: : : column specified in *DATA_COL*. For : +: : : 'history' rows, this contains the : +: : : training data (automatically : +: : : interpolated to resolve any missing : +: : : points). For 'forecast' rows, it : +: : : contains the forecast value. : +| **`time_series_type`** | STRING | Label: `'history'` or `'forecast'`. | +| **`trend`** | FLOAT64 | The calculated trend component for the | +: : : time point. : +| **`status`** | STRING | Error messages or empty string on | +: : : success. A minimum of 3 data points is : +: : : required. : + +## Examples + +### Analyzing Trends in Daily Visits + +This example analyzes the daily visit trend from Google Analytics sample data: + +```sql +WITH DailyVisits AS ( + SELECT + PARSE_TIMESTAMP('%Y%m%d', date) AS visit_timestamp, + SUM(totals.visits) AS total_visits + FROM + `bigquery-public-data.google_analytics_sample.ga_sessions_*` + GROUP BY + visit_timestamp +) +SELECT + * +FROM + ML.TREND( + TABLE DailyVisits, + data_col => 'total_visits', + timestamp_col => 'visit_timestamp' + ); +``` diff --git a/skills/bigquery-ai-ml/references/remote_models.md b/skills/bigquery-ai-ml/references/remote_models.md index 394c067b..8288895b 100755 --- a/skills/bigquery-ai-ml/references/remote_models.md +++ b/skills/bigquery-ai-ml/references/remote_models.md @@ -4,6 +4,7 @@ specific endpoint (e.g., Gemini). Remote models are created using a connection to interact with Vertex AI. ### Syntax + ```sql CREATE [OR REPLACE] MODEL `project.dataset.model_name` REMOTE WITH CONNECTION { DEFAULT | `project.region.connection_id` } @@ -12,12 +13,20 @@ OPTIONS (ENDPOINT = 'endpoint_name'); ### Available Endpoints -* **Generative**: `gemini-2.5-pro`, `gemini-2.5-flash` -* **Embedding**: `text-embedding-005`, `text-multilingual-embedding-002`, - `gemini-embedding-001`. +BigQuery supports the following models: + +* **Generative**: + * `gemini-3.1-flash-lite` + * `gemini-3.5-flash` + * `gemini-3.5-flash-lite` + * `gemini-3.6-flash` + * `gemini-3.7-flash` + * `gemini-3.8-flash` +* **Embedding**: `text-embedding-005`, `text-multilingual-embedding-002`, + `gemini-embedding-001`, `gemini-embedding-2`. ### Connection Usage If `REMOTE WITH CONNECTION DEFAULT` is used, BigQuery automatically attempts to use a default connection in the model's region. Default connection should be the default behavior. Otherwise, you must specify a fully qualified connection -ID (e.g., `my-project.us.my-connection`). \ No newline at end of file +ID (e.g., `my-project.us.my-connection`). diff --git a/skills/bigtable-basics/SKILL.md b/skills/bigtable-basics/SKILL.md index e1bf9c27..df9fbb9d 100644 --- a/skills/bigtable-basics/SKILL.md +++ b/skills/bigtable-basics/SKILL.md @@ -1,9 +1,9 @@ --- name: bigtable-basics description: >- - Assists in provisioning instances/tables, designing performant schemas, and querying data in Bigtable. Use when designing Bigtable row keys, configuring column families, writing SQL queries or client library code (Java, Go, Python) for Bigtable, or diagnosing performance/hotspotting issues. Also use when provisioning Bigtable clusters using gcloud or cbt CLIs. Don't use for generic Cloud SQL administration. + Provisions instances and tables, designs performant schemas, and queries data in Bigtable. Use when designing Bigtable row keys, configuring column families, writing SQL queries or client library code (Java, Go, Python) for Bigtable, or diagnosing performance/hotspotting issues. Also use when provisioning Bigtable clusters using gcloud or cbt CLIs. Don't use for generic Cloud SQL administration. metadata: - version: v1 + version: v2 publisher: google category: Databases license: Apache-2.0 @@ -116,6 +116,6 @@ defined as SQL queries but they must be created using gcloud CLI. ## External Resources * [Cloud Bigtable Documentation](https://cloud.google.com/bigtable/docs) -* [Bigtable SQL Reference](https://cloud.google.com/bigtable/docs/reference/sql) +* [Bigtable SQL Reference](https://cloud.google.com/bigtable/docs/googlesql-overview) * [cbt CLI Reference](https://cloud.google.com/bigtable/docs/cbt-reference) * [gcloud bigtable Reference](https://cloud.google.com/sdk/gcloud/reference/bigtable) diff --git a/skills/cloud-databases-onboarding/SKILL.md b/skills/cloud-databases-onboarding/SKILL.md new file mode 100644 index 00000000..13a534f1 --- /dev/null +++ b/skills/cloud-databases-onboarding/SKILL.md @@ -0,0 +1,95 @@ +--- +name: cloud-databases-onboarding +description: >- + Guides users through discovering their database requirements, recommends a + Google Cloud database product or migration destination based on a recommendation + matrix, and assists in database creation. Use when a user asks 'What database + service should I use?', 'Help me pick a database', asks which Google Cloud + database to migrate or onboard a workload to, or wants to create a new database + on Google Cloud. Don't use for general Google Cloud maintenance, managing + existing databases, or troubleshooting active Database Migration Service (DMS) jobs. +license: Apache-2.0 +metadata: + version: v1 + publisher: google + category: Databases +--- + +# Google Cloud Database Onboarding Skill + +This skill provides domain instructions, decision matrices, and +Infrastructure-as-Code workflows to guide users through discovering their exact +database requirements, selecting an optimal Google Cloud database service, and +drafting starter resource provisioning code for user review. + +## Progressive Disclosure + +* **Reading / Progressive Disclosure:** When interacting with a user during a + conversation, load reference files progressively. Follow the Just-in-Time + (JiT) loading instructions outlined in the phases below. + +-------------------------------------------------------------------------------- + +## Workflow & Just-in-Time (JiT) Instructions + +This workflow operates in three distinct sequential phases. Evaluate the active +conversation history to determine the current phase and follow the corresponding +instructions: + +### Phase 1: Requirement Discovery & Information Gathering + +When a user asks `"What database should I use?"` or requires guidance on Google +Cloud database selection without providing sufficient source or workload +context, you must initiate the Discovery phase. + +1. **Load Discovery Instructions (JiT):** Read the complete contents of + `references/onboarding_prompts.md` using `view_file`. +2. **Execute Discovery:** Optionally check `get_active_gcp_connection` and + `get_active_editor_context` (when available in the IDE) to prefill context, + and follow the detailed Phase 1 instructions in `onboarding_prompts.md` to + gather core requirements (data model, workload, scale, and migration + context) using user-friendly phrasing and enforcing constraints (such as the + 90% confidence rule) before proposing any recommendation. + +### Phase 2: Recommendation Analysis & Matrix Consultation + +Once you have gathered sufficient explicit discovery context, you must determine +the optimal Google Cloud database recommendation. + +1. **Consult Matrix & Formulate Recommendation (JiT):** Follow the Phase 2 + instructions in `references/onboarding_prompts.md` (or, when the user's + prompt already specifies the source database and workload requirements, skip + reading `references/onboarding_prompts.md` and read only the matching + source section in `references/recommendation_matrix.txt` via `view_file` + `StartLine`/`EndLine` — MySQL: `44-118`, PostgreSQL: `119-236`, SQL Server: + `237-305`, Oracle: `306-463`, HBase/DynamoDB/Cassandra: `464-616`, + Redis/Valkey/Memcached/Aerospike: `617-751`, MongoDB/DocumentDB/CosmosDB: + `752-797`, Neo4j/Neptune/Graph: `798-836` — to formulate a single primary + recommendation). +2. **Deliver Recommendation:** Deliver the recommendation to the user, mapping + destination codes to plain English, explaining the reasoning, and offering + to help with provisioning. + +### Phase 3: Implementation & Provisioning (Plan-Validate-Execute Pattern) + +When the user accepts the recommendation and requests to provision or modify +cloud resources, you must initiate the Implementation & Provisioning phase. + +1. **Load Provisioning Instructions (JiT):** Read the Phase 3 section of + `references/onboarding_prompts.md` (if not already loaded) using + `view_file`. +2. **Execute Plan-Validate-Execute Workflow:** Follow the detailed Phase 3 + instructions in `references/onboarding_prompts.md` to analyze the workspace, + confirm target file paths with the user, draft and validate Terraform or + `gcloud` provisioning scripts according to the required formatting and + tagging rules, and prepare a Change List (CL) or Pull Request for user + review without applying live infrastructure changes directly. + +-------------------------------------------------------------------------------- + +## Supporting Resources & Documentation + +- [Onboarding Prompts & Phase Rules](references/onboarding_prompts.md) +- [Database Recommendation Matrix](references/recommendation_matrix.txt) +- [Google Cloud Databases Overview](https://cloud.google.com/products/databases) +- [Google Cloud SDK (gcloud) Reference](https://cloud.google.com/sdk/gcloud/reference) diff --git a/skills/cloud-databases-onboarding/references/onboarding_prompts.md b/skills/cloud-databases-onboarding/references/onboarding_prompts.md new file mode 100644 index 00000000..dc3c5278 --- /dev/null +++ b/skills/cloud-databases-onboarding/references/onboarding_prompts.md @@ -0,0 +1,261 @@ +# Database onboarding agent instructions + +Your primary role is to assist users in selecting the best GCP database service +for their use case from the Google Cloud (GCP) portfolio. + +You operate in one of three distinct phases: **Phase 1 (Discovery)**, **Phase 2 +(Recommendation)**, or **Phase 3 (Implementation & Provisioning)**. Evaluate the +active conversation history to determine your current phase and strictly follow +the instructions for that phase. + +## CRITICAL CONSTRAINTS + +**NEVER provide a premature database recommendation or guess a database choice +during Phase 1 when the user's prompt is open-ended and lacks source/workload +context.** *Why this matters:* Recommending a target database without knowing +the data model, transaction consistency (ACID vs. eventual), and workload/scale +leads to incompatible architectural choices, failed migrations, and costly +re-architecting. However, when the user's prompt **already specifies** their +source database (or application context) and workload/migration requirements +matching `references/recommendation_matrix.txt`, immediately proceed to +**Phase 2 (Recommendation)** in the same turn. + +**GLOBAL OVERRIDE: Instance Creation** **Condition:** At ANY point in the +conversation (Phase 1, Phase 2, or Phase 3), if the user explicitly requests to +create or provision a database instance (e.g., "Create a PostgreSQL instance" or +"Provision Cloud Spanner"). **Your Action:** + +1. IMMEDIATELY proceed to **Phase 3: Implementation & Provisioning + (Plan-Validate-Execute)** using the requested database type and any + requirements gathered so far in the conversation. + +-------------------------------------------------------------------------------- + +## PHASE 1: DISCOVERY (Information Gathering) + +**Condition:** +**DO NOT JUMP TO RECOMMENDATION WITHOUT ASKING FOR THE BELOW INFORMATION.** +You are in this phase when the user's request is open-ended and does not yet +provide enough information to match a scenario in +`references/recommendation_matrix.txt`. *A "complete picture" generally requires +knowing: 1) Data model (e.g., relational, document, key-value, vector), 2) +Primary workload (OLTP, OLAP, HTAP, vector search), 3) Scale, latency, and +throughput requirements (e.g., QPS, vector count, number of dimensions, index +size), 4) On-prem requirements and 5) Current database/migration context.* + +**Stay in this phase until you have gathered enough +information to confidently recommend a database product with 90% confidence.** + +**Your Actions & Rules:** + +1. **Optional IDE & Connection Pre-check:** When available (such as in the Data + Cloud VS Code extension), optionally call `get_active_gcp_connection` (to + resolve the active GCP project and region) and `get_active_editor_context` + (to inspect open files or database resources) so you can skip redundant + discovery questions and prefill context for Phase 3. Treat these tools as + strictly optional and proceed normally if they are unavailable. + +2. **Efficiency:** Gather requirements efficiently by asking multiple relevant + follow-up questions in a single turn. Your goal is to get a "complete + picture" (data model, workload, scale, and context) as quickly as possible. + +3. **Constraint:** Aim to gather enough information to make a confident + recommendation within **1-2 turns maximum**. + +4. **User-Friendly Phrasing:** Frame your questions in plain, accessible + language. If you must use technical jargon (e.g., "OLTP," "HTAP," + "relational," "horizontal scaling"), you MUST provide a brief, + easy-to-understand definition in plain English. + +5. **Provide Examples:** Along with each question, provide 3 or 4 options for + the users to choose from. Users may also respond with free text. + + * *Example:* "Are you dealing with highly structured data (like strict + accounting tables where every row looks the same), flexible/unstructured + data (like user profiles with varying attributes, JSON documents, or IoT + device logs), or both?" + + * *Example:* "What does the database need to do with the stored + information? For example, will you primarily be adding to and updating + it (like a shopping cart or user profile), searching and transforming it + (like running reports or analytics), or both?" + +6. If the user mentions a specific source database, consult the + `discovery_questions` under the corresponding `# Source:` section in + `references/recommendation_matrix.txt` as a baseline, adapting them to fit + these user-friendly rules. + +7. Maintain a conversational, empathetic, and helpful tone. + +8. Ask the user if they have any additional context or requirements to add + about their workflow. + +**CRITICAL CONSTRAINT:** Do NOT consult `references/recommendation_matrix.txt` +to output a database recommendation during this phase. *Why this matters:* +Proposing a target database without a complete discovery context results in +inaccurate or generic recommendations that fail to account for critical +migration complexity, schema constraints, or IOPS requirements. + +-------------------------------------------------------------------------------- + +## PHASE 2: RECOMMENDATION (Analysis & Delivery) + +**Condition:** You are in this phase when the user has provided enough explicit +information (either in their initial prompt or across 1-2 discovery turns) to +identify the matching recommendation in `references/recommendation_matrix.txt`. + +**Your Actions (Execute in strict order):** + +### **Step A: Requirement Summary & Matrix Lookup** + +1. Distill all the user's requirements from the conversation history into a + single, comprehensive summary sentence. + +2. **CONSULT THE RECOMMENDATION MATRIX:** Search for and read the relevant + `# Source:` section in `references/recommendation_matrix.txt` to match the + distilled requirements against the `selection_criteria`, `best_when`, + `migration_complexity`, and `benefits` fields. + +*💡 Examples of effective requirement summaries for matching against the matrix:* + +* "I need a low-cost, 100% open-source compatible, fully managed relational + database for a small internal web app. It will have low traffic and doesn't + need to scale much. MySQL or PostgreSQL is fine. We are on MySQL 5.7." + +* "We're migrating a large (10TB+) on-prem PostgreSQL 15 database to GCP. We + need a managed service with significantly better performance for mixed OLTP + and analytical (HTAP) workloads than standard PostgreSQL, plus low-lag read + replicas for reporting." + +* "We are modernizing from a self-hosted Oracle 19c database and want to move + to an open-source compatible database on GCP to reduce licensing costs. Key + needs are robust migration tooling, high availability, HTAP capabilities, + and a fully managed service." + +* "For our core transaction system, I need a globally distributed relational + database capable of horizontal write scaling with strong consistency, and it + must offer at least 99.999% availability. We're migrating from a sharded + MySQL setup." + +* "I'm looking for a managed NoSQL database on GCP for a high-throughput (1M+ + OPS), low-latency (<10ms) key-value store, primarily for time-series data + from IoT devices. We're migrating from on-prem HBase." + +* "I'm building a new mobile app and need a serverless NoSQL document database + that's easy to use, handles real-time data synchronization with clients, + offers offline support, and has a generous free tier to start." + +### **Step B: The Final Response & Recommendation Delivery** + +1. **CRITICAL INSTRUCTION:** After evaluating + `references/recommendation_matrix.txt`, you must generate a final, + user-facing response explaining the recommendation. + +2. Provide **exactly ONE** top database recommendation based on the matrix + output (do not provide a menu of multiple primary recommendations). + +3. **Map Destination Codes to Plain English:** The matrix uses destination + codes (enums). You MUST translate them to plain-English product names in + your response to the user according to this mapping: + + * `ALLOYDB_FOR_POSTGRESQL` -> AlloyDB for PostgreSQL + * `ALLOYDB_OMNI` -> AlloyDB Omni + * `BIGTABLE` -> Cloud Bigtable + * `CLOUD_SQL_FOR_MYSQL_ENTERPRISE` -> Cloud SQL for MySQL Enterprise + * `CLOUD_SQL_FOR_MYSQL_ENTERPRISE_PLUS` -> Cloud SQL for MySQL Enterprise + Plus + * `CLOUD_SQL_FOR_POSTGRESQL_ENTERPRISE` -> Cloud SQL for PostgreSQL + Enterprise + * `CLOUD_SQL_FOR_POSTGRESQL_ENTERPRISE_PLUS` -> Cloud SQL for PostgreSQL + Enterprise Plus + * `CLOUD_SQL_FOR_SQL_SERVER` -> Cloud SQL for SQL Server + * `FIRESTORE` -> Google Cloud Firestore + * `MEMORYSTORE` -> Memorystore + * `MEMORYSTORE_FOR_MEMCACHED` -> Memorystore for Memcached + * `MEMORYSTORE_FOR_REDIS` -> Memorystore for Redis + * `MEMORYSTORE_FOR_VALKEY` -> Memorystore for Valkey + * `SPANNER` -> Cloud Spanner + * `SPANNER_GRAPH` -> Cloud Spanner Graph + * `ORACLE_AT_DATABASES` -> Oracle Database@Google Cloud + * `ORACLE_SELF_MANAGED` -> Oracle (Self-Managed) + * `SPANNER_OMNI` -> Spanner Omni + +4. Clearly explain your reasoning by mapping the GCP database features, + selection criteria, migration complexity (including specific protocols, + APIs, or migration tools), and benefits from + `references/recommendation_matrix.txt` to the user's stated requirements. + Keep the explanation accessible, just like in the discovery phase. + +5. Offer to help the customer create a starter instance of the recommended + database (transitioning to Phase 3 upon user acceptance). + +-------------------------------------------------------------------------------- + +## PHASE 3: IMPLEMENTATION & PROVISIONING (Plan-Validate-Execute) + +**Condition:** You are in this phase when the user accepts the recommendation +and requests to provision, create, or modify database resources, or triggers the +global override. + +**Your Actions & Rules (Follow strict Plan-Validate-Execute):** + +Follow the **Plan-Validate-Execute Pattern** to draft the infrastructure +modifications. Do NOT apply changes to production directly: + +0. **Environment Verification:** Check if required tools (e.g., `gcloud`, + `terraform`, `python3`) are installed and properly authenticated on the + system. If tools are missing or not authenticated, guide the user on how to + install or log in before proceeding. + +1. **Analyze the Workspace & Active Context:** If not already retrieved during + discovery, optionally call `get_active_gcp_connection` (to prefill the + default `` and region) and `get_active_editor_context` (when + available in the IDE). Scan the user's workspace, open files, and related + directories for existing database resource scripts (e.g., Terraform + configuration files). + +2. **Obtain User Confirmation:** If the target infrastructure files or + directory are not clear, ask the user explicitly to confirm the file + paths or target directory before modifying anything. + +3. **Draft Infrastructure Plan (Plan):** Create or edit the necessary Terraform + configuration files or shell scripts to provision the resources. + + - When generating `gcloud` CLI commands or shell scripts, you MUST follow + and apply these `gcloud` CLI rules: + * Always use `gcloud beta` command group for database provisioning + (e.g., `gcloud beta create`). + * Validate leaf-level command syntax using `gcloud help + ` prior to proposing commands. + * Explicitly append `--project=` and explicit location + flags (`--region`, `--zone`, `--location`, or `--config`). + * Include `--dry-run` or `--validate-only` for pre-execution + verification if supported by the command. + * Include custom label/tag flags (e.g., + `--labels=resource_generated_by=cloud_db_onboarding_skill`) on + generated `gcloud` provisioning commands. + * Do NOT include `--quiet` (`-q`) flags in generated `gcloud` commands + (commands are drafted for interactive user review). + * The skill MUST ONLY draft provisioning commands or code for user + review and MUST NOT execute mutating/write infrastructure operations + directly. + - When creating or editing Terraform files or shell scripts, you MUST: + * Add a stamped header comment at the top of every generated file + (e.g., `# Generated with cloud onboarding skills selector + @timestamp`). + * Add a custom default label/tag `resource_generated_by = + "cloud_db_onboarding_skill"` under the `default_labels` / `labels` + block or as a resource label (GCP label values only allow lowercase + letters, numeric characters, underscores, and hyphens—no spaces). + +4. **Validate Infrastructure Code (Validate):** Before finalizing, validate + the drafted infrastructure code to verify syntax and configuration + correctness. *Why this matters:* Validating Terraform code ensures that + configuration blocks, IAM bindings, and instance sizing are + syntax-error-free and strictly enforceable before code review. + +5. **Create Pull Request / Change List (Execute):** After validation succeeds + with zero errors, prepare the changes as a **Change List (CL)** or Pull + Request for user review. Never apply live infrastructure changes + (`terraform apply` or mutating `gcloud` commands) directly to production + yourself. diff --git a/skills/cloud-databases-onboarding/references/recommendation_matrix.txt b/skills/cloud-databases-onboarding/references/recommendation_matrix.txt new file mode 100644 index 00000000..c034b6c5 --- /dev/null +++ b/skills/cloud-databases-onboarding/references/recommendation_matrix.txt @@ -0,0 +1,836 @@ +# Recommendation Matrix for Database Migrations to Google Cloud + +# =================================================================================================== +# TABLE OF CONTENTS (SUPPORTED SOURCE DATABASES) +# Search for the corresponding `# Source:` header below to read a specific section: +# +# - Source Database: MySQL +# - Primary GCP Destinations: Cloud SQL for MySQL Enterprise / Plus, Spanner +# +# - Source Database: PostgreSQL +# - Primary GCP Destinations: Cloud SQL for PostgreSQL, AlloyDB, Spanner +# +# - Source Database: Microsoft SQL Server +# - Primary GCP Destinations: Cloud SQL for SQL Server, AlloyDB for PostgreSQL, Spanner +# +# - Source Database: Oracle +# - Primary GCP Destinations: Oracle Database@Google Cloud, Oracle Self-Managed, AlloyDB for PostgreSQL, Spanner +# +# - Source Database: HBase +# - Primary GCP Destinations: Bigtable +# +# - Source Database: Amazon DynamoDB +# - Primary GCP Destinations: Bigtable, Spanner +# +# - Source Database: Apache Cassandra +# - Primary GCP Destinations: Bigtable, Spanner +# +# - Source Database: Redis / Valkey +# - Primary GCP Destinations: Memorystore for Valkey +# +# - Source Database: Memcached +# - Primary GCP Destinations: Memorystore for Valkey +# +# - Source Database: Aerospike +# - Primary GCP Destinations: Bigtable, Memorystore for Valkey +# +# - Source Database: MongoDB / Amazon DocumentDB / Azure CosmosDB +# - Primary GCP Destinations: Firestore, Spanner +# +# - Source Database: Neo4j / Amazon Neptune / Aerospike Graph +# - Primary GCP Destinations: Spanner Graph +# =================================================================================================== + +# Source: MySQL +source_recommendations { + source: MYSQL + discovery_questions: [ + "What MySQL version?", + "What is the size of the database?", + "What are the availability requirements?", + "Is it a sharded MySQL deployment?", + "Are you willing to refactor the application?", + "What is the planned growth rate in the next 6-12 months?", + "Does your application require temporary scaling up to handle workload surges?" + ] + recommendations { + selection_criteria: [ + "100% open source compatible", + "Standard production performance and availability (99.95% SLA)", + "Also includes versions predating MySQL 8" + ] + primary_destination: CLOUD_SQL_FOR_MYSQL_ENTERPRISE + migration_complexity: [ + "Low complexity. Easy with Database Migration Service" + ] + best_when: "Existing MySQL customers are looking to stay on MySQL. Cloud SQL for MySQL is a great choice for straightforward migrations, offering an easy and accessible path to a fully managed, secure, and highly available cloud database." + benefits: [ + "Fully managed", + "Secure", + "H/A & DR", + "99.95% SLA", + "Broad MySQL version support", + "Lower entry point" + ] + } + recommendations { + selection_criteria: [ + "100% open source compatible", + "High performance (up to 3x faster reads, 2x faster writes) and high availability (99.99% SLA, near-zero maintenance downtime)", + "Continuous availability is needed", + "MySQL 8+ only" + ] + primary_destination: CLOUD_SQL_FOR_MYSQL_ENTERPRISE_PLUS + migration_complexity: [ + "Low complexity. Easy with Database Migration Service" + ] + best_when: "Cloud SQL Enterprise Plus is the premier choice for mission-critical MySQL workloads, delivering dramatic performance gains with up to 3x faster reads and 2x improved write latency, while ensuring maximum availability through near-zero maintenance downtime and a 99.99% SLA." + benefits: [ + "All the benefits of Cloud SQL for MySQL Enterprise", + "Optimized performance (3x faster)", + "Near zero maintenance downtime", + "99.99% SLA" + ] + } + recommendations { + selection_criteria: [ + "Sharded MySQL workloads (e.g. Vitess, app sharding) or large scale workloads", + "Ability to handle extremely large databases", + "Customers looking for highest scale and availability (e.g. payments use-cases)", + "Customer pain point around scale and management of existing sharded MySQL environment", + "Workloads with highly spiky or unpredictable traffic patterns that benefit from autoscaling to optimize costs" + ] + primary_destination: SPANNER + migration_complexity: [ + "Medium to Higher Complexity. Requires query and transaction handling changes, but Google SQL similarity to MySQL limits the rewrite effort. We offer free Spanner Migration Tool and Black Belt team expertise" + ] + best_when: "The source DB is sharded MySQL - Spanner transforms the operational burden of sharding. Instead of managing individual database shards, Spanner operates as a single, logical database that automatically handles horizontal scaling, delivering virtually infinite capacity and multi-region consistent writes. This shift eliminates the pain of re-sharding and cross-shard management, replacing it with a system built for the highest reliability, backed by an up to 99.999% SLA and no maintenance downtime. Spanner's native autoscaling also makes it a highly cost-effective solution for spiky workloads." + benefits: [ + "Virtually infinite scale", + "Multi-region consistent writes", + "No maintenance downtime", + "99.999% SLA", + "Online schema changes", + "Automatic scaling to handle spiky workloads and optimize costs" + ] + } +} + +# Source: PostgreSQL +source_recommendations { + source: POSTGRESQL + discovery_questions: [ + "What PostgreSQL version?", + "What is the size of the database?", + "Is it an analytical/HTAP workload?", + "Is it a sharded PostgreSQL deployment?", + "Do you need to leave a portion of your workload on-premises?", + "Do you need a multi cloud solution?", + "Do you need to build Gen AI applications with pgvector?", + "Does your application require temporary scaling up to handle workload surges?" + ] + recommendations { + selection_criteria: [ + "100% open source compatible", + "Standard production performance and availability (99.95% SLA)", + "Supports PostgreSQL 9.6 and above (including legacy versions 9.6, 10, 11)" + ] + primary_destination: CLOUD_SQL_FOR_POSTGRESQL_ENTERPRISE + migration_complexity: [ + "Low complexity. Easy with Database Migration Service" + ] + best_when: "Existing PostgreSQL customers who are looking to stay on PostgreSQL. Cloud SQL is a great choice for straightforward migrations, offering an easy and accessible path to a fully managed, secure, and highly available cloud database." + benefits: [ + "Fully managed", + "Secure", + "H/A & DR", + "99.95% SLA", + "Broadest PostgreSQL version support", + "Up to 64TB storage", + "pgvector support for Gen AI apps" + ] + } + recommendations { + selection_criteria: [ + "100% open source compatible", + "High performance (up to 4x reads, 2x writes) and high availability (99.99% SLA, near-zero maintenance downtime)", + "Supports PostgreSQL 12 and above" + ] + primary_destination: CLOUD_SQL_FOR_POSTGRESQL_ENTERPRISE_PLUS + migration_complexity: [ + "Low complexity. Easy with Database Migration Service" + ] + best_when: "Existing PostgreSQL customers who are looking to stay on PostgreSQL. Cloud SQL Enterprise Plus delivers dramatic performance gains, while ensuring maximum availability through a 99.99% SLA." + benefits: [ + "All the features and capabilities of Enterprise edition with up to 2x improved write throughput and up to 4x improved read throughput compared to Enterprise edition", + "Near zero maintenance downtime", + "99.99% SLA", + "pgvector support for Gen AI apps" + ] + } + recommendations { + selection_criteria: [ + "Higher level of performance and manageability than open source", + "Fully open source compatible", + "Extreme performance (4x transactional, 100x analytical HTAP) and high availability (99.99% SLA, near-zero maintenance downtime)", + "Hybrid Transactional and Analytical (HTAP) workloads", + "Do you need to scale out for reads?" + ] + primary_destination: ALLOYDB_FOR_POSTGRESQL + migration_complexity: [ + "Low complexity. Easy with Database Migration Service" + ] + best_when: "AlloyDB is a next-generation, PostgreSQL-compatible database that modernizes the most demanding enterprise workloads by delivering extreme performance for transactions and analytics (HTAP), a cloud-native elastic architecture, and a fully integrated platform for generative AI, all backed by enterprise-grade availability." + benefits: [ + "Superior performance, columnar engine, improved manageability, autopilot optimizations", + "4x performance for transactions and 100x faster analytical queries compared to stock PostgreSQL", + "Up to 64TB storage (>64TB coming soon)", + "Cloud-first architecture with complete disaggregation of compute and storage", + "Read replicas and read replica pools", + "Vertex AI integration for real-time model inferencing, BQ integration with Datastream", + "pgvector support for Gen AI apps", + "Near zero maintenance downtime" + ] + } + recommendations { + selection_criteria: [ + "Need to run PostgreSQL workloads on-premises and/or in any public cloud", + "Need to break free from legacy databases faster than you can replatform in the Cloud", + "Developers looking for a local / on-premises database" + ] + primary_destination: ALLOYDB_OMNI + migration_complexity: [ + "Low complexity" + ] + best_when: "PostgreSQL customers who are looking to stay on PostgreSQL. Looking to run on-prem, in any cloud, or in an air-gapped environment." + benefits: [ + "Run AlloyDB anywhere you run Databases", + "2x performance for OLTP and 100x for analytical queries compared to stock PostgreSQL", + "Support for edge computing, hybrid and multi-cloud customers who have regulatory restrictions or data sovereignty requirements", + "No cost for development and testing", + "Fully compatible with AlloyDB in the Cloud" + ] + } + recommendations { + selection_criteria: [ + "Ability to handle extremely large databases or sharded PostgreSQL workloads", + "Customers looking for highest scale and availability (e.g. payments use-cases)", + "Workloads with highly spiky or unpredictable traffic patterns that benefit from autoscaling to optimize costs" + ] + primary_destination: SPANNER + migration_complexity: [ + "Medium complexity. While Spanner is a distributed database, the Spanner PostgreSQL dialect significantly reduces the migration effort. We offer free Spanner Migration Tool and Black Belt team expertise" + ] + best_when: "The source is a large or sharded PostgreSQL - Spanner transforms the operational burden of sharding. Instead of managing individual database shards, Spanner operates as a single, logical database that automatically handles horizontal scaling, delivering virtually infinite capacity and multi-region consistent writes. This shift eliminates the pain of re-sharding and cross-shard management, replacing it with a system built for the highest reliability, backed by an up to 99.999% SLA and no maintenance downtime. Spanner's native autoscaling also makes it a highly cost-effective solution for spiky workloads." + benefits: [ + "Virtually infinite scale", + "Multi-region consistent writes", + "No maintenance downtime", + "99.999% SLA", + "Online schema changes", + "Analyze operational data isolated from transactional workloads with BigQuery", + "Automatic scaling to handle spiky workloads and optimize costs" + ] + } +} + +# Source: SQL Server +source_recommendations { + source: SQL_SERVER + discovery_questions: [ + "What SQL Server version?", + "Are you concerned about licensing costs and do you have an existing ELA with Microsoft?", + "SQL Server-specific options needed?", + "Is this OLTP, analytical, or mixed workload?", + "Are you willing to refactor the application?", + "Are you interested in moving to open source?" + ] + recommendations { + selection_criteria: [ + "100% SQL Server compatible", + "Supports legacy or packaged apps that need to run on SQL Server", + "Data center exit" + ] + primary_destination: CLOUD_SQL_FOR_SQL_SERVER + migration_complexity: [ + "Easy homogeneous migrations with Database Migration Service coming soon; Current solution is to use SQL Server technologies" + ] + best_when: "Existing SQL Server users are looking to stay on SQL Server, Cloud SQL for SQL Server is a great choice for straightforward migrations, offering an easy and accessible path to a fully managed, secure, and highly available cloud database." + benefits: [ + "Fully managed", + "Secure", + "H/A & DR", + "99.95% SLA", + "Multi-regional DR" + ] + } + recommendations { + selection_criteria: [ + "Looking to break free from legacy, proprietary database with expensive licensing and modernize to open source", + "Support for Hybrid Transactional and Analytical (HTAP) workloads via an open (PostgreSQL) interface" + ] + primary_destination: ALLOYDB_FOR_POSTGRESQL + migration_complexity: [ + "Heterogeneous migration required with application code changes; No cost Database Migration Assessment via first-party tool" + ] + best_when: "(Modernization opportunity) For SQL Server customers looking to break free from legacy licensing costs, AlloyDB for PostgreSQL offers the premier path to modernization. It provides a fully managed, PostgreSQL-compatible experience that combines the freedom of open-source with performance far exceeding the standard version. AlloyDB is built to handle the most demanding enterprise workloads with its extreme speed for transactions and analytics (HTAP), a cloud-native elastic architecture, and a powerful, integrated AI platform." + benefits: [ + "Fully PostgreSQL compatible, with no licensing costs or vendor lock-in", + "Superior performance, columnar engine, improved manageability, autopilot optimizations", + "Cloud-first architecture with complete disaggregation of compute and storage", + "Vertex AI integration for real-time model inferencing, BigQuery integration with Datastream", + "99.99% SLA", + "Monetize your operational data with AI-driven insights: Build transformative applications with vector search, foundation models integration, and generative AI, while maintaining open-source compatibility", + "Near zero maintenance downtime" + ] + } + recommendations { + selection_criteria: [ + "Highest scale and availability requirements (e.g. global payments workloads)", + "Willing to modernize and migrate to a distributed database to eliminate legacy SQL Server licensing costs" + ] + primary_destination: SPANNER + migration_complexity: [ + "Higher complexity. Requires heterogeneous migration and application rewrite from T-SQL to Google SQL or PostgreSQL dialect. We offer database migration assessment tools and expertise" + ] + best_when: "SQL Server customers with extreme scale, global availability, and strong consistency requirements that exceed the capabilities of traditional relational databases. Spanner provides virtually infinite horizontal scale, multi-region consistent writes, and an up to 99.999% SLA, making it the premier choice for mission-critical modernizations." + benefits: [ + "Virtually infinite scale and global consistency", + "99.999% SLA with no maintenance downtime", + "Eliminate expensive SQL Server licensing costs", + "Support for Google SQL and PostgreSQL dialects" + ] + } +} + +# Source: Oracle +source_recommendations { + source: ORACLE + discovery_questions: [ + "What Oracle Database version and do you have an existing ULA?", + "What is the size of the database?", + "Oracle Database options used?", + "Are you willing to do some refactoring of the application?", + "Are you running Oracle Apps and/or other COTS apps that use Oracle DB?", + "Need to run databases on-premises and/or in any public cloud?" + ] + # source: ORACLE - recommendation #1 + recommendations { + selection_criteria: [ + "Mission critical databases", + "Runs Exadata today", + "Highest performance or availability requirements", + "Dedicated infrastructure requirements", + "Needs to run DW or mixed workload with Oracle" + ] + primary_destination: ORACLE_AT_DATABASES + migration_complexity: [ + "Low complexity, supports physical and logical migrations" + ] + benefits: [ + "Highest performance and availability of any Oracle database platform", + "Deploy Oracle Database 19c or 23ai with automation from Oracle", + "License included or BYOL pricing for flexible compliance", + "Dedicated infrastructure provides highest levels of security and consistent performance", + "Run the latest version of Oracle Database, including 23ai", + "Take advantage of Google Cloud’s differentiated AI and analytics capabilities" + ] + } + # source: ORACLE - recommendation #2 + recommendations { + selection_criteria: [ + "Runs Oracle RAC today", + "Needs access to operating system", + "Has a limited number of Oracle licenses available", + "Wants to run RAC without negotiating a contract with Oracle" + ] + primary_destination: ORACLE_AT_DATABASES + migration_complexity: [ + "Low complexity, supports physical and logical migrations" + ] + benefits: [ + "Deploy Oracle Database 19c or 23ai with automation from Oracle", + "High availability with the lowest cost of entry", + "License included or BYOL pricing for flexible compliance", + "Scalable deployments take advantage of cloud economics", + "Run the latest version of Oracle Database, including 23ai", + "Take advantage of Google Cloud’s differentiated AI and analytics capabilities", + "Available with public offer to run month-to-month without long term contracts" + ] + } + # source: ORACLE - recommendation #3 + recommendations { + selection_criteria: [ + "Smaller departmental or dev/test databases", + "Needs access to operating system", + "Has a limited number of Oracle licenses available" + ] + primary_destination: ORACLE_AT_DATABASES + migration_complexity: [ + "Low complexity, supports physical and logical migrations" + ] + benefits: [ + "Lowest cost of entry for Oracle Database", + "License included or BYOL pricing for flexible compliance", + "Run the latest version of Oracle Database, including 23ai", + "Take advantage of Google Cloud’s differentiated AI and analytics capabilities" + ] + } + # source: ORACLE - recommendation #4 + recommendations { + selection_criteria: [ + "Smaller departmental or dev/test databases", + "Looking for a fully managed database", + "Has a limited number of Oracle licenses available" + ] + primary_destination: ORACLE_AT_DATABASES + migration_complexity: [ + "Low complexity, supports physical and logical migrations" + ] + benefits: [ + "Fully managed deployment of Oracle database, including backup and recovery, patching, and system management", + "License included or BYOL pricing for flexible compliance", + "Scalable deployments ranging from 1 CPU to hundreds of cores", + "Run the latest version of Oracle Database, including 23ai", + "Take advantage of Google Cloud’s differentiated AI and analytics capabilities", + "Available with public offer to run month-to-month without long term contracts" + ] + } + # source: ORACLE - recommendation #5 + recommendations { + selection_criteria: [ + "Lift and shift your Oracle database and associated workload to GCE/GKE", + "Data center exit", + "In-house and department app workloads running on Oracle database" + ] + primary_destination: ORACLE_SELF_MANAGED + migration_complexity: [ + "Low complexity, lift and shift" + ] + benefits: [ + "Run Oracle in any Google Cloud region globally", + "Maintain full control over database and OS versions, enabling support for critical legacy Oracle instances or non-Oracle Linux operating systems", + "Elastic, on-demand infrastructure lets you scale compute and storage independently", + "Ideal for applications moving to microservice architecture with GKE" + ] + } + # source: ORACLE - recommendation #6 + recommendations { + selection_criteria: [ + "Mission critical databases", + "Highest performance or availability requirements", + "100% open source compatible (need to reduce proprietary database license expenditures)", + "Future proofing apps for AI", + "Hybrid Transactional and Analytical (HTAP) workloads" + ] + primary_destination: ALLOYDB_FOR_POSTGRESQL + migration_complexity: [ + "Heterogeneous migration with schema and data migration with Database Migration Service;", + "Application code changes required and support for automated code conversion coming soon" + ] + best_when: "Oracle customers looking to move away from the expensive licensing and vendor lock-in technology to an open standards based database with a rich set of capabilities to meet their enterprise workloads needs. AlloyDB is built to handle the most demanding enterprise workloads with its extreme speed for transactions and analytics (HTAP), a cloud-native elastic architecture, and a powerful, integrated AI platform." + benefits: [ + "Fully PostgreSQL compatible, with no vendor lock-in", + "Superior performance, columnar engine, improved manageability, autopilot optimizations", + "4x transaction throughput and up to 100x faster analytical queries compared to stock PostgreSQL", + "Cloud-first architecture with complete disaggregation of compute and storage", + "Vertex AI integration for real-time model inferencing, BigQuery integration with Datastream", + "Monetize your operational data with AI-driven insights. Build transformative applications with vector search, foundation models integration, and generative AI, while maintaining open-source compatibility.", + "Near zero maintenance downtime", + "Run AlloyDB anywhere you run databases with AlloyDB Omni", + "Suitable for edge computing, hybrid and multi-cloud, customers who have regulatory restrictions or data sovereignty requirements" + ] + } + # source: ORACLE - recommendation #7 + recommendations { + selection_criteria: [ + "Highest scale and availability requirements (e.g. global payments workloads)", + "Looking to modernize and eliminate expensive Oracle licensing and vendor lock-in" + ] + primary_destination: SPANNER + migration_complexity: [ + "Higher complexity. Requires heterogeneous migration and application rewrite from PL/SQL to Google SQL or PostgreSQL dialect. We offer database migration assessment tools and expertise" + ] + best_when: "Oracle customers with extreme scale, global availability, and strong consistency requirements that exceed the capabilities of traditional relational databases. Spanner provides virtually infinite horizontal scale, multi-region consistent writes, and an up to 99.999% SLA, making it the premier choice for mission-critical modernizations." + benefits: [ + "Virtually infinite scale and global consistency", + "99.999% SLA with no maintenance downtime", + "Eliminate Oracle licensing costs and vendor lock-in", + "Support for Google SQL and PostgreSQL dialects" + ] + } +} + +# Source: HBase +source_recommendations { + source: HBASE + discovery_questions: [ + "What distribution of HBase?", + "What is the database size, number of nodes, CPU and disk per cluster, and hardware configuration?", + "Are coprocessors being used, and for what purpose?" + ] + recommendations { + selection_criteria: [ + "Looking to move from self managed to fully managed", + "HBase compatible key-value store suitable for all your application", + "Cost sensitive app with high write throughput & large data volumes", + "Storage-driven apps needing low cost storage with moderate to high read/write throughput" + ] + primary_destination: BIGTABLE + migration_complexity: [ + "Tools for assessment and migration.", + "Support for live & offline migrations", + "Direct import tools from HBase to Bigtable", + "Ability to migrate coprocessor functionality" + ] + best_when: "For customers running HBase, Bigtable is the ideal migration destination, offering a seamless path to modernization with its native support for the HBase API. This allows customers the ability to migrate existing applications with minimal code changes while eliminating the operational burden of managing complex HBase clusters. By moving to a fully managed platform, you gain virtually infinite scaling with single-digit millisecond latency, an up to 99.999% SLA with no maintenance downtime, and a lower TCO at any scale, driven by a native autoscaler and flexible SSD/HDD storage options." + benefits: [ + "Fully managed key-value DB", + "Support for HBase APIs", + "Lower TCO at any scale", + "Single digit millisecond read and write latencies", + "Multi-writer support", + "Virtually infinite scaling including writes", + "Global replication and 99.999% SLA", + "Autoscaler to reduce costs", + "No maintenance downtime", + "Low storage costs with both SSD and HDD" + ] + } +} + +# Source: DynamoDB +source_recommendations { + source: DYNAMODB + discovery_questions: [ + "What is the read/write workload ratio?", + "What are the latency requirements?", + "What capacity mode: on-demand or provisioned capacity?", + "Are secondary indexes being used?", + "Are multi-row transactions being used?", + "Are Global Tables being used?" + ] + recommendations { + selection_criteria: [ + "High write ratio (for Clickstream or Time Series style workloads)", + "Large storage volume, needing low cost storage", + "Flexible schema", + "Multi-primary eventually consistent writes", + "Multi-region write latency sensitive and using DynamoDB Global Tables" + ] + primary_destination: BIGTABLE + migration_complexity: [ + "Heterogeneous migration", + "Application code conversion required" + ] + best_when: "Customers are looking for similar access patterns including retrieving data via a unique key that is identical in both systems. Application logic designed for DynamoDB's key-based lookups translates naturally to Bigtable's row key model. This conceptual alignment means the migration effort is less about re-architecting the application's core logic and more about adapting to a platform designed for superior scale and performance, building upon the familiar principles of a wide-column, key-value data store." + benefits: [ + "Virtually infinite scaling including writes", + "Single digit millisecond read and write latencies", + "Flexible single or multi-replica deployment", + "Low storage costs with both SSD and HDD", + "Lower TCO for most workloads" + ] + } + recommendations { + selection_criteria: [ + "High read ratio", + "Application leverages advanced database capabilities (indexes, composite keys, typed values, constraints, JSON, views, ACID transactions, SQL)", + "Need strong and/or eventual consistency options for reads", + "Need replication with strong consistency guarantees" + ] + primary_destination: SPANNER + migration_complexity: [ + "Heterogeneous migration", + "Application code conversion required" + ] + best_when: "For DynamoDB users, migrating to Spanner is a strategic modernization that provides a significant upgrade in database capabilities for roughly half the cost. This move allows customers the ability to evolve from a NoSQL K/V store to a globally distributed multi-model database that unlocks advanced features not native to DynamoDB." + benefits: [ + "Virtually infinite scaling including writes", + "Advanced database capabilities (e.g. SQL query, indexes, transactions, ACID, and typed values)", + "Single digit milliseconds regional read and write latencies (strongly consistent)", + "Single-region and multi-region strong consistency", + "1/2 cost of DynamoDB" + ] + } +} + +# Source: Cassandra +source_recommendations { + source: CASSANDRA + discovery_questions: [ + "What is the read/write workload ratio?", + "What are the latency requirements?", + "Are secondary indexes being used?", + "Are strong consistency reads needed?", + "How many replicas are needed?", + "What is the quorum consistency?", + "What is the data volume?", + "Do you need CQL querying dialect?" + ] + recommendations { + selection_criteria: [ + "High write ratio (for Clickstream or Time Series style workloads)", + "Large storage volume, needing low cost storage", + "Flexible schema", + "Multi-primary eventually consistent writes", + "Multi-region write latency sensitive" + ] + primary_destination: BIGTABLE + migration_complexity: [ + "Heterogeneous migration", + "Application code conversion required" + ] + best_when: "When users want to maintain a familiar architecture. Both are wide-column NoSQL databases, meaning the core concepts of schema design and key-based access patterns are highly transferable, simplifying the transition. The primary driver for migrating is to move from a self-managed cluster to a fully managed, auto-scaling service. This allows you to offload the complexities of maintenance and tuning, while gaining Bigtable’s consistent single-digit millisecond latency and a lower total cost of ownership at virtually any scale." + benefits: [ + "Virtually infinite scaling including writes", + "Single digit milliseconds read and write latency", + "Flexible single or multi-replica deployment", + "Low storage costs with both SSD and HDD", + "Lower TCO for most workloads" + ] + } + recommendations { + selection_criteria: [ + "High read ratio", + "Application leverages advanced database capabilities (indexes, composite keys, typed values, constraints, JSON, views, ACID transactions, SQL)", + "Use SQL API", + "Need strong and/or eventual consistency options for reads", + "Need replication with strong consistency guarantees" + ] + primary_destination: SPANNER + migration_complexity: [ + "Low complexity", + "Near lift-and-shift migration using Spanner's native Cassandra-compatible interface (CQL and wire protocol support) with minimal application changes" + ] + best_when: "When users are looking to gain capabilities that are fundamentally impossible with a NoSQL architecture. The primary driver is upgrading from eventual consistency to Spanner's global, fully ACID-compliant transactions, which ensures absolute data integrity and simplifies application development. A key advantage of this modernization is Spanner's unified multi-model platform, allowing you to combine relational (SQL), full-text search, graph, and vector search queries directly on your live transactional data. This eliminates the need for separate specialized databases and complex data pipelines. Furthermore, the columnar storage engine in Spanner, which enables true HTAP (Hybrid Transactional/Analytical Processing) allows you to run fast, real-time analytical queries directly on your transactional data, a capability far beyond Cassandra's scope. The transition is also more practical than ever, as Spanner's native Cassandra interface allows developers to reuse their CQL queries and client libraries, minimizing code refactoring while eliminating the operational burden of Cassandra with a fully managed service." + benefits: [ + "Virtually infinite scaling including writes", + "Single digit milliseconds regional read and write latencies", + "Single-region and multi-region strong consistency", + "Advanced database capabilities (e.g. SQL query, indexes, transactions, ACID, and typed values)", + "Lower TCO for most workloads" + ] + } +} + +# Source: Redis, Valkey +source_recommendations { + source: [REDIS, VALKEY] + discovery_questions: [ + "What is the primary use case (caching, session store, pub/sub, real-time data)?", + "What are the latency and throughput requirements?", + "Is high availability and automatic failover critical?", + "Are you looking for a fully managed service to reduce operational overhead?", + "Which version of Redis/Valkey?", + "Are you using clustering?", + "What is the read/write throughput?", + "Size of data stored in cache?" + ] + recommendations { + selection_criteria: [ + "Need for a fully managed, in-memory data store.", + "High throughput and ultra-low latency required.", + "Compatibility with existing Redis commands and clients.", + "High availability (automatic failover, replication) is essential.", + "Use cases include caching, session management, real-time analytics, leaderboards, message queues.", + "Moving from Redis to Valkey for OSS or for better performance", + "Moving from self managed to fully managed", + "Looking for an integrated experience with IAM Auth and other services" + ] + primary_destination: MEMORYSTORE_FOR_VALKEY + migration_complexity: [ + "Low complexity migrations.", + "Compatible with existing Redis and Valkey APIs.", + "Data migration typically using RIOT or import a RDB file using RIOT tooling.", + "No application code changes required." + ] + best_when: "For customers running Redis or Valkey, Memorystore for Valkey is the best choice. Valkey is fully compatible with Redis 7.2, making migration easy and seamless with no application code changes required. It offers a fully open-source solution with no lock-in, while delivering substantial performance enhancements (such as 3x throughput improvements compared to Memorystore for Redis Cluster in Valkey 9.0) and extreme performance with sub-millisecond latency, high availability (99.99% SLA), and zero-downtime scaling." + benefits: [ + "Fully managed service, reducing operational burden.", + "Fully managed experience to accelerate innovation and reduce risk and TCO (usually customers save >40-50% from Redis Enterprise)", + "Full open-source compatibility with no lock-in", + "Extreme performance with sub-millisecond latency and millions of operations per second.", + "High availability (99.99% SLA) with automatic failover and replicas.", + "Scalable (zero-downtime scaling).", + "Zero downtime maintenance", + "Integrated with Google Cloud ecosystem (monitoring, logging, security).", + "Open-source compatible (Redis 7.2 and below, Valkey 7.2, 8.0, and 9.0).", + "Supports persistence (RDB snapshot, AOF logging). And backups", + "Support cross regional replication", + "Valkey supports both clustering and noncluster mode.", + "Valkey 9.0 supports 3x throughput improvements compared to Memorystore for Redis Cluster.", + "Valkey supports up to 14.5TB data in cache" + ] + } +} + +# Source: Memcached +source_recommendations { + source: MEMCACHED + discovery_questions: [ + "What is the read/write throughput?", + "How many nodes?", + "Size of data stored in Memcached?" + ] + recommendations { + selection_criteria: [ + "Moving to the cloud?", + "Moving from self managed to fully managed?", + "Looking for a next-gen engine to replace deprecated Memcached" + ] + primary_destination: MEMORYSTORE_FOR_VALKEY + migration_complexity: [ + "Medium complexity migration.", + "Requires updating application code to use a Valkey- or Redis-compatible client library (RESP protocol) instead of Memcached protocol." + ] + best_when: "For customers running Memcached, Memorystore for Valkey is the recommended migration path as the Memcached service is being deprecated. Valkey serves as our next-generation engine, providing a modern, fully open-source in-memory solution." + benefits: [ + "Fully open source engine with no vendor lock-in.", + "Zero-downtime horizontal and vertical scaling (from 1.25 GB instances to 27.5TB clusters).", + "High availability architecture with a 99.99% SLA.", + "Rich and flexible data structures (beyond simple key-value).", + "Supports multi-command transactions and Lua scripting.", + "Access to advanced modules.", + "Persistence options including RDB snapshots and AOF (Append Only File)." + ] + } +} + +# Source: Aerospike +source_recommendations { + source: AEROSPIKE + discovery_questions: [ + "How large is the data?", + "What are latency requirements?", + "Is multi-region or multi-writer deployment required?", + "What are availability requirements?" + ] + recommendations { + selection_criteria: [ + "Looking to move from self managed to fully managed", + "Data needs to be persisted", + "Data volume makes in-memory options cost-prohibitive", + "Does not require sub-millisecond latencies" + ] + primary_destination: BIGTABLE + migration_complexity: [ + "Heterogeneous migration", + "Application code conversion required" + ] + best_when: "Bigtable is a great destination for Aerospike workloads as it delivers single-digit millisecond latency and with significantly lower TCO and reduced risk of a fully managed, auto-scaling platform. You get matching performance while completely eliminating the operational overhead of managing complex clusters." + benefits: [ + "Fully managed experience and reduced risk and TCO", + "Open-source compatibility with no lock-in", + "Single digit millisecond read and write latency", + "Globally replicated and up to 99.999% SLA", + "Low storage cost and native autoscaler to reduce cost" + ] + } + recommendations { + selection_criteria: [ + "Moving from self managed to fully managed", + "Non-volatile storage is not required", + "Requires sub-millisecond latencies", + "Dataset is relatively small" + ] + primary_destination: MEMORYSTORE_FOR_VALKEY + migration_complexity: [ + "Heterogeneous migration", + "Application code conversion required" + ] + best_when: "For Aerospike workloads used primarily as a high-speed cache where durable, persistent storage is not a requirement, Memorystore for Valkey is an excellent choice for modernization and cost reduction." + benefits: [ + "Fully managed experience and reduced risk and TCO", + "Open-source compatibility with no lock-in", + "Sub-millisecond read and write latencies", + "Clustering support for increased scale and throughput with 99.99% SLA" + ] + } +} + +# Source: MongoDB, Amazon DocumentDB, Azure CosmosDB +source_recommendations { + source: [MONGODB, AMAZON_DOCUMENTDB, AZURE_COSMOSDB] + discovery_questions: [ + "What is the size of your DocumentDB deployment?", + "What is the read/write throughput?" + ] + recommendations { + selection_criteria: [ + "Large instance or large storage requirements", + "Lower combined costs for storage and I/O", + "High availability" + ] + primary_destination: FIRESTORE + migration_complexity: [ + "Heterogeneous migration", + "Application code conversion required" + ] + best_when: "For customers with MongoDB workloads, Firestore is the best choice for modernizing your application and eliminating operational overhead. It provides a truly serverless platform that scales elastically to any size, which significantly lowers the total cost of ownership through a pay-for-what-you-use model. This simplicity is backed by enterprise-grade power, including an industry-leading 99.999% SLA and the ability to perform global, multi-document ACID transactions with strong consistency. With deep integration for event-driven architectures, Firestore allows you to build highly scalable and reliable applications without managing servers." + benefits: [ + "Serverless, elastic capacity with no scaling limits", + "Lower storage costs and no additional throughput (I/O) fees", + "Up to 99.999% SLA with automated failure recovery through built-in data replication and zero RPO/zero RTO", + "Global, ACID-compliant, multi-document transactions with strong consistency", + "Event triggering support for GKE, Cloud Run and Cloud Functions" + ] + } + recommendations { + selection_criteria: [ + "Highest scale and availability requirements", + "Complex query requirements, including indexes and JSON handling, at extreme scale" + ] + primary_destination: SPANNER + migration_complexity: [ + "Higher complexity. Requires heterogeneous migration from NoSQL document model to relational/hybrid model, and application rewrite" + ] + best_when: "MongoDB customers with extreme scaling requirements, demanding strong consistency, high availability (up to 99.999% SLA), and complex query capabilities (including JSON support and indexing) that exceed MongoDB's capabilities at scale." + benefits: [ + "Virtually infinite scale with global consistency", + "99.999% SLA with no maintenance downtime", + "Native support for JSON data type and indexing", + "Strong consistency for multi-region deployments" + ] + } +} + +# Source: Neo4j, Amazon Neptune, Aerospike Graph +source_recommendations { + source: [NEO4J, AEROSPIKE_GRAPH, AMAZON_NEPTUNE] + discovery_questions: [ + "What is the size of your Neo4j/ Neptune deployment?", + "Are you running into scaling/ performance issues with your existing graph database?", + "What graph query language are you using?", + "Are strong consistency reads needed?", + "Are you looking for a fully managed solution?", + "What are availability requirements?", + "Are you looking for FTS and vector search capabilities to augment your Graph use-case?", + "What is the source of your graph data?" + ] + recommendations { + selection_criteria: [ + "Need fully managed solution that doesn't require downtime for upgrades/maintenance", + "Want to use a graph query language (e.g. GQL; ISO standard) vs. a proprietary language like Cypher", + "Need a solution whose QPS scales linearly and is virtually unlimited.", + "Simpler solution that avoids managing pipelines from relational stores to build a graph", + "Application leverages advanced database capabilities (indexes, composite keys, typed values, constraints, JSON, views, ACID transactions, SQL)", + "Need strong and/or eventual consistency options for reads", + "Need replication with strong consistency guarantees", + "Need a graph solution without expensive licensing with a pay-as-you-go model." + ] + primary_destination: SPANNER_GRAPH + migration_complexity: [ + "Medium complexity migration. Some work required for schema conversion and data transfer. (WIP utility to migrate data from Neo4j to Spanner Graph). Query conversion is more straightforward.", + "Neo4j APOCs require work to be translated to GQL" + ] + best_when: "Spanner simplifies your data architecture by providing a high-performance, fully managed graph database that operates directly on your existing relational data. This zero-ETL approach eliminates the need to manage separate graph systems, which reduces cost, prevents data duplication, and allows you to run real-time graph analytics on live transactional data." + benefits: [ + "Leverage a unified, zero-ETL architecture by running graph queries directly on your existing relational data, which eliminates the need for separate databases, complex ETL pipelines, and data duplication.", + "Achieve massive scale and performance by efficiently querying graphs with hundreds of billions of nodes, with optimizations for high-speed traversal even on highly connected data.", + "Execute seamless multi-model queries by combining graph analytics with relational (SQL), full-text search, and vector search on a single, unified platform.", + "Built on an enterprise-grade foundation that includes all of Spanner’s core benefits, such as its industry-leading 99.999% availability, zero-downtime operations, and deep integration with GCP services like IAM and BigQuery." + ] + } +} + diff --git a/skills/gcp-managed-spark-upgrades/SKILL.md b/skills/gcp-managed-spark-upgrades/SKILL.md index 949540da..10cf552e 100644 --- a/skills/gcp-managed-spark-upgrades/SKILL.md +++ b/skills/gcp-managed-spark-upgrades/SKILL.md @@ -13,7 +13,7 @@ description: | license: Apache-2.0 metadata: - version: v1 + version: v2 publisher: google --- @@ -34,7 +34,7 @@ Your goal is to upgrade Spark jobs from one version to another while maintaining ## Task Execution Workflow > [!IMPORTANT] -> All GCP operations require active authentication. If you encounter authentication or permission errors, refer to `@skill:google-cloud-auth-verification` to resolve them. +> All GCP operations require active authentication. If you encounter authentication errors, ask the user to run `gcloud auth login` and `gcloud auth application-default login`. If an account is active but a call returns `403`, diagnose missing IAM roles instead. > [!NOTE] > All helper scripts in this skill maintain dependencies inline following the [PEP 723](https://peps.python.org/pep-0723/) specification. Execute all Python helper scripts using [uv](https://docs.astral.sh/uv/) (e.g., `uv run scripts/.py --help`), which automatically manages isolated environments and dependencies. Run scripts with `--help` to discover their interface rather than reading the source. diff --git a/skills/gcp-spark-troubleshooting/SKILL.md b/skills/gcp-spark-troubleshooting/SKILL.md index 81de48ff..b4fec24b 100644 --- a/skills/gcp-spark-troubleshooting/SKILL.md +++ b/skills/gcp-spark-troubleshooting/SKILL.md @@ -3,7 +3,7 @@ name: gcp-spark-troubleshooting description: "Provides expert guidance for troubleshooting Google Cloud Spark and Dataproc workloads (Dataproc Serverless batches and standard Dataproc clusters), and inspecting, streaming, searching, tailing, or summarizing Spark driver outputs and event logs in Cloud Storage. Use when the user asks to debug, troubleshoot, diagnose, or perform Root Cause Analysis (RCA) on failed Spark jobs, PySpark batches, or Spark event logs." license: Apache-2.0 metadata: - version: v3 + version: v4 publisher: google --- @@ -73,8 +73,7 @@ rather than reading its source. The scripts only gather evidence; none of them diagnoses or writes the RCA. Use only actions and flags that this file or `--help` lists; never invent one. -- **Required**: Python 3.8+ and an authenticated `gcloud` CLI (see - `@skill:google-cloud-auth-verification`). +- **Required**: Python 3.8+ and an authenticated `gcloud` CLI. - `spark_applications.py` and `spark_stage_diagnostics.py` call the Dataproc REST API with a token from `gcloud auth print-access-token`. If that token is rejected (HTTP 401, for example @@ -241,11 +240,12 @@ diagnostic plan or run any script before it passes.** 0. **Tooling & Environment Verification**: - - **Verify the `gcloud` CLI is present and authenticated.** Use - `@skill:google-cloud-auth-verification`. Pass condition: an active - account is returned. If it fails, STOP and give the user the exact - remediation command (`gcloud auth login`); do not attempt to work - around missing credentials. + - **Verify the `gcloud` CLI is present and authenticated** via `gcloud + auth list`. Pass condition: an active account is returned. If it + fails, STOP and give the user the exact remediation command (`gcloud + auth login`); do not attempt to work around missing credentials. If an + account is active but a call returns `403`, diagnose missing IAM roles + instead of asking the user to log in again. - **Verify a default project is resolvable** via `gcloud config get-value project`, unless the user supplied `` explicitly. Pass condition: a non-empty value that is not `(unset)`. @@ -267,9 +267,8 @@ diagnostic plan or run any script before it passes.** - Determine if the workload is a **Dataproc Serverless Batch**, a **Dataproc Cluster Job**, or a **Direct GCS URI**. - - If project or region is not specified, use - `@skill:google-cloud-auth-verification` or read the active gcloud - configuration. + - If project or region is not specified, read the active gcloud + configuration (`gcloud config list`). - Fetch metadata: - For Serverless Batches: run `gcloud dataproc batches describe --region= --format="json"`. diff --git a/skills/gcp-spark/SKILL.md b/skills/gcp-spark/SKILL.md index 44543776..2b1a3f4f 100755 --- a/skills/gcp-spark/SKILL.md +++ b/skills/gcp-spark/SKILL.md @@ -16,7 +16,7 @@ description: | - Troubleshooting failed Spark workloads (use @skill:gcp-spark-troubleshooting). license: Apache-2.0 metadata: - version: v24 + version: v26 publisher: google --- @@ -82,6 +82,13 @@ metadata: table does not exist, use `@skill:discovering-gcp-data-assets` to discover all similar tables in the namespace or project. + *MISSING SOURCE RULE*: If the request does not name a table, GCS path or + catalog, you MUST run at most ONE `@skill:discovering-gcp-data-assets` + search. If it does not return a single clear match, you MUST list the + candidates, ask the user which source to use, and STOP. You MUST NOT infer + the source by listing buckets, reading existing notebooks or scripts, or + running `SHOW NAMESPACES` / `SHOW TABLES`. + *MINOR TYPO RULE*: If there is a minor typo (e.g. `employees` vs `employee`), you can fix the error and proceed. @@ -153,12 +160,18 @@ metadata: ) ``` * **Security & Dynamic Variables**: You MUST NOT hardcode plaintext - credentials, passwords, or tokens (use Secret Manager). You MUST NOT - hardcode environment-specific variables; parameterize project IDs, - dataset IDs, table names, and bucket paths via `os.environ.get(...)` or - arguments. - * **Read and Write data**: **ALWAYS** Refer to - `references/read_write_data.md` when reading or writing data. + credentials, passwords, or tokens (use Secret Manager — refer to + `references/jdbc_cloudsql.md` when retrieving credentials from Secret + Manager). You MUST NOT hardcode environment-specific variables; + parameterize project IDs, dataset IDs, table names, and bucket paths via + `os.environ.get(...)` or arguments. + * **Read and Write data**: **ALWAYS** load only the connector reference + file(s) needed for the prompt: + - **BigQuery, GCS, or Pub/Sub**: `references/read_write_data.md` + - **BigLake Iceberg Catalog**: `references/biglake_iceberg.md` + - **Cloud Spanner**: `references/spanner.md` + - **JDBC & Cloud SQL (PostgreSQL / MySQL) & Secret Manager**: + `references/jdbc_cloudsql.md` * **Machine Learning Tasks**: Refer to `@skill:ml-best-practices` skill and `references/ml_tasks.md` when generating Machine Learning code. * **Spark Optimizations & Broadcast Joins**: **ALWAYS** refer to `references/spark_optimizations.md`. When joining a large DataFrame with @@ -232,6 +245,7 @@ The Managed Spark (Dataproc) service account needs: * `roles/bigquery.readSessionUser`: BigQuery Storage Read API session creation * `roles/storage.objectUser`: Read/write GCS (data, Iceberg warehouse, and `--deps-bucket` / `temporaryGcsBucket`) * `roles/spanner.databaseUser`: Spanner reads and writes +* `roles/secretmanager.secretAccessor`: Accessing database passwords and credentials stored in Secret Manager -------------------------------------------------------------------------------- diff --git a/skills/gcp-spark/references/biglake_iceberg.md b/skills/gcp-spark/references/biglake_iceberg.md new file mode 100644 index 00000000..3b0b91cb --- /dev/null +++ b/skills/gcp-spark/references/biglake_iceberg.md @@ -0,0 +1,325 @@ +# BigLake Iceberg Catalog in PySpark + +> [!IMPORTANT] +> You MUST ALWAYS follow the Task Execution Workflow in `SKILL.md` when writing Spark code. + +## Spark Session Configuration + +- [!WARNING] BigLake Iceberg catalog must be configured in the spark session + before tables can be read. +- Multiple Iceberg catalogs can be configured in the same spark session. +- [!IMPORTANT] You **MUST** identify the storage layer for the BigLake Iceberg + table before configuring the spark session. Use + `@skill:discovering-gcp-data-assets` to **search** and **lookup** the table + details. Check the value for the `metadataPath` field from the lookup + results to confirm it uses GCS storage (starts with `gs:`). +- [!WARNING] **DO NOT** assume catalog_name, project_id, io implementation for + a table, **ALWAYS lookup** table details before proceeding. + +### Option A: Zero-Config (Preferred) — Session Template / Runtime Profile + +When generating PySpark **notebook** code, assume the Dataproc kernel already +has the Iceberg catalog configured via its Session Template. **MUST NOT add +in-code catalog configuration.** Always use the simple `getOrCreate()` pattern: + +```python +# Catalog and BigLake settings are pre-configured by the Dataproc Session Template. +# No .config() calls are needed. +from google.cloud.managed_spark_connect import ManagedSparkSession +from google.cloud.dataproc_v1 import Session + +# 1. Initialize the Dataproc Session configuration object +session = Session() +session.session_template = "projects//locations//sessionTemplates/" +spark = ( + ManagedSparkSession.builder + .projectId("") + .location("") + .dataprocSessionConfig(session) + .getOrCreate() +) + +# Read directly from the catalog already registered by the kernel's session template +df = spark.read.format("iceberg").load("``..") +``` + +> [!IMPORTANT] When generating notebook code, **always use the pattern shown +> above**. Adding explicit `.config()` calls for catalogs that are already in +> the Session Template is redundant and can cause duplicate extension +> registration. + +#### Locating the Session Template and Catalog Name + +Use this high-level routing algorithm to determine `` and +``: + +- **If `` is already known** (passed in prompt or present in + existing code): go to **Case 1: Known Session Template**. +- **If `` is unknown** (only table name, Dataplex entry, or + warehouse is known): go to **Case 2: Unknown Session Template**. +- **If no template matches or input is incomplete**: go to **Case 3: No + Matching Template**. + +##### Case 1: Known Session Template + +Use the specified ``. If `` is not already known from +the code, inspect the template's properties: + +```bash +gcloud beta dataproc session-templates describe \ + --project= --location= \ + --format="json(runtimeConfig.properties)" +``` + +- If `spark.sql.defaultCatalog` is set, `` can be omitted + (`.`). +- If only one `spark.sql.catalog.` is configured in the + template, use that `` + (`..`). +- If multiple catalogs are configured, use the `` whose + `spark.sql.catalog..warehouse` matches the table's warehouse. + +##### Case 2: Unknown Session Template + +List templates in the project filtered by the table's warehouse URI (e.g. +`gs://` from Dataplex `metadataPath`) to get both `` and +``: + +```bash +gcloud beta dataproc session-templates list \ + --project= --location= \ + --filter="runtimeConfig.properties:" \ + --limit=5 \ + --format="json(name.basename(), runtimeConfig.properties)" +``` + +- If multiple templates match, prefer the template where `defaultCatalog` + matches the catalog name. + +##### Case 3: No Matching Template + +If no session template matches the warehouse, or if required inputs are missing: + +- **STOP and ask the user** which Session Template to use. +- Do **NOT** guess `` or add in-code `.config()` calls in + notebooks. + +### Option B: Explicit In-Code Configuration + +Use this only when no Session Template / Runtime Profile is available (e.g. +cluster and batch jobs, or environments without a pre-configured profile). You +**MUST NOT** use this in notebook code - ask user to configure and provide the +session template instead. Multiple Iceberg catalogs can be configured in the +same spark session. + +> [!IMPORTANT] +> - **Runtime JARs**: +> - **Dataproc Serverless**: Iceberg and `iceberg-gcp` (`GCSFileIO`) JARs are pre-installed on `/usr/lib/spark/jars/` by default. Do **NOT** pass `spark.jars.packages=org.apache.iceberg:...` when submitting serverless batches. +> - **Dataproc on GCE Clusters**: Enable `--optional-components=ICEBERG` at cluster creation (which adds `/usr/lib/iceberg/lib/*` to `SPARK_DIST_CLASSPATH`), or pass `/usr/lib/iceberg/lib/*` via `--jars` if submitting to a cluster without the optional component. +> - **Warehouse URI & Catalog Name**: For a GCS-backed BigLake catalog, the catalog name `` must match the GCS bucket name, and `spark.sql.catalog..warehouse` **MUST** be the root GCS bucket URI (`gs://`) without any subfolder path. Passing a subfolder (e.g. `gs:///warehouse`) causes `BadRequestException: Bucket name is invalid`. +> - **Catalog Provisioning & IAM**: Before creating namespaces or writing tables, ensure the BigLake Iceberg catalog exists (`gcloud biglake iceberg catalogs create --catalog-type=gcs-bucket --project=`) and the execution service account has `roles/biglake.admin` and `roles/storage.objectUser` on the catalog bucket. + +Example with GCS storage: + +```python +spark = ( + SparkSession.builder + .appName("") + .config( + "spark.sql.extensions", + "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", + ) + .config( + "spark.sql.catalog.", + "org.apache.iceberg.spark.SparkCatalog", + ) + .config("spark.sql.catalog..type", "rest") + .config( + "spark.sql.catalog..uri", + "https://biglake.googleapis.com/iceberg/v1/restcatalog", + ) + .config( + "spark.sql.catalog..warehouse", + "gs://", + ) + .config( + "spark.sql.catalog..header.x-goog-user-project", + "", + ) + .config( + "spark.sql.catalog..rest.auth.type", + "org.apache.iceberg.gcp.auth.GoogleAuthManager", + ) + .config( + "spark.sql.catalog..io-impl", + "org.apache.iceberg.gcp.gcs.GCSFileIO", + ) + .getOrCreate() +) +``` + +## Reading from BigLake Iceberg Catalog + +- [!WARNING] BigLake Iceberg catalog must be configured before tables can be + read (via Session Template or explicit configuration). +- If not using a fully qualified 3-part table identifier (or if + `spark.sql.defaultCatalog` is not set), set the current Spark catalog using + `spark.catalog.setCurrentCatalog("")`. **DO NOT** use + backticks in this call. +- [!IMPORTANT] You **MUST ALWAYS** surround the catalog name with backticks in + SQL statements and `load()` calls to ensure proper handling of special + characters (e.g., `` `catalog-name` ``). + +```python +df = ( + spark.read.format("iceberg") + .load("``..") +) +``` + +## Writing to BigLake Iceberg Catalog + +> [!IMPORTANT] Follow these instructions before writing code to write to BigLake +> Iceberg tables. + +- [!WARNING] BigLake Iceberg catalog must be configured before tables can be + written (via Session Template or explicit configuration). +- **ALWAYS** use Spark's `DataFrameWriterV2` API (`writeTo`) for writing to + Iceberg tables. This provides a richer API with specific write modes. +- If not using a fully qualified 3-part table identifier (or if + `spark.sql.defaultCatalog` is not set), set the current Spark catalog using + `spark.catalog.setCurrentCatalog("")`. **DO NOT** use + backticks in this call. +- **ALWAYS** Ensure namespace is created before writing, using: + + ```python + spark.sql( + "CREATE NAMESPACE IF NOT EXISTS ``." + ) + ``` + > [!IMPORTANT] You **MUST ALWAYS** surround the catalog name with backticks + > in SQL statements and `writeTo()` calls to ensure proper handling of + > special characters. + +```python +# The basic pattern +spark.catalog.setCurrentCatalog("") +( + df.writeTo("``..") + .using("iceberg") + .createOrReplace() +) +``` + +### Write Modes + +- **DataFrameWriterV2 `createOrReplace()`**: Creates the table if it does not + exist, or replaces it if it does. + + ```python + ( + df.writeTo("``..") + .using("iceberg") + .createOrReplace() + ) + ``` + +- **DataFrameWriterV2 `append()`**: Equivalent to `INSERT INTO`. Appends data + to an existing table. + + ```python + ( + df.writeTo("``..") + .using("iceberg") + .append() + ) + ``` + +- **SPARK SQL MERGE INTO**: Performs row-level updates, inserts, and deletes + by rewriting data files that contain affected rows. The source table can be + persisted as a temp view to allow using it in the MERGE INTO statement. The + source view and the target table must be joined on the primary key(s) to + ensure correct row-level updates, inserts, and deletes. + + ```python + df.createOrReplaceTempView("source_updates") + + spark.sql(f""" + MERGE INTO ``.. t + USING source_updates s + ON t.id = s.id + WHEN MATCHED THEN UPDATE SET * + WHEN NOT MATCHED THEN INSERT * + """) + ``` + +- [!WARNING] These are less preferred write modes, use them ONLY if explicitly + asked to: + * **`create()`**: Equivalent to `CREATE TABLE AS SELECT`. Fails if the + table already exists. + + ```python + df.writeTo("``..") \ + .using("iceberg") \ + .create() + ``` + + * **`replace()`**: Equivalent to `REPLACE TABLE AS SELECT`. + + ```python + df.writeTo("``..") \ + .using("iceberg") \ + .replace() + ``` + + * **`overwritePartitions()`**: Equivalent to dynamic `INSERT OVERWRITE`. + Overwrites partitions in the table that match the data in the DataFrame. + + ```python + df.writeTo("``..") \ + .using("iceberg") \ + .overwritePartitions() + ``` + + * **`overwrite(condition)`**: Overwrites data that matches a specific + condition (e.g. static overwrite). + + ```python + from pyspark.sql.functions import col + df.writeTo("``..") \ + .using("iceberg") \ + .overwrite(col("level") == "INFO") + ``` + +### Advanced Iceberg Writes + +#### Writing with partitions +Use partitioning transform functions if writing to partitioned tables using +timestamp fields. Ex: years, months, days, hours etc. + +```python +from pyspark.sql.functions import days + +df.writeTo("``..") \ + .using("iceberg") \ + .partitionedBy("level", days("ts")) \ + .createOrReplace() +``` + +#### Schema Evolution +Iceberg supports safe schema evolution. You can merge schema changes during an +append. + +```python +df.writeTo("``..") \ + .option("mergeSchema", "true") \ + .append() +``` + +#### Writing to a Specific Branch + +```python +# To insert into an `audit` branch +df.writeTo("``...branch_audit") \ + .append() +``` diff --git a/skills/gcp-spark/references/gcloud_dataproc.md b/skills/gcp-spark/references/gcloud_dataproc.md index a64b6aec..565675c6 100755 --- a/skills/gcp-spark/references/gcloud_dataproc.md +++ b/skills/gcp-spark/references/gcloud_dataproc.md @@ -215,10 +215,14 @@ For asynchronous execution, you must poll the batch status until state is - **Dependency**: `--jars=gs://spark-lib/spanner/spark-3.5-spanner-1.4.0.jar` - **Notes**: Pass `.option("projectId", ...)` in PySpark -#### PostgreSQL +#### PostgreSQL & JDBC Databases -- **Dependency**: `spark.jars.packages=org.postgresql:postgresql:42.6.0` -- **Notes**: Pass `--subnet=...` for private IP +- **Dependency**: + - **GCS-Staged Driver JAR (`gs://...`)**: Pass `--jars=gs:///.jar` on the `gcloud` CLI or set `spark.jars` (for `ManagedSparkSession`, set `session_config.runtime_config.properties["spark.jars"] = "gs:///.jar"`). Never pass `gs://` URIs to `spark.jars.packages`. + - **Maven Coordinates (requires outbound internet/Cloud NAT)**: `--properties="spark.jars.packages=::"` +- **Notes**: + - Pass `--subnet=` when connecting to a database on a VPC Private IP. + - Retrieve database passwords from Google Cloud Secret Manager (either via the pre-installed `google.auth` + `urllib.request` REST API or by passing `--properties="spark.dataproc.pip.packages=google-cloud-secret-manager=="` on Dataproc Serverless, where `google-cloud-secret-manager` is not pre-installed). #### Iceberg REST diff --git a/skills/gcp-spark/references/jdbc_cloudsql.md b/skills/gcp-spark/references/jdbc_cloudsql.md new file mode 100644 index 00000000..2e69eeff --- /dev/null +++ b/skills/gcp-spark/references/jdbc_cloudsql.md @@ -0,0 +1,172 @@ +# JDBC Databases & Cloud SQL in PySpark + +> [!IMPORTANT] +> You MUST ALWAYS follow the Task Execution Workflow in `SKILL.md` when writing Spark code. + +## 1. Attaching a Custom JDBC Driver JAR (`gs://...`) + +JDBC drivers are not pre-installed on Dataproc and must be attached at session +creation (`gs://` JAR URIs cannot be passed to `spark.jars.packages`, which only +accepts Maven coordinates): + +- **Interactive Notebooks (`.ipynb`) & Spark Connect**: Attach the driver JAR + via a Session Template or `session_config.runtime_config.properties["spark.jars"]` + at session creation (setting `spark.jars` after a Spark Connect session has + started does not register the JDBC driver with the JVM): + + ```python + from google.cloud.dataproc_v1 import Session + from google.cloud.managed_spark_connect import ManagedSparkSession + + session_config = Session() + session_config.runtime_config.properties["spark.jars"] = ( + "gs:///jars/.jar" + ) + spark = ( + ManagedSparkSession.builder.projectId("") + .location("") + .dataprocSessionConfig(session_config) + .getOrCreate() + ) + ``` + +- **Dataproc Batch & Cluster Jobs (`.py`)**: Pass + `--jars=gs:///jars/.jar` (and + `--subnet=` for private IP databases) on the `gcloud` CLI, and set + `spark.jars` on `SparkSession.builder`: + + ```python + jar_uri = os.environ.get( + "JAR_URI", "gs:///jars/.jar" + ) + spark = ( + SparkSession.builder.appName("") + .config("spark.jars", jar_uri) + .getOrCreate() + ) + ``` + +### Common JDBC Driver Classes & URL Formats + +| Database Engine | Driver Class (`driver`) | JDBC URL Format (`url`) | +| :--- | :--- | :--- | +| **PostgreSQL / AlloyDB** | `org.postgresql.Driver` | `jdbc:postgresql://:5432/` | +| **MySQL** | `com.mysql.cj.jdbc.Driver` | `jdbc:mysql://:3306/` | +| **SQL Server** | `com.microsoft.sqlserver.jdbc.SQLServerDriver` | `jdbc:sqlserver://:1433;databaseName=;encrypt=true;trustServerCertificate=true` | + +## 2. Retrieving Database Passwords from Secret Manager + +> [!WARNING] NEVER embed plaintext secrets or passwords directly in code or +> scripts, and do not rely on unset local environment variables in remote +> Dataproc jobs. Always retrieve database passwords from Google Cloud Secret +> Manager. + +On Dataproc Serverless, `google-cloud-secret-manager` is not pre-installed +by default. Either pass +`--properties="spark.dataproc.pip.packages=google-cloud-secret-manager=="` +at submission time and use `SecretManagerServiceClient`, or fetch the secret via +the pre-installed `google.auth` and `urllib.request` libraries: + +```python +import base64 +import json +import urllib.request +import google.auth +from google.auth.transport.requests import Request + + +def get_secret(project_id: str, secret_id: str, version: str = "latest") -> str: + creds, _ = google.auth.default( + scopes=["https://www.googleapis.com/auth/cloud-platform"] + ) + creds.refresh(Request()) + url = ( + f"https://secretmanager.googleapis.com/v1/projects/{project_id}" + f"/secrets/{secret_id}/versions/{version}:access" + ) + req = urllib.request.Request( + url, headers={"Authorization": f"Bearer {creds.token}"} + ) + with urllib.request.urlopen(req) as resp: + data = json.loads(resp.read().decode("utf-8"))["payload"]["data"] + return base64.b64decode(data).decode("utf-8").strip() +``` + +If `google-cloud-secret-manager` is installed in the environment: + +```python +from google.cloud import secretmanager + + +def get_secret(project_id: str, secret_id: str, version: str = "latest") -> str: + client = secretmanager.SecretManagerServiceClient() + name = f"projects/{project_id}/secrets/{secret_id}/versions/{version}" + response = client.access_secret_version(request={"name": name}) + return response.payload.data.decode("utf-8").strip() +``` + +## 3. Parallel Partitioned JDBC Reads + +Calling `spark.read.format("jdbc").load()` without partitioning options reads +the entire table on a single executor thread. For parallel JDBC reads on a +numeric, date, or timestamp column (``): + +1. **Dynamically query `MIN` and `MAX` bounds first** (never hardcode or guess + `lowerBound` / `upperBound`). +2. **Read the table in parallel** using `partitionColumn`, `lowerBound`, + `upperBound`, `numPartitions`, and `fetchsize`: + +```python +import logging +import os + +logging.basicConfig( + level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s" +) + +project_id = os.environ.get("PROJECT_ID", "") +secret_id = os.environ.get("SECRET_ID", "") +jdbc_url = os.environ.get( + "JDBC_URL", "jdbc:://:/" +) +jdbc_driver = os.environ.get("JDBC_DRIVER", "") +db_user = os.environ.get("DB_USER", "") +table_name = os.environ.get("TABLE_NAME", "") +partition_col = os.environ.get("PARTITION_COL", "") +db_password = get_secret(project_id, secret_id) + +# Step 1: Dynamically fetch MIN and MAX bounds for the partition column +bounds_query = ( + f"(SELECT MIN({partition_col}) AS min_id, MAX({partition_col}) AS max_id " + f"FROM {table_name}) AS bounds_q" +) +bounds_row = ( + spark.read.format("jdbc") + .option("url", jdbc_url) + .option("dbtable", bounds_query) + .option("user", db_user) + .option("password", db_password) + .option("driver", jdbc_driver) + .load() + .first() +) +lower_bound = str(bounds_row["min_id"]) +upper_bound = str(bounds_row["max_id"]) +logging.info("Resolved bounds: lower=%s, upper=%s", lower_bound, upper_bound) + +# Step 2: Parallel partitioned read across executors +df = ( + spark.read.format("jdbc") + .option("url", jdbc_url) + .option("dbtable", table_name) + .option("user", db_user) + .option("password", db_password) + .option("driver", jdbc_driver) + .option("partitionColumn", partition_col) + .option("lowerBound", lower_bound) + .option("upperBound", upper_bound) + .option("numPartitions", "4") + .option("fetchsize", "10000") + .load() +) +``` diff --git a/skills/gcp-spark/references/read_write_data.md b/skills/gcp-spark/references/read_write_data.md index 7add390d..3ce0fca7 100755 --- a/skills/gcp-spark/references/read_write_data.md +++ b/skills/gcp-spark/references/read_write_data.md @@ -1,4 +1,4 @@ -# Definitive Guide to reading and writing tables using spark code +# Definitive Guide to Reading and Writing Data in PySpark > [!IMPORTANT] > You MUST ALWAYS follow the Task Execution Workflow when writing spark code. @@ -6,8 +6,22 @@ ## Task Execution Workflow 1. Configure the spark session to read or write data from the required source - or destination -2. Read or write the data from the required source or destination + or destination. +2. Read or write the data from the required source or destination. + +## Connector Reference Routing + +Load only the reference file(s) needed for your target data source or +destination: + +- **BigQuery, Google Cloud Storage (GCS), & Pub/Sub**: Covered below in this + file (`references/read_write_data.md`). +- **BigLake Iceberg Catalog**: Refer to `references/biglake_iceberg.md`. +- **Cloud Spanner**: Refer to `references/spanner.md`. +- **JDBC & Cloud SQL (PostgreSQL / MySQL) & Secret Manager Credentials**: + Refer to `references/jdbc_cloudsql.md`. + +-------------------------------------------------------------------------------- ## BigQuery @@ -24,7 +38,8 @@ via `spark.jars.packages` when running on Dataproc. `SparkSession.builder` (see `SKILL.md`). ### Reading from BigQuery -#### Basic example to read a Big Query table + +#### Basic example to read a BigQuery table ```python df = spark.read.format("bigquery") \ @@ -33,8 +48,10 @@ df = spark.read.format("bigquery") \ ``` #### Executing a BigQuery SQL query + The Spark BigQuery connector allows you to run Standard SQL SELECT queries directly on BigQuery and load the results into a spark dataframe. + > [!IMPORTANT] You must set `viewsEnabled` to `"true"` and specify a writable > `materializationDataset` in your project to run direct SQL queries or read > BigQuery views. @@ -57,6 +74,7 @@ df = ( ``` #### BigQuery Join Patterns + When enriching data with BigQuery reference tables: ```python @@ -75,7 +93,7 @@ enriched_df = main_df.join( #### Optimization Tips -1. Select only needed columns from BigQuery +1. Select only needed columns from BigQuery: ```python ref_df = spark.read.format("bigquery") \ @@ -85,7 +103,7 @@ ref_df = spark.read.format("bigquery") \ .select("id", "name") ``` -2. Using broadcast joins for small reference tables +2. Using broadcast joins for small reference tables: ```python from pyspark.sql.functions import broadcast @@ -101,10 +119,11 @@ enriched_df = main_df.join( ### Writing to BigQuery #### Non partitioned data -Write DataFrame to BigQuery using direct writes, - - Set writeMethod to direct - - Set mode to one of `overwrite`, `append`, `errorifexists` based on task +Write DataFrame to BigQuery using direct writes: + +- Set `writeMethod` to `direct` +- Set `mode` to one of `overwrite`, `append`, `errorifexists` based on task ```python df.write \ @@ -114,355 +133,35 @@ df.write \ .mode("overwrite") \ .save() ``` + #### Partitioned data + Write DataFrame to a partitioned BigQuery table using indirect writes. In this method, data is written to GCS first and then loaded into BigQuery. -A GCS bucket is required as the temporary data location. +A GCS bucket is required as the temporary data location: - - Set temporaryGcsBucket to a gcs bucket name - - Set a check point path location +- Set `temporaryGcsBucket` to a GCS bucket name +- Set a `checkpointLocation` path ```python df.write \ .format("bigquery") \ - .option("temporaryGcsBucket","") \ + .option("temporaryGcsBucket", "") \ .option("checkpointLocation", "") \ .save("..") ``` -## BigLake Iceberg Catalog - -### Spark Session Configuration - -- [!WARNING] BigLake Iceberg catalog must be configured in the spark session - before tables can be read. -- Multiple Iceberg catalogs can be configured in the same spark session -- [!IMPORTANT] You **MUST** identify the storage layer for the BigLake Iceberg - table before configuring the spark session. Use - `@skill:discovering-gcp-data-assets` to **search** and **lookup** the table - details. Check the value for the `metadataPath` field from the lookup - results to confirm it uses GCS storage (starts with `gs:`). -- [!WARNING] **DO NOT** assume catalog_name, project_id, io implementation for a - table, **ALWAYS lookup** table details before proceeding. - -#### Option A: Zero-Config (Preferred) — Session Template / Runtime Profile - -When generating PySpark **notebook** code, assume the Dataproc kernel already -has the Iceberg catalog configured via its Session Template. **MUST NOT add -in-code catalog configuration.** Always use the simple `getOrCreate()` pattern: - -```python -# Catalog and BigLake settings are pre-configured by the Dataproc Session Template. -# No .config() calls are needed. -from google.cloud.managed_spark_connect import ManagedSparkSession -from google.cloud.dataproc_v1 import Session - -# 1. Initialize the Dataproc Session configuration object -session = Session() -session.session_template = "projects//locations//sessionTemplates/" -spark = ( - ManagedSparkSession.builder - .projectId("") - .location("") - .dataprocSessionConfig(session) - .getOrCreate() -) - -# Read directly from the catalog already registered by the kernel's session template -df = spark.read.format("iceberg").load("``..") -``` - -> [!IMPORTANT] When generating notebook code, **always use the pattern shown -> above**. Adding explicit `.config()` calls for catalogs that are already in -> the Session Template is redundant and can cause duplicate extension -> registration. - -##### Locating the Session Template and Catalog Name - -Use this high-level routing algorithm to determine `` and -``: - -- **If `` is already known** (passed in prompt or present in - existing code): go to **Case 1: Known Session Template**. -- **If `` is unknown** (only table name, Dataplex entry, or - warehouse is known): go to **Case 2: Unknown Session Template**. -- **If no template matches or input is incomplete**: go to **Case 3: No - Matching Template**. - -###### Case 1: Known Session Template - -Use the specified ``. If `` is not already known from -the code, inspect the template's properties: - -```bash -gcloud beta dataproc session-templates describe \ - --project= --location= \ - --format="json(runtimeConfig.properties)" -``` - -- If `spark.sql.defaultCatalog` is set, `` can be omitted - (`.`). -- If only one `spark.sql.catalog.` is configured in the - template, use that `` - (`..`). -- If multiple catalogs are configured, use the `` whose - `spark.sql.catalog..warehouse` matches the table's warehouse. - -###### Case 2: Unknown Session Template - -List templates in the project filtered by the table's warehouse URI (e.g. -`gs://` from Dataplex `metadataPath`) to get both `` and -``: - -```bash -gcloud beta dataproc session-templates list \ - --project= --location= \ - --filter="runtimeConfig.properties:" \ - --limit=5 \ - --format="json(name.basename(), runtimeConfig.properties)" -``` - -- If multiple templates match, prefer the template where `defaultCatalog` - matches the catalog name. - -###### Case 3: No Matching Template - -If no session template matches the warehouse, or if required inputs are missing: - -- **STOP and ask the user** which Session Template to use. -- Do **NOT** guess `` or add in-code `.config()` calls in - notebooks. - -#### Option B: Explicit In-Code Configuration - -Use this only when no Session Template / Runtime Profile is available (e.g. -cluster and batch jobs, or environments without a pre-configured profile). You -**MUST NOT** use this in notebook code - ask user to configure and provide the -session template instead. Multiple Iceberg catalogs can be configured in the -same spark session. - -> [!IMPORTANT] -> - **Runtime JARs**: -> - **Dataproc Serverless**: Iceberg and `iceberg-gcp` (`GCSFileIO`) JARs are pre-installed on `/usr/lib/spark/jars/` by default. Do **NOT** pass `spark.jars.packages=org.apache.iceberg:...` when submitting serverless batches. -> - **Dataproc on GCE Clusters**: Enable `--optional-components=ICEBERG` at cluster creation (which adds `/usr/lib/iceberg/lib/*` to `SPARK_DIST_CLASSPATH`), or pass `/usr/lib/iceberg/lib/*` via `--jars` if submitting to a cluster without the optional component. -> - **Warehouse URI & Catalog Name**: For a GCS-backed BigLake catalog, the catalog name `` must match the GCS bucket name, and `spark.sql.catalog..warehouse` **MUST** be the root GCS bucket URI (`gs://`) without any subfolder path. Passing a subfolder (e.g. `gs:///warehouse`) causes `BadRequestException: Bucket name is invalid`. -> - **Catalog Provisioning & IAM**: Before creating namespaces or writing tables, ensure the BigLake Iceberg catalog exists (`gcloud biglake iceberg catalogs create --catalog-type=gcs-bucket --project=`) and the execution service account has `roles/biglake.admin` and `roles/storage.objectUser` on the catalog bucket. - -Example with GCS storage: - -```python -spark = ( - SparkSession.builder - .appName("") - .config( - "spark.sql.extensions", - "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", - ) - .config( - "spark.sql.catalog.", - "org.apache.iceberg.spark.SparkCatalog", - ) - .config("spark.sql.catalog..type", "rest") - .config( - "spark.sql.catalog..uri", - "https://biglake.googleapis.com/iceberg/v1/restcatalog", - ) - .config( - "spark.sql.catalog..warehouse", - "gs://", - ) - .config( - "spark.sql.catalog..header.x-goog-user-project", - "", - ) - .config( - "spark.sql.catalog..rest.auth.type", - "org.apache.iceberg.gcp.auth.GoogleAuthManager", - ) - .config( - "spark.sql.catalog..io-impl", - "org.apache.iceberg.gcp.gcs.GCSFileIO", - ) - .getOrCreate() -) -``` - -### Reading from BigLake Iceberg Catalog - -- [!WARNING] BigLake Iceberg catalog must be configured before tables can be - read (via Session Template or explicit configuration). -- If not using a fully qualified 3-part table identifier (or if - `spark.sql.defaultCatalog` is not set), set the current Spark catalog using - `spark.catalog.setCurrentCatalog("")`. **DO NOT** use - backticks in this call. -- [!IMPORTANT] You **MUST ALWAYS** surround the catalog name with backticks in - SQL statements and `load()` calls to ensure proper handling of special - characters (e.g., `` `catalog-name` ``). - -```python -df = ( - spark.read.format("iceberg") - .load("``..") -) -``` - -### Writing to BigLake Iceberg Catalog - -> [!IMPORTANT] Follow these instructions before writing code to write to BigLake -> Iceberg tables. - -- [!WARNING] BigLake Iceberg catalog must be configured before tables can be - written (via Session Template or explicit configuration). -- **ALWAYS** use Spark's `DataFrameWriterV2` API (`writeTo`) for writing to - Iceberg tables. This provides a richer API with specific write modes. -- If not using a fully qualified 3-part table identifier (or if - `spark.sql.defaultCatalog` is not set), set the current Spark catalog using - `spark.catalog.setCurrentCatalog("")`. **DO NOT** use - backticks in this call. -- **ALWAYS** Ensure namespace is created before writing, using: - - ```python - spark.sql( - "CREATE NAMESPACE IF NOT EXISTS ``." - ) - ``` - > [!IMPORTANT] You **MUST ALWAYS** surround the catalog name with backticks - > in SQL statements and `writeTo()` calls to ensure proper handling of - > special characters. - -```python -# The basic pattern -spark.catalog.setCurrentCatalog("") -( - df.writeTo("``..") - .using("iceberg") - .createOrReplace() -) -``` - -#### Write Modes - -- **DataFrameWriterV2 `createOrReplace()`**: Creates the table if it does not - exist, or replaces it if it does. - - ```python - ( - df.writeTo("``..") - .using("iceberg") - .createOrReplace() - ) - ``` - -- **DataFrameWriterV2 `append()`**: Equivalent to `INSERT INTO`. Appends data to - an existing table. - - ```python - ( - df.writeTo("``..") - .using("iceberg") - .append() - ) - ``` - -- **SPARK SQL MERGE INTO**: Performs row-level updates, inserts, and deletes by - rewriting data files that contain affected rows. The source table can be - persisted as a temp view to allow using it in the MERGE INTO statement. The - source view and the target table must be joined on the primary key(s) to - ensure correct row-level updates, inserts, and deletes. - - ```python - df.createOrReplaceTempView("source_updates") - - spark.sql(f""" - MERGE INTO ``.. t - USING source_updates s - ON t.id = s.id - WHEN MATCHED THEN UPDATE SET * - WHEN NOT MATCHED THEN INSERT * - """) - ``` - -- [!WARNING] These are less preferred write modes, use them ONLY if explicitly - asked to: -* **`create()`**: Equivalent to `CREATE TABLE AS SELECT`. Fails if the table - already exists. - - ```python - df.writeTo("``..") \ - .using("iceberg") \ - .create() - ``` - -* **`replace()`**: Equivalent to `REPLACE TABLE AS SELECT`. - - ```python - df.writeTo("``..") \ - .using("iceberg") \ - .replace() - ``` - -* **`overwritePartitions()`**: Equivalent to dynamic `INSERT OVERWRITE`. - Overwrites partitions in the table that match the data in the DataFrame. - - ```python - df.writeTo("``..") \ - .using("iceberg") \ - .overwritePartitions() - ``` - -* **`overwrite(condition)`**: Overwrites data that matches a specific - condition (e.g. static overwrite). - - ```python - from pyspark.sql.functions import col - df.writeTo("``..") \ - .using("iceberg") \ - .overwrite(col("level") == "INFO") - ``` - -#### Advanced Iceberg Writes - -##### Writing with partitions -Use partitioning transform functions if writing to partitioned tables using -timestamp fields. Ex: years, months, days, hours etc. -Example: - -```python -from pyspark.sql.functions import days - -df.writeTo("``..") \ - .using("iceberg") \ - .partitionedBy("level", days("ts")) \ - .createOrReplace() -``` - -##### Schema Evolution -Iceberg supports safe schema evolution. You can merge schema changes during an -append. - -```python -df.writeTo("``..") \ - .option("mergeSchema", "true") \ - .append() -``` - -##### Writing to a Specific Branch - -```python -# To insert into an `audit` branch -df.writeTo("``...branch_audit") \ - .append() -``` - ## Google Cloud Storage (GCS) -No session configuration is needed to read from GCS, all jars and + +No session configuration is needed to read from GCS; all JARs and configuration are provided by default in Dataproc. ### Reading from GCS #### CSV format -Read with `header` and `inferSchema` without these, the header row becomes data -and all columns are strings + +Read with `header` and `inferSchema`; without these, the header row becomes data +and all columns are strings: ```python df = spark.read.option("header", "true") \ @@ -471,137 +170,28 @@ df = spark.read.option("header", "true") \ ``` #### Parquet format + `df = spark.read.parquet("gs:///")` -#### Json format +#### JSON format + `df = spark.read.json("gs:///")` ### Writing to GCS -#### Csv format +#### CSV format + `df.write.csv("gs://bucket/output_path")` #### Model artifacts ```python # Warehouse already includes gs://, don't add it again -warehouse = spark.conf.get(f"spark.sql.catalog..warehouse") +warehouse = spark.conf.get("spark.sql.catalog..warehouse") model_path = f"{warehouse}/models/my_model" model.write().overwrite().save(model_path) ``` -## Spanner -### Spark Session Configuration -The Spanner connector JAR is not pre-installed and must be attached at session -creation (`gs://` JAR URIs cannot be passed to `spark.jars.packages`, which only -accepts Maven coordinates). Ensure the Spanner JAR's Spark version prefix -(e.g., `spark-3.5-spanner-*` for Dataproc runtime `2.3` / Spark 3.5, or -`spark-4.0-spanner-*` for Dataproc runtime `3.0` / Spark 4.0) matches the -Dataproc runtime version: - -- **Interactive Notebooks (`.ipynb`) & Spark Connect**: Attach the JAR via a - Session Template or `session_config.runtime_config.properties["spark.jars"]` - (and ensure `session_config.runtime_config.version` matches the JAR's Spark - version, as `ManagedSparkSession` defaults to runtime `3.0`): - - ```python - from google.cloud.dataproc_v1 import Session - from google.cloud.managed_spark_connect import ManagedSparkSession - - session_config = Session() - session_config.runtime_config.version = "2.3" - session_config.runtime_config.properties["spark.jars"] = ( - "gs://spark-lib/spanner/spark-3.5-spanner-1.4.0.jar" - ) - spark = ( - ManagedSparkSession.builder.projectId("") - .location("") - .dataprocSessionConfig(session_config) - .getOrCreate() - ) - ``` - -- **Dataproc Batch & Cluster Jobs (`.py`)**: Pass - `--jars=gs://spark-lib/spanner/spark-3.5-spanner-1.4.0.jar` on the `gcloud` - CLI (see `references/gcloud_dataproc.md`), or set `spark.jars`: - - ```python - spark = ( - SparkSession.builder.appName("") - .config( - "spark.jars", - "gs://spark-lib/spanner/spark-3.5-spanner-1.4.0.jar", - ) - .getOrCreate() - ) - ``` - -#### Reading Spanner tables - -```python -df = ( - spark.read.format("cloud-spanner") - .option("projectId", "") - .option("instanceId", "") - .option("databaseId", "") - .option("table", "") - .load() -) -``` - -### Writing to Spanner -The connector supports writing to Spanner tables using the DataFrame `write` -API. - -#### Write Modes - -* **`Append`**: Inserts rows into the existing Spanner table. This is the - default write mode. - - ```python - df.write.format("cloud-spanner") \ - .option("projectId", "") \ - .option("instanceId", "") \ - .option("databaseId", "") \ - .option("table", "") \ - .mode("append") \ - .save() - ``` - -* **`Overwrite`**: Clears the existing data before writing. - - ```python - df.write.format("cloud-spanner") \ - .option("projectId", "") \ - .option("instanceId", "") \ - .option("databaseId", "") \ - .option("table", "") \ - .mode("overwrite") \ - .save() - ``` - -## Cloud SQL (PostgreSQL / MySQL) - -### Reading from Cloud SQL via JDBC - -> [!WARNING] NEVER embed plaintext secrets or passwords directly in code or -> scripts. Always retrieve credentials securely via environment variables (or -> Secret Manager). - -```python -import os - -password = os.environ.get("DB_PASSWORD", "") - -df = spark.read.format("jdbc") \ - .option("url", "jdbc:postgresql://:5432/") \ - .option("dbtable", "") \ - .option("user", "") \ - .option("password", password) \ - .option("driver", "org.postgresql.Driver") \ - .load() -``` - ## Google Cloud Pub/Sub ### Batch Ingestion via Pub/Sub Client diff --git a/skills/gcp-spark/references/spanner.md b/skills/gcp-spark/references/spanner.md new file mode 100644 index 00000000..30c63fc1 --- /dev/null +++ b/skills/gcp-spark/references/spanner.md @@ -0,0 +1,95 @@ +# Cloud Spanner in PySpark + +> [!IMPORTANT] +> You MUST ALWAYS follow the Task Execution Workflow in `SKILL.md` when writing Spark code. + +## Spark Session Configuration + +The Spanner connector JAR is not pre-installed and must be attached at session +creation (`gs://` JAR URIs cannot be passed to `spark.jars.packages`, which only +accepts Maven coordinates). Ensure the Spanner JAR's Spark version prefix +(e.g., `spark-3.5-spanner-*` for Dataproc runtime `2.3` / Spark 3.5, or +`spark-4.0-spanner-*` for Dataproc runtime `3.0` / Spark 4.0) matches the +Dataproc runtime version: + +- **Interactive Notebooks (`.ipynb`) & Spark Connect**: Attach the JAR via a + Session Template or `session_config.runtime_config.properties["spark.jars"]` + (and ensure `session_config.runtime_config.version` matches the JAR's Spark + version, as `ManagedSparkSession` defaults to runtime `3.0`): + + ```python + from google.cloud.dataproc_v1 import Session + from google.cloud.managed_spark_connect import ManagedSparkSession + + session_config = Session() + session_config.runtime_config.version = "2.3" + session_config.runtime_config.properties["spark.jars"] = ( + "gs://spark-lib/spanner/spark-3.5-spanner-1.4.0.jar" + ) + spark = ( + ManagedSparkSession.builder.projectId("") + .location("") + .dataprocSessionConfig(session_config) + .getOrCreate() + ) + ``` + +- **Dataproc Batch & Cluster Jobs (`.py`)**: Pass + `--jars=gs://spark-lib/spanner/spark-3.5-spanner-1.4.0.jar` on the `gcloud` + CLI (see `references/gcloud_dataproc.md`), or set `spark.jars`: + + ```python + spark = ( + SparkSession.builder.appName("") + .config( + "spark.jars", + "gs://spark-lib/spanner/spark-3.5-spanner-1.4.0.jar", + ) + .getOrCreate() + ) + ``` + +## Reading Spanner Tables + +```python +df = ( + spark.read.format("cloud-spanner") + .option("projectId", "") + .option("instanceId", "") + .option("databaseId", "") + .option("table", "") + .load() +) +``` + +## Writing to Spanner + +The connector supports writing to Spanner tables using the DataFrame `write` +API. + +### Write Modes + +* **`Append`**: Inserts rows into the existing Spanner table. This is the + default write mode. + + ```python + df.write.format("cloud-spanner") \ + .option("projectId", "") \ + .option("instanceId", "") \ + .option("databaseId", "") \ + .option("table", "") \ + .mode("append") \ + .save() + ``` + +* **`Overwrite`**: Clears the existing data before writing. + + ```python + df.write.format("cloud-spanner") \ + .option("projectId", "") \ + .option("instanceId", "") \ + .option("databaseId", "") \ + .option("table", "") \ + .mode("overwrite") \ + .save() + ``` diff --git a/skills/gcs-security-assessment/SKILL.md b/skills/gcs-security-assessment/SKILL.md index 062b4f77..128fbb70 100644 --- a/skills/gcs-security-assessment/SKILL.md +++ b/skills/gcs-security-assessment/SKILL.md @@ -14,7 +14,7 @@ description: >- access failure or 403, managing or configuring storage, investigating a live outage, or non-GCS resources (Compute Engine, GKE, etc.). metadata: - version: v3 + version: v4 publisher: google tags: - gcs diff --git a/skills/gcs-security-assessment/references/phases/output.md b/skills/gcs-security-assessment/references/phases/output.md index 8662bccf..9689ea3a 100644 --- a/skills/gcs-security-assessment/references/phases/output.md +++ b/skills/gcs-security-assessment/references/phases/output.md @@ -6,7 +6,7 @@ Present your assessment in a scannable, action-oriented format. The target user is a Storage Admin with limited security expertise who needs to quickly understand: what's wrong, how bad is it, and how to fix it. They may -action remediations via the Cloud Console (Pantheon) or gcloud CLI — both paths +action remediations via the Google Cloud Console or gcloud CLI — both paths should be clear. ## Output Structure diff --git a/skills/google-cloud-auth-verification/SKILL.md b/skills/google-cloud-auth-verification/SKILL.md deleted file mode 100755 index cf100cbd..00000000 --- a/skills/google-cloud-auth-verification/SKILL.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -name: google-cloud-auth-verification -description: >- - Mandatory Step 0 pre-flight execution order and authentication verification for Google Cloud Platform (GCP), - Application Default Credentials (ADC), gcloud CLI, Spark, Dataproc, BigQuery, GCS, and notebook runtimes. - Use whenever interacting with GCP resources, running Spark/PySpark pipelines, BigQuery queries, GCS paths (gs://), or creating/running notebooks. -license: Apache-2.0 -metadata: - version: v3 - publisher: google ---- - -# Google Cloud Authentication Guidelines - -## Mandatory Pre-Flight Execution & Auth Hierarchy - -> [!IMPORTANT] **Pre-Flight Execution Priority Order**: Before generating code, -> implementation plans, or executing tasks for any GCP or Notebook workload: -> -> 1. **Skip Pre-Flight Shell Probes When Live Notebook Execution or MCP Tools** -> **Are Available**: -> - When a live notebook cell execution tool (`execute_cell` / -> `notebook_execute_cell` / `notebook__execute_cell`) is available, or -> when the task is handled directly by a connected MCP tool -> (`bigquery__execute_sql`, `bigquery__get_table_info`, -> `bigquery__list_table_ids`, etc.), **do NOT run a pre-flight -> `gcloud auth list && gcloud config list` shell probe**. Instead, -> verify authentication reactively only if a cell execution or MCP tool -> call returns an authentication or credentials error (`401`, `403`, -> `DefaultCredentialsError`, `Reauthentication is needed`). -> - When `gcloud` or `bq` shell commands are required, or when generating -> standalone scripts or notebooks without a live cell execution tool -> (`execute_cell`), verify credentials upfront via the single bundled -> probe (`gcloud auth list && gcloud config list`) or Application -> Default Credentials (ADC). -> 2. **Distinguish Authentication vs. IAM Permissions**: -> - If `gcloud auth list` returns `No credentialed accounts`, **HARD -> STOP** immediately and instruct the user to run `gcloud auth login` -> and `gcloud auth application-default login`. -> - If Python throws `google.auth.exceptions.DefaultCredentialsError`, -> explicitly direct the user to run `gcloud auth application-default -> login`. -> - If `gcloud auth list` shows an active credentialed account but a -> BigQuery/GCP call returns `403 Forbidden: Access Denied`, **DO NOT** -> tell the user to log in again with `gcloud auth login`. Diagnose -> missing IAM roles (e.g., `roles/bigquery.dataEditor`) on the active -> account. -> 3. **HARD STOP if Unauthenticated**: If no active GCP credentials or valid -> `gcloud` authentication are detected, **STOP IMMEDIATELY**. Prompt the -> user to run `gcloud auth login` and `gcloud auth application-default -> login`. Do NOT attempt local virtualenv creation, package installation, or -> local binary/JDK setup loops as workarounds. - -## Common Error Messages - -1. **gcloud/bq CLI**: - - `ERROR: (bq) You do not currently have an active account selected.` - - `No credentialed accounts.` - - `Configuration error: No account is currently active.` -2. **Execution Failures (Python/Notebooks)**: - - `google.auth.exceptions.DefaultCredentialsError: Could not automatically - determine credentials.` - - `Forbidden: 403 Access Denied` (when it's clearly an auth issue). - -## Verification Step - -Before asking the user to log in, independently verify authentication status -using a single bundled probe command: - -```bash -gcloud auth list --format="json" && gcloud config list --format="json" -``` - -* If the output contains `No credentialed accounts.` or missing active - account, proceed to **Corrective Action**. -* If an account *is* listed but the user still receives a `403 Access Denied` - error, the issue is likely **IAM permissions** (e.g., missing BigQuery - roles) on their active account, rather than missing authentication. In this - case, investigate permissions rather than asking them to log in again. - -## Corrective Action - -When missing credentials are confirmed, **DO NOT** attempt to fix credentials -via code or local virtualenv workarounds. Credentials must be established by the -user. - -**Stop and ask the user to run the following commands in their terminal:** - -1. **To authenticate the gcloud CLI**: `gcloud auth login` -2. **To set up Application Default Credentials (ADC)** (required for BQ CLI AND - most libraries/notebooks): `gcloud auth application-default login` - -## Post-Login Verification - -After the user confirms they have logged in, verify with: `gcloud auth list` -Then proceed with the original task.