diff --git a/CHANGELOG.md b/CHANGELOG.md index 14af03209..fd1054d5b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,11 @@ these packages. All five distributions share one version and one entry. ### Added +- Snowflake Container Runtime quickstarts for Kumo Relational and Kumo Tabular + in `examples/snowflake/`. The relational example samples uploaded + AdventureWorks tables through `SnowflakeSource`; the tabular example reads + `Cleaned_data.csv` from a stage. Both call a deployed NIM. + - `SDMClient.in_process()` runs `kumo-tabular` and `kumo-relational` in this Python process with structured-data-models, through the same calls as a NIM: predictions, fitted sessions, rankings, forecasts and explanations. It loads each model diff --git a/docs/get-started/quickstart.md b/docs/get-started/quickstart.md index 102af90d4..bbd52c5e2 100644 --- a/docs/get-started/quickstart.md +++ b/docs/get-started/quickstart.md @@ -164,6 +164,21 @@ The relational notebook installs the tabular notebook installs `structured-data-models-client[databricks,databricks-serving]`. +## Snowflake notebooks + +The Snowflake quickstarts run on Container Runtime with an active Snowpark +session and a deployed NIM endpoint: + +- [Kumo Relational](../../examples/snowflake/kumo_relational_quickstart.ipynb) + samples uploaded AdventureWorks tables in place through `SnowflakeSource`. +- [Kumo Tabular](../../examples/snowflake/kumo_tabular_quickstart.ipynb) + reads `Cleaned_data.csv` from a Snowflake stage and predicts through direct + requests and PQL. + +Download and upload the sample data yourself, following the notebook's source +links and schema instructions. Configure external access for package +installation and any NIM endpoint outside Snowflake. + ## A model in this process The model can also run in your own Python process, with no NIM at all. The diff --git a/examples/snowflake/kumo_relational_quickstart.ipynb b/examples/snowflake/kumo_relational_quickstart.ipynb new file mode 100644 index 000000000..0b4cbe4c2 --- /dev/null +++ b/examples/snowflake/kumo_relational_quickstart.ipynb @@ -0,0 +1,996 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "1e4bc018f9b64464", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "# Kumo Relational Quickstart with structured-data-models-client\n", + "\n", + "**Kumo Relational** is a relational foundation model for\n", + "machine learning on enterprise data. With your relational data and a\n", + "Predictive Query Language (PQL) query, you can generate predictions without\n", + "training a task-specific model.\n", + "\n", + "This quickstart uses four tables from Microsoft's **AdventureWorks** OLTP\n", + "sample database: customers, products, sales-order headers, and sales-order\n", + "details. Predictions are submitted to a deployed Kumo Relational NVIDIA NIM through\n", + "`structured-data-models-client`.\n", + "\n", + "Each user is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use." + ] + }, + { + "cell_type": "markdown", + "id": "c333f1688de74b17", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## Introduction\n", + "\n", + "Kumo Relational is grounded in three key world views:\n", + "\n", + "> **1. Enterprise data is a graph.**\n", + "\n", + "Enterprise data is a graph where tables are connected by keys. In this notebook, `sales_order_headers` connects customers to orders, while `sales_order_details` connects each order to its products. Together, these relationships give Kumo Relational the business context surrounding every transaction.\n", + "\n", + "Once we structure enterprise data this way, we can apply pre-trained Relational Graph Transformers to extract insights and patterns.\n", + "\n", + "> **2. With timestamps, we place events on a timeline.**\n", + "\n", + "By placing events on a timeline, we unlock the ability to model how things evolve. We can select any point in time and predict what is likely to happen next from the sequence and patterns in the historical data.\n", + "\n", + "> **3. Machine learning tasks can be described via predictive queries.**\n", + "\n", + "Major machine learning tasks, including regression, classification, forecasting, and recommendation, can be expressed using *Predictive Query Language (PQL)*. A PQL query identifies the outcome to predict, the entity to predict it for, and, for temporal tasks, the future prediction window.\n", + "\n", + "If you know SQL, PQL should feel familiar. Learn more in the [query-writing guide](https://docs.nvidia.com/sdgm/kumo-relational/user-guide/make-predictions/writing-predictive-queries)." + ] + }, + { + "cell_type": "markdown", + "id": "454eac1b23f249df", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "**Let's get started!**" + ] + }, + { + "cell_type": "markdown", + "id": "fc9ae315749f4538", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## Step 1. Install structured-data-models-client\n", + "\n", + "Use a Snowflake notebook on **Container Runtime** with an active Snowpark\n", + "session. Configure an external access integration for PyPI package installation\n", + "and attach it to the notebook service. If the NIM is outside Snowflake, allow\n", + "its host through an external access integration too. See Snowflake's\n", + "[external access setup](https://docs.snowflake.com/en/user-guide/ui-snowsight/notebooks-external-access).\n", + "The notebook itself does not need a GPU; inference runs on the NIM service.\n", + "\n", + "Install the relational and Snowflake extras. See\n", + "[Install and Connect](https://docs.nvidia.com/sdgm/client/kumo-relational/get-started/sdk-getting-started)\n", + "for supported Python versions and platforms. Restart the notebook kernel after\n", + "installation if these packages were already imported, then continue below." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a24f971d5d724e15", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "%pip install \"structured-data-models-client[relational,snowflake]\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6aac3912c4a144b8", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "import os\n", + "\n", + "import pandas as pd\n", + "from sdm_client import (\n", + " KumoRelational,\n", + " PredictiveQuery,\n", + " Relationship,\n", + " Schema,\n", + " SDMClient,\n", + ")\n", + "from sdm_connectors import SnowflakeSource" + ] + }, + { + "cell_type": "markdown", + "id": "1e6663f0c2574e5e", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "Build schemas and queries through `sdm_client`'s top-level names: `Schema`, `Relationship`, and `PredictiveQuery`. Submit predictions through `SDMClient`, naming the model with `KumoRelational()`." + ] + }, + { + "cell_type": "markdown", + "id": "40af00956c7342d2", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## Step 2. Configure the Kumo Relational NIM endpoint\n", + "\n", + "Deploy the NIM first using the\n", + "[Snowflake deployment guide](https://docs.nvidia.com/sdgm/client/kumo-relational/deployment/snowflake).\n", + "Set `KUMO_RELATIONAL_API_ENDPOINT` to its reachable address. The\n", + "`http://:8000` placeholder below is a private Snowpark Container Services (SPCS) service address,\n", + "not a container running in the notebook.\n", + "\n", + "For endpoints outside that private service network, use an `https://` URL with\n", + "TLS termination. Plain HTTP transmits table rows unencrypted. Set\n", + "`KUMO_RELATIONAL_API_KEY` only when an authenticating gateway protects the\n", + "endpoint; keep credentials in environment configuration." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e1bcc8f7a5024ff5", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "NIM_URL = os.environ.get(\n", + " 'KUMO_RELATIONAL_API_ENDPOINT',\n", + " 'http://:8000',\n", + ")\n", + "NIM_API_KEY = os.environ.get('KUMO_RELATIONAL_API_KEY')" + ] + }, + { + "cell_type": "markdown", + "id": "bb2867af180a486f", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## Step 3. Connect to the NIM\n", + "\n", + "An `SDMClient` owns the HTTP connection to one NIM endpoint. Keep credentials\n", + "in environment configuration rather than notebook source." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1f6054ae0524596", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "client = SDMClient(\n", + " url=NIM_URL,\n", + " api_key=NIM_API_KEY,\n", + ")\n", + "\n", + "print('Ready:', client.health_ready())\n", + "print('Capabilities:', client.capabilities(KumoRelational()))" + ] + }, + { + "cell_type": "markdown", + "id": "f83cc342e84b44f0", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "
\n", + "Security: Never hard-code NIM or gateway credentials in a\n", + "notebook. Use environment variables or your platform's secret manager.\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "456e3686d07948be", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## Step 4. Prepare the Snowflake tables\n", + "\n", + "Download `Customer.csv`, `Product.csv`, `SalesOrderHeader.csv`, and\n", + "`SalesOrderDetail.csv` from the pinned\n", + "[Microsoft AdventureWorks OLTP sample files](https://github.com/microsoft/sql-server-samples/tree/b47eadc85277f72843647223cde53aa1b85ffcb8/samples/databases/adventure-works/oltp-install-script).\n", + "These files are **tab-separated and have no header row**, despite their `.csv`\n", + "extension. The table definitions below provide the column names in file order.\n", + "\n", + "Upload the four files yourself to an internal Snowflake stage. In a SQL\n", + "worksheet, replace ``, ``, and `` with your destination\n", + "names, then create and load these tables in a dedicated schema. The quoted\n", + "table and column identifiers preserve the capitalization used by the notebook.\n", + "Use unquoted database and schema names in both SQL and the Python configuration\n", + "below. The Python cell normalizes them to uppercase, matching the SQL setup\n", + "and the names passed to `SnowflakeSource`. Your role needs\n", + "access to the database, schema, warehouse, and stage, permission to create and\n", + "load these tables, and permission to create the temporary context table used below.\n", + "\n", + "Each user is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use.\n", + "Microsoft's samples are provided under the\n", + "[MIT License](https://github.com/microsoft/sql-server-samples/blob/b47eadc85277f72843647223cde53aa1b85ffcb8/license.txt).\n", + "\n", + "```sql\n", + "CREATE TABLE IF NOT EXISTS ..\"Customer\" (\n", + " \"CustomerID\" INTEGER,\n", + " \"PersonID\" INTEGER,\n", + " \"StoreID\" INTEGER,\n", + " \"TerritoryID\" INTEGER,\n", + " \"AccountNumber\" VARCHAR,\n", + " \"rowguid\" VARCHAR,\n", + " \"ModifiedDate\" TIMESTAMP_NTZ\n", + ");\n", + "\n", + "CREATE TABLE IF NOT EXISTS ..\"Product\" (\n", + " \"ProductID\" INTEGER,\n", + " \"Name\" VARCHAR,\n", + " \"ProductNumber\" VARCHAR,\n", + " \"MakeFlag\" INTEGER,\n", + " \"FinishedGoodsFlag\" INTEGER,\n", + " \"Color\" VARCHAR,\n", + " \"SafetyStockLevel\" INTEGER,\n", + " \"ReorderPoint\" INTEGER,\n", + " \"StandardCost\" FLOAT,\n", + " \"ListPrice\" FLOAT,\n", + " \"Size\" VARCHAR,\n", + " \"SizeUnitMeasureCode\" VARCHAR,\n", + " \"WeightUnitMeasureCode\" VARCHAR,\n", + " \"Weight\" FLOAT,\n", + " \"DaysToManufacture\" INTEGER,\n", + " \"ProductLine\" VARCHAR,\n", + " \"Class\" VARCHAR,\n", + " \"Style\" VARCHAR,\n", + " \"ProductSubcategoryID\" INTEGER,\n", + " \"ProductModelID\" INTEGER,\n", + " \"SellStartDate\" TIMESTAMP_NTZ,\n", + " \"SellEndDate\" TIMESTAMP_NTZ,\n", + " \"DiscontinuedDate\" TIMESTAMP_NTZ,\n", + " \"rowguid\" VARCHAR,\n", + " \"ModifiedDate\" TIMESTAMP_NTZ\n", + ");\n", + "\n", + "CREATE TABLE IF NOT EXISTS ..\"SalesOrderHeader\" (\n", + " \"SalesOrderID\" INTEGER,\n", + " \"RevisionNumber\" INTEGER,\n", + " \"OrderDate\" TIMESTAMP_NTZ,\n", + " \"DueDate\" TIMESTAMP_NTZ,\n", + " \"ShipDate\" TIMESTAMP_NTZ,\n", + " \"Status\" INTEGER,\n", + " \"OnlineOrderFlag\" INTEGER,\n", + " \"SalesOrderNumber\" VARCHAR,\n", + " \"PurchaseOrderNumber\" VARCHAR,\n", + " \"AccountNumber\" VARCHAR,\n", + " \"CustomerID\" INTEGER,\n", + " \"SalesPersonID\" INTEGER,\n", + " \"TerritoryID\" INTEGER,\n", + " \"BillToAddressID\" INTEGER,\n", + " \"ShipToAddressID\" INTEGER,\n", + " \"ShipMethodID\" INTEGER,\n", + " \"CreditCardID\" INTEGER,\n", + " \"CreditCardApprovalCode\" VARCHAR,\n", + " \"CurrencyRateID\" INTEGER,\n", + " \"SubTotal\" FLOAT,\n", + " \"TaxAmt\" FLOAT,\n", + " \"Freight\" FLOAT,\n", + " \"TotalDue\" FLOAT,\n", + " \"Comment\" VARCHAR,\n", + " \"rowguid\" VARCHAR,\n", + " \"ModifiedDate\" TIMESTAMP_NTZ\n", + ");\n", + "\n", + "CREATE TABLE IF NOT EXISTS ..\"SalesOrderDetail\" (\n", + " \"SalesOrderID\" INTEGER,\n", + " \"SalesOrderDetailID\" INTEGER,\n", + " \"CarrierTrackingNumber\" VARCHAR,\n", + " \"OrderQty\" INTEGER,\n", + " \"ProductID\" INTEGER,\n", + " \"SpecialOfferID\" INTEGER,\n", + " \"UnitPrice\" FLOAT,\n", + " \"UnitPriceDiscount\" FLOAT,\n", + " \"LineTotal\" FLOAT,\n", + " \"rowguid\" VARCHAR,\n", + " \"ModifiedDate\" TIMESTAMP_NTZ\n", + ");\n", + "\n", + "COPY INTO ..\"Customer\"\n", + " FROM @../Customer.csv\n", + " FILE_FORMAT = (TYPE = CSV FIELD_DELIMITER = '\\t' SKIP_HEADER = 0\n", + " FIELD_OPTIONALLY_ENCLOSED_BY = NONE NULL_IF = (''));\n", + "\n", + "COPY INTO ..\"Product\"\n", + " FROM @../Product.csv\n", + " FILE_FORMAT = (TYPE = CSV FIELD_DELIMITER = '\\t' SKIP_HEADER = 0\n", + " FIELD_OPTIONALLY_ENCLOSED_BY = NONE NULL_IF = (''));\n", + "\n", + "COPY INTO ..\"SalesOrderHeader\"\n", + " FROM @../SalesOrderHeader.csv\n", + " FILE_FORMAT = (TYPE = CSV FIELD_DELIMITER = '\\t' SKIP_HEADER = 0\n", + " FIELD_OPTIONALLY_ENCLOSED_BY = NONE NULL_IF = (''));\n", + "\n", + "COPY INTO ..\"SalesOrderDetail\"\n", + " FROM @../SalesOrderDetail.csv\n", + " FILE_FORMAT = (TYPE = CSV FIELD_DELIMITER = '\\t' SKIP_HEADER = 0\n", + " FIELD_OPTIONALLY_ENCLOSED_BY = NONE NULL_IF = (''));\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9b4cbe03709f4914", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "from snowflake.snowpark import functions as F\n", + "from snowflake.snowpark.context import get_active_session\n", + "\n", + "session = get_active_session()\n", + "# Match Snowflake's handling of the unquoted names in the SQL setup.\n", + "DATABASE = ''.upper()\n", + "SCHEMA = ''.upper()\n", + "DATA_LOCATION = f'{DATABASE}.{SCHEMA}'" + ] + }, + { + "cell_type": "markdown", + "id": "55240ba7af5f4412", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "Create a temporary table that joins each order line to its parent order's\n", + "`CustomerID` and `OrderDate`. The uploaded source tables stay unchanged.\n", + "`SnowflakeSource` uses this same session, so the temporary table remains visible when the\n", + "client samples context. Rerun this cell after restarting the session." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5e56b1a7770c46d9", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "session.sql(\n", + " f'DROP VIEW IF EXISTS {DATA_LOCATION}.\"SDM_SalesOrderDetail\"'\n", + ").collect()\n", + "session.sql(f\"\"\"CREATE OR REPLACE TEMPORARY TABLE\n", + " {DATA_LOCATION}.\"SDM_SalesOrderDetail\" AS\n", + "SELECT d.*, h.\"CustomerID\", h.\"OrderDate\"\n", + "FROM {DATA_LOCATION}.\"SalesOrderDetail\" AS d\n", + "JOIN {DATA_LOCATION}.\"SalesOrderHeader\" AS h\n", + " ON d.\"SalesOrderID\" = h.\"SalesOrderID\"\n", + "\"\"\").collect()" + ] + }, + { + "cell_type": "markdown", + "id": "e383e679775c4356", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## Step 5. Build a schema from Snowflake\n", + "\n", + "`Schema.from_source()` describes the tables where they are stored. It reads\n", + "metadata and a small preview to infer types, and checks the configured keys in\n", + "Snowflake. Prediction requests sample related rows with SQL rather than copying\n", + "all four source tables into pandas. Review the inferred semantic types before\n", + "prediction." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d56d61bdd28e4630", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "source = SnowflakeSource(session, database=DATABASE, schema=SCHEMA)\n", + "schema = Schema.from_source(\n", + " source,\n", + " {\n", + " 'customers': 'Customer',\n", + " 'products': 'Product',\n", + " 'sales_order_headers': 'SalesOrderHeader',\n", + " 'sales_order_details': 'SDM_SalesOrderDetail',\n", + " },\n", + " primary_keys={\n", + " 'customers': 'CustomerID',\n", + " 'products': 'ProductID',\n", + " 'sales_order_headers': 'SalesOrderID',\n", + " 'sales_order_details': 'SalesOrderDetailID',\n", + " },\n", + " time_columns={\n", + " # Customer and product attributes have no event timeline here.\n", + " 'customers': None,\n", + " 'products': None,\n", + " 'sales_order_headers': 'OrderDate',\n", + " 'sales_order_details': 'OrderDate',\n", + " },\n", + " relationships=[\n", + " Relationship(\n", + " source_table='sales_order_headers',\n", + " source_column='CustomerID',\n", + " target_table='customers',\n", + " target_column='CustomerID',\n", + " ),\n", + " Relationship(\n", + " source_table='sales_order_details',\n", + " source_column='SalesOrderID',\n", + " target_table='sales_order_headers',\n", + " target_column='SalesOrderID',\n", + " ),\n", + " Relationship(\n", + " source_table='sales_order_details',\n", + " source_column='ProductID',\n", + " target_table='products',\n", + " target_column='ProductID',\n", + " ),\n", + " Relationship(\n", + " source_table='sales_order_details',\n", + " source_column='CustomerID',\n", + " target_table='customers',\n", + " target_column='CustomerID',\n", + " ),\n", + " ],\n", + ")\n", + "\n", + "print(schema)\n", + "for name in schema.tables:\n", + " print(schema[name])" + ] + }, + { + "cell_type": "markdown", + "id": "9549e5a69d5e4576", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "Keys and relationship columns use the `ID` semantic type. The next cell\n", + "sets product `Name` to categorical and `OrderQty` to numerical. See\n", + "[Data Types](https://docs.nvidia.com/sdgm/client/kumo-relational/user-guide/prepare-data/data-types).\n", + "\n", + "This example retains the other inferred columns. Before a point-in-time\n", + "evaluation, exclude fields that reveal information recorded after the anchor,\n", + "such as modification times and ship dates. An anchor does not reconstruct\n", + "earlier values of untimed customer and product attributes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "343f65d6fa1e4421", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "schema['products']['Name'].stype = 'categorical'\n", + "schema['sales_order_details']['OrderQty'].stype = 'numerical'" + ] + }, + { + "cell_type": "markdown", + "id": "b8d9bde03c0043dd", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "Call `schema.check_consistent()` after all metadata changes. An incompatible\n", + "semantic type, a missing column, or a primary key that is also configured as\n", + "a time column raises an error." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "76207d2f8c1440bc", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "schema.check_consistent()\n", + "print('Schema OK.')\n", + "for name in schema.tables:\n", + " print(schema[name])" + ] + }, + { + "cell_type": "markdown", + "id": "daf381f78c0c42a3", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## Step 6. Submit prediction requests\n", + "\n", + "The schema is a reusable description of the data. For each prediction:\n", + "\n", + "1. Write the task in PQL.\n", + "2. Build a `PredictiveQuery` from the query text and the schema.\n", + "3. Call `.materialize(indices=...)` to gather a `Request`.\n", + "4. Call `client.predict(request, model=KumoRelational())`.\n", + "\n", + "The result's `to_pandas()` method returns a pandas DataFrame." + ] + }, + { + "cell_type": "markdown", + "id": "2018ab497db1403f", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "**Note:** AdventureWorks is sample data, and the queries and results below are intended for demonstration. Benchmark Kumo Relational on data representative of your own use case before drawing performance conclusions." + ] + }, + { + "cell_type": "markdown", + "id": "9a3b840a14124567", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "#### PQL: How you describe a prediction task\n", + "\n", + "Learn more in the [query-writing guide](https://docs.nvidia.com/sdgm/kumo-relational/user-guide/make-predictions/writing-predictive-queries).\n", + "\n", + "The short reference below covers the entity, target, aggregation,\n", + "prediction-window, and condition syntax used by the examples in this\n", + "quickstart." + ] + }, + { + "cell_type": "markdown", + "id": "77a82381c7c543fb", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "
\n", + "💡 Click here for a short introduction to Predictive Query!\n", + "\n", + "**Entities** can be specified in three ways:\n", + "\n", + "* `FOR .=1` embeds one entity ID in PQL.\n", + "* `FOR . IN (1, 2, 3)` embeds several\n", + " entity IDs in PQL.\n", + "* `FOR EACH .` declares the entity column,\n", + " while `materialize(indices=[1, 2, 3])` supplies the entity IDs separately.\n", + "\n", + "**Preferred:** For application code, use `FOR EACH` and pass primary-key values\n", + "through `materialize()`'s `indices` argument. This keeps the prediction task separate\n", + "from the entity batch and makes the same PQL reusable. Despite its name,\n", + "`indices` contains source primary-key values, not DataFrame row numbers.\n", + "If `indices` is provided, it overrides entity IDs embedded in the PQL.\n", + "\n", + "**Targets** can be specified in two common ways:\n", + "\n", + "* `PREDICT .` imputes a missing value.\n", + "* `PREDICT (, , , )` predicts an aggregate in a future window.\n", + "\n", + "Supported aggregations include `COUNT`, `SUM`, `AVG`, `MIN`, `MAX`, and `LIST_DISTINCT`. For example, `PREDICT COUNT(sales_order_headers.*, 0, 7, days)` predicts the number of orders in the next seven days.\n", + "\n", + "Conditions can turn an aggregate into an event. For example, appending `=0` predicts the probability that no orders occur in the specified window.\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "7bdf5317192d401c", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "### Example 1A: Forecast 30-day product demand\n", + "\n", + "Predict how many units of product `ProductID=870` will be ordered in the next 30 days." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6f6a049b1a2245c1", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "query = PredictiveQuery(\n", + " 'PREDICT SUM(sales_order_details.OrderQty, 0, 30, days) FOR EACH products.ProductID',\n", + " schema,\n", + ")\n", + "\n", + "request = query.materialize(indices=[870])\n", + "result = client.predict(request, model=KumoRelational())\n", + "result.to_pandas()" + ] + }, + { + "cell_type": "markdown", + "id": "c909c8c8ce3f413a", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "How to interpret the result:\n", + "\n", + "1. `id`: the product with `ProductID=870`\n", + "2. `prediction`: the predicted number of units ordered during the next 30 days\n", + "\n", + "This result can support inventory and demand planning." + ] + }, + { + "cell_type": "markdown", + "id": "bb42f92f2c2c4126", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "### Example 1B: Forecast from a historical anchor time\n", + "\n", + "By default, predictions use the maximum timestamp in the temporal schema.\n", + "Set a historical `anchor_time` to make a prediction relative to an earlier date.\n", + "Temporal sampling uses the configured `OrderDate` columns for sales-order\n", + "headers and details to restrict their records to the anchor time.\n", + "\n", + "The `customers` and `products` tables have no time column in this schema, so\n", + "their current attributes remain available at every anchor. Setting an anchor\n", + "does not recover earlier attribute values. Use historical snapshots of those\n", + "tables for a point-in-time evaluation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fb4d3b45bdfb439e", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "# Query only the latest timestamp rather than loading the order table.\n", + "latest_order = (\n", + " session.table(f'{DATA_LOCATION}.\"SalesOrderHeader\"')\n", + " .select(F.max(F.col('\"OrderDate\"')))\n", + " .collect()[0][0]\n", + ")\n", + "anchor_time = pd.Timestamp(latest_order) - pd.Timedelta(days=60)\n", + "\n", + "request = query.materialize(indices=[870], anchor_time=anchor_time)\n", + "result = client.predict(request, model=KumoRelational())\n", + "result.to_pandas()" + ] + }, + { + "cell_type": "markdown", + "id": "1afe70c0bf7c4654", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "### Example 1C: Forecast for multiple steps" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "43a67a8a138949ec", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "query_multi_step = PredictiveQuery(\n", + " 'PREDICT SUM(sales_order_details.OrderQty, 0, 30, days) '\n", + " 'FORECAST 3 TIMEFRAMES FOR EACH products.ProductID',\n", + " schema,\n", + ")\n", + "\n", + "request = query_multi_step.materialize(indices=[870])\n", + "result = client.predict(request, model=KumoRelational())\n", + "result.to_pandas()" + ] + }, + { + "cell_type": "markdown", + "id": "2458e94f330c4ce0", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "A multi-step forecast returns one row per timeframe.\n", + "`forecast_step` is the one-based sequence number and `prediction` is the\n", + "forecast value for that period." + ] + }, + { + "cell_type": "markdown", + "id": "f4ecd48298c04830", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "### Example 2: Predict customer inactivity\n", + "\n", + "Predict the likelihood that customers `CustomerID=11176` and `CustomerID=11091` will place zero orders in the next 90 days." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "68b73fba28a54daf", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "query_inactivity = PredictiveQuery(\n", + " 'PREDICT COUNT(sales_order_headers.*, 0, 90, days)=0 FOR EACH customers.CustomerID',\n", + " schema,\n", + ")\n", + "\n", + "request = query_inactivity.materialize(indices=[11176, 11091])\n", + "result = client.predict(request, model=KumoRelational())\n", + "result.to_pandas()" + ] + }, + { + "cell_type": "markdown", + "id": "89c2f6703bdc4af7", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "How to interpret the result:\n", + "\n", + "1. `id`: the customer represented by each `CustomerID`\n", + "2. `prediction`: whether the zero-order event is predicted to occur\n", + "3. `probability_*` columns: the estimated likelihood of each outcome\n", + "\n", + "This result can support retention or re-engagement workflows." + ] + }, + { + "cell_type": "markdown", + "id": "f4a9687b1c554efd", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "### Example 3: Recommend products\n", + "\n", + "Rank which of a small candidate set of products customer `CustomerID=11176` is most likely to order in the next 30 days.\n", + "\n", + "Ranking runs as one classification call per candidate, so the candidate set is\n", + "named explicitly with `candidates=` rather than left for the client to infer.\n", + "This example candidates the 15 most frequently ordered products so each has\n", + "enough purchase history for the model to learn from; an arbitrary or rarely\n", + "ordered candidate set can fail with \"nothing to rank\" because its context\n", + "holds only one outcome. See\n", + "[Prediction Types](https://docs.nvidia.com/sdgm/kumo-relational/user-guide/make-predictions/prediction-types#recommendations-ranking)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "62cadc402ad84193", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "# Collect only the 15 candidate IDs; aggregate purchase counts in Snowflake.\n", + "candidate_product_ids = (\n", + " session.table(f'{DATA_LOCATION}.\"SalesOrderDetail\"')\n", + " .group_by(F.col('\"ProductID\"'))\n", + " .agg(F.count('*').alias('purchase_count'))\n", + " .sort(F.col('purchase_count').desc(), F.col('\"ProductID\"'))\n", + " .limit(15)\n", + " .to_pandas()['ProductID']\n", + ")\n", + "\n", + "query_rank = PredictiveQuery(\n", + " 'PREDICT LIST_DISTINCT(sales_order_details.ProductID, 0, 30, days) '\n", + " 'RANK TOP 10 FOR EACH customers.CustomerID',\n", + " schema,\n", + ")\n", + "\n", + "request = query_rank.materialize(\n", + " indices=[11176],\n", + " candidates=candidate_product_ids,\n", + " num_context_samples=5000,\n", + ")\n", + "result = client.predict(request, model=KumoRelational())\n", + "print(result.to_pandas())\n", + "print('Unscored candidates:', result.unscored_candidates)" + ] + }, + { + "cell_type": "markdown", + "id": "c591ab8a0d9948f8", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "How to interpret the result:\n", + "\n", + "1. `id`: the customer with `CustomerID=11176`\n", + "2. `rankings`: the ranked candidates, each with a `candidate` product ID and a `score`\n", + "\n", + "A candidate whose sampled context holds only one outcome can't be scored and\n", + "is left out rather than failing the whole ranking; check\n", + "`result.unscored_candidates` for which ones were skipped." + ] + }, + { + "cell_type": "markdown", + "id": "9ccdfb90708143fb", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "### Example 4: Infer a missing product attribute\n", + "\n", + "Predict the missing `Color` value for product `ProductID=870`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a872685251844a6c", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "query_attribute = PredictiveQuery(\n", + " 'PREDICT products.Color FOR EACH products.ProductID',\n", + " schema,\n", + ")\n", + "\n", + "request = query_attribute.materialize(indices=[870])\n", + "result = client.predict(request, model=KumoRelational())\n", + "result.to_pandas()" + ] + }, + { + "cell_type": "markdown", + "id": "8f2a1fe04ce34513", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "How to interpret the result:\n", + "\n", + "1. `id`: the product with `ProductID=870`\n", + "2. `prediction`: the predicted color category\n", + "3. `probability_*` columns: the estimated likelihood of each category\n", + "\n", + "Attribute inference can help enrich incomplete catalog records, but predicted values should be reviewed before being written back to a source system." + ] + }, + { + "cell_type": "markdown", + "id": "863c7f0fa50d4557", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "### Example 5: Predict customer churn\n", + "\n", + "Predict the likelihood that customers `CustomerID=11176` and `CustomerID=11091` will place zero orders in the next 90 days, reusing Example 2's query with one addition. The `WHERE` condition does not filter which entities are scored, both `indices` are still predicted; it filters which rows qualify as labeled context examples, restricting the model's learned comparison to customers who themselves placed at least one order in the previous 90 days." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1574bbb37ce747d6", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "query_churn = PredictiveQuery(\n", + " 'PREDICT COUNT(sales_order_headers.*, 0, 90, days) = 0 '\n", + " 'FOR EACH customers.CustomerID '\n", + " 'WHERE COUNT(sales_order_headers.*, -90, 0, days) > 0',\n", + " schema,\n", + ")\n", + "\n", + "request = query_churn.materialize(indices=[11176, 11091])\n", + "result = client.predict(request, model=KumoRelational())\n", + "result.to_pandas()" + ] + }, + { + "cell_type": "markdown", + "id": "f217317179fc486f", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "How to interpret the result:\n", + "\n", + "1. `id`: the customer represented by each `CustomerID`\n", + "2. `prediction`: whether the customer is predicted to place zero orders during the next 90 days\n", + "3. `probability_*` columns: the estimated likelihood of each outcome\n", + "\n", + "This churn prediction can support retention or re-engagement workflows." + ] + }, + { + "cell_type": "markdown", + "id": "0af9353c28134884", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## Step 7. Close the client\n", + "\n", + "Close the client when the notebook is finished to release its HTTP connection." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6f15f0bff032428e", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "client.close()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "pygments_lexer": "ipython3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/snowflake/kumo_tabular_quickstart.ipynb b/examples/snowflake/kumo_tabular_quickstart.ipynb new file mode 100644 index 000000000..b4693d774 --- /dev/null +++ b/examples/snowflake/kumo_tabular_quickstart.ipynb @@ -0,0 +1,431 @@ +{ + "cells": [ + { + "attachments": { + "logo.png": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIAAAACGCAYAAAAVZ4LWAAAehUlEQVR4nO1deXAb53V/i5MAD4AkeIEgCZISdVkOJSu21cgRaTtOnDgduXWatNOZyHHbpE4ykdv8k5lOLWWmnUkvy9NJp3GT1G4mySRNY3smTRw7iahEPuRDpixblkiJBG+KBEkcJG7s9r0lF1wAi3uxC1L9zWBA7AIgsN/73vd7v/feBx3chHh2/ISVDQS+gn8eZzRw8g/2PPUM3KRg4CbDs5c//xWW5U7iN7cKxziAYa1G89iDe/59CG4y3DQG8NP3/+KzHAsn8U9npucwHDPEaJmH0RBccJNg2xvAs+9/YYCLc49zDDeQ/6uYpzUa5tTNYAjb1gBw4J0syz6Bfx6DYsCBB6/OaY3Z/OSD3ac9sE2x7QxgneAFceC54yAH0BAYLZzYrkRx2xhAgtlzcEJM8GSES8PAYw/ufeo52EbYFgaQD8GTCxtE8dR2iRi2tAEQwcN1/j9BgYFPx/YgilvSAJ5979F+jos/URizLxM4JIrV5lNblShqYIvh0R9D/9jC6/9aEYOP4Jj4wwvuSz+ELQodbDFo4mC9PvfmkanF94b3OD4MNktXP6gET2jq7FpksZ8D7mOwRVHxBsBLtxxn/cN9T50SH4/Egv0XXb8Eg840fGjnMZvJUOsAhbAWXTrvCU62c8AeTT3H6w9x9isareb5rUAUK5YDSDB7FyVufjP81ATLwZnU59eabOcO9j6wX6c1WKBMCMW8wytBF8S5WJrX6Wy9qz41DN0K0nLFGUAu6TYaCw9fmnipf2V1VuIs42219ry9y3HXATkNIc6GpxfXRmZiXOQOqfM1hmawGjs8mfWHyo0YKsYACpVuV1bn4PLUEIQifomzjLe37bZLzuaDR6AEcBD3LgXGhkMx31Gp8yadFSxVHaDTGPJ5s4qUllU3gFKl27nlERiZfRVi8XDaOQ1opvc7P+IuhiguB8eGglHPASR4aZ7EqK2FOmMbGHW1UDAqTFpWzQDklG5j8QhMLl6CKfe7koZARDHfiMEfmTvnC807keClkUotYwRrlQNMelmU5oqQllUxgHJJt2QIIzOvwNzKiOR5IoroEZxSEQMRvKXgmInj2F2p5zQ4ZeuMdn6tlxtqS8uKGoBS0m0QecH7U2dBmigCtFp7hwSiSARvYe2KW4rZ08DXGFr4gae/ywt1iKIiBqCWdEtEcfzGW5ApYtjt/OBrqOR9VOq1Zn0jP+vzInhyQmFpuawGsM7sucdly80XCSKKY2gIqRHDzq7+YY1GmzTzieBZkdnrtSZQDRsRQ6r4VQ6UzQCePPvhjzsad3+/TLn5onDR9XNwe6cTj/f33rsYYVeahMdM3Ajt9bdApSAaD51fWRv5zF8efs0FZULZkkFXp397+5l3vr3qWrhwDlTGDe8ouJbPg8lcnXTcyNQ0iR9fmXwF3rz2LKyGlkFNkP5AYejC2uXdMYg5oYyQzQCI4FFoJz7GYihFiZvfvfe9Ybd3YhgUhts/AWPuVyHK+ECnS0576LRGdPPGtNd41xbh/NWfwNvj/0v5BlAaFIbO+i5CILoykKpB0JJKN5ARJS8BSdItrl3IZE8+uPdbTz76faDa+8fFz8Xw6/yB3gfay524WQuvwIJ/BPSG5EEPhFZhau4qdNj2Q0/rbWgEBgjH/OALz0E47oer428lPV+r0UFrQy/0tNxOWgKUE6IEU9K10QAz6LAeciVzKYwYzKbH5CCKRRtADunWNbl4cXh09rykrFuuxA3N2FnPu6AzSDs2DVcF9aZOMsS0c2QIb47+TFJa1uuqoLPpFkBpGeRGtgSTBg2wydT3tF5jOpbGpWSSlgs2AF7BWws8jq88keu5HgzDxrKEYXImbiaW3gJGy+JFSx/8QqTbbNIyGcJuxxFotvRAqSD9wR0Yc0XZgGS+otrQCBZDB36fHPpDidJy3gZQinS76HXBKF7UoMTsIr2+u+2gq9jEzezKexDhVtPWeAJJtzTwdDELQS5pucZkhT77EaivsUOhIIK3Epx4m9Z4qfNmlJnrjA7UH4xQIFxo/A8XqijmZQBySbf87JrD2RUrPXGzvDod9IanTHp9ulAjl3SbS1puqGuHXfYPrZmN1up83i9bgqkKvVOtwY5eqgZKQaHSclYDIIIXx3UenyRb2VUshrPLjbNr6V1JQ6DEza3dHzVZzM27pF6fTbolCAMvp3RbiLQsdT5bgolmOnkpUh7lRX7SsqQBKCHd0kV1IT+YXc4vcZNrzVRCus0lLXfY9g33tf9eonYgF8Gr07dCjbEFyoyT2YhikgGoId1mn13rRLGtuRuy5eaVlm4zScsEWsp2th8e1xpjtVm9lL45N8GTC1mkZd4AFGirygmKGIh9+4PutHO7um9LO0YEr8HUVVxRhkwYm39LkijWVFuH25t70wafyGitoa0YgicXXKkbYjA/euf4n6Pb/IdK0eyliKLYAOJsHGzm3oKZfblARHF09jVcyq4kjjXUtl1tstkTHMagNaOXcuK9igkmEYgoGhntH39i37/Na86++/Q/Xp195SJUAEi6DbJLSAQzr+PhcBDedf0aFrxjUAlY8I1CdW3ywNYZO5MI7Kz7Orw3+RuIqiAtpwK5nW9+7bJuzPf6bnpMwbNl2v3u0Vn35eldjiOz9sbdt4PCEEu3BkNuErca9MAl16/AUt2EwsxRqKlqAKVBxroanef1B01KSmU9AoklHVv2zcDL7/8QpeUe6G25A0UlZb0BDfxKaPJCMLqMXAqOaDboX0I9ocTN+9O/dVyff2N4d8eHmaa6rg9AmSGWblN1e0KTxQk77YfxYhlRr5/FBMlS0nkhcdNc343x+JGy6/UEGnj6LGSoUgmmDtst0NW8h88teELTGAFsLmVxNgYz7hFY8Ezic/B5TfJLy1JYD0Nn9+PAD6SeYx79Ae2RlI71xM0nHCYDqh1lQDbp1uv1gr1+DyptbUnHSa9f8I/ByOT5tNeUO3GTKcEkwOfxA4aAfIIp6XWRJYwY3oSF5Ym018gpLUshGPVeRPGpUUp/oCTTlw+/OZTRAAQ01LSf3e+8F0UOYx3IgFkPSrdsadItScsUMSiRuCEvNee9DFq99Pl8egPibBQmFt7JLi23fwjqq+WZazTwnvAkF2cjGQW8vA1gA97W+t63d7UfOVisIZDr9EdmQU7pNlvipspQg8vHnSXNLrkSTAJI8xjH0DGbtNzXdhiqi+Q0OOAz7sD18UximRiFGsDGiwpP3JDrvOG/kpHclSrdliNxU44Ekxi5pOV2W19BRFEgeIHo8gDkiaIMQIBWq7+6r/PuUDaiqLR0m2t2ddT+HnS17QajKXNDNMNW8cUh3mh6iFmO3gCSlkcziF8Mw/gcjfveFkvLUlgOjg9tMPuCUuolGYAAStzs777PbDW39AnHhHSnWtJtttml19RAt/UI2FsduKZvunWGM4AvdENy4AnlSDCJkUtalvK62RJM+UAWAxBQg4mbW533dsdgbdwXmtsvNfBKS7fZZhcZws7Gu6HN3h4Ph0PaxfA7ku+hdG9AJmmZILS3VZutTC6Clw9kNQBCJv1bjjWzFGSbXQd7fx/W2Jm04yU1f5YI4jTrhnBJ8rxUXqQYCAZQth1ClG2ryoy2hj7+lm12CaiEBBPpCH3th6EDQ9lsnEYuyFYWbqvauV/82OtfAf/asqqDL0ZdtQ1abJ1Jxxgm+evP3BhFvT4KlQAStqqrazEkLG+OTj4PwFBym008DEfX4MriyzDtfg+6Ww6VTe3KBbF0mxrPp1bDqCEtSyERhup1RKqhnCj7JlFC4oZEjp0ociiVuMmVYGqr7wOz0QKRsIfX7cVYWBmHJe+UYj0BAsQJJp1C+3cptksYZcPO+8o/u8TSrZRuT4LQno6jid6AJl0fVRvxiRsx1hM3VzFxM1G2ngAB3sAcLAVckgmmckPxbeLEs2t3+1GQE4J0K47xBRCzb7D2grk5fU2lHT/oFrNzGDFcSCKK0VgIrs+9CTNLV0qWllNRaBq8HFBln0C5Z5d4zUzltYWEoR1N+zFi2CUpLYciq/xSVkpPgIBcXkpJ6JAIPYNCwGdBBZQ6u5Jy8ym2LIShNPiFgMIw6huk0FEqDCNOc+H6z4rmNFPLF4HTRCS9lILwgp6bpz+0b/wPPPfBT8PzSOD3QAmNH1azA3TGzSiAGjGDIX9eryXxY8EzBh7vEqZWHTlnBUm3i6vXIcwtYV4iPcwk9a7B1ANVuuIz2NQ5TAUpdAuEvWlCUjDsh/mVazidTVBrbgCNNnuPDRvTwaz/Il7xKOn8kC98q0s4USKJx7b64j0PDw6erDPVHvv8ba/ypCfxSabmpgauuV944sXRk/3e0EzB7+u03QHG2s0yKPcKEhvPLBSDTIkbhtPiQIRADek2P2m5Ne1ctgRTPpiaG8HJtGl8xSqBXTWDsLP249+1Ww78dX19faJHgJmbm3OirkzNnseFg+/M/zf8zvUEFGIIchoAQZy40en0EMO3XgzhuswF0p6rpHSbTVo2a5thV+sgNNgsvJcikrccuQKloFQDsFXdAvssn8b7ffxjjm/hh9P21s5voCcKMedeG/pde5u9W683JpWjhOM+eH3qO/D69Hf4UqxckNsABJAh9DkPSA68WtJtrhqE3oZBVmfxybLIF2sAZl0TfKDhc2A3pe9uG2PjFw0a7Zfs9q5zunAodGRsfAxq6yxDLbaWg1qdll84jXh3l/MxuL3jEXhp9BR6hZ+AGoiyq2mDTwTPYuxQLcEkEMVOjBqkEjdh1otCrjp7cOo0Zuivf4R3+angOHYmFAkuIg/ojzIMv74mrNTv8w5cGxvhZudnh8QvIkN4YPc/wxfvfBn6bPeB2ojH4qCJV1dEY4iQuNlhT56VWq3yoR0N/F7rp+ETjqfSBp8GPhwJnguFg+00+EmvS3kfCxnCiN833dRoc9U32BJFCMTOH7rlP2DS8xr8FvkB3auBcDQI1+aH+fBMSWlZCgnp1phCVhllZz8N+F7LZ3i3LwbHcb5oPHohHosewIeSlVmS6xS+0LHgXjyCHmF4ddWf1DXUab0T/rT/R+gV/gmNInMVa9MG6SgXeGkZEzeXJl9SfDMnkm5p86kQ5y67dNtcszfjOSJ497d/Cw41fjlt8HHQccYHOLwfgCzlYlk/Pbrb/pnZaRRa9Ocd7R0OMVG8tfVT/O0NJInkEQR0VQ/CHnRFF6a/D1fgJSg3yiktp0IN6fYjO/4WAg4/vHTt67Cwepk/ZtE7ob/hkQSzFyMej7weicZQLODyKtzNy3wjkegdRBSNVVXnOuydtwpEkfBBxyNwa9unYHTpRWip240zX/mNFsuduFFbuu2yHoY/O/QLPjy31NgkCR4x+2g0xOEaX1BrX0HfhiIGXBa8UhHDLc0PgdooR+KmQqRbHuRxU0EELxINj7MsW9QeS8V8K54oXh8f9a0suyV3AeVY9moszhYuJ8oEIXFzfuTHRe/6SQkm2l2U0cUkG0OUwuKi+614PO5NPU4ELxIJniVmX+zgE4r+ZgJRHBm9Mi0QRTw2zYcbkeCuO9u/0P7Q3qd5aVYtUOKGiOLF8RcgGAzl9Zog5jBcS68Dqw0pnpsXwxo/CH2Rr4I22HLbtesjMDs7MySci8QiQzzBY9mSSU/J35AMAYmiw2QyvdjU1ETrT8IaO+puh0cO/Aredz8Pr059E7zhaVADbt8keFd/KtkTIGBTur2KaWX16hjrdbugMTwAVbGk+kWLf9U3cHXUP9nZ0UGPB0AmyGni1DKTVG1BRZfUC3jQ8Sewt/UBPmJ4ffq7KC37QGmQojiy/AKMe2pgT/PHoKllXT/YTDC9BWrCrN2Qbs3r0u3KyjK4lxYB3fvmkziuE2SGbAaA6lfSh4tGY2A2VYN2YyMkSs2StExRw0vXTsElFaVlkrXNi81waOeD7FLkXY1UnkEp6FHBu7X+c+CsuTvpeH19A5jMJnL9eC3LV6ksG7tBopQkyQUCazDuus5bsRhkCJ9EafnRO8+pKi0H4guo4s2pNvg08HswS/cxFHJSBz8SiYDXt0IMv+wEtKwsh9zX0pIbvF4PNDQ0Qr1100aoP5Ck5QnPq5h6Pq2KtMyolLARxLJqXXKjaQxz3msB6krWKiY0KUJz6YstLNwAn88LjY02qKneTN+SyNHVf3ijBuE0eEPKEUWlNXubcR8csn05beAJfr+PrypC1RWUhKJxTigUgpmZaTCbzdDS3IpfdnO/PEFapvoDMoRMRFHFPfaKBkm3RPCkVNK1NT9FUqpFHqooHIFAAPnBGFCegbyDGLcjSaTU813O9N3od9Q+ALfb/goqEY8c+jmfKBODmP1tjV+Ce+3/kjb4xJF8Pg/Oeg2gogpqgXqPvLgUlu0Xt7MBBST+QlgsFmiotyWEFyFi2N/6EO8N2qqdiTWTOEMloqVmH58lpc938cYPYIflCJ+fTwURvGBojV/j9Qq7+2RwvDKm04DRyXIhnG7MCTUMgYjiysoKroF+sFrrobHBljhHRJEihq0EntPgLRUo58Iqunta49VqAtnABHKfkzu69/CsWzM4OOi5Z/D+kxrg+quMRtV+4YuWArd7Ea6OvA/LK5n1+1i8vM2SxYKWtUxAeRwi0ZDiBC8FMY1O+82dPXudOPhPCwd5nzs7O3mMBfr9H8ZJosPc/Cxq5+qJI4uLNzB8XISWllaoq113SiwbxwsZAnv1Abiv9+/gtelv8k0haqOa7Ybm2D0wNT0BJpMZWlvaEjMcs3Q482OgNvQ6PS6vfB77i1OzrnYdY3ysra3NReeYq6Pv/X11dc3XUl8URIu+sTCPFz2c1z+pq7OgC99cQWizR7qV/OH1erDb7TxTFoOihAvz34O35/4rrbs3XxzacQz88SkoBgZogNbo/VDHplfsoN7hq62rkWVfxRs3bkA4vDkGnZ35q8FUm6jXGdLCXQ44r0Fruq2lpeW6Znpm6mtSpV8mDNWczh7eovUqZsXII6UOPsFksMJAz1fhi4df4cmiUliXbh+GT3Z9G/qa7uFnVyowRauqr6ffIagymsGgN6YNPl8qFgr6fYFlPqukWz+4XvpFFT+pPQIWi5W/kUumtTkpOaEC6P9XVZk3cwyadWmZwsZfXfs6jLhfhHKABn5H7SdhR90DYNCs/0SQcG3IU3o8K4nnaqX61RQADbZBXyUpH/MVQ5Fw42apWMpm0QShRwCFmrOYNj0gLv1qbGzCBEUjT9RWPOr9tCqFUfPz84pKy7mk25qa6iQDwBmoLM1naBcxo2Q5OseyI6EoEjpO+nefJH07MtqjQumXvdU+IBwny2pubuEzVSTtUlijBpSSlkm6JQXPauhOO6eUdJs1GcRQEytP8NJO5Vsqlm1xz9gjQMSsvd3BE8VFXBpSIwYiHjpd+ZfBfKRl6nEstAahkqTbVoyElpaXeWMXg2Y7rfGpEPUCDODDnLtP52R3G6VfjsVl91V7a3uopqY2sT0sEcVOcxev6AVDAX7QdWiRSidZBGkZPxtvCOJSLqEG4Y2NHEM2kHRLrj41Pbv+P9bQ80QVV++0eE3bWu1gQ083Nz/HEzwpckeIxiJn8TOSqx/I8+3zTwZxLLeLiCLyguGujq4mMVGkC083tZGvtJxajCJF8ARUinRLlVWdHV2S54jZR2JRJ45SwTWCBcd3FDEIPQJSXcUCYtHoJN7JXsKUC2JpWYooUsRw64YhECjBRLM+deAFgqe2dIsh27Vqc80OqXOiXoCiq4KLDvA3Ioa0HoE4Gx+ORMK26prqzhiqYGtra6AGBKK4vLzEK4pSRDHCrqUNPMGLWbr1ogz1ZjwZMhkximk7UGs47+jobDfoDfzG0DzBi4SmWY67A0pEqQoPTxTx5m1rbvu13qRvFMINcr+NjY1QW1vLz0ixmqUkyBCIKBKbFkvLhDR3vyHdqqzZ8woqDb6guURj0TvGx68j2a0+29zcpN1g9rL8vIhcEp/F4/c0NlU1pcWapI/X1Vr5SGFh8YZqhkAXc25ultcxWlvsfOQggLaHjYn24VELxOyJSNehs4pGohBIia6Q3xxlWXnb4sum8VKoRIqdZmM/XrRecHb18PWB7iU3z6jVAEnLQuIG2XUoFo9Ugcog70SxvKBu0rWimx9J7Twm5sqpvpatIsjn8/EzLpKSTCLp1NnVzQs4arZckUeKRIOybJVfLHjp1lAFRoMpMfgCKJrhuDivuZQTZc3yUFhGt/r6+qSwjKRyG0nLyNCpalgtaZkB9QyQwjqdxE+RBUNBdP9hPuTUa8rPRRRJ8xEJJGJD8Xlz0+ZWamQI69Jy/Ya0vAqKgmEU3w48k1imlt6gWJ43W3xOs6G9vYN3e8QP1CxGKRcy5ebV6AUQQ3EfKMTnE5PjackkIj6kdtntDslcAl08JXIMxSATp6FjRqNJUr71+b18uRiFnWrxIdUqPYRETlVVFa91ixM5tSgr001okCQICSY2ro6wlAvEaSx1Ft6DEVKZvRhCQkmvV3ejaIJOw3GDGGQ8jebZBSqADEGUyOGQKCamCaWdqdSMLqbSCaZiQEsZGTN5Oam9BdRKKEngeR1UDdMfmsHB+4fuGbzfiSb5MN8joBIoWhgbv+anjRDEO2IQURQPPsZtCjPFwpE6+PzGGeHAeZJEVB78s3gxB3f27D3W3d3N7xecWHjQCJ6mHgH8tKfUMgS8UHW0EQLtiLEksf0Mzp5zLBv3G42V1R5GEQ5+5mGMYobFx6n4cn0bl4BDDt2+BEzgHHoQB35gZ/eeIfEJSb965swZKweh0xzD5P07AiaTabipaVMKlqMqGD/ctL29w43vTYKSDS9p4lcyaemgqKKU3vmOjk4oZWVZW13jv2NMVPptNFads7fam/DDz8XZGG3QWFKzTSlVwQB819fJnd17MxZCSFJPaha5e/D+48gPunFangWVgO7eMTMz1Y+Zr37x4BOIPNIAtrS0KR4ZECchchfG2D2WUvcfDoeOzC/MBXHwBwDUabkDGnj05DrG5Mw2+ISsNBT5gQvvBs6c+cUAEsWTOF3KuxNjnhCSJpRnsFoM6xEDqokrZa5aJi6ix3BOYPbtGK4SsSM1M1Ah2gVOmmf0DHOyu2evK5/n5xV8bhDFgXWiyE2Aigij6/d4PFShlDgmSMs93TuSBCY5QQyfau1Twzr636RwVgDO6hjdgb6evce7u/e48n1RQeoDEcWNiOGUmhEDzfSx8VE+vSyGIC33dPdCTXUNyAHyNDTwqbp9NLq+jQtLCRsVmT0SzYsbzH6gu7tvuNDXFyU/rTeTbkQM0jgbCAafgTKCRQ9AhnB9bDSpJp8gSMsdjk4+7VsMaJkhBS9VvqUuX6/fwxePKCHdGo2Gb2TwutTl+3Bfz77+VGZfCIrWHxNdxUgUGY57ZvNDrYcb0XDYBQqARBfqzCFpea1AaVkKYulWk/LbwiTdYkiH55STbq31DS9ooKpf5HUFgtcv7vItFiVrkRtE8fiFyxe6cOBV4wcUFk5vSMsYhqFbziwtSxFFYU9DSek24EdXr550S5MN705ieH4aBz7U3dOd37aneUC2b3Rw70FVyaEAMoQxlJapFrGluS3pZ+UEaVmIGAhCgkkqN0/MPoZrPa3x6m8VnTAEWVG274VvPKQmUaS0s0AU46JNJYSIgaqS6O/MBM+DxsGpLt3iDHVBGVE2A6DQUW1pWSCKKNOGl5eWkrpFyd0bDcnlgNRWFY4EzsfiUbUrgxNcqpCQrhgokmIrRloWQyx/kg5wY2EBigG6+2l7W/uSuL2NkPLbOkUF9ZOTk4m/U2XxApBTupUbiuZYUVF0bqSeC1IU5TIAAdTe1trcxpAhiNqqHFACSjQAYvandRrzaSFLpxQUpbWVIi0LG2J0dnadLaafTk4UKt3KDVXiGuIHeDfw6zO/OA7rhqBKMQrHsVUqFpqQdHuiGPVOTqga3VSKtKwwJkqRbuVGJYS3+UjL2wG8dEv79JUi3cqNijAAQgZpOQ1xln0ZKhGZs6SySrdyo2IMQAARxY1ilMGUYhS+nm15cfFvoAIhVVdJBI8vyujdd1Jpdp8vKr7U9syZXx7rcHbAju69z60/xgiCYc6ADOjo6DiPblmWWj107fy1JM3DVKP7oxab7cVyizhyQP3C9BwYHPzoc7CFsKHXPwVbBBW3BOSG1lOBEUNFJMKKwZYzgMHB+4ZpZ/NsRFFBkHT7GDF72KKo/HabLChWWhZQEgcgZq+CdCs3trQBCFiXlpnT+G0+UMjrijGAhHS7BQhePtgWBiCgUGm5QAOoCOlWbmwrAxCAhnAyn5/AydMASLo9XknqnZzYglFAbmxKy/AkFI+KlG7lxrb0AGIQUcR1+6RUMUoGD6Babl4NbHsDECBVg5BqAOsEz3TiZhh4ATeNAQggaZkF9jQRRZEBPK9jmBPbhdkXgpvOAARQxOBwOG7R6HQ/285rfC7ctAbw/1jH/wEHnCVcWuW6QwAAAABJRU5ErkJggg==" + } + }, + "cell_type": "markdown", + "id": "4ed0a233", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "![NVIDIA Structured Data Models logo](attachment:logo.png)\n", + "\n", + "# Kumo Tabular Quickstart with structured-data-models-client\n", + "\n", + "**Kumo Tabular** is a structured-data foundation model for predictive AI on a single table. It uses labelled context rows to predict outcomes for new rows without training a task-specific model.\n", + "\n", + "This notebook runs a **binary classification** task with the Mammographic Mass dataset from **Kaggle**, using `structured-data-models-client` to call a deployed **Kumo Tabular NVIDIA NIM**. It demonstrates both a direct `Request` and a Predictive Query Language (PQL) workflow.\n", + "\n", + "**A GPU is not required on the notebook machine.** The model runs on the NIM server. This notebook reads the dataset from an existing Snowflake stage using an active Snowpark session.\n", + "\n", + "The client identifies the model as `KumoTabular()`; its size is determined by the server's deployed checkpoint. Local PyTorch `size`, `device`, and ensemble settings do not configure the NIM.\n", + "\n", + "See the [Kumo Tabular overview](https://docs.nvidia.com/sdgm/kumo-tabular/about-kumo-tabular/overview), [quickstart](https://docs.nvidia.com/sdgm/kumo-tabular/get-started/quickstart), and [client repository](https://github.com/NVIDIA/structured-data-models-client).\n", + "\n", + "Each user is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use." + ] + }, + { + "cell_type": "markdown", + "id": "561d19ec", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## 1. Install and connect\n", + "\n", + "Use a Snowflake notebook on **Container Runtime** with an active Snowpark\n", + "session. Configure an external access integration for PyPI package installation\n", + "and attach it to the notebook service. If the NIM is outside Snowflake, allow\n", + "its host through an external access integration too. See Snowflake's\n", + "[external access setup](https://docs.snowflake.com/en/user-guide/ui-snowsight/notebooks-external-access).\n", + "The notebook itself does not need a GPU; inference runs on the NIM service.\n", + "\n", + "Use a supported Python kernel with pandas, NumPy, and scikit-learn.\n", + "\n", + "Install the base client; the relational extra is not needed. See [Install and Connect](https://docs.nvidia.com/sdgm/kumo-tabular/get-started/sdk-getting-started).\n", + "\n", + "The `http://:8000` placeholder is the NIM's private Snowpark Container Services (SPCS)\n", + "service address. For endpoints outside that private service network, use `https://`\n", + "with TLS termination. Plain HTTP transmits table rows unencrypted." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "39f1ec88-eee7-40e9-9c9b-4b2d36e0031c", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "%pip install structured-data-models-client scikit-learn" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "80909c75-dfb5-4051-b037-4a6b92e92049", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "import os\n", + "\n", + "from sdm_client import KumoTabular, PredictiveQuery, Request, Schema, SDMClient\n", + "\n", + "# Override the endpoint through environment configuration when needed.\n", + "nim_endpoint = os.environ.get(\n", + " 'KUMO_TABULAR_API_ENDPOINT',\n", + " 'http://:8000',\n", + ")\n", + "\n", + "nim_api_key = os.environ.get('KUMO_TABULAR_API_KEY')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1ef18d21-c18a-44e6-b8f3-078ac5a2cc30", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "# Check the models served by the configured NIM endpoint.\n", + "with SDMClient(url=nim_endpoint, api_key=nim_api_key) as client:\n", + " print(client.models())" + ] + }, + { + "cell_type": "markdown", + "id": "e94f92c0", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## 2. Load the dataset\n", + "\n", + "**Dataset:** [Mammographic Mass Data Set (Kaggle)](https://www.kaggle.com/datasets/overratedgman/mammographic-mass-data-set), version 3\n", + "\n", + "**Task:** Binary classification of breast masses: benign (0) or malignant (1)\n", + "\n", + "Use four predictors: `Age`, `Shape`, `Margin`, and `Density`.\n", + "\n", + "Download `Cleaned_data.csv` from the Kaggle dataset page and upload it yourself\n", + "to a Snowflake stage with that exact filename. Replace the stage path in the\n", + "next cell with its location. Your notebook role needs permission to read it.\n", + "\n", + "Each user is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bfdc049a", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "import pandas as pd\n", + "from sklearn.model_selection import train_test_split\n", + "from snowflake.snowpark.context import get_active_session\n", + "\n", + "# Read the Kaggle CSV uploaded to the stage\n", + "session = get_active_session()\n", + "with session.file.get_stream(\n", + " '@../Cleaned_data.csv'\n", + ") as f:\n", + " data = pd.read_csv(f)\n", + "\n", + "# Select predictors and readable class labels\n", + "X = data[['Age', 'Shape', 'Margin', 'Density']]\n", + "y = data['Severity'].map({0: 'benign', 1: 'malignant'})\n", + "classes = sorted(y.unique())\n", + "df = X.assign(target=pd.Categorical(y, categories=classes))\n", + "\n", + "# Split ICL examples (\"train\") and test sets\n", + "train, test = train_test_split(\n", + " data.index,\n", + " test_size=0.2,\n", + " stratify=y,\n", + " random_state=0,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "9b2d96a3", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## 3. Prepare rows and run Kumo Tabular through the NIM\n", + "\n", + "Use the training split as labelled `context` and the held-out test split as `predict`. The target column must be present only in the context. A row ID lets PQL identify entities and keeps the held-out rows distinct from the context.\n", + "\n", + "The explicit schema preserves the original numerical and categorical feature types. The NIM accepts `task=\"classification\"` for both binary and multiclass tasks. Direct requests support up to 10,000 context rows; this dataset fits within that limit.\n", + "\n", + "See [Make Predictions](https://docs.nvidia.com/sdgm/kumo-tabular/user-guide/make-predictions/make-predictions)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "91fccd58", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "context = df.iloc[train].copy()\n", + "to_score = df.iloc[test].drop(columns='target').copy()\n", + "context.insert(0, 'row_id', train)\n", + "to_score.insert(0, 'row_id', test)\n", + "\n", + "schema = Schema.from_data(\n", + " {'masses': context},\n", + " primary_keys={'masses': 'row_id'},\n", + " relationships=[],\n", + ")\n", + "for column, stype in {\n", + " 'Age': 'numerical',\n", + " 'Density': 'numerical',\n", + " 'Shape': 'categorical',\n", + " 'Margin': 'categorical',\n", + " 'target': 'categorical',\n", + "}.items():\n", + " schema['masses'][column].stype = stype\n", + "schema.check_consistent()\n", + "\n", + "request = Request(\n", + " context=context,\n", + " predict=to_score,\n", + " target='target',\n", + " task='classification',\n", + " schema=schema,\n", + " entity_table='masses',\n", + ")\n", + "\n", + "with SDMClient(url=nim_endpoint, api_key=nim_api_key) as client:\n", + " result = client.predict(\n", + " request,\n", + " model=KumoTabular(),\n", + " outputs=['prediction', 'probabilities'],\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "9e64dc5a", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## 4. Read the predictions\n", + "\n", + "`PredictionResult.to_pandas()` returns `row_index`, `prediction`, and `probability_` columns. `row_index` refers to the zero-based position within `to_score`, not its pandas index. Align on it before evaluating against the test labels.\n", + "\n", + "See [Understand Prediction Results](https://docs.nvidia.com/sdgm/kumo-tabular/user-guide/work-with-prediction-results/prediction-results)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c46ac604", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "answers = (\n", + " result.to_pandas().set_index('row_index').reindex(range(len(to_score)))\n", + ")\n", + "predictions = answers['prediction'].to_numpy()\n", + "probabilities = answers[\n", + " [f'probability_{label}' for label in classes]\n", + "].to_numpy()\n", + "answers.head()" + ] + }, + { + "cell_type": "markdown", + "id": "6dc8e858", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## 5. Evaluate predictions" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "039b95c2", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "from sklearn.dummy import DummyClassifier\n", + "from sklearn.metrics import accuracy_score, roc_auc_score\n", + "\n", + "baseline = DummyClassifier(strategy='most_frequent').fit(\n", + " X.iloc[train], y.iloc[train]\n", + ")\n", + "pd.DataFrame(\n", + " {\n", + " 'Model': ['Kumo Tabular (NIM)', 'Majority class'],\n", + " 'Accuracy': [\n", + " accuracy_score(y.iloc[test], predictions),\n", + " accuracy_score(y.iloc[test], baseline.predict(X.iloc[test])),\n", + " ],\n", + " 'ROC AUC': [\n", + " roc_auc_score(\n", + " y.iloc[test], probabilities[:, classes.index('malignant')]\n", + " ),\n", + " roc_auc_score(\n", + " y.iloc[test],\n", + " baseline.predict_proba(X.iloc[test])[\n", + " :, list(baseline.classes_).index('malignant')\n", + " ],\n", + " ),\n", + " ],\n", + " }\n", + ").set_index('Model').round(4)" + ] + }, + { + "cell_type": "markdown", + "id": "f1d2284c", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## 6. Run the same task with a predictive query (PQL)\n", + "\n", + "PQL names the prediction target and entity. Here `PREDICT masses.target FOR EACH masses.row_id` asks for each mass's benign/malignant label. `materialize()` builds the request, and `client.predict()` sends it to the NIM.\n", + "\n", + "Reuse the schema built from **training rows only** and supply the held-out feature rows with `predict=to_score`. This keeps test labels out of the prediction context. Pass `task=\"classification\"` explicitly. Both workflows use the same context, test split, and semantic types; predictions can vary between calls.\n", + "\n", + "For rows already in a schema's table, `materialize(indices=[...], task=\"classification\")` selects rows by primary-key value and excludes them from context. The `predict=` form below scores rows supplied separately.\n", + "\n", + "See [Make Predictions with PQL](https://docs.nvidia.com/sdgm/kumo-tabular/user-guide/make-predictions/make-predictions#ask-in-pql)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c7d38349", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "query = PredictiveQuery(\n", + " 'PREDICT masses.target FOR EACH masses.row_id',\n", + " schema,\n", + ")\n", + "pql_request = query.materialize(\n", + " predict=to_score,\n", + " task='classification',\n", + " model=KumoTabular(),\n", + ")\n", + "\n", + "with SDMClient(url=nim_endpoint, api_key=nim_api_key) as client:\n", + " pql_result = client.predict(\n", + " pql_request,\n", + " model=KumoTabular(),\n", + " outputs=['prediction', 'probabilities'],\n", + " )\n", + "\n", + "pql_answers = (\n", + " pql_result.to_pandas().set_index('row_index').reindex(range(len(to_score)))\n", + ")\n", + "pql_answers.head()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "21562091", + "metadata": { + "language": "python" + }, + "outputs": [], + "source": [ + "pd.DataFrame(\n", + " {\n", + " 'Model': ['Kumo Tabular (NIM, PQL)'],\n", + " 'Accuracy': [accuracy_score(y.iloc[test], pql_answers['prediction'])],\n", + " 'ROC AUC': [\n", + " roc_auc_score(y.iloc[test], pql_answers['probability_malignant'])\n", + " ],\n", + " }\n", + ").set_index('Model').round(4)" + ] + }, + { + "cell_type": "markdown", + "id": "425c9d32", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "## Next Steps" + ] + }, + { + "cell_type": "markdown", + "id": "053b0106", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "### Check out other available Structured Data Foundation Models:\n", + "- [Kumo-Relational](https://huggingface.co/nvidia/Kumo-Relational)\n", + "- [Kumo-Anomaly](https://huggingface.co/nvidia/Kumo-Anomaly)\n", + "- [Kumo-Forecast](https://huggingface.co/nvidia/Kumo-Forecast)" + ] + }, + { + "cell_type": "markdown", + "id": "27aae15a", + "metadata": { + "codeCollapsed": true + }, + "source": [ + "### Check out Kumo Engine\n", + "To run foundation models for structured data in production at scale, check out Kumo Engine\n", + "- [Documentation](https://docs.nvidia.com/sdgm/kumo-engine/get-started-with-kumo-engine/overview)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.15" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/packages/structured-data-models-client/tests/test_snowflake_notebooks.py b/packages/structured-data-models-client/tests/test_snowflake_notebooks.py new file mode 100644 index 000000000..563346506 --- /dev/null +++ b/packages/structured-data-models-client/tests/test_snowflake_notebooks.py @@ -0,0 +1,132 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. +# All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Check the Snowflake example's table layout and source-backed schema offline.""" + +import ast +import datetime +import json +import re +from pathlib import Path + +import pytest +from sdm_connectors import DuckDBSource + +from sdm_client import Relationship, Schema, SourceTable + +NOTEBOOK = ( + Path(__file__).parents[3] + / 'examples/snowflake/kumo_relational_quickstart.ipynb' +) + + +def test_uploaded_tables_join_and_build_source_schema(): + duckdb = pytest.importorskip('duckdb') + pytest.importorskip('sdm_relational_engine') + cells = json.loads(NOTEBOOK.read_text())['cells'] + sources = [''.join(cell['source']) for cell in cells] + setup = next( + source for source in sources if 'CREATE TABLE IF NOT EXISTS' in source + ) + connection = duckdb.connect() + try: + for statement in re.findall( + r'CREATE TABLE IF NOT EXISTS .*?;', setup, re.DOTALL + ): + statement = statement.replace('..', '') + connection.execute(statement.replace('TIMESTAMP_NTZ', 'TIMESTAMP')) + + for table in ( + 'Customer', + 'Product', + 'SalesOrderHeader', + 'SalesOrderDetail', + ): + columns = connection.execute(f'DESCRIBE "{table}"').fetchall() + row = {} + for name, kind, *_ in columns: + if kind == 'TIMESTAMP': + row[name] = datetime.datetime(2024, 1, 1) + elif kind in ('BIGINT', 'INTEGER', 'FLOAT', 'DOUBLE'): + row[name] = 1 + else: + row[name] = 'example' + placeholders = ', '.join('?' for _ in row) + connection.execute( + f'INSERT INTO "{table}" VALUES ({placeholders})', + list(row.values()), + ) + + context_setup = next( + source for source in sources if source.startswith('session.sql(') + ) + for statement in ast.parse(context_setup).body: + sql_call = statement.value.func.value + sql = eval( + compile(ast.Expression(sql_call.args[0]), '', 'eval'), + {'DATA_LOCATION': 'main'}, + ) + # DuckDB cursors do not share temporary tables across connections. + connection.execute(sql.replace('TEMPORARY ', '')) + joined = connection.execute( + 'SELECT "SalesOrderDetailID", "CustomerID", "OrderDate" ' + 'FROM "SDM_SalesOrderDetail"' + ).fetchall() + assert joined == [(1, 1, datetime.datetime(2024, 1, 1))] + + source = DuckDBSource(connection) + namespace = { + 'session': object(), + 'DATABASE': 'DB', + 'SCHEMA': 'PUBLIC', + 'SnowflakeSource': lambda session, *, database, schema: source, + 'Schema': Schema, + 'Relationship': Relationship, + } + for marker in ('source = SnowflakeSource(', "schema['products']"): + code = next(item for item in sources if item.startswith(marker)) + exec(compile(code, '', 'exec'), namespace) + schema = namespace['schema'] + schema.check_consistent() + assert schema.source is source + assert all( + isinstance(table, SourceTable) for table in schema.tables.values() + ) + assert set(schema.tables) == { + 'customers', + 'products', + 'sales_order_headers', + 'sales_order_details', + } + assert schema['sales_order_details'].time_column == 'OrderDate' + assert schema['products']['Name'].stype == 'categorical' + assert schema['sales_order_details']['OrderQty'].stype == 'numerical' + finally: + connection.close() + + +def test_lowercase_namespace_matches_unquoted_snowflake_setup(): + cells = json.loads(NOTEBOOK.read_text())['cells'] + configuration = next( + ''.join(cell['source']) + for cell in cells + if "DATABASE = ''" in ''.join(cell['source']) + ) + configuration = configuration.replace('', 'example_db').replace( + '', 'example_schema' + ) + # Exercise the configuration without requiring Snowpark or a live session. + assignments = ast.Module( + body=[ + node + for node in ast.parse(configuration).body + if isinstance(node, ast.Assign) + ], + type_ignores=[], + ) + namespace = {'get_active_session': lambda: object()} + exec(compile(assignments, '', 'exec'), namespace) + assert namespace['DATABASE'] == 'EXAMPLE_DB' + assert namespace['SCHEMA'] == 'EXAMPLE_SCHEMA' + assert namespace['DATA_LOCATION'] == 'EXAMPLE_DB.EXAMPLE_SCHEMA'