diff --git a/README.md b/README.md index 0a52df445..3d45f6384 100644 --- a/README.md +++ b/README.md @@ -16,13 +16,47 @@ **An open accelerator for context generation over structured data.** -NVIDIA Kumo Engine prepares connected records and their history for predictive modeling at scale. Its Rust-based DiskGraph engine indexes data on local SSDs and retrieves time-correct neighborhoods for training and batch prediction, so teams can run structured data models over enterprise-scale datasets at lower cost by feeding the GPU from NVMe instead of RAM. +NVIDIA Kumo Engine prepares connected records and their history for predictive modeling at scale. Its Rust-based DiskGraph engine indexes data on local SSDs and retrieves time-correct neighborhoods for training and batch prediction, without keeping the full graph index in RAM. -Define your tables, relationships, and prediction task in Python. Kumo Engine handles ingestion from the data lake, graph construction, context sampling, training, evaluation, and prediction for a wide range of predictive use cases. Examples include recommendations, fraud detection, churn prediction, propensity modeling, and risk scoring, among many others. +Define your tables, relationships, and prediction task in Python. Kumo Engine handles ingestion, graph construction, context sampling, training, evaluation, and prediction. + +For example, to predict whether a customer will order in the next 30 days, Kumo Engine connects the customer to their order history and samples eligible records as model inputs. + + + + Connected tables become model context: customer and order tables define relationships, matching keys connect individual records, and selected historical records form model inputs. Future outcomes stay outside the inputs. + + +*Schematic example: the model plan determines sampling, and future outcomes remain outside the model inputs.* **Benchmark highlight:** [One million three-hop neighborhoods in 19 seconds](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/resources/benchmarks). -[Documentation](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/get-started-with-kumo-engine/overview) · [Installation](#installation) · [Quick Tour](#quick-tour) · [Benchmarks](#benchmarks) +**[Documentation](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/get-started-with-kumo-engine/overview)**
+**[Installation](#installation)**
+**[Quick Tour](#quick-tour)**
+**[Benchmarks](#benchmarks)** + +## Key Capabilities + +- **Context at scale:** Retrieve connected records from an NVMe-backed graph index without keeping the full index in RAM. +- **End-to-end workflows:** Train and evaluate models, run batch predictions, generate embeddings, and save reloadable model bundles. +- **Production ready:** Ingest in parallel from S3, Snowflake, Databricks, BigQuery, and GCS. +- **Existing data and infrastructure:** Read files or connect to cloud storage and warehouses using existing compute and credentials. +- **Compatible with agentic workflows:** Agent skills let AI agents set up and run Kumo Engine workflows. + +## Quick Tour + +See how to install Kumo Engine in a notebook, connect customer and order tables, and render their schema. + + + + + A notebook user flow shows a PyPI install command, defines customer and order tables, connects their foreign keys, and renders the table schema in a cell. + + +Predict which customers will order in the next 30 days using their connected order history. Try the [Python quickstart](packages/kumo-engine/README.md#quickstart) with the bundled sample dataset. + +For complete walkthroughs, follow the [Google Colab](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/user-code-guide/code-examples/google-colab-notebook), [Snowflake](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/user-code-guide/code-examples/snowflake-notebook), or [Databricks](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/user-code-guide/code-examples/databricks-notebook) notebook. Explore more tasks in [Predictive Query Examples](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/get-started-with-kumo-engine/workflows-and-solutions/predictive-query-examples). ## Part of NVIDIA Kumo @@ -51,109 +85,6 @@ The base package reads local CSV and Parquet files. To read from a remote data s Refer to [prerequisites](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/get-started-with-kumo-engine/prerequisites) and [GPU sizing](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/get-started-with-kumo-engine/gpu-sizing) for a compatible runtime and storage capacity. Follow the setup guide for [Linux](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/get-started-with-kumo-engine/installation/linux-machine-deployment), [Snowflake notebooks](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/get-started-with-kumo-engine/installation/snowflake-notebook), or [Databricks notebooks](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/get-started-with-kumo-engine/installation/databricks-notebook). For an ARM64 source build, refer to the [DGX Spark guide](https://github.com/NVIDIA/kumo-engine/blob/main/packages/kumo-engine/dgx-spark.md). -## Key Capabilities - -- **Context at scale:** Retrieve connected records from an NVMe-backed graph index without keeping the full index in RAM. -- **End-to-end workflows:** Train and evaluate models, run batch predictions, generate embeddings, and save reloadable model bundles. -- **Production ready:** Ingest in parallel from S3, Snowflake, Databricks, BigQuery, and GCS. -- **Existing data and infrastructure:** Read files or connect to cloud storage and warehouses using existing compute and credentials. -- **Compatible with agentic workflows:** Agent skills let AI agents set up and run Kumo Engine workflows. - -## Quick Tour - -Predict which customers will order in the next 30 days. This example creates two small synthetic CSV tables, builds their graph, trains a classifier, and prints predictions. It uses the installed `kumo.engine` API without a Kumo account or warehouse credentials. - -First, create the sample data in a new working directory: - -
-Create 1,000 customers and 12,000 synthetic orders - -```python -import csv -from datetime import datetime, timedelta -from pathlib import Path -from random import Random - -data = Path("kumo-readme-data") -data.mkdir(exist_ok=False) -rng = Random(42) - -with (data / "customers.csv").open("w", newline="") as f: - writer = csv.writer(f) - writer.writerow(["customer_id", "segment"]) - writer.writerows( - (i, rng.choice(["consumer", "business"])) for i in range(1000) - ) - -with (data / "orders.csv").open("w", newline="") as f: - writer = csv.writer(f) - writer.writerow(["order_id", "customer_id", "order_time"]) - for i in range(12000): - customer_id = rng.randrange(1000) - time = datetime(2025, 1, 1) + timedelta( - seconds=rng.randrange(365 * 24 * 60 * 60) - ) - writer.writerow([i, customer_id, time.strftime("%Y-%m-%d %H:%M:%S")]) -``` - -
- -Run the following in the same working directory after completing setup. Replace the scratch path with a writable directory on your **local SSD**, separate from the sample-data directory. - -```python -from pathlib import Path - -from kumo.engine import ( - Edge, FileConnector, Graph, PredictiveQuery, Table, Trainer, configure, -) -from kumoapi.model_plan import RunMode -from kumoapi.typing import Dtype - -configure(scratch_dir="/path/to/local-ssd/kumo-scratch") -files = FileConnector(root_dir=str(Path("kumo-readme-data").resolve())) -customers = Table.from_source_table( - files["customers"], primary_key="customer_id" -) -orders = Table.from_source_table( - files["orders"], primary_key="order_id", time_column="order_time" -).with_column( - "order_time", dtype=Dtype.time, timestamp_format="%Y-%m-%d %H:%M:%S" -) - -graph = Graph( - tables={"customers": customers, "orders": orders}, - edges=[Edge( - src_table="orders", fkey="customer_id", dst_table="customers" - )], -) -snapshot = graph.snapshot() - -query = PredictiveQuery( - graph, - "PREDICT COUNT(orders.*, 0, 30, days) > 0 " - "FOR EACH customers.customer_id", -) -training_table = query.generate_training_table(snapshot) -plan = query.suggest_model_plan(snapshot, run_mode=RunMode.FAST) -plan.training_job.num_experiments = 1 -plan.training_job.refit_trainval = False -plan.training_job.refit_full = False -plan.optimization.max_epochs = 3 - -trainer = Trainer(plan) -trained = trainer.fit(training_table, snapshot=snapshot) -predictions = trainer.predict( - query.generate_prediction_table(snapshot), snapshot=snapshot, - binary_classification_threshold=0.5, -) -print(predictions.frame.head()) -print("Saved model bundle:", trained.bundle_dir) -``` - -The prediction frame contains entity identifiers and model outputs. These scores cover a future 30-day window from the snapshot's latest event time. Keep the saved bundle and any results you need on durable storage. - -For a complete walkthrough with data preparation, diagnostics, and retained outputs, follow the [Google Colab](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/user-code-guide/code-examples/google-colab-notebook), [Snowflake](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/user-code-guide/code-examples/snowflake-notebook), or [Databricks](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/user-code-guide/code-examples/databricks-notebook) notebook. Explore more tasks in [Predictive Query Examples](https://docs.nvidia.com/sdgm/kumo-engine/1.0.0/get-started-with-kumo-engine/workflows-and-solutions/predictive-query-examples). - ## Benchmarks ### Context Generation on a 50-Billion-Node Graph diff --git a/assets/kumo-notebook-demo.gif b/assets/kumo-notebook-demo.gif new file mode 100644 index 000000000..58ef31a06 Binary files /dev/null and b/assets/kumo-notebook-demo.gif differ diff --git a/assets/kumo-notebook-demo.png b/assets/kumo-notebook-demo.png new file mode 100644 index 000000000..fd61c5dcf Binary files /dev/null and b/assets/kumo-notebook-demo.png differ diff --git a/assets/model-context-mobile.svg b/assets/model-context-mobile.svg new file mode 100644 index 000000000..2478b37ff --- /dev/null +++ b/assets/model-context-mobile.svg @@ -0,0 +1 @@ +Connected tables become model contextThree schematic stages show customers and orders tables, customer 42 connected to three orders, and a selected historical context retaining customer 42 and two orders. The future prediction window is separate.Connected tablesbecome model context1 Define tablescustomerscustomer_id PKsegmentFK → PKordersorder_id PKcustomer_id FKorder_timeKeys define table relationships.2 Connect recordscustomer_id = 42C42O1O2O3Nov 8Nov 19Nov 27Each circle is one record.3 Build model contextSelected historical contextC42O2O3Sampling shown is illustrative.HistoryNext 30 daysPrediction timeFuture outcomes stay outside.Schematic exampleC = customer · O = order \ No newline at end of file diff --git a/assets/model-context.svg b/assets/model-context.svg new file mode 100644 index 000000000..c3c4f9ad6 --- /dev/null +++ b/assets/model-context.svg @@ -0,0 +1 @@ +Connected tables become model contextThree schematic stages show customers and orders tables, customer 42 connected to three orders, and a selected historical context retaining customer 42 and two orders. The future prediction window is separate.Connected tables become model contextWill this customer order in the next 30 days?1 Define tablescustomerscustomer_id PKsegmentFK → PKordersorder_id PKcustomer_id FKorder_timeKeys define table relationships.2 Connect recordscustomer_id = 42C42O1O2O3Nov 8Nov 19Nov 27Each circle is one record.3 Build model contextSelected historical contextC42O2O3Sampling shown is illustrative.HistoryNext 30 daysPrediction timeFuture outcomes stay outside.C = customer · O = orderSchematic example · IDs and dates are illustrative \ No newline at end of file