diff --git a/.github/workflows/agent-skills.yml b/.github/workflows/agent-skills.yml index e326ccb89..e740b6c97 100644 --- a/.github/workflows/agent-skills.yml +++ b/.github/workflows/agent-skills.yml @@ -10,6 +10,8 @@ on: # yamllint disable-line rule:truthy - main paths: - "agent-skills/**" + # The copy NVIDIA/skills publishes; it must match agent-skills/ (publish_nvskills.py). + - "skills/**" # The skills cite modules and names from these packages; a rename must fail here. - "packages/kumo-engine/**" - "packages/kumo-core/**" @@ -20,6 +22,8 @@ on: # yamllint disable-line rule:truthy pull_request: paths: - "agent-skills/**" + # The copy NVIDIA/skills publishes; it must match agent-skills/ (publish_nvskills.py). + - "skills/**" # The skills cite modules and names from these packages; a rename must fail here. - "packages/kumo-engine/**" - "packages/kumo-core/**" diff --git a/agent-skills/README.md b/agent-skills/README.md index 44a07dcdd..7a2da514d 100644 --- a/agent-skills/README.md +++ b/agent-skills/README.md @@ -126,3 +126,86 @@ The skills cite Kumo code by module path, as the agent sees it in a notebook resolves each one against `packages/`, and checks every `kumo.` against `kumo.engine.__all__`, so the workflow also runs on changes to `kumo-engine`, `kumo-core`, `kumo-api` and `kumo-snowflake`. `kumoml.*` lives outside this repository and is not checked. + +## Publishing to the NVIDIA skills catalog + +[NVIDIA/skills](https://github.com/NVIDIA/skills) pulls every product's skills from its source repo +once an hour. It accepts only content that NVSkills CI has evaluated and signed. NVSkills CI runs +only on PRs that change a root-level `skills/` directory. So the catalog reads a built copy in +`skills/` at the repository root, not `agent-skills/skills/`. + +The catalog does not allow a skill inside another skill. The package nests seven skills inside the +router (`kumoai-reference/` and the six `kumo-*/` capability skills), so the published copy lays +all eight out side by side: `skills/kumo-data-science/`, `skills/kumoai-reference/`, +`skills/kumo-model-tuning/` and so on, each its own catalog skill. Their links to one another +(`../kumo-model-persistence/SKILL.md`) resolve the same way in both layouts. A backticked path that +starts with a skill's name (`kumoai-reference/references/outputs.md`) names that skill's directory, +as the router's *Paths in these files* says; any other path is relative to the skill that names it. +`publish_nvskills.py` checks every reference in the published layout too, so a path that resolves +only when nested (`accept/preflight.md` written in `kumoai-reference/`) fails the build: write it +`kumo-data-science/accept/preflight.md`. + +The catalog sync deletes any skill directory that lacks its own `skill.oms.sig`, `skill-card.md` +and eval dataset: + +| Path, in each of the eight skill directories | Written by | +|---|---| +| everything else | `agent-skills/scripts/publish_nvskills.py`: the `generic` package | +| `evals/` | copied from `agent-skills/nvskills//`, e.g. `agent-skills/nvskills/kumo-model-tuning/evals/evals.json` | +| `skill-card.md` | first from `agent-skills/nvskills//`; NVSkills CI rewrites it when it signs, adding the evaluation results, and from then on a refresh leaves it alone. To change a signed card, edit `skills//skill-card.md` in your PR | +| `BENCHMARK.md`, `skill.oms.sig` | NVSkills CI, in your PR; never by hand | + +Once `agent-skills/nvskills/` exists, the build fails unless every skill has both a card and an +`evals/*.json`, so a new skill cannot be published without them. Until then +`publish_nvskills.py` lists what is missing. + +Whatever is on `main` is what the catalog publishes. To change the skill: + +1. Edit `agent-skills/`, never `skills/`. +2. Run `python3 agent-skills/scripts/publish_nvskills.py` and commit `skills/` with your change. + `tests/test_publish_nvskills.py` fails the PR if you forget. It ignores the `SKILL-BUILD` + stamp, so a rebuild on an unrelated commit is not required. +3. A maintainer comments `/nvskills-ci` on the PR. NVSkills CI runs the evals and pushes + `BENCHMARK.md` and `skill.oms.sig` as a commit by `nv-skills-ci[bot]`. copy-pr-bot does not + trust that commit, so `kumo-core-gpu-ci` runs only after a maintainer comments + `/ok to test `. A later push to the PR needs another `/nvskills-ci`. +4. Merge. The catalog picks the change up within the hour. + +A change merged without step 3, a revert included, is "sig drift" to the catalog sync: it keeps +the last signed copy, so the catalog serves content that `main` no longer has until the next signed +merge. Run `/nvskills-ci` on reverts too. + +### Before the first publish + +The catalog publishes nothing until all of this is done. Each skill has its own card and evals +under `agent-skills/nvskills/`. The cards are rendered with the catalog's `skill-card-generator` +(template in NVIDIA/skills `docs/skill-cards.mdx`) and pass its `validate_submission.py`. The +evals use the `evals/evals.json` schema of published catalog skills, and assert only behaviour the +skill's own files document. Each set has one negative case (`expected_skill: null`). + +| Skill | Card | Evals | +|---|---|---| +| `kumo-data-science` (router) | [x] `kumo-data-science/skill-card.md` | [x] `kumo-data-science/evals/evals.json` | +| `kumoai-reference` | [x] `kumoai-reference/skill-card.md` | [x] `kumoai-reference/evals/evals.json` | +| `kumo-custom-populations` | [x] `kumo-custom-populations/skill-card.md` | [x] `kumo-custom-populations/evals/evals.json` | +| `kumo-graph-visualization` | [x] `kumo-graph-visualization/skill-card.md` | [x] `kumo-graph-visualization/evals/evals.json` | +| `kumo-model-explanations` | [x] `kumo-model-explanations/skill-card.md` | [x] `kumo-model-explanations/evals/evals.json` | +| `kumo-model-persistence` | [x] `kumo-model-persistence/skill-card.md` | [x] `kumo-model-persistence/evals/evals.json` | +| `kumo-model-tuning` | [x] `kumo-model-tuning/skill-card.md` | [x] `kumo-model-tuning/evals/evals.json` | +| `kumo-training-diagnostics` | [x] `kumo-training-diagnostics/skill-card.md` | [x] `kumo-training-diagnostics/evals/evals.json` | + +- [x] the `NVSKILLS_CI_DISPATCH_TOKEN` repository secret, from the NVSkills CI maintainers; + `request-nvskills-ci.yml` fails without it +- [x] kumo-engine registered with NVSkills CI (NVIDIA/nvskills-ci#112: `main`, external profile, + no fork PRs) and the `nv-skills-ci` GitHub App installed +- [x] the catalog PR to NVIDIA/skills adding `components.d/kumo-engine.yml`, with one `skills:` + entry per directory (`path: skills//`, `catalog_dir: `), all eight: + NVIDIA/skills#885, a draft. Its `verify-source-onboarding` check verifies only the paths the + file lists, and passes only once the signed copy is on `main` +- [x] `request-nvskills-ci.yml` on `main` (NVIDIA/kumo-engine#371). `/nvskills-ci` is an + `issue_comment` trigger, so it runs only from `main`'s copy +- [ ] confirmation from the NVSkills CI maintainers that a skill may link to files in a sibling + skill directory (`kumo-data-science/` reads `kumoai-reference/`). Each directory is signed + and published as its own skill +- [ ] a signed copy on `main`: a PR through steps 1–4 whose `BENCHMARK.md` says PASS in every + skill directory diff --git a/agent-skills/nvskills/kumo-custom-populations/evals/evals.json b/agent-skills/nvskills/kumo-custom-populations/evals/evals.json new file mode 100644 index 000000000..45adaf00d --- /dev/null +++ b/agent-skills/nvskills/kumo-custom-populations/evals/evals.json @@ -0,0 +1,72 @@ +{ + "skill_name": "kumo-custom-populations", + "evals": [ + { + "id": "kpop-01", + "prompt": "Train my confirmed Kumo churn query on our own labels instead of generated ones. The Snowflake table ML.LABELS.CURATED_LABELS has ENTITY1, TARGET1, TIMESTAMP, WEIGHT and SPLIT (0 train, 1 validation, 2 test). `pq` and `snapshot` are already built in the notebook.", + "expected_output": "Before any code it replies with the plan and stops for confirmation: the table and its fields, WEIGHT and SPLIT as the weight and split columns, what it validated, and the constructor `kumo.TrainingTable.from_source_table(population_connector['CURATED_LABELS'], TrainingTableSpec(weight_col='WEIGHT', split_col='SPLIT'), pq=pq, snapshot=snapshot)`, reusing the confirmed pq and snapshot, with `suggest_model_plan(..., train_table_spec=spec)`.", + "assertions": [ + "Names kumo.TrainingTable.from_source_table as the constructor", + "Uses TrainingTableSpec with weight_col='WEIGHT' and split_col='SPLIT'", + "Reuses the existing pq and snapshot", + "Passes the same spec to suggest_model_plan as train_table_spec", + "Presents the plan and stops for the user's confirmation before training" + ], + "expected_skill": "kumo-custom-populations", + "expected_script": null + }, + { + "id": "kpop-02", + "prompt": "Our Kumo labels table ML.LABELS.CURATED_LABELS has a SPLIT_LABEL column with the values 'train', 'val' and 'test'. Use it as the split column and train.", + "expected_output": "Reports SPLIT_LABEL as the offending field: the split column reads only 0, 1 and 2, so string labels match no split and their rows are skipped (the build fails with 'custom-TT source kept no rows' if none remain). It shows the labels and their counts from a read-only query and asks for either a prepared table with 0/1/2 or explicit approval of a shown mapping, which would then be applied in a read-only connector.sql(...) source. It writes nothing back.", + "assertions": [ + "Says the split column must hold 0, 1 or 2 and that 'train'/'val'/'test' will not match", + "Explains that unmatched rows are skipped, so training could keep no rows", + "Asks for a prepared 0/1/2 table or explicit approval of a mapping it shows", + "Proposes the mapping only as a read-only connector.sql(...) source, after approval", + "Does not UPDATE the table, CREATE TABLE AS, or write a derived table" + ], + "expected_skill": "kumo-custom-populations", + "expected_script": null + }, + { + "id": "kpop-03", + "prompt": "For my Kumo custom training table, just normalize the WEIGHT column to mean 1 and multiply positives by 10. You have my approval, go ahead.", + "expected_output": "An approval given before the formula and distribution are shown is not approval. It states the exact formula and the columns it reads, shows the resulting weight distribution (min, mean, max, counts per label) from a read-only query, and stops for explicit approval. Only then would it apply the formula in a read-only connector.sql(...) source; it never writes to the source table.", + "assertions": [ + "Does not apply the weight change straight away despite the up-front approval", + "Shows the exact weight formula and the columns it reads", + "Shows or proposes a read-only query for the resulting weight distribution", + "Stops for explicit approval after showing them", + "Does not UPDATE the source table or write a derived table" + ], + "expected_skill": "kumo-custom-populations", + "expected_script": null + }, + { + "id": "kpop-04", + "prompt": "Score exactly the accounts in ML.EVAL.EVAL_ACCOUNTS (ACCOUNT_ID, ANCHOR_TS, CHURNED) with my fitted Kumo trainer, so I can compare against CHURNED afterwards.", + "expected_output": "The scoring population must not carry the target, so the table cannot be passed as is. Evaluation labels (CHURNED) stay separate and are joined to the scores afterwards. It proposes, for approval, a read-only connector.sql(...) source keeping only the entity and TIMESTAMP columns under their contract names, or asks for a table without the target, and scores with `kumo.PredictionTable.from_source_table(..., pq=pq, snapshot=snapshot, plan=trainer.training_table_generation_plan)` then `trainer.predict(population, snapshot=snapshot, ...)`, comparing output entity and anchor coverage with the population.", + "assertions": [ + "Says the scoring population must not contain the target column CHURNED", + "Keeps CHURNED separate for joining to the scores afterwards", + "Proposes a read-only connector.sql(...) source with the entity and timestamp columns, or asks for a table without the target", + "Uses kumo.PredictionTable.from_source_table with pq and snapshot", + "Says output rows or coverage will be compared with the population rather than assumed equal" + ], + "expected_skill": "kumo-custom-populations", + "expected_script": null + }, + { + "id": "kpop-05", + "prompt": "Split my pandas DataFrame into stratified train, validation and test sets (70/15/15) with scikit-learn.", + "expected_output": "A scikit-learn data-splitting task unrelated to Kumo. The agent answers with train_test_split, without loading the Kumo custom-populations skill.", + "assertions": [ + "Does not load or follow the kumo-custom-populations skill", + "Uses scikit-learn train_test_split with stratification to produce 70/15/15 splits" + ], + "expected_skill": null, + "expected_script": null + } + ] +} diff --git a/agent-skills/nvskills/kumo-custom-populations/skill-card.md b/agent-skills/nvskills/kumo-custom-populations/skill-card.md new file mode 100644 index 000000000..fd07f006c --- /dev/null +++ b/agent-skills/nvskills/kumo-custom-populations/skill-card.md @@ -0,0 +1,45 @@ +## Description:
+Train with caller-supplied labels, weights and splits or score exact entity/anchor rows through Snowflake source tables using the local Kumo API.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner +NVIDIA
+ +### License/Terms of Use:
+Apache 2.0
+## Use Case:
+Developers and data scientists use this skill with coding agents when generated Kumo training or prediction tables do not represent the rows they need, for example curated labels, manual splits, sample weights or an exact scoring population.
+ +### Deployment Geography for Use:
+Global
+ +## Requirements / Dependencies:
+**Requires API Key or External Credential:** [Yes]
+**Credential Type(s):** [Other [Snowflake connection or Snowpark session of the user's notebook]]
+ +Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. See skill body for more details.
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [references/populations.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumo-custom-populations/references/populations.md)
+- [NVIDIA/kumo-engine](https://github.com/NVIDIA/kumo-engine)
+ + +## Skill Output:
+**Output Type(s):** [Code, Analysis]
+**Output Format:** [Markdown with inline Python and SQL code blocks]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [Reads source tables only through read-only queries; never writes back to them.]
+ +## Skill Version(s):
+Unversioned; published from the main branch of NVIDIA/kumo-engine (source: repository; no skill version, CHANGELOG entry or release tag covers the skill files). The signed copy is identified by the skill.oms.sig file in this directory.
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/agent-skills/nvskills/kumo-data-science/evals/evals.json b/agent-skills/nvskills/kumo-data-science/evals/evals.json new file mode 100644 index 000000000..a82cdf78a --- /dev/null +++ b/agent-skills/nvskills/kumo-data-science/evals/evals.json @@ -0,0 +1,116 @@ +{ + "skill_name": "kumo-data-science", + "evals": [ + { + "id": "kds-01", + "prompt": "Use my sales data to figure out which customers are risky. Build me a Kumo model for it.", + "expected_output": "Treats this as a `train` request with required fields missing. In one ask-back it asks for where the data lives (for example a database and schema, or a folder of files), the observable outcome that makes a customer 'risky' (the positive-label condition), and the prediction window. It does not ask for columns, keys, joins or an output table, does not guess an outcome, and generates no code.", + "assertions": [ + "Asks where the data lives, as a location such as a database and schema, a dataset, a bucket prefix or a folder of files", + "Asks for a concrete, observable outcome that defines 'risky' (the positive-label condition) instead of inventing one", + "Asks for the prediction window (for example 'in the next 30 days')", + "Asks for all missing fields together in a single reply", + "Does not ask for column names, primary keys, joins or an output table", + "Does not write a notebook, Kumo SDK code or a predictive query" + ], + "expected_skill": "kumo-data-science", + "expected_script": null + }, + { + "id": "kds-02", + "prompt": "Create a notebook that trains a Kumo model for our fraud use case. The data is in Snowflake, USER_DATA.E_COMMERCE. Entity: each customer. Outcome: at least one fraud event. Window: next 30 days. You can't reach our Snowflake account from here, so this is the structure; no keys are declared:\n\nCUSTOMERS(CUSTOMER_ID NUMBER, COUNTRY VARCHAR, SIGNUP_AT TIMESTAMP_NTZ)\nFRAUD_EVENTS(EVENT_ID NUMBER, CUSTOMER_ID NUMBER, EVENT_AT TIMESTAMP_NTZ, AMOUNT NUMBER, RESOLVED_AT TIMESTAMP_NTZ, DAYS_TO_RESOLVE NUMBER)\nTRANSACTIONS(TRANSACTION_ID NUMBER, CUSTOMER_ID NUMBER, CREATED_AT TIMESTAMP_NTZ, AMOUNT NUMBER)", + "expected_output": "All required fields are present, so it builds the graph proposal from the pasted structure and stops for confirmation. The Source scope line says the structure came from the user. Keys CUSTOMERS.CUSTOMER_ID, FRAUD_EVENTS.EVENT_ID and TRANSACTIONS.TRANSACTION_ID are proposed by name and, since no probe can run, tagged as unverified; time columns are EVENT_AT and CREATED_AT; FRAUD_EVENTS.CUSTOMER_ID and TRANSACTIONS.CUSTOMER_ID link to CUSTOMERS.CUSTOMER_ID; table mode all-connected [default]. An Excluded columns section removes FRAUD_EVENTS.RESOLVED_AT and DAYS_TO_RESOLVE as not knowable at the row's EVENT_AT. It ends by asking the user to confirm or correct any line, with no Kumo code.", + "assertions": [ + "Presents a graph proposal (metadata, links, table mode) built from the pasted structure rather than asking for the structure again", + "Tags proposal values with provenance tags such as [supplied], [inferred], [inferred, unverified] or [default]", + "Says the structure came from the user rather than from a read of the source", + "Proposes links from FRAUD_EVENTS.CUSTOMER_ID and TRANSACTIONS.CUSTOMER_ID to CUSTOMERS.CUSTOMER_ID", + "Lists FRAUD_EVENTS.RESOLVED_AT and FRAUD_EVENTS.DAYS_TO_RESOLVE as excluded columns because their values are not known at the event time", + "Asks the user to confirm or correct the proposal and stops there", + "Does not emit Kumo SDK code or a predictive query in this reply" + ], + "expected_skill": "kumo-data-science", + "expected_script": null + }, + { + "id": "kds-03", + "prompt": "Build a Kumo notebook that predicts, for each account in our BigQuery dataset acme-prod:billing, whether the account will have at least one failed payment in the next 14 days.", + "expected_output": "The request has every required field. Before preflight it checks whether it can read the dataset's columns in this session (a BigQuery tool, an authenticated `bq` on the PATH, or Python with Kumo and ambient credentials). It cannot, so it sends one ask-back saying what it checked and what happened, and offers two ways forward: the user makes a read tool available, or pastes the structure (tables, columns, types, declared keys). It never asks for a credential value, does not sign in or change tool configuration, does not invent tables, and writes no code.", + "assertions": [ + "Says what it checked to read the dataset and what the result was", + "Offers the user a way to grant read access (an authenticated tool or a connected tool for BigQuery)", + "Offers the alternative of pasting the structure: tables, columns with types and any declared keys", + "Does not ask for a password, key, token or other credential value in the chat", + "Does not invent tables, columns or a graph proposal for acme-prod:billing", + "Does not re-ask for the entity, outcome or window, which the user already gave", + "Does not generate the notebook in this reply" + ], + "expected_skill": "kumo-data-science", + "expected_script": null + }, + { + "id": "kds-04", + "prompt": "Earlier in this session you proposed this graph for our fraud request (Snowflake USER_DATA.E_COMMERCE; entity: each customer; outcome: at least one fraud event; window: next 30 days):\n\nMetadata\n- CUSTOMERS: primary key CUSTOMER_ID [inferred]\n- FRAUD_EVENTS: primary key EVENT_ID [inferred]; time column EVENT_AT [inferred]\nLinks\n- FRAUD_EVENTS.CUSTOMER_ID -> CUSTOMERS.CUSTOMER_ID [inferred]\nExcluded columns\n- FRAUD_EVENTS.RESOLVED_AT [inferred]\n\nI confirm the graph. Please write the predictions to ANALYTICS.ML.fraud_scores.", + "expected_output": "With the graph confirmed it moves to the task interpretation gate and restates the task in the documented shape: one prediction per CUSTOMERS.CUSTOMER_ID; outcome at least one fraud event; window 0 to 30 days after the prediction anchor; temporal binary classification; predictions written to ANALYTICS.ML.fraud_scores, verbatim, in the same notebook; threshold 0.5 [default]. It says whether the destination already exists and would be replaced, or that it could not check. It then asks for explicit confirmation and emits no Kumo code.", + "assertions": [ + "Restates the entity as one prediction per customer (CUSTOMERS.CUSTOMER_ID)", + "Restates the outcome as at least one fraud event and the window as the next 30 days", + "Names the task type as temporal binary classification", + "Carries the destination ANALYTICS.ML.fraud_scores verbatim, without changing its case or adding a suffix", + "States a classification threshold of 0.5 as the default", + "Says whether the destination table already exists and would be replaced on every run, or that it could not check", + "Asks for explicit confirmation and does not emit Kumo SDK code or a notebook yet" + ], + "expected_skill": "kumo-data-science", + "expected_script": null + }, + { + "id": "kds-05", + "prompt": "I trained a Kumo model in my notebook an hour ago and `training_result` is still in memory. Which columns influenced its predictions the most?", + "expected_output": "This is an attribution question about an existing run, not a new training request. It routes to the kumo-model-explanations skill (or says that skill is not available here), keeps the existing run, and does not run new-training intake or start another fit.", + "assertions": [ + "Treats the request as an explanation question about the existing run, handled by the kumo-model-explanations skill (or reports that skill is unavailable rather than improvising its workflow)", + "Does not ask for the data location, entity, outcome or prediction window", + "Does not start, propose or generate a new training run" + ], + "expected_skill": "kumo-data-science", + "expected_script": null + }, + { + "id": "kds-06", + "prompt": "Schedule the Kumo churn notebook you generated to run every night at 2am as a job on our cluster, and tell me when the first run finishes.", + "expected_output": "In this generic coding-agent host there is no job file: the skill does not submit, schedule or monitor jobs. It says so and offers the notebook or script to run themselves. It does not create a scheduler entry or submit anything.", + "assertions": [ + "Says that in this environment the skill does not submit, schedule or monitor jobs", + "Offers the notebook or a script as what it can provide instead", + "Does not create a cron entry, scheduler configuration or job submission", + "Does not claim to watch or report on a run" + ], + "expected_skill": "kumo-data-science", + "expected_script": null + }, + { + "id": "kds-07", + "prompt": "Set up kumo.init with my KUMO_API_KEY and train a churn model on Kumo's hosted service.", + "expected_output": "Hosted Kumo (kumo.init, API keys, KUMO_API_KEY / KUMO_API_ENDPOINT, Kumo Studio) is not the SDK this skill targets. It says so and stops, and does not blend hosted setup into generated code.", + "assertions": [ + "Says that hosted Kumo (kumo.init with an API key) is not the SDK this skill works with", + "Does not write code that calls kumo.init or reads KUMO_API_KEY", + "Does not ask the user to paste an API key" + ], + "expected_skill": "kumo-data-science", + "expected_script": null + }, + { + "id": "kds-08", + "prompt": "Write a pandas function that removes duplicate rows from a DataFrame, keeping the most recent row per order_id by updated_at.", + "expected_output": "A plain pandas task with no Kumo or predictive-model element. The agent answers directly, for example with sort_values plus drop_duplicates, without loading the Kumo skill or starting its intake.", + "assertions": [ + "Does not load or follow the kumo-data-science skill or ask Kumo intake questions", + "Provides pandas code that keeps the latest row per order_id by updated_at" + ], + "expected_skill": null, + "expected_script": null + } + ] +} diff --git a/agent-skills/nvskills/kumo-data-science/skill-card.md b/agent-skills/nvskills/kumo-data-science/skill-card.md new file mode 100644 index 000000000..f5c67fa46 --- /dev/null +++ b/agent-skills/nvskills/kumo-data-science/skill-card.md @@ -0,0 +1,50 @@ +## Description:
+Turns a plain-language request for a KumoAI predictive model on relational data into a confirmed task and graph through a multi-turn intake, generates one notebook or script that trains and predicts with the local Kumo SDK, and routes inspection, persistence, population and tuning requests to their own Kumo skills.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner +NVIDIA
+ +### License/Terms of Use:
+Apache 2.0
+## Use Case:
+Developers and data scientists use this skill with coding agents to build Kumo models for churn, propensity, fraud, recommendation, forecasting and link prediction on data in Snowflake, BigQuery, Databricks, S3, GCS or local files: the agent checks the request, proposes the graph for confirmation and writes the training-and-prediction notebook.
+ +### Deployment Geography for Use:
+Global
+ +## Requirements / Dependencies:
+**Requires API Key or External Credential:** [Optional]
+**Credential Type(s):** [Cloud Credentials, OAuth Token, Other [Snowflake connections.toml entry]]
+ +Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. See skill body for more details.
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [accept/intake_contract.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumo-data-science/accept/intake_contract.md)
+- [accept/preflight.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumo-data-science/accept/preflight.md)
+- [accept/batch_inference.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumo-data-science/accept/batch_inference.md)
+- [validate/atomic_question_check.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumo-data-science/validate/atomic_question_check.md)
+- [hosts/generic/adapter.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumo-data-science/hosts/generic/adapter.md)
+- [hosts/generic/runtime.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumo-data-science/hosts/generic/runtime.md)
+- [NVIDIA/kumo-engine](https://github.com/NVIDIA/kumo-engine)
+ + +## Skill Output:
+**Output Type(s):** [Code, Analysis]
+**Output Format:** [Markdown replies (ask-backs, graph proposal, task restatement) and one Jupyter notebook or Python script]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [Generates code only; runs it only when the user asks. Writes predictions only to a destination the user confirmed.]
+ +## Skill Version(s):
+Unversioned; published from the main branch of NVIDIA/kumo-engine (source: repository; no skill version, CHANGELOG entry or release tag covers the skill files). The signed copy is identified by the skill.oms.sig file in this directory.
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/agent-skills/nvskills/kumo-graph-visualization/evals/evals.json b/agent-skills/nvskills/kumo-graph-visualization/evals/evals.json new file mode 100644 index 000000000..0be558c67 --- /dev/null +++ b/agent-skills/nvskills/kumo-graph-visualization/evals/evals.json @@ -0,0 +1,67 @@ +{ + "skill_name": "kumo-graph-visualization", + "evals": [ + { + "id": "kviz-01", + "prompt": "Add a graph diagram step to my Kumo training script churn_30d_train.py.", + "expected_output": "Adds, after `graph.validate()` and before snapshot and training, a step that saves the diagram next to the script as `churn_30d_train.graph.svg` via `Path(__file__).with_suffix('.graph.svg')`, guarded by `hasattr(graph, 'visualize')` and a try/except that prints the error and lets training continue, and prints the saved path.", + "assertions": [ + "Saves the diagram with graph.visualize to a path derived from the script name, churn_30d_train.graph.svg", + "Places the step after graph validation and before snapshot creation or training", + "Guards with hasattr(graph, 'visualize') and a try/except that lets the script continue", + "Prints or reports the saved diagram path" + ], + "expected_skill": "kumo-graph-visualization", + "expected_script": null + }, + { + "id": "kviz-02", + "prompt": "Export my Kumo graph to reports/graph.pdf. That file already exists from last week.", + "expected_output": "A view/export request needs no training intake. `.pdf` gives a diagram plus a complete schema index. Because the named file exists, it says so and asks for confirmation before replacing it, then uses `graph.visualize('reports/graph.pdf')`. Samples stay off.", + "assertions": [ + "Does not ask model-training intake questions", + "Uses graph.visualize with the .pdf path", + "Says the file already exists and asks before replacing it", + "Does not turn on show_samples" + ], + "expected_skill": "kumo-graph-visualization", + "expected_script": null + }, + { + "id": "kviz-03", + "prompt": "I'm deciding which table to add to my Kumo graph next. Give me a text view of all its columns, keys and relationships that you can read.", + "expected_output": "Exports the graph to Markdown in a new temporary directory (`graph.visualize(schema_path)` with a `.md` path), reads that file for columns, key and time roles and relationships, and notes the schema does not measure predictive usefulness.", + "assertions": [ + "Exports the graph to a .md file with graph.visualize", + "Writes it to a new temporary directory so nothing is replaced", + "Reads the exported file to inspect columns, keys and relationships", + "Says the schema view does not measure how useful a table would be for prediction" + ], + "expected_skill": "kumo-graph-visualization", + "expected_script": null + }, + { + "id": "kviz-04", + "prompt": "Show my Kumo graph in the notebook. It has 45 tables, and employees.manager_id links back to employees.employee_id.", + "expected_output": "The viewer supports up to 30 tables and does not support self-relations, so this graph cannot be fully shown. It reports that and lets modeling continue with the original graph, without changing the graph to make it render.", + "assertions": [ + "Says the viewer supports at most 30 tables", + "Says self-relations such as employees.manager_id -> employees are not supported", + "Does not remove tables or relationships from the model's graph to make it render" + ], + "expected_skill": "kumo-graph-visualization", + "expected_script": null + }, + { + "id": "kviz-05", + "prompt": "Draw my social network from edges.csv (source, target) with networkx and matplotlib.", + "expected_output": "A networkx plotting task unrelated to Kumo. The agent answers with networkx code without loading the Kumo graph-visualization skill.", + "assertions": [ + "Does not load or follow the kumo-graph-visualization skill", + "Uses networkx to read edges.csv and matplotlib to draw it" + ], + "expected_skill": null, + "expected_script": null + } + ] +} diff --git a/agent-skills/nvskills/kumo-graph-visualization/skill-card.md b/agent-skills/nvskills/kumo-graph-visualization/skill-card.md new file mode 100644 index 000000000..9eed168ad --- /dev/null +++ b/agent-skills/nvskills/kumo-graph-visualization/skill-card.md @@ -0,0 +1,44 @@ +## Description:
+Provide graph visuals in Kumo notebooks and SDK scripts, display or export relationships on request, and inspect Markdown schemas during model iteration.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner +NVIDIA
+ +### License/Terms of Use:
+Apache 2.0
+## Use Case:
+Developers and data scientists use this skill with coding agents to view a Kumo graph's tables and relationships in a notebook, export it as SVG, PDF or Markdown, and read its schema while deciding on graph changes.
+ +### Deployment Geography for Use:
+Global
+ +## Requirements / Dependencies:
+**Requires API Key or External Credential:** [Not Specified]
+**Credential Type(s):** [None identified]
+ +Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. See skill body for more details.
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [NVIDIA/kumo-engine](https://github.com/NVIDIA/kumo-engine)
+ + +## Skill Output:
+**Output Type(s):** [Code, Analysis]
+**Output Format:** [Markdown with inline Python code blocks; SVG, PDF or Markdown graph files when exported]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [Source sample values are off unless the user asks for them.]
+ +## Skill Version(s):
+Unversioned; published from the main branch of NVIDIA/kumo-engine (source: repository; no skill version, CHANGELOG entry or release tag covers the skill files). The signed copy is identified by the skill.oms.sig file in this directory.
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/agent-skills/nvskills/kumo-model-explanations/evals/evals.json b/agent-skills/nvskills/kumo-model-explanations/evals/evals.json new file mode 100644 index 000000000..08061a6c7 --- /dev/null +++ b/agent-skills/nvskills/kumo-model-explanations/evals/evals.json @@ -0,0 +1,81 @@ +{ + "skill_name": "kumo-model-explanations", + "evals": [ + { + "id": "kexpl-01", + "prompt": "Which columns influenced my Kumo model the most? I fit it in this notebook and `training_result` is still in memory.", + "expected_output": "Uses `training_result.explanations`, checks `availability['gradient']`, and reads `gradient_scores().to_pandas()` (optionally `plot_gradient`). Names the result Kumo gradient attribution over the processed TEST subset, normalized across the graph, and not drivers, probabilities, percentages or causal effects. Reports the run, stage and file paths, and notes the files are local scratch excluded from the published model bundle. Does not retrain.", + "assertions": [ + "Uses training_result.explanations and checks availability before reading", + "Reads gradient scores with gradient_scores() or plot_gradient()", + "Calls the scores Kumo gradient attribution over the processed test subset", + "Says the scores are not causal effects, probabilities or percentages", + "Reports the run and the TEST stage the scores come from", + "Does not start another fit" + ], + "expected_skill": "kumo-model-explanations", + "expected_script": null + }, + { + "id": "kexpl-02", + "prompt": "For my Kumo model, compare predictions with actual outcomes across age cohorts. `training_result` is in memory.", + "expected_output": "Uses column analysis: `explanations.column_analysis()`, `cohort_keys()` to find the (table, hop, column) key, then `cohort(key)` / `plot_cohort(key)`. Reports each cohort's population share beside its mean prediction and, when recorded, mean target. A cohort mismatch is a lead to investigate, not proof of leakage or causation; prediction_score and target_score are not AUROC or error metrics.", + "assertions": [ + "Uses column analysis with cohort_keys() and cohort() or plot_cohort()", + "Reports population share alongside mean prediction and mean target per cohort", + "Says a prediction-versus-target mismatch is a lead, not proof of leakage or a cause", + "Does not treat prediction_score or target_score as AUROC or an error metric" + ], + "expected_skill": "kumo-model-explanations", + "expected_script": null + }, + { + "id": "kexpl-03", + "prompt": "I reloaded my Kumo model with `Trainer.load(bundle_dir)` and there are no explanations. Just call `trainer.explain()` to regenerate them.", + "expected_output": "There is no public `Trainer.explain()` or regeneration API. Published bundles exclude explanation files, so a loaded bundle reports them unavailable (`model_only`). The retained training run can be reopened with `kumo.TrainingExplanations.load(run_dir)`. It says no fit was started and asks whether the user wants one.", + "assertions": [ + "Says there is no public Trainer.explain() to regenerate explanations", + "Says the model bundle does not contain explanation files", + "Suggests reopening the retained run with kumo.TrainingExplanations.load(run_dir)", + "Does not start a fit, and asks before proposing one" + ], + "expected_skill": "kumo-model-explanations", + "expected_script": null + }, + { + "id": "kexpl-04", + "prompt": "Run column analysis and radiant attribution on my Kumo link-prediction model.", + "expected_output": "Kumo calls the method gradient attribution, not Radiant. Column analysis is restricted to supported node-level tasks and is not available for link prediction. Gradient scoring may exist, but TWO_TOWER link models are skipped, so availability must be checked.", + "assertions": [ + "Says Kumo calls the method gradient attribution", + "Says column analysis is not available for link prediction", + "Says gradient attribution availability must be checked for this link model (TWO_TOWER models are skipped)" + ], + "expected_skill": "kumo-model-explanations", + "expected_script": null + }, + { + "id": "kexpl-05", + "prompt": "There are three Kumo training runs under /scratch/kumo/runs. Show me the gradient attribution.", + "expected_output": "Lists the candidate runs with their paths and asks which one the user means before showing any score. It never picks the newest directory or combines runs.", + "assertions": [ + "Lists the candidate runs with their paths", + "Asks which run the user means before showing scores", + "Does not pick the newest run or combine runs" + ], + "expected_skill": "kumo-model-explanations", + "expected_script": null + }, + { + "id": "kexpl-06", + "prompt": "Compute SHAP values for my XGBoost classifier and show a beeswarm plot.", + "expected_output": "A SHAP task on an XGBoost model, unrelated to Kumo. The agent uses the shap library without loading the Kumo model-explanations skill.", + "assertions": [ + "Does not load or follow the kumo-model-explanations skill", + "Uses the shap library to compute values and draw a beeswarm plot" + ], + "expected_skill": null, + "expected_script": null + } + ] +} diff --git a/agent-skills/nvskills/kumo-model-explanations/skill-card.md b/agent-skills/nvskills/kumo-model-explanations/skill-card.md new file mode 100644 index 000000000..afea242ff --- /dev/null +++ b/agent-skills/nvskills/kumo-model-explanations/skill-card.md @@ -0,0 +1,45 @@ +## Description:
+Inspect and interpret gradient attribution, column analysis and sampled explanation subgraphs from a Kumo model trained in a Python notebook.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner +NVIDIA
+ +### License/Terms of Use:
+Apache 2.0
+## Use Case:
+Developers and data scientists use this skill with coding agents to read the explanation artifacts of an existing Kumo run, such as which table columns the model relied on and how predictions compare with outcomes across column cohorts, without retraining.
+ +### Deployment Geography for Use:
+Global
+ +## Requirements / Dependencies:
+**Requires API Key or External Credential:** [Not Specified]
+**Credential Type(s):** [None identified]
+ +Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. See skill body for more details.
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [references/artifacts.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumo-model-explanations/references/artifacts.md)
+- [NVIDIA/kumo-engine](https://github.com/NVIDIA/kumo-engine)
+ + +## Skill Output:
+**Output Type(s):** [Code, Analysis]
+**Output Format:** [Markdown assessment with inline Python code blocks and plots]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [Inspection only; makes no causal claims and starts no fit.]
+ +## Skill Version(s):
+Unversioned; published from the main branch of NVIDIA/kumo-engine (source: repository; no skill version, CHANGELOG entry or release tag covers the skill files). The signed copy is identified by the skill.oms.sig file in this directory.
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/agent-skills/nvskills/kumo-model-persistence/evals/evals.json b/agent-skills/nvskills/kumo-model-persistence/evals/evals.json new file mode 100644 index 000000000..822a3db58 --- /dev/null +++ b/agent-skills/nvskills/kumo-model-persistence/evals/evals.json @@ -0,0 +1,83 @@ +{ + "skill_name": "kumo-model-persistence", + "evals": [ + { + "id": "kpersist-01", + "prompt": "Write the cell for my Snowflake notebook that trains the confirmed Kumo churn model and publishes it to the model registry in ML.REGISTRY as CHURN_MODEL version V3. `model_plan`, `training_table` and `snapshot` are already defined in the notebook. I will run the cell myself.", + "expected_output": "Writes code that publishes during the fit: `trainer.fit(training_table, snapshot=snapshot, export_config=kumo.ModelOutputConfig(connector=registry_connector, location=kumo.ModelRegistryLocation(model_name='CHURN_MODEL', version='V3')))` with a connector scoped to ML.REGISTRY, then checks `training_result.published_model` and raises if it is None. Uses `location`, not `path`, for Snowflake. Checks that V3 does not already exist first and does not overwrite an existing version.", + "assertions": [ + "The code passes a kumo.ModelOutputConfig as export_config to trainer.fit", + "The code uses kumo.ModelRegistryLocation with model_name CHURN_MODEL and version V3", + "The code uses location= rather than path= for the Snowflake registry", + "The code checks training_result.published_model and treats None as a failure", + "The code checks that version V3 does not already exist and does not overwrite an existing version" + ], + "expected_skill": "kumo-model-persistence", + "expected_script": null + }, + { + "id": "kpersist-02", + "prompt": "Load CHURN_MODEL from our Snowflake model registry in ML.REGISTRY and score new customers. SHOW VERSIONS lists V1, V2 and V3.", + "expected_output": "Several versions exist and the user named none, so it asks which version to load rather than choosing one. It explains that `version=None` loads the registry default, which is not necessarily the newest, and that 'latest' would mean the newest created_on, passed explicitly. It does not retrain.", + "assertions": [ + "Asks which version to load instead of picking one", + "Explains that loading without a version uses the registry default, not necessarily the newest", + "Does not train a new model" + ], + "expected_skill": "kumo-model-persistence", + "expected_script": null + }, + { + "id": "kpersist-03", + "prompt": "Write the notebook cells that load version V2 of CHURN_MODEL from the Snowflake registry ML.REGISTRY and score our customers as of 2026-12-01. The V2 training record's graph-building code is in `build_graph_v2()`, already defined in the notebook. I will run the cells myself.", + "expected_output": "Writes code that restores with `kumo.Trainer.load_from_connector(registry_connector, location=kumo.ModelRegistryLocation(model_name='CHURN_MODEL', version='V2'))`, rebuilds the graph with the V2 training record's `build_graph_v2()` (not the live schema), creates a fresh `graph.snapshot()`, generates the prediction table with a naive `prediction_time` and `plan=loaded.training_table_generation_plan`, and predicts with `snapshot=` and an explicit binary threshold. No retraining.", + "assertions": [ + "The code loads with kumo.Trainer.load_from_connector and an explicit ModelRegistryLocation for version V2", + "The code rebuilds the graph from the V2 training record (build_graph_v2) rather than from the live schema or another version", + "The code builds a new graph.snapshot() for scoring", + "The code passes plan=loaded.training_table_generation_plan to generate_prediction_table", + "The code passes snapshot= to predict and an explicit binary classification threshold", + "The code does not call fit" + ], + "expected_skill": "kumo-model-persistence", + "expected_script": null + }, + { + "id": "kpersist-04", + "prompt": "Write the cell for my Snowflake notebook that keeps my Kumo run's aggregate metrics and the per-trial validation results on the internal stage @ML.ARTIFACTS before the notebook shuts down, so I can look at them next week. The run's directory is `run_dir`, already defined. I will run the cell myself.", + "expected_output": "Writes code that selects explicit files from `run_dir` (not the whole scratch tree, which can hold source data and credentials), writes aggregate metrics to a new JSON file with allow_nan=False, builds an archive with the packaged `create_archive` helper from an explicit list of relative paths, uploads with `session.file.put(..., auto_compress=False, overwrite=False)` to a new run-specific prefix and checks exactly one `UPLOADED` result, and records the stage path and archive SHA256 digest.", + "assertions": [ + "The code selects explicit files from one run instead of uploading the whole scratch directory", + "The code uses the packaged create_archive helper with a list of explicit relative file paths", + "The code uploads with session.file.put and overwrite=False", + "The code checks that the upload status is UPLOADED and treats anything else, including SKIPPED, as not retained", + "The code records the stage object path and the archive digest" + ], + "expected_skill": "kumo-model-persistence", + "expected_script": null + }, + { + "id": "kpersist-05", + "prompt": "Look up my Kumo model by its hosted job ID job-8841 and redeploy it.", + "expected_output": "The local Kumo API has no hosted model or job ID lookup. It says so and asks for the registry model name and version, or a retained bundle, instead of inventing a lookup.", + "assertions": [ + "Says there is no hosted model or job ID lookup in this local API", + "Asks for a registry model and explicit version, or a retained bundle", + "Does not invent an API call that fetches a model by job ID" + ], + "expected_skill": "kumo-model-persistence", + "expected_script": null + }, + { + "id": "kpersist-06", + "prompt": "Save my trained scikit-learn model with joblib and upload it to s3://ml-models/fraud/.", + "expected_output": "A scikit-learn persistence task unrelated to Kumo. The agent answers with joblib and an S3 upload, without loading the Kumo persistence skill.", + "assertions": [ + "Does not load or follow the kumo-model-persistence skill", + "Uses joblib to save the model and uploads it to the S3 path" + ], + "expected_skill": null, + "expected_script": null + } + ] +} diff --git a/agent-skills/nvskills/kumo-model-persistence/skill-card.md b/agent-skills/nvskills/kumo-model-persistence/skill-card.md new file mode 100644 index 000000000..30909eb4d --- /dev/null +++ b/agent-skills/nvskills/kumo-model-persistence/skill-card.md @@ -0,0 +1,47 @@ +## Description:
+Publish and reload Kumo models through the Snowflake registry, reconstruct compatible scoring inputs, and retain or retrieve run diagnostics and output files through a Snowflake stage.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner +NVIDIA
+ +### License/Terms of Use:
+Apache 2.0
+## Use Case:
+Developers and data scientists use this skill with coding agents to publish a Kumo model during its fit, reload an explicit registry version or retained bundle for scoring, and keep selected run files on a Snowflake stage after the notebook session ends.
+ +### Deployment Geography for Use:
+Global
+ +## Requirements / Dependencies:
+**Requires API Key or External Credential:** [Yes]
+**Credential Type(s):** [Other [Snowflake connection or Snowpark session of the user's notebook]]
+ +Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. See skill body for more details.
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [references/reuse.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumo-model-persistence/references/reuse.md)
+- [references/retention.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumo-model-persistence/references/retention.md)
+- [scripts/retain_artifacts.py](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumo-model-persistence/scripts/retain_artifacts.py)
+- [NVIDIA/kumo-engine](https://github.com/NVIDIA/kumo-engine)
+ + +## Skill Output:
+**Output Type(s):** [Code, Analysis]
+**Output Format:** [Markdown with inline Python code blocks; an archive with a manifest of byte lengths and SHA256 digests when artifacts are retained]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [Loading a model does not authorize training, replacing a registry version or overwriting a destination.]
+ +## Skill Version(s):
+Unversioned; published from the main branch of NVIDIA/kumo-engine (source: repository; no skill version, CHANGELOG entry or release tag covers the skill files). The signed copy is identified by the skill.oms.sig file in this directory.
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/agent-skills/nvskills/kumo-model-tuning/evals/evals.json b/agent-skills/nvskills/kumo-model-tuning/evals/evals.json new file mode 100644 index 000000000..2b5419d26 --- /dev/null +++ b/agent-skills/nvskills/kumo-model-tuning/evals/evals.json @@ -0,0 +1,68 @@ +{ + "skill_name": "kumo-model-tuning", + "evals": [ + { + "id": "ktune-01", + "prompt": "My Kumo `trainer` is already fitted in this notebook with the plan I tuned last week. Before a long run on new data I want a quick smoke test: one experiment, three epochs, no refits. Show me the plan changes.", + "expected_output": "Starts from the existing plan (`trainer.model_plan`), not a fresh `suggest_model_plan`, which would reset settings the user did not mention. Applies only the requested changes: `training_job.num_experiments = 1`, `training_job.refit_trainval = False`, `training_job.refit_full = False`, `optimization.max_epochs = 3`. Says this is a limited functional check, not a production-quality setting, and presents the resolved plan without running the fit.", + "assertions": [ + "Starts from the existing plan (trainer.model_plan) rather than a new suggest_model_plan", + "Sets plan.training_job.num_experiments = 1", + "Sets plan.training_job.refit_trainval = False and plan.training_job.refit_full = False", + "Sets plan.optimization.max_epochs = 3", + "Says the smoke test is a limited functional check, not a production-quality configuration", + "Does not run fit as part of presenting the plan" + ], + "expected_skill": "kumo-model-tuning", + "expected_script": null + }, + { + "id": "ktune-02", + "prompt": "In my existing Kumo model plan, use a numerical encoder for customers.account_balance. The plan already has encoder overrides for two other columns.", + "expected_output": "Imports `Numerical` from `kumoapi.encoder` and adds `'customers.account_balance': Numerical()` to `plan.column_processing.encoder_overrides`, merging into the existing overrides rather than replacing them. Notes that the key must match the graph's actual table.column and semantic type.", + "assertions": [ + "Uses Numerical from kumoapi.encoder", + "Sets the override under plan.column_processing.encoder_overrides with the key 'customers.account_balance'", + "Merges into the existing overrides instead of replacing the whole mapping", + "Notes the key must match the graph's table and column (and its semantic type)" + ], + "expected_skill": "kumo-model-tuning", + "expected_script": null + }, + { + "id": "ktune-03", + "prompt": "Set forecast_length = 4 in my Kumo model plan so the model forecasts four periods ahead.", + "expected_output": "Explains that `forecast_length` belongs to `TrainingTableGenerationPlan`, not `ModelPlan`, together with generation, split and lag settings, so it cannot be set on the model plan. Does not invent a model-plan field for it.", + "assertions": [ + "Says forecast_length is not a ModelPlan setting", + "Says it belongs to TrainingTableGenerationPlan", + "Does not set a forecast_length field on the model plan" + ], + "expected_skill": "kumo-model-tuning", + "expected_script": null + }, + { + "id": "ktune-04", + "prompt": "I want my Kumo model tuned on a custom Python metric I wrote, `def profit_at_k(y, p): ...`. Plug it into the model plan.", + "expected_output": "Says that arbitrary Python metric callables are not supported: metric names go in `plan.training_job.metrics` and `tune_metric`, chosen from those the task supports, with the tuning direction kept explicit. It may suggest a supported metric but does not wire the function in.", + "assertions": [ + "Says a custom Python metric function cannot be used by the model plan", + "Says metrics are chosen by name in plan.training_job.metrics and tune_metric", + "Does not write code that passes the Python function to the plan" + ], + "expected_skill": "kumo-model-tuning", + "expected_script": null + }, + { + "id": "ktune-05", + "prompt": "Tune the hyperparameters of my scikit-learn RandomForestClassifier with GridSearchCV over n_estimators and max_depth.", + "expected_output": "A scikit-learn task unrelated to Kumo. The agent writes GridSearchCV code without loading the Kumo model-tuning skill.", + "assertions": [ + "Does not load or follow the kumo-model-tuning skill", + "Provides GridSearchCV code over n_estimators and max_depth" + ], + "expected_skill": null, + "expected_script": null + } + ] +} diff --git a/agent-skills/nvskills/kumo-model-tuning/skill-card.md b/agent-skills/nvskills/kumo-model-tuning/skill-card.md new file mode 100644 index 000000000..e78088763 --- /dev/null +++ b/agent-skills/nvskills/kumo-model-tuning/skill-card.md @@ -0,0 +1,45 @@ +## Description:
+Configure a Kumo model plan for a user-defined task, including encoders, run mode, bounded experiments, sampling, optimization, architecture, metrics and refits.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner +NVIDIA
+ +### License/Terms of Use:
+Apache 2.0
+## Use Case:
+Developers and data scientists use this skill with coding agents to change a Kumo model plan for a confirmed task, such as a smoke test, encoder overrides or bounded experiments, while keeping the target and evaluation protocol unchanged.
+ +### Deployment Geography for Use:
+Global
+ +## Requirements / Dependencies:
+**Requires API Key or External Credential:** [Not Specified]
+**Credential Type(s):** [None identified]
+ +Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. See skill body for more details.
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [references/plans.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumo-model-tuning/references/plans.md)
+- [NVIDIA/kumo-engine](https://github.com/NVIDIA/kumo-engine)
+ + +## Skill Output:
+**Output Type(s):** [Code, Analysis]
+**Output Format:** [Markdown with inline Python code blocks]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [Suggesting a plan does not run its experiments.]
+ +## Skill Version(s):
+Unversioned; published from the main branch of NVIDIA/kumo-engine (source: repository; no skill version, CHANGELOG entry or release tag covers the skill files). The signed copy is identified by the skill.oms.sig file in this directory.
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/agent-skills/nvskills/kumo-training-diagnostics/evals/evals.json b/agent-skills/nvskills/kumo-training-diagnostics/evals/evals.json new file mode 100644 index 000000000..641ff73e1 --- /dev/null +++ b/agent-skills/nvskills/kumo-training-diagnostics/evals/evals.json @@ -0,0 +1,66 @@ +{ + "skill_name": "kumo-training-diagnostics", + "evals": [ + { + "id": "kdiag-01", + "prompt": "My Kumo churn model just finished training in this notebook and `training_result` is in memory. How did it do?", + "expected_output": "Reads `training_result.metrics` as an attribute and lists the metric keys and finite values that are actually present, treating missing values as unavailable, not zero. For binary classification it explains that AUROC measures ranking and AP/AUPRC depends on prevalence. It does not assume metrics that were not computed and does not retrain or tune.", + "assertions": [ + "Reads training_result.metrics as an attribute, not a method call", + "Reports only the metric keys actually present, treating missing values as unavailable rather than zero", + "Explains that AUROC measures ranking quality and AP/AUPRC depends on class prevalence", + "Does not start a new fit or tuning run" + ], + "expected_skill": "kumo-training-diagnostics", + "expected_script": null + }, + { + "id": "kdiag-02", + "prompt": "Is my Kumo model overfitting? `training_result` from the fit is still in memory; look at the learning curves.", + "expected_output": "Locates the run from `training_result.bundle_dir`, finds `auto_train_progress.json` under it, and reads each trial's epoch records from `trial_progress`, keeping trials and refits separate. A widening train/validation gap can suggest overfitting, but distribution shift, split design and sampling can also explain it. It does not join unrelated trials into one curve and does not retrain.", + "assertions": [ + "Finds the run directory from training_result.bundle_dir and reads auto_train_progress.json", + "Reads per-trial epoch metrics and keeps trial and refit histories separate", + "Says a widening train/validation gap can suggest overfitting but has other possible explanations", + "Does not plot unrelated trials as one continuous curve", + "Does not start a new fit" + ], + "expected_skill": "kumo-training-diagnostics", + "expected_script": null + }, + { + "id": "kdiag-03", + "prompt": "Pick the Kumo trial with the best test AUROC from my last run and use that one instead of the one Kumo selected.", + "expected_output": "Declines to select by test-set performance. Trial selection uses the configured validation tuning metric and its direction, recorded in `best_trainer.json`; the agent should not replace the SDK's selection with its own ranking. Test results are held for final evaluation.", + "assertions": [ + "Says a model should not be chosen using test-set performance", + "Points to best_trainer.json and the validation tuning metric as the selection evidence", + "Does not override Kumo's selected trial with its own ranking" + ], + "expected_skill": "kumo-training-diagnostics", + "expected_script": null + }, + { + "id": "kdiag-04", + "prompt": "What's the job ID of my Kumo training run, and can you poll its status until it's done?", + "expected_output": "Kumo trains synchronously in the notebook process: there is no hosted job ID, status API or polling. A progress file can show recorded elapsed and estimated time, but a stale file is not proof the process is running or dead.", + "assertions": [ + "Says Kumo runs synchronously in the notebook process, with no job ID or status API", + "Does not invent a job ID, status call or polling loop" + ], + "expected_skill": "kumo-training-diagnostics", + "expected_script": null + }, + { + "id": "kdiag-05", + "prompt": "Plot the ROC curve and confusion matrix for my scikit-learn LogisticRegression on X_test, y_test.", + "expected_output": "A scikit-learn evaluation task unrelated to Kumo. The agent answers with sklearn.metrics, without loading the Kumo training-diagnostics skill.", + "assertions": [ + "Does not load or follow the kumo-training-diagnostics skill", + "Uses scikit-learn metrics to plot the ROC curve and confusion matrix" + ], + "expected_skill": null, + "expected_script": null + } + ] +} diff --git a/agent-skills/nvskills/kumo-training-diagnostics/skill-card.md b/agent-skills/nvskills/kumo-training-diagnostics/skill-card.md new file mode 100644 index 000000000..381026a7b --- /dev/null +++ b/agent-skills/nvskills/kumo-training-diagnostics/skill-card.md @@ -0,0 +1,45 @@ +## Description:
+Evaluate an existing Kumo training run in a Python notebook using aggregate metrics, trial results, learning curves, refit evidence and training progress.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner +NVIDIA
+ +### License/Terms of Use:
+Apache 2.0
+## Use Case:
+Developers and data scientists use this skill with coding agents to answer how a Kumo model performed, which trial was selected and what its learning curves and progress show, without retraining or tuning.
+ +### Deployment Geography for Use:
+Global
+ +## Requirements / Dependencies:
+**Requires API Key or External Credential:** [Not Specified]
+**Credential Type(s):** [None identified]
+ +Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. See skill body for more details.
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [references/diagnostics.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumo-training-diagnostics/references/diagnostics.md)
+- [NVIDIA/kumo-engine](https://github.com/NVIDIA/kumo-engine)
+ + +## Skill Output:
+**Output Type(s):** [Code, Analysis]
+**Output Format:** [Markdown assessment with inline Python code blocks]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [Inspection only; does not tune, rerun or deploy a model.]
+ +## Skill Version(s):
+Unversioned; published from the main branch of NVIDIA/kumo-engine (source: repository; no skill version, CHANGELOG entry or release tag covers the skill files). The signed copy is identified by the skill.oms.sig file in this directory.
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/agent-skills/nvskills/kumoai-reference/evals/evals.json b/agent-skills/nvskills/kumoai-reference/evals/evals.json new file mode 100644 index 000000000..1272a510a --- /dev/null +++ b/agent-skills/nvskills/kumoai-reference/evals/evals.json @@ -0,0 +1,77 @@ +{ + "skill_name": "kumoai-reference", + "evals": [ + { + "id": "kref-01", + "prompt": "This request has passed kumo-data-science intake and both gates. Confirmed: entity is each account (accounts.account_id); outcome is at least one transaction with transactions.is_fraud = 1; window is the next 30 days; transactions links to accounts by account_id. Write the Kumo predictive query (PQL) for it.", + "expected_output": "`PREDICT COUNT(transactions.* WHERE transactions.is_fraud = 1, 0, 30, days) > 0 FOR EACH accounts.account_id`. The fraud condition is inline in the aggregation because it describes the future event; a trailing WHERE would instead filter which accounts are eligible and predict any transaction. The window is in the aggregation arguments with the plural unit `days`.", + "assertions": [ + "Writes PREDICT ... FOR EACH accounts.account_id", + "Puts the is_fraud = 1 condition inside the aggregation (inline WHERE), not as a trailing entity WHERE", + "Expresses the horizon as aggregation arguments 0, 30, days", + "Uses a comparison such as > 0 so the target is binary classification", + "Does not use keywords such as WITHIN or a singular unit such as day" + ], + "expected_skill": "kumoai-reference", + "expected_script": null + }, + { + "id": "kref-02", + "prompt": "Validated and confirmed by kumo-data-science: Parquet files in /data/shop (customers, orders). customers: primary key customer_id. orders: primary key order_id, time column ordered_at, link orders.customer_id -> customers.customer_id. Query: PREDICT COUNT(orders.*, 0, 30, days) = 0 FOR EACH customers.customer_id. Prediction time 2026-11-01. Threshold 0.5 [default]. Destination: the empty directory /data/scores/. Generate the Python script with the Kumo SDK.", + "expected_output": "A script that imports `kumo.engine`, connects with `FileConnector(root_dir='/data/shop')`, builds Tables with `Table.from_source_table`, links them with an `Edge`, validates the graph and the query (and checks `get_task_type()`), calls `graph.snapshot()`, generates the training and prediction tables (prediction_time a naive datetime), fits exactly once with `trainer.fit(training_table, snapshot=snapshot)`, and predicts with `trainer.predict(prediction_table, snapshot=snapshot, binary_classification_threshold=0.5, output_config=...)` writing to /data/scores/. No credentials, no environment setup, no superseded API.", + "assertions": [ + "Imports kumo.engine (not kumoai)", + "Connects with FileConnector rooted at /data/shop", + "Declares the orders -> customers relationship with an Edge object, not a dict", + "Calls graph.snapshot() and passes snapshot= to both fit and predict", + "Calls fit exactly once, on a training table generated from the predictive query", + "Passes binary_classification_threshold=0.5 to predict, not to PredictionOutputConfig", + "Uses a naive datetime (no timezone) for the 2026-11-01 prediction time", + "Writes predictions to /data/scores/ through a PredictionOutputConfig", + "Does not call set_backend, LocalBackend, kumo.init or OutputConfig" + ], + "expected_skill": "kumoai-reference", + "expected_script": null + }, + { + "id": "kref-03", + "prompt": "My old Kumo notebook fails with `AttributeError: module 'kumo' has no attribute 'set_backend'`. It does:\n\n```python\nkumo.set_backend(kumo.LocalBackend())\ngraph = kumo.Graph(tables=tables, edges=[{'src_table': 'orders', 'fkey': 'customer_id', 'dst_table': 'customers'}])\ntrainer.fit(graph, pquery=pq)\nout = OutputConfig(table_name='churn')\n```\n\nCan you patch it so it runs on the current Kumo SDK?", + "expected_output": "It says the notebook targets a superseded build of the Kumo SDK and must be rewritten, not patched line by line: there is no backend to set; edges are `Edge` objects; training is `fit(training_table, snapshot=snapshot)` after `graph.snapshot()` and table generation; the output config is `kumo.PredictionOutputConfig`. It does not keep `set_backend`, dict edges or `OutputConfig`.", + "assertions": [ + "Says the code targets a superseded (old) Kumo SDK build", + "Removes set_backend / LocalBackend rather than finding a replacement call for it", + "Rewrites the edge as an Edge object", + "Rewrites training as fit(training_table, snapshot=snapshot) with an explicit graph.snapshot()", + "Replaces OutputConfig with kumo.PredictionOutputConfig" + ], + "expected_skill": "kumoai-reference", + "expected_script": null + }, + { + "id": "kref-04", + "prompt": "Validated request: forecast weekly sales amount for each store for the next 4 weeks, from sales(store_id, sold_at, amount) linked to stores(store_id). Write the PQL and the Kumo SDK steps for the training and prediction tables.", + "expected_output": "Forecasting is a temporal regression query plus a generation plan: `PREDICT SUM(sales.amount, 0, 7, days) FOR EACH stores.store_id`, with `TrainingTableGenerationPlan(forecast_length=4)` passed to `generate_training_table(snapshot, plan=...)`, and the fitted `trainer.training_table_generation_plan` passed to `generate_prediction_table`. `forecast_length` is not a model-plan setting, and `weeks` is not a valid unit.", + "assertions": [ + "Writes a SUM aggregation over a 7-day window, e.g. PREDICT SUM(sales.amount, 0, 7, days) FOR EACH stores.store_id", + "Does not use weeks as the unit", + "Sets forecast_length=4 on a TrainingTableGenerationPlan passed to generate_training_table", + "Does not put forecast_length on the model plan", + "Reuses trainer.training_table_generation_plan when generating the prediction table" + ], + "expected_skill": "kumoai-reference", + "expected_script": null + }, + { + "id": "kref-05", + "prompt": "Write a PyTorch Geometric GCN from scratch for node classification on the Cora dataset.", + "expected_output": "A generic graph neural network task, unrelated to the Kumo SDK. The agent writes PyTorch Geometric code without loading the Kumo reference skill.", + "assertions": [ + "Does not load or follow the kumoai-reference skill", + "Writes PyTorch Geometric code for a GCN on Cora", + "Does not use kumo.engine" + ], + "expected_skill": null, + "expected_script": null + } + ] +} diff --git a/agent-skills/nvskills/kumoai-reference/skill-card.md b/agent-skills/nvskills/kumoai-reference/skill-card.md new file mode 100644 index 000000000..fdc400f7e --- /dev/null +++ b/agent-skills/nvskills/kumoai-reference/skill-card.md @@ -0,0 +1,50 @@ +## Description:
+Reference for translating a validated ML request into KumoAI graph configuration, PQL (predictive query language) and Kumo SDK code.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner +NVIDIA
+ +### License/Terms of Use:
+Apache 2.0
+## Use Case:
+Developers and data scientists use this skill, normally through the kumo-data-science skill, so a coding agent writes Kumo graph definitions, predictive queries and training and prediction code that match the shipped Kumo SDK.
+ +### Deployment Geography for Use:
+Global
+ +## Requirements / Dependencies:
+**Requires API Key or External Credential:** [Optional]
+**Credential Type(s):** [Cloud Credentials, OAuth Token, Other [Snowflake connections.toml entry]]
+ +Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. See skill body for more details.
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [references/graph_construction.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumoai-reference/references/graph_construction.md)
+- [references/pql_writing.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumoai-reference/references/pql_writing.md)
+- [references/kumo_finetuning_sdk.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumoai-reference/references/kumo_finetuning_sdk.md)
+- [references/task_journeys.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumoai-reference/references/task_journeys.md)
+- [references/outputs.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumoai-reference/references/outputs.md)
+- [references/troubleshooting_signals.md](https://github.com/NVIDIA/kumo-engine/blob/main/skills/kumoai-reference/references/troubleshooting_signals.md)
+- [NVIDIA/kumo-engine](https://github.com/NVIDIA/kumo-engine)
+ + +## Skill Output:
+**Output Type(s):** [Code, Analysis]
+**Output Format:** [Markdown with inline Python and PQL code blocks]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [Generates code only; does not run jobs or interpret metrics.]
+ +## Skill Version(s):
+Unversioned; published from the main branch of NVIDIA/kumo-engine (source: repository; no skill version, CHANGELOG entry or release tag covers the skill files). The signed copy is identified by the skill.oms.sig file in this directory.
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/agent-skills/scripts/publish_nvskills.py b/agent-skills/scripts/publish_nvskills.py new file mode 100644 index 000000000..66b687966 --- /dev/null +++ b/agent-skills/scripts/publish_nvskills.py @@ -0,0 +1,293 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. +# All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Refresh the copy of the skills that the NVIDIA skills catalog publishes. + + python3 agent-skills/scripts/publish_nvskills.py # rewrite skills/ + python3 agent-skills/scripts/publish_nvskills.py --check # exit 1 if that copy is out of date + +The NVIDIA/skills catalog (github.com/NVIDIA/skills) pulls skills from each product repo +every hour. It reads them from `components.d/.yml` in the catalog, which names one +directory in this repository per skill. It accepts a skill only if the skill's files match the +signature that NVSkills CI writes. NVSkills CI runs only on pull requests that change `skills/` or +`team-skills/` at the repository root. `agent-skills/skills/` is the source layout, not a +package, so the catalog cannot read it directly. This script writes the `generic` package (the one +for Claude Code, Codex and other coding agents) to `skills/`. A PR that changes the skill runs it +and commits the result, and `tests/test_publish_nvskills.py` fails the PR when it did not. +NVSkills CI signs that same PR, so merging it to `main` is what publishes. + +The catalog does not allow a skill inside another skill. The package nests the seven other skills +inside the router, `kumo-data-science/`, so this script lays them out beside it instead: eight +directories under `skills/`, each holding one `SKILL.md`. Their links to one another resolve the +same way in both layouts (`../kumo-model-persistence/SKILL.md`), and a backticked path that starts +with a skill's name is read from the directory holding the skills, so the build checks every +reference again in the flat layout. + +The catalog sync deletes any skill directory that lacks its own `skill.oms.sig`, `skill-card.md` +and eval dataset. Three kinds of file sit in each published skill directory: + +* the package, rebuilt from `agent-skills/skills/` on every run; +* the catalog files that NVSkills requires and that we write: the skill's `skill-card.md` and + `evals/*.json`. Their source is `agent-skills/nvskills//`, and they are copied in on + every run. A catalog file must belong to a skill and may not replace a package file. Once that + source exists, the build fails unless every skill has both, so the sync never drops one. + The card is the exception to "every run": NVSkills CI rewrites it when it signs, adding the + evaluation results, so once a skill's directory holds a `skill.oms.sig` its card is NVSkills + CI's. A refresh then leaves it as it is and `--check` does not compare it. The source card + seeds a skill's first card; to change a signed one, edit `skills//skill-card.md`; +* `BENCHMARK.md` and `skill.oms.sig`, which NVSkills CI writes into the pull request. They are + never edited here, and the build fails if the package or the catalog files carry either + name at any depth. A refresh leaves them in place at any depth, so they describe the previous + content until NVSkills CI runs again. The catalog then keeps serving the last signed version + rather than dropping the skill. + +Everything else under `skills/` is this script's output: a refresh deletes it. + +`--check` ignores the `SKILL-BUILD` stamp, which names the commit that built the package and so +differs on every commit even when no skill file changed. +""" +from __future__ import annotations + +import argparse +import importlib.util +import shutil +import sys +import tempfile +from pathlib import Path + +HERE = Path(__file__).resolve().parent +spec = importlib.util.spec_from_file_location('package_skill', HERE / 'package_skill.py') +assert spec and spec.loader +package_skill = importlib.util.module_from_spec(spec) +sys.modules.setdefault('package_skill', package_skill) +spec.loader.exec_module(package_skill) + +HOST = 'generic' +ROUTER = package_skill.ROUTER +REPO = package_skill.ROOT.parent +PUBLISHED = REPO / 'skills' +CATALOG_FILES = package_skill.ROOT / 'nvskills' +# Written by NVSkills CI into the pull request; never generated or deleted here. +SIGNED = frozenset({'BENCHMARK.md', 'skill.oms.sig'}) +# What each skill directory needs from the catalog files; the sync drops a skill without them. +CARD = 'skill-card.md' +EVALS = 'evals' + + +def skill_dirs(root: Path) -> list[str]: + """Each skill directory of the flat layout under `root`, by name.""" + return sorted(p.parent.name for p in root.glob('*/SKILL.md')) + + +def nested(root: Path) -> list[str]: + """Every `SKILL.md` under `root` that is not at the top of its own skill directory.""" + return sorted(p.relative_to(root).as_posix() for p in root.rglob('SKILL.md') + if len(p.relative_to(root).parts) != 2) + + +def owner(rel: Path) -> str | None: + """The skill a catalog file belongs to, or None if it is not a catalog input.""" + parts = rel.parts + if len(parts) == 2 and parts[1] == CARD: + return parts[0] + if len(parts) > 2 and parts[1] == EVALS: + return parts[0] + return None + + +def missing(root: Path) -> list[str]: + """The catalog inputs a skill directory under `root` lacks, as `root`-relative paths.""" + out = [] + for skill in skill_dirs(root): + d = root / skill + if not (d / CARD).is_file(): + out.append(f'{skill}/{CARD}') + if not any((d / EVALS).glob('*.json')): + out.append(f'{skill}/{EVALS}/*.json') + return out + + +def unresolved(root: Path) -> list[str]: + """Return every reference in the flat layout under `root` that does not resolve. + + A markdown link resolves from the file that holds it. A backticked path resolves from `root` + when it starts with a skill's name, and from its own skill directory when it starts with one + of that skill's entries. A path that starts with another skill's entry would be read from the + wrong directory, so it is an error. + """ + root = root.resolve() + skills = set(skill_dirs(root)) + entries = {s: {p.name for p in (root / s).iterdir()} for s in skills} + errors = [] + for doc in sorted(root.rglob('*.md')): + text = doc.read_text(encoding='utf-8') + where = doc.relative_to(root).as_posix() + skill = doc.relative_to(root).parts[0] + for target in package_skill.MD_LINK.findall(text): + if '://' in target or target.startswith('mailto:'): + continue + resolved = (doc.parent / target).resolve() + if not resolved.is_file() or not resolved.is_relative_to(root): + errors.append(f'{where}: link ({target}) does not resolve') + for target in package_skill.CODE_PATH.findall(text): + head = target.split('/', 1)[0] + if head in skills: + path = root / target + elif head in entries.get(skill, ()): + path = root / skill / target + elif any(head in names for names in entries.values()): + errors.append(f'{where}: `{target}` is another skill\'s file: start it with ' + f'that skill\'s name') + continue + else: + continue # not a package path, e.g. a Kumo source module + if not path.is_file(): + errors.append(f'{where}: `{target}` does not resolve') + return errors + + +def flatten(pkg: Path, out: Path) -> Path: + """Lay the package `pkg` out under `out` with every skill beside the router.""" + inner = sorted(p.parent.relative_to(pkg) for p in pkg.rglob('SKILL.md') if p.parent != pkg) + deep = [p.as_posix() for p in inner if len(p.parts) != 1] + if deep: + raise SystemExit(f'{HOST} package nests skills below its top level: {deep}') + names = {p.name for p in inner} + shutil.copytree(pkg, out / ROUTER, + ignore=lambda d, entries: names & set(entries) if Path(d) == pkg else ()) + for name in sorted(names): + shutil.copytree(pkg / name, out / name) + return out + + +def build(out: Path, catalog_files: Path = CATALOG_FILES) -> Path: + """Build the flat layout the catalog publishes into `out` and return it.""" + pkg = package_skill.build(HOST, out / 'package') + errors = package_skill.unresolved(pkg) + if errors: + raise SystemExit(f'{HOST}: {len(errors)} unresolved reference(s): {errors[:3]}') + # A signed name in the package would overwrite what NVSkills CI wrote, unseen by --check. + signed = [p.relative_to(pkg).as_posix() for p in pkg.rglob('*') if p.name in SIGNED] + if signed: + raise SystemExit(f'{HOST} package ships {signed}: written by NVSkills CI, not by hand') + root = flatten(pkg, out / 'skills') + errors = unresolved(root) + if errors: + raise SystemExit(f'{HOST}, one directory per skill: {len(errors)} unresolved ' + f'reference(s): {errors[:3]}') + if catalog_files.is_dir(): + skills = skill_dirs(root) + for path in sorted(catalog_files.rglob('*')): + if path.is_file(): + rel = path.relative_to(catalog_files) + if path.name in SIGNED: + raise SystemExit(f'{path}: {rel} is written by NVSkills CI, not by hand') + if owner(rel) not in skills: + raise SystemExit(f'{path}: {rel} is not the {CARD} or {EVALS}/ of a skill ' + f'in the {HOST} package ({skills})') + target = root / rel + if target.exists(): + raise SystemExit(f'{path}: {rel} is already in the {HOST} package') + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(path, target) + # The catalog sync deletes a skill directory without its own card and evals, and with + # it the files the other skills link to. Before the first one is written nothing is + # published, so the gap is reported (`not_ready`) rather than fatal. + gaps = missing(root) + if gaps: + raise SystemExit(f'{catalog_files}: the catalog would drop these skills; add ' + f'{gaps}') + return root + + +def not_ready(catalog_files: Path = CATALOG_FILES) -> list[str]: + """Catalog inputs still to write before the first publish; empty once they all exist.""" + with tempfile.TemporaryDirectory() as tmp: + return missing(build(Path(tmp), catalog_files)) + + +def signed_cards(root: Path) -> frozenset[str]: + """The cards under `root` that NVSkills CI rewrote when it signed their skill.""" + return frozenset(f'{skill}/{CARD}' for skill in skill_dirs(root) + if (root / skill / 'skill.oms.sig').is_file() + and (root / skill / CARD).is_file()) + + +def files(root: Path, kept: frozenset[str] = frozenset()) -> dict[str, bytes]: + """Map each file under `root` that we own to its bytes, with the build stamp normalized. + + `kept` names files NVSkills CI owns besides the signed ones: they are left out. + """ + if not root.is_dir(): + return {} + out = {} + for path in sorted(root.rglob('*')): + rel = path.relative_to(root).as_posix() + if path.is_file() and path.name not in SIGNED and rel not in kept: + data = path.read_bytes() + if rel == f'{ROUTER}/SKILL.md': + text = package_skill.MARKER.sub('SKILL-BUILD: x', data.decode('utf-8')) + data = text.encode('utf-8') + out[rel] = data + return out + + +def stale(dest: Path = PUBLISHED, catalog_files: Path = CATALOG_FILES) -> list[str]: + """Return what `dest` would gain, lose or change if it were refreshed now.""" + kept = signed_cards(dest) + with tempfile.TemporaryDirectory() as tmp: + want = files(build(Path(tmp), catalog_files), kept) + have = files(dest, kept) + return ([f'+ {p}' for p in sorted(want.keys() - have.keys())] + + [f'- {p}' for p in sorted(have.keys() - want.keys())] + + [f'~ {p}' for p in sorted(want.keys() & have.keys()) if want[p] != have[p]]) + + +def publish(dest: Path = PUBLISHED, catalog_files: Path = CATALOG_FILES) -> list[str]: + """Rewrite `dest` with the current skills, keeping the files NVSkills CI wrote.""" + changes = stale(dest, catalog_files) + if not changes: + return [] + kept = signed_cards(dest) + with tempfile.TemporaryDirectory() as tmp: + root = build(Path(tmp), catalog_files) + if dest.is_dir(): + for path in sorted(dest.rglob('*'), reverse=True): + rel = path.relative_to(dest).as_posix() + if path.is_file() and path.name not in SIGNED and rel not in kept: + path.unlink() + elif path.is_dir() and not any(path.iterdir()): + path.rmdir() + for path in sorted(root.rglob('*')): + if path.is_file() and path.relative_to(root).as_posix() not in kept: + target = dest / path.relative_to(root) + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(path, target) + return changes + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__.split('\n\n')[0]) + ap.add_argument('--check', action='store_true', + help=f'exit 1 if {PUBLISHED.relative_to(REPO)}/ is out of date') + args = ap.parse_args() + where = PUBLISHED.relative_to(REPO) + changes = stale() if args.check else publish() + for line in changes: + print(line) + if not changes: + print(f'{where}/ is up to date') + elif args.check: + print(f'{where}/ is out of date: run {Path(__file__).relative_to(REPO)}', file=sys.stderr) + raise SystemExit(1) + else: + print(f'{where}/ refreshed: {len(changes)} file(s)') + todo = not_ready() + if todo: + print(f'not publishable yet: the catalog would drop every skill until ' + f'{CATALOG_FILES.relative_to(REPO)}/ supplies {todo}', file=sys.stderr) + + +if __name__ == '__main__': + main() diff --git a/agent-skills/skills/hosts/databricks-genie/runtime.md b/agent-skills/skills/hosts/databricks-genie/runtime.md index f86091c00..742919cb9 100644 --- a/agent-skills/skills/hosts/databricks-genie/runtime.md +++ b/agent-skills/skills/hosts/databricks-genie/runtime.md @@ -172,6 +172,8 @@ without it, even for a static query, whatever the §7.1 Snowflake example shows, anchor as null. Without a time column, select `ENTITY1` alone. ```python +if f'{POP_CATALOG}.{POP_SCHEMA}.{POP_TABLE}' != POPULATION_CONFIRMED: # approved at Node 1d + raise RuntimeError('The population table changed after Node 1d: confirm the new one first.') run_sql(f'CREATE OR REPLACE TABLE {q(POP_CATALOG, POP_SCHEMA, POP_TABLE)} AS ' f'SELECT {q(ENTITY_KEY)} AS ENTITY1, CAST({q(ENTITY_TIME)} AS TIMESTAMP) AS TIMESTAMP ' f'FROM {q(CATALOG, SCHEMA, ENTITY_TABLE)}') @@ -182,7 +184,9 @@ prediction_table = kumo.PredictionTable.from_source_table( ``` `ENTITY_TIME` is the entity `Table`'s `time_column`; `POP_CATALOG` and `POP_SCHEMA` are the -population's confirmed location, which need not be the source's. +population's confirmed location, which need not be the source's. `POPULATION_CONFIRMED` is the +population exactly as Node 1d confirmed it (`kumoai-reference/references/kumo_finetuning_sdk.md` +§7.1). **A population the user supplies** — a table of exact scoring rows, or curated labels, weights and splits (`kumo-custom-populations/SKILL.md`) — is read through a `DatabricksConnector` on that diff --git a/agent-skills/skills/hosts/generic/runtime.md b/agent-skills/skills/hosts/generic/runtime.md index 59e5791d1..398fee17c 100644 --- a/agent-skills/skills/hosts/generic/runtime.md +++ b/agent-skills/skills/hosts/generic/runtime.md @@ -28,7 +28,9 @@ still waits for an explicit request. ## 1. Config cell The first code cell holds the imports and every location as a named constant: the confirmed -source, the confirmed destination, and any population table — each written verbatim. +source, the confirmed destination, and any population table — each written verbatim. A +population also gets `POPULATION_CONFIRMED`, its name exactly as Node 1d confirmed it, which the +staging step checks before replacing anything (§3.1). Values only the runtime knows — a Snowflake warehouse or `connections.toml` entry, a Databricks SQL warehouse or cluster — are constants here too. S3, GCS and BigQuery have none: the ambient @@ -69,7 +71,7 @@ connector = kumo.GCSConnector(root_dir=SOURCE_URI) # 'gs://bucket/pref # default). No billing_project_id (§1). location= is the dataset's own, read by bq_location below. connector = kumo.BigQueryConnector(project_id=PROJECT_ID, dataset_id=DATASET_ID, identity=BQ_IDENTITY, - location=bq_location(DATASET_ID, BQ_CREDENTIALS)) + location=bq_location(PROJECT_ID, DATASET_ID, BQ_CREDENTIALS)) # Databricks — exactly one of warehouse_id / cluster_id connector = kumo.DatabricksConnector( catalog=CATALOG, schema_name=SCHEMA, identity=kumo.DatabricksIdentity(), @@ -90,17 +92,29 @@ else — `EU`, or a single region such as `us-central1`, which is not `US` — f not-found, and the prediction write only after training, at the sink's location check. Whenever any connector in the notebook is BigQuery — source, population or destination — define this and the identities below once, in the connect step before the first connector, and pass -`location=bq_location(, )` to every `BigQueryConnector`: +`location=bq_location(, , )` to every +`BigQueryConnector`: ```python from google.cloud import bigquery -def bq_location(dataset, credentials=None): +def bq_location(project, dataset, credentials=None): """The location a dataset lives in; the dataset must already exist.""" - return bigquery.Client(project=PROJECT_ID, credentials=credentials).get_dataset( - f'{PROJECT_ID}.{dataset}').location + return bigquery.Client(project=project, credentials=credentials).get_dataset( + f'{project}.{dataset}').location + +def bq_name(*parts): + """`project`.`dataset`.`table`, or one `column`, for BigQuery SQL. A backtick, backslash or + line break is in no BigQuery name and would end the quoting, so such a part is refused.""" + for part in parts: + if not part or any(c in part for c in '`\\\n\r'): + raise ValueError(f'not a BigQuery name: {part!r}') + return '.'.join(f'`{part}`' for part in parts) ``` +Build every BigQuery table or column name in SQL with `bq_name`, never by pasting a name into the +query text. + Each BigQuery identity comes with the Google credentials its direct clients use: the location lookups, the population query (§3.1) and the read-back (§3). With an explicit `BigQueryIdentity` (*Credentials* below), build those credentials from the inputs the identity was given, as the SDK @@ -135,6 +149,13 @@ BQ_CREDENTIALS = impersonated_credentials.Credentials( OUTPUT_BQ_IDENTITY, OUTPUT_BQ_CREDENTIALS = BQ_IDENTITY, BQ_CREDENTIALS ``` +Each BigQuery location keeps the project it was confirmed with: `PROJECT_ID` for the source, +`OUTPUT_PROJECT` and `POP_PROJECT` for the destination and the population, equal to `PROJECT_ID` +only when the confirmed names say so. Use that project for its connector, location, staging query +and read-back. Project and credentials are independent: a location in another project still uses +its own identity's credentials as above — the population the source's, the destination +`OUTPUT_BQ_CREDENTIALS`. + Name file and object-store tables without their suffix — `connector['accounts']` for `accounts.parquet` — and use those names in the tables, edges and PQL (`adapter.md` §3). @@ -147,6 +168,20 @@ identity = kumo.S3Identity(access_key_id=os.environ[AWS_KEY_ID_VAR], secret_access_key=os.environ[AWS_SECRET_VAR]) ``` +Arrow does not see that identity. Every `pyarrow` read or write of S3 in §3 and §3.1 then takes +the same keys, with the path written without `s3://`: + +```python +import pyarrow.fs +s3 = pyarrow.fs.S3FileSystem(access_key=os.environ[AWS_KEY_ID_VAR], + secret_key=os.environ[AWS_SECRET_VAR]) +written = ds.dataset(OUTPUT_LOCATION.removeprefix('s3://'), format=OUTPUT_FORMAT, + filesystem=s3).count_rows() +``` + +Without an explicit identity, emit `identity = None` and `s3 = None`: §3.1's population staging +then lets Kumo and Arrow resolve the ambient credentials themselves. + The Snowflake factory `from_spcs_env` and Snowflake's notebook secret store belong to Snowflake's own runtime. Emit them only when the user says the code runs there. @@ -179,9 +214,11 @@ output_config = kumo.PredictionOutputConfig( OUTPUT_LOCATION = OUTPUT_PARENT + OUTPUT_LEAF # the confirmed path, for read-back ``` -A BigQuery destination's connector is scoped to `OUTPUT_DATASET` and takes -`identity=OUTPUT_BQ_IDENTITY` and `location=bq_location(OUTPUT_DATASET, OUTPUT_BQ_CREDENTIALS)` -(§2); the location may differ from the source's. +A BigQuery destination's connector is scoped to `OUTPUT_PROJECT` and `OUTPUT_DATASET` and takes +`identity=OUTPUT_BQ_IDENTITY` and +`location=bq_location(OUTPUT_PROJECT, OUTPUT_DATASET, OUTPUT_BQ_CREDENTIALS)` (§2); the location +may differ from the source's. A Databricks destination's connector is scoped to `OUTPUT_CATALOG` +and `OUTPUT_SCHEMA`. Rooting the connector at the destination itself and passing a `table_name` writes one level below it, to a path the user never confirmed. A directory destination must be empty or absent; @@ -199,8 +236,9 @@ written = ds.dataset(OUTPUT_LOCATION, format=OUTPUT_FORMAT).count_rows() # full # BigQuery — as the destination's identity (§2) from google.cloud import bigquery -written = next(iter(bigquery.Client(project=PROJECT_ID, credentials=OUTPUT_BQ_CREDENTIALS).query( - f'SELECT COUNT(*) FROM `{PROJECT_ID}.{OUTPUT_DATASET}.{OUTPUT_TABLE}`').result()))[0] +bq = bigquery.Client(project=OUTPUT_PROJECT, credentials=OUTPUT_BQ_CREDENTIALS) +count_sql = f'SELECT COUNT(*) FROM {bq_name(OUTPUT_PROJECT, OUTPUT_DATASET, OUTPUT_TABLE)}' +written = next(iter(bq.query(count_sql).result()))[0] # Databricks — with the confirmed SQL warehouse; wait_timeout is at most '50s', so poll import time @@ -214,25 +252,45 @@ def q(*parts): ws = WorkspaceClient() r = ws.statement_execution.execute_statement( warehouse_id=DATABRICKS_WAREHOUSE_ID, wait_timeout='50s', - statement=f'SELECT COUNT(*) FROM {q(CATALOG, OUTPUT_SCHEMA, OUTPUT_TABLE)}') + statement=f'SELECT COUNT(*) FROM {q(OUTPUT_CATALOG, OUTPUT_SCHEMA, OUTPUT_TABLE)}') while r.status.state in (StatementState.PENDING, StatementState.RUNNING): time.sleep(5) r = ws.statement_execution.get_statement(r.statement_id) assert r.status.state == StatementState.SUCCEEDED, r.status written = int(r.result.data_array[0][0]) -# Snowflake — through the same connections.toml entry +# Snowflake — through the same connections.toml entry and warehouse. Kumo keeps the table +# name's case, so quote it (kumo_finetuning_sdk.md §9.1); database and schema as in §2. +import re import snowflake.connector -with snowflake.connector.connect(connection_name=SNOWFLAKE_CONNECTION) as conn: - written = conn.cursor().execute( - f'SELECT COUNT(*) FROM {OUTPUT_DATABASE}.{OUTPUT_SCHEMA}.{OUTPUT_TABLE}').fetchone()[0] -assert written == prediction_result.num_rows, (written, prediction_result.num_rows) +def sf_name(database, schema, table): + """The table Kumo writes, as database.schema."table". Database and schema are written as + the connector was given them (§2): a plain identifier, which Snowflake upper-cases, or a + lower- or mixed-case name already in double quotes ('"my_schema"'), kept as it is. Anything + else is refused. The table is quoted, with any double quote in it doubled.""" + for part in (database, schema): + if not re.fullmatch(r'[A-Za-z_][A-Za-z0-9_$]*|"(?:[^"]|"")+"', part or ''): + raise ValueError(f'not a Snowflake name: {part!r}; confirm it with the user') + return f'{database}.{schema}."' + table.replace('"', '""') + '"' + +with snowflake.connector.connect(connection_name=SNOWFLAKE_CONNECTION, + warehouse=SNOWFLAKE_WAREHOUSE) as conn: + count_sql = f'SELECT COUNT(*) FROM {sf_name(OUTPUT_DATABASE, OUTPUT_SCHEMA, OUTPUT_TABLE)}' + written = conn.cursor().execute(count_sql).fetchone()[0] + +assert written - before == prediction_result.num_rows, (written, before, prediction_result.num_rows) ``` +`before` is 0 for OVERWRITE. For APPEND (`kumoai-reference/references/outputs.md`) it is the same +count taken before `predict` — 0 when the destination does not exist yet, since the first APPEND +creates it — so the check compares the rows this run added. The delta holds only while nothing +else writes to the destination; when the user says others do, confirm a run-specific destination +instead. + - `OUTPUT_LOCATION` for files, S3 and GCS is the confirmed destination itself. - A Databricks destination is passed to Kumo as `'__kumo__'` and read back as - `..
`. With a cluster and no SQL warehouse, there is no read-back here: + `..
`, in its own catalog. With a cluster and no SQL warehouse, there is no read-back here: say so in the artifact and in your reply. - Report the destination in its own form, unchanged — `DB.SCHEMA.TABLE`, or the path. @@ -257,25 +315,74 @@ system; the destination may be in another system (`adapter.md` §4) and plays no ```python import os + import re import pyarrow as pa import pyarrow.compute as pc + import pyarrow.csv import pyarrow.dataset as ds + import pyarrow.fs import pyarrow.parquet as papq + def arrow_fs(uri): + """Arrow's filesystem and path for uri; S3 with an explicit S3Identity takes §2's `s3`.""" + if uri.startswith('s3://') and s3 is not None: # s3 = None under ambient credentials + return s3, uri.removeprefix('s3://') + return pyarrow.fs.FileSystem.from_uri(uri if '://' in uri else os.path.abspath(uri)) + + if POPULATION_PATH != POPULATION_CONFIRMED: # Node 1d approved replacing this path only + raise RuntimeError('POPULATION_PATH changed after Node 1d: confirm the new path first.') keep = {ENTITY_KEY: 'ENTITY1'} if ENTITY_TIME: # the entity Table's time_column, else None keep[ENTITY_TIME] = 'TIMESTAMP' - population = ds.dataset(ENTITY_SOURCE, format=ENTITY_FORMAT).to_table(columns=list(keep)) - population = population.rename_columns([keep[c] for c in population.column_names]) - if 'TIMESTAMP' in population.column_names: # a DATE anchor reads as null - i = population.column_names.index('TIMESTAMP') - population = population.set_column(i, 'TIMESTAMP', - pc.cast(population['TIMESTAMP'], pa.timestamp('us'))) - if '://' not in POPULATION_PATH: - os.makedirs(POPULATION_PATH, exist_ok=True) # object stores need no directory - papq.write_table(population, POPULATION_PATH + 'population.parquet') + source_fs, source_path = arrow_fs(ENTITY_SOURCE) + # Arrow parses C-style directives only; chrono's own (%.f, %-m, %:z, %3f, %f) it cannot. + ARROW_FORMAT = (ENTITY_TIME_FORMAT + if ENTITY_TIME_FORMAT and not re.search(r'%[-_0^#.:+\d]|%f', ENTITY_TIME_FORMAT) + else None) + source_format = ENTITY_FORMAT + if ENTITY_FORMAT == 'csv' and ENTITY_TIME and ARROW_FORMAT: # text, parsed with the format below + source_format = ds.CsvFileFormat(convert_options=pyarrow.csv.ConvertOptions( + column_types={ENTITY_TIME: pa.string()})) + NULL_TOKENS = pa.array(pyarrow.csv.ConvertOptions().null_values) # '', 'NA', 'null', ... + scanner = ds.dataset(source_path, format=source_format, + filesystem=source_fs).scanner(columns=list(keep)) + + def anchor_type(t): # a timestamp keeps its unit; a DATE anchor reads as null: cast it + return t if pa.types.is_timestamp(t) else pa.timestamp('us') + + schema = pa.schema([(keep[f.name], anchor_type(f.type) if keep[f.name] == 'TIMESTAMP' else f.type) + for f in scanner.projected_schema]) + + def to_column(col, field): # text in the entity's format reads as the graph reads it + if not (field.name == 'TIMESTAMP' and ARROW_FORMAT + and (pa.types.is_string(col.type) or pa.types.is_large_string(col.type))): + return pc.cast(col, field.type) + unit = field.type.unit + col = pc.if_else(pc.is_in(col, value_set=NULL_TOKENS), pa.scalar(None, col.type), col) + if '%z' not in ARROW_FORMAT: + return pc.strptime(col, format=ARROW_FORMAT, unit=unit) # raises on a misfit + # As the graph does, a value without an offset is read by the format without %z. + out = pc.coalesce( + pc.strptime(col, format=ARROW_FORMAT, unit=unit, error_is_null=True), + pc.strptime(col, format=ARROW_FORMAT.replace('%z', ''), unit=unit, error_is_null=True)) + misfit = pc.and_(pc.is_valid(col), pc.is_null(out)) + if pc.any(misfit).as_py(): + raise pa.ArrowInvalid(f'{pc.filter(col, misfit)[0]} does not fit {ARROW_FORMAT}') + return out + + pop_fs, pop_dir = arrow_fs(POPULATION_PATH) + if isinstance(pop_fs, pyarrow.fs.LocalFileSystem): + pop_fs.create_dir(pop_dir) # object stores need no directory + final = pop_dir.rstrip('/') + '/population.parquet' + with papq.ParquetWriter(final + '.partial', schema, filesystem=pop_fs) as writer: + for batch in scanner.to_batches(): # in batches: bounded memory + writer.write_batch(pa.RecordBatch.from_arrays( + [to_column(batch[f.name], t) for f, t in zip(batch.schema, schema)], + schema=schema)) + pop_fs.move(final + '.partial', final) # replace only once every batch is in if POPULATION_PATH.startswith('s3://'): # the source's kind, in the §2 forms - population_connector = kumo.S3Connector(root_dir=POPULATION_PATH) + population_connector = kumo.S3Connector(root_dir=POPULATION_PATH, + identity=identity) # §2's; None under ambient elif POPULATION_PATH.startswith('gs://'): population_connector = kumo.GCSConnector(root_dir=POPULATION_PATH) else: @@ -285,16 +392,33 @@ system; the destination may be in another system (`adapter.md` §4) and plays no ``` `ENTITY_SOURCE` is the entity table's file or directory as preflight listed it, and - `ENTITY_FORMAT` its format as preflight found it, `'csv'` or `'parquet'`. The population is + `ENTITY_FORMAT` its format as preflight found it, `'csv'` or `'parquet'`. `ENTITY_TIME_FORMAT` + is the `timestamp_format` the entity `Table` declares on its time column + (`kumoai-reference/references/graph_construction.md`, *A time column must be a real date*), else + `None`: a CSV reads `03/15/2024` as text, and a plain cast parses only ISO-8601. The format is + Kumo's (chrono) syntax. When Arrow can parse it too, a CSV time column is read as text and + parsed with the format, because ISO-looking text such as `2024-01-02` under `%Y-%d-%m` means + what the format says, as it does in the graph; a cell the CSV reader takes as null (empty, + `NA`, `null`) stays null, and no other column changes. Arrow cannot parse + chrono's own directives (`%.f`, `%-m`, `%:z`); with one of those the column is read and cast as + without a format, which keeps the type the reader inferred: nanoseconds, or a UTC offset as a + time zone. Under `%z`, a value without an offset is read without it, as the graph reads it. + If the parse or the cast fails, report the format and the failing value rather than + guessing another format. The population is written as Parquet either way. `POPULATION_PATH` is the confirmed path, ending in `/`, in the - source's file system or bucket. + source's file system or bucket, and `POPULATION_CONFIRMED` the same path as Node 1d confirmed + it, both in the config cell. Each run replaces its `population.parquet`, as Node 1d says. The + rows go to `population.parquet.partial` first, so a scan or cast that fails leaves the previous + population in place. A source timestamp keeps its own unit: a `timestamp[ns]` with + sub-microsecond values would fail a safe cast to microseconds. - **Snowflake source:** the §7.1 SQL, run through - `snowflake.connector.connect(connection_name=SNOWFLAKE_CONNECTION)` as in the read-back above, + `snowflake.connector.connect(connection_name=SNOWFLAKE_CONNECTION, warehouse=SNOWFLAKE_WAREHOUSE)` + as in the read-back above, and the `PredictionTable` built from a `SnowflakeConnector` made as in §2 but scoped to the population's database and schema. - **Databricks source:** create the table through the source's SQL warehouse with the statement API, wait for it, and build the `PredictionTable` from a `DatabricksConnector` scoped to the - population's schema. A source reached through a cluster has no SQL warehouse: say at Node 1d + population's catalog and schema, `POP_CATALOG` and `POP_SCHEMA`. A source reached through a cluster has no SQL warehouse: say at Node 1d that staging is not covered there, and offer the rows whose target is NULL instead. The same `TIMESTAMP` rule applies: project `CAST(
']` tries the exact name, then its upper-cased form, so `connector['ENTITY_PROFILE']` never finds a lower-case @@ -219,7 +220,7 @@ table names as `table_names()` returns them and column names as `.columns` retur **Never exclude a column by a case-sensitive `!=` against a hand-typed literal.** A filter such as `c.name != 'END_DATE'` silently keeps a lower-case `end_date` — the comparison never matches, the column is never dropped, and nothing raises. Pass the confirmed `Excluded columns` from preflight -(`accept/preflight.md` §8) as a literal `column_names=` list built from the catalog's own spelling +(`kumo-data-science/accept/preflight.md` §8) as a literal `column_names=` list built from the catalog's own spelling — it has no hand-typed comparison to get wrong. If a filter is used instead, compare case-insensitively: `c.name.lower() != 'end_date'.lower()`. @@ -228,6 +229,10 @@ Source: shipped package (`kumo.engine.connector.snowflake`, `.base`, `.source_ta ### 3.3 The Snowpark session: take it once, first +Only where the host's `runtime.md` provides an active Snowpark session. A host without one +follows its `runtime.md` for SQL and read-back instead, and this section and the `session` +calls in §7.1 and §9.1 do not apply. + Kumo's Snowflake connector opens its own Snowpark session. Once Kumo has read or written Snowflake, the process holds two sessions, and a later `get_active_session()` raises `SnowparkSessionException: (1409): More than one active session is detected`. @@ -466,9 +471,14 @@ says how a population is staged, or that it is not covered there: `ENTITY1` in the confirmed population table and pass it explicitly: ```python +def q(name): # a quoted Snowflake identifier + return '"' + name.replace('"', '""') + '"' + +if f'{POP_DATABASE}.{POP_SCHEMA}.{POP_TABLE}' != POPULATION_CONFIRMED: # approved at Node 1d + raise RuntimeError('The population table changed after Node 1d: confirm the new one first.') session.sql( # the session from §3.3 f'CREATE OR REPLACE TABLE {POP_DATABASE}.{POP_SCHEMA}.{POP_TABLE} AS ' - f'SELECT "{ENTITY_KEY}" AS ENTITY1 FROM {SOURCE_DATABASE}.{SOURCE_SCHEMA}."{ENTITY_TABLE}"' + f'SELECT {q(ENTITY_KEY)} AS ENTITY1 FROM {SOURCE_DATABASE}.{SOURCE_SCHEMA}.{q(ENTITY_TABLE)}' ).collect() population_connector = kumo.SnowflakeConnector.from_spcs_env( # or from_connection_toml, §3 warehouse=WAREHOUSE, database=POP_DATABASE, schema_name=POP_SCHEMA, @@ -478,6 +488,10 @@ prediction_table = kumo.PredictionTable.from_source_table( ) ``` +- `CREATE OR REPLACE` replaces an existing table; Node 1d names that before it is confirmed, + and the approval covers that one name. The config cell carries it verbatim as + `POPULATION_CONFIRMED`, and every staging recipe stops before writing when the name it would + replace differs — an edit after confirmation needs a new confirmation, not a silent replace. - Stage the population only where Node 1d confirmed it — never in the source schema. Write `POP_TABLE` in upper case, so the unquoted `CREATE` and the connector lookup agree. - Never change the query, the entity or the target to get rows. An empty population is a fact @@ -573,6 +587,9 @@ PredictionOutputConfig(connector=, artifact_type=, table_name=, write_mode=, ext PredictionResult.export(output_config) ``` +- `write_mode`: `OVERWRITE` replaces an existing table at the destination. Use the mode Node 1d + confirmed: `OVERWRITE` for a new table or an approved replacement, `APPEND` when the user chose + it (`kumoai-reference/references/outputs.md`). - For Snowflake, always pass `artifact_type` and `table_name`. `output_paths` is for file and object-store connectors only; with a `SnowflakeConnector` it raises `ValueError`. Every connector's form is in §9.2. @@ -591,8 +608,9 @@ PredictionResult.export(output_config) (`kumo_core.prediction_options.MAX_LINKS_TO_PREDICT`); for more than 500 per entity, export embeddings instead. Multiclass and multilabel accept it optionally and fall back to the query's `RANK TOP K` when it has one. Regression and binary classification reject it. -- Binary classification requires an explicit `binary_classification_threshold`; there is no SDK - default. Use the one confirmed at Node 1d (§12). +- Binary classification: without `binary_classification_threshold` (and none recorded with the + model), `predict` returns `True_PROB` and leaves `TARGET_PRED` null; it has no 0.5 default of + its own. Always pass the one confirmed at Node 1d (§12), written or local alike. - `binary_classification_threshold` and `num_classes_to_return` are arguments of `predict`, and only of `predict`. `PredictionOutputConfig` describes where to write, not how to predict: it takes neither, and passing one is a `TypeError`. Whatever the destination, the config carries @@ -615,9 +633,14 @@ def quoted(name): return '"' + name.replace('"', '""') + '"' destination = session.table([OUTPUT_DATABASE, OUTPUT_SCHEMA, quoted(output_table)]) -assert destination.count() == prediction_result.num_rows +assert destination.count() - before == prediction_result.num_rows ``` +`before` is 0 for `OVERWRITE`. For `APPEND` it is the same count taken before `predict` — 0 when +the table does not exist yet, since the first APPEND creates it — so the check compares the rows +this run added. The delta holds only while nothing else writes to the table +(`kumoai-reference/references/outputs.md`). + Scope the output connector to the destination database and schema; `table_name` is a bare name. Always read the table back — it is the only proof the write-back worked. `session` is the one taken in the first cell (§3.3); calling `get_active_session()` here, after Kumo has written, @@ -645,8 +668,10 @@ the write, as the host's `runtime.md` says (§9.1 for Snowflake). ## 10. Complete verified example -On Snowflake. On any other connector, replace steps 1, 7 and 8 with the host `runtime.md`'s -connect step, the §9.2 destination and its read-back; nothing else changes. +On Snowflake, in a host whose `runtime.md` provides an active Snowpark session (§3.3). On any +other host — Snowflake reached from outside it included — drop the `get_active_session()` line and +replace steps 1, 7 and 8 with that host `runtime.md`'s connect step, the §9.2 destination and its +SQL read-back; nothing else changes. Self-contained: `OUTPUT_DATABASE`, `OUTPUT_SCHEMA` and `OUTPUT_TABLE` are placeholders for the confirmed output destination (§12). @@ -664,7 +689,7 @@ from snowflake.snowpark.context import get_active_session OUTPUT_DATABASE, OUTPUT_SCHEMA, OUTPUT_TABLE = '', '', '
' # confirmed destination -session = get_active_session() +session = get_active_session() # §3.3: only where the host provides the session # 1. Scratch directory, then connect (inside SPCS; elsewhere use the §3 order). No credentials in the notebook. kumo.configure(scratch_dir=Path('/tmp/kumo-run') / 'scratch') @@ -808,7 +833,7 @@ targets a superseded build and needs rewriting against §10 — do not copy its | PQL `query=` | Generated per `kumoai-reference/references/pql_writing.md` | | `prediction_time=` | A known in-range **naive** `datetime`; **cannot be computed** | | `predict(num_classes_to_return=N)` | Results per entity the user asked for; defaults to `RANK TOP K` but may differ. Link prediction writing predictions requires it, 1–500; no cap otherwise (§9) | -| `predict(binary_classification_threshold=)` | Required for binary classification; never on `PredictionOutputConfig`. The threshold restated at Node 1d: the user's, else `0.5 [default]` (`kumo-data-science/accept/intake_contract.md`) | +| `predict(binary_classification_threshold=)` | Always passed for binary classification, written or local; never on `PredictionOutputConfig`. The threshold restated at Node 1d: the user's, else `0.5 [default]` (`kumo-data-science/accept/intake_contract.md`) | | Population table (static task, rows that already have a target) | Confirmed at Node 1d (§7.1) | | `PredictionOutputConfig(...)` + output connector | Output destination in its connector's form (§9.2); report it back unchanged — `DB.SCHEMA.TABLE`, or the path | @@ -824,14 +849,14 @@ targets a superseded build and needs rewriting against §10 — do not copy its `PredictionArtifactType` / `WriteMode` from `kumoapi.jobs`; `RunMode` from `kumoapi.model_plan`. - [ ] The connect step is the host `runtime.md`'s, with no credential literal (§3, §3.4). -- [ ] Snowflake: connection tries `from_spcs_env`, then `from_connection_toml`, then - `SnowflakeIdentity` from secrets (§3), with explicit `warehouse`, `database`, - `schema_name`; `SPCS_TOKEN_PATH` imported, never hardcoded. +- [ ] Snowflake: the factory the host's `runtime.md` names (§3), with explicit `warehouse`, + `database`, `schema_name`; `SPCS_TOKEN_PATH` imported, never hardcoded. - [ ] Snowflake: lower-case or mixed-case `database` / `schema_name` passed with their double quotes (§3). - [ ] Table and column names in the catalog's exact spelling (§3.2). -- [ ] Snowflake: `session = get_active_session()` once, in the first code cell; never again, no - `Session.builder` (§3.3). +- [ ] Snowflake, on a host that provides a Snowpark session: `session = get_active_session()` + once, in the first code cell; never again, no `Session.builder` (§3.3). Elsewhere, the + host `runtime.md`'s connection for SQL and read-back. - [ ] `connector.table_names()` asserted before building tables. - [ ] Every foreign key corrected with `.with_column(col, stype=Stype.ID)`. - [ ] `with_column(...)` results assigned or chained — it returns a new Table. @@ -848,9 +873,12 @@ targets a superseded build and needs rewriting against §10 — do not copy its - [ ] No code cell writes anywhere unless a destination was confirmed; an example of how to add one later is markdown text, not a code cell. - [ ] Smoke-test sizing called out as such. +- [ ] `write_mode` and any population replacement as Node 1d confirmed them (§9, §7.1); the + staging cell stops when the population name differs from `POPULATION_CONFIRMED`. - [ ] Destination in its connector's form (§9.2), reported with its exact name — no `_predictions` suffix. -- [ ] Written destination read back and row count compared to `prediction_result.num_rows`. +- [ ] Written destination read back and row count compared to `prediction_result.num_rows` — + for APPEND, the count added since before `predict` (§9.1). - [ ] Success judged by blocking calls returning without raising — not by a status value. --- diff --git a/agent-skills/skills/kumoai-reference/references/outputs.md b/agent-skills/skills/kumoai-reference/references/outputs.md index 2c0caf96f..04583d8d1 100644 --- a/agent-skills/skills/kumoai-reference/references/outputs.md +++ b/agent-skills/skills/kumoai-reference/references/outputs.md @@ -51,9 +51,10 @@ print(written.num_rows, written.files) ``` **APPEND rows say which run wrote them.** An APPEND write to a table adds `PREDICTION_ID` (this -prediction run) and `BUNDLE_ID` (the model that scored it), and the `extra_fields` above add -`JOB_TIMESTAMP` (when the run started) and `ANCHOR_TIMESTAMP` (the time its predictions are for), -so the table's history can be ordered and filtered by run, and a retried run shows as a second +prediction run) and `BUNDLE_ID` (the model that scored it), whatever `extra_fields` says (Source: +shipped package (`kumo.engine.trainer.output_config.PredictionOutputConfig`)). The `extra_fields` +above add `JOB_TIMESTAMP` (when the run started) and `ANCHOR_TIMESTAMP` (the time its predictions +are for), so the table's history can be ordered and filtered by run, and a retried run shows as a second `PREDICTION_ID` and `JOB_TIMESTAMP`. `BUNDLE_ID` is the model id the run's prediction manifest names. When there is none — `trainer.result.id` is `None` because the id could not be computed at fit, or `predict` warns `Predicting without a prediction manifest` — the writer leaves the @@ -80,6 +81,9 @@ other. Column types: | Databricks | `STRING` | `TIMESTAMP_NTZ` | | BigQuery | `STRING` | `TIMESTAMP` | +The timestamps take the type the destination's sink writes: on BigQuery a `DATETIME` column fails +the APPEND (Source: shipped package (`kumo_bigquery.write`, `kumo_databricks.write`)). + OVERWRITE onto an existing Snowflake table keeps the table and its grants, so the run's columns must be exactly the table's: an OVERWRITE run writes no run columns, so a table that earlier APPEND runs gave them fails, after scoring. Offer OVERWRITE there only when the columns match; diff --git a/agent-skills/skills/kumoai-reference/references/pql_writing.md b/agent-skills/skills/kumoai-reference/references/pql_writing.md index eabb16be2..e2100560d 100644 --- a/agent-skills/skills/kumoai-reference/references/pql_writing.md +++ b/agent-skills/skills/kumoai-reference/references/pql_writing.md @@ -212,7 +212,7 @@ Kumo determines the task type automatically. `pquery.get_task_type()` returns it | `LIST_DISTINCT` over a **non-foreign-key** categorical column | Multilabel classification | | `LIST_DISTINCT` over a **foreign key** column (+ `RANK TOP K`) | Link prediction / ranking | | `LIST_DISTINCT(...) CLASSIFY` | Separate binary classification per class | -| Temporal regression + model-plan `forecast_length` | Forecasting | +| Temporal regression meant as a forecast | Regression: `get_task_type()` reads the query alone, so it does not change when the generation plan sets `forecast_length`; that plan is what makes it a forecast (*Forecasting* below) | Static vs temporal: a query without a time window is **static** (default 80/10/10 random split, no time column required). A query aggregating over a window is **temporal** (time-based splits, anchor times sampled back through history). @@ -367,7 +367,7 @@ Produces a separate binary classification per article ID. Without `TOP K`, class ### Forecasting -Forecasting is **not a distinct PQL keyword**. Write a temporal regression query and set `forecast_length` in the model plan; `forecast_length=3` on a 7-day window forecasts the next three 7-day windows. +Forecasting is **not a distinct PQL keyword**. Write a temporal regression query, set `forecast_length` on the `TrainingTableGenerationPlan` passed to `generate_training_table(snapshot, plan=...)`, and pass the fitted trainer's `trainer.training_table_generation_plan` to `generate_prediction_table(snapshot, plan=...)` — both take a `TrainingTableGenerationPlan`, not a `PredictionTableGenerationPlan`, and never the model plan (Source: shipped package (`kumoapi.model_plan`, `kumo.engine.pquery.predictive_query`)); `forecast_length=3` on a 7-day window forecasts the next three 7-day windows. The flow is in `kumoai-reference/references/task_journeys.md` *Forecasting*. ```text PREDICT SUM(TRANSACTIONS.AMOUNT, 0, 7, days) @@ -429,7 +429,7 @@ Source: shipped package (`kumo_core.model_plan`) 8. **`LAST`/`FIRST` cannot be applied to foreign key columns.** Use `LIST_DISTINCT` + `RANK` for FK targets. 9. **`LIST_DISTINCT` over a non-FK column silently becomes multilabel classification, not ranking.** Check `get_task_type()`. 10. **`LIST_DISTINCT` beyond ~1000 distinct values is not recommended.** -11. **Horizon vs. split health:** too large a window leaves fewer than 3 anchor times; too small a window on a long history generates too many examples (mitigate with model-plan `train_start_offset`) and, for link prediction, too few positives. +11. **Horizon vs. split health:** too large a window leaves fewer than 3 anchor times; too small a window on a long history generates too many examples (mitigate with `train_start_offset` on the `TrainingTableGenerationPlan` passed to `generate_training_table(snapshot, plan=...)`; the model plan has no such field) and, for link prediction, too few positives. 12. **Always validate before generating tables:** `pquery.validate()` for syntax, `pquery.get_task_type()` for semantics. Required — see "Required pre-flight" under SDK usage. 13. **Inline `WHERE` vs trailing `WHERE` are not interchangeable.** A row-level condition on the predicted event goes inside the aggregation; an entity-eligibility condition goes in the trailing `WHERE`. Swapping them produces a query that validates and trains on the wrong question. diff --git a/agent-skills/tests/test_no_environment_setup.py b/agent-skills/tests/test_no_environment_setup.py index 5f1c1bb6d..15637274d 100644 --- a/agent-skills/tests/test_no_environment_setup.py +++ b/agent-skills/tests/test_no_environment_setup.py @@ -7,14 +7,17 @@ package-file path, and the agent asked the user for one anyway: the placeholder existed only inside the rule that forbade it. So the words are checked by absence, in every file under `skills/`, including the host files about Snowflake ML Jobs, where Kumo reaches the job through host-supplied -values (`KUMO_JOB_RUNTIME`, `KUMO_JOB_REQUIREMENTS`) that the skill only names. +values (`KUMO_JOB_RUNTIME`, `KUMO_JOB_REQUIREMENTS`) that the skill only names. The catalog's skill +cards and evals under `nvskills/` are published beside the skills, so they are checked too. """ import re from pathlib import Path import pytest -SKILLS = Path(__file__).resolve().parents[1] / 'skills' +ROOT = Path(__file__).resolve().parents[1] +SKILLS = ROOT / 'skills' +CATALOG = ROOT / 'nvskills' # `pip` as a word or as the `pip_` prefix of a submission keyword; `.zip` / `.whl` as extensions, # so `zipfile.ZipFile` in the retention script does not count. @@ -22,16 +25,17 @@ def skill_files() -> list[Path]: - return sorted(p for p in SKILLS.rglob('*') + return sorted(p for root in (SKILLS, CATALOG) for p in root.rglob('*') if p.is_file() and '__pycache__' not in p.parts) def test_files_are_found() -> None: names = {p.name for p in skill_files()} - assert {'SKILL.md', 'adapter.md', 'ml_job.md', 'retain_artifacts.py'} <= names + assert {'SKILL.md', 'adapter.md', 'ml_job.md', 'retain_artifacts.py', + 'skill-card.md', 'evals.json'} <= names -@pytest.mark.parametrize('path', skill_files(), ids=lambda p: str(p.relative_to(SKILLS))) +@pytest.mark.parametrize('path', skill_files(), ids=lambda p: str(p.relative_to(ROOT))) def test_no_setup_words(path: Path) -> None: hits = [f'{n}: {line.strip()}' for n, line in enumerate(path.read_text().splitlines(), 1) if SETUP_WORDS.search(line)] diff --git a/agent-skills/tests/test_publish_nvskills.py b/agent-skills/tests/test_publish_nvskills.py new file mode 100644 index 000000000..1c4bef53e --- /dev/null +++ b/agent-skills/tests/test_publish_nvskills.py @@ -0,0 +1,235 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. +# All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +import importlib.util +import sys +from pathlib import Path + +import pytest + +SCRIPT = Path(__file__).resolve().parents[1] / 'scripts' / 'publish_nvskills.py' +spec = importlib.util.spec_from_file_location('publish_nvskills', SCRIPT) +assert spec and spec.loader +publish_nvskills = importlib.util.module_from_spec(spec) +sys.modules['publish_nvskills'] = publish_nvskills +spec.loader.exec_module(publish_nvskills) + + +# The router and every skill the generic package nests inside it, each published beside it. +ROUTER = publish_nvskills.ROUTER +SKILLS = [ROUTER, 'kumoai-reference', *publish_nvskills.package_skill.CAPABILITY] + + +@pytest.fixture +def catalog(tmp_path: Path) -> Path: + root = tmp_path / 'catalog' + for skill in SKILLS: + (root / skill / 'evals').mkdir(parents=True, exist_ok=True) + (root / skill / 'evals' / 'evals.json').write_text(f'{{"skill_name": "{skill}"}}\n') + (root / skill / 'skill-card.md').write_text('## Description\n') + return root + + +def test_publish_writes_the_generic_package_and_catalog_files(tmp_path: Path, + catalog: Path) -> None: + dest = tmp_path / 'skills' + assert publish_nvskills.publish(dest, catalog) + assert (dest / ROUTER / 'SKILL.md').is_file() + assert (dest / ROUTER / 'hosts' / 'generic' / 'adapter.md').is_file() + assert not (dest / ROUTER / 'hosts' / 'snowflake-coco').exists() + assert sorted(publish_nvskills.skill_dirs(dest)) == sorted(SKILLS) + for skill in SKILLS: + assert (dest / skill / 'evals' / 'evals.json').is_file() + assert (dest / skill / 'skill-card.md').is_file() + assert publish_nvskills.missing(dest) == [] + assert publish_nvskills.stale(dest, catalog) == [] + + +def test_no_skill_is_published_inside_another(tmp_path: Path, catalog: Path) -> None: + """The catalog does not allow a skill inside a skill: each SKILL.md has its own directory.""" + dest = tmp_path / 'skills' + publish_nvskills.publish(dest, catalog) + assert publish_nvskills.nested(dest) == [] + assert not (dest / ROUTER / 'kumoai-reference').exists() + assert (dest / 'kumoai-reference' / 'references' / 'kumo_finetuning_sdk.md').is_file() + assert publish_nvskills.unresolved(dest) == [] + + +def test_the_package_cannot_nest_a_skill_two_levels_down(tmp_path: Path, catalog: Path, + monkeypatch: pytest.MonkeyPatch) -> None: + build = publish_nvskills.package_skill.build + + def deeper(host: str, dist: Path) -> Path: + pkg = build(host, dist) + (pkg / 'kumo-model-tuning' / 'extra').mkdir() + (pkg / 'kumo-model-tuning' / 'extra' / 'SKILL.md').write_text('---\nname: extra\n---\n') + return pkg + + monkeypatch.setattr(publish_nvskills.package_skill, 'build', deeper) + with pytest.raises(SystemExit, match='nests skills below its top level'): + publish_nvskills.publish(tmp_path / 'skills', catalog) + + +@pytest.mark.parametrize('text, error', [ + ('read `accept/preflight.md`', "is another skill's file"), + ('read `kumoai-reference/references/absent.md`', 'does not resolve'), + ('read `references/absent.md`', 'does not resolve'), + ('see [plans](../kumo-model-tuning/references/absent.md)', 'does not resolve'), +]) +def test_references_are_checked_one_directory_per_skill(tmp_path: Path, catalog: Path, + text: str, error: str) -> None: + """A path a nested package resolves can miss once the skills sit side by side.""" + dest = tmp_path / 'skills' + publish_nvskills.publish(dest, catalog) + doc = dest / 'kumo-model-tuning' / 'references' / 'plans.md' + doc.write_text(doc.read_text() + f'\n{text}\n') + errors = publish_nvskills.unresolved(dest) + assert len(errors) == 1 and error in errors[0], errors + + +@pytest.mark.parametrize('rel', ['kumoai-reference/skill-card.md', + 'kumo-model-tuning/evals/evals.json', + 'kumo-data-science/evals/evals.json']) +def test_every_skill_directory_needs_its_own_card_and_evals(tmp_path: Path, catalog: Path, + rel: str) -> None: + """The catalog sync deletes a SKILL.md directory without its own card and evals. + + That also breaks the other skills' links into it, so the build refuses. + """ + (catalog / rel).unlink() + dest = tmp_path / 'skills' + with pytest.raises(SystemExit, match='the catalog would drop these skills'): + publish_nvskills.publish(dest, catalog) + assert not dest.exists() + + +def test_without_catalog_files_every_input_is_reported_missing(tmp_path: Path) -> None: + absent = tmp_path / 'no-catalog' + dest = tmp_path / 'skills' + assert publish_nvskills.publish(dest, absent) + todo = publish_nvskills.not_ready(absent) + for skill in SKILLS: + assert f'{skill}/skill-card.md' in todo + assert f'{skill}/evals/*.json' in todo + assert len(todo) == 2 * len(SKILLS) + + +@pytest.mark.parametrize('rel', ['notes.md', 'skill-card.md', 'evals/evals.json', + 'kumo-data-science/SKILL.md', + 'kumo-data-science/hosts/generic/adapter.md', + 'kumo-data-science/hosts/skill-card.md', + 'kumo-data-science/hosts/evals/evals.json', + 'kumo-data-science/kumo-model-tuning/skill-card.md', + 'kumo-model-tunning/skill-card.md']) +def test_catalog_files_must_belong_to_a_skill(tmp_path: Path, catalog: Path, rel: str) -> None: + (catalog / rel).parent.mkdir(parents=True, exist_ok=True) + (catalog / rel).write_text('stray\n') + with pytest.raises(SystemExit, match='is not the skill-card.md or evals/ of a skill'): + publish_nvskills.publish(tmp_path / 'skills', catalog) + + +def test_publish_keeps_signed_files_and_drops_files_no_longer_shipped(tmp_path: Path, + catalog: Path) -> None: + dest = tmp_path / 'skills' + publish_nvskills.publish(dest, catalog) + signed = [dest / skill / name for skill in SKILLS for name in publish_nvskills.SIGNED] + for path in signed: + path.write_text('from NVSkills CI\n') + # The nested layout this directory held before it was split into one directory per skill. + (dest / ROUTER / 'kumoai-reference').mkdir() + (dest / ROUTER / 'kumoai-reference' / 'SKILL.md').write_text('gone\n') + + assert publish_nvskills.stale(dest, catalog) == [f'- {ROUTER}/kumoai-reference/SKILL.md'] + publish_nvskills.publish(dest, catalog) + assert not (dest / ROUTER / 'kumoai-reference').exists() + for path in signed: + assert path.read_text() == 'from NVSkills CI\n' + + +def test_a_signed_skill_keeps_the_card_nvskills_ci_wrote(tmp_path: Path, + catalog: Path) -> None: + """NVSkills CI rewrites a skill's card when it signs; the source card only seeds it.""" + dest = tmp_path / 'skills' + publish_nvskills.publish(dest, catalog) + signed, unsigned = dest / 'kumo-model-tuning', dest / 'kumoai-reference' + (signed / 'skill.oms.sig').write_text('from NVSkills CI\n') + for skill in (signed, unsigned): + (skill / 'skill-card.md').write_text('## Description\n\n## Evaluation Results\n') + + assert publish_nvskills.stale(dest, catalog) == ['~ kumoai-reference/skill-card.md'] + (dest / ROUTER / 'SKILL.md').write_text('edited by hand\n') + publish_nvskills.publish(dest, catalog) + assert (signed / 'skill-card.md').read_text() == '## Description\n\n## Evaluation Results\n' + assert (unsigned / 'skill-card.md').read_text() == '## Description\n' + assert publish_nvskills.stale(dest, catalog) == [] + + +def test_check_ignores_the_build_stamp_but_not_content(tmp_path: Path, catalog: Path) -> None: + dest = tmp_path / 'skills' + publish_nvskills.publish(dest, catalog) + router = dest / ROUTER / 'SKILL.md' + text = router.read_text(encoding='utf-8') + router.write_text(publish_nvskills.package_skill.MARKER.sub('SKILL-BUILD: 2000-01-01-0', text), + encoding='utf-8') + assert publish_nvskills.stale(dest, catalog) == [] + + router.write_text(text + '\nedited by hand\n', encoding='utf-8') + assert publish_nvskills.stale(dest, catalog) == [f'~ {ROUTER}/SKILL.md'] + + +@pytest.mark.parametrize('rel', ['kumo-data-science/skill.oms.sig', + 'kumo-data-science/evals/skill.oms.sig', + 'kumo-data-science/evals/BENCHMARK.md', + 'kumoai-reference/skill.oms.sig', + 'kumo-model-tuning/BENCHMARK.md']) +def test_catalog_files_cannot_supply_signed_files(tmp_path: Path, catalog: Path, + rel: str) -> None: + (catalog / rel).write_text('forged\n') + with pytest.raises(SystemExit, match='written by NVSkills CI'): + publish_nvskills.publish(tmp_path / 'skills', catalog) + + +@pytest.mark.parametrize('rel', ['skill-card.md', 'kumo-model-tuning/evals/evals.json']) +def test_catalog_files_cannot_replace_package_files(tmp_path: Path, catalog: Path, + monkeypatch: pytest.MonkeyPatch, + rel: str) -> None: + build = publish_nvskills.package_skill.build + + def shipping(host: str, dist: Path) -> Path: + pkg = build(host, dist) + (pkg / rel).parent.mkdir(parents=True, exist_ok=True) + (pkg / rel).write_text('from the package\n') + return pkg + + monkeypatch.setattr(publish_nvskills.package_skill, 'build', shipping) + with pytest.raises(SystemExit, match='already in the generic package'): + publish_nvskills.publish(tmp_path / 'skills', catalog) + + +def test_the_package_cannot_ship_signed_files(tmp_path: Path, catalog: Path, + monkeypatch: pytest.MonkeyPatch) -> None: + build = publish_nvskills.package_skill.build + + def with_signature(host: str, dist: Path) -> Path: + pkg = build(host, dist) + (pkg / 'kumoai-reference' / 'BENCHMARK.md').write_text('forged\n') + return pkg + + monkeypatch.setattr(publish_nvskills.package_skill, 'build', with_signature) + dest = tmp_path / 'skills' + with pytest.raises(SystemExit, match='written by NVSkills CI'): + publish_nvskills.publish(dest, catalog) + assert not dest.exists() + + +def test_the_published_copy_matches_the_source() -> None: + """The copy NVIDIA/skills syncs from main is rebuilt in the same PR as the change. + + A PR that edits agent-skills/ without refreshing skills/ would merge a published copy that + no longer matches its source, so it fails here instead. + """ + changes = publish_nvskills.stale() + assert changes == [], ( + 'skills/ is out of date; run ' + f'python3 agent-skills/scripts/publish_nvskills.py and commit the result: {changes}' + ) diff --git a/skills/kumo-custom-populations/BENCHMARK.md b/skills/kumo-custom-populations/BENCHMARK.md new file mode 100644 index 000000000..b86790de1 --- /dev/null +++ b/skills/kumo-custom-populations/BENCHMARK.md @@ -0,0 +1,155 @@ +# Skill Benchmark: kumo-custom-populations + +> ✅ **Overall verdict: PASS — Recommended for publication** + +## Publication Recommendation + +Recommended for publication based on the completed evaluation evidence in this report. + +## Evaluation Metadata + +- Skill: `kumo-custom-populations` +- Validation status: `passed` +- Evaluation date: 2026-11-19 +- Evaluator version: `1.6.10` +- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-5-5`), Codex (`openai/openai/gpt-6.1-sol`) +- Tasks: 5 evaluation tasks (4 positive, 1 negative) +- Dataset digest: `sha256:ed3c0e36c21c14588ddb79e7ed42113b76bddfa7ad9f90f3e27392b0fffdf2f2` (skill-evaluator-dataset-snapshot/1) +- Attempts per task: 1 +- Environment: `k8s-sandbox` +- Tier 2 evidence: required for publication +- Tier 3 evidence: required for publication + +Each task attempt ran in its own isolated sandbox pod. + +## Tier 1 LLM Provenance + +SkillSpector provider, model, analyzer revision, and determinism evidence retained for this scan. + +| Skill | Provider | Adapter | Analyzer | Model | Revision | Determinism | +| --- | --- | --- | --- | --- | --- | --- | +| kumo-custom-populations | nv_inference | openai | mcp_tool_poisoning | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.5` | nondeterministic / provider_defaults | +| kumo-custom-populations | nv_inference | openai | semantic_developer_intent | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.5` | nondeterministic / provider_defaults | +| kumo-custom-populations | nv_inference | openai | semantic_quality_policy | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.5` | nondeterministic / provider_defaults | +| kumo-custom-populations | nv_inference | openai | semantic_security_discovery | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.5` | nondeterministic / provider_defaults | +| kumo-custom-populations | nv_inference | openai | meta_analyzer | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.5` | nondeterministic / provider_defaults | + +- Controls for `kumo-custom-populations`: temperature=not set (source=provider_default, forwarded=no); seed=not set (source=unset, forwarded=no); reasoning_effort=not set (source=provider_default, forwarded=no) + +## What This Report Answers + +The three-tier evaluation checks whether the skill: + +- is safe to use; +- produces correct answers; +- is discovered and activated when needed; +- helps the agent complete the user's goal and expected workflow; and +- avoids wasted skill and tool usage. + +## Results at a Glance + +| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | +|---|---:|---:| +| Overall | 96.3% — baseline ran, but no comparable score was available; uplift unavailable | 92.4% — baseline ran, but no comparable score was available; uplift unavailable | +| Security | 100.0% → 100.0% (±0.0 points) | 100.0% → 100.0% (±0.0 points) | +| Correctness | 52.0% → 100.0% (+48.0 points) | 72.0% → 96.0% (+24.0 points) | +| Discoverability | 97.5% — baseline ran, but no comparable score was available; uplift unavailable | 93.8% — baseline ran, but no comparable score was available; uplift unavailable | +| Effectiveness | 52.0% → 86.0% (+34.0 points) | 46.0% → 76.0% (+30.0 points) | +| Efficiency | 98.1% — baseline ran, but no comparable score was available; uplift unavailable | 96.0% — baseline ran, but no comparable score was available; uplift unavailable | + +**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points. + +Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline. + +A partial dimension was calculated from only the available configured signals; review the detailed report before relying on it. + +### Comparison details + +Shared-source comparisons use the same signals and weights on both conditions. They do not replace the full dimension scores above or change the publication verdict. + +| Agent | Dimension | Basis | Baseline → With skill (uplift) | Explanation | +| --- | --- | --- | --- | --- | +| claude-code | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| claude-code | Efficiency | Token-only comparison | 98.0% → 96.3% (-1.7 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | +| codex | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| codex | Efficiency | Token-only comparison | 88.5% → 92.1% (+3.6 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | + +## Token Usage + +Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions. + +| Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage | +|---|---|---:|---:|---:|---:|---| +| claude-code | All cases | 612,662 | 615,210 | -2,548 | -0.41% | skill 5/5; base 5/5 | +| claude-code | kpop-01 | 154,125 | 110,465 | +43,660 | +39.52% | skill 1/1; base 1/1 | +| claude-code | kpop-02 | 153,436 | 144,504 | +8,932 | +6.18% | skill 1/1; base 1/1 | +| claude-code | kpop-03 | 154,844 | 142,115 | +12,729 | +8.96% | skill 1/1; base 1/1 | +| claude-code | kpop-04 | 114,529 | 182,472 | -67,943 | -37.23% | skill 1/1; base 1/1 | +| claude-code | kpop-05 | 35,728 | 35,654 | +74 | +0.21% | skill 1/1; base 1/1 | +| codex | All cases | 243,157 | 280,860 | -37,703 | -13.42% | skill 5/5; base 5/5 | +| codex | kpop-01 | 60,533 | 65,793 | -5,260 | -7.99% | skill 1/1; base 1/1 | +| codex | kpop-02 | 60,696 | 60,548 | +148 | +0.24% | skill 1/1; base 1/1 | +| codex | kpop-03 | 60,404 | 59,522 | +882 | +1.48% | skill 1/1; base 1/1 | +| codex | kpop-04 | 43,726 | 77,426 | -33,700 | -43.53% | skill 1/1; base 1/1 | +| codex | kpop-05 | 17,798 | 17,571 | +227 | +1.29% | skill 1/1; base 1/1 | +| ALL AGENTS | Dataset aggregate | 855,819 | 896,070 | -40,251 | -4.49% | skill 10/10; base 10/10 | + +Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. Change is `100 * (with - without) / without`; negative means fewer tokens. N/A comparisons include their reason in the coverage column; coverage is never estimated. + +## Tier Status + +| Tier | Purpose | Status | Evidence | +|---|---|---|---| +| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 11 finding(s) | +| Tier 2 | Semantic deduplication | **PASSED** | 2 validator(s); 0 finding(s) | +| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 5 task(s) | + +## Findings and Observations + +
+Show detailed findings and successful checks + +- **MEDIUM** QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.author' (`skills/kumo-custom-populations/SKILL.md`) +- **MEDIUM** QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.tags' (`skills/kumo-custom-populations/SKILL.md`) +- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/kumo-custom-populations/SKILL.md`) +- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/kumo-custom-populations/SKILL.md`) +- **MEDIUM** SCHEMA/author_missing: Author not specified in metadata (`skills/kumo-custom-populations/SKILL.md`) +- 6 additional finding(s) are available in the full evaluation artifacts. + +
+ +## Scoring Methodology + +
+Show dimension definitions, source signals, and thresholds + +| Dimension | Question | Scored signals | +|---|---|---| +| Security | Is it safe to use? | `security` (100%) | +| Correctness | Is the answer correct? | `accuracy` (100%) | +| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) | +| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) | +| Efficiency | Did it avoid wasted tool calls and token usage? | `skill_efficiency` (50%) + `token_efficiency` (50%) | + +- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%. +- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL. +- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate. +- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold. +- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`). +- Efficiency is 50% tool-call productivity (the backward-compatible `skill_efficiency` wire id) and 50% `token_efficiency`. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial. + +Signals present in this run: + +- `security` (Security): unsafe operations, secret leakage, and unauthorized access. +- `skill_execution` (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed. +- `skill_efficiency` (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability). +- `accuracy` (Accuracy): final-answer correctness against the reference answer. +- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved. +- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed. +- `token_efficiency` (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency). + +
+ +## Freshness + +Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes. diff --git a/skills/kumo-custom-populations/SKILL.md b/skills/kumo-custom-populations/SKILL.md new file mode 100644 index 000000000..91eefa007 --- /dev/null +++ b/skills/kumo-custom-populations/SKILL.md @@ -0,0 +1,34 @@ +--- +name: kumo-custom-populations +description: Train with caller-supplied labels, weights and splits or score exact entity/anchor rows through Snowflake source tables using the local Kumo API. Use when generated populations do not represent the requested rows. +--- + +# Caller-supplied populations + +Preserve the query, graph and evaluation protocol. Read +[the population contract](references/populations.md) before creating a supplied table. +Snowflake sources stage local Parquet for this API. Do not switch to another connector or +claim that a local bundle is a source table. + +Resolve the supplied source table and required column semantics. These constructors consume +existing tables; they do not authorize creating labels, renaming data, cleaning nulls or +changing splits or weights. Use a user-approved prepared table when the source lacks required +fields. + +Validate schema, row types, keys, timestamps, labels, weights and splits and report the exact +bad field. Do not silently drop rows, cast IDs, normalize weights or replace manual splits: a +field that needs changing is reported and asked about. Split labels and weight derivations follow +the same rule: show the mapping or the exact formula, the columns it reads and the resulting +distribution from a read-only query, and stop for the user's explicit approval. An approval given +before that is shown is not approval. Only then apply it in a read-only `connector.sql(...)` +source, never as a write to the source table, an UPDATE, a CREATE TABLE AS or a derived table +(`references/populations.md`). Validation still applies to the result. + +Before any code, reply with the plan and stop for confirmation: the table and its fields, the +weight and split columns, what you validated, the constructor and that it reuses the confirmed +`pq` and `snapshot`, and any shape the host adapter says is unverified there. For scoring, add +that validation does not guarantee every row is scored, that no id is dropped, cast or remapped, +and that the output's entity and anchor coverage will be compared with the population. A bad +field gets the ask-back instead. +Confirm the snapshot used to create the population and the snapshot passed to fit/predict +belong to the intended graph. Rebuild populations after incompatible graph changes. diff --git a/skills/kumo-custom-populations/evals/evals.json b/skills/kumo-custom-populations/evals/evals.json new file mode 100644 index 000000000..45adaf00d --- /dev/null +++ b/skills/kumo-custom-populations/evals/evals.json @@ -0,0 +1,72 @@ +{ + "skill_name": "kumo-custom-populations", + "evals": [ + { + "id": "kpop-01", + "prompt": "Train my confirmed Kumo churn query on our own labels instead of generated ones. The Snowflake table ML.LABELS.CURATED_LABELS has ENTITY1, TARGET1, TIMESTAMP, WEIGHT and SPLIT (0 train, 1 validation, 2 test). `pq` and `snapshot` are already built in the notebook.", + "expected_output": "Before any code it replies with the plan and stops for confirmation: the table and its fields, WEIGHT and SPLIT as the weight and split columns, what it validated, and the constructor `kumo.TrainingTable.from_source_table(population_connector['CURATED_LABELS'], TrainingTableSpec(weight_col='WEIGHT', split_col='SPLIT'), pq=pq, snapshot=snapshot)`, reusing the confirmed pq and snapshot, with `suggest_model_plan(..., train_table_spec=spec)`.", + "assertions": [ + "Names kumo.TrainingTable.from_source_table as the constructor", + "Uses TrainingTableSpec with weight_col='WEIGHT' and split_col='SPLIT'", + "Reuses the existing pq and snapshot", + "Passes the same spec to suggest_model_plan as train_table_spec", + "Presents the plan and stops for the user's confirmation before training" + ], + "expected_skill": "kumo-custom-populations", + "expected_script": null + }, + { + "id": "kpop-02", + "prompt": "Our Kumo labels table ML.LABELS.CURATED_LABELS has a SPLIT_LABEL column with the values 'train', 'val' and 'test'. Use it as the split column and train.", + "expected_output": "Reports SPLIT_LABEL as the offending field: the split column reads only 0, 1 and 2, so string labels match no split and their rows are skipped (the build fails with 'custom-TT source kept no rows' if none remain). It shows the labels and their counts from a read-only query and asks for either a prepared table with 0/1/2 or explicit approval of a shown mapping, which would then be applied in a read-only connector.sql(...) source. It writes nothing back.", + "assertions": [ + "Says the split column must hold 0, 1 or 2 and that 'train'/'val'/'test' will not match", + "Explains that unmatched rows are skipped, so training could keep no rows", + "Asks for a prepared 0/1/2 table or explicit approval of a mapping it shows", + "Proposes the mapping only as a read-only connector.sql(...) source, after approval", + "Does not UPDATE the table, CREATE TABLE AS, or write a derived table" + ], + "expected_skill": "kumo-custom-populations", + "expected_script": null + }, + { + "id": "kpop-03", + "prompt": "For my Kumo custom training table, just normalize the WEIGHT column to mean 1 and multiply positives by 10. You have my approval, go ahead.", + "expected_output": "An approval given before the formula and distribution are shown is not approval. It states the exact formula and the columns it reads, shows the resulting weight distribution (min, mean, max, counts per label) from a read-only query, and stops for explicit approval. Only then would it apply the formula in a read-only connector.sql(...) source; it never writes to the source table.", + "assertions": [ + "Does not apply the weight change straight away despite the up-front approval", + "Shows the exact weight formula and the columns it reads", + "Shows or proposes a read-only query for the resulting weight distribution", + "Stops for explicit approval after showing them", + "Does not UPDATE the source table or write a derived table" + ], + "expected_skill": "kumo-custom-populations", + "expected_script": null + }, + { + "id": "kpop-04", + "prompt": "Score exactly the accounts in ML.EVAL.EVAL_ACCOUNTS (ACCOUNT_ID, ANCHOR_TS, CHURNED) with my fitted Kumo trainer, so I can compare against CHURNED afterwards.", + "expected_output": "The scoring population must not carry the target, so the table cannot be passed as is. Evaluation labels (CHURNED) stay separate and are joined to the scores afterwards. It proposes, for approval, a read-only connector.sql(...) source keeping only the entity and TIMESTAMP columns under their contract names, or asks for a table without the target, and scores with `kumo.PredictionTable.from_source_table(..., pq=pq, snapshot=snapshot, plan=trainer.training_table_generation_plan)` then `trainer.predict(population, snapshot=snapshot, ...)`, comparing output entity and anchor coverage with the population.", + "assertions": [ + "Says the scoring population must not contain the target column CHURNED", + "Keeps CHURNED separate for joining to the scores afterwards", + "Proposes a read-only connector.sql(...) source with the entity and timestamp columns, or asks for a table without the target", + "Uses kumo.PredictionTable.from_source_table with pq and snapshot", + "Says output rows or coverage will be compared with the population rather than assumed equal" + ], + "expected_skill": "kumo-custom-populations", + "expected_script": null + }, + { + "id": "kpop-05", + "prompt": "Split my pandas DataFrame into stratified train, validation and test sets (70/15/15) with scikit-learn.", + "expected_output": "A scikit-learn data-splitting task unrelated to Kumo. The agent answers with train_test_split, without loading the Kumo custom-populations skill.", + "assertions": [ + "Does not load or follow the kumo-custom-populations skill", + "Uses scikit-learn train_test_split with stratification to produce 70/15/15 splits" + ], + "expected_skill": null, + "expected_script": null + } + ] +} diff --git a/skills/kumo-custom-populations/references/populations.md b/skills/kumo-custom-populations/references/populations.md new file mode 100644 index 000000000..dbcf742ad --- /dev/null +++ b/skills/kumo-custom-populations/references/populations.md @@ -0,0 +1,174 @@ +# Supplied Snowflake population contract + +The constructors take a `SourceTable`, not a pandas `data_df` and not a table name string. +Use a Snowflake connector scoped to the supplied table, which may differ from the graph's +source schema. The query `pq` and `snapshot` describe the already confirmed graph/task. + +## Columns and task shapes + +| Field | Contract | +| --- | --- | +| `ENTITY1` | Non-null entity key, with numeric/string type compatible with the graph entity PK | +| `TARGET1` | Required for training; scalar for binary, regression and multiclass. Multilabel uses a scalar string encoded with the target's multicategorical delimiter | +| `TIMESTAMP` | Required temporal anchor. For a static node task it is required, training or prediction, when the entity table has a time column, and omitted when it has none. For static link prediction, training never needs it; prediction needs it when the entity table has a time column | +| Named weight | Optional training-only finite nonnegative scalar for scalar targets; at least one positive weight is required | +| Named split | Optional training-only values 0=train, 1=validation, 2=test (numeric values or digit strings) | + +Training link targets require a typed list of RHS IDs, not a scalar string or arbitrary JSON. +Link weights must be lists matching target lengths; temporal link tasks permit signed weights, +with at least one positive weight. Forecast rows require a scalar `TARGET1` plus precomputed `future_labels` for training, +`autoregression_labels` when lags are enabled, and `year_over_year_labels` when enabled. +The constructors preserve these sequences; they do not derive them from scalar anchors. + +Snowflake ARRAY/VARIANT source columns are exposed as strings by this connector. Do not +promise that an ordinary ARRAY label or forecast-sequence column supplies the typed lists +these paths require. Use generated link/forecast populations for those shapes in this +Snowflake workflow, or report the supplied-shape limitation. Do not silently replace an +explicit exact-population request with generated rows. Scalar supplied populations and +entity/anchor prediction populations remain supported for tasks without sequence requirements. +Exact forecast prediction still needs enabled lag and year-over-year sequences. + +A prediction population must not contain numbered target columns (`TARGET1`, `TARGET2`, etc.). +Keep evaluation labels separately rather than leaking them into the scoring population. +If the supplied table carries a target column, say that the scoring population must not carry it +and that evaluation labels stay separate, joined to the scores afterwards. Never pass the table as +is. Propose, for the user's approval, a read-only `connector.sql(...)` source (Snowflake, +Databricks and BigQuery only) that keeps only the entity column, the `TIMESTAMP` where the +task needs one (a temporal task, or a static task, node or link, on an entity table with a time +column), and any sequence columns the task requires +(`autoregression_labels`, `year_over_year_labels`), or ask for a table without the target. A temporal prediction population with no +`TIMESTAMP` per row cannot be built from the table as it stands: ask for the anchors, and do not +substitute a generated population with one `prediction_time`. +No public column-renaming map exists on these constructors. Fields need their contract names. + +## Train with supplied labels, weights and manual splits + +The following consumes an existing scalar-label table `CURATED_LABELS` containing `ENTITY1`, +`TARGET1`, the applicable `TIMESTAMP`, `WEIGHT` and `SPLIT`. Select the source only after its +semantics and split design are confirmed. + +```python +import kumo.engine as kumo +from kumoapi.train import TrainingTableSpec +from kumoapi.model_plan import RunMode + +spec = TrainingTableSpec(weight_col="WEIGHT", split_col="SPLIT") +training_table = kumo.TrainingTable.from_source_table( + population_connector["CURATED_LABELS"], spec, pq=pq, snapshot=snapshot, +) +model_plan = pq.suggest_model_plan(snapshot, run_mode=RunMode.FAST, train_table_spec=spec) +trainer = kumo.Trainer(model_plan) +training_result = trainer.fit(training_table, snapshot=snapshot) +``` + +Use `TrainingTableSpec()` when no weights or manual splits are supplied. Without `split_col`, +normal generation-plan split inference applies; a public `TrainingTableGenerationPlan` may +be passed as `plan=`. Do not assume every task uses the same random split. A supplied split +column overrides split assignment with 0/1/2. Inspect required nonempty splits for the +requested evaluation. Generated training tables are a distinct path and do not accept a +caller-supplied weight column by passing `TrainingTableSpec` to generation. + +## String or other non-numeric splits + +`split_col` reads only 0 (train), 1 (validation), 2 (test), as numbers or digit strings. A column +holding labels such as `'train'` / `'val'` / `'test'` is accepted as a string column if passed +as is, but no row matches a split. Ingestion skips every row it cannot read, so the build then +fails with `custom-TT source kept no rows`. When only some labels are unreadable, those rows are +skipped: the build succeeds on fewer rows and logs a warning with the skipped count per reason +(also recorded in `row_failures`). So report the split column as the offending field, with its +labels and counts (from the host's SQL tool), and ask: a prepared table holding 0/1/2, or the +user's explicit approval of a mapping you show. Only after that approval, map in a read-only +query; the result is a source table, so nothing is written: + +```sql +SELECT SPLIT_LABEL, COUNT(*) AS N FROM CURATED_LABELS GROUP BY SPLIT_LABEL; +``` + +```python +labeled = population_connector.sql( + "SELECT ENTITY1, TARGET1, TIMESTAMP, WEIGHT, " + "CASE SPLIT_LABEL WHEN 'train' THEN 0 WHEN 'val' THEN 1 WHEN 'test' THEN 2 END AS SPLIT " + "FROM CURATED_LABELS" +) +training_table = kumo.TrainingTable.from_source_table( + labeled, TrainingTableSpec(weight_col="WEIGHT", split_col="SPLIT"), pq=pq, snapshot=snapshot, +) +``` + +Any value in that result that matches no branch (including NULL) becomes a NULL `SPLIT` and the +row is skipped: ask the user how to map it before building the training table, and never leave it +out unreported. List the labels you mapped with their counts. The mapping stays in the read-only +`connector.sql(...)` source: write nothing back, and create no mapped copy or derived table. + +## Derived or normalized weights + +Weights follow the same rule as splits. Asked to derive, scale or normalize weights (for example +positives x10, or `WEIGHT / mean(WEIGHT)`), report the request, then show the exact formula, the +columns it reads and the resulting weight distribution (min, mean, max, and counts per distinct +weight or per label) from a read-only query, and stop for the user's explicit approval. An +approval given before the formula and distribution were shown is not approval. Unapproved, report +and ask; do not transform. Only after approval, apply it in a read-only `connector.sql(...)` +source, as for approved split mappings above; keep the table's `SPLIT` (or the generated +training table's, if exported), and write nothing back: no UPDATE, no CREATE TABLE AS, no +derived table. The result must still be finite and nonnegative, except the signed weights +temporal link prediction permits above, with at least one positive weight. + +## Export a generated training table + +`TrainingTable.export` writes the rows of a table Kumo generated, with a `SPLIT` column +(0 train, 1 validation, 2 test), so people can review or edit them and fit on the edited copy: + +```python +training_table.export(kumo.TrainingTableOutputConfig( + connector=population_connector, table_name="REVIEW_TABLE", +)) +``` + +`training_table` is the table this session already generated. Export is not training: do not call +`fit`, regenerate the table or change the plan unless asked. If the session has no generated +table, say so and ask before generating one. `table_name` is the destination. `write_mode` +(`kumoapi.jobs.WriteMode`) defaults to `OVERWRITE`; replacing an existing table needs the user's +explicit approval. `APPEND` is warehouse-only, and only when they ask for it. A file connector +takes a directory and supports only `OVERWRITE`: to keep an existing export, write to a new +directory. Report the destination the call returns. + +To refit on an edited copy, read it back with `TrainingTable.from_source_table` and +`TrainingTableSpec(weight_col=..., split_col="SPLIT")` as above. Keep the exported `SPLIT` values. +Call `suggest_model_plan` with that same spec, because only the weighted spec changes the plan. + +## Score exact rows + +`SCORING_POPULATION` contains entity IDs and row-specific timestamps for a temporal task, +without target columns. It can be used with either a fitted or loaded trainer: + +```python +population = kumo.PredictionTable.from_source_table( + population_connector["SCORING_POPULATION"], pq=pq, snapshot=snapshot, + plan=trainer.training_table_generation_plan, +) +# Binary example; other task arguments follow kumoai-reference/references/task_journeys.md. +scores = trainer.predict( + population, snapshot=snapshot, binary_classification_threshold=THRESHOLD, +) +print(population.row_count, scores.num_rows) +``` + +Do not assume output rows equal population rows: class, ranking and forecast outputs expand +rows. Preserve every requested anchor; do not replace row anchors with one global time. + +## Actionable validation + +Empty sources, missing required columns, incompatible key types and list/scalar target mismatch +are errors. Row-level problems are not: a null entity or anchor, a nonfinite or disallowed +negative weight, a null split, or a split value outside 0/1/2 skips that row. The build logs a +warning with the skipped count per reason, records it in `row_failures`, and continues on the +remaining rows; if no row remains it fails with `custom-TT source kept no rows`. Check these +values before building, and report the offending field and required representation before +proposing a correction. +Verify that non-null IDs actually exist in the intended graph when exact coverage matters; +schema validation alone does not guarantee membership or prediction coverage. + +Population construction rejects a snapshot from another graph. Later fit/predict receives a +snapshot separately, so retain explicit lineage rather than assuming the population object +pins all graph data. A loaded trainer checks the target/entity query contract; do not bypass a +mismatch by rewriting the target. Inspect output key/anchor coverage as well as total rows. diff --git a/skills/kumo-custom-populations/skill-card.md b/skills/kumo-custom-populations/skill-card.md new file mode 100644 index 000000000..3c4f731a9 --- /dev/null +++ b/skills/kumo-custom-populations/skill-card.md @@ -0,0 +1,116 @@ +## Description:
+Train with caller-supplied labels, weights and splits or score exact entity/anchor rows through Snowflake source tables using the local Kumo API.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner +NVIDIA
+ +### License/Terms of Use:
+Apache 2.0
+## Use Case:
+Developers and data scientists using Kumo Engine use this skill to train models on their own Snowflake labels, weights and manual splits, or to score exact entity/anchor rows, when generated populations do not represent the requested rows.
+ +### Deployment Geography for Use:
+Global
+ +## Requirements / Dependencies:
+**Requires API Key or External Credential:** [Not Specified]
+**Credential Type(s):** [None identified]
+ +Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate.
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [Supplied Snowflake population contract](references/populations.md)
+- [Kumo Engine documentation](https://docs.nvidia.com/sdgm/kumo-engine/get-started-with-kumo-engine/overview)
+ + +## Skill Output:
+**Output Type(s):** [Analysis, Code, Configuration instructions]
+**Output Format:** [Markdown with inline Python and SQL code blocks]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [Presents a plan and stops for user confirmation before any code; read-only source queries only]
+ +## Evaluation Agents Used:
+- Claude Code (`aws/anthropic/bedrock-claude-opus-5-5`)
+- Codex (`openai/openai/gpt-6.1-sol`)
+ + + +## Evaluation Tasks:
+5 evaluation tasks (4 positive, 1 negative), 1 attempt per task, each in an isolated sandbox pod (`k8s-sandbox`); evaluator version `1.6.10`, evaluated 2026-11-19. Baseline is the same task attempted without the skill.
+ +## Evaluation Metrics Used:
+Reported benchmark dimensions:
+- Security: Is it safe to use? Scored from `security` (100%).
+- Correctness: Is the answer correct? Scored from `accuracy` (100%).
+- Discoverability: Was the right skill loaded when needed? Scored from `skill_execution` (100%).
+- Effectiveness: Did the skill help complete the task? Scored from `goal_accuracy` (50%) + `behavior_check` (50%).
+- Efficiency: Did it avoid wasted tool calls and token usage? Scored from `skill_efficiency` (50%) + `token_efficiency` (50%).
+ +Underlying evaluation signals used in this run:
+- `security`: Unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution`: Whether the expected skill was selected, decoys were avoided, and the workflow executed.
+- `skill_efficiency`: Tool-call productivity (legacy wire id; routing is scored under Discoverability).
+- `accuracy`: Final-answer correctness against the reference answer.
+- `goal_accuracy`: Whether the user's goal was achieved.
+- `behavior_check`: Whether the expected workflow behavior was followed.
+- `token_efficiency`: Actual uncached prompt plus completion usage (50% of Efficiency).
+ + + +## Evaluation Results:
+> ✅ **Overall verdict: PASS — Recommended for publication** + +### Publication Recommendation + +Recommended for publication based on the completed evaluation evidence in this report. + +### Results at a Glance + +| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | +|---|---:|---:| +| Overall | 96.3% — baseline ran, but no comparable score was available; uplift unavailable | 92.4% — baseline ran, but no comparable score was available; uplift unavailable | +| Security | 100.0% → 100.0% (±0.0 points) | 100.0% → 100.0% (±0.0 points) | +| Correctness | 52.0% → 100.0% (+48.0 points) | 72.0% → 96.0% (+24.0 points) | +| Discoverability | 97.5% — baseline ran, but no comparable score was available; uplift unavailable | 93.8% — baseline ran, but no comparable score was available; uplift unavailable | +| Effectiveness | 52.0% → 86.0% (+34.0 points) | 46.0% → 76.0% (+30.0 points) | +| Efficiency | 98.1% — baseline ran, but no comparable score was available; uplift unavailable | 96.0% — baseline ran, but no comparable score was available; uplift unavailable | + +**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points. + +Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline. + +A partial dimension was calculated from only the available configured signals; review the detailed report before relying on it. + +#### Comparison details + +Shared-source comparisons use the same signals and weights on both conditions. They do not replace the full dimension scores above or change the publication verdict. + +| Agent | Dimension | Basis | Baseline → With skill (uplift) | Explanation | +| --- | --- | --- | --- | --- | +| claude-code | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| claude-code | Efficiency | Token-only comparison | 98.0% → 96.3% (-1.7 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | +| codex | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| codex | Efficiency | Token-only comparison | 88.5% → 92.1% (+3.6 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | + +### Tier Status + +| Tier | Purpose | Status | Evidence | +|---|---|---|---| +| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 11 finding(s) | +| Tier 2 | Semantic deduplication | **PASSED** | 2 validator(s); 0 finding(s) | +| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 5 task(s) | + +## Skill Version(s):
+0.1.0 (source: pyproject.toml); git SHA 2bbedbe6, committed 2026-11-18
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/skills/kumo-custom-populations/skill.oms.sig b/skills/kumo-custom-populations/skill.oms.sig new file mode 100644 index 000000000..f4cf52ed6 --- /dev/null +++ b/skills/kumo-custom-populations/skill.oms.sig @@ -0,0 +1 @@ +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAia3Vtby1jdXN0b20tcG9wdWxhdGlvbnMiLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiYzIxMDc4ZDY1OGYxOTE5NjI5MDI0Y2Q3YzQ4YzU4ZTc1YTc1NGE3N2IxOTgwNzZjMjdkYTUwODEyNWUxNjI4MCIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInNlcmlhbGl6YXRpb24iOiB7CiAgICAgICJpZ25vcmVfcGF0aHMiOiBbCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdGlnbm9yZSIsCiAgICAgICAgIi5naXRodWIiLAogICAgICAgICIuZ2l0IgogICAgICBdLAogICAgICAibWV0aG9kIjogImZpbGVzIiwKICAgICAgImFsbG93X3N5bWxpbmtzIjogZmFsc2UsCiAgICAgICJoYXNoX3R5cGUiOiAic2hhMjU2IgogICAgfSwKICAgICJyZXNvdXJjZXMiOiBbCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICJkZjMwNzNiY2M3NmE5NzE3NDk1N2U3N2I0NjI5ZGU0ZGUwN2E0NWE4ZWY3MGViNzBiODUzZjVhNmU1ZWM1NmFkIiwKICAgICAgICAibmFtZSI6ICJCRU5DSE1BUksubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI2MjI4MmZkMzY0MmI0MzMzNjY1NjJiYTU5YTUzYmE1YjkzMDI1N2U3M2UzYWQ1OGI5NmQ2OGMyNzAwZjM5NzU4IiwKICAgICAgICAibmFtZSI6ICJTS0lMTC5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjcwNWY1NmMwZDcyZjM1YmVhOWQyMTMwZTU2MmU1MGY5OWExM2I5MTE2ZDlmOWQ0ZDNkN2U1Zjk3ZGFjOWQ3ZDgiLAogICAgICAgICJuYW1lIjogImV2YWxzL2V2YWxzLmpzb24iCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI3ZjIyZmI0MzdlMGNmMDBkMWMwY2VlZTYyYTBiYjdlZjBhZjM1NDhkMzQyMjllZmZkMWMwNTVjYjllMmFkMjdhIiwKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL3BvcHVsYXRpb25zLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiNDQ2ZGUyMzcyNWJlZDA3ZGI2MWZiMjRjNGRkMjhkMzQ2ZGFlYTY4NjM3M2EzNjY4N2IzZjBkNDkzNWYzNDYzZCIsCiAgICAgICAgIm5hbWUiOiAic2tpbGwtY2FyZC5tZCIKICAgICAgfQogICAgXQogIH0KfQ==","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMC5xb67Gq4//X4nF7jlMM1+K9BEyjHxMVUAM/0p29tbleVTwudOzYCcb1RM6hUZKBgIxAI+0DzXiKVu/uY/giDHarEANpG3wDL0NsoDoWoIVcTkP/Bz5bioBRVKpCRqp2Zsglw==","keyid":""}]}} \ No newline at end of file diff --git a/skills/kumo-data-science/BENCHMARK.md b/skills/kumo-data-science/BENCHMARK.md new file mode 100644 index 000000000..c4ddbcb06 --- /dev/null +++ b/skills/kumo-data-science/BENCHMARK.md @@ -0,0 +1,171 @@ +# Skill Benchmark: kumo-data-science + +> ✅ **Overall verdict: PASS — Recommended for publication** + +## Publication Recommendation + +Recommended for publication based on the completed evaluation evidence in this report. + +## Evaluation Metadata + +- Skill: `kumo-data-science` +- Validation status: `passed` +- Evaluation date: 2026-11-19 +- Evaluator version: `1.6.10` +- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-5-5`), Codex (`openai/openai/gpt-6.1-sol`) +- Tasks: 8 evaluation tasks (7 positive, 1 negative) +- Dataset digest: `sha256:f72dc6983fe9c6ff4f46b5d40bebd4ceb50356c04e83ebc88dc2cd383486d627` (skill-evaluator-dataset-snapshot/1) +- Attempts per task: 1 +- Environment: `k8s-sandbox` +- Tier 2 evidence: required for publication +- Tier 3 evidence: required for publication + +Each task attempt ran in its own isolated sandbox pod. + +## Tier 1 LLM Provenance + +SkillSpector provider, model, analyzer revision, and determinism evidence retained for this scan. + +| Skill | Provider | Adapter | Analyzer | Model | Revision | Determinism | +| --- | --- | --- | --- | --- | --- | --- | +| kumo-data-science | nv_inference | openai | mcp_tool_poisoning | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.5` | nondeterministic / provider_defaults | +| kumo-data-science | nv_inference | openai | semantic_developer_intent | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.5` | nondeterministic / provider_defaults | +| kumo-data-science | nv_inference | openai | semantic_quality_policy | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.5` | nondeterministic / provider_defaults | +| kumo-data-science | nv_inference | openai | semantic_security_discovery | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.5` | nondeterministic / provider_defaults | +| kumo-data-science | nv_inference | openai | meta_analyzer | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.5` | nondeterministic / provider_defaults | + +- Controls for `kumo-data-science`: temperature=not set (source=provider_default, forwarded=no); seed=not set (source=unset, forwarded=no); reasoning_effort=not set (source=provider_default, forwarded=no) + +## What This Report Answers + +The three-tier evaluation checks whether the skill: + +- is safe to use; +- produces correct answers; +- is discovered and activated when needed; +- helps the agent complete the user's goal and expected workflow; and +- avoids wasted skill and tool usage. + +## Results at a Glance + +| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | +|---|---:|---:| +| Overall | 98.7% — baseline ran, but no comparable score was available; uplift unavailable | 94.0% — baseline ran, but no comparable score was available; uplift unavailable | +| Security | 100.0% → 100.0% (±0.0 points) | 75.0% → 100.0% (+25.0 points) | +| Correctness | 52.5% → 97.5% (+45.0 points) | 50.0% → 95.0% (+45.0 points) | +| Discoverability | 100.0% — baseline ran, but no comparable score was available; uplift unavailable | 92.9% — baseline ran, but no comparable score was available; uplift unavailable | +| Effectiveness | 39.3% → 100.0% (+60.7 points) | 37.8% → 86.6% (+48.8 points) | +| Efficiency | 96.1% — baseline ran, but no comparable score was available; uplift unavailable | 95.3% — baseline ran, but no comparable score was available; uplift unavailable | + +**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points. + +Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline. + +A partial dimension was calculated from only the available configured signals; review the detailed report before relying on it. + +### Comparison details + +Shared-source comparisons use the same signals and weights on both conditions. They do not replace the full dimension scores above or change the publication verdict. + +| Agent | Dimension | Basis | Baseline → With skill (uplift) | Explanation | +| --- | --- | --- | --- | --- | +| claude-code | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| claude-code | Efficiency | Token-only comparison | 97.0% → 92.3% (-4.7 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | +| codex | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| codex | Efficiency | Token-only comparison | 82.0% → 90.7% (+8.7 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | + +## Token Usage + +Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions. + +| Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage | +|---|---|---:|---:|---:|---:|---| +| claude-code | All cases | 1,546,761 | 961,349 | +585,412 | +60.89% | skill 8/8; base 8/8 | +| claude-code | kds-01 | 225,288 | 143,106 | +82,182 | +57.43% | skill 1/1; base 1/1 | +| claude-code | kds-02 | 184,106 | 161,500 | +22,606 | +14.00% | skill 1/1; base 1/1 | +| claude-code | kds-03 | 226,123 | 190,667 | +35,456 | +18.60% | skill 1/1; base 1/1 | +| claude-code | kds-04 | 267,817 | 145,317 | +122,500 | +84.30% | skill 1/1; base 1/1 | +| claude-code | kds-05 | 217,017 | 70,650 | +146,367 | +207.17% | skill 1/1; base 1/1 | +| claude-code | kds-06 | 216,857 | 107,418 | +109,439 | +101.88% | skill 1/1; base 1/1 | +| claude-code | kds-07 | 173,669 | 107,032 | +66,637 | +62.26% | skill 1/1; base 1/1 | +| claude-code | kds-08 | 35,884 | 35,659 | +225 | +0.63% | skill 1/1; base 1/1 | +| codex | All cases | 510,192 | 3,729,493 | -3,219,301 | -86.32% | skill 8/8; base 8/8 | +| codex | kds-01 | 70,556 | 45,022 | +25,534 | +56.71% | skill 1/1; base 1/1 | +| codex | kds-02 | 96,732 | 2,075,477 | -1,978,745 | -95.34% | skill 1/1; base 1/1 | +| codex | kds-03 | 96,681 | 1,411,737 | -1,315,056 | -93.15% | skill 1/1; base 1/1 | +| codex | kds-04 | 49,701 | 41,202 | +8,499 | +20.63% | skill 1/1; base 1/1 | +| codex | kds-05 | 67,801 | 30,289 | +37,512 | +123.85% | skill 1/1; base 1/1 | +| codex | kds-06 | 65,684 | 54,786 | +10,898 | +19.89% | skill 1/1; base 1/1 | +| codex | kds-07 | 49,385 | 57,575 | -8,190 | -14.22% | skill 1/1; base 1/1 | +| codex | kds-08 | 13,652 | 13,405 | +247 | +1.84% | skill 1/1; base 1/1 | +| ALL AGENTS | Dataset aggregate | 2,056,953 | 4,690,842 | -2,633,889 | -56.15% | skill 16/16; base 16/16 | + +Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. Change is `100 * (with - without) / without`; negative means fewer tokens. N/A comparisons include their reason in the coverage column; coverage is never estimated. + +## Tier Status + +| Tier | Purpose | Status | Evidence | +|---|---|---|---| +| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 26 finding(s) | +| Tier 2 | Semantic deduplication | **PASSED WITH OBSERVATIONS** | 2 validator(s); 4 finding(s) | +| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 8 task(s) | + +## Findings and Observations + +
+Show detailed findings and successful checks + +- **HIGH** DUPLICATE/duplicate: Duplicate content found across SKILL.md and accept/preflight.md: + "### Node 1c — Preflight & graph gate" in SKILL.md (lines 178-194) + vs "## 5. Column availability" in accept/preflight.md (lines 157-171) (`SKILL.md:178`) +- **HIGH** DUPLICATE/duplicate: Duplicate content found across accept/intake_contract.md and validate/examples/good_request.md: + "## Canonical good example (`train`)" in accept/intake_contract.md (lines 137-148) + vs "## Input" in validate/examples/good_request.md (lines 3-12) + vs "## Node 1a — Required fields: PASS" in validate/examples/good_request.md (lines 13-19) (`accept/intake_contract.md:137`) +- **HIGH** DUPLICATE/duplicate: Duplicate content found within accept/batch_inference.md: + "## Node 1a — fields" in accept/batch_inference.md (lines 7-29) + vs "### `batch_inference` — required fields (ask if missing)" in accept/batch_inference.md (lines 30-42) + vs "### `batch_inference` — resolved, defaulted, or inferred (never ask)" in accept/batch_inference.md (lines 43-54) (`accept/batch_inference.md:7`) +- **HIGH** DUPLICATE/duplicate: Duplicate content found across validate/atomic_question_check.md and validate/examples/bad_request.md: + "### Fail 1 — Multiple targets bundled" in validate/atomic_question_check.md (lines 42-50) + vs "## 6. Bundled goals → REJECT" in validate/examples/bad_request.md (lines 64-68) (`validate/atomic_question_check.md:42`) +- **MEDIUM** QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.author' (`skills/kumo-data-science/SKILL.md`) +- 25 additional finding(s) are available in the full evaluation artifacts. + +
+ +## Scoring Methodology + +
+Show dimension definitions, source signals, and thresholds + +| Dimension | Question | Scored signals | +|---|---|---| +| Security | Is it safe to use? | `security` (100%) | +| Correctness | Is the answer correct? | `accuracy` (100%) | +| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) | +| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) | +| Efficiency | Did it avoid wasted tool calls and token usage? | `skill_efficiency` (50%) + `token_efficiency` (50%) | + +- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%. +- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL. +- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate. +- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold. +- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`). +- Efficiency is 50% tool-call productivity (the backward-compatible `skill_efficiency` wire id) and 50% `token_efficiency`. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial. + +Signals present in this run: + +- `security` (Security): unsafe operations, secret leakage, and unauthorized access. +- `skill_execution` (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed. +- `skill_efficiency` (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability). +- `accuracy` (Accuracy): final-answer correctness against the reference answer. +- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved. +- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed. +- `token_efficiency` (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency). + +
+ +## Freshness + +Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes. diff --git a/skills/kumo-data-science/SKILL.md b/skills/kumo-data-science/SKILL.md new file mode 100644 index 000000000..6aae6b86b --- /dev/null +++ b/skills/kumo-data-science/SKILL.md @@ -0,0 +1,367 @@ +--- +name: kumo-data-science +description: "Use for KumoAI graphs and predictive models on relational data in a warehouse, an object store or local files: inspect, visualize or export a graph's schema and relationships; train a model, score data with a trained one, explain its predictions, diagnose its training, publish, reload or retain it, train or score on custom labels or rows, or tune it. Covers churn, propensity, fraud, recommendation, forecasting and link prediction. For a new model it checks that the request names the data, outcome and time window, confirms the inferred tables and task, then generates one notebook or script; it runs it only when asked, in the notebook or as a host job. Also use it, before any general jobs or workspace skill, for a job, schedule, run or cluster that runs a Kumo notebook or scores a model published from one: creating, scheduling, pausing, submitting or debugging it." +--- + +# Data Science Agent — Router Skill + +## Build marker + +**`SKILL-BUILD: 2026-11-19-bef5ab4e-dirty`** + +Quote this string verbatim whenever asked which build of the skill is loaded. If you cannot read +it from this file, you have not loaded the skill — say so rather than guessing. + +Targets the local `kumo.engine` API; the baseline bundle is `20261007+2e1eb12`. +Graph visualization requires a source build including [NVIDIA/kumo-engine#134](https://github.com/NVIDIA/kumo-engine/pull/134); it is absent from that bundle and the `1.0.0` release. +One SDK, no variants. + +## Paths in these files + +A path that starts with a skill's name — `kumo-data-science/`, `kumoai-reference/`, or a skill in +*Routing* below — is in that skill's directory, which sits inside this one or beside it. Any other +path, such as `accept/` or `hosts/`, is relative to the skill whose file names it. + +## Read the host adapter first + +Before you answer any request this skill covers — a new model, scoring, a request routed to a +capability skill below, a follow-up in a session that already has a notebook, or a question +about a job — read `hosts//adapter.md`, the one directory under `hosts/`. Where it has a +section on what you may do before the user confirms (the Databricks Genie adapter's §0), that +holds for every reply. Where the adapter names a job file, a request about a job reads it before +you act on any job, run or cluster. + +## When to use + +Requests that mention any of: kumo, predict, churn, recommend, forecast, fraud, train a model, run +inference, explain a prediction, feature importance, feature attribution, column cohorts, model +metrics, trials, learning curves, publish or load a model, registry publishing, artifact retention, +caller-supplied labels, weights or splits, encoders, run mode, experiments, tune a model, semantic +view, metric view, ML Job, compute pool, Databricks job, graph diagram, visualize a graph, +inspect graph schema, export graph relationships. + +## Routing + +Some requests are not new-training intake. Route these to their own skill, which has its own +build and artifact checks: + +| Request | Skill | +|---|---| +| Show or export a graph; inspect its schema during model iteration | `kumo-graph-visualization/SKILL.md` | +| Attribution or column cohorts for an existing run | `kumo-model-explanations/SKILL.md` | +| Metrics, trial comparisons, learning curves, recorded progress | `kumo-training-diagnostics/SKILL.md` | +| Publish, reload, retain or retrieve a model or its artifacts | `kumo-model-persistence/SKILL.md` | +| Caller-supplied labels, weights, splits, or exact scoring rows | `kumo-custom-populations/SKILL.md` | +| Encoders, run mode, bounded experiments, model-plan options | `kumo-model-tuning/SKILL.md` | + +None of these require new-training intake, and none authorizes another fit. Preserve intent the +user has already given — task, target, run, model version, source, destination — and ask only +for what the selected workflow is missing. If the host has not provided the referenced skill, +report that limitation rather than inventing its workflow. **New training requests still follow +the gates below**, unchanged. + +A request to run the artifact as a **job** — a Snowflake ML Job on a compute pool, a Databricks +job on a cluster, unattended, or on a schedule — or a question about a job already submitted (status, logs, cost) also reads the +host's job file, when the host adapter names one. It adds settings to the gates below and changes +where Step 3 runs; it never replaces the gates. If the adapter names no job file, say that this +host does not run jobs and offer the notebook. A request that does not mention a job stays in the +notebook. + +## Purpose +Take a plain-language ML request, get it to a **confirmed** task and graph through a multi-turn +intake, and generate **one** runnable KumoAI artifact — a notebook for training, a Python script +for batch inference. The agent asks only for what is missing, infers structure rather than asking +for it, confirms before writing code, and does not run jobs unless told to. + +## What this skill does +1. **Accept** — a subDAG of checks; each node emits ACCEPT / ASK BACK / REJECT +2. **Preflight & graph gate** — infer tables, keys, time columns, links; user confirms +3. **Task gate** — restate entity / outcome / window / task type; user confirms +4. **Generate** — one notebook that **trains and then predicts** in the same process, with + exactly one blocking `fit` call; the predict call is part of the same artifact +5. **Run** — *only on explicit request* + +**The P1 deliverable is train → predict in one notebook.** Batch inference from a *saved* model +(`batch_inference`) is also supported: `Trainer.load` and `Trainer.load_from_connector` restore a +trainer, and `predict` runs against an explicitly rebuilt graph snapshot (see Step 2). + +## What this skill does NOT do +- Choose the predictive problem, target, horizon, graph, or tuning for the user — refuse and name + the minimum inputs needed +- Join, clean, deduplicate, aggregate, create labels, or transform data **as model input**. + Diagnostic scalar aggregates during preflight — reported as proposal evidence, never written and + never fed to training — are not this; they exist so a key can be checked before it silently + discards rows +- Search for, rank, or curate tables by business relevance +- Retry or resume jobs, or keep state across sessions. The one execution the agent performs is + Step 3, and only when the user asks. +- Schedule anything on its own initiative (a recurring run is proposed and created only on explicit + confirmation, and only on a host whose job file covers it) +- RFM / zero-shot workflows +- Audit the user's schema for leakage at intake beyond the mandatory availability guards in + Node 1c. The codegen guard in `kumoai-reference/SKILL.md` (the target never appears as a + feature) still applies +- Turn a vague question into a complete request beyond the ask-back, or repair runtime job failures + +## Task scope +**Temporal binary classification**, **link prediction**, regression, ranking, static link +prediction, and embeddings as an output. Treat them all with the same intake; `time_window` is `not_critical` for +static shapes. + +--- + +## Step 1 — Accept subDAG + +Accept is a subDAG of check nodes run in a multi-turn chat. Each node produces one of: + +- **ACCEPT** — proceed to the next node. +- **ASK BACK** — ask one targeted question for exactly what is missing; re-run on the next turn. +- **REJECT** — cannot be fixed by clarification. Stop with a clear explanation. + +Ask **only** for missing, ambiguous, or contradictory inputs; never request secrets. +One ask-back turn lists every missing required field at once. + +### Node 1a — Operation and required fields +Reference: `accept/intake_contract.md` + +First decide the operation: `train` or `batch_inference`. + +**`train` — required (ask if missing):** +- `source_scope` — where the tables live, in a form the host accepts (`accept/intake_contract.md`) +- `prediction_entity` — what gets one prediction +- `prediction_target` — the outcome or interaction, with the positive-label condition +- `time_window` — `specified` / `not_critical` (static task) / `missing` → ask only when `missing`. + Static vs. temporal is decided by *what is predicted*, never by a date column in the data: a + value the entity row carries ("final price of each listing") or embeddings / observed links are + static; anything that happens or accumulates after a point in time is temporal + (`accept/intake_contract.md`, *static or temporal*). + +**`train` — never ask for:** column names, types, keys, joins, table mode (inferred at 1c); +output table (optional; preserve if given); metrics; embeddings (add only if asked). + +**`batch_inference`:** its required and resolved fields, and what changes at Nodes 1c, 1d and +Step 2, are in `accept/batch_inference.md`. Read it when the operation is `batch_inference`; a +training request never needs it. + +**Never fill a required field from the schema.** A preflight result (or a schema listing handed +to you) can tell you which tables exist, not what the user wants predicted. If the user has not +named the outcome, the positive-label condition, the item, the interaction, or the window, it is +missing — ASK BACK, even when the tables make a guess look obvious. + +**Quote test:** a required field is present only if you can quote the user's words for it; a +table or column name is not a quote. **Conversely, never ask the user which column or value +implements a field they already named** — infer the mapping and show it `[inferred]` at the gate +(`accept/intake_contract.md`). + +| Condition | Transition | +|---|---| +| Any required field missing | ASK BACK — list exactly the missing fields | +| `source_scope` given but does not resolve (location or semantic view not found) | ASK BACK — ask for a valid source; do not guess a similar name | +| All required present | proceed to Node 1b | + +### Node 1b — Atomic question check +Reference: `validate/atomic_question_check.md` + +| Condition | Transition | +|---|---| +| Goal is vague but potentially clarifiable | ASK BACK — ask the user to define the target concretely | +| Multiple goals bundled that cannot be split | REJECT — explain and ask for separate requests | +| "Figure out what to predict / pick the best target / tune and deploy" | REJECT — the agent does not choose the problem; list the minimum inputs for one supported request | +| Single, well-defined target | proceed to Node 1c | + +### Node 1c — Preflight & graph gate +Reference: `accept/preflight.md` + +First establish that you can read `source_scope` (`accept/preflight.md` §0). Then run read-only +preflight inside it. + +Present the graph proposal — seeds, tables, +primary keys, time columns, links, `table_mode: all-connected [default]` — each line tagged +`[supplied]` / `[declared]` / `[inferred]` / `[inferred, unverified]` / `[default]`. A semantic +view is resolved by `accept/semantic_view.md`, read only for a semantic view. If the user asked for embeddings, say they are included. **Stop and wait for confirmation.** +User-supplied values take precedence over inferred ones. + +**Mandatory column-availability pass.** Before presenting the graph, decide the selected columns +for every included table and add an `Excluded columns` section to the proposal. + +- When a table's `time_column` marks the beginning of a row's validity or an interval, exclude any + candidate feature whose value is determined only after that time: the interval's eventual + endpoint, its realized duration, or a summary of the completed interval. At the row timestamp + those values are future information. Decide by semantic role, never by matching column names. +- Separate the target column from columns derived from it; both rules apply at once. + - The target column itself **always stays** in the positional column list. The query references + it by name, so dropping it fails with a missing column. + - **Exclude** every other column whose value is a function of the target — a margin, ratio, + rate, share, rank or flag computed from it. Such a column reproduces the answer under a + different name, and the dependency is often visible only in the naming, so treat a plausible + derivation as a derivation and show it as an exclusion the user can overturn. +- Apply the active host adapter's runtime column constraints (`hosts//adapter.md`). A host + constraint may exclude a column; it must never invent a replacement semantic type. Exclude only + on evidence the adapter accepts as establishing the constrained property — a physical catalog + type generally does not establish a semantic one — or on a prior runtime-guard result. Where the + first pass cannot establish it, say so rather than guessing, and if the host's runtime guard + later resolves it, return to this node with a revised proposal rather than proceeding. + +Tag each exclusion `[inferred]`, give the reason, and let the user correct it at this same gate. +Carry the confirmed exclusion set into code generation: a table with any exclusion must be built +with an explicit positional column list, because omitting it silently restores every source +column. + +**Assert what preflight could not find.** The proposal must state these explicitly rather than +leaving them unmentioned, because each one changes whether the task is expressible: + +- a table with no **confirmed** primary key — say so. Until the user confirms one it is treated + as keyless: it may be an edge source, but not an edge destination and not the `FOR EACH` + entity. An `[inferred, unverified]` candidate does not lift that restriction; confirming it + does. Never promote a merely unique-looking column to primary key, and do not + conclude from a failed heuristic that the table must be an event or fact table. +- a time-ranged target whose referenced table has no usable time column — say so, and say what + would make the task expressible. A timestamp on another table does not repair the target. + +| Condition | Transition | +|---|---| +| You cannot read `source_scope` | ASK BACK — say what you checked; offer access or the structure in the user's own words (`accept/preflight.md` §0) | +| Seeds cannot be connected without transforming data | REJECT — explain; generate nothing | +| A semantic view fails a check in `accept/semantic_view.md` | ASK BACK — as that file says; do not pick a schema, drop tables, or substitute a link | +| User corrects a line | re-present the proposal with the correction `[supplied]` | +| Confirmed proposal still lacks metadata the task requires (entity key, target time column) | remain at Node 1c — explain the correction needed; generate nothing | +| User confirms | proceed to Node 1d | + +### Node 1d — Task interpretation gate +Restate, in this shape, then **stop and wait for explicit confirmation** — no code is +generated before the task is confirmed: + +``` +- Entity: one prediction per
. +- Outcome: / +- Window: <0 to N days after the prediction anchor> | static +- Task type: +- Output: predictions written to in the same notebook | notebook only, no predictions written +- Scoring population (static tasks): rows whose is NULL | all rows, staged at +- Threshold (binary classification, written or local): +- Embeddings: +``` + +**Static tasks: confirm the scoring population.** A static query scores only the rows whose +target is NULL (`kumoai-reference/references/kumo_finetuning_sdk.md` §7.1). If the user wants rows that +already have a target scored (for example "every order"), the notebook stages an explicit +population table. That is a write, and it goes in the source's system, so name it here: next to the confirmed +destination by default when the destination is in the source's system but not in the source schema +or under the source path — the table +`_POPULATION`, or for a path the sibling directory +`_population/` (`s3://b/scores/` → +`s3://b/scores_population/`), never inside the destination, which must stay empty; otherwise, or +when there is no destination, ask for a schema or path in the source's system. Never stage it in +the source schema or under the source path, where it would read as a source table. + +**Name what a run replaces.** The notebook writes predictions with `OVERWRITE` and stages a +population with `CREATE OR REPLACE` (a file: overwritten), on every run. Check from the catalog +whether the destination and the population already exist, and say so on their lines — +` (exists: replaced on every run)` — so confirming them approves the +replacement. When the user does not want an existing table replaced, ask for another name, or +for a destination, APPEND (`kumoai-reference/references/outputs.md`). If the check cannot run, +say that instead. The approval covers the confirmed population name only: the notebook carries it +as `POPULATION_CONFIRMED` and stops before staging if the name it would replace differs +(`kumoai-reference/references/kumo_finetuning_sdk.md` §7.1). + +| Condition | Transition | +|---|---| +| User confirms | ACCEPT — proceed to Step 2 | +| User changes intent | back to Node 1a with the new facts | + +**Do not emit any Kumo SDK code before this ACCEPT.** In the eval harness and in production alike, +a reply that reaches Node 1d ends with the restatement and the confirmation question. + +### Default metrics by task type +The generated notebook does not set metrics; the SDK's default model plan supplies them. Use +this table to tell the user what the model will be evaluated and tuned on. + +| Task type | Reported metrics | Tuned on | +|---|---|---| +| Binary classification | `auroc`, `ap`, `auprc` | `auroc` | +| Multiclass classification | `acc` | `acc` | +| Multilabel classification | `ap`, `auroc`, `auprc`, each `_macro` / `_micro` / `_per_label` | `ap_macro` | +| Regression, forecasting | `mae`, `mse`, `rmse` | `mae` | +| Ranking, link prediction | `map`, `precision`, `recall`, `f1`, each `@1` / `@10` / `@100` | `map@10` | +| Link prediction with a training-table weight column | `weighted_ndcg`, `weighted_recall`, `precision`, each `@1` / `@10` / `@100` | `weighted_ndcg@10` | + +Source: shipped package (`kumo_core.model_plan`) + +--- + +## Step 2 — Generate via KumoAI + +Read the host's `runtime.md` (`hosts//runtime.md`) before generating. + +After ACCEPT, load `kumoai-reference/SKILL.md` and hand off the confirmed payload. Produce +**one** artifact. + +### `train` → one notebook that fits, then predicts + +1. Connector from environment credentials; confirmed tables, metadata, links; graph and PQL + validation; SDK-default training table and model plan, no tuning. + After graph construction and validation, include the interactive graph cell from `kumo-graph-visualization/SKILL.md` for the user, before snapshot creation and training. + Visualization adds no approval gate; continue unless the user requests a pause for review. +2. **Exactly one** training call — `trainer.fit(...)`. It blocks until training finishes and + returns a `TrainingResult`; print its `metrics`. +3. **Predict in the same notebook** — same `Trainer`, same process, immediately after + `fit`: + - `output_location` confirmed at Node 1d → the notebook includes the `predict` call with a + `PredictionOutputConfig` pointing at that destination, written **verbatim**. + - no destination → predictions stay local; the notebook ends with a short markdown cell + describing the one line to add a destination later. That example is markdown text, never a + code cell: a notebook with no confirmed destination contains no code that writes. Inside + the markdown, show the code as text (a `text` fence), never as a `python` code block — + one that would run is a code cell that writes. + The predict call is part of the artifact, not a separate job the agent manages. The agent + still executes nothing (Step 3 is opt-in). + +The user may ask for a script instead of a notebook. +Include the SVG export from `kumo-graph-visualization/SKILL.md` before training, saving it alongside run outputs and reporting its path. + +### `batch_inference` + +See `accept/batch_inference.md`, *Step 2*. + +The artifact must preserve every confirmed input verbatim — entity, outcome, window, table mode, +metadata, links, destination. No invented link, horizon, or table. + +--- + +## Step 3 — Run **(only on explicit user request)** + +Default is generate-only. Run only when the user says to run it, +in the same or a later turn. Then, per `kumoai-reference/references/kumo_finetuning_sdk.md`: +- Run the notebook. `fit` blocks until training finishes; there is no Kumo job to poll. When the + user asked for a job, running means submitting it and waiting for its status as the host's job + file describes. +- Report the outcome: the training metrics, or the exception and the cell that raised it +- **Read what Kumo said about the data first.** A warning or error about the data — duplicate + primary keys, a foreign key that matched few rows, invalid values read as null, a float or + mismatched key, an empty training table — leads the report, with the table, column and count, + and the decision goes to the user: fix the source, change the key or link, or accept the loss. + A run that finished is not a success until they have decided. Never change the data, a column's + type or the query to make such a message go away + (`kumoai-reference/references/troubleshooting_signals.md` §1a). A failure that is not about the + data — an import, a misspelled argument — is the code's: fix it and run again +- Let the notebook's own predict step run; predictions land at the confirmed destination — **the + name is written verbatim**, no suffix, no case change; report it in the destination's own + form (`DB.SCHEMA.TABLE`, or a path) +- Include embeddings if they were confirmed at Node 1d + +--- + +## Host adapter +This package carries exactly one host: the single directory under `hosts/`. Everything specific to +where you run and where the data lives is there — never assume another host's rules. + +- `hosts//adapter.md` — read at intake: how to check access to the source, the source and + destination forms the host accepts, how to run preflight, and runtime column constraints +- `hosts//runtime.md` — read at Step 2: how the generated code connects, authenticates, + writes and verifies + +--- + +## Extension points +- **Host expansion:** add a directory under `hosts/` with its own `adapter.md`. The portable core + (this file + `accept/` + `validate/` + `kumoai-reference/`) does not change. diff --git a/skills/kumo-data-science/accept/batch_inference.md b/skills/kumo-data-science/accept/batch_inference.md new file mode 100644 index 000000000..fb5374c88 --- /dev/null +++ b/skills/kumo-data-science/accept/batch_inference.md @@ -0,0 +1,191 @@ +# Batch inference — scoring with a saved model + +Read this file only when the operation is `batch_inference`: the user names a trained model and +wants it to score data. A training request never needs it. It adds to `SKILL.md` and +`intake_contract.md`; everything they say still applies unless this file overrides it. + +## Node 1a — fields + +**`batch_inference` — required (ask if missing):** +- `model_address` — the model the user names: a name or ID, a registry location with a version, or + a retained local bundle directory +- `output_location` — a destination the active host can write (`hosts//adapter.md`, + *Destinations*); a connector plus a table name is complete + +**`batch_inference` — resolved, never asked** (`accept/intake_contract.md`): +- `source_scope`: a source the user named, else the connector named for this request (only on a + host whose *Source scope* accepts a connector name), else the source in the team's training + record for the model (`kumo-model-persistence/references/reuse.md`) — the registry does not + keep one. The bundle restores trainer state, **not** the graph, so the graph is rebuilt from + source at 1c. The record is read at 1c, not at intake: a source that is not in the conversation + is not missing; ask for it only if 1c finds no training record. +- `scoring_population`: defaults to every entity of the trained entity table, anchored at the + latest timestamp in the rebuilt snapshot, tagged `[default]`. A subset or date the user names + replaces it. +- The target, positive-label condition, window and task type are fixed by the trained model and + must not be reopened. + +A request with a model and a destination goes to the gates. It does not get an ask-back. + +### `batch_inference` — required fields (ask if missing) + +Two fields are the user's to give. Everything else is reused from the model, resolved from what +the user already said, or defaulted — and shown at the gates for confirmation. + +| Field | What it must include | +|---|---| +| **`model_address`** | The model the user names, in their words: a model name or ID ("trainingjob-benchmark-001", "the benchmark model"), a registry location with a version, or a retained local bundle directory | +| **`output_location`** | Where scores are written, in a form the active host can write (`hosts//adapter.md`, *Destinations*). A connector plus a table name ("connector `benchmark_sf`, table `MODEL_SCORES`") is complete, because the connector supplies the database and schema. A fully qualified table (`DB.SCHEMA.TABLE`), or a path on a host that writes files, is also complete. | + +Both pass the quote test through earlier turns too. "Run the benchmark model again and put the +preds in the scores table", after a turn that named the model and the table, supplies both. + +### `batch_inference` — resolved, defaulted, or inferred (never ask) + +| Field | How it is obtained | Shown as | +|---|---|---| +| **task** — entity, outcome, window, task type | Reused from the model unchanged. Never reopened. | restated at 1d | +| **`source_scope`** | In order: a source the user named → the connector the user named for this request, on a host whose *Source scope* accepts a connector name (one connector reads the source and writes the scores unless the user separates them) → the source scope in the team's training record for the model (`kumo-model-persistence/references/reuse.md`); the registry keeps only `spec_version`, the query and `snapshot_id`, not the source. When neither of the first two applies, restate `Source: from the training record of [inferred]` and do not ask: the record is read at 1c. Ask only if 1c then finds no training record, or one that names no source. On a host that does not accept a connector name, a connector named alone is not a source: ask for the system and location as that host's adapter says. | `[supplied]` / `[inferred]` at 1c | +| **graph** | Rebuilt from the model's recorded graph definition inside `source_scope`, then confirmed | `[inferred]` at 1c | +| **`scoring_population`** | Default: every entity of the trained entity table, anchored at the latest timestamp in the rebuilt snapshot ("current predictions"). A static task has no anchor. If the user names a subset or an anchor date, use theirs. | `[default]` / `[supplied]` at 1d | +| **model version** | The version the user named. If the address resolves to more than one version and none was named, ask which one. Never pick the latest yourself. An address with a single version uses it. Either way, the generated code names the version explicitly. | `[supplied]` / `[inferred]` (the only version) at 1d | +| **embeddings** | Only when the user explicitly asks ("export the entity and target-item embeddings") | restated at 1d | +| **scoring options** | Set by the trained task type, below | restated at 1d | + +### `batch_inference` — scoring options by task type + +These are `predict` arguments, chosen at scoring time; none needs retraining, and the trained +query is never changed to get them (`kumoai-reference/references/kumo_finetuning_sdk.md` §9; +Source: shipped package (`kumo_core.prediction_options`)). The count is +`num_classes_to_return`; it is **independent of the query's `RANK TOP K`** (a `RANK TOP 10` +model can return 50), and when unstated it falls back to that K. + +| Task type | Count of results per entity | Threshold | +|---|---|---| +| Binary classification | none: the output already carries both class probabilities and the thresholded label, so a request for "both classes" needs no option | required, in [0, 1]: the user's value, else the one recorded with the model, else 0.5 `[default]` (`predict` has no default) | +| Temporal or static link prediction | **required** when predictions are written, 1 to 500; unstated, K from `RANK TOP K` `[default]`. Above 500, say so and offer up to 500 per entity, or the entity and item embeddings for a nearest-neighbour search over longer lists; ask which | none | +| Multilabel ranking | optional; unstated, K from `RANK TOP K`. No 500 cap | none | +| Multiclass, multilabel classification | optional; unstated, K from `RANK TOP K` when the query has one, else every class | multilabel only, in [0, 1]: the user's value, else 0.5 `[default]` | +| Regression, forecasting | none: one predicted value (or forecast series) per entity. A "top N" request does not apply; say so and ask whether that is what the user wants | none | + +A threshold outside [0, 1], or a count where the task takes none, is invalid: say which and ask. +A count or threshold the user asks for that fits these rules is carried as given, never turned +into a question and never a reason to retrain. + +**The saved bundle does not carry the graph.** It restores trainer state; `predict` takes an +explicit `snapshot`, so the source data must be available and the graph rebuilt and confirmed at +Node 1c exactly as for `train`. "Reuse the model" needs reading precisely here: the *task* is +reused unchanged and must not be reopened. The *graph* is rebuilt, but from the recorded definition, not re-asked. + +**When the request has a model and a destination, it goes to the gates. It does not get an +ask-back.** Do not ask for the source schema, the scoring population, the anchor date, or a fully +qualified output path when the rules above resolve them. Put each resolved value in the proposal, +tagged, so the user can correct it at confirmation. + +### `batch_inference` — invalid values (ask back, do not substitute) + +| Condition | Response | +|---|---| +| `model_address` does not resolve in the registry or as a bundle directory, or the user calls it rejected/invalid | Name it as unusable and ask for a corrected model identifier. Do not guess a similarly named model. | +| `output_location` is a kind the host cannot write (the adapter's *Destinations* lists what it can) | Name it as unusable here, say what the host writes instead, and ask for such a destination. Do not silently swap in another one. | + +When one field is invalid and the other is valid, ask only about the invalid one. When one is +missing and the other is present, ask only for the missing one. Do not also list the resolved +fields as questions. + +| Condition | Transition | +|---|---| +| `batch_inference`: `model_address` does not resolve, or the user calls it rejected | ASK BACK — name it as unusable, ask for a corrected identifier; do not guess | +| `batch_inference`: `output_location` is a kind the host cannot write | ASK BACK — name it as unusable, ask for a destination the host writes; do not substitute | + +## Node 1c — rebuilt graph + +For `batch_inference`, do not re-derive this set. The selected columns are the ones the model +was trained on: take them from the graph-building code and column metadata recorded with the +model (`kumo-model-persistence/references/reuse.md`), and show each table's list at this gate. +Re-inferring them can build the table with a different column list than the model saw, so +`predict` either fails on the mismatch or scores on different features. If no recorded column +selection is available, say so and remain at this node rather than substituting a fresh one. + +This node runs for `batch_inference` too. A saved bundle does not carry graph storage or source +data, and `predict` requires an explicit `snapshot`, so the graph is rebuilt and confirmed the +same way. Confirm that the rebuilt structure matches what the model was trained on; a bundle +from an older SDK, or source tables whose shape has drifted, fails at load or predict rather +than scoring silently. + +| Condition | Transition | +|---|---| +| `batch_inference` and no column selection is recorded with the model | remain at Node 1c — say the training-time selection is unavailable; generate nothing | + +## Node 1d — restatement + +For `batch_inference`, restate in this shape and stop for confirmation: + +``` +- Model: — trained task reused unchanged +- Source: [supplied | inferred] +- Population: as of latest snapshot timestamp [default] | the user's subset/date [supplied]> +- Output: predictions written to +- Scoring options: results per entity [supplied | default: RANK TOP K] (link prediction, ranking, multiclass, multilabel); none (regression, forecasting)> +- Embeddings: +``` + +Without tools, where 1c cannot run, fold the source and graph lines into this one restatement. +Do not turn them into questions. + +## Step 2 + +Follow `kumo-model-persistence/SKILL.md` for the load and predict code. Restore the trainer with +`Trainer.load` (retained bundle) or `Trainer.load_from_connector` (registry), materialize the +snapshot for the graph confirmed at 1c, build the prediction table for the confirmed scoring +population — `kumo-custom-populations/SKILL.md` when the user supplied exact rows — and call +`predict(prediction_table, snapshot=snapshot, output_config=...)`. + +Do not refit. If the bundle cannot be loaded, or the rebuilt graph is incompatible with it, +report the failing stage and stop; do not substitute a fresh training run without being asked. + +## Examples + +### Model and destination given → gate + +> "Load trained model `trainingjob-benchmark-001` and write current batch predictions through +> connector `benchmark_sf` to table `MODEL_SCORES`." + +Node 1a: `model_address` ✓ and `output_location` ✓. A connector plus a table name is a complete +destination. Nothing is asked. The rest is resolved and goes straight to the gate: + +``` +- Model: trainingjob-benchmark-001 — trained task reused unchanged +- Source: benchmark_sf [inferred — the connector named for this request] +- Population: all entities of the trained entity table, as of the latest snapshot timestamp [default] +- Output: predictions written to MODEL_SCORES via benchmark_sf +Shall I generate the batch-inference notebook with this setup? +``` + +With "…export the entity and target-item embeddings" added, the gate also carries +`Embeddings: included`. It is still one confirmation, not an ask-back. + +### Destination missing → ASK BACK + +> "Use trained model `trainingjob-abc123` to score current accounts." + +- `model_address` ✓ · `output_location` ✗ +- Not asked: the source (from the model's training record), the population ("current accounts" means all + accounts at the latest snapshot timestamp, the default), and the task. The task is never + reopened. + +> "Where should the scores be written? For example a connector plus a table name, or +> `DB.SCHEMA.TABLE`." + +### Destination unusable → ASK BACK + +On a host whose *Destinations* are Snowflake tables only: + +> "Score with `trainingjob-abc123` and write predictions to `s3://tmp/scores`." + +- `model_address` ✓ · `output_location` invalid: not a Snowflake table + +> "`s3://tmp/scores` isn't a destination I can write to here, because scores go to a Snowflake +> table. Which Snowflake table should receive them?" + +Only the invalid field is raised. Do not add questions about the source or the population. diff --git a/skills/kumo-data-science/accept/intake_contract.md b/skills/kumo-data-science/accept/intake_contract.md new file mode 100644 index 000000000..cafae4f8f --- /dev/null +++ b/skills/kumo-data-science/accept/intake_contract.md @@ -0,0 +1,177 @@ +# Input Contract — What a request must contain + +The contract is keyed by **operation**. Decide the operation first, then apply that operation's +table. The tables below are the complete contract: a field that is not in them is not required. + +Three kinds of field exist, and the distinction is the whole point of this file: + +| Kind | Who supplies it | If absent | +|---|---|---| +| **Required** | the user | ASK BACK — ask for exactly these, nothing else | +| **Inferred, then confirmed** | the agent, by read-only preflight (`accept/preflight.md`) | never asked; shown as `[inferred]` and confirmed at the graph gate — except a key that preflight could only fail to disqualify, which is shown `[inferred, unverified]` and cannot be used until confirmed | +| **Optional** | the user, if they want | never asked; defaults apply; if supplied it must be preserved verbatim | + +**Infer structure, never infer intent.** Column names, keys, time columns and links are structure +— measure them. Entity, outcome, positive-label condition, interaction, item, window and +destination are intent — the user's, and only theirs. No outcome definition is ever invented. + +A preflight result or schema listing **never fills a required field.** Seeing a `sales_events` +table does not tell you the positive interaction is "a sale"; seeing a `target_label` column does +not tell you the positive-label condition. If the user did not state it, it is missing — ask, +even when the schema makes the answer look obvious. Preflight may resolve *which table holds* +what the user named; it may not supply what they did not name. + +**The quote test — is a required field present?** A required field is present only if you can +quote the user's own words for it, from any turn, in any vocabulary: "each customer", "a +cancelled booking", "at least one fraud event", "next 30 days". A table name, a column name, or "the only +entity in the schema" is not a quote — the field is missing, and the reply is an ask-back, not a +graph proposal. + +**Named intent, inferred mapping — never ask the user to map their words onto columns.** Once the +user has named a field, turning their words into a physical table, column, or value is structure. +Preflight infers it — "a cancelled booking" → `bookings` where `status` marks a cancellation — +and it is shown as `[inferred]` in the proposal for the user to confirm or correct. Do +not ask which column or value implements something the user already named, and do not ask them to +re-define it more precisely. Inferred values are fine as long as the complete proposal is shown +and confirmed; questions are only for missing, ambiguous, or contradictory inputs. + +The two rules are one rule seen from both sides: the **user's words** decide *whether* a field is +present; the **data** decides *how* a present field is implemented. + +--- + +## Operation + +`train` or `batch_inference`. Read it from the request: "train / build a model / notebook that +predicts" is `train`; "use my trained model / score / run inference with job X" is +`batch_inference`. One session covers one operation, one task, one graph, one artifact. +Ask only if the request genuinely reads both ways. + +--- + +## `train` — required fields (ask if missing) + +| Field | What it must include | +|---|---| +| **`source_scope`** | Where the tables live, in a form the active host accepts (`hosts//adapter.md`, *Source scope*): for example a database + schema, a dataset, a catalog + schema, a bucket prefix or a directory of files, or a connector name the host knows. On a host that supports them, a fully qualified semantic view (`DB.SCHEMA.VIEW`) contributes exactly its base tables, which must share one schema (`semantic_view.md`). This is where tables are enumerated. Credentials are never requested in chat. | +| **`prediction_entity`** | What receives one prediction — "each account", "each customer". A physical table/key is welcome but not required; preflight resolves it. | +| **`prediction_target`** | The observable outcome (classification/regression) or the positive interaction (link prediction), **stated by the user**. For classification this includes the positive-label condition — "at least one fraud event"; "predict fraud" alone leaves it missing. For link prediction and ranking it includes the **item** being recommended and the **interaction** that counts as positive — "spare-part families the customer ordered"; "recommend to customers" alone leaves both missing. | +| **`time_window`** | Tri-state. `specified` ("next 30 days") → pass. `not_critical` → the task is static (no temporal component: static link prediction, static regression/classification) → pass, do not ask. `missing` → the task is temporal and no window was given → ASK BACK. Decide static vs. temporal with the test below. | + +**Static or temporal — decide from what is being predicted, never from the data.** Almost every +table has a date column (`booked_date`, `event_at`); its presence says nothing about the task. + +- **Static** (`not_critical`, do not ask): + - the target is a value the entity row itself carries — "the final price of each listing", + "the category of each article". The entity is one record; nothing has to happen after it. + - the user asks for **embeddings**, or for links **learned from observed activity** without a + forecasting framing — "learn user and product embeddings from past clicks", + "suggest tags for each article from the article-tag links we already have". +- **Temporal** (window required): the target is something that happens, or is aggregated, over a + period after a point in time — "will churn", "will have a fraud event", "total spend", + "what each customer will reorder", and recommenders, which by default predict future + interactions ("a recommender for each member"). + +If the task is static, restate `Window: static` at the task gate and do not ask. A time column +may still be used as the graph's time column — that is structure, not a window. + +A request that names the dataset ("the e-commerce fraud dataset", a schema name) has supplied +`source_scope` if that name resolves to a location the host accepts. Do not ask the user to restate a +location they have already given in an earlier turn. + +## `train` — inferred, then confirmed (never ask) + +| Field | How it is obtained | +|---|---| +| **seed tables** | The physical tables behind the entity and the target, found by preflight inside `source_scope` | +| **primary keys, time columns** | Preflight; each shown as `[inferred]` unless the user stated it (`[supplied]`) or a semantic view declares it (`[declared]`). A key that preflight could only fail to disqualify is shown `[inferred, unverified]` and must be confirmed before the table can be an entity or an edge destination | +| **links** | Preflight; shown as `[inferred]`, or `[declared]` when read from a semantic view | +| **`table_mode`** | Default `all-connected` — every table reachable from the seeds. Shown as `[default]` in the graph proposal; the user may switch to `named-table` there. | + +User-supplied values always win over inferred ones. +The complete proposal — every table, key, time column, link, and the table mode — is shown and +confirmed **before** any PQL is written. + +## `train` — optional (never ask; preserve if given) + +| Field | Default | +|---|---| +| **`output_location`** | none — training yields a notebook; when given, the same notebook also predicts into it (SKILL.md Step 2) | +| **artifact format** | one notebook (user may ask for a script) | +| **embeddings** | not included; add only when the user explicitly asks (see below) | +| **binary classification threshold** | `0.5`, shown `[default]` at Node 1d for every binary classification, whether predictions are written or stay local. Without one, `predict` leaves `TARGET_PRED` null; it has no default of its own (`kumoai-reference/references/kumo_finetuning_sdk.md` §9) | +| **metrics** | by task type, see `SKILL.md` | + +If the user supplies an `output_location`, carry it verbatim into the task interpretation gate +and the generated artifact — the notebook's predict call writes there. If none is supplied, +predictions stay local and the notebook shows how to add a destination. The destination must be +one the host can write (`hosts//adapter.md`, *Destinations*). **A table is written with +exactly the name passed** — no suffix, no case change (the host's `runtime.md`, +`kumoai-reference/references/kumo_finetuning_sdk.md` §9.1). A file path follows §9.2. Older SDK builds appended +`_predictions`; the current one does not. + +--- + +## `batch_inference` + +Required, resolved and invalid fields for scoring with a saved model are in +`batch_inference.md`. Read it only when the operation is `batch_inference`. + +--- + +## Embeddings + +Default to predictions alone. Add embeddings **only when the user explicitly asks for them** — +never ask back for this, and never infer it from the task being recommendation-shaped. + +Recognise: "we need shopper and item embeddings", "learn customer and product embeddings from +completed orders", "export the entity and target-item embeddings from that model". + +When embeddings are requested, say so explicitly in every reply that restates the request — the +graph proposal and the task interpretation gate — the user must see that embeddings are part of +the output. Carry it into codegen as +`PredictionOutputConfig(artifact_type=...)`; see `kumoai-reference/references/kumo_finetuning_sdk.md` §9. + +--- + +## Canonical good example (`train`) + +> "Create a notebook that trains a Kumo model for our fraud use case. Data is in +> `USER_DATA.E_COMMERCE`. Entity: each customer. Outcome: at least one fraud event. Window: next 30 days." + +What makes it good: +- `source_scope`: `USER_DATA.E_COMMERCE` ✓ +- `prediction_entity`: each customer ✓ +- `prediction_target` with positive-label condition: at least one fraud event ✓ +- `time_window`: `specified`, 30 days ✓ +- No columns, keys or joins described — **correct**; preflight infers them and the user confirms. + +## Canonical bad example (`train`) + +> "Use my sales data to figure out which customers are risky." + +What is missing: +- `source_scope`: "sales data" does not resolve to a location ✗ +- `prediction_target`: "risky" is not an observable outcome ✗ +- `time_window`: `missing` for what reads as a temporal question ✗ + +Not missing, and not to be asked for: schema, join keys, output table. + +--- + +## What to do when a required field is missing + +Ask for **exactly** the missing required fields, once, in one turn. Do not ask for structure, +table mode, or an output table on a training request. On a `batch_inference` request, do not ask +for the source schema, population, or anchor date (see the *resolved, defaulted, or inferred* +table). Do not invent values. Ask in the user's +vocabulary ("what should be recommended to each customer?"), and do not attach questions +about columns or values for the fields that *are* present — those are confirmed later, as +`[inferred]`. Example: + +> "To proceed I need: +> 1. **Where the data lives** — for example a database and schema, or a folder of files. +> 2. **The prediction window** — e.g. 'in the next 30 days'." + +Ask only for missing, ambiguous, or contradictory inputs, restate the resolved contract, and never +request secrets in chat. diff --git a/skills/kumo-data-science/accept/preflight.md b/skills/kumo-data-science/accept/preflight.md new file mode 100644 index 000000000..abed8b5ba --- /dev/null +++ b/skills/kumo-data-science/accept/preflight.md @@ -0,0 +1,241 @@ +# Read-only Preflight — inferring structure before the graph gate + +Once the required fields are present, the agent resolves **structure** itself. It does not ask +the user for column names, keys, or joins, and it builds no Kumo table or graph at this point — +preflight is a catalog read of the source plus explicitly targeted, read-only probes against +source data, and it is read-only throughout. A probe exists only to produce evidence for the +proposal; it is never written anywhere and never becomes a model input. + +**What** preflight establishes is portable and written here. **How** it reads the source — which +tool, which catalog query — belongs to the active host, in the *Access* and *Preflight* sections +of `hosts//adapter.md`. The examples below are SQL; a host whose source has no SQL answers +the same questions with its own reader. + +Preflight produces the **graph proposal** the user confirms at Node 1c. Every value carries a tag: + +| Tag | Meaning | +|---|---| +| `[supplied]` | the user stated it; never overridden | +| `[declared]` | read from a semantic view's definition (`semantic_view.md`); shown for confirmation | +| `[inferred]` | measured by preflight; shown for confirmation | +| `[inferred, unverified]` | a candidate preflight could not establish, only fail to disqualify; must be confirmed before use | +| `[default]` | a contract default (`table_mode: all-connected`); shown for confirmation | + +If the harness or host has already run preflight and handed you a result (for example a +"Preflight result for " message), use that result as-is — do not ask the user to repeat it. + +--- + +## 0. Access — can you read the source? + +Before §1, establish whether **you**, in this session, can read `source_scope` with a read-only +tool you already have. The host adapter's *Access* section says which tools count and the check +that proves each one reaches the data. + +- **You can read it** → run §1-§6 through that tool, and name the tool in the proposal's + `Source scope` line. +- **You cannot** — no suitable tool, or the tool reaches the system but not this scope (an + authorization error, an unknown schema or bucket) → stop before §1. In one message, tell the + user exactly what you checked and what happened, quoting any error, and offer both ways forward: + 1. **Access** — they make a read tool available, as the adapter describes, and tell you; you + then repeat the check. + 2. **Details** — they paste the structure: tables, columns with types, and any declared keys, + in whatever form they have (the adapter lists useful ones). You build the proposal from it. + + This is one ask-back. Do not also ask for anything §1-§8 would infer. +- **Never** ask for a credential's value or invite one into the chat, sign in, change a tool's + configuration, or set up a tool on the user's behalf. Never broaden the search beyond + `source_scope` because the scope itself was unreadable. + +A proposal built from the user's details follows §1-§8 unchanged, except that nothing that needs +row data can run: the §3 rule 5 probe is skipped, so its candidates stay `[inferred, unverified]`. +Say in the `Source scope` line that the structure came from the user, not from a read. + +This check concerns your session only. The generated artifact authenticates at run time through +its own connect step (the host's `runtime.md`), so your access — or its absence — changes +neither the generated code nor any gate. + +--- + +## 1. Resolve `source_scope` + +If `source_scope` is a semantic view (`DB.SCHEMA.VIEW`), read `accept/semantic_view.md` and +resolve it there instead of here; then continue at §2. + +```sql +SELECT SCHEMA_NAME FROM .INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME ILIKE ''; +SELECT TABLE_NAME, ROW_COUNT FROM .INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA ILIKE ''; +``` + +- Not found, or zero tables → **ASK BACK**: the source cannot be resolved; ask for a valid + location. Do not guess a similarly named schema, dataset, bucket or directory. +- Found → enumerate tables **only inside this scope**. Never search other schemas, and never + rank or curate tables by business relevance. + +## 2. Columns and types + +```sql +SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE +FROM .INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA ILIKE '' +ORDER BY TABLE_NAME, ORDINAL_POSITION; +``` + +## 3. Primary key per table + +In order of preference, first match wins: +1. A declared primary key, where the source declares any (the adapter's preflight shows how to + read them; files and object stores declare none). Declared is not enforced: it is probed + before use (**Probe every key the proposal uses**, below). +2. `_ID`, where the stem is the table name without a `_DIM` / `_PROFILE` / `_EVENTS` + suffix, singular or plural (`CUSTOMER_ID` in `CUSTOMERS`, `PAGE_VIEW_ID` in `PAGE_VIEW_EVENTS`). +3. A column named `ROW_ID`, `ID`, or `EVENT_ID`. +4. For dimension / profile tables only: `_ID` for any token of the table name + (`ACCOUNT_ID` in `ACCOUNT_PROFILE`). Never for event tables — there such a column is a link. +5. A column that one or more *other* tables in scope carry under the same name is a **candidate** + only. Shared naming does not make it a key, and neither does uniqueness: `COUNTRY`, `STATUS` or + `MERCHANT` can be ordinary attributes, and an accidentally unique attribute is still not an + identifier. A duplicated inferred key silently discards rows. A column that rules 1-4 already + established as another in-scope table's key is a link to that table, not a candidate here — + skip it, as rule 4 does for event tables. + + Use a targeted read-only probe to **reject** candidates, never to establish identity: + + ```sql + SELECT COUNT(*) AS N, COUNT(DISTINCT ) AS N_DISTINCT, COUNT() AS N_NON_NULL + FROM ..
; + ``` + + Anything other than `N > 0 AND N_DISTINCT = N_NON_NULL = N` disqualifies the candidate outright + — an empty table returns three zeros and does not qualify. Surviving the probe makes it a + **proposed** key, tagged `[inferred, unverified]`, which the user confirms or corrects at the + gate before it is used as the `FOR EACH` entity or an edge destination. If several candidates + survive on one table, or the same candidate survives on more than one side, report the + ambiguity and propose none. If the probe cannot run, list the column under `Not found` as an + unverified candidate. + + Rules 2-4 match names only, so run the same probe on the column they pick: a column that + fails it is a link, not a key (`CUSTOMER_ID` in `CUSTOMER_EVENTS`); go on to the next rule. + If the probe cannot run, the rule's pick is `[inferred, unverified]`, as here. + +6. Otherwise preflight **inferred no primary key**. Report that as its own line and treat the + table as keyless unless corrected. A keyless table may be an edge *source*, but not an + `Edge.dst_table` and not the `FOR EACH` entity. Do not conclude from this failure alone that + the table is an event or fact table — it may have a natural, composite or unconventional key + the heuristic missed. Never promote a merely unique-looking column to primary key: a column + unique in the rows you can see is not a declared key, and a child table may hold several rows + per parent. + +Rules 1-4 establish a key and name it `[inferred]` (rules 2-4 once their probe passes). Rule 5 +never establishes one: it proposes a survivor as `[inferred, unverified]`. + +**Probe every key the proposal uses**, not only rule-5 candidates. A declared key (rule 1) or a +`[supplied]` key that will be an edge destination or the `FOR EACH` entity gets the same read-only +probe; rules 2-4 picks already had it. Snowflake, BigQuery and Databricks record a declared +primary key without enforcing it, so declared does not mean unique. If `N_DISTINCT < N_NON_NULL` +or `N_NON_NULL < N`, keep it in the proposal with the counts and list it under `Not found` — +"`orders.order_id`: 792 repeated values, 0 nulls" — because Kumo keeps one row per key and drops +the rest without failing. The user decides: a different key, or the source fixed upstream. The +agent does not deduplicate or filter. Until the user confirms it, the table is treated exactly as +keyless — usable as an edge source, but not as an edge destination and not as the `FOR EACH` +entity. Confirmation is what lifts that restriction, not the existence of a candidate. + +## 4. Time column per table + +The first `TIMESTAMP*`, `DATE`, or `DATETIME` column, preferring names ending in `_AT`, `_TS`, +`_TIME`, `_DATE`, or named `EVENT_TIME` / `TIMESTAMP`. A table with none is static. + +For every time-ranged target aggregation, the table it references must have a usable +`time_column`. If it does not, say so in the proposal and say what would make the task +expressible: a timestamp on some other table does not repair the target. + +A catalog `TEXT` column may still be usable when its date semantics are established and the +generated code casts it with `dtype=Dtype.date`; the absence of a timestamp-typed catalog column +is not by itself grounds for rejection. Integers that merely order or describe events — a +sequence number, a day-of-week, an hour, a gap in days since a previous event — are not +timestamps and must not be treated as one. + +## 5. Column availability + +For each table, place every candidate feature in time relative to that table's proposed +`time_column`, then exclude the ones that are not yet knowable at it. + +- When the time column marks the beginning of a validity period or interval, exclude a feature + whose value is only determined once that interval later changes or ends: its eventual endpoint, + its realized duration, or any summary of the completed interval. +- Exclude a feature computed from, or containing, the confirmed prediction target. +- Apply the active host adapter's runtime column constraints. + +Decide by semantic role, never by a fixed column-name list. Record every exclusion and its reason +in the proposal. Where the role is genuinely ambiguous, do not invent one — show the column as an +exclusion the user can overturn at the gate, rather than asking the user for column names. + +## 6. Links + +A link is proposed when a column in table A has the **same name** as the primary key of table B +(`FRAUD_EVENTS.CUSTOMER_ID → CUSTOMERS.CUSTOMER_ID`). Foreign-key constraints, if declared, are +taken as well, where the source declares any. Nothing else is inferred — no fuzzy matching, +no value overlap checks, no joins executed. + +## 7. Seeds and table mode + +- **Seeds** are the tables behind `prediction_entity` and `prediction_target` (for link + prediction: source entity, item entity, and the interaction table). Resolve them by name + against the enumerated tables; if none matches, ask the user which table holds the entity or + the outcome — that is intent, not structure. +- **Preflight runs only after Node 1a passes.** It resolves tables for fields the user already + supplied; it never back-fills a field the user left out. If the required-fields check found + `positive_interaction`, `recommended_item`, `positive_label_condition` or `time_window` + missing, ask for them first — do not read them off the schema. +- **`table_mode: all-connected [default]`** — every table reachable from the seeds through the + proposed links is included. List any table that is **not** reachable so the user can drop it + or name a link. +- If the seeds cannot be connected through inferred links, say so in the proposal and ask + which existing columns link them; a name mismatch can hide a real link. Generate nothing until + the seeds connect. If no existing column pair connects them, **reject**: connecting them would + require creating or transforming data. + +## 8. The graph proposal + +Present exactly this shape, then **stop and wait** for confirmation: + +``` +Source scope: USER_DATA.E_COMMERCE [supplied] — 7 tables found, read via SQL +Table mode: all-connected [default] — 7 of 7 tables reachable from the seeds + +Metadata +- customers: primary key CUSTOMER_ID [inferred] +- fraud_events: primary key EVENT_ID [inferred] +- fraud_events: time column EVENT_AT [inferred] +- ... + +Links +- fraud_events.CUSTOMER_ID -> customers.CUSTOMER_ID [inferred] +- ... + +Excluded columns +- fraud_events.RESOLVED_AT [inferred] — not knowable at the row's EVENT_AT +- fraud_events.DAYS_TO_RESOLVE [inferred] — duration of an interval that has not ended yet +- ... + +Not found +- payments: no primary key inferred [inferred] — treated as keyless; edge source only unless corrected +- events: no usable time column [inferred] — a time-ranged target cannot reference this table +- ... + +Confirm, or correct any line (your values take precedence). +``` + +`Excluded columns` and `Not found` are local availability-gate extensions to the confirmation +format. + +## 9. Never + +- Prepare, transform, aggregate or materialize **model inputs** — joining, cleaning, + deduplicating, creating labels. A diagnostic scalar aggregate over source rows is permitted + solely as evidence for the proposal: row, distinct and null counts for a proposed or candidate key. Its + result is reported and then discarded; it is never written anywhere and never reaches training. +- Rank tables by relevance or drop tables the user did not ask to drop. +- Proceed to PQL before the proposal is confirmed. +- Ask the user for columns, types, keys or joins — with one exception: when §0 finds you cannot + read the source, offering to take its structure from the user is the §0 ask-back. Whatever the + proposal is built from, it is still tagged, still confirmed, and contains no invented values. diff --git a/skills/kumo-data-science/accept/semantic_view.md b/skills/kumo-data-science/accept/semantic_view.md new file mode 100644 index 000000000..c77af159c --- /dev/null +++ b/skills/kumo-data-science/accept/semantic_view.md @@ -0,0 +1,146 @@ +# Semantic view as `source_scope` + +Read this file only when `source_scope` is a fully qualified semantic view (`DB.SCHEMA.VIEW`). It +replaces `preflight.md` §1 for that case; every other preflight section still applies. Nothing here +applies to a schema or connector. + +## Resolve the view + +A semantic view is not a schema to search. It is a definition that points at physical tables and +declares their keys and relationships. Read the definition; do not query the view itself. + +```sql +DESCRIBE SEMANTIC VIEW ..; +``` + +The result holds one `property` / `property_value` pair per object, not one column per field. +Group the rows by `object_name` for each `TABLE` and `RELATIONSHIP` object, then read each field +as "the row whose `property` is X, taking `property_value`": for a `TABLE` object read +`BASE_TABLE_DATABASE_NAME`, `BASE_TABLE_SCHEMA_NAME`, `BASE_TABLE_NAME` and `PRIMARY_KEY` this way; +for a `RELATIONSHIP` object read `TABLE`, `REF_TABLE`, `FOREIGN_KEY` and `REF_KEY` this way — +`TABLE` and `REF_TABLE` are the view's logical names too; map each to its physical table through +the extracted `TABLE` records before using it anywhere downstream. + +- Not found or unreadable → **ASK BACK**, exactly as for a schema that does not resolve. +- **A logical table with no `BASE_TABLE_*` properties does not name a physical table.** One + defined by a query carries its own definition instead. The SDK can read a query through + `connector.sql(...)`, but this skill does not build graph tables from SQL it composed itself, so + the view alone is not enough. **ASK BACK**: name the logical table, say it is defined by a query + rather than a base table, and ask for a view whose tables are physical or for the physical table + to use. Do not parse the definition or rebuild it as a query of your own. +- **Only a plain column-to-column equality relationship becomes an edge.** A Kumo `Edge` is + foreign key to primary key, matched on equality. If a relationship declares any other kind of + match, do not read its `FOREIGN_KEY` / `REF_KEY` as an ordinary key pair: a non-equality payload + can read as a one-column key and become an edge that does not mean what the view says. + **ASK BACK**: name the relationship and its kind, and say only equality joins become edges. +- **The base tables must all sit in one database and schema.** The connector reads from a single + schema. If they span more than one, **ASK BACK**: name the schemas found and ask for a semantic + view or a schema whose tables share one schema. Do not pick one schema, and do not drop tables + to make it fit. +- That one schema becomes the effective scope: it is the `database` / `schema_name` passed to + the connector, and `preflight.md` §2 reads columns from it. +- **The tables in scope are exactly the view's base tables** — not every table in that schema. + The user chose them by naming the view; that is their selection, not curation by the agent. +- **Always use physical names.** The view's logical names (`USERS`, `TRANSACTIONS`) do not exist + for Kumo. Every table, key, link, edge and PQL reference uses the base table and column names, + with the same case and quoting the catalog reports. +- Each `PRIMARY_KEY` and each relationship is `[declared]`. They replace `preflight.md` §3 and §5 + for the tables they cover; those sections still apply to any base table the view gives no key + or link for. +- A key with more than one column, on either side of a relationship, cannot become a Kumo edge. + **ASK BACK**: name the relationship and say a single-column key is needed, the same way as a + view that does not resolve — ask for a corrected view or schema. Do not offer to drop the + table, concatenate the columns, or substitute a different link as options. Those are repairs, + not a corrected source, and a menu of repairs is not an ask-back. +- A declared `PRIMARY_KEY` with more than one column only matters when it is what some + relationship's `REF_KEY` points **at**. If no relationship's `REF_KEY` is this composite key, it + needs no ask-back by itself — not even when this table is the **foreign-key** side of its own + relationships and one of those `FOREIGN_KEY` columns happens to also be part of the composite + key; pointing out of a table never depends on that table's own primary key. Leave the table in + the graph exactly as `preflight.md` §3 treats a table with no usable primary key: keyless, or + with a single-column `[inferred]` key (e.g. an `*_ID` column) offered for confirmation. That + keyless or `[inferred]` treatment applies only while **no** relationship references this table + at all. If some relationship's `REF_KEY` is a different single column of it, that column is the + key, by the bullet below; do not infer another, because an edge points at the key Kumo is given + and a guessed one joins the foreign key to the wrong column. Ask back only when some + relationship's `REF_KEY` is the composite key — that is the previous bullet, not this one. +- A relationship whose **single-column** `REF_KEY` is not the referenced table's declared + `PRIMARY_KEY` (for example it references a `UNIQUE` key instead) is still expressible. A Kumo + primary key is declared by the caller in `Table.from_source_table(..., primary_key=)` and does + not have to match the catalog, so that column becomes the table's Kumo primary key, shown + `[declared]`. Say in the proposal that it differs from the catalog's primary key so the user can + correct it. This is not a repair; it is reading the view as written. +- That holds only while one column can serve the whole table. When **two relationships reference + two different columns of the same table**, a Kumo table has one primary key and cannot carry + both. **ASK BACK**: name both relationships and the column each references, say only one can be + an edge destination, and ask for a corrected view, or for explicit authorization to build without + one named relationship. Asking only which column to key on is not enough, because either answer + still leaves the other relationship unbuildable. Do not silently key on one and leave the + other relationship unbuilt, and do not emit the table twice to satisfy both: separate copies are + not joined to each other, so `A -> Entity <- B` would silently become two disconnected + components, and an entity split this way is ambiguous as the `FOR EACH` target. +- Ignore the view's dimensions, facts and metrics. They describe aggregate reporting, not the + graph. A metric is not the user's outcome: if the user has not described one, `prediction_target` + is missing, as for any source. If they have ("completed orders", "fraud events"), map their + words to columns as for any source and show the mapping `[inferred]` at the gate — do not ask + which column or value implements it. + +## Tag precedence + +Precedence: `[supplied]` > `[declared]` > `[inferred]`. `[inferred, unverified]` is not a rung +on that ladder but a weaker form of `[inferred]`: whatever its source, a value carrying it must +be confirmed before the table can be an entity or an edge destination. + +## In the graph proposal + +When `source_scope` is a semantic view, the first line names the view and the schema it resolved +to, and keys and links read from it carry `[declared]`: + +``` +Source scope: semantic view SALES_DB.ML.CHURN_SV [supplied] — 3 base tables in SALES_DB.ML +... +- orders: primary key order_id [declared] +- orders.customer_id -> customer_profile.customer_id [declared] +``` + +## Examples + +### Valid view → graph proposal + +> "Train a churn model on the semantic view `SALES_DB.ML.CHURN_SV`. Entity: each customer. +> Outcome: no order in the next 60 days." + +Node 1a: `source_scope` is the semantic view ✓ — nothing else is asked. At Node 1c the agent runs +`DESCRIBE SEMANTIC VIEW`, finds three base tables, all in `SALES_DB.ML`, and replies — then stops: + +> Source scope: semantic view `SALES_DB.ML.CHURN_SV` [supplied] — 3 base tables in `SALES_DB.ML` +> Table mode: all-connected [default] — 3 of 3 tables reachable from the seeds +> +> **Metadata** +> - customer_profile: primary key customer_id [declared] +> - orders: primary key order_id [declared]; time column ordered_at [inferred] +> - support_tickets: primary key ticket_id [declared]; time column opened_at [inferred] +> +> **Links** +> - orders.customer_id -> customer_profile.customer_id [declared] +> - support_tickets.customer_id -> customer_profile.customer_id [declared] +> +> Confirm, or correct any line — your values take precedence. + +The proposal uses the physical names (`customer_profile`, not the view's logical `CUSTOMERS`), +lists only the view's three tables even if `SALES_DB.ML` holds more, and the connector is built on +`SALES_DB` / `ML`. The view's metrics play no part — the outcome is still the user's words. + +### Base tables span more than one schema → ASK BACK + +> "Predict 30-day churn for each customer using the semantic view `SALES_DB.ML.CHURN_SV`. +> Churn means no order in the window." + +Preflight: `DESCRIBE SEMANTIC VIEW` finds base tables in two schemas — `customer_profile` in +`SALES_DB.CRM`, `orders` in `SALES_DB.ML`. + +> "`CHURN_SV` reads from two schemas — `SALES_DB.CRM` (customer_profile) and `SALES_DB.ML` +> (orders). I can only build a graph from tables in a single schema. Could you point me to a +> semantic view or a schema where these tables sit together?" + +Not done: picking one schema, dropping a table, or copying data across schemas. diff --git a/skills/kumo-data-science/evals/evals.json b/skills/kumo-data-science/evals/evals.json new file mode 100644 index 000000000..a82cdf78a --- /dev/null +++ b/skills/kumo-data-science/evals/evals.json @@ -0,0 +1,116 @@ +{ + "skill_name": "kumo-data-science", + "evals": [ + { + "id": "kds-01", + "prompt": "Use my sales data to figure out which customers are risky. Build me a Kumo model for it.", + "expected_output": "Treats this as a `train` request with required fields missing. In one ask-back it asks for where the data lives (for example a database and schema, or a folder of files), the observable outcome that makes a customer 'risky' (the positive-label condition), and the prediction window. It does not ask for columns, keys, joins or an output table, does not guess an outcome, and generates no code.", + "assertions": [ + "Asks where the data lives, as a location such as a database and schema, a dataset, a bucket prefix or a folder of files", + "Asks for a concrete, observable outcome that defines 'risky' (the positive-label condition) instead of inventing one", + "Asks for the prediction window (for example 'in the next 30 days')", + "Asks for all missing fields together in a single reply", + "Does not ask for column names, primary keys, joins or an output table", + "Does not write a notebook, Kumo SDK code or a predictive query" + ], + "expected_skill": "kumo-data-science", + "expected_script": null + }, + { + "id": "kds-02", + "prompt": "Create a notebook that trains a Kumo model for our fraud use case. The data is in Snowflake, USER_DATA.E_COMMERCE. Entity: each customer. Outcome: at least one fraud event. Window: next 30 days. You can't reach our Snowflake account from here, so this is the structure; no keys are declared:\n\nCUSTOMERS(CUSTOMER_ID NUMBER, COUNTRY VARCHAR, SIGNUP_AT TIMESTAMP_NTZ)\nFRAUD_EVENTS(EVENT_ID NUMBER, CUSTOMER_ID NUMBER, EVENT_AT TIMESTAMP_NTZ, AMOUNT NUMBER, RESOLVED_AT TIMESTAMP_NTZ, DAYS_TO_RESOLVE NUMBER)\nTRANSACTIONS(TRANSACTION_ID NUMBER, CUSTOMER_ID NUMBER, CREATED_AT TIMESTAMP_NTZ, AMOUNT NUMBER)", + "expected_output": "All required fields are present, so it builds the graph proposal from the pasted structure and stops for confirmation. The Source scope line says the structure came from the user. Keys CUSTOMERS.CUSTOMER_ID, FRAUD_EVENTS.EVENT_ID and TRANSACTIONS.TRANSACTION_ID are proposed by name and, since no probe can run, tagged as unverified; time columns are EVENT_AT and CREATED_AT; FRAUD_EVENTS.CUSTOMER_ID and TRANSACTIONS.CUSTOMER_ID link to CUSTOMERS.CUSTOMER_ID; table mode all-connected [default]. An Excluded columns section removes FRAUD_EVENTS.RESOLVED_AT and DAYS_TO_RESOLVE as not knowable at the row's EVENT_AT. It ends by asking the user to confirm or correct any line, with no Kumo code.", + "assertions": [ + "Presents a graph proposal (metadata, links, table mode) built from the pasted structure rather than asking for the structure again", + "Tags proposal values with provenance tags such as [supplied], [inferred], [inferred, unverified] or [default]", + "Says the structure came from the user rather than from a read of the source", + "Proposes links from FRAUD_EVENTS.CUSTOMER_ID and TRANSACTIONS.CUSTOMER_ID to CUSTOMERS.CUSTOMER_ID", + "Lists FRAUD_EVENTS.RESOLVED_AT and FRAUD_EVENTS.DAYS_TO_RESOLVE as excluded columns because their values are not known at the event time", + "Asks the user to confirm or correct the proposal and stops there", + "Does not emit Kumo SDK code or a predictive query in this reply" + ], + "expected_skill": "kumo-data-science", + "expected_script": null + }, + { + "id": "kds-03", + "prompt": "Build a Kumo notebook that predicts, for each account in our BigQuery dataset acme-prod:billing, whether the account will have at least one failed payment in the next 14 days.", + "expected_output": "The request has every required field. Before preflight it checks whether it can read the dataset's columns in this session (a BigQuery tool, an authenticated `bq` on the PATH, or Python with Kumo and ambient credentials). It cannot, so it sends one ask-back saying what it checked and what happened, and offers two ways forward: the user makes a read tool available, or pastes the structure (tables, columns, types, declared keys). It never asks for a credential value, does not sign in or change tool configuration, does not invent tables, and writes no code.", + "assertions": [ + "Says what it checked to read the dataset and what the result was", + "Offers the user a way to grant read access (an authenticated tool or a connected tool for BigQuery)", + "Offers the alternative of pasting the structure: tables, columns with types and any declared keys", + "Does not ask for a password, key, token or other credential value in the chat", + "Does not invent tables, columns or a graph proposal for acme-prod:billing", + "Does not re-ask for the entity, outcome or window, which the user already gave", + "Does not generate the notebook in this reply" + ], + "expected_skill": "kumo-data-science", + "expected_script": null + }, + { + "id": "kds-04", + "prompt": "Earlier in this session you proposed this graph for our fraud request (Snowflake USER_DATA.E_COMMERCE; entity: each customer; outcome: at least one fraud event; window: next 30 days):\n\nMetadata\n- CUSTOMERS: primary key CUSTOMER_ID [inferred]\n- FRAUD_EVENTS: primary key EVENT_ID [inferred]; time column EVENT_AT [inferred]\nLinks\n- FRAUD_EVENTS.CUSTOMER_ID -> CUSTOMERS.CUSTOMER_ID [inferred]\nExcluded columns\n- FRAUD_EVENTS.RESOLVED_AT [inferred]\n\nI confirm the graph. Please write the predictions to ANALYTICS.ML.fraud_scores.", + "expected_output": "With the graph confirmed it moves to the task interpretation gate and restates the task in the documented shape: one prediction per CUSTOMERS.CUSTOMER_ID; outcome at least one fraud event; window 0 to 30 days after the prediction anchor; temporal binary classification; predictions written to ANALYTICS.ML.fraud_scores, verbatim, in the same notebook; threshold 0.5 [default]. It says whether the destination already exists and would be replaced, or that it could not check. It then asks for explicit confirmation and emits no Kumo code.", + "assertions": [ + "Restates the entity as one prediction per customer (CUSTOMERS.CUSTOMER_ID)", + "Restates the outcome as at least one fraud event and the window as the next 30 days", + "Names the task type as temporal binary classification", + "Carries the destination ANALYTICS.ML.fraud_scores verbatim, without changing its case or adding a suffix", + "States a classification threshold of 0.5 as the default", + "Says whether the destination table already exists and would be replaced on every run, or that it could not check", + "Asks for explicit confirmation and does not emit Kumo SDK code or a notebook yet" + ], + "expected_skill": "kumo-data-science", + "expected_script": null + }, + { + "id": "kds-05", + "prompt": "I trained a Kumo model in my notebook an hour ago and `training_result` is still in memory. Which columns influenced its predictions the most?", + "expected_output": "This is an attribution question about an existing run, not a new training request. It routes to the kumo-model-explanations skill (or says that skill is not available here), keeps the existing run, and does not run new-training intake or start another fit.", + "assertions": [ + "Treats the request as an explanation question about the existing run, handled by the kumo-model-explanations skill (or reports that skill is unavailable rather than improvising its workflow)", + "Does not ask for the data location, entity, outcome or prediction window", + "Does not start, propose or generate a new training run" + ], + "expected_skill": "kumo-data-science", + "expected_script": null + }, + { + "id": "kds-06", + "prompt": "Schedule the Kumo churn notebook you generated to run every night at 2am as a job on our cluster, and tell me when the first run finishes.", + "expected_output": "In this generic coding-agent host there is no job file: the skill does not submit, schedule or monitor jobs. It says so and offers the notebook or script to run themselves. It does not create a scheduler entry or submit anything.", + "assertions": [ + "Says that in this environment the skill does not submit, schedule or monitor jobs", + "Offers the notebook or a script as what it can provide instead", + "Does not create a cron entry, scheduler configuration or job submission", + "Does not claim to watch or report on a run" + ], + "expected_skill": "kumo-data-science", + "expected_script": null + }, + { + "id": "kds-07", + "prompt": "Set up kumo.init with my KUMO_API_KEY and train a churn model on Kumo's hosted service.", + "expected_output": "Hosted Kumo (kumo.init, API keys, KUMO_API_KEY / KUMO_API_ENDPOINT, Kumo Studio) is not the SDK this skill targets. It says so and stops, and does not blend hosted setup into generated code.", + "assertions": [ + "Says that hosted Kumo (kumo.init with an API key) is not the SDK this skill works with", + "Does not write code that calls kumo.init or reads KUMO_API_KEY", + "Does not ask the user to paste an API key" + ], + "expected_skill": "kumo-data-science", + "expected_script": null + }, + { + "id": "kds-08", + "prompt": "Write a pandas function that removes duplicate rows from a DataFrame, keeping the most recent row per order_id by updated_at.", + "expected_output": "A plain pandas task with no Kumo or predictive-model element. The agent answers directly, for example with sort_values plus drop_duplicates, without loading the Kumo skill or starting its intake.", + "assertions": [ + "Does not load or follow the kumo-data-science skill or ask Kumo intake questions", + "Provides pandas code that keeps the latest row per order_id by updated_at" + ], + "expected_skill": null, + "expected_script": null + } + ] +} diff --git a/skills/kumo-data-science/hosts/generic/adapter.md b/skills/kumo-data-science/hosts/generic/adapter.md new file mode 100644 index 000000000..f907eb1ee --- /dev/null +++ b/skills/kumo-data-science/hosts/generic/adapter.md @@ -0,0 +1,149 @@ +# Host Adapter — Generic coding agent + +For a coding agent working beside the user — Claude Code, Codex, or any agent with a shell and +file tools — rather than inside a data platform. The agent produces code that runs in **any** +Python environment where Kumo is present, against whichever of the six Kumo connectors the data +calls for. The agent does not know where that code will run; the user does. + +Kumo is already present in the environment that runs the artifact; never raise environment setup +with the user. Generating the artifact — connect step, credentials, destinations and read-back — +is in `runtime.md` in this directory; read it at Step 2. + +This host has no job file: it does not submit, schedule or monitor jobs. When asked for one, say +so and offer the notebook or script. + +--- + +## 1. Source scope + +`source_scope` names a system and a location in it. Map it to its connector +(`kumoai-reference/references/kumo_finetuning_sdk.md` §3, §3.4): + +| The user names | Connector | +|---|---| +| A local directory of CSV or Parquet files | `FileConnector` | +| `s3://bucket/prefix` | `S3Connector` | +| `gs://bucket/prefix` | `GCSConnector` | +| A BigQuery project and dataset | `BigQueryConnector` | +| A Databricks catalog and schema | `DatabricksConnector` | +| A Snowflake database and schema | `SnowflakeConnector` | + +- A dotted name with no system named (`SALES.EVENTS`) fits Snowflake, BigQuery and Databricks + alike. Ask which system holds it — that one question, nothing else. A system already named in + any turn, or evident from the user's tools or files, settles it. +- A connector name on its own is not a location on this host, which keeps no registry of named + connectors. Ask what system and location it points at. +- A Snowflake semantic view resolves through `kumo-data-science/accept/semantic_view.md`, which + needs Snowflake SQL (§2). Without it, the §2 details are the view's base tables, their keys and + its relationships. +- Values only the runtime needs — a Snowflake warehouse, a Databricks SQL warehouse or cluster, a + `connections.toml` entry — are not part of `source_scope` and are never asked for at intake. + `runtime.md` §1 handles them. + +ADLS has no Kumo connector. Data there is outside this host's reach: say so. + +--- + +## 2. Access + +Implements `kumo-data-science/accept/preflight.md` §0. Check, in this order, and stop at the +first that reads the **columns and types** inside `source_scope`. A listing of object names +(`aws s3 ls`, `gcloud storage ls`) proves access, not structure: keep going down the list for a +reader, and if none is found, send the ask-back below with **Details** as the way forward. + +1. **A tool in this session** that reads the system — a server or connector tool for that + warehouse or storage. Use only its read operations. +2. **A command-line tool on the `PATH`** that is already authenticated (table below). Find it + with `command -v `, then run its check command against `source_scope` itself — a + general "who am I" call proves the login, not access to the data. +3. **Python in this session** in which `import kumo.engine` works and the provider's ambient + credentials resolve. Build the connector exactly as the artifact would (`runtime.md` §2) and + read `connector.table_names()` and `connector['
'].columns`, which gives each column's + name and source type. Catalog reads only: no `Table`, `Graph` or snapshot. + +| Source | Tool | Check, inside `source_scope` | Columns and types | Declared keys | §3 probe | +|---|---|---|---|---|---| +| Local files | your file tools; `duckdb`, or Python with `pyarrow` | list the directory | CSV header line; Parquet schema (`duckdb -c "DESCRIBE SELECT * FROM ''"`) | none | `duckdb`, or `pyarrow` | +| S3 | `aws` | `aws s3 ls s3:////` | file names only — not enough on its own; for columns, a reader of `s3://` with the same credentials (`duckdb`, or check 3), else the ask-back | none | the same reader, else skip | +| GCS | `gcloud` | `gcloud storage ls gs:////` | as S3, with `gs://` | none | as S3 | +| BigQuery | `bq` | `bq ls :` | `bq show --schema --format=prettyjson :.
` | `INFORMATION_SCHEMA.TABLE_CONSTRAINTS` and `KEY_COLUMN_USAGE` of the dataset | `bq query --nouse_legacy_sql` | +| Databricks | `databricks` | `databricks tables list ` | `databricks tables get ..
` | the same output's table constraints | only through a SQL warehouse the user named; else skip | +| Snowflake | `snow` | `snow sql -q ""` | the preflight §2 query | `SHOW PRIMARY KEYS` / `SHOW IMPORTED KEYS IN SCHEMA .` | `snow sql -q` | + +A skipped probe leaves its candidates `[inferred, unverified]`, as preflight §0 says. A BigQuery +probe is billed by the bytes it scans; probe only the candidate columns. + +For BigQuery, also read each dataset's `location` with `bq show --format=prettyjson +:` — the source, and the population and destination datasets once confirmed. +A population dataset in a different location from the source cannot be filled by one query: say +so before accepting it. The notebook reads locations itself (`runtime.md` §2), so they are not +config constants. + +**What a check may do.** Read catalog metadata and run the preflight §3 probe — nothing else. +Never run a command that writes, creates, copies or exports data; never sign in, switch a +profile or account, or edit a tool's configuration; never add or update software to get a +reader. Tool permission prompts are the user's to answer; a refused prompt means that path is +unavailable. + +**When nothing reads the source's columns**, send the preflight §0 ask-back. State what you checked and +what happened — "no BigQuery tool in this session; `bq ls my-proj:sales` returned `Access +Denied`" — then offer: + +1. **Access**: authenticate the named command-line tool as an identity that can read the scope, + or connect a tool for that system to this session, and tell you when it is done. The user + does this; you never handle the credential. +2. **Details**: paste the structure. No rows are needed. Useful forms: DDL, `DESCRIBE` output, + `bq show --schema` JSON, a Parquet schema, CSV header lines — with any declared keys. + +--- + +## 3. Preflight + +Answer `kumo-data-science/accept/preflight.md` §1-§6 with the tool §2 found. Write table and +column names exactly as that tool returns them; the artifact must use the same spelling +(`kumoai-reference/references/kumo_finetuning_sdk.md` §3.2). Probe results are evidence for the +proposal only: never written, never saved to a file, never fed to the artifact. + +For files and object stores, a table is a file or a directory of parts in one format, below the +source location. **Name it without its file suffix**: `accounts.parquet` is the table `accounts`. +The connector finds the file from the bare name, and the name you pass becomes the table's name +in the graph, the edges and the PQL, where a suffix's dot does not parse. Show the bare names in +the proposal. + +--- + +## 4. Destinations + +Any destination a Kumo connector writes +(`kumoai-reference/references/kumo_finetuning_sdk.md` §9.2): a table in Snowflake, BigQuery or +Databricks, or a path in a local directory, S3 or GCS. The destination may be in a different +system from the source; it gets its own connector. Not destinations here: ADLS, and BigQuery for +anything but predictions and embeddings. + +A path destination ends in `/` (a directory) or in `.parquet` / `.csv` (one file). The sink adds +`.parquet` to any other path, so the file would not be where the user said. When the user gives +a path with neither, restate it with a trailing `/` at Node 1d and let them confirm or correct it. + +--- + +## 5. Runtime column constraints + +None known for this host. Do not exclude text columns pre-emptively. If a run fails while loading +a text encoder, report the error and the column; that finding belongs at Node 1c. + +--- + +## 6. Capability skills + +The capability skills were written for, and measured on, Snowflake. Their Kumo calls hold for +every connector. Where a step names a Snowflake registry, stage, session or table, follow it only +when the source or destination is Snowflake. For any other connector, say that the step is not +covered on this host; do not adapt it. + +--- + +## 7. Other deployment modes — out of scope + +Hosted Kumo — `kumo.init(...)`, an API key, `KUMO_API_KEY` / `KUMO_API_ENDPOINT`, Kumo Studio — and +the Snowflake Native App are not this SDK. If the user is on one of them, say so and stop. Never +blend their setup into generated code. diff --git a/skills/kumo-data-science/hosts/generic/runtime.md b/skills/kumo-data-science/hosts/generic/runtime.md new file mode 100644 index 000000000..398fee17c --- /dev/null +++ b/skills/kumo-data-science/hosts/generic/runtime.md @@ -0,0 +1,491 @@ +# Host Runtime — Generic coding agent + +Read at Step 2, when generating the artifact: where it runs, the config cell, the connect step, +credentials, destinations and read-back. Intake is in `adapter.md`. + +--- + +## 0. Where the code runs + +Unknown to you, and it does not matter: the artifact must run unchanged in any Python environment +where `import kumo.engine` works and the source's credentials resolve — a workstation, a VM, a +hosted notebook. So it assumes no platform: + +- no Snowpark session from `get_active_session()`, no `dbutils`, no SPCS token, no notebook + magics, no platform secret store; +- no path outside the ones confirmed at intake, except Kumo's scratch default + (`kumoai-reference/references/kumo_finetuning_sdk.md` §2.1). + +Your own session's access (`adapter.md` §2) says nothing about the runtime's. The connect step +resolves credentials where the code runs. + +Write the artifact to the path the user names, else to the current working directory, named after +the task (`churn_30d_train.ipynb`). Write only that file. Creating it is not running it: Step 3 +still waits for an explicit request. + +--- + +## 1. Config cell + +The first code cell holds the imports and every location as a named constant: the confirmed +source, the confirmed destination, and any population table — each written verbatim. A +population also gets `POPULATION_CONFIRMED`, its name exactly as Node 1d confirmed it, which the +staging step checks before replacing anything (§3.1). + +Values only the runtime knows — a Snowflake warehouse or `connections.toml` entry, a Databricks +SQL warehouse or cluster — are constants here too. S3, GCS and BigQuery have none: the ambient +chain resolves credentials and region where the code runs (an instance or job role, the +environment, `aws configure`, `gcloud`), and BigQuery bills queries to the confirmed dataset's own +project. So add no region, profile, project or billing constant for them beyond the confirmed +location, and pass `region=` or `billing_project_id=` only when the user named one. A BigQuery +dataset's location is not a constant either: the notebook reads it from the dataset (§2). For the +values that are constants, write the user's value if they gave one; otherwise a placeholder of +the form `''`, and list each placeholder in your reply as something to +fill in before running. A placeholder never stands in for a confirmed +value, and never for a secret. + +End the cell with a guard, so a run with an unfilled value stops before reading anything: + +```python +CONFIG = dict(SOURCE_DIR=SOURCE_DIR, OUTPUT_DIR=OUTPUT_DIR) # every constant in this cell +unfilled = [k for k, v in CONFIG.items() if isinstance(v, str) and v.startswith('<')] +assert not unfilled, f'Fill in before running: {unfilled}' +``` + +--- + +## 2. Connect step + +One connector per system, built from the confirmed `source_scope` with ambient credentials +(`kumoai-reference/references/kumo_finetuning_sdk.md` §3, §3.4). Signatures are there; do not +add arguments the user did not confirm. + +```python +# Local files +connector = kumo.FileConnector(root_dir=SOURCE_DIR) +# S3 / GCS — rooted at the confirmed prefix, so table_names() works +connector = kumo.S3Connector(root_dir=SOURCE_URI) # 's3://bucket/prefix/'; no region= +connector = kumo.GCSConnector(root_dir=SOURCE_URI) # 'gs://bucket/prefix/' +# BigQuery — PROJECT_ID and DATASET_ID are the two parts of the confirmed `project.dataset` scope, +# never placeholders. BQ_IDENTITY and BQ_CREDENTIALS are the source's, defined below (ambient by +# default). No billing_project_id (§1). location= is the dataset's own, read by bq_location below. +connector = kumo.BigQueryConnector(project_id=PROJECT_ID, dataset_id=DATASET_ID, + identity=BQ_IDENTITY, + location=bq_location(PROJECT_ID, DATASET_ID, BQ_CREDENTIALS)) +# Databricks — exactly one of warehouse_id / cluster_id +connector = kumo.DatabricksConnector( + catalog=CATALOG, schema_name=SCHEMA, identity=kumo.DatabricksIdentity(), + warehouse_id=DATABRICKS_WAREHOUSE_ID) +# Snowflake — outside Snowflake's own runtime, through a connections.toml entry +connector = kumo.SnowflakeConnector.from_connection_toml( + connection_name=SNOWFLAKE_CONNECTION, warehouse=SNOWFLAKE_WAREHOUSE, + database=DATABASE, schema_name=SCHEMA) # lower case: keep the quotes, §3 + +assert connector.table_names(), 'Connector reached no tables.' +``` + +Emit only the line for the confirmed system. Never emit a rootless S3 or GCS connector for a +source: it cannot list tables. + +**BigQuery location.** `BigQueryConnector` defaults to `location='US'`, and a dataset anywhere +else — `EU`, or a single region such as `us-central1`, which is not `US` — fails: reads with +not-found, and the prediction write only after training, at the sink's location check. Whenever +any connector in the notebook is BigQuery — source, population or destination — define this +and the identities below once, in the connect step before the first connector, and pass +`location=bq_location(, , )` to every +`BigQueryConnector`: + +```python +from google.cloud import bigquery + +def bq_location(project, dataset, credentials=None): + """The location a dataset lives in; the dataset must already exist.""" + return bigquery.Client(project=project, credentials=credentials).get_dataset( + f'{project}.{dataset}').location + +def bq_name(*parts): + """`project`.`dataset`.`table`, or one `column`, for BigQuery SQL. A backtick, backslash or + line break is in no BigQuery name and would end the quoting, so such a part is refused.""" + for part in parts: + if not part or any(c in part for c in '`\\\n\r'): + raise ValueError(f'not a BigQuery name: {part!r}') + return '.'.join(f'`{part}`' for part in parts) +``` + +Build every BigQuery table or column name in SQL with `bq_name`, never by pasting a name into the +query text. + +Each BigQuery identity comes with the Google credentials its direct clients use: the location +lookups, the population query (§3.1) and the read-back (§3). With an explicit `BigQueryIdentity` +(*Credentials* below), build those credentials from the inputs the identity was given, as the SDK +does (`kumo_bigquery.auth`); `None` means ambient. A BigQuery source uses `BQ_IDENTITY` and +`BQ_CREDENTIALS` for its connector, the population staged in it, and their lookups. A BigQuery +destination uses `OUTPUT_BQ_IDENTITY` and `OUTPUT_BQ_CREDENTIALS` for its connector, its lookup +and the read-back: the source's pair unless the user named another identity for the destination, +then built from that identity's inputs in the same forms. With no BigQuery source, define only the +output pair, ambient unless the user named one. + +```python +BQ_IDENTITY = kumo.BigQueryIdentity() # ambient; otherwise replace both with a form below +BQ_CREDENTIALS = None + +# BigQueryIdentity(service_account_key=SERVICE_ACCOUNT_KEY), the key read as Credentials says +import json, os +from google.oauth2 import service_account +SERVICE_ACCOUNT_KEY = json.loads(os.environ[BQ_KEY_VAR]) +BQ_IDENTITY = kumo.BigQueryIdentity(service_account_key=SERVICE_ACCOUNT_KEY) +BQ_CREDENTIALS = service_account.Credentials.from_service_account_info(SERVICE_ACCOUNT_KEY) + +# BigQueryIdentity(workload_service_account_email=SERVICE_ACCOUNT_EMAIL) +import google.auth +from google.auth import impersonated_credentials +SCOPE = 'https://www.googleapis.com/auth/cloud-platform' +BQ_IDENTITY = kumo.BigQueryIdentity(workload_service_account_email=SERVICE_ACCOUNT_EMAIL) +BQ_CREDENTIALS = impersonated_credentials.Credentials( + source_credentials=google.auth.default(scopes=[SCOPE])[0], + target_principal=SERVICE_ACCOUNT_EMAIL, target_scopes=[SCOPE]) + +# A BigQuery destination: the source's identity unless the user named another +OUTPUT_BQ_IDENTITY, OUTPUT_BQ_CREDENTIALS = BQ_IDENTITY, BQ_CREDENTIALS +``` + +Each BigQuery location keeps the project it was confirmed with: `PROJECT_ID` for the source, +`OUTPUT_PROJECT` and `POP_PROJECT` for the destination and the population, equal to `PROJECT_ID` +only when the confirmed names say so. Use that project for its connector, location, staging query +and read-back. Project and credentials are independent: a location in another project still uses +its own identity's credentials as above — the population the source's, the destination +`OUTPUT_BQ_CREDENTIALS`. + +Name file and object-store tables without their suffix — `connector['accounts']` for +`accounts.parquet` — and use those names in the tables, edges and PQL (`adapter.md` §3). + +**Credentials.** The ambient forms above name no secret. Use an explicit identity (§3.4) only +when the user asks for one, and then read each secret at run time from an environment variable +whose **name** is a config-cell constant — never a literal, never a value asked for in chat: + +```python +identity = kumo.S3Identity(access_key_id=os.environ[AWS_KEY_ID_VAR], + secret_access_key=os.environ[AWS_SECRET_VAR]) +``` + +Arrow does not see that identity. Every `pyarrow` read or write of S3 in §3 and §3.1 then takes +the same keys, with the path written without `s3://`: + +```python +import pyarrow.fs +s3 = pyarrow.fs.S3FileSystem(access_key=os.environ[AWS_KEY_ID_VAR], + secret_key=os.environ[AWS_SECRET_VAR]) +written = ds.dataset(OUTPUT_LOCATION.removeprefix('s3://'), format=OUTPUT_FORMAT, + filesystem=s3).count_rows() +``` + +Without an explicit identity, emit `identity = None` and `s3 = None`: §3.1's population staging +then lets Kumo and Arrow resolve the ambient credentials themselves. + +The Snowflake factory `from_spcs_env` and Snowflake's notebook secret store belong to +Snowflake's own runtime. Emit them only when the user says the code runs there. + +### 2.1 Storage + +Kumo's scratch directory defaults to `~/.cache/kumo-engine`. Emit `kumo.configure(scratch_dir=...)` +only for a scratch path the user named. A `FileConnector` source or destination must not overlap +it (§3.4). Scratch is disposable: anything that must outlive the run is written to the confirmed +destination. + +--- + +## 3. Destinations and read-back + +Build the output connector for the destination's own system, and the `PredictionOutputConfig` in +that connector's form (`kumoai-reference/references/kumo_finetuning_sdk.md` §9.2). Then read the +destination back and compare its row count to `prediction_result.num_rows` — the only proof the +write worked. + +**A path destination is written at exactly the confirmed path**, never inside it. Root the output +connector at the destination's **parent** and name the destination's last segment in +`output_paths`, keeping its trailing `/` for a directory: + +```python +# confirmed destination: s3://bucket/prediction_outputs/churn_scores/ +OUTPUT_PARENT, OUTPUT_LEAF = 's3://bucket/prediction_outputs/', 'churn_scores/' +output_connector = kumo.S3Connector(root_dir=OUTPUT_PARENT) # FileConnector / GCSConnector alike +output_config = kumo.PredictionOutputConfig( + connector=output_connector, output_paths={'predictions': OUTPUT_LEAF}) +OUTPUT_LOCATION = OUTPUT_PARENT + OUTPUT_LEAF # the confirmed path, for read-back +``` + +A BigQuery destination's connector is scoped to `OUTPUT_PROJECT` and `OUTPUT_DATASET` and takes +`identity=OUTPUT_BQ_IDENTITY` and +`location=bq_location(OUTPUT_PROJECT, OUTPUT_DATASET, OUTPUT_BQ_CREDENTIALS)` (§2); the location +may differ from the source's. A Databricks destination's connector is scoped to `OUTPUT_CATALOG` +and `OUTPUT_SCHEMA`. + +Rooting the connector at the destination itself and passing a `table_name` writes one level +below it, to a path the user never confirmed. A directory destination must be empty or absent; +the sink creates it. + +`OUTPUT_LEAF` ends in `/`, `.parquet` or `.csv` (`adapter.md` §4). File, S3 and GCS sinks alike +add `.parquet` to any other leaf, and the read-back would then miss the file. The suffix also +sets the format the sink writes, so read back in that format. + +```python +# Local files, S3, GCS — directory or single-file output +import pyarrow.dataset as ds +OUTPUT_FORMAT = 'csv' if OUTPUT_LOCATION.rstrip('/').lower().endswith('.csv') else 'parquet' +written = ds.dataset(OUTPUT_LOCATION, format=OUTPUT_FORMAT).count_rows() # full path or URI + +# BigQuery — as the destination's identity (§2) +from google.cloud import bigquery +bq = bigquery.Client(project=OUTPUT_PROJECT, credentials=OUTPUT_BQ_CREDENTIALS) +count_sql = f'SELECT COUNT(*) FROM {bq_name(OUTPUT_PROJECT, OUTPUT_DATASET, OUTPUT_TABLE)}' +written = next(iter(bq.query(count_sql).result()))[0] + +# Databricks — with the confirmed SQL warehouse; wait_timeout is at most '50s', so poll +import time +from databricks.sdk import WorkspaceClient +from databricks.sdk.service.sql import StatementState + +def q(*parts): + """`catalog`.`schema`.`table`: Unity Catalog names may contain hyphens.""" + return '.'.join('`' + p.replace('`', '``') + '`' for p in parts) + +ws = WorkspaceClient() +r = ws.statement_execution.execute_statement( + warehouse_id=DATABRICKS_WAREHOUSE_ID, wait_timeout='50s', + statement=f'SELECT COUNT(*) FROM {q(OUTPUT_CATALOG, OUTPUT_SCHEMA, OUTPUT_TABLE)}') +while r.status.state in (StatementState.PENDING, StatementState.RUNNING): + time.sleep(5) + r = ws.statement_execution.get_statement(r.statement_id) +assert r.status.state == StatementState.SUCCEEDED, r.status +written = int(r.result.data_array[0][0]) + +# Snowflake — through the same connections.toml entry and warehouse. Kumo keeps the table +# name's case, so quote it (kumo_finetuning_sdk.md §9.1); database and schema as in §2. +import re +import snowflake.connector + +def sf_name(database, schema, table): + """The table Kumo writes, as database.schema."table". Database and schema are written as + the connector was given them (§2): a plain identifier, which Snowflake upper-cases, or a + lower- or mixed-case name already in double quotes ('"my_schema"'), kept as it is. Anything + else is refused. The table is quoted, with any double quote in it doubled.""" + for part in (database, schema): + if not re.fullmatch(r'[A-Za-z_][A-Za-z0-9_$]*|"(?:[^"]|"")+"', part or ''): + raise ValueError(f'not a Snowflake name: {part!r}; confirm it with the user') + return f'{database}.{schema}."' + table.replace('"', '""') + '"' + +with snowflake.connector.connect(connection_name=SNOWFLAKE_CONNECTION, + warehouse=SNOWFLAKE_WAREHOUSE) as conn: + count_sql = f'SELECT COUNT(*) FROM {sf_name(OUTPUT_DATABASE, OUTPUT_SCHEMA, OUTPUT_TABLE)}' + written = conn.cursor().execute(count_sql).fetchone()[0] + +assert written - before == prediction_result.num_rows, (written, before, prediction_result.num_rows) +``` + +`before` is 0 for OVERWRITE. For APPEND (`kumoai-reference/references/outputs.md`) it is the same +count taken before `predict` — 0 when the destination does not exist yet, since the first APPEND +creates it — so the check compares the rows this run added. The delta holds only while nothing +else writes to the destination; when the user says others do, confirm a run-specific destination +instead. + +- `OUTPUT_LOCATION` for files, S3 and GCS is the confirmed destination itself. +- A Databricks destination is passed to Kumo as `'__kumo__
'` and read back as + `..
`, in its own catalog. With a cluster and no SQL warehouse, there is no read-back here: + say so in the artifact and in your reply. +- Report the destination in its own form, unchanged — `DB.SCHEMA.TABLE`, or the path. + +> **Not verified:** none of these read-backs, and no connector other than Snowflake, has been run +> end to end with this skill. Say so when you hand over an artifact for one of them. + +### 3.1 Static scoring population + +When Node 1d confirmed a population table (`kumoai-reference/references/kumo_finetuning_sdk.md` +§7.1), stage it in the **source's** system — where the entity table is read from — whatever the +destination. Each recipe below reads the entity table and writes the population inside that one +system; the destination may be in another system (`adapter.md` §4) and plays no part here. + +- **Files, S3, GCS source:** read the entity key — and, when the entity table has a time column, that + column too — from the source with `pyarrow.dataset`; name them `ENTITY1` and `TIMESTAMP`; write + one Parquet file inside the confirmed population path, creating the directory first for local + files; and build the `PredictionTable` from a connector rooted there. Decide from the entity's + `Table` as you built it: if it has a `time_column`, the population **must** carry it as + `TIMESTAMP`, **cast to a timestamp**. Kumo rejects a population without it, even for a static + query, whatever the §7.1 Snowflake example shows (`ValueError: ... missing TIMESTAMP`), and + reads a `DATE` anchor as null (`Column 'TIMESTAMP' is date32[day]`): a date column is cast. + + ```python + import os + import re + import pyarrow as pa + import pyarrow.compute as pc + import pyarrow.csv + import pyarrow.dataset as ds + import pyarrow.fs + import pyarrow.parquet as papq + + def arrow_fs(uri): + """Arrow's filesystem and path for uri; S3 with an explicit S3Identity takes §2's `s3`.""" + if uri.startswith('s3://') and s3 is not None: # s3 = None under ambient credentials + return s3, uri.removeprefix('s3://') + return pyarrow.fs.FileSystem.from_uri(uri if '://' in uri else os.path.abspath(uri)) + + if POPULATION_PATH != POPULATION_CONFIRMED: # Node 1d approved replacing this path only + raise RuntimeError('POPULATION_PATH changed after Node 1d: confirm the new path first.') + keep = {ENTITY_KEY: 'ENTITY1'} + if ENTITY_TIME: # the entity Table's time_column, else None + keep[ENTITY_TIME] = 'TIMESTAMP' + source_fs, source_path = arrow_fs(ENTITY_SOURCE) + # Arrow parses C-style directives only; chrono's own (%.f, %-m, %:z, %3f, %f) it cannot. + ARROW_FORMAT = (ENTITY_TIME_FORMAT + if ENTITY_TIME_FORMAT and not re.search(r'%[-_0^#.:+\d]|%f', ENTITY_TIME_FORMAT) + else None) + source_format = ENTITY_FORMAT + if ENTITY_FORMAT == 'csv' and ENTITY_TIME and ARROW_FORMAT: # text, parsed with the format below + source_format = ds.CsvFileFormat(convert_options=pyarrow.csv.ConvertOptions( + column_types={ENTITY_TIME: pa.string()})) + NULL_TOKENS = pa.array(pyarrow.csv.ConvertOptions().null_values) # '', 'NA', 'null', ... + scanner = ds.dataset(source_path, format=source_format, + filesystem=source_fs).scanner(columns=list(keep)) + + def anchor_type(t): # a timestamp keeps its unit; a DATE anchor reads as null: cast it + return t if pa.types.is_timestamp(t) else pa.timestamp('us') + + schema = pa.schema([(keep[f.name], anchor_type(f.type) if keep[f.name] == 'TIMESTAMP' else f.type) + for f in scanner.projected_schema]) + + def to_column(col, field): # text in the entity's format reads as the graph reads it + if not (field.name == 'TIMESTAMP' and ARROW_FORMAT + and (pa.types.is_string(col.type) or pa.types.is_large_string(col.type))): + return pc.cast(col, field.type) + unit = field.type.unit + col = pc.if_else(pc.is_in(col, value_set=NULL_TOKENS), pa.scalar(None, col.type), col) + if '%z' not in ARROW_FORMAT: + return pc.strptime(col, format=ARROW_FORMAT, unit=unit) # raises on a misfit + # As the graph does, a value without an offset is read by the format without %z. + out = pc.coalesce( + pc.strptime(col, format=ARROW_FORMAT, unit=unit, error_is_null=True), + pc.strptime(col, format=ARROW_FORMAT.replace('%z', ''), unit=unit, error_is_null=True)) + misfit = pc.and_(pc.is_valid(col), pc.is_null(out)) + if pc.any(misfit).as_py(): + raise pa.ArrowInvalid(f'{pc.filter(col, misfit)[0]} does not fit {ARROW_FORMAT}') + return out + + pop_fs, pop_dir = arrow_fs(POPULATION_PATH) + if isinstance(pop_fs, pyarrow.fs.LocalFileSystem): + pop_fs.create_dir(pop_dir) # object stores need no directory + final = pop_dir.rstrip('/') + '/population.parquet' + with papq.ParquetWriter(final + '.partial', schema, filesystem=pop_fs) as writer: + for batch in scanner.to_batches(): # in batches: bounded memory + writer.write_batch(pa.RecordBatch.from_arrays( + [to_column(batch[f.name], t) for f, t in zip(batch.schema, schema)], + schema=schema)) + pop_fs.move(final + '.partial', final) # replace only once every batch is in + if POPULATION_PATH.startswith('s3://'): # the source's kind, in the §2 forms + population_connector = kumo.S3Connector(root_dir=POPULATION_PATH, + identity=identity) # §2's; None under ambient + elif POPULATION_PATH.startswith('gs://'): + population_connector = kumo.GCSConnector(root_dir=POPULATION_PATH) + else: + population_connector = kumo.FileConnector(root_dir=POPULATION_PATH) + prediction_table = kumo.PredictionTable.from_source_table( + population_connector['population'], pq=pquery, snapshot=snapshot) + ``` + + `ENTITY_SOURCE` is the entity table's file or directory as preflight listed it, and + `ENTITY_FORMAT` its format as preflight found it, `'csv'` or `'parquet'`. `ENTITY_TIME_FORMAT` + is the `timestamp_format` the entity `Table` declares on its time column + (`kumoai-reference/references/graph_construction.md`, *A time column must be a real date*), else + `None`: a CSV reads `03/15/2024` as text, and a plain cast parses only ISO-8601. The format is + Kumo's (chrono) syntax. When Arrow can parse it too, a CSV time column is read as text and + parsed with the format, because ISO-looking text such as `2024-01-02` under `%Y-%d-%m` means + what the format says, as it does in the graph; a cell the CSV reader takes as null (empty, + `NA`, `null`) stays null, and no other column changes. Arrow cannot parse + chrono's own directives (`%.f`, `%-m`, `%:z`); with one of those the column is read and cast as + without a format, which keeps the type the reader inferred: nanoseconds, or a UTC offset as a + time zone. Under `%z`, a value without an offset is read without it, as the graph reads it. + If the parse or the cast fails, report the format and the failing value rather than + guessing another format. The population is + written as Parquet either way. `POPULATION_PATH` is the confirmed path, ending in `/`, in the + source's file system or bucket, and `POPULATION_CONFIRMED` the same path as Node 1d confirmed + it, both in the config cell. Each run replaces its `population.parquet`, as Node 1d says. The + rows go to `population.parquet.partial` first, so a scan or cast that fails leaves the previous + population in place. A source timestamp keeps its own unit: a `timestamp[ns]` with + sub-microsecond values would fail a safe cast to microseconds. +- **Snowflake source:** the §7.1 SQL, run through + `snowflake.connector.connect(connection_name=SNOWFLAKE_CONNECTION, warehouse=SNOWFLAKE_WAREHOUSE)` + as in the read-back above, + and the `PredictionTable` built from a `SnowflakeConnector` made as in §2 but scoped to the + population's database and schema. +- **Databricks source:** create the table through the source's SQL warehouse with the statement + API, wait for it, and build the `PredictionTable` from a `DatabricksConnector` scoped to the + population's catalog and schema, `POP_CATALOG` and `POP_SCHEMA`. A source reached through a cluster has no SQL warehouse: say at Node 1d + that staging is not covered there, and offer the rows whose target is NULL instead. The same `TIMESTAMP` rule applies: project `CAST(
']` tries the +exact name, then its upper-cased form, so `connector['ENTITY_PROFILE']` never finds a lower-case +`entity_profile` (`SourceNotFoundError: Invalid table ENTITY_PROFILE: it has no columns`). Write +table names as `table_names()` returns them and column names as `.columns` returns them — in +`primary_key=`, `time_column=`, column lists and filters. + +**Never exclude a column by a case-sensitive `!=` against a hand-typed literal.** A filter such as +`c.name != 'END_DATE'` silently keeps a lower-case `end_date` — the comparison never matches, the +column is never dropped, and nothing raises. Pass the confirmed `Excluded columns` from preflight +(`kumo-data-science/accept/preflight.md` §8) as a literal `column_names=` list built from the catalog's own spelling +— it has no hand-typed comparison to get wrong. If a filter is used instead, compare +case-insensitively: `c.name.lower() != 'end_date'.lower()`. + +Source: shipped package (`kumo.engine.connector.snowflake`, `.base`, `.source_table`; +`kumo_snowflake.identity` for the lookup order) + +### 3.3 The Snowpark session: take it once, first + +Only where the host's `runtime.md` provides an active Snowpark session. A host without one +follows its `runtime.md` for SQL and read-back instead, and this section and the `session` +calls in §7.1 and §9.1 do not apply. + +Kumo's Snowflake connector opens its own Snowpark session. Once Kumo has read or written +Snowflake, the process holds two sessions, and a later `get_active_session()` raises +`SnowparkSessionException: (1409): More than one active session is detected`. + +```python +from snowflake.snowpark.context import get_active_session + +session = get_active_session() # first code cell, before any connector +``` + +- Call `get_active_session()` **once**, in the first code cell, before building any connector. + Reuse that `session` for every later Snowflake call: the write-back check (§9.1) and any + population table (§7.1). +- Never call `get_active_session()` again later in the notebook, and never open another session + with `Session.builder`. + +Source: shipped package (`kumo_snowflake.session`, which opens its session with +`Session.builder.configs(...).create()`) + +### 3.4 Every other connector + +Constructors and identities, all keyword-only. Source: shipped package +(`kumo.engine.connector.file`, `kumo.engine.connector.s3`, `kumo.engine.connector.gcs`, +`kumo.engine.connector.bigquery`, `kumo.engine.connector.databricks`, +`kumo.engine.connector.identity`): + +``` +FileConnector(*, root_dir) +S3Connector(*, root_dir=None, identity=None, region=None) +GCSConnector(*, root_dir=None, identity=None, requester_pays_project=None) +BigQueryConnector(*, project_id, dataset_id, identity=None, location='US', + billing_project_id=None, staging_uri=None, require_staged=False) +DatabricksConnector(*, catalog, schema_name, identity, warehouse_id=None, cluster_id=None, + staging_volume_path=None, require_staged=False) +DatabricksConnector.from_notebook(*, catalog, schema_name, warehouse_id=None, cluster_id=None, + staging_volume_path=None, require_staged=False) + +S3Identity(*, access_key_id=None, secret_access_key=None, session_token=None, role_arn=None, + partition='aws', connection=None) +GCSIdentity(*, service_account_key=None, workload_service_account_email=None) +BigQueryIdentity(*, service_account_key=None, workload_service_account_email=None) +DatabricksIdentity(*, host=None, client_id=None, client_secret=None) +``` + +**Prefer ambient credentials.** Each connector resolves its provider's own credential chain when +no secret is passed, so the connect step names no secret at all: + +| Connector | Ambient form | Credentials it resolves | Table names | `sql()` | +|---|---|---|---|---| +| `FileConnector` | `FileConnector(root_dir=...)` | none needed | relative paths below `root_dir`, optionally with `.csv` / `.parquet`, or a directory of parts in one format | no | +| `S3Connector` | `identity=None` | AWS chain: instance profile, `aws configure`, environment | relative paths below `root_dir`; with `root_dir=None`, full `s3://` URIs and no `table_names()` | no | +| `GCSConnector` | `identity=None` | Application Default Credentials | as S3, with `gs://` | no | +| `BigQueryConnector` | `identity=None` | Application Default Credentials | tables of `dataset_id` | yes | +| `DatabricksConnector` | `from_notebook(...)`, or `identity=DatabricksIdentity()` | the Databricks SDK's chain: a notebook's context, a CLI profile, environment | tables of `catalog.schema_name` | yes | + +- The explicit identity forms carry secrets (an access key, a service-account key dict, an OAuth + client secret). Use one only when the user asks for it, and read each secret from the + environment's secret store at run time — never a literal, never a value asked for in chat. +- `FileConnector.root_dir` is a filesystem directory, not a URI, and must not overlap the + scratch directory (§2.1). Absolute table paths, URIs, `..` and globs are rejected. A CSV time + column needs `.with_column(col, dtype=Dtype.date, timestamp_format=...)`. +- `region` on `S3Connector` is optional and normally **omitted**: the ambient chain resolves the + bucket's region where the code runs. Pass it only when the user named one, and only with + `root_dir`. It is never a value to ask for, and never a placeholder. +- `DatabricksConnector` takes exactly one of `warehouse_id` and `cluster_id`. +- `require_staged=True` needs `staging_uri` (BigQuery) or `staging_volume_path` (Databricks). +- ADLS has no connector in this package. + +Where each one writes is §9.2. + +--- + +## 4. Tables + +`Table` is **immutable**. `with_column(...)` returns a *new* table — assign or chain it, never +call it for its side effect. + +```python +from kumoapi.typing import Dtype, Stype + +account_profile = kumo.Table.from_source_table( + connector['account_profile'], primary_key='account_id', time_column='created_date', +) + +transaction_events = ( + kumo.Table.from_source_table( + connector['transaction_events'], primary_key='event_id', time_column='event_date', + ) + .with_column('account_id', stype=Stype.ID) + .with_column('device_id', stype=Stype.ID) +) +``` + +Source: shipped package (`kumo.engine.table`): + +``` +Table.from_source_table(source_table, column_names=None, *, primary_key=None, + time_column=None, end_time_column=None) +Table.with_column(name, *, stype=None, dtype=None, timestamp_format=None, + multicategorical_delimiter=None) +Table.infer_metadata() +``` + +- **The source table and column list are positional**; everything else is keyword-only. +- Omit `column_names` to take every column. Pass a list to restrict. +- **Foreign keys infer as numerical.** FK-ness is a graph-level fact, so per-table inference sees + only values. Correct every FK with `.with_column(col, stype=Stype.ID)`. +- **A TEXT time column needs `dtype=Dtype.date`** declared, or staging will not cast it. +- `infer_metadata()` takes no arguments. +- Read-back attributes live under `.definition`: `.definition.pkey`, `.definition.time_col`, + `.definition.end_time_col`. `.table_name` is on the table itself. + +> **Not verified:** `timestamp_format` and `multicategorical_delimiter` are in the signature but +> exercised nowhere. + +Source: shipped package (`kumo.engine.table`) + +--- + +## 5. Graph + +```python +from kumo.engine import Edge + +graph = kumo.Graph( + tables=[account_profile, transaction_events, device_profile], + edges=[ + Edge(src_table='transaction_events', fkey='account_id', dst_table='account_profile'), + Edge(src_table='transaction_events', fkey='device_id', dst_table='device_profile'), + ], +) +graph.validate() +``` + +- **Emit `tables` as a list.** Each table's graph name is then its source table name + (`.table_name`), which is what edges and PQL reference. +- `tables` may also be a mapping `{alias: table}`. The alias then replaces the source name in edges + and PQL; the connector still reads the source name. It is valid in a user's notebook — do not + call it superseded — but emit the list form unless two tables share a source name, which the + list form rejects with `Duplicate graph table name`. +- **`edges` are `Edge` objects** from `kumo.engine`, not dicts or tuples. +- Edges point **foreign key -> primary key**: `src_table.fkey` references `dst_table`'s primary key. +- **`graph.validate()` takes no arguments.** There is no `verbose=`. +- Emit `validate()` explicitly. It is a structural check needing only schemas — no snapshot yet, + so it is cheap and catches edge typos before any data moves. + +Surface includes `.tables`, `.edges`, `.id`, `.native_graph`, `.validate()`, `.visualize()`, `.snapshot()`. +**There is no `add_table`, `remove_table`, `link`, `unlink`, `infer_links`, `get_edge_stats`, or +`get_table_stats`.** + +`Graph.visualize(path=None, *, show_cols=False, show_samples=False, layout='auto')` displays or exports the graph. +Follow `kumo-graph-visualization/SKILL.md` for generated notebook visuals, script exports, and schema inspection during model iteration. + +Source: shipped package (`kumo.engine.graph`) + +--- + +## 6. Predictive query + +```python +pquery = kumo.PredictiveQuery(graph=graph, query=QUERY) +pquery.validate() +print('task type:', pquery.get_task_type()) +``` + +- The graph is bound at construction; parsing is lazy. +- **Emit `validate()` with no arguments.** It raises on an invalid query; otherwise it returns a + `ValidationResponse` and logs its warnings. +- **Always emit `validate()` then `get_task_type()`** before training. The task type is never + written in the query — it is inferred from the target's shape — so print it and confirm it + matches intent. A query that validates can still be the wrong task. +- Print `.query` back so the user can check the generated PQL. + +See `kumoai-reference/references/pql_writing.md` for the query language itself. + +Source: shipped package (`kumo.engine.pquery.predictive_query`) + +--- + +## 7. Snapshot, training and prediction tables + +Materialization is **explicit**. `fit` does not do it for you. + +```python +snapshot = graph.snapshot() + +training_table = pquery.generate_training_table(snapshot) +prediction_table = pquery.generate_prediction_table( + snapshot, prediction_time=PREDICTION_TIME, +) +``` + +Source: shipped package (`kumo.engine.graph`, `kumo.engine.pquery.predictive_query`): + +``` +Graph.snapshot() +PredictiveQuery.generate_training_table(snapshot, plan=None) +PredictiveQuery.generate_prediction_table(snapshot, *, prediction_time=None, plan=None) +``` + +- `snapshot` is **positional** on both. `prediction_time` and `plan` are keyword-only on + `generate_prediction_table`. +- `graph.snapshot()` reads every table through its connector and builds an on-disk DiskGraph index + under the scratch directory. An unchanged valid snapshot is reused on later runs. +- `GraphSnapshot` exposes `.id`, `.index_path`, `.graph_config`. +- `TrainingTable` exposes `.train_table_path` (ordinary parquet), `.manifest_path`, + `.required_stages`, `.generation_plan`. +- `PredictionTable` exposes `.table_path`, `.row_count`, `.timestamps`, `.required_stages`. +- **`prediction_time` must be a NAIVE `datetime` — passing a timezone-aware one raises** + `ValueError: prediction_time must not include a timezone`. Kumo applies UTC itself + (`kumo_core.prediction_options.prediction_timestamps`), so write `datetime(2026, 8, 1)` and + **never** `datetime(2026, 8, 1, tzinfo=timezone.utc)`. Do not import `timezone` for this. +- It must not be earlier than the start of the data's event history (an earlier anchor raises + `Invalid Prediction Anchor Time`; a later date is accepted), and it **cannot be computed from + the connector** — `connector.sql(query)` returns a source table to feed `Table.from_source_table`, + not rows. Hardcode a known in-range value and say why. Omitted, a temporal query anchors at + the latest source time of its target; state that choice if you rely on it. +- `prediction_time` is rejected outright for a **static** query — it is temporal-only. + +Source: shipped package (`kumo.engine.pquery.predictive_query`, +`kumo.engine.pquery.prediction_table`, `kumo_core.prediction_options`, `kumo_core.training.validation`) + +### 7.1 A static query scores the rows whose target is NULL + +In prediction mode a static query keeps only the rows whose target is NULL: Kumo trains on the +labeled rows and scores the unlabeled ones. If every row already has its target, +`generate_prediction_table` raises `RuntimeError: prediction query produced no rows`. + +The scoring population confirmed at the task gate (`kumo-data-science/SKILL.md` Node 1d) decides +the path. The staging example below is Snowflake's; on any other connector the host's `runtime.md` +says how a population is staged, or that it is not covered there: + +- **Rows whose target is NULL** → `pquery.generate_prediction_table(snapshot)`, no + `prediction_time`. +- **Rows that already have a target** (for example "every order") → stage their keys as + `ENTITY1` in the confirmed population table and pass it explicitly: + +```python +def q(name): # a quoted Snowflake identifier + return '"' + name.replace('"', '""') + '"' + +if f'{POP_DATABASE}.{POP_SCHEMA}.{POP_TABLE}' != POPULATION_CONFIRMED: # approved at Node 1d + raise RuntimeError('The population table changed after Node 1d: confirm the new one first.') +session.sql( # the session from §3.3 + f'CREATE OR REPLACE TABLE {POP_DATABASE}.{POP_SCHEMA}.{POP_TABLE} AS ' + f'SELECT {q(ENTITY_KEY)} AS ENTITY1 FROM {SOURCE_DATABASE}.{SOURCE_SCHEMA}.{q(ENTITY_TABLE)}' +).collect() +population_connector = kumo.SnowflakeConnector.from_spcs_env( # or from_connection_toml, §3 + warehouse=WAREHOUSE, database=POP_DATABASE, schema_name=POP_SCHEMA, +) +prediction_table = kumo.PredictionTable.from_source_table( + population_connector[POP_TABLE], pq=pquery, snapshot=snapshot, +) +``` + +- `CREATE OR REPLACE` replaces an existing table; Node 1d names that before it is confirmed, + and the approval covers that one name. The config cell carries it verbatim as + `POPULATION_CONFIRMED`, and every staging recipe stops before writing when the name it would + replace differs — an edit after confirmation needs a new confirmation, not a silent replace. +- Stage the population only where Node 1d confirmed it — never in the source schema. Write + `POP_TABLE` in upper case, so the unquoted `CREATE` and the connector lookup agree. +- Never change the query, the entity or the target to get rows. An empty population is a fact + about the data to report, not a query to rewrite. +- The population table must carry `ENTITY1` and, when the task samples against seed times (a + temporal query, or a static one on a timestamped entity), a `TIMESTAMP` column; it must not + contain a target column. The SQL above stages only `ENTITY1`, which suffices for a static query + on an entity without a timestamp; otherwise also select the anchor time as `TIMESTAMP`. +- Rows the model trained on are scored too. Say so: those predictions are not out-of-sample. + +Source: shipped package (`kumo_core.prediction.generate_prediction_table`, +`kumo.engine.pquery.prediction_table.PredictionTable.from_source_table`) + +--- + +## 8. Model plan and training + +```python +from kumoapi.model_plan import RunMode + +model_plan = pquery.suggest_model_plan(snapshot, run_mode=RunMode.FAST) + +# Smoke-test sizing — say so, and remove for a real run. +model_plan.training_job.num_experiments = 1 +model_plan.training_job.refit_trainval = False +model_plan.training_job.refit_full = False +model_plan.optimization.max_epochs = 3 + +trainer = kumo.Trainer(model_plan) +training_result = trainer.fit(training_table, snapshot=snapshot) +print(training_result.metrics) +``` + +Source: shipped package (`kumo.engine.pquery.predictive_query`, `kumo.engine.trainer.trainer`): + +``` +PredictiveQuery.suggest_model_plan(snapshot, *, run_mode=RunMode.FAST, train_table_spec=None) +Trainer(model_plan) +Trainer.fit(train_table, *, snapshot, export_config=None, warm_start=None) +``` + +- **`suggest_model_plan` is implemented.** Prefer it over hand-building a plan: it inspects the + materialized graph and query. `snapshot` is positional; `run_mode` is keyword-only. +- **`fit` requires `snapshot=` as a keyword.** `fit(training_table)` alone is a `TypeError`. This + is the single easiest signature to get wrong. +- Mutate the returned plan in place for sizing. Always tell the user when numbers are smoke-test + values rather than a real run. +- `TrainingResult` exposes `.metrics` (a `Dict[str, float]` **attribute**, not a call), + `.artifact_paths`, `.bundle_dir`, `.published_model`. +- `export_config=ModelOutputConfig(...)` publishes the bundle during `fit`. +- Success is a blocking call returning without raising. **There is no job status enum — never + branch on one.** + +`Trainer` also exposes `.model_plan` (settable), `.training_table_generation_plan`, `.scratch_dir`, +and reloading: `Trainer.load(bundle_path, embedding_models=None)` and +`Trainer.load_from_connector(model_registry, *, path=None, location=None, +embedding_models=None)`. + +Source: shipped package (`kumo.engine.trainer.trainer`) + +--- + +## 9. Prediction and write-back + +```python +from kumoapi.jobs import PredictionArtifactType, WriteMode + +output_connector = kumo.SnowflakeConnector.from_spcs_env( + warehouse=WAREHOUSE, database=OUTPUT_DATABASE, schema_name=OUTPUT_SCHEMA, +) +output_config = kumo.PredictionOutputConfig( + connector=output_connector, + artifact_type=PredictionArtifactType.PREDICTIONS, + table_name=output_table, + write_mode=WriteMode.OVERWRITE, +) + +prediction_result = trainer.predict( + prediction_table, snapshot=snapshot, output_config=output_config, +) +print('rows:', prediction_result.num_rows) +``` + +Source: shipped package (`kumo.engine.trainer.trainer`, `kumo.engine.trainer.output_config`): + +``` +Trainer.predict(prediction_table, *, snapshot, output_config=None, + num_classes_to_return=None, num_workers=1, + binary_classification_threshold=None) + +PredictionOutputConfig(connector=, artifact_type=, table_name=, write_mode=, extra_fields=, + output_paths=None) +PredictionResult.export(output_config) +``` + +- `write_mode`: `OVERWRITE` replaces an existing table at the destination. Use the mode Node 1d + confirmed: `OVERWRITE` for a new table or an approved replacement, `APPEND` when the user chose + it (`kumoai-reference/references/outputs.md`). +- For Snowflake, always pass `artifact_type` and `table_name`. `output_paths` is for file and + object-store connectors only; with a `SnowflakeConnector` it raises `ValueError`. Every + connector's form is in §9.2. +- `extra_fields=[MetadataField.JOB_TIMESTAMP, MetadataField.ANCHOR_TIMESTAMP]` (`kumoapi.jobs`) + adds those two columns: when the run started, and the time each prediction is anchored at. + `ANCHOR_TIMESTAMP` exists only for a query with a time window; requesting it for one without + raises `ValueError` after scoring. Source: shipped package (`kumo_core.train.output`, + `kumo_core.prediction_options`). + +- **`snapshot=` is required.** `output_config` is optional — omit it to keep results local. +- `predict` requires a trainer holding fitted state — from `fit` on that same `Trainer`, or from + `Trainer.load` / `Trainer.load_from_connector`; there is no job registry to look a model up in. +- `num_classes_to_return` is the number of results returned per entity, and it is independent + of the query's `RANK TOP K`: a `RANK TOP 10` model can return 50 results. Link prediction + requires it whenever predictions are written, as an integer from 1 to 500 + (`kumo_core.prediction_options.MAX_LINKS_TO_PREDICT`); for more than 500 per entity, export + embeddings instead. Multiclass and multilabel accept it optionally and fall back to the + query's `RANK TOP K` when it has one. Regression and binary classification reject it. +- Binary classification: without `binary_classification_threshold` (and none recorded with the + model), `predict` returns `True_PROB` and leaves `TARGET_PRED` null; it has no 0.5 default of + its own. Always pass the one confirmed at Node 1d (§12), written or local alike. +- `binary_classification_threshold` and `num_classes_to_return` are arguments of `predict`, and + only of `predict`. `PredictionOutputConfig` describes where to write, not how to predict: it + takes neither, and passing one is a `TypeError`. Whatever the destination, the config carries + only the fields in its signature above. +- `PredictionResult` exposes `.num_rows`, `.files`, `.frame`, `.selection`, `.artifact_type`. + Prefer `.num_rows` and `.files` — `.frame` collects every row into pandas. +- `prediction_result.export(output_config)` writes an existing result to a destination. Passing + `output_config=` to `predict` does the same write in one call; that is the form to emit. + +### 9.1 The output table keeps its name + +**The table is written with exactly the name you pass.** No suffix, no case change, no quoting +needed on the write. Verify by reading it back and comparing the row count. Quote the table name +in the read: `session.table` upper-cases an unquoted part, so `churn_scores` would look for +`CHURN_SCORES` and not find the table Kumo wrote. The database and schema stay unquoted; both +sides resolve those the same way. + +```python +def quoted(name): + return '"' + name.replace('"', '""') + '"' + +destination = session.table([OUTPUT_DATABASE, OUTPUT_SCHEMA, quoted(output_table)]) +assert destination.count() - before == prediction_result.num_rows +``` + +`before` is 0 for `OVERWRITE`. For `APPEND` it is the same count taken before `predict` — 0 when +the table does not exist yet, since the first APPEND creates it — so the check compares the rows +this run added. The delta holds only while nothing else writes to the table +(`kumoai-reference/references/outputs.md`). + +Scope the output connector to the destination database and schema; `table_name` is a bare name. +Always read the table back — it is the only proof the write-back worked. `session` is the one +taken in the first cell (§3.3); calling `get_active_session()` here, after Kumo has written, +raises. + +Source: shipped package (`kumo.engine.trainer`) + +### 9.2 Destinations by connector + +The output connector may differ from the source connector. Source: shipped package +(`kumo.engine.trainer.output_config`, `kumo.engine.connector.file`, `kumo_core.object_store_sink`, +`kumo_databricks.sink`, `kumo_databricks.output`): + +| Connector | `PredictionOutputConfig` form | Rules | +|---|---|---| +| `SnowflakeConnector` | `artifact_type=` + `table_name=` (bare name) | connector scoped to the destination database and schema; §9.1 | +| `FileConnector`, `S3Connector`, `GCSConnector` | `output_paths={'predictions': ..., 'embeddings': ...}` — the keys select the artifacts — or `artifact_type=` + `table_name=` | do not combine the two forms; a path ending in `/` is a directory and must be empty; otherwise one file, written in the format its `.parquet` or `.csv` suffix names, and `.parquet` is added when it has neither — by all three connectors; paths are relative to a rooted connector, full URIs for a rootless one | +| `BigQueryConnector` | `artifact_type=` + `table_name=` (unqualified table ID in its dataset) | predictions and embeddings only — not a model `export_config` and not a `Trainer.load_from_connector` source | +| `DatabricksConnector` | `artifact_type=` + `table_name=`: a bare name, or `'__kumo__
'` | a bare name lands in the connector's schema; a qualified name must name the connector's own schema, or it is refused; writes stage through the volume `.._kumo_staging`, created on first use | + +`output_paths` with a non-file connector raises `ValueError`. Read every destination back after +the write, as the host's `runtime.md` says (§9.1 for Snowflake). + +--- + +## 10. Complete verified example + +On Snowflake, in a host whose `runtime.md` provides an active Snowpark session (§3.3). On any +other host — Snowflake reached from outside it included — drop the `get_active_session()` line and +replace steps 1, 7 and 8 with that host `runtime.md`'s connect step, the §9.2 destination and its +SQL read-back; nothing else changes. + +Self-contained: `OUTPUT_DATABASE`, `OUTPUT_SCHEMA` and `OUTPUT_TABLE` are placeholders for the +confirmed output destination (§12). + +```python +from datetime import datetime +from pathlib import Path + +import kumo.engine as kumo +from kumo.engine import Edge +from kumoapi.jobs import PredictionArtifactType, WriteMode +from kumoapi.model_plan import RunMode +from kumoapi.typing import Stype +from snowflake.snowpark.context import get_active_session + +OUTPUT_DATABASE, OUTPUT_SCHEMA, OUTPUT_TABLE = '', '', '
' # confirmed destination + +session = get_active_session() # §3.3: only where the host provides the session + +# 1. Scratch directory, then connect (inside SPCS; elsewhere use the §3 order). No credentials in the notebook. +kumo.configure(scratch_dir=Path('/tmp/kumo-run') / 'scratch') +connector = kumo.SnowflakeConnector.from_spcs_env( + warehouse='WH_S', database='SYNTHETIC_DATASETS', schema_name='"my_schema"', +) +assert connector.table_names(), 'Connector reached no tables.' + +# 2. Tables — correct every foreign key to Stype.ID. +account_profile = kumo.Table.from_source_table( + connector['account_profile'], primary_key='account_id', time_column='created_date', +) +transaction_events = ( + kumo.Table.from_source_table( + connector['transaction_events'], primary_key='event_id', time_column='event_date', + ) + .with_column('account_id', stype=Stype.ID) + .with_column('location_id', stype=Stype.ID) +) +location_profile = kumo.Table.from_source_table( + connector['location_profile'], primary_key='location_id', +) + +# 3. Graph — list of tables, Edge objects, explicit validate. +graph = kumo.Graph( + tables=[account_profile, transaction_events, location_profile], + edges=[ + Edge(src_table='transaction_events', fkey='account_id', dst_table='account_profile'), + Edge(src_table='transaction_events', fkey='location_id', dst_table='location_profile'), + ], +) +graph.validate() + +# 4. Query — validate, then confirm the inferred task type. +pquery = kumo.PredictiveQuery( + graph=graph, + query=('PREDICT LIST_DISTINCT(transaction_events.location_id, 0, 30, days) ' + 'RANK TOP 10 FOR EACH account_profile.account_id'), +) +pquery.validate() +print('task type:', pquery.get_task_type()) + +# 5. Materialize. PREDICTION_TIME is NAIVE — a tz-aware datetime raises ValueError. +PREDICTION_TIME = datetime(2026, 8, 1) +snapshot = graph.snapshot() +training_table = pquery.generate_training_table(snapshot) +prediction_table = pquery.generate_prediction_table( + snapshot, prediction_time=PREDICTION_TIME, # NAIVE datetime, inside the event history +) + +# 6. Train — snapshot= is required. +model_plan = pquery.suggest_model_plan(snapshot, run_mode=RunMode.FAST) +model_plan.optimization.max_epochs = 3 # smoke-test sizing +trainer = kumo.Trainer(model_plan) +training_result = trainer.fit(training_table, snapshot=snapshot) +print(training_result.metrics) + +# 7. Predict and write back — snapshot= is required here too. +output_connector = kumo.SnowflakeConnector.from_spcs_env( + warehouse='WH_S', database=OUTPUT_DATABASE, schema_name=OUTPUT_SCHEMA, +) +prediction_result = trainer.predict( + prediction_table, + snapshot=snapshot, + output_config=kumo.PredictionOutputConfig( + connector=output_connector, + artifact_type=PredictionArtifactType.PREDICTIONS, + table_name=OUTPUT_TABLE, + write_mode=WriteMode.OVERWRITE, + ), +) +print('rows:', prediction_result.num_rows) + +# 8. Verify the write-back. The table name is quoted (§9.1). +destination = session.table([OUTPUT_DATABASE, OUTPUT_SCHEMA, '"' + OUTPUT_TABLE.replace('"', '""') + '"']) +assert destination.count() == prediction_result.num_rows +``` + +Source: shipped package (`kumo.engine`, `kumoapi`, `kumo_snowflake.spcs`) + +--- + +## 11. Never emit + +### 11.1 The old SDK — recognise and refuse + +These were the **previous** Kumo SDK. They are gone from the current package; each one is an +`AttributeError` or `TypeError`. If a user's existing notebook contains them, say plainly that it +targets a superseded build and needs rewriting against §10 — do not copy its shape. + +| Old | Now | +|---|---| +| `kumo.set_backend(...)`, `kumo.LocalBackend(...)`, `kumo.get_backend()` | nothing — there is no backend | +| `kumo.Storage(...)`, `kumo.FilesystemArtifactLocation(...)` | `kumo.configure(scratch_dir=...)` (§2.1) | +| `kumo.create_identity(CreateIdentityArgs(...))`, `kumo.test_identity_credentials(...)` | `SnowflakeConnector.from_spcs_env()`, else `from_connection_toml()`, else `SnowflakeIdentity` from secrets (§3) | +| `SnowflakeConnector(name=..., account=..., credentials={...})` | keyword-only ctor, no `name`/`account` (§3.1) | +| `kumo.Table(source_table=..., primary_key=...)` | `Table.from_source_table(...)` (§4) | +| `table['COL'].stype = kumo.Stype.ID` | `.with_column('COL', stype=Stype.ID)` (§4) | +| `kumo.Stype`, `kumo.Dtype` | `from kumoapi.typing import Dtype, Stype` | +| `kumo.Graph(..., edges=[{...}])` — edges as dicts | `Edge` objects (§5). A `tables={alias: t}` mapping is still valid | +| `trainer.fit(graph, pquery=...)` | `fit(train_table, snapshot=snapshot)` (§8) | +| `OutputConfig` from `kumoai.artifact_export` | `kumo.PredictionOutputConfig` (§9) | +| `.metrics()` as a call, `.job_id`, `.summary()` | `.metrics` attribute, `.num_rows` (§8, §9) | +| output table renamed to `"_predictions"` | the name is kept verbatim (§9.1) | + +### 11.2 Hosted Kumo — out of scope entirely + +- `kumo.init(...)` in any form (`url=`, `api_key=`, `snowflake_credentials=`, + `snowflake_application=`), API keys, `KUMO_API_KEY` / `KUMO_API_ENDPOINT`, Kumo Studio. +- `training_job.tracking_url` or any Kumo UI reference. +- `kumo.Session(...)`, `kumo.ComputeType`, RBAC groups, `project_id`. +- The Snowflake Native App and Marketplace deployment. +- `kumoai.codegen`, `DistillationTrainer`, online serving. + +### 11.3 Also never + +- **Any environment-setup step.** Kumo is already present (§2); begin at `import kumo.engine`. +- Branching on a Kumo job status string or enum — Kumo has none. A Snowflake ML Job's status + is Snowflake's, read from `snowflake.ml.jobs`, and only the host's job file uses it. +- `connector['X'].head(...)` — it does not exist. `connector.sql(query)` exists but returns a + custom-SQL source table, never rows; do not use it to read data (Source: shipped package + (`kumo.engine.connector.source_table`)). +- **`import kumoai`.** That is the hosted SDK's name, and this SDK's name before the rename. Where + it resolves at all, it is the wrong package or a pre-rename build; import `kumo.engine`. + +--- + +## 12. Parameters to inject from the validated request + +| Code location | Source | +|---|---| +| The connect step | The host's `runtime.md`, from the confirmed `source_scope` (§3, §3.4) | +| `TARGET = dict(warehouse=, database=, schema_name=)` (Snowflake) | Data location from the intake contract; a lower-case name keeps its double quotes (§3) | +| `from_connection_toml(connection_name=)` (Snowflake) | The user's `[connections.]` entry; ask | +| `SnowflakeIdentity(account=, username=)`, secret names (Snowflake) | Account identifier, key-pair user, and the names of the attached key and passphrase secrets; ask only when neither factory applies, never for a secret's value | +| `configure(scratch_dir=)` | A local path; `/tmp/` is fine | +| `SnowflakeConnector(role=)` (Snowflake) | Role from the intake contract | +| `connector['TABLE']` names, column names | Exactly as `table_names()` / `.columns` return them (§3.2) | +| Column selection, `stype` / `dtype` | Data structure from the intake contract | +| `Edge(src_table=, fkey=, dst_table=)` | Relationships from the intake contract | +| PQL `query=` | Generated per `kumoai-reference/references/pql_writing.md` | +| `prediction_time=` | A known in-range **naive** `datetime`; **cannot be computed** | +| `predict(num_classes_to_return=N)` | Results per entity the user asked for; defaults to `RANK TOP K` but may differ. Link prediction writing predictions requires it, 1–500; no cap otherwise (§9) | +| `predict(binary_classification_threshold=)` | Always passed for binary classification, written or local; never on `PredictionOutputConfig`. The threshold restated at Node 1d: the user's, else `0.5 [default]` (`kumo-data-science/accept/intake_contract.md`) | +| Population table (static task, rows that already have a target) | Confirmed at Node 1d (§7.1) | +| `PredictionOutputConfig(...)` + output connector | Output destination in its connector's form (§9.2); report it back unchanged — `DB.SCHEMA.TABLE`, or the path | + +--- + +## 13. Pre-return checklist + +- [ ] Generated code begins at `import kumo.engine as kumo`; `import kumoai` appears nowhere + (§11.3). +- [ ] `kumo.init(...)` appears nowhere; no API keys; no `tracking_url`. +- [ ] No call from §11.1 appears anywhere. +- [ ] `Stype` / `Dtype` imported from `kumoapi.typing`; `Column` / `Edge` from `kumo.engine`; + `PredictionArtifactType` / `WriteMode` from `kumoapi.jobs`; `RunMode` from + `kumoapi.model_plan`. +- [ ] The connect step is the host `runtime.md`'s, with no credential literal (§3, §3.4). +- [ ] Snowflake: the factory the host's `runtime.md` names (§3), with explicit `warehouse`, + `database`, `schema_name`; `SPCS_TOKEN_PATH` imported, never hardcoded. +- [ ] Snowflake: lower-case or mixed-case `database` / `schema_name` passed with their double + quotes (§3). +- [ ] Table and column names in the catalog's exact spelling (§3.2). +- [ ] Snowflake, on a host that provides a Snowpark session: `session = get_active_session()` + once, in the first code cell; never again, no `Session.builder` (§3.3). Elsewhere, the + host `runtime.md`'s connection for SQL and read-back. +- [ ] `connector.table_names()` asserted before building tables. +- [ ] Every foreign key corrected with `.with_column(col, stype=Stype.ID)`. +- [ ] `with_column(...)` results assigned or chained — it returns a new Table. +- [ ] `Graph(tables=[...], edges=[Edge(...)])` — list form; `graph.validate()` emitted. +- [ ] `pquery.validate()` and `get_task_type()` run before training, both called with no arguments. +- [ ] `graph.snapshot()` before generating any table. +- [ ] **`trainer.fit(training_table, snapshot=snapshot)`** — `snapshot=` present. +- [ ] **`trainer.predict(prediction_table, snapshot=snapshot, ...)`** — `snapshot=` present. +- [ ] `binary_classification_threshold` / `num_classes_to_return` on `predict`, never on + `PredictionOutputConfig` (§9). +- [ ] `prediction_time` hardcoded and **naive** — no `tzinfo`, no `timezone.utc` — with the reason stated. +- [ ] Static query: generated population only for rows whose target is NULL; otherwise the + population table confirmed at Node 1d (§7.1). +- [ ] No code cell writes anywhere unless a destination was confirmed; an example of how to add + one later is markdown text, not a code cell. +- [ ] Smoke-test sizing called out as such. +- [ ] `write_mode` and any population replacement as Node 1d confirmed them (§9, §7.1); the + staging cell stops when the population name differs from `POPULATION_CONFIRMED`. +- [ ] Destination in its connector's form (§9.2), reported with its exact name — no + `_predictions` suffix. +- [ ] Written destination read back and row count compared to `prediction_result.num_rows` — + for APPEND, the count added since before `predict` (§9.1). +- [ ] Success judged by blocking calls returning without raising — not by a status value. + +--- + +## 14. Known gaps + +**Not verified:** + +- `Table`'s `timestamp_format` and `multicategorical_delimiter` — in the signature, exercised + nowhere. +- `train_table_spec=` on `suggest_model_plan`; `plan=` on `generate_training_table` / + `generate_prediction_table`. +- `export_config=` on `fit`, and `ModelOutputConfig`'s `path` vs `location`. +- `Trainer.load` / `load_from_connector` round-trips — implemented in the package, exercised + nowhere. `load_from_connector` needs a connector whose sink implements `ModelFetchingSink` + (otherwise `TypeError`); `FileConnector` provides `fetch_model`, other sinks are unverified. +- `num_workers` above 1. +- `PredictionResult.export` against Snowflake — in the package, exercised nowhere. +- Whether a snapshot can be reused across graphs or persisted between processes. +- The written column schema for a ranking task. +- `FileConnector`, `S3Connector`, `GCSConnector`, `BigQueryConnector`, `DatabricksConnector`: + §3.4 and §9.2 are read from the package source; none has been run end to end with this skill. + +**Known absent** — do not reach for these: `SourceTable.head(...)`, +`graph.infer_links()`, `graph.get_edge_stats()`, `graph.get_table_stats()`, `table.validate()`, +`table.metadata`, `table.get_stats()`, `graph.validate(verbose=)`, any job-status enum. diff --git a/skills/kumoai-reference/references/outputs.md b/skills/kumoai-reference/references/outputs.md new file mode 100644 index 000000000..04583d8d1 --- /dev/null +++ b/skills/kumoai-reference/references/outputs.md @@ -0,0 +1,140 @@ +# Predictions, embeddings and destinations + +`Trainer.predict` selects one artifact per call. Without `output_config`, it produces local +prediction Parquet files. `num_rows` is the selected output row count; `.frame` collects all +files into pandas and should be used only for bounded results. Local files require explicit +retention to survive runtime teardown. + +Inspect schema without collecting all rows: + +```python +import pyarrow.parquet as parquet + +for filename in predictions.files: + print(filename, parquet.ParquetFile(filename).schema_arrow) +``` + +Binary/regression results have scalar output columns. Multiclass/multilabel and rankings +use long class rows. Forecasts use expanded timestamp rows. Embeddings include vector-valued +EMBEDDING columns and entity metadata; link embeddings can include ID, TYPE and NAME for +self-links. Preserve the actual schema and task rather than assuming one row per entity. + +## Write predictions + +Inputs: `output_connector` scoped to the confirmed destination database/schema, exact bare +`OUTPUT_TABLE`, explicit `WRITE_MODE`, and task-specific `PREDICT_OPTIONS`. WriteMode has +APPEND and OVERWRITE. Select the mode deliberately: File export OVERWRITE replaces existing data while retaining the table and grants when its schema matches, and fails on Snowflake when it does not; on BigQuery it removes the table's row access policies (below); +APPEND can duplicate outputs on retry and never changes the existing table's schema (below). Check destination +existence and access first. + +```python +import warnings +import kumo.engine as kumo +from kumoapi.jobs import MetadataField, PredictionArtifactType, WriteMode + +# APPEND only; JOB_TIMESTAMP alone for a query with no time window. +RUN_COLUMNS = [MetadataField.JOB_TIMESTAMP, MetadataField.ANCHOR_TIMESTAMP] +output = kumo.PredictionOutputConfig( + connector=output_connector, + artifact_type=PredictionArtifactType.PREDICTIONS, + table_name=OUTPUT_TABLE, write_mode=WRITE_MODE, + extra_fields=RUN_COLUMNS if WRITE_MODE == WriteMode.APPEND else [], +) +with warnings.catch_warnings(): + if WRITE_MODE == WriteMode.APPEND: # no model id, no write: see below + assert trainer.result.id, 'No model id: APPEND would write rows without BUNDLE_ID' + warnings.filterwarnings('error', message='Predicting without a prediction manifest') + written = trainer.predict( + population, snapshot=snapshot, output_config=output, **PREDICT_OPTIONS, + ) +print(written.num_rows, written.files) +``` + +**APPEND rows say which run wrote them.** An APPEND write to a table adds `PREDICTION_ID` (this +prediction run) and `BUNDLE_ID` (the model that scored it), whatever `extra_fields` says (Source: +shipped package (`kumo.engine.trainer.output_config.PredictionOutputConfig`)). The `extra_fields` +above add `JOB_TIMESTAMP` (when the run started) and `ANCHOR_TIMESTAMP` (the time its predictions +are for), so the table's history can be ordered and filtered by run, and a retried run shows as a second +`PREDICTION_ID` and `JOB_TIMESTAMP`. `BUNDLE_ID` is the model id the run's prediction manifest +names. When there is none — `trainer.result.id` is `None` because the id could not be computed at +fit, or `predict` warns `Predicting without a prediction manifest` — the writer leaves the +`BUNDLE_ID` column out of the rows; `PREDICTION_ID` is still set and the row count still matches. +An existing table with the column gets nulls in it. A table the run creates has no `BUNDLE_ID` +column at all, so every later APPEND that names a model fails on BigQuery and Databricks, or loses +the id on Snowflake (below). The snippet above checks the model id before scoring and turns the +warning into an error, so APPEND writes neither; report it as a failed run. A query with no time +window has no anchor: pass `JOB_TIMESTAMP` alone, or the write fails after scoring. + +The write never adds a column to an existing table. When an APPEND destination already exists, +read its columns before generating the artifact and check every column the run emits: +`PREDICTION_ID`, `BUNDLE_ID`, `JOB_TIMESTAMP`, and `ANCHOR_TIMESTAMP` when the query has a time +window. What a missing one does depends on the destination. Snowflake loads by column name and +drops a column the table lacks, silently: the rows land without it, and a row-count check still +passes. Databricks and BigQuery fail the write, after scoring. Either way, say so at the gate and +let the user pick: add the missing columns (`ALTER TABLE … ADD COLUMN`; earlier rows hold nulls), +a new table the first run creates, or OVERWRITE. Adding columns is a write, confirmed like any +other. Column types: + +| Destination | `PREDICTION_ID`, `BUNDLE_ID` | `JOB_TIMESTAMP`, `ANCHOR_TIMESTAMP` | +|---|---|---| +| Snowflake | `VARCHAR` | `TIMESTAMP_NTZ` | +| Databricks | `STRING` | `TIMESTAMP_NTZ` | +| BigQuery | `STRING` | `TIMESTAMP` | + +The timestamps take the type the destination's sink writes: on BigQuery a `DATETIME` column fails +the APPEND (Source: shipped package (`kumo_bigquery.write`, `kumo_databricks.write`)). + +OVERWRITE onto an existing Snowflake table keeps the table and its grants, so the run's columns +must be exactly the table's: an OVERWRITE run writes no run columns, so a table that earlier +APPEND runs gave them fails, after scoring. Offer OVERWRITE there only when the columns match; +otherwise offer a new table, or dropping the existing one first (a write, confirmed like any +other). Databricks OVERWRITE replaces the table, columns included. + +BigQuery OVERWRITE replaces the table's rows and schema with one `WRITE_TRUNCATE` job +(`kumo_bigquery.write`). It keeps column descriptions and policy tags, but BigQuery drops the +table's row access policies with it, so every reader of the table then sees every prediction row. Before offering OVERWRITE onto an existing BigQuery table, list its policies +(`bq ls --row_access_policies :.
`, or ask the user). When it has any, say +so at the gate and offer a new table, with the access controls it needs, instead. + +The exact table name is used; no suffix is appended and its case is kept. Read back, with the +session taken in the first cell (`kumoai-reference/references/kumo_finetuning_sdk.md` §3.3), using +the quoted table name of §9.1 there, inspect schema and bounded values. For a new/replaced table compare its count +with `written.num_rows`. For APPEND compare the count delta against a recorded pre-write count +only without concurrent writers, or use a run-specific destination for unambiguous verification. +Check entity/anchor/class coverage too. A count alone does not prove semantic correctness. + +## Requested embeddings + +Predictions and embeddings are separate calls and destinations. For a fitted or loaded trainer, +using the same compatible snapshot and population: + +```python +embedding_output = kumo.PredictionOutputConfig( + connector=output_connector, + artifact_type=PredictionArtifactType.EMBEDDINGS, + table_name=EMBEDDING_TABLE, write_mode=WRITE_MODE, +) +embeddings = trainer.predict( + population, snapshot=snapshot, output_config=embedding_output, **PREDICT_OPTIONS, +) +print(embeddings.num_rows, embeddings.files) +``` + +Preserve task-specific arguments, including the explicit binary threshold where the API +requires it. Do not promise embeddings for every module/task: report an unsupported selection +error without silently switching models or refitting. Link models can expose entity and item +vectors; inspect TYPE and identifiers before calling them customer/product embeddings. + +## Serialization boundaries + +On Snowflake, the public prediction path exports selected Parquet through Snowflake's file sink, +with schema inference and COPY. This differs from the separate in-memory DataFrame writer, which rejects +list/dict columns. Do not apply that rejection indiscriminately to prediction export, and do +not send nested cohort JSON through the scalar DataFrame path unchanged. + +For vectors or other nested fields, inspect the destination's actual column type and a +bounded read-back value; inference is not a guarantee of the consumer's expected representation. +Use [stage retention](../../kumo-model-persistence/references/retention.md) to preserve the original +Parquet or JSON when a destination cannot represent the required shape. Conversion/flattening +requires an explicit schema decision. Do not claim that all artifact families fit an ordinary +table unchanged or automatically rerun prediction after an export failure. diff --git a/skills/kumoai-reference/references/pql_writing.md b/skills/kumoai-reference/references/pql_writing.md new file mode 100644 index 000000000..e2100560d --- /dev/null +++ b/skills/kumoai-reference/references/pql_writing.md @@ -0,0 +1,502 @@ +# PQL — Predictive Query Language + +Constructs marked ``Source: shipped package (``)`` are verified against the Kumo package source and are ground truth. The rest of this page is the PQL surface as previously compiled; it is **not** re-verifiable against the published Kumo documentation, which is stale for this version and is not a source here — do not cite it, fetch it, or reconcile against it. Anything explicitly not verified is marked with a `>` note. + +**Never emit a construct that is not on this page.** If the user's request needs one that is absent, say so rather than reaching for the docs or memory. + +PQL is the query language; the surrounding Python is in `kumoai-reference/references/graph_construction.md` and `kumoai-reference/references/kumo_finetuning_sdk.md`. + +**Never invent PQL syntax.** If the user's request cannot be expressed with the constructs on this page, say so and ask for a reformulation. + +**Scope rule:** KumoRFM is a different product with a different validator (`FOR customers.customer_id=42`, `RANK TOP K` capped at 20, no joined static filters). Do not mix it in. Platform/fine-tuning PQL uses `FOR EACH`. + +--- + +## Grammar + +```text +PREDICT +FOR EACH . +[WHERE ] +[ASSUMING ] +``` + +- `PREDICT` and `FOR EACH` are **required**. `WHERE` and `ASSUMING` are optional. +- `FOR EACH` accepts **only a primary key column**. The entity table must be directly connected to the target table by a PK→FK relationship. +- `RANK TOP ` / `CLASSIFY` are suffixes on the target expression, inside the `PREDICT` clause. + +> `RANK TOP ` has been seen both on its own line between `PREDICT` and `FOR EACH` and inline at the end of the `PREDICT` line. Prefer the inline form. + +### Target forms + +```text +PREDICT
. -- static / column target +PREDICT (
., , [, ]) -- temporal aggregation target +PREDICT -- conditional -> classification +PREDICT LIST_DISTINCT(
., , ) RANK TOP -- ranking / link prediction +PREDICT LIST_DISTINCT(
., , ) CLASSIFY -- per-class binary classification +``` + +--- + +## Aggregation functions + +Valid in the target and in temporal filters. + +| Function | Applies to | +|---|---| +| `SUM` | numerical columns | +| `AVG` | numerical columns | +| `MAX` | numerical, timestamp | +| `MIN` | numerical, timestamp | +| `COUNT` | all columns (use `TABLE.*` to count rows regardless of N/A) | +| `COUNT_DISTINCT` | all columns | +| `LIST_DISTINCT` | all columns; recommended only if fewer than ~1000 possible values | +| `LAST` | all columns **except foreign keys** | +| `FIRST` | all columns **except foreign keys** | + +Forecasting targets support **only** `SUM`, `COUNT`, `COUNT_DISTINCT` (`AVG` is undefined when an entity has no fact rows in a timeframe). + +### Null / empty-window behavior (critical, differs per function) + +- `SUM`, `COUNT`, `COUNT_DISTINCT`: if no fact rows exist in the window, the training label is **0**. +- `AVG`, `MIN`, `MAX`, `FIRST`, `LAST`: if no fact rows exist, the training example is **dropped**. This implicitly assumes every predicted entity has at least one fact row in the horizon. Use `IS NULL` / `IS NOT NULL` if you need to predict periods with no interactions. + +This is the difference between "will spend $0" (`SUM`/`COUNT`, keeps inactive entities) and "what will they buy next" (`FIRST`/`LAST`, silently drops inactive entities). + +--- + +## Time window / horizon + +The horizon is expressed **inside the aggregation function**, not as a separate clause. There is no `WITHIN` keyword. + +```text +(
., , , ) +``` + +- `start` is **exclusive**, `end` is **inclusive**. +- Both must be **non-negative integers**; `end` must be strictly greater than `start`. +- In the **target**, the window is forward-looking (positive). In a **temporal filter**, use negative offsets for the past. +- `unit` defaults to `days`. Valid: `minutes`, `hours`, `days`, `months` (uppercase also accepted). **Not** accepted: `seconds`, `weeks`, `years`, or any singular form (`day`, `week`). + +Worked example: predicting on `2020-01-01 00:00:00` with `SUM(TRANSACTIONS.PRICE, 10, 30)` aggregates rows with timestamp `t` where `2020-01-11 00:00:00 < t <= 2020-01-31 00:00:00`. + +```text +PREDICT COUNT(PURCHASES.*, 0, 48, hours) > 20 +PREDICT SUM(Purchases.amount, 0, 6, months) > 500 +``` + +Guidance: the fact table should hold history at least an order of magnitude longer than the prediction horizon (e.g. 1 year of rows for a 30-day horizon). At least 3 anchor times are needed for a healthy train/val/test split, so very large horizons can starve the splits. + +--- + +## Comparison operators + +Used to turn a target into a conditional (classification) target, and in filters. + +| Operator | Applies to | +|---|---| +| `>` `<` `>=` `<=` | numeric, timestamp | +| `=` `!=` | all column types | +| `STARTS WITH` | text | +| `ENDS WITH` | text | +| `CONTAINS` / `NOT CONTAINS` | text | +| `LIKE` / `NOT LIKE` | text, `%` wildcard | +| `IN` | e.g. `LOAN.STATUS IN ('A', 'C')` | +| `IS NULL` / `IS NOT NULL` | all column types (`= NULL` / `!= NULL` also shown) | + +Logical combinators: `AND`, `OR`, `NOT`. Each operand must itself be a filter operation. Supported in all filter fields and in targets. + +```text +LAST(LOAN.STATUS, 0, 30) CONTAINS 'SUCCESS' AND LAST(LOAN.AMOUNT, 0, 30) > 30 +NOT LAST(LOAN.AMOUNT, 0, 30) > 30 +ARTICLES.DESCRIPTION NOT CONTAINS 'polyester' +LAST(LOAN.STATUS, 0, 30) LIKE '%SUCCESS' +``` + +--- + +## Filters + +### `WHERE` — entity / temporal filter + +Restricts which entities (and therefore which training examples) are included. + +```text +PREDICT SUM(transactions.price, 0, 30) +FOR EACH customers.customer_id +WHERE COUNT(transactions.*, -30, 0) > 0 +``` + +```text +PREDICT fraud_reports.is_fraud +FOR EACH transaction.id +WHERE transaction.type = "bank transfer" +``` + +Temporal filters use negative offsets (look-back). Static filters compare a column to a constant. + +### Inline `WHERE` inside an aggregation + +Restricts the rows that the aggregation counts, rather than the entities. + +```text +PREDICT LIST_DISTINCT( + transactions.article_id + WHERE transactions.price > 100, +0, 30, days) +RANK TOP 25 +FOR EACH customers.customer_id +``` + +Note the syntax: the inline `WHERE` goes **after the column reference and before the `start, end, unit` arguments**. It may also be written on one line: + +```text +COUNT_DISTINCT(Ratings.rating WHERE Ratings.rating < 3, 0, 60, days) +``` + +Source: shipped package (`kumopql.grammar`) + +**Inline vs trailing `WHERE` is the same keyword doing two different jobs** — inline selects rows for the aggregation, trailing selects eligible entities. A single query may use both; see "Both `WHERE`s in one query" under the worked examples. + +### `ASSUMING` — hypothetical / future condition + +Followed by one or more future-looking temporal aggregations assumed true at prediction time. + +```text +PREDICT SUM(TRANSACTIONS.PRICE, 0, 30) +FOR EACH CUSTOMERS.CUSTOMER_ID +ASSUMING COUNT(NOTIFICATIONS.*, 0, 7) > 2 +``` + +Also usable as a way to drop zero-heavy training rows in forecasting: + +```text +PREDICT SUM(Sales.quantity, START_DAYS, END_DAYS) +FOR EACH Groups.group_id +ASSUMING SUM(Sales.quantity, START_DAYS, END_DAYS) > 0 +``` + +`ASSUMING` is not a causal estimate. Do not present a difference between assumed and unassumed predictions as proof of causation. + +--- + +## Date and time literals in filters (gotcha) + +Timestamp constants are written as **bare ISO 8601, unquoted** — no quotes, no `DATE '...'` wrapper, no `TIMESTAMP` cast. Hours/minutes/seconds may be included and default to `0`. + +```text +PREDICT COUNT(transactions.*, 0, 10) +FOR EACH customers.customer_id +WHERE customers.date_joined < 2022-01-01 12:45:30 +``` + +```text +PREDICT CUSTOMERS.industry = 'IT' AND CUSTOMERS.date_of_birth <= 1990-01-01 +``` + +String literals in filters appear both single-quoted (`'bank transfer'`, `'IT'`) and double-quoted (`"bank transfer"`). + +--- + +## Task type is inferred from query shape + +Kumo determines the task type automatically. `pquery.get_task_type()` returns it — always call it to confirm intent. + +| Query shape | Task type | +|---|---| +| `PREDICT
.` | Regression (static) | +| `PREDICT SUM/AVG/MIN/MAX/COUNT(...)` with no comparison | Regression (temporal) | +| Any target with a conditional operator (`=`, `>`, `<`, …) | Binary classification | +| `PREDICT FIRST/LAST(, …)` over a high-cardinality categorical | Multiclass classification | +| `LIST_DISTINCT` over a **non-foreign-key** categorical column | Multilabel classification | +| `LIST_DISTINCT` over a **foreign key** column (+ `RANK TOP K`) | Link prediction / ranking | +| `LIST_DISTINCT(...) CLASSIFY` | Separate binary classification per class | +| Temporal regression meant as a forecast | Regression: `get_task_type()` reads the query alone, so it does not change when the generation plan sets `forecast_length`; that plan is what makes it a forecast (*Forecasting* below) | + +Static vs temporal: a query without a time window is **static** (default 80/10/10 random split, no time column required). A query aggregating over a window is **temporal** (time-based splits, anchor times sampled back through history). + +--- + +## Worked examples by task type + +### Binary classification — churn + +```text +PREDICT COUNT(transactions.*, 0, 90, days) = 0 +FOR EACH customers.customer_id +WHERE COUNT(transactions.*, -60, 0, days) > 0 +``` + +Predicts whether a customer active in the last 60 days makes zero transactions in the next 90. `=` makes it classification; swapping to `>` changes which class is positive. Output is the probability of the positive class. + +### Binary classification — will a *qualifying* event occur (temporal, filtered) + +Use this shape for "will entity X have at least one event of a particular kind in the next N days" — fraud, conversion, unsubscribe, churn-by-event. The condition that defines *which* rows count goes **inside** the aggregation. + +```text +PREDICT COUNT(transactions.* WHERE transactions.is_fraud = 1, 0, 30, days) > 0 +FOR EACH accounts.account_id +``` + +When the label is already a numeric 0/1 flag, summing it is equivalent: + +```text +PREDICT SUM(fraud_reports.LABEL, 0, 30, days) > 0 +FOR EACH accounts.account_id +``` + +**Do not** express the event condition as an entity-level `WHERE`. These mean different things and only the first is correct here: + +| Query | Meaning | +|---|---| +| `PREDICT COUNT(t.* WHERE t.is_fraud = 1, 0, 30, days) > 0` | will this account have a **fraudulent** transaction | +| `PREDICT COUNT(t.*, 0, 30, days) > 0 ... WHERE t.is_fraud = 1` | will this account have **any** transaction, training only on accounts that already had fraud | + +The second leaks the label into entity selection and predicts the wrong thing. If the target condition describes the *future event*, it belongs inline; if it describes *which entities are eligible*, it belongs in the trailing `WHERE`. + +Also do not reach for the static form (`PREDICT fraud_reports.is_fraud`) when the request names a time window — static queries have no horizon and cannot answer "in the next 30 days". + +#### Both `WHERE`s in one query — the canonical example + +Source: shipped package (`kumopql.grammar`, `kumo.engine.pquery.predictive_query`) + +This query parses under the PQL grammar (`PredictiveQuery.validate()` parses and validates the query string against the graph) and uses **both** jobs of the `WHERE` keyword at once. Copy this shape when a request has a row-level target condition *and* an entity-level cohort condition. + +```text +PREDICT COUNT_DISTINCT(Ratings.rating WHERE Ratings.rating < 3, 0, 60, days) > 0 +FOR EACH Movies.MovieID +WHERE AVG(Ratings.rating, -60, 0, days) > 3 +``` + +Read it clause by clause: + +| Clause | Job | +|---|---| +| `WHERE Ratings.rating < 3` (inline, inside `COUNT_DISTINCT`) | selects **which rows** the aggregation counts — only low ratings | +| `0, 60, days` | forward target window | +| `> 0` | makes it binary classification: at least one low rating occurs | +| `WHERE AVG(Ratings.rating, -60, 0, days) > 3` (trailing) | selects **which entities** are eligible — only movies well rated over the past 60 days | + +Same keyword, two different jobs, in one query: "will a currently well-rated movie start getting low ratings?". Moving the inline condition to the trailing `WHERE` would instead train only on movies that already had low ratings and predict whether they get *any* rating — a different and wrong question. + +Note also the offsets: the target window is non-negative (`0, 60`), the entity filter looks back (`-60, 0`). + +### Binary classification — static / non-temporal + +```text +PREDICT fraud_reports.is_fraud +FOR EACH transaction.id +WHERE transaction.type = "bank transfer" +``` + +```text +PREDICT customer_item_pairs.likes = 1 +FOR EACH customer_item_pairs.uuid +``` + +Rows where the label is NULL are dropped from training and become the prediction set. + +### Multiclass classification — next purchase category + +```text +PREDICT FIRST(purchase.type, 0, 7) +FOR EACH user.user_id +``` + +`FIRST` is undefined when a user made no purchase in the window, so those examples are dropped — every prediction assumes the user buys something. + +### Regression — static + +```text +PREDICT customers.age +FOR EACH customers.customer_id +``` + +### Regression — temporal (LTV) + +```text +PREDICT SUM(transactions.price, 0, 30) +FOR EACH customers.customer_id +WHERE COUNT(transactions.*, -30, 0) > 0 +``` + +### Temporal link prediction / recommendation + +```text +PREDICT LIST_DISTINCT(transactions.article_id, 0, 30, days) +RANK TOP 25 +FOR EACH customers.customer_id +``` + +```text +PREDICT LIST_DISTINCT(transactions.article_id, 0, 7) RANK TOP 10 +FOR EACH customers.customer_id +``` + +With an inline filter on candidate items: + +```text +PREDICT LIST_DISTINCT( + transactions.article_id + WHERE transactions.price > 100, +0, 30, days) +RANK TOP 25 +FOR EACH customers.customer_id +``` + +Output is K rows per entity by default; `num_classes_to_return` at prediction time can return a different count (1–500 for link prediction; `kumoai-reference/references/kumo_finetuning_sdk.md` §9). Ranking tasks have a limit of 10 million target items. `RANK` on a foreign key is the only case where new targets may be added at batch-prediction time (by adding rows to the item table). + +### Static link prediction — item-to-item similarity + +```text +PREDICT LIST_DISTINCT(copurchase.dst_item_id) RANK TOP 10 +FOR EACH src_item.item_id +``` + +`LIST_DISTINCT` with **no window arguments** → static link prediction. + +### Multilabel via CLASSIFY + +```text +PREDICT LIST_DISTINCT(TRANSACTIONS.ARTICLE_ID, 0, 30) CLASSIFY +FOR EACH CUSTOMERS.CUSTOMER_ID +``` + +Produces a separate binary classification per article ID. Without `TOP K`, classes are generated for all available classes up to the limit. + +### Forecasting + +Forecasting is **not a distinct PQL keyword**. Write a temporal regression query, set `forecast_length` on the `TrainingTableGenerationPlan` passed to `generate_training_table(snapshot, plan=...)`, and pass the fitted trainer's `trainer.training_table_generation_plan` to `generate_prediction_table(snapshot, plan=...)` — both take a `TrainingTableGenerationPlan`, not a `PredictionTableGenerationPlan`, and never the model plan (Source: shipped package (`kumoapi.model_plan`, `kumo.engine.pquery.predictive_query`)); `forecast_length=3` on a 7-day window forecasts the next three 7-day windows. The flow is in `kumoai-reference/references/task_journeys.md` *Forecasting*. + +```text +PREDICT SUM(TRANSACTIONS.AMOUNT, 0, 7, days) +FOR EACH CUSTOMERS.CUSTOMER_ID +``` + +```text +PREDICT COUNT(Sales.*, 0, 28) +FOR EACH Items.item_id +``` + +```text +-- 6th week out +PREDICT SUM(Sales.quantity, 35, 42) +FOR EACH Groups.group_id +``` + +Targets restricted to `SUM`, `COUNT`, `COUNT_DISTINCT`. `forecast_type` (`mlp` default, `implicit`, `autoregressive`) is a model-plan option. Treat forecasting as **experimental**. + +--- + +## Evaluation metrics per task type + +| Task type | Metrics Kumo reports | +|---|---| +| Binary classification | Accuracy, AUROC, AUPRC, confusion matrix, gain chart | +| Multiclass classification | Accuracy | +| Multilabel classification | AUPRC, AUROC, AP (macro / micro / per-label); also F1, Hamming score | +| Link prediction | F1@K, MAP@K, Precision@K, Recall@K for K = 1, 10, 100 | +| Regression | MAE, MSE, RMSE, MAPE, SMAPE, prediction-distribution histogram | + +Additional selectable metrics (model-plan `metrics` field): binary adds `ap`, `f1`, `ndcg[@k]`, `precision[@k]`, `recall[@k]`; multiclass adds `f1`, `precision`, `recall`; link prediction adds `mrr@k`, `hit_ratio@k`, `coverage@k`, `avg_popularity@k`, `personalization@k`, `diversity[col]@k`; forecasting adds `neg_binomial`, `normal`, `lognormal`. + +**Defaults:** the SDK's default model plan sets `metrics` and `tune_metric` per task type, for +every run mode: + +| Task type | Reported metrics | Tuned on | +|---|---|---| +| Binary classification | `auroc`, `ap`, `auprc` | `auroc` | +| Multiclass classification | `acc` | `acc` | +| Multilabel classification | `ap`, `auroc`, `auprc`, each `_macro` / `_micro` / `_per_label` | `ap_macro` | +| Regression, forecasting | `mae`, `mse`, `rmse` | `mae` | +| Ranking, link prediction | `map`, `precision`, `recall`, `f1`, each `@1` / `@10` / `@100` | `map@10` | +| Link prediction with a training-table weight column | `weighted_ndcg`, `weighted_recall`, `precision`, each `@1` / `@10` / `@100` | `weighted_ndcg@10` | + +Source: shipped package (`kumo_core.model_plan`) + +--- + +## Gotchas and malformed-query traps + +1. **No `WITHIN` / `ON` / `TOP_K` / `NEXT_ITEM` keywords.** The horizon lives in the aggregation arguments; the entity clause is `FOR EACH`; ranking is `RANK TOP K`. +2. **Do not mix static conditions and temporal aggregations in one target.** Temporal aggregations depend on anchor time, static conditions do not; they need different sampling strategies. Documented as invalid practice. +3. **`start` must be non-negative and `end` strictly greater than `start`** in the target. Negative offsets only in temporal filters. +4. **Unit names must be plural and from the allowed set.** `day`, `week`, `weeks`, `years`, `seconds` are rejected. +5. **Date literals are unquoted ISO 8601.** See the date/time section. +6. **`FOR EACH` requires a primary key.** If the entity table has duplicate rows, pre-build a `SELECT DISTINCT` view first. +7. **FK direction matters.** The entity table must connect to the target table via PK→FK. In the fraud example, `transaction` holds the FK to `fraud_reports`, not the reverse — otherwise multiple reports could link to one transaction. +8. **`LAST`/`FIRST` cannot be applied to foreign key columns.** Use `LIST_DISTINCT` + `RANK` for FK targets. +9. **`LIST_DISTINCT` over a non-FK column silently becomes multilabel classification, not ranking.** Check `get_task_type()`. +10. **`LIST_DISTINCT` beyond ~1000 distinct values is not recommended.** +11. **Horizon vs. split health:** too large a window leaves fewer than 3 anchor times; too small a window on a long history generates too many examples (mitigate with `train_start_offset` on the `TrainingTableGenerationPlan` passed to `generate_training_table(snapshot, plan=...)`; the model plan has no such field) and, for link prediction, too few positives. +12. **Always validate before generating tables:** `pquery.validate()` for syntax, `pquery.get_task_type()` for semantics. Required — see "Required pre-flight" under SDK usage. +13. **Inline `WHERE` vs trailing `WHERE` are not interchangeable.** A row-level condition on the predicted event goes inside the aggregation; an entity-eligibility condition goes in the trailing `WHERE`. Swapping them produces a query that validates and trains on the wrong question. + +--- + +## SDK usage + +`PredictiveQuery(graph=graph, query=)` binds the graph at construction. Call `validate()` and `get_task_type()` with no arguments. + +```python +import kumo.engine as kumo + +pquery = kumo.PredictiveQuery( + graph=graph, + query=( + "PREDICT MAX(transaction.Quantity, 0, 30, days)\n" + "FOR EACH customer.CustomerID\n" + "ASSUMING SUM(transaction.UnitPrice, 0, 7, days) > 15" + ), +) + +pquery.validate() +print(pquery.get_task_type()) +``` + +Build multi-line query strings with a triple-quoted string and `.strip()`. Explicit `\n` separators also work. + +```python +QUERY = """ +PREDICT COUNT_DISTINCT(Ratings.rating WHERE Ratings.rating < 3, 0, 60, days) > 0 +FOR EACH Movies.MovieID +WHERE AVG(Ratings.rating, -60, 0, days) > 3 +""".strip() +``` + +Table names used in PQL are the **source table names** — `Graph(tables=[...])` has no keys. See `kumoai-reference/references/graph_construction.md`. + +Source: shipped package (`kumo.engine.pquery.predictive_query`) + +### Required pre-flight, before any training step + +Source: shipped package (`kumo.engine.pquery.predictive_query`) + +Always emit these two calls, in this order, immediately after constructing the `PredictiveQuery` and before generating training tables or fitting: + +```python +pquery.validate() # syntax + schema; call with no arguments +print('task type:', pquery.get_task_type()) +``` + +`validate()` catches syntax and graph-mismatch errors; `get_task_type()` is the semantic check: the task type is never written in the query, only inferred from its shape, so print it and confirm it matches the user's intent before spending a training run. A query that validates can still be the wrong task (see the `LIST_DISTINCT` multilabel-vs-ranking trap below). + +> There is no `KumoFineTuning` class. Do not emit `kumo.init()` — hosted-edition only, out of scope. + +> `RANK BY` appears in one stray example elsewhere but is defined nowhere and contradicts the `LIST_DISTINCT(...) RANK TOP K` form used here. Treat `RANK BY` as unverified; **do not emit it.** + +--- + +## Mapping a validated request to PQL + +| Validated field | PQL element | +|---|---| +| Prediction target | `PREDICT` expression — pick the aggregation from the target column type | +| Primary entity | `FOR EACH .` | +| Prediction horizon | `start`, `end`, `unit` arguments inside the aggregation | +| Eligibility / cohort constraint (which **entities**) | trailing `WHERE` (temporal look-back or static column filter) | +| Condition on the predicted event (which **rows** count) | inline `WHERE` inside the aggregation, before the window arguments | +| Hypothetical future condition | `ASSUMING` | +| Number of recommendations | `RANK TOP ` | +| Task type | Not written explicitly — inferred from query shape; confirm with `get_task_type()` | diff --git a/skills/kumoai-reference/references/task_journeys.md b/skills/kumoai-reference/references/task_journeys.md new file mode 100644 index 000000000..5d888d736 --- /dev/null +++ b/skills/kumoai-reference/references/task_journeys.md @@ -0,0 +1,111 @@ +# Complete a task journey + +Start with a confirmed graph, query, source scope, population, evaluation protocol and workload. +The examples below share the [SDK setup](kumo_finetuning_sdk.md). Symbols in uppercase are +user-selected inputs; table/column names in PQL are illustrative and must match the actual +graph. Each journey validates the query, generates populations, suggests a model plan, fits, +predicts and inspects output. Do not execute every example as one notebook. + +## Classification and regression + +Choose one query from [PQL forms](pql_writing.md), then set `EXPECTED_TASK` to its confirmed +public `TaskType`, imported from `kumoapi.task`. `IS_TEMPORAL` comes from the confirmed query; +`SCORING_TIME` is the requested naive datetime in UTC. `PREDICT_OPTIONS` contains only supported +options for that task, as listed below. + +```python +import kumo.engine as kumo +from kumoapi.model_plan import RunMode + +pq = kumo.PredictiveQuery(graph=graph, query=QUERY) +pq.validate() +if pq.get_task_type() != EXPECTED_TASK: + raise ValueError("Resolved query task differs from confirmed intent") +snapshot = graph.snapshot() +training_table = pq.generate_training_table(snapshot) +plan = pq.suggest_model_plan(snapshot, run_mode=RunMode.FAST) +# Apply only previously requested workload/configuration choices here. +trainer = kumo.Trainer(plan) +training_result = trainer.fit(training_table, snapshot=snapshot) +anchors = {"prediction_time": SCORING_TIME} if IS_TEMPORAL else {} +population = pq.generate_prediction_table(snapshot, **anchors) +predictions = trainer.predict(population, snapshot=snapshot, **PREDICT_OPTIONS) +print(training_result.metrics) +print(predictions.num_rows, predictions.files) +``` + +| Journey | Task confirmation, scoring options and interpretation | +| --- | --- | +| Static or temporal binary | Confirm BINARY_CLASSIFICATION and positive condition. Supply `binary_classification_threshold=THRESHOLD` in [0,1]. `True_PROB`/`False_PROB` are probabilities and `TARGET_PRED` is the thresholded label. Compare AUROC/AP and threshold metrics only when recorded | +| Static or temporal multiclass | Confirm MULTICLASS_CLASSIFICATION and class meaning. No binary threshold. Optional `num_classes_to_return` limits returned classes only when requested. Long rows carry CLASS, SCORE and PREDICTED with entity metadata. Compare compatible class/averaging metrics and per-class cohorts when present | +| Static or temporal multilabel | Confirm MULTILABEL_CLASSIFICATION (or explicitly intended MULTILABEL_RANKING) and label set. No explicit binary threshold for multilabel prediction output: the API rejects it. Long CLASS/SCORE/PREDICTED rows represent separate label outcomes; do not interpret scores as exclusive class probabilities | +| Static or temporal regression | Confirm REGRESSION and units. No threshold or top-class count. Inspect TARGET_PRED, MAE/RMSE when recorded, and matching temporal/split semantics | + +Static tasks omit prediction_time entirely, and a static query scores only rows whose target is +NULL; to score rows that already have one, use the population table confirmed at the task gate +(`kumoai-reference/references/kumo_finetuning_sdk.md` §7.1). Supplied exact populations use +[custom populations](../../kumo-custom-populations/SKILL.md) instead of generated populations; +never attach evaluation labels to a scoring table. For generated static tasks, verify which +rows are eligible before claiming all entities were scored. + +## Links and ranking + +Use the same journey with a confirmed FK target, for example +`PREDICT LIST_DISTINCT(orders.product_id, 0, 30, days) RANK TOP 10 FOR EACH customers.customer_id`. +For static links use `LIST_DISTINCT(pairs.product_id)` without a time window and omit the +scoring timestamp. Confirm TEMPORAL_LINK_PREDICTION or STATIC_LINK_PREDICTION respectively. +Set `PREDICT_OPTIONS = {"num_classes_to_return": N}`, where N is the confirmed number of +results per entity: the query's `RANK TOP K` unless the user asked for a different count +(1–500; `kumoai-reference/references/kumo_finetuning_sdk.md` §9). No binary threshold. +Preserve the item universe, positive interaction, eligibility and evaluation sampling. Inspect +entity/CLASS and available SCORE/time columns. Returned rows can be fewer than N when +candidates are unavailable. Compare recorded precision/MAP/recall at the same K. + +For a non-FK category ranking, confirm the distinct MULTILABEL_RANKING task rather than treating +it as a link to a product table. Do not create a graph edge solely to force a task type. + +## Forecasting + +Confirm interval, number of steps, target units, required history and evaluation horizon. +Use SUM, COUNT or COUNT_DISTINCT over that interval. The initial query can resolve as +REGRESSION before generation settings promote it to forecasting; use the generation plan in this workflow, not a `ModelPlan.training_table_generation` field. + +This example forecasts `FORECAST_STEPS` consecutive windows for the confirmed temporal query: + +```python +from kumoapi.model_plan import TrainingTableGenerationPlan, RunMode +from kumoapi.task import TaskType + +pq = kumo.PredictiveQuery(graph=graph, query=FORECAST_QUERY) +pq.validate() +if pq.get_task_type() not in (TaskType.REGRESSION, TaskType.FORECASTING): + raise ValueError("Forecast query must have a supported numerical temporal target") +snapshot = graph.snapshot() +generation = TrainingTableGenerationPlan(forecast_length=FORECAST_STEPS) +training_table = pq.generate_training_table(snapshot, plan=generation) +plan = pq.suggest_model_plan(snapshot, run_mode=RunMode.FAST) +trainer = kumo.Trainer(plan) +training_result = trainer.fit(training_table, snapshot=snapshot) +population = pq.generate_prediction_table( + snapshot, prediction_time=SCORING_TIME, + plan=trainer.training_table_generation_plan, +) +predictions = trainer.predict(population, snapshot=snapshot) +print(trainer.training_table_generation_plan) +print(training_result.metrics, predictions.num_rows, predictions.files) +``` + +Inspect effective generation metadata to confirm the intended forecast length, lags and +intervals. Prediction must reuse the fitted generation settings. Outputs expand into one row +per entity/forecast timestamp with TARGET_PRED and ANCHOR_TIMESTAMP; configured quantiles can +add q_* columns. Compare horizon-aligned errors and temporal splits, not one-window regression +scores as if they established all horizons. Column analysis is not supported for forecasting. + +## Finish each journey + +Use [outputs](outputs.md) for requested write-back and a separate embedding call. +Inspect selected Parquet schemas before interpreting columns; do not hardcode optional fields. +Use [diagnostics](../../kumo-training-diagnostics/SKILL.md) for actual metrics and trial histories, +and [explanations](../../kumo-model-explanations/SKILL.md) for available attribution/cohorts. +Missing explanations do not imply training failed. Retain requested artifacts via +[persistence](../../kumo-model-persistence/SKILL.md) before notebook teardown. diff --git a/skills/kumoai-reference/references/troubleshooting_signals.md b/skills/kumoai-reference/references/troubleshooting_signals.md new file mode 100644 index 000000000..a57911a67 --- /dev/null +++ b/skills/kumoai-reference/references/troubleshooting_signals.md @@ -0,0 +1,306 @@ +# Troubleshooting Signals + +Observable symptom → likely cause → fix. Ordered by when the agent hits them: table definition, graph construction, query validation, then job time. + +This file is the signal set as previously compiled. It is **not** re-verifiable against the published Kumo documentation, which is stale for this version of the local SDK and is not a source here — do not cite it, fetch it, or reconcile against it. + +> No verbatim Kumo exception text is established for this SDK. Entries below are keyed on the **observable condition**, not on a quoted message. **Do not fabricate Kumo error strings.** +> The five messages quoted in §1a are the exceptions: they are the SDK's own text, read from its source. + +**Who applies a fix.** A fix that changes the data — casting a column, deduplicating, flattening, +filling or filtering values, combining columns into a key, normalising key types — is the user's +to make in their source. The agent reports the symptom, its cause in the data, and that fix, and +asks; it never applies it in the generated code, and never declares a `dtype` to make a data +problem pass (router *What this skill does NOT do*). Choosing a different primary key, time column +or link, or leaving a column out, is a graph change: propose it and let the user confirm it. + +--- + +## 0. The validation calls — run these, do not guess + +These exist in the current package. **Call every one with no arguments.** + +| Call | Checks | +|---|---| +| `table.infer_metadata()` | Re-resolves `dtype` / `stype` from the source | +| `graph.validate()` | Graph structure, before any snapshot. Cheap — schemas only | +| `pquery.validate()` | Query syntax and its fit against the graph | +| `pquery.get_task_type()` | The inferred task type — confirms the ML problem matches intent | + +Standard order: build tables -> `graph.validate()` -> `pquery.validate()` -> +`get_task_type()` -> `graph.snapshot()` -> generate tables -> train. + +**These do not exist.** Do not emit them, and do not tell the user to run them: +`table.validate()`, `table.metadata`, `table.get_stats()`, `graph.infer_links()`, +`graph.get_edge_stats()`, `graph.get_table_stats()`, `connector.sql(...)`, +`SourceTable.head(...)`. + +Inspect a table through `.definition` (`.pkey`, `.time_col`, `.end_time_col`) and a graph through +`.tables` / `.edges`. To look at data, read the parquet that `snapshot()` and +`generate_training_table()` leave under the scratch directory. + +--- + +## 1. Table definition + +### Symptom: `connector.table("X")` raises `ValueError` +**Cause:** The table name does not exist in the connector. +**Fix:** List what is actually there before constructing tables: `connector.table_names()`, or guard with `connector.has_table(name)`. Snowflake object names, for example, are typically uppercase (`connector.table("ORDERS")`). + +### Symptom: a column's `stype` is unset, or the query fails because a column has no semantic type +**Cause:** `Table.from_source_table()` and the explicit `Table(...)` constructor do **not** fully populate `dtype`/`stype`. A partially specified table is legal until it is used in a predictive query. +**Fix:** Call `.infer_metadata()`, or set the type explicitly: +```python +table.column("col").stype = "categorical" +``` +Fully specify tables in production code to avoid unexpected re-inference. + +### Symptom: "invalid data type error" / type-mismatch recommendation on a column +**Cause:** The semantic type (`Type`) does not align with the data type (`Data Type`). +**Fix:** Align them. Valid `Stype` values: `numerical`, `categorical`, `multicategorical`, `ID`, `text`, `timestamp`, `sequence`, `image`, `unsupported`. + +### Symptom: a column holds URLs, lat/long, IP addresses, or phone numbers +**Cause:** These types are **not supported** for preprocessing. +**Fix (the user's, upstream):** derive domain/path, geographic area, subnet, or area code in the source and use that as a categorical column. The agent can leave the column out of the graph meanwhile, and says so. + +### Symptom: a column holds nested schemas, arrays, or maps +**Cause:** Not supported. +**Fix (the user's, upstream):** flatten to a delimited string in the source. `["TV", "electronics", "promotion"]` → `"TV, electronics, promotion"` (multicategorical). + +### Symptom: timestamps arrive as strings or are misparsed +**Cause:** Non-ISO-8601 format, or Parquet columns not cast to a `DATE`/`TIME`/`TIMESTAMP` type. +**Fix:** Declare a TEXT date column with `dtype=Dtype.date` (a declaration the skill makes). Otherwise the user casts the column in the source (ISO 8601 or epoch time). + +### Symptom: row count silently drops after ingestion +**Cause:** Duplicate primary key values — Kumo retains one row per key and drops the rest. +**Fix:** Report how many rows the key repeats on (the preflight probe, or the snapshot warning in +§1a) and put the choice to the user: a different, genuinely unique key, or the source deduplicated +upstream. The agent does not deduplicate. + +### Symptom: a categorical column has very high cardinality +**Cause:** Categorical supports up to **4,000 unique values by default**. +**Fix:** Re-type the column (`ID` for identifiers, `text` for multi-token strings) — a declaration the agent proposes. Bucketing the values is a data change for the user to make upstream. + +### Symptom: missing values are filled with `N/A`, `none`, `null`, or `-1` +**Cause:** These placeholders are interpreted as real values. +**Fix (the user's, upstream):** leave the field blank in the source. Report the placeholder to the user; a high-missingness column can still be useful when absence itself is informative. + +--- + +## 1a. Kumo reports a problem in the data — stop and put it to the user + +These messages are the SDK's own (`kumo.engine.graph`, `kumo_core.pquery`, +`kumo_core.diskgraph.validation`). Each says the data is not what the graph declares. The run may +still finish. Do not report success before the user has seen the message and decided, and do not +change data, types or the query to make it go away (*Who applies a fix*, above). + +| Message (begins) | What it means | Put to the user | +|---|---|---| +| `Snapshot contains duplicate primary-key row(s) for
.` | `` rows share a key; Kumo keeps one per key and drops the rest | a different key, or the source deduplicated; or continue knowing `` rows are gone | +| `Foreign key
. matched of non-null rows` | only `` of `` rows reach the parent; the rest get no edge | the link may be wrong (wrong column, or values differing in format, e.g. `007` vs `7`), or the parent rows are missing | +| ` invalid values found in
.` | `` values did not parse as the column's type and are read as null (see §5) | whether those rows are bad data, or the column's type is wrong | +| `key
. has a floating-point dtype` | an ID column is stored as a float (often an integer key with nulls) | the column fixed upstream as an integer or string key | +| `foreign key
. (dtype=…) cannot link to primary key
. (dtype=…)` | the two sides of a link hold different types | which side is right, fixed upstream; never cast one side in the notebook | + +A float or mismatched key fails `graph.validate()` and stops the notebook. The other three are +warnings: the notebook keeps going and trains on what is left. Either way, the report the user +reads starts with the message, the table and column, and the count. + +--- + +## 2. Graph construction (highest priority — fail here, not at job time) + +### Symptom: a table cannot be linked into the graph at all +**Cause:** To connect tables into a graph, **at least one table must contain a primary key**. Edges map a `src_table`'s foreign key to a `dst_table`'s primary key. +**Fix:** Set a primary key on the entity/dimension table: +```python +table.primary_key = 'new_primary_key' # must already be a column of the table +print(table.has_primary_key()) +``` + +### Symptom: an edge is declared but the model learns nothing from it +**Cause:** FK values do not actually match PK values. Rows with missing or unmatched key values are +**silently ignored** when linkages are built, so a broken join looks like a sparse one. If the two +linked columns have different types, both are **cast to strings** — values equal as numbers may not +match as strings (`1.0` vs `1`). +**Fix:** There is **no `get_edge_stats()` in this SDK.** The snapshot reports the match rate +(§1a). Report it, and put a low one to the user before training: a wrong link to correct, or keys +to fix upstream so both sides hold the same values in the same type. Do not cast either side. +Set every foreign key to `Stype.ID`; left as numerical it is treated as a feature rather than a +link. + +### Symptom: the edge direction looks wrong / no neighbors are sampled +**Cause:** `src_table` must be the table holding the **foreign key**; `dst_table` holds the matching **primary key**. Edges are bidirectional once declared, but the declaration itself is directional. +**Fix:** +```python +edges=[ + dict(src_table='transaction', fkey='StockCode', dst_table='stock'), + dict(src_table='transaction', fkey='CustomerID', dst_table='customer'), +] +``` + +### Symptom: graph exceeds known limits +**Cause:** Default quotas: 12 tables per graph, 15B rows per table, 2,048 columns per table, 2 TB per table, 30B rows per graph, 2 TB per graph, 30B edges. Environments may be configured lower. +**Fix:** Reduce table count or prune columns. Limits are admin-configurable; a limit increase requires the Kumo point of contact. + +### Symptom: an event table has no time column +**Cause:** Kumo uses time columns to enforce the anchor-time boundary. For event tables (each row a discrete occurrence), the time column should be when the event occurred or when its information became available. +**Fix:** Set `time_column` on the event table: +```python +transaction = kumo.Table.from_source_table( + source_table=transaction_src, + time_column='InvoiceDate', +).infer_metadata() +``` +Entity tables may use the entity creation time. Tables with no meaningful timeline can remain without a time column — that is legal. Do **not** designate an administrative "last modified" timestamp unless it represents when the row became valid for modeling. + +### Symptom: the user supplies pre-aggregated features (`logins_last_30_days`, etc.) +**Cause:** Aggregating before ingestion removes the sequence/relationship information the GNN learns from. Kumo reports aggregated features can perform **5–10% worse** than an equivalent GBDT model, while raw data performs better. +**Fix:** Ask for the raw timestamped event table instead. Do not do feature engineering on Kumo's behalf. + +### Symptom: the user has downsampled tables before ingestion +**Cause:** Independently sampling rows from each table breaks FK coverage and leaves entities without their histories. +**Fix:** Sample around the prediction entity and keep all related rows across every table — e.g. "all rows in all tables where `user_id % 10 == 1`". Never sample each table independently. + +--- + +## 3. Predictive query validation + +### Symptom: `pquery.validate()` reports a syntax error +**Cause:** PQL is `PREDICT FOR EACH ` with an optional filter clause. The query must reference the **names of tables as registered in the graph's `tables` dict**, not the source table names. +**Fix:** Re-check table names against `graph.tables`. Minimal valid form: +```python +pquery = kumo.PredictiveQuery( + graph=graph, + query=( + "PREDICT MAX(transaction.Quantity, 0, 30, days)\n" + "FOR EACH customer.CustomerID\n" + "ASSUMING SUM(transaction.UnitPrice, 0, 7, days) > 15" + ), +) +pquery.validate() +``` + +### Symptom: aggregation window rejected +**Cause:** Both *start* and *end* must be **non-negative integers**, and *end* must be **greater than** *start*. +**Fix:** Correct the bounds. Semantics: start is exclusive, end is inclusive. `SUM(TRANSACTIONS.PRICE, 10, 30)` from a `2020-01-01` anchor aggregates `2020-01-11 00:00:00 < t <= 2020-01-31 00:00:00`. + +### Symptom: the user asks for a horizon in an unsupported unit +**Cause:** The accepted time units are `days` (default), `months`, and `hours`. +**Fix:** Convert. Do not emit `weeks`, `quarters`, or `years` — they are not accepted. + +> `weeks` is not an accepted unit. Convert to `days`. + +### Symptom: `get_task_type()` returns a different task than intended +**Cause:** The PQL formulation determines the ML task, not the user's prose. The same business problem has several valid formulations. +**Fix:** Always call `get_task_type()` after `validate()` and confirm it against the user's intent before generating a training table. Reformulate the query if it disagrees. + +### Symptom: unclear whether the generated training data is what the user meant +**Cause:** Validation checks syntax, not semantics. +**Fix:** Inspect the generated data before training: +```python +training_table = pquery.generate_training_table(non_blocking=False) +print(training_table.data_df().head()) +``` +`data_df()` and `data_urls()` both expose the generated data. + +### Symptom: an `END_DATE` column is set on a non-entity table and appears to be ignored +**Cause:** End-date-aware sampling is currently supported **only** on the entity table (after `FOR EACH`) and on the RHS table in link prediction (after `LIST_DISTINCT`). +**Fix:** Do not rely on end dates elsewhere. + +### Symptom: date-only boundary includes one day too few / too many +**Cause:** A date such as `2025-10-01` is interpreted at the **start** of that day. The target window is left-open, right-closed. +**Fix:** To include all of October 1, use `2025-09-30 23:59:59` rather than midnight on October 1. + +--- + +## 4. Data leakage + +### Symptom: evaluation results are unexpectedly strong, offline and production performance diverge, or one column dominates explanations +**Possible cause:** Data leakage; these symptoms alone do not establish it. Check for: +- A feature directly or indirectly contains the future target. +- An event timestamp marks the start of a period while another column summarizes what happened by the end of it. +- A supposedly static entity column is overwritten over time, so its historical value is unavailable. +- Training and holdout periods overlap or are ordered incorrectly. + +**Fix:** Verify when each source value became available relative to the anchor time before +correcting the timeline or removing a confirmed leaking field. **Do not tune the model to +compensate for temporally invalid data.** For interpretation limits of recorded metrics and +explanations, follow the existing-run analysis route in `kumo-data-science/SKILL.md`. + +### Pre-flight data quality checklist +Run before building the graph: +- [ ] A source table contains the label needed for positive and negative training examples. +- [ ] Enough history to learn seasonal patterns — typically 1–2 years, sometimes more. +- [ ] The data fits within the quotas and limits above. +- [ ] PII is hashed, obfuscated, or removed per policy. +- [ ] No target leakage — columns mutated/updated over time must carry the timestamp at which the value first became known. +- [ ] The dataset contains the signals a human would expect to be predictive. +- [ ] Tables can all be linked through stable primary/foreign keys. + +Column-selection criteria: **clean** (no synthetic data, no other models' predictions, no columns whose definition changed over time), **relevant and mutually exclusive** (drop duplicate/irrelevant columns — graph size drives compute cost), **complete** (full history, not just first/last interaction). + +--- + +## 5. Training and prediction time + +### Symptom: `ValueError: prediction_time must not include a timezone` +**Cause:** `prediction_time` was passed as a timezone-aware `datetime`. `kumo_core` rejects any +`tzinfo` and applies UTC itself. +**Fix:** Pass a **naive** datetime. +```python +PREDICTION_TIME = datetime(2026, 8, 1) # correct +# PREDICTION_TIME = datetime(2026, 8, 1, tzinfo=timezone.utc) # raises +``` + +### Symptom: `ValueError: prediction_time is not supported for static queries` +**Cause:** `prediction_time` was passed for a non-temporal query. +**Fix:** Drop the argument. Confirm the intended task with `pquery.get_task_type()`. + +### Symptom: `fit()` or `predict()` raises `TypeError` about a missing argument +**Cause:** Both require `snapshot=` as a keyword. `fit(training_table)` and +`predict(prediction_table)` alone are errors. +**Fix:** +```python +training_result = trainer.fit(training_table, snapshot=snapshot) +prediction_result = trainer.predict( + prediction_table, snapshot=snapshot, output_config=output_config) +``` + +### Symptom: code from an older notebook raises `AttributeError` on `kumo.` +**Cause:** It targets the previous SDK — `set_backend`, `LocalBackend`, `create_identity`, +`OutputConfig`, `kumo.Stype`, edges written as dicts. A `Graph(tables={alias: table})` mapping +is current, not old. +**Fix:** Do not patch it call by call. Say it targets a superseded build and rewrite against +`kumoai-reference/references/kumo_finetuning_sdk.md` §10. The recognise-and-refuse table is §11.1 there. + +### Symptom: `snapshot()` warns ` invalid values found in
.` +**Cause:** Values in that column did not parse as its declared type. They are read as null; the +snapshot still completes. +**Fix:** Report the table, column and count to the user. A large count usually means a wrong +`dtype` (a TEXT date without `dtype=Dtype.date`, or a numeric column with string placeholders); +correct the declaration or the source and rebuild. Do not ignore it silently. + +### Symptom: `snapshot()` warns the scratch directory is on network storage +**Cause:** The scratch root is on network-backed storage such as EBS. Sampling and training-table +generation run much slower there. +**Fix:** It is a warning, not an error — a small run can continue. For a large run, point +`kumo.configure(scratch_dir=...)` at local NVMe if the runtime has it. + +### Symptom: link prediction job fails with an explicit out-of-memory error +**Cause:** Link prediction has **no** batch-size fallback. Node prediction, classification, regression, and forecasting catch OOM, halve the batch size, and retry; link prediction does not. +**Fix:** Reduce `output_embedding_dim` — the first remedy to try. + +### Symptom: results change between two training runs on the same query +**Cause:** The source data moved underneath the graph. +**Fix:** `graph.snapshot()` locks in the current version of all tables so repeated runs use the same data. It also reduces resource usage across multiple jobs. + +--- + +## 6. Known gaps + +> Not verified: +> - Verbatim exception text and error codes raised by `Graph.validate()` and +> `PredictiveQuery.validate()`. **Do not fabricate them.** +> - The exception types raised by graph or PQL validation failures. +> - Whether `pquery.validate()` catches nonexistent column references, or only syntax. diff --git a/skills/kumoai-reference/skill-card.md b/skills/kumoai-reference/skill-card.md new file mode 100644 index 000000000..0c5098882 --- /dev/null +++ b/skills/kumoai-reference/skill-card.md @@ -0,0 +1,121 @@ +## Description:
+Reference for translating a validated ML request into KumoAI graph configuration, PQL (predictive query language), and Kumo Predict SDK code; normally loaded by the kumo-data-science skill after its intake checks pass, rather than invoked directly.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner +NVIDIA
+ +### License/Terms of Use:
+Apache 2.0
+## Use Case:
+Developers and data scientists use this skill, after a request passes kumo-data-science intake checks, to generate Kumo table/graph definitions, PQL predictive queries, and version-matched Kumo SDK training and prediction code for the local in-process Kumo runtime.
+ +### Deployment Geography for Use:
+Global
+ +## Requirements / Dependencies:
+**Requires API Key or External Credential:** [Yes]
+**Credential Type(s):** [Cloud Credentials, Other [Snowflake runtime token, connections.toml entry, or key pair from a secret store]]
+ +Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate.
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [Graph construction](references/graph_construction.md)
+- [PQL writing](references/pql_writing.md)
+- [Kumo SDK code generation reference](references/kumo_finetuning_sdk.md)
+- [Task journeys](references/task_journeys.md)
+- [Predictions, embeddings and destinations](references/outputs.md)
+- [Troubleshooting signals](references/troubleshooting_signals.md)
+- [NVIDIA/kumo-engine#134 (graph visualization)](https://github.com/NVIDIA/kumo-engine/pull/134)
+ + +## Skill Output:
+**Output Type(s):** [Code, Configuration instructions]
+**Output Format:** [Markdown with inline Python code blocks and PQL statements]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [Generated code uses only SDK signatures documented in the skill references; it does not run jobs or interpret results.]
+ +## Evaluation Agents Used:
+- Claude Code (`aws/anthropic/bedrock-claude-opus-5-5`)
+- Codex (`openai/openai/gpt-6.1-sol`)
+ + + +## Evaluation Tasks:
+5 evaluation tasks (4 positive, 1 negative), 1 attempt per task, each in an isolated `k8s-sandbox` pod; each task also run without the skill as a baseline. Evaluated 2026-11-19 with evaluator version `1.6.10`.
+ +## Evaluation Metrics Used:
+Reported benchmark dimensions:
+- Security: Is it safe to use? Scored from `security` (100%).
+- Correctness: Is the answer correct? Scored from `accuracy` (100%).
+- Discoverability: Was the right skill loaded when needed? Scored from `skill_execution` (100%).
+- Effectiveness: Did the skill help complete the task? `goal_accuracy` (50%) + `behavior_check` (50%).
+- Efficiency: Did it avoid wasted tool calls and token usage? `skill_efficiency` (50%) + `token_efficiency` (50%).
+ +Underlying evaluation signals used in this run:
+- `security`: Unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution`: Whether the expected skill was selected, decoys were avoided, and the workflow executed.
+- `skill_efficiency`: Tool-call productivity (legacy wire id; routing is scored under Discoverability).
+- `accuracy`: Final-answer correctness against the reference answer.
+- `goal_accuracy`: Whether the user's goal was achieved.
+- `behavior_check`: Whether the expected workflow behavior was followed.
+- `token_efficiency`: Actual uncached prompt plus completion usage (50% of Efficiency).
+ + + +## Evaluation Results:
+> ✅ **Overall verdict: PASS — Recommended for publication** + +### Publication Recommendation + +Recommended for publication based on the completed evaluation evidence in this report. + +### Results at a Glance + +| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | +|---|---:|---:| +| Overall | 90.1% — baseline ran, but no comparable score was available; uplift unavailable | 86.3% — baseline ran, but no comparable score was available; uplift unavailable | +| Security | 100.0% → 80.0% (-20.0 points) | 100.0% → 80.0% (-20.0 points) | +| Correctness | 68.0% → 100.0% (+32.0 points) | 80.0% → 100.0% (+20.0 points) | +| Discoverability | 96.3% — baseline ran, but no comparable score was available; uplift unavailable | 95.0% — baseline ran, but no comparable score was available; uplift unavailable | +| Effectiveness | 52.2% → 84.0% (+31.8 points) | 60.2% → 71.6% (+11.4 points) | +| Efficiency | 90.1% — baseline ran, but no comparable score was available; uplift unavailable | 85.2% — baseline ran, but no comparable score was available; uplift unavailable | + +**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points. + +Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline. + +A partial dimension was calculated from only the available configured signals; review the detailed report before relying on it. + +### Comparison details + +Shared-source comparisons use the same signals and weights on both conditions. They do not replace the full dimension scores above or change the publication verdict. + +| Agent | Dimension | Basis | Baseline → With skill (uplift) | Explanation | +| --- | --- | --- | --- | --- | +| claude-code | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| claude-code | Efficiency | Token-only comparison | 97.2% → 88.9% (-8.3 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | +| codex | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| codex | Efficiency | Token-only comparison | 80.3% → 87.0% (+6.7 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | + +### Tier Status + +| Tier | Purpose | Status | Evidence | +|---|---|---|---| +| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 15 finding(s) | +| Tier 2 | Semantic deduplication | **PASSED WITH OBSERVATIONS** | 2 validator(s); 2 finding(s) | +| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 5 task(s) | + +## Skill Version(s):
+0.1.0 (source: pyproject.toml); git SHA 2bbedbe6, committed 2026-11-18
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/skills/kumoai-reference/skill.oms.sig b/skills/kumoai-reference/skill.oms.sig new file mode 100644 index 000000000..61c2c9666 --- /dev/null +++ b/skills/kumoai-reference/skill.oms.sig @@ -0,0 +1 @@ +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAia3Vtb2FpLXJlZmVyZW5jZSIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICJkNWE1OWExYTc5YWMyYTY5M2E3OWE5ZmY5NTNiYjg3N2YyZTM0ZjQ2NjA0N2MxNWViY2NkNmUzY2Q4NmVlYjg3IgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiLAogICAgICAibWV0aG9kIjogImZpbGVzIiwKICAgICAgImFsbG93X3N5bWxpbmtzIjogZmFsc2UsCiAgICAgICJpZ25vcmVfcGF0aHMiOiBbCiAgICAgICAgIi5naXQiLAogICAgICAgICIuZ2l0aWdub3JlIiwKICAgICAgICAiLmdpdGF0dHJpYnV0ZXMiLAogICAgICAgICIuZ2l0aHViIgogICAgICBdCiAgICB9LAogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiNTk5M2M5MTIxZGY5ZDZkYzhmOTI2NDg4YWIzNDIzMTZhYTIzY2VhMjBmMjRmMDFmNjQ0YjQxYzQ5NDAyYjc5NCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiMDQzMTYyYmFiYzc1NWFiYTEzNTgyNWFmM2NhMzc1ZDhmMGQ1MWQ5MDRjZGNhYmU2M2Y3MzlhNjNmODRjZDRhZiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICJlODBjYWNhY2NjNjA3YjIwMDFmYzE5MDQ0ZGNkNzEwOTJiNWEwNTQzYmY1NzYzMDJkZmRmMDgxMmY0NzhkZTZjIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiOGU4OGUyM2IxN2EyMGUxN2E5Mjc4Njg3NDJmYzQwNTA2NGE4OGQxNDVmMmNiODgzNmE4NTJlMzQwMDVlNGJiYyIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvZ3JhcGhfY29uc3RydWN0aW9uLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICJlMDQxYWY1ZTkyZmI2NmYyMjE1NWU0NTk0Mjk5NmFlZjM4ZWFjNGE0Y2E3ZDE0YjlkOWNhZWU4ZWFmNmI0OWI4IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9rdW1vX2ZpbmV0dW5pbmdfc2RrLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICJmMjlmM2QyZDJiY2E5NTUwZGU1MjViYjViYzYyZTU1ODc4ZTBmNTRiYTdmNWYxMDliMDJmYzk3OGJiYjg1ZWJhIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9vdXRwdXRzLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICI0NTc4N2RhY2I4ZTk3NzJhZTk1YjA1NzRmMjk3ODM0ODdlNGNjYzliNzBhZWUxYzkwNGQ4MGVkMmU4NWMxM2FmIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9wcWxfd3JpdGluZy5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiYzA5YzIzYmExMGM3YWEwMDg5OTY0NjAwMGExNmVkYjMyYzg5ZDI5YzYzNTllNjE0ZGM5YWM2YTJlYTkyMGYwOCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvdGFza19qb3VybmV5cy5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiNzliNTU1ZGJmMmQ0N2MyOThlY2U2NGY0ZjVkODg0ZGU3N2Q0Zjc2YTY1M2JhOWUyMTgzMzBjM2YxYzE0N2ZiNSIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvdHJvdWJsZXNob290aW5nX3NpZ25hbHMubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjEwNzExNTI0ODYxNjNhNTczYTY0NjUzNjI0ZGE4ZDg5NTM5MDg0YWE2NTFiZTg2MDQwOWU0NjJiM2I4NzM1NDQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJza2lsbC1jYXJkLm1kIgogICAgICB9CiAgICBdCiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMQC2VkMzp+y8hSdcED0i4y3NkEcU1OeDsmAK3JGZr3tNC+/ovfO2F/8Q8Bnciq24vOoCMCgBP14M2N8Igj9QVXUi+tQuQ94dZBYZvedIdwQJ6C+R82fPhYoAxM8gbKr9dC2R7Q==","keyid":""}]}} \ No newline at end of file