Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
f52998a
Close the gaps before 2.0.0
Manushpm8 Nov 5, 2026
2f59ae2
Remove the empty env block from the rust-lint job
Manushpm8 Nov 6, 2026
f241e1b
Take the thrift fix the connector now allows
Manushpm8 Nov 6, 2026
7116dbf
Take the oauthlib fix the connector now allows
Manushpm8 Nov 6, 2026
12c22da
Clear the open dependency alerts the lock can reach
Manushpm8 Nov 6, 2026
89b9162
Tie the contract decisions to the v2-candidate merge base
Manushpm8 Nov 6, 2026
e986f46
Indent doc list continuations so clippy passes
Manushpm8 Nov 6, 2026
74eb425
Take arrow 58.4.1 for RUSTSEC-2026-0442
Manushpm8 Nov 6, 2026
73f7d35
Close the gaps the Databricks parity runs found
Manushpm8 Nov 6, 2026
5c7b874
Name a refused column by its declared type
Manushpm8 Nov 9, 2026
38a81eb
Keep a payload view to the dictionary contract
Manushpm8 Nov 9, 2026
11ec821
Give each analysed instance its own id across calls
Manushpm8 Nov 9, 2026
e1ce779
Say decimals are exact through the SQL warehouse too
Manushpm8 Nov 9, 2026
c38f8e5
Take rusqlite 0.41 for RUSTSEC-2026-0459
Manushpm8 Nov 9, 2026
4f7393b
Name the field a relational session's refusal finds changed
Manushpm8 Nov 9, 2026
b450c95
Format the session refusal test
Manushpm8 Nov 9, 2026
368555c
Read a described table's catalog again, and the rest only if it changed
Manushpm8 Nov 9, 2026
ff17f07
Bring the live tests to what the endpoints serve
Manushpm8 Nov 9, 2026
1e0e8ec
Reuse a table's facts only while its catalog declares it unchanged
Manushpm8 Nov 10, 2026
f541b4c
Read a table in full where a fact rests on its keys
Manushpm8 Nov 10, 2026
f7a445f
Address review: cleanup errors, payload reads, local option checks, docs
Manushpm8 Nov 10, 2026
27d1cfc
Widen forecast analysis to every column a step sent
Manushpm8 Nov 10, 2026
a8e4bd5
Refuse candidate-free temporal link prediction up front
Manushpm8 Nov 10, 2026
39e7906
Expect the link refusal in the in-process and Java result tests
Manushpm8 Nov 10, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,6 @@ jobs:
defaults:
run:
working-directory: rust
env:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
Expand Down
47 changes: 18 additions & 29 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,12 @@ name: publish

on:
push:
tags: ['v[0-9]+.[0-9]+.[0-9]+']
tags:
- 'v[0-9]+.[0-9]+.[0-9]+'
# PEP 440 pre-releases: v2.0.0a1, v2.0.0b1, v2.0.0rc1.
- 'v[0-9]+.[0-9]+.[0-9]+a[0-9]+'
- 'v[0-9]+.[0-9]+.[0-9]+b[0-9]+'
- 'v[0-9]+.[0-9]+.[0-9]+rc[0-9]+'
workflow_dispatch:

# Least privilege at the top: a job that needs more says so itself.
Expand Down Expand Up @@ -201,31 +206,6 @@ jobs:
cp native-*/*.whl dist/
cp native-*/*.whl.json dist/

- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
github-token: ${{ github.token }}
run-id: ${{ steps.candidate-ci.outputs.run_id }}
name: java-jar
path: java-release
- name: Verify and assemble the tested Java artifacts
run: |
python - <<'PY'
import hashlib, json, os, pathlib, shutil, xml.etree.ElementTree as ET
root = pathlib.Path('java-release')
record = json.loads((root / 'java-artifacts.json').read_text())
assert record['commit'] == os.environ['GITHUB_SHA']
version = os.environ['GITHUB_REF_NAME'].removeprefix('v')
stem = f'structured-data-models-client-{version}'
assert set(record['files']) == {stem + '.jar', stem + '.pom'}
pom = ET.parse(root / (stem + '.pom')).getroot()
assert pom.findtext('{http://maven.apache.org/POM/4.0.0}version') == version
for name, digest in record['files'].items():
path = root / name
assert hashlib.sha256(path.read_bytes()).hexdigest() == digest
shutil.copy2(path, pathlib.Path('dist') / name)
shutil.copy2(root / 'java-artifacts.json', 'dist/java-artifacts.json')
PY

- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: built-distributions
Expand Down Expand Up @@ -257,8 +237,8 @@ jobs:
ref = os.environ['GITHUB_REF_NAME']
# `lstrip` would also have accepted `vv1.0.0`, and a dispatch can be
# aimed at a tag called `1.0.0`.
if not re.fullmatch(r'v\d+\.\d+\.\d+', ref):
sys.exit(f'{ref} is not a release tag of the form v1.2.3')
if not re.fullmatch(r'v\d+\.\d+\.\d+((a|b|rc)\d+)?', ref):
sys.exit(f'{ref} is not a release tag of the form v1.2.3 or v1.2.3rc1')
tag = ref.removeprefix('v')
wheels = sorted(p.name for p in pathlib.Path('dist').glob('*.whl'))
sdists = sorted(p.name for p in pathlib.Path('dist').glob('*.tar.gz'))
Expand Down Expand Up @@ -371,11 +351,16 @@ jobs:
# the Kitmaker publish has worked. This only withholds the
# promotion: with no earlier release to stay Latest, GitHub still
# reports the newest non-draft release as the latest one.
prerelease=()
if [[ "$GITHUB_REF_NAME" =~ (a|b|rc)[0-9]+$ ]]; then
prerelease=(--prerelease)
fi
gh release create "$GITHUB_REF_NAME" \
--repo "$GITHUB_REPOSITORY" \
--title "$GITHUB_REF_NAME" \
--verify-tag \
--latest=false \
${prerelease[@]+"${prerelease[@]}"} \
--notes-file dist/notes.md
fi
# The SBOM is a release asset and was never meant for the index; the
Expand Down Expand Up @@ -419,5 +404,9 @@ jobs:
echo "NVIDIA's network. RELEASING.md says who runs it and what it"
echo "has to do."
echo
echo "Mark this release Latest once the index has all three distributions."
if [[ "$GITHUB_REF_NAME" =~ (a|b|rc)[0-9]+$ ]]; then
echo "This is a pre-release: leave it unmarked."
else
echo "Mark this release Latest once the index has all three distributions."
fi
} >> "$GITHUB_STEP_SUMMARY"
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# All rights reserved.
# SPDX-License-Identifier: Apache-2.0

exclude: ^(packages/structured-data-models-pql/src/predictive_query/|packages/structured-data-models-api/src/sdm_api/generated/tfm_api\.py$|packages/structured-data-models-relational-engine/src/sdm_relational_engine/rfm/assets/mermaid\.min\.js$)
exclude: ^(packages/structured-data-models-api/src/sdm_api/generated/tfm_api\.py$|packages/structured-data-models-client/src/sdm_client/assets/mermaid\.min\.js$)

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
Expand Down
125 changes: 121 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,12 @@
Kept in the [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) shape, and
versioned by [semantic versioning](https://semver.org) as
[`docs/reference/stability.md`](docs/reference/stability.md) defines it for
these packages. The three Python distributions and Java library share one version.
these packages. The three Python distributions share one version.

## Unreleased

Release candidate versions are `2.0.0rc1` for Python and Java and `2.0.0-rc.1` for Rust.
The release candidate version is `2.0.0rc1`, spelled `2.0.0-rc.1` in Rust's `Cargo.toml`.
The Java library is built and tested in CI but is not released.
The public API snapshot records this version update; publication remains pending.

### Added
Expand Down Expand Up @@ -39,17 +40,90 @@ The public API snapshot records this version update; publication remains pending
yourself is used as it is.
- The client now contains the Rust core instead of depending on separate
relational-engine and Python PQL packages.
Python wheels and the Java JAR target Linux x86-64/ARM64, macOS
x86-64/ARM64 and Windows x86-64.
Python wheels target Linux x86-64/ARM64, macOS x86-64/ARM64 and Windows
x86-64; the Java library, built from source, supports the same platforms.
Native Windows ARM64 is deferred pending upstream Arrow dependencies.

- Both models support gradient explanation collections when the model runtime supports them.
Python exposes `result.explanations` and its pandas view; Java exposes `explanations()` and its Arrow view.
`EXPLAIN PREDICT` requests explanations, including each forecast step; client-assembled rankings refuse them.
Python also accepts manually supplied `PredictionResult(explanations=...)` mappings.

- `Schema` gains `infer_metadata`, `infer_links`, `link_report`,
`metadata_report` and `print_metadata`/`print_links`, `link_origins`
(as `RelationshipOrigin` pairs), `with_relationship`, `without_relationship`,
`without_table`, `with_source` and `with_sql_column`, and `to_mermaid`,
`to_html` and `visualize` for an offline diagram.
- `RelationalTask` and `Schema.materialize_examples` ask a relational
question from labelled examples without PQL; `build_timeseries` builds a
forecast's input from a series.
- `PredictiveQuery.labels` computes a query's historical labels without
calling a model.
- `InferenceConfig` holds relational inference settings.
- `ExplanationSummarizer` writes a short summary of an explanation with an
LLM endpoint.
- `Table.end_time_column` and `SourceTable.end_time_column` name a row's end
time.
- `PredictionResult.merge(parts, columns=...)` takes the column analysis to
keep on the merged result.

### Changed

- `structured-data-models-client` is now a native wheel, `cp310-abi3`, built
for Linux x86-64 and aarch64 (manylinux), macOS arm64 and x86-64, and Windows
AMD64. 1.0.0 was pure Python. It no longer installs on PyPy, musl Linux,
32-bit platforms or Windows ARM64.
- A failure in the data source a schema reads, such as a warehouse query that
errors, raises `StructuredDataError` with code `PROVIDER_ERROR`, and
`details['driver_class']` names the driver's exception class. 1.0.0 raised
these from the relational engine as `INTERNAL_ERROR`. A semantic view or
metric view whose expression names a column its base table lacks is one of
them: the warehouse refuses the read, where 1.0.0 refused the view as
`INVALID_REQUEST` before reading it.
- A column of a type a described table does not take, such as a list of
booleans, a struct or a map, raises `UnsupportedRequestError` with code
`UNSUPPORTED_REQUEST` and `details['rule']` `column_type`. The message and
details name the column, its table and its type, as the source declares it
where it does, such as a Snowflake `VARIANT`. 1.0.0 raised a `ValueError`
naming the column and table.
- Describing an empty table raises `StructuredDataError` with code
`INVALID_REQUEST` and `details['rule']` `empty_table`. 1.0.0 raised a
`RuntimeError`, so `except RuntimeError` no longer catches it.
- A column whose name holds a dot is kept and sent. 1.0.0 left it out with a
warning.
- `predict()` on a request gathered in several calls that share one context
scores them through a temporary endpoint session, so the context is
uploaded once. 1.0.0 sent every call stateless. An expired session is
recreated once; if the endpoint loses it again, as replicas that do not
share sessions do, the remaining calls are sent stateless.

- A related table sampled from frames sends its own primary key, where
1.0.0 sent a generated `__node_id`.
- Each `materialize()` on a schema from `Schema.from_source` reads its
tables' catalog again, one query per table, so a column dropped or
retyped since is found. A table whose catalog declares a column otherwise
is read again in full, its keys and first rows included, as are SQLite's
tables and a semantic model's, with its definition. 1.0.0 read them once
and kept them on the schema. A table emptied since it was described is
refused as empty by a warehouse's row count; elsewhere it reads as a
table with no times, as 1.0.0 read it.
- An HTTP 408 from `POST /v1/predictions` is retried, as 429, 502, 503 and
504 are.
- A foreign key holding integers and nulls is sent as exact integers beside
nulls. 1.0.0 sent such a column as `float64`, so its keys arrived as floats
and any past 2**53 was rounded.
- A relational request reached over three or more hops fills a foreign key
from every path that reaches the row, where 1.0.0 took the first path and
left the rest null. The model sees more links, so predictions can differ.
- Related rows with the same timestamp keep their source order. 1.0.0 used
NumPy's unstable sort, whose order depends on the platform, so sampled
rows and predictions can differ where timestamps tie.
- `SUM` and `AVG` labels, and the lagged target columns built from them, are
summed in float32 with Kahan compensation, as
pandas' Cython grouped sum computes them. 1.0.0 left this to the installed
pandas, whose answer depends on its version: pandas 3.1 sums sorted groups
with NumPy and averages in float64. A label can differ in its last digit,
and a threshold on it can flip.
- Sampled Tabular context uses stable Rust index ordering, including duplicate labels and mixed numeric labels.
Equal labels retain sampled order; mixed numbers compare by their represented values.
Context order and predictions can differ from platform-specific NumPy ordering; sampled membership and warehouse sampling are unchanged.
Expand All @@ -63,6 +137,16 @@ The public API snapshot records this version update; publication remains pending
- Gradient explanations have no separate one-row or 1,000-context-row client restriction.
Endpoint capabilities and ordinary request limits still apply.
The singular raw `result.explanation` view remains supported.
- A temporal link prediction asked without candidates, such as
`PREDICT LIST_DISTINCT(orders.item_id, 0, 30, days) FOR EACH users.user_id`,
raises `UnsupportedRequestError` with code `UNSUPPORTED_REQUEST` and
`details['rule']` `temporal_link_prediction`. `materialize()` and
`materialize_examples()` refuse it before any source is read, and
`predict()` refuses a request that names the task before anything is sent.
No released Kumo Relational model serves it, so `kumo-relational` no longer
lists it among its tasks. Rank an explicit candidate set with `RANK TOP k`
and `candidates=` instead. 1.0.0 built the request, and the endpoint
refused it.

### Fixed

Expand All @@ -82,6 +166,39 @@ The public API snapshot records this version update; publication remains pending
- A request's description leaves out a wire table carrying an integer past
the float range on every pandas release. pandas 3.1 had started to build
a frame around one, where earlier releases refused it.
- On Databricks through Spark, a `TIMESTAMP` column is read in UTC, so the
anchor time and the rows a window takes no longer move with the client
machine's time zone. 1.0.0 read its range as local times.
- On Databricks, through Spark or a SQL warehouse, a `DECIMAL` value with a
fraction is sent exactly, such as `997.67`. 1.0.0 sent it through a float, such as
`997.6700000000001`.

### Removed

- `structured-data-models-pql` (`predictive_query`) and
`structured-data-models-relational-engine` (`sdm_relational_engine`) are no
longer distributed, and importing either fails. Write predictive queries
through `sdm_client.PredictiveQuery`.
- The `[relational]` extra installs nothing extra. It is kept so existing
install commands still work.
- Features that lived only in the relational engine are gone: Snowflake SPCS
serving, the engine's telemetry and progress logging, and its extras
`snowflake-serving`, `explain` and `relbench`. RelBench datasets load with
`Schema.from_relbench`. The LLM summary of an explanation is now
`ExplanationSummarizer`, a different API: `summarize(view, query, prediction)`
takes an `ExplanationView` and raises on failure, where 1.0.0's
`generate_summary` returned fallback text.

### Upgrading from 1.0.0

Uninstall the two retired packages before upgrading:

```bash
pip uninstall structured-data-models-relational-engine structured-data-models-pql
```

The 1.0.0 engine pins `structured-data-models-api==1.0.0`, which conflicts
with 2.0.

## 1.0.0 - 2026-10-15

Expand Down
2 changes: 1 addition & 1 deletion NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ the MIT license, and its bundled dependencies.
The upstream license, source and full dependency attribution are retained in
packages/structured-data-models-client/src/sdm_client/assets/MERMAID_LICENSE,
VENDORED.md and THIRD-PARTY-NOTICES-MERMAID.txt in that directory.
Both Python and Java distributions include these resources.
The Python client wheel includes these resources, as does the Java library built from source.

--------------------------------------------------------------------------------
Test data
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
get an answer without training a model for each dataset.**

```bash
pip install 'structured-data-models-client[relational]'
pip install structured-data-models-client
```

## Overview
Expand Down
46 changes: 23 additions & 23 deletions RELEASING.md

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RELEASING.md:217

P2: Release steps say to mark pre-releases Latest

Nit: this procedure also covers alpha, beta and RC releases, but the After section always says to mark the release Latest. The workflow summary explicitly says to leave prereleases unmarked. Could this step say it applies only to final releases?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f7a445f. Latest is now marked for final releases only; alpha, beta and RC releases stay unmarked.

Original file line number Diff line number Diff line change
Expand Up @@ -47,36 +47,35 @@ SDM_TABULAR_URL=... SDM_RELATIONAL_URL=... \
python -m pytest packages/structured-data-models-client/tests -m live_nim -q
```

## Java artifacts
## Java

The same successful CI run supplies the Java JAR and POM.
The release workflow checks their source commit, version and SHA-256 hashes before attaching them to the GitHub Release.
The JAR must contain Linux and macOS x86-64/ARM64 libraries plus Windows x86-64.
Windows ARM64 is deferred for both languages because required Arrow binaries are not published upstream.
The Java library is built and tested in CI, but it is not released yet.
The release workflow attaches no JAR or POM, and nothing is published to a Maven repository.
Keep `java/pom.xml` at the same version as the rest, so the library is ready when it is released.

Attaching a JAR is not Maven repository publication.
The Java repository destination and release credentials remain to be configured before the first Java release; install the published coordinates in a clean consumer before marking Java publication complete.
Keep the JAR, POM, Rust workspace and three Python distribution versions aligned.
## Pre-releases

## Current release-candidate preparation

Prepare `2.0.0rc1` for Python and Java; Rust uses the equivalent SemVer spelling `2.0.0-rc.1`.
Publication and tagging are on hold until the owner explicitly authorizes them.
Java initially targets GitHub Release assets; Maven Central is a follow-up.
The existing publication workflow accepts final-release tags only; RC publication would need a separately reviewed prerelease path.
A tag such as `v2.0.0rc1`, `v2.0.0b1` or `v2.0.0a1` runs the same workflow with the same checks.
The version is the PEP 440 spelling, as in `_version.py`; Rust's `Cargo.toml` writes the same version in SemVer (`2.0.0-rc.1`).
The GitHub Release is marked as a pre-release.
`CHANGELOG.md` needs a `## 2.0.0rc1` heading, since the release notes are taken from it.
Tagging `v2.0.0rc1` is on hold until the owner approves it.

## Bumping the version

One literal per distribution, in `_version.py`. All three must match, and
`tests/test_release_contract.py` fails if they do not.
The version is written in several places, and all of them move together:

```bash
for f in packages/*/src/*/_version.py; do
sed -i '' "s/__version__ = '.*'/__version__ = '1.2.0'/" "$f"
done
```
| Where | Spelling for `2.0.0rc1` |
| --- | --- |
| `packages/*/src/*/_version.py` (one per distribution) | `2.0.0rc1` |
| `rust/Cargo.toml`, `[workspace.package] version`, which maturin stamps on the client wheel | `2.0.0-rc.1` |
| `rust/Cargo.lock`, the workspace crates' entries (`cargo update --workspace` rewrites them) | `2.0.0-rc.1` |
| `java/pom.xml`, `<version>` | `2.0.0rc1` |
| `packages/structured-data-models-client/pyproject.toml`, the pins on `structured-data-models-api` and the connectors extras | `2.0.0rc1` |
| `tests/golden/public_api.json`, rewritten by `scripts/snapshot_public_api.py --write` | `2.0.0rc1` |

Internal pins (`structured-data-models-api==1.0.0` in `structured-data-models-client`'s dependencies) move with them.
`tests/test_release_contract.py` fails if `_version.py`, the Rust workspace, `java/pom.xml` or a first-party pin disagree.
Search for the old version afterwards to catch anything else, such as the coordinates in `java/README.md`.

Update `CHANGELOG.md`: the entries for the release go under its own heading
with the date you are tagging on, not the date you wrote them, so the tag
Expand Down Expand Up @@ -215,7 +214,8 @@ Check the three projects on pypi.org and confirm each shows NVIDIA organization
ownership. Portal reporting that a release completed is not the same as the
package being installable, so install it the way a user would.

Then mark the GitHub Release Latest. The workflow leaves it unmarked on
For a final release, then mark the GitHub Release Latest; leave an alpha, beta
or release candidate unmarked. The workflow leaves every release unmarked on
purpose, because until the publish has worked it would advertise a version the
index does not have. One exception is worth knowing: when there is no earlier
release to hold the position, GitHub reports the newest non-draft release as
Expand Down
Loading
Loading