Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,14 @@ The public API snapshot records this version update; publication remains pending
and `candidates=` instead. 1.0.0 built the request, and the endpoint
refused it.

### Removed

- The client no longer enforces its own copies of the endpoint's limits, so a deployment that raises them is not held back: the 10,000 relational context rows (`num_context_samples` may be larger), the 10,000 predict rows per call (`rows_per_call` may be larger; 10,000 stays the default batch), the tabular refusal past 10,000 context rows (`materialize()` still samples to 10,000 by default, with a warning), the 30 MiB request body, the 15-table cap, and the 100,000-entity and 100-call guards on `materialize()`.
The endpoint refuses what it does not take.
Comment thread
adi-kumo marked this conversation as resolved.
Resolved
Sampling still supports at most six hops, including explicit neighbor-list depth.
The Databricks gateway's 16 MB check, `MOST_PARTS`, `top_k` of at most 20 and the 64 MiB response cap stay.
The contract is version 74.

### Fixed

- Explicit empty outputs remain invalid with `EXPLAIN`; unsupported local-runtime outputs are refused before model weights load.
Expand Down
6 changes: 6 additions & 0 deletions contract/expected/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,12 @@ Their differences from DuckDB follow the fixture loader's explicit mappings in `
| SQLite / Postgres | Malformed text-time values are refused; the US-date fixture completes under these backends' existing read behavior |
| Postgres | NaN-key fixtures refuse at materialize; a negative neighbor bound retains the existing `PROVIDER_ERROR` outcome |

`sample.num-hops.seven` and `sample.num-neighbors.seven-hops` refuse at materialize with `UNSUPPORTED_REQUEST`: sampling supports at most six hops.
The `multicall.body-30mib` cases reach predict on every backend: the client holds a call to no size of its own.
`multicall.entities.100001` and `multicall.calls.101` are gathered: the client sets no most on the entities or the calls of one `materialize()`.
`rank.entities-by-candidates.over` is refused at materialize only for the entities that name no user, as `rank.entities-by-candidates.at` is, since the client sets no entity ceiling over a ranking.
These outcomes were recorded from the runner's observations (`run_python.py`, then `check_observations.py --expect`) on DuckDB and SQLite; the Postgres entries take the same outcomes, as no Postgres server was available.

These expectations describe the declared fixture mappings; they do not claim that every backend supports identical physical types.
`explain.two-entities` now succeeds on all three backends: The client supports explanations for multiple prediction rows.
`contract/tests/test_query_options.py::test_each_explained_entity_retains_its_own_raw_response` independently checks both rows' distinct explanation values, the request batch size and the public success outcome.
Expand Down
10 changes: 5 additions & 5 deletions contract/expected/duckdb-python.json
Original file line number Diff line number Diff line change
Expand Up @@ -108,13 +108,13 @@
"metadata.items": {"code": null, "error": null, "result": "success", "stage": "predict"},
"metadata.items-across-calls": {"code": null, "error": null, "result": "success", "stage": "predict"},
"metadata.items-explained": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.body-30mib.one-row": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"multicall.body-30mib.several-rows": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"multicall.body-30mib.one-row": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.body-30mib.several-rows": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.calls.100": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.calls.101": {"code": "UNSUPPORTED_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"multicall.calls.101": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.entities-per-call.10001": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.entities.100000": {"code": null, "error": null, "result": "success", "stage": "materialize"},
"multicall.entities.100001": {"code": "UNSUPPORTED_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"multicall.entities.100001": {"code": null, "error": null, "result": "success", "stage": "materialize"},
"multicall.fail-call-0": {"code": null, "error": "NimRequestError", "result": "refusal", "stage": "predict"},
"multicall.fail-call-2": {"code": null, "error": "NimRequestError", "result": "refusal", "stage": "predict"},
"multicall.forecast.batched": {"code": "UNSUPPORTED_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
Expand Down Expand Up @@ -160,7 +160,7 @@
"rank.candidates-not-a-ranking": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.decimal-key": {"code": null, "error": null, "result": "success", "stage": "predict"},
"rank.entities-by-candidates.at": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.entities-by-candidates.over": {"code": "UNSUPPORTED_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.entities-by-candidates.over": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.most-parts": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.no-candidates": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.odd-column": {"code": null, "error": null, "result": "success", "stage": "predict"},
Expand Down
10 changes: 5 additions & 5 deletions contract/expected/postgres-python.json
Original file line number Diff line number Diff line change
Expand Up @@ -108,13 +108,13 @@
"metadata.items-across-calls@postgres": {"code": null, "error": null, "result": "success", "stage": "predict"},
"metadata.items-explained@postgres": {"code": null, "error": null, "result": "success", "stage": "predict"},
"metadata.items@postgres": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.body-30mib.one-row@postgres": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"multicall.body-30mib.several-rows@postgres": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"multicall.body-30mib.one-row@postgres": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.body-30mib.several-rows@postgres": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.calls.100@postgres": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.calls.101@postgres": {"code": "UNSUPPORTED_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"multicall.calls.101@postgres": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.entities-per-call.10001@postgres": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.entities.100000@postgres": {"code": null, "error": null, "result": "success", "stage": "materialize"},
"multicall.entities.100001@postgres": {"code": "UNSUPPORTED_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"multicall.entities.100001@postgres": {"code": null, "error": null, "result": "success", "stage": "materialize"},
"multicall.fail-call-0@postgres": {"code": null, "error": "NimRequestError", "result": "refusal", "stage": "predict"},
"multicall.fail-call-2@postgres": {"code": null, "error": "NimRequestError", "result": "refusal", "stage": "predict"},
"multicall.forecast.batched@postgres": {"code": "UNSUPPORTED_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
Expand Down Expand Up @@ -160,7 +160,7 @@
"rank.candidates-not-a-ranking@postgres": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.decimal-key@postgres": {"code": null, "error": null, "result": "success", "stage": "predict"},
"rank.entities-by-candidates.at@postgres": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.entities-by-candidates.over@postgres": {"code": "UNSUPPORTED_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.entities-by-candidates.over@postgres": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.most-parts@postgres": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.no-candidates@postgres": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.odd-column@postgres": {"code": null, "error": null, "result": "success", "stage": "predict"},
Expand Down
10 changes: 5 additions & 5 deletions contract/expected/sqlite-python.json
Original file line number Diff line number Diff line change
Expand Up @@ -108,13 +108,13 @@
"metadata.items-across-calls@sqlite": {"code": null, "error": null, "result": "success", "stage": "predict"},
"metadata.items-explained@sqlite": {"code": null, "error": null, "result": "success", "stage": "predict"},
"metadata.items@sqlite": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.body-30mib.one-row@sqlite": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"multicall.body-30mib.several-rows@sqlite": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"multicall.body-30mib.one-row@sqlite": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.body-30mib.several-rows@sqlite": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.calls.100@sqlite": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.calls.101@sqlite": {"code": "UNSUPPORTED_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"multicall.calls.101@sqlite": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.entities-per-call.10001@sqlite": {"code": null, "error": null, "result": "success", "stage": "predict"},
"multicall.entities.100000@sqlite": {"code": null, "error": null, "result": "success", "stage": "materialize"},
"multicall.entities.100001@sqlite": {"code": "UNSUPPORTED_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"multicall.entities.100001@sqlite": {"code": null, "error": null, "result": "success", "stage": "materialize"},
"multicall.fail-call-0@sqlite": {"code": null, "error": "NimRequestError", "result": "refusal", "stage": "predict"},
"multicall.fail-call-2@sqlite": {"code": null, "error": "NimRequestError", "result": "refusal", "stage": "predict"},
"multicall.forecast.batched@sqlite": {"code": "UNSUPPORTED_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
Expand Down Expand Up @@ -160,7 +160,7 @@
"rank.candidates-not-a-ranking@sqlite": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.decimal-key@sqlite": {"code": null, "error": null, "result": "success", "stage": "predict"},
"rank.entities-by-candidates.at@sqlite": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.entities-by-candidates.over@sqlite": {"code": "UNSUPPORTED_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.entities-by-candidates.over@sqlite": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.most-parts@sqlite": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.no-candidates@sqlite": {"code": "INVALID_REQUEST", "error": "StructuredDataError", "result": "refusal", "stage": "materialize"},
"rank.odd-column@sqlite": {"code": null, "error": null, "result": "success", "stage": "predict"},
Expand Down
2 changes: 0 additions & 2 deletions contract/support/error_wording_scenarios.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,12 +65,10 @@
'sample.max-pq-iterations.zero',
'sample.num-hops.bool',
'sample.num-hops.fraction',
'sample.num-hops.seven',
'sample.num-hops.text',
'sample.num-hops.zero',
'sample.num-neighbors.empty',
'sample.num-neighbors.number',
'sample.num-neighbors.seven-hops',
'sample.seed.negative',
'shape.star.customers',
'target.filter.sum',
Expand Down
38 changes: 22 additions & 16 deletions contract/tests/test_prediction_batching.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,26 +65,10 @@
#: reference gives (the ceilings' counts, the 30 MiB body's mark).
REFUSED = {
**{case: ('INVALID_REQUEST', {}) for case in INVALID},
'multicall.entities.100001': (
'UNSUPPORTED_REQUEST',
{'entities': 100_001, 'gathered_times': 1, 'total': 100_001},
),
'multicall.calls.101': (
'UNSUPPORTED_REQUEST',
{'entities': 101, 'calls': 101, 'rows_per_call': 1},
),
'multicall.forecast.batched': (
'UNSUPPORTED_REQUEST',
{'entities': 8, 'rows_per_call': 3},
),
'multicall.body-30mib.one-row': (
'INVALID_REQUEST',
{'batch_too_large_to_send': True},
),
'multicall.body-30mib.several-rows': (
'INVALID_REQUEST',
{'batch_too_large_to_send': True},
),
}
#: The failed calls, by case: the call failed, and what the error carries
#: of the calls before it (``calls_answered`` and ``rows_answered``, or a
Expand All @@ -102,6 +86,14 @@
'the call: materialize(rows_per_call=N) below the 3 this one holds'
),
}
#: The cases this client departs from the reference on, on purpose: it holds a
#: request to no size of its own (no entity, call or body ceiling), so each is
#: gathered where the reference refused it at materialize.
DEPARTED = (
'multicall.entities.100001',
'multicall.calls.101',
*BODY,
)
CASE_IDS = (
'multicall.rows-per-call-3',
'multicall.fail-call-2',
Expand Down Expand Up @@ -404,6 +396,8 @@ def test_current_client_batches_without_label_or_entity_leakage(
database = case_key.partition('@')[2] or 'duckdb'
if database == 'postgres' and not _server():
pytest.skip('Postgres test DSN is not configured')
if case_key.split('@')[0] in DEPARTED:
pytest.skip('a departure from the reference, checked on its own')

def check() -> None:
run = core_run(database)
Expand Down Expand Up @@ -511,6 +505,8 @@ def test_batching_preserves_rows_limits_and_partial_answers(
case_id, _, database = case_key.partition('@')
if database == 'postgres' and not _server():
pytest.skip('Postgres test DSN is not configured')
if case_id in DEPARTED:
pytest.skip('a departure from the reference, checked on its own')
case, _ = cases()[case_id]
got = core_carried(case_key)
assert same_outcome(_expected(case), _outcome(got))
Expand Down Expand Up @@ -584,3 +580,13 @@ def test_transport_errors_keep_completed_answers(case_key: str) -> None:
steps = [1] if 'parts_answered' in RAISED[case_id] else None
assert_scored_rows(got['answered'], indices, steps)
assert got['result'] is None


@pytest.mark.parametrize('case_id', DEPARTED)
def test_python_core_gathers_and_predicts_what_the_reference_refused(
case_id: str,
) -> None:
r"""The client sets no most on the entities, the calls or the body: each
case is answered."""
got = core_carried(case_id)
assert got['outcome']['result'] == 'success', got['outcome']
2 changes: 0 additions & 2 deletions contract/tests/test_public_error_messages.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,12 +110,10 @@ def runner() -> ModuleType:
'sample.max-pq-iterations.zero': ['max_pq_iterations', '0'],
'sample.num-hops.bool': ['num_hops', 'True'],
'sample.num-hops.fraction': ['num_hops', '1.5'],
'sample.num-hops.seven': ['7 hops', '6'],
'sample.num-hops.text': ['num_hops', "'2'"],
'sample.num-hops.zero': ['num_hops', 'num_neighbors=[8, 8]'],
'sample.num-neighbors.empty': ['num_neighbors', 'no hop'],
'sample.num-neighbors.number': ['num_neighbors', 'sequence'],
'sample.num-neighbors.seven-hops': ['7 hops', '6'],
'sample.seed.negative': ['random_seed', '-1'],
'wording.rank-text-candidate': ['buys.item_id', "'1'"],
'wording.float32-classification': ['non-finite', 'inf'],
Expand Down
Loading
Loading