Repository navigation
Stop enforcing client-side copies of server limits - #93
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
fd4fdcf to
c9593fa
Compare
The client kept its own copies of the NIM's size limits and refused requests past them, so a deployment that raised its limits was still held to the old ones. Remove those copies; the endpoint refuses what it does not take. - Relational: no 10,000-row context cap (num_context_samples may be larger), no 10,000 predict-rows cap (it stays the default batch, and rows_per_call may be larger), no 15-table or 6-hop cap, and no 100,000-entity or 100-call guard on materialize(). Hops past the six sizes a run mode lists keep its last size, read lazily, and a walk ends at the first hop that reaches no row. - Tabular: no refusal past 10,000 context rows; materialize() still samples to 10,000 by default, with a warning, unless num_context_samples is given. A direct Request is sent as given. - No 30 MiB request-body refusal (at materialize, at send, or for lags). - Keep the Databricks 16 MB check, top_k at most 20, MOST_PARTS and the 64 MiB response cap. - Contract v73: drop the removed sentences, the check_tabular_cap operation and the most_entities / most_calls fields; regenerate the schema and lock. Record the contract cases that now depart from the reference. - Update the tests that asserted the removed caps, limits.md, the skills and the CHANGELOG. Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
c9593fa to
8c4f20a
Compare
JGagacheva
left a comment
There was a problem hiding this comment.
P0: commit ef0708dee ("Sample payload shapes from cases that stay small") is missing a Signed-off-by line. Unsigned commits can't be merged here. Please amend it with git commit --amend --signoff (or rebase with --signoff) and force-push.
| | hops | relational | 6 | at `materialize()` | | ||
| | tables | relational | 15 | against the graph the request carries, not the schema it was built from | | ||
|
|
||
| A client opened with `SDMClient.in_process()` is held to this table as well, |
There was a problem hiding this comment.
P1: the in-process client loses all size checks. This paragraph said a client opened with SDMClient.in_process() is held to the client's limits. Those checks were its only ones, because there is no endpoint behind it to refuse anything. With them removed, an in-process run takes any context size, hop count or row count. Nothing has tested what the model does with, for example, a 50,000-row context on a laptop. Could the client keep its checks for the in-process backend, or replace them with limits that backend states itself? This also conflicts with #76: its in-process references say these limits still apply in-process.
There was a problem hiding this comment.
The PR description now distinguishes SDMClient.in_process() from the separate MLflow path. The in-process client is affected. Conservative defaults and input validation remain, but there is no fixed maximum row count based on assumed hardware. SDM has internal memory controls, not a guarantee that arbitrary inputs fit. docs/reference/limits.md now explains that explicit larger requests must fit available host and device memory. The old server-sized row cap is intentionally not restored.
With no client size limits, multicall.body-30mib.*, multicall.calls.101 and multicall.entities.100001 now materialize in full. Taking one of them as a family's sample made the shape-mutation test walk a 30 MiB payload and run for hours. Skip them; they carry no shape the others lack. Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
8c4f20a to
914cf48
Compare
Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
Why
Deployments can raise their request limits, but hard-coded copies in the client still reject larger requests. This change lets the deployment enforce its configured limits while retaining supported client sampling boundaries.
Changes
NIM can provide detailed limit errors. Databricks Model Serving sanitizes some error text. Gateway errors are not guaranteed to identify the server configuration variable.
Validation
The initial implementation's Rust, Python, Java and contract checks are recorded in the commit history. After restoring six-hop validation, local checks passed 51 Python limits/session tests, 18 Java relational tests, 19 Rust option tests, the materialization rejection test, schema checks, formatting and repository hygiene.
New live row-limit tests run a successful at-limit prediction before checking refusal above the limit. Their request fixtures were checked locally for both models. Live NIM/Databricks execution and payload-byte boundary validation are still pending rebuilt artifacts. This PR is not yet claiming those checks passed.
Supersedes #84.