Skip to content

Stop enforcing client-side copies of server limits - #93

Merged
adi-kumo merged 6 commits into
v2-candidatefrom
awadaskar/raise-limits
Nov 11, 2026
Merged

adi-kumo merged 6 commits into
v2-candidatefrom
awadaskar/raise-limits

Conversation

@adi-kumo

@adi-kumo adi-kumo commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Deployments can raise their request limits, but hard-coded copies in the client still reject larger requests. This change lets the deployment enforce its configured limits while retaining supported client sampling boundaries.

Changes

  • Remove fixed client maxima for context rows, prediction rows per call, request bytes, table count, total materialized entities and call count.
  • Keep conservative defaults, including 10,000 prediction rows per batch and Tabular context sampling to 10,000 unless overridden.
  • Keep the six-hop sampling maximum, the Databricks 16 MB envelope guard, top_k ≤ 20, MOST_PARTS and the 64 MiB response cap.
  • Document that SDMClient.in_process() is also affected. Explicit larger inputs must fit available host and device memory.
  • Keep lag_timesteps under caller control and document its preparation cost. No automatic retries, splitting or new memory budget is added.

NIM can provide detailed limit errors. Databricks Model Serving sanitizes some error text. Gateway errors are not guaranteed to identify the server configuration variable.

Validation

The initial implementation's Rust, Python, Java and contract checks are recorded in the commit history. After restoring six-hop validation, local checks passed 51 Python limits/session tests, 18 Java relational tests, 19 Rust option tests, the materialization rejection test, schema checks, formatting and repository hygiene.

New live row-limit tests run a successful at-limit prediction before checking refusal above the limit. Their request fixtures were checked locally for both models. Live NIM/Databricks execution and payload-byte boundary validation are still pending rebuilt artifacts. This PR is not yet claiming those checks passed.

Supersedes #84.

@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
Pre-merge checks: Inconclusive 2

Failed Failed checks: 2 inconclusive

Check name Status Explanation Resolution
Title check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
Description check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@adi-kumo
adi-kumo marked this pull request as ready for review November 10, 2026 05:46
@adi-kumo
adi-kumo force-pushed the awadaskar/raise-limits branch from fd4fdcf to c9593fa Compare November 10, 2026 21:31
@adi-kumo
adi-kumo changed the base branch from v2-candidate to rust-fixture-cleanup November 10, 2026 21:31
Base automatically changed from rust-fixture-cleanup to v2-candidate November 10, 2026 21:54
The client kept its own copies of the NIM's size limits and refused
requests past them, so a deployment that raised its limits was still
held to the old ones. Remove those copies; the endpoint refuses what it
does not take.

- Relational: no 10,000-row context cap (num_context_samples may be
  larger), no 10,000 predict-rows cap (it stays the default batch, and
  rows_per_call may be larger), no 15-table or 6-hop cap, and no
  100,000-entity or 100-call guard on materialize(). Hops past the six
  sizes a run mode lists keep its last size, read lazily, and a walk
  ends at the first hop that reaches no row.
- Tabular: no refusal past 10,000 context rows; materialize() still
  samples to 10,000 by default, with a warning, unless
  num_context_samples is given. A direct Request is sent as given.
- No 30 MiB request-body refusal (at materialize, at send, or for lags).
- Keep the Databricks 16 MB check,
  top_k at most 20, MOST_PARTS and the 64 MiB response cap.
- Contract v73: drop the removed sentences, the check_tabular_cap
  operation and the most_entities / most_calls fields; regenerate the
  schema and lock. Record the contract cases that now depart from the
  reference.
- Update the tests that asserted the removed caps, limits.md, the
  skills and the CHANGELOG.

Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
@adi-kumo
adi-kumo force-pushed the awadaskar/raise-limits branch from c9593fa to 8c4f20a Compare November 10, 2026 21:56

@JGagacheva JGagacheva left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P0: commit ef0708dee ("Sample payload shapes from cases that stay small") is missing a Signed-off-by line. Unsigned commits can't be merged here. Please amend it with git commit --amend --signoff (or rebase with --signoff) and force-push.

Comment thread rust/crates/sdm-core/src/lag.rs Resolved
Comment thread rust/crates/sdm-request/src/options.rs Resolved
Comment thread CHANGELOG.md Resolved
Comment thread docs/reference/limits.md
| hops | relational | 6 | at `materialize()` |
| tables | relational | 15 | against the graph the request carries, not the schema it was built from |

A client opened with `SDMClient.in_process()` is held to this table as well,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: the in-process client loses all size checks. This paragraph said a client opened with SDMClient.in_process() is held to the client's limits. Those checks were its only ones, because there is no endpoint behind it to refuse anything. With them removed, an in-process run takes any context size, hop count or row count. Nothing has tested what the model does with, for example, a 50,000-row context on a laptop. Could the client keep its checks for the in-process backend, or replace them with limits that backend states itself? This also conflicts with #76: its in-process references say these limits still apply in-process.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description now distinguishes SDMClient.in_process() from the separate MLflow path. The in-process client is affected. Conservative defaults and input validation remain, but there is no fixed maximum row count based on assumed hardware. SDM has internal memory controls, not a guarantee that arbitrary inputs fit. docs/reference/limits.md now explains that explicit larger requests must fit available host and device memory. The old server-sized row cap is intentionally not restored.

With no client size limits, multicall.body-30mib.*, multicall.calls.101
and multicall.entities.100001 now materialize in full. Taking one of them
as a family's sample made the shape-mutation test walk a 30 MiB payload
and run for hours. Skip them; they carry no shape the others lack.

Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
@adi-kumo
adi-kumo force-pushed the awadaskar/raise-limits branch from 8c4f20a to 914cf48 Compare November 11, 2026 04:41
@adi-kumo

Copy link
Copy Markdown
Collaborator Author

The missing sign-off is fixed and pushed. Commit ef0708d is now 34041d6. Every PR commit has a Signed-off-by trailer. The sign-off rewrite changed no source files. Follow-up client fixes are in 914cf48. Live boundary validation remains in progress.

Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
@adi-kumo
adi-kumo merged commit ccdd261 into v2-candidate Nov 11, 2026
80 checks passed
@adi-kumo
adi-kumo deleted the awadaskar/raise-limits branch November 11, 2026 06:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants