Skip to content

Gather over frames without the warehouse package - #101

Merged
Manushpm8 merged 2 commits into
v2-candidatefrom
frames-gather-without-connectors
Nov 12, 2026
Merged

Manushpm8 merged 2 commits into
v2-candidatefrom
frames-gather-without-connectors

Conversation

@Manushpm8

Copy link
Copy Markdown
Collaborator

What breaks

On a client installed without a driver extra, a relational gather over tables
already in memory fails:

ModuleNotFoundError: No module named 'sdm_connectors'

Reproduced on a clean install of this branch's base, with no connectors
package present:

tabular over frames:     ok; sdm_connectors imported = False
relational schema.source = None
relational over frames:  ModuleNotFoundError: No module named 'sdm_connectors'

The chain, every link at module scope:

materializers/relational.py:76    from sdm_client._rust.frame_materialize import gather
_rust/frame_materialize.py:17     from sdm_client._rust.materialize import (...)
_rust/materialize.py:21           from sdm_client._rust import codec, errors, providers, results
_rust/providers/__init__.py:20    from sdm_client._rust.providers.databricks import (...)
_rust/providers/databricks.py:45  from sdm_connectors import providers

relational.py:76 is the branch taken when schema.source is None, so the
route that reads nothing from a warehouse is the one that required the
warehouse package. structured-data-models-connectors is in no base
dependency of the client, and every extra that brings it also brings a driver
this route never uses, so there is nothing a caller can install to mean "I
have frames".

Why it is a bug and not a dependency

_rust/run.py:46 keeps its own import of the package under TYPE_CHECKING,
saying so: the warehouse package is an optional dependency, and a core route
over frames needs none of it
. tests/test_core_route_without_connectors.py
already asserted that invariant by name. It only ran the tabular route, whose
materializer builds the request directly and never reaches the gather, so the
only route that could break the rule was the only route the test did not run.

The change

_rust/providers/__init__.py imports a provider module on the first read of
its name instead of when the package is imported. for_source resolves the
name through the module's own attribute lookup, so a test that binds a
provider still wins; a first attempt that called __getattr__ directly
bypassed module globals and broke 32 tests that monkeypatch
providers.SnowflakeProvider and friends.

__all__, from-imports and the AttributeError for an unknown name are
unchanged.

The test now runs both routes, parametrised.

Checks

  • The new [relational] case fails on the base and passes with the change;
    [tabular] passes either way.
  • Client suite with the change: 1 failed, 4356 passed. Without it, same
    suite: 1 failed, 4355 passed. The failure diff is empty, and that one
    failure (test_materialize_payloads timestamptz) is pre-existing on
    v2-candidate.
  • ruff check, ruff format --check, mypy over the three src trees, all
    five hygiene checks, and the architecture-policy and tooling tests
    (170 passed) are green.

Found while verifying the in-process skill references in #76, which had to
tell readers to install the connectors package to work around this.

🤖 Generated with Claude Code

A schema built from frames names no source, so the relational gather takes
the branch that reads nothing from a warehouse. It still imported the data
providers, and each of those imports sdm_connectors at module scope, so a
gather over tables already in memory raised ModuleNotFoundError on a client
installed without a driver extra. The package is in no base dependency, and
every extra carrying it also carries a driver the route never uses.

run.py already names the rule this breaks, keeping its own import of the
package under TYPE_CHECKING because a core route over frames needs none of
it, and test_core_route_without_connectors.py already asserted it. That test
only ran the tabular route, whose materializer builds the request directly
and never reaches the gather, so the one route that could break it was the
one route it did not cover.

The providers module now imports a provider on the first read of its name,
through the module's own attribute lookup so a test that binds one still
wins, and the test runs both routes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Manush Murali <manushm@nvidia.com>
@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The provider package now loads provider classes when accessed instead of importing them eagerly. Parametrized subprocess tests cover tabular and relational query materialization while blocking imports of sdm_connectors.

Changes: Provider imports and route tests

Layer / File(s) Summary
Resolve provider classes on access
packages/structured-data-models-client/src/sdm_client/_rust/providers/__init__.py
The package maps provider class names to their defining modules and imports a module when its class is accessed. for_source retrieves the selected provider through module attribute lookup.
Check tabular and relational routes
packages/structured-data-models-client/tests/test_core_route_without_connectors.py
Parametrized subprocess tests block sdm_connectors imports, materialize tabular and relational requests, and check that the package is absent from sys.modules.

Change: Bug fix

Merge Risk: Minimal · up to 48dea

The change defers provider imports so in-memory relational gathering works without the optional connectors package. No concrete merge-blocking risk was identified, and the new tests cover both the tabular and relational routes.

Security architecture risk: Low · up to 48dea

Warehouse dependencies are now loaded only when needed. The reviewed changes do not broaden database access, but verification coverage remains incomplete.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — Within the reviewed dispatch path, source-controlled kind values cannot select additional import targets or gain a different connection authority. Both revisions select the same provider classes through the same fixed source-kind table.

Trust Boundaries and Controls

  • observed — The lazy importer rejects names outside its fixed map before constructing an import path. for_source derives the requested class name from the existing source-kind map rather than accepting an arbitrary module name.

Resilience and Maintainability Implications

  • inferred — A failed class import does not reach provider construction or create a package-level class cache. Successful repeated lookup uses normal module caching. SQLite's existing connection-scoped temporary-table ownership registry remains in its unchanged provider module; lazy lookup does not introduce a separate registry or reset it on each read.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title "Gather over frames without the warehouse package" matches the main change: relational gathering over in-memory frames no longer requires the optional sdm_connectors package. The change defe…
Description check Passed The description explains the bug (ModuleNotFoundError for sdm_connectors on relational gathers over in-memory tables), traces the import chain, and describes the fix (lazy provider imports via __getat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "Gather over frames without the warehouse package" matches the main change: relational gathering over in-memory frames no longer requires the optional sdm_connectors package. The change defers provider imports in _rust/providers/init.py, and the test now covers the relational route without connectors. The title is short, specific, and contains no file lists or vague terms.

Full details: Description check

Explanation

The description explains the bug (ModuleNotFoundError for sdm_connectors on relational gathers over in-memory tables), traces the import chain, and describes the fix (lazy provider imports via getattr and for_source lookup). It also covers the test changes and the verification results. It relates directly to the changeset.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 1b8ef13 and 48deae3.

Files selected for processing: 2

  • packages/structured-data-models-client/src/sdm_client/_rust/providers/__init__.py
  • packages/structured-data-models-client/tests/test_core_route_without_connectors.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

The lazy read bound the class it imported into this module's globals. A
test that swaps a driver out and back imports the provider module again
inside that window, so the binding outlived the module it came from and a
later run answered with a class the swap had made. Python's module cache
makes the repeated import a dictionary read.

Signed-off-by: Manush Murali <manushm@nvidia.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 12, 2026

Copy link
Copy Markdown

Add Carrot credits or activate Agent usage billing to use Autofix

@Manushpm8
Manushpm8 merged commit 693647f into v2-candidate Nov 12, 2026
80 checks passed
@Manushpm8
Manushpm8 deleted the frames-gather-without-connectors branch November 12, 2026 05:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant