Skip to content

fix(rfm): keep Snowflake TIME values as ISO text instead of NaT - #102

Open
pcbeingused333 wants to merge 1 commit into
NVIDIA:mainfrom
pcbeingused333:fix/snowflake-time-text
Open

pcbeingused333 wants to merge 1 commit into
NVIDIA:mainfrom
pcbeingused333:fix/snowflake-time-text

Conversation

@pcbeingused333

Copy link
Copy Markdown

Fixes #88

Materialising a Snowflake graph with a TIME column replaced every value of that column with NaT, so a column of times of day reached the model empty.

SnowTable mapped TIME to Dtype.time, which is the timestamp data type. Table._sanitize then sent the column through to_datetime, and pd.to_datetime(..., errors='coerce') cannot place a datetime.time on a timeline, so each value was coerced to NaT (with the "could not parse N value(s)" warning). A time of day has no date, so it is not a timestamp.

TIME is now mapped to Dtype.string, and _sanitize turns an Arrow time-of-day column into ISO text: HH:MM:SS, followed by the fraction of a second only where there is one, exact to the column's unit (seconds to nanoseconds). Nulls stay null. contains_timestamp also stops treating a column made only of times of day as a timestamp, since pd.to_datetime would otherwise place such text on today's date and infer a time column from it.

Tests: test_time_columns_are_kept_as_text_not_dropped (Snowflake graph with a TIME column holding nulls, for second, millisecond, microsecond and nanosecond precision: dtype, stype, no warning, and the sampled values) and test_times_of_day_are_no_timestamp. On main these 5 fail (all values NaT / inferred as timestamp); with the change they pass. Both suites from AGENTS.md pass: relational engine 2370 passed, 81 skipped; the other packages plus the root tests/ 2663 passed, 423 skipped, and one failure in test_databricks_helpers.py that comes from the optional psycopg not being installed in the test environment and is unrelated to this change. ruff check, ruff format --check, mypy on the engine, and check_repo_hygiene.py all are clean. A CHANGELOG entry is added under Unreleased / Fixed. The commit is signed off (DCO).

The other backends that map TIME to Dtype.time (DuckDB, Postgres) are left as they are; this PR covers the Snowflake case in the issue.

This change was prepared, and the tests above were run, with the assistance of an AI coding tool.

A TIME column was mapped to Dtype.time, so materialising it ran every
value through pd.to_datetime(errors='coerce'), which turned all times of
day into NaT. It is now read as ISO text exact to the column's precision,
nulls stay null, and a column of times of day is no longer inferred as a
timestamp.

Signed-off-by: Alex Castillo González <alex.castillog33@gmail.com>
@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Snowflake TIME columns now use string dtype. Arrow time values are formatted as text with nulls and fractional seconds preserved. Timestamp inference excludes sampled values that all match the time-of-day pattern.

Changes: Time value handling

Layer / File(s) Summary
Map and sanitize Snowflake TIME values
packages/structured-data-models-relational-engine/src/sdm_relational_engine/rfm/backend/snow/table.py, packages/structured-data-models-relational-engine/src/sdm_relational_engine/rfm/base/table.py, packages/structured-data-models-relational-engine/tests/backend/snow/test_snow_graph.py, CHANGELOG.md
Snowflake TIME types map to string dtype. Arrow time values in string columns are formatted as text with nulls preserved. Parameterized tests check second, millisecond, microsecond, and nanosecond precision. The changelog records the behavior.
Exclude time-of-day strings from timestamp inference
packages/structured-data-models-relational-engine/src/sdm_relational_engine/rfm/metadata_inference/timestamp.py, packages/structured-data-models-relational-engine/tests/metadata_inference/test_timestamp.py
Timestamp inference returns false when all remaining sampled values match the time-of-day pattern. The test includes fractional seconds and a null value.

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: Minimal · up to c5291

Snowflake TIME columns are now kept as ISO text instead of being turned into missing values, and time-only strings are no longer inferred as timestamps. Regression tests cover the changed behavior, and no merge-blocking risk is identified.

Security architecture risk: Low · up to c5291

TIME columns intentionally become text, and time-only samples no longer infer as timestamps. These changes can affect consumers of column metadata, but the reviewed paths do not introduce new data-access or execution authority.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The demonstrated exposure is data-dependent representation and metadata classification within already retrieved samples. Because timestamp inference is shared, its changed classification is not limited to Snowflake, but no reviewed path turns that classification into additional access authority.

Trust Boundaries and Controls

  • observed — Retrieval continues to use the existing Snowflake connection and query paths. The changed logic operates after fetch and does not add a connection, credential source, query-execution capability, or authorization decision.

Resilience and Maintainability Implications

  • inferred — On the traced Snowflake sample path, conversion operates on a freshly materialized frame before cache publication, containing partial-conversion failures locally. Repetition skips the new branch once the column becomes Arrow string. Sanitization remains sequential and in-place, so this does not establish whole-frame atomicity for arbitrary external callers supplying shared frames.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: Snowflake TIME values are kept as ISO text instead of being converted to NaT. It matches the changeset (SnowTable dtype mapping, Table._sanitize ISO conversion, and co…
Description check Passed The description states the bug, the cause (TIME mapped to Dtype.time, then coerced to NaT by to_datetime), and the fix (Dtype.string plus ISO text conversion and a contains_timestamp change). It also …
Linked Issues check Passed Issue #88 (direct, open) requires that Snowflake TIME values not be dropped as missing. It accepts text as a model-compatible form. The PR maps Snowflake TIME to Dtype.string in SnowTable._to_dtype, s…
Out of Scope Changes check Passed All changes connect to issue #88. The changes are: the TIME dtype mapping in snow/table.py; the time-of-day conversion in rfm/base/table.py; the time-of-day exclusion in metadata_inference/timestamp.p…
Full details: Title check

Explanation

The title names the main change: Snowflake TIME values are kept as ISO text instead of being converted to NaT. It matches the changeset (SnowTable dtype mapping, Table._sanitize ISO conversion, and contains_timestamp change) and the linked objectives. It uses the conventional fix(rfm) prefix and is 63 characters long.

Full details: Description check

Explanation

The description states the bug, the cause (TIME mapped to Dtype.time, then coerced to NaT by to_datetime), and the fix (Dtype.string plus ISO text conversion and a contains_timestamp change). It also lists the tests, the verification results, and the scope limit to Snowflake. It is directly related to the changeset.

Full details: Linked Issues check

Explanation

Issue #88 (direct, open) requires that Snowflake TIME values not be dropped as missing. It accepts text as a model-compatible form. The PR maps Snowflake TIME to Dtype.string in SnowTable._to_dtype, so the column no longer goes through to_datetime. Table._sanitize converts Arrow time-of-day columns to ISO text (HH:MM:SS, with fractional seconds at the column's unit) and keeps nulls null. contains_timestamp no longer infers a timestamp for columns made only of times of day. The test test_time_columns_are_kept_as_text_not_dropped covers second, millisecond, microsecond and nanosecond precision, and checks dtype, stype, absence of warnings and sampled values. The PR reports that these tests fail on main and pass with the change. The raw summary is consistent with this. Verification relies on the PR description and summary, not on a repository read of the final code.

Full details: Out of Scope Changes check

Explanation

All changes connect to issue #88. The changes are: the TIME dtype mapping in snow/table.py; the time-of-day conversion in rfm/base/table.py; the time-of-day exclusion in metadata_inference/timestamp.py; the Snowflake regression tests; the timestamp inference test; and the CHANGELOG entry. The timestamp inference change supports the requirement that a time-of-day text column is not inferred as a timestamp. The CHANGELOG entry is documentation for the fix. DuckDB and Postgres mappings are unchanged, as the PR states. This matches the Snowflake scope of #88. No unrelated change is evident from the summary.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 0f95e84 and c52919b.

Files selected for processing: 6

  • CHANGELOG.md
  • packages/structured-data-models-relational-engine/src/sdm_relational_engine/rfm/backend/snow/table.py
  • packages/structured-data-models-relational-engine/src/sdm_relational_engine/rfm/base/table.py
  • packages/structured-data-models-relational-engine/src/sdm_relational_engine/rfm/metadata_inference/timestamp.py
  • packages/structured-data-models-relational-engine/tests/backend/snow/test_snow_graph.py
  • packages/structured-data-models-relational-engine/tests/metadata_inference/test_timestamp.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Snowflake TIME columns lose every value

1 participant