Repository navigation
fix(rfm): keep Snowflake TIME values as ISO text instead of NaT - #102
pcbeingused333 wants to merge 1 commit into
Conversation
A TIME column was mapped to Dtype.time, so materialising it ran every value through pd.to_datetime(errors='coerce'), which turned all times of day into NaT. It is now read as ISO text exact to the column's precision, nulls stay null, and a column of times of day is no longer inferred as a timestamp. Signed-off-by: Alex Castillo González <alex.castillog33@gmail.com>
|
No actionable comments were generated in the recent review. Walkthrough
Change: Bug fix · Severity of issue fixed: Medium Merge Risk:
|
Fixes #88
Materialising a Snowflake graph with a
TIMEcolumn replaced every value of that column withNaT, so a column of times of day reached the model empty.SnowTablemappedTIMEtoDtype.time, which is the timestamp data type.Table._sanitizethen sent the column throughto_datetime, andpd.to_datetime(..., errors='coerce')cannot place adatetime.timeon a timeline, so each value was coerced toNaT(with the "could not parse N value(s)" warning). A time of day has no date, so it is not a timestamp.TIMEis now mapped toDtype.string, and_sanitizeturns an Arrow time-of-day column into ISO text:HH:MM:SS, followed by the fraction of a second only where there is one, exact to the column's unit (seconds to nanoseconds). Nulls stay null.contains_timestampalso stops treating a column made only of times of day as a timestamp, sincepd.to_datetimewould otherwise place such text on today's date and infer a time column from it.Tests:
test_time_columns_are_kept_as_text_not_dropped(Snowflake graph with aTIMEcolumn holding nulls, for second, millisecond, microsecond and nanosecond precision: dtype, stype, no warning, and the sampled values) andtest_times_of_day_are_no_timestamp. On main these 5 fail (all valuesNaT/ inferred as timestamp); with the change they pass. Both suites from AGENTS.md pass: relational engine 2370 passed, 81 skipped; the other packages plus the roottests/2663 passed, 423 skipped, and one failure intest_databricks_helpers.pythat comes from the optionalpsycopgnot being installed in the test environment and is unrelated to this change.ruff check,ruff format --check,mypyon the engine, andcheck_repo_hygiene.py allare clean. A CHANGELOG entry is added under Unreleased / Fixed. The commit is signed off (DCO).The other backends that map
TIMEtoDtype.time(DuckDB, Postgres) are left as they are; this PR covers the Snowflake case in the issue.This change was prepared, and the tests above were run, with the assistance of an AI coding tool.