Repository navigation
fix(semantic): duplicate row labels, non-string column labels, tz-aware date checks - #363
Merged
Merged
Conversation
…and tz-mixed dates Duplicate row index labels (#231): the cross-field checks read cells with df.at[row, col] / series.at[row], which return a Series when a row label is duplicated (common after pd.concat). The date-ordering check then raised "truth value of a Series is ambiguous" and aborted fd.clean. The Fahrenheit, timezone-transition and policy-duration checks share the same pattern; they silently missed findings instead of crashing. All four now iterate positionally and still report the original row labels. The Fahrenheit check tests only the column maximum, which is the only value that can exceed every other value by more than 10. The result is the same as the old per-row scan, in O(n). Non-string column labels (#232): the semantic context and every backend name columns str(label), and run_semantic indexed the frame with that string, so integer labels raised KeyError. run_semantic now maps each name back to the real label. Distinct labels that stringify alike (1 and "1") cannot be attributed by name, so those columns are left out of semantic repair and a report warning names them. tz-aware vs naive datetimes (#233): the date-ordering and future-start checks compared a tz-aware column with a naive column or reference timestamp, which raised TypeError. Parsed columns and the reference date are now normalized to naive UTC, with naive values read as UTC. When strings with mixed UTC offsets parse to an object column, parsing is retried with utc=True. The reported reference date is still the declared date. Refs #231 (part 1: semantic cross-field checks) Refs #232 (part 1: run_semantic column labels) Refs #233 (part 1: semantic date comparisons)
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
kevincostner17
added a commit
that referenced
this pull request
Sep 15, 2026
This was referenced Sep 15, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Semantic-module parts of three bundle issues.
semantic/consistency.pyread cells withdf.at[row, col]/series.at[row]. On a duplicated row label (common afterpd.concat) that returns a Series.ValueError: The truth value of a Series is ambiguousand abortedfd.clean. The Fahrenheit check had the same flaw.str(label), andrun_semanticindexed the frame with that string, so integer labels (fromread_csv(header=None)ordf.T) raisedKeyError.run_semanticnow maps names back to the real labels.1and"1") can't be attributed by name. Those columns are left out of semantic repair, a report warning names them, and other columns are still repaired._check_date_pair_orderingand_check_future_start_datescompared tz-aware and naive values and raisedTypeError.utc=True.User-visible changes:
semantic_modeenabled instead of raising.Tests
New module
tests/test_semantic_label_and_tz_robustness.py:fd.cleancompared with a unique-index control, plus unit tests for date ordering, Fahrenheit (including a tied maximum), timezone transitions and policy durations._as_datetimenormalization for a single offset, mixed offsets and a tz-aware dtype.Verification
ruff check .: all checks passedmypy src/freshdata: no issues in 202 source filesRefs #231 (part 1: semantic cross-field checks)
Refs #232 (part 1: run_semantic column labels)
Refs #233 (part 1: semantic date comparisons)