Skip to content

fix(semantic): duplicate row labels, non-string column labels, tz-aware date checks - #363

Merged
kevincostner17 merged 1 commit into
mainfrom
fix/semantic-crossfield-labels
Sep 15, 2026
Merged

kevincostner17 merged 1 commit into
mainfrom
fix/semantic-crossfield-labels

Conversation

@kevincostner17

Copy link
Copy Markdown
Contributor

Summary

Semantic-module parts of three bundle issues.

  • Duplicate row index labels (Duplicate row index labels crash or overwrite rows in four APIs #231 part 1). The cross-field checks in semantic/consistency.py read cells with df.at[row, col] / series.at[row]. On a duplicated row label (common after pd.concat) that returns a Series.
    • The date-ordering check raised ValueError: The truth value of a Series is ambiguous and aborted fd.clean. The Fahrenheit check had the same flaw.
    • The timezone-transition and policy-duration checks silently missed findings.
    • All four now iterate positionally and report the original row labels.
    • The Fahrenheit check tests only the column maximum, the only value that can exceed every other value by more than 10. Results match the old per-row scan, which was O(n²); the new check is O(n).
  • Non-string column labels (Non-string column labels crash or silently misbehave in six APIs #232 part 1). The semantic context and backends name columns str(label), and run_semantic indexed the frame with that string, so integer labels (from read_csv(header=None) or df.T) raised KeyError.
    • run_semantic now maps names back to the real labels.
    • Distinct labels that stringify alike (1 and "1") can't be attributed by name. Those columns are left out of semantic repair, a report warning names them, and other columns are still repaired.
  • tz-aware vs naive datetimes (tz-aware vs naive datetimes crash contracts, semantic checks, domain checks and cdc_profile #233 part 1). _check_date_pair_ordering and _check_future_start_dates compared tz-aware and naive values and raised TypeError.
    • Parsed columns and the reference date are normalized to naive UTC, with naive values read as UTC.
    • Mixed-offset strings that parse to an object column are re-parsed with utc=True.
    • The warning still shows the declared reference date.

User-visible changes:

  • These frames now clean with semantic_mode enabled instead of raising.
  • One new report warning when labels stringify alike.
  • Timezone-transition and policy-duration findings are no longer missed on duplicate-label frames.

Tests

New module tests/test_semantic_label_and_tz_robustness.py:

  • Duplicate row labels: end-to-end fd.clean compared with a unique-index control, plus unit tests for date ordering, Fahrenheit (including a tied maximum), timezone transitions and policy durations.
  • Integer column labels: semantic repair is applied.
  • Labels that stringify alike: those columns are skipped with a warning, and other columns are still repaired.
  • tz handling:
    • tz-aware start vs naive end, both end-to-end and at the check level.
    • _as_datetime normalization for a single offset, mixed offsets and a tz-aware dtype.
    • Offset strings in the ordering check.
    • Future-start checks with a tz-aware column vs naive reference, and a naive column vs tz-aware reference.

Verification

  • ruff check .: all checks passed
  • mypy src/freshdata: no issues in 202 source files
  • py3.12 / pandas 2.3.3: 4254 passed, 6 skipped, 12 deselected
  • py3.9 / pandas 1.5.3: 4250 passed, 10 skipped, 12 deselected
  • Truthbench and golden expectations unchanged.

Refs #231 (part 1: semantic cross-field checks)
Refs #232 (part 1: run_semantic column labels)
Refs #233 (part 1: semantic date comparisons)

…and tz-mixed dates

Duplicate row index labels (#231): the cross-field checks read cells
with df.at[row, col] / series.at[row], which return a Series when a row
label is duplicated (common after pd.concat). The date-ordering check
then raised "truth value of a Series is ambiguous" and aborted fd.clean.
The Fahrenheit, timezone-transition and policy-duration checks share the
same pattern; they silently missed findings instead of crashing. All four
now iterate positionally and still report the original row labels. The
Fahrenheit check tests only the column maximum, which is the only value
that can exceed every other value by more than 10. The result is the same
as the old per-row scan, in O(n).

Non-string column labels (#232): the semantic context and every backend
name columns str(label), and run_semantic indexed the frame with that
string, so integer labels raised KeyError. run_semantic now maps each
name back to the real label. Distinct labels that stringify alike (1 and
"1") cannot be attributed by name, so those columns are left out of
semantic repair and a report warning names them.

tz-aware vs naive datetimes (#233): the date-ordering and future-start
checks compared a tz-aware column with a naive column or reference
timestamp, which raised TypeError. Parsed columns and the reference date
are now normalized to naive UTC, with naive values read as UTC. When
strings with mixed UTC offsets parse to an object column, parsing is
retried with utc=True. The reported reference date is still the declared
date.

Refs #231 (part 1: semantic cross-field checks)
Refs #232 (part 1: run_semantic column labels)
Refs #233 (part 1: semantic date comparisons)
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: ed1fad7e-0148-442d-a752-cbe8e5de663d


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

FreshData benchmark report — performance

  • freshdata: ?
  • python: ?
  • platform: ?
fixture n_rows n_cols p50 s p95 s peak MB repair % false-repair % preserve % trust monotonic export %

Authored-code reduction (Metric 6)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant