Skip to content

fix(timeseries): epoch and mixed-offset timestamps, MAD zero-scale, stable anomaly columns, cdc tz - #360

Merged
kevincostner17 merged 1 commit into
mainfrom
fix/timeseries-event-time
Sep 15, 2026
Merged

kevincostner17 merged 1 commit into
mainfrom
fix/timeseries-event-time

Conversation

@kevincostner17

Copy link
Copy Markdown
Contributor

Summary

Time-series streaming parsed timestamps with a bare pd.to_datetime(errors="coerce"). Epoch seconds became 1970, bad strings silently became NaT, and mixed UTC offsets crashed. This PR fixes that, two anomaly-detection bugs, and time zone handling in cdc_profile.

Behaviour changes:

  • Numeric timestamp columns produce real dates instead of 1970.
  • Unparseable timestamps are now reported.
  • Mixed-offset batches become a UTC-aware column.
  • anomaly_window_size=1 now fails at config time.
  • MAD no longer flags the minority value of two-valued series.
  • cdc_profile measures freshness against UTC by default.
  • New config field timestamp_unit and new report step timeseries_timestamp_parse.

Tests

  • tests/test_streaming_timeseries.py:
    • epoch unit inference (s/ms/us/ns), float and nullable epochs, explicit timestamp_unit
    • unparseable and out-of-range timestamps are reported and their rows kept; sensitive masking
    • DST mixed offsets, and a naive + aware mix
    • epoch event-time late data
    • anomaly_window_size=1
    • MAD on a two-valued series, and a flat-window spike that is still capped
    • stable anomaly columns when a batch dtype changes; text cells neither scored nor capped
  • tests/test_cdc_profile.py:
    • default now is UTC on a non-UTC host
    • naive now and naive watermark with aware events, and aware now with naive events
    • mixed offsets normalised, with ordering still detected
  • 22 of the new tests fail on main on both lanes.

Verification

  • ruff check .: all checks passed
  • mypy src/freshdata: no issues (202 files)
  • pytest -m "not online and not large", py3.12 / pandas 2.3.3: 4263 passed, 6 skipped
  • pytest -m "not online and not large", py3.9 / pandas 1.5.3: 4259 passed, 10 skipped

Found but not fixed here, as follow-ups:

Closes #227
Closes #250
Closes #290
Closes #291
Refs #248 (part b: time-series anomaly columns)
Refs #233 (part 4: cdc_profile)

…lags

Time-series streaming parsed the timestamp column with a bare
pd.to_datetime(errors="coerce"). Integer epoch seconds were read as
nanoseconds (1970), unparseable strings silently became NaT, and a batch
with mixed UTC offsets (a DST change) raised ValueError.

- #227: numeric timestamp and event-time columns are read as epochs. The
  unit comes from the new TimeSeriesCleanConfig.timestamp_unit
  ("s"/"ms"/"us"/"ns"), or is inferred from the median magnitude
  (<1e11 s, <1e14 ms, <1e17 us, else ns); a "timeseries_timestamp_parse"
  action records the unit. Values that still fail to parse keep their row
  and are recorded in report.coerced_cells / coerced_rows (masked for
  sensitive_columns) with a medium-risk action and a report warning.
- #250: string parsing goes through coerce_datetimes(). When the plain
  parse raises, returns an object column, or loses values, it retries with
  utc=True and, on pandas >= 2 (which infers one format from the first
  value), with format="ISO8601", keeping the attempt that loses the fewest
  values. Clean batches still pay for a single parse, and a single-offset
  or naive batch keeps its zone.
- #290: anomaly_window_size=1 is rejected in the config ("must be 0 or
  >= 2") instead of failing inside pandas on the first batch.
- #291: MAD treated a zero MAD as a flat window, so the minority value of
  any sparse or two-valued series was flagged and, with cap, overwritten
  by the median. A window is now flat only when every other point in it
  is identical. A zero MAD in a non-flat window gives no scale, so the
  point is not flagged and its fences are NaN. A spike in a truly flat
  window is still flagged and capped.
- #248 part b: anomaly target columns are chosen from the (locked)
  numeric role, not from the batch dtype, so a numeric column holding a
  stray string still emits <col>_anomaly. It is scored with
  pd.to_numeric(errors="coerce"): non-numeric cells are neither flagged
  nor capped. Interpolation and seasonal imputation still require a
  numeric dtype.
- #233 part 4: cdc_profile parses event times with the same
  coerce_datetimes() helper, so mixed offsets no longer crash cummax. The
  default now is the current UTC time instead of local wall-clock time
  for naive events. now= and watermark= are converted to the event
  times' zone, reading naive values (and naive events) as UTC.

Closes #227
Closes #250
Closes #290
Closes #291
Refs #248 (part b: time-series anomaly columns)
Refs #233 (part 4: cdc_profile)
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 09e5bd64-cc59-4278-8f68-22ff1bbc0589


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

FreshData benchmark report — performance

  • freshdata: ?
  • python: ?
  • platform: ?
fixture n_rows n_cols p50 s p95 s peak MB repair % false-repair % preserve % trust monotonic export %

Authored-code reduction (Metric 6)

@kevincostner17
kevincostner17 merged commit b93aefc into main Sep 15, 2026
32 of 33 checks passed
kevincostner17 added a commit that referenced this pull request Sep 15, 2026
#366)

The generated-code sandbox reported a failed child with the last 800
characters of its stderr. faulthandler writes the crashing thread's stack
first and a long "Extension modules: ..." line last, so a native crash
was reported with only the module list and without the frame that
located it. The finance/copilot case on PR #360 failed this way.

Keep a faulthandler dump from its header, drop the module list line and
cap it at 1500 characters. Other failures still keep the traceback tail.
kevincostner17 added a commit that referenced this pull request Sep 15, 2026
Add Unreleased Added, Changed and Fixed entries for merged PRs #351,
#352, #353, #354, #356, #357, #358, #360, #361, #362, #363, #364, #365,
#367, #368, #369, #370, #371, #372, #374, #375, #377, #378 and #379.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant