Repository navigation
fix(timeseries): epoch and mixed-offset timestamps, MAD zero-scale, stable anomaly columns, cdc tz - #360
Merged
Conversation
…lags Time-series streaming parsed the timestamp column with a bare pd.to_datetime(errors="coerce"). Integer epoch seconds were read as nanoseconds (1970), unparseable strings silently became NaT, and a batch with mixed UTC offsets (a DST change) raised ValueError. - #227: numeric timestamp and event-time columns are read as epochs. The unit comes from the new TimeSeriesCleanConfig.timestamp_unit ("s"/"ms"/"us"/"ns"), or is inferred from the median magnitude (<1e11 s, <1e14 ms, <1e17 us, else ns); a "timeseries_timestamp_parse" action records the unit. Values that still fail to parse keep their row and are recorded in report.coerced_cells / coerced_rows (masked for sensitive_columns) with a medium-risk action and a report warning. - #250: string parsing goes through coerce_datetimes(). When the plain parse raises, returns an object column, or loses values, it retries with utc=True and, on pandas >= 2 (which infers one format from the first value), with format="ISO8601", keeping the attempt that loses the fewest values. Clean batches still pay for a single parse, and a single-offset or naive batch keeps its zone. - #290: anomaly_window_size=1 is rejected in the config ("must be 0 or >= 2") instead of failing inside pandas on the first batch. - #291: MAD treated a zero MAD as a flat window, so the minority value of any sparse or two-valued series was flagged and, with cap, overwritten by the median. A window is now flat only when every other point in it is identical. A zero MAD in a non-flat window gives no scale, so the point is not flagged and its fences are NaN. A spike in a truly flat window is still flagged and capped. - #248 part b: anomaly target columns are chosen from the (locked) numeric role, not from the batch dtype, so a numeric column holding a stray string still emits <col>_anomaly. It is scored with pd.to_numeric(errors="coerce"): non-numeric cells are neither flagged nor capped. Interpolation and seasonal imputation still require a numeric dtype. - #233 part 4: cdc_profile parses event times with the same coerce_datetimes() helper, so mixed offsets no longer crash cummax. The default now is the current UTC time instead of local wall-clock time for naive events. now= and watermark= are converted to the event times' zone, reading naive values (and naive events) as UTC. Closes #227 Closes #250 Closes #290 Closes #291 Refs #248 (part b: time-series anomaly columns) Refs #233 (part 4: cdc_profile)
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
kevincostner17
added a commit
that referenced
this pull request
Sep 15, 2026
#366) The generated-code sandbox reported a failed child with the last 800 characters of its stderr. faulthandler writes the crashing thread's stack first and a long "Extension modules: ..." line last, so a native crash was reported with only the module list and without the frame that located it. The finance/copilot case on PR #360 failed this way. Keep a faulthandler dump from its header, drop the module list line and cap it at 1500 characters. Other failures still keep the traceback tail.
This was referenced Sep 15, 2026
Merged
kevincostner17
added a commit
that referenced
this pull request
Sep 15, 2026
This was referenced Sep 15, 2026
Closed
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Time-series streaming parsed timestamps with a bare
pd.to_datetime(errors="coerce"). Epoch seconds became 1970, bad strings silently became NaT, and mixed UTC offsets crashed. This PR fixes that, two anomaly-detection bugs, and time zone handling incdc_profile.TimeSeriesCleanConfig.timestamp_unit(s/ms/us/ns), or is inferred from the median magnitude.timeseries_timestamp_parseaction records the unit used.report.coerced_cells/coerced_rows(masked forsensitive_columns), with a medium-risk action and a warning.coerce_datetimes(). If the plain parse raises, returns an object column, or loses values, it retries withutc=True, and on pandas ≥ 2 also withformat="ISO8601". It keeps the attempt that loses the fewest values. Clean batches parse once, and naive or single-offset batches keep their zone.anomaly_window_size=1is rejected when the config is built ("must be 0 or >= 2").<col>_anomaly. Values are scored withpd.to_numeric(errors="coerce"), so text cells are never flagged or capped.cdc_profiletime zones.coerce_datetimes, so mixed offsets no longer crash.nowis the current UTC time, not local time.now=andwatermark=are converted to the event times' zone, reading naive values as UTC.Behaviour changes:
anomaly_window_size=1now fails at config time.cdc_profilemeasures freshness against UTC by default.timestamp_unitand new report steptimeseries_timestamp_parse.Tests
tests/test_streaming_timeseries.py:timestamp_unitanomaly_window_size=1tests/test_cdc_profile.py:nowis UTC on a non-UTC hostnowand naivewatermarkwith aware events, and awarenowwith naive eventsmainon both lanes.Verification
ruff check .: all checks passedmypy src/freshdata: no issues (202 files)-m "not online and not large", py3.12 / pandas 2.3.3: 4263 passed, 6 skipped-m "not online and not large", py3.9 / pandas 1.5.3: 4259 passed, 10 skippedFound but not fixed here, as follow-ups:
anomaly_method="iqr"has the same zero-scale flaw as MAD anomaly detection flags and caps every minority value when the rolling MAD is 0 #291 when Q1 equals Q3.Closes #227
Closes #250
Closes #290
Closes #291
Refs #248 (part b: time-series anomaly columns)
Refs #233 (part 4: cdc_profile)