fix(ha-ws): correct cumulative sum handling on incremental import - #111
Open
jrozelle wants to merge 1 commit into
Open
fix(ha-ws): correct cumulative sum handling on incremental import#111jrozelle wants to merge 1 commit into
jrozelle wants to merge 1 commit into
Conversation
The HA WebSocket exporter (`recorder/import_statistics`) has 3 distinct
bugs that together break the Energy dashboard for users running incremental
imports. They compound: each one masks the next.
## Bug 1: HA timestamps are milliseconds, not seconds
`get_last_statistic_dates` parses the `start` field of
`recorder/statistics_during_period` responses with
`datetime.fromtimestamp(start_ts)` — but HA core 2023+ returns these
timestamps as a numeric epoch in MILLISECONDS, not seconds.
Effect: `OverflowError: year 58244 is out of range`. The function falls
through to the bare `except`, returns `success: False`, and the caller
silently falls back to a full re-import on every scheduled run.
Fix: auto-detect ms vs s by magnitude (`start_ts > 1e12 → ms`). Safe
until ~year 33658 in seconds.
## Bug 2: Cumulative sums restart at 0 in incremental mode
`_get_consumption_statistics_by_tariff`, `_get_cost_statistics_by_tariff`
and `_get_production_statistics` initialize their cumulative accumulators
to 0.0 unconditionally. In full mode this is correct. In incremental mode,
the new sums clobber the existing monotonic series in HA — the Energy
dashboard sees a giant negative delta on the boundary hour
(e.g. -11705 kWh on TEMPO blue_hp at the season transition) and stops
displaying anything sensible.
Fix:
- `get_last_statistic_dates` now also returns `last_sums` (extracted from
the same `statistics_during_period` response, no extra round-trip).
- A new helper `_extract_per_pdl_state` splits the flat
`{statistic_id: sum/date}` maps into per-category buckets needed by the
3 export functions.
- The 3 builders accept `initial_sums` (or `initial_sum` for production)
and seed their cumulative accumulators with those values instead of 0.0.
## Bug 3: Per-tariff records re-processed and double-counted
After fixing bug 2, the cumulative continuity worked but each new MED run
shifted the series upward by the consumption delta between the global
oldest_date and each tariff's own last_date.
Root cause: incremental mode uses ONE global `since_date` (the oldest
across all tariffs) for the SQL fetch. But each individual tariff has its
own newer last_date — and the previous code re-iterated all fetched
records, re-importing entries that were already in HA and overwriting
their sums with shifted-up values. After enough runs, totals are 2-3x
the real value.
Fix: pass `last_dates_by_tariff` (and `last_date` for production) to the
builders, and skip records whose timestamp is `<= last_date` for that
tariff. The SQL fetch stays wide (global oldest) so every tariff still
sees its new records, but per-tariff filtering inside the loop prevents
re-processing.
Cost statistics inherit the filter for free since they're derived from
the (already filtered) `consumption_by_tariff`.
## Effect
In incremental mode, MED now correctly:
- Reads existing sums from HA and uses them as cumulative starting points
- Skips records that are already exported (per tariff)
- Only ADDS strictly new records on top of the existing series
- The sum stays monotonic, no compensation event in HA Energy
Verified locally on a TEMPO production setup: after the fix, the Energy
dashboard shows correct deltas across all 4 season-transition days
(2026-03-31, 2026-04-01, 2026-04-03), and the 12-month rolling total
matches the user's actual consumption (~10 MWh/year).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The HA WebSocket exporter (
recorder/import_statistics) has 3 distinct bugs that together break the Energy dashboard for users running incremental imports. They compound: each one masks the next.This PR fixes all three with a single localized change to
home_assistant.py.Bug 1: HA timestamps are milliseconds, not seconds
get_last_statistic_datesparses thestartfield ofrecorder/statistics_during_periodresponses withdatetime.fromtimestamp(start_ts)— but HA core 2023+ returns these timestamps as a numeric epoch in milliseconds, not seconds.Effect:
OverflowError: year 58244 is out of range. The function falls through to the bareexcept, returnssuccess: False, and the caller silently falls back to a full re-import on every scheduled run.Fix: auto-detect ms vs s by magnitude (
start_ts > 1e12 → ms). Safe until ~year 33658 in seconds.Bug 2: Cumulative sums restart at 0 in incremental mode
_get_consumption_statistics_by_tariff,_get_cost_statistics_by_tariffand_get_production_statisticsinitialize their cumulative accumulators to0.0unconditionally. In full mode this is correct. In incremental mode, the new sums clobber the existing monotonic series in HA — the Energy dashboard sees a giant negative delta on the boundary hour (e.g. -11705 kWh on TEMPO blue_hp at the season transition) and stops displaying anything sensible.Fix:
get_last_statistic_datesnow also returnslast_sums(extracted from the samestatistics_during_periodresponse, no extra round-trip)._extract_per_pdl_statesplits the flat{statistic_id: sum/date}maps into per-category buckets needed by the 3 export functions.initial_sums(orinitial_sumfor production) and seed their cumulative accumulators with those values instead of0.0.Bug 3: Per-tariff records re-processed and double-counted
After fixing bug 2, the cumulative continuity worked but each new MED run shifted the series upward by the consumption delta between the global
oldest_dateand each tariff's ownlast_date.Root cause: incremental mode uses ONE global
since_date(the oldest across all tariffs) for the SQL fetch. But each individual tariff has its own newerlast_date— and the previous code re-iterated all fetched records, re-importing entries that were already in HA and overwriting their sums with shifted-up values. After enough runs, totals are 2-3x the real value.Fix: pass
last_dates_by_tariff(andlast_datefor production) to the builders, and skip records whose timestamp is<= last_datefor that tariff. The SQL fetch stays wide (global oldest) so every tariff still sees its new records, but per-tariff filtering inside the loop prevents re-processing.Cost statistics inherit the filter for free since they're derived from the (already filtered)
consumption_by_tariff.Effect
In incremental mode, MED now correctly:
Test plan
Verified locally on a TEMPO production setup running for 26 months:
year 58244error in MED logs (was happening every 30 min)Found last dates for 12/13 statisticsand runs in incremental mode (not full-import on loop)Note: this PR only contains the code fix. Existing users whose recorder DB was already corrupted by the bugs will need a one-shot data repair (purge
med:*stats from HA recorder and let MED full-reimport). A separate runbook can be added if needed.