Skip to content

fix(ha-ws): correct cumulative sum handling on incremental import - #111

Open
jrozelle wants to merge 1 commit into
MyElectricalData:mainfrom
jrozelle:fix-ha-ws-cumulative-sum
Open

fix(ha-ws): correct cumulative sum handling on incremental import#111
jrozelle wants to merge 1 commit into
MyElectricalData:mainfrom
jrozelle:fix-ha-ws-cumulative-sum

Conversation

@jrozelle

Copy link
Copy Markdown

Summary

The HA WebSocket exporter (recorder/import_statistics) has 3 distinct bugs that together break the Energy dashboard for users running incremental imports. They compound: each one masks the next.

This PR fixes all three with a single localized change to home_assistant.py.

Bug 1: HA timestamps are milliseconds, not seconds

get_last_statistic_dates parses the start field of recorder/statistics_during_period responses with datetime.fromtimestamp(start_ts) — but HA core 2023+ returns these timestamps as a numeric epoch in milliseconds, not seconds.

Effect: OverflowError: year 58244 is out of range. The function falls through to the bare except, returns success: False, and the caller silently falls back to a full re-import on every scheduled run.

Fix: auto-detect ms vs s by magnitude (start_ts > 1e12 → ms). Safe until ~year 33658 in seconds.

Bug 2: Cumulative sums restart at 0 in incremental mode

_get_consumption_statistics_by_tariff, _get_cost_statistics_by_tariff and _get_production_statistics initialize their cumulative accumulators to 0.0 unconditionally. In full mode this is correct. In incremental mode, the new sums clobber the existing monotonic series in HA — the Energy dashboard sees a giant negative delta on the boundary hour (e.g. -11705 kWh on TEMPO blue_hp at the season transition) and stops displaying anything sensible.

Fix:

  • get_last_statistic_dates now also returns last_sums (extracted from the same statistics_during_period response, no extra round-trip).
  • A new helper _extract_per_pdl_state splits the flat {statistic_id: sum/date} maps into per-category buckets needed by the 3 export functions.
  • The 3 builders accept initial_sums (or initial_sum for production) and seed their cumulative accumulators with those values instead of 0.0.

Bug 3: Per-tariff records re-processed and double-counted

After fixing bug 2, the cumulative continuity worked but each new MED run shifted the series upward by the consumption delta between the global oldest_date and each tariff's own last_date.

Root cause: incremental mode uses ONE global since_date (the oldest across all tariffs) for the SQL fetch. But each individual tariff has its own newer last_date — and the previous code re-iterated all fetched records, re-importing entries that were already in HA and overwriting their sums with shifted-up values. After enough runs, totals are 2-3x the real value.

Fix: pass last_dates_by_tariff (and last_date for production) to the builders, and skip records whose timestamp is <= last_date for that tariff. The SQL fetch stays wide (global oldest) so every tariff still sees its new records, but per-tariff filtering inside the loop prevents re-processing.

Cost statistics inherit the filter for free since they're derived from the (already filtered) consumption_by_tariff.

Effect

In incremental mode, MED now correctly:

  • Reads existing sums from HA and uses them as cumulative starting points
  • Skips records that are already exported (per tariff)
  • Only ADDS strictly new records on top of the existing series
  • The sum stays monotonic, no compensation event in HA Energy

Test plan

Verified locally on a TEMPO production setup running for 26 months:

  • After the fix, no year 58244 error in MED logs (was happening every 30 min)
  • After the fix, MED reports Found last dates for 12/13 statistics and runs in incremental mode (not full-import on loop)
  • After the fix, sums stay monotonic across all 4 season-transition days (2026-03-31, 2026-04-01, 2026-04-03)
  • After the fix, the 12-month rolling total in HA Energy matches actual consumption (~10 MWh/year vs ~21 MWh that the bugs were producing)
  • After the fix, repeated incremental runs do not shift existing entries (verified by querying the recorder DB directly between runs)
  • Compose-build the patched container, confirm Python startup is clean

Note: this PR only contains the code fix. Existing users whose recorder DB was already corrupted by the bugs will need a one-shot data repair (purge med:* stats from HA recorder and let MED full-reimport). A separate runbook can be added if needed.

The HA WebSocket exporter (`recorder/import_statistics`) has 3 distinct
bugs that together break the Energy dashboard for users running incremental
imports. They compound: each one masks the next.

## Bug 1: HA timestamps are milliseconds, not seconds

`get_last_statistic_dates` parses the `start` field of
`recorder/statistics_during_period` responses with
`datetime.fromtimestamp(start_ts)` — but HA core 2023+ returns these
timestamps as a numeric epoch in MILLISECONDS, not seconds.

Effect: `OverflowError: year 58244 is out of range`. The function falls
through to the bare `except`, returns `success: False`, and the caller
silently falls back to a full re-import on every scheduled run.

Fix: auto-detect ms vs s by magnitude (`start_ts > 1e12 → ms`). Safe
until ~year 33658 in seconds.

## Bug 2: Cumulative sums restart at 0 in incremental mode

`_get_consumption_statistics_by_tariff`, `_get_cost_statistics_by_tariff`
and `_get_production_statistics` initialize their cumulative accumulators
to 0.0 unconditionally. In full mode this is correct. In incremental mode,
the new sums clobber the existing monotonic series in HA — the Energy
dashboard sees a giant negative delta on the boundary hour
(e.g. -11705 kWh on TEMPO blue_hp at the season transition) and stops
displaying anything sensible.

Fix:
- `get_last_statistic_dates` now also returns `last_sums` (extracted from
  the same `statistics_during_period` response, no extra round-trip).
- A new helper `_extract_per_pdl_state` splits the flat
  `{statistic_id: sum/date}` maps into per-category buckets needed by the
  3 export functions.
- The 3 builders accept `initial_sums` (or `initial_sum` for production)
  and seed their cumulative accumulators with those values instead of 0.0.

## Bug 3: Per-tariff records re-processed and double-counted

After fixing bug 2, the cumulative continuity worked but each new MED run
shifted the series upward by the consumption delta between the global
oldest_date and each tariff's own last_date.

Root cause: incremental mode uses ONE global `since_date` (the oldest
across all tariffs) for the SQL fetch. But each individual tariff has its
own newer last_date — and the previous code re-iterated all fetched
records, re-importing entries that were already in HA and overwriting
their sums with shifted-up values. After enough runs, totals are 2-3x
the real value.

Fix: pass `last_dates_by_tariff` (and `last_date` for production) to the
builders, and skip records whose timestamp is `<= last_date` for that
tariff. The SQL fetch stays wide (global oldest) so every tariff still
sees its new records, but per-tariff filtering inside the loop prevents
re-processing.

Cost statistics inherit the filter for free since they're derived from
the (already filtered) `consumption_by_tariff`.

## Effect

In incremental mode, MED now correctly:
- Reads existing sums from HA and uses them as cumulative starting points
- Skips records that are already exported (per tariff)
- Only ADDS strictly new records on top of the existing series
- The sum stays monotonic, no compensation event in HA Energy

Verified locally on a TEMPO production setup: after the fix, the Energy
dashboard shows correct deltas across all 4 season-transition days
(2026-03-31, 2026-04-01, 2026-04-03), and the 12-month rolling total
matches the user's actual consumption (~10 MWh/year).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant