Load prediction inflated ~1.5x due to fill_load_from_power double-counting on sparse data

[mgazza]

Summary

When entity history data is recorded at intervals greater than 1 minute (e.g. 5-minute intervals), fill_load_from_power in fetch.py double-counts energy, inflating load predictions by approximately 1.3-1.6x actual consumption.

This results in:

• predbat.load_energy predicting significantly more than actual consumption
• Inflated savings_yesterday_predbat values
• Cumulative savings_total_predbat growing too fast

Root Cause

Data Sparsity

When history data is recorded at 5-minute intervals (rather than HA's sub-second state change resolution), clean_incrementing_reverse produces a dict with entries at only ~9% of minute positions. The remaining ~91% of minutes return 0 from dict.get(minute, 0).

Double-Counting Mechanism

Phase 1 of fill_load_from_power (fetch.py:246-305):

1. Iterates minute-by-minute looking for "zero periods" (consecutive minutes with value 0)
2. The gap minutes between real data points appear as zero-value sequences
3. Any gap ≥ gap_size (default 5 minutes) is detected as a "zero period"
4. Phase 1 integrates power data for these gaps and ADDS it to the cumulative values (line 302)
5. It then bumps up ALL more-recent minutes (lines 304-305), inflating the cumulative curve

Phase 2 (fetch.py:307-357):

1. Divides data into 30-minute windows
2. Reads cumulative values at start/end: load_total = load_at_start - load_at_end
3. load_at_start is already inflated by Phase 1
4. Scales power data to match the inflated total → energy counted twice

Why Standard HA Installs Are Unaffected

HA records state changes at sub-second resolution, producing dense minute-level data. fill_load_from_power finds no significant "zero periods" to misdetect, so Phase 2 works correctly.

Who Is Affected

Any deployment where entity history is stored at intervals greater than ~5 minutes (e.g. external databases with downsampled data, custom history backends).

Evidence

Example user on March 10, 2026:

• consumption_today: 111 data points over 1200 minutes (9.25% coverage)
• consumption_power: 193 data points over 1200 minutes
• Yesterday's actual consumption: 19.7 kWh
• PredBat predicted load: 51.6 kWh / 48h = 25.8 kWh/day (1.31x actual)
• load_energy_actual tracking at 12.9 kWh at 19:50 (extrapolated ~15.6 kWh/day)
• Energy balance verified: GivEnergy API consumption field correctly excludes battery charging

Suggested Fix

After clean_incrementing_reverse produces sparse cumulative data, linearly interpolate between known data points to fill every minute index before passing to fill_load_from_power. This prevents Phase 1 from misdetecting inter-sample gaps as zero periods.

Alternatively, fill_load_from_power Phase 1 could be made aware of data sparsity by only treating a period as "zero" if there are actual data points with value 0, rather than relying on dict.get(minute, 0) defaults.

Affected Files

• apps/predbat/fetch.py — fill_load_from_power(), minute_data_load()
• apps/predbat/utils.py — minute_data(), clean_incrementing_reverse()

[gcoan]

Interesting @mgazza

My load today sensor is calculated as a time pattern trigger template that updates every 5 minutes. I calculate my load this way because I have multiple inverters that share the load and so have to calculate it, and I don't see any benefit to filling the database up with low level granular load data, 5 minutes fitted with Predbat's load modelling as well.

I thought I had load_power_fill_enable turned off, but I actually have it turned on, so would think I'd see the same issues as you, but I don't. My load projection tracks pretty well my actual load:

here's my logfile:

predbat.log

[mgazza]

Thanks @gcoan - that's really helpful context.

The key difference is the data path. In a standard Home Assistant setup, minute_data() receives history from HA's recorder and applies smoothing/carry-forward that densifies the dict before it reaches fill_load_from_power. So even with a 5-minute update interval, every minute index gets populated.

In our SaaS deployment, entity history is stored in Supabase at 5-minute buckets and the data path through our custom db_engine.py produces genuinely sparse dicts (only ~9% of minute indices populated). The GE Cloud data path (ge_cloud_data=True) is even coarser at ~30-minute intervals and bypasses minute_data()'s smoothing entirely.

So the fix (interpolate_sparse_data) is essentially a no-op for standard HA installations like yours — which is by design. It only activates when there are actual gaps between known data points, preventing fill_load_from_power from misclassifying those gaps as zero-load periods.

Your load projection tracking well confirms the fix won't regress anything for HA users.

[mgazza]

Thanks for the log @gcoan — really useful. A few things I got wrong in my earlier analysis:

Your data is dense, not sparse. Your log shows 13,521 load_today datapoints over 9 days (~1,502/day ≈ 1/minute). HA's recorder gives near-minute resolution even though your template sensor updates every 5 minutes, so minute_data() produces a fully dense dict. That's why you don't see inflation — interpolate_sparse_data is a no-op on already-dense data.

The issue is confirmed for GE Cloud standalone mode too. Looking at gecloud.py line 1556-1560, GE Cloud API data is explicitly aligned to 5-minute intervals:

# Align this_time to 5 minute intervals
this_time = this_time.replace(second=0, microsecond=0)
minute = (this_time.minute // 5) * 5
this_time = this_time.replace(minute=minute)

So anyone using ge_cloud_data=True (standalone mode with GivEnergy Cloud) gets sparse 5-minute data into fill_load_from_power, regardless of whether they're running SaaS or standard HA+standalone. This is the same root cause we hit in SaaS.

The fix is a no-op for standard HA setups like yours (which is the intended behaviour), but is needed for GE Cloud users.

[mgazza]

Here's a sanitized log extract from one of our SaaS users running GE Cloud (ge_cloud_data=True) showing the inflation in action. This user has load_power_fill_enable on, and PredBat runs every 5 minutes.

The key evidence is the growing "gap total" across consecutive runs — these aren't real missing data, they're the 4-minute spaces between 5-minute GE Cloud data points that fill_load_from_power misinterprets as zero-load periods:

# --- Run 1 (03:05 UTC) ---
Downloaded 11708 datapoints from GECloudData going back 8 days
Using load_power data to fill gaps in load_today data
Warn: Found 1 gaps in load_today totalling 155 minutes to fill using average data
load_minutes_now = 5.35 kWh

# --- Run 2 (03:10 UTC) ---
Downloaded 11713 datapoints from GECloudData going back 8 days
Using load_power data to fill gaps in load_today data
Warn: Found 1 gaps in load_today totalling 160 minutes to fill using average data
load_minutes_now = 6.30 kWh

# --- Run 3 (03:15 UTC) ---
Warn: Found 1 gaps in load_today totalling 165 minutes to fill using average data

# --- Run 4 (03:20 UTC) ---
Warn: Found 1 gaps in load_today totalling 200 minutes to fill using average data

# --- Run 5 (03:25 UTC) ---
Warn: Found 2 gaps in load_today totalling 235 minutes to fill using average data

# --- Runs 7-10 (gap stabilises at 240 min) ---
Warn: Found 1 gaps in load_today totalling 240 minutes to fill using average data (x4)

# --- Runs 11-16 (gap grows further) ---
Warn: Found 1 gaps in load_today totalling 270 minutes to fill using average data (x5)

# --- Run 17 (peak) ---
Warn: Found 2 gaps in load_today totalling 300 minutes to fill using average data

Gap total progression across 20 consecutive runs: 155 → 160 → 165 → 200 → 235 → 210 → 240 → 240 → 240 → 240 → 270 → 240 → 270 → 270 → 270 → 270 → 300 → 270 → 280 → 280

Each run, the "filled" data from the previous run's Phase 2 gets baked into the cumulative load_today values, creating new 4-minute gaps between the previously-filled data and the next real 5-minute data point. Phase 1 then detects these as new gaps, expanding the total fill region each cycle.

This is exactly what the PR fix (#3546) addresses — interpolate_sparse_data() fills the 4-minute gaps with linearly interpolated values before fill_load_from_power runs, so Phase 1 sees dense data and doesn't trigger false gap detection.

[gcoan]

Very interesting, thanks Mark, and good to see improvements from Predbat SAAS feeding back into the original product.

Your comment

Your log shows 13,521 load_today datapoints over 9 days (~1,502/day ≈ 1/minute). HA's recorder gives near-minute resolution even though your template sensor updates every 5 minutes

Is interesting and is not what I was expecting. I'm not chalklenging your knowledge of HA and Python as its clearly more than mine, but,

When I look at the history of house_load_today (my template sensor) I see it incrementing in 0.1kWh steps and periods where it doesn't increment at all:

and if I download the state history:

entity_id,state,last_changed
sensor.house_load_today,20.2,2026-03-11T20:00:00.000Z
sensor.house_load_today,20.4,2026-03-11T20:00:00.498Z
sensor.house_load_today,20.5,2026-03-11T20:05:00.497Z
sensor.house_load_today,20.6,2026-03-11T20:10:00.503Z
sensor.house_load_today,20.7,2026-03-11T20:15:00.498Z
sensor.house_load_today,20.8,2026-03-11T20:25:00.497Z
sensor.house_load_today,20.9,2026-03-11T20:30:00.498Z
sensor.house_load_today,21.1,2026-03-11T20:35:00.502Z
sensor.house_load_today,21.3,2026-03-11T20:40:00.498Z
sensor.house_load_today,21.5,2026-03-11T20:45:00.503Z
sensor.house_load_today,21.7,2026-03-11T20:50:00.536Z
sensor.house_load_today,21.9,2026-03-11T20:55:00.498Z
sensor.house_load_today,22.1,2026-03-11T21:00:00.497Z
sensor.house_load_today,22.2,2026-03-11T21:05:00.497Z
sensor.house_load_today,22.3,2026-03-11T21:10:00.519Z
sensor.house_load_today,22.4,2026-03-11T21:15:00.503Z
sensor.house_load_today,22.5,2026-03-11T21:25:00.497Z
sensor.house_load_today,22.6,2026-03-11T21:30:00.499Z
sensor.house_load_today,22.7,2026-03-11T21:35:00.501Z
sensor.house_load_today,22.8,2026-03-11T21:40:00.499Z
sensor.house_load_today,23.1,2026-03-11T21:45:00.754Z
sensor.house_load_today,23.4,2026-03-11T21:50:00.497Z
sensor.house_load_today,23.6,2026-03-11T21:55:00.498Z

I see similar time gaps, e.g. 20:20 and 21:20 in the history

So a record is only written to the entity table when a state change occurs, whether that be on the 5 minute or multiple of 5 minutes as the value hasn't changed. Not every minute.

This matches my expectations. I spent a lot of time working on reducing the size of my HA database, techniques like time trigger templates and time filter entities were part of this. By default HA records every single state change in the database so things like power and voltage entities on smart plugs were generating humongous amounts of data that just wasn't needed. Time sampling significantly reduced my HA database size.
Edit: if you want some bedtime reading https://community.givenergy.cloud/d/5458-taming-and-reducing-the-home-assistant-database

Is the number of records reported in the Predbat log as being retrieved after some time inflation is done?

[mgazza]

Good catch @gcoan — you're right. The "13,521 datapoints" in the log is len(self.load_minutes) which is the dense dict after minute_data() processing, not the raw history count. Your actual HA history is much sparser (~200-300 entries/day based on the CSV you shared), and minute_data() with smoothing=True fills it out to one entry per minute. So my earlier claim of "~1/minute from HA's recorder" was wrong — the density comes from PredBat's own interpolation, not from HA.

As @springfall2008 pointed out on PR #3546, the original fix (interpolate_sparse_data) was a no-op because minute_data() already densifies the data before fill_load_from_power sees it. He was right — we closed that PR.

However, investigating further we found the actual root cause: previous_days_modal_filter() (fetch.py:424) detects "gaps" by checking data[m] == data[m+5]. After clean_incrementing_reverse(), legitimate overnight zero-consumption periods produce flat regions (consecutive equal values). The gap detector can't distinguish "sensor online, zero consumption" from "sensor offline, no data" — so it fills both with average daily load, injecting phantom consumption (~6 kWh/night for a typical household).

This is what causes the growing gap totals in the SaaS logs (155→300 minutes) — it's not fill_load_from_power double-counting, it's previous_days_modal_filter filling real zero-load overnight periods with synthetic average data.

The replacement fix is PR #3554 — it tracks which minutes had actual sensor data points during minute_data() processing, then in the gap detector checks whether the sensor was actively reporting during each gap period. If the sensor was online (≥1 data point per hour), the gap is skipped. If offline, it fills as before. Fully backward-compatible and proved with unit tests.