Skip to content

fix: prevent false-positive gap detection on zero-consumption periods#3554

Open
mgazza wants to merge 11 commits intomainfrom
fix/false-positive-gap-detection
Open

fix: prevent false-positive gap detection on zero-consumption periods#3554
mgazza wants to merge 11 commits intomainfrom
fix/false-positive-gap-detection

Conversation

@mgazza
Copy link
Collaborator

@mgazza mgazza commented Mar 11, 2026

Summary

  • Fixes false-positive gap detection in previous_days_modal_filter() that injected phantom load (~6 kWh/night) during legitimate zero-consumption overnight periods
  • Tracks sensor data point provenance during minute_data() processing via a new data_point_minutes set
  • In the gap detector, checks whether the sensor was actively reporting during each gap. If online (≥1 data point/hour), skips filling. If offline, fills as before.
  • Fully backward-compatible: existing callers unaffected, no attribute = old behavior

Root Cause

The gap detector checks data[m] == data[m+5] to find missing data. After clean_incrementing_reverse(), zero-consumption periods have equal consecutive cumulative values. The detector falsely flags these as "gaps" and fills them with average_day / 1440 kWh/minute of synthetic consumption.

Test 10 proves this: 8 false gaps per 8 days, 3360 minutes (29.2% of data) incorrectly filled.

Approach: Data Point Provenance

Rather than changing the gap detection heuristic (which serves a legitimate purpose for sensor-offline periods), we record which minutes had actual sensor readings during minute_data(). The gap filter then distinguishes:

  • Sensor online, zero consumption → skip filling (correct)
  • Sensor offline, no data → fill with average (correct, existing behavior)

Files Changed

File Change
utils.py Add data_point_minutes param to minute_data()
fetch.py Init set, pass through minute_data_load() + download_ge_data(), filter in previous_days_modal_filter()
tests/test_previous_days_modal.py Tests 6-8: sensor active → skip, sensor offline → fill, backward compat
tests/test_fill_load_from_power.py Test 11: e2e verification all 8 false-positive gaps filtered out

Test plan

  • All existing tests pass (previous_days_modal, fill_load_from_power, minute_data, minute_data_load, get_now_cumulative)
  • Test 6: Overnight zeros with sensor data points → gaps skipped (12 kWh stays 12 kWh)
  • Test 7: Gap with no sensor data points → gap correctly filled (12 → 18 kWh)
  • Test 8: No load_data_point_minutes attribute → backward compat (gaps filled as before)
  • Test 11: End-to-end proof all 8 false-positive overnight gaps filtered out

Supersedes #3546

🤖 Generated with Claude Code

The gap detector in previous_days_modal_filter() checks if consecutive
values are equal (data[m] == data[m+5]) to find missing data. After
clean_incrementing_reverse(), zero-consumption overnight periods have
equal consecutive values, triggering false gap detection and injecting
phantom load (~6 kWh/night for a 24 kWh/day average).

Track sensor data point provenance during minute_data() processing via
a new data_point_minutes set parameter. In the gap detector, check
whether the sensor was actively reporting during each gap period. If
the sensor was online (≥1 data point/hour), skip filling. If offline,
fill as before.

Supersedes #3546 which attempted to fix the symptom via interpolation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
mgazza and others added 10 commits March 13, 2026 06:16
… plan

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… decode and commands

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GatewayMQTT now inherits from ComponentBase with full instance methods:
- initialize() stores config and builds MQTT topic strings
- run() starts background MQTT listener on first call, does housekeeping after
- _mqtt_loop() connects with TLS, subscribes to /status and /online, reconnects
- _process_telemetry() decodes protobuf and publishes entities via set_state_wrapper
- publish_plan/publish_command for outbound control
- is_alive() checks MQTT connected + telemetry freshness
- select_event/number_event for UI-driven mode/rate/SOC changes
- final() sends AUTO mode and cancels listener on shutdown
- Token refresh via Supabase edge function (same pattern as OAuthMixin)

All existing static methods and tests preserved unchanged.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ewal

Add extract_jwt_expiry and token_needs_refresh static methods. Wire into
_check_token_refresh to extract expiry from JWT claims directly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…erter entities

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix MQTT topic prefix: gw/ -> predbat/devices/ (critical)
- Add retain=True on schedule topic publish
- Set api_started=True on first telemetry decode
- Add get_error_count() with error tracking
- Fix edge function name: refresh-mqtt-token (matching spec)
- Remove unused _TOKEN_REFRESH_THRESHOLD constant

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consistent with Fox OAuth pattern — single oauth-refresh edge function
handles all providers including gateway MQTT token refresh.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
oauth-refresh edge function reads refresh token from instance secrets,
so the component doesn't need to hold or pass it. Consistent with
how Fox OAuth works via OAuthMixin.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant