fix: prevent false-positive gap detection on zero-consumption periods#3554
Open
fix: prevent false-positive gap detection on zero-consumption periods#3554
Conversation
The gap detector in previous_days_modal_filter() checks if consecutive values are equal (data[m] == data[m+5]) to find missing data. After clean_incrementing_reverse(), zero-consumption overnight periods have equal consecutive values, triggering false gap detection and injecting phantom load (~6 kWh/night for a 24 kWh/day average). Track sensor data point provenance during minute_data() processing via a new data_point_minutes set parameter. In the gap detector, check whether the sensor was actively reporting during each gap period. If the sensor was online (≥1 data point/hour), skip filling. If offline, fill as before. Supersedes #3546 which attempted to fix the symptom via interpolation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4 tasks
… plan Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… decode and commands Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GatewayMQTT now inherits from ComponentBase with full instance methods: - initialize() stores config and builds MQTT topic strings - run() starts background MQTT listener on first call, does housekeeping after - _mqtt_loop() connects with TLS, subscribes to /status and /online, reconnects - _process_telemetry() decodes protobuf and publishes entities via set_state_wrapper - publish_plan/publish_command for outbound control - is_alive() checks MQTT connected + telemetry freshness - select_event/number_event for UI-driven mode/rate/SOC changes - final() sends AUTO mode and cancels listener on shutdown - Token refresh via Supabase edge function (same pattern as OAuthMixin) All existing static methods and tests preserved unchanged. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ewal Add extract_jwt_expiry and token_needs_refresh static methods. Wire into _check_token_refresh to extract expiry from JWT claims directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…erter entities Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix MQTT topic prefix: gw/ -> predbat/devices/ (critical) - Add retain=True on schedule topic publish - Set api_started=True on first telemetry decode - Add get_error_count() with error tracking - Fix edge function name: refresh-mqtt-token (matching spec) - Remove unused _TOKEN_REFRESH_THRESHOLD constant Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consistent with Fox OAuth pattern — single oauth-refresh edge function handles all providers including gateway MQTT token refresh. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
oauth-refresh edge function reads refresh token from instance secrets, so the component doesn't need to hold or pass it. Consistent with how Fox OAuth works via OAuthMixin. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
previous_days_modal_filter()that injected phantom load (~6 kWh/night) during legitimate zero-consumption overnight periodsminute_data()processing via a newdata_point_minutessetRoot Cause
The gap detector checks
data[m] == data[m+5]to find missing data. Afterclean_incrementing_reverse(), zero-consumption periods have equal consecutive cumulative values. The detector falsely flags these as "gaps" and fills them withaverage_day / 1440kWh/minute of synthetic consumption.Test 10 proves this: 8 false gaps per 8 days, 3360 minutes (29.2% of data) incorrectly filled.
Approach: Data Point Provenance
Rather than changing the gap detection heuristic (which serves a legitimate purpose for sensor-offline periods), we record which minutes had actual sensor readings during
minute_data(). The gap filter then distinguishes:Files Changed
utils.pydata_point_minutesparam tominute_data()fetch.pyminute_data_load()+download_ge_data(), filter inprevious_days_modal_filter()tests/test_previous_days_modal.pytests/test_fill_load_from_power.pyTest plan
load_data_point_minutesattribute → backward compat (gaps filled as before)Supersedes #3546
🤖 Generated with Claude Code