feat: client mode improvements — sync, HA export, scheduler, display fixes - #110
Open
jrozelle wants to merge 12 commits into
Open
feat: client mode improvements — sync, HA export, scheduler, display fixes#110jrozelle wants to merge 12 commits into
jrozelle wants to merge 12 commits into
Conversation
When a Tempo year (e.g. 2024-2025) appeared both as "previous" of a newer block and as "current" of its own block, the deduplication kept the first (incomplete) version, causing gaps in the chart curve. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These metadata files created by Synology NAS were polluting the repo and causing git pull errors (bad object refs/@eaDir/heads@SynoEAStream). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… stats fallback) - Fix off-by-one in _find_missing_ranges: use <= instead of < so yesterday's data is properly detected as missing and synced from Enedis - Add DETAILED granularity fallback in get_day_total when DAILY data is missing - Use self.prefix instead of hardcoded "myelectricaldata_" in all MQTT topics, unique_ids, and subscription patterns to prevent v2 overwriting v1 sensors - Add queryClient.invalidateQueries after HA export mutation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous session accidentally replaced the entire home_assistant.py and sync.py files instead of making targeted prefix edits, losing ~600 lines of content-card-linky compatible attributes (HC/HP breakdown, costs, max power, tempo colors, weekly/monthly/yearly stats, evolution %). This commit: - Restores home_assistant.py from e73fbb4 (with rich attributes) - Restores sync.py from e73fbb4 (with deduplication, MaxPower, placeholder handling) - Restores HomeAssistant.tsx from e73fbb4 (with prefix-aware entity preview) - Keeps the <= fix in _find_missing_ranges (applied on top of restored sync.py) - Keeps statistics.py DETAILED fallback and simplified queries (no duplicates possible now thanks to NULLS NOT DISTINCT) - Adds NULLS NOT DISTINCT migration for consumption_data and production_data unique constraints (fixes ON CONFLICT never firing for daily records) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous session accidentally overwrote multiple files, losing: - MaxPowerData model + migration - TrustedHost security middleware - Adapter cache parameters - enedis_client local-first cache, dedup, backoff logic - MQTT exporter model imports and defensive coding - Scheduler startup sync + export-after-sync - Export router schedule fields - PDLDetails XSS protection - MQTT/VictoriaMetrics env-based defaults - Ecowatt return format handling All files restored from e73fbb4, with only intentional changes kept: - sync.py: <= fix for date boundaries - statistics.py: DETAILED fallback + simplified queries - NULLS NOT DISTINCT migration (new) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ware Scheduler: replace aggressive polling (24 Tempo/day, 6h intervals) with targeted cron jobs aligned to data publication times. Remove TrustedHostMiddleware that caused issues behind reverse proxy. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…y mismatch) Cherry-pick of 03b3616 that was lost when files were restored from e73fbb4: - local_data.py: inclusive end_date (<=), MaxPower queries, placeholder handling - Frontend hooks: endDate = today (not yesterday) for inclusive backend queries - Frontend pages: same endDate fix in ConsumptionEuro, ConsumptionKwh, Production - invalidateQueries keys: consumptionDaily/productionDaily (not consumption/production) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jrozelle
changed the title
Author
|
J'ai modifié la PR car elle m'a écrasé des commit précédents... |
…today) Enedis API returns error "end date must be earlier than current date" when end_date is today+1. Since _find_missing_power_ranges adds +1 day for exclusive end bounds, using date.today() resulted in API calls with end=tomorrow, which Enedis always rejected with data=None. Fix: use yesterday as end_date since today's max_power isn't published yet. Also remove debug logging from previous commits. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
`recorder/statistics_during_period` returns the `start` field as a numeric
epoch in MILLISECONDS since Home Assistant Core 2023+, not seconds. The
existing code passed it directly to `datetime.fromtimestamp()` which expects
seconds, causing OverflowError ("year 58244 is out of range") and the
exporter to silently fall back to a full re-import every 30 minutes.
The full re-import wrote sums that didn't align with HA's existing
statistics, so the Energy dashboard could not compute deltas reliably and
no consumption data was visible.
Auto-detect ms vs s based on magnitude (> 1e12 = ms) — safe until ~year
33658, simple, no need to know HA core version.
In incremental mode the three statistics builders (_get_consumption_statistics_by_tariff, _get_cost_statistics_by_tariff, _get_production_statistics) initialised their cumulative sums to 0.0 and only added the new records on top. Because incremental imports only ship records since the last known date, the resulting `sum` field starts back at 0 and clobbers the existing monotonic series in HA. The Energy Dashboard then sees a giant negative delta on the boundary hour (e.g. -11705 kWh on TEMPO blue_hp at the season transition) and stops being usable. Fix: - get_last_statistic_dates now also returns last_sums (extracted from the same statistics_during_period response, no extra round-trip) - import_statistics and import_statistics_with_progress build per-PDL initial sums via a new helper _extract_initial_sums_for_pdl that maps full statistic_ids back to tariff tags - The three builders accept initial_sums (or initial_sum for production) and seed cumulative_by_tariff with those values instead of 0.0 Effect: in incremental mode the new series continues the existing cumulative history, sum stays monotonic, no more compensation events in HA Energy. Note: this commit only prevents NEW regressions. Existing damage (historical regressions already written to HA recorder) needs a separate one-shot data repair.
…al import The previous fix (dc3f556) seeded cumulative_by_tariff with the LAST sums already in HA, but still iterated all records since the GLOBAL since_date (= oldest last_date across all tariffs). For tariffs with newer last_dates, this re-processed records that were already in HA and overwrote their sums with shifted-up values — effectively double-counting consumption every time MED ran an incremental import. Symptom: each new MED run shifted the cumulative series upward by the delta of consumption between the global oldest_date and each tariff's own last_date. After enough runs, the totals were 2-3x the real value. Fix: - get_last_statistic_dates already returned per-stat last dates; we now also forward them to the builders. - _extract_initial_sums_for_pdl is replaced by _extract_per_pdl_state which returns initial sums AND per-tariff last_dates (consumption) + single last_date (production). - _get_consumption_statistics_by_tariff accepts last_dates_by_tariff and skips records whose hour <= the tariff's last_date. - _get_production_statistics accepts last_date and applies the same filter to production hourly aggregation. - import_statistics and import_statistics_with_progress wire all of this through. Cost statistics naturally inherit the filter because they're derived from the (already filtered) consumption_by_tariff. Effect: incremental imports now only ADD strictly new records; existing ones are preserved bit-for-bit. No more double-counting, no more shifting on each run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Client mode — Sync & data pipeline
<=instead of<in_find_missing_ranges(sync.py + local_data.py) so yesterday's data is properly detected as missing and syncedON CONFLICTnever firing for daily records (interval_start is NULL)end_date = yesterday— Enedis API rejects end dates >= todayuse_cache=trueparameter on API callsHome Assistant export — content-card-linky compatibility
dailyweek,dailyweek_HC,dailyweek_HP,dailyweek_cost,dailyweek_MP,dailyweek_MP_time,dailyweek_Tempo, evolution percentages (yearly/monthly/weekly/yesterday), HP/HC breakdown, costs, peak/offpeak percentself.prefixfrom config (not hardcodedmyelectricaldata_) — allows v1 and v2 instances to coexist without topic collisionget_day_totaltries DAILY first, falls back to summing DETAILED records (W/2 = Wh)Scheduler (~20 API calls/day)
_run_sync()triggers_run_all_exports()automaticallyFrontend fixes
today(notyesterday) as exclusive end boundconsumptionDaily/productionDailykeys for proper cache invalidationwindow.__ENV__VITE_DEFAULT_*env vars for runtime configOther
.gitignoreentrymodel_fields_setlogiccreated/updatedcountsTest plan
git pullno longer fails with Synology @eadir errors🤖 Generated with Claude Code