From 55cae6d3d4420945b6b0f338c1ecbc0528f677c6 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 7 Sep 2026 19:44:22 +0000 Subject: [PATCH 1/3] ci: bump github/issue-labeler from 3.4 to 3.5 Bumps [github/issue-labeler](https://github.com/github/issue-labeler) from 3.4 to 3.5. - [Release notes](https://github.com/github/issue-labeler/releases) - [Commits](https://github.com/github/issue-labeler/compare/v3.4...v3.5) --- updated-dependencies: - dependency-name: github/issue-labeler dependency-version: '3.5' dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- .github/workflows/label-issues.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/label-issues.yml b/.github/workflows/label-issues.yml index b2f2761..56504d2 100644 --- a/.github/workflows/label-issues.yml +++ b/.github/workflows/label-issues.yml @@ -12,7 +12,7 @@ jobs: name: Automatically label issues runs-on: ubuntu-latest steps: - - uses: github/issue-labeler@v3.4 + - uses: github/issue-labeler@v3.5 with: repo-token: ${{ secrets.GITHUB_TOKEN }} configuration-path: .github/labeler.yml From a52c63b38a98f8aaad7f45a15addcedc4e7ca649 Mon Sep 17 00:00:00 2001 From: bvweerd Date: Mon, 21 Sep 2026 18:47:16 +0200 Subject: [PATCH 2/3] docs: add Venus A production-measured curve and DB analysis methodology MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a production-measured efficiency curve for the Marstek Venus A derived from HA recorder data on two units (Sept 2026): SoC-based calibration showed efficiency_factor > 1.0 on both batteries, calibration correction factor capped at 1.0492. Community curve scaled by ×1.0492 gives one-way efficiency of 0.954 at the 500–800 W plateau (RTE 91 %) vs 0.909 (RTE 83 %) in the community curve. Also add docs/efficiency-analysis-from-db.md explaining the two-pronged methodology: hourly SQL queries against the statistics table for dispatch fidelity, and Python parsing of .storage calibration files for the SoC-based efficiency factor — so the analysis can be repeated in future. --- docs/efficiency-analysis-from-db.md | 204 ++++++++++++++++++++++++++++ docs/efficiency-curves.md | 32 ++++- 2 files changed, 235 insertions(+), 1 deletion(-) create mode 100644 docs/efficiency-analysis-from-db.md diff --git a/docs/efficiency-analysis-from-db.md b/docs/efficiency-analysis-from-db.md new file mode 100644 index 0000000..bf260d3 --- /dev/null +++ b/docs/efficiency-analysis-from-db.md @@ -0,0 +1,204 @@ +# Verifying the efficiency curve from HA recorder data + +How to use SQLite queries and the stored calibration files to compare actual +battery efficiency against the configured values. + +--- + +## 1. What you want to know + +The battery controller uses an efficiency curve (RTE, e.g. 91 %) as input for +the DP. If the actual battery performs better or worse than that curve, you +either miss opportunities or plan too optimistically. + +There are **two metrics** you can check: + +| Metric | What it measures | Data source | +|---|---|---| +| **Dispatch fidelity** | Actually delivered / commanded (AC side) | `marstek_total_energy_discharged` / setpoint | +| **Efficiency correction** | SoC change / planned SoC change | `.storage/battery_controller_*_eff` | + +They measure different things. Dispatch fidelity is on the AC side of the +inverter; the efficiency correction measures across the full conversion +(including internal battery losses). + +--- + +## 2. Dispatch fidelity via hourly averages + +```sql +-- File: home-assistant_v2.db +-- Returns: per hour, per SoC band, the ratio actual/commanded energy + +WITH setpoint AS ( + SELECT start_ts, mean as setpoint_w + FROM statistics + WHERE metadata_id = ( + SELECT id FROM statistics_meta + WHERE statistic_id = 'sensor.marstek_battery_setpoint_marstek' + ) + AND start_ts >= strftime('%s', '2026-08-01') +), +energy_dchg AS ( + SELECT start_ts, + sum - LAG(sum) OVER (ORDER BY start_ts) as delta_dchg_kwh + FROM statistics + WHERE metadata_id = ( + SELECT id FROM statistics_meta + WHERE statistic_id = 'sensor.marstek_total_energy_discharged' + ) + AND start_ts >= strftime('%s', '2026-08-01') +), +energy_chg AS ( + SELECT start_ts, + sum - LAG(sum) OVER (ORDER BY start_ts) as delta_chg_kwh + FROM statistics + WHERE metadata_id = ( + SELECT id FROM statistics_meta + WHERE statistic_id = 'sensor.marstek_total_energy_charged' + ) + AND start_ts >= strftime('%s', '2026-08-01') +), +soc_data AS ( + SELECT start_ts, mean as soc_pct + FROM statistics + WHERE metadata_id = ( + SELECT id FROM statistics_meta + WHERE statistic_id = 'sensor.marstek_battery_soc' + ) + AND start_ts >= strftime('%s', '2026-08-01') +), +combined AS ( + SELECT + sp.setpoint_w, + so.soc_pct, + ed.delta_dchg_kwh, + ec.delta_chg_kwh, + ABS(sp.setpoint_w) / 1000.0 as commanded_kwh + FROM setpoint sp + LEFT JOIN soc_data so ON so.start_ts = sp.start_ts + LEFT JOIN energy_dchg ed ON ed.start_ts = sp.start_ts + LEFT JOIN energy_chg ec ON ec.start_ts = sp.start_ts + WHERE ABS(sp.setpoint_w) > 300 + AND so.soc_pct IS NOT NULL +) +SELECT + CASE WHEN setpoint_w > 0 THEN 'discharge' ELSE 'charge' END as direction, + CAST(soc_pct / 10 AS INT) * 10 as soc_band, + COUNT(*) as n, + ROUND(AVG( + CASE WHEN setpoint_w > 0 THEN delta_dchg_kwh / commanded_kwh + ELSE delta_chg_kwh / commanded_kwh END + ), 4) as avg_fidelity +FROM combined +WHERE + (setpoint_w > 0 AND delta_dchg_kwh > 0.01) + OR (setpoint_w < 0 AND delta_chg_kwh > 0.01) +GROUP BY direction, soc_band +ORDER BY direction, soc_band; +``` + +### Results (September 2026, Marstek 1) + +| Direction | SoC band | Average fidelity | +|---|---|---| +| discharge | 10–90 % | ~0.91–0.94 (flat, no SoC dependency) | +| charge | 10–80 % | ~0.99 | +| charge | 90 %+ | ~0.89 (CV-phase taper near full) | + +**Dispatch conclusion**: battery delivers ~93.7 % of the commanded discharge +power; charging is nearly lossless up to ~80 % SoC, then taper-effect kicks in. + +--- + +## 3. SoC-based efficiency correction (stored calibration) + +The coordinator stores one file per battery per direction: + +``` +/media/data/homeassistant/config/.storage/ + battery_controller___charge_eff + battery_controller___discharge_eff +``` + +```python +import json, statistics as stats + +files = { + 'marstek_charge': '...charge_eff', + 'marstek_discharge': '...discharge_eff', +} +for label, fname in files.items(): + with open(fname) as f: + d = json.load(f) + data = d['data'] + samples = data.get('samples', []) + corr = data.get('correction') + schema = data.get('schema', 'raw_ratio') + print(f'{label}: correction={corr:.4f}, n={len(samples)}, schema={schema}') + if samples: + print(f' mean={stats.mean(samples):.4f}, ' + f'min={min(samples):.3f}, max={max(samples):.3f}') +``` + +### Schema interpretation + +| Schema in file | Interpretation | +|---|---| +| `"schema": "efficiency_factor"` | Samples are already efficiency factors (< 1.0 = worse than configured) | +| No schema key (`"raw_ratio"`) | Old format: samples are `measured/planned` ratios | + +For **discharge** in old format: `efficiency_factor = 1 / raw_ratio` +(when discharging, less SoC drop per commanded kWh means better efficiency). + +### Results (September 2026) + +| Battery | Direction | Raw ratio | Efficiency factor | Applied | +|---|---|---|---|---| +| Marstek 1 | charge | 1.050 (capped) | 1.050 | False | +| Marstek 1 | discharge | 0.796 | **1.256** | False | +| Marstek 2 | charge | 1.048 | 1.048 | False | +| Marstek 2 | discharge | 0.920 | **1.087** | False | + +`applied: False` because efficiency_factor ≥ `CALIBRATION_APPLY_THRESHOLD` +(0.995): the battery performs at or above the configured curve, so the +correction is not applied — the optimizer is never made more optimistic than +the values the user entered. + +**Efficiency conclusion**: the configured curve is conservative. Both batteries +store more energy per commanded kWh and lose less SoC per commanded discharge +kWh than planned. The corrections are capped at 105 % (CALIBRATION_APPLY_MAX) +and not passed to the DP. + +--- + +## 4. What to do with this + +| Finding | Meaning | Action | +|---|---|---| +| Dispatch fidelity 94 % | Battery delivers slightly less than commanded | Normal for Marstek; confirmed by `dispatch_fidelity` sensor | +| Efficiency factor > 1 | Battery outperforms configured curve | No automatic correction (by design) — consider raising the curve manually | +| Charge fidelity 89 % at SoC > 90 % | CV-phase taper near full charge | Normal LFP behaviour; calibration skips samples that cross the high-SoC derate threshold | + +To raise the curve: multiply every point in the configured curve by 1.05 +(the measured but capped correction factor). See the updated curves in +`efficiency-curves.md` §6. + +--- + +## 5. Limitations + +- **Hourly setpoint averages**: the hourly mean setpoint is not the same as + a constant setpoint for the whole hour (BC switches every 15 min). Errors + of ±5 % are normal. +- **SoC resolution**: 1 % step ≈ 0.05 kWh on a 5 kWh battery. Short steps + are unreliable; the calibration logic filters them via + `CALIBRATION_SOC_QUANTUM_FACTOR = 4.0`. +- **`statistics_short_term`**: only kept for ~10 days; use the `statistics` + table (hourly averages) for longer periods. +- **Energy counter gaps**: the `states` table can have gaps for + `total_energy_charged/discharged`; the `statistics` table (column `sum`) + is more reliable for cumulative sensors. +- **Old `raw_ratio` schema**: storage files written before the + `STORED_EFFICIENCY_FACTOR` marker was introduced need the inversion + described above. Check `data.get('schema')` before interpreting samples. diff --git a/docs/efficiency-curves.md b/docs/efficiency-curves.md index 997ed37..1224427 100644 --- a/docs/efficiency-curves.md +++ b/docs/efficiency-curves.md @@ -366,15 +366,45 @@ Venus E's 7 W — seventy times less — yet its operating overhead is only 1.8 | System | Provenance | | --- | --- | -| Marstek Venus A | **User-measured**, two independent owners: a wall-meter test at 100/200 W plus ~300 kWh of counter data at 1200 W — the best-constrained curve here | +| Marstek Venus A (community) | **User-measured**, two independent owners: a wall-meter test at 100/200 W plus ~300 kWh of counter data at 1200 W — the best-constrained curve here | +| Marstek Venus A (production) | Community curve ×1.0492, derived from HA recorder data on two Venus A units (SoC calibration + dispatch fidelity, Sept 2026) | | Marstek Venus E | **User-measured** charge curve; discharge scaled to the measured full-power RTE | | Zendure, HomeWizard | RTE anchor is measured; the *shape* is borrowed from the Marstek fit | **Marstek Venus A** — 1500 W bidirectional, overhead 30 W, plateau 500–800 W + +Two variants are available. The **community curve** is the baseline from forum +and review measurements. The **production-measured curve** is the community +curve scaled by the correction factor measured from HA recorder data on two +Venus A units over several weeks of normal operation (×1.0492, see +`efficiency-analysis-from-db.md` for methodology). Both units consistently +showed efficiency factors > 1.0 (battery beats the community curve), so the +production curve is the better prior for these specific units. + +*Community curve (baseline):* ``` charge: 0.05:0.623, 0.1:0.764, 0.2:0.857, 0.3:0.890, 0.5:0.909, 0.8:0.908, 1.2:0.892, 1.5:0.878 discharge: 0.05:0.623, 0.1:0.764, 0.2:0.857, 0.3:0.890, 0.5:0.909, 0.8:0.908, 1.2:0.892, 1.5:0.878 ``` + +*Production-measured curve (×1.0492, two units, Sept 2026):* +``` +charge: 0.05:0.654, 0.1:0.802, 0.2:0.899, 0.3:0.934, 0.5:0.954, 0.8:0.953, 1.2:0.936, 1.5:0.921 +discharge: 0.05:0.654, 0.1:0.802, 0.2:0.899, 0.3:0.934, 0.5:0.954, 0.8:0.953, 1.2:0.936, 1.5:0.921 +``` + +| AC power | RTE (community) | RTE (production-measured) | +|---:|---:|---:| +| 100 W | 58 % | 64 % | +| 300 W | 79 % | 87 % | +| 500 W | 83 % | 91 % | +| 800 W | 82 % | 91 % | +| 1200 W | 80 % | 88 % | +| 1500 W | 77 % | 85 % | + +If the integration's `discharge_efficiency_correction` sensor shows +`applied: False` for your Venus A units, use the production curve — the +battery is already performing at that level. Set both power limits to 1.5 kW. The derived `round_trip_efficiency` the integration reports from this curve is 0.765 — the mean over 5–95 % of rated power, which sits below the 0.83 plateau because it includes the poor bottom end. From a074dc760f16cf028636a9a799e86f72537e451d Mon Sep 17 00:00:00 2001 From: bvweerd Date: Sun, 27 Sep 2026 19:07:57 +0200 Subject: [PATCH 3/3] fix: always persist calibration after sampling, not only when correction moves When the efficiency correction is at CALIBRATION_APPLY_MAX (1.05) every new sample is added to the rolling window but record() returns moved=False, so async_save() is never called. The file on disk then keeps the old raw_ratio schema indefinitely and is re-migrated on every HA restart, resetting to correction=1.05 each time. The sensor appears permanently stuck. Fix: save after every successful sample (CALIBRATION_SAMPLED), not only when the correction shifts by more than CALIBRATION_SIGNIFICANT_CHANGE. This keeps the file in the current efficiency_factor schema and the sample count current, so a reset is not needed to unblock the sensor. --- .../battery_controller/coordinator_optimization.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/custom_components/battery_controller/coordinator_optimization.py b/custom_components/battery_controller/coordinator_optimization.py index ca57c14..31c77cb 100644 --- a/custom_components/battery_controller/coordinator_optimization.py +++ b/custom_components/battery_controller/coordinator_optimization.py @@ -30,6 +30,7 @@ CALIBRATION_NO_SOC_SOURCE, CALIBRATION_NOT_DISPATCHED, CALIBRATION_PLAN_NOT_EXECUTED, + CALIBRATION_SAMPLED, CALIBRATION_STEP_INCOMPLETE, CHARGE_CALIBRATION, DISCHARGE_CALIBRATION, @@ -2218,7 +2219,12 @@ def _update_battery_eff_calibration( ) actual_delta = max(0.0, measured) - if calibration.record(planned_delta, actual_delta, "SoC delta"): + moved = calibration.record(planned_delta, actual_delta, "SoC delta") + # Always persist after a sample, not only when the correction moves. + # When the correction is at the cap (1.05) every new sample is added to + # the rolling window but never triggers a save, so the file on disk stays + # in the old raw_ratio format and is re-migrated on every restart. + if moved or calibration.last_result == CALIBRATION_SAMPLED: self.hass.async_create_task(calibration.async_save()) def _update_charge_eff_calibration(