Match the real CT on zero-MAC probes and normalize CT_MAC

CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Next
 
+- **Fixed** a Marstek Venus E Mini failing to pair with the emulated CT ("configuration failed" in the app) when `CT_MAC` is set, and a `CT_MAC` entered in uppercase or with colons never pairing ([#727](https://github.com/tomquist/astrameter/pull/727), [#728](https://github.com/tomquist/astrameter/pull/728)).
+
 - **Fixed** a battery's Home Assistant entities staying unavailable until a restart since 2.3.1, after a network hiccup briefly dropped the battery and the MQTT connection at the same time ([#718](https://github.com/tomquist/astrameter/issues/718), [#719](https://github.com/tomquist/astrameter/pull/719)).
 
 - **Fixed** a battery sometimes staying stuck below its share since 2.3.1 while the other carried the rest, letting their charge levels drift apart; a battery that only paused briefly is now back to sharing within a minute ([#704](https://github.com/tomquist/astrameter/issues/704), [#707](https://github.com/tomquist/astrameter/pull/707)).

config.ini.example

@@ -69,7 +69,8 @@ THROTTLE_INTERVAL = 0
 ## Note: CT003 can optionally use its own [CT003] section with the same keys.
 ## CT MAC (12 hex digits, from Marstek app).
 ## If empty, the emulator accepts any request CT MAC and echoes the request's
-## CT MAC in responses. If set, the emulator responds only to the configured MAC.
+## CT MAC in responses. If set, the emulator responds only to the configured MAC
+## and to an unpaired battery's 000000000000 probe, answering with its own MAC.
 # CT_MAC = 001122334455
 ## UDP port to bind for CT002/CT003 (default 12345).
 # UDP_PORT = 12345

docs/ct002-ct003-protocol.md

@@ -704,7 +704,25 @@ slv_ip=<ip>,slv_t=<type>,slv_p=<phase>,slv_id=<mac>;
 ## CT MAC behavior
 
 The CT responds when the request `hhm_mac_code` is the wildcard `000000000000`
-**or** matches the CT's own MAC. In AstraMeter: if `CT_MAC` is configured, the
-emulator only responds when the request `hhm_mac_code` matches `CT_MAC`; if
-`CT_MAC` is empty, it accepts requests for any CT MAC and echoes the request CT
-MAC back in responses.
+**or** matches the CT's own MAC, compared byte for byte. An empty
+`hhm_mac_code` does not parse, so the CT drops it. Either way the response
+carries the CT's **own** type and MAC, never the request's.
+
+A battery sends the wildcard while it has no CT bound and has not learned one
+yet. It accepts any response then, and puts that response's CT MAC into its
+next requests. Once a CT MAC is bound (set from the app over BLE or the
+cloud), it drops any response whose CT MAC is not byte‑for‑byte the bound one,
+so case and separators matter.
+
+> Verified in the CT002 (`HME-4` v120/v124) and CT003 (`HME-3`
+> v116/v117/v118/v122) images, and on the battery side in Venus E 3.0
+> (`VNSE3-0` v151) and Venus E 2.0 (`HMG-50` v156), from the firmware
+> archives cited above. The Venus E Mini (`VNSEM-0`), which needs the CT's own
+> MAC in the wildcard reply to pair, has no public image.
+
+In AstraMeter: if `CT_MAC` is configured, the emulator responds to the wildcard
+and to requests naming `CT_MAC` (case‑insensitively), always with `CT_MAC` in
+the response, and ignores every other CT MAC. `CT_MAC` is normalized to
+lowercase hex without separators, the form a battery stores. If `CT_MAC` is
+empty, it accepts requests for any CT MAC and echoes the request CT MAC back
+in responses.

docs/ct002.md

@@ -17,7 +17,8 @@ For the UDP wire protocol, see
 # CT type is derived from the emulated device (ct002 -> HME-4, ct003 -> HME-3).
 # CT MAC (12 hex digits, from Marstek app).
 # If empty, the emulator accepts any request CT MAC and echoes the request’s
-# CT MAC in responses. If set, the emulator responds only to that CT MAC.
+# CT MAC in responses. If set, the emulator responds only to that CT MAC and to
+# an unpaired battery's 000000000000 probe, answering with its own MAC.
 CT_MAC = 001122334455
 # UDP port to bind for CT002/CT003 (default 12345).
 UDP_PORT = 12345

esphome/components/ct002/ct002.cpp

@@ -48,6 +48,10 @@ constexpr float PEAKSHAVING_IDLE_BAND_W = 100.0f;
 // match between the two stacks.
 constexpr float POLL_INTERVAL_EMA_ALPHA = 0.3f;
 
+// The CT MAC a battery sends while it has no CT bound yet; answered like our
+// own MAC. Mirrors WILDCARD_CT_MAC in src/astrameter/ct002/ct002.py.
+constexpr const char *WILDCARD_CT_MAC = "000000000000";
+
 std::optional<float> ema_interval(std::optional<float> previous, float raw) {
   const auto round_tenth = [](float v) {
     return static_cast<float>(round_half_even(static_cast<double>(v) * 10.0)) / 10.0f;
@@ -569,7 +573,7 @@ bool CT002Component::validate_ct_mac_(const std::vector<std::string> &fields) co
   if (this->ct_mac_.empty()) return true;
   if (fields.size() < 4) return false;
   std::string req(fields[3]);
-  if (req.empty() || req == "000000000000") return true;
+  if (req == WILDCARD_CT_MAC) return true;
   for (auto &c : req) c = static_cast<char>(std::tolower(c));
   std::string cfg(this->ct_mac_);
   for (auto &c : cfg) c = static_cast<char>(std::tolower(c));

ha_addon/translations/en.yaml

@@ -10,7 +10,7 @@ configuration:
     description: "Comma-separated list of devices to emulate (ct002, ct003, shellypro3em, shellyemg3, shellyproem50, shellypro3em_old, shellypro3em_new). Use ct002 or ct003. The device type shellypro3em_old uses port 1010 (used up until B2500 firmware version 224). The device type shellypro3em_new uses the new port 2220 (used for B2500 firmware >=226). If you use shellypro3em, it will use shellypro3em_old and shellypro3em_new simultaneously for backwards compatibility."
   ct_mac:
     name: CT MAC Override
-    description: "Optional MAC address of the emulated CT device (the CT MAC addressed in incoming requests). Leave empty to respond regardless of addressed CT MAC and mirror that addressed CT MAC in the response. If set, only requests addressed to this CT MAC are answered."
+    description: "Optional MAC address of the emulated CT device (the CT MAC addressed in incoming requests). Leave empty to respond regardless of addressed CT MAC and mirror that addressed CT MAC in the response. If set, only requests addressed to this CT MAC, or to 000000000000 from a battery that is not paired yet, are answered."
   throttle_interval:
     name: Throttling Interval
     description: "Minimum time interval (in seconds) between power meter readings. When set to 0, throttling is disabled for maximum performance. For slower data sources or to prevent control instability, set to 1-3 seconds. Supports decimal values (e.g., 1.5 for 1.5 seconds). Increase this value if you're seeing oscillations or instability with slow data sources."

src/astrameter/ct002/ct002.py

@@ -64,6 +64,11 @@
 CLEANUP_INTERVAL_SECONDS = 5
 POLL_INTERVAL_EMA_ALPHA = 0.3
 
+# The CT MAC a battery sends while it has no CT bound yet. A real CT answers
+# it as well as its own MAC, and always replies with its own MAC — which is
+# how the battery learns it (verified in CT002/CT003 firmware v116-v124).
+WILDCARD_CT_MAC = "000000000000"
+
 # Under peak shaving, how close to 0 W (W) every battery of a pool trading
 # power must already be before each is held at 0 W on its own (see
 # CT002._peakshaving).  Sized to what the shared correction leaves stuck: its
@@ -115,6 +120,16 @@ def normalize_phase(raw: object) -> str:
     return phase if phase in STEERED_PHASES else "0"
 
 
+def normalize_ct_mac(raw: str) -> str:
+    """``CT_MAC`` as the battery stores it: lowercase hex, no separators.
+
+    The battery compares the MAC in our reply to its bound CT MAC byte for
+    byte, so ``AA:BB:CC:DD:EE:FF`` would never pair. Mirrors
+    ``_validate_ct_mac`` in esphome/components/ct002/__init__.py.
+    """
+    return raw.strip().replace(":", "").replace("-", "").lower()
+
+
 def _bucket_for_phase(phase: str) -> str:
     """Map a stored consumer phase to its aggregation bucket."""
     p = normalize_phase(phase)
@@ -457,7 +472,7 @@ def __init__(
         reset_fn: Callable[[], None] | None = None,
     ) -> None:
         self.udp_port = udp_port
-        self.ct_mac = ct_mac
+        self.ct_mac = normalize_ct_mac(ct_mac)
         self.ct_type = ct_type
         self.wifi_rssi = wifi_rssi
         self.dedupe_time_window = dedupe_time_window
@@ -1314,11 +1329,9 @@ async def _call_before_send(
 
     def _ct_mac_matches(self, request: CT002Request) -> bool:
         """Whether *request* names the CT MAC we are configured to answer for."""
-        if not self.ct_mac:
-            return True
-        if request.ct_mac in ("", "000000000000"):
+        if not self.ct_mac or request.ct_mac == WILDCARD_CT_MAC:
             return True
-        return bool(request.ct_mac) and request.ct_mac.lower() == self.ct_mac.lower()
+        return request.ct_mac.lower() == self.ct_mac
 
     async def _safe_handle_request(
         self, data: bytes, addr: tuple[str, int], transport: DatagramSink

src/astrameter/main.py

@@ -414,7 +414,7 @@ async def run_device(
                 ct,
                 device_id,
                 powermeters,
-                marstek_mac or ct.ct_mac,
+                marstek_mac or device.ct_mac,
                 insights is not None,
                 registry,
             )

tests/components/ct002/test.host.yaml

@@ -24,3 +24,5 @@ sensor:
 ct002:
   id: ct002_main
   power_sensor_l1: grid_l1
+  # A bound CT, so test_host_e2e.py can check which requests it answers.
+  ct_mac: "112233445566"

tests/components/ct002/test_host_e2e.py

@@ -27,6 +27,7 @@
 
 import pytest
 
+from astrameter.ct002.protocol import build_payload, parse_request
 from astrameter.simulator.battery import BatterySimulator
 
 HERE = Path(__file__).parent
@@ -156,6 +157,35 @@ async def go() -> list[str] | None:
     assert len(fields) == 24, f"expected 24 fields, got {len(fields)}: {fields}"
     # ct_type and ct_mac come back as configured on the ct002 component.
     assert fields[0] == "HME-4", fields[0]
+    assert fields[1] == "112233445566", fields[1]
     # meter_dev_type/meter_mac echo the request.
     assert fields[2] == "HMG-50", fields[2]
     assert fields[3].lower() == "aabbccddeeff", fields[3]
+
+
+def _ask(ct_mac: str) -> list[str] | None:
+    """Send one request addressed to *ct_mac*; the reply's fields, or None."""
+    request = build_payload(["HMG-50", "AABBCCDDEEFF", "HME-4", ct_mac, "0", "0"])
+    with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:
+        sock.settimeout(1.0)
+        sock.sendto(request, ("127.0.0.1", 12345))
+        try:
+            data, _ = sock.recvfrom(1024)
+        except TimeoutError:
+            return None
+    fields, error = parse_request(data)
+    assert fields is not None, error
+    return fields
+
+
+@pytest.mark.timeout(30, func_only=True)
+def test_ct_mac_policy(running_binary: subprocess.Popen) -> None:
+    """test.host.yaml binds ct_mac 112233445566. Like a real CT, the binary
+    answers its own MAC and the unbound-battery probe 000000000000, replying
+    with its own MAC either way, and ignores any other CT MAC."""
+    for ct_mac in ("112233445566", "000000000000"):
+        fields = _ask(ct_mac)
+        assert fields is not None, f"no reply to a request for {ct_mac}"
+        assert fields[1] == "112233445566", fields
+    for ct_mac in ("", "aabbccddeeff"):
+        assert _ask(ct_mac) is None, f"answered a request for {ct_mac!r}"

tests/test_ct002_mac_policy.py

@@ -1,6 +1,6 @@
 from unittest.mock import MagicMock
 
-from astrameter.ct002.ct002 import CT002, build_payload
+from astrameter.ct002.ct002 import CT002, build_payload, parse_request
 
 
 def make_request(ct_mac):
@@ -33,3 +33,40 @@ async def test_ct002_configured_mac_accepts_match() -> None:
         make_request("AABBCCDDEEFF"), ("1.1.1.1", 12345), transport
     )
     transport.sendto.assert_called_once()
+
+
+def reply_fields(transport: MagicMock) -> list[str]:
+    transport.sendto.assert_called_once()
+    fields, error = parse_request(transport.sendto.call_args[0][0])
+    assert fields is not None, error
+    return fields
+
+
+async def test_ct002_answers_zero_mac_probe_with_configured_mac() -> None:
+    # An unbound battery probes with CT MAC 000000000000; a real CT answers it
+    # with its own MAC, which is how the battery learns which CT to pair with.
+    device = CT002(ct_mac="02b250aabbcc")
+    transport = MagicMock()
+    await device._handle_request(
+        make_request("000000000000"), ("1.1.1.1", 12345), transport
+    )
+    assert reply_fields(transport)[1] == "02b250aabbcc"
+
+
+async def test_ct002_configured_mac_ignores_empty_ct_mac() -> None:
+    # Only 000000000000 is a wildcard: a real CT drops an empty CT MAC field.
+    device = CT002(ct_mac="02b250aabbcc")
+    transport = MagicMock()
+    await device._handle_request(make_request(""), ("1.1.1.1", 12345), transport)
+    transport.sendto.assert_not_called()
+
+
+async def test_ct002_replies_with_normalized_configured_mac() -> None:
+    # The battery compares our reply's CT MAC byte for byte with the one it
+    # stores (lowercase, no separators), so CT_MAC is normalized to that form.
+    device = CT002(ct_mac="AA:BB:CC:DD:EE:FF")
+    transport = MagicMock()
+    await device._handle_request(
+        make_request("aabbccddeeff"), ("1.1.1.1", 12345), transport
+    )
+    assert reply_fields(transport)[1] == "aabbccddeeff"

tests/test_ct_settings_wiring.py

@@ -112,7 +112,8 @@ def distinct_settings() -> CtSettings:
         elif isinstance(default, float):
             values[field.name] = 0.1001 + index / 10000
         elif isinstance(default, str):
-            values[field.name] = f"value-{index}"
+            # Lowercase, no separators: CT002 normalizes ct_mac to that form.
+            values[field.name] = f"value{index}"
         elif default is None:  # consumer_ttl
             values[field.name] = 1000 + index
     return replace(CtSettings(), **values)