Fix 1849

.audit/oberstet_fix_1849.md

@@ -0,0 +1,8 @@
+- [ ] I did **not** use any AI-assistance tools to help create this pull request.
+- [x] I **did** use AI-assistance tools to *help* create this pull request.
+- [x] I have read, understood and followed the projects' [AI Policy](https://github.com/crossbario/autobahn-python/blob/main/AI_POLICY.md) when creating code, documentation etc. for this pull request.
+
+Submitted by: @oberstet
+Date: 2025-06-13
+Related issue(s): #1849
+Branch: oberstet:fix_1849

README.md

@@ -343,14 +343,14 @@ masking) and UTF-8 validation.
 
 ### WAMP Serializers
 
-**As of v25.11.1, all WAMP serializers are included by default** - batteries included!
+**The JSON, MessagePack, CBOR and FlatBuffers serializers are included by default** - batteries included! UBJSON is available via the optional `autobahn[serialization]` extra.
 
-Autobahn|Python now ships with full support for all WAMP serializers out-of-the-box:
+Autobahn|Python ships with the following WAMP serializers:
 
 - **JSON** (standard library) - always available
 - **MessagePack** - high-performance binary serialization
 - **CBOR** - IETF standard binary serialization (RFC 8949)
-- **UBJSON** - Universal Binary JSON
+- **UBJSON** - Universal Binary JSON *(optional: `pip install autobahn[serialization]`)*
 - **Flatbuffers** - Google's zero-copy serialization (vendored)
 
 #### Architecture & Performance
@@ -363,12 +363,12 @@ The serializer dependencies are optimized for both **CPython** and **PyPy**:
 | **msgpack** | Binary wheel (C extension) | u-msgpack-python (pure Python) | Native + Universal | PyPy JIT makes pure Python faster than C |
 | **ujson** | Binary wheel | Binary wheel | Native | Available for both implementations |
 | **cbor2** | Binary wheel | Pure Python fallback | Native + Universal | Binary wheels + py3-none-any |
-| **ubjson** | Pure Python | Pure Python | Source | Set `PYUBJSON_NO_EXTENSION=1` to skip C build |
+| **ubjson** *(optional)* | C ext (from sdist) | ❌ n/a (CPython only) | Source only — no wheels | Optional `autobahn[serialization]` extra (bjdata, pulls numpy). CPython-only: bjdata can't install on PyPy ([NeuroJSON/pybj#6](https://github.com/NeuroJSON/pybj/issues/6)); on PyPy use cbor/msgpack. On CPython without a compiler set `PYBJDATA_NO_EXTENSION=1` |
 | **flatbuffers** | Vendored | Vendored | Included | Always available, no external dependency |
 
 **Key Design Principles:**
 
-1. **Batteries Included**: All serializers available without extra install steps
+1. **Batteries Included**: Core serializers (JSON, MessagePack, CBOR, FlatBuffers) available without extra install steps; UBJSON via the optional `autobahn[serialization]` extra
 2. **PyPy Optimization**: Pure Python implementations leverage PyPy's JIT for superior performance
 3. **Binary Wheels**: Native wheels for all major platforms (Linux x86_64/ARM64, macOS x86_64/ARM64, Windows x86_64)
 4. **Zero System Pollution**: All dependencies install cleanly via wheels or pure Python
@@ -429,15 +429,15 @@ All dependencies follow these design principles:
 
 ### WAMP Serializers (Batteries Included)
 
-All serializers are now **included by default** in the base installation:
+All serializers **except UBJSON** are included by default in the base installation; UBJSON is an optional extra (`pip install autobahn[serialization]`):
 
 | Serializer | Purpose | CPython | PyPy | Wheel Coverage | Notes |
 |------------|---------|---------|------|----------------|-------|
 | **json** | JSON serialization | stdlib | stdlib | ✅ Always available | Python standard library |
 | **msgpack** | MessagePack serialization | msgpack (binary wheel) | u-msgpack-python (pure Python) | ✅ Excellent | 50+ wheels for CPython; PyPy JIT optimized |
 | **ujson** | Fast JSON (optional) | Binary wheel | Binary wheel | ✅ Excellent | 30+ wheels; both implementations |
 | **cbor2** | CBOR serialization (RFC 8949) | Binary wheel | Pure Python fallback | ✅ Excellent | 30+ binary wheels + universal fallback |
-| **py-ubjson** | UBJSON serialization | Pure Python | Pure Python | ✅ Good | Optional C extension (can skip with `PYUBJSON_NO_EXTENSION=1`) |
+| **bjdata** | UBJSON serialization *(optional)* | C ext (from sdist) | ❌ n/a (CPython only) | ⚠️ sdist only — no wheels | `autobahn[serialization]` extra; pulls numpy. CPython-only: can't install on PyPy ([NeuroJSON/pybj#6](https://github.com/NeuroJSON/pybj/issues/6)). On CPython without a compiler set `PYBJDATA_NO_EXTENSION=1`. For wheels-only/cross-arch/PyPy installs prefer cbor/msgpack |
 | **flatbuffers** | Google Flatbuffers | **Vendored** | **Vendored** | ✅ Perfect | Included in our wheel, zero external dependency |
 
 ### Optional: Twisted Framework

docs/changelog.rst

@@ -8,6 +8,13 @@ Changelog
 26.6.1
 ------
 
+**WAMP Serialization**
+
+* ``py-ubjson`` (unmaintained, sdist-only) is no longer an unconditional dependency. A base ``pip install autobahn`` — and the wheels-only / cross-arch case from #1849 (``pip download --only-binary :all: --platform ...``) — now resolves entirely from binary wheels (#1849)
+* The WAMP ``ubjson`` serializer is now backed by the maintained ``bjdata`` (Binary JData) package, provided as the OPTIONAL ``autobahn[serialization]`` extra (it also pulls in numpy), keeping both out of a minimal install (#1849)
+* ``bjdata`` is published sdist-only (no PyPI wheels) and is currently **CPython-only**: on PyPy its sdist build pulls an unbuildable numpy (upstream ``NeuroJSON/pybj#6``), so the ``ubjson`` serializer is unavailable on PyPy - use ``cbor``/``msgpack`` there. On CPython without a compiler, set ``PYBJDATA_NO_EXTENSION=1`` for a pure-Python build. For wheels-only or cross-arch deployments, also prefer ``cbor``/``msgpack`` (#1849)
+* ⚠️ **Wire-level change to watch out for:** bjdata's octet-level encoding is NOT identical to the previous py-ubjson/UBJSON bytes (different integer markers, little-endian). The WAMP serializer id remains ``ubjson`` for transport negotiation. The ``wamp-proto`` UBJSON test vectors will be regenerated in a follow-up PR after this release; until then the ``ubjson`` serializer is excluded from the byte-vector conformance suite (round-trip and cross-serializer coverage retained) (#1849)
+
 **FlatBuffers**
 
 * Bump vendored FlatBuffers from v25.9.23 to v25.12.19, restoring the version-sync with zlmdb 26.6.1 (#1853)

docs/installation.rst

@@ -152,7 +152,7 @@ Install Variants
    * - ``scram``
      - Install WAMP-SCRAM authentication packages.
    * - ``serialization``
-     - Backwards-compatible no-op; WAMP serializers are included by default.
+     - Install ``bjdata`` to enable the optional UBJSON WAMP serializer. CPython-only - it cannot install on PyPy (upstream ``NeuroJSON/pybj#6``); on CPython without a compiler set ``PYBJDATA_NO_EXTENSION=1`` for a pure-Python build. JSON, MessagePack, CBOR and FlatBuffers are always available without this extra (use them on PyPy).
    * - ``nvx``
      - Backwards-compatible no-op; NVX acceleration is included in binary wheels where supported.
    * - ``all``

examples/serdes/tests/conftest.py

@@ -12,6 +12,20 @@
 from .utils import load_test_vector, get_serializer_ids
 
 
+# The WAMP "ubjson" serializer is backed by bjdata (autobahn #1849). The
+# wamp-proto canonical vectors (via the .proto submodule) now carry BOTH the
+# legacy py-ubjson bytes and the new bjdata/BJData bytes for "ubjson" (each
+# tagged with a "note"), matched with "at least one must match" semantics. The
+# deserialize-direction tests use utils.require_decodable() to skip byte
+# variants this backend cannot decode (the two encodings are not mutually
+# decodable), so no serializer needs to be excluded from the byte-vector suite.
+_VECTOR_EXCLUDED_SERIALIZERS = ()
+
+
+def _conformance_serializer_ids():
+    return [s for s in get_serializer_ids() if s not in _VECTOR_EXCLUDED_SERIALIZERS]
+
+
 @pytest.fixture(scope="session")
 def wamp_test_vector_publish():
     """Load PUBLISH test vector"""
@@ -48,12 +62,12 @@ def pytest_generate_tests(metafunc):
     This generates test parameters for serializer_id based on available serializers.
     """
     if "serializer_id" in metafunc.fixturenames:
-        serializer_ids = get_serializer_ids()
+        serializer_ids = _conformance_serializer_ids()
         metafunc.parametrize("serializer_id", serializer_ids)
 
     if "serializer_pair" in metafunc.fixturenames:
         # Generate all unique pairs of serializers for cross-serializer tests
-        serializer_ids = get_serializer_ids()
+        serializer_ids = _conformance_serializer_ids()
         pairs = []
         for i, ser1 in enumerate(serializer_ids):
             for ser2 in serializer_ids[i + 1 :]:

examples/serdes/tests/test_abort.py

@@ -14,6 +14,7 @@
 from autobahn.wamp.serializer import create_transport_serializer
 
 from .utils import (
+    require_decodable,
     load_test_vector,
     bytes_from_hex,
     matches_any_byte_representation,
@@ -62,7 +63,7 @@ def test_abort_deserialize_from_bytes(serializer_id, abort_samples, create_seria
         byte_variants = sample["serializers"][serializer_id]
 
         # Try deserializing each byte variant
-        for variant in byte_variants:
+        for variant in require_decodable(serializer, byte_variants):
             # Get bytes
             if "bytes_hex" in variant:
                 test_bytes = bytes_from_hex(variant["bytes_hex"])
@@ -129,7 +130,7 @@ def test_abort_roundtrip(serializer_id, abort_samples, create_serializer):
         if not byte_variants:
             continue
 
-        variant = byte_variants[0]
+        variant = require_decodable(serializer, byte_variants)[0]
         if "bytes_hex" in variant:
             original_bytes = bytes_from_hex(variant["bytes_hex"])
         elif "bytes" in variant:

examples/serdes/tests/test_authenticate.py

@@ -14,6 +14,7 @@
 from autobahn.wamp.serializer import create_transport_serializer
 
 from .utils import (
+    require_decodable,
     load_test_vector,
     bytes_from_hex,
     matches_any_byte_representation,
@@ -64,7 +65,7 @@ def test_authenticate_deserialize_from_bytes(
         byte_variants = sample["serializers"][serializer_id]
 
         # Try deserializing each byte variant
-        for variant in byte_variants:
+        for variant in require_decodable(serializer, byte_variants):
             # Get bytes
             if "bytes_hex" in variant:
                 test_bytes = bytes_from_hex(variant["bytes_hex"])
@@ -133,7 +134,7 @@ def test_authenticate_roundtrip(serializer_id, authenticate_samples, create_seri
         if not byte_variants:
             continue
 
-        variant = byte_variants[0]
+        variant = require_decodable(serializer, byte_variants)[0]
         if "bytes_hex" in variant:
             original_bytes = bytes_from_hex(variant["bytes_hex"])
         elif "bytes" in variant:

examples/serdes/tests/test_call.py

@@ -14,6 +14,7 @@
 from autobahn.wamp.serializer import create_transport_serializer
 
 from .utils import (
+    require_decodable,
     load_test_vector,
     bytes_from_hex,
     matches_any_byte_representation,
@@ -62,7 +63,7 @@ def test_call_deserialize_from_bytes(serializer_id, call_samples, create_seriali
         byte_variants = sample["serializers"][serializer_id]
 
         # Try deserializing each byte variant
-        for variant in byte_variants:
+        for variant in require_decodable(serializer, byte_variants):
             # Get bytes
             if "bytes_hex" in variant:
                 test_bytes = bytes_from_hex(variant["bytes_hex"])
@@ -149,7 +150,7 @@ def test_call_roundtrip(serializer_id, call_samples, create_serializer):
         if not byte_variants:
             continue
 
-        variant = byte_variants[0]
+        variant = require_decodable(serializer, byte_variants)[0]
         if "bytes_hex" in variant:
             original_bytes = bytes_from_hex(variant["bytes_hex"])
         elif "bytes" in variant:

examples/serdes/tests/test_cancel.py

@@ -14,6 +14,7 @@
 from autobahn.wamp.serializer import create_transport_serializer
 
 from .utils import (
+    require_decodable,
     load_test_vector,
     bytes_from_hex,
     matches_any_byte_representation,
@@ -64,7 +65,7 @@ def test_cancel_deserialize_from_bytes(
         byte_variants = sample["serializers"][serializer_id]
 
         # Try deserializing each byte variant
-        for variant in byte_variants:
+        for variant in require_decodable(serializer, byte_variants):
             # Get bytes
             if "bytes_hex" in variant:
                 test_bytes = bytes_from_hex(variant["bytes_hex"])
@@ -131,7 +132,7 @@ def test_cancel_roundtrip(serializer_id, cancel_samples, create_serializer):
         if not byte_variants:
             continue
 
-        variant = byte_variants[0]
+        variant = require_decodable(serializer, byte_variants)[0]
         if "bytes_hex" in variant:
             original_bytes = bytes_from_hex(variant["bytes_hex"])
         elif "bytes" in variant:

examples/serdes/tests/test_challenge.py

@@ -14,6 +14,7 @@
 from autobahn.wamp.serializer import create_transport_serializer
 
 from .utils import (
+    require_decodable,
     load_test_vector,
     bytes_from_hex,
     matches_any_byte_representation,
@@ -64,7 +65,7 @@ def test_challenge_deserialize_from_bytes(
         byte_variants = sample["serializers"][serializer_id]
 
         # Try deserializing each byte variant
-        for variant in byte_variants:
+        for variant in require_decodable(serializer, byte_variants):
             # Get bytes
             if "bytes_hex" in variant:
                 test_bytes = bytes_from_hex(variant["bytes_hex"])
@@ -133,7 +134,7 @@ def test_challenge_roundtrip(serializer_id, challenge_samples, create_serializer
         if not byte_variants:
             continue
 
-        variant = byte_variants[0]
+        variant = require_decodable(serializer, byte_variants)[0]
         if "bytes_hex" in variant:
             original_bytes = bytes_from_hex(variant["bytes_hex"])
         elif "bytes" in variant:

examples/serdes/tests/test_error.py

@@ -14,6 +14,7 @@
 from autobahn.wamp.serializer import create_transport_serializer
 
 from .utils import (
+    require_decodable,
     load_test_vector,
     bytes_from_hex,
     matches_any_byte_representation,
@@ -62,7 +63,7 @@ def test_error_deserialize_from_bytes(serializer_id, error_samples, create_seria
         byte_variants = sample["serializers"][serializer_id]
 
         # Try deserializing each byte variant
-        for variant in byte_variants:
+        for variant in require_decodable(serializer, byte_variants):
             # Get bytes
             if "bytes_hex" in variant:
                 test_bytes = bytes_from_hex(variant["bytes_hex"])
@@ -135,7 +136,7 @@ def test_error_roundtrip(serializer_id, error_samples, create_serializer):
         if not byte_variants:
             continue
 
-        variant = byte_variants[0]
+        variant = require_decodable(serializer, byte_variants)[0]
         if "bytes_hex" in variant:
             original_bytes = bytes_from_hex(variant["bytes_hex"])
         elif "bytes" in variant:

examples/serdes/tests/test_event.py

@@ -14,6 +14,7 @@
 from autobahn.wamp.serializer import create_transport_serializer
 
 from .utils import (
+    require_decodable,
     load_test_vector,
     bytes_from_hex,
     validate_message_with_code,
@@ -73,7 +74,7 @@ def test_event_deserialize_from_bytes(serializer_id, event_samples, create_seria
         byte_variants = sample["serializers"][serializer_id]
 
         # Try deserializing each byte variant
-        for variant in byte_variants:
+        for variant in require_decodable(serializer, byte_variants):
             # Get bytes
             if "bytes_hex" in variant:
                 test_bytes = bytes_from_hex(variant["bytes_hex"])
@@ -223,7 +224,7 @@ def test_event_cross_serializer_preservation(serializer_pair, event_samples):
             pytest.skip(f"No byte variants for {ser1_id} or {ser2_id}")
 
         # Take the first canonical byte representation from ser1
-        variant1 = ser1_variants[0]
+        variant1 = require_decodable(ser1, ser1_variants)[0]
         if "bytes_hex" in variant1:
             bytes1 = bytes_from_hex(variant1["bytes_hex"])
         elif "bytes" in variant1:

examples/serdes/tests/test_eventreceived.py

@@ -14,6 +14,7 @@
 from autobahn.wamp.serializer import create_transport_serializer
 
 from .utils import (
+    require_decodable,
     load_test_vector,
     bytes_from_hex,
     matches_any_byte_representation,
@@ -64,7 +65,7 @@ def test_eventreceived_deserialize_from_bytes(
         byte_variants = sample["serializers"][serializer_id]
 
         # Try deserializing each byte variant
-        for variant in byte_variants:
+        for variant in require_decodable(serializer, byte_variants):
             # Get bytes
             if "bytes_hex" in variant:
                 test_bytes = bytes_from_hex(variant["bytes_hex"])
@@ -135,7 +136,7 @@ def test_eventreceived_roundtrip(
         if not byte_variants:
             continue
 
-        variant = byte_variants[0]
+        variant = require_decodable(serializer, byte_variants)[0]
         if "bytes_hex" in variant:
             original_bytes = bytes_from_hex(variant["bytes_hex"])
         elif "bytes" in variant:

examples/serdes/tests/test_goodbye.py

@@ -14,6 +14,7 @@
 from autobahn.wamp.serializer import create_transport_serializer
 
 from .utils import (
+    require_decodable,
     load_test_vector,
     bytes_from_hex,
     matches_any_byte_representation,
@@ -64,7 +65,7 @@ def test_goodbye_deserialize_from_bytes(
         byte_variants = sample["serializers"][serializer_id]
 
         # Try deserializing each byte variant
-        for variant in byte_variants:
+        for variant in require_decodable(serializer, byte_variants):
             # Get bytes
             if "bytes_hex" in variant:
                 test_bytes = bytes_from_hex(variant["bytes_hex"])
@@ -131,7 +132,7 @@ def test_goodbye_roundtrip(serializer_id, goodbye_samples, create_serializer):
         if not byte_variants:
             continue
 
-        variant = byte_variants[0]
+        variant = require_decodable(serializer, byte_variants)[0]
         if "bytes_hex" in variant:
             original_bytes = bytes_from_hex(variant["bytes_hex"])
         elif "bytes" in variant:

examples/serdes/tests/test_hello.py

@@ -20,6 +20,7 @@
 )
 
 from .utils import (
+    require_decodable,
     load_test_vector,
     bytes_from_hex,
     matches_any_byte_representation,
@@ -95,7 +96,7 @@ def test_hello_deserialize_from_bytes(serializer_id, hello_samples, create_seria
         byte_variants = sample["serializers"][serializer_id]
 
         # Try deserializing each byte variant
-        for variant in byte_variants:
+        for variant in require_decodable(serializer, byte_variants):
             # Get bytes
             if "bytes_hex" in variant:
                 test_bytes = bytes_from_hex(variant["bytes_hex"])
@@ -165,7 +166,7 @@ def test_hello_roundtrip(serializer_id, hello_samples, create_serializer):
         if not byte_variants:
             continue
 
-        variant = byte_variants[0]
+        variant = require_decodable(serializer, byte_variants)[0]
         if "bytes_hex" in variant:
             original_bytes = bytes_from_hex(variant["bytes_hex"])
         elif "bytes" in variant: