From 58c80ac03c388c8beaa6e7119ddb7172c036dcd1 Mon Sep 17 00:00:00 2001 From: Tobias Oberstein Date: Wed, 17 Jun 2026 14:48:19 +0200 Subject: [PATCH 1/3] start new dev branch; add audit file --- .audit/oberstet_fix_1830.md | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 .audit/oberstet_fix_1830.md diff --git a/.audit/oberstet_fix_1830.md b/.audit/oberstet_fix_1830.md new file mode 100644 index 000000000..77e3e4461 --- /dev/null +++ b/.audit/oberstet_fix_1830.md @@ -0,0 +1,8 @@ +- [ ] I did **not** use any AI-assistance tools to help create this pull request. +- [x] I **did** use AI-assistance tools to *help* create this pull request. +- [x] I have read, understood and followed the projects' [AI Policy](https://github.com/crossbario/autobahn-python/blob/main/AI_POLICY.md) when creating code, documentation etc. for this pull request. + +Submitted by: @oberstet +Date: 2026-06-17 +Related issue(s): #1830 +Branch: oberstet:fix_1830 From 662d9308f4f22871a2af6f0dcbd312b64d1af8a9 Mon Sep 17 00:00:00 2001 From: Tobias Oberstein Date: Wed, 17 Jun 2026 15:02:49 +0200 Subject: [PATCH 2/3] Make FlatBuffers generated-code verification actually detect drift (#1830) Two bugs let stale generated files accumulate undetected: 1. `build-fbs` invoked a system `flatc` (whatever version is on PATH) instead of the vendored, version-matched flatc. A version mismatch can silently produce different generated code. 2. The CI verification ran `build-fbs` on top of the committed tree WITHOUT cleaning first, so any orphaned file (no longer produced by the schema) survived in both the "before" and "after" states, matched its own checksum, and was never flagged. Fixes: - justfile: build-fbs now uses the bundled flatc (${VENV_PATH}/bin/flatc), built from deps/flatbuffers during install, with a clear error if it is missing. - main.yml: run `just clean-fbs` before `just build-fbs` so a clean regeneration is compared against the committed tree. - main.yml: replace the opaque checksum diff with an actionable, categorized table derived from `git status` of the regenerated tree: content differs -> regenerate & commit (e.g. newer flatc) orphan / not generated -> delete (#1828) new, not committed -> commit Validated locally with the vendored v25.12.19 flatc: a clean regenerate differs from the committed tree by exactly two files, both orphans (ChannelBinding.py, Kdf.py) - confirming no flatc content drift, only the #1828 leftovers. NOTE: this commit intentionally leaves those two orphan files in place, so CI is EXPECTED to fail on this commit - that red run proves the detection mechanism works and that the files are genuinely stale. The orphans are removed in the next commit (#1828), which turns CI green. Refs #1828. Note: This work was completed with AI assistance (Claude Code). --- .github/workflows/main.yml | 57 +++++++++++++++++++++++--------------- docs/changelog.rst | 1 + justfile | 10 ++++++- 3 files changed, 45 insertions(+), 23 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index ce1a7f26f..e4acafc58 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -466,11 +466,11 @@ jobs: just create ${{ matrix.python-env }} just install-tools ${{ matrix.python-env }} - - name: Store original generated files checksum - run: | - echo "==> Computing checksums of committed FlatBuffers files..." - find src/autobahn/wamp/gen/ -name "*.py" -o -name "*.bfbs" | sort | xargs sha256sum > /tmp/original-checksums.txt - echo "Found $(wc -l < /tmp/original-checksums.txt) committed generated files" + - name: Clean generated FlatBuffers files (so orphans become detectable) + # Without this, `build-fbs` regenerates on top of the committed tree and + # any orphaned/stale file (no longer produced by the schema) survives in + # both the before and after states, hiding the drift (#1830). + run: just clean-fbs - name: Build FlatBuffers binary schema & Python wrappers run: just build-fbs @@ -489,24 +489,37 @@ jobs: - name: Verify regenerated files match committed files run: | - echo "==> Computing checksums of regenerated files..." - find src/autobahn/wamp/gen/ -name "*.py" -o -name "*.bfbs" | sort | xargs sha256sum > /tmp/regenerated-checksums.txt - - echo "==> Comparing checksums..." - if diff -u /tmp/original-checksums.txt /tmp/regenerated-checksums.txt; then - echo "✅ SUCCESS: All regenerated FlatBuffers files match committed versions exactly!" - echo "This confirms the build process is reproducible and up-to-date." - else - echo "❌ FAILURE: Regenerated files differ from committed versions!" - echo "" - echo "This indicates either:" - echo "1. The committed files were generated with a different flatc version" - echo "2. The committed files were manually modified" - echo "3. The build process has changed since files were last committed" - echo "" - echo "Please run 'just clean-fbs && just build-fbs' locally and commit the results." - exit 1 + # Compare a clean regeneration (clean-fbs + build-fbs, above) against the + # committed generated tree using git's own classification, and fail with + # an actionable, categorized table (#1830): + # M -> content differs (e.g. newer vendored flatc) -> regenerate & commit + # D -> committed but NOT produced by flatc -> orphan/attic leftover, delete (#1828) + # ?? -> newly generated, not yet committed -> commit + status=$(git status --porcelain -- src/autobahn/wamp/gen/) + if [ -z "${status}" ]; then + echo "✅ SUCCESS: committed FlatBuffers generated code matches a clean regenerate." + exit 0 fi + echo "❌ FAILURE: committed FlatBuffers generated code drifted from a clean regenerate." + echo "" + printf '%-36s %-21s %s\n' "CATEGORY" "ACTION" "FILE" + printf '%-36s %-21s %s\n' "------------------------------------" "---------------------" "----" + while IFS= read -r line; do + code="${line:0:2}" + file="${line:3}" + case "${code}" in + " M"|"M "|"MM") category="content differs" ; action="regenerate & commit" ;; + " D"|"D ") category="orphan / not generated [#1828]" ; action="delete" ;; + "??") category="new, not committed" ; action="commit" ;; + *) category="changed (${code})" ; action="review" ;; + esac + printf '%-36s %-21s %s\n' "${category}" "${action}" "${file}" + done <<< "${status}" + echo "" + count=$(printf '%s\n' "${status}" | grep -c .) + echo "FAIL: ${count} generated file(s) drifted." + echo "Run 'just clean-fbs && just build-fbs' locally and commit the result." + exit 1 # Upload FlatBuffers artifacts (split into two uploads since verified action # doesn't support multi-line paths like actions/upload-artifact@v4) diff --git a/docs/changelog.rst b/docs/changelog.rst index 1f2590b37..83be4fddd 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -38,6 +38,7 @@ Changelog * Build and publish the missing CPython 3.12 and 3.14 ``manylinux_*_aarch64`` (ARM64) wheels. The per-version ARM64 build matrix (added in commit 3d856f5 to deduplicate wheels) only covered cp311 and cp313, so no cp312 aarch64 wheel was ever published (and cp314 was never added) - e.g. ``pip download autobahn --platform manylinux_2_34_aarch64`` for CPython 3.12 found no matching distribution. The strict release fileset manifest shared the same gap and so could not catch it; it now requires the cp312/cp314 aarch64 wheels (fail-closed). Thanks to @norrisjeremy for the report (#1848) * Make the release fileset symmetric across all four platforms: every supported interpreter (cp311, cp312, cp313, cp314, pypy311) is now required on macOS/arm64, Linux/x86_64, Linux/aarch64, and Windows/amd64. The macOS job already built all interpreters via ``just build-all``, but the manifest only required cp313/cp314/pypy311, so the cp311/cp312 macOS wheels were built and then dropped as "extra" rather than published; they are now kept and required (#1848) * Remove orphaned/attic files left over from the pre-``justfile``/``uv`` CI/CD system: ``Makefile.orig``, ``Dockerfile.wheels``, ``mypy.ini``, ``test-docker-builds.sh``, ``versions.sh``, ``deploy.sh``, ``.prettierrc.json``, ``.coveragerc``, ``docs/DOCKER_BUILDS.md``, ``docker/README.md`` and the ``pyinstaller/`` PyInstaller hooks, plus the unused ``pyinstaller`` dev dependency. The ``.coveragerc`` ``omit = */test/*.py`` setting was preserved by migrating it to ``[tool.coverage.run]`` in ``pyproject.toml`` (so coverage still excludes in-package test modules), and the stale ``DOCKER_BUILDS.md`` entry was dropped from the Sphinx ``exclude_patterns``. ``mypy`` is unaffected: the typing recipe already passes ``--config-file pyproject-static-typing.toml`` explicitly. ``setuptools`` was added explicitly to the ``dev`` extra: it is required by ``cffi``'s ``ffi.compile()`` to build the NVX extensions in an editable install on Python >= 3.12 (stdlib ``distutils`` was removed in 3.12) and had been pulled in only transitively via the removed ``pyinstaller`` (#1831) +* Fix the FlatBuffers generated-code verification so it actually detects drift. The ``build-fbs`` recipe now uses the vendored, version-matched ``flatc`` bundled in the venv (``${VENV_PATH}/bin/flatc``) instead of an arbitrary system ``flatc``, and the CI job runs ``just clean-fbs`` before ``just build-fbs`` so orphaned/stale generated files no longer survive in both the before and after states (previously they matched checksums and went undetected). On drift the job now fails with an actionable, categorized table - ``content differs`` (regenerate & commit, e.g. after a vendored-``flatc`` bump), ``orphan / not generated`` (delete), ``new, not committed`` (commit) - derived from ``git status`` of the regenerated tree (#1830) 25.12.2 ------- diff --git a/justfile b/justfile index d8d6d08cd..cf339a6a0 100644 --- a/justfile +++ b/justfile @@ -1941,7 +1941,15 @@ build-fbs venv="": (install-tools venv) VENV_PATH="{{ VENV_DIR }}/${VENV_NAME}" FBSFILES="./src/autobahn/wamp/flatbuffers/*.fbs" - FLATC="flatc" + # Use the vendored, version-matched flatc bundled in the venv (built from + # deps/flatbuffers during install), NOT a system flatc which may be a + # different version and silently produce different generated code (#1830). + FLATC="${VENV_PATH}/bin/flatc" + if [ ! -x "${FLATC}" ]; then + echo "ERROR: bundled flatc not found at ${FLATC}" >&2 + echo " Run 'just install-tools ${VENV_NAME}' first (it builds the vendored flatc)." >&2 + exit 1 + fi echo "==> Generating FlatBuffers binary schema and Python wrappers using $(${FLATC} --version)..." # Generate schema binary type library (*.bfbs files) From 783a1c6769b58f4e7d139b637391b0cbc7bf14d4 Mon Sep 17 00:00:00 2001 From: Tobias Oberstein Date: Wed, 17 Jun 2026 16:11:34 +0200 Subject: [PATCH 3/3] Delete orphaned generated FlatBuffers files Kdf.py / ChannelBinding.py (#1828) These two files in src/autobahn/wamp/gen/wamp/proto/ are stale leftovers from before the schema renamed the tables to KDF and TLSChannelBinding. They are no longer produced by `flatc`, and their case-insensitive collisions with the current KDF.py / TLSChannelBinding.py broke `git clone` and directory copies on case-insensitive filesystems (APFS/macOS, some Docker containers). Produced by `just clean-fbs && just build-fbs` with the vendored v25.12.19 flatc: a clean regeneration of the canonical tree differs from the committed tree by exactly these two deletions (every other generated file is byte-identical), so this commit is precisely "drop the orphans". With the detection fix from the previous commit (#1830) now in place, CI's clean-regenerate-and-compare turns green, proving the two files were genuine attic leftovers. Fixes #1828. Note: This work was completed with AI assistance (Claude Code). --- docs/changelog.rst | 1 + src/autobahn/wamp/gen/wamp/proto/ChannelBinding.py | 8 -------- src/autobahn/wamp/gen/wamp/proto/Kdf.py | 9 --------- 3 files changed, 1 insertion(+), 17 deletions(-) delete mode 100644 src/autobahn/wamp/gen/wamp/proto/ChannelBinding.py delete mode 100644 src/autobahn/wamp/gen/wamp/proto/Kdf.py diff --git a/docs/changelog.rst b/docs/changelog.rst index 83be4fddd..42654e323 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -25,6 +25,7 @@ Changelog * Commit the binary schemas (``reflection.bfbs``, ``wamp.bfbs``) to the source tree and ship them as-is; the package build no longer runs ``flatc``, which fixes cross-compilation from the PyPI sdist (e.g. Buildroot/Yocto/aarch64) (#1853) * Add ``just generate-reflection`` to regenerate the committed binary schemas with a version-matched ``flatc`` built from ``deps/flatbuffers`` (#1853) * Add ``just check-flatbuffers-sync`` and a unit test exercising ``check_zlmdb_flatbuffers_version_in_sync()`` (#1853) +* Delete two orphaned generated files, ``Kdf.py`` and ``ChannelBinding.py``, left in ``src/autobahn/wamp/gen/wamp/proto/`` after the schema renamed those tables to ``KDF`` and ``TLSChannelBinding``. Their case-insensitive collisions with the current ``KDF.py`` / ``TLSChannelBinding.py`` broke ``git clone`` and directory copies on case-insensitive filesystems (APFS/macOS, some Docker setups). They are no longer produced by ``flatc``, and the verification from #1830 now keeps such orphans from recurring. Thanks to @dcki for the report (#1828) **Build & CI/CD** diff --git a/src/autobahn/wamp/gen/wamp/proto/ChannelBinding.py b/src/autobahn/wamp/gen/wamp/proto/ChannelBinding.py deleted file mode 100644 index 69a2cb449..000000000 --- a/src/autobahn/wamp/gen/wamp/proto/ChannelBinding.py +++ /dev/null @@ -1,8 +0,0 @@ -# automatically generated by the FlatBuffers compiler, do not modify - -# namespace: proto - - -class ChannelBinding(object): - NONE = 0 - TLS_UNIQUE = 1 diff --git a/src/autobahn/wamp/gen/wamp/proto/Kdf.py b/src/autobahn/wamp/gen/wamp/proto/Kdf.py deleted file mode 100644 index 5f58d0cca..000000000 --- a/src/autobahn/wamp/gen/wamp/proto/Kdf.py +++ /dev/null @@ -1,9 +0,0 @@ -# automatically generated by the FlatBuffers compiler, do not modify - -# namespace: proto - - -class Kdf(object): - NONE = 0 - PBKDF2 = 1 - ARGON2 = 2