M7 follow-up: Port ArrayShuffle for BHC C++ parity (#116)

[kalwalt]

Summary

Closes the BHC RNG divergence noted in PR #114 / PR #115 by porting vision::FastRandom and vision::ArrayShuffle from math/rand.h to Rust with byte-identical semantics. With this change:

• The Rust BHC index produces a tree topology byte-identical to the C++ baseline given the same seed.
• The match_indexed dual-mode test now asserts sorted pair equality with C++ (matching the brute-force and guided variants).
• A new CI step runs dual-mode tests to enforce the parity guarantee on every push.

Changes

Algorithm port (commit 6146ab8)

clustering.rs:

• fast_random(seed: &mut i32) -> i32 — exact port of vision::FastRandom. Uses i32 + wrapping arithmetic so the bit pattern matches C++ int.
• array_shuffle<T>(v, sample_size, seed) — exact port of vision::ArrayShuffle. Note: NOT Fisher–Yates; k is drawn from [0, pop_size) per C++.
• KMedoids::assign() reworked to:
  • Initialize rand_indices once before the hypothesis loop (matches C++ kmedoids.h:163)
  • Use array_shuffle with persistent rand_seed (mutated across calls and recursive BHC builds)
  • rand_seed type changed from u64 to i32 to match C++ int
• BHC query_recursive now visits all children with the minimum Hamming distance, matching C++ Node::nearest (""Any nodes that are the SAME distance as the minimum node are added"").

kpm_c_api.cpp/.h: Add webarkit_cpp_fast_random and webarkit_cpp_array_shuffle FFI shims.

matcher.rs: Strengthen dual_mode_match_indexed_within_tolerance from count-only to count + sorted pair equality.

CI integration (commit f15f30a)

.github/workflows/ci.yml: Add cargo test -p webarkitlib-rs --features dual-mode step to the existing kpm-build job (runs on ubuntu, macOS, windows).

Tests

New unit tests (always run in CI)

• test_fast_random_first_call — frozen expected values (seed 1234 → 4068)
• test_fast_random_deterministic
• test_fast_random_range
• test_fast_random_negative_seed
• test_array_shuffle_preserves_elements
• test_array_shuffle_deterministic
• test_array_shuffle_empty_pop
• test_array_shuffle_zero_sample
• test_array_shuffle_seed_evolves

New dual-mode tests (now run in CI via the new job)

• dual_mode_fast_random_byte_identical — 1000 calls × 9 starting seeds, value AND seed state match C++ at every step
• dual_mode_array_shuffle_byte_identical — 8 (pop, sample, seed) cases, permutation AND seed match C++
• dual_mode_array_shuffle_persistent_seed — 10 sequential shuffles, state evolution matches C++ at every round

Strengthened existing test

• dual_mode_match_indexed_within_tolerance — was count-only; now asserts sorted pair equality with C++

Verification

• ✅ cargo test -p webarkitlib-rs --lib → 274 passed
• ✅ cargo test --features dual-mode -p webarkitlib-rs --lib → 292 passed
• ✅ cargo clippy --workspace -- -D warnings → clean
• ✅ cargo fmt --all -- --check → clean

Why this matters

Before this PR: match_indexed dual-mode could only assert non-negative match counts because BHC RNG diverged. A bug in the indexed matcher would have been invisible to the dual-mode validation gate.

After this PR: the indexed matcher is held to the same correctness standard as brute-force and guided — sorted pair equality with the C++ baseline. The CI dual-mode job ensures regressions are caught automatically, not just when someone happens to run --features dual-mode locally.

Related

• Fixes feat(kpm): port ArrayShuffle for BHC parity in dual-mode tests #116
• Closes the divergence noted in PR M7-2: Implement K-Medoids clustering and Binary Hierarchical Clustering for FREAK descriptors #114 (M7-2) and PR M7-3: Port FeatureStore and FeatureMatcher with three match variants #115 (M7-3)
• Parent feat(kpm): M7 — Hough voting & feature matching in pure Rust #108 (M7 milestone)

Base Branch

Target: feat/milestone7-hough-voting-feature-match (M7 milestone branch)

[kalwalt]

Final state — all 16 CI checks green ✅

Beyond the planned ArrayShuffle port, this PR's new dual-mode CI gate exposed three dormant pre-existing bugs that were lying in wait. Documenting them here for the record:

1. <limits> include order in kpm_c_api.cpp

Symptom: error: 'numeric_limits' is not a member of 'std' on Linux GCC during build.

Root cause: The matcher headers I added (feature_matcher-inline.h etc.) transitively pull in hamming.h, which uses std::numeric_limits without including <limits> itself. MSVC pulled it in transitively; GCC did not. My initial fix added #include <limits> after the matcher headers — too late, because include guards meant hamming.h had already been parsed and failed.

Fix: Moved all standard library headers to the top of kpm_c_api.cpp, before project headers.

2. Wrong C++ stdlib on macOS in build.rs

Symptom: ld: library 'stdc++' not found on macOS when linking test binaries.

Root cause: build.rs unconditionally emitted cargo:rustc-link-lib=stdc++ for all non-MSVC targets. macOS uses LLVM's libc++, not GNU's libstdc++. The pre-existing kpm-build job didn't catch this because cargo build for a library produces an .rlib (no linking step). The new cargo test --features dual-mode --lib step links a test binary and exposed the long-standing bug.

Fix: Branch on target triple — *-apple-* links c++, others link stdc++.

3. M6-2 dual-mode test tolerance too tight for cross-platform FMA

Symptom: solve_linear_system_2x2_matches_cpp failed on Apple Silicon ARM64:

rust=-358.62683, cpp=-358.62802, diff=1.19e-3

Root cause: The fixed 1e-5 absolute tolerance was sub-f32-precision for results with magnitude > ~10. ARM64 (Apple Silicon) uses FMA differently from x86_64 (Linux runners), producing rounding differences that are real (~3.3e-6 relative) but exceed an absolute 1e-5 for larger values. Random near-singular 2×2 systems exacerbate this — small determinants produce large solutions that are very sensitive to FMA order.

Fix: Combined absolute + relative tolerance:

let tol = 1e-5_f32.max(x_rust[i].abs() * 1e-5);

Applied to both solve_linear_system_2x2_matches_cpp and solve_symmetric_linear_system_3x3_matches_cpp.

---

Why this matters for the project going forward

All three issues were dormant — they would have eventually bitten somebody trying to run integration tests with FFI on macOS, or building dual-mode validation on a fresh GCC. The new dual-mode CI gate (commit f15f30a) now exercises the full FFI test path on ubuntu/macOS/windows on every push, so:

• Future regressions in cross-platform C++ integration → caught in CI, not after a release
• Future ports that affect FMA-sensitive math → tolerance failures surface immediately, not after someone happens to run dual-mode locally
• The match_indexed parity guarantee is now enforced, not just observable

This is exactly the kind of payoff a strong validation gate provides — you find latent issues at integration time, not at use time.

Refs #116