Keep ML feature values paired with their nearest timestamps

docs/design/ML_BUNDLE_METADATA.md

@@ -72,3 +72,24 @@ truthful description of the supplied values.
 metadata to three actual small Parquet witnesses and the public analysis-job
 API. All three window files remain byte-identical to the original producer;
 only the metadata dictionary changes.
+
+
+## Feature rows and nearest lookup
+
+Retained feature tables can contain session timestamps in a different order
+from their stored rows. MCAP message log order, for example, does not establish
+the order of embedded session timestamps. Before nearest-feature lookup, the
+loader checks the integer timestamps and stably orders timestamp/value pairs
+in memory when it finds an inversion. The original Parquet file and the
+feature writer's row order remain intact.
+
+Already ordered rows follow the same lookup path. Equal-distance choices
+continue to select the later timestamp. Equal timestamps retain their original
+relative row order: an exact query selects the first equal row, while a query
+just after them compares the last preceding equal row. No interpolation,
+aggregation, timestamp conversion, validity or target-column rule changes.
+
+[Feature-order receiving](../evidence/ml-bundle-feature-order-713adaab/README.md)
+includes actual Parquet inputs and an MCAP-to-feature-table-to-public-job
+witness. This qualification concerns deterministic data alignment; it does not
+establish physical clock synchronization or hardware accuracy.

docs/design/research/PROGRESS.md

@@ -1,3 +1,29 @@
+## 2026-10-08 — Preserve nearest feature pairs across retained row orders
+
+Contributor `chatgpt:/root/production_execution`, issue #86, repaired the
+ML-bundle loader's assumption that retained feature timestamps were ordered.
+A real private MCAP fixture with different log and embedded timestamp orders
+passes through the radar extractor, native feature writer and public Job API;
+the original bundle selects energy 4 where the nearest sample has energy 9.
+
+Only feature ingestion adds an inversion check and stable in-memory ordering
+of the timestamp and value arrays together. Ordered inputs, duplicate relative
+order and the existing equal-distance choice retain their behavior. Stored
+inputs, feature writers, kinematics/evaluation, validity, metadata, schema and
+the unrelated internal self-digest convention remain unchanged.
+
+The same 17 final native Python 3.12.10 receivers record 6 original passes and
+11 failures, then 17 candidate passes without errors or skips. A separate 14
+inherited metadata, pose, kinematics, bundle, CLI and actual evaluator cases
+pass. Paired file readback verifies all 43 original inputs, six byte-identical
+ordered output controls, eleven corrected unordered outputs and identical
+non-energy columns. The initial receiver's four schema-invalid 2 ns windows
+and their correction remain distinct from the final paired qualification.
+[Source and receiving custody](../../evidence/ml-bundle-feature-order-713adaab/README.md)
+retains exact files and limitations. Independent review and supported Windows
+CI remain separate integration gates; no installed or hardware result is
+claimed here.
+
 ## 2026-10-08 — Inspect and compare retained analysis parameters
 
 Contributor `chatgpt-ac386303dce2/product_execution`, issue #70, implements

docs/evidence/ml-bundle-feature-order-713adaab/README.md

@@ -0,0 +1,116 @@
+# ML feature ordering: source and native receiving
+
+Issue [#86](https://github.com/Jacob-Met/CaptureSuite/issues/86) addresses a
+data-pairing defect in the ML-bundle producer on current main
+`11cc78297d8d214407aea693548af4de855d61da`.
+
+The loader previously handed stored row order to `numpy.searchsorted`.
+Actual radar features can retain an embedded-session timestamp order different
+from MCAP log order. A private MCAP fixture passes through the real radar
+extractor, native feature-table writer and public analysis Job API: the final
+window originally selected energy 4 instead of the nearest sample's energy 9.
+
+Only four lines are added to `_load_feature_series`: detect an inversion,
+then apply the same stable integer timestamp ordering to both arrays. Ordered
+inputs avoid sorting. Duplicate relative order and the existing later-time
+equal-distance choice remain unchanged. Feature writers, stored source files,
+kinematics, target medians, validity, centers, public arguments, schemas and
+the unrelated internal manifest self-digest are unchanged.
+
+## Exact source
+
+| Input | SHA-256 |
+|---|---|
+| Original producer | `38b7161f081ef038e6ce464dd3b82ca2b49f775ad8988010243743274d6fc5e6` |
+| Candidate producer | `ed67d94ded834d63599797bbd8efb7b26f125bd216f52bcb7f01febb928922ce` |
+| Final paired test | `7a5e226afb9c0f49eadf6d4c25b8bc22575f1fdd04afdac854cf8aa4adee58be` |
+| Runtime patch | `3232f15dc5b9c443b04d63aa0f23503b9fc19926cda0f83882a4708fa7b5c35e` |
+| Paired artifact readback | `e9f1b94c200261104ce9504b8f568a6738b957b26ee8506e8d317a8f6cb8b355` |
+
+## Actual native results
+
+All runs use an isolated Linux Python 3.12.10 environment. The exact standalone
+runtime archive identity, declared package pins, installed versions, successful
+dependency consistency check and complete installation log are retained.
+This environment does not alter an installed application or shared runtime.
+
+| Run | Passed | Failed | Errors / skips |
+|---|---:|---:|---:|
+| Initial original-source receiver | 4 | 11 | 0 / 0 |
+| Corrected final original-source receiver | 6 | 11 | 0 / 0 |
+| Same final receiver on candidate | 17 | 0 | 0 / 0 |
+| Separate inherited consumers on candidate | 14 | 0 | 0 / 0 |
+
+The initial receiver included four 2 ns windows below the existing schema
+minimum of 0.1 s. Those four cases stopped at an unexpected schema warning
+before their intended alignment assertion. Its original test, source, log,
+JUnit, artifacts and correction note remain in `evidence/baseline/` and
+`evidence/initial-receiver-correction.json`. No schema or product change was
+made to accommodate that receiver error.
+
+The final tests use valid 0.2 s windows. They cover shuffled and reversed rows,
+ordered controls, exact high integer timestamps, adjacent nanoseconds above
+2^53, stable duplicates, equal-distance choices, numeric-column mean fallback,
+a singleton, and the actual MCAP-to-Job API path. The final same-test baseline
+and candidate receipts retain process exits, full source maps and unchanged
+source checks.
+
+The separate inherited run selects the complete metadata and
+pose/kinematics/ML test modules plus
+`test_real_pose_kinematics_bundle_to_external_eval`. It does not repeat the
+17 focused receivers.
+
+Readback of all 17 retained output Parquets found six ordered controls
+byte-identical and eleven unordered outputs corrected. Every non-energy
+column remains exact. All 43 original input files in the paired runs match
+byte-for-byte. Input and final output inventory assertions also execute inside
+the tests.
+
+## Capsule
+
+[receiving.tar.gz](receiving.tar.gz), SHA-256
+`d80a4d2b79d4491b66ea8227ff92471ebdbb6edd8fe2fce75327bbe755485097`,
+contains 296 regular files, 1,502,844 expanded bytes and 265,323 compressed
+bytes. [manifest.json](manifest.json) inventories every member's exact bytes
+and SHA-256. Every member was reopened and compared after archive creation.
+
+The capsule includes original/candidate producer snapshots, identical final
+test bytes, the original rejected receiver, exact runtime/documentation
+patches, receiving/readback drivers, all three initial/final paired fixture
+and output sets, process/log/JUnit/source receipts, and runtime/dependency
+provenance. The inherited run's complete logs, process, JUnit, source maps and
+artifact-inventory receipt are included; its 309 runtime outputs remain in
+native custody at the recorded private path.
+
+Independent review and the existing full supported Windows workflow are
+separate integration gates. These native results establish deterministic
+alignment on the declared fixture inputs; they do not establish hardware
+synchronization, clinical validity or installed-desktop adoption.
+
+
+## Independent review
+
+[Peer review](peer-source-review.json), SHA-256
+`3c39e1f3ba00614ed67f68003e44bc184c072380afa965088401f59ace59f60a`,
+is copied unchanged from `chatgpt:/root/mac_execution` custody. The reviewer
+read the full producer, patch, all 17 receivers and documentation, independently
+restored the original producer by removing exactly the four added lines, and
+read back all 43 original input pairs and all 17 output identities. It found no
+source blocker. Numeric Parquet decoding and runtime receiving remain the
+author's separate qualification; the reviewer did not rerun tests or builds.
+
+
+## Current-main composition
+
+Before publication, current main advanced to
+`a2c89957c2cfe6fd5347d4ac317982449d3baa3a` with the source picker and saved-job
+parameter comparison. [The composition receipt](current-main-composition.json)
+binds the genuine three-way merge to those owner changes. All 1,157 unrelated
+main leaves remain exact; the complete current progress document is preserved
+after the exact owned addition. All 96 Python library leaves are unchanged
+between the originally received parent and this current main.
+
+The receipt identifies the underlying merged tree before adding this receipt
+and paragraph. Runtime and test bytes retain their reviewed identities.
+The earlier native results remain bound to their original source maps;
+the complete published union receives fresh supported Windows CI.

docs/evidence/ml-bundle-feature-order-713adaab/current-main-composition.json

@@ -0,0 +1,30 @@
+{
+  "schema": "capturesuite.nearest_feature_current_main_composition/1",
+  "base": "11cc78297d8d214407aea693548af4de855d61da",
+  "qualified_native": "e1128160ae9ddc767d3235867bcd1647ddf047b0",
+  "current_main": "a2c89957c2cfe6fd5347d4ac317982449d3baa3a",
+  "tree": "a56d16deacabfdc2d062e76fd49b31c972b6ebb2",
+  "resolved_path": "docs/design/research/PROGRESS.md",
+  "resolution": "Prepend the exact owned entry to current main's complete progress bytes.",
+  "current_main_leaves": 1160,
+  "preserved_unrelated_main_leaves": 1157,
+  "owned_paths": [
+    "docs/design/ML_BUNDLE_METADATA.md",
+    "docs/design/research/PROGRESS.md",
+    "docs/evidence/ml-bundle-feature-order-713adaab/README.md",
+    "docs/evidence/ml-bundle-feature-order-713adaab/freeze_capsule.py",
+    "docs/evidence/ml-bundle-feature-order-713adaab/manifest.json",
+    "docs/evidence/ml-bundle-feature-order-713adaab/peer-source-review.json",
+    "docs/evidence/ml-bundle-feature-order-713adaab/prepublication-static.json",
+    "docs/evidence/ml-bundle-feature-order-713adaab/receiving.tar.gz",
+    "docs/evidence/ml-bundle-feature-order-713adaab/source-verification.json",
+    "docs/evidence/ml-bundle-feature-order-713adaab/verify_source.py",
+    "libs/python/capture_analysis/capture_analysis/ml_bundle/job.py",
+    "tests/analysis/test_ml_bundle_feature_order.py"
+  ],
+  "all_other_main_leaves_exact": true,
+  "python_library_paths_unchanged_between_received_base_and_main": 96,
+  "producer_sha256": "ed67d94ded834d63599797bbd8efb7b26f125bd216f52bcb7f01febb928922ce",
+  "test_sha256": "7a5e226afb9c0f49eadf6d4c25b8bc22575f1fdd04afdac854cf8aa4adee58be",
+  "limits": "Source composition and dependency-byte preservation, not a new runtime execution. Existing native17+14 results retain their original source maps; the complete published composition requires fresh supported Windows CI."
+}

docs/evidence/ml-bundle-feature-order-713adaab/freeze_capsule.py

@@ -0,0 +1,66 @@
+# SPDX-License-Identifier: GPL-3.0-only
+"""Freeze exact native receiving files into a small verified archive."""
+from pathlib import Path
+import gzip
+import hashlib
+import io
+import json
+import tarfile
+
+root = Path(__file__).resolve().parent.parent
+evidence = root / "evidence"
+destination = root / "source/docs/evidence/ml-bundle-feature-order-713adaab"
+destination.mkdir(parents=True, exist_ok=True)
+def sha(data):
+    return hashlib.sha256(data).hexdigest()
+members = {}
+for name in [
+    "receive_feature_order.py", "receive_related.py", "read_paired_artifacts.py",
+    "baseline-producer.py", "candidate-producer.py", "paired-test.py",
+    "test_feature_order_initial.py", "initial-receiver-correction.json",
+    "runtime.patch", "documentation.patch", "paired-artifacts.json",
+]:
+    members["evidence/" + name] = (evidence / name).read_bytes()
+for stage in ["baseline", "baseline-final", "candidate", "related"]:
+    for name in ["receipt.json","process.json","pytest.xml","pytest.log",
+                 "source-before.json","source-after.json"]:
+        members[f"evidence/{stage}/{name}"] = (evidence/stage/name).read_bytes()
+    if stage != "related":
+        for path in sorted((evidence/stage/"pytest-temp").rglob("*")):
+            if path.is_file() and not path.is_symlink():
+                relative = path.relative_to(evidence).as_posix()
+                members["evidence/" + relative] = path.read_bytes()
+for name in ["runtime-receipt.json","dependency-receipt.json",
+             "requirements-receiver.txt","runtime.sha256","dependencies.log"]:
+    members["environment/"+name]=(root/"environment"/name).read_bytes()
+buffer=io.BytesIO()
+with tarfile.open(fileobj=buffer,mode="w") as archive:
+    for name,data in sorted(members.items()):
+        info=tarfile.TarInfo(name);info.size=len(data);info.mode=0o644
+        archive.addfile(info,io.BytesIO(data))
+packed=gzip.compress(buffer.getvalue(),mtime=0)
+target=destination/"receiving.tar.gz"
+target.write_bytes(packed)
+with tarfile.open(target,"r:gz") as archive:
+    restored={row.name:archive.extractfile(row).read() for row in archive.getmembers()}
+assert restored==members
+manifest={
+    "schema":"capturesuite.ml_feature_order_capsule/1","issue":86,
+    "base_commit":"11cc78297d8d214407aea693548af4de855d61da",
+    "original_producer_sha256":"38b7161f081ef038e6ce464dd3b82ca2b49f775ad8988010243743274d6fc5e6",
+    "candidate_producer_sha256":"ed67d94ded834d63599797bbd8efb7b26f125bd216f52bcb7f01febb928922ce",
+    "final_test_sha256":"7a5e226afb9c0f49eadf6d4c25b8bc22575f1fdd04afdac854cf8aa4adee58be",
+    "archive":{"path":target.name,"bytes":len(packed),"sha256":sha(packed),
+               "members":len(members),"expanded_file_bytes":sum(map(len,members.values())),
+               "all_members_reopened_and_verified":True},
+    "files":[{"path":name,"bytes":len(data),"sha256":sha(data)}
+             for name,data in sorted(members.items())],
+    "limits":["Archive contains the complete initial/final paired test artifacts. Related inherited test outputs remain in native custody; their complete process/log/JUnit/source maps and artifact-inventory receipt are included.",
+              "Private Linux Python3.12.10 runtime only; no Windows, physical sensor, installed application or scientific-validity qualification.",
+              "Original mixed negative with four schema-invalid 2ns windows is retained separately from corrected final17 test receiving."],
+}
+output=destination/"manifest.json"
+output.write_text(json.dumps(manifest,indent=2)+"\n")
+print(json.dumps({"archive":str(target),"bytes":len(packed),"sha256":sha(packed),
+"members":len(members),"expanded_bytes":manifest["archive"]["expanded_file_bytes"],
+"manifest_sha256":sha(output.read_bytes())}))