Skip to content

Move MEMS-Nav preprocessing and data paper to jbrodovsky/mems-nav-dataset - #421

Open
jbrodovsky wants to merge 1 commit into
mainfrom
claude/mems-dataset-repo-integration-cd01ef
Open

jbrodovsky wants to merge 1 commit into
mainfrom
claude/mems-dataset-repo-integration-cd01ef

Conversation

@jbrodovsky

Copy link
Copy Markdown
Owner

Summary

The MEMS-Nav dataset now lives in its own repository,
jbrodovsky/mems-nav-dataset. The data itself goes on Zenodo (record not yet created).
That repository's memsnav package now owns fetching, raw → processed, and release packaging. analysis depends on it, pinned to the git tag v0.1.0.

  • analyze preprocess is removed.
    • memsnav preprocess builds the real arm, data/input_real.
    • The new analyze synthesize builds the synthetic data/input from the real arm. It replaces only grav_*/mag_*, using the same synthesize_geophysical.
  • just preprocess runs both steps. data/input_real is now rebuilt from data/raw instead of being a frozen copy; clean still leaves it alone.
  • New just recipes:
    • fetch-data and fetch-processed download from Zenodo through memsnav fetch. They work once the record exists.
    • dataset-benchmark stages the release's benchmark/ directory from data/output_real2.
  • papers/mems_nav_data moved to the dataset repo's paper/. There it builds from the release layout instead of importing anom_combined's scripts.
  • Docs follow the move: CLAUDE.md, the book's data-format page, config comments, and the dataset link in integration_tests.rs.

The old dataset repository has been renamed to mems-nav-dataset-legacy and archived.

Verification

  • Real arm: memsnav preprocess on data/raw reproduces all 27 frozen data/input_real CSVs byte for byte. segments.json matches as a set; it is now sorted by recording.
  • Synthetic arm: analyze synthesize over data/input_real reproduces all 27 data/input CSVs byte for byte. synthetic.json is identical except for its generator string and trajectory order.
  • Paper: it builds from the release layout and regenerates every table, figure and number identically.
  • Release round trip: memsnav package builds 32 archives (8.4 GB). Fetching processed and maps back out of them gives 109 files identical to data/input_real.
  • Checks: uv run ruff check, ruff format --check and pytest pass (69). cargo clippy -p strapdown-core --tests -D warnings is clean.

Follow-ups (after the Zenodo record is published)

  • Bump the memsnav pin to v1.0.0, whose manifest carries the record.
  • Put the new DOI in papers/anom_combined/references.bib (mems-nav-dataset) and in sections/09-availability.tex.

🤖 Generated with Claude Code

The MEMS-Nav recordings get their own Zenodo record and their own repository,
jbrodovsky/mems-nav-dataset, whose `memsnav` package now owns raw -> processed.
analysis depends on it by git tag.

- `analyze preprocess` is gone; `memsnav preprocess` builds the real arm in
  data/input_real (byte-identical to the frozen copy on all 27 trajectories),
  and the new `analyze synthesize` builds the synthetic data/input from it
  (byte-identical to the one-step `--synthetic` on all 27; synthetic.json
  differs only in its generator name and trajectory order)
- `just preprocess` runs both steps, so data/input_real is now reproducible
  instead of frozen; `just fetch-data` / `fetch-processed` download from Zenodo
  and `just dataset-benchmark` stages the release's benchmark/
- papers/mems_nav_data moved to that repository's paper/
- docs and config comments follow; integration_tests.rs's dataset link fixed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant