Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 20 additions & 8 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,14 +63,24 @@ Command-line tool for running INS simulations with GNSS degradation:

### 4. `analysis` (/analysis, Python)
Post-processing and experiment tooling, exposed as the `analyze` CLI:
- **preprocess.py**: rebuilds `data/input` from the Sensor Logger exports in `data/raw` --
- **Preprocessing lives in the dataset's own package, `memsnav`**
([jbrodovsky/mems-nav-dataset](https://github.com/jbrodovsky/mems-nav-dataset), a git
dependency of `analysis` pinned by tag in `analysis/pyproject.toml`). `memsnav preprocess`
rebuilds the **real arm**, `data/input_real`, from the Sensor Logger exports in `data/raw` --
resampling, splitting recordings at IMU dropouts, and downloading the `_gravity.nc` /
`_magnetic.nc` maps beside each trajectory. `just preprocess` runs it at **10 Hz**, which
gives 10 Hz inertial propagation against the ~1 Hz the GNSS was actually recorded at (the
GNSS columns stay NaN in nine rows out of ten; do not interpolate them up). `data/input` is
the one directory every consumer reads -- all 21 `conf/*.toml`, `geo-stats`, `postprocess`
and `geoperf-*` -- so change the rate, not the path. `just preprocess-1hz` is the 1 Hz
variant, which is the rate every result before the geophysical fixes used
`_magnetic.nc` maps beside each trajectory. `memsnav fetch` downloads the recordings
(`just fetch-data`) or the published processed data (`just fetch-processed`) from Zenodo.
Change preprocessing there, tag a release, and bump the pin here -- the data paper and the
Zenodo record describe what that package produces, so it is not edited from this repo
- **synthetic.py** (`analyze synthesize`): builds the **synthetic arm**, `data/input`, from the
real one by replacing only `grav_*`/`mag_*` with a modelled ADXL355/RM3100. Draws are keyed
by seed and file stem, so it is a pure function of `data/input_real`. `just preprocess` runs
both steps at **10 Hz**, which gives 10 Hz inertial propagation against the ~1 Hz the GNSS
was actually recorded at (the GNSS columns stay NaN in nine rows out of ten; do not
interpolate them up). `data/input` is the one directory every consumer reads -- all 21
`conf/*.toml`, `geo-stats`, `postprocess` and `geoperf-*` -- so change the rate, not the
path. `just preprocess-1hz` is the 1 Hz variant, which is the rate every result before the
geophysical fixes used
- **geostats.py**: characterises the geophysical measurements against those maps -- per-field
bias, measurement noise and signal-to-noise, plus the de-correlation length that sets
`geo_interval_s`. Its anomaly models mirror `core/src/earth.rs` and `geonav/src/lib.rs` term
Expand Down Expand Up @@ -152,7 +162,9 @@ cargo fmt-check
```

### Datasets
Sample datasets are no longer vendored in this repository. Test fixtures live in
Sample datasets are no longer vendored in this repository. The MEMS-Nav recordings are a
separate Zenodo record with its own repository and package (`memsnav`, see the `analysis`
section); `just fetch-data` downloads them into `data/raw`. Test fixtures live in
`core/tests/`; generate your own trajectories with the `syn` subcommand:

```bash
Expand Down
6 changes: 6 additions & 0 deletions analysis/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ dependencies = [
"cartopy>=0.25.0",
"haversine>=2.9.0",
"matplotlib>=3.10.8",
# The MEMS-Nav dataset's own package: `memsnav fetch` downloads the recordings and
# `memsnav preprocess` turns them into data/input_real. Pinned by tag in [tool.uv.sources].
"memsnav[maps]",
# NetCDF backend for xarray, so `geostats` can read the .nc maps `preprocess` writes.
# xarray arrives via pygmt but brings no engine of its own.
"netcdf4>=1.7.0",
Expand All @@ -25,6 +28,9 @@ dependencies = [
"tqdm>=4.67.1",
]

[tool.uv.sources]
memsnav = { git = "https://github.com/jbrodovsky/mems-nav-dataset", tag = "v0.1.0" }

[tool.pytest.ini_options]
testpaths = ["tests"]

Expand Down
44 changes: 11 additions & 33 deletions analysis/src/analysis/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,12 @@
save_detailed_results_to_csv,
)
from analysis.geostats import add_geostats_arguments, geostats_analysis
from analysis.preprocess import add_preprocess_arguments, preprocess_data
from analysis.synthetic import add_synthesize_arguments, synthesize_directory

# `analysis.plotting` is NOT imported here. It imports pygmt at module scope, and pygmt
# dlopens the GMT C library on import -- so an eager import made *every* subcommand, and
# importing the package at all, fail on a machine without GMT installed. Only `performance`
# and `geoperformance` actually draw maps; `preprocess` and `geostats` do not, and CI has no
# and `geoperformance` actually draw maps; `synthesize` and `geostats` do not, and CI has no
# GMT. The two call sites import it where they use it.

__version__ = "0.1.0"
Expand Down Expand Up @@ -96,7 +96,7 @@ def read_timeseries(path) -> DataFrame:

unsupported operand type(s) for -: 'str' and 'str'

Every timestamp this package reads is written by `analyze preprocess` or by
Every timestamp this package reads is written by `memsnav preprocess` or by
`strapdown-sim`, both of which emit ISO 8601, so the format is named rather than guessed.
The index is checked before the frame is handed back, because the whole point is that the
silent version of this failure costs an hour: `performance_analysis` caught it in a broad
Expand Down Expand Up @@ -166,35 +166,13 @@ def main() -> None:

command = parser.add_subparsers(title="command", dest="command")

preprocess = command.add_parser("preprocess", help="Preprocess raw trajectory data.")

preprocess.add_argument(
"-i",
"--input",
type=str,
default="data/raw",
help="Base directory for the sensor logger app data.",
)
preprocess.add_argument(
"-o",
"--output",
type=str,
default="data",
help="Output directory for the cleaned data.",
)
preprocess.add_argument(
"-b",
"--buffer",
type=float,
default=0.1,
help="Buffer amount to inflate the bounding box by (as a percentage). Default is 0.1 (10 percent).",
)
preprocess.add_argument(
"--getmaps",
action="store_true",
help="Download geophysical maps for each trajectory.",
# Raw -> processed lives in the dataset's own package, `memsnav` (`uv run memsnav
# preprocess`). What stays here is what mirrors this repository's measurement models.
synthesize = command.add_parser(
"synthesize",
help="Build the synthetic gravimeter/magnetometer arm from the real processed data.",
)
add_preprocess_arguments(preprocess)
add_synthesize_arguments(synthesize)

dataset_summary = command.add_parser(
"dataset-summary",
Expand Down Expand Up @@ -352,8 +330,8 @@ def main() -> None:

args = parser.parse_args()

if args.command == "preprocess":
preprocess_data(args)
if args.command == "synthesize":
synthesize_directory(args.input, args.output, seed=args.seed, prune=args.prune)
elif args.command == "dataset-summary":
dataset_summary_analysis(args)
elif args.command == "performance":
Expand Down
14 changes: 7 additions & 7 deletions analysis/src/analysis/geostats.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
analyze geostats -i data/input -o data/output/geostats

Each trajectory CSV is read with the sibling ``<stem>_gravity.nc`` and ``<stem>_magnetic.nc``
that ``analyze preprocess`` wrote beside it -- the same files, found the same way,
that ``memsnav preprocess`` wrote beside it -- the same files, found the same way,
``find_gravity_map`` and ``find_magnetic_map`` in ``sim/src/main.rs`` look for.

Why the formulas below are duplicated from Rust
Expand Down Expand Up @@ -253,7 +253,7 @@ def wmm_total_field_nt(latitude_deg, longitude_deg, altitude_m, years) -> np.nda

#: Effective resolution of the gravity grid over land, metres.
#:
#: ``analyze preprocess`` downloads ``earth_faa`` at ``01m`` -- a 1 arc-minute cell, about
#: ``memsnav preprocess`` downloads ``earth_faa`` at ``01m`` -- a 1 arc-minute cell, about
#: 1855 m of latitude. That is the *grid* spacing. ``earth_faa`` is Sandwell & Smith's
#: altimetry-derived marine anomaly, and **over land it is filled in from EGM2008**, whose
#: terrestrial input is a 5 arc-minute area-mean grid. For a land vehicle the 1 arc-minute
Expand All @@ -263,7 +263,7 @@ def wmm_total_field_nt(latitude_deg, longitude_deg, altitude_m, years) -> np.nda

#: Altitude the magnetic grid is referenced to, metres.
#:
#: ``analyze preprocess`` downloads ``earth_wdmam`` at ``03m``. WDMAM v2 is a 3 arc-minute
#: ``memsnav preprocess`` downloads ``earth_wdmam`` at ``03m``. WDMAM v2 is a 3 arc-minute
#: grid **at 5 km above sea level**. A phone at ground level is not measuring the same
#: quantity: upward continuation attenuates an anomaly of wavelength L by exp(-2*pi*h/L), so
#: at 5 km everything shorter than about 31 km is suppressed in the map but present in the
Expand Down Expand Up @@ -566,7 +566,7 @@ def analyse_trajectory(
raise ValueError(f"{csv_path.name}: missing column(s) {missing}")

# Parsed explicitly rather than through `read_csv(parse_dates=...)`. On a column of
# `2025-03-01 12:00:00+00:00` -- which is exactly what `analyze preprocess` writes --
# `2025-03-01 12:00:00+00:00` -- which is exactly what `memsnav preprocess` writes --
# pandas leaves `parse_dates` columns as **strings** when it cannot infer a format, with
# no error, so the failure surfaced a hundred lines later as
# `unsupported operand type(s) for -: 'str' and 'str'`.
Expand Down Expand Up @@ -1144,7 +1144,7 @@ def add_geostats_arguments(parser) -> None:
Register the geostats options on a parser.

Shared by this module's standalone entry point and the ``analyze geostats`` subcommand,
following `add_preprocess_arguments` so the two cannot drift apart.
following `add_synthesize_arguments` so the two cannot drift apart.
"""
parser.add_argument(
"-i",
Expand All @@ -1153,8 +1153,8 @@ def add_geostats_arguments(parser) -> None:
required=True,
help=(
"Directory of preprocessed trajectory CSVs, each with its `<stem>_gravity.nc` "
"and `<stem>_magnetic.nc` beside it -- that is, the output of `analyze "
"preprocess`, not data/raw."
"and `<stem>_magnetic.nc` beside it -- that is, `memsnav preprocess --getmaps` "
"or `analyze synthesize` output, not data/raw."
),
)
parser.add_argument(
Expand Down
Loading
Loading