Skip to content

About

Error-state Kalman filter and calibration harness for GNSS-denied navigation. Shows when a filter is confidently wrong. Synthetic data only.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

666 Commits

Folders and files

Repository files navigation

contested-nav

Is the filter right about how wrong it is?

A navigation filter reports its position and also how sure it is. Here the filter says it is sure and is far off, so camera aiding is shipped switched off.

A small red circle, the position the filter claims, far from a blue dot, where the vehicle is, in one synthetic run

CI Security Docs License: MIT

In 60 seconds

Problem. A navigation filter has to be right about its own position and also right about how sure it is. A drone that has lost GNSS and is steered by a filter that is wrongly confident will trust a position it should not.

Decision. Should camera-based motion tracking be switched on when GNSS is denied? With the single-anchor model it does not reliably make the position error smaller, and the filter claims to know its position to within centimetres while the error is metres (the table below is one run with one start draw). It is confidently wrong, so it ships off by default (ADR-0003).

Evidence. In the sweeps the overconfidence held for every one of 10 seeds and every one of 40 outage runs (measured once on one machine, not checked by CI, at 946d661; see what the sweeps show). The tables on this page are generated by navkit run --markdown, and CI fails if a cell disagrees with the code. They are synthetic. The filter has also been run on recorded IMU data (EuRoC and TUM VI) with GNSS simulated from the ground truth, and on UrbanNav with the receiver's own fixes; see Status and limits. None of it is a field test.

Status. The bar for switching vision on was fixed before any candidate was tested, and no candidate met it. The question is closed as characterised, not solved (ADR-0036), and vision stays off. The candidates and their gates are in the roadmap and CONSTRAINTS.md.

Position error against the uncertainty the filter claims, through a GNSS outage, without and with vision

A 21-state error-state Kalman filter and an evaluation harness for GNSS-denied navigation. It measures not only how accurate the trajectory is, but whether the uncertainty the filter reports is honest. With GNSS denied and visual odometry on (the single-anchor model), the filter claims centimetres of uncertainty while its error is metres. Every ESKF case here starts at the truth plus an error drawn from the filter's own declared prior (ADR-0027), so the table below is one run with one start draw (seed 0). It is descriptive:

Configuration ATE RMSE, full run Claimed 1σ, full run Mean NEES, full run (expected 3) 2σ-per-axis coverage, full run (expected 99.3%) Mean NEES, outage window (expected 3) 2σ-per-axis coverage, outage window (expected 99.3%)
GNSS denied, vision off 6.530 m 0.567 m 3.2 100.0% 4.3 100.0%
GNSS denied, vision on 17.568 m 0.156 m 14018.6 19.2% 2783.3 5.0%

Calibration failed, and accuracy did not improve. The filter is confidently wrong, so visual fusion ships off by default. In this single run the aided case is also less accurate than the case without vision. Over 10 seeds, each with its own start draw, it was less accurate in 9 (measured once on one machine, not checked by CI, at 946d661), so whether vision helps accuracy depends on the start draw, and here it mostly does not. The overconfidence held in every run of every sweep (see what the sweeps show).

A stochastic clone of the previous pose (opt-in, vision_model: clone, ADR-0017) is the candidate fix. The pre-registered held-out gate (ADR-0028) was run once and failed, and the result was accepted rather than tuned away. A single run cannot establish calibration, so the table lists what was measured and its status. It is all synthetic, the default is unchanged, and visual fusion still ships off. B1 is closed as characterised, not solved (ADR-0036): work towards a default-on configuration has stopped, and the tools stay in the repository.

What the evidence shows

Configuration What was measured Status
Single anchor overconfident in every run of every sweep; does not recover from a start attitude error inside its own declared prior (the ensemble over seeds, what the sweeps show) fails; kept as the control
Stochastic clone, independent visual errors passes the S1 gate of ADR-0030 on the development seeds (25 to 49) (the candidate gate) not evaluated on held-out seeds
Clone with the measured-front-end noise preset conservative; failed the held-out gate of ADR-0028 on condition 2, on the fixture before ADR-0033 (the decision run) recorded, accepted as final (B1 in CONSTRAINTS.md)
Stochastic clone, visual errors correlated over time fails the S2 gate of ADR-0030 on the development seeds (the candidate gate) not solved; work stopped (ADR-0036)
Adaptive visual noise scale (ADR-0031, ADR-0032) failed on development seeds, on the fixture before ADR-0033 recorded
Gauss-Markov model of the correlated visual error (ADR-0034) restores consistency with the NEES conditions in scenario S2 when its correlation time matches the data, and is conservative when it does not (the tuning run and the robustness diagnostic); the usefulness conditions of the gate are not met in S2 even with a near-exact model (the diagnostic) failed on development seeds, recorded

Mostly synthetic evidence. The tables on this page come from a deterministic known-answer fixture: an analytic trajectory with generated sensor streams. The recorded-data pages use a real IMU with simulated GNSS (UrbanNav: the receiver's own fixes) and, in one study, real camera frames. No number here is a field measurement.

How it works

flowchart TB
    IMU["IMU, always on"] --> F
    GNSS["GNSS, can be denied"] --> FDIR["Fault detection: screens each measurement"]
    CAM["Camera motion, opt-in"] -.-> FDIR
    FDIR --> F["Navigation filter: position, and how sure it is"]
    F --> ACC["Accuracy: error against the truth"]
    F --> HON["Honesty: is the truth inside the claimed area?"]
    TRUTH["Known truth: synthetic runs, recorded datasets"] --> ACC
    TRUTH --> HON
    ACC --> GATE{"Gate B1"}
    HON --> GATE
    GATE -- "met" --> ON["Camera fusion on"]
    GATE -- "not met" --> OFF["Camera fusion off by default (ADR-0003)"]
Loading

Every run is scored twice: how far the estimate is from the truth, and whether the truth falls inside the area the filter claims. A feature that improves the first and breaks the second stays off. The project page has an animated figure of the same outage; the module-level design is in docs/architecture.md.

Quickstart

Python 3.11 to 3.13. From a fresh clone this took 18.7 s (clone, environment, install and a three-case run), measured once on one machine, not checked by CI:

git clone https://github.com/Telschow/contested-nav.git && cd contested-nav
python -m venv .venv && .venv/bin/pip install -e .
.venv/bin/navkit run --only gnss_only outage_control outage_visual --markdown

make repro regenerates every figure and table from scratch in about 5 minutes (321 s, measured once on one machine, not checked by CI). More in Getting started.

Results

Synthetic 30 s scenario, 5 Hz GNSS, 20 Hz vision, noisy IMU, seed 0, one start draw, so a single sample: each ESKF case starts at the truth plus an error drawn from its own declared prior, so this table is descriptive and not a typical case. ATE uses alignment=none: no fitted transform absorbs an offset, so the start error is part of what the table scores.

Scenario ATE RMSE (m) Claimed 1σ (m) Mean NEES (exp. 3) Coverage @ 2σ Verdict
GNSS only (control) 0.512 0.254 4.2 100.0% mixed: bulk overconfident, tail consistent
Dead reckoning (no aiding) 1.248 n/a n/a n/a no covariance reported
Anchor as measurement noise (defect) 11.497 0.100 13554.9 8.7% overconfident†
Vision only 26.727 0.158 32045.8 1.5% overconfident†
GNSS denied 5–20 s, vision off (control) 6.530 0.567 3.2 100.0% consistent
GNSS denied 5–20 s, vision on 17.568 0.156 14018.6 19.2% overconfident†
GNSS denied, 30% camera frames dropped 7.434 0.192 1621.3 22.2% overconfident†
GNSS denied 5–20 s, vision on, stochastic clone 0.730 0.375 2.2 100.0% consistent
GNSS denied, 30% frames dropped, re-referenced, single anchor 7.609 0.192 1698.4 22.7% overconfident†
GNSS denied, 30% frames dropped, re-referenced, stochastic clone 0.403 0.306 1.8 100.0% mixed: bulk underconfident, tail consistent
Vision only, stochastic clone 1.785 1.325 1.6 100.0% consistent†
Stochastic clone, 5% gross visual outliers 0.816 0.382 2.6 100.0% consistent
Stochastic clone, drifting translation scale 0.905 0.383 3.2 100.0% consistent
Stochastic clone, visual errors correlated over 2 s 3.061 0.395 30.7 34.1% overconfident†
Stochastic clone, correlated errors, visual noise assumed 4x 2.710 0.532 3.4 100.0% consistent

This table and the two-row table above it are generated (navkit run --markdown and python scripts/check_doc_tables.py --print-configuration-table), and CI fails if any cell disagrees with the benchmark. On this page, the documentation index, Getting started, the Results page and Status and limits, CI also fails on any other measured number in the text unless it is quoted through a metric marker that scripts/metrics.py checks. Timings and figures labelled as one-off measurements are not checked. Interpretation, the four ATE alignments and the figures are on the Results page. A dagger after a verdict means that the effective sample size behind it is truncated, so an upper bound (ADR-0029). consistent means that the run did not reject calibration: one run cannot establish it, and a single-run verdict is descriptive.

What the sweeps show

The seed, outage and scene sweeps below were re-run on the corrected translation noise (ADR-0033), with the code at 946d661. Their outputs are not committed, and a single run is descriptive until the ensemble statistics that will commit them are decided. navkit sweep seeds, navkit sweep outages and navkit sweep scenes regenerate them. A sentence below that was not re-run is labelled as produced under the earlier draw.

  • Not a bad seed. Over 10 seeds, each with its own start draw, outage_visual and vision_only were overconfident at every one (measured once on one machine, not checked by CI, at a992aa7).
  • Not a long-outage effect. Over 8 outage windows (start 5 s or 10 s, length 5 to 20 s, 5 seeds each) the visual case was overconfident in all 40 runs, including 5 s outages (measured once on one machine, not checked by CI, at 946d661).
  • Not one path. Over 8 synthetic scenes (trajectories), each of the five single-anchor rows read overconfident in every scene (measured once on one machine, not checked by CI, at 946d661).
  • No accuracy gain from the single anchor. outage_visual had a larger ATE than outage_control in 9 of 10 seeds, and in all 8 outage windows with the seeds averaged. vision_only was less accurate than dead reckoning in 10 of 10 seeds (measured once on one machine, not checked by CI, at 946d661).
  • The start attitude draw is what the anchor model fails on (MEASUREMENT). Over 10 seeds the median ATE of outage_visual and of vision_only is several times larger with the attitude drawn than without it, and close to the full-draw value with only the attitude drawn. The attitude prior is 2 degrees per axis, so the anchor model does not recover from a start attitude error inside its own declared prior. The clone's ATE and NEES do not follow the attitude draw. The table is generated from docs/data/start_draw_study.csv and shown on the Results page. Why the anchor model fails is a HYPOTHESIS and was not tested: gravity leakage from the start tilt error plus absolute information that the anchor model invents.
  • A control's verdict is one draw. Over 10 seeds the outage_control verdict was consistent in 5, mixed (bulk overconfident, tail consistent) in 2 and mixed (bulk underconfident, tail consistent) in 3. The mean NEES of outage_visual spanned about two orders of magnitude over the seeds and that of vision_only about one, and the verdict of both anchor rows did not change (measured once on one machine, not checked by CI, at 946d661).

Limits

  • Not field validated. The benchmark is a synthetic fixture. The recorded-data studies use real IMU and camera data with GNSS simulated from the ground truth, and the data is not vendored.
  • Visual fusion under GNSS denial is not trustworthy. The cause is structural (the visual anchor is a pose the filter itself produced) and the established remedies are not implemented here. With the start drawn from the declared prior, the single-anchor model also does not recover from a start attitude error inside that prior (see what the sweeps show). See Concepts.
  • FDIR detects implausible updates but mostly cannot say why. A signature for the white-noise surrogate of multipath exists as a function (ADR-0022). A slow-ramp position offset (spoof surrogate) is followed by the filter with an IMU and GNSS alone (ADR-0019), and a simulated visual source catches it only when it is fast and the visual errors are independent (ADR-0020). Produced under the earlier draw of the synthetic visual translation noise (ADR-0033); not regenerated.
  • Not flight-ready: no sensor driver, no live front end, no real-time loop.

All of it, with sources, is in Status and limits and Limitations.

How this was built

The code was written mostly with an AI coding assistant (Claude Code). The maintainer sets the scope and the order of work, and the project's rule is that the maintainer writes and accepts the decision in each ADR. The evidence to trust is the gates, not the authorship: finite-difference checks on the measurement Jacobians, property tests, a golden snapshot of the benchmark, and CI. Most changes land through a pull request that passes CI. That is the practice, not a rule the repository enforces: a few commits went to main directly, and the Sandcastle automation merges its own pull requests.

Scope and responsible use

This is a research and evaluation library built from public sources, synthetic data and two public recorded datasets that are fetched, never vendored. It contains no real sensor data or drivers, no radio-frequency model (a GNSS outage is a gap in a stream; spoofing tests inject offsets into the filter), and no guidance, targeting or platform integration code. It is not a navigation product and is not qualified for any platform. To the maintainer's knowledge nothing in it is classified or restricted, but that is not a legal determination: check your own obligations before using it in a regulated or operational setting. Details in Scope and responsible use; report a vulnerability through the security policy.

Documentation

The site is at telschow.github.io/contested-nav; every page is also a Markdown file under docs/.

If you want Read
To run it Getting started
The argument and the mechanism Concepts, calibration
What the filter assumes, every config key, the result format Model and interface contract
Evidence on recorded data EuRoC, TUM VI, UrbanNav (real GNSS outage), where the outage error comes from, a real front end
Spoofing, multipath and fault classes Honest returns, slow ramps, classification, multipath
How fast it is, measured Performance
Decisions of record Architecture decision records
What is planned ROADMAP, CHANGELOG
How to contribute CONTRIBUTING, code of conduct

Layout

src/navkit/   estimators (ESKF, dead reckoning), fdir, sensors, degrade, eval, geometry, io
configs/      benchmark scenarios (configs/benchmark.yaml)
scripts/      thin wrappers, coverage ratchet, doc-table and link checks, secret scan
benchmarks/   timing harness for docs/PERFORMANCE.md
docs/         site pages, ADRs, figures, defense assessment, product management
tests/        unit, property, golden-snapshot and documentation consistency tests

Acknowledgements

Recorded data and its citations:

  • EuRoC (page): Burri, M. et al. (2016). The EuRoC micro aerial vehicle datasets. The International Journal of Robotics Research, 35(10), 1157-1163. Rights statement: "In Copyright - Non-Commercial Use Permitted", set by the dataset's authors (ETH Zurich ASL). See the License section below for the committed result files.
  • TUM VI (page): Schubert, D. et al. (2018). The TUM VI benchmark for evaluating visual-inertial odometry. IROS 2018. Licence: CC BY 4.0, as the dataset page states; the TUM VI page gives the attribution.
  • UrbanNav (page): Hsu, L.-T. et al. (2023). UrbanNav: an open-sourced multisensory dataset for urban positioning. NAVIGATION. Licence: MIT, as stated by the authors.

The recorded GNSS and IMU data on the UrbanNav page is from the UrbanNav dataset (github.com/IPNL-POLYU/UrbanNavDataset). The authors ask for a citation of Hsu et al. (2023), NAVIGATION. UrbanNav: an open-sourced multisensory dataset for urban positioning. The authors state that the dataset files are MIT licensed. The data is fetched with navkit urbannav fetch and is not part of this repository. Only aggregate results are published here.

License

MIT. See LICENSE.

The MIT licence covers the code. Many files in docs/data/ contain measurements computed from the EuRoC recordings, for example euroc_*.csv, vo_*.csv, fault_classification*.csv, multipath_*.csv, separability.csv, slow_ramp.csv, vision_ramp.csv and timing.csv. They are results, not the dataset. The EuRoC rights statement is "In Copyright - Non-Commercial Use Permitted", so use those files for non-commercial purposes only, whatever the MIT licence says about the code. The terms are the dataset authors' to set. Read them on the dataset page before any other use. Some of those files also hold rows computed from TUM VI. Whether the per-run and per-frame rows may stay published is an open maintainer decision: ADR-0042, issue #278. No answer from the rights holder is recorded.

About

Error-state Kalman filter and calibration harness for GNSS-denied navigation. Shows when a filter is confidently wrong. Synthetic data only.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages