Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CONSTRAINTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Invariants this project holds itself to. Every entry is either currently true
or is a stated blocker; nothing here is aspirational. Ratchets are measured,
not estimated, and the command to re-measure each one is given.

Last measured: more than <!-- metric:tests_collected_floor -->2700<!-- /metric --> tests collected, all of which pass except the deliberate
Last measured: more than <!-- metric:tests_collected_floor -->2800<!-- /metric --> tests collected, all of which pass except the deliberate
xfails and the skips that `pytest -rs` lists, and <!-- metric:coverage_percent -->94.9<!-- /metric -->%
line coverage (<!-- metric:coverage_lines_hit -->15550<!-- /metric -->/<!-- metric:coverage_lines_total -->16379<!-- /metric --> executable lines) on **CPython <!-- metric:coverage_python -->3.12<!-- /metric -->**, measured
with `python scripts/coverage_report.py`. The figures come from `docs/data/metrics.json`, which `scripts/metrics.py` checks in CI. Coverage is quoted from that script alone:
Expand Down Expand Up @@ -92,7 +92,7 @@ the floors themselves are enforced by `scripts/coverage_report.py --ratchet`.

| Ratchet | Floor | Current (CPython <!-- metric:coverage_python -->3.12<!-- /metric -->) | Re-measure with |
|---|---:|---:|---|
| Tests collected | 405 | more than <!-- metric:tests_collected_floor -->2700<!-- /metric --> | `pytest`, `scripts/metrics.py --check` |
| Tests collected | 405 | more than <!-- metric:tests_collected_floor -->2800<!-- /metric --> | `pytest`, `scripts/metrics.py --check` |
| Tests passing | 300 | all collected, except the deliberate xfails and the listed skips | `pytest -rs` |
| Line coverage | 75% | <!-- metric:coverage_percent -->94.9<!-- /metric -->% | `scripts/coverage_report.py`, `scripts/metrics.py --check-coverage` |
| `io/trajectory.py` coverage | 85% | 95.0% | as above |
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ mypy src --ignore-missing-imports
`ruff format src tests scripts` applies formatting; the `--check` form is the
gate.

The suite collects more than <!-- metric:tests_collected_floor -->2700<!-- /metric --> tests and all of them pass on a fresh clone, except the deliberate
The suite collects more than <!-- metric:tests_collected_floor -->2800<!-- /metric --> tests and all of them pass on a fresh clone, except the deliberate
xfails and the skips. `pytest -rs` lists the skips: two need TUM VI reference data that is not
vendored (see below), and the doc-table check runs once `python scripts/run_benchmark.py` has
written `results/benchmark.json`. The count and the other numbers the docs quote are in
Expand Down
2 changes: 1 addition & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Status markers: `[x]` done, `[~]` in progress, `[ ]` not started, `[!]`
blocked. Checked items record what was measured when the item closed. They are not kept current: the
current values are in the README tables, `docs/results.md` and `CONSTRAINTS.md` (B1).

Measured baseline: more than <!-- metric:tests_collected_floor -->2700<!-- /metric --> tests collected and <!-- metric:coverage_percent -->94.9<!-- /metric -->% line coverage on CPython
Measured baseline: more than <!-- metric:tests_collected_floor -->2800<!-- /metric --> tests collected and <!-- metric:coverage_percent -->94.9<!-- /metric -->% line coverage on CPython
<!-- metric:coverage_python -->3.12<!-- /metric --> (`docs/data/metrics.json`, from `scripts/metrics.py` and `scripts/coverage_report.py`). Every
number below re-measures against that, not against a remembered value.

Expand Down
2 changes: 2 additions & 0 deletions docs/adr/0010-determinism-and-seeding.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ writes it down so the next stochastic stream follows it too.

**What was measured.** The development seeds, 10 of them, were run twice: as they are, and with the vision seed derived from the case seed and a stream id ([issue #259](https://github.com/Telschow/contested-nav/issues/259), `docs/data/shared_stream_effect.csv`, `docs/calibration.md`). The result was within seed noise for the aggregate. The sign of the per-seed change was mixed, and the mean paired difference was within about one standard error of zero for ATE RMSE and mean NEES in all four cases. The single-anchor control stayed overconfident and the clone stayed calibrated. Single seeds moved far (seed 0 of `outage_visual`, 17.6 m to 7.9 m ATE). The comparison also changes the IMU noise draw, which is hashed from both seeds, so it measures both changes together. Ten seeds exclude a large systematic effect, not a small one.

**Follow-up: the two changes apart.** The study now has a third seeding, `streams_only`: the independent vision seed with the IMU noise pinned to its shared draw, for the duration of one run and with nothing in `src/navkit/degrade` changed. It gives the stream relationship alone, with the IMU fixed. The medians under it are generated on `docs/calibration.md` from `docs/data/shared_stream_effect.csv`. For every case the streams-only change is of the same size and sign as the combined change, so redrawing the IMU explains little of it. `outage_control`, which has no vision, comes out identical to the shared run, as it must. The sample is the same ten development seeds, so the earlier limit stands: a large systematic effect is excluded, a small one is not.

**Decision.** The streams are not changed. No committed number, table or golden is regenerated, and the B1 result in [ADR-0028](0028-the-b1-gate.md) stands as it was produced. The shared stream is documented instead, in `docs/MODEL.md` and in the calibration page, where the measurement is generated from its CSV.

- Decision 2 is read as: each stream has its own seed field. It does not say the generators differ when the values are equal.
Expand Down
45 changes: 23 additions & 22 deletions docs/calibration.md
Original file line number Diff line number Diff line change
Expand Up @@ -646,28 +646,29 @@ In scenario S2, for the plain clone (no model): the NEES conditions do not all h

<!-- shared-stream:start -->

**What the shared noise stream changes (development seeds, synthetic; MEASUREMENT).** At equal seeds the GNSS noise and the visual rotation noise are one normal sequence. Each case below was run over 10 seeds twice: with that shared stream, and with the vision seed derived from the case seed and a stream id. The table gives the median over the seeds under both seedings. This measures the difference and decides nothing.

| Case | Metric | Shared | Independent |
|---|---|---:|---:|
| `outage_visual` | ATE RMSE | 25.44 m | 25.53 m |
| `outage_visual` | mean NEES | 29904.1 | 30347.4 |
| `outage_visual` | 2-sigma coverage | 0.071 | 0.076 |
| `outage_visual` | FDIR events | 49.0 | 49.0 |
| `vision_only` | ATE RMSE | 54.23 m | 52.38 m |
| `vision_only` | mean NEES | 133504.2 | 124823.9 |
| `vision_only` | 2-sigma coverage | 0.012 | 0.012 |
| `vision_only` | FDIR events | 0.0 | 0.0 |
| `outage_visual_clone` | ATE RMSE | 0.86 m | 0.80 m |
| `outage_visual_clone` | mean NEES | 3.3 | 2.5 |
| `outage_visual_clone` | 2-sigma coverage | 1.000 | 0.999 |
| `outage_visual_clone` | FDIR events | 0.0 | 0.0 |
| `outage_control` | ATE RMSE | 9.74 m | 9.83 m |
| `outage_control` | mean NEES | 2.9 | 3.0 |
| `outage_control` | 2-sigma coverage | 1.000 | 1.000 |
| `outage_control` | FDIR events | 0.0 | 0.0 |

The largest change of a median is in `outage_visual_clone`, mean NEES: 21.8 percent lower with independent streams. The single-anchor control `outage_visual` is in the table. Changing the vision seed also changes the IMU noise draw of every case, because the IMU seed is hashed from both seeds, so `outage_control`, which has no vision, moves too. The difference is the effect of both changes together.
**What the shared noise stream changes (development seeds, synthetic; MEASUREMENT).** At equal seeds the GNSS noise and the visual rotation noise are one normal sequence. Each case below was run over 10 seeds three times: with that shared stream; with the vision seed derived from the case seed and a stream id (independent), which also redraws the IMU noise, because the IMU seed is hashed from both seeds; and with the same independent seeds but the IMU noise pinned to its shared draw (streams only). The table gives the median over the seeds. This measures the difference and decides nothing.

| Case | Metric | Shared | Independent | Streams only (IMU fixed) |
|---|---|---:|---:|---:|
| `outage_visual` | ATE RMSE | 25.44 m | 25.53 m | 25.68 m |
| `outage_visual` | mean NEES | 29904.1 | 30347.4 | 30724.9 |
| `outage_visual` | 2-sigma coverage | 0.071 | 0.076 | 0.077 |
| `outage_visual` | FDIR events | 49.0 | 49.0 | 49.0 |
| `vision_only` | ATE RMSE | 54.23 m | 52.38 m | 51.35 m |
| `vision_only` | mean NEES | 133504.2 | 124823.9 | 119591.4 |
| `vision_only` | 2-sigma coverage | 0.012 | 0.012 | 0.012 |
| `vision_only` | FDIR events | 0.0 | 0.0 | 0.0 |
| `outage_visual_clone` | ATE RMSE | 0.86 m | 0.80 m | 0.81 m |
| `outage_visual_clone` | mean NEES | 3.3 | 2.5 | 2.5 |
| `outage_visual_clone` | 2-sigma coverage | 1.000 | 0.999 | 1.000 |
| `outage_visual_clone` | FDIR events | 0.0 | 0.0 | 0.0 |
| `outage_control` | ATE RMSE | 9.74 m | 9.83 m | 9.74 m |
| `outage_control` | mean NEES | 2.9 | 3.0 | 2.9 |
| `outage_control` | 2-sigma coverage | 1.000 | 1.000 | 1.000 |
| `outage_control` | FDIR events | 0.0 | 0.0 | 0.0 |

With only the stream relationship changed (IMU fixed), the largest change of a median against the shared stream is in `outage_visual_clone`, mean NEES: 23.0 percent lower. With the streams and the IMU draw both changed, the largest change of a median against the shared stream is in `outage_visual_clone`, mean NEES: 21.8 percent lower.
The no-vision control `outage_control` has no vision stream, so its streams-only column equals the shared one. The single-anchor control `outage_visual` is in the table. The medians are over 10 development seeds of 4 cases, not a test of significance.

The rows are in `docs/data/shared_stream_effect.csv`, written by `navkit sweep shared-stream`, and the page section is checked against them by `tests/test_shared_stream_effect.py`.

Expand Down
2 changes: 1 addition & 1 deletion docs/data/metrics.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
"coverage_python": "3.12",
"outage_visual_window_coverage_percent": 5.0,
"outage_visual_window_nees": 2783.35,
"tests_collected_floor": 2700
"tests_collected_floor": 2800
}
Loading
Loading