diff --git a/CONSTRAINTS.md b/CONSTRAINTS.md index e047062..d4b646d 100644 --- a/CONSTRAINTS.md +++ b/CONSTRAINTS.md @@ -4,7 +4,7 @@ Invariants this project holds itself to. Every entry is either currently true or is a stated blocker; nothing here is aspirational. Ratchets are measured, not estimated, and the command to re-measure each one is given. -Last measured: more than 2700 tests collected, all of which pass except the deliberate +Last measured: more than 2800 tests collected, all of which pass except the deliberate xfails and the skips that `pytest -rs` lists, and 94.9% line coverage (15550/16379 executable lines) on **CPython 3.12**, measured with `python scripts/coverage_report.py`. The figures come from `docs/data/metrics.json`, which `scripts/metrics.py` checks in CI. Coverage is quoted from that script alone: @@ -92,7 +92,7 @@ the floors themselves are enforced by `scripts/coverage_report.py --ratchet`. | Ratchet | Floor | Current (CPython 3.12) | Re-measure with | |---|---:|---:|---| -| Tests collected | 405 | more than 2700 | `pytest`, `scripts/metrics.py --check` | +| Tests collected | 405 | more than 2800 | `pytest`, `scripts/metrics.py --check` | | Tests passing | 300 | all collected, except the deliberate xfails and the listed skips | `pytest -rs` | | Line coverage | 75% | 94.9% | `scripts/coverage_report.py`, `scripts/metrics.py --check-coverage` | | `io/trajectory.py` coverage | 85% | 95.0% | as above | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e1c8d6d..27a1865 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -52,7 +52,7 @@ mypy src --ignore-missing-imports `ruff format src tests scripts` applies formatting; the `--check` form is the gate. -The suite collects more than 2700 tests and all of them pass on a fresh clone, except the deliberate +The suite collects more than 2800 tests and all of them pass on a fresh clone, except the deliberate xfails and the skips. `pytest -rs` lists the skips: two need TUM VI reference data that is not vendored (see below), and the doc-table check runs once `python scripts/run_benchmark.py` has written `results/benchmark.json`. The count and the other numbers the docs quote are in diff --git a/ROADMAP.md b/ROADMAP.md index f0c782c..48ae94f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -5,7 +5,7 @@ Status markers: `[x]` done, `[~]` in progress, `[ ]` not started, `[!]` blocked. Checked items record what was measured when the item closed. They are not kept current: the current values are in the README tables, `docs/results.md` and `CONSTRAINTS.md` (B1). -Measured baseline: more than 2700 tests collected and 94.9% line coverage on CPython +Measured baseline: more than 2800 tests collected and 94.9% line coverage on CPython 3.12 (`docs/data/metrics.json`, from `scripts/metrics.py` and `scripts/coverage_report.py`). Every number below re-measures against that, not against a remembered value. diff --git a/docs/adr/0010-determinism-and-seeding.md b/docs/adr/0010-determinism-and-seeding.md index 58d09e0..e345182 100644 --- a/docs/adr/0010-determinism-and-seeding.md +++ b/docs/adr/0010-determinism-and-seeding.md @@ -59,6 +59,8 @@ writes it down so the next stochastic stream follows it too. **What was measured.** The development seeds, 10 of them, were run twice: as they are, and with the vision seed derived from the case seed and a stream id ([issue #259](https://github.com/Telschow/contested-nav/issues/259), `docs/data/shared_stream_effect.csv`, `docs/calibration.md`). The result was within seed noise for the aggregate. The sign of the per-seed change was mixed, and the mean paired difference was within about one standard error of zero for ATE RMSE and mean NEES in all four cases. The single-anchor control stayed overconfident and the clone stayed calibrated. Single seeds moved far (seed 0 of `outage_visual`, 17.6 m to 7.9 m ATE). The comparison also changes the IMU noise draw, which is hashed from both seeds, so it measures both changes together. Ten seeds exclude a large systematic effect, not a small one. +**Follow-up: the two changes apart.** The study now has a third seeding, `streams_only`: the independent vision seed with the IMU noise pinned to its shared draw, for the duration of one run and with nothing in `src/navkit/degrade` changed. It gives the stream relationship alone, with the IMU fixed. The medians under it are generated on `docs/calibration.md` from `docs/data/shared_stream_effect.csv`. For every case the streams-only change is of the same size and sign as the combined change, so redrawing the IMU explains little of it. `outage_control`, which has no vision, comes out identical to the shared run, as it must. The sample is the same ten development seeds, so the earlier limit stands: a large systematic effect is excluded, a small one is not. + **Decision.** The streams are not changed. No committed number, table or golden is regenerated, and the B1 result in [ADR-0028](0028-the-b1-gate.md) stands as it was produced. The shared stream is documented instead, in `docs/MODEL.md` and in the calibration page, where the measurement is generated from its CSV. - Decision 2 is read as: each stream has its own seed field. It does not say the generators differ when the values are equal. diff --git a/docs/calibration.md b/docs/calibration.md index 0c07e9d..9e7480c 100644 --- a/docs/calibration.md +++ b/docs/calibration.md @@ -646,28 +646,29 @@ In scenario S2, for the plain clone (no model): the NEES conditions do not all h -**What the shared noise stream changes (development seeds, synthetic; MEASUREMENT).** At equal seeds the GNSS noise and the visual rotation noise are one normal sequence. Each case below was run over 10 seeds twice: with that shared stream, and with the vision seed derived from the case seed and a stream id. The table gives the median over the seeds under both seedings. This measures the difference and decides nothing. - -| Case | Metric | Shared | Independent | -|---|---|---:|---:| -| `outage_visual` | ATE RMSE | 25.44 m | 25.53 m | -| `outage_visual` | mean NEES | 29904.1 | 30347.4 | -| `outage_visual` | 2-sigma coverage | 0.071 | 0.076 | -| `outage_visual` | FDIR events | 49.0 | 49.0 | -| `vision_only` | ATE RMSE | 54.23 m | 52.38 m | -| `vision_only` | mean NEES | 133504.2 | 124823.9 | -| `vision_only` | 2-sigma coverage | 0.012 | 0.012 | -| `vision_only` | FDIR events | 0.0 | 0.0 | -| `outage_visual_clone` | ATE RMSE | 0.86 m | 0.80 m | -| `outage_visual_clone` | mean NEES | 3.3 | 2.5 | -| `outage_visual_clone` | 2-sigma coverage | 1.000 | 0.999 | -| `outage_visual_clone` | FDIR events | 0.0 | 0.0 | -| `outage_control` | ATE RMSE | 9.74 m | 9.83 m | -| `outage_control` | mean NEES | 2.9 | 3.0 | -| `outage_control` | 2-sigma coverage | 1.000 | 1.000 | -| `outage_control` | FDIR events | 0.0 | 0.0 | - -The largest change of a median is in `outage_visual_clone`, mean NEES: 21.8 percent lower with independent streams. The single-anchor control `outage_visual` is in the table. Changing the vision seed also changes the IMU noise draw of every case, because the IMU seed is hashed from both seeds, so `outage_control`, which has no vision, moves too. The difference is the effect of both changes together. +**What the shared noise stream changes (development seeds, synthetic; MEASUREMENT).** At equal seeds the GNSS noise and the visual rotation noise are one normal sequence. Each case below was run over 10 seeds three times: with that shared stream; with the vision seed derived from the case seed and a stream id (independent), which also redraws the IMU noise, because the IMU seed is hashed from both seeds; and with the same independent seeds but the IMU noise pinned to its shared draw (streams only). The table gives the median over the seeds. This measures the difference and decides nothing. + +| Case | Metric | Shared | Independent | Streams only (IMU fixed) | +|---|---|---:|---:|---:| +| `outage_visual` | ATE RMSE | 25.44 m | 25.53 m | 25.68 m | +| `outage_visual` | mean NEES | 29904.1 | 30347.4 | 30724.9 | +| `outage_visual` | 2-sigma coverage | 0.071 | 0.076 | 0.077 | +| `outage_visual` | FDIR events | 49.0 | 49.0 | 49.0 | +| `vision_only` | ATE RMSE | 54.23 m | 52.38 m | 51.35 m | +| `vision_only` | mean NEES | 133504.2 | 124823.9 | 119591.4 | +| `vision_only` | 2-sigma coverage | 0.012 | 0.012 | 0.012 | +| `vision_only` | FDIR events | 0.0 | 0.0 | 0.0 | +| `outage_visual_clone` | ATE RMSE | 0.86 m | 0.80 m | 0.81 m | +| `outage_visual_clone` | mean NEES | 3.3 | 2.5 | 2.5 | +| `outage_visual_clone` | 2-sigma coverage | 1.000 | 0.999 | 1.000 | +| `outage_visual_clone` | FDIR events | 0.0 | 0.0 | 0.0 | +| `outage_control` | ATE RMSE | 9.74 m | 9.83 m | 9.74 m | +| `outage_control` | mean NEES | 2.9 | 3.0 | 2.9 | +| `outage_control` | 2-sigma coverage | 1.000 | 1.000 | 1.000 | +| `outage_control` | FDIR events | 0.0 | 0.0 | 0.0 | + +With only the stream relationship changed (IMU fixed), the largest change of a median against the shared stream is in `outage_visual_clone`, mean NEES: 23.0 percent lower. With the streams and the IMU draw both changed, the largest change of a median against the shared stream is in `outage_visual_clone`, mean NEES: 21.8 percent lower. +The no-vision control `outage_control` has no vision stream, so its streams-only column equals the shared one. The single-anchor control `outage_visual` is in the table. The medians are over 10 development seeds of 4 cases, not a test of significance. The rows are in `docs/data/shared_stream_effect.csv`, written by `navkit sweep shared-stream`, and the page section is checked against them by `tests/test_shared_stream_effect.py`. diff --git a/docs/data/metrics.json b/docs/data/metrics.json index a4189bd..df3fc31 100644 --- a/docs/data/metrics.json +++ b/docs/data/metrics.json @@ -6,5 +6,5 @@ "coverage_python": "3.12", "outage_visual_window_coverage_percent": 5.0, "outage_visual_window_nees": 2783.35, - "tests_collected_floor": 2700 + "tests_collected_floor": 2800 } diff --git a/docs/data/shared_stream_effect.csv b/docs/data/shared_stream_effect.csv index c505072..2e8e0b8 100644 --- a/docs/data/shared_stream_effect.csv +++ b/docs/data/shared_stream_effect.csv @@ -1,41 +1,41 @@ -case,seed,ate_rmse_m_shared,ate_rmse_m_independent,ate_rmse_m_diff,nees_mean_shared,nees_mean_independent,nees_mean_diff,coverage_2sigma_shared,coverage_2sigma_independent,coverage_2sigma_diff,fdir_events_shared,fdir_events_independent,fdir_events_diff -outage_visual,0,17.56790334065547,7.935858836227262,-9.632044504428208,14018.64282439698,2494.647020262347,-11523.995804134633,0.1916347275454091,0.20996500583236127,0.018330278286952162,49.0,36.0,-13.0 -outage_visual,1,19.02375145626059,16.753963773197576,-2.269787683063015,16536.85022052687,12858.274116814191,-3678.5761037126795,0.06982169638393601,0.07532077987002166,0.005499083486085646,49.0,49.0,0.0 -outage_visual,2,6.5658380322158685,6.537582992083011,-0.028255040132857623,1762.6807355170142,1665.967558552416,-96.71317696459823,0.09165139143476088,0.1158140309948342,0.02416263956007332,35.0,23.0,-12.0 -outage_visual,3,47.80995458801466,54.35643478735286,6.546480199338198,102705.79849603804,133168.55971570144,30462.7612196634,0.04282619563406099,0.04349275120813198,0.0006665555740709914,50.0,50.0,0.0 -outage_visual,4,37.00873922005613,39.89390869354247,2.885169473486343,62661.865626238185,72866.58925977634,10204.72363353816,0.06565572404599233,0.06832194634227629,0.0026662222962839516,49.0,49.0,0.0 -outage_visual,5,21.542041655104992,22.2515264907168,0.7094848356118071,21179.107861354456,22614.122616473476,1435.0147551190203,0.17163806032327944,0.17663722712881186,0.004999166805532418,49.0,49.0,0.0 -outage_visual,6,23.39197121045072,21.03853128289814,-2.3534399275525786,25182.381205684935,20336.88139745252,-4845.499808232416,0.049158473587735375,0.04999166805532411,0.000833194467588734,49.0,49.0,0.0 -outage_visual,7,27.49488879427997,28.81677860064089,1.3218898063609217,34625.77920962559,38080.6469860678,3454.86777644221,0.12014664222629562,0.11248125312447925,-0.0076653891018163695,49.0,49.0,0.0 -outage_visual,8,53.6879693051257,55.039481943380046,1.3515126382543485,130301.90453234632,136869.11049112192,6567.205958775594,0.07132144642559574,0.07698716880519914,0.005665722379603402,49.0,49.0,0.0 -outage_visual,9,29.98994606093875,29.401676853189315,-0.588269207749434,41446.40141108007,40099.35039688116,-1347.0510141989143,0.04665889018496917,0.04665889018496917,0.0,49.0,49.0,0.0 -vision_only,0,26.726529467263198,24.173681433535375,-2.552848033727823,32045.81679592718,26376.77448771701,-5669.04230821017,0.014997500416597235,0.014997500416597235,0.0,0.0,0.0,0.0 -vision_only,1,43.962081082883486,39.473107557412874,-4.4889735254706125,86572.55548729422,69743.36013885577,-16829.195348438443,0.0033327778703549408,0.0033327778703549408,0.0,0.0,0.0,0.0 -vision_only,2,32.45946939899342,27.1569294445454,-5.302539954448019,47537.30392301714,33136.61674177516,-14400.687181241985,0.06832194634227629,0.0733211131478087,0.004999166805532418,0.0,0.0,0.0 -vision_only,3,62.2603185095318,65.99030596488397,3.7299874553521732,175707.34637830517,198117.23947735075,22409.893099045585,0.004999166805532411,0.004999166805532411,0.0,0.0,0.0,0.0 -vision_only,4,73.24843972035387,76.91969435847406,3.671254638120189,240177.45617683226,264887.8205651644,24710.36438833212,0.0066655557407098815,0.0066655557407098815,0.0,0.0,0.0,0.0 -vision_only,5,51.85297515541508,52.11135672282054,0.2583815674054577,120178.79331986172,121555.75546318747,1376.9621433257416,0.018330278286952176,0.016663889351774704,-0.0016663889351774715,0.0,0.0,0.0 -vision_only,6,60.046421111013025,61.16883615287566,1.122415041862638,162064.66552752285,168220.59388975118,6155.928362228326,0.008331944675887352,0.008331944675887352,0.0,0.0,0.0,0.0 -vision_only,7,39.82135583058336,38.959497688438155,-0.8618581421452021,71411.171169793,68407.45828541275,-3003.7128843802493,0.0033327778703549408,0.0033327778703549408,0.0,0.0,0.0,0.0 -vision_only,8,94.23631501364375,91.51621379587549,-2.72010121776826,399590.5343538818,377518.902378856,-22071.63197502581,0.021163139476753873,0.019996667222129644,-0.0011664722546242297,0.0,0.0,0.0 -vision_only,9,56.60290838144201,52.645392348912985,-3.9575160325290284,146829.56465827808,128091.98932877449,-18737.575329503583,0.026162306282286284,0.01749708381936344,-0.008665222462922846,0.0,0.0,0.0 -outage_visual_clone,0,0.7302553482601657,1.0282643115914,0.2980089633312344,2.1853018824808372,3.8265142670313113,1.641212384550474,1.0,1.0,0.0,0.0,0.0,0.0 -outage_visual_clone,1,1.0740614523038756,0.8300041903844854,-0.24405726191939026,3.916510724757279,2.7467235337080202,-1.1697871910492585,1.0,1.0,0.0,0.0,0.0,0.0 -outage_visual_clone,2,0.993323448119455,1.2507674340485573,0.2574439859291022,3.762089326142326,4.347850890088849,0.585761563946523,1.0,0.9983336110648225,-0.0016663889351774541,0.0,0.0,0.0 -outage_visual_clone,3,0.8404964182425271,0.6854366283075631,-0.15505978993496394,3.2018760080436293,2.092128219049551,-1.109747788994078,0.9931678053657723,0.9933344442592901,0.00016663889351775651,0.0,0.0,0.0 -outage_visual_clone,4,0.8741008902503773,1.2530962839544666,0.3789953937040893,3.3050779901567378,5.989293991390467,2.684216001233729,1.0,0.9400099983336111,-0.059990001666388904,0.0,0.0,0.0 -outage_visual_clone,5,1.0732069247728613,0.5513815617034783,-0.521825363069383,3.597842037481508,1.5679904523563613,-2.029851585125147,1.0,1.0,0.0,0.0,0.0,0.0 -outage_visual_clone,6,1.0869973652007667,0.7444685751718269,-0.3425287900289399,3.9416408840128865,2.342461627136799,-1.5991792568760874,1.0,0.9988335277453758,-0.0011664722546241846,0.0,0.0,0.0 -outage_visual_clone,7,0.6089181898354253,0.7033532949830767,0.09443510514765141,2.126764675683063,2.057700745135718,-0.06906393054734483,1.0,1.0,0.0,0.0,0.0,0.0 -outage_visual_clone,8,0.8510309399377713,0.8367118520736964,-0.014319087864074986,3.126834580210158,3.0786597669854996,-0.04817481322465822,0.9803366105649058,0.9866688885185803,0.0063322779536744145,0.0,0.0,0.0 -outage_visual_clone,9,0.6211295204125348,0.7674382956848396,0.1463087752723048,1.8168792619312162,1.9863659617029654,0.16948669977174924,1.0,1.0,0.0,0.0,0.0,0.0 -outage_control,0,6.529711423279851,6.529711423279851,0.0,3.181460385783359,3.181460385783359,0.0,1.0,1.0,0.0,0.0,0.0,0.0 -outage_control,1,12.831808221149172,12.784656879036206,-0.04715134211296679,3.953088803262574,3.511281452421199,-0.4418073508413749,1.0,1.0,0.0,0.0,0.0,0.0 -outage_control,2,9.756930553147333,9.883522065689732,0.1265915125423991,2.1767128277612504,2.5944644736136775,0.4177516458524271,1.0,1.0,0.0,0.0,0.0,0.0 -outage_control,3,10.223654966929733,10.371903984296017,0.14824901736628426,3.528667279892539,3.3133210232661665,-0.2153462566263724,0.9933344442592901,0.9933344442592901,0.0,0.0,0.0,0.0 -outage_control,4,15.04611543663912,14.852557004666306,-0.1935584319728143,5.342271676699745,5.298089887598451,-0.044181789101294555,0.9933344442592901,0.9933344442592901,0.0,0.0,0.0,0.0 -outage_control,5,6.373587460034523,5.948357931672375,-0.42522952836214767,1.9409752290510491,1.9528393053044832,0.011864076253434108,1.0,1.0,0.0,0.0,0.0,0.0 -outage_control,6,3.152179510112384,3.705485339800482,0.5533058296880977,1.4188392162496777,1.5454749264300756,0.12663571018039788,1.0,1.0,0.0,0.0,0.0,0.0 -outage_control,7,9.71809409607755,9.769107805135917,0.051013709058366885,2.5386096969748158,2.892051300835571,0.3534416038607553,1.0,1.0,0.0,0.0,0.0,0.0 -outage_control,8,17.613232953403458,17.499854801472484,-0.11337815193097356,5.976067149253434,5.927980230249868,-0.04808691900356621,0.9600066655557407,0.9733377770371605,0.013331111481419855,0.0,0.0,0.0 -outage_control,9,1.0930464187018925,1.4506536041455302,0.3576071854436378,1.3316066945737073,1.3210146301873162,-0.010592064386391042,1.0,1.0,0.0,0.0,0.0,0.0 +case,seed,ate_rmse_m_shared,ate_rmse_m_independent,ate_rmse_m_diff,nees_mean_shared,nees_mean_independent,nees_mean_diff,coverage_2sigma_shared,coverage_2sigma_independent,coverage_2sigma_diff,fdir_events_shared,fdir_events_independent,fdir_events_diff,ate_rmse_m_streams_only,ate_rmse_m_diff_streams,nees_mean_streams_only,nees_mean_diff_streams,coverage_2sigma_streams_only,coverage_2sigma_diff_streams,fdir_events_streams_only,fdir_events_diff_streams +outage_visual,0,17.567903340655395,7.9358588362272595,-9.632044504428135,14018.642824396871,2494.6470202623354,-11523.995804134536,0.1916347275454091,0.20996500583236127,0.018330278286952162,49.0,36.0,-13.0,7.859194957313447,-9.708708383341948,2441.1271322009356,-11577.515692195935,0.20996500583236127,0.018330278286952162,37.0,-12.0 +outage_visual,1,19.023751456260815,16.753963773197558,-2.2697876830632566,16536.85022052747,12858.274116814104,-3678.576103713367,0.06982169638393601,0.07532077987002166,0.005499083486085646,49.0,49.0,0.0,17.02464030156905,-1.9991111546917644,13273.19583985244,-3263.6543806750306,0.0761539743376104,0.006332277953674387,49.0,0.0 +outage_visual,2,6.56583803221568,6.537582992083256,-0.028255040132424192,1762.6807355168553,1665.9675585526434,-96.71317696421193,0.09165139143476088,0.1158140309948342,0.02416263956007332,35.0,23.0,-12.0,7.395755779436978,0.8299177472212982,2216.6956011678503,454.01486565099503,0.09498416930511581,0.003332777870354936,35.0,0.0 +outage_visual,3,47.80995458801468,54.356434787352924,6.546480199338241,102705.79849603806,133168.55971570153,30462.761219663473,0.04282619563406099,0.04349275120813198,0.0006665555740709914,50.0,50.0,0.0,51.80461645540296,3.994661867388274,120805.97302470756,18100.174528669508,0.04349275120813198,0.0006665555740709914,50.0,0.0 +outage_visual,4,37.00873922005581,39.89390869354242,2.885169473486613,62661.865626236555,72866.58925977597,10204.723633539412,0.06565572404599233,0.06832194634227629,0.0026662222962839516,49.0,49.0,0.0,39.63444248910055,2.6257032690447417,71807.14959271328,9145.283966476723,0.06865522412931178,0.0029995000833194507,49.0,0.0 +outage_visual,5,21.542041655105013,22.25152649071676,0.7094848356117467,21179.10786135452,22614.122616473276,1435.0147551187547,0.17163806032327944,0.17663722712881186,0.004999166805532418,49.0,49.0,0.0,22.822685717609513,1.2806440625045,23773.852284084434,2594.744422729913,0.1749708381936344,0.0033327778703549638,49.0,0.0 +outage_visual,6,23.391971210450862,21.038531282897843,-2.353439927553019,25182.381205685426,20336.88139745165,-4845.499808233777,0.049158473587735375,0.04999166805532411,0.000833194467588734,49.0,49.0,0.0,23.154465040053847,-0.23750617039701538,24695.11867413609,-487.2625315493351,0.04999166805532411,0.000833194467588734,49.0,0.0 +outage_visual,7,27.49488879427951,28.81677860064107,1.3218898063615576,34625.77920962384,38080.64698606858,3454.867776444742,0.12014664222629562,0.11248125312447925,-0.0076653891018163695,49.0,49.0,0.0,29.204311444176803,1.7094226498972915,39015.66894301538,4389.889733391537,0.11414764205965672,-0.0059990001666389015,49.0,0.0 +outage_visual,8,53.68796930512569,55.03948194338007,1.351512638254377,130301.90453234655,136869.1104911219,6567.205958775332,0.07132144642559574,0.07698716880519914,0.005665722379603402,49.0,49.0,0.0,54.42120747028923,0.7332381651635416,133897.36116198334,3595.456629636785,0.07698716880519914,0.005665722379603402,49.0,0.0 +outage_visual,9,29.989946060938514,29.401676853189958,-0.5882692077485565,41446.40141107905,40099.350396884016,-1347.0510141950363,0.04665889018496917,0.04665889018496917,0.0,49.0,49.0,0.0,28.210408901425144,-1.7795371595133709,36754.68057047882,-4691.720840600232,0.04665889018496917,0.0,49.0,0.0 +vision_only,0,26.726529467263095,24.17368143353514,-2.5528480337279547,32045.816795926843,26376.77448771624,-5669.042308210603,0.014997500416597235,0.014997500416597235,0.0,0.0,0.0,0.0,24.258849819601796,-2.467679647661299,26436.068928772074,-5609.747867154769,0.014997500416597235,0.0,0.0,0.0 +vision_only,1,43.962081082883635,39.4731075574128,-4.488973525470833,86572.55548729505,69743.36013885547,-16829.195348439578,0.0033327778703549408,0.0033327778703549408,0.0,0.0,0.0,0.0,38.82593622612558,-5.136144856758058,67430.96315923033,-19141.59232806471,0.0033327778703549408,0.0,0.0,0.0 +vision_only,2,32.45946939899332,27.15692944454457,-5.302539954448751,47537.303923016654,33136.616741771955,-14400.687181244699,0.06832194634227629,0.0733211131478087,0.004999166805532418,0.0,0.0,0.0,27.416907006214675,-5.042562392778645,33929.65632869143,-13607.647594325223,0.07282119646725546,0.004499250124979176,0.0,0.0 +vision_only,3,62.260318509531906,65.99030596488412,3.7299874553522088,175707.34637830604,198117.2394773518,22409.89309904576,0.004999166805532411,0.004999166805532411,0.0,0.0,0.0,0.0,66.23675035838768,3.976431848855775,199200.26452227874,23492.918143972696,0.004999166805532411,0.0,0.0,0.0 +vision_only,4,73.24843972035376,76.91969435847382,3.671254638120061,240177.45617683145,264887.8205651625,24710.364388331072,0.0066655557407098815,0.0066655557407098815,0.0,0.0,0.0,0.0,77.2869292114778,4.038489491124039,267270.9853989671,27093.529222135665,0.0066655557407098815,0.0,0.0,0.0 +vision_only,5,51.852975155415095,52.11135672282054,0.2583815674054435,120178.7933198619,121555.75546318742,1376.9621433255234,0.018330278286952176,0.016663889351774704,-0.0016663889351774715,0.0,0.0,0.0,51.032941609034815,-0.8200335463802801,116389.5348721795,-3789.2584476823977,0.016663889351774704,-0.0016663889351774715,0.0,0.0 +vision_only,6,60.04642111101356,61.16883615287558,1.1224150418620198,162064.66552752812,168220.59388975042,6155.928362222301,0.008331944675887352,0.008331944675887352,0.0,0.0,0.0,0.0,59.87386662003416,-0.172554490979401,160951.74434921655,-1112.9211783115752,0.008331944675887352,0.0,0.0,0.0 +vision_only,7,39.82135583058349,38.95949768843818,-0.8618581421453158,71411.17116979367,68407.45828541281,-3003.7128843808605,0.0033327778703549408,0.0033327778703549408,0.0,0.0,0.0,0.0,38.40066876976285,-1.4206870608206401,66459.44761328035,-4951.723556513316,0.0033327778703549408,0.0,0.0,0.0 +vision_only,8,94.23631501364397,91.51621379587556,-2.7201012177684163,399590.5343538858,377518.9023788572,-22071.631975028606,0.021163139476753873,0.019996667222129644,-0.0011664722546242297,0.0,0.0,0.0,92.5814433380587,-1.6548716755852695,385835.56834230287,-13754.966011582932,0.019996667222129644,-0.0011664722546242297,0.0,0.0 +vision_only,9,56.602908381441424,52.64539234891225,-3.9575160325291705,146829.56465827316,128091.98932876892,-18737.575329504238,0.026162306282286284,0.01749708381936344,-0.008665222462922846,0.0,0.0,0.0,51.659523388881055,-4.943384992560368,122793.34634227506,-24036.218315998092,0.017163806032327946,-0.008998500249958338,0.0,0.0 +outage_visual_clone,0,0.7302553482600196,1.0282643115914147,0.29800896333139515,2.185301882480629,3.826514267031367,1.6412123845507383,1.0,1.0,0.0,0.0,0.0,0.0,0.9977164773123999,0.2674611290523803,3.791462540357359,1.60616065787673,1.0,0.0,0.0,0.0 +outage_visual_clone,1,1.0740614523040282,0.8300041903846871,-0.24405726191934107,3.9165107247584636,2.7467235337087708,-1.1697871910496929,1.0,1.0,0.0,0.0,0.0,0.0,0.839423413316584,-0.23463803898744418,2.6559412261499213,-1.2605694986085423,1.0,0.0,0.0,0.0 +outage_visual_clone,2,0.9933234481194868,1.250767434048554,0.25744398592906714,3.7620893261433146,4.347850890088792,0.5857615639454776,1.0,0.9983336110648225,-0.0016663889351774541,0.0,0.0,0.0,1.239733206940966,0.24640975882147909,4.335688492662537,0.5735991665192222,0.9983336110648225,-0.0016663889351774541,0.0,0.0 +outage_visual_clone,3,0.8404964182427889,0.6854366283075447,-0.15505978993524416,3.2018760080448425,2.092128219049619,-1.1097477889952234,0.9931678053657723,0.9933344442592901,0.00016663889351775651,0.0,0.0,0.0,0.6603325684432989,-0.18016384979948996,1.8310001018072393,-1.3708759062376032,0.9933344442592901,0.00016663889351775651,0.0,0.0 +outage_visual_clone,4,0.8741008902502918,1.2530962839542925,0.3789953937040007,3.3050779901551968,5.989293991388893,2.684216001233696,1.0,0.9400099983336111,-0.059990001666388904,0.0,0.0,0.0,1.239606452064482,0.36550556181419025,6.053336426449411,2.748258436294214,0.9455090818196967,-0.05449091818030327,0.0,0.0 +outage_visual_clone,5,1.0732069247730964,0.551381561703529,-0.5218253630695674,3.5978420374830606,1.5679904523559733,-2.029851585127087,1.0,1.0,0.0,0.0,0.0,0.0,0.5784107003143639,-0.4947962244587325,1.7869357423676866,-1.810906295115374,1.0,0.0,0.0,0.0 +outage_visual_clone,6,1.0869973652008305,0.7444685751719007,-0.3425287900289298,3.941640884014958,2.342461627137628,-1.59917925687733,1.0,0.9988335277453758,-0.0011664722546241846,0.0,0.0,0.0,0.7356316074894482,-0.35136575771138223,2.3543275388282665,-1.5873133451866916,0.9990001666388936,-0.000999833361106428,0.0,0.0 +outage_visual_clone,7,0.6089181898353216,0.7033532949828624,0.09443510514754072,2.1267646756824536,2.0577007451340226,-0.06906393054843107,1.0,1.0,0.0,0.0,0.0,0.0,0.7070861829725341,0.09816799313721247,2.1562877213012954,0.02952304561884178,1.0,0.0,0.0,0.0 +outage_visual_clone,8,0.8510309399377116,0.8367118520736855,-0.014319087864026137,3.126834580209927,3.078659766984515,-0.04817481322541184,0.9803366105649058,0.9866688885185803,0.0063322779536744145,0.0,0.0,0.0,0.8842748602826336,0.03324392034492196,3.1455563851825863,0.018721804972659406,0.9800033327778703,-0.00033327778703551303,0.0,0.0 +outage_visual_clone,9,0.6211295204123862,0.7674382956847348,0.14630877527234853,1.8168792619307603,1.9863659617023892,0.1694866997716289,1.0,1.0,0.0,0.0,0.0,0.0,0.7826196413583383,0.16149012094595205,2.1812563020601825,0.3643770401294222,1.0,0.0,0.0,0.0 +outage_control,0,6.529711423279986,6.529711423279986,0.0,3.181460385784476,3.181460385784476,0.0,1.0,1.0,0.0,0.0,0.0,0.0,6.529711423279986,0.0,3.181460385784476,0.0,1.0,0.0,0.0,0.0 +outage_control,1,12.831808221149094,12.784656879036236,-0.04715134211285843,3.9530888032596283,3.511281452422505,-0.4418073508371232,1.0,1.0,0.0,0.0,0.0,0.0,12.831808221149094,0.0,3.9530888032596283,0.0,1.0,0.0,0.0,0.0 +outage_control,2,9.756930553147225,9.883522065689778,0.12659151254255363,2.176712827761094,2.5944644736096114,0.4177516458485173,1.0,1.0,0.0,0.0,0.0,0.0,9.756930553147225,0.0,2.176712827761094,0.0,1.0,0.0,0.0,0.0 +outage_control,3,10.223654966929745,10.371903984295917,0.14824901736617235,3.528667279892544,3.3133210232640873,-0.21534625662845652,0.9933344442592901,0.9933344442592901,0.0,0.0,0.0,0.0,10.223654966929745,0.0,3.528667279892544,0.0,0.9933344442592901,0.0,0.0,0.0 +outage_control,4,15.046115436639141,14.852557004666306,-0.19355843197283562,5.3422716766974885,5.298089887597506,-0.044181789099982716,0.9933344442592901,0.9933344442592901,0.0,0.0,0.0,0.0,15.046115436639141,0.0,5.3422716766974885,0.0,0.9933344442592901,0.0,0.0,0.0 +outage_control,5,6.373587460034851,5.94835793167238,-0.42522952836247097,1.940975229055362,1.9528393053046207,0.011864076249258781,1.0,1.0,0.0,0.0,0.0,0.0,6.373587460034851,0.0,1.940975229055362,0.0,1.0,0.0,0.0,0.0 +outage_control,6,3.152179510112326,3.7054853398005796,0.5533058296882536,1.4188392162501882,1.5454749264290615,0.12663571017887332,1.0,1.0,0.0,0.0,0.0,0.0,3.152179510112326,0.0,1.4188392162501882,0.0,1.0,0.0,0.0,0.0 +outage_control,7,9.718094096077518,9.769107805135928,0.05101370905840952,2.5386096969735292,2.892051300835884,0.3534416038623549,1.0,1.0,0.0,0.0,0.0,0.0,9.718094096077518,0.0,2.5386096969735292,0.0,1.0,0.0,0.0,0.0 +outage_control,8,17.61323295340344,17.49985480147253,-0.11337815193090961,5.97606714925277,5.927980230250838,-0.04808691900193196,0.9600066655557407,0.9733377770371605,0.013331111481419855,0.0,0.0,0.0,17.61323295340344,0.0,5.97606714925277,0.0,0.9600066655557407,0.0,0.0,0.0 +outage_control,9,1.0930464187019173,1.4506536041455351,0.3576071854436178,1.3316066945736493,1.321014630187469,-0.010592064386180322,1.0,1.0,0.0,0.0,0.0,0.0,1.0930464187019173,0.0,1.3316066945736493,0.0,1.0,0.0,0.0,0.0 diff --git a/src/navkit/shared_stream_effect.py b/src/navkit/shared_stream_effect.py index 8c01368..86a7248 100644 --- a/src/navkit/shared_stream_effect.py +++ b/src/navkit/shared_stream_effect.py @@ -20,8 +20,18 @@ A side effect of the method: :func:`navkit.degrade.inject._derive_seed` hashes both the GNSS and the vision seed, so changing the vision seed also changes the IMU noise draw of every case, ``outage_control`` included. The difference -below is therefore the effect of the new vision seed on the vision noise and on the IMU noise together. It is not -the effect of the vision rotation noise alone. +``independent`` minus ``shared`` is therefore the effect of the new vision seed on the vision noise and on the IMU +noise together. It is not the effect of the vision rotation noise alone. + +``streams_only`` separates the two (post-remediation audit, CN-AUD-01). It uses the seeds of ``independent`` and, for +the duration of one run, pins the IMU noise draws to the ones ``shared`` has, by answering the two IMU channels of +``_derive_seed`` with the hash of the shared seeds. Nothing in ``src/navkit/degrade`` changes, and the pin is +undone after the run. Then: + +* ``streams_only`` minus ``shared`` is the effect of the GNSS/vision stream relationship alone, with the IMU fixed; +* ``independent`` minus ``streams_only`` is the effect of redrawing the IMU alone. + +On ``outage_control``, which has no vision, ``streams_only`` equals ``shared`` exactly: a known answer the tests check. Seeds from 100 are refused through :func:`navkit.seed_policy.require_development`. The output is deterministic. """ @@ -29,12 +39,16 @@ from __future__ import annotations import argparse +import contextlib import copy import csv +import importlib import io import math import statistics +from collections.abc import Iterator from pathlib import Path +from types import SimpleNamespace from typing import Any import numpy as np @@ -52,6 +66,11 @@ CASES = ("outage_visual", "vision_only", "outage_visual_clone", "outage_control") SINGLE_ANCHOR = "outage_visual" SEEDINGS = ("shared", "independent") +#: ``independent``'s seeds with the IMU noise pinned to the ``shared`` draw. The third column, not in ``SEEDINGS``. +STREAMS_ONLY = "streams_only" +#: ``navkit.degrade`` re-exports a function named ``inject``, so ``from .degrade import inject`` gives the function. +inject: Any = importlib.import_module("navkit.degrade.inject") +IMU_CHANNELS = ("imu", "imu_bias0") DEFAULT_SEEDS = range(0, 10) #: The second entry of the ``SeedSequence`` that gives the vision stream its own seed. After 1013 and 1014 in models.py. VISION_STREAM = 1015 @@ -62,7 +81,13 @@ "coverage_2sigma": "2-sigma coverage", "fdir_events": "FDIR events", } -CSV_FIELDS = ("case", "seed", *(f"{m}_{s}" for m in METRICS for s in ("shared", "independent", "diff"))) +#: The first fourteen columns are the ones of the original study, unchanged. The ``streams_only`` ones follow. +CSV_FIELDS = ( + "case", + "seed", + *(f"{m}_{s}" for m in METRICS for s in ("shared", "independent", "diff")), + *(f"{m}_{s}" for m in METRICS for s in (STREAMS_ONLY, "diff_streams")), +) TABLE = Table( (Column("case"), Column("seed", int), *(Column(f, float) for f in CSV_FIELDS[2:])), key="case", @@ -75,10 +100,10 @@ def stream_seeds(seed: int, seeding: str) -> tuple[int, int]: """The (GNSS, vision) seeds for a case seed. ``shared`` is what the benchmark does.""" if seeding == "shared": return seed, seed - if seeding == "independent": + if seeding in ("independent", STREAMS_ONLY): vision = int(np.random.SeedSequence([seed, VISION_STREAM]).generate_state(1)[0]) return seed, vision - raise ValueError(f"seeding must be one of {', '.join(SEEDINGS)}, got {seeding!r}") + raise ValueError(f"seeding must be one of {', '.join((*SEEDINGS, STREAMS_ONLY))}, got {seeding!r}") def _metrics(rec: dict[str, Any]) -> dict[str, float]: @@ -91,7 +116,29 @@ def _metrics(rec: dict[str, Any]) -> dict[str, float]: } +@contextlib.contextmanager +def imu_pinned_to_shared(seed: int) -> Iterator[None]: + """Answer the IMU channels of ``_derive_seed`` as if the GNSS and vision seeds were both ``seed``, then undo it.""" + original = inject._derive_seed + + def pinned(scenario: Any, channel: str) -> int: + if channel in IMU_CHANNELS: + scenario = SimpleNamespace( + name=scenario.name, gnss=SimpleNamespace(seed=seed), vision=SimpleNamespace(seed=seed) + ) + return original(scenario, channel) + + inject._derive_seed = pinned + try: + yield + finally: + inject._derive_seed = original + + def _run_one(name: str, case: dict[str, Any], defaults: dict[str, Any], seed: int, seeding: str) -> dict[str, float]: + if seeding == STREAMS_ONLY: + with imu_pinned_to_shared(seed): + return _run_one(name, case, defaults, seed, "independent") if seeding == "shared": return _metrics(run_case(name, copy.deepcopy(case), defaults, seed=seed)) gnss, vision = stream_seeds(seed, seeding) @@ -113,12 +160,14 @@ def run_study(doc: dict[str, Any], seeds: list[int] | None = None) -> list[dict[ rows: list[dict[str, Any]] = [] for name in CASES: for seed in use: - by = {s: _run_one(name, cases[name], defaults, seed, s) for s in SEEDINGS} + by = {s: _run_one(name, cases[name], defaults, seed, s) for s in (*SEEDINGS, STREAMS_ONLY)} row: dict[str, Any] = {"case": name, "seed": seed} for m in METRICS: row[f"{m}_shared"] = by["shared"][m] row[f"{m}_independent"] = by["independent"][m] row[f"{m}_diff"] = by["independent"][m] - by["shared"][m] + row[f"{m}_{STREAMS_ONLY}"] = by[STREAMS_ONLY][m] + row[f"{m}_diff_streams"] = by[STREAMS_ONLY][m] - by["shared"][m] rows.append(row) return rows @@ -157,13 +206,13 @@ def median_of(rows: list[dict[str, Any]], case: str, field: str) -> float: return statistics.median(r[field] for r in rows if r["case"] == case) -def largest_change(rows: list[dict[str, Any]]) -> tuple[str, str, float]: - """The (case, metric, relative change of the median) with the largest absolute relative change of the median.""" +def largest_change(rows: list[dict[str, Any]], seeding: str = "independent") -> tuple[str, str, float]: + """The (case, metric, relative change of the median against ``shared``) with the largest absolute change.""" best = ("", "", 0.0) for case in CASES: for m in ("ate_rmse_m", "nees_mean"): base = median_of(rows, case, f"{m}_shared") - rel = (median_of(rows, case, f"{m}_independent") - base) / base + rel = (median_of(rows, case, f"{m}_{seeding}") - base) / base if abs(rel) > abs(best[2]): best = (case, m, rel) return best @@ -177,31 +226,41 @@ def _fmt(m: str, v: float) -> str: return f"{v:.1f}" +def _change_sentence(rows: list[dict[str, Any]], seeding: str, what: str) -> str: + case, metric, rel = largest_change(rows, seeding) + return ( + f"With {what}, the largest change of a median against the shared stream is in `{case}`, {LABELS[metric]}: " + f"{abs(rel) * 100:.1f} percent {'lower' if rel < 0 else 'higher'}." + ) + + def markdown_section(rows: list[dict[str, Any]]) -> str: - """The generated section of ``docs/calibration.md``: a table and one sentence, both from the rows.""" + """The generated section of ``docs/calibration.md``: a table and two sentences, all from the rows.""" n = len({r["seed"] for r in rows}) - case, metric, rel = largest_change(rows) + columns = (*SEEDINGS, STREAMS_ONLY) lines = [ f"**What the shared noise stream changes ({LABEL}; MEASUREMENT).** At equal seeds the GNSS noise and the " "visual rotation noise are one normal sequence. Each case below was run over " - f"{n} seeds twice: with that shared stream, and with the vision seed derived from the case seed and a stream " - "id. The table gives the median over the seeds under both seedings. This measures the difference and decides " - "nothing.", + f"{n} seeds three times: with that shared stream; with the vision seed derived from the case seed and a " + "stream id (independent), which also redraws the IMU noise, because the IMU seed is hashed from both seeds; " + "and with the same independent seeds but the IMU noise pinned to its shared draw (streams only). The table " + "gives the median over the seeds. This measures the difference and decides nothing.", "", - "| Case | Metric | Shared | Independent |", - "|---|---|---:|---:|", + "| Case | Metric | Shared | Independent | Streams only (IMU fixed) |", + "|---|---|---:|---:|---:|", ] for c in CASES: for m in METRICS: - a, b = (median_of(rows, c, f"{m}_{s}") for s in SEEDINGS) - lines.append(f"| `{c}` | {LABELS[m]} | {_fmt(m, a)} | {_fmt(m, b)} |") + a, b, d = (median_of(rows, c, f"{m}_{s}") for s in columns) + lines.append(f"| `{c}` | {LABELS[m]} | {_fmt(m, a)} | {_fmt(m, b)} | {_fmt(m, d)} |") lines += [ "", - f"The largest change of a median is in `{case}`, {LABELS[metric]}: {abs(rel) * 100:.1f} percent " - f"{'lower' if rel < 0 else 'higher'} with independent streams. " - "The single-anchor control `outage_visual` is in the table. Changing the vision seed also changes the IMU " - "noise draw of every case, because the IMU seed is hashed from both seeds, so `outage_control`, which has no " - "vision, moves too. The difference is the effect of both changes together.", + _change_sentence(rows, STREAMS_ONLY, "only the stream relationship changed (IMU fixed)") + + " " + + _change_sentence(rows, "independent", "the streams and the IMU draw both changed"), + "The no-vision control `outage_control` has no vision stream, so its streams-only column equals the shared " + "one. The single-anchor control `outage_visual` is in the table. The medians are over " + f"{n} development seeds of {len(CASES)} cases, not a test of significance.", "", "The rows are in `docs/data/shared_stream_effect.csv`, written by `navkit sweep shared-stream`, and the " "page section is checked against them by `tests/test_shared_stream_effect.py`.", diff --git a/tests/test_shared_stream_effect.py b/tests/test_shared_stream_effect.py index 686e2d3..35840ca 100644 --- a/tests/test_shared_stream_effect.py +++ b/tests/test_shared_stream_effect.py @@ -83,3 +83,63 @@ def test_the_independent_seeding_leaves_the_default_seeds_alone(): assert sse.stream_seeds(5, "shared") == (5, 5) with pytest.raises(ValueError): sse.stream_seeds(5, "other") + + +# ------------------------------------------------------------------------------- the streams-only seeding (CN-AUD-01) + + +class _Scenario: + def __init__(self, gnss: int, vision: int) -> None: + from types import SimpleNamespace + + self.name = "case" + self.gnss = SimpleNamespace(seed=gnss) + self.vision = SimpleNamespace(seed=vision) + + +def test_streams_only_uses_the_independent_seeds(): + assert sse.stream_seeds(4, sse.STREAMS_ONLY) == sse.stream_seeds(4, "independent") + + +def test_the_pin_gives_the_imu_channels_the_shared_draw_and_nothing_else_changes(): + original = sse.inject._derive_seed + shared = _Scenario(4, 4) + independent = _Scenario(*sse.stream_seeds(4, "independent")) + assert original(independent, "imu") != original(shared, "imu") + with sse.imu_pinned_to_shared(4): + for channel in sse.IMU_CHANNELS: + assert sse.inject._derive_seed(independent, channel) == original(shared, channel) + # a channel outside the pin keeps its own hash + assert sse.inject._derive_seed(independent, "other") == original(independent, "other") + assert sse.inject._derive_seed is original + + +def test_the_pin_is_undone_after_an_error(): + original = sse.inject._derive_seed + with pytest.raises(RuntimeError), sse.imu_pinned_to_shared(1): + raise RuntimeError + assert sse.inject._derive_seed is original + + +@pytest.mark.docs_only +def test_the_original_columns_are_the_first_fourteen(rows): + assert sse.CSV_FIELDS[:14] == ( + "case", + "seed", + *(f"{m}_{s}" for m in sse.METRICS for s in ("shared", "independent", "diff")), + ) + + +@pytest.mark.docs_only +def test_each_streams_difference_is_streams_only_minus_shared(rows): + for r in rows: + for m in sse.METRICS: + assert r[f"{m}_diff_streams"] == pytest.approx(r[f"{m}_streams_only"] - r[f"{m}_shared"]) + + +@pytest.mark.docs_only +def test_without_vision_the_streams_only_run_is_the_shared_run(rows): + """Known answer: ``outage_control`` has no vision, so pinning the IMU leaves nothing that differs.""" + for r in (r for r in rows if r["case"] == "outage_control"): + for m in sse.METRICS: + assert r[f"{m}_diff_streams"] == pytest.approx(0.0, abs=1e-9 * max(1.0, abs(r[f"{m}_shared"])))