Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,9 @@ dist
.hypothesis
opencode.json
.opencode
# Local secrets and tooling outside the package. The Dockerfile copies the whole context.
.env
**/.env
node_modules
.sandcastle/logs
.sandcastle/worktrees
13 changes: 11 additions & 2 deletions .sandcastle/main.mts
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,18 @@ import { z } from "zod";
// and validates it against this schema. We use Zod here, but any Standard
// Schema validator works just as well, Valibot, ArkType, etc. See
// https://standardschema.dev.
//
// The id must be an issue number. The branch is not taken from the planner:
// the review prompt runs it inside shell blocks, and the planner reads issue
// bodies, so a branch name from its text could carry a command.
const planSchema = z.object({
issues: z.array(
z.object({ id: z.string(), title: z.string(), branch: z.string() }),
z.object({ id: z.string().regex(/^[1-9][0-9]*$/), title: z.string() }),
),
});

const branchFor = (id: string) => `sandcastle/issue-${id}`;

// ---------------------------------------------------------------------------
// Configuration
// ---------------------------------------------------------------------------
Expand Down Expand Up @@ -93,7 +99,10 @@ for (let iteration = 1; iteration <= MAX_ITERATIONS; iteration++) {
output: sandcastle.Output.object({ tag: "plan", schema: planSchema }),
});

const issues = plan.output.issues;
const issues = plan.output.issues.map((issue) => ({
...issue,
branch: branchFor(issue.id),
}));

if (issues.length === 0) {
// No unblocked work: either everything is done or everything is blocked.
Expand Down
4 changes: 2 additions & 2 deletions .sandcastle/merge-prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,11 @@ If something fails, fix it and rerun only what failed. If the doc tables or metr

# PUSH

`main` rejects pushes from this sandbox (403), so deliver through a pull request. Only when every check above passes:
Deliver through a pull request. Do not count on `main` rejecting a push: the branch rules do not stop this token today. Only when every check above passes:

1. `git fetch origin`
2. `git merge origin/main --no-edit`. If this brings in changes, rerun the checks.
3. Check `git branch --show-current` is not `main`, then `git push origin HEAD` (the current branch, never `main`). If the current branch is `main`, stop and say so: a push to `main` skips the pull request, and the 403 is not guaranteed.
3. Check `git branch --show-current` is not `main`, then `git push origin HEAD` (the current branch, never `main`). If the current branch is `main`, stop and say so: a push to `main` skips the pull request and nothing on the server stops it.

Never force-push and never rewrite history. If the push is rejected, fetch, merge `origin/main` once more and try once more. If it is rejected again, stop and say so.

Expand Down
15 changes: 9 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,8 +208,9 @@ All of it, with sources, is in [Status and limits](docs/status.md) and
The code was written mostly with an AI coding assistant (Claude Code). The maintainer sets the scope and the order of
work, and the project's rule is that the maintainer writes and accepts the decision in each ADR. The evidence to trust
is the gates, not the authorship: finite-difference checks on the measurement Jacobians, property tests, a golden
snapshot of the benchmark, and CI. Since the first week of the repository, every change lands through a pull request
that must pass CI and that the maintainer merges.
snapshot of the benchmark, and CI. Most changes land through a pull request that passes CI. That is the practice, not
a rule the repository enforces: a few commits went to `main` directly, and the Sandcastle automation merges its own
pull requests.

## Scope and responsible use

Expand Down Expand Up @@ -256,10 +257,10 @@ tests/ unit, property, golden-snapshot and documentation consistency test
Recorded data and its citations:

- **EuRoC** ([page](docs/euroc.md)): Burri, M. et al. (2016). The EuRoC micro aerial vehicle datasets. *The International
Journal of Robotics Research*, 35(10), 1157-1163. Licence: set by the dataset's authors (ETH Zurich ASL); read the terms on the
dataset page before reuse. Not verified here.
Journal of Robotics Research*, 35(10), 1157-1163. Rights statement: "In Copyright - Non-Commercial Use
Permitted", set by the dataset's authors (ETH Zurich ASL). See the License section below for the committed result files.
- **TUM VI** ([page](docs/tumvi.md)): Schubert, D. et al. (2018). The TUM VI benchmark for evaluating visual-inertial odometry.
*IROS 2018*. Licence: set by the dataset's authors; read the terms on the dataset page before reuse. Not verified here.
*IROS 2018*. Licence: CC BY 4.0, as the dataset page states; the [TUM VI page](docs/tumvi.md) gives the attribution.
- **UrbanNav** ([page](docs/urbannav.md)): Hsu, L.-T. et al. (2023). UrbanNav: an open-sourced multisensory dataset for urban
positioning. *NAVIGATION*. Licence: MIT, as stated by the authors.

Expand All @@ -278,4 +279,6 @@ for example `euroc_*.csv`, `vo_*.csv`, `fault_classification*.csv`, `multipath_*
`slow_ramp.csv`, `vision_ramp.csv` and `timing.csv`. They are results, not the dataset. The EuRoC rights statement is
"In Copyright - Non-Commercial Use Permitted", so use those files for non-commercial purposes only, whatever the MIT
licence says about the code. The terms are the dataset authors' to set. Read them on the dataset page before any other
use.
use. Some of those files also hold rows computed from TUM VI. Whether the per-run and per-frame rows may stay
published is an open maintainer decision: [ADR-0042](docs/adr/0042-publishing-rows-derived-from-euroc.md),
[issue #278](https://github.com/Telschow/contested-nav/issues/278). No answer from the rights holder is recorded.
2 changes: 1 addition & 1 deletion docs/adr/0010-determinism-and-seeding.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ writes it down so the next stochastic stream follows it too.

**What was measured.** The development seeds, 10 of them, were run twice: as they are, and with the vision seed derived from the case seed and a stream id ([issue #259](https://github.com/Telschow/contested-nav/issues/259), `docs/data/shared_stream_effect.csv`, `docs/calibration.md`). The result was within seed noise for the aggregate. The sign of the per-seed change was mixed, and the mean paired difference was within about one standard error of zero for ATE RMSE and mean NEES in all four cases. The single-anchor control stayed overconfident and the clone stayed calibrated. Single seeds moved far (seed 0 of `outage_visual`, 17.6 m to 7.9 m ATE). The comparison also changes the IMU noise draw, which is hashed from both seeds, so it measures both changes together. Ten seeds exclude a large systematic effect, not a small one.

**Follow-up: the two changes apart.** The study now has a third seeding, `streams_only`: the independent vision seed with the IMU noise pinned to its shared draw, for the duration of one run and with nothing in `src/navkit/degrade` changed. It gives the stream relationship alone, with the IMU fixed. The medians under it are generated on `docs/calibration.md` from `docs/data/shared_stream_effect.csv`. For every case the streams-only change is of the same size and sign as the combined change, so redrawing the IMU explains little of it. `outage_control`, which has no vision, comes out identical to the shared run, as it must. The sample is the same ten development seeds, so the earlier limit stands: a large systematic effect is excluded, a small one is not.
**Follow-up: the two changes apart.** The study now has a third seeding, `streams_only`: the independent vision seed with the IMU noise pinned to its shared draw, for the duration of one run and with nothing in `src/navkit/degrade` changed. It gives the stream relationship alone, with the IMU fixed. The medians under it are generated on `docs/calibration.md` from `docs/data/shared_stream_effect.csv`. For the three cases with vision, the streams-only change of the ATE RMSE and mean NEES medians has the same sign as the combined change and the same order of size, so redrawing the IMU explains little of it. Both changes are within seed noise. `outage_control`, which has no vision, comes out identical to the shared run, as it must. The sample is the same ten development seeds, so the earlier limit stands: a large systematic effect is excluded, a small one is not.

**Decision.** The streams are not changed. No committed number, table or golden is regenerated, and the B1 result in [ADR-0028](0028-the-b1-gate.md) stands as it was produced. The shared stream is documented instead, in `docs/MODEL.md` and in the calibration page, where the measurement is generated from its CSV.

Expand Down
3 changes: 2 additions & 1 deletion docs/euroc.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ decision behind it is [ADR-0013](adr/0013-recorded-imu-with-simulated-gnss.md);
are discussed in [ADR-0014](adr/0014-textbook-imu-process-noise-by-default.md).

Citation: Burri, M. et al. (2016). The EuRoC micro aerial vehicle datasets. *The International Journal of Robotics
Research*, 35(10), 1157-1163. Licence: set by the dataset's authors; read the terms on the dataset page before reuse. Not verified here.
Research*, 35(10), 1157-1163. Rights statement: "In Copyright - Non-Commercial Use Permitted", set by the dataset's authors. Whether the committed
result rows may stay published is open ([ADR-0042](adr/0042-publishing-rows-derived-from-euroc.md)).

## Read this before quoting anything

Expand Down
4 changes: 3 additions & 1 deletion docs/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,9 @@ navkit euroc compare --markdown # default vs preset over every fet
```

The data is not vendored. Its rights statement is "In Copyright - Non-Commercial Use Permitted", so
the files go under `data/raw/`, which git ignores, and only aggregate results may be committed. A
the files go under `data/raw/`, which git ignores. Results computed from them are committed, including per-frame
front-end errors in `docs/data/vo_errors.csv`; whether those rows may stay is open
([ADR-0042](adr/0042-publishing-rows-derived-from-euroc.md)). A
result from this path is labelled `real_imu_simulated_gnss`. It tests the filter against real
inertial noise and a known reference. It is not GNSS-denied navigation in the field. The decision
behind it is [ADR-0013](adr/0013-recorded-imu-with-simulated-gnss.md).
Expand Down
2 changes: 1 addition & 1 deletion docs/tumvi.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ ADIS16448). The decision behind it is [ADR-0016](adr/0016-tum-vi-as-a-second-rec
the one of [ADR-0013](adr/0013-recorded-imu-with-simulated-gnss.md).

Citation: Schubert, D. et al. (2018). The TUM VI benchmark for evaluating visual-inertial odometry. *IROS 2018*.
Licence: set by the dataset's authors; read the terms on the dataset page before reuse. Not verified here.
Licence: CC BY 4.0, as the dataset page states. The attribution is under "Read this before quoting anything" below.

## Read this before quoting anything

Expand Down
2 changes: 1 addition & 1 deletion docs/urbannav.md
Original file line number Diff line number Diff line change
Expand Up @@ -322,7 +322,7 @@ What the section tests:
attitude normalised errors over the whole run and by third of the run, the pooled 2-sigma coverage per axis, the
sigma the filter claims against the receiver's own, and the fixes' own normalised error against the receiver's
reported sigma.
2. **How the gate behaves on real multipath fixes.** The rows are the fixes it rejects, whether the channel is
2. **How the gate behaves on real urban fixes.** Multipath is the likely cause of their errors, but it is not measured here. The rows are the fixes it rejects, whether the channel is
declared faulty, and the NIS against the gate's level.

**As declared**, one scalar GNSS sigma for every fix:
Expand Down
2 changes: 1 addition & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ site_url: https://telschow.github.io/contested-nav/
repo_url: https://github.com/Telschow/contested-nav
repo_name: Telschow/contested-nav
edit_uri: edit/main/docs/
copyright: MIT licensed. Not field validated.
copyright: Code MIT licensed. Results derived from EuRoC are for non-commercial use (see the README). Not field validated.

docs_dir: docs
site_dir: site
Expand Down
28 changes: 21 additions & 7 deletions src/navkit/b1_gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@
applies the conditions.

Seeds 100 to 124 are the held-out set. ADR-0028 allows one evaluation of it. The command refuses them unless
``--held-out`` is given, and so does :func:`run_gate`.
``--held-out`` is given, and so does :func:`run_gate`. ``--held-out`` takes exactly that set. Every other seed must be
a development seed: the later held-out blocks belong to :mod:`navkit.candidate_gate` (ADR-0030).

Two further rows use the same function on the development seeds and are labelled DIAGNOSTIC: the single-anchor
``outage_visual`` (the gate has to be able to say no) and the plain clone ``outage_visual_clone``. They are not gate
Expand Down Expand Up @@ -108,6 +109,18 @@ def _touches_held_out(seeds: list[int]) -> bool:
return any(s in HELD_OUT_SEEDS for s in seeds)


def check_seeds(seeds: list[int], held_out: bool) -> None:
"""Raise ``ValueError`` unless ``seeds`` are development seeds, or exactly the ADR-0028 set with ``held_out``."""
span = f"{HELD_OUT_SEEDS.start} to {HELD_OUT_SEEDS.stop - 1}"
if held_out:
if list(seeds) != list(HELD_OUT_SEEDS):
raise ValueError(f"with --held-out the seeds must be exactly {span}, the held-out set of ADR-0028")
return
if _touches_held_out(seeds):
raise ValueError(f"seeds {span} are the held-out set of ADR-0028; give --held-out to evaluate it (single use)")
seed_policy.require_development(seeds)


# --------------------------------------------------------------------------------------------- the runs


Expand Down Expand Up @@ -169,8 +182,7 @@ def run_arrays(name: str, case: dict[str, Any], defaults: dict[str, Any], seeds:

def run_gate(seeds: list[int], held_out: bool = False, diagnostics: bool = True) -> list[dict[str, Any]]:
"""Run the candidate, its control and (on development seeds) the two diagnostics, and return the CSV rows."""
if _touches_held_out(seeds) and not held_out:
raise ValueError("seeds 100 to 124 are the held-out set of ADR-0028 and need held_out=True")
check_seeds(seeds, held_out)
if len(seeds) < 2:
raise ValueError("the gate needs at least two seeds")
doc = _document()
Expand Down Expand Up @@ -297,14 +309,16 @@ def markdown_table(rows: list[dict[str, Any]]) -> str:


class _Parser(argparse.ArgumentParser):
"""Refuses the held-out seeds unless ``--held-out`` is given, and says the set is single use when it is."""
"""Refuses seeds :func:`check_seeds` refuses, and says the set is single use when ``--held-out`` is given."""

def parse_args(self, args: Any = None, namespace: Any = None) -> argparse.Namespace: # type: ignore[override]
ns = super().parse_args(args, namespace)
try:
check_seeds(ns.seeds, ns.held_out)
except ValueError as exc:
self.error(str(exc))
if ns.held_out:
print("warning: the held-out set (seeds 100 to 124) is single use per ADR-0028.", file=sys.stderr)
elif _touches_held_out(ns.seeds):
self.error("seeds 100 to 124 are the held-out set of ADR-0028; give --held-out to evaluate it (single use)")
return ns


Expand All @@ -320,7 +334,7 @@ def build_parser() -> argparse.ArgumentParser:
ap.add_argument("--seeds", type=_seeds_arg, required=True, help="A-B or a comma list; the gate uses exactly 25")
ap.add_argument("--csv", type=Path, required=True, help="write the rows here")
ap.add_argument("--page", type=Path, default=None, help="write the table between the b1-gate-dev markers")
ap.add_argument("--held-out", action="store_true", help="allow seeds 100 to 124 (single use, ADR-0028)")
ap.add_argument("--held-out", action="store_true", help="take exactly seeds 100 to 124 (single use, ADR-0028)")
return ap


Expand Down
20 changes: 20 additions & 0 deletions src/navkit/benchmark.py
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,14 @@ def _eskf_config(scenario: Scenario, keys: dict[str, Any]) -> EskfConfig:
)


def stream_seeds(scenario: Scenario) -> list[int]:
"""The seeds of a scenario's stochastic streams: GNSS, vision and, when it has one, camera drop."""
seeds = [scenario.gnss.seed, scenario.vision.seed]
if scenario.camera_drop is not None:
seeds.append(scenario.camera_drop.seed)
return seeds


def run_case(
name: str,
case: dict[str, Any],
Expand Down Expand Up @@ -357,6 +365,10 @@ def run_case(
scenario.vision.seed = seed
if scenario.camera_drop is not None:
scenario.camera_drop.seed = seed
else:
# The case's own seeds are used as written, so they must be development seeds. Only an explicit ``seed``
# replaces them: the sweeps check theirs, and the two gates pass their own sets.
seed_policy.require_development(stream_seeds(scenario))

# Clean streams from the analytic reference, then degraded by the scenario.
reference = synthetic_trajectory(syn, name=f"{name}-reference")
Expand Down Expand Up @@ -629,6 +641,14 @@ def main(argv: list[str] | None = None) -> int:
unknown = [k for k in selected if k not in cases]
if unknown:
raise SystemExit(f"unknown case(s): {', '.join(unknown)}; known: {', '.join(cases)}")
if args.seed is None:
# Refuse before the first case runs, not when run_case reaches the offending one.
for name in selected:
scenario = scenario_from_dict(cases[name].get("scenario", {}))
try:
seed_policy.require_development(stream_seeds(scenario) if scenario is not None else [])
except ValueError as exc:
ap.error(f"case {name!r} in {args.config}: {exc}")

results: list[dict[str, Any]] = []
for name in selected:
Expand Down
6 changes: 4 additions & 2 deletions src/navkit/seed_policy.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@
Seeds 0 to 99 are development seeds. Seeds 100 to 124 are the held-out set of ADR-0028, spent. From 125 the seeds are
held-out blocks of 25 (ADR-0030), run one at a time through :mod:`navkit.candidate_gate`. Every command that runs a
case on seed numbers calls :func:`require_development` before it starts: the ``navkit sweep`` entry points,
``navkit run --seed`` and ``--scene-seed``, ``navkit euroc run`` and ``navkit euroc compare``. The B1 gate and the
candidate gate are the two procedures allowed to use held-out seeds.
``navkit run --seed`` and ``--scene-seed``, ``navkit euroc run`` and ``navkit euroc compare``. Seeds a config file
sets itself are checked too: by ``navkit run`` before the first case, and by :func:`navkit.benchmark.run_case`
whenever no ``seed`` replaces them. The B1 gate and the candidate gate are the two procedures allowed to use held-out
seeds; the B1 gate takes only its own set, 100 to 124, and only with ``--held-out``.
"""

from __future__ import annotations
Expand Down
4 changes: 2 additions & 2 deletions src/navkit/urbannav_imposed.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@
question to answer here.

Results are rows in the format of :func:`urbannav_run.rows_for`, under a sequence name of their own
(``deepurban1_imposed_<start>s_<duration>s``). This module writes no CSV and no page. Rows in the committed
UrbanNav CSVs are not touched.
(``deepurban1_imposed_<start>s_<duration>s``). This module writes no page. With ``--csv`` it merges its rows
into that file under those names; rows of other sequences in the file are not touched.
"""

from __future__ import annotations
Expand Down
Loading
Loading