Skip to content

Replay MOOS-IvP missions recorded under pHelmIvP against the helm library - #5

Merged
tsaubergine merged 8 commits into
claude/admiring-clarke-s2ay8b-helm-libraryfrom
claude/admiring-clarke-s2ay8b-mission-replay
Nov 4, 2026
Merged

tsaubergine merged 8 commits into
claude/admiring-clarke-s2ay8b-helm-libraryfrom
claude/admiring-clarke-s2ay8b-mission-replay

Conversation

@tsaubergine

Copy link
Copy Markdown
Member

Stacked on #4 (helm library), and targets its branch. Merge #2, #3 and #4 first.

This is the second half of Phase 1: MOOS-IvP's own missions become IvPico's regression suite. Each mission runs once under real pHelmIvP and is recorded. The replay test then feeds the same inputs to ivpico::helm::Helm at pHelmIvP's iteration times, and checks that it posts and decides the same things. src/test/missions/README.md has an overview diagram.

Results

Eight single-vehicle missions from ivp/missions are recorded, about 3.2 MB in all, and all eight pass. Every compared stream matches, and DESIRED_* matches on all but a handful of iterations:

Mission Iterations DESIRED_* iterations off
s1_alpha 2331 0
s2_bravo 2301 0
s3_charlie 2319 0
s4_delta (depth) 2287 0
s5_echo (BearingLine, spawning) 2289 7 (max 2 in a row)
s7_golf 2312 0
s10_juliet (GoToDepth) 2278 2
x1_legs 2310 n/a

Two of these don't run as a mission would:

  • s7_golf parks soon after deploying, because no behavior decides course or speed (MissingDecVars).
  • x1_legs is MALCONFIG: its .bhv uses BHV_LegRun parameters this MOOS-IvP version rejects.

pHelmIvP does exactly the same in both, so they now test those paths. x1_legs could move to another mission later.

The replay found three real fidelity bugs, now fixed in the helm library

  1. Inputs from before startup were applied after "initialize". pHelmIvP posts its initialize values before it registers for anything, so a value posted before the helm started (e.g. DEPLOY=true) is replaced by the helm's own initialize value.
  2. The helm's own initialize posts weren't fed back. pHelmIvP doesn't store them again, but it still acts on them: s5_echo's initialize MOOS_MANUAL_OVERRIDE = false starts it in DRIVE.
  3. Startup posts came back one iteration late. pHelmIvP's arrive before its first iteration.

These change src/lib/helm/helm.cpp, which #4 adds. I can move them into #4 if you'd rather.

Recording (record.sh, missions.txt, "Record missions" workflow)

record.sh builds all of MOOS-IvP from the submodule into build/missions (build-moos.sh --minrobot, build-ivp.sh --nogui). It runs each mission with time warp 10 and keeps each vehicle's trimmed log (xz), .moos and .bhv. Getting reliable logs out of MOOS took a few adjustments to the mission files:

  • Pokes from inside the community. uPokeDB's posts reached pHelmIvP 2–3 s after their timestamps under time warp. The pokes now come from a uTimerScript added to the mission, which waits until pHelmIvP is up.
  • Two runs per mission. pLogger's wildcard logging misses a variable's first posts. A first run finds pHelmIvP's variables, and the second logs each of them explicitly with @ 0.
  • pLogger launches right after MOOSDB, so the log includes pHelmIvP's startup, and the missions' WildCardOmitPatterns are dropped so that BHV_STATUS is logged too.
  • A mission that hangs on exit gets killed. pAntler runs in its own process group, which is killed if it hasn't exited 20 s after SIGINT; s7_golf's left MOOSDB running.

The "Record missions" workflow (workflow_dispatch, with an optional list of missions) runs record.sh on GitHub and uploads the fixtures as an artifact to commit. The recordings should be redone whenever external/moos-ivp moves.

Replay (replay.cpp, one ctest per fixture)

  • Inputs: every logged post to a variable pHelmIvP registered, except pHelmIvP's own, goes in as a Variable once it is 5 ms old at an iteration. 5 ms matched best across the missions. pHelmIvP's own posts come back through the helm's own echo.

  • Configuration: HelmConfig comes from the .moos file's pHelmIvP block, read in reverse as MOOS's mission reader hands it over. That order sets the order of the domain variables.

  • Compared as sequences of changes, within ±1 iteration:

    • IVPHELM_STATE and IVPHELM_ALLSTOP;
    • mode variables, and initialize posts;
    • every behavior's posts, including BHV_* events and VIEW_*;
    • life events, and active behaviors.

    Numbers inside values match within 0.1% (at least 0.01). A stream with 20 or more changes may differ on up to 5% of them, because pHelmIvP sometimes gets a nav update slightly before or after its timestamp, which shows up in distances, ETAs and track points.

  • DESIRED_* is compared iteration by iteration: within one domain step on at least 99% of iterations, and never off for more than 3 in a row.

  • On failure it prints the first mismatches in each stream. replay <fixture> "<behavior> <VAR>" prints one stream in full.

Also

init.sh installs xz-utils, which the Ubuntu base images lack.

Test plan

  • ctest: helm_libs, helm and the eight replay_* tests pass (about 12 s).
  • All eight missions recorded with record.sh in this sandbox (MOOSGeodesy without PROJ, since PROJ couldn't be downloaded here).
  • CI on all four jobs.
  • The "Record missions" workflow on GitHub (manual trigger; it runs only once this is on main, or by selecting this branch).

Generated by Claude Code

…rary

- src/test/missions/record.sh builds all of MOOS-IvP from the submodule,
  runs each mission in missions.txt headless with time warp, and keeps the
  vehicle's helm inputs and outputs as a fixture. A uTimerScript added to
  the mission makes the pokes, since uPokeDB's posts reach pHelmIvP
  seconds after their timestamps under warp. A first run finds pHelmIvP's
  variables and the second logs each of them explicitly, since pLogger's
  wildcard logging misses a variable's first posts.
- replay feeds the recorded inputs to ivpico::helm::Helm at pHelmIvP's
  iteration times and compares state, allstop, modes, initialize,
  behavior postings, life events, active behaviors and DESIRED_*.
- The "Record missions" workflow runs record.sh and uploads the fixtures.
- init.sh installs xz-utils for the compressed fixtures.

First fixture: s1_alpha, which matches on every stream, and on
DESIRED_HEADING for 1463 of 1464 iterations (the other is 2 degrees off).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VLcYK4e29B5FiPbDDJy5fp
pHelmIvP posts its initialize variables before it registers for
anything, so a value posted before the helm started (e.g. DEPLOY=true)
is replaced by the helm's own initialize value. The helm library applied
queued inputs after initialize instead.

record.sh now launches pLogger right after MOOSDB, so the log has
pHelmIvP's startup, starts the pokes once pHelmIvP is up, finds the log
directories wherever the mission's pLogger writes them, and warns if a
log misses pHelmIvP's startup. replay names fixture directories given
with a trailing slash.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VLcYK4e29B5FiPbDDJy5fp
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VLcYK4e29B5FiPbDDJy5fp
pHelmIvP's initialize posts come back from the MOOSDB like its other
posts. It doesn't store them again, but it still acts on them, so
"initialize MOOS_MANUAL_OVERRIDE = false" (s5_echo) starts the helm in
DRIVE. They, and the other startup posts, arrive before pHelmIvP's first
iteration, so the helm library now feeds them back on its first one.

replay:
- mail latency 5 ms (best across the recorded missions);
- a different value on the same iteration counts as one mismatch, not
  two;
- streams with 20 or more changes may differ on up to 5% of them, since
  continuous values (distances, ETAs, track points) differ whenever
  pHelmIvP's navigation arrived slightly off its timestamp (DESIRED_*
  stays at 1%);
- an optional second argument prints one stream in full.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VLcYK4e29B5FiPbDDJy5fp
- Drop the missions' WildCardOmitPattern lines, so BHV_STATUS and the
  like are logged and compared.
- Run pAntler in its own process group, and kill the group if it hasn't
  exited 20 s after SIGINT (s7_golf's left MOOSDB and pLogger running).
- The check for pHelmIvP's startup no longer trips on SIGPIPE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VLcYK4e29B5FiPbDDJy5fp
setsid may fork, leaving $! pointing at a process that already exited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VLcYK4e29B5FiPbDDJy5fp
- s2_bravo, s3_charlie, s4_delta, s5_echo, s7_golf, s10_juliet and x1_legs
  join s1_alpha. Every stream matches in all eight, and DESIRED_* matches
  on all but 0-7 iterations of each.
- s7_golf parks soon after deploying (MissingDecVars), and x1_legs is
  MALCONFIG (its .bhv uses BHV_LegRun parameters this MOOS-IvP rejects),
  in pHelmIvP as in IvPico.
- replay reads pHelmIvP's configuration in reverse, as MOOS's mission
  reader hands it over, which sets the order of the domain variables
  (e.g. in "MissingDecVars:speed,course").
- record.sh copies the mission's .bhv files when pHelmIvP never started
  far enough to ask pLogger for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VLcYK4e29B5FiPbDDJy5fp
@tsaubergine
tsaubergine added this pull request to stack #8 November 3, 2026 23:30
@tsaubergine
tsaubergine merged commit be97f06 into main Nov 4, 2026
4 checks passed
@tsaubergine
tsaubergine deleted the claude/admiring-clarke-s2ay8b-mission-replay branch November 4, 2026 04:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants