feat: live control-loop rate readout during inference

[ZouzouWP]

The problem

There is currently no way to answer a basic question about a running policy: is it actually stepping at 30 Hz?

lerobot does notice when the control loop misses its target, but it reports it as a warning per missed iteration, buried in the inference log:

WARNING Record loop is running slower (0.4 Hz) than the target FPS (30.0 Hz).

That number is 1/dt for a single slow iteration, so it swings wildly and says nothing about the run as a whole. Meanwhile the UI shows the same thing whether the policy is holding 30 Hz or crawling at 8 — a stopwatch and a progress bar. Since a policy behaves differently when it is stepped at a fraction of the rate it was trained at, that gap matters when you are trying to work out whether a disappointing rollout is a bad checkpoint or a starved control loop.

What this adds

A live readout of the real control-loop rate, measured rather than assumed.

On the Inference page

Three figures under the timer, plus the last minute of history against the target:

 CONTROL LOOP                                    target 30 Hz
┌──────────────┬──────────────┬──────────────┐
│ 28.6         │ 27.9         │ 11.4         │
│ FPS now      │ average      │ worst second │
└──────────────┴──────────────┴──────────────┘
   ╭╮      ╭─╮                            ╌╌╌╌╌╌╌ 30 Hz
 ──╯╰──────╯ ╰────╮   ╭──────────────────
                  ╰───╯
 longest stall 452 ms · 1 742 loop ticks

figure	what it answers
FPS now	is it keeping up right now — colour-coded against the target
average	did the run as a whole hold its rate
worst second	did it ever drop, and how far
longest stall	was it one long freeze or a steady sag — the number that points at the cause

The average is also carried into the completion toast, so a finished run states the rate it actually ran at.

In demo mode

The same measurement, presented for a different audience. Demo mode (#75) is meant to be watched from across a room, so it gets a single colour-coded pill next to the running indicator rather than a panel:

  ● Robot running   ( 28 fps )

Green while the loop holds its target, amber below 85%, red below 55% — enough to catch a degrading demo at a glance, small enough to stay out of the way.

Ordering note: the pill lives in Demo.tsx, which does not exist on main yet — it arrives with #75. This PR is deliberately standalone and touches only the inference path, so the two can be reviewed and merged in either order; whichever lands second picks up the other half. Happy to fold them together if you would rather review it as one change.

How it works

The measurement point is ThreadSafeRobot.get_observation: the rollout control loop calls it exactly once per iteration, so counting those calls over a window is the loop rate. No lerobot patch, no instrumentation inside the loop body.

  rollout subprocess          lelab server              browser
 ┌──────────────────┐      ┌────────────────┐      ┌──────────────┐
 │ rollout_metrics  │      │ _pump_stdout   │      │ Inference /  │
 │  counts ticks    │─────▶│  parses the    │─────▶│ Demo         │
 │  prints 1 line/s │stdout│  line          │ HTTP │  render      │
 └──────────────────┘      └────────────────┘      └──────────────┘
      [LELAB_FPS]            /inference-status         1 Hz poll

• lelab/rollout_metrics.py — a thin wrapper around lerobot.scripts.lerobot_rollout (the module the inference subprocess now spawns). It installs the counter, then hands over. Every failure path is swallowed: a run whose meter fails to install simply reports no rate.
• lelab/rollout.py — _pump_stdout already read every output line to find the rollout-start marker, so it now also picks up the rate samples and lerobot's own Robot: … | FPS: 30 | … banner. The target is read, not assumed from RolloutConfig's default, so the display stays honest if the config changes.
• Frontend — both pages already polled /inference-status once a second. Nothing new is plumbed; they just render fields that are now in the response.

New fields on /inference-status: target_fps, fps_now, fps_avg, fps_min, fps_worst_gap_ms, fps_ticks, fps_stale.

Two details worth a reviewer's eye

Windows are timed by their own ticks, not by the reporter's clock. The obvious implementation — count ticks, divide by one second — reports a phantom slowdown for the half-filled window left behind when a run ends. A healthy 20 Hz run finishing mid-window read as 10 Hz, which would have put a scary "worst second" on every clean run. Each window is now timed from its own first to its own last tick, and a window with fewer than two ticks reports nothing at all. That silence is also what lets the server mark the sample stale (3 s) instead of showing a frozen number as live.

This is not lerobot's warning number. lerobot logs 1/dt for one slow iteration; a single 230 ms hiccup among 29 healthy iterations makes it shout 0.4 Hz while the second genuinely ran at 24 Hz. Both are true, they answer different questions. That is exactly why the longest-stall figure is reported separately — it is where a single freeze shows up, instead of being averaged away.

Testing

• 27 unit tests pass, including new coverage for the line format (the contract between the subprocess and the pump), the target-FPS parsing, and the staleness/finished-run branches of the status payload.
• The meter was driven against a simulated control loop with an injected 400 ms stall, verifying it reports the true rate, captures the stall, and goes quiet — rather than reporting a phantom slowdown — once the loop stops.
• Exercised on a real SO-101 rollout: the meter tracks 30 Hz with intermittent stalls, consistent with what the profiler reports for the same run.

frontend/dist/ is intentionally untouched — build_frontend.yml rebuilds it on main.

[nicolas-rabault]

Two blockers inline. The meter can't currently see a stall: I drove _RateMeter with a 30 Hz stream frozen for 4 seconds of a 10 second run and got avg=30.0 and worst second=30.0. A starved run reading as clean 30 Hz is the one outcome this feature exists to prevent.

And this collides with #75 over the same -m slot, see inline. Both wrappers swallow failures by design, so whichever loses just goes silent rather than erroring. Decide how you want the two to coexist before either lands.

[nicolas-rabault]

A window timed from its own first tick cannot see a stall: the frozen time falls outside span, so it never reaches avg or fps_min. Feeding this meter a 30 Hz stream frozen from t=3.0 to t=7.0 of a 10 second run gives avg=30.0 and worst second=30.0, on 150 ticks over 10 seconds of wall clock, so 15 Hz actually delivered.

Timing each window from the previous window's last tick fixes it, roughly:

span = (self._newest - prev_last) if self.ticks >= 1 else 0.0

with intervals = ticks rather than ticks - 1. Frozen time then lands in the window where the loop resumed. It also measures the trailing partial window at end of run honestly, which is the phantom slowdown you were working around, so the not started skip below can go with it and you get that second of data back.

max_gap needs the same treatment, see the note on the ticks < 2 branch.

[nicolas-rabault]

Second half of the same fix. drain() has already run by the time this continue fires, and it zeroed max_gap, so a gap this window measured is silently discarded.

Same 4 second freeze as in the note above, but resumed 20 ms before a reporter tick so this window drained holding a single tick: longest stall came back 33 ms and every figure read healthy. Whether a freeze is visible at all ends up depending on where it lands against the 1 Hz reporter, and with the current windowing this is the only figure carrying stalls at all.

Accumulate the worst gap for the run's lifetime and read it without clearing, rather than resetting it per drain.

[nicolas-rabault]

#75 replaces this same element with "lelab.rollout_frames", its own wrapper for the demo page's camera tee, and that module has the identical shape: main() installs its hooks then calls lerobot.scripts.lerobot_rollout.main. Only one name can occupy this slot.

Git will conflict here, but the conflict is the harmless part. The trap is the resolution: picking one module name leaves the other wrapper never imported, and since both swallow every failure by design, nothing raises. The losing feature just reports nothing, forever, on a run that otherwise looks fine.

Worth deciding now, whichever way you prefer: one wrapper module that installs both sets of hooks, or a chain where one wrapper imports the other's install function. Whoever lands first should probably leave the seam in place for the second.