The no-cut watchdog is inline in the loop it watches, so a starved origin leaves it unable to decide rather than late

[superuser404notfound]

Split out of #405, where it was the fourth of four findings and the only one I measured instead of
building. The other three shipped in 6.36.0 and are confirmed by the reporter against a shipping
client; this one is a separate round because it moves a decision off the read thread.

The defect

HLSSegmentProducer's no-cut stall watchdog is evaluated inline in the read loop it watches. It
ticks between av_read_frame calls, and av_read_frame does not return until a whole packet is
assembled. There is no interrupt_callback on the format context, so that call has no upper bound
at all. An origin too slow to complete a packet inside the watchdog window does not make the
watchdog late, it makes it unable to run.

Structurally the same defect #309 fixed on the reader side, where the precondition that had to go
was "a consumer must be blocked on it".

Measurement

Loopback trickle origin, 300 s MPEG-TS fixture: 200 KB/s, then 100 bytes once per second for 45 s
on the connection it already holds, then normal. One connection for the whole run, Range: bytes=0-, no reconnects, which is the shape of the reporter's trace (gen=1->1, reconnects=0,
every wait signalled with nothing to show for it).

Arm A, 6.36.0 as shipped:

[AVIOReader] slow read: 46874ms at offset=4194304 stallWaits=71(46874ms,71signaled) tailWaits=0 reconnects=0 backoff=0ms iters=143 gen=1->1
[HLSSegmentProducer] no-cut stall: no segment finalized for 48s (packetsRead=1473, sinceFinalize=110, rate=2.3pkt/s, source starvation); videoPtsAdvance=1.6s; exiting for host retune

Those two lines are 30 ms apart. liveSourceStarvationTimeoutSeconds is 35 s and it fired at
48 s: the 13 s of overrun are exactly the time the read was blocked.

Arm B, short read. readPersistent patched to return what is in hand rather than hold out for
requestSize (live only, totalRead > 0). The slow-read line disappeared, so the patch did its
job, and the watchdog fired at 47 s. One second. The short read moves the block up one layer, it
does not remove it. Not the lever, do not re-attempt it as one.

Fix shape

Lift the watchdog onto a timer that evaluates the existing classifier (noCutStallAction) off the
read thread, and abort the parked read when it decides .exitForRetune.

The abort does not need building. Demuxer.markClosed() already does exactly this: the AVIO read
callback returns -1 and av_read_frame returns at once, no resources freed. It is what the #79
reopen path uses to unblock a wedged read, and HLSVideoEngine's session teardown already pairs it
with the producer's own stop(), because a cancel flag alone cannot reach a thread parked in a
read. The no-cut watchdog is the one decision on that path that sits inside the loop with no such
pairing, and its exit is a teardown anyway (exitReason = .segmentStall, break, host retune), so it
has nothing to preserve across the abort.

What makes it its own round

• The Three live-path fixes from the #168 investigation (high-bitrate live HLS on tvOS) #177 slow-delivery hold re-arms the window from the read thread, and its anchor
  (noCutHoldRearmedAt, consecutiveNoCutHolds) plus the per-window packet counters are locals of
  the read loop. Moving the decision off that thread means moving that state under stateLock,
  and the read thread's re-arm then races the timer's evaluation. The hold exists so a slow but
  live source is not retuned, so getting that race wrong turns a hold into a retune.
• The timer must not fire against a producer that has not started its pump or is being stopped, and
  it stays live-only.

Explicitly out of scope

• The short read above, measured at one second.
• A reader-side throughput floor. Both of the reader's stall detectors are zero-byte detectors, not
  throughput detectors, which is what made the reporter's case invisible at that layer, but the
  proposed floor of "less than one AVIO read in 20 s" is a 102 kbit/s lower bound and would hang up
  on healthy low-bitrate live audio. Proposal withdrawn by the reporter in Live TS trace: a trickling origin holds both stall detectors, a wrap-corrected restart is not a source replay, 407 is untyped, and the stage-2 reload replays a frozen playlist #405 after the
  measurement; the watchdog above is the whole answer.

Reported and traced by @tschuegy in #405.

[superuser404notfound]

Built and measured. Released in 6.37.0 (merge 7224be92, tag d958ba8f), both consumers pinned to it.

Reproduced the case first, then paired the arms on the same origin: a 300 s MPEG-TS behind a loopback server at 200 KB/s that drops to 100 bytes once per second for 45 s on the connection it already holds, one connection, Range: bytes=0-, no reconnects.

6.36.0:

[AVIOReader] slow read: 46642ms at offset=3670016 stallWaits=57(46642ms,57signaled) reconnects=0 iters=115 gen=1->1
[HLSSegmentProducer] no-cut stall: no segment finalized for 46s (packetsRead=1386, sinceFinalize=24, rate=0.5pkt/s, source starvation); exiting for host retune

6.37.0, same origin, same run:

[HLSSegmentProducer] no-cut stall: no segment finalized for 35s (packetsRead=1493, sinceFinalize=131, rate=3.7pkt/s, source starvation); aborting the source read and exiting for host retune
[AVIOReader] slow read: 34586ms at offset=3932160 stallWaits=36(34585ms,36signaled) reconnects=0 iters=72 gen=1->2

The order of the two lines is the result. Before, the verdict could only be reached after the read returned, and the eleven seconds of overrun on a 35 s window were exactly the time it was blocked. Now the verdict is reached while the read is still parked, and the read line that follows is the abort ending it (gen=1->2, cancelled). Both arms exit reason=segmentStall into the host retune; this one gets there 11.4 s earlier, 53.2 s of pump against 64.6 s.

The window state moved into a NoCutStallWatchdog the read thread reports into and a 1 s timer evaluates. The classifier, both timeouts, the #177 hold ladder and the log vocabulary are unchanged.

Three things that came with it, since each could have been a quiet regression:

• The abort did not need building. Demuxer.markClosed() makes the AVIO read callback return -1 and av_read_frame return at once; it is the #79 reopen path's abort, and the session teardown already pairs it with the producer's stop() for the same reason.
• That abort surfaces in the loop as a read error, and .readError would send a reopenable source into a URL reopen of the origin that just starved. The latched verdict overrides it on the way out.
• A deliberately parked pump is not judged. The live headroom park sleeps the read thread for a consumer that stopped polling; counting that time would report a dead consumer as a dead source. It brackets itself now, and the window re-anchors on release.

Both things that were measured and dropped stay dropped: the short read (one second) and a reader-side throughput floor (102 kbit/s, hangs up low-bitrate audio).

Leaving this open for your re-measure against the same panel. Fair warning on what to expect: with a 10 s playhead-progress watchdog on your side, this should be invisible in Syravo, because that watchdog still fires long before either 48 or 35. If it does show up in your trace, the thing to look for is the pair above, the stall line landing before the slow-read line rather than after it.