A segment inside a keyframe drought is cut by audio and starts mid-GOP

[superuser404notfound]

Split out of AE#408, where it was found while measuring the fix at the serving layer.

What happens

Video packets are routed into segments by the keyframe-gated cutter (VODSegmentCutter.index, #92): a segment opens at the IRAP that reaches its plan boundary, so a boundary no IRAP reaches is stepped over. Audio packets are routed by segmentIndex(forSourcePts:), which is a plain boundary lookup with no such gate.

Where every boundary has an IRAP the two agree and nothing shows. Where they do not, the audio still crosses the boundary, ensureMuxer(forSegmentIndex:) cuts there and opens the folded index, and the video packets that follow land in a segment whose first video sample is not a sync sample.

Measured on a fixture built with Scripts/mkv-cue-fixture.py (cue points at non-sync positions, a deliberate 12 s keyframe drought at 43 s), fetched from the loopback server and read back as init.mp4 + segN:

segment	advertised	first video packet	sync samples inside
seg10	40.0-44.0	38.417000,K__	38.417
seg11	44.0-48.0	44.000000,___	none
seg12	48.0-52.0	48.042000,___	none
seg13	52.0-56.0	52.000000,___	55.000

The cuts sit exactly on the plan boundaries, which is the audio crossing them; the #65 ledger lines for the same run show the video cutter folding 10 straight to 13.

What it costs today

Less than it looks, on this client:

• A consumer that arrives sequentially is already decoding, so a mid-GOP segment costs it nothing.
• A cold arrival on a folded index (a seek) takes the out-of-range path, which restarts the producer there. Since AE#408 that restart opens on a sync sample, so what it gets back is decodable and covers its advertised start.
• A cold arrival on an index that is already cached from a sequential pass gets the mid-GOP segment, and AVPlayer resolves it by fetching the PREVIOUS segment first: GET /seg10.mp4 immediately after seek#2 programmatic began target=46.00, then landed rendered=46.00 target=46.00.

So the visible cost is one extra segment fetch on that last path. Not free: on the AE#408 reporting asset the segments run about 17 MB.

The playlist does not carry EXT-X-INDEPENDENT-SEGMENTS, so nothing in the manifest claims otherwise. Measured on macOS only; a tvOS reading is worth having before deciding this is benign, since the backward walk is the client's choice, not a guarantee.

Direction

The producer knows, at adopt time, whether the first video sample it routed into a segment was a sync sample. Recording that with the cache entry would let a COLD arrival treat a non-decodable cached segment the way the #358 fold path already treats a missing one, and re-produce it with a restart at that index. Sequential arrivals keep what they have, since it costs them nothing.

Not proposing to let the video fold win over audio: that would leave the folded indices with no media at all, which turns a benign extra fetch into a hole.

[superuser404notfound]

Measured, and it corrects the cost model above. Fixed in 87eac750 (merged, not released yet).

The extra fetch is not a repair

aetherctl play --picture-probe on a fixture built with Scripts/timecode-fixture.sh + Scripts/mkv-cue-fixture.py (12 s keyframe drought at 43 s, cues at every integer second): AVPlayer fetches from a fixed span below a cold seek target, measured at target - 8 s in one run shape and target - 6 s in another. It does that on the CONTROL fixture too, where every segment opens on a real sync sample. So the GET /seg10.mp4 this issue read as AVPlayer resolving the mid-GOP segment is what it does on every cold seek, and the defect does not cost one extra fetch.

What it costs is the landing

Both arms cut from the same 120 s source with the same drought, started at 0 so the whole session runs on a zero axis, played to 88 s so the drought was cached from a sequential pass, then seeked back. no restart in all six runs, so these are the cached segments being served.

target	cues lie (case)	cues ARE sync samples (control)
46.0	46.00	46.00
50.0	55.00	50.00
54.0	54.96	54.00

A seek to 50.0 s plays from 55.0 s. capErr stays small in both arms, so the clock is not lying, it is describing the wrong position correctly.

The control has the same drought. What it does not have is the cut inside it: its plan boundary sits on the real sync sample, so the drought is ONE segment beginning at its covering IRAP and one fetch serves any target inside it. In the case, audio cut the drought into seg11/seg12/seg13 and stranded the covering IRAP at 43.0 back in seg10, out of reach for targets late in the drought. Keeping that IRAP at the head of the segment that spans the drought is exactly what the keyframe-gated fold buys, and the audio cut is what spends it.

So the trigger is not "the drought is wider than a segment", it is "the drought is wider than AVPlayer's reach back".

What shipped

The producer records, per adopted segment, where its first random-access point sits, as an offset from the segment's ADVERTISED start so the claim carries no axis with it. A cold seek asks that of the segment it lands in and of the ones within reach below it, and re-cuts from the covering point only when nothing down there can open a run at the target. After: 46.00 / 50.00 / 54.00, capErr +0.075 to +0.117, identical to the control. The re-cut fires on 50 and 54 only, never on the control, never on 46 (which already landed).

A Bool for "is this segment independent" would not have been enough in either direction: seg10 starts mid-GOP and still serves a target of 46.0, while seg13 does carry a sync sample (at 55.0) and still cannot serve 54.0.

One thing worth recording separately

The seek target is deliberately NOT moved to compensate for the re-cut. Moving it by the gate's offset, which is what the AE#418 model predicts, landed exactly that far past the target: capErr +0.975 / +5.075 / +9.117 at offsets of 1, 5 and 9 s. Measured 3 of 3, AVPlayer puts the re-cut segment at its own tfdt inside the timeline it is already building (axisErr 0.000). Placement by playlist reproduces on the FIRST segment of a timeline (a resume reads axisErr -1.000 / -5.000 / -9.000 on the same fixture) and not on a segment placed into a running one. For the same reason a re-cut epoch records no axis offset.

Staying open for a tvOS reading of the same seeks.