diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b8a9dbc..0c8fbec7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,30 @@ the public-API contract. ## [Unreleased] -_Nothing yet._ +### Fixed + +- **A picture that is not a whole number of ticks long left #409's repair with nothing to stand on, + so the reporting asset still juddered from the first frame (AE#409).** The repair reads a rank out + of the bitstream and puts it back on the ladder the container wrote, and it needed that ladder to + advance by one constant. A constant frame rate does not always produce one: at a 1200000 timescale + the retest asset's pictures are `200202/5` ticks apart, so its sample table can only alternate + between 40040 and 40041, and the classifier fell closed on a ladder it read as variable frame + timing. A two-valued ladder is now read as the quantization it is: the cycle it repeats names the + fraction (a cycle counts only when it is seen through twice), and the pattern it rounds to names + the phase of the lattice it was quantized from, which is the one thing a whole-tick ladder cannot + carry and this one can. Ranks are then placed on that lattice instead of on a step, so the repair + reproduces the muxer exactly rather than a tick beside it, and the whole-tick ladder stays the + special case it always was, untouched. The phase also makes the verdict independent of where the + sample was taken, so a session that starts inside the file describes the same axis as one that + starts at byte 0. Nothing else changed: how far the ladder runs ahead of presentation is still + read from the container header (the ladder fits every alignment equally well, so it cannot answer + that), the container index is still folded by one constant so an index entry can never disagree + with the packet it points at, and a picture the lattice cannot place still falls back to the + rounded step rather than being handed on in decode order. Genuine variable frame timing, a ladder + with a dropped picture, and a wobble that never repeats are all still left exactly as the container + delivered them. Verified against a fractional twin pair (33 packets, three coded video sequences, + both writer shapes, from the head and after a seek): every repaired packet carries the healthy + twin's PTS and DTS exactly. Reported and diagnosed by @orut34iop. ## [6.43.0] - 2026-08-25 diff --git a/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift b/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift index a81f74cd..698862a2 100644 --- a/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift +++ b/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift @@ -27,8 +27,16 @@ import Libavutil /// fMP4 muxer and its output sanitizer enforce; a healthy file carries exactly the same negative head /// (the twin's first packet is `pts=0 dts=-2002`), so this is the shape the pipeline already handles. /// -/// Verified against three fixture pairs (432 packets, both edit-list shapes, 7 IDR boundaries): every -/// repaired packet matches its healthy twin's PTS and DTS exactly. +/// A picture is not always a whole number of ticks long, and then no `step` describes the ladder: the +/// sample table has to alternate between the two neighbouring tick counts, and the file's second +/// report against this issue was exactly that (`200202/5` ticks at a 1200000 timescale). Such a ladder +/// carries something a uniform one cannot, its rounding pattern, which is the phase of the lattice it +/// was quantized from. Reading that phase turns `displayIndex * step` into a place on the lattice and +/// the arithmetic above into its whole-tick special case. The one thing the ladder cannot say is how +/// far it runs ahead of presentation, so that still comes from the container header. +/// +/// Verified against four fixture pairs (498 packets, both edit-list shapes, both whole and fractional +/// cadences, 10 IDR boundaries): every repaired packet matches its healthy twin's PTS and DTS exactly. enum H264CompositionOffsetRepair { /// One sampled picture. Deliberately values only, so the decision is testable without FFmpeg. @@ -39,6 +47,59 @@ enum H264CompositionOffsetRepair { var isKeyframe: Bool } + /// A frame cadence that is constant but does not land on a whole number of ticks. #409's second + /// asset is one: at `time_base=1/1200000` its pictures are `200202/5` ticks apart, so the sample + /// table can only alternate between 40040 and 40041 in a five-picture cycle. Nothing about the + /// defect changes, only the ladder the repair has to read. + struct Cadence: Equatable, Sendable { + let numerator: Int64 + let denominator: Int64 + + init?(numerator: Int64, denominator: Int64) { + guard numerator > 0, denominator > 0 else { return nil } + let divisor = H264CompositionOffsetRepair.greatestCommonDivisor(numerator, denominator) + self.numerator = numerator / divisor + self.denominator = denominator / divisor + } + + /// Whole ticks per picture, rounded down. + var wholeTicks: Int64 { numerator / denominator } + + /// Where the picture with this ordinal sits, relative to ordinal zero. Ties round away from + /// zero, which is how FFmpeg rescales, so the lattice reproduces what a healthy muxer wrote + /// for the same source rather than something a half tick beside it. + func timestamp(at ordinal: Int64) -> Int64? { + Self.roundedQuotient(ordinal, times: numerator, dividedBy: denominator) + } + + /// The inverse, and it is exact for every point the lattice produced: a lattice value is at + /// most half a tick away from the real product, so dividing it by a cadence of two ticks or + /// more can never reach the neighbouring ordinal. Cadences below that are refused for this + /// reason, not for a lack of precision in the multiply. + func ordinal(at timestamp: Int64) -> Int64? { + Self.roundedQuotient(timestamp, times: denominator, dividedBy: numerator) + } + + /// `round(value * factor / divisor)` at full width, so a long timeline cannot overflow the + /// intermediate product and land the repair on a wrapped timestamp. + private static func roundedQuotient(_ value: Int64, times factor: Int64, dividedBy divisor: Int64) -> Int64? { + guard value != Int64.min, factor > 0, divisor > 0 else { return nil } + let negative = value < 0 + let magnitude = UInt64(negative ? -value : value) + let product = magnitude.multipliedFullWidth(by: UInt64(factor)) + guard product.high < UInt64(divisor) else { return nil } + let division = UInt64(divisor).dividingFullWidth(product) + var quotient = division.quotient + if division.remainder * 2 >= UInt64(divisor) { + let (rounded, overflow) = quotient.addingReportingOverflow(1) + guard !overflow else { return nil } + quotient = rounded + } + guard quotient <= UInt64(Int64.max) else { return nil } + return negative ? -Int64(quotient) : Int64(quotient) + } + } + /// What the rewrite needs, all of it derived once at the head of the session. struct Plan: Equatable, Sendable { /// Ticks between two consecutive pictures in decode order. @@ -52,6 +113,47 @@ enum H264CompositionOffsetRepair { /// Ticks a picture order count advances per displayed picture. 2 for frame coding, but /// measured rather than assumed. var pocStep: Int64 + /// Set only when the ladder is a quantization of a fractional cadence. `step` then carries + /// the rounded cadence and still describes the stream to within a tick, which is what the + /// arithmetic falls back to; the lattice is what reproduces the muxer exactly. + var cadence: Cadence? + /// Where lattice ordinal 0 sits on this timeline, and which lattice ordinal the first + /// sampled picture sits on. A whole-tick ladder has no phase to speak of; a quantized one + /// does, and its rounding pattern is the only place that phase is written down. It is not + /// implied by anything else: the reporting asset for this case sits on phase 2 while its + /// reorder delay is 1. + var latticeOrigin: Int64 = 0 + var ladderPhase: Int64 = 0 + /// How many ordinals the decode ladder runs ahead of presentation: 0 when the writer left the + /// ladder on the presentation axis, `videoDelay` when it kept the edit list that trims the + /// reorder head, and the values between when it trims part of one. The ladder cannot say + /// which, so this is read from the container the same way `shift` is. + var ladderOrdinalOffset: Int64 = 0 + + /// The timestamp of a presentation ordinal, nil unless this plan carries a lattice. + func presentationTimestamp(ordinal: Int64) -> Int64? { + guard let cadence else { return nil } + let (phased, phaseOverflow) = ordinal.addingReportingOverflow(ladderPhase) + let (placed, placeOverflow) = phased.addingReportingOverflow(ladderOrdinalOffset) + guard !phaseOverflow, !placeOverflow, let offset = cadence.timestamp(at: placed) else { + return nil + } + let (timestamp, overflow) = latticeOrigin.addingReportingOverflow(offset) + return overflow ? nil : timestamp + } + + /// Reads a ladder timestamp back into the presentation ordinal of the picture it belongs to. + /// nil when the timestamp is not on the lattice, which is deliberate: a ladder that drifted + /// off its own cadence must fall back to the scalar anchor instead of being placed a whole + /// picture away from where it belongs. + func presentationOrdinal(ladderTimestamp: Int64) -> Int64? { + guard let cadence else { return nil } + let (offset, offsetOverflow) = ladderTimestamp.subtractingReportingOverflow(latticeOrigin) + guard !offsetOverflow, let ordinal = cadence.ordinal(at: offset), + cadence.timestamp(at: ordinal) == offset else { return nil } + let (presentation, overflow) = ordinal.subtractingReportingOverflow(ladderPhase) + return overflow ? nil : presentation + } } enum Verdict: Equatable, Sendable { @@ -123,15 +225,35 @@ enum H264CompositionOffsetRepair { return .inconclusive("sample does not start on a picture-order origin") } - // A uniform ladder is what lets a rank be turned back into a timestamp without buffering - // packets. Variable frame timing with no composition offsets is not repairable this way, and - // is left alone rather than guessed at. - var step: Int64 = 0 + // A ladder that advances by one constant is what lets a rank be turned back into a timestamp + // without buffering packets. A picture whose length is not a whole number of ticks cannot + // produce such a ladder at all: the sample table has to alternate between the two + // neighbouring counts, and the cycle it repeats names the fraction it is quantizing (#409's + // retest asset: 200202/5 ticks at a 1200000 timescale, cycling 40041,40040,40040,40041,40040). + // Everything else, genuine variable frame timing included, is left alone rather than guessed at. + var deltas: [Int64] = [] + deltas.reserveCapacity(samples.count - 1) for index in 1.. 0 else { return .inconclusive("decode timestamps do not advance") } - if step == 0 { step = delta } - guard delta == step else { return .inconclusive("decode ladder is not uniform") } + let (delta, overflow) = samples[index].dts + .subtractingReportingOverflow(samples[index - 1].dts) + guard !overflow, delta > 0 else { + return .inconclusive("decode timestamps do not advance") + } + deltas.append(delta) + } + guard let shortestStep = deltas.min(), let longestStep = deltas.max() else { + return .inconclusive("no ladder step") + } + var cadence: Cadence? + var step = shortestStep + if shortestStep != longestStep { + guard longestStep - shortestStep == 1, + let quantized = quantizedCadence(deltas: deltas), + let roundedStep = quantized.timestamp(at: 1) else { + return .inconclusive("decode ladder is not uniform") + } + cadence = quantized + step = roundedStep } guard step > 0 else { return .inconclusive("no ladder step") } @@ -173,21 +295,118 @@ enum H264CompositionOffsetRepair { return .inconclusive("display indices do not fill the sampled window") } - let decodeLead = Int64(videoDelay) * step + // The reorder head is one decode lead long, and with a fractional cadence that lead is a + // rounded distance on the lattice rather than a multiple of a whole step. + let ladderOrigin = ladderStart == Int64.min ? (samples.first?.dts ?? Int64.min) : ladderStart + let decodeLead: Int64 + if let cadence { + guard let head = cadence.timestamp(at: -Int64(videoDelay)), head < 0 else { + return .inconclusive("cadence does not describe a reorder head") + } + decodeLead = -head + } else { + let (lead, overflow) = Int64(videoDelay).multipliedReportingOverflow(by: step) + guard !overflow else { return .inconclusive("reorder head does not fit the timeline") } + decodeLead = lead + } + guard decodeLead > 0 else { return .inconclusive("no decode lead") } + let shift = presentationShift( + streamStartTime: streamStartTime, ladderStart: ladderOrigin, decodeLead: decodeLead) + + // A fractional cadence is read back off a lattice, so the lattice has to be located first, + // and that takes two readings the ladder cannot both give. The phase and the origin come from + // the ladder itself: its rounding pattern is the phase, and only one phase of the period can + // reproduce the sampled window picture for picture. How far the ladder runs ahead of + // presentation does NOT come from the ladder, because every alignment fits it equally well; + // that is the same question `presentationShift` already answers from the container header, + // and here it has to resolve to a whole number of ordinals to be usable at all. + var latticeOrigin: Int64 = 0 + var ladderPhase: Int64 = 0 + var ladderOrdinalOffset: Int64 = 0 + if let cadence { + guard let fit = latticeFit(samples: samples, cadence: cadence) else { + return .inconclusive( + "ladder does not follow one \(cadence.numerator)/\(cadence.denominator) phase") + } + ladderPhase = fit.phase + latticeOrigin = fit.origin + let alignment = (0...Int64(videoDelay)).first { candidate in + cadence.timestamp(at: -candidate).map { -$0 } == shift + } + guard let alignment else { + return .inconclusive("reorder head is not a whole number of pictures") + } + ladderOrdinalOffset = alignment + } + return .repair( Plan( step: step, decodeLead: decodeLead, - shift: presentationShift( - streamStartTime: streamStartTime, - ladderStart: ladderStart == Int64.min ? (samples.first?.dts ?? Int64.min) : ladderStart, - decodeLead: decodeLead - ), - pocStep: pocStep + shift: shift, + pocStep: pocStep, + cadence: cadence, + latticeOrigin: latticeOrigin, + ladderPhase: ladderPhase, + ladderOrdinalOffset: ladderOrdinalOffset ) ) } + /// Which phase of the cadence the sampled ladder starts on, and where that puts lattice ordinal + /// zero. Every sampled picture has to land exactly, and the answer has to be the only phase in + /// the period that does, or the ladder is not evidence of a lattice at all. This is what lets a + /// sample taken anywhere describe the same axis as a sample taken at the head. + static func latticeFit(samples: [Sample], cadence: Cadence) -> (phase: Int64, origin: Int64)? { + guard let head = samples.first?.dts else { return nil } + var found: (phase: Int64, origin: Int64)? + for phase in 0.. Cadence? { + guard deltas.count >= 4 else { return nil } + for period in 2...(deltas.count / 2) { + guard deltas.indices.allSatisfy({ deltas[$0] == deltas[$0 % period] }) else { continue } + var ticks: Int64 = 0 + for delta in deltas.prefix(period) { + let (sum, overflow) = ticks.addingReportingOverflow(delta) + guard !overflow else { return nil } + ticks = sum + } + guard let cadence = Cadence(numerator: ticks, denominator: Int64(period)), + cadence.denominator == Int64(period), cadence.wholeTicks >= 2 else { continue } + return cadence + } + return nil + } + static func greatestCommonDivisor(_ a: Int64, _ b: Int64) -> Int64 { var x = abs(a), y = abs(b) while y != 0 { (x, y) = (y, x % y) } @@ -201,6 +420,9 @@ enum H264CompositionOffsetRepair { let plan: Plan /// Set at the first keyframe seen, and again whenever the picture order restarts. private(set) var sequenceAnchorDTS: Int64? + /// The presentation ordinal this sequence starts at, read back off the lattice. Only a + /// fractional cadence has one; without it the scalar anchor above carries the sequence. + private(set) var sequenceBaseOrdinal: Int64? /// A seek leaves the parser and the sequence anchor behind; the next keyframe re-anchors. private var awaitingReanchor = true /// Pictures emitted untouched because no anchor was available or the arithmetic did not @@ -213,6 +435,7 @@ enum H264CompositionOffsetRepair { mutating func noteSeek() { awaitingReanchor = true sequenceAnchorDTS = nil + sequenceBaseOrdinal = nil } /// nil when the picture cannot be placed; the caller then emits it untouched. @@ -232,13 +455,20 @@ enum H264CompositionOffsetRepair { // GOP would restart the display axis under a picture that has not moved. if isKeyframe, pictureOrderCount == 0 { sequenceAnchorDTS = dts + sequenceBaseOrdinal = plan.presentationOrdinal(ladderTimestamp: dts) awaitingReanchor = false } else if awaitingReanchor, isKeyframe { guard pictureOrderCount % plan.pocStep == 0 else { unrepairedPictures += 1 return nil } - sequenceAnchorDTS = dts - (pictureOrderCount / plan.pocStep) * plan.step + let landingIndex = pictureOrderCount / plan.pocStep + sequenceAnchorDTS = dts - landingIndex * plan.step + sequenceBaseOrdinal = plan.presentationOrdinal(ladderTimestamp: dts) + .flatMap { ordinal -> Int64? in + let (base, overflow) = ordinal.subtractingReportingOverflow(landingIndex) + return overflow ? nil : base + } awaitingReanchor = false } guard let anchor = sequenceAnchorDTS, !awaitingReanchor, @@ -247,7 +477,16 @@ enum H264CompositionOffsetRepair { return nil } let displayIndex = pictureOrderCount / plan.pocStep - let pts = anchor &+ plan.shift &+ displayIndex &* plan.step + // On a lattice the picture is placed by its ordinal, which reproduces the muxer exactly + // even where the fraction rounds the other way. Without one, or when the sequence could + // not be read back onto the lattice, the rounded step still describes the stream to + // within a tick, and a tick beside the twin beats a picture left in decode order. + let pts = sequenceBaseOrdinal + .flatMap { base -> Int64? in + let (ordinal, overflow) = base.addingReportingOverflow(displayIndex) + return overflow ? nil : plan.presentationTimestamp(ordinal: ordinal) + } + ?? (anchor &+ plan.shift &+ displayIndex &* plan.step) let newDTS = dts &+ plan.shift &- plan.decodeLead // The muxer invariant. A picture that lands before its own decode time means the // arithmetic no longer describes this stream, and passing it through unchanged is diff --git a/Tests/AetherEngineTests/H264CompositionOffsetRepairTests.swift b/Tests/AetherEngineTests/H264CompositionOffsetRepairTests.swift index 0abd36a3..757ed943 100644 --- a/Tests/AetherEngineTests/H264CompositionOffsetRepairTests.swift +++ b/Tests/AetherEngineTests/H264CompositionOffsetRepairTests.swift @@ -343,4 +343,280 @@ struct H264CompositionOffsetRepairTests { BwGft0BtBmAAAAAIQZu8NEB9BmAAAAAJQZ/aRRUodQZgAAAABwGf+UBtBmAAAAAHAZ/7QG0GYAAAAAhBm+A0QH0GYAAAAAlBnh5F FSh1BmAAAAAHAZ49QG0GYAAAAAcBnj9AbQZgAAAACEGaITRAfQZg """ + + // MARK: - a cadence that does not land on whole ticks + + /// #409's retest asset. It is constant rate, but a picture is not a whole number of ticks long, + /// so the sample table alternates between the two neighbouring counts: at `time_base=1/1200000` + /// the pictures are `200202/5` ticks apart and the ladder repeats `40041,40040,40040,40041,40040`. + /// Nothing about the defect changed, only the ladder the repair has to read, and a classifier + /// that demanded one identical step left the file exactly as broken as it found it. + private func quantizedLadderSamples() -> [H264CompositionOffsetRepair.Sample] { + let ladder: [Int64] = [ + -80081, -40040, 0, 40040, 80081, 120121, 160162, 200202, 240242, 280283, 320323, 360364, + ] + let pocs: [Int64] = [0, 8, 4, 2, 6, 16, 12, 10, 14, 24, 20, 18] + return zip(ladder, pocs).enumerated().map { index, pair in + H264CompositionOffsetRepair.Sample( + dts: pair.0, pts: pair.0, pictureOrderCount: pair.1, isKeyframe: index == 0) + } + } + + @Test("a ladder quantized from a fractional cadence is repaired, not called nonuniform") + func quantizedLadderIsRepaired() { + let verdict = verdict( + quantizedLadderSamples(), videoDelay: 2, streamStartTime: 0, ladderStart: -80081) + guard case .repair(let plan) = verdict else { + Issue.record("expected a repair, got \(verdict)") + return + } + #expect(plan.cadence == H264CompositionOffsetRepair.Cadence(numerator: 200202, denominator: 5)) + #expect(plan.decodeLead == 80081) + #expect(plan.shift == 80081) + #expect(plan.pocStep == 2) + // The ladder starts on phase 3 of the five-picture period, and the container's retained edit + // list puts presentation one whole reorder head above it. + #expect(plan.ladderPhase == 3) + #expect(plan.ladderOrdinalOffset == 2) + // Which lands the axis where the healthy twin writes it: the first picture at zero. + #expect(plan.presentationTimestamp(ordinal: 0) == 0) + #expect(plan.presentationTimestamp(ordinal: 1) == 40040) + } + + @Test("the fractional plan reproduces the presentation lattice, including across a sequence") + func quantizedPlanFollowsTheLattice() { + guard case .repair(let plan) = verdict( + quantizedLadderSamples(), videoDelay: 2, streamStartTime: 0, ladderStart: -80081) else { + Issue.record("expected a repair") + return + } + var rewriter = H264CompositionOffsetRepair.Rewriter(plan: plan) + // The head, against the healthy twin's first pictures. + #expect(rewriter.rewrite(dts: -80081, pictureOrderCount: 0, isKeyframe: true).map(\.pts) == 0) + #expect(rewriter.rewrite(dts: -40040, pictureOrderCount: 8, isKeyframe: false).map(\.pts) == 160162) + #expect(rewriter.rewrite(dts: 0, pictureOrderCount: 4, isKeyframe: false).map(\.pts) == 80081) + // A second coded video sequence starts on a ladder point whose distance from the first is + // not a whole multiple of any integer step. Read back from the lattice it still lands on + // the twin's timestamp; counted in rounded steps it would be a tick out. + rewriter.noteSeek() + #expect(rewriter.rewrite(dts: 560566, pictureOrderCount: 0, isKeyframe: true).map(\.pts) == 640646) + #expect(rewriter.rewrite(dts: 600606, pictureOrderCount: 8, isKeyframe: false).map(\.pts) == 800808) + } + + @Test("a sequence the lattice cannot place is still repaired, by the rounded step") + func offLatticeSequenceFallsBackToTheStep() { + guard case .repair(let plan) = verdict( + quantizedLadderSamples(), videoDelay: 2, streamStartTime: 0, ladderStart: -80081) else { + Issue.record("expected a repair") + return + } + var rewriter = H264CompositionOffsetRepair.Rewriter(plan: plan) + rewriter.noteSeek() + // A sequence opening one tick beside the lattice: the ladder has stopped describing itself. + // The rounded step still describes it to within a tick, and it anchors on the container's own + // timestamp, so it cannot drift. Handing the picture on in decode order would be the defect. + let head = rewriter.rewrite(dts: 560567, pictureOrderCount: 0, isKeyframe: true) + #expect(head?.pts == 640648) + #expect(head?.dts == 560567) + #expect(rewriter.rewrite(dts: 600607, pictureOrderCount: 8, isKeyframe: false)?.pts == 800808) + #expect(rewriter.unrepairedPictures == 0) + } + + @Test("a sample taken away from the head describes the same axis as one taken at it") + func quantizedLadderClassifiesFromAnywhere() { + // The same fixture, sampled from its second IDR instead of its first. The ladder starts on a + // different phase of the five-picture cycle there, and a repair that assumed the head would + // place every picture a tick beside the twin, or refuse the file outright. + let ladder: [Int64] = [ + 560566, 600606, 640646, 680687, 720727, 760768, 800808, 840848, 880889, 920929, 960970, + 1001010, + ] + let pocs: [Int64] = [0, 8, 4, 2, 6, 16, 12, 10, 14, 24, 20, 18] + let samples = zip(ladder, pocs).enumerated().map { index, pair in + H264CompositionOffsetRepair.Sample( + dts: pair.0, pts: pair.0, pictureOrderCount: pair.1, isKeyframe: index == 0) + } + guard case .repair(let plan) = verdict( + samples, videoDelay: 2, streamStartTime: 0, ladderStart: -80081) else { + Issue.record("a sample away from the head must still classify") + return + } + #expect(plan.ladderPhase == 4) + var rewriter = H264CompositionOffsetRepair.Rewriter(plan: plan) + // The healthy twin's timestamps for those same three pictures. + #expect(rewriter.rewrite(dts: 560566, pictureOrderCount: 0, isKeyframe: true).map(\.pts) == 640646) + #expect(rewriter.rewrite(dts: 600606, pictureOrderCount: 8, isKeyframe: false).map(\.pts) == 800808) + #expect(rewriter.rewrite(dts: 640646, pictureOrderCount: 4, isKeyframe: false).map(\.pts) == 720727) + } + + @Test("a fractional ladder left on the presentation axis needs no shift either") + func quantizedLadderOnPresentationAxis() { + // The other writer shape, and the container is the only thing that says which one it is: the + // same ladder lifted to non-negative timestamps, reporting a start time on its own head. + let ladder: [Int64] = [ + 0, 40041, 80081, 120121, 160162, 200202, 240243, 280283, 320323, 360364, 400404, 440445, + ] + let pocs: [Int64] = [0, 8, 4, 2, 6, 16, 12, 10, 14, 24, 20, 18] + let samples = zip(ladder, pocs).enumerated().map { index, pair in + H264CompositionOffsetRepair.Sample( + dts: pair.0, pts: pair.0, pictureOrderCount: pair.1, isKeyframe: index == 0) + } + guard case .repair(let plan) = verdict( + samples, videoDelay: 2, streamStartTime: 0, ladderStart: 0) else { + Issue.record("expected a repair") + return + } + #expect(plan.shift == 0) + #expect(plan.ladderOrdinalOffset == 0) + #expect(plan.decodeLead == 80081) + var rewriter = H264CompositionOffsetRepair.Rewriter(plan: plan) + let head = rewriter.rewrite(dts: 0, pictureOrderCount: 0, isKeyframe: true) + #expect(head?.pts == 0) + #expect(head?.dts == -80081) + } + + @Test("a ladder that wobbles by a tick without repeating is left alone") + func nonRepeatingWobbleIsNotACadence() { + let ladder: [Int64] = [0, 40040, 80081, 120121, 160161, 200202, 240242, 280282, 320323, 360364, 400404, 440445] + let pocs: [Int64] = [0, 8, 4, 2, 6, 16, 12, 10, 14, 24, 20, 18] + let samples = zip(ladder, pocs).enumerated().map { index, pair in + H264CompositionOffsetRepair.Sample( + dts: pair.0, pts: pair.0, pictureOrderCount: pair.1, isKeyframe: index == 0) + } + guard case .inconclusive = verdict(samples, videoDelay: 2, streamStartTime: 0, ladderStart: 0) else { + Issue.record("a ladder with no repeating cycle must not be repaired") + return + } + } + + @Test("a ladder with a dropped picture is left alone") + func droppedPictureIsNotACadence() { + let ladder: [Int64] = [0, 40040, 80081, 120121, 160162, 240242, 280283, 320323, 360364, 400404, 440444, 480485] + let pocs: [Int64] = [0, 8, 4, 2, 6, 16, 12, 10, 14, 24, 20, 18] + let samples = zip(ladder, pocs).enumerated().map { index, pair in + H264CompositionOffsetRepair.Sample( + dts: pair.0, pts: pair.0, pictureOrderCount: pair.1, isKeyframe: index == 0) + } + guard case .inconclusive = verdict(samples, videoDelay: 2, streamStartTime: 0, ladderStart: 0) else { + Issue.record("a two-tick gap is not a quantization") + return + } + } + + @Test("the fractional twin carries the healthy twin's timestamps, packet for packet") + func repairedRationalTwinMatchesHealthyTwin() throws { + let healthy = try Self.videoTimestamps(base64: Self.healthyRationalFixtureBase64) + let repaired = try Self.videoTimestamps(base64: Self.missingRationalFixtureBase64) + #expect(healthy.count == 33) + #expect(repaired.count == healthy.count) + #expect(repaired == healthy) + } + + @Test("the fractional healthy twin is delivered exactly as the container wrote it") + func rationalHealthyTwinIsUntouched() throws { + let healthy = try Self.videoTimestamps(base64: Self.healthyRationalFixtureBase64) + #expect(healthy.first == Timestamps(pts: 0, dts: -80081)) + #expect(healthy.contains { $0.pts != $0.dts }) + // The point of this pair: the decode ladder does not advance by one constant. + let steps = Set(zip(healthy, healthy.dropFirst()).map { $1.dts - $0.dts }) + #expect(steps == [40040, 40041]) + } + + @Test("the repaired fractional stream presents every picture exactly once, in order") + func repairedRationalStreamIsABijection() throws { + let repaired = try Self.videoTimestamps(base64: Self.missingRationalFixtureBase64) + let presentation = repaired.map(\.pts).sorted() + #expect(Set(presentation).count == repaired.count) + #expect(Set(zip(presentation, presentation.dropFirst()).map { $1 - $0 }) == [40040, 40041]) + #expect(repaired.allSatisfy { $0.pts >= $0.dts }) + #expect(zip(repaired, repaired.dropFirst()).allSatisfy { $1.dts > $0.dts }) + } + + /// 96x64 H.264, 33 frames at 1000000/33367 fps in a 1200000 timescale, so a picture is 200202/5 + /// ticks long and the sample table has to quantize it. Three B pictures per group, a keyframe + /// every 16, and the composition offsets stream-copied away in the second file, the same way + /// @orut34iop's original pair was made. + /// + /// ffmpeg -f lavfi -i 'color=c=gray:s=96x64:rate=1000000/33367' -frames:v 33 \ + /// -c:v libx264 -preset ultrafast -pix_fmt yuv420p -bf 3 -b_strategy 0 -g 16 \ + /// -sc_threshold 0 -crf 40 -video_track_timescale 1200000 -r 1000000/33367 \ + /// -movflags +faststart healthy.mp4 + /// ffmpeg -i healthy.mp4 -map 0:v:0 -c:v copy -bsf:v 'setts=pts=DTS' \ + /// -movflags +faststart missing.mp4 + + private static let healthyRationalFixtureBase64 = """ + AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAWObW9vdgAAAGxtdmhkAAAAAAAAAAAAAAAAAAAD6AAABE4AAQ + AAAQAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA + AAAAAAAAAgAABLh0cmFrAAAAXHRraGQAAAADAAAAAAAAAAAAAAABAAAAAAAABE4AAAAAAAAAAAAAAAAAAAAAAAEAAAAAAA + AAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAGAAAABAAAAAAAAkZWR0cwAAABxlbHN0AAAAAAAAAAEAAAROAAE40QAB + AAAAAAQwbWRpYQAAACBtZGhkAAAAAAAAAAAAAAAAABJPgAAUKXVVxAAAAAAALWhkbHIAAAAAAAAAAHZpZGUAAAAAAAAAAA + AAAABWaWRlb0hhbmRsZXIAAAAD221pbmYAAAAUdm1oZAAAAAEAAAAAAAAAAAAAACRkaW5mAAAAHGRyZWYAAAAAAAAAAQAA + AAx1cmwgAAAAAQAAA5tzdGJsAAAAt3N0c2QAAAAAAAAAAQAAAKdhdmMxAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAGAAQA + BIAAAASAAAAAAAAAABFUxhdmM2Mi4yOC4xMDEgbGlieDI2NAAAAAAAAAAAAAAAGP//AAAALWF2Y0MBTUAK/+EAFWdNQArs + oxNgIgABBK4APQkAHiRLLAEABWjOA5yAAAAAEHBhc3AAAAABAAAAAQAAABRidHJ0AAAAAAAAH48AAAAAAAAA4HN0dHMAAA + AAAAAAGgAAAAEAAJxpAAAAAgAAnGgAAAABAACcaQAAAAEAAJxoAAAAAQAAnGkAAAACAACcaAAAAAEAAJxpAAAAAQAAnGgA + AAABAACcaQAAAAIAAJxoAAAAAQAAnGkAAAABAACcaAAAAAEAAJxpAAAAAgAAnGgAAAABAACcaQAAAAEAAJxoAAAAAQAAnG + kAAAACAACcaAAAAAEAAJxpAAAAAQAAnGgAAAABAACcaQAAAAIAAJxoAAAAAQAAnGkAAAABAACcaAAAAAEAAJxpAAAAAgAA + nGgAAAAcc3RzcwAAAAAAAAADAAAAAQAAABEAAAAhAAABGGN0dHMAAAAAAAAAIQAAAAEAATjRAAAAAQADDgoAAAABAAE40Q + AAAAEAAAAAAAAAAQAAnGgAAAABAAMOCgAAAAEAATjQAAAAAQAAAAAAAAABAACcaQAAAAEAAw4KAAAAAQABONEAAAABAAAA + AAAAAAEAAJxoAAAAAQACcaIAAAABAACcaAAAAAEAAJxpAAAAAQABONAAAAABAAMOCgAAAAEAATjRAAAAAQAAAAAAAAABAA + CcaQAAAAEAAw4KAAAAAQABONEAAAABAAAAAAAAAAEAAJxoAAAAAQADDgoAAAABAAE40AAAAAEAAAAAAAAAAQAAnGkAAAAB + AAJxoQAAAAEAAJxpAAAAAQAAnGgAAAABAAE40QAAABxzdHNjAAAAAAAAAAEAAAABAAAAIQAAAAEAAACYc3RzegAAAAAAAA + AAAAAAIQAAAr4AAAALAAAACwAAAAsAAAALAAAADAAAAA0AAAALAAAACwAAAAwAAAANAAAACwAAAAsAAAAMAAAADQAAAAsA + AAAfAAAACwAAAAsAAAALAAAACwAAAAwAAAANAAAACwAAAAsAAAAMAAAADQAAAAsAAAALAAAADAAAAA0AAAALAAAAHwAAAB + RzdGNvAAAAAAAAAAEAAAW+AAAAYnVkdGEAAABabWV0YQAAAAAAAAAhaGRscgAAAAAAAAAAbWRpcmFwcGwAAAAAAAAAAAAA + AAAtaWxzdAAAACWpdG9vAAAAHWRhdGEAAAABAAAAAExhdmY2Mi4xMi4xMDEAAAAIZnJlZQAABGBtZGF0AAACnQYF//+Z3E + XpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE2NSByMzIyMiBiMzU2MDVhIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAt + IENvcHlsZWZ0IDIwMDMtMjAyNSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYm + FjPTAgcmVmPTEgZGVibG9jaz0wOjA6MCBhbmFseXNlPTA6MCBtZT1kaWEgc3VibWU9MCBwc3k9MSBwc3lfcmQ9MS4wMDow + LjAwIG1peGVkX3JlZj0wIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MCA4eDhkY3Q9MCBjcW09MCBkZWFkem + 9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD0wIHRocmVhZHM9MiBsb29rYWhlYWRfdGhyZWFkcz0x + IHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYW + luZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0wIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9 + MCBvcGVuX2dvcD0wIHdlaWdodHA9MCBrZXlpbnQ9MTYga2V5aW50X21pbj0xIHNjZW5lY3V0PTAgaW50cmFfcmVmcmVzaD + 0wIHJjPWNyZiBtYnRyZWU9MCBjcmY9NDAuMCBxY29tcD0wLjYwIHFwbWluPTAgcXBtYXg9NjkgcXBzdGVwPTQgaXBfcmF0 + aW89MS40MCBwYl9yYXRpbz0xLjMwIGFxPTAAgAAAABlliIQA6JuTk5OTk6666666666666666668AAAAB0GaJADqDMAAAA + AHQZ5CQDaDMAAAAAcBnmFAXQZgAAAABwGeY0BdBmAAAAAIQZpoNEB1BmAAAAAJQZ6GRREobQZgAAAABwGepUBlBmAAAAAH + AZ6nQGUGYAAAAAhBmqw0QH0GYAAAAAlBnspFFShtBmAAAAAHAZ7pQGUGYAAAAAcBnutAZQZgAAAACEGa7zRAfQZgAAAACU + GfDUUVKHUGYAAAAAcBny5AZQZgAAAAG2WIggAPomKMnJycnJ1111111111111111114AAAAAdBmiQA6gzAAAAAB0GeQkA2 + gzAAAAAHAZ5hQGUGYAAAAAcBnmNAZQZgAAAACEGaaDRAdQZgAAAACUGehkURKG0GYAAAAAcBnqVAZQZgAAAABwGep0BlBm + AAAAAIQZqsNEB9BmAAAAAJQZ7KRRUobQZgAAAABwGe6UBlBmAAAAAHAZ7rQGUGYAAAAAhBmu80QH0GYAAAAAlBnw1FFSh1 + BmAAAAAHAZ8uQG0GYAAAABtliIQAEKJijJycnJyddddddddddddddddddeA= + """ + + private static let missingRationalFixtureBase64 = """ + AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAR2bW9vdgAAAGxtdmhkAAAAAAAAAAAAAAAAAAAD6AAABAsAAQ + AAAQAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA + AAAAAAAAAgAAA6B0cmFrAAAAXHRraGQAAAADAAAAAAAAAAAAAAABAAAAAAAABAsAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAA + AAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAGAAAABAAAAAAAAkZWR0cwAAABxlbHN0AAAAAAAAAAEAAAQLAAE40QAB + AAAAAAMYbWRpYQAAACBtZGhkAAAAAAAAAAAAAAAAABJPgAAUKXVVxAAAAAAALWhkbHIAAAAAAAAAAHZpZGUAAAAAAAAAAA + AAAABWaWRlb0hhbmRsZXIAAAACw21pbmYAAAAUdm1oZAAAAAEAAAAAAAAAAAAAACRkaW5mAAAAHGRyZWYAAAAAAAAAAQAA + AAx1cmwgAAAAAQAAAoNzdGJsAAAAt3N0c2QAAAAAAAAAAQAAAKdhdmMxAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAGAAQA + BIAAAASAAAAAAAAAABFUxhdmM2Mi4yOC4xMDEgbGlieDI2NAAAAAAAAAAAAAAAGP//AAAALWF2Y0MBTUAK/+EAFWdNQArs + oxNgIgABBK4APQkAHiRLLAEABWjOA5yAAAAAEHBhc3AAAAABAAAAAQAAABRidHJ0AAAAAAAAH48AAB+PAAAA4HN0dHMAAA + AAAAAAGgAAAAEAAJxpAAAAAgAAnGgAAAABAACcaQAAAAEAAJxoAAAAAQAAnGkAAAACAACcaAAAAAEAAJxpAAAAAQAAnGgA + AAABAACcaQAAAAIAAJxoAAAAAQAAnGkAAAABAACcaAAAAAEAAJxpAAAAAgAAnGgAAAABAACcaQAAAAEAAJxoAAAAAQAAnG + kAAAACAACcaAAAAAEAAJxpAAAAAQAAnGgAAAABAACcaQAAAAIAAJxoAAAAAQAAnGkAAAABAACcaAAAAAEAAJxpAAAAAgAA + nGgAAAAcc3RzcwAAAAAAAAADAAAAAQAAABEAAAAhAAAAHHN0c2MAAAAAAAAAAQAAAAEAAAAhAAAAAQAAAJhzdHN6AAAAAA + AAAAAAAAAhAAACvgAAAAsAAAALAAAACwAAAAsAAAAMAAAADQAAAAsAAAALAAAADAAAAA0AAAALAAAACwAAAAwAAAANAAAA + CwAAAB8AAAALAAAACwAAAAsAAAALAAAADAAAAA0AAAALAAAACwAAAAwAAAANAAAACwAAAAsAAAAMAAAADQAAAAsAAAAfAA + AAFHN0Y28AAAAAAAAAAQAABKYAAABidWR0YQAAAFptZXRhAAAAAAAAACFoZGxyAAAAAAAAAABtZGlyYXBwbAAAAAAAAAAA + AAAAAC1pbHN0AAAAJal0b28AAAAdZGF0YQAAAAEAAAAATGF2ZjYyLjEyLjEwMQAAAAhmcmVlAAAEYG1kYXQAAAKdBgX//5 + ncRem95tlIt5Ys2CDZI+7veDI2NCAtIGNvcmUgMTY1IHIzMjIyIGIzNTYwNWEgLSBILjI2NC9NUEVHLTQgQVZDIGNvZGVj + IC0gQ29weWxlZnQgMjAwMy0yMDI1IC0gaHR0cDovL3d3dy52aWRlb2xhbi5vcmcveDI2NC5odG1sIC0gb3B0aW9uczogY2 + FiYWM9MCByZWY9MSBkZWJsb2NrPTA6MDowIGFuYWx5c2U9MDowIG1lPWRpYSBzdWJtZT0wIHBzeT0xIHBzeV9yZD0xLjAw + OjAuMDAgbWl4ZWRfcmVmPTAgbWVfcmFuZ2U9MTYgY2hyb21hX21lPTEgdHJlbGxpcz0wIDh4OGRjdD0wIGNxbT0wIGRlYW + R6b25lPTIxLDExIGZhc3RfcHNraXA9MSBjaHJvbWFfcXBfb2Zmc2V0PTAgdGhyZWFkcz0yIGxvb2thaGVhZF90aHJlYWRz + PTEgc2xpY2VkX3RocmVhZHM9MCBucj0wIGRlY2ltYXRlPTEgaW50ZXJsYWNlZD0wIGJsdXJheV9jb21wYXQ9MCBjb25zdH + JhaW5lZF9pbnRyYT0wIGJmcmFtZXM9MyBiX3B5cmFtaWQ9MiBiX2FkYXB0PTAgYl9iaWFzPTAgZGlyZWN0PTEgd2VpZ2h0 + Yj0wIG9wZW5fZ29wPTAgd2VpZ2h0cD0wIGtleWludD0xNiBrZXlpbnRfbWluPTEgc2NlbmVjdXQ9MCBpbnRyYV9yZWZyZX + NoPTAgcmM9Y3JmIG1idHJlZT0wIGNyZj00MC4wIHFjb21wPTAuNjAgcXBtaW49MCBxcG1heD02OSBxcHN0ZXA9NCBpcF9y + YXRpbz0xLjQwIHBiX3JhdGlvPTEuMzAgYXE9MACAAAAAGWWIhADom5OTk5OTrrrrrrrrrrrrrrrrrrwAAAAHQZokAOoMwA + AAAAdBnkJANoMwAAAABwGeYUBdBmAAAAAHAZ5jQF0GYAAAAAhBmmg0QHUGYAAAAAlBnoZFEShtBmAAAAAHAZ6lQGUGYAAA + AAcBnqdAZQZgAAAACEGarDRAfQZgAAAACUGeykUVKG0GYAAAAAcBnulAZQZgAAAABwGe60BlBmAAAAAIQZrvNEB9BmAAAA + AJQZ8NRRUodQZgAAAABwGfLkBlBmAAAAAbZYiCAA+iYoycnJycnXXXXXXXXXXXXXXXXXXgAAAAB0GaJADqDMAAAAAHQZ5C + QDaDMAAAAAcBnmFAZQZgAAAABwGeY0BlBmAAAAAIQZpoNEB1BmAAAAAJQZ6GRREobQZgAAAABwGepUBlBmAAAAAHAZ6nQG + UGYAAAAAhBmqw0QH0GYAAAAAlBnspFFShtBmAAAAAHAZ7pQGUGYAAAAAcBnutAZQZgAAAACEGa7zRAfQZgAAAACUGfDUUV + KHUGYAAAAAcBny5AbQZgAAAAG2WIhAAQomKMnJycnJ1111111111111111114A== + """ }