Skip to content

fix(playback): stop unverified bytes from adopting a release - #455

Closed
randrini wants to merge 9 commits into
mainfrom
fix/no-identity-adoption-on-mismatch
Closed

randrini wants to merge 9 commits into
mainfrom
fix/no-identity-adoption-on-mismatch

Conversation

@randrini

Copy link
Copy Markdown
Collaborator

Summary

A size-gap serve no longer establishes a release's identity. A row's own candidate can still play and carry its measured inventory when a provider posts it short, but unverified bytes cannot write another row's inventory or move the plan to a different release.

Problem

Measured on the build with #447/#448/#452 (2026-11-15 01:39–02:30Z):

  • served_bytes=16961343519 was returned for four candidate URIs (59241, 59242, 59274, 1662283). A 720p / 2.36 GB request received a 16.96 GB object.
  • The 720p row reported resolution=2160p, codec_video=hevc, hdr=t while keeping its 720p release name and size. 538 of 27,397 rows showed a …1080p… name with a probed 2160p resolution.
  • An explicit 720p H.264/AAC pick was planned as a 2160p Dolby Vision HEVC transcode with substituted_from_file_id=59274, reason=version_not_playable.
  • 150 size-mismatch lines in 50 minutes, including virtual probe evidence rotated to the candidate's owner row … reason=sibling_owner_rotation.

#447 made a size gap usable so it does not block probing or playing the row's own bytes. That part is right. Two consequences were not acceptable: the unverified inventory was written onto the requested row, and the plan and persist seams rotated onto a row that owns an unrelated object.

Rule

When served bytes contradict the declared size, the verdict is unverified. Unverified bytes are playable only for the row whose own candidate measured them.

  • The persist seam refuses an unverified write unless the candidate belongs to the row's release. It never rotates to a foreign owner row.
  • resolveVirtualPlaybackSource and the stale-source fallback refuse a candidate whose effective row is a different release when the served size does not confirm that row. They try the next candidate of the requested release; when none serves, the start fails with the mismatch instead of playing another release.
  • A concrete selection is never fulfilled by a different release on unverified evidence. A neutral listing row, which owns no bytes, may still bind to the candidate's owner; that is auto selection, not a substitution.
  • fix(playback): keep size-gapped probe bytes usable #447's intent is preserved: a row whose own candidate is short still probes, plays, and writes its measured inventory metadata-only, without a stamp or pin.
  • fix(playback): keep version rotations from dropping selected tracks #448's track-loss and duration guard and fix(playback): keep virtual resolve fallbacks and substitutions honest #452's budget fix and substitution reporting are unchanged. fix(playback): keep version rotations from dropping selected tracks #448's guard still runs for verified rotations, so it is not redundant.
  • A refused row's stored probe inventory is invalidated with a CAS-fenced ClearProbe and empty inventory, so the next start re-probes instead of presenting the foreign tracks.

Mechanism

  • internal/api/handlers/playback_virtual.go:745 classifyVirtualServedEvidenceV3 calls internal/scanner/virtual_probe_guard.go:469 ClassifyVirtualServedEvidenceForProbe, which computes Unverified from a size gap and Mismatch from a duration or track-layout contradiction.
  • playback_virtual.go:790 virtualProbeEvidenceWithholdsIdentity turns that verdict into "no stamp, no pin". Before this change that was the only place the verdict was consulted: the persist seam still wrote the inventory, and virtualProbeEvidenceRotateTarget (:6212) rotated on path ownership alone. The sibling_owner_rotation line at :6319 never saw the verdict.
  • virtualProbeEvidenceArgs (:6163) now takes identityVerified and refuses an unverified write whose candidate does not belong to the row. invalidateVirtualProbeInventory (:6055) performs the repair write.
  • The plan-side guards are in resolveVirtualPlaybackSource (:4074) and the stale fallback (:7791).

Repair

The 538 contaminated rows are not rewritten by a data migration. A row is invalidated the next time the bad candidate recurs: the refusal clears its probe stamp and inventory, and the next start re-probes. Rows that never see the bad candidate again re-probe after the existing 12-hour probe staleness bound, when played again. The new rule prevents new contamination on both paths.

Tests

New tests in internal/api/handlers/playback_virtual_unverified_identity_test.go:

  • TestUnverifiedForeignEvidenceDoesNotRotateOntoTheOwnerRow: an unverified write for a foreign owner writes no inventory and invalidates the requested row.
  • TestUnverifiedExplicitPickIsNotFulfilledByAnotherRelease: the 720p pick with a 2160p owner candidate fails with the retryable virtual_source_unavailable terminal.
  • TestUnverifiedSizeGapOnOwnBytesStillInventoriesItsRow: a row's own short bytes still play and write their measured inventory metadata-only.

Fail-before on the prerequisite tip (bfa78e596): the first test persisted one inventory onto the owner row; the second resolved successfully with URI=…?result=foreign2160 and SubstitutedFromFileID=710.

Two #447 cache-replay tests now assert that an unverified foreign cache hit writes only a probe invalidation, not inventory.

Verification

  • go build ./...
  • go vet ./internal/api/handlers ./internal/scanner ./internal/playback
  • go test -count=1 -skip 'TestHandleStartPlaybackV3ReturnsBoundedOnSlowProbe|TestHandleStartPlaybackV3DoesNotWaitForSessionSync' ./internal/api/handlers
  • go test -count=1 ./internal/scanner
  • go test -count=1 -skip 'TestServeExtractDiscardsACappedDemuxCopy|TestOnDemandSubtitleSegmentsDoNotWaitForTheStream' ./internal/playback
  • go test -race -count=1 -run TestUnverified ./internal/api/handlers
  • make verify-playback-fixtures
  • make lint-changed BASE_REF=kipsilabs/main reports 0 issues.
  • gofmt and git diff --check

The skipped tests also fail on bfa78e596 in this environment: the two handler tests measure a 40 ms probe bound where planning takes seconds, and the two playback tests cover subtitle extraction and a stream-timing assertion. This change does not touch any of them.

Stacking

The branch starts at main (303f3eccc) and merges #447 and #452, which contains #448, as prerequisites because the bug is in their composition. The fix commit is the tip.

Related issue: N/A

Posted by an AI agent (Muse Spark, OpenCode harness).

randrini and others added 9 commits November 14, 2026 12:04
Separate release identity from usability in the virtual probe gate. A
served-vs-declared size gap no longer rejects a candidate: the probe's
measured inventory is written metadata-only, planned and played, while
the verdict stays Unverified and never stamps the row or pins the
candidate. Duration or track-layout contradictions still reject as a
different file, and every acceptance path now uses one classification
entry point so the fresh-probe, cache, and persist seams agree.
…erequisite

#448 (fix/rotation-respects-selection) is open, not merged. This change
extends the same guard to the resolve-timeout fallback, so it must reuse
virtualRotationWouldDropTracks/virtualRequestedRowIsSelection instead of
growing a second policy. Whichever of the two lands first, the other's
call sites keep working.
A virtual resolve attempt took its bound from a nested WithTimeout on a
deadline an earlier stage had already spent. The derived context expired
before the wait began, the resolver logged a negative bound
(bound_ms=-34/-65/-95/-126 on four consecutive attempts), and every attempt
failed immediately with context deadline exceeded. Callers read that as a
provider listing failure: the failover loop ran on a spent budget, the version
fallback used it to justify a release substitution, and the transport reported
the last attempt as a non-retryable transcode_start_failed.

Derive each attempt's budget from the request's remaining time under an
explicit cap (virtualResolveAttemptBudget), refuse an attempt when nothing is
left, and stop the cold-path candidate loop once its single deadline is spent.
Logs report 0 ms left with an explicit bound_exhausted flag instead of a
negative wait. A transport startup with no budget left answers the retryable
provider_unavailable instead of a fabricated transcode verdict.

The cold-path and stale-pin tests now pin the new truth: a spent budget is
refused, and the recovery listing is identified by its own sub-budget rather
than by a global call ordinal.
The cold-start version fallback walked alternate releases on any provider
listing failure, including a resolve timeout, and served the first that
resolved. Production: a 3-audio/4-subtitle selection played as a
1-audio/0-subtitle sibling.

Apply the rotation guard #448 introduced to the fallback: a substitute that
contradicts the item's runtime (duration_contradiction) or offers fewer audio
or subtitle tracks than the selected row (track_loss) is refused, and when every
substitute would lose the selection the start answers a retryable terminal that
names the version list instead of serving a poorer menu.

A walk whose own budget is spent does not run at all: a timeout is not evidence
that the selected release cannot serve, and every alternate would be resolved
on the same expired deadline.
The plan's substituted_from_file_id and substitution_reason were set in the
domain type, but the v2 projection rebuilt the plan field by field and dropped
both. Across 35 production plan/replan decisions with 14 effective-id changes
the client saw them zero times, so it could only guess which release it got.

Project both fields into the v2 plan and publish them on every replan that
commits a row other than the requested one, not only the explicit version-pick
hunt. The cause comes from a reason the plan already carries, a decode
rotation, or the replan's failure class.

Additive: two optional plan fields, regenerated OpenAPI and web types, and a
short playback API section naming the substitution vocabulary. Tests pin the
serialized body, the omitted-when-unsubstituted case, and a replan rotation.
# Conflicts:
#	internal/api/handlers/playback_virtual.go
A served size that contradicts the declared release keeps the row's own candidate playable, but it never establishes that row's identity. The persist seam now refuses an unverified write whose bytes do not belong to the row it would repaint, the resolve and stale-fallback paths refuse to plan a foreign owner row from unverified evidence, and a refused row's stale inventory is invalidated so the next probe re-measures it.
@randrini

Copy link
Copy Markdown
Collaborator Author

Superseded by #459, which contains this change plus the rest of the playback reliability bundle. Closing to keep the review surface to one PR.

@randrini randrini closed this Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants