Skip to content

Latent-space chaining seams for chunked video continuation - #40

Merged
TroyHernandez merged 3 commits into
mainfrom
feat/ltx-latent-chain
Jul 21, 2026
Merged

Latent-space chaining seams for chunked video continuation#40
TroyHernandez merged 3 commits into
mainfrom
feat/ltx-latent-chain

Conversation

@TroyHernandez

Copy link
Copy Markdown
Contributor

Chained multi-chunk generation today round-trips every join through pixels: decode chunk N, H.264-encode it, re-read its tail frames, VAE re-encode them into conditioning latents for chunk N+1 — plus a full onload/offload cycle of every component per chunk. This PR adds the seams to keep a chain in latent space end to end and to hold components resident across back-to-back calls.

New surface

  • condition_latents= on txt2vid_ltx2(): an already-encoded conditioning prefix [1, 128, k, H/32, W/32], bypassing the read/preprocess/VAE-encode block entirely. Mutually exclusive with image/condition_video; geometry-validated.
  • ltx23_tail_latents() (exported): slices the trailing k latent frames of a result into condition_latents form. Results now carry latent_shape to make that self-contained. The docs flag the semantics caveat: sliced continuation latents represent 8 pixel frames each, a fresh tail encode represents 1 + 8(k−1) with a first-frame-form leading latent — A/B on real content before trusting latent-only joins.
  • resident=: component names to keep on the compute device after their phase. Onload is now idempotent (device check, tryCatch-guarded so anything odd falls back to the old path), offload and the dequant-buffer release skip resident components. A chunk loop pays one transformer onload per track instead of one per chunk.
  • trim_frames=: drop leading pixel frames (the conditioning-head overlap) from the decoded video and the saved file, in memory — replacing a lossy subclip re-encode downstream. The returned latents keep the full sequence, since tail slicing needs it.

Head trim stays in pixel space by design: the causal VAE decodes latent frame 1 to a single pixel frame and later frames to 8 each, so dropping leading latent frames before decode yields the wrong frame count (7 short) and pushes a continuation-form latent through the decoder's first-frame path.

Tests

Extends the tiny-pipeline CPU fixture: condition_latents reproduces the condition_video path bit-for-bit when fed the same encoded tail; tail slicing matches a manual unpack+narrow; a sliced tail feeds back end to end; trim_frames output equals the untrimmed video minus its head, with latents and latent_shape intact; guardrails for the three-way source exclusivity, bad geometry, bad trim, and unknown resident names. Full suite: 923 green (CPU).

GPU validation still owed (deferred while the card is busy): resident-transformer chain timing, and the sliced-tail vs fresh-encoded-tail A/B on identity/lip-sync stability.

@TroyHernandez
TroyHernandez merged commit cdb562f into main Jul 21, 2026
2 checks passed
@TroyHernandez
TroyHernandez deleted the feat/ltx-latent-chain branch July 21, 2026 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant