From eb183af80a790bb941a238ba004829cdb7287ff0 Mon Sep 17 00:00:00 2001 From: TroyHernandez Date: Wed, 22 Jul 2026 17:30:51 -0500 Subject: [PATCH 01/10] Release prep 0.2.0: SD tiers labeled fp16, performance-levers vignette, README hardware table, NEWS consolidation --- DESCRIPTION | 4 +- NEWS.md | 303 ++++++++++---------------------- R/recommend.R | 23 +-- README.md | 16 ++ inst/tinytest/test_recommend.R | 9 +- man/recommend.Rd | 8 +- vignettes/performance-levers.md | 119 +++++++++++++ 7 files changed, 256 insertions(+), 226 deletions(-) create mode 100644 vignettes/performance-levers.md diff --git a/DESCRIPTION b/DESCRIPTION index 3ea4d54..e7da035 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,6 +1,6 @@ Package: diffuseR Title: Functional Interface to Diffusion Models in R -Version: 0.1.0.16 +Version: 0.2.0 Authors@R: c( person("Troy", "Hernandez", email = "troy@cornball.ai", role = c("aut", "cre"), comment = c(ORCID = "0009-0005-4248-604X")), @@ -30,5 +30,7 @@ Suggests: av, hfhub, safetensors, + simplermarkdown, tinytest +VignetteBuilder: simplermarkdown RoxygenNote: 7.3.3 diff --git a/NEWS.md b/NEWS.md index 33d7f7c..45d508b 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,211 +1,98 @@ -# diffuseR 0.1.0.16 (development) +# diffuseR 0.2.0 -* `recommend()` is now host-RAM-aware: it returns `pin` (page-lock the - phase-swapped host weight copies), `pinned_set_gb`, and - `host_ram_gb`. Pinned pages are unswappable, so the policy pins only - when available RAM covers the model's pinned set twice over — small- - RAM machines get `pin = FALSE` instead of OOM-kill risk, the cpu - tier never pins, and undetectable RAM leaves pinning on (it already - fails soft per component). - -# diffuseR 0.1.0.15 (development) - -* Pinned staging now allocates page-locked host memory via - `torch_empty_strided(pin_memory = TRUE)` instead of the deprecated - `Tensor$pin_memory(device)` overload, silencing the two libtorch - deprecation warnings per tensor (thousands of lines per pipeline - load). Validated at 25.1 GB/s H2D (the card's DMA ceiling) with a - clean stderr. -* `txt2vid_ltx2()` gains `connector_embeds=`: precomputed - text-connector outputs (the prompt is constant across a chained - track) skip the per-call connectors phase. Benchmarked at 960x960 - this cuts the denoise-phase peak by ~2.5 GiB — the per-call phase - moved the raw Gemma3 hidden-state stack to the GPU, which is what - OOMed every resident-transformer chain attempt. -* The `resident=` onload check now probes the staging pair's live - tensor instead of `module$parameters` (which the NF4 transformer - doesn't expose), so a resident component is never silently - re-onloaded over itself. -* The Gemma3 loaders gain `pin=` (default: the `diffuseR.pin_staging` - option): a CPU-resident encoder is page-locked once and - `encode_with_gemma3()` swaps it to the GPU per encode (~0.3 s on, - free off) instead of holding VRAM or reloading (~15 s). - -# diffuseR 0.1.0.14 (development) - -* Latent-space chaining seams for chunked video continuation: - `txt2vid_ltx2()` gains `condition_latents=` (an already-encoded - prefix, bypassing the VAE), `resident=` (keep named components on - the compute device across back-to-back calls; onload is now - idempotent), and `trim_frames=` (deliver head-free pixels while the - returned latents keep the full sequence). Results carry - `latent_shape`, and the new `ltx23_tail_latents()` slices a - result's trailing latent frames into `condition_latents` form — so - a chunk chain can stay in latent space end to end: encode once, - denoise every chunk with the transformer resident, decode once. - -# diffuseR 0.1.0.13 (development) - -* `gemma3_quantize_nf4()` + `load_gemma3_nf4()` put the Gemma3 12B text - encoder on the GPU: the 336 projection weights quantize to a ~8 GB - NF4 artifact (one-time 52 s; vision tower dropped), which loads in - ~12 s and encodes in ~7 s on CUDA vs ~30 s load + ~24 s encode for - the fp32 CPU path, at 8 GB host RAM instead of 45. Pinned phase swap - costs 0.3 s on / 0 s off after a one-time 3.3 s page-lock. - `load_gemma3_text_encoder()` dispatches to the artifact - automatically. Embedding drift (cosine 0.989 vs fp32) renders - scene-equivalent same-seed videos. -* The eager NF4 dequant now shares the byte-LUT with the jit block - stack (one index per packed byte, gather straight into the output), - speeding every eager NF4 forward (Gemma3, FLUX.1, LTX fallbacks). -* Fixed `txt2vid_ltx2(decode_audio = FALSE, filename = )`: partial - matching handed the raw audio latents to the WAV writer. -* `gemma3_encode_batch()` encodes a prompt vector in sub-batches with - optional per-prompt disk caching (resumable; returns cache paths): - encode a whole episode list once, then swap the encoder off and - render from the cached embeddings. Works around a torch_save bug - where storage-offset views serialize from the base storage. - -# diffuseR 0.1.0.12 (development) - -* LTX video decode runs untiled when the estimated activation cost - (~1 GB + ~360 B per output pixel-frame) fits the card - tiling - bounds VRAM, and at decode time the transformer is phase-offloaded, - so 768x512x49 decodes in one full-latent forward. The per-tile - explicit `gc()` is gone (storm-free with the allocator gates live). - In-render decode phase 12.3 s -> 1.7-1.8 s; warm render 54 -> ~44 s. - `options(diffuseR.vae_untiled = TRUE/FALSE)` forces either path. - -# diffuseR 0.1.0.11 (development) - -* `ltx23_tune_gc()` now takes effect: it used to set the allocator gate - options one call after `torch::cuda_is_available()` had started torch - (which reads them exactly once at init), so every option it set was - inert. Options now land before the first torch call, the three CUDA - gates are pushed into the live allocator, and `.onLoad` defaults - `torch.threshold_call_gc` (which has no live setter). Measured: ~4-5% - off LTX render walls (63.7/62.2/61.9 s vs 67.9/64.2/65.6 s at - 768x512x49 NF4). -* `diffuseR.pin_staging` now defaults to TRUE for LTX phase offload: - page-locked host copies make onload a DMA transfer and offload a - pointer swap, saving ~7 s per render for +9 s of one-time - page-locking at pipeline load (break-even on the second render). - Opt out with `options(diffuseR.pin_staging = FALSE)` under host - memory pressure. - -# diffuseR 0.1.0.10 (development) - -* The generators accept a three-level `verbose`: "silent", "progress", - "steps" (logicals still work: TRUE = "steps", FALSE = "silent"). - "progress" prints a one-line generation summary plus a denoise - progress bar interactively, or a tick line every ~n/10 steps in - captured logs, so a long run is distinguishable from a hang without - per-step noise (#30). - -# diffuseR 0.1.0.9 (development) - -* The LTX-2.3 JIT block stack dequantizes NF4 weights through a - precomputed [256, 2] byte lookup table (one embedding gather in the - compute dtype) instead of the int64 shift/stack/gather chain, cutting - per-step dequant memory traffic ~6x (isolated benchmark: 5.7x; the - dequant was the measured per-step wall at ~4.4 s of a 6.5 s step). - -# diffuseR 0.1.0.8 (development) - -* The Qwen3 encoder builds its additive attention mask in the query - dtype, fixing the "invalid dtype for bias" CUDA error every FLUX.2 - prompt encode hit through the fused SDPA path (since #33). - -# diffuseR 0.1.0.7 (development) - -## SDXL native pipeline from safetensors - -* `sdxl_pipeline_from_safetensors()` and `txt2img_sdxl(diffusers_dir=)` - run SDXL end to end from a diffusers safetensors directory (no - TorchScript, so it works on Blackwell): the two CLIP encoders - (ViT-L with `quick_gelu`, OpenCLIP bigG) are concatenated at their - penultimate hidden state into the 2048-dim UNet conditioning, with - pooled `text_embeds` + `time_ids` added conditioning and the VAE - `scaling_factor` (0.13025) read from `vae/config.json`. -* `text_encoder_native()` and `text_encoder2_native()` gain - `return_penultimate` for the SDXL `hidden_states[-2]` prompt embeds; - the encoder-2 pooled output still comes from the full stack. -* `text_encoder2_native_from_safetensors()` and - `load_text_encoder2_safetensors()` load the OpenCLIP bigG - `text_encoder_2` (including its top-level `text_projection`); the - encoder-1 and encoder-2 loaders now share one key remap and core. -* `reshard_safetensors()` splits an oversize (>= 2 GB) safetensors into - sub-2 GB diffusers shards plus an index, so large fp16 weights load on - stock CRAN safetensors. -* `download_sdxl()` fetches the SDXL diffusers weights from the - `cornball-ai/sdxl-R` dataset. - -## LTX-2.3 loading and attention +## Native safetensors pipelines +* SD 2.1, SDXL, FLUX.1-schnell, FLUX.2 Klein, Z-Image-Turbo, and + LTX-2.3 all run fully natively from diffusers-layout safetensors — + no TorchScript step, so everything works on Blackwell (RTX 50xx). + `download_sd21()`/`download_sdxl()` fetch diffusers weights; + `sd_pipeline_from_safetensors()`/`sdxl_pipeline_from_safetensors()` + build the pipelines; `txt2img_sd21()`/`txt2img_sdxl()` take + `diffusers_dir=`. * Checkpoint loaders build module skeletons (uninitialized weights at - the target dtype) instead of initializing full-precision modules and - casting: the LTX-2.3 NF4 pipeline load drops from ~108 GB host RAM - (kernel OOM on a 125 GB machine) to ~21 GB, and Gemma3 from ~72 GB of - transient writes to its resident size. `load_gemma3_text_encoder()` - now errors on any parameter the checkpoint does not fill. -* LTX-2.3 attention now uses R torch fused scaled-dot-product attention - when available and no explicit query chunk is requested. The adaptive - chunked/scratch-buffer implementation remains the fallback for older - torch builds, explicit `attn_chunk`, or `options(diffuseR.ltx23_fused_sdpa = FALSE)`. - -# diffuseR 0.1.0.6 (development) - -## Uniform native-safetensors + hosted quantization (in progress) - -* New `recommend(model, vram_gb, st_caps)`: one VRAM- and - capability-aware precision/device recommendation for every model - (sd21, sdxl, flux1, flux2, zimage, ltx). nf4 is the default tier; fp8 - or bf16 is recommended only when the card fits it *and* the installed - safetensors can **read** that dtype, otherwise it recommends nf4 and - surfaces the `cornball-ai/safetensors` suggestion in `$note` (never an - error). -* `flux_memory_profile()` now delegates to `recommend("flux1")`, - correcting the stale tiers that placed fp8 (GPU-resident now, not - streamed) in a narrow low-VRAM band it can no longer fit. -* Quantizer shards default to `shard_bytes = 1.9e9` (`flux_quantize`, - `ltx23_quantize_nf4`, `ltx23_quantize_fp8`). This is *what makes* nf4 - artifacts load on stock CRAN safetensors: R safetensors overflows a - 32-bit offset on any file at or above 2^31 bytes (~2.15 GB), so the - old 4e9 default produced shards only the fork could read. nf4 is - CRAN-readable **because of** the sub-2 GB shards, not automatically; - 4e9 remains available for local fork builds. -* Explicitly requesting fp8/bf16 without the needed safetensors support - now warns and falls back to nf4 instead of failing - (`download_flux1`, `download_flux2_klein`, `download_zimage_turbo`). -* Reading a legacy oversize (>2 GB) shard on stock safetensors raises an - actionable "rebuild with smaller shards or install the fork" message - instead of a raw 32-bit overflow error. -* Native SD21/SDXL UNet weights now load from diffusers safetensors - (`load_unet_safetensors`, `load_unet_sdxl_safetensors`, and the - `unet_native_from_safetensors` / `unet_sdxl_native_from_safetensors` - constructors), with no TorchScript step (Blackwell-safe). The VAE - decoder and CLIP text encoder already had safetensors loaders; the - UNet was the gap. Validated against the cached SDXL base UNet (all - 1680 keys map with matching shapes). -* `vae_decoder_native_from_safetensors` and - `text_encoder_native_from_safetensors` (config-driven CLIP arch - detection) complete the native SD component set. -* `download_sd21()` + `sd_pipeline_from_safetensors()` run Stable - Diffusion 2.1 fully natively from diffusers safetensors; - `txt2img_sd21(diffusers_dir=)` uses it. The SD VAE decode now applies - the `post_quant_conv` the FLUX-derived native decoder omitted (the - decode was badly wrong without it). SD 2.1 defaults to float32 on this - path (fp16 attention overflows to NaN). -* Fixed a native SD 2.1 UNet **tiling** bug (pre-existing; the - `.pt`-native path shared it). The timestep embedding used - `flip_sin_to_cos=FALSE, downscale_freq_shift=1`, scrambling the - sin/cos ordering the trained `time_embedding` weights expect (standard - diffusers SD, and the native SDXL UNet, use `TRUE/0`). Constant-input - parity tests could not see it (GroupNorm sits at its bias, attention is - uniform); it compounded through the spatial path into tiled output. - The native UNet now matches the TorchScript reference at cos 0.99999, - and `test_unet.R` gains a random-input parity check. - -All of the above is capability-**probed**, not version-pinned, so the -fork requirement self-heals when the safetensors fixes reach CRAN -(mlverse/safetensors#11, #13). + the target dtype) instead of initializing fp32 and casting: the + LTX-2.3 NF4 pipeline load drops from ~108 GB host RAM to ~21 GB. + Loaders hard-error on any parameter the checkpoint does not fill. +* `reshard_safetensors()` splits oversize weights into sub-2 GB shards + readable by stock CRAN safetensors; quantizer shards default to + 1.9e9 bytes for the same reason. Requesting fp8/bf16 without a + capable safetensors warns and falls back to nf4; legacy oversize + shards raise an actionable message. +* Fixed a long-standing native SD 2.1 UNet tiling bug: the timestep + embedding used the wrong sin/cos ordering + (`flip_sin_to_cos`/`downscale_freq_shift`), which compounded through + the spatial path into tiled output. Now matches the TorchScript + reference at cosine 0.99999. + +## Machine-aware configuration + +* `recommend(model)`: one VRAM-, host-RAM-, and capability-aware + recommendation for every model — precision tier, per-component + device map, phase offload, pixel budget, attention chunking, and + `pin` (page-lock the phase-swapped host copies; pinned pages are + unswappable, so pinning is recommended only when available RAM + covers the model's pinned set twice over). fp8/bf16 tiers are gated + on the *installed* safetensors' read capabilities; when a card fits + a tier the reader can't load, the fork suggestion is surfaced in + `$note`, never as an error. +* The SD tiers are labeled fp16 at every VRAM level: the SD models + ship no quantized weights, so placement varies, not precision. +* New "Performance Levers" vignette documenting the three axes + (precision ladder, device placement, memory residency) and a + hardware-requirements table in the README. + +## LTX-2.3 video performance + +Warm 768x512x49 renders went from ~90 s to ~44 s across this cycle: + +* NF4 dequantization via a precomputed [256, 2] byte lookup table + (one embedding gather in the compute dtype) instead of an int64 + shift/stack/gather chain — ~6x less per-step memory traffic. +* The CUDA allocator gc gates now actually take effect + (`ltx23_tune_gc()` used to set its options one call after torch had + read them) and are pushed into the live allocator. +* Pinned staging for phase offload, on by default: page-locked host + copies make onload a DMA transfer (25.1 GB/s measured; 11 GB + re-onloads in 0.5 s) and offload a pointer swap. Page-locked memory + is allocated via `torch_empty_strided(pin_memory = TRUE)`, avoiding + a deprecated overload that printed two warnings per tensor. Opt out + with `options(diffuseR.pin_staging = FALSE)`. +* Video decode runs untiled when the estimated activation cost fits + the card: in-render decode 12.3 s -> ~1.8 s. +* Attention uses R torch fused scaled-dot-product attention when + available; the chunked implementation remains the fallback. + +## Chained video generation + +* `txt2vid_ltx2()` gains `condition_latents=` (an already-encoded + conditioning prefix, bypassing the VAE), `connector_embeds=` + (precomputed text-connector outputs — the prompt is constant across + a chained track, and skipping the per-call connectors phase cuts + the denoise peak by ~2.5 GiB), `resident=` (keep components on the + compute device across back-to-back calls, with an idempotent + onload), and `trim_frames=`. Results carry `latent_shape`; + `ltx23_tail_latents()` slices a result's trailing latent frames + into `condition_latents` form. + +## Gemma3 text encoder + +* `gemma3_quantize_nf4()` + `load_gemma3_nf4()`: the 12B encoder + quantizes to a ~8 GB NF4 artifact that loads in ~12 s and encodes + in ~7 s on CUDA (vs ~30 s + ~24 s for fp32 CPU) at 8 GB host RAM + instead of 45. `load_gemma3_text_encoder()` dispatches to the + artifact automatically and gains `pin=`: a CPU-resident encoder is + page-locked once and `encode_with_gemma3()` swaps it to the GPU per + encode (~0.3 s on, free off). +* `gemma3_encode_batch()`: sub-batched, disk-cached, resumable prompt + encoding for episode-scale workloads. + +## Fixes + +* Qwen3 encoder attention masks build in the query dtype, fixing the + "invalid dtype for bias" error on every FLUX.2 prompt encode through + fused SDPA. +* `txt2vid_ltx2(decode_audio = FALSE, filename = )` no longer hands + the raw audio latents to the WAV writer via partial matching. +* Generators accept a three-level `verbose` ("silent", "progress", + "steps"); "progress" gives a one-line summary plus a progress bar + or periodic ticks in captured logs. diff --git a/R/recommend.R b/R/recommend.R index 6aaca39..38a7045 100644 --- a/R/recommend.R +++ b/R/recommend.R @@ -4,9 +4,11 @@ #' model. The policy: #' #' \itemize{ -#' \item nf4 is the default tier. Its weights are packed uint8 plus -#' float32 blocks in sub-2 GB shards, which every safetensors reads, -#' so it always loads. +#' \item nf4 is the default tier for the quantized families (flux1, +#' flux2, zimage, ltx). Its weights are packed uint8 plus float32 +#' blocks in sub-2 GB shards, which every safetensors reads, so it +#' always loads. The SD models ship no quantized weights; their +#' floor is fp16 with placement varying by VRAM. #' \item When the card has room for a higher-quality tier (fp8 or #' bf16) AND the installed safetensors can \emph{read} that dtype #' (\code{\link{.st_can_read}}), that tier is recommended instead. @@ -161,8 +163,8 @@ recommend <- function(model = c("sd21", "sdxl", "flux1", "flux2", "zimage", # order of magnitude, not the byte. .pinned_set_gb <- function(model, precision) { sets <- list( - sd21 = c(fp16 = 3, nf4 = 2), - sdxl = c(fp16 = 8, nf4 = 4), + sd21 = c(fp16 = 3), + sdxl = c(fp16 = 8), flux1 = c(bf16 = 43, fp8 = 31, nf4 = 26), # DiT + T5 fp32 host copy flux2 = c(bf16 = 17, fp8 = 12, nf4 = 10), # DiT + Qwen3 bf16 zimage = c(bf16 = 21, fp8 = 14, nf4 = 12), # DiT + Qwen3 bf16 @@ -213,18 +215,19 @@ recommend <- function(model = c("sd21", "sdxl", "flux1", "flux2", "zimage", ) } -# SD-family ladder: fp16 for cards that fit the full model, nf4 default -# for tighter cards, cpu otherwise. Both dtypes are CRAN-readable, so no -# fork gate. Device maps reuse the auto_devices strategy builder. +# SD-family ladder: the SD models ship no quantized weights, so every +# tier is fp16 - what varies is placement (all-GPU, UNet-only-GPU, or +# CPU) and the area budget. Device maps reuse the auto_devices +# strategy builder. .sd_tiers <- function(model, fp16_vram, nf4_vram, max_fp16, max_nf4, max_cpu) { list( list(precision = "fp16", min_vram = fp16_vram, needs = NULL, devices = .build_fallback_devices(model, "full_gpu"), offload = FALSE, max_pixels = max_fp16), - list(precision = "nf4", min_vram = nf4_vram, needs = NULL, + list(precision = "fp16", min_vram = nf4_vram, needs = NULL, devices = .build_fallback_devices(model, "unet_gpu"), offload = TRUE, max_pixels = max_nf4), - list(precision = "nf4", min_vram = 0, needs = NULL, cpu = TRUE, + list(precision = "fp16", min_vram = 0, needs = NULL, cpu = TRUE, devices = .build_fallback_devices(model, "cpu_only"), offload = FALSE, max_pixels = max_cpu) ) diff --git a/README.md b/README.md index 2c6687b..62fc0a8 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,22 @@ targets::install_github("cornball-ai/diffuseR") - **Device Support**: CPU and CUDA GPUs (including Blackwell RTX 50xx) - **R-native Interface**: Functional programming approach that feels natural in R +## Hardware requirements + +Approximate, per model, from the floor configuration (lowest precision, +CPU-only) to the best-quality tier. `recommend("model")` picks the right +configuration for your machine automatically; the `vignette("performance-levers")` +explains the levers. + +| model | minimum (floor quality) | best quality | +|---|---|---| +| SD 2.1 | 4 GB RAM, no GPU | 6 GB VRAM + 4 GB RAM (fp16) | +| SDXL | 10 GB RAM, no GPU | 12 GB VRAM + 8 GB RAM (fp16) | +| FLUX.2 Klein | 12 GB RAM, no GPU | 16 GB VRAM + 20 GB RAM (bf16) | +| Z-Image-Turbo | 14 GB RAM, no GPU | 18 GB VRAM + 24 GB RAM (bf16) | +| FLUX.1-schnell | 28 GB RAM, no GPU | 24 GB VRAM + 45 GB RAM (bf16) | +| LTX-2.3 video | 40 GB RAM, no GPU | 16 GB VRAM + 45 GB RAM (fp8, streamed) | + ## Quick Start ### Basic Usage diff --git a/inst/tinytest/test_recommend.R b/inst/tinytest/test_recommend.R index eafb74d..ec09b14 100644 --- a/inst/tinytest/test_recommend.R +++ b/inst/tinytest/test_recommend.R @@ -19,12 +19,13 @@ expect_true(all(c("precision", "devices", "offload", "max_pixels", names(r))) expect_true(is.integer(r$max_pixels) || is.numeric(r$max_pixels)) -# --- nf4 is the floor / default --------------------------------------------------- +# --- floors: nf4 for the quantized families, fp16 for SD ------------------------- -# No GPU: nf4 on CPU, no fork nag, smallest area. +# No GPU: everything runs on CPU at its floor precision, no fork nag. +# SD ships no quantized weights, so its floor is fp16. for (m in c("sd21", "sdxl", "flux1", "flux2", "zimage")) { z <- recommend(m, vram_gb = 0, st_caps = cran) - expect_equal(z$precision, "nf4") + expect_equal(z$precision, if (m %in% c("sd21", "sdxl")) "fp16" else "nf4") expect_false(z$fork_suggested) expect_true(all(unlist(z$devices) == "cpu")) } @@ -65,7 +66,7 @@ expect_equal(nob$precision, "nf4") # --- SD ladder: fp16 for cards that fit, nf4 default, cpu ------------------------- expect_equal(recommend("sdxl", 16, cran)$precision, "fp16") -expect_equal(recommend("sdxl", 8, cran)$precision, "nf4") +expect_equal(recommend("sdxl", 8, cran)$precision, "fp16") expect_equal(recommend("sdxl", 16, cran)$devices$unet, "cuda") expect_true("text_encoder2" %in% names(recommend("sdxl", 16, cran)$devices)) expect_false("text_encoder2" %in% names(recommend("sd21", 16, cran)$devices)) diff --git a/man/recommend.Rd b/man/recommend.Rd index f0dc845..d24b3d9 100644 --- a/man/recommend.Rd +++ b/man/recommend.Rd @@ -35,9 +35,11 @@ model. The policy: } \details{ \itemize{ -\item nf4 is the default tier. Its weights are packed uint8 plus -float32 blocks in sub-2 GB shards, which every safetensors reads, -so it always loads. +\item nf4 is the default tier for the quantized families (flux1, +flux2, zimage, ltx). Its weights are packed uint8 plus float32 +blocks in sub-2 GB shards, which every safetensors reads, so it +always loads. The SD models ship no quantized weights; their +floor is fp16 with placement varying by VRAM. \item When the card has room for a higher-quality tier (fp8 or bf16) AND the installed safetensors can \emph{read} that dtype (\code{\link{.st_can_read}}), that tier is recommended instead. diff --git a/vignettes/performance-levers.md b/vignettes/performance-levers.md new file mode 100644 index 0000000..d2a9843 --- /dev/null +++ b/vignettes/performance-levers.md @@ -0,0 +1,119 @@ + +--- +title: "Performance Levers" +--- + +# Performance Levers + +diffuseR runs 20B-parameter models on 16 GB consumer GPUs, and most +models on no GPU at all. It does that with three independent levers: +weight precision, per-component device placement, and memory residency. +This vignette is the map. The machine-readable version of the same +policy is `recommend()`, which inspects your VRAM, host RAM, and +installed `safetensors` capabilities and returns a configuration; the +`txt2img_*`/`txt2vid_*` defaults consume it. + +## Lever 1: weight precision + +The ladder that exists in code: fp32 → bf16/fp16 → fp8 (e4m3fn) → nf4. +Quantization applies to the big diffusion transformers only — VAEs, +vocoders, connectors, and modulation/embedding layers always stay at +16/32 bits (they are small and precision-sensitive; each quantizer +carries an exact census of which weights it may touch). + +| model | DiT / UNet | text encoder(s) | VAE | +|---|---|---|---| +| FLUX.1 (12B) | nf4, fp8 (streamed), bf16, fp32 | T5: fp32 (CPU) · CLIP-L: fp16/fp32 | 16/32 | +| FLUX.2 klein (4B) | nf4, fp8 (resident), bf16, fp32 | Qwen3-4B: bf16/fp32 | 16/32 | +| Z-Image (6B) | nf4, fp8 (resident), bf16, fp32 | Qwen3-4B: bf16/fp32 | 16/32 | +| LTX-2.3 (22B video) | nf4, fp8 (streamed), bf16, fp32 | Gemma3-12B: nf4, bf16, fp32 | 16/32 | +| SD 2.1 / SDXL | fp16, fp32 | CLIP: fp16/fp32 | 16/32 | + +Two readability rules govern the ladder: + +- **nf4 always loads.** Its artifacts are packed uint8 plus float32 + scale blocks in sub-2 GB shards — every `safetensors` build reads + them. It is the default tier for the quantized families. +- **fp8 needs a capable `safetensors`.** The float8 dtypes are not yet + readable by the CRAN `safetensors`; `recommend()` probes the + installed build and, when a card could run fp8 but the reader + cannot, recommends nf4 and surfaces the suggestion in `$note` + (never an error). + +The SD-family models ship no quantized weights: their floor is fp16, +and what varies across VRAM is placement, not precision. + +## Lever 2: device placement + +Every component takes its own device. The SD family uses explicit +device maps (`auto_devices()` strategies: `full_gpu`, `unet_gpu`, +`cpu_only`); the flux family and LTX use phase offloading, where each +component holds the GPU only for its own phase — text encoding, +denoising, decoding — and the denoiser is the sole GPU tenant during +the loop. Text encoders earn special placement: FLUX.1's T5 runs +fp32 on the CPU for quality; the Qwen3 encoders phase-onload in bf16; +the Gemma3 encoder can be GPU-resident or CPU-resident with a staged +swap (see below). + +## Lever 3: residency + +From most to least VRAM: + +1. **Fully GPU-resident** — everything fits, nothing moves (large + cards; the fp8 residents of FLUX.2/Z-Image on 16 GB). +2. **Resident quantized DiT, phase-swapped everything else** — the + LTX nf4 default on 16 GB. +3. **Pinned phase swap** — component weights live in page-locked + ("pinned") host RAM and move to the GPU for their phase at DMA + rate. Measured on a PCIe 5.0 x8 card: 25 GB/s host-to-device + (11 GB of transformer re-onloads in 0.5 s), and the offload is a + pointer swap back to the still-valid pinned copy — zero bytes + moved, because inference never mutates weights. Page-locking costs + ~0.6 s/GB once at load. Built today for the LTX pipeline + components and the Gemma3 encoder. +4. **Pageable phase swap** — the same movement through ordinary + memory, at roughly 2-16 GB/s depending on tensor layout. What the + flux-family encoders do today. +5. **Streamed weights** — the bigger-than-VRAM tier: weights stay in + (pinned) host RAM permanently and stream across PCIe during each + forward pass, about one byte per parameter per step. LTX fp8 and + FLUX.1 fp8 run this way. +6. **CPU-only** — every model runs without a GPU, at its floor + precision. + +There is no disk tier at inference time: weights load from disk once, +and the closest thing to "swap to disk" is *unpinned* host memory +being paged out by the OS — which is exactly the trade `recommend()` +weighs. + +## Pinning: when and when not + +Pinned pages are unswappable — they subtract from what the OS can page +out, so on small-RAM machines they convert memory pressure into +process kills rather than slowdowns. `recommend()` therefore returns +`pin = TRUE` only when available host RAM covers the model's estimated +pinned set twice over, `FALSE` on the CPU tier (nothing stages), and +`TRUE` when RAM cannot be detected, because page-locking already fails +soft per component. The global switch is +`options(diffuseR.pin_staging = FALSE)` — reach for it under host +memory pressure, in containers with hard memory caps, or for +single-generation sessions where the one-time page-lock never pays +itself back. Today the LTX pipeline and Gemma3 loaders consume the +decision; the image-model loaders do not stage weights yet. + +## Putting it together + +```r +r <- recommend("ltx") # or "flux1", "flux2", "zimage", "sdxl", "sd21" +r$precision # tier the card + safetensors support +r$devices # per-component placement +r$pin # page-lock the phase-swapped host copies? +r$note # fork suggestion when fp8 wanted but unreadable +``` + +The generation functions apply this automatically via their defaults; +every field can be overridden by hand. From 0fcffe6f5a62d6c091ffd546b622628076665b2d Mon Sep 17 00:00:00 2001 From: TroyHernandez Date: Wed, 22 Jul 2026 17:59:59 -0500 Subject: [PATCH 02/10] Check fixes: roxygen pairing, return_penultimate doc, quantizer output_dir defaults, fixtures excluded from tarball, README images --- .Rbuildignore | 1 + R/fp8_ltx23.R | 10 +++++++--- R/nf4_ltx23.R | 10 +++++++--- R/staging_ltx23.R | 16 ++++++++-------- R/text_encoder.R | 3 +++ README.md | 4 +--- ...23_pin_host.Rd => dot-ltx23_pin_component.Rd} | 6 +++--- man/ltx23_quantize_fp8.Rd | 8 ++++---- man/ltx23_quantize_nf4.Rd | 8 ++++---- man/text_encoder2_native.Rd | 4 ++++ 10 files changed, 42 insertions(+), 28 deletions(-) rename man/{dot-ltx23_pin_host.Rd => dot-ltx23_pin_component.Rd} (82%) diff --git a/.Rbuildignore b/.Rbuildignore index 1d0835b..749785d 100644 --- a/.Rbuildignore +++ b/.Rbuildignore @@ -12,3 +12,4 @@ ^inst/validation/\.venv$ ^ref$ ^tasks$ +^inst/tinytest/fixtures$ diff --git a/R/fp8_ltx23.R b/R/fp8_ltx23.R index 726c181..ad07264 100644 --- a/R/fp8_ltx23.R +++ b/R/fp8_ltx23.R @@ -104,7 +104,8 @@ ltx23_fp8_linear <- torch::nn_module( #' original key names plus a manifest for skip-if-exists. #' #' @param checkpoint_path Source .safetensors (46 GB bf16 single file). -#' @param output_dir Output directory for shards + manifest. +#' @param output_dir Output directory for shards + manifest; NULL (the +#' default) resolves under tools::R_user_dir("diffuseR", "data"). #' @param shard_bytes Numeric. Target shard size in bytes. The default #' 1.9e9 keeps every shard under the 2^31-byte (~2.15 GB) ceiling that #' stock CRAN safetensors can read. Pass a larger value (e.g. 4e9) only @@ -115,10 +116,13 @@ ltx23_fp8_linear <- torch::nn_module( #' @return Invisibly, the manifest list. #' #' @export -ltx23_quantize_fp8 <- function(checkpoint_path, - output_dir = file.path(tools::R_user_dir("diffuseR", "data"), "ltx2.3-fp8"), +ltx23_quantize_fp8 <- function(checkpoint_path, output_dir = NULL, shard_bytes = 1.9e9, force = FALSE, verbose = TRUE) { + if (is.null(output_dir)) { + output_dir <- file.path(tools::R_user_dir("diffuseR", "data"), + "ltx2.3-fp8") + } manifest_path <- file.path(output_dir, "manifest.json") if (!force && file.exists(manifest_path)) { manifest <- jsonlite::fromJSON(manifest_path) diff --git a/R/nf4_ltx23.R b/R/nf4_ltx23.R index cac2ab4..d2360d6 100644 --- a/R/nf4_ltx23.R +++ b/R/nf4_ltx23.R @@ -251,7 +251,8 @@ ltx23_nf4_linear <- torch::nn_module( #' \code{format = "nf4"}. #' #' @param checkpoint_path Source .safetensors (bf16 single file). -#' @param output_dir Output directory for shards + manifest. +#' @param output_dir Output directory for shards + manifest; NULL (the +#' default) resolves under tools::R_user_dir("diffuseR", "data"). #' @param shard_bytes Numeric. Target shard size in bytes. The default #' 1.9e9 keeps every shard under the 2^31-byte (~2.15 GB) ceiling that #' stock CRAN safetensors can read. Pass a larger value (e.g. 4e9) only @@ -262,10 +263,13 @@ ltx23_nf4_linear <- torch::nn_module( #' @return Invisibly, the manifest list. #' #' @export -ltx23_quantize_nf4 <- function(checkpoint_path, - output_dir = file.path(tools::R_user_dir("diffuseR", "data"), "ltx2.3-nf4"), +ltx23_quantize_nf4 <- function(checkpoint_path, output_dir = NULL, shard_bytes = 1.9e9, force = FALSE, verbose = TRUE) { + if (is.null(output_dir)) { + output_dir <- file.path(tools::R_user_dir("diffuseR", "data"), + "ltx2.3-nf4") + } manifest_path <- file.path(output_dir, "manifest.json") if (!force && file.exists(manifest_path)) { manifest <- jsonlite::fromJSON(manifest_path) diff --git a/R/staging_ltx23.R b/R/staging_ltx23.R index 680ca70..5c95888 100644 --- a/R/staging_ltx23.R +++ b/R/staging_ltx23.R @@ -25,14 +25,6 @@ #' @name staging_ltx23 NULL -#' Pin a component's tensors for fast phase transfer -#' -#' @param module An nn_module on the CPU. -#' -#' @return A list of \code{list(live, pinned)} tensor pairs, or NULL -#' if pinning is unavailable (no CUDA, or page-locking failed). -#' -#' @keywords internal # Allocate pinned host memory without Tensor$pin_memory(): this torch # build's binding requires the deprecated device argument (omitting it # errors with "Expected a torch_device"; passing it prints two libtorch @@ -57,6 +49,14 @@ NULL }) } +#' Pin a component's tensors for fast phase transfer +#' +#' @param module An nn_module on the CPU. +#' +#' @return A list of \code{list(live, pinned)} tensor pairs, or NULL +#' if pinning is unavailable (no CUDA, or page-locking failed). +#' +#' @keywords internal .ltx23_pin_component <- function(module) { if (!torch::cuda_is_available()) { return(NULL) diff --git a/R/text_encoder.R b/R/text_encoder.R index 0d83728..53b6615 100644 --- a/R/text_encoder.R +++ b/R/text_encoder.R @@ -586,6 +586,9 @@ load_text_encoder_weights <- function(native_encoder, torchscript_path, #' @param num_layers Number of transformer layers (default 32) #' @param num_heads Number of attention heads (default 20) #' @param mlp_dim MLP hidden dimension (default 5120) +#' @param return_penultimate Return the penultimate hidden state +#' alongside the pooled output (SDXL's \code{hidden_states[-2]} +#' prompt embeds; the pooled output still comes from the full stack) #' #' @return An nn_module representing the text encoder #' @export diff --git a/README.md b/README.md index 62fc0a8..62b56ef 100644 --- a/README.md +++ b/README.md @@ -11,9 +11,7 @@ ## Example output -![](man/figures/20250602_231518_retro_tin_toy_robot.png) -![](man/figures/20250528_200344_Calvin_and_Hobbes_on_a_beach__Calvin_wearing_a_red.png) -![](man/figures/20250601_111817_Beach_at_night__glowing_waves__stars_in_the_sky__h.png) +![](man/figures/sdxl_vs_flux2_radio_studio.jpg) ## Installation diff --git a/man/dot-ltx23_pin_host.Rd b/man/dot-ltx23_pin_component.Rd similarity index 82% rename from man/dot-ltx23_pin_host.Rd rename to man/dot-ltx23_pin_component.Rd index b3701a7..048aa48 100644 --- a/man/dot-ltx23_pin_host.Rd +++ b/man/dot-ltx23_pin_component.Rd @@ -1,9 +1,9 @@ % tinyrox says don't edit this manually, but it can't stop you! -\name{.ltx23_pin_host} -\alias{.ltx23_pin_host} +\name{.ltx23_pin_component} +\alias{.ltx23_pin_component} \title{Pin a component's tensors for fast phase transfer} \usage{ -.ltx23_pin_host(p) +.ltx23_pin_component(module) } \arguments{ \item{module}{An nn_module on the CPU.} diff --git a/man/ltx23_quantize_fp8.Rd b/man/ltx23_quantize_fp8.Rd index 69d6bc1..06e722e 100644 --- a/man/ltx23_quantize_fp8.Rd +++ b/man/ltx23_quantize_fp8.Rd @@ -3,14 +3,14 @@ \alias{ltx23_quantize_fp8} \title{Quantize an LTX-2.3 checkpoint to FP8 shards} \usage{ -ltx23_quantize_fp8(checkpoint_path, - output_dir = file.path(tools::R_user_dir("diffuseR", "data"), "ltx2.3-fp8"), - shard_bytes = 1.9e+09, force = FALSE, verbose = TRUE) +ltx23_quantize_fp8(checkpoint_path, output_dir = NULL, shard_bytes = 1.9e+09, + force = FALSE, verbose = TRUE) } \arguments{ \item{checkpoint_path}{Source .safetensors (46 GB bf16 single file).} -\item{output_dir}{Output directory for shards + manifest.} +\item{output_dir}{Output directory for shards + manifest; NULL (the +default) resolves under tools::R_user_dir("diffuseR", "data").} \item{shard_bytes}{Numeric. Target shard size in bytes. The default 1.9e9 keeps every shard under the 2^31-byte (~2.15 GB) ceiling that diff --git a/man/ltx23_quantize_nf4.Rd b/man/ltx23_quantize_nf4.Rd index 4d72932..7c169b5 100644 --- a/man/ltx23_quantize_nf4.Rd +++ b/man/ltx23_quantize_nf4.Rd @@ -3,14 +3,14 @@ \alias{ltx23_quantize_nf4} \title{Quantize an LTX-2.3 checkpoint to NF4 shards} \usage{ -ltx23_quantize_nf4(checkpoint_path, - output_dir = file.path(tools::R_user_dir("diffuseR", "data"), "ltx2.3-nf4"), - shard_bytes = 1.9e+09, force = FALSE, verbose = TRUE) +ltx23_quantize_nf4(checkpoint_path, output_dir = NULL, shard_bytes = 1.9e+09, + force = FALSE, verbose = TRUE) } \arguments{ \item{checkpoint_path}{Source .safetensors (bf16 single file).} -\item{output_dir}{Output directory for shards + manifest.} +\item{output_dir}{Output directory for shards + manifest; NULL (the +default) resolves under tools::R_user_dir("diffuseR", "data").} \item{shard_bytes}{Numeric. Target shard size in bytes. The default 1.9e9 keeps every shard under the 2^31-byte (~2.15 GB) ceiling that diff --git a/man/text_encoder2_native.Rd b/man/text_encoder2_native.Rd index 35c7070..f9a7563 100644 --- a/man/text_encoder2_native.Rd +++ b/man/text_encoder2_native.Rd @@ -19,6 +19,10 @@ text_encoder2_native(vocab_size = 49408, context_length = 77, embed_dim = 1280, \item{num_heads}{Number of attention heads (default 20)} \item{mlp_dim}{MLP hidden dimension (default 5120)} + +\item{return_penultimate}{Return the penultimate hidden state +alongside the pooled output (SDXL's \code{hidden_states[-2]} +prompt embeds; the pooled output still comes from the full stack)} } \value{ An nn_module representing the text encoder From 7ac2a3535d2979487bf8850914d21e316f8f19b9 Mon Sep 17 00:00:00 2001 From: TroyHernandez Date: Wed, 22 Jul 2026 18:23:43 -0500 Subject: [PATCH 03/10] serve(): base-socket HTTP server (images + ltx video), systemd unit; consent audit documented --- NAMESPACE | 1 + NEWS.md | 13 ++ R/serve.R | 345 ++++++++++++++++++++++++++++++++++++++++++ inst/diffuser.service | 43 ++++++ man/serve.Rd | 56 +++++++ 5 files changed, 458 insertions(+) create mode 100644 R/serve.R create mode 100644 inst/diffuser.service create mode 100644 man/serve.Rd diff --git a/NAMESPACE b/NAMESPACE index ec55a2b..1dcbfba 100644 --- a/NAMESPACE +++ b/NAMESPACE @@ -189,6 +189,7 @@ export(scheduler_add_noise) export(sd_pipeline_from_safetensors) export(sdxl_memory_profile) export(sdxl_pipeline_from_safetensors) +export(serve) export(t5_encoder) export(text_encoder_native) export(text_encoder_native_from_safetensors) diff --git a/NEWS.md b/NEWS.md index 45d508b..91f9fcc 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,5 +1,18 @@ # diffuseR 0.2.0 +## Serving + +* `serve()`: a zero-dependency HTTP server (base R sockets, one + persistent process, model loaded once) answering OpenAI-style + requests - `/v1/images/generations` for flux2/zimage/flux1, + `/v1/videos/generations` for ltx, `GET /health`. Never downloads + weights; an example systemd unit ships as + `system.file("diffuser.service", package = "diffuseR")`. Port 7812 + in the cornball serve range. +* Every model download is consent-gated: interactive prompt with the + size stated, and non-interactive sessions require + `options(diffuseR.consent = TRUE)`. Generation functions never + download implicitly. ## Native safetensors pipelines * SD 2.1, SDXL, FLUX.1-schnell, FLUX.2 Klein, Z-Image-Turbo, and diff --git a/R/serve.R b/R/serve.R new file mode 100644 index 0000000..c89d03e --- /dev/null +++ b/R/serve.R @@ -0,0 +1,345 @@ +# serve.R +# Minimal HTTP server exposing diffuseR over OpenAI-style endpoints. +# Built on base R sockets (serverSocket/socketAccept) so it adds no +# dependencies and runs as a single persistent process: the model loads +# once and stays resident (no fork, so the CUDA context is never +# invalidated). Requests are served one at a time - the natural fit for +# a single-GPU box. Same skeleton as whisper::serve / chatterbox::serve +# (cornball serve ports: whisper 7809, chatterbox 7810, qwen3 7811, +# diffuseR 7812). + +# Session memo for per-prompt text embeddings (the expensive part of a +# video request; a track's chunks share one prompt) +.dserve_embed_cache <- new.env(parent = emptyenv()) + +#' Serve diffuseR over HTTP +#' +#' Starts a blocking HTTP server that loads one model and answers +#' OpenAI-style generation requests. Never downloads weights: if the +#' model's artifacts are missing, startup stops with the loader's +#' pointer to the explicit \code{download_*()} function. +#' +#' Endpoints: +#' \itemize{ +#' \item \code{GET /health} - liveness probe, returns +#' \code{{"status":"ok","model":...}}. The server is +#' single-threaded, so health only answers between requests. +#' \item \code{POST /v1/images/generations} - image models. JSON body +#' \code{{prompt, size, seed, steps}} (\code{size} like +#' \code{"1024x1024"}; \code{n > 1} is not supported). Returns +#' \code{{created, data: [{b64_json}]}} with a base64 PNG. +#' \item \code{POST /v1/videos/generations} - \code{model = "ltx"} +#' only. JSON body \code{{prompt, width, height, num_frames, +#' frame_rate, seed}}. Returns raw \code{video/mp4} bytes. Video +#' generation takes minutes: give your client a matching timeout. +#' } +#' +#' The server is single-threaded and runs until interrupted. Run it +#' under a process supervisor (systemd, tmux); an example unit ships +#' with the package: \code{system.file("diffuser.service", +#' package = "diffuseR")}. +#' +#' @param port Integer. TCP port. Default 7812 (cornball serve range: whisper 7809, chatterbox 7810, qwen3 TTS 7811). +#' @param model One of "flux2", "zimage", "flux1" (images) or "ltx" +#' (video). SD 2.1/SDXL are not served yet. +#' @param device Character. "cuda" or "cpu". +#' @param timeout Integer. Per-connection I/O timeout in seconds. +#' @param max_body Integer. Maximum request body bytes. Default 1 MB +#' (bodies are JSON). +#' @param warmup Logical. Image models: run one small generation at +#' startup so the first request doesn't pay tracing and allocator +#' growth. Ignored for "ltx". +#' +#' @return Does not return normally; runs until interrupted. +#' @export +serve <- function(port = 7812L, + model = c("flux2", "zimage", "flux1", "ltx"), + device = "cuda", timeout = 300L, + max_body = 1024L^2, warmup = TRUE) { + model <- match.arg(model) + + message("Loading ", model, " on ", device, " (no downloads; run the ", + "download_*() function first if artifacts are missing)...") + state <- .dserve_load(model, device) + message("Model loaded.") + + if (isTRUE(warmup) && model != "ltx") { + message("Warming up...") + tryCatch({ + invisible(state$generate("a red square", width = 256L, + height = 256L, seed = 1L)) + message("Warmup done.") + }, error = function(e) { + message("warmup skipped: ", conditionMessage(e)) + }) + } + + srv <- serverSocket(port) + on.exit(close(srv), add = TRUE) + message("diffuseR::serve listening on port ", port, + " (interrupt to stop)") + + repeat { + con <- tryCatch( + socketAccept(srv, blocking = TRUE, open = "r+b", + timeout = timeout), + error = function(e) { + message("accept error: ", conditionMessage(e)) + Sys.sleep(0.5) + NULL + } + ) + if (is.null(con)) { + next + } + tryCatch({ + req <- .dserve_read_request(con, max_body) + if (!is.null(req)) { + resp <- tryCatch( + .dserve_route(req, state), + error = function(e) { + .dserve_err(500L, conditionMessage(e)) + } + ) + .dserve_send(con, resp$status, resp$content_type, resp$body) + } + }, + error = function(e) message("request error: ", conditionMessage(e)), + finally = { + try(close(con), silent = TRUE) + gc(verbose = FALSE) # bound dead handles; keep the warm pool + }) + } +} + +# Load the served model once; return generate closures + metadata +.dserve_load <- function(model, device) { + if (model == "ltx") { + pipe <- ltx23_load_pipeline( + file.path(tools::R_user_dir("diffuseR", "data"), "ltx2.3-nf4"), + device = device, verbose = FALSE + ) + te_dir <- file.path(tools::R_user_dir("diffuseR", "data"), + "gemma3-nf4") + if (!dir.exists(te_dir)) { + stop("Gemma3 NF4 artifact not found at ", te_dir, + "; run gemma3_quantize_nf4() first.", call. = FALSE) + } + if (!requireNamespace("hfhub", quietly = TRUE)) { + stop("Serving ltx requires the hfhub package (tokenizer files).", + call. = FALSE) + } + tok_dir <- dirname(hfhub::hub_download("Lightricks/LTX-2", + "tokenizer/tokenizer.json", local_files_only = TRUE)) + te <- load_gemma3_text_encoder(te_dir, device = "cpu", + verbose = FALSE) + tok <- gemma3_tokenizer(tok_dir) + embeds <- function(prompt) { + key <- paste0("p:", prompt) + e <- .dserve_embed_cache[[key]] + if (is.null(e)) { + e <- encode_with_gemma3(prompt, model = te, tokenizer = tok, + max_sequence_length = 1024L, + device = if (torch::cuda_is_available()) "cuda" else "cpu", + verbose = FALSE) + e$prompt_embeds <- e$prompt_embeds$to(device = "cpu") + e$prompt_attention_mask <- + e$prompt_attention_mask$to(device = "cpu") + .dserve_embed_cache[[key]] <- e + } + e + } + list(model = model, video = TRUE, pipe = pipe, embeds = embeds, + device = device) + } else { + loader <- switch(model, flux1 = flux_load_pipeline, + flux2 = flux2_load_pipeline, + zimage = zimage_load_pipeline) + pipe <- loader(device = device, verbose = FALSE) + genfn <- switch(model, flux1 = txt2img_flux, flux2 = txt2img_flux2, + zimage = txt2img_zimage) + generate <- function(prompt, width, height, seed = NULL, + steps = NULL) { + args <- list(prompt = prompt, pipeline = pipe, + width = as.integer(width), + height = as.integer(height), seed = seed, + save_file = FALSE, verbose = FALSE) + if (!is.null(steps)) { + args$num_inference_steps <- as.integer(steps) + } + res <- do.call(genfn, args) + # The generators return list(image, metadata) + if (is.list(res) && !is.null(res$image)) { + res$image + } else { + res + } + } + list(model = model, video = FALSE, generate = generate, + device = device) + } +} + +.dserve_route <- function(req, state) { + if (isTRUE(req$too_large)) { + return(.dserve_err(413L, "request body too large")) + } + path <- sub("\\?.*$", "", req$path) + + if (identical(req$method, "GET") && path == "/health") { + return(.dserve_json(list(status = "ok", model = state$model))) + } + if (identical(req$method, "POST") && path == "/v1/images/generations") { + if (isTRUE(state$video)) { + return(.dserve_err(400L, "this server hosts a video model; POST /v1/videos/generations")) + } + return(.dserve_image(req, state)) + } + if (identical(req$method, "POST") && path == "/v1/videos/generations") { + if (!isTRUE(state$video)) { + return(.dserve_err(400L, "this server hosts an image model; POST /v1/images/generations")) + } + return(.dserve_video(req, state)) + } + .dserve_err(404L, "not found") +} + +.dserve_body <- function(req) { + if (!length(req$body)) { + return(NULL) + } + tryCatch(jsonlite::fromJSON(rawToChar(req$body), simplifyVector = TRUE), + error = function(e) NULL) +} + +.dserve_image <- function(req, state) { + body <- .dserve_body(req) + if (is.null(body) || is.null(body$prompt) || !nzchar(body$prompt)) { + return(.dserve_err(400L, "body must be JSON with a prompt")) + } + if (!is.null(body$n) && as.integer(body$n) > 1L) { + return(.dserve_err(400L, "n > 1 is not supported")) + } + size <- if (is.null(body$size)) "1024x1024" else body$size + wh <- suppressWarnings(as.integer(strsplit(size, "x", fixed = TRUE)[[1]])) + if (length(wh) != 2L || anyNA(wh)) { + return(.dserve_err(400L, "size must look like 1024x1024")) + } + seed <- if (is.null(body$seed)) NULL else as.integer(body$seed) + img <- state$generate(body$prompt, width = wh[1], height = wh[2], + seed = seed, steps = body$steps) + png <- png::writePNG(img) + .dserve_json(list( + created = as.integer(Sys.time()), + data = list(list(b64_json = jsonlite::base64_enc(png))) + )) +} + +.dserve_video <- function(req, state) { + body <- .dserve_body(req) + if (is.null(body) || is.null(body$prompt) || !nzchar(body$prompt)) { + return(.dserve_err(400L, "body must be JSON with a prompt")) + } + out <- tempfile(fileext = ".mp4") + on.exit(unlink(out), add = TRUE) + txt2vid_ltx2( + prompt = body$prompt, + pipeline = state$pipe, + prompt_embeds = state$embeds(body$prompt), + width = as.integer(body$width %||% 768L), + height = as.integer(body$height %||% 512L), + num_frames = as.integer(body$num_frames %||% 121L), + frame_rate = as.numeric(body$frame_rate %||% 24), + seed = if (is.null(body$seed)) NULL else as.integer(body$seed), + device = state$device, dtype = "bfloat16", + filename = out, verbose = FALSE + ) + bytes <- readBin(out, "raw", n = file.size(out)) + list(status = 200L, content_type = "video/mp4", body = bytes) +} + +# ---- HTTP plumbing (same shape as whisper::serve) --------------------------- + +.dserve_read_request <- function(con, max_body) { + term <- as.raw(c(13L, 10L, 13L, 10L)) + buf <- raw(0) + max_header <- 65536L + repeat { + b <- readBin(con, "raw", n = 1L) + if (length(b) == 0L) { + return(NULL) + } + buf <- c(buf, b) + n <- length(buf) + if (n >= 4L && identical(buf[(n - 3L):n], term)) { + break + } + if (n > max_header) { + return(NULL) + } + } + + lines <- strsplit(rawToChar(buf), "\r\n", fixed = TRUE)[[1]] + req_line <- strsplit(lines[1], " ", fixed = TRUE)[[1]] + if (length(req_line) < 2L) { + return(NULL) + } + hdr <- list() + if (length(lines) > 1L) { + for (ln in lines[-1L]) { + if (!nzchar(ln)) { + next + } + pos <- regexpr(":", ln, fixed = TRUE) + if (pos < 1L) { + next + } + hdr[[tolower(trimws(substr(ln, 1L, pos - 1L)))]] <- + trimws(substr(ln, pos + 1L, nchar(ln))) + } + } + clen <- suppressWarnings(as.integer(hdr[["content-length"]] %||% "0")) + if (length(clen) != 1L || is.na(clen) || clen < 0L) { + clen <- 0L + } + if (clen > max_body) { + return(list(method = req_line[1], path = req_line[2], headers = hdr, + body = raw(0), too_large = TRUE)) + } + body <- raw(0) + while (length(body) < clen) { + chunk <- readBin(con, "raw", n = clen - length(body)) + if (length(chunk) == 0L) { + break + } + body <- c(body, chunk) + } + list(method = req_line[1], path = req_line[2], headers = hdr, body = body) +} + +.dserve_send <- function(con, status, content_type, body) { + if (is.character(body)) { + body <- charToRaw(enc2utf8(body)) + } + reason <- switch(as.character(status), "200" = "OK", + "400" = "Bad Request", "404" = "Not Found", + "405" = "Method Not Allowed", + "413" = "Payload Too Large", + "500" = "Internal Server Error", "Unknown") + head <- paste0( + sprintf("HTTP/1.1 %d %s\r\n", status, reason), + sprintf("Content-Type: %s\r\n", content_type), + sprintf("Content-Length: %d\r\n", length(body)), + "Connection: close\r\n\r\n") + writeBin(c(charToRaw(head), body), con) + flush(con) +} + +.dserve_json <- function(x, status = 200L) { + list(status = status, content_type = "application/json", + body = jsonlite::toJSON(x, auto_unbox = TRUE)) +} + +.dserve_err <- function(status, msg) { + .dserve_json(list(error = list(message = msg)), status = status) +} diff --git a/inst/diffuser.service b/inst/diffuser.service new file mode 100644 index 0000000..75be441 --- /dev/null +++ b/inst/diffuser.service @@ -0,0 +1,43 @@ +# Example systemd unit for diffuseR::serve(). +# +# Serves one diffuseR model over HTTP as a persistent service (model +# loaded once, auto-restart on failure, starts on boot). Copy from +# system.file("diffuser.service", package = "diffuseR") +# edit the CHANGEME placeholders, then install: +# +# sudo cp diffuser.service /etc/systemd/system/diffuser.service +# sudo systemctl daemon-reload +# sudo systemctl enable --now diffuser.service +# journalctl -u diffuser.service -f +# +# The server never downloads weights: run the model's download_*() +# function once as the service user before first start. +# +# Port 7812 sits in the cornball serve range (7809-7829): whisper 7809, +# chatterbox 7810, qwen3 TTS 7811. +# +# Video serving (model = "ltx") answers /v1/videos/generations and takes +# minutes per request - point clients at it with matching timeouts. + +[Unit] +Description=diffuseR generation server (R package, base-socket HTTP) +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +# The user that owns the R library and model artifacts: +User=CHANGEME +Group=CHANGEME +WorkingDirectory=/home/CHANGEME +# littler (/usr/bin/r) does not read ~/.Renviron/.Rprofile, so set what +# the server needs explicitly: +Environment=HOME=/home/CHANGEME +ExecStart=/usr/bin/r -e "diffuseR::serve(model = 'flux2', port = 7812L)" +Restart=on-failure +RestartSec=5 +# Pipeline load + warmup take a while; don't let a watchdog kill startup. +TimeoutStartSec=300 + +[Install] +WantedBy=multi-user.target diff --git a/man/serve.Rd b/man/serve.Rd new file mode 100644 index 0000000..29d7ec8 --- /dev/null +++ b/man/serve.Rd @@ -0,0 +1,56 @@ +% tinyrox says don't edit this manually, but it can't stop you! +\name{serve} +\alias{serve} +\title{Serve diffuseR over HTTP} +\usage{ +serve(port = 7812L, model = c("flux2", "zimage", "flux1", "ltx"), + device = "cuda", timeout = 300L, max_body = 1024L^2, warmup = TRUE) +} +\arguments{ +\item{port}{Integer. TCP port. Default 7812 (cornball serve range: whisper 7809, chatterbox 7810, qwen3 TTS 7811).} + +\item{model}{One of "flux2", "zimage", "flux1" (images) or "ltx" +(video). SD 2.1/SDXL are not served yet.} + +\item{device}{Character. "cuda" or "cpu".} + +\item{timeout}{Integer. Per-connection I/O timeout in seconds.} + +\item{max_body}{Integer. Maximum request body bytes. Default 1 MB +(bodies are JSON).} + +\item{warmup}{Logical. Image models: run one small generation at +startup so the first request doesn't pay tracing and allocator +growth. Ignored for "ltx".} +} +\value{ +Does not return normally; runs until interrupted. +} +\description{ +Starts a blocking HTTP server that loads one model and answers +OpenAI-style generation requests. Never downloads weights: if the +model's artifacts are missing, startup stops with the loader's +pointer to the explicit \code{download_*()} function. +} +\details{ +Endpoints: +\itemize{ +\item \code{GET /health} - liveness probe, returns +\code{{"status":"ok","model":...}}. The server is +single-threaded, so health only answers between requests. +\item \code{POST /v1/images/generations} - image models. JSON body +\code{{prompt, size, seed, steps}} (\code{size} like +\code{"1024x1024"}; \code{n > 1} is not supported). Returns +\code{{created, data: [{b64_json}]}} with a base64 PNG. +\item \code{POST /v1/videos/generations} - \code{model = "ltx"} +only. JSON body \code{{prompt, width, height, num_frames, +frame_rate, seed}}. Returns raw \code{video/mp4} bytes. Video +generation takes minutes: give your client a matching timeout. +} + +The server is single-threaded and runs until interrupted. Run it +under a process supervisor (systemd, tmux); an example unit ships +with the package: \code{system.file("diffuser.service", +package = "diffuseR")}. + +} From 9ded3dd9cdbb159e35e91d41b3c7f38f5ac578f2 Mon Sep 17 00:00:00 2001 From: TroyHernandez Date: Wed, 22 Jul 2026 18:39:03 -0500 Subject: [PATCH 04/10] NULL-default the long constructor defaults (Rd width note) --- R/vae_ltx23.R | 27 ++++++++++++++++++++------- R/vocoder_ltx23.R | 12 +++++++++--- man/ltx23_video_decoder3d.Rd | 3 +-- man/ltx23_video_encoder3d.Rd | 7 +++---- man/ltx23_video_vae.Rd | 3 +-- man/ltx23_vocoder_with_bwe.Rd | 11 +++++------ 6 files changed, 39 insertions(+), 24 deletions(-) diff --git a/R/vae_ltx23.R b/R/vae_ltx23.R index 1428a11..f9d2201 100644 --- a/R/vae_ltx23.R +++ b/R/vae_ltx23.R @@ -35,14 +35,17 @@ ltx23_video_encoder3d <- torch::nn_module( block_out_channels = c(256L, 512L, 1024L, 1024L), spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE), layers_per_block = c(4L, 6L, 4L, 2L, 2L), - downsample_type = c("spatial", "temporal", "spatiotemporal", - "spatiotemporal"), + downsample_type = NULL, patch_size = 4L, patch_size_t = 1L, resnet_norm_eps = 1e-6, is_causal = TRUE, spatial_padding_mode = "zeros" ) { + if (is.null(downsample_type)) { + downsample_type <- c("spatial", "temporal", "spatiotemporal", + "spatiotemporal") + } self$patch_size <- as.integer(patch_size) self$patch_size_t <- as.integer(patch_size_t) self$in_channels <- in_channels * patch_size ^ 2 @@ -145,8 +148,7 @@ ltx23_video_decoder3d <- torch::nn_module( block_out_channels = c(256L, 512L, 512L, 1024L), spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE), layers_per_block = c(4L, 6L, 4L, 2L, 2L), - upsample_type = c("spatiotemporal", "spatiotemporal", "temporal", - "spatial"), + upsample_type = NULL, patch_size = 4L, patch_size_t = 1L, resnet_norm_eps = 1e-6, @@ -155,6 +157,10 @@ ltx23_video_decoder3d <- torch::nn_module( upsample_factor = c(2L, 2L, 1L, 2L), spatial_padding_mode = "zeros" ) { + if (is.null(upsample_type)) { + upsample_type <- c("spatiotemporal", "spatiotemporal", "temporal", + "spatial") + } self$patch_size <- as.integer(patch_size) self$patch_size_t <- as.integer(patch_size_t) self$out_channels <- out_channels * patch_size ^ 2 @@ -263,9 +269,8 @@ ltx23_video_vae <- torch::nn_module( decoder_layers_per_block = c(4L, 6L, 4L, 2L, 2L), spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE), decoder_spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE), - downsample_type = c("spatial", "temporal", "spatiotemporal", - "spatiotemporal"), - upsample_type = c("spatiotemporal", "spatiotemporal", "temporal", "spatial"), + downsample_type = NULL, + upsample_type = NULL, upsample_residual = c(FALSE, FALSE, FALSE, FALSE), upsample_factor = c(2L, 2L, 1L, 2L), patch_size = 4L, @@ -276,6 +281,14 @@ ltx23_video_vae <- torch::nn_module( encoder_spatial_padding_mode = "zeros", decoder_spatial_padding_mode = "zeros" ) { + if (is.null(downsample_type)) { + downsample_type <- c("spatial", "temporal", "spatiotemporal", + "spatiotemporal") + } + if (is.null(upsample_type)) { + upsample_type <- c("spatiotemporal", "spatiotemporal", "temporal", + "spatial") + } self$latent_channels <- as.integer(latent_channels) self$encoder <- ltx23_video_encoder3d( diff --git a/R/vocoder_ltx23.R b/R/vocoder_ltx23.R index 35c0a29..765b2da 100644 --- a/R/vocoder_ltx23.R +++ b/R/vocoder_ltx23.R @@ -417,14 +417,13 @@ ltx23_vocoder_with_bwe <- torch::nn_module( upsample_kernel_sizes = c(11L, 4L, 4L, 4L, 4L, 4L), upsample_factors = c(5L, 2L, 2L, 2L, 2L, 2L), resnet_kernel_sizes = c(3L, 7L, 11L), - resnet_dilations = list(c(1L, 3L, 5L), c(1L, 3L, 5L), c(1L, 3L, 5L)), + resnet_dilations = NULL, bwe_in_channels = 128L, bwe_hidden_channels = 512L, bwe_upsample_kernel_sizes = c(12L, 11L, 4L, 4L, 4L), bwe_upsample_factors = c(6L, 5L, 2L, 2L, 2L), bwe_resnet_kernel_sizes = c(3L, 7L, 11L), - bwe_resnet_dilations = list(c(1L, 3L, 5L), c(1L, 3L, 5L), - c(1L, 3L, 5L)), + bwe_resnet_dilations = NULL, filter_length = 512L, hop_length = 80L, window_length = 512L, @@ -432,6 +431,13 @@ ltx23_vocoder_with_bwe <- torch::nn_module( input_sampling_rate = 16000L, output_sampling_rate = 48000L ) { + if (is.null(resnet_dilations)) { + resnet_dilations <- list(c(1L, 3L, 5L), c(1L, 3L, 5L), c(1L, 3L, 5L)) + } + if (is.null(bwe_resnet_dilations)) { + bwe_resnet_dilations <- list(c(1L, 3L, 5L), c(1L, 3L, 5L), + c(1L, 3L, 5L)) + } self$hop_length <- as.integer(hop_length) self$rate_ratio <- as.integer(output_sampling_rate %/% input_sampling_rate) diff --git a/man/ltx23_video_decoder3d.Rd b/man/ltx23_video_decoder3d.Rd index 7b34f94..818ee26 100644 --- a/man/ltx23_video_decoder3d.Rd +++ b/man/ltx23_video_decoder3d.Rd @@ -7,8 +7,7 @@ ltx23_video_decoder3d(in_channels = 128L, out_channels = 3L, block_out_channels = c(256L, 512L, 512L, 1024L), spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE), layers_per_block = c(4L, 6L, 4L, 2L, 2L), - upsample_type = c("spatiotemporal", "spatiotemporal", "temporal", "spatial"), - patch_size = 4L, patch_size_t = 1L, + upsample_type = NULL, patch_size = 4L, patch_size_t = 1L, resnet_norm_eps = 1e-06, is_causal = FALSE, upsample_residual = c(FALSE, FALSE, FALSE, FALSE), upsample_factor = c(2L, 2L, 1L, 2L), diff --git a/man/ltx23_video_encoder3d.Rd b/man/ltx23_video_encoder3d.Rd index ca384a9..e3ddacc 100644 --- a/man/ltx23_video_encoder3d.Rd +++ b/man/ltx23_video_encoder3d.Rd @@ -7,10 +7,9 @@ ltx23_video_encoder3d(in_channels = 3L, out_channels = 128L, block_out_channels = c(256L, 512L, 1024L, 1024L), spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE), layers_per_block = c(4L, 6L, 4L, 2L, 2L), - downsample_type = c("spatial", "temporal", "spatiotemporal", "spatiotemporal"), - patch_size = 4L, patch_size_t = 1L, - resnet_norm_eps = 1e-06, is_causal = TRUE, - spatial_padding_mode = "zeros") + downsample_type = NULL, patch_size = 4L, + patch_size_t = 1L, resnet_norm_eps = 1e-06, + is_causal = TRUE, spatial_padding_mode = "zeros") } \arguments{ \item{block_out_channels}{Integer vector. Per-block output channels.} diff --git a/man/ltx23_video_vae.Rd b/man/ltx23_video_vae.Rd index bc3226a..6f4d047 100644 --- a/man/ltx23_video_vae.Rd +++ b/man/ltx23_video_vae.Rd @@ -10,8 +10,7 @@ ltx23_video_vae(in_channels = 3L, out_channels = 3L, latent_channels = 128L, decoder_layers_per_block = c(4L, 6L, 4L, 2L, 2L), spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE), decoder_spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE), - downsample_type = c("spatial", "temporal", "spatiotemporal", "spatiotemporal"), - upsample_type = c("spatiotemporal", "spatiotemporal", "temporal", "spatial"), + downsample_type = NULL, upsample_type = NULL, upsample_residual = c(FALSE, FALSE, FALSE, FALSE), upsample_factor = c(2L, 2L, 1L, 2L), patch_size = 4L, patch_size_t = 1L, resnet_norm_eps = 1e-06, diff --git a/man/ltx23_vocoder_with_bwe.Rd b/man/ltx23_vocoder_with_bwe.Rd index 81550b5..4cb2f6b 100644 --- a/man/ltx23_vocoder_with_bwe.Rd +++ b/man/ltx23_vocoder_with_bwe.Rd @@ -8,15 +8,14 @@ ltx23_vocoder_with_bwe(in_channels = 128L, hidden_channels = 1536L, upsample_kernel_sizes = c(11L, 4L, 4L, 4L, 4L, 4L), upsample_factors = c(5L, 2L, 2L, 2L, 2L, 2L), resnet_kernel_sizes = c(3L, 7L, 11L), - resnet_dilations = list(c(1L, 3L, 5L), c(1L, 3L, 5L), c(1L, 3L, 5L)), - bwe_in_channels = 128L, bwe_hidden_channels = 512L, + resnet_dilations = NULL, bwe_in_channels = 128L, + bwe_hidden_channels = 512L, bwe_upsample_kernel_sizes = c(12L, 11L, 4L, 4L, 4L), bwe_upsample_factors = c(6L, 5L, 2L, 2L, 2L), bwe_resnet_kernel_sizes = c(3L, 7L, 11L), - bwe_resnet_dilations = list(c(1L, 3L, 5L), c(1L, 3L, 5L), c(1L, 3L, 5L)), - filter_length = 512L, hop_length = 80L, - window_length = 512L, num_mel_channels = 64L, - input_sampling_rate = 16000L, + bwe_resnet_dilations = NULL, filter_length = 512L, + hop_length = 80L, window_length = 512L, + num_mel_channels = 64L, input_sampling_rate = 16000L, output_sampling_rate = 48000L) } \arguments{ From ad3071f7be05073ab12e20ce681989fd0ebada23 Mon Sep 17 00:00:00 2001 From: TroyHernandez Date: Wed, 22 Jul 2026 20:36:36 -0500 Subject: [PATCH 05/10] serve hardening: bounded connector-embed LRU, token auth, size limits, OOM exit, artifact resolution, device respect; consent gates on legacy download paths; vignette accuracy; serve tests --- NEWS.md | 6 +- R/download_component.R | 19 +++-- R/download_model.R | 14 ++-- R/serve.R | 142 +++++++++++++++++++++++++------- README.md | 2 +- inst/tinytest/test_serve.R | 109 ++++++++++++++++++++++++ man/serve.Rd | 24 +++++- vignettes/performance-levers.md | 13 ++- 8 files changed, 278 insertions(+), 51 deletions(-) create mode 100644 inst/tinytest/test_serve.R diff --git a/NEWS.md b/NEWS.md index 91f9fcc..143441c 100644 --- a/NEWS.md +++ b/NEWS.md @@ -8,7 +8,11 @@ `/v1/videos/generations` for ltx, `GET /health`. Never downloads weights; an example systemd unit ships as `system.file("diffuser.service", package = "diffuseR")`. Port 7812 - in the cornball serve range. + in the cornball serve range. Hardened for + persistence: optional bearer-token auth, hard pixel/frame limits + (400 on oversize), a bounded LRU of per-prompt connector embeds + (~9 MB each, never the raw Gemma stacks), and a clean process exit + on CUDA OOM so a supervisor restarts with sane GPU state. * Every model download is consent-gated: interactive prompt with the size stated, and non-interactive sessions require `options(diffuseR.consent = TRUE)`. Generation functions never diff --git a/R/download_component.R b/R/download_component.R index 142ff14..2f8b026 100644 --- a/R/download_component.R +++ b/R/download_component.R @@ -20,16 +20,23 @@ download_component <- function(model_name = "sd21", component, device = "cpu", overwrite = FALSE, show_progress = TRUE) { filename <- paste0(component, "-", device, ".pt") + if (!requireNamespace("hfhub", quietly = TRUE)) { + stop("Package 'hfhub' is required. Install with: install.packages('hfhub')") + } + repo_id <- paste0("cornball-ai/", model_name, "-R") + + cached <- !overwrite && !is.null(tryCatch( + hfhub::hub_download(repo_id, filename, repo_type = "dataset", + local_files_only = TRUE), + error = function(e) NULL)) + if (!cached && + !.ltx23_consent(paste0(model_name, " ", component, " (", filename, ")"))) { + stop("Download cancelled.", call. = FALSE) + } if (overwrite) { - # Force download (bypass cache) - if (!requireNamespace("hfhub", quietly = TRUE)) { - stop("Package 'hfhub' is required. Install with: install.packages('hfhub')") - } - repo_id <- paste0("cornball-ai/", model_name, "-R") return(hfhub::hub_download(repo_id, filename, repo_type = "dataset", force_download = TRUE)) } - hf_download_pt(model_name, filename, download = TRUE) } diff --git a/R/download_model.R b/R/download_model.R index e84d288..8134881 100644 --- a/R/download_model.R +++ b/R/download_model.R @@ -154,16 +154,12 @@ download_model <- function(model_name = "sd21", needs_download <- !already_available | overwrite - # Interactive consent when files actually need downloading + # Consent when files actually need downloading: interactive prompt, + # or the diffuseR.consent option in non-interactive sessions if (download_models && any(needs_download)) { - if (interactive()) { - ans <- utils::askYesNo( - paste0("Download '", model_name, "' model files from HuggingFace?"), - default = TRUE - ) - if (!isTRUE(ans)) { - stop("Download cancelled.", call. = FALSE) - } + if (!.ltx23_consent(paste0("'", model_name, + "' model files from HuggingFace"))) { + stop("Download cancelled.", call. = FALSE) } } diff --git a/R/serve.R b/R/serve.R index c89d03e..ce82640 100644 --- a/R/serve.R +++ b/R/serve.R @@ -8,9 +8,10 @@ # (cornball serve ports: whisper 7809, chatterbox 7810, qwen3 7811, # diffuseR 7812). -# Session memo for per-prompt text embeddings (the expensive part of a -# video request; a track's chunks share one prompt) -.dserve_embed_cache <- new.env(parent = emptyenv()) +# (Per-prompt caches live in server state, created per serve() call: +# a bounded LRU of connector outputs (~9 MB each), never the raw +# Gemma stacks (~0.4 GB each) - a persistent server must not grow +# with prompt count.) #' Serve diffuseR over HTTP #' @@ -39,10 +40,27 @@ #' with the package: \code{system.file("diffuser.service", #' package = "diffuseR")}. #' +#' Security: base R's \code{serverSocket} binds all interfaces, so the +#' server is reachable by anything that can reach the machine. Keep it +#' behind a firewall or reverse proxy, and/or set \code{token}: when +#' set, every request must carry \code{Authorization: Bearer } +#' or it is refused with 401. Generation size is capped by +#' \code{max_pixels}/\code{max_frames}; oversized requests get 400. A +#' CUDA out-of-memory during a request answers 500 and then exits the +#' process (status 70) so a supervisor restarts it with clean GPU +#' state rather than serving on with stranded components. +#' #' @param port Integer. TCP port. Default 7812 (cornball serve range: whisper 7809, chatterbox 7810, qwen3 TTS 7811). #' @param model One of "flux2", "zimage", "flux1" (images) or "ltx" #' (video). SD 2.1/SDXL are not served yet. #' @param device Character. "cuda" or "cpu". +#' @param token Character or NULL. Shared secret; when set, requests +#' must send \code{Authorization: Bearer }. +#' @param max_pixels Integer. Maximum width x height accepted (images +#' and video frames). Default 1024^2. +#' @param max_frames Integer. Maximum video frame count. Default 161. +#' @param max_prompts Integer. Bound on the per-prompt connector-embed +#' cache for "ltx" (~9 MB per entry, LRU-evicted). Default 32. #' @param timeout Integer. Per-connection I/O timeout in seconds. #' @param max_body Integer. Maximum request body bytes. Default 1 MB #' (bodies are JSON). @@ -54,13 +72,18 @@ #' @export serve <- function(port = 7812L, model = c("flux2", "zimage", "flux1", "ltx"), - device = "cuda", timeout = 300L, + device = "cuda", token = NULL, + max_pixels = 1024L^2, max_frames = 161L, + max_prompts = 32L, timeout = 300L, max_body = 1024L^2, warmup = TRUE) { model <- match.arg(model) message("Loading ", model, " on ", device, " (no downloads; run the ", "download_*() function first if artifacts are missing)...") - state <- .dserve_load(model, device) + state <- .dserve_load(model, device, max_prompts = max_prompts) + state$token <- token + state$max_pixels <- as.integer(max_pixels) + state$max_frames <- as.integer(max_frames) message("Model loaded.") if (isTRUE(warmup) && model != "ltx") { @@ -92,13 +115,20 @@ serve <- function(port = 7812L, if (is.null(con)) { next } + resp <- NULL tryCatch({ req <- .dserve_read_request(con, max_body) if (!is.null(req)) { resp <- tryCatch( .dserve_route(req, state), error = function(e) { - .dserve_err(500L, conditionMessage(e)) + r <- .dserve_err(500L, conditionMessage(e)) + if (grepl("CUDA out of memory", + conditionMessage(e), + fixed = TRUE)) { + attr(r, "fatal") <- TRUE + } + r } ) .dserve_send(con, resp$status, resp$content_type, resp$body) @@ -109,21 +139,34 @@ serve <- function(port = 7812L, try(close(con), silent = TRUE) gc(verbose = FALSE) # bound dead handles; keep the warm pool }) + if (exists("resp", inherits = FALSE) && isTRUE(attr(resp, "fatal"))) { + message("CUDA out of memory: exiting for a clean supervisor restart") + quit(save = "no", status = 70L) + } } } # Load the served model once; return generate closures + metadata -.dserve_load <- function(model, device) { +.dserve_load <- function(model, device, max_prompts = 32L) { if (model == "ltx") { - pipe <- ltx23_load_pipeline( - file.path(tools::R_user_dir("diffuseR", "data"), "ltx2.3-nf4"), - device = device, verbose = FALSE - ) - te_dir <- file.path(tools::R_user_dir("diffuseR", "data"), - "gemma3-nf4") + data_dir <- tools::R_user_dir("diffuseR", "data") + # Serve whichever LTX artifact is built (nf4 preferred; fp8 is + # what download_ltx2() produces by default) + ckpt_dir <- Filter(dir.exists, + file.path(data_dir, c("ltx2.3-nf4", "ltx2.3-fp8"))) + if (!length(ckpt_dir)) { + stop("No LTX-2.3 artifact under ", data_dir, + "; run download_ltx2() (fp8) or ltx23_quantize_nf4() first.", + call. = FALSE) + } + pipe <- ltx23_load_pipeline(ckpt_dir[[1]], device = device, + verbose = FALSE) + te_dir <- file.path(data_dir, "gemma3-nf4") if (!dir.exists(te_dir)) { stop("Gemma3 NF4 artifact not found at ", te_dir, - "; run gemma3_quantize_nf4() first.", call. = FALSE) + "; run gemma3_quantize_nf4() first (serving loads the ", + "quantized encoder, not the raw fp32 weights).", + call. = FALSE) } if (!requireNamespace("hfhub", quietly = TRUE)) { stop("Serving ltx requires the hfhub package (tokenizer files).", @@ -134,19 +177,41 @@ serve <- function(port = 7812L, te <- load_gemma3_text_encoder(te_dir, device = "cpu", verbose = FALSE) tok <- gemma3_tokenizer(tok_dir) + enc_dev <- if (identical(device, "cuda") && + torch::cuda_is_available()) { + "cuda" + } else { + "cpu" + } + # Bounded LRU of CONNECTOR outputs (~9 MB each): the raw Gemma + # stack (~0.4 GB) is encoded, projected, and dropped per prompt + cache <- new.env(parent = emptyenv()) + order <- character(0) embeds <- function(prompt) { key <- paste0("p:", prompt) - e <- .dserve_embed_cache[[key]] - if (is.null(e)) { - e <- encode_with_gemma3(prompt, model = te, tokenizer = tok, - max_sequence_length = 1024L, - device = if (torch::cuda_is_available()) "cuda" else "cpu", - verbose = FALSE) - e$prompt_embeds <- e$prompt_embeds$to(device = "cpu") - e$prompt_attention_mask <- - e$prompt_attention_mask$to(device = "cpu") - .dserve_embed_cache[[key]] <- e + e <- cache[[key]] + if (!is.null(e)) { + order <<- c(setdiff(order, key), key) + return(e) + } + raw <- encode_with_gemma3(prompt, model = te, tokenizer = tok, + max_sequence_length = 1024L, + device = enc_dev, verbose = FALSE) + conn <- torch::with_no_grad(pipe$connectors( + raw$prompt_embeds$to(device = "cpu", + dtype = torch::torch_bfloat16()), + raw$prompt_attention_mask$to(device = "cpu"))) + e <- list(video_text_embedding = conn$video_text_embedding, + audio_text_embedding = conn$audio_text_embedding, + attention_mask = conn$attention_mask) + rm(raw, conn) + cache[[key]] <- e + order <<- c(order, key) + while (length(order) > max_prompts) { + rm(list = order[[1]], envir = cache) + order <<- order[-1] } + gc(verbose = FALSE) e } list(model = model, video = TRUE, pipe = pipe, embeds = embeds, @@ -184,6 +249,12 @@ serve <- function(port = 7812L, if (isTRUE(req$too_large)) { return(.dserve_err(413L, "request body too large")) } + if (!is.null(state$token)) { + auth <- req$headers[["authorization"]] + if (!identical(auth, paste("Bearer", state$token))) { + return(.dserve_err(401L, "unauthorized")) + } + } path <- sub("\\?.*$", "", req$path) if (identical(req$method, "GET") && path == "/health") { @@ -225,6 +296,10 @@ serve <- function(port = 7812L, if (length(wh) != 2L || anyNA(wh)) { return(.dserve_err(400L, "size must look like 1024x1024")) } + if (wh[1] * wh[2] > state$max_pixels) { + return(.dserve_err(400L, sprintf( + "request exceeds limits (max %d pixels)", state$max_pixels))) + } seed <- if (is.null(body$seed)) NULL else as.integer(body$seed) img <- state$generate(body$prompt, width = wh[1], height = wh[2], seed = seed, steps = body$steps) @@ -240,15 +315,23 @@ serve <- function(port = 7812L, if (is.null(body) || is.null(body$prompt) || !nzchar(body$prompt)) { return(.dserve_err(400L, "body must be JSON with a prompt")) } + w <- as.integer(body$width %||% 768L) + h <- as.integer(body$height %||% 512L) + nf <- as.integer(body$num_frames %||% 121L) + if (w * h > state$max_pixels || nf > state$max_frames) { + return(.dserve_err(400L, sprintf( + "request exceeds limits (max %d pixels, %d frames)", + state$max_pixels, state$max_frames))) + } out <- tempfile(fileext = ".mp4") on.exit(unlink(out), add = TRUE) txt2vid_ltx2( prompt = body$prompt, pipeline = state$pipe, - prompt_embeds = state$embeds(body$prompt), - width = as.integer(body$width %||% 768L), - height = as.integer(body$height %||% 512L), - num_frames = as.integer(body$num_frames %||% 121L), + connector_embeds = state$embeds(body$prompt), + width = w, + height = h, + num_frames = nf, frame_rate = as.numeric(body$frame_rate %||% 24), seed = if (is.null(body$seed)) NULL else as.integer(body$seed), device = state$device, dtype = "bfloat16", @@ -322,7 +405,8 @@ serve <- function(port = 7812L, body <- charToRaw(enc2utf8(body)) } reason <- switch(as.character(status), "200" = "OK", - "400" = "Bad Request", "404" = "Not Found", + "400" = "Bad Request", "401" = "Unauthorized", + "404" = "Not Found", "405" = "Method Not Allowed", "413" = "Payload Too Large", "500" = "Internal Server Error", "Unknown") diff --git a/README.md b/README.md index 62b56ef..7a36890 100644 --- a/README.md +++ b/README.md @@ -243,7 +243,7 @@ so negative prompts do not apply. When the image needs legible text ### Downloading Models -Models are automatically downloaded from HuggingFace on first use. For gated models (like FLUX.1-schnell), you need to: +Model downloads are explicit and consent-gated: call the model's `download_*()` function once (interactive sessions are prompted with the size; non-interactive sessions require `options(diffuseR.consent = TRUE)`). For gated models (like FLUX.1-schnell), you need to: 1. Create a HuggingFace account at https://huggingface.co 2. Accept the model's license agreement (visit the model page and click "Agree") diff --git a/inst/tinytest/test_serve.R b/inst/tinytest/test_serve.R new file mode 100644 index 0000000..50b0b85 --- /dev/null +++ b/inst/tinytest/test_serve.R @@ -0,0 +1,109 @@ +# serve(): HTTP parser, router, auth, and limits - pure R, no sockets, +# no models (the generate closure is faked). + +library(diffuseR) + +mkreq <- function(method, path, body = "", headers = character(0)) { + raw_body <- charToRaw(body) + # NB: paste0(character(0), "\r\n", collapse = "") is "\r\n", not "" + # (with collapse set, zero-length args become "") - hence the guard + hdr_block <- if (length(headers)) { + paste0(paste0(headers, "\r\n"), collapse = "") + } else { + "" + } + head <- paste0(method, " ", path, " HTTP/1.1\r\n", + "Content-Length: ", length(raw_body), "\r\n", + hdr_block, "\r\n") + con <- rawConnection(c(charToRaw(head), raw_body), open = "rb") + on.exit(close(con)) + diffuseR:::.dserve_read_request(con, max_body = 1024L^2) +} +jbody <- function(resp) jsonlite::fromJSON(as.character(resp$body), simplifyVector = FALSE) + +# --- parser ------------------------------------------------------------------------ + +req <- mkreq("GET", "/health") +expect_equal(req$method, "GET") +expect_equal(req$path, "/health") +expect_equal(length(req$body), 0L) + +req <- mkreq("POST", "/x", body = '{"a":1}', + headers = "Content-Type: application/json") +expect_equal(rawToChar(req$body), '{"a":1}') +expect_equal(req$headers[["content-type"]], "application/json") + +# too-large bodies are flagged, not read. (Assign the raw first: +# rawConnection uses deparse(substitute(object)) as the connection +# name, and a long inline expression deparses to length > 1.) +big_bytes <- charToRaw(paste0( + "POST /x HTTP/1.1\r\nContent-Length: 100\r\n\r\n", + paste(rep("a", 100), collapse = ""))) +con <- rawConnection(big_bytes, open = "rb") +big <- diffuseR:::.dserve_read_request(con, max_body = 4L) +close(con) +expect_true(isTRUE(big$too_large)) + +# --- router: health, 404, auth ----------------------------------------------------- + +state <- list(model = "fake", video = FALSE, + max_pixels = 1024L^2, max_frames = 161L) + +r <- diffuseR:::.dserve_route(mkreq("GET", "/health"), state) +expect_equal(r$status, 200L) +expect_equal(jbody(r)$model, "fake") + +expect_equal(diffuseR:::.dserve_route(mkreq("GET", "/nope"), state)$status, + 404L) + +auth_state <- state +auth_state$token <- "s3cret" +expect_equal(diffuseR:::.dserve_route(mkreq("GET", "/health"), + auth_state)$status, 401L) +ok <- diffuseR:::.dserve_route( + mkreq("GET", "/health", headers = "Authorization: Bearer s3cret"), + auth_state) +expect_equal(ok$status, 200L) + +# --- image handler: caps, validation, b64 round trip ------------------------------- + +state$generate <- function(prompt, width, height, seed = NULL, steps = NULL) { + array(runif(height * width * 3), dim = c(height, width, 3L)) +} + +bad <- diffuseR:::.dserve_route( + mkreq("POST", "/v1/images/generations", body = '{"size":"64x64"}'), state) +expect_equal(bad$status, 400L) # missing prompt + +over <- diffuseR:::.dserve_route( + mkreq("POST", "/v1/images/generations", + body = '{"prompt":"x","size":"4096x4096"}'), state) +expect_equal(over$status, 400L) +expect_true(grepl("exceeds limits", jbody(over)$error$message)) + +good <- diffuseR:::.dserve_route( + mkreq("POST", "/v1/images/generations", + body = '{"prompt":"x","size":"64x64","seed":1}'), state) +expect_equal(good$status, 200L) +png_bytes <- jsonlite::base64_dec(jbody(good)$data[[1]]$b64_json) +expect_identical(png_bytes[1:4], as.raw(c(0x89, 0x50, 0x4e, 0x47))) +img <- png::readPNG(png_bytes) +expect_equal(dim(img)[1:2], c(64L, 64L)) + +# wrong endpoint for the hosted model type +expect_equal(diffuseR:::.dserve_route( + mkreq("POST", "/v1/videos/generations", body = '{"prompt":"x"}'), + state)$status, 400L) + +# --- video caps (no model needed: limits check precedes generation) ---------------- + +vstate <- list(model = "fake", video = TRUE, + max_pixels = 1024L^2, max_frames = 161L) +overv <- diffuseR:::.dserve_route( + mkreq("POST", "/v1/videos/generations", + body = '{"prompt":"x","width":2048,"height":2048}'), vstate) +expect_equal(overv$status, 400L) +overf <- diffuseR:::.dserve_route( + mkreq("POST", "/v1/videos/generations", + body = '{"prompt":"x","num_frames":9999}'), vstate) +expect_equal(overf$status, 400L) diff --git a/man/serve.Rd b/man/serve.Rd index 29d7ec8..6fdf68a 100644 --- a/man/serve.Rd +++ b/man/serve.Rd @@ -4,7 +4,8 @@ \title{Serve diffuseR over HTTP} \usage{ serve(port = 7812L, model = c("flux2", "zimage", "flux1", "ltx"), - device = "cuda", timeout = 300L, max_body = 1024L^2, warmup = TRUE) + device = "cuda", token = NULL, max_pixels = 1024L^2, max_frames = 161L, + max_prompts = 32L, timeout = 300L, max_body = 1024L^2, warmup = TRUE) } \arguments{ \item{port}{Integer. TCP port. Default 7812 (cornball serve range: whisper 7809, chatterbox 7810, qwen3 TTS 7811).} @@ -14,6 +15,17 @@ serve(port = 7812L, model = c("flux2", "zimage", "flux1", "ltx"), \item{device}{Character. "cuda" or "cpu".} +\item{token}{Character or NULL. Shared secret; when set, requests +must send \code{Authorization: Bearer }.} + +\item{max_pixels}{Integer. Maximum width x height accepted (images +and video frames). Default 1024^2.} + +\item{max_frames}{Integer. Maximum video frame count. Default 161.} + +\item{max_prompts}{Integer. Bound on the per-prompt connector-embed +cache for "ltx" (~9 MB per entry, LRU-evicted). Default 32.} + \item{timeout}{Integer. Per-connection I/O timeout in seconds.} \item{max_body}{Integer. Maximum request body bytes. Default 1 MB @@ -53,4 +65,14 @@ under a process supervisor (systemd, tmux); an example unit ships with the package: \code{system.file("diffuser.service", package = "diffuseR")}. +Security: base R's \code{serverSocket} binds all interfaces, so the +server is reachable by anything that can reach the machine. Keep it +behind a firewall or reverse proxy, and/or set \code{token}: when +set, every request must carry \code{Authorization: Bearer } +or it is refused with 401. Generation size is capped by +\code{max_pixels}/\code{max_frames}; oversized requests get 400. A +CUDA out-of-memory during a request answers 500 and then exits the +process (status 70) so a supervisor restarts it with clean GPU +state rather than serving on with stranded components. + } diff --git a/vignettes/performance-levers.md b/vignettes/performance-levers.md index d2a9843..c36dbe0 100644 --- a/vignettes/performance-levers.md +++ b/vignettes/performance-levers.md @@ -14,8 +14,11 @@ models on no GPU at all. It does that with three independent levers: weight precision, per-component device placement, and memory residency. This vignette is the map. The machine-readable version of the same policy is `recommend()`, which inspects your VRAM, host RAM, and -installed `safetensors` capabilities and returns a configuration; the -`txt2img_*`/`txt2vid_*` defaults consume it. +installed `safetensors` capabilities and returns a configuration. The +SD device strategies and the LTX memory profile consume it today; the +flux-family loaders select artifacts by what is built on disk, and +full consumption of the recommendation (including `pin`) is arriving +model by model. ## Lever 1: weight precision @@ -64,9 +67,11 @@ swap (see below). From most to least VRAM: 1. **Fully GPU-resident** — everything fits, nothing moves (large - cards; the fp8 residents of FLUX.2/Z-Image on 16 GB). + cards only). 2. **Resident quantized DiT, phase-swapped everything else** — the - LTX nf4 default on 16 GB. + 16 GB sweet spot: LTX nf4 and the fp8 FLUX.2/Z-Image + configurations keep the DiT on the card while text encoders swap + per phase. 3. **Pinned phase swap** — component weights live in page-locked ("pinned") host RAM and move to the GPU for their phase at DMA rate. Measured on a PCIe 5.0 x8 card: 25 GB/s host-to-device From ac8e2a76c61777c874faf0731451de5927485929 Mon Sep 17 00:00:00 2001 From: TroyHernandez Date: Wed, 22 Jul 2026 21:23:31 -0500 Subject: [PATCH 06/10] Round-2 hardening: consent gates probe every file (no sentinel/artifact bypass), steps/frame-rate/joint-budget limits, warmup OOM fails startup, machine-aware LTX artifact selection with manifest validation, token'd systemd example, vignette accuracy --- NEWS.md | 3 +- R/download_flux.R | 8 +-- R/download_flux2.R | 8 +-- R/download_ltx23.R | 15 ++++++ R/download_zimage.R | 8 +-- R/sd_pipeline_safetensors.R | 6 +-- R/sdxl_pipeline_safetensors.R | 10 ++-- R/serve.R | 91 ++++++++++++++++++++++++++------- inst/diffuser.service | 6 ++- inst/tinytest/test_serve.R | 35 +++++++++++++ man/serve.Rd | 10 +++- vignettes/performance-levers.md | 15 +++--- 12 files changed, 160 insertions(+), 55 deletions(-) diff --git a/NEWS.md b/NEWS.md index 143441c..5b5b8e4 100644 --- a/NEWS.md +++ b/NEWS.md @@ -10,7 +10,8 @@ `system.file("diffuser.service", package = "diffuseR")`. Port 7812 in the cornball serve range. Hardened for persistence: optional bearer-token auth, hard pixel/frame limits - (400 on oversize), a bounded LRU of per-prompt connector embeds + (400 on oversize, including a steps cap, frame-rate bounds, and a + joint pixels-x-frames video budget), a bounded LRU of per-prompt connector embeds (~9 MB each, never the raw Gemma stacks), and a clean process exit on CUDA OOM so a supervisor restarts with sane GPU state. * Every model download is consent-gated: interactive prompt with the diff --git a/R/download_flux.R b/R/download_flux.R index ed5a276..a22b18f 100644 --- a/R/download_flux.R +++ b/R/download_flux.R @@ -103,7 +103,7 @@ download_flux1 <- function(quantize = TRUE, precision = c("nf4", "fp8"), local_files_only = TRUE), error = function(e) NULL ) - if (is.null(cached) && !have_artifact) { + if (is.null(cached) || !.hub_all_cached(.flux1_repo, .flux1_transformer_files)) { free <- .ltx23_disk_free_gb(path.expand("~")) if (!is.na(free) && free < 45) { warning(sprintf( @@ -148,11 +148,7 @@ download_flux1 <- function(quantize = TRUE, precision = c("nf4", "fp8"), if (text_encoders) { files <- c(.flux1_support_files, .flux1_t5_files) - have_t5 <- !is.null(tryCatch( - hfhub::hub_download(.flux1_repo, .flux1_t5_files[[3]], - local_files_only = TRUE), - error = function(e) NULL - )) + have_t5 <- .hub_all_cached(.flux1_repo, files) if (!have_t5) { ok <- .ltx23_consent( "the FLUX text encoders, VAE, and tokenizer (~10 GB)" diff --git a/R/download_flux2.R b/R/download_flux2.R index 2a1c155..e1a1f5a 100644 --- a/R/download_flux2.R +++ b/R/download_flux2.R @@ -75,7 +75,7 @@ download_flux2_klein <- function(quantize = TRUE, local_files_only = TRUE), error = function(e) NULL ) - if (is.null(cached) && !have_artifact) { + if (is.null(cached) || !.hub_all_cached(.flux2_repo, .flux2_transformer_files)) { free <- .ltx23_disk_free_gb(path.expand("~")) if (!is.na(free) && free < 25) { warning(sprintf( @@ -119,11 +119,7 @@ download_flux2_klein <- function(quantize = TRUE, } if (text_encoders) { - have_te <- !is.null(tryCatch( - hfhub::hub_download(.flux2_repo, .flux2_support_files[[5]], - local_files_only = TRUE), - error = function(e) NULL - )) + have_te <- .hub_all_cached(.flux2_repo, .flux2_support_files) if (!have_te) { ok <- .ltx23_consent( "the Qwen3-4B text encoder, tokenizer, and VAE (~8.3 GB)" diff --git a/R/download_ltx23.R b/R/download_ltx23.R index 879c60e..ab84e22 100644 --- a/R/download_ltx23.R +++ b/R/download_ltx23.R @@ -147,3 +147,18 @@ download_ltx2 <- function(quantize = TRUE, invisible(result) } + +# TRUE when every file is already in the local hub cache, i.e. no +# network fetch will happen. Consent gates key on this - never on a +# single sentinel file, and never on artifact presence (a completed +# artifact must not license re-downloading sources). +.hub_all_cached <- function(repo, files, repo_type = NULL) { + all(vapply(files, function(f) { + args <- list(repo, f, local_files_only = TRUE) + if (!is.null(repo_type)) { + args$repo_type <- repo_type + } + !is.null(tryCatch(do.call(hfhub::hub_download, args), + error = function(e) NULL)) + }, logical(1))) +} diff --git a/R/download_zimage.R b/R/download_zimage.R index 78b6739..571189b 100644 --- a/R/download_zimage.R +++ b/R/download_zimage.R @@ -81,7 +81,7 @@ download_zimage_turbo <- function(quantize = TRUE, local_files_only = TRUE), error = function(e) NULL ) - if (is.null(cached) && !have_artifact) { + if (is.null(cached) || !.hub_all_cached(.zimage_repo, .zimage_transformer_files)) { free <- .ltx23_disk_free_gb(path.expand("~")) if (!is.na(free) && free < 35) { warning(sprintf( @@ -125,11 +125,7 @@ download_zimage_turbo <- function(quantize = TRUE, } if (text_encoders) { - have_te <- !is.null(tryCatch( - hfhub::hub_download(.zimage_repo, .zimage_support_files[[5]], - local_files_only = TRUE), - error = function(e) NULL - )) + have_te <- .hub_all_cached(.zimage_repo, .zimage_support_files) if (!have_te) { ok <- .ltx23_consent( "the Qwen3-4B text encoder, tokenizer, and VAE (~8.2 GB)" diff --git a/R/sd_pipeline_safetensors.R b/R/sd_pipeline_safetensors.R index ab4151b..205259a 100644 --- a/R/sd_pipeline_safetensors.R +++ b/R/sd_pipeline_safetensors.R @@ -74,11 +74,7 @@ download_sd21 <- function(verbose = TRUE) { if (!requireNamespace("hfhub", quietly = TRUE)) { stop("The hfhub package is required to download model weights.") } - have <- !is.null(tryCatch( - hfhub::hub_download(.sd21_repo, .sd21_files[[1]], - repo_type = "dataset", local_files_only = TRUE), - error = function(e) NULL - )) + have <- .hub_all_cached(.sd21_repo, .sd21_files, repo_type = "dataset") if (!have) { ok <- .ltx23_consent( "Stable Diffusion 2.1 UNet + VAE + CLIP text encoder (~2.5 GB)" diff --git a/R/sdxl_pipeline_safetensors.R b/R/sdxl_pipeline_safetensors.R index 9346e12..a721175 100644 --- a/R/sdxl_pipeline_safetensors.R +++ b/R/sdxl_pipeline_safetensors.R @@ -159,10 +159,7 @@ download_sdxl <- function(verbose = TRUE) { if (!requireNamespace("hfhub", quietly = TRUE)) { stop("The hfhub package is required to download model weights.") } - have <- !is.null(tryCatch( - hfhub::hub_download(.sdxl_repo, .sdxl_files[[1]], - repo_type = "dataset", local_files_only = TRUE), - error = function(e) NULL)) + have <- .hub_all_cached(.sdxl_repo, .sdxl_files, repo_type = "dataset") if (!have) { ok <- .ltx23_consent( "Stable Diffusion XL UNet + VAE + 2 CLIP text encoders (~7 GB)") @@ -180,6 +177,11 @@ download_sdxl <- function(verbose = TRUE) { index_path <- paths[[length(.sdxl_files)]] idx <- jsonlite::fromJSON(index_path, simplifyVector = TRUE) shards <- unique(unlist(idx$weight_map)) + shard_files <- paste0("diffusers/unet/", shards) + if (!.hub_all_cached(.sdxl_repo, shard_files, repo_type = "dataset") && + !.ltx23_consent("the remaining SDXL UNet shards")) { + stop("Download cancelled.", call. = FALSE) + } for (s in shards) { hfhub::hub_download(.sdxl_repo, paste0("diffusers/unet/", s), repo_type = "dataset") diff --git a/R/serve.R b/R/serve.R index ce82640..1f8fa47 100644 --- a/R/serve.R +++ b/R/serve.R @@ -59,6 +59,11 @@ #' @param max_pixels Integer. Maximum width x height accepted (images #' and video frames). Default 1024^2. #' @param max_frames Integer. Maximum video frame count. Default 161. +#' @param max_steps Integer. Maximum image inference steps. Default 50. +#' @param max_pixel_frames Numeric. Joint video budget: width x height +#' x frames must stay under it (NULL = max_pixels x 121, so full- +#' resolution clips top out at 121 frames and longer clips must +#' shrink spatially). #' @param max_prompts Integer. Bound on the per-prompt connector-embed #' cache for "ltx" (~9 MB per entry, LRU-evicted). Default 32. #' @param timeout Integer. Per-connection I/O timeout in seconds. @@ -74,9 +79,15 @@ serve <- function(port = 7812L, model = c("flux2", "zimage", "flux1", "ltx"), device = "cuda", token = NULL, max_pixels = 1024L^2, max_frames = 161L, + max_steps = 50L, max_pixel_frames = NULL, max_prompts = 32L, timeout = 300L, max_body = 1024L^2, warmup = TRUE) { model <- match.arg(model) + if (is.null(max_pixel_frames)) { + # Joint video budget: full max_pixels only up to 121 frames; + # longer clips must shrink spatially + max_pixel_frames <- as.numeric(max_pixels) * 121 + } message("Loading ", model, " on ", device, " (no downloads; run the ", "download_*() function first if artifacts are missing)...") @@ -84,6 +95,8 @@ serve <- function(port = 7812L, state$token <- token state$max_pixels <- as.integer(max_pixels) state$max_frames <- as.integer(max_frames) + state$max_steps <- as.integer(max_steps) + state$max_pixel_frames <- as.numeric(max_pixel_frames) message("Model loaded.") if (isTRUE(warmup) && model != "ltx") { @@ -93,6 +106,14 @@ serve <- function(port = 7812L, height = 256L, seed = 1L)) message("Warmup done.") }, error = function(e) { + # A CUDA OOM here strands GPU components before the socket + # even opens: fail startup so the supervisor sees it + if (grepl("CUDA out of memory", conditionMessage(e), + fixed = TRUE)) { + stop("warmup hit CUDA out of memory; refusing to serve ", + "with stranded GPU state: ", conditionMessage(e), + call. = FALSE) + } message("warmup skipped: ", conditionMessage(e)) }) } @@ -150,17 +171,35 @@ serve <- function(port = 7812L, .dserve_load <- function(model, device, max_prompts = 32L) { if (model == "ltx") { data_dir <- tools::R_user_dir("diffuseR", "data") - # Serve whichever LTX artifact is built (nf4 preferred; fp8 is - # what download_ltx2() produces by default) - ckpt_dir <- Filter(dir.exists, - file.path(data_dir, c("ltx2.3-nf4", "ltx2.3-fp8"))) - if (!length(ckpt_dir)) { - stop("No LTX-2.3 artifact under ", data_dir, + # Serve a COMPLETE artifact only (a partial/interrupted quantize + # must not shadow a valid one), and when both precisions are + # built, follow the machine-aware recommendation - on a card + # too small for resident nf4, recommend() prescribes streamed + # fp8 (which is also what download_ltx2() produces by default) + valid_artifact <- function(d) { + mp <- file.path(d, "manifest.json") + file.exists(mp) && isTRUE(tryCatch({ + m <- jsonlite::fromJSON(mp) + length(m$shards) > 0 && + all(file.exists(file.path(d, m$shards))) + }, error = function(e) FALSE)) + } + dirs <- c(nf4 = file.path(data_dir, "ltx2.3-nf4"), + fp8 = file.path(data_dir, "ltx2.3-fp8")) + avail <- dirs[vapply(dirs, valid_artifact, logical(1))] + if (!length(avail)) { + stop("No complete LTX-2.3 artifact under ", data_dir, "; run download_ltx2() (fp8) or ltx23_quantize_nf4() first.", call. = FALSE) } - pipe <- ltx23_load_pipeline(ckpt_dir[[1]], device = device, - verbose = FALSE) + pick <- if (length(avail) == 1L) { + avail[[1]] + } else { + prec <- tryCatch(recommend("ltx")$precision, + error = function(e) "nf4") + if (prec %in% names(avail)) avail[[prec]] else avail[[1]] + } + pipe <- ltx23_load_pipeline(pick, device = device, verbose = FALSE) te_dir <- file.path(data_dir, "gemma3-nf4") if (!dir.exists(te_dir)) { stop("Gemma3 NF4 artifact not found at ", te_dir, @@ -296,13 +335,22 @@ serve <- function(port = 7812L, if (length(wh) != 2L || anyNA(wh)) { return(.dserve_err(400L, "size must look like 1024x1024")) } - if (wh[1] * wh[2] > state$max_pixels) { + if (anyNA(wh) || any(wh < 16L) || wh[1] * wh[2] > state$max_pixels) { return(.dserve_err(400L, sprintf( - "request exceeds limits (max %d pixels)", state$max_pixels))) + "request exceeds limits (max %d pixels, min side 16)", + state$max_pixels))) + } + steps <- body$steps + if (!is.null(steps)) { + steps <- suppressWarnings(as.integer(steps)) + if (is.na(steps) || steps < 1L || steps > state$max_steps) { + return(.dserve_err(400L, sprintf( + "steps must be between 1 and %d", state$max_steps))) + } } seed <- if (is.null(body$seed)) NULL else as.integer(body$seed) img <- state$generate(body$prompt, width = wh[1], height = wh[2], - seed = seed, steps = body$steps) + seed = seed, steps = steps) png <- png::writePNG(img) .dserve_json(list( created = as.integer(Sys.time()), @@ -315,13 +363,20 @@ serve <- function(port = 7812L, if (is.null(body) || is.null(body$prompt) || !nzchar(body$prompt)) { return(.dserve_err(400L, "body must be JSON with a prompt")) } - w <- as.integer(body$width %||% 768L) - h <- as.integer(body$height %||% 512L) - nf <- as.integer(body$num_frames %||% 121L) - if (w * h > state$max_pixels || nf > state$max_frames) { + w <- suppressWarnings(as.integer(body$width %||% 768L)) + h <- suppressWarnings(as.integer(body$height %||% 512L)) + nf <- suppressWarnings(as.integer(body$num_frames %||% 121L)) + fr <- suppressWarnings(as.numeric(body$frame_rate %||% 24)) + if (anyNA(c(w, h, nf)) || w < 32L || h < 32L || nf < 9L || + w * h > state$max_pixels || nf > state$max_frames || + as.numeric(w) * h * nf > state$max_pixel_frames) { return(.dserve_err(400L, sprintf( - "request exceeds limits (max %d pixels, %d frames)", - state$max_pixels, state$max_frames))) + "request exceeds limits (max %d pixels, %d frames, %.0f pixel-frames)", + state$max_pixels, state$max_frames, state$max_pixel_frames))) + } + # frame_rate scales the audio-latent length inversely: bound it + if (is.na(fr) || fr < 12 || fr > 60) { + return(.dserve_err(400L, "frame_rate must be between 12 and 60")) } out <- tempfile(fileext = ".mp4") on.exit(unlink(out), add = TRUE) @@ -332,7 +387,7 @@ serve <- function(port = 7812L, width = w, height = h, num_frames = nf, - frame_rate = as.numeric(body$frame_rate %||% 24), + frame_rate = fr, seed = if (is.null(body$seed)) NULL else as.integer(body$seed), device = state$device, dtype = "bfloat16", filename = out, verbose = FALSE diff --git a/inst/diffuser.service b/inst/diffuser.service index 75be441..32214b9 100644 --- a/inst/diffuser.service +++ b/inst/diffuser.service @@ -33,7 +33,11 @@ WorkingDirectory=/home/CHANGEME # littler (/usr/bin/r) does not read ~/.Renviron/.Rprofile, so set what # the server needs explicitly: Environment=HOME=/home/CHANGEME -ExecStart=/usr/bin/r -e "diffuseR::serve(model = 'flux2', port = 7812L)" +# Shared-secret auth: put DIFFUSER_TOKEN= in the env +# file (chmod 600). serverSocket binds all interfaces, so run with a +# token or keep the port firewalled to localhost/LAN. +EnvironmentFile=/etc/diffuser.env +ExecStart=/usr/bin/r -e "diffuseR::serve(model = 'flux2', port = 7812L, token = Sys.getenv('DIFFUSER_TOKEN'))" Restart=on-failure RestartSec=5 # Pipeline load + warmup take a while; don't let a watchdog kill startup. diff --git a/inst/tinytest/test_serve.R b/inst/tinytest/test_serve.R index 50b0b85..13009a2 100644 --- a/inst/tinytest/test_serve.R +++ b/inst/tinytest/test_serve.R @@ -107,3 +107,38 @@ overf <- diffuseR:::.dserve_route( mkreq("POST", "/v1/videos/generations", body = '{"prompt":"x","num_frames":9999}'), vstate) expect_equal(overf$status, 400L) + +# --- steps, frame_rate, and joint-budget limits ------------------------------------ + +state$max_steps <- 50L +overstep <- diffuseR:::.dserve_route( + mkreq("POST", "/v1/images/generations", + body = '{"prompt":"x","size":"64x64","steps":5000}'), state) +expect_equal(overstep$status, 400L) +expect_true(grepl("steps", jbody(overstep)$error$message)) + +okstep <- diffuseR:::.dserve_route( + mkreq("POST", "/v1/images/generations", + body = '{"prompt":"x","size":"64x64","steps":4}'), state) +expect_equal(okstep$status, 200L) + +vstate$max_steps <- 50L +vstate$max_pixel_frames <- 1024^2 * 121 +badfr <- diffuseR:::.dserve_route( + mkreq("POST", "/v1/videos/generations", + body = '{"prompt":"x","frame_rate":1}'), vstate) +expect_equal(badfr$status, 400L) +expect_true(grepl("frame_rate", jbody(badfr)$error$message)) + +# joint budget: each dimension legal alone, product over budget +joint <- diffuseR:::.dserve_route( + mkreq("POST", "/v1/videos/generations", + body = '{"prompt":"x","width":1024,"height":1024,"num_frames":161}'), + vstate) +expect_equal(joint$status, 400L) +expect_true(grepl("pixel-frames", jbody(joint)$error$message)) + +tiny <- diffuseR:::.dserve_route( + mkreq("POST", "/v1/videos/generations", + body = '{"prompt":"x","width":8,"height":8}'), vstate) +expect_equal(tiny$status, 400L) diff --git a/man/serve.Rd b/man/serve.Rd index 6fdf68a..e1bdfe2 100644 --- a/man/serve.Rd +++ b/man/serve.Rd @@ -5,7 +5,8 @@ \usage{ serve(port = 7812L, model = c("flux2", "zimage", "flux1", "ltx"), device = "cuda", token = NULL, max_pixels = 1024L^2, max_frames = 161L, - max_prompts = 32L, timeout = 300L, max_body = 1024L^2, warmup = TRUE) + max_steps = 50L, max_pixel_frames = NULL, max_prompts = 32L, + timeout = 300L, max_body = 1024L^2, warmup = TRUE) } \arguments{ \item{port}{Integer. TCP port. Default 7812 (cornball serve range: whisper 7809, chatterbox 7810, qwen3 TTS 7811).} @@ -23,6 +24,13 @@ and video frames). Default 1024^2.} \item{max_frames}{Integer. Maximum video frame count. Default 161.} +\item{max_steps}{Integer. Maximum image inference steps. Default 50.} + +\item{max_pixel_frames}{Numeric. Joint video budget: width x height +x frames must stay under it (NULL = max_pixels x 121, so full- +resolution clips top out at 121 frames and longer clips must +shrink spatially).} + \item{max_prompts}{Integer. Bound on the per-prompt connector-embed cache for "ltx" (~9 MB per entry, LRU-evicted). Default 32.} diff --git a/vignettes/performance-levers.md b/vignettes/performance-levers.md index c36dbe0..7c7d1bb 100644 --- a/vignettes/performance-levers.md +++ b/vignettes/performance-levers.md @@ -14,11 +14,11 @@ models on no GPU at all. It does that with three independent levers: weight precision, per-component device placement, and memory residency. This vignette is the map. The machine-readable version of the same policy is `recommend()`, which inspects your VRAM, host RAM, and -installed `safetensors` capabilities and returns a configuration. The -SD device strategies and the LTX memory profile consume it today; the -flux-family loaders select artifacts by what is built on disk, and -full consumption of the recommendation (including `pin`) is arriving -model by model. +installed `safetensors` capabilities and returns a configuration. +Today `flux_memory_profile()` delegates to it and `serve()` consults +it to pick between built LTX artifacts; for everything else it is +advisory — call it and pass its fields to the loaders yourself. Full +consumption (including `pin`) is arriving model by model. ## Lever 1: weight precision @@ -120,5 +120,6 @@ r$pin # page-lock the phase-swapped host copies? r$note # fork suggestion when fp8 wanted but unreadable ``` -The generation functions apply this automatically via their defaults; -every field can be overridden by hand. +Treat the result as the machine's advice: pass its fields to the +loaders and generators (only `flux_memory_profile()` and `serve()`'s +LTX artifact selection consume it automatically today). From db8d6d227df117116057401da5970c0d5a65642d Mon Sep 17 00:00:00 2001 From: TroyHernandez Date: Wed, 22 Jul 2026 21:53:24 -0500 Subject: [PATCH 07/10] Round-3: LTX Gemma consent probes all files, fp8-artifact no longer suppresses source-checkpoint consent, scalar-field validation (arrays are 400s) --- R/download_ltx23.R | 8 ++--- R/serve.R | 61 ++++++++++++++++++++++++++++++-------- inst/tinytest/test_serve.R | 22 ++++++++++++++ 3 files changed, 72 insertions(+), 19 deletions(-) diff --git a/R/download_ltx23.R b/R/download_ltx23.R index ab84e22..4d10972 100644 --- a/R/download_ltx23.R +++ b/R/download_ltx23.R @@ -76,7 +76,7 @@ download_ltx2 <- function(quantize = TRUE, local_files_only = TRUE), error = function(e) NULL ) - if (is.null(cached) && !have_fp8) { + if (is.null(cached)) { free <- .ltx23_disk_free_gb(path.expand("~")) if (!is.na(free) && free < 75) { warning(sprintf( @@ -122,11 +122,7 @@ download_ltx2 <- function(quantize = TRUE, "tokenizer/tokenizer.json", "tokenizer/tokenizer_config.json", "tokenizer/special_tokens_map.json" ) - have_te <- !is.null(tryCatch( - hfhub::hub_download(.ltx23_text_encoder_repo, te_files[[2]], - local_files_only = TRUE), - error = function(e) NULL - )) + have_te <- .hub_all_cached(.ltx23_text_encoder_repo, te_files) if (!have_te) { ok <- .ltx23_consent( "the Gemma3 text encoder and tokenizer (~25 GB, Lightricks/LTX-2)" diff --git a/R/serve.R b/R/serve.R index 1f8fa47..1f0f334 100644 --- a/R/serve.R +++ b/R/serve.R @@ -314,6 +314,23 @@ serve <- function(port = 7812L, .dserve_err(404L, "not found") } +# One length-1 atomic value or the default; JSON arrays/objects sent +# for scalar fields become NA so validation answers 400, not 500 +.dserve_scalar <- function(x, default = NULL) { + if (is.null(x)) { + return(default) + } + if (!is.atomic(x) || length(x) != 1L) { + return(NA) + } + x +} + +.dserve_prompt_ok <- function(p) { + is.atomic(p) && length(p) == 1L && is.character(p) && !is.na(p) && + nzchar(p) +} + .dserve_body <- function(req) { if (!length(req$body)) { return(NULL) @@ -324,13 +341,18 @@ serve <- function(port = 7812L, .dserve_image <- function(req, state) { body <- .dserve_body(req) - if (is.null(body) || is.null(body$prompt) || !nzchar(body$prompt)) { - return(.dserve_err(400L, "body must be JSON with a prompt")) + if (is.null(body) || !.dserve_prompt_ok(body$prompt)) { + return(.dserve_err(400L, "body must be JSON with a single string prompt")) } - if (!is.null(body$n) && as.integer(body$n) > 1L) { + n <- .dserve_scalar(body$n, 1L) + if (is.na(suppressWarnings(as.integer(n))) || + as.integer(n) > 1L) { return(.dserve_err(400L, "n > 1 is not supported")) } - size <- if (is.null(body$size)) "1024x1024" else body$size + size <- .dserve_scalar(body$size, "1024x1024") + if (!is.character(size) || is.na(size)) { + return(.dserve_err(400L, "size must look like 1024x1024")) + } wh <- suppressWarnings(as.integer(strsplit(size, "x", fixed = TRUE)[[1]])) if (length(wh) != 2L || anyNA(wh)) { return(.dserve_err(400L, "size must look like 1024x1024")) @@ -340,7 +362,7 @@ serve <- function(port = 7812L, "request exceeds limits (max %d pixels, min side 16)", state$max_pixels))) } - steps <- body$steps + steps <- .dserve_scalar(body$steps) if (!is.null(steps)) { steps <- suppressWarnings(as.integer(steps)) if (is.na(steps) || steps < 1L || steps > state$max_steps) { @@ -348,7 +370,13 @@ serve <- function(port = 7812L, "steps must be between 1 and %d", state$max_steps))) } } - seed <- if (is.null(body$seed)) NULL else as.integer(body$seed) + seed <- .dserve_scalar(body$seed) + if (!is.null(seed)) { + seed <- suppressWarnings(as.integer(seed)) + if (is.na(seed)) { + return(.dserve_err(400L, "seed must be a single integer")) + } + } img <- state$generate(body$prompt, width = wh[1], height = wh[2], seed = seed, steps = steps) png <- png::writePNG(img) @@ -360,13 +388,13 @@ serve <- function(port = 7812L, .dserve_video <- function(req, state) { body <- .dserve_body(req) - if (is.null(body) || is.null(body$prompt) || !nzchar(body$prompt)) { - return(.dserve_err(400L, "body must be JSON with a prompt")) + if (is.null(body) || !.dserve_prompt_ok(body$prompt)) { + return(.dserve_err(400L, "body must be JSON with a single string prompt")) } - w <- suppressWarnings(as.integer(body$width %||% 768L)) - h <- suppressWarnings(as.integer(body$height %||% 512L)) - nf <- suppressWarnings(as.integer(body$num_frames %||% 121L)) - fr <- suppressWarnings(as.numeric(body$frame_rate %||% 24)) + w <- suppressWarnings(as.integer(.dserve_scalar(body$width, 768L))) + h <- suppressWarnings(as.integer(.dserve_scalar(body$height, 512L))) + nf <- suppressWarnings(as.integer(.dserve_scalar(body$num_frames, 121L))) + fr <- suppressWarnings(as.numeric(.dserve_scalar(body$frame_rate, 24))) if (anyNA(c(w, h, nf)) || w < 32L || h < 32L || nf < 9L || w * h > state$max_pixels || nf > state$max_frames || as.numeric(w) * h * nf > state$max_pixel_frames) { @@ -378,6 +406,13 @@ serve <- function(port = 7812L, if (is.na(fr) || fr < 12 || fr > 60) { return(.dserve_err(400L, "frame_rate must be between 12 and 60")) } + vseed <- .dserve_scalar(body$seed) + if (!is.null(vseed)) { + vseed <- suppressWarnings(as.integer(vseed)) + if (is.na(vseed)) { + return(.dserve_err(400L, "seed must be a single integer")) + } + } out <- tempfile(fileext = ".mp4") on.exit(unlink(out), add = TRUE) txt2vid_ltx2( @@ -388,7 +423,7 @@ serve <- function(port = 7812L, height = h, num_frames = nf, frame_rate = fr, - seed = if (is.null(body$seed)) NULL else as.integer(body$seed), + seed = vseed, device = state$device, dtype = "bfloat16", filename = out, verbose = FALSE ) diff --git a/inst/tinytest/test_serve.R b/inst/tinytest/test_serve.R index 13009a2..e1db6ee 100644 --- a/inst/tinytest/test_serve.R +++ b/inst/tinytest/test_serve.R @@ -142,3 +142,25 @@ tiny <- diffuseR:::.dserve_route( mkreq("POST", "/v1/videos/generations", body = '{"prompt":"x","width":8,"height":8}'), vstate) expect_equal(tiny$status, 400L) + +# --- scalar validation: JSON arrays for scalar fields are 400s --------------------- + +arrprompt <- diffuseR:::.dserve_route( + mkreq("POST", "/v1/images/generations", + body = '{"prompt":["a","b"],"size":"64x64"}'), state) +expect_equal(arrprompt$status, 400L) + +arrsize <- diffuseR:::.dserve_route( + mkreq("POST", "/v1/images/generations", + body = '{"prompt":"x","size":["64x64","128x128"]}'), state) +expect_equal(arrsize$status, 400L) + +arrseed <- diffuseR:::.dserve_route( + mkreq("POST", "/v1/images/generations", + body = '{"prompt":"x","size":"64x64","seed":[1,2]}'), state) +expect_equal(arrseed$status, 400L) + +arrw <- diffuseR:::.dserve_route( + mkreq("POST", "/v1/videos/generations", + body = '{"prompt":"x","width":[512,640]}'), vstate) +expect_equal(arrw$status, 400L) From c27274e2aabc99549e45221ee8ce61c66706cb53 Mon Sep 17 00:00:00 2001 From: TroyHernandez Date: Wed, 22 Jul 2026 22:03:40 -0500 Subject: [PATCH 08/10] README: cu128, remotes install, dangling images repointed/removed, LTX video and serve sections --- README.md | 42 ++++++++++++++++++++++++++++++++---------- 1 file changed, 32 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 7a36890..b921baf 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,3 @@ -![](man/figures/diffuseRlogo.png) # diffuseR @@ -21,7 +20,7 @@ First install torch. As per [this comment](https://github.com/mlverse/torch/issu options(timeout = 600) # increasing timeout is recommended since we will be downloading a 2GB file. # For Windows and Linux: "cpu", "cu128" are the only currently supported # For MacOS the supported are: "cpu-intel" or "cpu-m1" -kind <- "cu124" +kind <- "cu128" version <- available.packages()["torch","Version"] options(repos = c( torch = sprintf("https://torch-cdn.mlverse.org/packages/%s/%s/", kind, version), @@ -33,11 +32,8 @@ install.packages("torch") You can install the development version of diffuseR from GitHub: ```r -# install.packages("devtools") -devtools::install_github("cornball-ai/diffuseR") -# Or -# install.packages("targets") -targets::install_github("cornball-ai/diffuseR") +# install.packages("remotes") +remotes::install_github("cornball-ai/diffuseR") ``` ## Features @@ -49,6 +45,8 @@ targets::install_github("cornball-ai/diffuseR") - **Scheduler Options**: DDIM and FlowMatch Euler (static and dynamic shifting) - **Device Support**: CPU and CUDA GPUs (including Blackwell RTX 50xx) - **R-native Interface**: Functional programming approach that feels natural in R +- **HTTP serving**: `serve()` exposes any model over OpenAI-style endpoints + (images and video) from one persistent process - see `?serve` ## Hardware requirements @@ -91,7 +89,7 @@ cat_img <- txt2img( pipeline <- NULL torch::cuda_empty_cache() ``` -![](man/figures/cat.png) + ### Advanced Usage with GPU @@ -145,8 +143,8 @@ gambling_cat <- img2img( pipeline <- NULL torch::cuda_empty_cache() ``` -![](man/figures/cat2.png) -![](man/figures/gambling_cat.png) +![](cat2.png) +![](gambling_cat.png) ### FLUX and Z-Image @@ -179,6 +177,30 @@ download_zimage_turbo() txt2img_zimage(paste("A storefront with a large wooden sign that reads", "\"DIFFUSER\" in bold carved letters"), seed = 42) +### Text-to-Video: LTX-2.3 + +LTX-2.3 (22B, distilled) generates video with synchronized audio in 8 +steps. NF4-quantized it renders 768x512x49 in ~44s warm on an RTX +5060 Ti; audio-driven talking-head generation and clip continuation +are supported (see ?txt2vid_ltx2). + +```r +download_ltx2() # ~46GB download, one-time fp8 quantize +txt2vid_ltx2("A river winding through a misty forest at dawn", + pipeline = ltx23_load_pipeline(), + filename = "river.mp4") +``` + +### Serving over HTTP + +```r +# One persistent process, OpenAI-style endpoints, never downloads: +serve(model = "flux2", port = 7812L, token = "my-secret") +# curl -H "Authorization: Bearer my-secret" \ +# -d '{"prompt":"a lighthouse at dusk","size":"512x512"}' \ +# http://localhost:7812/v1/images/generations +``` + # Or through the common dispatcher txt2img("a lighthouse at dusk", model_name = "flux2") ``` From 855dd44a948a5693a67b86c7745dff64875c948f Mon Sep 17 00:00:00 2001 From: TroyHernandez Date: Wed, 22 Jul 2026 22:13:05 -0500 Subject: [PATCH 09/10] is_blackwell_gpu: probe CUDA softly (lantern-less torch errors instead of returning FALSE) --- R/vram.R | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/R/vram.R b/R/vram.R index f7975cc..b254934 100644 --- a/R/vram.R +++ b/R/vram.R @@ -21,8 +21,12 @@ NULL #' } #' } is_blackwell_gpu <- function() { - # Check compute capability via torch - if (torch::cuda_is_available()) { + # Check compute capability via torch. cuda_is_available() ERRORS + # (not FALSE) when the torch package is installed without its + # lantern binaries - fresh installs, win-builder, CRAN checks - so + # probe soft: no lantern means no CUDA means not Blackwell. + avail <- tryCatch(torch::cuda_is_available(), error = function(e) FALSE) + if (avail) { cap <- tryCatch(torch::cuda_get_device_capability(0L), error = function(e) NULL) if (!is.null(cap)) { From 4ba32aeb99fbb1c23517eed54aa08bb6a2f1b23f Mon Sep 17 00:00:00 2001 From: TroyHernandez Date: Wed, 22 Jul 2026 22:16:13 -0500 Subject: [PATCH 10/10] Guard the five unguarded torch test files (lantern-less installs: win-builder, CRAN) --- inst/tinytest/test_flowmatch_python_validation.R | 4 ++++ inst/tinytest/test_flowmatch_scheduler.R | 4 ++++ inst/tinytest/test_text_encoder.R | 4 ++++ inst/tinytest/test_unet.R | 4 ++++ inst/tinytest/test_vae_decoder.R | 4 ++++ 5 files changed, 20 insertions(+) diff --git a/inst/tinytest/test_flowmatch_python_validation.R b/inst/tinytest/test_flowmatch_python_validation.R index fe20c1b..6c538bc 100644 --- a/inst/tinytest/test_flowmatch_python_validation.R +++ b/inst/tinytest/test_flowmatch_python_validation.R @@ -1,3 +1,7 @@ +if (!requireNamespace("torch", quietly = TRUE) || !torch::torch_is_installed()) { + exit_file("torch not fully installed") +} + # Validation tests for FlowMatch scheduler against Python diffusers # These tests ensure numerical equivalence with HuggingFace implementation diff --git a/inst/tinytest/test_flowmatch_scheduler.R b/inst/tinytest/test_flowmatch_scheduler.R index a3431f5..e0327f9 100644 --- a/inst/tinytest/test_flowmatch_scheduler.R +++ b/inst/tinytest/test_flowmatch_scheduler.R @@ -1,3 +1,7 @@ +if (!requireNamespace("torch", quietly = TRUE) || !torch::torch_is_installed()) { + exit_file("torch not fully installed") +} + # Tests for FlowMatch Euler Discrete Scheduler # Test scheduler creation diff --git a/inst/tinytest/test_text_encoder.R b/inst/tinytest/test_text_encoder.R index 32a865d..f3c185e 100644 --- a/inst/tinytest/test_text_encoder.R +++ b/inst/tinytest/test_text_encoder.R @@ -1,3 +1,7 @@ +if (!requireNamespace("torch", quietly = TRUE) || !torch::torch_is_installed()) { + exit_file("torch not fully installed") +} + # Test native CLIP text encoder # Skip if models not available diff --git a/inst/tinytest/test_unet.R b/inst/tinytest/test_unet.R index 133b9b6..65699a7 100644 --- a/inst/tinytest/test_unet.R +++ b/inst/tinytest/test_unet.R @@ -1,3 +1,7 @@ +if (!requireNamespace("torch", quietly = TRUE) || !torch::torch_is_installed()) { + exit_file("torch not fully installed") +} + # Test native UNet # Skip if models not available diff --git a/inst/tinytest/test_vae_decoder.R b/inst/tinytest/test_vae_decoder.R index 64ee9e5..c5fafbf 100644 --- a/inst/tinytest/test_vae_decoder.R +++ b/inst/tinytest/test_vae_decoder.R @@ -1,3 +1,7 @@ +if (!requireNamespace("torch", quietly = TRUE) || !torch::torch_is_installed()) { + exit_file("torch not fully installed") +} + # Test native VAE decoder # Skip if no GPU or models not available