diff --git a/AGENTS.md b/AGENTS.md index ebb9f15e..2ea9cdf1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -360,9 +360,10 @@ Rules for `render_frame`: - It runs once per frame on several threads. No panics, no I/O, no heavy allocation. Precompute in the constructor and read from `self`; memoise per-video data in a `std::sync::OnceLock` field (only store a measurement once it succeeded). -- `svgr!` is SVG with Rust interpolation: attributes and text children accept `{expr}` where - `expr` is a string, number, `Color`, `Transform`, another `Svgr`, or an iterator of `Svgr` - (`.collect::>()`). Comments are `// ...` lines inside the markup. +- `svgr!` is SVG with Rust interpolation: attributes accept `{expr}` where `expr` is a + string, number, `Color`, `Transform` or an image `href()`; children accept a string, + another `Svgr`, an `Option`, or an iterator of `Svgr` (`.collect::>()`). A + number as a child needs `.to_string()`. Comments are `// ...` lines inside the markup. - Subtrees whose markup contains no `{}` get a compile-time `static_hash` and are cached by the renderers across frames (paths, paints, whole groups). Keep decorative geometry lexically static and put the animated values on a wrapping ``. @@ -550,7 +551,7 @@ Two renderers draw the same picture: - The CPU rasterizer runs everywhere, also in the editor (at half size there). - With `fframes = { features = ["3d", "wgpu"] }` the renderer uses wgpu (Vulkan, Metal, DX12). A frame differs from the CPU frame by less than one 8-bit step on average. Without a GPU - it falls back to the CPU. `Renderer::cpu()` forces the CPU. + it falls back to the CPU. `Renderer::new(w, h).cpu()` forces the CPU. - With `fframes_skia_renderer = { features = ["vulkan", "wgpu"] }` as well, the 3D stays on the GPU: Skia draws the wgpu texture in place and converts the frame for the encoder. Nothing is read back. Other backends (the CPU backend, Metal, the editor) get pixels. diff --git a/fframes-three-d/src/render/post.rs b/fframes-three-d/src/render/post.rs index beef4824..dc7f8c0b 100644 --- a/fframes-three-d/src/render/post.rs +++ b/fframes-three-d/src/render/post.rs @@ -45,7 +45,20 @@ pub enum ToneMapping { /// white. #[default] Aces, - /// three.js `NeutralToneMapping` (Khronos PBR Neutral). It keeps base colors accurate. + /// three.js `NeutralToneMapping` (Khronos PBR Neutral). It keeps the hue of base colors + /// and the saturation of mid colors. Above 0.76 (linear) it compresses the brightest + /// channel toward 1 and mixes the color with white, so highlights lose saturation. + /// + /// The curve subtracts 0.04 (linear) from every channel. This removes the 4% white + /// reflection that a lit dielectric gets from a bright environment. A color whose darkest + /// channel `x` is below 0.08 (sRGB `#505050`) loses `x - 6.25 * x * x` instead, so that + /// channel becomes `6.25 * x * x`. Dark colors go toward black and toward their strongest + /// channel: an unlit `#1a1c20` becomes `#02060f`. Unlit materials and surfaces without + /// that reflection lose the same amount, so they are darker than their color. + /// + /// For dark grounds and dark products use [`ToneMapping::Aces`] with a higher + /// [`Renderer::exposure`](super::Renderer::exposure), or [`ToneMapping::Linear`] for exact + /// flat colors. Neutral, } diff --git a/skills/fframes-video/SKILL.md b/skills/fframes-video/SKILL.md index 985147ad..c12ad1e7 100644 --- a/skills/fframes-video/SKILL.md +++ b/skills/fframes-video/SKILL.md @@ -236,7 +236,9 @@ warning gives the `frame ... --crop ...` command to look at it; repeats of the s Rules: - Look at the PNGs before saying anything looks good. -- Prefer `strip` to many `frame` calls; add `--scale 0.5` when composition is all you need. +- Prefer `strip` to many `frame` calls. The size of its cells is `--width` (480 px by + default; `--scale` does not change a strip). Add `--scale 0.5` to `frame` when composition + is all you need. - Add `--json` when parsing output: the result goes to stdout, progress to stderr. - Keep `--release`: debug builds render many times slower. diff --git a/skills/fframes-video/references/3d.md b/skills/fframes-video/references/3d.md index 53007eb4..b95d94d4 100644 --- a/skills/fframes-video/references/3d.md +++ b/skills/fframes-video/references/3d.md @@ -30,8 +30,8 @@ fframes = { version = "", features = ["3d", "wgpu"] } | the CPU backend, or no usable GPU | the CPU rasterizer, 60 to 150 ms per 1080p frame on 28 threads | | the web editor (wasm) | the CPU rasterizer at half size | -Without the `wgpu` feature everything renders on the CPU. `Renderer::cpu()` forces the CPU for -one renderer. With a GPU, a 1080p frame with shadows, reflections, depth of field and 4x +Without the `wgpu` feature everything renders on the CPU. `Renderer::new(w, h).cpu()` forces the +CPU for one renderer. With a GPU, a 1080p frame with shadows, reflections, depth of field and 4x MSAA takes 5 to 20 ms on an integrated GPU; the arm film (2216 frames) renders in 14 s on an M4 Max. @@ -119,6 +119,9 @@ In `render_frame`: `let image = self.shot.image(t, frame.animate(&self.open));` - **Media in 3D.** `Texture::from_image(&image)` puts an fframes image on a material (`.map(texture)`, `.emissive_map(texture)` for a glowing screen). For a video playing on a screen, make a new texture from each synced frame, resized to the screen first. + `Texture::from_image` does not exist on wasm: put the call behind + `#[cfg(not(target_arch = "wasm32"))]` in a video with an editor bridge. `Texture::decode` + and `Texture::from_fn` work everywhere. - **Fonts and models.** `Geometry::text(include_bytes!("../media/Inter.ttf"), "Hi", 0.5, 0.1, 0.01)?` (size, depth and bevel in meters) makes extruded text from the same font file as the SVG. `Model::from_gltf(include_bytes!(..))` @@ -127,19 +130,29 @@ In `render_frame`: `let image = self.shot.image(t, frame.animate(&self.open));` Z up: `Geometry::from_stl(bytes)?.scale(0.001, 0.001, 0.001).rotate_x(-FRAC_PI_2)`. - **Matching the look.** Pick one lighting setup per video: `Environment::studio()` for products, an HDR panorama (`Environment::map`) for real places. `ToneMapping::Neutral` - keeps brand colors, ACES (the default) is filmic and shifts saturated colors. Match the SVG - palette with `fframes::Color::from(rgb)` of the material colors. + keeps the hue of mid and bright brand colors on light grounds, but it crushes dark colors: + below sRGB `#505050` they go toward black and shift in hue (`#1a1c20` renders as `#02060f` on an + unlit material, and near `#080c13` on a standard material under a white directional + light). For a dark ground or a dark product use ACES (the default, filmic, shifts + saturated colors) with `.exposure(1.3)` to `.exposure(1.6)`, or `ToneMapping::Linear` when + flat and unlit colors must match their hex exactly. Match the SVG palette with + `fframes::Color::from(rgb)` of the material colors. ## Do - Build geometry, textures, models and text in the constructor. The GPU caches meshes and textures by their `Arc`; new ones every frame upload again every frame. - Clone the scene in `render_frame` and change only what moves. -- Set `.shadows(false, ..)` on floors, walls and far scenery and give `Shadow::bounds` a box - around what casts shadows, so the shadow map stays sharp. Only `Light::sun` (or - `Light::with_shadow`) casts shadows; point and spot lights shine through objects. -- Use `Reflector::floor()` with `Material::reflection(..)` and a roughness map for polished - floors, one reflector per scene. +- Set `.shadows(false, ..)` on floors, walls and far scenery, so the shadow map stays sharp: + by default it covers the box of all shadow casters. When the casters are far apart, set + the box that must receive sharp shadows: + `Light::directional(color, intensity, from).with_shadow(Shadow { bounds: Some(Aabb::from_points([min, max])), ..Default::default() })`. + Only a directional light casts shadows (`Light::sun`, or `Light::directional` with + `.with_shadow(..)`); point and spot lights shine through objects. +- Use `Reflector::floor()` with `Material::reflection(..)` for polished floors, one + reflector per scene. For a floor with rough and polished areas set the field + `material.metal_rough_map = Some(Arc::new(texture))` (the glTF texture: green multiplies + the roughness, blue the metalness; there is no builder method for it). - Give small glowing particles a few pixels of size, or keep them in focus: out of focus, a very bright single-pixel dot turns into a pattern of dots in the depth of field. - Look at the result: `strip` and `frame` show the 3D as pixels; `inspect` cannot see inside @@ -171,7 +184,7 @@ In `render_frame`: `let image = self.shot.image(t, frame.animate(&self.open));` | `Scene`, `Group`, `Mesh`, `InstancedMesh`, `Points` | `Scene`, `Node::group`, `Node::mesh`, `Node::instanced`, `Node::particles` | | Box, Sphere, Cylinder, Cone, Capsule, Torus, Plane, Circle, Ring, Lathe, Extrude, Tube, Icosahedron, Text, RoundedBox geometry | `Geometry::cuboid`, `sphere`, `cylinder`, `cone`, `capsule`, `torus`, `plane`, `floor`, `circle`, `ring`, `lathe`, `extrude`, `tube`, `icosahedron`, `text`, `rounded_box`; `merge`, `translate`, `rotate_*`, `scale`, `smooth`, `flat` | | `MeshStandardMaterial`, clearcoat, `MeshBasicMaterial`, `ShadowMaterial`, `MeshNormalMaterial` | `Material::standard`, `metal`, `.clearcoat`, `unlit`, `glow` (additive), `shadow_catcher`, `normals` | -| `map`, `normalMap`, `aoMap`, `emissiveMap`, metal-roughness map | `.map`, `.normal_map`, `.ao_map`, `.emissive_map`, `metal_rough_map`; `Texture::decode`, `from_fn`, `from_image` | +| `map`, `normalMap`, `aoMap`, `emissiveMap`, metal-roughness map | `.map`, `.normal_map`, `.ao_map`, `.emissive_map`, the field `metal_rough_map`; `Texture::decode`, `from_fn`, `from_image` | | `AmbientLight`, `HemisphereLight`, `DirectionalLight` + shadow, `PointLight`, `SpotLight` | `Light::ambient`, `hemisphere`, `directional`, `sun`, `point`, `spot` | | `RoomEnvironment`, `RGBELoader`, `Fog`, `FogExp2`, `Reflector` | `Environment::studio`, `Environment::map`, `Fog::Linear`, `Fog::Exp2`, `Reflector::floor` | | `GLTFLoader` + `AnimationMixer`, `STLLoader` | `Model::from_gltf`, `ModelInstance::play`, `Geometry::from_stl` | diff --git a/skills/fframes-video/references/api.md b/skills/fframes-video/references/api.md index 4fe35352..3a99b699 100644 --- a/skills/fframes-video/references/api.md +++ b/skills/fframes-video/references/api.md @@ -11,8 +11,10 @@ shaders). ## svgr! -SVG markup with Rust in braces. Attributes and text children take strings, numbers, `Color`, -`Transform`, another `Svgr`, or a `Vec` / iterator collected to a `Vec`. +SVG markup with Rust in braces. An attribute takes a string, a number, a `Color`, a +`Transform` or an image `href()`. A child takes a `String` or `&str`, another `Svgr`, an +`Option` or a `Vec` (an iterator collected to a `Vec`). A number as a child does +not compile: write `{count.to_string()}`. ```rust fframes::svgr!( @@ -32,8 +34,9 @@ fframes::svgr!( decoration static, wrap animated values around it. - Gradients, clip paths, masks, filters go into `` like normal SVG and are referenced with `url(#id)`. Give ids unique names per component instance if repeated. -- Images: `href={self.media.logo_png.href()}` (field named after the file in `media/`) or - `ctx.get_image("logo.png").map(|i| i.href())`. +- Images: `href={self.media.logo_png.href()}` (field named after the file in `media/`), or + `let Some(logo) = ctx.get_image("logo.png") else { return Svgr::empty() };` and + `href={logo.href()}`. An attribute does not take an `Option`. ## Video and Scene @@ -123,8 +126,9 @@ fframes::svgr!( caps the natural settle time; omit the end in `timeline!` to let the spring settle. - For precomputed geometry, motion-blur subframes or a custom clock, `runtime.solve(&elapsed_seconds)` evaluates the same easing - directly. Handle times before the start and after `get_duration()` explicitly: `solve` - does not clamp them. Use a custom curve only when the required motion is not represented + directly. Handle times before the start and after `get_duration()` explicitly: `Linear`, + `Spring` and a diagonal `CubicBezier` (`x1 == y1` and `x2 == y2`) do not clamp them; the + other bezier easings do. Use a custom curve only when the required motion is not represented by the available easings. - Easing: `Linear`, `EaseIn`, `EaseOut`, `EaseInOut`, `CubicBezier(x1, y1, x2, y2)`, `Spring {..}`. Use the named ones and `Spring`; `design.md` says which fits where. @@ -178,7 +182,10 @@ family, size and weight in the `` attributes. `text-anchor="middle"` cente and combine providers with `CombinedMediaProvider::from([&static_media as &dyn MediaProvider, &media])`. - Video frames: `frame.get_synced_video_frame(ctx, "clip.mp4", &SyncVideoFrameInput { start_from, looping, speed, editor_fallback_image })` (`speed: 2.0` plays twice as fast, `..Default::default()` keeps 1.0) - then `` (needs the `compile-time-svgtree` feature). + returns an `Option`: `let Some(video_frame) = .. else { return Svgr::empty() };` then + `let image = video_frame.into_image();` and `` (needs the + `compile-time-svgtree` feature). The video frame itself has no `href()`, and its methods + need `use fframes::FFramesSyncedVideoFrame;`. `start_from` is when the video starts on the timeline, not a position in the file. - Footage: `Clip::new("IMG_0926.mov", 19.2..24.7)` is seconds 19.2 to 24.7 of the file, `Clip::new("cam.mp4", 0.0..)` the whole file (`.at(1.2)` starts it at 1.2 s of the scene, `.speed(1.5)`, `.reverse()`, `.looping()`). @@ -301,7 +308,8 @@ let problems = previewer.inspect(frame)?.diagnostics; Skia: `SkiaFFramesRenderer::new_metal(&SkiaMetalCtx::new(W, H)?, SkiaPipelineConfig::default())` (`fframes_skia_renderer` feature `metal`) or `new_vulkan` with `SkiaVulkanCtx` (feature `vulkan`). `backend.frame_renderer()` -returns a renderer for single frames that matches the backend (Skia on the GPU for Skia). +returns an `Option` of a renderer for single frames that matches the backend (Skia on the +GPU for Skia); `None` means the built-in CPU renderer. The Skia backend picks the fastest way to hand frames to the encoder on its own: hardware frames (`*_videotoolbox` encoders with `new_metal`; `h264_vulkan`/`hevc_vulkan` with @@ -336,4 +344,5 @@ fframes::cli::new(&video, options) - Heavy filters (`feGaussianBlur`, `feDropShadow`) on animated subtrees are slow on the CPU backend: keep them static or use Skia. - Durations from audio (`Duration::FromAudio`, `Eof`) need the file in the media provider; - missing files fail early with `RequiredAudioNotFound`. + a missing file fails early with `CanNotProcessAudioDuration` (`RequiredAudioNotFound` when + `RenderOptions::media` is `None`). diff --git a/skills/fframes-video/references/captions.md b/skills/fframes-video/references/captions.md index fdf30895..dd1f4552 100644 --- a/skills/fframes-video/references/captions.md +++ b/skills/fframes-video/references/captions.md @@ -41,11 +41,17 @@ per-word settings: Loading is the same as for any other media: `include_media_dir!` embeds the file, `MediaDirectory::read_folder` reads it at runtime, the editor loads it over HTTP. JSON -whose top-level object has `cues` is a caption track; any other JSON is not. -`ctx.get_subtitles("speech.json")` returns the `&CaptionTrack`: `cues()`, -`active_cues(seconds)`, and on each `CaptionCue` `phrase()`, `words`, `metadata`. -A track can also be built in code: `CaptionTrack::new([CaptionCue::words(0.0, 2.0, -[CaptionWord::new("Choose", 0.0, 0.7), ..]), CaptionCue::text(2.4, 4.0, "..")])`. +whose top-level object has `cues` is a caption track. A file named `subtitles.json`, +`captions.json`, `*.subtitles.json` or `*.captions.json` is always a caption track and fails +to load without `cues`. Any other JSON is a Lottie animation with the `lottie` feature; +without it `include_media_dir!` fails to compile and `MediaDirectory` skips the file. The +top-level object, cues and words accept only the keys shown here: an unknown key (a +`language` next to `cues`) fails the load. Put your own data into `metadata`. +`ctx.get_subtitles("speech.json")` returns an `Option<&CaptionTrack>` (`None` when the +file is missing): `cues()`, `active_cues(seconds)`, and on each `CaptionCue` `phrase()`, +`words`, `metadata`. A track can also be built in code: `CaptionTrack::new([CaptionCue::words(0.0, 2.0, +[CaptionWord::new("Choose", 0.0, 0.7), ..]), CaptionCue::text(2.4, 4.0, "..")])` returns a +`Result`, because it validates the timing like the parser. For a plain phrase caption `frame.get_subtitle_phrase(track)` returns the current cue text; this older helper includes the cue's end timestamp, `Captions` and `active_cues` @@ -72,7 +78,8 @@ let captions = Captions::from_media("speech.json", CaptionRect::new(120.0, 760.0 .underline("#ffd166", 5.0); // bar under the active word ``` -`Captions::new(track, rect)` takes an owned `CaptionTrack` instead of a media name. +`Captions::new(track, rect)` takes an owned `CaptionTrack` (not the `Result` of +`CaptionTrack::new` or `parse`) instead of a media name. The rectangle is in composition pixels. Each phrase is wrapped to its width, lines are centred, and the block is anchored to the bottom edge of the rectangle; simultaneous cues are stacked in order. Nothing moves when the active word changes. @@ -89,7 +96,8 @@ Modes: (`upcoming_fill`, `active_fill`, `spoken_fill`). - `Karaoke`: like Highlight, the active word fills left to right with its progress. - `Reveal`: words appear as they are spoken and stay; upcoming words are hidden. -- `Word`: only the active word, centred in the rectangle; nothing during gaps. +- `Word`: only the active word, centred horizontally on the bottom edge of the rectangle + like the other modes; nothing during gaps. A VTT phrase has no word timing: every mode shows the whole phrase in `spoken_fill`. @@ -133,8 +141,9 @@ What the scope offers: - `scope.word`: `text`, `state` (`Upcoming | Active | Spoken`), `progress` (0..=1 of the word's duration), `start`/`end` (`None` for a VTT phrase), `metadata`, `cue_index`, - `word_index`. `scope.cue_metadata` is the cue's object. `scope.is_active()`. -- `scope.time` (caption seconds), `scope.elapsed()` (seconds since the word started), + `token_index`, `word_index` (`None` for a VTT phrase). `scope.cue_metadata` is the cue's object. `scope.is_active()`. +- `scope.time` (caption seconds), `scope.elapsed()` (`Option`: seconds since the word started, + `None` for a VTT phrase), `scope.frame` (the `Frame`, for `frame.animate` or `Shader::draw`). - `scope.composition_rect`: where the word lands on the canvas, if something else must align with it. @@ -144,9 +153,9 @@ What the scope offers: - `render_at_with` and `highlight_at_with` take explicit seconds; every variant has a `try_` form that returns the error. -For a layout of your own, `captions.words_at(seconds, &mut frame, ctx)` returns the -positioned words (`rect` in composition pixels, state, progress, metadata) and you draw -them yourself. +For a layout of your own, `captions.words_at(seconds, &mut frame, ctx)` returns a +`Result` with the positioned words (`rect` in composition pixels, state, progress, +metadata) and you draw them yourself. It does not report its errors to `inspect`. ## Metadata diff --git a/skills/fframes-video/references/design.md b/skills/fframes-video/references/design.md index 90952ba2..ca5eefa1 100644 --- a/skills/fframes-video/references/design.md +++ b/skills/fframes-video/references/design.md @@ -52,7 +52,7 @@ problems already looks finished. `+2` to `+6` for small caps labels. - Max ~8 words per line, ~3 lines per card. Measure with `frame.text_width` or wrap with `frame.text_break_lines`; shrink to fit with a loop over sizes and memoize the result - in a `OnceLock`. Generated projects include a `FittedSizes` helper that does this. + in a `OnceLock`. The `greeting_size` method of a generated single-scene project does this. ## Color