Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -360,9 +360,10 @@ Rules for `render_frame`:
- It runs once per frame on several threads. No panics, no I/O, no heavy allocation. Precompute
in the constructor and read from `self`; memoise per-video data in a `std::sync::OnceLock`
field (only store a measurement once it succeeded).
- `svgr!` is SVG with Rust interpolation: attributes and text children accept `{expr}` where
`expr` is a string, number, `Color`, `Transform`, another `Svgr`, or an iterator of `Svgr`
(`.collect::<Vec<_>>()`). Comments are `// ...` lines inside the markup.
- `svgr!` is SVG with Rust interpolation: attributes accept `{expr}` where `expr` is a
string, number, `Color`, `Transform` or an image `href()`; children accept a string,
another `Svgr`, an `Option<Svgr>`, or an iterator of `Svgr` (`.collect::<Vec<_>>()`). A
number as a child needs `.to_string()`. Comments are `// ...` lines inside the markup.
- Subtrees whose markup contains no `{}` get a compile-time `static_hash` and are cached by
the renderers across frames (paths, paints, whole groups). Keep decorative geometry lexically
static and put the animated values on a wrapping `<g transform={...} opacity={...}>`.
Expand Down Expand Up @@ -550,7 +551,7 @@ Two renderers draw the same picture:
- The CPU rasterizer runs everywhere, also in the editor (at half size there).
- With `fframes = { features = ["3d", "wgpu"] }` the renderer uses wgpu (Vulkan, Metal, DX12).
A frame differs from the CPU frame by less than one 8-bit step on average. Without a GPU
it falls back to the CPU. `Renderer::cpu()` forces the CPU.
it falls back to the CPU. `Renderer::new(w, h).cpu()` forces the CPU.
- With `fframes_skia_renderer = { features = ["vulkan", "wgpu"] }` as well, the 3D stays on
the GPU: Skia draws the wgpu texture in place and converts the frame for the encoder.
Nothing is read back. Other backends (the CPU backend, Metal, the editor) get pixels.
Expand Down
15 changes: 14 additions & 1 deletion fframes-three-d/src/render/post.rs
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,20 @@ pub enum ToneMapping {
/// white.
#[default]
Aces,
/// three.js `NeutralToneMapping` (Khronos PBR Neutral). It keeps base colors accurate.
/// three.js `NeutralToneMapping` (Khronos PBR Neutral). It keeps the hue of base colors
/// and the saturation of mid colors. Above 0.76 (linear) it compresses the brightest
/// channel toward 1 and mixes the color with white, so highlights lose saturation.
///
/// The curve subtracts 0.04 (linear) from every channel. This removes the 4% white
/// reflection that a lit dielectric gets from a bright environment. A color whose darkest
/// channel `x` is below 0.08 (sRGB `#505050`) loses `x - 6.25 * x * x` instead, so that
/// channel becomes `6.25 * x * x`. Dark colors go toward black and toward their strongest
/// channel: an unlit `#1a1c20` becomes `#02060f`. Unlit materials and surfaces without
/// that reflection lose the same amount, so they are darker than their color.
///
/// For dark grounds and dark products use [`ToneMapping::Aces`] with a higher
/// [`Renderer::exposure`](super::Renderer::exposure), or [`ToneMapping::Linear`] for exact
/// flat colors.
Neutral,
}

Expand Down
4 changes: 3 additions & 1 deletion skills/fframes-video/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,9 @@ warning gives the `frame ... --crop ...` command to look at it; repeats of the s

Rules:
- Look at the PNGs before saying anything looks good.
- Prefer `strip` to many `frame` calls; add `--scale 0.5` when composition is all you need.
- Prefer `strip` to many `frame` calls. The size of its cells is `--width` (480 px by
default; `--scale` does not change a strip). Add `--scale 0.5` to `frame` when composition
is all you need.
- Add `--json` when parsing output: the result goes to stdout, progress to stderr.
- Keep `--release`: debug builds render many times slower.

Expand Down
33 changes: 23 additions & 10 deletions skills/fframes-video/references/3d.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ fframes = { version = "<the version of fframes>", features = ["3d", "wgpu"] }
| the CPU backend, or no usable GPU | the CPU rasterizer, 60 to 150 ms per 1080p frame on 28 threads |
| the web editor (wasm) | the CPU rasterizer at half size |

Without the `wgpu` feature everything renders on the CPU. `Renderer::cpu()` forces the CPU for
one renderer. With a GPU, a 1080p frame with shadows, reflections, depth of field and 4x
Without the `wgpu` feature everything renders on the CPU. `Renderer::new(w, h).cpu()` forces the
CPU for one renderer. With a GPU, a 1080p frame with shadows, reflections, depth of field and 4x
MSAA takes 5 to 20 ms on an integrated GPU; the arm film (2216 frames) renders in 14 s on an
M4 Max.

Expand Down Expand Up @@ -119,6 +119,9 @@ In `render_frame`: `let image = self.shot.image(t, frame.animate(&self.open));`
- **Media in 3D.** `Texture::from_image(&image)` puts an fframes image on a material
(`.map(texture)`, `.emissive_map(texture)` for a glowing screen). For a video playing on a
screen, make a new texture from each synced frame, resized to the screen first.
`Texture::from_image` does not exist on wasm: put the call behind
`#[cfg(not(target_arch = "wasm32"))]` in a video with an editor bridge. `Texture::decode`
and `Texture::from_fn` work everywhere.
- **Fonts and models.** `Geometry::text(include_bytes!("../media/Inter.ttf"), "Hi", 0.5, 0.1,
0.01)?` (size, depth and bevel in meters) makes extruded text from the same font file as
the SVG. `Model::from_gltf(include_bytes!(..))`
Expand All @@ -127,19 +130,29 @@ In `render_frame`: `let image = self.shot.image(t, frame.animate(&self.open));`
Z up: `Geometry::from_stl(bytes)?.scale(0.001, 0.001, 0.001).rotate_x(-FRAC_PI_2)`.
- **Matching the look.** Pick one lighting setup per video: `Environment::studio()` for
products, an HDR panorama (`Environment::map`) for real places. `ToneMapping::Neutral`
keeps brand colors, ACES (the default) is filmic and shifts saturated colors. Match the SVG
palette with `fframes::Color::from(rgb)` of the material colors.
keeps the hue of mid and bright brand colors on light grounds, but it crushes dark colors:
below sRGB `#505050` they go toward black and shift in hue (`#1a1c20` renders as `#02060f` on an
unlit material, and near `#080c13` on a standard material under a white directional
light). For a dark ground or a dark product use ACES (the default, filmic, shifts
saturated colors) with `.exposure(1.3)` to `.exposure(1.6)`, or `ToneMapping::Linear` when
flat and unlit colors must match their hex exactly. Match the SVG palette with
`fframes::Color::from(rgb)` of the material colors.

## Do

- Build geometry, textures, models and text in the constructor. The GPU caches meshes and
textures by their `Arc`; new ones every frame upload again every frame.
- Clone the scene in `render_frame` and change only what moves.
- Set `.shadows(false, ..)` on floors, walls and far scenery and give `Shadow::bounds` a box
around what casts shadows, so the shadow map stays sharp. Only `Light::sun` (or
`Light::with_shadow`) casts shadows; point and spot lights shine through objects.
- Use `Reflector::floor()` with `Material::reflection(..)` and a roughness map for polished
floors, one reflector per scene.
- Set `.shadows(false, ..)` on floors, walls and far scenery, so the shadow map stays sharp:
by default it covers the box of all shadow casters. When the casters are far apart, set
the box that must receive sharp shadows:
`Light::directional(color, intensity, from).with_shadow(Shadow { bounds: Some(Aabb::from_points([min, max])), ..Default::default() })`.
Only a directional light casts shadows (`Light::sun`, or `Light::directional` with
`.with_shadow(..)`); point and spot lights shine through objects.
- Use `Reflector::floor()` with `Material::reflection(..)` for polished floors, one
reflector per scene. For a floor with rough and polished areas set the field
`material.metal_rough_map = Some(Arc::new(texture))` (the glTF texture: green multiplies
the roughness, blue the metalness; there is no builder method for it).
- Give small glowing particles a few pixels of size, or keep them in focus: out of focus, a
very bright single-pixel dot turns into a pattern of dots in the depth of field.
- Look at the result: `strip` and `frame` show the 3D as pixels; `inspect` cannot see inside
Expand Down Expand Up @@ -171,7 +184,7 @@ In `render_frame`: `let image = self.shot.image(t, frame.animate(&self.open));`
| `Scene`, `Group`, `Mesh`, `InstancedMesh`, `Points` | `Scene`, `Node::group`, `Node::mesh`, `Node::instanced`, `Node::particles` |
| Box, Sphere, Cylinder, Cone, Capsule, Torus, Plane, Circle, Ring, Lathe, Extrude, Tube, Icosahedron, Text, RoundedBox geometry | `Geometry::cuboid`, `sphere`, `cylinder`, `cone`, `capsule`, `torus`, `plane`, `floor`, `circle`, `ring`, `lathe`, `extrude`, `tube`, `icosahedron`, `text`, `rounded_box`; `merge`, `translate`, `rotate_*`, `scale`, `smooth`, `flat` |
| `MeshStandardMaterial`, clearcoat, `MeshBasicMaterial`, `ShadowMaterial`, `MeshNormalMaterial` | `Material::standard`, `metal`, `.clearcoat`, `unlit`, `glow` (additive), `shadow_catcher`, `normals` |
| `map`, `normalMap`, `aoMap`, `emissiveMap`, metal-roughness map | `.map`, `.normal_map`, `.ao_map`, `.emissive_map`, `metal_rough_map`; `Texture::decode`, `from_fn`, `from_image` |
| `map`, `normalMap`, `aoMap`, `emissiveMap`, metal-roughness map | `.map`, `.normal_map`, `.ao_map`, `.emissive_map`, the field `metal_rough_map`; `Texture::decode`, `from_fn`, `from_image` |
| `AmbientLight`, `HemisphereLight`, `DirectionalLight` + shadow, `PointLight`, `SpotLight` | `Light::ambient`, `hemisphere`, `directional`, `sun`, `point`, `spot` |
| `RoomEnvironment`, `RGBELoader`, `Fog`, `FogExp2`, `Reflector` | `Environment::studio`, `Environment::map`, `Fog::Linear`, `Fog::Exp2`, `Reflector::floor` |
| `GLTFLoader` + `AnimationMixer`, `STLLoader` | `Model::from_gltf`, `ModelInstance::play`, `Geometry::from_stl` |
Expand Down
27 changes: 18 additions & 9 deletions skills/fframes-video/references/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,10 @@ shaders).

## svgr!

SVG markup with Rust in braces. Attributes and text children take strings, numbers, `Color`,
`Transform`, another `Svgr`, or a `Vec<Svgr>` / iterator collected to a `Vec`.
SVG markup with Rust in braces. An attribute takes a string, a number, a `Color`, a
`Transform` or an image `href()`. A child takes a `String` or `&str`, another `Svgr`, an
`Option<Svgr>` or a `Vec<Svgr>` (an iterator collected to a `Vec`). A number as a child does
not compile: write `{count.to_string()}`.

```rust
fframes::svgr!(
Expand All @@ -32,8 +34,9 @@ fframes::svgr!(
decoration static, wrap animated values around it.
- Gradients, clip paths, masks, filters go into `<defs>` like normal SVG and are referenced
with `url(#id)`. Give ids unique names per component instance if repeated.
- Images: `href={self.media.logo_png.href()}` (field named after the file in `media/`) or
`ctx.get_image("logo.png").map(|i| i.href())`.
- Images: `href={self.media.logo_png.href()}` (field named after the file in `media/`), or
`let Some(logo) = ctx.get_image("logo.png") else { return Svgr::empty() };` and
`href={logo.href()}`. An attribute does not take an `Option`.

## Video and Scene

Expand Down Expand Up @@ -123,8 +126,9 @@ fframes::svgr!(
caps the natural settle time; omit the end in `timeline!` to let the spring settle.
- For precomputed geometry, motion-blur subframes or a custom clock,
`runtime.solve(&elapsed_seconds)` evaluates the same easing
directly. Handle times before the start and after `get_duration()` explicitly: `solve`
does not clamp them. Use a custom curve only when the required motion is not represented
directly. Handle times before the start and after `get_duration()` explicitly: `Linear`,
`Spring` and a diagonal `CubicBezier` (`x1 == y1` and `x2 == y2`) do not clamp them; the
other bezier easings do. Use a custom curve only when the required motion is not represented
by the available easings.
- Easing: `Linear`, `EaseIn`, `EaseOut`, `EaseInOut`, `CubicBezier(x1, y1, x2, y2)`, `Spring {..}`.
Use the named ones and `Spring`; `design.md` says which fits where.
Expand Down Expand Up @@ -178,7 +182,10 @@ family, size and weight in the `<text>` attributes. `text-anchor="middle"` cente
and combine providers with `CombinedMediaProvider::from([&static_media as &dyn MediaProvider, &media])`.
- Video frames: `frame.get_synced_video_frame(ctx, "clip.mp4", &SyncVideoFrameInput { start_from, looping, speed, editor_fallback_image })`
(`speed: 2.0` plays twice as fast, `..Default::default()` keeps 1.0)
then `<image href={video_frame.href()} .../>` (needs the `compile-time-svgtree` feature).
returns an `Option`: `let Some(video_frame) = .. else { return Svgr::empty() };` then
`let image = video_frame.into_image();` and `<image href={image.href()} .../>` (needs the
`compile-time-svgtree` feature). The video frame itself has no `href()`, and its methods
need `use fframes::FFramesSyncedVideoFrame;`.
`start_from` is when the video starts on the timeline, not a position in the file.
- Footage: `Clip::new("IMG_0926.mov", 19.2..24.7)` is seconds 19.2 to 24.7 of the file,
`Clip::new("cam.mp4", 0.0..)` the whole file (`.at(1.2)` starts it at 1.2 s of the scene, `.speed(1.5)`, `.reverse()`, `.looping()`).
Expand Down Expand Up @@ -301,7 +308,8 @@ let problems = previewer.inspect(frame)?.diagnostics;
Skia: `SkiaFFramesRenderer::new_metal(&SkiaMetalCtx::new(W, H)?, SkiaPipelineConfig::default())`
(`fframes_skia_renderer` feature `metal`) or `new_vulkan` with `SkiaVulkanCtx` (feature
`vulkan`). `backend.frame_renderer()`
returns a renderer for single frames that matches the backend (Skia on the GPU for Skia).
returns an `Option` of a renderer for single frames that matches the backend (Skia on the
GPU for Skia); `None` means the built-in CPU renderer.

The Skia backend picks the fastest way to hand frames to the encoder on its own: hardware
frames (`*_videotoolbox` encoders with `new_metal`; `h264_vulkan`/`hevc_vulkan` with
Expand Down Expand Up @@ -336,4 +344,5 @@ fframes::cli::new(&video, options)
- Heavy filters (`feGaussianBlur`, `feDropShadow`) on animated subtrees are slow on the CPU
backend: keep them static or use Skia.
- Durations from audio (`Duration::FromAudio`, `Eof`) need the file in the media provider;
missing files fail early with `RequiredAudioNotFound`.
a missing file fails early with `CanNotProcessAudioDuration` (`RequiredAudioNotFound` when
`RenderOptions::media` is `None`).
33 changes: 21 additions & 12 deletions skills/fframes-video/references/captions.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,17 @@ per-word settings:

Loading is the same as for any other media: `include_media_dir!` embeds the file,
`MediaDirectory::read_folder` reads it at runtime, the editor loads it over HTTP. JSON
whose top-level object has `cues` is a caption track; any other JSON is not.
`ctx.get_subtitles("speech.json")` returns the `&CaptionTrack`: `cues()`,
`active_cues(seconds)`, and on each `CaptionCue` `phrase()`, `words`, `metadata`.
A track can also be built in code: `CaptionTrack::new([CaptionCue::words(0.0, 2.0,
[CaptionWord::new("Choose", 0.0, 0.7), ..]), CaptionCue::text(2.4, 4.0, "..")])`.
whose top-level object has `cues` is a caption track. A file named `subtitles.json`,
`captions.json`, `*.subtitles.json` or `*.captions.json` is always a caption track and fails
to load without `cues`. Any other JSON is a Lottie animation with the `lottie` feature;
without it `include_media_dir!` fails to compile and `MediaDirectory` skips the file. The
top-level object, cues and words accept only the keys shown here: an unknown key (a
`language` next to `cues`) fails the load. Put your own data into `metadata`.
`ctx.get_subtitles("speech.json")` returns an `Option<&CaptionTrack>` (`None` when the
file is missing): `cues()`, `active_cues(seconds)`, and on each `CaptionCue` `phrase()`,
`words`, `metadata`. A track can also be built in code: `CaptionTrack::new([CaptionCue::words(0.0, 2.0,
[CaptionWord::new("Choose", 0.0, 0.7), ..]), CaptionCue::text(2.4, 4.0, "..")])` returns a
`Result`, because it validates the timing like the parser.

For a plain phrase caption `frame.get_subtitle_phrase(track)` returns the current cue
text; this older helper includes the cue's end timestamp, `Captions` and `active_cues`
Expand All @@ -72,7 +78,8 @@ let captions = Captions::from_media("speech.json", CaptionRect::new(120.0, 760.0
.underline("#ffd166", 5.0); // bar under the active word
```

`Captions::new(track, rect)` takes an owned `CaptionTrack` instead of a media name.
`Captions::new(track, rect)` takes an owned `CaptionTrack` (not the `Result` of
`CaptionTrack::new` or `parse`) instead of a media name.
The rectangle is in composition pixels. Each phrase is wrapped to its width, lines are
centred, and the block is anchored to the bottom edge of the rectangle; simultaneous cues
are stacked in order. Nothing moves when the active word changes.
Expand All @@ -89,7 +96,8 @@ Modes:
(`upcoming_fill`, `active_fill`, `spoken_fill`).
- `Karaoke`: like Highlight, the active word fills left to right with its progress.
- `Reveal`: words appear as they are spoken and stay; upcoming words are hidden.
- `Word`: only the active word, centred in the rectangle; nothing during gaps.
- `Word`: only the active word, centred horizontally on the bottom edge of the rectangle
like the other modes; nothing during gaps.

A VTT phrase has no word timing: every mode shows the whole phrase in `spoken_fill`.

Expand Down Expand Up @@ -133,8 +141,9 @@ What the scope offers:

- `scope.word`: `text`, `state` (`Upcoming | Active | Spoken`), `progress` (0..=1 of the
word's duration), `start`/`end` (`None` for a VTT phrase), `metadata`, `cue_index`,
`word_index`. `scope.cue_metadata` is the cue's object. `scope.is_active()`.
- `scope.time` (caption seconds), `scope.elapsed()` (seconds since the word started),
`token_index`, `word_index` (`None` for a VTT phrase). `scope.cue_metadata` is the cue's object. `scope.is_active()`.
- `scope.time` (caption seconds), `scope.elapsed()` (`Option<f64>`: seconds since the word started,
`None` for a VTT phrase),
`scope.frame` (the `Frame`, for `frame.animate` or `Shader::draw`).
- `scope.composition_rect`: where the word lands on the canvas, if something else must
align with it.
Expand All @@ -144,9 +153,9 @@ What the scope offers:
- `render_at_with` and `highlight_at_with` take explicit seconds; every variant has a
`try_` form that returns the error.

For a layout of your own, `captions.words_at(seconds, &mut frame, ctx)` returns the
positioned words (`rect` in composition pixels, state, progress, metadata) and you draw
them yourself.
For a layout of your own, `captions.words_at(seconds, &mut frame, ctx)` returns a
`Result` with the positioned words (`rect` in composition pixels, state, progress,
metadata) and you draw them yourself. It does not report its errors to `inspect`.

## Metadata

Expand Down
2 changes: 1 addition & 1 deletion skills/fframes-video/references/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ problems already looks finished.
`+2` to `+6` for small caps labels.
- Max ~8 words per line, ~3 lines per card. Measure with `frame.text_width` or wrap with
`frame.text_break_lines`; shrink to fit with a loop over sizes and memoize the result
in a `OnceLock`. Generated projects include a `FittedSizes` helper that does this.
in a `OnceLock`. The `greeting_size` method of a generated single-scene project does this.

## Color

Expand Down
Loading