Skip to content

Add FSRCNNX-8 neural upscale option via libplacebo custom_shader_path #37

Description

@prekabreki

Context

The current "upscale" path is pure interpolation: libplacebo=…:upscaler=ewa_lanczos plus
cas=0.5 (fetchforge/server.py:590-604). Measured on a real 720p→1080p job
(FFX-2 walkthrough, 38 min), cas=0.5 adds only +14–16% high-frequency energy over
the bare scaler — visually near-invisible, which is what prompted this.

libplacebo in this ffmpeg exposes custom_shader_path (mpv .hook format), so an
actual learned upscaler can be dropped in for free. Measured with FSRCNNX
(igv/FSRCNN-TensorFlow, LGPL-3.0) on three frames of that same file, identical NVENC
settings (p4/uhq/cq24/maxrate 3M):

chain HF gain vs bare scaler 30 s clip full 38-min file
ewa_lanczos only baseline
+ cas=0.5 (current) +14 to +16% 3.60× realtime 11 min
FSRCNNX-8 +40 to +48% 3.68× realtime 10 min
FSRCNNX-16 +38 to +49% 3.42× realtime 11 min
FSRCNNX-8 + cas=0.5 +55 to +72%

So ~3× the sharpening benefit at no measurable cost — the pipeline is bound by
decode/encode and the CUDA→RAM→Vulkan→RAM→NVENC round trip, not shader math.
Use the _8-0-4-1 variant: statistically identical to _16-0-4-1, 71 KB vs 247 KB,
marginally faster.

Acceptance criteria

  • fetchforge/shaders/FSRCNNX_x2_8-0-4-1.glsl is committed, and pyproject.toml
    [tool.setuptools.package-data] includes it so pip install fetchforge ships it.
  • _decode_filter_args(...) accepts a new neural: bool = False keyword and, when
    neural=True and HAS_LIBPLACEBO and target_height > 0, appends
    :custom_shader_path=<abs path under PKG_DIR> to the existing libplacebo=… filter.
  • neural=True with HAS_LIBPLACEBO=False (or target_height=0) changes nothing —
    falls back to the current chain, no crash, no bogus arg on scale_cuda/CPU branches.
  • build_video_ffmpeg_args(...) forwards neural through to _decode_filter_args.
  • /download and /convert-local both accept a neural: str = Form("false") field,
    parsed exactly like the existing sharpen field (_neural = neural == "true"), and
    pass it down.
  • Card 04 gets a Neural upscale toggle next to the existing Sharpen toggle; both
    startDownload/local-convert opts objects send form.append('neural', opts.neural).
  • The toggle is disabled/greyed with a hint when the selected target resolution is
    less than 1.3× the source height (see danger zone Move import-time side effects (mkdir, logging) into run_server() #2) — the shader silently
    no-ops there, and a silent no-op is worse than a disabled control.
  • Unit tests in tests/test_server.py (alongside the existing test_upscale_libplacebo
    family) assert: custom_shader_path present when neural=True + libplacebo +
    upscale; absent when neural=False; absent on the scale_cuda and CPU branches;
    and that the referenced shader file actually exists on disk.
  • Existing quality floor (CQ≤24, maxrate≥3M, preset≥p4 — issue Quality floor for sharpen/upscale: never below CQ24 / 3M / p4 / hq / yuv420p #29) still applies
    unchanged when neural=True.

Files / where this lives

  • fetchforge/server.py:577-642_decode_filter_args, the libplacebo branch at 593-604
  • fetchforge/server.py:900-920build_video_ffmpeg_args
  • fetchforge/server.py:1514-1515, 2345-2346 — the target_res / sharpen form fields
    on /download and /convert-local; mirror them for neural
  • fetchforge/server.py:1585-1586_sharpen = sharpen == "true" parse site
  • fetchforge/server.py:1959-1969, 2223-2233 — the four call sites that thread
    target_res= / sharpen= into calc_encode_params and build_video_ffmpeg_args
  • fetchforge/index.html:1290-1297 — the Sharpen toggle markup to mirror
  • fetchforge/index.html:2245-2246, 2264-2265, 2347-2348form.append sites
  • fetchforge/index.html:2432-2433 — where targetRes/sharpen are read from the DOM
  • pyproject.toml:32-33package-data
  • tests/test_server.py:313-345 — the libplacebo scaling test block

Shader source (download, do not hand-transcribe):
https://github.com/igv/FSRCNN-TensorFlow/releases/download/1.1/FSRCNNX_x2_8-0-4-1.glsl
(sha: 71296 bytes, 426 lines). Keep the LGPL-3.0 header comment intact and add a line
crediting igv/FSRCNN-TensorFlow to the README or a fetchforge/shaders/LICENSE note.

How to verify

  • .venv/bin/python -m unittest discover -s tests -v — full suite green
  • python3 -m py_compile fetchforge/server.py
  • Argv shape, no GPU needed:
    .venv/bin/python -c "from pathlib import Path; from fetchforge import server; server.HAS_LIBPLACEBO=True; server.HAS_SCALE_CUDA=False; print(server._decode_filter_args(Path('in.mkv'),'yuv420p',None,target_height=1080,sharpen=True,neural=True))"
    → must contain custom_shader_path= and the existing upscaler=ewa_lanczos and cas=0.5
  • Shader ships: pip install -e . && .venv/bin/python -c "from fetchforge import server; import pathlib; p=server.PKG_DIR/'shaders'/'FSRCNNX_x2_8-0-4-1.glsl'; print(p, p.exists(), p.stat().st_size)"

Constraints

  • Preserve: neural=False must produce byte-identical argv to today for every branch —
    the existing test_passthrough_* / test_downscale_* / test_upscale_* tests must pass
    untouched.
  • Preserve: the sharpen/cas path stays. neural and sharpen are independent
    toggles and must compose (FSRCNNX + cas=0.5 is a valid, measured-good combination).
  • Do not touch: calc_encode_params bitrate/CQ math, the quality floor from Quality floor for sharpen/upscale: never below CQ24 / 3M / p4 / hq / yuv420p #29,
    or the scale_cuda / CPU fallback filter strings.
  • Match: the existing sharpen plumbing exactly — same param position, same
    Form("false") + == "true" parse idiom, same toggle markup/CSS classes.
  • Resolve the shader path from PKG_DIR, never STATE_DIR/cwd — it's read-only
    installed package data.

⚠ Danger zones (review focus)

  1. The shader fails silently, and that is the whole risk here. FSRCNNX_x2_8-0-4-1.glsl
    is //!HOOK LUMA, so it only fires when libplacebo's input has a luma plane. Feed it
    RGB and it is a no-op with no warning and no error — I hit exactly this while
    benchmarking (RGB PNG input produced byte-identical output to the plain scaler; the
    real yuv420p video path worked). A unit test asserting the argv string is therefore
    necessary but not sufficient: it proves the flag is passed, not that it did anything.
    ⚠ Review: do not accept "tests pass" as evidence the feature works. Real verification
    needs a GPU run — see the follow-up note below.
  2. Scale-factor guard inside the shader:
    //!WHEN OUTPUT.w LUMA.w / 1.300 > OUTPUT.h LUMA.h / 1.300 > * — it only activates
    above 1.3× in both axes. 720p→1080p (1.5×) fires; 1080p→1440p (1.33×) barely fires;
    720p→900p (1.25×) silently does not. This is why the UI must gate the toggle on the
    source/target ratio rather than letting the user enable a no-op.
  3. Chroma is not shader-upscaled — FSRCNNX is luma-only, chroma still rides
    ewa_lanczos. That's standard and correct; don't "fix" it by adding a chroma shader.
  4. Build integrity: a missing package-data entry means the shader is absent from a
    real pip install while working fine from a source clone. The pip install -e .
    check above does not catch that — verify with a built wheel or pip install .
    into a scratch venv.

Out of scope

  • Any other shader family (Anime4K, ravu, FSRCNNX-16) or a shader-picker UI. Ship the
    one measured-best default; more options can follow if wanted.
  • Changing the default CAS_STRENGTH (0.5). Separate question — the strength ladder
    showed 0.7–0.8 is the sweet spot and cas=1.0 blows up into chroma speckle, but
    that's its own issue.
  • Making neural the default. Ship it opt-in.
  • Downscale-time use of the shader (it's an upscaler; the //!WHEN guard blocks it anyway).

Notes

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestscopedIssue drafted and specced

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions