Migrate image rendering to vim.ui.img (staged plan)

[delphinus]

Upstream tracking: neovim/neovim#30889

vim.ui.img (introduced in Neovim 0.13 as EXPERIMENTAL) provides a minimal, official API for displaying images via the Kitty graphics protocol. This plugin currently ships its own ~1700-line lua/md-render/image.lua covering image transmission, cropping, animation, conversion, URL caching, Mermaid rendering, and video frame extraction. Most of that is out of scope for vim.ui.img today and likely will remain so for some time, so a wholesale switch is not feasible. This issue tracks a staged migration.

Status

Phase	Scope	Status
1. I/O & capability layer	Replace /dev/tty writes with vim.api.nvim_ui_send; opportunistically use vim.tty.query_apc for capability detection	Done on main (commit 146d89b, will ship in v3.0.0). Bumps minimum Neovim to 0.12.
2. Route plain PNG cases via vim.ui.img.set	When the call has no crop, no animation, no Ghostty workaround, no path transmission needed: delegate to vim.ui.img.set(bytes, opts). Two-path coexistence with the rich path.	Pending. Recommended only after vim.ui.img loses the EXPERIMENTAL marker, since the API may shift.
3. Become a vim.ui.img backend	Override vim.ui.img.set/get/del with this plugin's implementation so other plugins (snacks.image, image.nvim) can share one image surface	Pending. Requires vim.ui.img.Opts to grow first (see gaps below).
4. Retire most of image.lua	Once upstream supports t=f (filepath transmission) and source-rectangle cropping, much of the current code can be deleted	Upstream-dependent. _kitty.lua already notes "A future filepath option (t=f) could let the terminal read the file directly."

What vim.ui.img does not provide today

These are the features that keep us on our own code path. Each is also a candidate for an upstream PR.

Feature	Used by md-render for	Workaround
Path transmission (t=f / t=t)	Avoid base64-encoding the entire PNG over the wire	Currently base64 the path, not the bytes. vim.ui.img only does t=d direct base64 in 4 KB chunks.
Source-rectangle cropping (x, y, w, h)	Clipping images at window edges (scroll, leftcol, winbar, border, textoff)	Custom put_image math in image.lua.
Animated GIF support	Frame extraction + a=T per placement	transmit_animated* in image.lua, ffmpeg/magick frame extraction.
Placement-only delete (d=a)	Re-rendering during scroll without retransmitting	clear_placements in image.lua. vim.ui.img.del removes the whole image.
Ghostty a=T workaround	Ghostty does not reliably re-place via a=p after a=t	Detect Ghostty and re-transmit each placement.
JPEG / WebP / GIF → PNG conversion	Non-PNG image inputs	ensure_png (sips → ffmpeg → magick fallback chain).
URL download with cache	Web image links	download_async + curl + on-disk cache.
Mermaid rendering	Inline diagrams	render_mermaid_async via mmdc or npx.
Video frame extraction	Inline video playback	transmit_animated* + ffmpeg.
Synchronized terminal update wrapper	Reduce flicker on bulk changes	begin_sync_update / end_sync_update (DEC 2026).
Batched writes	Coalesce many small APC sequences into one TTY write	begin_batch / flush_batch. With nvim_ui_send this matters less but is still useful.

Phase 2 design sketch

function display_image(path, opts)
  if opts.has_crop or opts.is_animated or opts.is_ghostty
     or not is_plain_png(path) then
    -- existing rich path: transmit_image_async + put_image
    return legacy_display(path, opts)
  end
  -- delegate to vim.ui.img for parity with the standard surface
  local bytes = vim.fn.readblob(path)
  return vim.ui.img.set(bytes, {
    row = opts.row, col = opts.col,
    width = opts.cols, height = opts.rows,
    zindex = opts.zindex,
  })
end

Trade-offs:

• Pro: progressively shrinks our code surface; easier to swap in future Neovim improvements; signals API parity.
• Con: large images get slower (full base64 over nvim_ui_send vs. base64 of a path). Until t=f lands upstream, the win is mostly symbolic.
• Decision: hold until vim.ui.img is no longer EXPERIMENTAL or until upstream adds t=f. Whichever comes first.

Phase 3 design sketch

vim.ui.img's docstring explicitly invites overrides: "To override the image backend, replace vim.ui.img with your own implementation providing set/get/del."

If md-render's renderer became the default backend, other Neovim plugins (snacks.image, image.nvim, future built-in features) would share one surface for cropping, animation, conversion, URL caching, etc. But the contract has to grow first:

• vim.ui.img.Opts needs at minimum a window/buffer association and source-rectangle fields.
• A delete-placements-only operation (separate from delete-image) needs to exist.
• An animation/multi-frame model needs a story.

This is best driven via PR / discussion against neovim/neovim rather than by md-render quietly adopting the override hook (which would diverge if upstream's shape changes).

Phase 4 (upstream-dependent)

Once vim.ui.img supports t=f and source-rectangle cropping, large parts of image.lua (the cropping math, the path-vs-bytes branching, possibly the Ghostty workaround if upstream handles it) can be deleted outright.

Open questions

1. When does Neovim drop the EXPERIMENTAL marker on vim.ui.img? Phase 2 should wait for that.
2. Is there appetite upstream for t=f and source-rectangle cropping in vim.ui.img.Opts? Worth opening a discussion before Phase 3.
3. Phase 2 adds two code paths (legacy + vim.ui.img). If Phase 4 is close, it may be worth skipping Phase 2 entirely and going straight from Phase 1 to Phase 4.

Decision log

• 2026-05-12: agreed to ship Phase 1 only for now. Phase 1 is bundled into the v3.0.0 release alongside Migrate to subcommand-based command structure (:MdRender <sub>) #11 (the subcommand consolidation), since both are breaking changes — Phase 1 raises the minimum Neovim version, Migrate to subcommand-based command structure (:MdRender <sub>) #11 reshapes the command surface. See discussion in the conversation that produced commit 146d89b.